From XXX Sat Feb 05 21:14:34 2000 Forwarded: Sat, 05 Feb 2000 16:14:34 -0500 Forwarded: "greg, garth " Return-Path: cheriton@cisco.com Delivery-Date: Fri Feb 4 02:12:06 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id CAA03588 for ; Fri, 4 Feb 2000 02:11:52 -0500 Received: from ruby.cisco.com (ruby.cisco.com [171.69.198.43]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA13536 for ; Fri, 4 Feb 2000 02:11:51 -0500 (EST) Received: from alfonso. ([10.19.129.228]) by ruby.cisco.com (8.8.8-Cisco List Logging/8.8.8) with SMTP id XAA20628; Thu, 3 Feb 2000 23:11:11 -0800 (PST) Message-Id: <3.0.3.32.20000203230605.00e5f5dc@ruby.cisco.com> X-Sender: cheriton@ruby.cisco.com X-Mailer: QUALCOMM Windows Eudora Pro Version 3.0.3 (32) Date: Thu, 03 Feb 2000 23:06:05 -0800 To: Dave_Lee@3com.com, Andreas Bechtolsheim From: "David R. Cheriton" Subject: Re: IP Storage Framework document Cc: mark_bradley@btc.adaptec.com, Carl_Madison@3com.com, Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, paulv@corp.adaptec.com, dwilson@corp.adaptec.com, raz@emc.com, avb@cisco.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, sob@harvard.edu, vern@aciri.org, csapuntz@cisco.com In-Reply-To: <8825687B.0004CCFA.00@hqoutbound.ops.3com.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" I actually dont think there is any other transport protocol to consider other than TCP. Here's why. 1) TCP has been developed, refined, evaluated and deployed over a period of 20 years, with some of the smartest folks in the Internet research community spending years of effort. There is NO hope in duplicating this effort, IMHO, and plenty of evidence to suggest you'll screw up or end up with TCP with gratutious differences if you dont. 2) TCP has been the most successful transport protocol in the history of mankind, and continues to grow dramatically in use. There are numerous other attempts now and in the past to create a new transport protocol for one reason or another, but the success level has been almost 0 i.e. they're not all dead YET. (I was even responsible for one old attempt, blush!) 3) The TCP fast path is reasonably simple, has been identified in software implementations and used successfully for years. A hardware implementation of that fast path seems feasible to me, and, even if a little more difficult than what one might imagine for some new fangled protocol, the pay-off for a fast hardware TCP is certain, and huge - not so with any other protocol. I think NFS/UDP is problematic. Even in a campus network, you can have systematic loss of IP fragments that UDP too often generates, causing communication failures and network overload. And, handling IP fragment reassembly is one of the more complex aspects of hardware IP. With a hardware TCP, I think one could rely on MTU discovery and punt IP reassembly to software as an unusual and unwarranted case. Finally, my perception is that many netadmins are trying to reduce and restrict non-TCP traffic on their networks because it does not adapt predictably to network loads, causing failures. A high-rate UDP or random new protocol load on a network would not be welcomed. I'd also mention that the same forces that are causing networks to converge on IP are pushing for convergence on TCP as well. Diversity is not nice in a mission-critical 7x24 enterprise network that is struggling with rapid scaling in users, bandwidth, services. So, the only thing that would convince me to consider another transport protocol would be some major flaw in the above reasoning, or proof that you just cant do hardware TCP at the required rates. The latter would really, really surprise me. What am I missing? At 04:54 PM 2/3/00 -0800, Dave_Lee@3com.com wrote: > > > >Andy - > >You point out excellent issues that must be considered. Let me provide some >additional comments for all to ponder. Before I start, I believe the document >authors are using the term IP in its most generic sense and not meant to imply a >particular transport such as TCP, UDP, or some other transport. Drew/Paul -- >feel free to correct me if I'm wrong. There are certainly many possibilities >and I think they all need to be considered at this point. > >While it may be theoretically possible to implement TCP in silicon, it is not at >all clear if TCP in silicon will occur anytime soon -- if at all. Betting on >that fact may not sit well with others, especially those concerned with higher >performance campus-based solutions that they want available now. Nothing, >however, precludes us from taking advantage of TCP in silicon in a future >revision of the protocol(s). > >For WAN traffic, I think most people would agree with you on your remark on the >need for bandwidth management. Further, I believe most people would probably >agree with you that if we were looking for a pure L2 solution (e.g., directly >attached LAN), the IETF is *not* the right place to take this. I definitely do. > >However, a customer may want to have a storage solution that can be accessed >over a campus network. In this case, a L2 solution is not sufficient and IP is >required. Further, it does not seem to be at all clear if TCP bandwidth >management is a requirement in this environment. The NFS version 4 requirements >document has some commentary on this discontinuity. In a campus network, there >are other possible ways to do bandwidth management, including traffic >engineering, differential services, etc. > >In light of the NFS discussion, perhaps looking to NFS may not be a bad thing to >do? NFS can operate over both TCP and UDP. Perhaps this model is also a good >model for IP storage. > >Dave > > > > >Andreas Bechtolsheim on 02/03/2000 09:14:54 AM > >Sent by: Andreas Bechtolsheim > > >To: See Below >cc: See Below >Subject: Re: IP Storage Framework document > > > >While I will not be able to attend this meeting in person, I would >like to propose an alternative framework for Storage over IP. >We will discuss this in more detail at the 2/16 Meeting in San Jose. > >The key characteristics of our proposal are as follows: > >1. It is based on TCP/IP, so all the issues of a reliable > end-to-end byte stream including error recovery and > congestion management are addressed without inventing > a new protocol to do the same functions. > >2. It works over any network topology and any distance, > addressing the needs of both local and remote connectivity > between servers and storage devices. > >3. The protocol can be implemented efficiently in hardware > inside network interface controllers allowing for > high-performance wirespeed network operation. > >While in the past several new protocols have been proposed to >accelerate certain types of data transfers, typically with >the argument that they are easier to implement in hardware, >experience has shown that such protocols have failed to gain >market acceptance. With the plummenting cost of silicon, >at this point in history the cost of implementing TCP/IP in >NIC silicon is in the order of a few $ worth of silicon >so there is simply no economic reason not to do it. > >In addition, proposing protocols that do not adhere to the >TCP/IP bandwidth management algorithms pose a serious risk >for overall network stability, in particular if they are used >for high bandwidth traffic such as disk I/O. > >Finally, designing new protocols for limited topologies such >as local area networks does not appear consistent with the >goals of the IETF. > > > >To: Dave Lee/Hq/3Com Raz @Emc.Com > Mark_Bradley @Btc.Adaptec.Com Avb @Cisco.Com > Carl Madison/Us/3Com Rcampbel @Cisco.Com > Arun Verma/Hq/3Com Jgw @Cisco.Com > Kquick @Iphase.Com Ddecapit @Cisco.Com > Bassoon @Ece.Cmu.Edu Sob @Harvard.Edu > Paulv @Corp.Adaptec.Com Vern @Aciri.Org > Dwilson @Corp.Adaptec.Com > >cc: Avb @Cisco.Com Csapuntz @Cisco.Com > Cheriton @Cisco.Com > > From XXX Sat Feb 05 21:14:38 2000 Forwarded: Sat, 05 Feb 2000 16:14:38 -0500 Forwarded: "greg, garth " Return-Path: mark_bradley@btc.adaptec.com Delivery-Date: Fri Feb 4 07:48:43 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id HAA29005 for ; Fri, 4 Feb 2000 07:48:42 -0500 Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA19539 for ; Fri, 4 Feb 2000 07:48:41 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.9.2/8.8.8) with ESMTP id EAA12829; Fri, 4 Feb 2000 04:45:42 -0800 (PST) Received: from btcexc01.btc.adaptec.com (btcexc01.btc.adaptec.com [162.62.147.10]) by kingdom.adaptec.com (8.9.2/8.8.8) with ESMTP id EAA23529; Fri, 4 Feb 2000 04:45:39 -0800 (PST) Received: by btcexc01.btc.adaptec.com with Internet Mail Service (5.5.2650.21) id <1BGA1C4W>; Fri, 4 Feb 2000 05:46:59 -0700 Message-ID: <3CFAB2A312A9D311BB300000D11ABE920D2FD9@btcexc01.btc.adaptec.com> From: "Bradley, Mark" To: "'David R. Cheriton'" , Dave_Lee@3com.com, Andreas Bechtolsheim Cc: "Bradley, Mark" , Carl_Madison@3com.com, Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, "VonStamwitz, Paul" , "Wilson, Andrew" , raz@emc.com, avb@cisco.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, sob@harvard.edu, vern@aciri.org, csapuntz@cisco.com Subject: RE: IP Storage Framework document Date: Fri, 4 Feb 2000 05:46:51 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Hi, David. Thank you for your thoughts. In answer to your question, I think that: 1)TCP in h/w is probably possible, but as others have found, it is very difficult to do. A simpler protocol in hardware is highly desirable 2)Cost. TCP h/w implementations are likely to be complex silicon designs that one could expect are costly to provide. 3)While it is critical that IPS be cohabitable and non-destructive to network performance, it is entirely possible that a separate physical layer would be used for performance reasons. I can envision many scenarios where we do this today, sans IPS. 4)Performance. Both in CPU utilization and for pure bandwidth, it is desirable to go light to minimize impact for those who might not even implement in H/W. Lastly, one envisions large networks in considering your points below. For small networks (SOHO, small business), your points are likely less important for many reasons, including network loading. Small networks with generall light traffic are less sensitive to the issues you bring up. They also tend to be more cost sensitive. These are the points that it I think you may not be sensitive to. -- markb -----Original Message----- From: David R. Cheriton [mailto:cheriton@cisco.com] Sent: Friday, February 04, 2000 12:06 AM To: Dave_Lee@3com.com; Andreas Bechtolsheim Cc: mark_bradley@btc.Adaptec.Com; Carl_Madison@3com.com; Arun_Verma@3com.com; kquick@iphase.com; bassoon@ece.cmu.edu; paulv@corp.adaptec.com; dwilson@corp.adaptec.com; raz@emc.com; avb@cisco.com; rcampbel@cisco.com; Jgw@cisco.com; ddecapit@cisco.com; sob@harvard.edu; vern@aciri.org; csapuntz@cisco.com Subject: Re: IP Storage Framework document I actually dont think there is any other transport protocol to consider other than TCP. Here's why. 1) TCP has been developed, refined, evaluated and deployed over a period of 20 years, with some of the smartest folks in the Internet research community spending years of effort. There is NO hope in duplicating this effort, IMHO, and plenty of evidence to suggest you'll screw up or end up with TCP with gratutious differences if you dont. 2) TCP has been the most successful transport protocol in the history of mankind, and continues to grow dramatically in use. There are numerous other attempts now and in the past to create a new transport protocol for one reason or another, but the success level has been almost 0 i.e. they're not all dead YET. (I was even responsible for one old attempt, blush!) 3) The TCP fast path is reasonably simple, has been identified in software implementations and used successfully for years. A hardware implementation of that fast path seems feasible to me, and, even if a little more difficult than what one might imagine for some new fangled protocol, the pay-off for a fast hardware TCP is certain, and huge - not so with any other protocol. I think NFS/UDP is problematic. Even in a campus network, you can have systematic loss of IP fragments that UDP too often generates, causing communication failures and network overload. And, handling IP fragment reassembly is one of the more complex aspects of hardware IP. With a hardware TCP, I think one could rely on MTU discovery and punt IP reassembly to software as an unusual and unwarranted case. Finally, my perception is that many netadmins are trying to reduce and restrict non-TCP traffic on their networks because it does not adapt predictably to network loads, causing failures. A high-rate UDP or random new protocol load on a network would not be welcomed. I'd also mention that the same forces that are causing networks to converge on IP are pushing for convergence on TCP as well. Diversity is not nice in a mission-critical 7x24 enterprise network that is struggling with rapid scaling in users, bandwidth, services. So, the only thing that would convince me to consider another transport protocol would be some major flaw in the above reasoning, or proof that you just cant do hardware TCP at the required rates. The latter would really, really surprise me. What am I missing? At 04:54 PM 2/3/00 -0800, Dave_Lee@3com.com wrote: > > > >Andy - > >You point out excellent issues that must be considered. Let me provide some >additional comments for all to ponder. Before I start, I believe the document >authors are using the term IP in its most generic sense and not meant to imply a >particular transport such as TCP, UDP, or some other transport. Drew/Paul -- >feel free to correct me if I'm wrong. There are certainly many possibilities >and I think they all need to be considered at this point. > >While it may be theoretically possible to implement TCP in silicon, it is not at >all clear if TCP in silicon will occur anytime soon -- if at all. Betting on >that fact may not sit well with others, especially those concerned with higher >performance campus-based solutions that they want available now. Nothing, >however, precludes us from taking advantage of TCP in silicon in a future >revision of the protocol(s). > >For WAN traffic, I think most people would agree with you on your remark on the >need for bandwidth management. Further, I believe most people would probably >agree with you that if we were looking for a pure L2 solution (e.g., directly >attached LAN), the IETF is *not* the right place to take this. I definitely do. > >However, a customer may want to have a storage solution that can be accessed >over a campus network. In this case, a L2 solution is not sufficient and IP is >required. Further, it does not seem to be at all clear if TCP bandwidth >management is a requirement in this environment. The NFS version 4 requirements >document has some commentary on this discontinuity. In a campus network, there >are other possible ways to do bandwidth management, including traffic >engineering, differential services, etc. > >In light of the NFS discussion, perhaps looking to NFS may not be a bad thing to >do? NFS can operate over both TCP and UDP. Perhaps this model is also a good >model for IP storage. > >Dave > > > > >Andreas Bechtolsheim on 02/03/2000 09:14:54 AM > >Sent by: Andreas Bechtolsheim > > >To: See Below >cc: See Below >Subject: Re: IP Storage Framework document > > > >While I will not be able to attend this meeting in person, I would >like to propose an alternative framework for Storage over IP. >We will discuss this in more detail at the 2/16 Meeting in San Jose. > >The key characteristics of our proposal are as follows: > >1. It is based on TCP/IP, so all the issues of a reliable > end-to-end byte stream including error recovery and > congestion management are addressed without inventing > a new protocol to do the same functions. > >2. It works over any network topology and any distance, > addressing the needs of both local and remote connectivity > between servers and storage devices. > >3. The protocol can be implemented efficiently in hardware > inside network interface controllers allowing for > high-performance wirespeed network operation. > >While in the past several new protocols have been proposed to >accelerate certain types of data transfers, typically with >the argument that they are easier to implement in hardware, >experience has shown that such protocols have failed to gain >market acceptance. With the plummenting cost of silicon, >at this point in history the cost of implementing TCP/IP in >NIC silicon is in the order of a few $ worth of silicon >so there is simply no economic reason not to do it. > >In addition, proposing protocols that do not adhere to the >TCP/IP bandwidth management algorithms pose a serious risk >for overall network stability, in particular if they are used >for high bandwidth traffic such as disk I/O. > >Finally, designing new protocols for limited topologies such >as local area networks does not appear consistent with the >goals of the IETF. > > > >To: Dave Lee/Hq/3Com Raz @Emc.Com > Mark_Bradley @Btc.Adaptec.Com Avb @Cisco.Com > Carl Madison/Us/3Com Rcampbel @Cisco.Com > Arun Verma/Hq/3Com Jgw @Cisco.Com > Kquick @Iphase.Com Ddecapit @Cisco.Com > Bassoon @Ece.Cmu.Edu Sob @Harvard.Edu > Paulv @Corp.Adaptec.Com Vern @Aciri.Org > Dwilson @Corp.Adaptec.Com > >cc: Avb @Cisco.Com Csapuntz @Cisco.Com > Cheriton @Cisco.Com > > From XXX Sat Feb 05 21:14:42 2000 Forwarded: Sat, 05 Feb 2000 16:14:42 -0500 Forwarded: "greg,garth " Return-Path: avb@cisco.com Delivery-Date: Fri Feb 4 09:49:51 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id JAA09698 for ; Fri, 4 Feb 2000 09:49:47 -0500 Received: from postoffice.cisco.com (postoffice.cisco.com [171.69.198.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA22391 for ; Fri, 4 Feb 2000 09:49:46 -0500 (EST) Received: (avb@localhost) by postoffice.cisco.com (8.8.7-Cisco.2-SunOS.5.5.1.sun4/8.6.5) id GAA13994; Fri, 4 Feb 2000 06:48:19 -0800 (PST) Date: Fri, 4 Feb 2000 06:48:19 -0800 (PST) From: Andreas Bechtolsheim Message-Id: <200002041448.GAA13994@postoffice.cisco.com> To: cheriton@cisco.com, Dave_Lee@3com.com, avb@cisco.com, mark_bradley@btc.adaptec.com Subject: RE: IP Storage Framework document Cc: Carl_Madison@3com.com, Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, paulv@corp.adaptec.com, dwilson@corp.adaptec.com, raz@emc.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, sob@harvard.edu, vern@aciri.org, csapuntz@cisco.com X-Sun-Charset: US-ASCII Mark, I think this whole argument comes down to the question "is it easier to put TCP/IP into hardware or is it easier to invent a new IP protocol to replace TCP functionality because such a protocol is easier to put into hardware?" Along these lines and in response to your points below, I would like to offer the following observations: 1. It appears straightforward to implement the "standard" TCP/IP control and data path in hardware, leaving more complex options to software. Both the latest 3COM and Intel Gigabit Ethernet NICs have added hardware support for the TCP/IP transmit path. There are several startups building full TCP/IP hardware implementations, and people are busy adding hardware functions such as IPSEC security. Seems to me the time has arrived to take advantage of contemporary silicon technology for the network interface controller. 2. The estimated cost for a TCP/IP hardware stack is 200K gates. This is about 2 mm^2 in 0.15um CMOS or less than $1 worth of silicon. To believe that TCP/IP hardware is a cost issue is a fallacy. 3. While one can always build separate networks for storage, just like Fibrechannel is today, doing so of course removes the strongest motivation to converge storage on IP in the first place. 4. One can implement Storage over TCP/IP in software today with off-the shelf 100 Mbit and Gigabit Ethernet NICs and switches. People are reporting close to 1 Gbps wirespeed TCP/IP performance with the latest off-the-shelf Gigabit Ethernet adaptors which is more than enough for small-office/home-office storage I/O. Bottomline, while designing TCP/IP hardware NICs requires some effort, the most important enabling factor is that TCP/IP has been a stable architecture that has achieved extremely wide world-wide deployment measured in the 100s of Millions and soon Billions of end nodes. Thus IMHO the opportunity is to address and leverage this large market, rather than inventing yet another network layer and building hardware for one particular purpose that today has an installed base of zero. > From mark_bradley@btc.adaptec.com Fri Feb 4 04:45:51 2000 > From: "Bradley, Mark" > To: "'David R. Cheriton'" , Dave_Lee@3com.com, > Andreas Bechtolsheim > Cc: "Bradley, Mark" , Carl_Madison@3com.com, > Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, > "VonStamwitz, Paul" , > "Wilson, Andrew" > , raz@emc.com, > avb@cisco.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, > sob@harvard.edu, vern@aciri.org, csapuntz@cisco.com > Subject: RE: IP Storage Framework document > Date: Fri, 4 Feb 2000 05:46:51 -0700 > MIME-Version: 1.0 > X-SMTP-HELO: magic.adaptec.com > X-SMTP-MAIL-FROM: mark_bradley@btc.adaptec.com > X-SMAP-Received-From: outside > X-SMTP-PEER-INFO: magic.adaptec.com [208.236.45.80] > > Hi, David. Thank you for your thoughts. In answer to your > question, I think that: > 1)TCP in h/w is probably possible, but as others have found, it > is very difficult to do. A simpler protocol in hardware is > highly desirable > 2)Cost. TCP h/w implementations are likely to be complex silicon > designs that one could expect are costly to provide. > 3)While it is critical that IPS be cohabitable and non-destructive > to network performance, it is entirely possible that a separate > physical layer would be used for performance reasons. I can > envision many scenarios where we do this today, sans IPS. > 4)Performance. Both in CPU utilization and for pure bandwidth, it > is desirable to go light to minimize impact for those who might > not even implement in H/W. > Lastly, one envisions large networks in considering your points > below. For small networks (SOHO, small business), your points > are likely less important for many reasons, including network > loading. Small networks with generall light traffic are less > sensitive to the issues you bring up. They also tend to be more > cost sensitive. > > These are the points that it I think you may not be sensitive to. > -- markb > > -----Original Message----- > From: David R. Cheriton [mailto:cheriton@cisco.com] > Sent: Friday, February 04, 2000 12:06 AM > To: Dave_Lee@3com.com; Andreas Bechtolsheim > Cc: mark_bradley@btc.Adaptec.Com; Carl_Madison@3com.com; > Arun_Verma@3com.com; kquick@iphase.com; bassoon@ece.cmu.edu; > paulv@corp.adaptec.com; dwilson@corp.adaptec.com; raz@emc.com; > avb@cisco.com; rcampbel@cisco.com; Jgw@cisco.com; ddecapit@cisco.com; > sob@harvard.edu; vern@aciri.org; csapuntz@cisco.com > Subject: Re: IP Storage Framework document > > > I actually dont think there is any other transport protocol to > consider other than TCP. Here's why. > 1) TCP has been developed, refined, evaluated and deployed over a > period of 20 years, with some of the smartest folks in the > Internet research community spending years of effort. > There is NO hope in duplicating this effort, IMHO, > and plenty of evidence to suggest you'll screw up or end up > with TCP with gratutious differences if you dont. > 2) TCP has been the most successful transport protocol in the history > of mankind, and continues to grow dramatically in use. > There are numerous other attempts now and in the past to create > a new transport protocol for one reason or another, but the > success level has been almost 0 i.e. they're not all dead YET. > (I was even responsible for one old attempt, blush!) > 3) The TCP fast path is reasonably simple, has been identified > in software implementations and used successfully for years. > A hardware implementation of that fast path seems feasible to me, > and, even if a little more difficult than what one might imagine > for some new fangled protocol, the pay-off for a fast hardware > TCP is certain, and huge - not so with any other protocol. > > I think NFS/UDP is problematic. Even in a campus network, you can > have systematic loss of IP fragments that UDP too often generates, > causing communication failures and network overload. > And, handling IP fragment reassembly is one of the more complex > aspects of hardware IP. With a hardware TCP, I think one could > rely on MTU discovery and punt IP reassembly to software as > an unusual and unwarranted case. > Finally, my perception is that many netadmins are trying to reduce > and restrict non-TCP traffic on their networks because it does not > adapt predictably to network loads, causing failures. A high-rate > UDP or random new protocol load on a network would not be welcomed. > I'd also mention that the same forces that are causing networks > to converge on IP are pushing for convergence on TCP as well. > Diversity is not nice in a mission-critical 7x24 enterprise network > that is struggling with rapid scaling in users, bandwidth, services. > > So, the only thing that would convince me to consider another > transport protocol would be some major flaw in the above reasoning, > or proof that you just cant do hardware TCP at the required rates. > The latter would really, really surprise me. > > What am I missing? > > At 04:54 PM 2/3/00 -0800, Dave_Lee@3com.com wrote: > > > > > > > >Andy - > > > >You point out excellent issues that must be considered. Let me provide > some > >additional comments for all to ponder. Before I start, I believe the > document > >authors are using the term IP in its most generic sense and not meant to > imply a > >particular transport such as TCP, UDP, or some other transport. Drew/Paul > -- > >feel free to correct me if I'm wrong. There are certainly many > possibilities > >and I think they all need to be considered at this point. > > > >While it may be theoretically possible to implement TCP in silicon, it is > not at > >all clear if TCP in silicon will occur anytime soon -- if at all. Betting > on > >that fact may not sit well with others, especially those concerned with > higher > >performance campus-based solutions that they want available now. Nothing, > >however, precludes us from taking advantage of TCP in silicon in a future > >revision of the protocol(s). > > > >For WAN traffic, I think most people would agree with you on your remark > on the > >need for bandwidth management. Further, I believe most people would > probably > >agree with you that if we were looking for a pure L2 solution (e.g., > directly > >attached LAN), the IETF is *not* the right place to take this. I > definitely do. > > > >However, a customer may want to have a storage solution that can be > accessed > >over a campus network. In this case, a L2 solution is not sufficient and > IP is > >required. Further, it does not seem to be at all clear if TCP bandwidth > >management is a requirement in this environment. The NFS version 4 > requirements > >document has some commentary on this discontinuity. In a campus network, > there > >are other possible ways to do bandwidth management, including traffic > >engineering, differential services, etc. > > > >In light of the NFS discussion, perhaps looking to NFS may not be a bad > thing to > >do? NFS can operate over both TCP and UDP. Perhaps this model is also a > good > >model for IP storage. > > > >Dave > > > > > > > > > >Andreas Bechtolsheim on 02/03/2000 09:14:54 AM > > > >Sent by: Andreas Bechtolsheim > > > > > >To: See Below > >cc: See Below > >Subject: Re: IP Storage Framework document > > > > > > > >While I will not be able to attend this meeting in person, I would > >like to propose an alternative framework for Storage over IP. > >We will discuss this in more detail at the 2/16 Meeting in San Jose. > > > >The key characteristics of our proposal are as follows: > > > >1. It is based on TCP/IP, so all the issues of a reliable > > end-to-end byte stream including error recovery and > > congestion management are addressed without inventing > > a new protocol to do the same functions. > > > >2. It works over any network topology and any distance, > > addressing the needs of both local and remote connectivity > > between servers and storage devices. > > > >3. The protocol can be implemented efficiently in hardware > > inside network interface controllers allowing for > > high-performance wirespeed network operation. > > > >While in the past several new protocols have been proposed to > >accelerate certain types of data transfers, typically with > >the argument that they are easier to implement in hardware, > >experience has shown that such protocols have failed to gain > >market acceptance. With the plummenting cost of silicon, > >at this point in history the cost of implementing TCP/IP in > >NIC silicon is in the order of a few $ worth of silicon > >so there is simply no economic reason not to do it. > > > >In addition, proposing protocols that do not adhere to the > >TCP/IP bandwidth management algorithms pose a serious risk > >for overall network stability, in particular if they are used > >for high bandwidth traffic such as disk I/O. > > > >Finally, designing new protocols for limited topologies such > >as local area networks does not appear consistent with the > >goals of the IETF. > > > > > > > >To: Dave Lee/Hq/3Com Raz @Emc.Com > > Mark_Bradley @Btc.Adaptec.Com Avb @Cisco.Com > > Carl Madison/Us/3Com Rcampbel @Cisco.Com > > Arun Verma/Hq/3Com Jgw @Cisco.Com > > Kquick @Iphase.Com Ddecapit @Cisco.Com > > Bassoon @Ece.Cmu.Edu Sob @Harvard.Edu > > Paulv @Corp.Adaptec.Com Vern @Aciri.Org > > Dwilson @Corp.Adaptec.Com > > > >cc: Avb @Cisco.Com Csapuntz @Cisco.Com > > Cheriton @Cisco.Com > > > > > > From XXX Sat Feb 05 21:14:47 2000 Forwarded: Sat, 05 Feb 2000 16:14:47 -0500 Forwarded: "greg,garth " Return-Path: Dave_Lee@3com.com Delivery-Date: Fri Feb 4 10:57:56 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id KAA32586 for ; Fri, 4 Feb 2000 10:57:55 -0500 From: Dave_Lee@3com.com Received: from seattle.3com.com (seattle.3com.com [129.213.128.97]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA24640 for ; Fri, 4 Feb 2000 10:57:53 -0500 (EST) Received: from new-york.3com.com (new-york.3com.com [129.213.157.12]) by seattle.3com.com (8.8.8/8.8.8) with ESMTP id HAA13984; Fri, 4 Feb 2000 07:56:52 -0800 (PST) Received: from hqoutbound.ops.3com.com (hqoutbound.OPS.3Com.COM [139.87.48.104]) by new-york.3com.com (8.8.8/8.8.8) with SMTP id HAA15864; Fri, 4 Feb 2000 07:56:50 -0800 (PST) Received: by hqoutbound.ops.3com.com(Lotus SMTP MTA v4.6.3 (778.2 1-4-1999)) id 8825687B.005760F7 ; Fri, 4 Feb 2000 07:54:24 -0800 X-Lotus-FromDomain: 3COM To: Andreas Bechtolsheim cc: cheriton@cisco.com, avb@cisco.com, mark_bradley@btc.adaptec.com, Carl_Madison@3com.com, Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, paulv@corp.adaptec.com, dwilson@corp.adaptec.com, raz@emc.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, sob@harvard.edu, vern@aciri.org, csapuntz@cisco.com, Peter_Wang@3com.com Message-ID: <8825687B.00574BF8.00@hqoutbound.ops.3com.com> Date: Fri, 4 Feb 2000 07:55:18 -0800 Subject: RE: IP Storage Framework document Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=tHmXIoPFLoxGgS8WvIpmVRWWYEoiA9Py9RwYyYSFZxiz59LsSOYz9vpn" Content-Disposition: inline --0__=tHmXIoPFLoxGgS8WvIpmVRWWYEoiA9Py9RwYyYSFZxiz59LsSOYz9vpn Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-transfer-encoding: quoted-printable My position is that we still need to consider TCP, UDP, and other optio= ns in the initial discussions. Otherwise, you give up on even knowing about poss= ible advances through an arbitrary decision without hearing them. Perhaps s= ome of those advances may even help a TCP based approach -- who knows until yo= u see them. This is one of the purposes of the pre-BOF meeting to try to set= tle some of this out before the BOF. We don't know if Sun, HP, Seagate, etc. ha= ve other proposals that may be better than anything this group has. To stimulate more discussion, let me approach the issue from another ve= ctor. Isn?t one of the nice features about IP is that it can be diverse and s= upport multiple transports? If we accept your arguments that TCP is the solut= ion, why not do away with IP and create TCP flows through the network? What if = we identify features or requirements for IP storage that would benefit fro= m capabilities that TCP does not provide, such as multicast? Should we i= gnore that and still stick with TCP? Not at all clear to me. I just don?t know if everybody will buy argument that TCP is the single= cure-all for all the network?s ills. That slow start thing is a particular conc= ern for me. The additional cost to do MTU discovery (which everybody doesn?t d= o currently) is a concern to me -- why bother if it?s always 1500? And s= o forth. I can find an argument for TCP and one against TCP, which seems to say = that we should be flexible in our thinking. I?ll grant you that some simplistic TCP accelerations do exist today. = Doing minor things to the transmit path is easy -- the receive processing, retransmits, and other things that make TCP what it is and account for = a nice part of the overhead are hard. There are a slew of vendors (many start= ups) promising us network processor solutions that will solve all our proble= ms. I?m sure they?re promising you guys as well. But they?re vaporware at this= point. Are you going to commit your products on the mere possibility that some= body unproven will provide a critical component? It?s just not clear to me that TCP in silicon is cost-effective, especi= ally at 10/100 speeds. For ultra-low cost devices, the cost of the extra silic= on may be a concern to some. Initially, the cost will be high and even then, if = I?m price sensitive and space sensitive, then it would be a very serious issue to= me. It?ll be awhile before we see fully integrated, single chip solutions. = Bottom line, focusing solely on the WAN solution as the only solution before w= e see the alternatives and betting on unproven technology may not be the best str= ategy. One might argue that if it is a simple UDP based protocol, it will be significantly easier to implement the *complete* protocol in hardware t= han something based ontop of TCP. I hope UDP is not new fangled protocol,= at any rate. :-) Some of the UDP-related problems that you are mentioning, we= can clearly provide optional mechanisms to cure it for those who want to cu= re it. Key word here is optional. Those who do not want to use these options,= they don?t -- and they still interoperate. It is probably not in the IETF?s interest to be dictating to vendors an= d customers on how they build their networks and storage systems. We wan= t to enable them to be able to provide and to use interoperable storage solu= tions as they see fit. To enable the most broadly interoperable solution would = be the best answer. Dave Andreas Bechtolsheim on 02/04/2000 06:48:19 AM Sent by: Andreas Bechtolsheim To: cheriton@cisco.com, Dave Lee/HQ/3Com, avb@cisco.com, mark_bradley@btc.adaptec.com cc: See Below Subject: RE: IP Storage Framework document = --0__=tHmXIoPFLoxGgS8WvIpmVRWWYEoiA9Py9RwYyYSFZxiz59LsSOYz9vpn Content-type: text/plain; charset=us-ascii Content-Disposition: inline Mark, I think this whole argument comes down to the question "is it easier to put TCP/IP into hardware or is it easier to invent a new IP protocol to replace TCP functionality because such a protocol is easier to put into hardware?" Along these lines and in response to your points below, I would like to offer the following observations: 1. It appears straightforward to implement the "standard" TCP/IP control and data path in hardware, leaving more complex options to software. Both the latest 3COM and Intel Gigabit Ethernet NICs have added hardware support for the TCP/IP transmit path. There are several startups building full TCP/IP hardware implementations, and people are busy adding hardware functions such as IPSEC security. Seems to me the time has arrived to take advantage of contemporary silicon technology for the network interface controller. 2. The estimated cost for a TCP/IP hardware stack is 200K gates. This is about 2 mm^2 in 0.15um CMOS or less than $1 worth of silicon. To believe that TCP/IP hardware is a cost issue is a fallacy. 3. While one can always build separate networks for storage, just like Fibrechannel is today, doing so of course removes the strongest motivation to converge storage on IP in the first place. 4. One can implement Storage over TCP/IP in software today with off-the shelf 100 Mbit and Gigabit Ethernet NICs and switches. People are reporting close to 1 Gbps wirespeed TCP/IP performance with the latest off-the-shelf Gigabit Ethernet adaptors which is more than enough for small-office/home-office storage I/O. Bottomline, while designing TCP/IP hardware NICs requires some effort, the most important enabling factor is that TCP/IP has been a stable architecture that has achieved extremely wide world-wide deployment measured in the 100s of Millions and soon Billions of end nodes. Thus IMHO the opportunity is to address and leverage this large market, rather than inventing yet another network layer and building hardware for one particular purpose that today has an installed base of zero. > From mark_bradley@btc.adaptec.com Fri Feb 4 04:45:51 2000 > From: "Bradley, Mark" > To: "'David R. Cheriton'" , Dave_Lee@3com.com, > Andreas Bechtolsheim > Cc: "Bradley, Mark" , Carl_Madison@3com.com, > Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, > "VonStamwitz, Paul" , > "Wilson, Andrew" > , raz@emc.com, > avb@cisco.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, > sob@harvard.edu, vern@aciri.org, csapuntz@cisco.com > Subject: RE: IP Storage Framework document > Date: Fri, 4 Feb 2000 05:46:51 -0700 > MIME-Version: 1.0 > X-SMTP-HELO: magic.adaptec.com > X-SMTP-MAIL-FROM: mark_bradley@btc.adaptec.com > X-SMAP-Received-From: outside > X-SMTP-PEER-INFO: magic.adaptec.com [208.236.45.80] > > Hi, David. Thank you for your thoughts. In answer to your > question, I think that: > 1)TCP in h/w is probably possible, but as others have found, it > is very difficult to do. A simpler protocol in hardware is > highly desirable > 2)Cost. TCP h/w implementations are likely to be complex silicon > designs that one could expect are costly to provide. > 3)While it is critical that IPS be cohabitable and non-destructive > to network performance, it is entirely possible that a separate > physical layer would be used for performance reasons. I can > envision many scenarios where we do this today, sans IPS. > 4)Performance. Both in CPU utilization and for pure bandwidth, it > is desirable to go light to minimize impact for those who might > not even implement in H/W. > Lastly, one envisions large networks in considering your points > below. For small networks (SOHO, small business), your points > are likely less important for many reasons, including network > loading. Small networks with generall light traffic are less > sensitive to the issues you bring up. They also tend to be more > cost sensitive. > > These are the points that it I think you may not be sensitive to. > -- markb > > -----Original Message----- > From: David R. Cheriton [mailto:cheriton@cisco.com] > Sent: Friday, February 04, 2000 12:06 AM > To: Dave_Lee@3com.com; Andreas Bechtolsheim > Cc: mark_bradley@btc.Adaptec.Com; Carl_Madison@3com.com; > Arun_Verma@3com.com; kquick@iphase.com; bassoon@ece.cmu.edu; > paulv@corp.adaptec.com; dwilson@corp.adaptec.com; raz@emc.com; > avb@cisco.com; rcampbel@cisco.com; Jgw@cisco.com; ddecapit@cisco.com; > sob@harvard.edu; vern@aciri.org; csapuntz@cisco.com > Subject: Re: IP Storage Framework document > > > I actually dont think there is any other transport protocol to > consider other than TCP. Here's why. > 1) TCP has been developed, refined, evaluated and deployed over a > period of 20 years, with some of the smartest folks in the > Internet research community spending years of effort. > There is NO hope in duplicating this effort, IMHO, > and plenty of evidence to suggest you'll screw up or end up > with TCP with gratutious differences if you dont. > 2) TCP has been the most successful transport protocol in the history > of mankind, and continues to grow dramatically in use. > There are numerous other attempts now and in the past to create > a new transport protocol for one reason or another, but the > success level has been almost 0 i.e. they're not all dead YET. > (I was even responsible for one old attempt, blush!) > 3) The TCP fast path is reasonably simple, has been identified > in software implementations and used successfully for years. > A hardware implementation of that fast path seems feasible to me, > and, even if a little more difficult than what one might imagine > for some new fangled protocol, the pay-off for a fast hardware > TCP is certain, and huge - not so with any other protocol. > > I think NFS/UDP is problematic. Even in a campus network, you can > have systematic loss of IP fragments that UDP too often generates, > causing communication failures and network overload. > And, handling IP fragment reassembly is one of the more complex > aspects of hardware IP. With a hardware TCP, I think one could > rely on MTU discovery and punt IP reassembly to software as > an unusual and unwarranted case. > Finally, my perception is that many netadmins are trying to reduce > and restrict non-TCP traffic on their networks because it does not > adapt predictably to network loads, causing failures. A high-rate > UDP or random new protocol load on a network would not be welcomed. > I'd also mention that the same forces that are causing networks > to converge on IP are pushing for convergence on TCP as well. > Diversity is not nice in a mission-critical 7x24 enterprise network > that is struggling with rapid scaling in users, bandwidth, services. > > So, the only thing that would convince me to consider another > transport protocol would be some major flaw in the above reasoning, > or proof that you just cant do hardware TCP at the required rates. > The latter would really, really surprise me. > > What am I missing? > > At 04:54 PM 2/3/00 -0800, Dave_Lee@3com.com wrote: > > > > > > > >Andy - > > > >You point out excellent issues that must be considered. Let me provide > some > >additional comments for all to ponder. Before I start, I believe the > document > >authors are using the term IP in its most generic sense and not meant to > imply a > >particular transport such as TCP, UDP, or some other transport. Drew/Paul > -- > >feel free to correct me if I'm wrong. There are certainly many > possibilities > >and I think they all need to be considered at this point. > > > >While it may be theoretically possible to implement TCP in silicon, it is > not at > >all clear if TCP in silicon will occur anytime soon -- if at all. Betting > on > >that fact may not sit well with others, especially those concerned with > higher > >performance campus-based solutions that they want available now. Nothing, > >however, precludes us from taking advantage of TCP in silicon in a future > >revision of the protocol(s). > > > >For WAN traffic, I think most people would agree with you on your remark > on the > >need for bandwidth management. Further, I believe most people would > probably > >agree with you that if we were looking for a pure L2 solution (e.g., > directly > >attached LAN), the IETF is *not* the right place to take this. I > definitely do. > > > >However, a customer may want to have a storage solution that can be > accessed > >over a campus network. In this case, a L2 solution is not sufficient and > IP is > >required. Further, it does not seem to be at all clear if TCP bandwidth > >management is a requirement in this environment. The NFS version 4 > requirements > >document has some commentary on this discontinuity. In a campus network, > there > >are other possible ways to do bandwidth management, including traffic > >engineering, differential services, etc. > > > >In light of the NFS discussion, perhaps looking to NFS may not be a bad > thing to > >do? NFS can operate over both TCP and UDP. Perhaps this model is also a > good > >model for IP storage. > > > >Dave > > > > > > > > > >Andreas Bechtolsheim on 02/03/2000 09:14:54 AM > > > >Sent by: Andreas Bechtolsheim > > > > > >To: See Below > >cc: See Below > >Subject: Re: IP Storage Framework document > > > > > > > >While I will not be able to attend this meeting in person, I would > >like to propose an alternative framework for Storage over IP. > >We will discuss this in more detail at the 2/16 Meeting in San Jose. > > > >The key characteristics of our proposal are as follows: > > > >1. It is based on TCP/IP, so all the issues of a reliable > > end-to-end byte stream including error recovery and > > congestion management are addressed without inventing > > a new protocol to do the same functions. > > > >2. It works over any network topology and any distance, > > addressing the needs of both local and remote connectivity > > between servers and storage devices. > > > >3. The protocol can be implemented efficiently in hardware > > inside network interface controllers allowing for > > high-performance wirespeed network operation. > > > >While in the past several new protocols have been proposed to > >accelerate certain types of data transfers, typically with > >the argument that they are easier to implement in hardware, > >experience has shown that such protocols have failed to gain > >market acceptance. With the plummenting cost of silicon, > >at this point in history the cost of implementing TCP/IP in > >NIC silicon is in the order of a few $ worth of silicon > >so there is simply no economic reason not to do it. > > > >In addition, proposing protocols that do not adhere to the > >TCP/IP bandwidth management algorithms pose a serious risk > >for overall network stability, in particular if they are used > >for high bandwidth traffic such as disk I/O. > > > >Finally, designing new protocols for limited topologies such > >as local area networks does not appear consistent with the > >goals of the IETF. > > > > > > > >To: Dave Lee/Hq/3Com Raz @Emc.Com > > Mark_Bradley @Btc.Adaptec.Com Avb @Cisco.Com > > Carl Madison/Us/3Com Rcampbel @Cisco.Com > > Arun Verma/Hq/3Com Jgw @Cisco.Com > > Kquick @Iphase.Com Ddecapit @Cisco.Com > > Bassoon @Ece.Cmu.Edu Sob @Harvard.Edu > > Paulv @Corp.Adaptec.Com Vern @Aciri.Org > > Dwilson @Corp.Adaptec.Com > > > >cc: Avb @Cisco.Com Csapuntz @Cisco.Com > > Cheriton @Cisco.Com > > > > > > cc: Carl Madison/Us/3Com Rcampbel @Cisco.Com Arun Verma/Hq/3Com Jgw @Cisco.Com Kquick @Iphase.Com Ddecapit @Cisco.Com Bassoon @Ece.Cmu.Edu Sob @Harvard.Edu Paulv @Corp.Adaptec.Com Vern @Aciri.Org Dwilson @Corp.Adaptec.Com Csapuntz @Cisco.Com Raz @Emc.Com --0__=tHmXIoPFLoxGgS8WvIpmVRWWYEoiA9Py9RwYyYSFZxiz59LsSOYz9vpn-- From XXX Sat Feb 05 21:14:54 2000 Forwarded: Sat, 05 Feb 2000 16:14:54 -0500 Forwarded: "greg,garth " Return-Path: vern@daffy.ee.lbl.gov Delivery-Date: Sat Feb 5 04:37:30 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id EAA24930 for ; Sat, 5 Feb 2000 04:37:25 -0500 Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA21308 for ; Sat, 5 Feb 2000 04:37:24 -0500 (EST) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.9.2/8.9.2) id BAA09865; Sat, 5 Feb 2000 01:37:15 -0800 (PST) Message-Id: <200002050937.BAA09865@daffy.ee.lbl.gov> To: "Bradley, Mark" Cc: "'David R. Cheriton'" , Dave_Lee@3com.com, Andreas Bechtolsheim , Carl_Madison@3com.com, Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, "VonStamwitz, Paul" , "Wilson, Andrew" , raz@emc.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, sob@harvard.edu, csapuntz@cisco.com Subject: Re: IP Storage Framework document In-reply-to: Your message of Fri, 04 Feb 2000 05:46:51 MST. Date: Sat, 05 Feb 2000 01:37:15 PST From: Vern Paxson > 1)TCP in h/w is probably possible, but as others have found, it > is very difficult to do. Once a protocol such as this runs over IP, it will inevitably run over richer and richer IP substrates; and, as it does, it will require congestion control, intelligent retransmission policies, high performance windowing, etc. So I think the path inevitably leads to TCP. Vern From XXX Sat Feb 05 21:15:00 2000 Forwarded: Sat, 05 Feb 2000 16:15:00 -0500 Forwarded: "greg, garth " Return-Path: vern@daffy.ee.lbl.gov Delivery-Date: Sat Feb 5 04:37:42 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id EAA24935 for ; Sat, 5 Feb 2000 04:37:41 -0500 Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA21318 for ; Sat, 5 Feb 2000 04:37:41 -0500 (EST) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.9.2/8.9.2) id BAA09857; Sat, 5 Feb 2000 01:36:33 -0800 (PST) Message-Id: <200002050936.BAA09857@daffy.ee.lbl.gov> To: Dave_Lee@3com.com Cc: Andreas Bechtolsheim , mark_bradley@btc.adaptec.com, Carl_Madison@3com.com, Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, paulv@corp.adaptec.com, dwilson@corp.adaptec.com, raz@emc.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, sob@harvard.edu, cheriton@cisco.com, csapuntz@cisco.com Subject: Re: IP Storage Framework document In-reply-to: Your message of Thu, 03 Feb 2000 16:54:08 PST. Date: Sat, 05 Feb 2000 01:36:33 PST From: Vern Paxson > NFS can operate over both TCP and UDP. Perhaps this model is also= > a good model for IP storage. FYI, NFSv4 is spec'd to only operate over TCP or a transport with equivalent congestion control. Vern From XXX Wed Feb 09 22:24:17 2000 Forwarded: Sat, 05 Feb 2000 16:15:07 -0500 Forwarded: "greg, garth " Return-Path: vern@daffy.ee.lbl.gov Delivery-Date: Sat Feb 5 04:39:43 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id EAA24942 for ; Sat, 5 Feb 2000 04:39:42 -0500 Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA21375 for ; Sat, 5 Feb 2000 04:39:42 -0500 (EST) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.9.2/8.9.2) id BAA09898; Sat, 5 Feb 2000 01:39:34 -0800 (PST) Message-Id: <200002050939.BAA09898@daffy.ee.lbl.gov> To: Dave_Lee@3com.com Cc: Andreas Bechtolsheim , cheriton@cisco.com, mark_bradley@btc.adaptec.com, Carl_Madison@3com.com, Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, paulv@corp.adaptec.com, dwilson@corp.adaptec.com, raz@emc.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, sob@harvard.edu, csapuntz@cisco.com, Peter_Wang@3com.com Subject: Re: IP Storage Framework document In-reply-to: Your message of Fri, 04 Feb 2000 07:55:18 PST. Date: Sat, 05 Feb 2000 01:39:34 PST From: Vern Paxson > If we accept your arguments that TCP is > the solution, why not do away with IP and create TCP flows through the > network? Because there's a wealth of applications that don't need strongly reliable, in-order delivery. But IP storage doesn't appear to be one of them. > What if we identify features or requirements for IP storage that > would benefit from capabilities that TCP does not provide, such as multicast? Worth discussing, but if you want reliability, for multicast this gets hairy very fast ... > I just don?t know if everybody will buy argument that TCP is the single > cure-all for all the network?s ills. That slow start thing is a particular > concern for me. Not an issue if you keep the connection open and active. > The additional cost to do MTU discovery (which everybody > doesn?t do currently) is a concern to me -- why bother if it?s always > 1500? Likewise, not an issue if you keep the connection open. > I can find an argument for TCP and one against TCP, > which seems to say that we should be flexible in our thinking. You need to be more detailed in these arguments. This is old ground for TCP folks, so it really does require specifics to convey why it might not be appropriate. > I hope UDP is not new fangled protocol, > at any rate. :-) It's not, but it does nothing other than graft port numbers and a weak checksum on top of IP, so it's hardly a rich protocol. > Some of the UDP-related problems that you are mentioning, > we can clearly provide optional mechanisms to cure it for those who want > to cure it. Key word here is optional. Those who do not want to use > these options, they don?t -- and they still interoperate. This fails for congestion control. It also fails for strong reliability - it's all too easy to do fairly weak reliability, and it works fine until you start stressing things. > It is probably not in the IETF?s interest to be dictating to vendors and > customers on how they build their networks and storage systems. Well, if this isn't something the IETF needs to standardize, fine, less work for me (as Transport co-AD). But if the IETF is going to standardize it, then it will be held to the usual requirements of soundness. All that said, it seems the #1 step is to get the requirements fully described. Vern From XXX Thu Feb 10 11:31:28 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 10 05:56:01 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id FAA19718 for ; Thu, 10 Feb 2000 05:56:00 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA15207 for ips-outgoing; Thu, 10 Feb 2000 05:55:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from jindo.cisco.com (jindo.cisco.com [171.69.11.73]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA15202 for ; Thu, 10 Feb 2000 05:55:52 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by jindo.cisco.com (8.8.8/2.5.1/Cisco List Logging/8.8.8) with ESMTP id CAA08300; Thu, 10 Feb 2000 02:55:19 -0800 (PST) Date: Thu, 10 Feb 2000 02:55:19 -0800 (PST) From: Costa Sapuntzakis To: ips@ece.cmu.edu, dave_lee@3com.com cc: djpotter@cisco.com, cheriton@cisco.com, avb@cisco.com, vern@aciri.org, sob@harvard.edu Subject: Re: BOF Request In-Reply-To: <200002092127.NAA07247@postoffice.cisco.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, I'm working with Andy Bechtolsheim on the IP SAN initiative at Cisco. I believe we (the industry) are behind at this point because we (Cisco) have been hearing from our customers (and I'm sure you've been hearing from yours) that they want these technologies now. As such, much of the discussion on the charter and goals should happen before the BOF (on the ips mailing list). Let's make the IETF meeting be for serious high-bandwidth engineering discussion of the various approaches so we can attempt to reach consensus on a technical direction. Proposed agenda for BOF: > 5 min Review of BOF agenda and objectives (D. Lee/M. Bradley) > 10 min IP Storage Goals (D. Nagle/A. Wilson) > 15 min Discussion > 30 min Approaches & Implementations (10 min presentations) > o SCSI/TCP (TBA) > o Adaptec (A. Wilson) > o ISI's NetSCSI (?) > 40 min discussion > 5 min Proposed charters (D. Lee/M. Bradley/C. Sapuntzakis) > 15 min Discussion Our formal interaction with T10 and T11 should be minimized. The IETF strongly discourages the republishing of IETF standards by other organizations. In addition, IETF can work much more quickly than ANSI standards organizations. The only interaction I see with T13 is if we define an IDE transport protocol. ATAPI command sets are now handled through T10. FCP is an impressive protocol, especially in the bulk of documentation required to understand it. It is very much tied to Fibre Channel Physical Layer. Understanding FCP, its limitations, how it and its associated discovery mechanisms need to be fixed up to work in an IP environment is a large undertaking. FCP will seriously slow down the generation of any standard and impede the clarity of the final standard. A proposed charter: ------------------- Description of Working Group: The goal of this working group is to define extensions to current IETF transports for storage networking. There are a several areas to consider: - Reducing protocol processing overhead at servers and clients - SCSI transport on top of TCP - QoS parameters for storage networks - Running storage transports at multi-gigabit speeds The objectives of this working group are: * A proposed standard for a SCSI transport on top of TCP * A proposed standard for extensions to TCP that reduce protocol processing overhead. * Produce an informational RFC describing the QoS requirements of storage networks. Propose QoS extensions to diffserv if necessary. * Produce an informational RFC discussing the issues behind running storage transports at multi-gigabit speeds. -------------------------------------- -Costa > > ----- Begin Included Message ----- > > >From Dave_Lee@3com.com Wed Feb 9 03:17:02 2000 > From: Dave_Lee@3com.com > X-Lotus-FromDomain: 3COM > To: agenda@ietf.org > cc: sob@harvard.edu, mankin@isi.edu, Vern@Aciri.Org, markb@btc.adaptec.com, > Carl_Madison@3com.com, avb@cisco.com > Date: Wed, 9 Feb 2000 03:15:23 -0800 > Subject: BOF Request > Mime-Version: 1.0 > Content-Disposition: inline > X-SMTP-HELO: seattle.3com.com > X-SMTP-MAIL-FROM: Dave_Lee@3com.com > X-SMAP-Received-From: outside > X-SMTP-PEER-INFO: seattle.3com.com [129.213.128.97] > > > > BOF Description > > The objective of this BOF is: > > 1. To consider the creation of a new IETF working group to enable storage > networking via IP-based storage interfaces of SCSI, ATAPI, FCP and other > interfaces. Only system-to-storage interfaces will be considered. > > 2. To develop and agree upon a charter for the new working group. > > 3. To consider the proposed operational characteristics of reliable and > unreliable communications, shared and non-shared, managed and unmanaged, and > integrity, privacy, and/or non-secure. Further, the protocol must > efficiently operate over LAN, campus, and WAN environments. > > 4. To consider proposed protocol features such as automatic discovery, > enumeration, configuration, and management of storage adapters and devices. > > 5. To discuss what collaboration with ANSI T10, T11 and T13 is appropriate. > > > BOF Agenda (2 hrs) > 5 min Review of BOF agenda and objectives (D. Lee/M. Bradley) > 20 min IP Storage Goals (D. Nagle/A. Wilson) > 35 min Discussion > 20 min Approaches (5 minutes each) > o Cisco (A. Bechtolsheim) > o Adaptec (A. Wilson) > o ISI's NetSCSI (?) > o Other (?) > 10 min Proposed charter (D. Lee/M. Bradley) > 30 min Discussion > > > > > > > ----- End Included Message ----- > > > From XXX Thu Feb 10 17:51:44 2000 Return-Path: bassoon@ece.cmu.edu Delivery-Date: Thu Feb 10 12:49:05 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA24332 for ; Thu, 10 Feb 2000 12:49:04 -0500 Received: from jurassic (JURASSIC.ECE.CMU.EDU [128.2.252.18]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id MAA26382 for ; Thu, 10 Feb 2000 12:49:03 -0500 (EST) Message-Id: <200002101749.MAA26382@ece.cmu.edu> X-Sender: bassoon@pepelepew.ece.cmu.edu (Unverified) X-Mailer: QUALCOMM Windows Eudora Pro Version 4.0.2 Date: Thu, 10 Feb 2000 12:50:38 -0500 To: bassoon@ece.cmu.edu From: David Nagle Subject: Fwd: Meeting Minutes from 2/4/00 Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=====================_88590866==_" --=====================_88590866==_ Content-Type: text/plain; charset="us-ascii" >From: "Bradley, Mark" >To: "'Dave_Lee@3com.com'" , > Andreas Bechtolsheim > >Cc: "Bradley, Mark" , Carl_Madison@3com.com, > Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu, > "VonStamwitz, Paul" , > "Wilson, Andrew" > , raz@emc.com, > avb@cisco.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, > sob@harvard.edu, vern@aciri.org, cheriton@cisco.com, > csapuntz@cisco.com >Subject: Meeting Minutes from 2/4/00 >Date: Fri, 4 Feb 2000 17:14:14 -0700 >X-Mailer: Internet Mail Service (5.5.2650.21) > >Attached in MS-Word. > > > --=====================_88590866==_ Content-Type: application/msword; name="IPS020400-1.doc" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="IPS020400-1.doc" 0M8R4KGxGuEAAAAAAAAAAAAAAAAAAAAAPgADAP7/CQAGAAAAAAAAAAAAAAABAAAANwAAAAAAAAAA EAAAOQAAAAEAAAD+////AAAAADYAAAD///////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// ///////////////////////////////////////////////////////////////////////////s pcEAWQAJBAAAABK/AAAAAAAAEAAAAAAABAAATRkAAA4AYmpiavNX81cAAAAAAAAAAAAAAAAAAAAA AAAJBBYAHjYAAJE9AQCRPQEAQBUAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAAAAAAAAAD//w8AAAAA AAAAAAD//w8AAAAAAAAAAAD//w8AAAAAAAAAAAAAAAAAAAAAAF0AAAAAAEYBAAAAAAAARgEAAEYB AAAAAAAARgEAAAAAAAD+AgAAAAAAAP4CAAAAAAAA/gIAABQAAAAAAAAAAAAAABIDAAAAAAAAEgMA AAAAAAASAwAAAAAAABIDAAAAAAAAEgMAABQAAAAmAwAATAAAABIDAAAAAAAAcQ8AADIBAACOAwAA AAAAAI4DAAAAAAAAjgMAAAAAAACOAwAAAAAAAI4DAAAAAAAAbQQAAAAAAABtBAAAAAAAAG0EAAAA AAAANg8AAAIAAAA4DwAAAAAAADgPAAAAAAAAOA8AAAAAAAA4DwAAAAAAADgPAAAAAAAAOA8AACQA AACjEAAA9AEAAJcSAACYAAAAXA8AABUAAAAAAAAAAAAAAAAAAAAAAAAA/gIAAAAAAABtBAAAAAAA AAAAAAAAAAAAAAAAAAAAAABpBAAABAAAAG0EAAAAAAAAbQQAAAAAAABtBAAAAAAAAFwPAAAAAAAA eQkAAAAAAABGAQAAAAAAAEYBAAAAAAAAjgMAAAAAAAAAAAAAAAAAAI4DAADbAAAAjgMAAAAAAAB5 CQAAAAAAAHkJAAAAAAAAeQkAAAAAAABtBAAAYAQAAEYBAAA8AQAAjgMAAAAAAAD+AgAAAAAAAI4D AAAAAAAANg8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAEgMAAAAAAAASAwAAAAAAAEYBAAAAAAAARgEA AAAAAABGAQAAAAAAAEYBAAAAAAAAbQQAAAAAAAA2DwAAAAAAAHkJAAB0AwAAeQkAAAAAAADtDAAA jgAAAIcOAACfAAAAggIAAHwAAAD+AgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANg8AAAAAAACOAwAAAAAAAHIDAAAcAAAAIEnAu0lv vwESAwAAAAAAABIDAAAAAAAAzQgAAKwAAAAmDwAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATmV0 d29yayBJL08gRXhwbG9yYXRvcnkgTWVldGluZw0xMC8xNC85OQ0zQ29tLCBNYXJsYm9yb3VnaCwg TWFzcy4NDUF0dGVuZGVlczogIE1hcmsgQnJhZGxleQ1EcmV3IFdpbHNvbg0NM0NvbToJCUNhcmwg TWFkaXNvbg1EYXZlIExlZQ1BcnVuIFZlcm1hDQ1JbnRlcnBoYXNlOiAqS2V2aW4gUXVpY2sNDUNp c2NvOglOb25lDQ1DTVU6CQlEYXZlIE5hZ2xlDQ1FTUM6CQlZb2F2IFJheg0JCUdlb3JnZSBFcmlj c29uDQkJSmFjayBIYXJ3b29kDQ1BZ2VuZGE6IAlJbnRyb2R1Y3Rpb25zDQkJQWN0aW9uIEl0ZW0g UmV2aWV3DQkJUGFja2V0aXplZCBTQ1NJIA1JUFMgR29hbHMNSUVURiBQcm9jZXNzDVByb3RvIGNo YXJ0ZXIgZGlzY3Vzc2lvbi9GcmFtZXdvcmsgZG9jDUNoYXJ0ZXIgQ3JlYXRpb24NDURpZmZlcmVu Y2VzIGJldHdlZW4gdGhpcyBtZWV0aW5nIGFuZCB0aGUgMi8xNiBtZWV0aW5nLg1CYWNrZ3JvdW5k IGFuZCBvdmVydmlldy4NDUdlbmVyYWwgZGlzY3Vzc2lvbiBvZiBPQlNEIGFuZCBob3cgaXQgbWF5 IG9yIG1heSBub3QgZml0IHdpdGggSVBTLiAgR2VvcmdlIGJyaW5ncyB1cCBhIHBvaW50IGFib3V0 IE9TVnMgYW5kIGFwcGwuIHByb3ZpZGVycyBhY2NlcHRpbmcgT0JTRC4gIFRoaXMgc2VlbXMgc29t ZXdoYXQgb3J0aG9nb25hbCB0byBJUFMsIGJ1dCBpcyBzdGlsbCBhIHZhbGlkIHBvaW50Lg0NVXNp bmcgSVBTLCBhbnkgbmF0aXZlIEZTIG91Z2h0IHRvIHdvcmsgdGhyb3VnaCB0aGUgZHJpdmVyIGlu IHRoZSBzYW1lIHdheSB0aGV5IGRvIHRvZGF5Lg0NRHJldyBnYXZlIGFuIG92ZXJ2aWV3IG9mIFBh Y2tldGl6ZWQgU0NTSS4NDUdlb3JnZSBzdWdnZXN0ZWQgdGhhdCBhbiBhYnN0cmFjdGlvbiBsYXll ciBzdWNoIGFzIHBpY3R1cmVkIGJlbG93IHdvdWxkIGJlIHVzZWZ1bC4NDVNDU0kgVUxQB0FUQQdB VEFQSQdFdGMuBwdNYXBwZXIvYWJzdHJhY3Rpb24gbGF5ZXIHB1NDU0kgTExQB1NQSQdGQwdFdGMu BwcNSXQgd2FzIHN1Z2dlc3RlZCB0aGF0IHRoZSBlZmZvcnQgRHJldyB3YXMgZGVzY3JpYmluZyB3 YXMgYXQgYSBkaWZmZXJlbnQgdHJhbnNwb3J0IGxldmVsLiAgTmVlZCBjb21tb24gdGVybXMgdG8g ZGVzY3JpYmUgdGhlc2UgdGhpbmdzLg0NU0NTSSBlbmNhcHN1bGF0aW9uBwdMVFAgbGF5ZXIHB0lQ BwdQaHlzaWNhbAcHCQ1EaXNjdXNzaW9uIG9mIHdoZXJlIHdlIHdlcmUgdy8gTkRBLCBldGMuIHNv IHRoYXQgd2UgY291bGQgc2hhcmUgb3VyIFBPQyB3b3JrIGVhcmx5LiAgQWRhcHRlYyBjYW4gcHJv YmFibHkgcmVsZWFzZSB0aGlzIGluZm9ybWF0aW9uIHB1YmxpY2x5IGluIDcvMDAuDQ1DaXNjbyB3 YW50cyB0byBwdXQgZm9yd2FyZCBhIHByb3Bvc2FsIChzb2x1dGlvbikgcHJpb3IgdG8gc3RhdGVt ZW50IG9mIHByb2JsZW0sIGNoYXJ0ZXIgYXBwcm92YWwsIGV0Yy4NDVByb3RvIGNoYXJ0ZXIgd2ls bCBiZSBwcmVzZW50ZWQgYXQgQm9GLiAgRmVlZGJhY2sgd2lsbCBiZSBnaWVuIG9uIHRoaXMgYXQg dGhhdCB0aW1lLiAgRm9ybWFsIGNoYXJ0ZXIgd2lsbCBiZSBuZWdvY2lhdGVkIGJldHdlZW4gdGhl IEFEknMgYW5kIHRoZSBXRyBjaGFpcnMuDQ1XZSBuZWVkOglXaGF0IHdlIHdhbnQgdG8gYWNoaWV2 ZSBhbmQgd2h5IChGcmFtZXdvcmsgZG9jIGRpc2N1c3NlcyBzb21lIG9mIHRoaXMuKQ0JCUlFVEYg cHJvY2Vzcw0JCURlY2lzaW9uIG9mIHdoZXRoZXIgdG8gY3JlYXRlIGEgY291bnRlci1wcm9wb3Nh bCB0byBDaXNjb5JzIGZvciAyLzE2DQ0NWW9hdiBiZWxpZXZlcyB0aGF0IHdlIHdhbnQgdG8gaGF2 ZSBhIHByb3Bvc2FsIGNvbWluZyBvdXQgYXQgdGhlIHNhbWUgdGltZSBhcyBDaWNzb5JzLg1EYXZl IE4uIGFuZCBHZW9yZ2Ugc3VnZ2VzdGVkIHRoYXQgcGVyaGFwcyB0aGUgTFRQIGlzIHNvbWUgbGlt aXRlZCBzdWJzZXQgb2YgVENQLiAgVGhhdJJzIGEgcG9zc2liaWxpdHksIG9mIGNvdXJzZS4NDShU aGlzIGlzIHRvIGJlIGEgYmVzdCBvZiBjbGFzcyBzcGVjLCBub3QgYSBzcGVjIHRvIHN1cHBvcnQg c29tZSBzcGVjaWZpYyBpbXBsZW1lbnRhdGlvbi4pDQ1JRVRGIFByb2Nlc3M6CVdHIGlzIGZvcm1l ZC4NVmVyeSBmb2N1c2VkIGNoYXJ0ZXIuDU9wZW4gdG8gYWxsIGludGVyZXN0ZWQuDQkJQ2hhaXIv Q29jaGFpcnMgbm9taW5hdGVkLg1SZWZsZWN0b3Igc2V0IHVwLg1Db25jZW5zdXMgZHJpdmVuLg1E cmFmdCBjcmVhdGVkLg1SZXZpZXcgYnkgQUSScw1BcHByb3ZlZCBvcg0JUmUtd29ya2VkIGlmIG5v dCBhcHByb3ZlZA1JbnRlcm5ldCBkcmFmdCBjcmVhdGVkICgoNiBtb250aHMpKFByb3Bvc2VkIHN0 YW5kYXJkDUludGVyb3BlcmFiaWxpdHkgdGVzdGluZyBzdWNjZXNzZnVsLg1BcHByb3ZlZCBzdGFu ZGFyZC4NDUJvRiBpcyBoZWxkIChBZGVsYWlkZSwgQXVzdHJhbGlhLCAzLzAwKS4NQm9GIGlzIGEg b25lLXNob3QgZGVhbC4gIElmIGl0IGZhaWxzLCBpdJJzIG5vdCBhbGxvd2VkIGZvciByZWNvbnNp ZGVyYXRpb24uDUFEknMgZGV0ZXJtaW5lIGlmIFdHIGlzIGZvcm1lZCBvciBub3QgYmFzZWQgb24g Qm9GIG91dGNvbWUuDQ1QcmUtQm9GIGlzIHRvIGRldGVybWluZSBpZiB0aGVyZSBpcyBhIGNvbnNl bnN1cyB0aGF0IHRoZXJlIGlzIGEgbmVlZC4NRGVzY3JpYmUgd2hhdCBJUFMgaXMsIGdlbmVyYWxs eSBzcGVha2luZy4NQWdyZWUgdG8gcHJvdG8tY2hhcnRlci4NDUlQUyBBcHBsaWNhdGlvbgdMQU4H Q2FtcHVzB1dBTgcHU2hhcmVkB1gHU0VQB1NFUAcHVW5zaGFyZWQHWAdTRVAHU0VQBwdUcnVzdAdT RVAHU0VQB1NFUAcHSW50ZWdyaXR5BwdQcml2YWN5BwdObyBwcm90ZWN0aW9uBwcNV2hhdDoJU2Vy dmVyIHRvIHN0b3JhZ2UNCUJsb2NrIEkvTyAobm90IGZpbGUgc3lzdGVtLCBvYmplY3QsIGV0Yy4g IEFsc28gbm90IHByZWNsdWRlIHRoZXNlLikNCVJlbGlhYmxlIGFuZCBVbnJlbGlhYmxlIChyZWxp YWJsZSBMVFAgb3Igc3RyYWlnaHQgdG8gSVAsIGlmIHJlbGlhYmxlIG5ldHdvcmspDQlTaGFyZWQg YW5kIG5vbi1zaGFyZWQgKHBhcnRpdGlvbmluZykNCUludGVncml0eSwgcHJpdmFjeSBhbmQgbm9u LXNlY3VyZSAoSVBTZWMgb3Igbm90KQ0JSVAgaXMgTEFOLCBjYW1wdXMsIFdBTiAoKElQIGlzKSAg Y2FuIGtlZXAgdG8gc3VibmV0IHZpYSBJUHY2IGFkZHJlc3MpDQ1XaHk6CUlQIGlzIGV2ZXJ5d2hl cmUsIHNvIHdoeSBub3QgdXNlIGl0DQkJTXVsdGlwbGUgcGh5c2ljYWwgdHJhbnNwb3J0cw0JTmVl ZCBsb3cgb3ZlcmhlYWQgYW5kIGJpdCBlZmZpY2llbmNpZXMgb24gSVAgd2hpbGUgc3RpbGwgZGVh bGluZyB3LyByZWxpYWJpbGl0eSBpc3N1ZXMpDQlVc2VzIGV4aXN0aW5nLCBwcm92ZW4gaW5mcmFz dHJ1Y3R1cmVzDQlMYXJnZSBzY2FsZSBpbnRlcm9wZXJhYmlsaXR5IGVuYWJsZWQvcmVxdWlyZWQg KHRoZSBJbnRlcm5ldCBoYXMgcHJvdmVuIHRoaXMpDQlCdXQgc3RvcmFnZSBoYXMgZGlmZmVyZW50 IHJlcXVpcmVtZW50DQkJTG93IHByb2Nlc3NvciBjb25zdW1wdGlvbiAoY29zdCBpbXBsaWNhdGlv bnMpDQkJTGF0ZW5jeSBzZW5zaXRpdmUNCQlMb3cgbWVtb3J5IGF2YWlsYWJpbGl0eQ0JQ2FuknQg dXNlIHNhbWUgb2xkIFNDU0kvVENQL0lQDQkJTmVlZCByZWxpYWJpbGl0eQ0JCU5lZWQgbGlnaHQg d2VpZ2h0DQkJCURyaXZlIGVuYWJsZW1lbnQNCU1hbmFnZWFiaWxpdHkgdG9vbHMgY2FuIGJlIGxl dmVyYWdlZA1XaGlsZSBGQ1AgbWlnaHQgd29yayBvdmVyIFRDUCwgb3RoZXIgc3RvcmFnZSBpbnRl cmZhY2VzIHRvIFRDUCBkbyBub3Qgd29yaw0JVGltaW5nLCByZWxpYWJpbGl0eSwgb3RoZXIgaXNz dWVzDQkNDU90aGVyOglMZXZlcmFnZXMgdHJlbmRzIG9mIHN0b3JhZ2UNCQlQYWNrZXRpemVkIFND U0kNCQlGQ1ANCQlQYXJ0aXRpb25pbmcvem9uaW5nDQlBZGRyZXNzZXMgbmVlZHMgb2YgY3VzdG9t ZXJzDQkJQ29ubmVjdGl2aXR5DQkJRWFzZSBvZiBhZGRpbmcgc3RvcmFnZQ0JCUxvd2VyIGFkbWlu aXN0cmF0aXZlIG92ZXJoZWFkDQkJCUNvbW1vbiB0b29scyBtYXkgYmUgdXNlZA0JKFNMVylBTiBh cmUgYWxsIGltcG9ydGFudCAmIERhdGEgaXMgIzEgYXNzZXQgYW5kIG11c3QgYmUgcHJvdGVjdGVk DQkJRGlzYXN0ZXIgcmVjb3ZlcnkNCQlNaXJyb3Igc2l0ZXMNCQlEYXRhIG1pZ3JhdGlvbiwgSFNN LCBldGMuDQkJVW5hdHRlbmRlZCBiYWNrdXANCURhdGEgc3RvcmFnZSBjYW4gYmUgZGVjb3VwbGVk IGZyb20gc3lzdGVtIHJlbGlhYmlsaXR5DQkJKGUuZy4gTVRETCA9IFsxL01UREwxICogTVRETDJd L01UREwxb3IyKQ0JQ2FuIHVzZSBJUFNlYw0NUHJvdG8tQ2hhcnRlciBPdXRsaW5lOg1BQlNUUkFD VDoNQ3JlYXRlIGFuIGVmZmljaWVudCBJUC1iYXNlZCBwcm90b2NvbCB0aGF0IGVuYWJsZXMgbmV0 d29yayBzdG9yYWdlLiAgDQ08TGlzdCBqdXN0aWZpY2F0aW9ucy9tb3RpdmF0aW9ucyBmcm9tIHRo ZSBhYm92ZSAzIGFyZWFzLj4NDVRoZSBjcml0aWNhbCBmYWN0b3JzIGZvciB0aGlzIHByb3RvY29s IGFyZToNCTEpDQkgfA0JbikNDVRoZSB0aW1lZnJhbWUgZm9yIHRoaXMgd29yayBpcyAobWlsZXN0 b25lIGRvYywgZGVzY3JpcHRpb24gb2YgZGVsaXZlcmFibGVzKQ0NTkZTIHY0IGNvbnNpZGVyYXRp b25zIG1heSBiZSBhIGdvb2QgdGVtcGxhdGUgZm9yIHRoaXMgZG9jdW1lbnQuDQ1TZWUgYWxzbyB3 d3cuaWV0Zi5vcmcvaWV0Zi8wMG1hci9ibG9ja3MtYWdlbmRhLnR4dA0NDQ0NQWN0aW9uIEl0ZW1z Og1EaXN0cmlidXRlIEZyYW1ld29yayBkb2MgdG8gSUVURiBwcmlvciB0byBCb0YgLS0gKERyZXcs IFBWUykNV3JpdGUgdXAgcHJvdG8tY2hhcnRlciAtLSAoRGF2ZSBMLCBNYXJrQikNQ3JlYXRlIGFu ZCBlZGl0IHNsaWRlcyBmb3IgcHJlLUJvRiAtLSAoRGF2ZSBOLCBNYXJrQiwgQ2FybE0sIEdlb3Jn ZUUpIJYgbm90ZSB0aGF0IHRoaXMgcGFydGljdWxhciBzZXQgb2YgDSAgICAgICAgICAgc2xpZGVz IGhhcyBsaW1pdGVkIGRpc3RyaWJ1dGlvbiBwcmlvciB0byAyLzE2DUludml0ZSBJbnRlbCBTQUwg Rm9sa3MsIExTSSAmIEthdGh5IEUuIG9mIE1TLCBDb252ZXJlbmV0LCBQYXVsIEJvcnJpbGwsIEdl b3JnZSBQZW5va2llIHRvIDIvMTYgDSAgICAgICAgICAgLS0gKERhdmUgTikNSW52aXRlIEludGVs IE5JQywgU3VuIChLYXRoeSBKZW5rcz8pLCBUdXJib0xpbnV4IChMaW5kYSBXYW5nKSB0byAyLjE2 IC0tIChNYXJrQikNDVNob3J0IHRlcm0gc2NoZWR1bGU6DTIvNC8wMCBXb3JrIGRlbGVnYXRlZA0y LzcvMDAgQ2hhcnRlciAxc3QgZHJhZnQNMi84LzAwIENoYXJ0ZXIgMm5kIGRyYWZ0DTIvOS8wMCBG cmFtZXdvcmsgMm5kIERyYWZ0DTIvMTAvMDAgU2xpZGVzIDFzdCBEcmFmdA0yLzExLzAwIEZyYW1l d29yayAzcmQgRHJhZnQNMi8xNC8wMCBTbGlkZXMgMm5kIERyYWZ0DTIvMTQvMDAgUE0gQWxsIGRv Y3Mgc2VudCBvdXQgcHJpb3IgdG8gbGVhdmluZyB0byB0cmF2ZWwNMi8xNS8wMCBUcmF2ZWwgRGF5 DTIvMTYvMDAgRjJGIHByZS1Cb0YNDQ0NDQ0NDQ0NDQ0NDQ0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAgBAAAQwQAAEwE AADJBAAAygQAAMsEAAD6BAAA+wQAAAIFAAAEBQAAigUAAIUHAACkCwAAsQsAAO4LAAD6CwAAnwwA AKEMAADUDAAA1QwAAN8MAADgDAAAZg4AAHYOAACPDgAAlw4AAKEOAACrDgAAsg4AAL4OAAC/DgAA yQ4AAMoOAADSDgAA0w4AAOEOAADjDgAA5w4AAC8QAAAyEAAAvBIAAMESAABzFAAAdBQAAHsUAAB8 FAAAghQAAIYUAACYFAAArRQAAMEVAAACFgAAChYAADcWAABFFgAAGhgAAC0YAABVGAAAVxgAAG4Y AABwGAAAiRgAAIsYAACiGAAApBgAAL4YAAAA9+z34tj32Oz3zskAxwDDAMMAvQC9ALsAuwC7ALsA xwDHAMcAuwC7ALsAuQC5ALkAwwC1r7WpAMMApwCnAKcApwADSCoBCjUIgT4qAUNKGAAACjUIgTYI gUNKEAAABzYIgUNKEAADSCoCAzUIgQoJagQA4PBtSAAEAAY1CIE+KgEAAzYIgQhPSgMAUUoDAAAS QioBT0oDAFFKAwBoCABuSAkEABNDShAAT0oDAFFKAwBoCABuSAkEEjcIgU9KAwBRSgMAaAgAbkgJ BAAVNQiBPioBT0oDAFFKAwBoCABuSAkED09KAwBRSgMAaAgAbkgJBABCAAQAACAEAAApBAAAQgQA AEMEAABcBAAAaAQAAGkEAAB9BAAAhgQAAJEEAACSBAAAqwQAAKwEAAC4BAAAuQQAAMoEAADLBAAA 2gQAAOsEAAD6BAAA+wQAABIFAAAnBQAAOgUAAEQFAABRBQAAeAUAAIkFAAD9AAAAAAAAAAAAAAAA +gAAAAAAAAAAAAAAAPoAAAAAAAAAAAAAAAD4AAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAPIAAAAA AAAAAAAAAAD4AAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAPIAAAAAAAAAAAAAAADyAAAAAAAAAAAA AAAA8gAAAAAAAAAAAAAAAPgAAAAAAAAAAAAAAAD4AAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAPgA AAAAAAAAAAAAAAD4AAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAPgAAAAAAAAAAAAAAAD4AAAAAAAA AAAAAAAA+AAAAAAAAAAAAAAAAPgAAAAAAAAAAAAAAAD4AAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAA APgAAAAAAAAAAAAAAADyAAAAAAAAAAAAAAAA8gAAAAAAAAAAAAAAAPIAAAAAAAAAAAAAAADyAAAA AAAAAAAAAAAAAAAAAAAFAAAPhNACEYTQAgABAAADAAADJAEAAQ8AABwABAAAIAQAACkEAABCBAAA QwQAAFwEAABoBAAAaQQAAH0EAACGBAAAkQQAAJIEAACrBAAArAQAALgEAAC5BAAAygQAAMsEAADa BAAA6wQAAPoEAAD7BAAAEgUAACcFAAA6BQAARAUAAFEFAAB4BQAAiQUAAIoFAADBBQAA2gUAANsF AACrBgAArAYAAAUHAAAGBwAAMAcAADEHAACEBwAAhQcAAI4HAACSBwAAmAcAAJ0HAACeBwAAtwcA ALgHAADBBwAAxQcAAMgHAADNBwAAzgcAAM8HAABVCAAAVggAAGkIAABqCAAAdAgAAHUIAAB4CAAA eQgAAIIIAACDCAAAhQgAABoJAAAbCQAAggkAAIMJAAAfCgAAIAoAAHEKAACACgAAxwoAAMgKAADJ CgAAHwsAAJILAACTCwAA7QsAAO4LAAAKDAAAIAwAADgMAABUDAAAZgwAAHgMAACHDAAAlgwAAKIM AAC9DAAA8gwAABcNAAAqDQAAKw0AAFQNAACgDQAA3A0AAN0NAAAjDgAATQ4AAP7+/v7+/v7+/v7+ /v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+ /v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v4AAAAAAgEBZIkFAACKBQAAwQUAANoFAADb BQAAqwYAAKwGAAAFBwAABgcAADAHAAAxBwAAhAcAAIUHAACOBwAAkgcAAJgHAACdBwAAngcAALcH AAC4BwAAwQcAAMUHAAD5AAAAAAAAAAAAAAAA9wAAAAAAAAAAAAAAAPcAAAAAAAAAAAAAAAD3AAAA AAAAAAAAAAAA9wAAAAAAAAAAAAAAAPcAAAAAAAAAAAAAAAD3AAAAAAAAAAAAAAAA9wAAAAAAAAAA AAAAAPcAAAAAAAAAAAAAAAD3AAAAAAAAAAAAAAAA9wAAAAAAAAAAAAAAAPcAAAAAAAAAAAAAAADy AAAAAAAAAAAAAAAA8gAAAAAAAAAAAAAAAPIAAAAAAAAAAAAAAADyAAAAAAAAAAAAAAAA1mgAAAAA AAAAAAAAAPIAAAAAAAAAAAAAAAC8WAAAAAAAAAAAAAAA8gAAAAAAAAAAAAAAAPIAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAaAAAWJAEXJAEClmwAAzQBBdYYBAEAAAQBAAAEAQAABAEAAAQBAAAE AQAACNYGAAGU/ywiABsAABYkARckAQKWbAAF1hgEAQAABAEAAAQBAAAEAQAABAEAAAQBAAAI1gwA BJT/OgjgEIYZLCIABAAAAyQBFiQBAAEAAAAFAAAPhBwCEYTk/QAVxQcAAMgHAADNBwAAzgcAAM8H AABVCAAAVggAAGkIAABqCAAAdAgAAHUIAAB4CAAAeQgAAIIIAACDCAAAhQgAABoJAAAbCQAAggkA AIMJAAAfCgAAIAoAAHEKAAD6AAAAAAAAAAAAAAAA+gAAAAAAAAAAAAAAAN4AAAAAAAAAAAAAAADc AAAAAAAAAAAAAAAA3AAAAAAAAAAAAAAAANwAAAAAAAAAAAAAAAD6AAAAAAAAAAAAAAAAwywAAAAA AAAAAAAAAPoAAAAAAAAAAAAAAADDEAAAAAAAAAAAAAAA+gAAAAAAAAAAAAAAAMMoAAAAAAAAAAAA AAD6AAAAAAAAAAAAAAAAwwAAAAAAAAAAAAAAANwAAAAAAAAAAAAAAADcAAAAAAAAAAAAAAAA3AAA AAAAAAAAAAAAANwAAAAAAAAAAAAAAADcAAAAAAAAAAAAAAAA3AAAAAAAAAAAAAAAANwAAAAAAAAA AAAAAADcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABgAABYkARckAQKWbAAF1hgEAQAABAEAAAQB AAAEAQAABAEAAAQBAAAI1gYAAZT/CAcAAQAAABsAABYkARckAQKWbAAF1hgEAQAABAEAAAQBAAAE AQAABAEAAAQBAAAI1gwABJT/OgjgEIYZLCIABAAAAyQBFiQBABZxCgAAgAoAAMcKAADICgAAyQoA AB8LAACSCwAAkwsAAO0LAADuCwAACgwAACAMAAA4DAAAVAwAAGYMAAB4DAAAhwwAAJYMAACiDAAA vQwAAPIMAAAXDQAAKg0AACsNAABUDQAAoA0AAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAA AAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAA AAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAADxAAAAAAAAAAAAAAAA 5QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAADfAAAAAAAAAAAAAAAA3wAAAAAAAAAAAAAAAN8AAAAA AAAAAAAAAADfAAAAAAAAAAAAAAAA2QAAAAAAAAAAAAAAAN8AAAAAAAAAAAAAAADfAAAAAAAAAAAA AAAA3wAAAAAAAAAAAAAAAN8AAAAAAAAAAAAAAADfAAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0A AAAAAAAAAAAAAAAAAAAAAAAFAAAPhKAFEYTQAgAFAAAPhNACEYTQAgwAAAomAAtGAwAPhNgJDcYH AWgBAdgJBgwAAAomAAtGAgAPhNgJDcYHAWgBAdgJBgABAAAAGaANAADcDQAA3Q0AACMOAABNDgAA ZQ4AAGYOAAB2DgAAeg4AAIEOAACFDgAAhg4AAI0OAACPDgAAkw4AAJcOAACYDgAAoQ4AAKMOAACn DgAAqw4AAKwOAACyDgAAtg4AALoOAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAA AAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD6AAAAAAAAAAAAAAAA 9QAAAAAAAAAAAAAAAPUAAAAAAAAAAAAAAAD1AAAAAAAAAAAAAAAA2UgAAAAAAAAAAAAAAPoAAAAA AAAAAAAAAADWAAAAAAAAAAAAAAAA9QAAAAAAAAAAAAAAAPUAAAAAAAAAAAAAAADZUAAAAAAAAAAA AAAA+gAAAAAAAAAAAAAAAPUAAAAAAAAAAAAAAAD1AAAAAAAAAAAAAAAA9QAAAAAAAAAAAAAAANlM AAAAAAAAAAAAAAD6AAAAAAAAAAAAAAAA9QAAAAAAAAAAAAAAAPUAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAwEAFiQBABsAABYkARckAQKWbAAF1hgEAQAABAEAAAQBAAAEAQAABAEAAAQBAAAI 1gwABJT/OgjgEIYZLCIABAAAAyQBFiQBAwAAFiQBAAEAAAAYTQ4AAGUOAABmDgAAdg4AAHoOAACB DgAAhQ4AAIYOAACNDgAAjw4AAJMOAACXDgAAmA4AAKEOAACjDgAApw4AAKsOAACsDgAAsg4AALYO AAC6DgAAvg4AAL8OAADJDgAAyg4AANIOAADTDgAA4Q4AAOIOAADjDgAA+w4AAEAPAACPDwAAtQ8A AOcPAAAuEAAALxAAAFgQAAB3EAAA0BAAAPcQAABFEQAAbBEAAJwRAACwEQAAyhEAAOoRAAD9EQAA ERIAACUSAABLEgAAlhIAALkSAAC7EgAAvBIAAN8SAADxEgAA9xIAAA0TAAArEwAAOhMAAFMTAABz EwAAjxMAANMTAADnEwAA9hMAABIUAAAmFAAAXRQAAIgUAACXFAAAmBQAAK8UAAC5FAAA/xQAAAAV AAA6FQAAOxUAAGcVAABrFQAAbxUAAHMVAAB0FQAAwBUAAMEVAAABFgAAAhYAADUWAAA2FgAANxYA ADgWAAA5FgAARxYAAIQWAACuFgAAGBcAAFEXAACxFwAAyBcAABkYAAD+/v7+/v7+/v7+/v7+/v7+ /v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+ /v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+AAAAAAIBAWS6DgAAvg4AAL8OAADJDgAAyg4AANIO AADTDgAA4Q4AAOIOAADjDgAA+w4AAEAPAACPDwAAtQ8AAOcPAAAuEAAALxAAAFgQAAB3EAAA0BAA APcQAABFEQAA+gAAAAAAAAAAAAAAAN4sAAAAAAAAAAAAAADWAAAAAAAAAAAAAAAAvSQAAAAAAAAA AAAAANYAAAAAAAAAAAAAAAC9PAAAAAAAAAAAAAAA1gAAAAAAAAAAAAAAAL0AAAAAAAAAAAAAAAC7 AAAAAAAAAAAAAAAAuwAAAAAAAAAAAAAAALsAAAAAAAAAAAAAAAC7AAAAAAAAAAAAAAAAuwAAAAAA AAAAAAAAALsAAAAAAAAAAAAAAAC7AAAAAAAAAAAAAAAAuwAAAAAAAAAAAAAAALsAAAAAAAAAAAAA AAC7AAAAAAAAAAAAAAAAuwAAAAAAAAAAAAAAALsAAAAAAAAAAAAAAAC7AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAABAAAAGAAAFiQBFyQBApZsAAXWGAQBAAAEAQAABAEAAAQBAAAEAQAABAEAAAjW BgABlP86CAgAAAMkAgomAAtGBQAWJAEAGwAAFiQBFyQBApZsAAXWGAQBAAAEAQAABAEAAAQBAAAE AQAABAEAAAjWDAAElP86COAQhhksIgAEAAADJAEWJAEAFUURAABsEQAAnBEAALARAADKEQAA6hEA AP0RAAAREgAAJRIAAEsSAACWEgAAuRIAALsSAAC8EgAA3xIAAPESAAD3EgAADRMAACsTAAA6EwAA UxMAAHMTAACPEwAA0xMAAOcTAAD2EwAAEhQAACYUAABdFAAAiBQAAP0AAAAAAAAAAAAAAAD9AAAA AAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAA AAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAPkAAAAAAAAAAAAAAAD5 AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAA AAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAA AAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAA AAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAA AAAAAAD9AAAAAAAAAAAAAAAAAAADAAARhNACAAEAAAAdiBQAAJcUAACYFAAArxQAALkUAAD/FAAA ABUAADoVAAA7FQAAZxUAAGsVAABvFQAAcxUAAHQVAADAFQAAwRUAAAEWAAACFgAANRYAADYWAAA3 FgAAOBYAADkWAABHFgAAhBYAAK4WAAAYFwAAURcAALEXAADIFwAA/QAAAAAAAAAAAAAAAP0AAAAA AAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAA AAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0A AAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAA AAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAA AP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAA AAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAA AAAAAP0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAB3IFwAAGRgAABoYAAAvGAAARRgAAF4YAAB3 GAAAkhgAAKsYAADHGAAA4BgAABgZAAArGQAAPxkAAEAZAABBGQAAQhkAAEMZAABEGQAARRkAAEYZ AABHGQAASBkAAEkZAABKGQAASxkAAEwZAABNGQAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9 AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAA AAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAA AAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAA AAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAA AAAAAAD9AAAAAAAAAAAAAAAA/QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAA /QAAAAAAAAAAAAAAAP0AAAAAAAAAAAAAAAD9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAGxkYAAAaGAAALxgAAEUYAABeGAAAdxgAAJIY AACrGAAAxxgAAOAYAAAYGQAAKxkAAD8ZAABAGQAAQRkAAEIZAABDGQAARBkAAEUZAABGGQAARxkA AEgZAABJGQAAShkAAEsZAABMGQAATRkAAP7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+AAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAQEavhgAAMAYAADXGAAA2RgAAE0ZAAD9AP0AAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADSCoBAAQcAB+w0C8gsOA9IbAIByKwCAcjkKAFJJCgBSWw AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIAEQAKAAEAWwAPAAIAAAAAAAAAJAAAQPH/AgAk AAAABgBOAG8AcgBtAGEAbAAAAAIAAAAEAG1ICQQ0AAFgAQACADQAAAAJAEgAZQBhAGQAaQBuAGcA IAAxAAAACwABAAMkAQYkAUAmAAADADUIgQAAAAAAAAAAAAAAAAAAAAAAPABBQPL/oQA8AAAAFgBE AGUAZgBhAHUAbAB0ACAAUABhAHIAYQBnAHIAYQBwAGgAIABGAG8AbgB0AAAAAAAAAAAAAAAAADQA PkABAPIANAAAAAUAVABpAHQAbABlAAAABQAPAAMkAQASADUIgU9KAwBRSgMAaAgAbkgJBEgAQ0AB AAIBSAAAABAAQgBvAGQAeQAgAFQAZQB4AHQAIABJAG4AZABlAG4AdAAAAAYAEAARhNACDwBPSgMA UUoDAGgIAG5ICQQAAAAAAAEAAAACAAAAAwAAAAQAAAAFAAAABgAAAAcAAAAIAAAACQAAAAoAAAAL AAAATRUAAP////8AAAAAAQD/////AAAAAAAAAAAAAAAAAQAAAAEA/////wAAAAAAAAAAAQAAAAIA AAABAP////8AAAAAAAAAAAIAAAADAAAAAQD/////AAAAAAAAAAADAAAABAAAAAEA/////wAAAAAA AAAABAAAAAUAAAABAP////8AAAAAAAAAAAUAAAAGAAAAAQD/////AAAAAAAAAAAGAAAABwAAAAEA /////wAAAAAAAAAABwAAAAgAAAABAP////8AAAAAAAAAAAgAAAAJAAAAAQD/////AAAAAAAAAAAJ AAAACgAAAAEA/////wAAAAAAAAAACgAAAAsAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAgAAAAMA AAAEAAAABQAAAAYAAAAHAAAACAAAAAkAAAAKAAAACwAAAA4AAAAAAAAAAAgBAAAAAAgCAAAAAAgD AAAAAAgEAAAAAAgFAAAAAAgGAAAAAAgHAAAAAAgIAAAAAAgJAAAAAAgKAAAAAAj//wAAAAAAAAAA TRUAAAgAADYAAAcA/////wAEAAC+GAAATRkAAA0AAAAaAAAAAAQAAIkFAADFBwAAcQoAAKANAAC6 DgAARREAAIgUAADIFwAATRkAAA4AAAAQAAAAEQAAABIAAAATAAAAFQAAABYAAAAXAAAAGAAAAAAE AABNDgAAGRgAAE0ZAAAPAAAAFAAAABkAAAAPAADwOAAAAAAABvAYAAAAAggAAAIAAAARAAAAAQAA AAEAAAASAAAAQAAe8RAAAAD//wAAAAD/AICAgAD3AAAQAA8AAvCSAAAAEAAI8AgAAAABAAAAEQQA AA8AA/AwAAAADwAE8CgAAAABAAnwEAAAAAAAAAAAAAAAAAAAAAAAAAACAArwCAAAAAAEAAAFAAAA DwAE8EIAAAASAArwCAAAAAEEAAAADgAAUwAL8B4AAAC/AQAAEADLAQAAAAD/AQAACAAEAwkAAAA/ AwEAAQAAABHwBAAAAAEAAABNFQAAAAAAAIYAAACKAAAAiwAAAJAAAAC5AAAAvAAAAMQAAADJAAAA 0QAAANUAAADWAAAA2QAAAOMAAADqAAAA8gAAAPkAAAApAQAAMwEAADQBAAA4AQAAOgEAAD0BAABE AQAASAEAAFEBAABWAQAA8QEAAPUBAAAZAgAAHAIAAD4CAABCAgAARwIAAEsCAABhAgAAZQIAAIoC AACNAgAAsgIAALUCAAAfAwAAKQMAACoDAAAuAwAAhQMAAIkDAACKAwAAjQMAAJIDAACXAwAAngMA AKQDAAC4AwAAvAMAAL0DAADAAwAAwQMAAMQDAADFAwAAxwMAAFYEAABaBAAAagQAAG0EAACkBAAA pwQAAMkEAADMBAAAgwUAAIgFAACmBQAAqQUAAL0FAADBBQAA8AUAAPoFAAAHBgAACwYAABQGAAAW BgAAcwYAAHcGAADJBgAAzQYAABYHAAAdBwAATQcAAFAHAADuBwAA8gcAAPwHAAD+BwAAQAgAAEgI AABmCAAAbwgAAJEIAACVCAAAKwkAAC4JAABUCQAAVwkAAKAJAACkCQAAsgkAALQJAADPCQAA0gkA AOEJAADkCQAAMQoAADQKAABWCgAAWwoAAGYKAABpCgAAYwsAAGYLAADZCwAA3gsAAN4NAADiDQAA Gg4AACQOAABRDgAAVA4AAOEOAADrDgAA7A4AAPAOAADzDgAA9g4AAJEPAACUDwAACBAAAAsQAAA7 EAAARBAAAGUQAABpEAAAkRAAAJYQAACYEAAAnRAAAMERAADEEQAAYxIAAGcSAABxEgAAdBIAAH8S AACCEgAAjRIAAJISAACnEgAArBIAAM0SAADQEgAA3RIAAOISAADkEgAA6RIAAOsSAADyEgAAXhMA AGETAABpEwAAbBMAAH8TAACJEwAAkBMAAJcTAACgEwAApxMAANUTAADYEwAA5RMAAOoTAADuEwAA +BMAABIUAAAXFAAAOxUAAD4VAABAFQAAThUAAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAH ABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcA HAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAc AAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwA BwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAH ABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcA HAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAcAAcAHAAHABwABwAH AAAAAADBAQAA2QEAANsBAAAdAgAABAQAAAcEAADyCAAAFgkAAAYMAAAMDAAAbg0AAHENAACQDwAA lQ8AAGAQAABkEAAANxEAADkRAABUEQAAZREAAHARAABxEQAAIxMAACkTAAB4EwAAsBMAAEAVAABO FQAABwAaAAcAGgAHABoABwAaAAcAGgAHABoABwAaAAcAGgAHABoABwAaAAcAGgAHABoABwAEAAcA BwD//xQAAAAMAE0AYQByAGsAIABCAHIAYQBkAGwAZQB5ABIAQQA6AFwASQBQAFMAMAAyADAANAAw ADAALQAxAC4AZABvAGMADABNAGEAcgBrACAAQgByAGEAZABsAGUAeQASAEEAOgBcAEkAUABTADAA MgAwADQAMAAwAC0AMQAuAGQAbwBjAAwATQBhAHIAawAgAEIAcgBhAGQAbABlAHkAIwBDADoAXABN AHkAIABEAG8AYwB1AG0AZQBuAHQAcwBcAEkAUABTAFwASQBQAFMAMAAyADAANAAwADAALQAxAC4A ZABvAGMADABNAGEAcgBrACAAQgByAGEAZABsAGUAeQAjAEMAOgBcAE0AeQAgAEQAbwBjAHUAbQBl AG4AdABzAFwASQBQAFMAXABJAFAAUwAwADIAMAA0ADAAMAAtADEALgBkAG8AYwAMAE0AYQByAGsA IABCAHIAYQBkAGwAZQB5ACMAQwA6AFwATQB5ACAARABvAGMAdQBtAGUAbgB0AHMAXABJAFAAUwBc AEkAUABTADAAMgAwADQAMAAwAC0AMQAuAGQAbwBjAAwATQBhAHIAawAgAEIAcgBhAGQAbABlAHkA NABDADoAXAB3AGkAbgBkAG8AdwBzAFwAVABFAE0AUABcAEEAdQB0AG8AUgBlAGMAbwB2AGUAcgB5 ACAAcwBhAHYAZQAgAG8AZgAgAEkAUABTADAAMgAwADQAMAAwAC0AMQAuAGEAcwBkAAwATQBhAHIA awAgAEIAcgBhAGQAbABlAHkAIwBDADoAXABNAHkAIABEAG8AYwB1AG0AZQBuAHQAcwBcAEkAUABT AFwASQBQAFMAMAAyADAANAAwADAALQAxAC4AZABvAGMADABNAGEAcgBrACAAQgByAGEAZABsAGUA eQASAEEAOgBcAEkAUABTADAAMgAwADQAMAAwAC0AMQAuAGQAbwBjAAwATQBhAHIAawAgAEIAcgBh AGQAbABlAHkAEgBBADoAXABJAFAAUwAwADIAMAA0ADAAMAAtADEALgBkAG8AYwAMAE0AYQByAGsA IABCAHIAYQBkAGwAZQB5ACMAQwA6AFwATQB5ACAARABvAGMAdQBtAGUAbgB0AHMAXABJAFAAUwBc AEkAUABTADAAMgAwADQAMAAwAC0AMQAuAGQAbwBjAAUA/v//////////D/8P/w//D/8P/w//D/8P /w8BAFNlpjwCuu7+/w8AAAAAAAAAAAAAAAAAAAAAAQAuJpVFArru/v8PAAAAAAAAAAAAAAAAAAAA AAEAUkTiXgK67v7/DwAAAAAAAAAAAAAAAAAAAAABABZkg3ICuu7+/w8AAAAAAAAAAAAAAAAAAAAA AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAqAAEAAAAXAAAAAAAAAAAAAAAAAAAAAAAA AAsQAAAPhGgBEYSY/hXGBQABaAEGT0oAAFFKAABvKAABABMgAQAAABcAAAAAAAAAAAAAAAAAAAAA AAAACxAAAA+EaAERhJj+FcYFAAFoAQZPSgAAUUoAAG8oAAEAEyABAAAAFwAAAAAAAAAAAAAAAAAA AAAAAAALEAAAD4RoARGEmP4VxgUAAWgBBk9KAABRSgAAbygAAQATIAEAAAAXAAAAAAAAAAAAAAAA AAAAAAAAAAsQAAAPhGgBEYSY/hXGBQABaAEGT0oAAFFKAABvKAABABMgBQAAAP7///8AAAAAjCnj AQEAAAAuJpVFAAAAAAAAAAAAAAAAFmSDcgAAAAAAAAAAAAAAAFJE4l4AAAAAAAAAAAAAAABTZaY8 AAAAAAAAAAAAAAAA/////5gp4wHg////AAAAABdAAAAAAAAAAAAAAAAAAAAAAAAADwAAAENKGABP SgMAUUoDAG8oAAEAIiD///////////////////////8FAAAAAAAAAAAAAAAAAP9AAYABAKUTAACl EwAA7KrFAAEAAQClEwAAAAAAAKQTAAAAAAAAAhAAAAAAAAAATRUAAIAAAAgAQAAABQAAAEcWkAEA AAICBgMFBAUCAwSHOgAAAAAAAAAAAAAAAAAA/wAAAAAAAABUAGkAbQBlAHMAIABOAGUAdwAgAFIA bwBtAGEAbgAAADUWkAECAAUFAQIBBwYCBQcAAAAAAAAAEAAAAAAAAAAAAAAAgAAAAABTAHkAbQBi AG8AbAAAADMmkAEAAAILBgQCAgICAgSHOgAAAAAAAAAAAAAAAAAA/wAAAAAAAABBAHIAaQBhAGwA AAA/NZABAAACBwMJAgIFAgQEhzoAAAAAAAAAAAAAAAAAAP8AAAAAAAAAQwBvAHUAcgBpAGUAcgAg AE4AZQB3AAAAOwaQAQIABQAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAACAAAAAAFcAaQBuAGcA ZABpAG4AZwBzAAAAIgAEAHAIiBgAANACAABoAQAAAAA3I0KmNyNCpgAAAAACAAEAAAASAwAAhREA AAEACAAAAAQAAxAlAAAAAAAAAAAAAAABAAEAAAABAAAAAAAAABEEAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAKUGwAe0ALQAgAAyMAAAEAAZAGQAAAAZAAAAgxUAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAD/ /xIAAAAAAAAAHwBOAGUAdAB3AG8AcgBrACAASQAvAE8AIABFAHgAcABsAG8AcgBhAHQAbwByAHkA IABNAGUAZQB0AGkAbgBnAAAAAAAAAAwATQBhAHIAawAgAEIAcgBhAGQAbABlAHkADABNAGEAcgBr ACAAQgByAGEAZABsAGUAeQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAA/v8AAAQKAgAAAAAAAAAAAAAAAAAAAAAAAQAAAOCFn/L5T2gQq5EI ACsns9kwAAAAfAEAABAAAAABAAAAiAAAAAIAAACQAAAAAwAAALgAAAAEAAAAxAAAAAUAAADcAAAA BwAAAOgAAAAIAAAA+AAAAAkAAAAQAQAAEgAAABwBAAAKAAAAOAEAAAwAAABEAQAADQAAAFABAAAO AAAAXAEAAA8AAABkAQAAEAAAAGwBAAATAAAAdAEAAAIAAADkBAAAHgAAACAAAABOZXR3b3JrIEkv TyBFeHBsb3JhdG9yeSBNZWV0aW5nAB4AAAABAAAAAGV0dx4AAAANAAAATWFyayBCcmFkbGV5AHhw bB4AAAABAAAAAGFyax4AAAAHAAAATm9ybWFsAGEeAAAADQAAAE1hcmsgQnJhZGxleQB4cGweAAAA AgAAADIAcmseAAAAEwAAAE1pY3Jvc29mdCBXb3JkIDguMAB0QAAAAABGwyMAAAAAQAAAAACCALhJ b78BQAAAAACCALhJb78BAwAAAAEAAAADAAAAEgMAAAMAAACFEQAAAwAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAP7/AAAECgIAAAAAAAAAAAAAAAAAAAAAAAIAAAAC1c3VnC4bEJOXCAArLPmuRAAA AAXVzdWcLhsQk5cIACss+a5gAQAAHAEAAAwAAAABAAAAaAAAAA8AAABwAAAABQAAAJAAAAAGAAAA mAAAABEAAACgAAAAFwAAAKgAAAALAAAAsAAAABAAAAC4AAAAEwAAAMAAAAAWAAAAyAAAAA0AAADQ AAAADAAAAPwAAAACAAAA5AQAAB4AAAAYAAAAQWRhcHRlYywgSW5jLiwgTG9uZ21vbnQAAwAAACUA AAADAAAACAAAAAMAAACDFQAAAwAAAOgQCAALAAAAAAAAAAsAAAAAAAAACwAAAAAAAAALAAAAAAAA AB4QAAABAAAAIAAAAE5ldHdvcmsgSS9PIEV4cGxvcmF0b3J5IE1lZXRpbmcADBAAAAIAAAAeAAAA BgAAAFRpdGxlAAMAAAABAAAAAACYAAAAAwAAAAAAAAAgAAAAAQAAADYAAAACAAAAPgAAAAEAAAAC AAAACgAAAF9QSURfR1VJRAACAAAA5AQAAEEAAABOAAAAewAzAEEANgBFADUANAAxAEIALQA4ADIA MAA2AC0AMQAxAEQAMwAtADkARAA2ADMALQA4ADAARgBCADUANgBDADEAMAA4ADAAMQB9AAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAABAAAAAgAAAAMAAAAEAAAABQAAAAYAAAAHAAAACAAAAAkAAAAKAAAACwAAAAwAAAANAAAADgAA AA8AAAAQAAAAEQAAABIAAAATAAAAFAAAABUAAAAWAAAAFwAAABgAAAAZAAAAGgAAABsAAAD+//// HQAAAB4AAAAfAAAAIAAAACEAAAAiAAAAIwAAACQAAAAlAAAA/v///ycAAAAoAAAAKQAAACoAAAAr AAAALAAAAC0AAAD+////LwAAADAAAAAxAAAAMgAAADMAAAA0AAAANQAAAP7////9////OAAAAP7/ ///+/////v////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// /1IAbwBvAHQAIABFAG4AdAByAHkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAWAAUB//////////8DAAAABgkCAAAAAADAAAAAAAAARgAAAABgnBD6LG+/ASCz2LtJ b78BOgAAAIAAAAAAAAAAMQBUAGEAYgBsAGUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA4AAgD///////////////8AAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAcAAAALxMAAAAAAABXAG8AcgBkAEQAbwBjAHUAbQBlAG4AdAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGgACAQUAAAD//////////wAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAeNgAAAAAAAAUAUwB1AG0AbQBhAHIA eQBJAG4AZgBvAHIAbQBhAHQAaQBvAG4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAoAAIBAgAA AAQAAAD/////AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJgAAAAAQAAAAAAAA BQBEAG8AYwB1AG0AZQBuAHQAUwB1AG0AbQBhAHIAeQBJAG4AZgBvAHIAbQBhAHQAaQBvAG4AAAAA AAAAAAAAADgAAgH///////////////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAuAAAAABAAAAAAAAABAEMAbwBtAHAATwBiAGoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAEgACAQEAAAAGAAAA/////wAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAABqAAAAAAAAAE8AYgBqAGUAYwB0AFAAbwBvAGwAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWAAEA////////////////AAAAAAAA AAAAAAAAAAAAAAAAAAAgs9i7SW+/ASCz2LtJb78BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD///// //////////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAxAgCg9C1HADwAQAAB AAAA/v////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// /////////////////////////////////////////////////////////////////////////wEA /v8DCgAA/////wYJAgAAAAAAwAAAAAAAAEYYAAAATWljcm9zb2Z0IFdvcmQgRG9jdW1lbnQACgAA AE1TV29yZERvYwAQAAAAV29yZC5Eb2N1bWVudC44APQ5snEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA --=====================_88590866==_ Content-Type: text/plain; charset="us-ascii" --=====================_88590866==_-- From XXX Thu Feb 10 17:51:44 2000 Return-Path: bassoon@ece.cmu.edu Delivery-Date: Thu Feb 10 12:50:14 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA24417 for ; Thu, 10 Feb 2000 12:50:13 -0500 Received: from jurassic (JURASSIC.ECE.CMU.EDU [128.2.252.18]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id MAA26435 for ; Thu, 10 Feb 2000 12:50:12 -0500 (EST) Message-Id: <200002101750.MAA26435@ece.cmu.edu> X-Sender: bassoon@pepelepew.ece.cmu.edu (Unverified) X-Mailer: QUALCOMM Windows Eudora Pro Version 4.0.2 Date: Thu, 10 Feb 2000 12:51:47 -0500 To: bassoon@ece.cmu.edu From: David Nagle Subject: Fwd: IP Storage Framework document Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=====================_88660246==_" --=====================_88660246==_ Content-Type: text/plain; charset="us-ascii" >From: "VonStamwitz, Paul" >To: "'Dave_Lee@3com.com'" , > "Bradley, Mark" > , > Carl_Madison@3com.com, Arun_Verma@3com.com, kquick@iphase.com, > bassoon@ece.cmu.edu, "VonStamwitz, Paul" , > "Wilson, Andrew" > , raz@emc.com, > avb@cisco.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com, > sob@harvard.edu, vern@aciri.org >Subject: IP Storage Framework document >Date: Wed, 2 Feb 2000 19:50:07 -0800 >X-Mailer: Internet Mail Service (5.5.2650.21) > >Please find attached the framework document for Friday's meeting. It is >scant on details for reasons discussed on previous emails. I hope it is >enough to provide discussion. > >Paul von Stamwitz >Adaptec, Inc. > > > > --=====================_88660246==_ Content-Type: application/msword; name="IPS Framework.doc" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="IPS Framework.doc" 0M8R4KGxGuEAAAAAAAAAAAAAAAAAAAAAPgADAP7/CQAGAAAAAAAAAAAAAAABAAAAMgAAAAAAAAAA EAAANAAAAAEAAAD+////AAAAADEAAAD///////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// ///////////////////////////////////////////////////////////////////////////s pcEAcQAJBAAAABK/AAAAAAAAEAAAAAAABAAAUx4AAA4AYmpianQrdCsAAAAAAAAAAAAAAAAAAAAA AAAJBBYAHjAAABZBAQAWQQEAUxoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD//w8AAAAA AAAAAAD//w8AAAAAAAAAAAD//w8AAAAAAAAAAAAAAAAAAAAAAF0AAAAAAHABAAAAAAAAcAEAAHAB AAAAAAAAcAEAAAAAAABwAQAAAAAAAHABAAAAAAAAcAEAABQAAAAAAAAAAAAAAIQBAAAAAAAAhAEA AAAAAACEAQAAAAAAAIQBAAAAAAAAhAEAAAwAAACQAQAAHAAAAIQBAAAAAAAAswoAALYAAADIAQAA AAAAAMgBAAAAAAAAyAEAAAAAAADIAQAAAAAAAMgBAAAAAAAAyAEAAAAAAADIAQAAAAAAAMgBAAAA AAAAeAoAAAIAAAB6CgAAAAAAAHoKAAAAAAAAegoAAAAAAAB6CgAAAAAAAHoKAAAAAAAAegoAACQA AABpCwAA9AEAAF0NAABwAAAAngoAABUAAAAAAAAAAAAAAAAAAAAAAAAAcAEAAAAAAADIAQAAAAAA AAAAAAAAAAAAAAAAAAAAAADIAQAAAAAAAMgBAAAAAAAAyAEAAAAAAADIAQAAAAAAAJ4KAAAAAAAA GAIAAAAAAABwAQAAAAAAAHABAAAAAAAAyAEAAAAAAAAAAAAAAAAAAMgBAAAAAAAAyAEAAAAAAAAY AgAAAAAAABgCAAAAAAAAGAIAAAAAAADIAQAAIgAAAHABAAAAAAAAyAEAAAAAAABwAQAAAAAAAMgB AAAAAAAAeAoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAhAEAAAAAAACEAQAAAAAAAHABAAAAAAAAcAEA AAAAAABwAQAAAAAAAHABAAAAAAAAyAEAAAAAAAB4CgAAAAAAABgCAAA6BQAAGAIAAAAAAABSBwAA OgAAAEIKAAAsAAAAcAEAAAAAAABwAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAeAoAAAAAAADIAQAAAAAAAKwBAAAcAAAA4KO0mflt vwGEAQAAAAAAAIQBAAAAAAAA6gEAAC4AAABuCgAACgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQWJz dHJhY3QNDVRoaXMgZG9jdW1lbnQgcHJvdmlkZXMgYSBnZW5lcmFsIGRlc2NyaXB0aW9uIG9mIGhv dyB0byBkZWxpdmVyIGVuaGFuY2VtZW50cyB0byBleGlzdGluZyBuZXR3b3JraW5nIHRlY2hub2xv Z3kgZm9yIHN0b3JhZ2Ugc29sdXRpb25zLg0NU3RydWN0dXJlIG9mIHRoaXMgRHJhZnQNDVRCRA0N SVAgU3RvcmFnZSCWIE1vdGl2YXRpb24NDVJlY2VudGx5LCB0aGUgdHJlbmQgdG93YXJkIG1lcmdp bmcgbmV0d29ya2luZyBhbmQgc3RvcmFnZSBpbnRlcmNvbm5lY3QgdGVjaG5vbG9naWVzIGhhcyBh Y2NlbGVyYXRlZC4gVGhpcyBtZXJnaW5nIGhhcyBnaXZlbiByaXNlIHRvIG5ldyB0ZWNobm9sb2dp ZXMgc3VjaCBhcyBTdG9yYWdlIEFyZWEgTmV0d29ya3MgKFNBTnMpIGFuZCBOZXR3b3JrIEF0dGFj aGVkIFN0b3JhZ2UgKE5BUy4pIE90aGVyIHRlY2hub2xvZ2llcywgc3VjaCBhcyBPYmplY3QgQmFz ZWQgU3RvcmFnZSBEZXZpY2UgKE9CU0QpIGFyZSBiZWluZyBkaXNjdXNzZWQuIER1ZSB0byB0aGUg aW5jcmVhc2VkIHVzYWdlIG9mIHRoZSBJbnRlcm5ldCwgc3RvcmFnZSBkZW1hbmRzIHdpbGwgY29u dGludWUgdG8gaW5jcmVhc2UuIEluIGxpZ2h0IG9mIHRoaXMsIGl0IGlzIHJlYXNvbmFibGUgdG8g YXNzdW1lIHRoYXQgdGhlIG1lcmdpbmcgb2YgbmV0d29ya2luZyBhbmQgc3RvcmFnZSB3aWxsIGNv bnRpbnVlLiBXaGlsZSBjdXJyZW50IFNBTiBpbXBsZW1lbnRhdGlvbnMgaGF2ZSBhcHBsaWVkIG5l dHdvcmtpbmcgY29uY2VwdHMgdG8gc3RvcmFnZSB0ZWNobm9sb2d5LCBhbm90aGVyIHBlcnNwZWN0 aXZlIGlzIHRvIGFwcGx5IHN0b3JhZ2UgcHJpbmNpcGxlcyBhbmQgY29uY2VwdHMgdG8gZXhpc3Rp bmcgbmV0d29ya2luZyB0ZWNobm9sb2d5Lg0NVGhlIElQIFN0b3JhZ2UgYXJjaGl0ZWN0dXJlIG9m ZmVycyBhIGZyYW1ld29yayB3aXRoaW4gd2hpY2ggZXhpc3RpbmcgbmV0d29yayBpbmZyYXN0cnVj dHVyZXMgY2FuIGJlIHVzZWQgYXMgYSBoaWdoLXF1YWxpdHkgc3RvcmFnZSBzdWJzeXN0ZW0gaW50 ZXJjb25uZWN0LiBUaGUgZ29hbCBpcyB0byBwcm92aWRlIGN1cnJlbnQgYW5kIGZ1dHVyZSBzdG9y YWdlIHJlcXVpcmVtZW50cyB3aGlsZSBsZXZlcmFnaW5nIHRoZSBicm9hZCBpbnN0YWxsZWQgYmFz ZSBhcyB3ZWxsIGFzIHRoZSBpbmhlcmVudCBtYW5hZ2VhYmlsaXR5LCBzY2FsYWJpbGl0eSwgYW5k IGF2YWlsYWJpbGl0eSBvZiB0aGUgbmV0d29yay4NDVJlcXVpcmVtZW50cw0NTG93IGxhdGVuY3ks IG9uIHRoZSBvcmRlciBvZiBhIGNvdXBsZSBodW5kcmVkIG1pY3Jvc2Vjb25kcyBvciBsZXNzDQ1U aGlzIHJlcXVpcmVtZW50IGlzIGVhc2lseSBtZXQgdXNpbmcgdG9kYXmScyBFdGhlcm5ldCBzd2l0 Y2hlcy4gVHlwaWNhbCB2YWx1ZXMgZm9yIGxhdGVuY3kgYXJlIDMtMjAgbWljcm9zZWNvbmRzIGZv ciBjb21tZXJjaWFsbHkgYXZhaWxhYmxlIEdpZ2FiaXQgRXRoZXJuZXQgc3dpdGNoZXMuIEV2ZW4g d2l0aCBzZXZlcmFsIHN3aXRjaGVzIGFuZCBhIGNvdXBsZSBodW5kcmVkIG1ldGVycyBvZiBjYWJs ZSwgdGhlIHJvdW5kIHRyaXAgZGVsYXkgd2lsbCBiZSBsZXNzIHRoYW4gMTAwIG1pY3Jvc2Vjb25k cyBpbiB0aGUgYWJzZW5jZSBvZiBjb25nZXN0aW9uLg0NRGlzdGFuY2VzIG9mIGEgZmV3IGh1bmRy ZWQgbWV0ZXJzDQ1UaGUgZnVsbCBkdXBsZXgsIGZpYmVyIG9wdGljIHZlcnNpb24gb2YgR2lnYWJp dCBFdGhlcm5ldCB3aWxsIGVhc2lseSBzcGFuIHNldmVyYWwga2lsb21ldGVycywgd2hpbGUgdGhl IGNvcHBlciB2ZXJzaW9uIHdpbGwgc3BhbiAxMDAgbWV0ZXJzLiBUaGVzZSBkaXN0YW5jZXMgc2hv dWxkIGJlIG1vcmUgdGhhbiBhZGVxdWF0ZSBmb3IgbW9zdCBzdG9yYWdlIGFwcGxpY2F0aW9ucy4g SG93ZXZlciwgc29tZSBhcHBsaWNhdGlvbnMgbWF5IHJlcXVpcmUgZ3JlYXRlciBkaXN0YW5jZXMu DQ1OZWVkcyB0byBzdXBwb3J0IGJhbmR3aWR0aCBvZiBkcml2ZXMgZm9yIGF0IGxlYXN0IHRoZSBu ZXh0IDUtMTAgeWVhcnMNDUJhbmR3aWR0aCByZXF1aXJlbWVudHMgYXJlIGRpZmZpY3VsdCB0byBw cmVkaWN0LCBzaW5jZSB0aGV5IGRlcGVuZCBvbiBhZHZhbmNlcyBpbiByZWNvcmRpbmcgZGVuc2l0 eSBhbmQgbWVjaGFuaWNhbCBzcGVlZCBvZiB0aGUgZGlzayBkcml2ZXMsIGFzIHdlbGwgYXMgdGhl IGFjY2VzcyBwYXR0ZXJucyBvZiBmdXR1cmUgYXBwbGljYXRpb25zLiAgUHJvamVjdGVkIGFjY2Vz cyB0aW1lcyBhbmQgc2VxdWVudGlhbCBhY2Nlc3MgcmF0ZXMgZm9yIGRpc2sgZHJpdmVzLCBhc3N1 bWluZyBoaXN0b3JpY2FsIGFubnVhbCBpbXByb3ZlbWVudCByYXRlcywgd2lsbCBhcHByb2FjaCAz IG1pbGxpc2Vjb25kcyBhbmQgMTAwIG1lZ2FieXRlcy9zZWNvbmQgcmVzcGVjdGl2ZWx5IHdpdGhp biB0aGUgbmV4dCA1IHllYXJzLiBGb3IgYXBwbGljYXRpb25zIHdoaWNoIGFjY2VzcyBkYXRhIHNl cXVlbnRpYWxseSwgb3IgaW4gdmVyeSBsYXJnZSByYW5kb20gcmVhZHMsIHRoZSBwcm9qZWN0aW9u cyBzaG93IHRoYXQgYSAxMCBnaWdhYml0IGxpbmsgd2lsbCBiZSByZXF1aXJlZC4gT24gdGhlIG90 aGVyIGhhbmQsIHRyYW5zYWN0aW9uIHByb2Nlc3NpbmcgYXBwbGljYXRpb25zIHdpbGwgc3RpbGwg YmUgbGltaXRlZCBieSBkaXNrIGFjY2VzcyB0aW1lIHRvIGFib3V0IDM1MCBJT1BTIHBlciBkcml2 ZSwgb3Igb25lIHRvIGZpdmUgbWVnYWJ5dGVzIGEgc2Vjb25kIGRlcGVuZGluZyBvbiByZXF1ZXN0 IHNpemUuIEV2ZW4gYXQgZml2ZSBtZWdhYnl0ZXMgYSBzZWNvbmQsIG92ZXIgMjAgZHJpdmVzIGNv dWxkIGJlIHN1cHBvcnRlZCBieSBhIHNpbmdsZSBnaWdhYml0IGxpbmsuDQ1DUFUgdXRpbGl6YXRp b24gb24gdGhlIG9yZGVyIG9mIGN1cnJlbnQgc3RvcmFnZSBzeXN0ZW1zDQ1JbiBvcmRlciB0byBl cXVhbCB0aGUgbG93IENQVSB1dGlsaXphdGlvbiBvZiBjdXJyZW50IGhvc3QgYWRhcHRlcnMsIHRo ZSB0cmFuc3BvcnQgcHJvdG9jb2wgd2lsbCBoYXZlIHRvIHJ1biBvbmJvYXJkIHRoZSBob3N0IGFk YXB0ZXIsIGlkZWFsbHkgd2l0aCBzcGVjaWFsIHB1cnBvc2UgaGFyZHdhcmUuIFRvIGtlZXAgdGhl IGhhcmR3YXJlIHNpbXBsZSBhbmQgaW5leHBlbnNpdmUsIGEgcHJvdG9jb2wgd2l0aCBsZXNzIGNv bXBsZXhpdHkgdGhhbiBUQ1AgbWF5IGJlIG5lY2Vzc2FyeS4NDURhdGEgbXVzdCBiZSBkZWxpdmVy ZWQgY29tcGxldGVseSBhbmQgd2l0aG91dCBjb3JydXB0aW9uDQ1UaGUgYmFzaWMgRXRoZXJuZXQg cHJvdG9jb2wgaXMgdGhhdCBvZiBhbiB1bnJlbGlhYmxlIERhdGFncmFtLCBidXQgYWRkaXRpb24g b2YgYW55IG9mIGEgbnVtYmVyIG9mIHRyYW5zcG9ydCBsYXllciBwcm90b2NvbHMgY2FuIHByb2R1 Y2UgYSByZWxpYWJsZSBkYXRhIGNvbmR1aXQgcXVpdGUgc3VpdGFibGUgZm9yIHN0b3JhZ2UgYXBw bGljYXRpb25zLiBUaGVyZWZvcmUsIEV0aGVybmV0IGFuZCBvdGhlciBuZXR3b3JrIHRlY2hub2xv Z2llcyBhcmUgcmVsaWFibGUgZW5vdWdoIGZvciBzdG9yYWdlIGludGVyY29ubmVjdHMuDQ1BYmxl IHRvIHVzZSBvZmYgdGhlIHNoZWxmIEV0aGVybmV0IHN3aXRjaGVzIGFuZCBJUCByb3V0ZXJzDQ1J biBzaW1wbGUgY29uZmlndXJhdGlvbnMsIGxldmVsIDIgc3dpdGNoZXMgd2lsbCBiZSBhZGVxdWF0 ZSBhbmQgY29zdCBlZmZlY3RpdmUuIEZvciBtb3JlIGNvbXBsZXggZW52aXJvbm1lbnRzIGFuZCB3 aGVyZSB0aGUgbWFuYWdlbWVudCBuZWVkcyByZXF1aXJlIHRoZSBmZWF0dXJlcyBvZiBJUCwgbGV2 ZWwgMyBzd2l0Y2hlcyBhbmQgcm91dGVycyB3aWxsIGJlIG5lY2Vzc2FyeS4NDU5lZWRzIGVhc3kg bWFuYWdlbWVudCBieSBzeXN0ZW0gYWRtaW5pc3RyYXRvcg0NU21hbGwgc3lzdGVtcyB3aWxsIHJl cXVpcmUgYXV0b21hdGljIGNvbmZpZ3VyYXRpb24gdGhhdCBjYW4gYmUgc2F0aXNmaWVkIGJ5IGFu IGF1dG9tYXRpYyBkaXNjb3ZlcnkgcHJvdG9jb2wgaW5kZXBlbmRlbnQgb2YgSVAuIExhcmdlciBz eXN0ZW1zIHdpbGwgaGF2ZSBzeXN0ZW0gYWRtaW5pc3RyYXRvcnMgd2hvIHdpbGwgd2FudCB0byBt YW5hZ2UgY2VydGFpbiBhc3BlY3RzIG9mIHRoZSBzdG9yYWdlIG5ldHdvcmssIHdoaWNoIGNhbiBi ZSBkb25lIHRocm91Z2ggdXNlIG9mIElQIGFuZCBleGlzdGluZyBJUCBiYXNlZCB0b29scy4gDQ1O ZWVkcyB0byBiZSBjb3N0IGNvbXBldGl0aXZlIHdpdGggb3RoZXIgc3RvcmFnZSBuZXR3b3JrIHRl Y2hub2xvZ2llcw0NVGhlIHVzZSBvZiBvZmYtdGhlLXNoZWxmIEV0aGVybmV0IHN3aXRjaGVzIGFu ZCB3aXJpbmcgd2lsbCBwcm9kdWNlIHN1YnN0YW50aWFsIGNvc3Qgc2F2aW5ncyBkdWUgdG8gdGhl IGxhcmdlIHZvbHVtZXMgYXNzb2NpYXRlZCB3aXRoIHRoZSB1c2Ugb2Ygc3VjaCBlcXVpcG1lbnQg Zm9yIG5ldHdvcmtzLg0NSm9pbnRseSBvcGVyYXRlIHdpdGggb3RoZXIgcHJvdG9jb2xzDQ1PbmUg Y2FuIGVhc2lseSBlbnZpc2lvbiBhIHNpbmdsZSBpbnRlcmNvbm5lY3QgdGVjaG5vbG9neSBwcm92 aWRpbmcgZm9yIHRoZSBuZWVkcyBvZiBib3RoIHN0b3JhZ2UgYW5kIG5ldHdvcmtpbmcuIFRoZXJl Zm9yZSwgYW55IHByb3RvY29scyB1c2VkIGZvciBJUCBTdG9yYWdlIG11c3QgZ3JhY2VmdWxseSBj b2V4aXN0IHdpdGggb3RoZXIgcHJvdG9jb2xzLiBJUCBzZXJ2aWNlcyBtYXkgYmUgbmVlZGVkIHRv IHByb3ZpZGUgZm9yIGJhbmR3aWR0aCBtYW5hZ2VtZW50Lg0NS2V5IElzc3Vlcw0NVHJhbnNwb3J0 DQ1UaGUgdHJhbnNwb3J0IHByb3RvY29sIGZvciBJUCBzdG9yYWdlIHNob3VsZCBtZWV0IHRoZSBy ZXF1aXJlbWVudHMgZGV0YWlsZWQgYWJvdmUuIEl0IHNob3VsZCBtYXhpbWl6ZSBhdmFpbGFibGUg YmFuZHdpZHRoIHdoaWxlIG1pbmltaXppbmcgbGF0ZW5jeS4gIEl0IHNob3VsZCBiZSBlYXN5IHRv IGVtYmVkIGluIGhhcmR3YXJlIHRvIGJlIGNvbXBldGl0aXZlIHdpdGggb3RoZXIgc3RvcmFnZSBu ZXR3b3JraW5nIHRlY2hub2xvZ2llcywgYm90aCBpbiBjb3N0IGFuZCBDUFUgdXRpbGl6YXRpb24u DQ1UaGUgdHJhbnNwb3J0IHByb3RvY29sIHNob3VsZCBiZSBhYmxlIHRvIHRha2UgYWR2YW50YWdl IG9mIHRoZSBzZXJ2aWNlcyB0aGF0IElQIHByb3ZpZGVzLiBIb3dldmVyLCBpdCBzaG91bGQgYmUg YWJsZSB0byBiZSBvcHRpb25hbGx5IGNvbmZpZ3VyZWQgdG8gb3BlcmF0ZSB3aXRob3V0IElQLg0N TWFwcGluZyBvZiBzdG9yYWdlIHByb3RvY29sIG9udG8gdHJhbnNwb3J0DQ1UaGUgc3RvcmFnZSBw cm90b2NvbCBzaG91bGQgYmUgaW5kZXBlbmRlbnQgZnJvbSB0aGUgdHJhbnNwb3J0IHByb3RvY29s LiBUaGUgYXJjaGl0ZWN0dXJlIHNob3VsZCBhbGxvdyBmb3IgbXVsdGlwbGUgcHJvdG9jb2xzIHRv IGJlIG1hcHBlZC4gVGhpcyB3aWxsIGFsbG93IGZvciB0aGUgZmxleGliaWxpdHkgb2YgcnVubmlu ZyBtdWx0aXBsZSBvciBkaWZmZXJlbnQgc3RvcmFnZSBwcm90b2NvbHMgb24gdG9wIG9mIHRoZSB0 cmFuc3BvcnQuIEl0IGFsc28gd2lsbCBub3QgcHJlY2x1ZGUgb3RoZXIgbm9uLXN0b3JhZ2UgcHJv dG9jb2xzIGZyb20gdGFraW5nIGFkdmFudGFnZSBvZiB0aGUgbG93LWxhdGVuY3ksIGhpZ2gtYmFu ZHdpZHRoIGF0dHJpYnV0ZXMgb2YgdGhlIHRyYW5zcG9ydC4NDURldmljZSBEaXNjb3ZlcnkNDVVu bGlrZSBjb21tdW5pY2F0aW9uIG5ldHdvcmtzLCB3aGljaCBhc3N1bWUgdGhhdCBhbGwgYXR0YWNo ZWQgZGV2aWNlcyBhcmUgcGVlcnMgYW5kIHRoYXQgYW55IGdpdmVuIGRldmljZSB3aWxsIHdhbnQg dG8gY29tbXVuaWNhdGUgd2l0aCBvbmx5IGEgc21hbGwgc3Vic2V0IG9mIG90aGVyIGRldmljZXMs IHN0b3JhZ2UgbmV0d29ya3MgYXJlIGhpZXJhcmNoaWNhbCwgd2l0aCBhIGZldyBJbml0aWF0b3Jz IGluIGNvbW11bmljYXRpb24gd2l0aCBtYW55IHRhcmdldHMsIGFuZCB3aXRoIHRhcmdldHMgaW4g Y29tbXVuaWNhdGlvbiBvbmx5IHdpdGggaW5pdGlhdG9ycywgb3Igd2l0aCBvdGhlciB0YXJnZXRz IHVuZGVyIHRoZSBjb250cm9sIG9mIGFuIGluaXRpYXRvci4gVGhpcyBzdG9yYWdlIHZpZXdwb2lu dCByZXF1aXJlcyB0aGF0IGluaXRpYXRvcnMgYmUgYWJsZSB0byBsb2NhdGUgYWxsIHRoZSB0YXJn ZXRzLCBhbmQgYmUgYWJsZSB0byBjb21tdW5pY2F0ZSB3aXRoIG1hbnkgb3IgYWxsIG9mIHRoZW0u DQ1TdG9yYWdlIHN5c3RlbXMgdHlwaWNhbGx5IHVzZSBzaW1wbGUgZGlzY292ZXJ5IHByb3RvY29s cyBkZXNpZ25lZCB0byBiZSBpbXBsZW1lbnRlZCBpbiBoYXJkd2FyZSwgYXQgbGVhc3Qgb24gdGhl IHRhcmdldCBlbmQsIHdoZXJlIGNvc3QgaXMgYSBtYWpvciBjb25jZXJuLiBUaHVzLCB0aGVyZSBt YXkgYmUgYSBuZWVkIHRvIGRldmVsb3AgYSBzdG9yYWdlIGNlbnRyaWMgcHJvdG9jb2wgZm9yIElQ IFN0b3JhZ2UsIHdoaWNoIHRha2VzIGludG8gYWNjb3VudCB0aGUgYmVoYXZpb3Igb2YgRXRoZXJu ZXQsIGJ1dCBzdGlsbCBrZWVwcyB0aGUgdGFyZ2V0knMgcm9sZSBzaW1wbGUgZW5vdWdoIHRvIGJl IGltcGxlbWVudGVkIGRpcmVjdGx5IGluIGhhcmR3YXJlLiBJdCBzaG91bGQgYmUgc2NhbGFibGUg dXAgdG8gc3lzdGVtcyBvZiBodW5kcmVkcyBvZiBpbml0aWF0b3JzIGFuZCB0aG91c2FuZHMgb2Yg dGFyZ2V0cy4NDVF1YWxpdHkgb2Ygc2VydmljZQ0NSVAgc2hvdWxkIGJlIGxldmVyYWdlZCBhcyBt dWNoIGFzIHBvc3NpYmxlIHRvIHByb3ZpZGUgdGhlIG5lY2Vzc2FyeSBzZXJ2aWNlcy4NDVNlY3Vy aXR5DQ1UQkQNDQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAkEAAAKBAAAjwQAAJAE AACpBAAArQQAAIMKAACWCwAAUQ8AAGYQAAChEAAAzBEAAAkSAADiEgAAEhMAAEIUAABDFAAAiRQA ADYVAABdFQAAaBYAAIAWAABFGAAADBoAABAcAADfHQAA4B0AAE4eAABSHgAAUx4AAP328fb9AP3x /fH98f3x/fEA/fH98f0A/fH9AP0A/QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAIT0oAAFFKAAAADENKGABPSgIAUUoCAAAEQ0oYAB4ABAAACQQAAAoEAACPBAAAkAQA AKgEAACpBAAArQQAAK4EAADGBAAAxwQAAHQHAAB1BwAAzggAAM8IAADcCAAA3QgAACAJAAAhCQAA XwoAAGAKAACCCgAAgwoAAJYLAACXCwAA3QsAAN4LAAD6AAAAAAAAAAAAAAAA9gAAAAAAAAAAAAAA APQAAAAAAAAAAAAAAAD2AAAAAAAAAAAAAAAA+gAAAAAAAAAAAAAAAPIAAAAAAAAAAAAAAADwAAAA AAAAAAAAAAAA9gAAAAAAAAAAAAAAAPoAAAAAAAAAAAAAAADyAAAAAAAAAAAAAAAA9gAAAAAAAAAA AAAAAPYAAAAAAAAAAAAAAAD2AAAAAAAAAAAAAAAA9gAAAAAAAAAAAAAAAPoAAAAAAAAAAAAAAADy AAAAAAAAAAAAAAAA6wAAAAAAAAAAAAAAAPIAAAAAAAAAAAAAAAD2AAAAAAAAAAAAAAAA8gAAAAAA AAAAAAAAAOsAAAAAAAAAAAAAAADyAAAAAAAAAAAAAAAA5wAAAAAAAAAAAAAAAPIAAAAAAAAAAAAA AADrAAAAAAAAAAAAAAAA8gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAw8AD4RoAQUAAAom AQtGAgAAAQEAAAEAAAABEAAAAwAAD4RoAQUAAAomAAtGAgAAGgAEAAAJBAAACgQAAI8EAACQBAAA qAQAAKkEAACtBAAArgQAAMYEAADHBAAAdAcAAHUHAADOCAAAzwgAANwIAADdCAAAIAkAACEJAABf CgAAYAoAAIIKAACDCgAAlgsAAJcLAADdCwAA3gsAABcPAAAYDwAAUA8AAFEPAABmEAAAZxAAAKAQ AAChEAAAzBEAAM0RAAD8APoA9ADx7ubj4N3a18/Mw8C9uq6rpqOXlJGOgn96d2toY2AAAAAAAAAA AAUG3fL//wgCDwAGCPT//wAFBgn0//8WBkL0//8HAQgCAAkBCgQAAAALaPj//wAFBkP0//8IAhAA Blj1//8ABQZZ9f//FgaR9f//BwEIAgAJAQoDAAAAC7f5//8ABQaS9f//BQbL+P//BQbM+P//FgYS +f//BwEIAgAJAQoCAAAACzj9//8ABQYT+f//CAIPAAYm+v//AAUGJ/r//xYGSfr//wcBCAIACQEK AQAAAAtv/v//AAUGSvr//wUGiPv//wUGifv//xEGzPv//wcBCAIACQEL8v///wUGzfv//w8G2vv/ /wgCAAkBCgMAAAAFBtv7//8FBjT9//8FBjX9//8FBuL///8FBuP///8PBvv///8IAgAJAQoCAAAA BQb8////BQIBAAUACggCAAkBCgEAAAAAAwIQAAUIAgAJAQAk3gsAABcPAAAYDwAAUA8AAFEPAABm EAAAZxAAAKAQAAChEAAAzBEAAM0RAAAIEgAACRIAAOISAADjEgAAERMAABITAABCFAAAQxQAAIgU AACJFAAANhUAADcVAABcFQAAXRUAAGgWAABpFgAA+wAAAAAAAAAAAAAAAPkAAAAAAAAAAAAAAAD0 AAAAAAAAAAAAAAAA+QAAAAAAAAAAAAAAAO4AAAAAAAAAAAAAAAD5AAAAAAAAAAAAAAAA9AAAAAAA AAAAAAAAAPkAAAAAAAAAAAAAAADqAAAAAAAAAAAAAAAA+QAAAAAAAAAAAAAAAPQAAAAAAAAAAAAA AAD5AAAAAAAAAAAAAAAA6AAAAAAAAAAAAAAAAPkAAAAAAAAAAAAAAAD0AAAAAAAAAAAAAAAA+QAA AAAAAAAAAAAAAOgAAAAAAAAAAAAAAAD5AAAAAAAAAAAAAAAA9AAAAAAAAAAAAAAAAPkAAAAAAAAA AAAAAADoAAAAAAAAAAAAAAAA+QAAAAAAAAAAAAAAAPQAAAAAAAAAAAAAAAD5AAAAAAAAAAAAAAAA 6AAAAAAAAAAAAAAAAPkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAARAAAAMPAA+EaAEA BRAADcYFAAEOAQAFAAAKJgELRgIAAAEAAAADAAAPhGgBABrNEQAACBIAAAkSAADiEgAA4xIAABET AAASEwAAQhQAAEMUAACIFAAAiRQAADYVAAA3FQAAXBUAAF0VAABoFgAAaRYAAHQWAAB1FgAAfxYA AIAWAACcFwAAnRcAAEUYAABGGAAAcRgAAHIYAAD5GQAA+hkAAAsaAAAMGgAAEBwAABEcAADgHQAA 9PHr6NzZ1NHFwr26rqumo5uYj4yJiYmGend0cWViXVpWAAAABwQABoz7//8FBo37//8IAhAABpH9 //8ABQaS/f//Fgaj/f//BwEIAgAJAQoCAAAAC2/8//8ABQak/f//BQYr////BQYs////FgZX//// BwEIAgAJAQoBAAAACyP+//8ABQZY////BQIBAAUABQYq7v//EQY07v//BwEIAgAJAQv0////BQY1 7v//DwZA7v//CAIACQEKBAAAAAUGQe7//wgCEAAGTO///wAFBk3v//8WBnLv//8HAQgCAAkBCggA AAALmPP//wAFBnPv//8IAhAABiDw//8ABQYh8P//FgZm8P//BwEIAgAJAQoHAAAAC4z0//8ABQZn 8P//CAIQAAaX8f//AAUGmPH//xYGxvH//wcBCAIACQEKBgAAAAvs9f//AAUGx/H//woCEAAEAAag 8v//AAUGofL//xYG3PL//wcBCAIACQEKBQAAAAsC9///IWkWAAB0FgAAdRYAAH8WAACAFgAAnBcA AJ0XAABFGAAARhgAAHEYAAByGAAA+RkAAPoZAAALGgAADBoAABAcAAARHAAA4B0AAOEdAAD0HQAA 9R0AAEMeAABEHgAATR4AAE4eAABSHgAAUx4AAPoAAAAAAAAAAAAAAAD4AAAAAAAAAAAAAAAA8wAA AAAAAAAAAAAAAPgAAAAAAAAAAAAAAADxAAAAAAAAAAAAAAAA8QAAAAAAAAAAAAAAAPEAAAAAAAAA AAAAAAD4AAAAAAAAAAAAAAAA8wAAAAAAAAAAAAAAAPgAAAAAAAAAAAAAAADtAAAAAAAAAAAAAAAA +AAAAAAAAAAAAAAAAPMAAAAAAAAAAAAAAAD4AAAAAAAAAAAAAAAA6wAAAAAAAAAAAAAAAO0AAAAA AAAAAAAAAADtAAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAPMAAAAAAAAAAAAAAAD4AAAAAAAAAAAA AAAA7QAAAAAAAAAAAAAAAPgAAAAAAAAAAAAAAADzAAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAPEA AAAAAAAAAAAAAAD4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEQAAADAAAP hGgBAAEBAAUAAAomAQtGAgAAAQAABQAACiYAC0YCAAAa4B0AAOEdAAD0HQAA9R0AAEMeAABEHgAA TR4AAE4eAABSHgAAUx4AAPzw7ern29jV0gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQYK AAAABQIBAAUABQb4+f//FgZZ+f//BwEIAgAJAQoEAAAACyX4//8ABQZa+f//BQao+f//BQap+f// Fga8+f//BwEIAgAJAQoDAAAAC4j4//8ABQa9+f//AAkcAB+w0C8gsOA9IbAIByKwCAcjkKAFJJCg BSWwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIAEQAKAAEAWwAPAAIAAAAAAAAAJAAAQPH/ AgAkAAAABgBOAG8AcgBtAGEAbAAAAAIAAAAEAG1ICQQ0AAFAAQACADQAAAAJAEgAZQBhAGQAaQBu AGcAIAAxAAAADAABAAYkAQ+EaAFAJgAEAENKGAAwAAJAAQACADAAAAAJAEgAZQBhAGQAaQBuAGcA IAAyAAAACAACAAYkAUAmAQMANQiBAAAAAAAAAAAAAAAAAAAAPABBQPL/oQA8AAAAFgBEAGUAZgBh AHUAbAB0ACAAUABhAHIAYQBnAHIAYQBwAGgAIABGAG8AbgB0AAAAAAAAAAAAAAAAADIAQkABAPIA MgAAAAkAQgBvAGQAeQAgAFQAZQB4AHQAAAACAA8ADABDShgAT0oCAFFKAgBEAENAAQACAUQAAAAQ AEIAbwBkAHkAIABUAGUAeAB0ACAASQBuAGQAZQBuAHQAAAAGABAAD4RoAQwAQ0oYAE9KAgBRSgIA AAAAAFMaAAAUAAAwAAAQAP////8ABAAAUx4AABEAAAAABAAA3gsAAGkWAABTHgAAEgAAABQAAAAW AAAAAAQAAM0RAADgHQAAUx4AABMAAAAVAAAAFwAAAAAAAAB7AQAAfwEAANYMAADeDAAAVRoAAAcA HAAHABwABwAAAAAA3QkAAOcJAAAMFgAAhxcAAEoYAABkGAAAVRoAAAcAGgAHABoABwAaAAcA//8U AAAAEQBQAGEAdQBsACAAdgBvAG4AIABTAHQAYQBtAHcAaQB0AHoALABDADoAXABNAHkAIABEAG8A YwB1AG0AZQBuAHQAcwBcAEUAdABoAGUAcgBTAEMAUwBJAFwAaQBlAHQAZgBcAEYAcgBhAG0AZQB3 AG8AcgBrAC4AZABvAGMAEQBQAGEAdQBsACAAdgBvAG4AIABTAHQAYQBtAHcAaQB0AHoAMgBDADoA XABXAEkATgBEAE8AVwBTAFwAVABFAE0AUABcAEEAdQB0AG8AUgBlAGMAbwB2AGUAcgB5ACAAcwBh AHYAZQAgAG8AZgAgAEYAcgBhAG0AZQB3AG8AcgBrAC4AYQBzAGQAEQBQAGEAdQBsACAAdgBvAG4A IABTAHQAYQBtAHcAaQB0AHoAMgBDADoAXABXAEkATgBEAE8AVwBTAFwAVABFAE0AUABcAEEAdQB0 AG8AUgBlAGMAbwB2AGUAcgB5ACAAcwBhAHYAZQAgAG8AZgAgAEYAcgBhAG0AZQB3AG8AcgBrAC4A YQBzAGQAEQBQAGEAdQBsACAAdgBvAG4AIABTAHQAYQBtAHcAaQB0AHoAMgBDADoAXABXAEkATgBE AE8AVwBTAFwAVABFAE0AUABcAEEAdQB0AG8AUgBlAGMAbwB2AGUAcgB5ACAAcwBhAHYAZQAgAG8A ZgAgAEYAcgBhAG0AZQB3AG8AcgBrAC4AYQBzAGQAEQBQAGEAdQBsACAAdgBvAG4AIABTAHQAYQBt AHcAaQB0AHoAMgBDADoAXABXAEkATgBEAE8AVwBTAFwAVABFAE0AUABcAEEAdQB0AG8AUgBlAGMA bwB2AGUAcgB5ACAAcwBhAHYAZQAgAG8AZgAgAEYAcgBhAG0AZQB3AG8AcgBrAC4AYQBzAGQAEQBQ AGEAdQBsACAAdgBvAG4AIABTAHQAYQBtAHcAaQB0AHoALABDADoAXABNAHkAIABEAG8AYwB1AG0A ZQBuAHQAcwBcAEUAdABoAGUAcgBTAEMAUwBJAFwAaQBlAHQAZgBcAEYAcgBhAG0AZQB3AG8AcgBr AC4AZABvAGMAEQBQAGEAdQBsACAAdgBvAG4AIABTAHQAYQBtAHcAaQB0AHoAMgBDADoAXABXAEkA TgBEAE8AVwBTAFwAVABFAE0AUABcAEEAdQB0AG8AUgBlAGMAbwB2AGUAcgB5ACAAcwBhAHYAZQAg AG8AZgAgAEYAcgBhAG0AZQB3AG8AcgBrAC4AYQBzAGQAEQBQAGEAdQBsACAAdgBvAG4AIABTAHQA YQBtAHcAaQB0AHoALABDADoAXABNAHkAIABEAG8AYwB1AG0AZQBuAHQAcwBcAEUAdABoAGUAcgBT AEMAUwBJAFwAaQBlAHQAZgBcAEYAcgBhAG0AZQB3AG8AcgBrAC4AZABvAGMAEQBQAGEAdQBsACAA dgBvAG4AIABTAHQAYQBtAHcAaQB0AHoALABDADoAXABNAHkAIABEAG8AYwB1AG0AZQBuAHQAcwBc AEUAdABoAGUAcgBTAEMAUwBJAFwAaQBlAHQAZgBcAEYAcgBhAG0AZQB3AG8AcgBrAC4AZABvAGMA EQBQAGEAdQBsACAAdgBvAG4AIABTAHQAYQBtAHcAaQB0AHoAMgBDADoAXABXAEkATgBEAE8AVwBT AFwAVABFAE0AUABcAEEAdQB0AG8AUgBlAGMAbwB2AGUAcgB5ACAAcwBhAHYAZQAgAG8AZgAgAEYA cgBhAG0AZQB3AG8AcgBrAC4AYQBzAGQAAgD6O0wVhG0EbP8P/w//D/8P/w//D/8P/w//DwAAsAB6 XA8ACQT/D/8P/w//D/8P/w//D/8P/w8BAAEAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAMQAAAPhGgB EYSY/hXGBQABaAEGbygAAgAAAC4AAQAAAAAEAQMAAAAAAAAAAAAAAAAAAAAABxAAAA+EaAERhJj+ FcYFAAFoAQZDShgAbygAAwAAAC4AAQABAAAAAAQBAwUAAAAAAAAAAAAAAAAAAAAHEAAAD4TQAhGE MP0VxgUAAdACBkNKGABvKAAFAAAALgABAC4AAgABAAAAAAQBAwUHAAAAAAAAAAAAAAAAAAAHEAAA D4TQAhGEMP0VxgUAAdACBkNKGABvKAAHAAAALgABAC4AAgAuAAMAAQAAAAAEAQMFBwkAAAAAAAAA AAAAAAAABxAAAA+EOAQRhMj7FcYFAAE4BAZDShgAbygACQAAAC4AAQAuAAIALgADAC4ABAABAAAA AAQBAwUHCQsAAAAAAAAAAAAAAAAHEAAAD4Q4BBGEyPsVxgUAATgEBkNKGABvKAALAAAALgABAC4A AgAuAAMALgAEAC4ABQABAAAAAAQBAwUHCQsNAAAAAAAAAAAAAAAHEAAAD4SgBRGEYPoVxgUAAaAF BkNKGABvKAANAAAALgABAC4AAgAuAAMALgAEAC4ABQAuAAYAAQAAAAAEAQMFBwkLDQ8AAAAAAAAA AAAABxAAAA+EoAURhGD6FcYFAAGgBQZDShgAbygADwAAAC4AAQAuAAIALgADAC4ABAAuAAUALgAG AC4ABwABAAAAAAQBAwUHCQsNDxEAAAAAAAAAAAAHEAAAD4QIBxGE+PgVxgUAAQgHBkNKGABvKAAR AAAALgABAC4AAgAuAAMALgAEAC4ABQAuAAYALgAHAC4ACAABAAAAAAABAAAAAAAAAAAAAAAAAAAA AAAHEAAAD4RoARGEmP4VxgUAAWgBBkNKFABvKAACAAAALgACAAAAsAB6XAAAAAAAAAAAAAAAAPo7 TBUAAAAAAAAAAAAAAAD/////////////AgAAAAAAAAD/QAGAAQCWGQAAlhkAACTNhAABAAEAlhkA AAAAAACWGQAAAAAAAAIQAAAAAAAAAFMaAABAAQAIAEAAAAMAAABHFpABAAACAgYDBQQFAgMEAwAA AAAAAAAAAAAAAAAAAAEAAAAAAAAAVABpAG0AZQBzACAATgBlAHcAIABSAG8AbQBhAG4AAAA1FpAB AgAFBQECAQcGAgUHAAAAAAAAABAAAAAAAAAAAAAAAIAAAAAAUwB5AG0AYgBvAGwAAAAzJpABAAAC CwYEAgICAgIEAwAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAQQByAGkAYQBsAAAAIgAEAHEIiBgAANAC AABoAQAAAADKEkJm8BRCZuMTQmYJABgBAADOAwAAtBUAAAEACwAAAAQAAxAuAAAAAAAAAAAAAAAB AAEAAAABAAAAAAAAACQDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKUGwAe0ALQAgAAy MAAAAAAAAAAAAAAAAAAApxoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAE0aAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAD//xIAAAAAAAAAAQAxAAAAAAAAABEAUABh AHUAbAAgAHYAbwBuACAAUwB0AGEAbQB3AGkAdAB6ABEAUABhAHUAbAAgAHYAbwBuACAAUwB0AGEA bQB3AGkAdAB6AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD+/wAABAACAAAAAAAAAAAAAAAAAAAAAAABAAAA4IWf8vlP aBCrkQgAKyez2TAAAACQAQAAEgAAAAEAAACYAAAAAgAAAKAAAAADAAAArAAAAAQAAAC4AAAABQAA ANQAAAAGAAAA4AAAAAcAAADsAAAACAAAAPwAAAAJAAAAGAEAABIAAAAkAQAACgAAAEABAAALAAAA TAEAAAwAAABYAQAADQAAAGQBAAAOAAAAcAEAAA8AAAB4AQAAEAAAAIABAAATAAAAiAEAAAIAAADk BAAAHgAAAAIAAAAxAHMAHgAAAAEAAAAAAHMAHgAAABIAAABQYXVsIHZvbiBTdGFtd2l0egBkAB4A AAABAAAAAGF1bB4AAAABAAAAAGF1bB4AAAAHAAAATm9ybWFsAG4eAAAAEgAAAFBhdWwgdm9uIFN0 YW13aXR6AGQAHgAAAAIAAAA5AHVsHgAAABMAAABNaWNyb3NvZnQgV29yZCA4LjAAAEAAAAAAkJQd JwAAAEAAAAAAKv0e1m2/AUAAAAAAtNkZsW2/AUAAAAAAWPl2+W2/AQMAAAABAAAAAwAAAM4DAAAD AAAAtBUAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAA/v8AAAQAAgAAAAAAAAAAAAAAAAAAAAAAAgAAAALVzdWcLhsQk5cIACss +a5EAAAABdXN1ZwuGxCTlwgAKyz5rjABAADsAAAADAAAAAEAAABoAAAADwAAAHAAAAAFAAAAgAAA AAYAAACIAAAAEQAAAJAAAAAXAAAAmAAAAAsAAACgAAAAEAAAAKgAAAATAAAAsAAAABYAAAC4AAAA DQAAAMAAAAAMAAAAzgAAAAIAAADkBAAAHgAAAAgAAABBREFQVEVDAAMAAAAuAAAAAwAAAAsAAAAD AAAApxoAAAMAAAAxFQgACwAAAAAAAAALAAAAAAAAAAsAAAAAAAAACwAAAAAAAAAeEAAAAQAAAAIA AAAxAAwQAAACAAAAHgAAAAYAAABUaXRsZQADAAAAAQAAAJgAAAADAAAAAAAAACAAAAABAAAANgAA AAIAAAA+AAAAAQAAAAIAAAAKAAAAX1BJRF9HVUlEAAIAAADkBAAAQQAAAE4AAAB7ADMANgA1AEMA OAA4AEEAQQAtAEQAOQA2ADMALQAxADEARAAzAC0AQQA2AEQAOAAtADAAMAA4ADAAQwA3AEUANABC ADkAQgA3AH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAEAAAACAAAAAwAAAAQAAAAFAAAABgAAAAcAAAAIAAAACQAAAAoAAAALAAAADAAAAA0A AAAOAAAADwAAABAAAAARAAAAEgAAABMAAAAUAAAAFQAAABYAAAAXAAAAGAAAAP7///8aAAAAGwAA ABwAAAAdAAAAHgAAAB8AAAAgAAAA/v///yIAAAAjAAAAJAAAACUAAAAmAAAAJwAAACgAAAD+//// KgAAACsAAAAsAAAALQAAAC4AAAAvAAAAMAAAAP7////9////MwAAAP7////+/////v////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// ////////UgBvAG8AdAAgAEUAbgB0AHIAeQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAABYABQH//////////wMAAAAGCQIAAAAAAMAAAAAAAABGAAAAAACUVcLAbb8B 4A3NmfltvwE1AAAAgAAAAAAAAAAxAFQAYQBiAGwAZQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgACAP///////////////wAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAABkAAAAAEAAAAAAAAFcAbwByAGQARABvAGMAdQBtAGUAbgB0 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaAAIBBQAAAP////////// AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB4wAAAAAAAABQBTAHUAbQBt AGEAcgB5AEkAbgBmAG8AcgBtAGEAdABpAG8AbgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgA AgECAAAABAAAAP////8AAAAAAAAAAAAAAAAAAAAAAAAAAMDm9szjbb8BwOb2zONtvwEhAAAAABAA AAAAAAAFAEQAbwBjAHUAbQBlAG4AdABTAHUAbQBtAGEAcgB5AEkAbgBmAG8AcgBtAGEAdABpAG8A bgAAAAAAAAAAAAAAOAACAf///////////////wcJAgAAAAAAwAAAAAAAAEYAAAAAwOb2zONtvwHA 5vbM422/ASkAAAAAEAAAAAAAAAEAQwBvAG0AcABPAGIAagAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASAAIBAQAAAAYAAAD/////AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGoAAAAAAAAATwBiAGoAZQBjAHQAUABvAG8AbAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABYAAQD///////////////8A AAAAAAAAAAAAAAAAAAAAAAAAAOANzZn5bb8B4A3NmfltvwEAAAAAaAAAAAAAAAADAE8AYgBqAEkA bgBmAG8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AP///////////////wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAEAAAA AAAAAAEAAAD+//////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// ////AQD+/wMKAAD/////BgkCAAAAAADAAAAAAAAARhgAAABNaWNyb3NvZnQgV29yZCBEb2N1bWVu dAAKAAAATVNXb3JkRG9jABAAAABXb3JkLkRvY3VtZW50LjgA9DmycQAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAA= --=====================_88660246==_ Content-Type: text/plain; charset="us-ascii" --=====================_88660246==_-- From XXX Sat Feb 12 16:41:52 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sat Feb 12 04:43:35 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id EAA23398 for ; Sat, 12 Feb 2000 04:43:35 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA18919 for ips-outgoing; Sat, 12 Feb 2000 04:43:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA18913 for ; Sat, 12 Feb 2000 04:43:29 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA104464; Sat, 12 Feb 2000 10:42:54 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay02.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id KAA50228; Sat, 12 Feb 2000 10:42:53 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256883.00355ABB ; Sat, 12 Feb 2000 10:42:46 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "Bradley, Mark" cc: "'Costa Sapuntzakis'" , ips@ece.cmu.edu, dave_lee@3com.com Message-ID: Date: Sat, 12 Feb 2000 11:42:19 +0200 Subject: RE: BOF Request Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, I am part of another organization that has some work going on in both the storage and networking areas (as are you). I think that Costa is right in assuming that most of us know the area and the requirements could be defined very fast. It would be a pity not to move to something more substantive during this meeting - as the area is mature and badly in need of good solutions. Regards, Julo Julian Satran - Senior Technical Staff Member - IBM Research "Bradley, Mark" on 11/02/2000 23:28:23 Please respond to "Bradley, Mark" To: "'Costa Sapuntzakis'" , ips@ece.cmu.edu, dave_lee@3com.com cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: BOF Request This presumes that everyone knows and understands the issues and problems from the various perspectives that may be provided. Since this is not the case, we need to provide a venue to those to be communicated and discussed. Once we have some common understandings of the issues, perhaps we can work on requirements. Then, and only then, would it make sense to entertain proposals that satisfy the agreed upon requirements. I believe we should attempt to do as thorough a job as possible in understanding the issues and defining requirements as is humanly possible. Only then can we evaluate proposals against needs. This might fly in the face of TTM (time to market) for someone's product, but it is the best method I can imagine to develop standards that meet the needs defined by the industry. Let's follow the process that the IETF has laid out for this work. -- markb > -----Original Message----- > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > Sent: Thursday, February 10, 2000 3:55 AM > To: ips@ece.cmu.edu; dave_lee@3com.com > Cc: djpotter@cisco.com; cheriton@cisco.com; avb@cisco.com; > vern@aciri.org; sob@harvard.edu > Subject: Re: BOF Request > > > > Hi, > > I'm working with Andy Bechtolsheim on the IP SAN initiative at Cisco. > I believe we (the industry) are behind at this point because we > (Cisco) have been hearing from our customers (and I'm sure you've been > hearing from yours) that they want these technologies now. As such, > much of the discussion on the charter and goals should happen before > the BOF (on the ips mailing list). > > Let's make the IETF meeting be for serious high-bandwidth engineering > discussion of the various approaches so we can attempt to reach > consensus on a technical direction. > > Proposed agenda for BOF: > > > 5 min Review of BOF agenda and objectives (D. Lee/M. Bradley) > > 10 min IP Storage Goals (D. Nagle/A. Wilson) > > 15 min Discussion > > 30 min Approaches & Implementations (10 min presentations) > > o SCSI/TCP (TBA) > > o Adaptec (A. Wilson) > > o ISI's NetSCSI (?) > > 40 min discussion > > 5 min Proposed charters (D. Lee/M. Bradley/C. Sapuntzakis) > > 15 min Discussion > > Our formal interaction with T10 and T11 should be minimized. The IETF > strongly discourages the republishing of IETF standards by other > organizations. In addition, IETF can work much more quickly than > ANSI standards organizations. > > The only interaction I see with T13 is if we define an IDE > transport protocol. > ATAPI command sets are now handled through T10. > > FCP is an impressive protocol, especially in the bulk of documentation > required to understand it. It is very much tied to Fibre Channel > Physical Layer. Understanding FCP, its limitations, how it and its > associated discovery mechanisms need to be fixed up to work in an IP > environment is a large undertaking. FCP will seriously slow down the > generation of any standard and impede the clarity of the final > standard. > > > A proposed charter: > ------------------- > > Description of Working Group: > > The goal of this working group is to define extensions to current IETF > transports for storage networking. > > There are a several areas to consider: > > - Reducing protocol processing overhead at servers and clients > - SCSI transport on top of TCP > - QoS parameters for storage networks > - Running storage transports at multi-gigabit speeds > > The objectives of this working group are: > > * A proposed standard for a SCSI transport on top of TCP > > * A proposed standard for extensions to TCP that reduce protocol > processing overhead. > > * Produce an informational RFC describing the QoS > requirements of storage > networks. Propose QoS extensions to diffserv if necessary. > > * Produce an informational RFC discussing the issues behind running > storage transports at multi-gigabit speeds. > > -------------------------------------- > > -Costa > > > > > ----- Begin Included Message ----- > > > > >From Dave_Lee@3com.com Wed Feb 9 03:17:02 2000 > > From: Dave_Lee@3com.com > > X-Lotus-FromDomain: 3COM > > To: agenda@ietf.org > > cc: sob@harvard.edu, mankin@isi.edu, Vern@Aciri.Org, > markb@btc.adaptec.com, > > Carl_Madison@3com.com, avb@cisco.com > > Date: Wed, 9 Feb 2000 03:15:23 -0800 > > Subject: BOF Request > > Mime-Version: 1.0 > > Content-Disposition: inline > > X-SMTP-HELO: seattle.3com.com > > X-SMTP-MAIL-FROM: Dave_Lee@3com.com > > X-SMAP-Received-From: outside > > X-SMTP-PEER-INFO: seattle.3com.com [129.213.128.97] > > > > > > > > BOF Description > > > > The objective of this BOF is: > > > > 1. To consider the creation of a new IETF working group to > enable storage > > networking via IP-based storage interfaces of SCSI, ATAPI, > FCP and other > > interfaces. Only system-to-storage interfaces will be considered. > > > > 2. To develop and agree upon a charter for the new working group. > > > > 3. To consider the proposed operational characteristics of > reliable and > > unreliable communications, shared and non-shared, managed > and unmanaged, and > > integrity, privacy, and/or non-secure. Further, the protocol must > > efficiently operate over LAN, campus, and WAN environments. > > > > 4. To consider proposed protocol features such as automatic > discovery, > > enumeration, configuration, and management of storage > adapters and devices. > > > > 5. To discuss what collaboration with ANSI T10, T11 and T13 > is appropriate. > > > > > > BOF Agenda (2 hrs) > > 5 min Review of BOF agenda and objectives (D. Lee/M. Bradley) > > 20 min IP Storage Goals (D. Nagle/A. Wilson) > > 35 min Discussion > > 20 min Approaches (5 minutes each) > > o Cisco (A. Bechtolsheim) > > o Adaptec (A. Wilson) > > o ISI's NetSCSI (?) > > o Other (?) > > 10 min Proposed charter (D. Lee/M. Bradley) > > 30 min Discussion > > > > > > > > > > > > > > ----- End Included Message ----- > > > > > > > From XXX Tue Feb 15 16:10:00 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 15 11:06:55 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA29558 for ; Tue, 15 Feb 2000 11:06:55 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA24211 for ips-outgoing; Tue, 15 Feb 2000 11:06:53 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA24204 for ; Tue, 15 Feb 2000 11:06:48 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.9.2/8.8.8) with ESMTP id IAA10608; Tue, 15 Feb 2000 08:06:15 -0800 (PST) Received: from btcexc01.btc.adaptec.com (btcexc01.btc.adaptec.com [162.62.147.10]) by kingdom.adaptec.com (8.9.2/8.8.8) with ESMTP id IAA23614; Tue, 15 Feb 2000 08:06:14 -0800 (PST) Received: by btcexc01.btc.adaptec.com with Internet Mail Service (5.5.2650.21) id <1SDTVRAH>; Tue, 15 Feb 2000 09:07:50 -0700 Message-ID: <3CFAB2A312A9D311BB300000D11ABE920D301A@btcexc01.btc.adaptec.com> From: "Bradley, Mark" To: "'Costa Sapuntzakis'" , ips@ece.cmu.edu Subject: RE: ATA/IP & ATAPI Date: Tue, 15 Feb 2000 09:07:41 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk About 85% of storage on IA32/64 systems is IDE/ATA. Ignoring this substantial a volume seems inappropriate. Further, there is a proposal for Serial ATA (SAT) that might lend better lend itself to this work. -- markb > -----Original Message----- > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > Sent: Tuesday, February 15, 2000 1:07 AM > To: ips@ece.cmu.edu > Subject: ATA/IP & ATAPI > > > > Hi, > > What are the arguments for ATA/IP? > > -Costa > From XXX Tue Feb 15 16:32:26 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 15 11:29:26 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA29783 for ; Tue, 15 Feb 2000 11:29:25 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA25078 for ips-outgoing; Tue, 15 Feb 2000 11:29:19 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA25073 for ; Tue, 15 Feb 2000 11:29:16 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA41648 for ; Tue, 15 Feb 2000 17:28:44 +0100 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id RAA75864 for ; Tue, 15 Feb 2000 17:28:44 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256886.005A8140 ; Tue, 15 Feb 2000 17:28:33 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 15 Feb 2000 18:28:23 +0200 Subject: RE: ATA/IP & ATAPI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I think we can include ATA/ATAPI within SCOT. I am not sure that we should "surface" both ends or make it transparent to the initiator. I would like to hear the "industry" on that. By making only one end visible we could have the industry converge to one "virtual" standard. Regards, Julo "Bradley, Mark" on 15/02/2000 18:07:41 Please respond to "Bradley, Mark" To: "'Costa Sapuntzakis'" , ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: ATA/IP & ATAPI About 85% of storage on IA32/64 systems is IDE/ATA. Ignoring this substantial a volume seems inappropriate. Further, there is a proposal for Serial ATA (SAT) that might lend better lend itself to this work. -- markb > -----Original Message----- > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > Sent: Tuesday, February 15, 2000 1:07 AM > To: ips@ece.cmu.edu > Subject: ATA/IP & ATAPI > > > > Hi, > > What are the arguments for ATA/IP? > > -Costa > From XXX Tue Feb 15 16:52:15 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 15 11:37:17 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA30128 for ; Tue, 15 Feb 2000 11:37:16 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA25512 for ips-outgoing; Tue, 15 Feb 2000 11:37:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA25506 for ; Tue, 15 Feb 2000 11:37:10 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.9.2/8.8.8) with ESMTP id IAA13855; Tue, 15 Feb 2000 08:36:39 -0800 (PST) Received: from btcexc01.btc.adaptec.com (btcexc01.btc.adaptec.com [162.62.147.10]) by kingdom.adaptec.com (8.9.2/8.8.8) with ESMTP id IAA26932; Tue, 15 Feb 2000 08:36:38 -0800 (PST) Received: by btcexc01.btc.adaptec.com with Internet Mail Service (5.5.2650.21) id <1SDTVRCK>; Tue, 15 Feb 2000 09:38:14 -0700 Message-ID: <3CFAB2A312A9D311BB300000D11ABE920D301D@btcexc01.btc.adaptec.com> From: "Bradley, Mark" To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: ATA/IP & ATAPI Date: Tue, 15 Feb 2000 09:38:09 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk It would be prudent to attempt to include any common interfaces in encapsulation, whether FC, SCSI, ATA or versions thereof. > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, February 15, 2000 9:28 AM > To: ips@ece.cmu.edu > Subject: RE: ATA/IP & ATAPI > > > > > I think we can include ATA/ATAPI within SCOT. I am not sure > that we should > "surface" both ends or make it transparent to the initiator. > I would like > to hear the "industry" on that. > By making only one end visible we could have the industry > converge to one > "virtual" standard. > > Regards, > Julo > > "Bradley, Mark" on 15/02/2000 18:07:41 > > Please respond to "Bradley, Mark" > > To: "'Costa Sapuntzakis'" , ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: ATA/IP & ATAPI > > > > > About 85% of storage on IA32/64 systems is IDE/ATA. Ignoring > this substantial a volume seems inappropriate. Further, there > is a proposal for Serial ATA (SAT) that might lend better lend > itself to this work. > -- markb > > > -----Original Message----- > > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > > Sent: Tuesday, February 15, 2000 1:07 AM > > To: ips@ece.cmu.edu > > Subject: ATA/IP & ATAPI > > > > > > > > Hi, > > > > What are the arguments for ATA/IP? > > > > -Costa > > > > > From XXX Tue Feb 15 17:35:56 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 15 12:35:44 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA21406 for ; Tue, 15 Feb 2000 12:35:43 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA27815 for ips-outgoing; Tue, 15 Feb 2000 12:35:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA27809 for ; Tue, 15 Feb 2000 12:35:37 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.9.2/8.8.8) with ESMTP id JAA20908; Tue, 15 Feb 2000 09:34:31 -0800 (PST) Received: from btcexc01.btc.adaptec.com (btcexc01.btc.adaptec.com [162.62.147.10]) by kingdom.adaptec.com (8.9.2/8.8.8) with ESMTP id JAA05658; Tue, 15 Feb 2000 09:34:30 -0800 (PST) Received: by btcexc01.btc.adaptec.com with Internet Mail Service (5.5.2650.21) id <1SDTVRGX>; Tue, 15 Feb 2000 10:36:06 -0700 Message-ID: <3CFAB2A312A9D311BB300000D11ABE920D301F@btcexc01.btc.adaptec.com> From: "Bradley, Mark" To: "'Costa Sapuntzakis'" , "Bradley, Mark" Cc: ips@ece.cmu.edu Subject: RE: ATA/IP & ATAPI Date: Tue, 15 Feb 2000 10:35:59 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk >>From what you write below, it seems you envision some ATA chip trying to drive packets onto a wire (perhaps using some strange bridge). Is this what you mean? > -----Original Message----- > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > Sent: Tuesday, February 15, 2000 10:06 AM > To: Bradley, Mark > Cc: ips@ece.cmu.edu > Subject: RE: ATA/IP & ATAPI > > > > Installed base is a very good argument. :) Here are some of the > issues: > > Many ATA drivers are written to do register-level accesses. Rewriting > them to send packets will involve some serious reworking. > > An ATA/IP device could maintain a register interface to the host with > a special ATA/ISA<->ATA/IP bridge (either emulated in BIOS or as a > real PCI device). > > Currently, all ATA devices plug into motherboards with > processors. These processors could act as SCSI<->ATA converters. > Or are there ATA-only functions that are lost by that sort of > bridging? > Perhaps SMART... > > Would an ATA/IP device be significantly simpler than a SCSI/IP device > from a firmware standpoint? > > -Costa > > On Tue, 15 Feb 2000, Bradley, Mark wrote: > > > About 85% of storage on IA32/64 systems is IDE/ATA. Ignoring > > this substantial a volume seems inappropriate. Further, there > > is a proposal for Serial ATA (SAT) that might lend better lend > > itself to this work. > > -- markb > > > > > -----Original Message----- > > > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > > > Sent: Tuesday, February 15, 2000 1:07 AM > > > To: ips@ece.cmu.edu > > > Subject: ATA/IP & ATAPI > > > > > > > > > > > > Hi, > > > > > > What are the arguments for ATA/IP? > > > > > > -Costa > > > > > > > > > > From XXX Tue Feb 15 17:58:26 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 15 12:56:44 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA21999 for ; Tue, 15 Feb 2000 12:56:43 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA28577 for ips-outgoing; Tue, 15 Feb 2000 12:56:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from jindo.cisco.com (jindo.cisco.com [171.69.11.73]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA28571 for ; Tue, 15 Feb 2000 12:56:38 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by jindo.cisco.com (8.8.8/2.5.1/Cisco List Logging/8.8.8) with ESMTP id JAA26404; Tue, 15 Feb 2000 09:56:05 -0800 (PST) Date: Tue, 15 Feb 2000 09:56:07 -0800 (PST) From: Costa Sapuntzakis To: "Bradley, Mark" cc: ips@ece.cmu.edu Subject: RE: ATA/IP & ATAPI In-Reply-To: <3CFAB2A312A9D311BB300000D11ABE920D301F@btcexc01.btc.adaptec.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk I should clarify. It seems there are two options here: 1) Rewrite the BIOS and OS ATA drivers to support ATA/IP 2) Have some hardware that bridges between ATA/ISA, which is what current BIOSes and ATA drivers expect, and ATA/IP. There is a middle ground perhaps where the BIOS emulates the old ATA/ISA interface. This is already done with USB to support USB keyboards and mice on legacy OSes. -Costa On Tue, 15 Feb 2000, Bradley, Mark wrote: > From what you write below, it seems you envision some ATA chip > trying to drive packets onto a wire (perhaps using some strange > bridge). Is this what you mean? > > > -----Original Message----- > > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > > Sent: Tuesday, February 15, 2000 10:06 AM > > To: Bradley, Mark > > Cc: ips@ece.cmu.edu > > Subject: RE: ATA/IP & ATAPI > > > > > > > > Installed base is a very good argument. :) Here are some of the > > issues: > > > > Many ATA drivers are written to do register-level accesses. Rewriting > > them to send packets will involve some serious reworking. > > > > An ATA/IP device could maintain a register interface to the host with > > a special ATA/ISA<->ATA/IP bridge (either emulated in BIOS or as a > > real PCI device). > > > > Currently, all ATA devices plug into motherboards with > > processors. These processors could act as SCSI<->ATA converters. > > Or are there ATA-only functions that are lost by that sort of > > bridging? > > Perhaps SMART... > > > > Would an ATA/IP device be significantly simpler than a SCSI/IP device > > from a firmware standpoint? > > > > -Costa > > > > On Tue, 15 Feb 2000, Bradley, Mark wrote: > > > > > About 85% of storage on IA32/64 systems is IDE/ATA. Ignoring > > > this substantial a volume seems inappropriate. Further, there > > > is a proposal for Serial ATA (SAT) that might lend better lend > > > itself to this work. > > > -- markb > > > > > > > -----Original Message----- > > > > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > > > > Sent: Tuesday, February 15, 2000 1:07 AM > > > > To: ips@ece.cmu.edu > > > > Subject: ATA/IP & ATAPI > > > > > > > > > > > > > > > > Hi, > > > > > > > > What are the arguments for ATA/IP? > > > > > > > > -Costa > > > > > > > > > > > > > > > > > > From XXX Tue Feb 15 18:03:55 2000 Forwarded: Tue, 15 Feb 2000 13:03:55 -0500 Forwarded: "ed.gronke@intel.com " Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 15 12:06:53 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA20552 for ; Tue, 15 Feb 2000 12:06:52 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA26731 for ips-outgoing; Tue, 15 Feb 2000 12:06:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from jindo.cisco.com (jindo.cisco.com [171.69.11.73]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA26703 for ; Tue, 15 Feb 2000 12:06:41 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by jindo.cisco.com (8.8.8/2.5.1/Cisco List Logging/8.8.8) with ESMTP id JAA23935; Tue, 15 Feb 2000 09:06:08 -0800 (PST) Date: Tue, 15 Feb 2000 09:06:09 -0800 (PST) From: Costa Sapuntzakis To: "Bradley, Mark" cc: ips@ece.cmu.edu Subject: RE: ATA/IP & ATAPI In-Reply-To: <3CFAB2A312A9D311BB300000D11ABE920D301A@btcexc01.btc.adaptec.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Installed base is a very good argument. :) Here are some of the issues: Many ATA drivers are written to do register-level accesses. Rewriting them to send packets will involve some serious reworking. An ATA/IP device could maintain a register interface to the host with a special ATA/ISA<->ATA/IP bridge (either emulated in BIOS or as a real PCI device). Currently, all ATA devices plug into motherboards with processors. These processors could act as SCSI<->ATA converters. Or are there ATA-only functions that are lost by that sort of bridging? Perhaps SMART... Would an ATA/IP device be significantly simpler than a SCSI/IP device from a firmware standpoint? -Costa On Tue, 15 Feb 2000, Bradley, Mark wrote: > About 85% of storage on IA32/64 systems is IDE/ATA. Ignoring > this substantial a volume seems inappropriate. Further, there > is a proposal for Serial ATA (SAT) that might lend better lend > itself to this work. > -- markb > > > -----Original Message----- > > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > > Sent: Tuesday, February 15, 2000 1:07 AM > > To: ips@ece.cmu.edu > > Subject: ATA/IP & ATAPI > > > > > > > > Hi, > > > > What are the arguments for ATA/IP? > > > > -Costa > > > > > From XXX Wed Feb 16 09:06:20 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Feb 16 01:07:59 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id BAA01659 for ; Wed, 16 Feb 2000 01:07:58 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA25980 for ips-outgoing; Wed, 16 Feb 2000 01:07:52 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25968 for ; Wed, 16 Feb 2000 01:07:46 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA95622; Wed, 16 Feb 2000 07:07:11 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id HAA74404; Wed, 16 Feb 2000 07:07:10 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256887.00219C46 ; Wed, 16 Feb 2000 07:07:06 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Costa Sapuntzakis cc: "Bradley, Mark" , ips@ece.cmu.edu Message-ID: Date: Wed, 16 Feb 2000 08:06:55 +0200 Subject: RE: ATA/IP & ATAPI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, The points you make are very relevant. The large installed base makes a good argument to have ATA/ATAPI devices as targets. For initiators (hosts) having a SCSI driver or an ATA driver makes no difference. I don't see any major issue with having a SCSI to ATA convertor on the target. This will cover most of the appliances, RAID or JBOD boxes. For new devices I think that SCSI has only a moderate level of complexity and costs are only a mater of volume. Regards, Julo Costa Sapuntzakis on 15/02/2000 19:06:09 Please respond to Costa Sapuntzakis To: "Bradley, Mark" cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: ATA/IP & ATAPI Installed base is a very good argument. :) Here are some of the issues: Many ATA drivers are written to do register-level accesses. Rewriting them to send packets will involve some serious reworking. An ATA/IP device could maintain a register interface to the host with a special ATA/ISA<->ATA/IP bridge (either emulated in BIOS or as a real PCI device). Currently, all ATA devices plug into motherboards with processors. These processors could act as SCSI<->ATA converters. Or are there ATA-only functions that are lost by that sort of bridging? Perhaps SMART... Would an ATA/IP device be significantly simpler than a SCSI/IP device from a firmware standpoint? -Costa On Tue, 15 Feb 2000, Bradley, Mark wrote: > About 85% of storage on IA32/64 systems is IDE/ATA. Ignoring > this substantial a volume seems inappropriate. Further, there > is a proposal for Serial ATA (SAT) that might lend better lend > itself to this work. > -- markb > > > -----Original Message----- > > From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] > > Sent: Tuesday, February 15, 2000 1:07 AM > > To: ips@ece.cmu.edu > > Subject: ATA/IP & ATAPI > > > > > > > > Hi, > > > > What are the arguments for ATA/IP? > > > > -Costa > > > > > From XXX Wed Feb 16 09:06:20 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Feb 16 02:35:04 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id CAA25037 for ; Wed, 16 Feb 2000 02:35:04 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA27821 for ips-outgoing; Wed, 16 Feb 2000 02:35:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA27817 for ; Wed, 16 Feb 2000 02:34:59 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA14620 for ; Wed, 16 Feb 2000 08:34:26 +0100 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id IAA24294 for ; Wed, 16 Feb 2000 08:34:20 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256887.002996D4 ; Wed, 16 Feb 2000 08:34:15 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: cherry@il.ibm.com, hrvoje@il.ibm.com, meth@il.ibm.com, Meir_Toledano@il.ibm.com Message-ID: Date: Wed, 16 Feb 2000 09:34:04 +0200 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The ietf-draft submitted yesterday is also available at: http://www.haifa.il.ibm.com/satran/draft-satran-scot-01.txt Regards, Julo From XXX Tue Feb 22 17:46:49 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 22 12:46:32 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA24194 for ; Tue, 22 Feb 2000 12:46:31 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA08341 for ips-outgoing; Tue, 22 Feb 2000 12:44:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA08337 for ; Tue, 22 Feb 2000 12:44:56 -0500 (EST) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA24140 for ; Tue, 22 Feb 2000 12:44:55 -0500 Message-Id: <200002221744.MAA24140@yogi.ece.cmu.edu> To: ips@ece.cmu.edu subject: web page / mail archieve Date: Tue, 22 Feb 2000 12:44:55 -0500 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, The mailing list archieve and web page are now available at http://www.ece.cmu.edu/~ips They will be edited throughout the next two days. dave.......... From XXX Tue Feb 22 17:49:58 2000 Forwarded: Tue, 22 Feb 2000 12:49:58 -0500 Forwarded: "ap@ece.cmu.edu, pdl-gen@cs.cmu.edu " Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 17 20:05:21 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA13022 for ; Thu, 17 Feb 2000 20:05:20 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA05000 for ips-outgoing; Thu, 17 Feb 2000 20:04:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub1.almaden.ibm.com (mailhub1.almaden.ibm.com [198.4.83.44]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA04993 for ; Thu, 17 Feb 2000 20:04:47 -0500 (EST) Received: from catalina.almaden.ibm.com (catalina.almaden.ibm.com [9.1.24.58]) by mailhub1.almaden.ibm.com (8.8.8/8.8.8) with ESMTP id RAA34800 for ; Thu, 17 Feb 2000 17:04:02 -0800 Received: (from dfsmith@localhost) by catalina.almaden.ibm.com (AIX4.3/UCB 8.8.8/8.7) id RAA23568 for ips@ece.cmu.edu; Thu, 17 Feb 2000 17:04:12 -0800 From: Daniel Smith Message-Id: <200002180104.RAA23568@catalina.almaden.ibm.com> Subject: FYI: SCSI over the network in the news To: ips@ece.cmu.edu Date: Thu, 17 Feb 2000 17:04:12 -0800 (PST) X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Just an interest piece. SCSI and the network was mentioned in the Linux Weekly News in a piece about Scheduled Transfer. Demos from SGI and Genroco are mentioned in hyperlinks. http://lwn.net/2000/0217/kernel.phtml -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 From XXX Tue Feb 22 18:32:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 22 13:31:31 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA16144 for ; Tue, 22 Feb 2000 13:31:30 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10059 for ips-outgoing; Tue, 22 Feb 2000 13:29:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from jindo.cisco.com (jindo.cisco.com [171.69.11.73]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10055 for ; Tue, 22 Feb 2000 13:29:45 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by jindo.cisco.com (8.8.8/2.5.1/Cisco List Logging/8.8.8) with ESMTP id KAA29458; Tue, 22 Feb 2000 10:29:02 -0800 (PST) Date: Tue, 22 Feb 2000 10:29:05 -0800 (PST) From: Costa Sapuntzakis To: Brian E Carpenter cc: ips@ece.cmu.edu, djpotter@cisco.com, cheriton@cisco.com, avb@cisco.com, sob@harvard.edu, vern@aciri.org, mankin@east.isi.edu Subject: Re: Proposed announcement to rest of IETF In-Reply-To: <38B2A8EF.F580F652@hursley.ibm.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > 3. I think it is very odd to refer to SCSI as if it is a protocol > at the same level as NFS, CIFS (which is proprietary), and AFS and DFS > that you don't mention. SCSI is in a completely different class, but you > only half-say this. You will get a lot of flak over this in the IETF, I agree that if feels odd at first to see SCSI referred to in the same sentence as NFS. However, once both are travelling over the network, they're not all that different, except that SCSI has a simpler model of sharing. I can access a disk as one large NFS file and it will look a lot like SCSI. > and you don't at all mention the requirements analysis that will > need to be Phase One. I don't think everybody agrees that requirements analysis is phase one. -Costa From XXX Tue Feb 22 21:43:20 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 22 16:37:06 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id QAA24102 for ; Tue, 22 Feb 2000 16:37:05 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA17651 for ips-outgoing; Tue, 22 Feb 2000 16:35:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from deliverator.sgi.com (deliverator.sgi.com [204.94.214.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA17640 for ; Tue, 22 Feb 2000 16:35:31 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by deliverator.sgi.com (980309.SGI.8.8.8-aspam-6.2/980310.SGI-aspam) via ESMTP id NAA04687; Tue, 22 Feb 2000 13:30:23 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id NAA30390; Tue, 22 Feb 2000 13:34:52 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id NAA01317; Tue, 22 Feb 2000 13:34:21 -0800 (PST) Message-ID: <38B300DD.3E4CF760@engr.sgi.com> Date: Tue, 22 Feb 2000 13:34:21 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: Dave_Lee@3com.com CC: aman@sgi.com, markb@btc.adaptec.com, csapuntz@cisco.com, "Mailing List, IP Storage" Subject: Re: Reminder: IP Storage Mtg / Wed. Feb. 16 @ 1:00pm PST References: <88256889.00069552.00@hqoutbound.ops.3com.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, Talking in context of the 3 layer protocol model, SI->Transport->IP, I'd like to bring to attention an ANSI draft standard protocol - Scheduled Transfer Protocol (STP). The design of STP has been driven by requirements very similar to those outlined in the Storage of IP Design Framework, namely low-latency, high bandwidth, low CPU utilization, a measure of congestion/flow control built in, etc. STP can very easily fit in as the transport layer in the 3 layer protocol model. STP can be IP encapsulated, and a draft specification for SCSI over STP also exists. STP has existed on Irix as part of the GSN infrastructure; SGI along with Genroco demonstrated a SCSI over STP solution interfacing a Origin 2000 server with GSN to a fibre channel raid network via a Genroco bridge, at CERN in October 1999. STP is truly media independent, with support for GbE in the works. We recently made an alpha release of STP on Linux to the open-source community (http://oss.sgi.com/projects/stp), and going forward intend to fully support it on a variety of interfaces, starting with GbE. More information about STP and the various drafts can be accessed off http://www.hippi.org/cDOCS.html. If there is interest in learning more about this protocol, we could arrange for a presentation/demo at one of the meetings for the proposed working group. (Note: STP is also refered to as ST in the marketing literature of various vendors - Genroco, Essential, PMR..) thanks, Aman Singla MTS, SGI From XXX Wed Feb 23 02:32:00 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 22 20:55:00 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA22358 for ; Tue, 22 Feb 2000 20:55:00 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA25387 for ips-outgoing; Tue, 22 Feb 2000 20:53:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail-gw.hursley.ibm.com (mail-gw.hursley.ibm.com [194.196.110.15]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA25376 for ; Tue, 22 Feb 2000 20:53:11 -0500 (EST) Received: from sp3at21.hursley.ibm.com (sp3at21.hursley.ibm.com [9.20.45.21]) by mail-gw.hursley.ibm.com (AIX4.3/UCB 8.8.8/8.8.8) with ESMTP id BAA264746; Wed, 23 Feb 2000 01:47:14 GMT Received: from hursley.ibm.com (lig32-226-113-14.us.lig-dial.ibm.com [32.226.113.14]) by sp3at21.hursley.ibm.com (AIX4.2/UCB 8.7/8.7.3) with ESMTP id BAA27510; Wed, 23 Feb 2000 01:47:00 GMT Message-ID: <38B33C15.24A6FC8B@hursley.ibm.com> Date: Tue, 22 Feb 2000 19:47:01 -0600 From: Brian E Carpenter Organization: IBM X-Mailer: Mozilla 4.61 [en] (Win98; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: Costa Sapuntzakis CC: ips@ece.cmu.edu, djpotter@cisco.com, cheriton@cisco.com, avb@cisco.com, sob@harvard.edu, vern@aciri.org, mankin@east.isi.edu Subject: Re: Proposed announcement to rest of IETF References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, without requirements analysis you will not get anywhere in the IETF. The way you have phrased it rings lots of alarm bells - it reads as if you are trying to set up SCSI/TCP as an architectural competitor or replacement for NFS etc, but that is absolutely not the case. That's why the requirements analysis is the first step. Brian Costa Sapuntzakis wrote: > > > 3. I think it is very odd to refer to SCSI as if it is a protocol > > at the same level as NFS, CIFS (which is proprietary), and AFS and DFS > > that you don't mention. SCSI is in a completely different class, but you > > only half-say this. You will get a lot of flak over this in the IETF, > > I agree that if feels odd at first to see SCSI referred to in the > same sentence as NFS. However, once both are travelling over the network, > they're not all that different, except that SCSI has a simpler model > of sharing. > > I can access a disk as one large NFS file and it will look a lot like > SCSI. > > > and you don't at all mention the requirements analysis that will > > need to be Phase One. > > I don't think everybody agrees that requirements analysis is phase one. > > -Costa From XXX Wed Feb 23 13:21:21 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Feb 22 22:06:48 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id WAA03565 for ; Tue, 22 Feb 2000 22:06:47 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA26953 for ips-outgoing; Tue, 22 Feb 2000 22:05:02 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from jindo.cisco.com (jindo.cisco.com [171.69.11.73]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA26939 for ; Tue, 22 Feb 2000 22:04:50 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by jindo.cisco.com (8.8.8/2.5.1/Cisco List Logging/8.8.8) with ESMTP id TAA20550; Tue, 22 Feb 2000 19:04:11 -0800 (PST) Date: Tue, 22 Feb 2000 19:04:14 -0800 (PST) From: Costa Sapuntzakis To: Brian E Carpenter cc: ips@ece.cmu.edu, sob@harvard.edu, vern@aciri.org, mankin@east.isi.edu Subject: Re: Proposed announcement to rest of IETF In-Reply-To: <38B33C15.24A6FC8B@hursley.ibm.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk On Tue, 22 Feb 2000, Brian E Carpenter wrote: > Costa, without requirements analysis you will not get anywhere in the IETF. Can you point me to some documents that indicate what goes into a requirements analysis? I'm fuzzy on the concept. > The way you have phrased it rings lots of alarm bells - it reads as if you are > trying to set up SCSI/TCP as an architectural competitor or replacement for > NFS etc, but that is absolutely not the case. That's why the requirements analysis > is the first step. > > Brian > Phrasing aside, here's what I meant for the record: The intent was to say "SCSI is joining the IP network bandwagon 15 years late." AFS and NFS have been doing IP networks for years. Now, SCSI wants to play over IP networks. Our goal is to make SCSI work over all kinds of IP networks. The goal is not to replace NFS. In fact, some of the proposals out of this proposed working group may actually help NFS. Again, though, the principle goal is to make SCSI work over IP networks. -Costa > Costa Sapuntzakis wrote: > > > > > 3. I think it is very odd to refer to SCSI as if it is a protocol > > > at the same level as NFS, CIFS (which is proprietary), and AFS and DFS > > > that you don't mention. SCSI is in a completely different class, but you > > > only half-say this. You will get a lot of flak over this in the IETF, > > > > I agree that if feels odd at first to see SCSI referred to in the > > same sentence as NFS. However, once both are travelling over the network, > > they're not all that different, except that SCSI has a simpler model > > of sharing. > > > > I can access a disk as one large NFS file and it will look a lot like > > SCSI. > > > > > and you don't at all mention the requirements analysis that will > > > need to be Phase One. > > > > I don't think everybody agrees that requirements analysis is phase one. > > > > -Costa > > > From XXX Wed Feb 23 13:21:21 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Feb 23 04:23:35 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id EAA14394 for ; Wed, 23 Feb 2000 04:23:34 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA04225 for ips-outgoing; Wed, 23 Feb 2000 04:21:55 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA04220 for ; Wed, 23 Feb 2000 04:21:52 -0500 (EST) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.9.2/8.9.2) id BAA06973; Wed, 23 Feb 2000 01:21:47 -0800 (PST) Message-Id: <200002230921.BAA06973@daffy.ee.lbl.gov> To: Costa Sapuntzakis Cc: Brian E Carpenter , ips@ece.cmu.edu, sob@harvard.edu, mankin@east.isi.edu Subject: Re: Proposed announcement to rest of IETF In-reply-to: Your message of Tue, 22 Feb 2000 19:04:14 PST. Date: Wed, 23 Feb 2000 01:21:47 PST From: Vern Paxson Sender: owner-ips@ece.cmu.edu Precedence: bulk > > Costa, without requirements analysis you will not get anywhere in the IETF. > > Can you point me to some documents that indicate what goes into > a requirements analysis? I'm fuzzy on the concept. I don't think we have such a document per se, but what goes into a requirements analysis is something like: 1. Here's the problem [domain] we're trying to solve 2. Solutions for this problem must do X because of ABC. 3. Solutions for this problem must do Y because of DEF. 4. Note that doing both X and Y means the solution must also do Z. 5. It's out of scope for solutions to do Q. 6. It's optional for solutions to do W. 7. Note that one possible approach is Foo, which has the following additional consideration: ... 8. There's also the Bar technology, but it's geared for a somewhat different problem domain, because of blah-blah-blah. These last two aren't strictly appropriate for a requirements document, but if done in objective terms can be natural discussions to include. > The goal is not to replace NFS. In fact, some of the proposals out of > this proposed working group may actually help NFS. Again, though, the > principle goal is to make SCSI work over IP networks. I think Brian's point is that until the requirements are nailed down, there will be confusion over exactly what problem is it we're trying to solve. In particular, I believe the problem is not "run SCSI over IP" but something a bit more abstract than that, for which one of the possible solutions will be "run SCSI over TCP over IP, with the following tweaks to the SCSI spec, and the following TCP options, and ...". And we want to understand how that abstract goal does indeed differ from the abstract goal that NFS attempts to address. Vern From XXX Wed Feb 23 16:54:47 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Feb 23 11:53:25 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA25439 for ; Wed, 23 Feb 2000 11:53:25 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA24135 for ips-outgoing; Wed, 23 Feb 2000 11:51:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from voltron.flexstar.com (mail.flexstar.com [205.178.7.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA24131 for ; Wed, 23 Feb 2000 11:51:32 -0500 (EST) Received: by VOLTRON with Internet Mail Service (5.5.2448.0) id ; Wed, 23 Feb 2000 08:51:17 -0800 Message-ID: From: "Morrow, Licia" To: "'ips@ece.cmu.edu'" Subject: BOF information Date: Wed, 23 Feb 2000 08:51:16 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I am seeking additional information about the BOF meeting being held in Adelaide, Australia. I need to know the date, times, agenda and a list of attendees if possible. Please let me know who I should contact. Thanks! Licia Morrow lmorrow@flexstar.com From XXX Wed Feb 23 19:33:54 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Feb 23 14:27:54 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id OAA31921 for ; Wed, 23 Feb 2000 14:27:53 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA01273 for ips-outgoing; Wed, 23 Feb 2000 14:26:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from seattle.3com.com (seattle.3com.com [129.213.128.97]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA01268 for ; Wed, 23 Feb 2000 14:26:10 -0500 (EST) From: Dave_Lee@3com.com Received: from new-york.3com.com (new-york.3com.com [129.213.157.12]) by seattle.3com.com (8.8.8/8.8.8) with ESMTP id LAA16502; Wed, 23 Feb 2000 11:26:09 -0800 (PST) Received: from hqoutbound.ops.3com.com (hqoutbound.OPS.3Com.COM [139.87.48.104]) by new-york.3com.com (8.8.8/8.8.8) with SMTP id LAA22628; Wed, 23 Feb 2000 11:26:08 -0800 (PST) Received: by hqoutbound.ops.3com.com(Lotus SMTP MTA v4.6.3 (778.2 1-4-1999)) id 8825688E.006A842E ; Wed, 23 Feb 2000 11:23:26 -0800 X-Lotus-FromDomain: 3COM To: "Morrow, Licia" cc: "'ips@ece.cmu.edu'" Message-ID: <8825688E.006A8119.00@hqoutbound.ops.3com.com> Date: Wed, 23 Feb 2000 11:25:08 -0800 Subject: Re: BOF information Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The BOF meeting is being in held at the March 2000 IETF meeting. http://www.ietf.org/meetings/IETF-47.html A draft agenda is posted there but the IP storage BOF is not currently on it but will be at some point. Current information on the BOF, such as the BOF description, is posted at: http://www.ece.cmu.edu/~ips/ Dave "Morrow, Licia" on 02/23/2000 08:51:16 AM Sent by: "Morrow, Licia" To: "'ips @ece.cmu.edu'" cc: (Dave Lee/HQ/3Com) Subject: BOF information I am seeking additional information about the BOF meeting being held in Adelaide, Australia. I need to know the date, times, agenda and a list of attendees if possible. Please let me know who I should contact. Thanks! Licia Morrow lmorrow@flexstar.com From XXX Thu Feb 24 00:37:31 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Feb 23 19:37:30 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA20408 for ; Wed, 23 Feb 2000 19:37:29 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA13941 for ips-outgoing; Wed, 23 Feb 2000 19:36:48 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA13937 for ; Wed, 23 Feb 2000 19:36:45 -0500 (EST) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id TAA04987; Wed, 23 Feb 2000 19:36:45 -0500 (EST) Date: Wed, 23 Feb 2000 19:36:45 -0500 (EST) From: Scott Bradner Message-Id: <200002240036.TAA04987@newdev.harvard.edu> To: ips@ece.cmu.edu, lmorrow@flexstar.com Subject: Re: BOF information Sender: owner-ips@ece.cmu.edu Precedence: bulk > a list of attendees if possible. an attendee list will be compiled in real time when the meeting is held - if you have not been to an IETF meeting before take a look at teh IETF home page for more information (www.ietf.org) Scott From XXX Thu Feb 24 02:35:46 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Feb 23 21:29:09 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id VAA02574 for ; Wed, 23 Feb 2000 21:29:08 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA16515 for ips-outgoing; Wed, 23 Feb 2000 21:26:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from jindo.cisco.com (jindo.cisco.com [171.69.11.73]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA16508 for ; Wed, 23 Feb 2000 21:26:53 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by jindo.cisco.com (8.8.8/2.5.1/Cisco List Logging/8.8.8) with ESMTP id SAA28611; Wed, 23 Feb 2000 18:25:45 -0800 (PST) Date: Wed, 23 Feb 2000 18:25:48 -0800 (PST) From: Costa Sapuntzakis To: ietf@ietf.org cc: ips@ece.cmu.edu Subject: Clarification of IP Storage announcement Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk To clarify a couple items in relation to the announcement posted earlier on IP storage: * it is NOT an announcement of an IETF working group. * IP Storage has not been sanctioned/endorsed by the IESG/IAB/IETF in any way My apologies for any confusion resulting from the original posting. -Costa From XXX Thu Feb 24 12:06:56 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 01:11:24 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id BAA31307 for ; Thu, 24 Feb 2000 01:11:23 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA20929 for ips-outgoing; Thu, 24 Feb 2000 01:10:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA20924 for ; Thu, 24 Feb 2000 01:10:50 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA34118; Thu, 24 Feb 2000 07:10:02 +0100 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id HAA43646; Thu, 24 Feb 2000 07:10:00 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125688F.0021DB37 ; Thu, 24 Feb 2000 07:09:48 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Dave_Lee@3com.com cc: ips@ece.cmu.edu Message-ID: Date: Thu, 24 Feb 2000 08:09:40 +0200 Subject: Re: BOF information Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, The agenda is hardly balanced. I suggested that you reduce your goals presentation to a 5 min summary and post your statement in full on this and/or the IETF board at least 2 weeks before the BOF. Allow also other teams to present their view. An alternative to this might be to have a position paper agreed by a larger population ahead of time. The charter discussion will also be hardly conclusive unless we start drawing it now. It seems also strange that "Storage Industry" is so lightly represented. Regards, Julo Dave_Lee@3com.com on 23/02/2000 21:25:08 Please respond to Dave_Lee@3com.com To: "Morrow, Licia" cc: "'ips@ece.cmu.edu'" (bcc: Julian Satran/Haifa/IBM) Subject: Re: BOF information The BOF meeting is being in held at the March 2000 IETF meeting. http://www.ietf.org/meetings/IETF-47.html A draft agenda is posted there but the IP storage BOF is not currently on it but will be at some point. Current information on the BOF, such as the BOF description, is posted at: http://www.ece.cmu.edu/~ips/ Dave "Morrow, Licia" on 02/23/2000 08:51:16 AM Sent by: "Morrow, Licia" To: "'ips @ece.cmu.edu'" cc: (Dave Lee/HQ/3Com) Subject: BOF information I am seeking additional information about the BOF meeting being held in Adelaide, Australia. I need to know the date, times, agenda and a list of attendees if possible. Please let me know who I should contact. Thanks! Licia Morrow lmorrow@flexstar.com From XXX Thu Feb 24 22:20:46 2000 Forwarded: Thu, 24 Feb 2000 17:20:46 -0500 Forwarded: "rochberg@cs.cmu.edu " Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 16:59:22 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id QAA02116 for ; Thu, 24 Feb 2000 16:59:22 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA24829 for ips-outgoing; Thu, 24 Feb 2000 16:58:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-mailhub-3.cisco.com (sj-mailhub-3.cisco.com [171.68.224.215]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA24819 for ; Thu, 24 Feb 2000 16:58:12 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by sj-mailhub-3.cisco.com (8.9.1a/8.9.1) with ESMTP id OAA01655; Thu, 24 Feb 2000 14:19:35 -0800 (PST) Received: (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) id NAA18955; Thu, 24 Feb 2000 13:56:46 -0800 (PST) Date: Thu, 24 Feb 2000 13:56:46 -0800 (PST) Message-Id: <200002242156.NAA18955@csapuntz-u1.cisco.com> From: Costa Sapuntzakis To: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov cc: gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com Subject: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Sender: owner-ips@ece.cmu.edu Precedence: bulk The TCP RDMA option reduces the overhead of receiving data over TCP-based protocols such as NFS and HTTP. It enables the construction of a simple hardware accelerator that copies data directly from the incoming packet into application buffers, avoiding expensive copies in the protocol stack. Even without hardware acceleration, the option enables the protocol stack to decrease the number of copies it must do. The TCP RDMA option is an annotation and requires no modifications to higher layer protocols. It can be used with popular protocols such as HTTP, NFS, and CIFS, along with new protocols. The TCP option also provides a bit to indicate application-level message boundaries. The bit enables out-of-order processing of the TCP receive queue, potentially decreasing service times in the presence of packet drops and improving performance on parallel systems. A draft describing the TCP RDMA option can be found at: ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt -Costa From XXX Thu Feb 24 23:25:46 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 18:01:09 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id SAA29291 for ; Thu, 24 Feb 2000 18:01:08 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA27029 for ips-outgoing; Thu, 24 Feb 2000 18:00:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA27021 for ; Thu, 24 Feb 2000 18:00:29 -0500 (EST) Received: from sunmail1.Sun.COM ([129.145.1.2]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id QAA26535; Thu, 24 Feb 2000 16:00:14 -0700 (MST) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.84.31]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id PAA15164; Thu, 24 Feb 2000 15:00:14 -0800 (PST) Received: from awe174-18 (awe174-18.AWE.Sun.COM [192.29.174.18]) by jurassic.eng.sun.com (8.9.3+Sun/8.9.3) with SMTP id PAA07352; Thu, 24 Feb 2000 15:00:12 -0800 (PST) Date: Thu, 24 Feb 2000 14:59:38 -0800 (PST) From: Erik Nordmark Reply-To: Erik Nordmark Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: Costa Sapuntzakis Cc: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com In-Reply-To: "Your message with ID" <200002242156.NAA18955@csapuntz-u1.cisco.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > The TCP RDMA option reduces the overhead of receiving data over > TCP-based protocols such as NFS and HTTP. Do you have any data (simulation, implementation) to back up this claim? Or did you mean to say "provides a capbility which an implementation can use to try to reduce the overhead"? > It enables the construction of a simple hardware accelerator that > copies data directly from the incoming packet into application > buffers, avoiding expensive copies in the protocol stack. Even > without hardware acceleration, the option enables the protocol stack > to decrease the number of copies it must do. This seems to be an overstatement as well. Are you saying that an implementation that currently has a single copy in its receive path (from kernel to user space) can "reduce" the number of copies without any hardware acceleration? That would imply that the number of copies could be reduces to zero which I have a hard time understanding (unless you add hardware acceleration). > The TCP RDMA option is an annotation and requires no modifications to > higher layer protocols. It can be used with popular protocols such as > HTTP, NFS, and CIFS, along with new protocols. How will the higher layer protocol peer know what RDMA offsets to use? For this to provide benefits to e.g. NFS it seems like you'd want the NFS peer to have offsets that would depend on which file is being written. Otherwise the only benefit would be for the NIC to be able to separate all the protocol headers from the data so that the data can be placed on e.g. contigiuos pages in memory. > The TCP option also provides a bit to indicate application-level > message boundaries. The bit enables out-of-order processing of the TCP > receive queue, potentially decreasing service times in the presence of > packet drops and improving performance on parallel systems. > > A draft describing the TCP RDMA option can be found at: > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt There is no DNS entry for ftpeng.cisco.com so I can't access the document. Thanks, Erik From XXX Fri Feb 25 00:22:03 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 19:09:25 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA09982 for ; Thu, 24 Feb 2000 19:09:24 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA29060 for ips-outgoing; Thu, 24 Feb 2000 19:09:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from caspian.plutotech.com (caspian.plutotech.com [206.168.67.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA29055 for ; Thu, 24 Feb 2000 19:09:17 -0500 (EST) Received: from caspian.plutotech.com (localhost [127.0.0.1]) by caspian.plutotech.com (8.9.3/8.9.1) with ESMTP id RAA01066; Thu, 24 Feb 2000 17:09:51 -0700 (MST) (envelope-from gibbs@caspian.plutotech.com) Message-Id: <200002250009.RAA01066@caspian.plutotech.com> X-Mailer: exmh version 2.1.0 09/18/1999 To: "David S. Miller" cc: Erik.Nordmark@eng.sun.com, csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. In-reply-to: Your message of "Thu, 24 Feb 2000 15:57:10 PST." <200002242357.PAA16973@pizda.ninka.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 24 Feb 2000 17:09:51 -0700 From: "Justin T. Gibbs" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Date: Thu, 24 Feb 2000 14:59:38 -0800 (PST) > From: Erik Nordmark > >As an aside I think the RDMA proposal has a lot of holes too. For >example, there are in-kernel HTTP accelerators that do the complete >client header parse and initial packet response in the hw interrupt >handler. There are no user buffers involved, and static response >data is DMA'd directly from the filesystem page cache. In the case of a server response, RDMA benefits the client, not the server, so I fail to see why your example is problematic. Zero copy send is not what this standard addresses. -- Justin From XXX Fri Feb 25 00:22:03 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 19:14:50 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA10087 for ; Thu, 24 Feb 2000 19:14:49 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA29221 for ips-outgoing; Thu, 24 Feb 2000 19:14:39 -0500 (EST) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA29217 for ; Thu, 24 Feb 2000 19:14:35 -0500 (EST) Received: (from davem@localhost) by pizda.ninka.net (8.9.3/8.9.3) id QAA16997; Thu, 24 Feb 2000 16:10:26 -0800 Date: Thu, 24 Feb 2000 16:10:26 -0800 Message-Id: <200002250010.QAA16997@pizda.ninka.net> X-Authentication-Warning: pizda.ninka.net: davem set sender to davem@redhat.com using -f From: "David S. Miller" To: gibbs@freebsd.org CC: Erik.Nordmark@eng.sun.com, csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com In-reply-to: <200002250009.RAA01066@caspian.plutotech.com> (gibbs@freebsd.org) Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: <200002250009.RAA01066@caspian.plutotech.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Date: Thu, 24 Feb 2000 17:09:51 -0700 From: "Justin T. Gibbs" In the case of a server response, RDMA benefits the client, not the server, so I fail to see why your example is problematic. Zero copy send is not what this standard addresses. With client memory bus bandwidth in the multi-gigabyte per second range, who needs to avoid the single copy? How much NFS and web surfing does one need to do before this is would really come into play? And the bus speeds will just be faster by the time something like this could be deployed widely. For example, look at SACK, only within the past year are there a decent number of systems out there implementing it. Now how many years ago did it enter RFC state? And there are still stacks out there even in their current development sources not implementing any form of it. Later, David S. Miller davem@redhat.com From XXX Fri Feb 25 00:22:03 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 19:03:02 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA28816 for ; Thu, 24 Feb 2000 19:03:01 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28874 for ips-outgoing; Thu, 24 Feb 2000 19:02:21 -0500 (EST) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28860 for ; Thu, 24 Feb 2000 19:02:15 -0500 (EST) Received: (from davem@localhost) by pizda.ninka.net (8.9.3/8.9.3) id PAA16973; Thu, 24 Feb 2000 15:57:10 -0800 Date: Thu, 24 Feb 2000 15:57:10 -0800 Message-Id: <200002242357.PAA16973@pizda.ninka.net> X-Authentication-Warning: pizda.ninka.net: davem set sender to davem@redhat.com using -f From: "David S. Miller" To: Erik.Nordmark@eng.sun.com CC: csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com In-reply-to: (message from Erik Nordmark on Thu, 24 Feb 2000 14:59:38 -0800 (PST)) Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: Sender: owner-ips@ece.cmu.edu Precedence: bulk Date: Thu, 24 Feb 2000 14:59:38 -0800 (PST) From: Erik Nordmark As an aside I think the RDMA proposal has a lot of holes too. For example, there are in-kernel HTTP accelerators that do the complete client header parse and initial packet response in the hw interrupt handler. There are no user buffers involved, and static response data is DMA'd directly from the filesystem page cache. > A draft describing the TCP RDMA option can be found at: > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt There is no DNS entry for ftpeng.cisco.com so I can't access the document. Here is what I get: ? host -a ftpeng.cisco.com Trying null domain rcode = 0 (Success), ancount=1 The following answer is not authoritative: The following answer is not verified as authentic by the server: ftpeng.cisco.com 84574 IN CNAME ftp-eng.cisco.com For authoritative answers, see: cisco.com 38435 IN NS NS1.cisco.com cisco.com 38435 IN NS NS2.cisco.com Additional information: NS1.cisco.com 78995 IN A 192.31.7.92 NS2.cisco.com 67536 IN A 192.135.250.69 rcode = 0 (Success), ancount=1 The following answer is not authoritative: The following answer is not verified as authentic by the server: ftp-eng.cisco.com 84574 IN A 198.92.30.33 For authoritative answers, see: CISCO.com 38435 IN NS NS1.CISCO.com CISCO.com 38435 IN NS NS2.CISCO.com Additional information: NS1.CISCO.com 78995 IN A 192.31.7.92 NS2.CISCO.com 67536 IN A 192.135.250.69 From XXX Fri Feb 25 00:51:45 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 19:50:47 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA10526 for ; Thu, 24 Feb 2000 19:50:46 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA00166 for ips-outgoing; Thu, 24 Feb 2000 19:50:38 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from caspian.plutotech.com (caspian.plutotech.com [206.168.67.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA00155 for ; Thu, 24 Feb 2000 19:50:32 -0500 (EST) Received: from caspian.plutotech.com (localhost [127.0.0.1]) by caspian.plutotech.com (8.9.3/8.9.1) with ESMTP id RAA01134; Thu, 24 Feb 2000 17:50:52 -0700 (MST) (envelope-from gibbs@caspian.plutotech.com) Message-Id: <200002250050.RAA01134@caspian.plutotech.com> X-Mailer: exmh version 2.1.0 09/18/1999 To: "David S. Miller" cc: gibbs@freebsd.org, Erik.Nordmark@eng.sun.com, csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. In-reply-to: Your message of "Thu, 24 Feb 2000 16:10:26 PST." <200002250010.QAA16997@pizda.ninka.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 24 Feb 2000 17:50:52 -0700 From: "Justin T. Gibbs" Sender: owner-ips@ece.cmu.edu Precedence: bulk > In the case of a server response, RDMA benefits the client, not the > server, so I fail to see why your example is problematic. Zero > copy send is not what this standard addresses. > >With client memory bus bandwidth in the multi-gigabyte per second >range, who needs to avoid the single copy? How much NFS and web >surfing does one need to do before this is would really come into >play? This is a very different argument than the one you "implied" before. I can certainly say that I'd rather make use of my memory bandwidth, regardless of how much I happen to have, in a more constructive manner than copying data for no good reason. RDMA is a general purpose feature. Don't shoe horn it into just an option to accelerate a few protocols. >And the bus speeds will just be faster by the time something like >this could be deployed widely. And the networks will be faster by then too and the demand for pulling more rich content will increase, etc. >For example, look at SACK, only within the past year are there a >decent number of systems out there implementing it. Now how many >years ago did it enter RFC state? And there are still stacks out >there even in their current development sources not implementing any >form of it. The performance impact of RDMA is quite a bit larger than SACK, so I don't know that your example is relevant. All the big vendors implement zero-copy in some shape or form and since RDMA is a scheme to make zero-copy work in more cases, I'm sure it will be picked up if the proposal is deemed sane. >Later, >David S. Miller >davem@redhat.com -- Justin From XXX Fri Feb 25 00:51:45 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 19:36:55 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA10329 for ; Thu, 24 Feb 2000 19:36:54 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA29812 for ips-outgoing; Thu, 24 Feb 2000 19:36:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from seattle.3com.com (seattle.3com.com [129.213.128.97]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA29807 for ; Thu, 24 Feb 2000 19:36:47 -0500 (EST) From: Dave_Lee@3com.com Received: from new-york.3com.com (new-york.3com.com [129.213.157.12]) by seattle.3com.com (8.8.8/8.8.8) with ESMTP id QAA01522; Thu, 24 Feb 2000 16:36:46 -0800 (PST) Received: from hqoutbound.ops.3com.com (hqoutbound.OPS.3Com.COM [139.87.48.104]) by new-york.3com.com (8.8.8/8.8.8) with SMTP id QAA02754; Thu, 24 Feb 2000 16:36:45 -0800 (PST) Received: by hqoutbound.ops.3com.com(Lotus SMTP MTA v4.6.3 (778.2 1-4-1999)) id 88256890.00032639 ; Thu, 24 Feb 2000 16:34:23 -0800 X-Lotus-FromDomain: 3COM To: julian_satran@il.ibm.com cc: ips@ece.cmu.edu Message-ID: <88256890.000322D9.00@hqoutbound.ops.3com.com> Date: Thu, 24 Feb 2000 16:36:05 -0800 Subject: Re: BOF information Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Alternative views in terms of the charter is what we'd like to see and hopefully they will be discussed on this list. Whatever is presented will hopefully be the rough consensus to-date so feel free to pitch in and/or initiate specific discussions. What was presented at the Feb 16 meeting was a strawman to help get the discussions started. I believe there will be a joint requirement/issues document produced by folks at Adaptec and Cisco prior to the BOF and circulated for comments. I'm sure they will welcome input and help in the drafting process (contact paulv@corp.adaptec.com or csapuntz@cisco.com). While there are a small number of people from the storage industry involved, keep in mind that the storage industry is not generally familiar with the IETF. Dave julian_satran@il.ibm.com on 02/23/2000 10:09:40 PM Sent by: julian_satran@il.ibm.com To: Dave Lee/HQ/3Com cc: ips@ece.cmu.edu Subject: Re: BOF information Dave, The agenda is hardly balanced. I suggested that you reduce your goals presentation to a 5 min summary and post your statement in full on this and/or the IETF board at least 2 weeks before the BOF. Allow also other teams to present their view. An alternative to this might be to have a position paper agreed by a larger population ahead of time. The charter discussion will also be hardly conclusive unless we start drawing it now. It seems also strange that "Storage Industry" is so lightly represented. Regards, Julo Dave_Lee@3com.com on 23/02/2000 21:25:08 Please respond to Dave_Lee@3com.com To: "Morrow, Licia" cc: "'ips@ece.cmu.edu'" (bcc: Julian Satran/Haifa/IBM) Subject: Re: BOF information The BOF meeting is being in held at the March 2000 IETF meeting. http://www.ietf.org/meetings/IETF-47.html A draft agenda is posted there but the IP storage BOF is not currently on it but will be at some point. Current information on the BOF, such as the BOF description, is posted at: http://www.ece.cmu.edu/~ips/ Dave "Morrow, Licia" on 02/23/2000 08:51:16 AM Sent by: "Morrow, Licia" To: "'ips @ece.cmu.edu'" cc: (Dave Lee/HQ/3Com) Subject: BOF information I am seeking additional information about the BOF meeting being held in Adelaide, Australia. I need to know the date, times, agenda and a list of attendees if possible. Please let me know who I should contact. Thanks! Licia Morrow lmorrow@flexstar.com From XXX Fri Feb 25 00:56:45 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 19:54:10 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA10533 for ; Thu, 24 Feb 2000 19:54:09 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA00306 for ips-outgoing; Thu, 24 Feb 2000 19:54:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA00298 for ; Thu, 24 Feb 2000 19:53:57 -0500 (EST) Received: from sunmail1.Sun.COM ([129.145.1.2]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id RAA27463; Thu, 24 Feb 2000 17:53:55 -0700 (MST) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.88.31]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id QAA07619; Thu, 24 Feb 2000 16:53:55 -0800 (PST) Received: from shield (shield.Eng.Sun.COM [129.146.85.114]) by jurassic.eng.sun.com (8.9.3+Sun/8.9.3) with SMTP id QAA29178; Thu, 24 Feb 2000 16:53:54 -0800 (PST) Date: Thu, 24 Feb 2000 16:53:53 -0800 (PST) From: Kacheong Poon Reply-To: Kacheong Poon Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: tcp-impl@grc.nasa.gov Cc: ips@ece.cmu.edu, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com In-Reply-To: "Your message with ID" <200002242357.PAA16973@pizda.ninka.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > As an aside I think the RDMA proposal has a lot of holes too. For > example, there are in-kernel HTTP accelerators that do the complete > client header parse and initial packet response in the hw interrupt > handler. There are no user buffers involved, and static response > data is DMA'd directly from the filesystem page cache. Even without the above special acceleration, section 2 in the draft overestimates the number of copies involved in most OSes nowadays. > In the case of a server response, RDMA benefits the client, not the > server, so I fail to see why your example is problematic. Zero copy > send is not what this standard addresses. Even without RDMA, people have done zero copy receive with TCP/IP before. What I see in RDMA is a way to provide message boundary in TCP so that apps do not need to spend time on that (I call them lazy apps). I don't see right away the other claims in the draft. It seems to me that the authors should re-evaluate their claims and provide support to them. I don't see any arguments in the draft. If message boundary is the only benefit, I think the SCTP proposal is more interesting. Check draft-ietf-sigtran-sctp-06.txt. K. Poon. kcpoon@eng.sun.com From XXX Fri Feb 25 01:06:32 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 20:06:10 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA00921 for ; Thu, 24 Feb 2000 20:06:09 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA00649 for ips-outgoing; Thu, 24 Feb 2000 20:06:00 -0500 (EST) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA00644 for ; Thu, 24 Feb 2000 20:05:56 -0500 (EST) Received: (from davem@localhost) by pizda.ninka.net (8.9.3/8.9.3) id RAA17054; Thu, 24 Feb 2000 17:01:34 -0800 Date: Thu, 24 Feb 2000 17:01:34 -0800 Message-Id: <200002250101.RAA17054@pizda.ninka.net> X-Authentication-Warning: pizda.ninka.net: davem set sender to davem@redhat.com using -f From: "David S. Miller" To: gibbs@freebsd.org CC: gibbs@freebsd.org, Erik.Nordmark@eng.sun.com, csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, zaitcev@metabyte.com, drich@fjst.com In-reply-to: <200002250050.RAA01134@caspian.plutotech.com> (gibbs@freebsd.org) Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: <200002250050.RAA01134@caspian.plutotech.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Date: Thu, 24 Feb 2000 17:50:52 -0700 From: "Justin T. Gibbs" The performance impact of RDMA is quite a bit larger than SACK, It depends who you are. For someone over a satellite link, I think SACK benefits them much more than RDMA. Later, David S. Miller davem@redhat.com From XXX Fri Feb 25 01:56:40 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 20:54:30 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA01531 for ; Thu, 24 Feb 2000 20:54:29 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA01721 for ips-outgoing; Thu, 24 Feb 2000 20:52:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA01717 for ; Thu, 24 Feb 2000 20:52:53 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id RAA07438; Thu, 24 Feb 2000 17:52:43 -0800 (PST) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (clock.engr.sgi.com [163.154.34.45]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id RAA93030; Thu, 24 Feb 2000 17:52:30 -0800 (PST) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (localhost [127.0.0.1]) by clock.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id RAA71116; Thu, 24 Feb 2000 17:56:26 -0800 (PST) Message-Id: <200002250156.RAA71116@clock.engr.sgi.com> X-Mailer: exmh version 2.0.2 2/24/98 To: "David S. Miller" Cc: Erik.Nordmark@eng.sun.com, csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. From: Zachary Amsden In-Reply-To: Your message of "Thu, 24 Feb 2000 16:10:26 PST." <200002250010.QAA16997@pizda.ninka.net> Date: Thu, 24 Feb 2000 17:56:26 -0800 Sender: owner-ips@ece.cmu.edu Precedence: bulk > Date: Thu, 24 Feb 2000 17:09:51 -0700 > From: "Justin T. Gibbs" > > In the case of a server response, RDMA benefits the client, not the > server, so I fail to see why your example is problematic. Zero > copy send is not what this standard addresses. > > With client memory bus bandwidth in the multi-gigabyte per second > range, who needs to avoid the single copy? How much NFS and web > surfing does one need to do before this is would really come into > play? With network bandwidth approaching memory bus bandwidth, it becomes an issue. A single copy receive uses 3x the bus bandwidth of a zero copy implementation (or worse if not aligned properly), and also poisons many more cachelines. -- Zachary Amsden zamsden@engr.sgi.com (650) 933-6919 09U-510 Core Protocols From XXX Fri Feb 25 01:56:40 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 20:34:38 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA01330 for ; Thu, 24 Feb 2000 20:34:37 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA01329 for ips-outgoing; Thu, 24 Feb 2000 20:34:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA01321 for ; Thu, 24 Feb 2000 20:34:25 -0500 (EST) Received: from sunmail1.Sun.COM ([129.145.1.2]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id SAA19583; Thu, 24 Feb 2000 18:34:17 -0700 (MST) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.86.31]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id RAA15294; Thu, 24 Feb 2000 17:34:17 -0800 (PST) Received: from shield (shield.Eng.Sun.COM [129.146.85.114]) by jurassic.eng.sun.com (8.9.3+Sun/8.9.3) with SMTP id RAA05249; Thu, 24 Feb 2000 17:34:15 -0800 (PST) Date: Thu, 24 Feb 2000 17:34:14 -0800 (PST) From: Kacheong Poon Reply-To: Kacheong Poon Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: tcp-impl@grc.nasa.gov, "Justin T. Gibbs" Cc: csapuntz@cisco.com, ips@ece.cmu.edu, zaitcev@metabyte.com, drich@fjst.com In-Reply-To: "Your message with ID" <200002250050.RAA01134@caspian.plutotech.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > The performance impact of RDMA is quite a bit larger than SACK, so > I don't know that your example is relevant. All the big vendors > implement zero-copy in some shape or form and since RDMA is a scheme > to make zero-copy work in more cases, I'm sure it will be picked up > if the proposal is deemed sane. Can you elaborate on this? Suppose TCP "blindly" does zero copy everything to an app's buffer (for example, to a web browser's receive buffer) without RDMA. Then the browser app looks at the data and displays it. What is the difference RDMA makes in this case? Yes, RDMA can separate different messages in the buffer. But this can also be done by the browser app, not by TCP. I guess this is what I suggest the authors to add to the draft. It is not clear to me how RDMA can make a difference, especially in those cases the authors claim a big performance difference. K. Poon. kcpoon@eng.sun.com From XXX Fri Feb 25 01:56:40 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 20:42:15 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA01433 for ; Thu, 24 Feb 2000 20:42:15 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA01508 for ips-outgoing; Thu, 24 Feb 2000 20:42:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA01504 for ; Thu, 24 Feb 2000 20:42:08 -0500 (EST) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id RAA07029; Thu, 24 Feb 2000 17:42:07 -0800 (PST) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwkb.EBay.Sun.COM [129.150.144.211]) by ebaymail2.EBay.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6) with ESMTP id RAA18385; Thu, 24 Feb 2000 17:42:05 -0800 (PST) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id RAA17745; Thu, 24 Feb 2000 17:42:05 -0800 (PST) Message-Id: <200002250142.RAA17745@ha10nwk.EBay.Sun.COM> Date: Thu, 24 Feb 2000 17:42:05 -0800 (PST) From: David Robinson Reply-To: David Robinson Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: ips@ece.cmu.edu Cc: tcp-impl@grc.nasa.gov MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: dRBoRwrHEPdRhrLF5rMnhA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk To efficently determine boundaries within a packet stream work must be done somewhere. In the RDMA proposal it is up to the clients to do the work to make the server's job easier. In traditional intelligent NIC cards the server does the work by parsing the headers. It seems that the design of RDMA is backwards as it relies on changes to the many clients to enable efficiency on the server. A traditional intelligent NIC card with a modest amount of hardware/firmware can handle 99+% of requests from unmodified clients. The existence proof is checksumming NICs and NFS accelerator boards. For an efficient IP storage device it will have to deal with legacy IP client stacks (no RDMA) and a competitive IP storage vendor will implement the smart NIC described above. Why is RDMA more compelling? -David From XXX Fri Feb 25 02:12:56 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 21:09:51 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id VAA24373 for ; Thu, 24 Feb 2000 21:09:51 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA02070 for ips-outgoing; Thu, 24 Feb 2000 21:08:08 -0500 (EST) Received: from pizda.ninka.net (IDENT:root@pizda.ninka.net [216.101.162.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA02064 for ; Thu, 24 Feb 2000 21:08:01 -0500 (EST) Received: (from davem@localhost) by pizda.ninka.net (8.9.3/8.9.3) id SAA17132; Thu, 24 Feb 2000 18:03:31 -0800 Date: Thu, 24 Feb 2000 18:03:31 -0800 Message-Id: <200002250203.SAA17132@pizda.ninka.net> X-Authentication-Warning: pizda.ninka.net: davem set sender to davem@redhat.com using -f From: "David S. Miller" To: zamsden@cthulhu.engr.sgi.com CC: Erik.Nordmark@eng.sun.com, csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com In-reply-to: <200002250156.RAA71116@clock.engr.sgi.com> (message from Zachary Amsden on Thu, 24 Feb 2000 17:56:26 -0800) Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: <200002250156.RAA71116@clock.engr.sgi.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk From: Zachary Amsden Date: Thu, 24 Feb 2000 17:56:26 -0800 With network bandwidth approaching memory bus bandwidth, it becomes an issue. A single copy receive uses 3x the bus bandwidth of a zero copy implementation (or worse if not aligned properly), and also poisons many more cachelines. Before we discuss this point further, does anyone have real hard evidence that cpu cycles on the client side are the issue for the vast majority of systems out there? In my experience cpu cycles are abundant on client machines. Server side is where getting precious cpu cycles back seems more important. Client side cpu cycles are typically being expent on tasks such as IDCT transforms to decode audio/visual streams, executing Java code, but not copying the data from the network. Next, have any other implementors investigated the cpu usage gains obtainable from deferring TCP receive packet processing to user context? I have seen it make significant improvements, and whats more this helps out all systems, this means old applications and unsophisticated hardware. (I am referring to Jakobson's idea of nearly 10 years ago, and it appears once again he was right.) Finally, what does RDMA do in the presence of SACK options? Which set of options gets kicked out? Should the RDMA options go in and we just drop the SACK blocks? Or the other way around? Later, David S. Miller davem@redhat.com From XXX Fri Feb 25 02:33:14 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 21:25:36 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id VAA24571 for ; Thu, 24 Feb 2000 21:25:35 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA02425 for ips-outgoing; Thu, 24 Feb 2000 21:24:01 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from calcite.rhyolite.com (calcite.rhyolite.com [38.159.140.3]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA02413 for ; Thu, 24 Feb 2000 21:23:55 -0500 (EST) Received: (from vjs@localhost) by calcite.rhyolite.com (8.9.3/calcite) id TAA25413 env-from ; Thu, 24 Feb 2000 19:23:47 -0700 (MST) Date: Thu, 24 Feb 2000 19:23:47 -0700 (MST) From: Vernon Schryver Message-Id: <200002250223.TAA25413@calcite.rhyolite.com> To: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: Erik Nordmark > > A draft describing the TCP RDMA option can be found at: > > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt > > There is no DNS entry for ftpeng.cisco.com so I can't access the document. ftpeng.cisco.com resolves for me to 198.92.30.33, and the URL works ftpeng.cisco.com does not answer ICMP Echo-Requests. It also seems that Cisco is filtering ICMP TTL Exceeded. Oh, well. I predict that soon traceroute and ping will be as effective as if the Internet were run by the old line telco managers who went great lengths to keep their technical problems quite. The recent security hassles will be a handy (and quite silly) excuse. (Yes, of course, Cisco has every right to filter however they want. I'm talking about technical sense, not rights.) I'm even less impressed about the proposal than Erik Nordmark, perhaps because more than 10 years ago I saw systems shipped by more than one competitor of Sun Microsystems that paged flipped NFS/UDP and user TCP data. (well, one of the other vendors might have been a little more recent 10 years.) The motive for the proposal seems to be that while only a very few CPU instructions are needed to page flip, the functions of those CPU instructions are very hard in hardware. I don't agree. In today's world of ASIC's, silicon to figure out where to drop incoming TCP segments or NFS/UDP/IP fragments based only on old fashioned TCP and RPC/XRD/UDP headers is nothing to write home about. It wasn't even all that big a deal more than 10 years ago, as everyone involved with or who watched Protocol Engines Inc. remembers. Hashing is almost as cool (and easy) in hardware as in software. Vernon Schryver vjs@rhyolite.com From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 04:56:08 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id EAA32027 for ; Fri, 25 Feb 2000 04:56:07 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA18077 for ips-outgoing; Fri, 25 Feb 2000 04:55:57 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA17988 for ; Fri, 25 Feb 2000 04:55:48 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA53310; Fri, 25 Feb 2000 10:55:12 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id KAA44992; Fri, 25 Feb 2000 10:55:10 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256890.00367577 ; Fri, 25 Feb 2000 10:54:50 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: David Robinson cc: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Message-ID: Date: Fri, 25 Feb 2000 11:54:25 +0200 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Message boundaries are only part of the proposal - and they don't imply additional work at the client. Doing zero copy based only on the information in current headers is certainly possible at low speed. Over 1 Gb/s it requires some innovation and lots of silicon. The RDMA option makes it possible at a far lower price. And the zero copy it enables might go deep into the application space as it is only an annotation on packets. It certainly makes sense on all the new applications (NFS4, SCSI, etc) and the retrofit into existing ones is not that difficult either. And placing it over TCP puts it on a safer ground that having to use at higher speed completely new and unproven protocols (VIA, NGIO etc.). Julo Julian Satran - IBM Research David Robinson on 25/02/2000 03:42:05 Please respond to David Robinson To: ips@ece.cmu.edu cc: tcp-impl@grc.nasa.gov (bcc: Julian Satran/Haifa/IBM) Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To efficently determine boundaries within a packet stream work must be done somewhere. In the RDMA proposal it is up to the clients to do the work to make the server's job easier. In traditional intelligent NIC cards the server does the work by parsing the headers. It seems that the design of RDMA is backwards as it relies on changes to the many clients to enable efficiency on the server. A traditional intelligent NIC card with a modest amount of hardware/firmware can handle 99+% of requests from unmodified clients. The existence proof is checksumming NICs and NFS accelerator boards. For an efficient IP storage device it will have to deal with legacy IP client stacks (no RDMA) and a competitive IP storage vendor will implement the smart NIC described above. Why is RDMA more compelling? -David From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 22:31:00 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id WAA15585 for ; Thu, 24 Feb 2000 22:31:00 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA04048 for ips-outgoing; Thu, 24 Feb 2000 22:29:07 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pneumatic-tube.sgi.com (pneumatic-tube.sgi.com [204.94.214.22]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA04044 for ; Thu, 24 Feb 2000 22:29:04 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id TAA00629; Thu, 24 Feb 2000 19:31:49 -0800 (PST) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (clock.engr.sgi.com [163.154.34.45]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id TAA68813; Thu, 24 Feb 2000 19:28:41 -0800 (PST) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (localhost [127.0.0.1]) by clock.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id TAA71655; Thu, 24 Feb 2000 19:32:41 -0800 (PST) Message-Id: <200002250332.TAA71655@clock.engr.sgi.com> X-Mailer: exmh version 2.0.2 2/24/98 To: "David S. Miller" Cc: Erik.Nordmark@eng.sun.com, csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. From: Zachary Amsden In-Reply-To: Your message of "Thu, 24 Feb 2000 18:03:31 PST." <200002250203.SAA17132@pizda.ninka.net> Date: Thu, 24 Feb 2000 19:32:40 -0800 Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: Zachary Amsden > Date: Thu, 24 Feb 2000 17:56:26 -0800 > > With network bandwidth approaching memory bus bandwidth, it becomes > an issue. A single copy receive uses 3x the bus bandwidth of a > zero copy implementation (or worse if not aligned properly), and > also poisons many more cachelines. > In my experience cpu cycles are abundant on client machines. Server > side is where getting precious cpu cycles back seems more important. Agreed for the most part. But reducing latency and cache misses are just as important on servers, and it would be interesting to see whether RDMA would make a difference there. Of course for file/web serving, the server receive fast path can all be done in-kernel anyways, and output volume dwarfs input volume. I can come up with a rigged scenario where RDMA should make a difference: Group server or database which needs large amounts of external data over the network via NFS/your protocol of choice. Internet search engines would possibly fall into this category. In this case, input/output ratio is drastically different than a typical server, and memory bandwidth used by copying could become an issue. However, this is a pretty limited use, and would only need to be deployed on a few machines. I don't see RDMA being beneficial for typical web/file servers, or clients, other than getting a couple % extra performance. -- Zachary Amsden zamsden@engr.sgi.com (650) 933-6919 09U-510 Core Protocols From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 03:07:35 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id DAA02133 for ; Fri, 25 Feb 2000 03:07:34 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA09114 for ips-outgoing; Fri, 25 Feb 2000 03:05:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from prue.eim.surrey.ac.uk (IDENT:exim@prue.eim.surrey.ac.uk [131.227.76.5]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA09109 for ; Fri, 25 Feb 2000 03:05:47 -0500 (EST) Received: from petra.ee.surrey.ac.uk ([131.227.88.13] ident=eep1lw) by prue.eim.surrey.ac.uk with esmtp (Exim 3.03 #1) id 12OFko-0004nz-00; Fri, 25 Feb 2000 08:05:38 +0000 Date: Fri, 25 Feb 2000 08:05:34 +0000 (GMT) From: Lloyd Wood X-Sender: eep1lw@petra.ee.surrey.ac.uk Reply-To: L.Wood@eim.surrey.ac.uk To: Vernon Schryver cc: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, ietf@ietf.org Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. In-Reply-To: <200002250223.TAA25413@calcite.rhyolite.com> Message-ID: Organization: speaking for none X-url: http://www.ee.surrey.ac.uk/Personal/L.Wood/ X-no-archive: yes MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk On Thu, 24 Feb 2000, Vernon Schryver wrote: > > From: Erik Nordmark > > > > A draft describing the TCP RDMA option can be found at: > > > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt If this _was_ a draft, it would be available from http://www.ietf.org/internet-drafts/ (and I imagine that it wouldn't go against RFC2026, or be copyright Cisco. I see that e.g. ftp://ftp.ietf.org/internet-drafts/draft-satran-scot-00.txt has similar wording. How widespread is this practice?) > > There is no DNS entry for ftpeng.cisco.com so I can't access the document. > > ftpeng.cisco.com resolves for me to 198.92.30.33, and the URL works It's an alias for ftp-eng. there doesn't appear to be a reverse DNS entry, though. > ftpeng.cisco.com does not answer ICMP Echo-Requests. It also seems that > Cisco is filtering ICMP TTL Exceeded. > > Oh, well. I predict that soon traceroute and ping will be as > effective as if the Internet were run by the old line telco managers > who went great lengths to keep their technical problems quite. Bear in mind that both traceroute and ping were effective one-person opportunistic hacks based on using an existing infrastructure in unexpected ways. If they'd first been tediously designed by a committee, standardised and mandated, things might be different. they probably wouldn't work as well, but they'd be on buzzword-compliant feature lists. > I'm even less impressed about the proposal than Erik Nordmark, Note the mentions of SCSI and SCSI/TCP and the tie-in with the proposed IP Storage efforts (recent ietf general list discussion). I'd still like to know _why_. L. SCSI DMA over TCP? What _is_ all this aiming for - trying to build distributed RAID arrays with really poor performance that are subject to WAN outages and DoS attacks? PGP From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 00:04:20 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id AAA29477 for ; Fri, 25 Feb 2000 00:04:20 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA06125 for ips-outgoing; Fri, 25 Feb 2000 00:02:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ns1.metabyte.com (ns1.metabyte.com [216.218.208.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA06120 for ; Fri, 25 Feb 2000 00:02:20 -0500 (EST) Received: (from zaitcev@localhost) by ns1.metabyte.com (8.9.1a/8.9.1/akolb/110398) id VAA28688; Thu, 24 Feb 2000 21:01:22 -0800 (PST) From: Pete Zaitcev Message-Id: <200002250501.VAA28688@ns1.metabyte.com> Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: sob@harvard.edu (Scott Bradner) Date: Thu, 24 Feb 2000 21:01:22 -0800 (PST) Cc: charlese@cvs.com.au, csapuntz@cisco.com, drich@fjst.com, gibbs@freebsd.org, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, zaitcev@metabyte.com In-Reply-To: <200002250344.WAA01141@newdev.harvard.edu> from "Scott Bradner" at Feb 24, 2000 10:44:55 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > > 1) Given that the first paragraph says section 10 of rfc2026 is irrelevant, > > under what conditions is the document published? > > actually, since it has not shown up in the Internet Drafts directory it > has not been published in the sense that it could be the topic > of a work item in an IETF working group > > so, while the topic may be interesting I'd suggest that it be ignored > in the contect of an IPS BOF (and note that when published it will not > say that 2026 is "irrelevant" cuz if it does it will not get published) > > Scott Very well, but what about its companion document (SCOT)? http://search.ietf.org/internet-drafts/draft-satran-scot-00.txt It is published, isn't it? It was somewhat disturbing to see the notice, but on the other hand it was honest. IBM could just as easily come up silently with a silly software patent for RDMA option or for SCSI over TCP idea as such. --Pete From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 07:41:22 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id HAA05200 for ; Fri, 25 Feb 2000 07:41:21 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA20904 for ips-outgoing; Fri, 25 Feb 2000 07:41:08 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from the-village.bc.nu (lightning.swansea.uk.linux.org [194.168.151.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA20900 for ; Fri, 25 Feb 2000 07:41:04 -0500 (EST) Received: from alan by the-village.bc.nu with local (Exim 2.12 #1) id 12OK1V-0007YW-00; Fri, 25 Feb 2000 12:39:09 +0000 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: gibbs@FreeBSD.org (Justin T. Gibbs) Date: Fri, 25 Feb 2000 12:39:05 +0000 (GMT) Cc: Kacheong.Poon@eng.sun.com (Kacheong Poon), tcp-impl@grc.nasa.gov, csapuntz@cisco.com, ips@ece.cmu.edu, zaitcev@metabyte.com, drich@fjst.com In-Reply-To: <200002250513.WAA01485@caspian.plutotech.com> from "Justin T. Gibbs" at Feb 24, 2000 10:13:07 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-ips@ece.cmu.edu Precedence: bulk > flip is then performed to get the data where the user wants it, > imposing the restriction that your payload be page sized so you don't > leave gaps in the user's destination buffer. Certainly, with a more Perhaps its about time the world put together an official, sane, ring buffer style mmap socket api. A lot of the requirement to align data is coming from the existing socket API. > * This technique has been implemented with custom firmware on Alteon > Gig-E cards for a product I work on. Nifty. From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 00:14:08 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id AAA29578 for ; Fri, 25 Feb 2000 00:14:07 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA06405 for ips-outgoing; Fri, 25 Feb 2000 00:12:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from caspian.plutotech.com (caspian.plutotech.com [206.168.67.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA06400 for ; Fri, 25 Feb 2000 00:12:32 -0500 (EST) Received: from caspian.plutotech.com (localhost [127.0.0.1]) by caspian.plutotech.com (8.9.3/8.9.1) with ESMTP id WAA01485; Thu, 24 Feb 2000 22:13:07 -0700 (MST) (envelope-from gibbs@caspian.plutotech.com) Message-Id: <200002250513.WAA01485@caspian.plutotech.com> X-Mailer: exmh version 2.1.0 09/18/1999 To: Kacheong Poon cc: tcp-impl@grc.nasa.gov, csapuntz@cisco.com, ips@ece.cmu.edu, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. In-reply-to: Your message of "Thu, 24 Feb 2000 17:34:14 PST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 24 Feb 2000 22:13:07 -0700 From: "Justin T. Gibbs" Sender: owner-ips@ece.cmu.edu Precedence: bulk >> The performance impact of RDMA is quite a bit larger than SACK, so >> I don't know that your example is relevant. All the big vendors >> implement zero-copy in some shape or form and since RDMA is a scheme >> to make zero-copy work in more cases, I'm sure it will be picked up >> if the proposal is deemed sane. > >Can you elaborate on this? Suppose TCP "blindly" does zero copy everything to >an app's buffer (for example, to a web browser's receive buffer) without >RDMA. Then the browser app looks at the data and displays it. What is the >difference RDMA makes in this case? Yes, RDMA can separate different messages >in the buffer. But this can also be done by the browser app, not by TCP. You seem to be saying that in the common case zero copy is achievable. Most implementations I've seen require the network driver to make a guess about where the payload will be in an incoming packet so the header can be stripped off and the payload dmaed to an aligned area. A page flip is then performed to get the data where the user wants it, imposing the restriction that your payload be page sized so you don't leave gaps in the user's destination buffer. Certainly, with a more intelligent network adapter that knows every protocol you can determine exactly where the data is in each packet. If you add connection tracking and sequence number sniffing to the nic with a mechanism to register user buffers to connections, you can get zero copy every time*. Unfortunately this is not very general purpose solution. The point of RDMA seems to be to allow nic manufacturers to add support for a single tcp option that, at the very least, allows the nic to align the payload for you. Add RID registration with the nic and you get the payload exactly where you want it too. All without too much state information kept by the nic. * This technique has been implemented with custom firmware on Alteon Gig-E cards for a product I work on. single protocol -- Justin From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 22:47:06 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id WAA15701 for ; Thu, 24 Feb 2000 22:47:05 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA04432 for ips-outgoing; Thu, 24 Feb 2000 22:45:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA04425 for ; Thu, 24 Feb 2000 22:45:31 -0500 (EST) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id WAA01141; Thu, 24 Feb 2000 22:44:55 -0500 (EST) Date: Thu, 24 Feb 2000 22:44:55 -0500 (EST) From: Scott Bradner Message-Id: <200002250344.WAA01141@newdev.harvard.edu> To: charlese@cvs.com.au, csapuntz@cisco.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Cc: drich@fjst.com, gibbs@freebsd.org, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, zaitcev@metabyte.com Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles asks: > 1) Given that the first paragraph says section 10 of rfc2026 is > irrelevant, > under what conditions is the document published? actually, since it has not shown up in the Internet Drafts directory it has not been published in the sense that it could be the topic of a work item in an IETF working group so, while the topic may be interesting I'd suggest that it be ignored in the contect of an IPS BOF (and note that when published it will not say that 2026 is "irrelevant" cuz if it does it will not get published) Scott From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 05:13:32 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id FAA22487 for ; Fri, 25 Feb 2000 05:13:31 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA18772 for ips-outgoing; Fri, 25 Feb 2000 05:13:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA18768 for ; Fri, 25 Feb 2000 05:13:23 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA113932; Fri, 25 Feb 2000 11:12:48 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id LAA69446; Fri, 25 Feb 2000 11:12:44 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256890.0038164D ; Fri, 25 Feb 2000 11:12:37 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Vernon Schryver cc: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Message-ID: Date: Fri, 25 Feb 2000 12:12:25 +0200 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk That is not completely accurate. You will need appreciably more silicon to do what you suggest. And you can do it only with information that "passes through the protocol" . The good thing about the proposal is that it can TAG whatever the application wants (and that can be several layers away from the protocol). You can't "page-flip" to buffers that you are not aware of. And page flipping wherever is applicable assumes also page boundaries for buffers. Julo Julian Satran - IBM Research Vernon Schryver on 25/02/2000 04:23:47 Please respond to Vernon Schryver To: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. > From: Erik Nordmark > > A draft describing the TCP RDMA option can be found at: > > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt > > There is no DNS entry for ftpeng.cisco.com so I can't access the document. ftpeng.cisco.com resolves for me to 198.92.30.33, and the URL works ftpeng.cisco.com does not answer ICMP Echo-Requests. It also seems that Cisco is filtering ICMP TTL Exceeded. Oh, well. I predict that soon traceroute and ping will be as effective as if the Internet were run by the old line telco managers who went great lengths to keep their technical problems quite. The recent security hassles will be a handy (and quite silly) excuse. (Yes, of course, Cisco has every right to filter however they want. I'm talking about technical sense, not rights.) I'm even less impressed about the proposal than Erik Nordmark, perhaps because more than 10 years ago I saw systems shipped by more than one competitor of Sun Microsystems that paged flipped NFS/UDP and user TCP data. (well, one of the other vendors might have been a little more recent 10 years.) The motive for the proposal seems to be that while only a very few CPU instructions are needed to page flip, the functions of those CPU instructions are very hard in hardware. I don't agree. In today's world of ASIC's, silicon to figure out where to drop incoming TCP segments or NFS/UDP/IP fragments based only on old fashioned TCP and RPC/XRD/UDP headers is nothing to write home about. It wasn't even all that big a deal more than 10 years ago, as everyone involved with or who watched Protocol Engines Inc. remembers. Hashing is almost as cool (and easy) in hardware as in software. Vernon Schryver vjs@rhyolite.com From XXX Fri Feb 25 13:05:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Feb 24 22:39:20 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id WAA15597 for ; Thu, 24 Feb 2000 22:39:19 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA04265 for ips-outgoing; Thu, 24 Feb 2000 22:37:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from kitty.netconnect.com.au (ren.netconnect.com.au [203.7.198.1]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id WAA04260 for ; Thu, 24 Feb 2000 22:37:42 -0500 (EST) Received: (qmail 17574 invoked from network); 25 Feb 2000 03:37:36 -0000 Received: from unknown (HELO cvs.com.au) (203.87.14.203) by mail.netconnect.com.au with SMTP; 25 Feb 2000 03:37:36 -0000 Message-ID: <38B5BA9B.1D7C5957@cvs.com.au> Date: Fri, 25 Feb 2000 10:11:23 +1100 From: Charles Esson X-Mailer: Mozilla 4.5 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: Costa Sapuntzakis CC: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: <200002242156.NAA18955@csapuntz-u1.cisco.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk 1) Given that the first paragraph says section 10 of rfc2026 is irrelevant, under what conditions is the document published? Costa Sapuntzakis wrote: > The TCP RDMA option reduces the overhead of receiving data over > TCP-based protocols such as NFS and HTTP. > > It enables the construction of a simple hardware accelerator that > copies data directly from the incoming packet into application > buffers, avoiding expensive copies in the protocol stack. Even > without hardware acceleration, the option enables the protocol stack > to decrease the number of copies it must do. > > The TCP RDMA option is an annotation and requires no modifications to > higher layer protocols. It can be used with popular protocols such as > HTTP, NFS, and CIFS, along with new protocols. > > The TCP option also provides a bit to indicate application-level > message boundaries. The bit enables out-of-order processing of the TCP > receive queue, potentially decreasing service times in the presence of > packet drops and improving performance on parallel systems. > > A draft describing the TCP RDMA option can be found at: > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt > > -Costa From XXX Fri Feb 25 13:28:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 08:21:35 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id IAA28430 for ; Fri, 25 Feb 2000 08:21:34 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA21691 for ips-outgoing; Fri, 25 Feb 2000 08:21:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from kitty.netconnect.com.au (ren.netconnect.com.au [203.7.198.1]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id IAA21682 for ; Fri, 25 Feb 2000 08:21:26 -0500 (EST) Received: (qmail 1074 invoked from network); 25 Feb 2000 13:21:59 -0000 Received: from unknown (HELO cvs.com.au) (203.87.14.203) by mail.netconnect.com.au with SMTP; 25 Feb 2000 13:21:59 -0000 Message-ID: <38B64383.7F0A213@cvs.com.au> Date: Fri, 25 Feb 2000 19:55:31 +1100 From: Charles Esson X-Mailer: Mozilla 4.5 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: Costa Sapuntzakis CC: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: <200002242156.NAA18955@csapuntz-u1.cisco.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I must have missed something. If we don't have this, you can take the destination port, convert to a table address, use the sequence number, do some calculations and come up with a buffer address and an offset. If you want to mess up the layering of your stack, they are all things you can do now. or using RDMA You tell the would the contents of your table, have the server play with the data so detailed, the server sends back info in this option. --->An attacker plays with the data so returned. The client instead of doing the port to table translation to determine where to send the data from the packet , does a port to table translation to find the table to detect malicious corruption's of the option. How can you assume anything in the option is nothing but rubbish until such a translation and check is done? 1) Haven't you only changed the reason to do the port to table translation. 2) Have completely messed up the layering of your stack. Ok people do it, but as an option? 3) Provided a whole new farm yard of bugs and potential attacks? As I read more and more RFC's one question I am constantly asking, why are people making this terrific protocol more and more complicated. Isn't it difficult enough to implement already ( I think it is anyway). Costa Sapuntzakis wrote: > The TCP RDMA option reduces the overhead of receiving data over > TCP-based protocols such as NFS and HTTP. > > It enables the construction of a simple hardware accelerator that > copies data directly from the incoming packet into application > buffers, avoiding expensive copies in the protocol stack. Even > without hardware acceleration, the option enables the protocol stack > to decrease the number of copies it must do. 1) I think your a bit game if you assume the data in the option is anything other than a story that aims to mislead. 2) If you want to play this game why can't you use the port, sequence number and a table? All things available to you without advertising the state of your system. > > > The TCP RDMA option is an annotation and requires no modifications to > higher layer protocols. It can be used with popular protocols such as > HTTP, NFS, and CIFS, along with new protocols. > > > The TCP option also provides a bit to indicate application-level > message boundaries. The bit enables out-of-order processing of the TCP > receive queue, potentially decreasing service times in the presence of > packet drops and improving performance on parallel systems. Out of order processing assumes the data that didn't arrive one day will. Is that a valid assumption on the internet? What happens with packets that are received more than once, do you DMA over the previous data thus corrupting the changes that have been made by the application? Do you make the rule the application may not alter the DMA buffers? Or do you check for this condition? Is this check in your acceleration hardware? I assume you believe you still only move the sequence number forward if all data has been received to that sequence number. Is this done in the acceleration hardware? By the time you have checked that the option is valid, updated your received sequence number, and made sure you are not DMAing over data the application already has, what have you gained that could not be had without this option and a restructure of a stack. P.S. Maximum of two copies in the stack I wrote, one from card to ip_buffer, and an optional copy from ip_buffer to application_buffer. I don't believe I am partially smart. I deceive the introduction is misleading. > > > A draft describing the TCP RDMA option can be found at: > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt > > -Costa From XXX Fri Feb 25 13:28:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 08:19:41 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id IAA28331 for ; Fri, 25 Feb 2000 08:19:40 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA21638 for ips-outgoing; Fri, 25 Feb 2000 08:19:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from the-village.bc.nu (lightning.swansea.uk.linux.org [194.168.151.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA21634 for ; Fri, 25 Feb 2000 08:19:27 -0500 (EST) Received: from alan by the-village.bc.nu with local (Exim 2.12 #1) id 12OKcF-0007bx-00; Fri, 25 Feb 2000 13:17:07 +0000 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: julian_satran@il.ibm.com Date: Fri, 25 Feb 2000 13:17:05 +0000 (GMT) Cc: David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: from "julian_satran@il.ibm.com" at Feb 25, 2000 11:54:25 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-ips@ece.cmu.edu Precedence: bulk > Gb/s it requires some innovation and lots of silicon. The RDMA option makes > it possible at a far lower price. And the zero copy it enables might go > deep into the application space as it is only an annotation on packets. I am not convinced the amount of silicon changes between the two. The RDMA id make be faked by an attacker so must still be verified. Va Jacobson proposed and to an extent implemented a system where the user context does all the TCP work. In that sort of situation and with a more sensible API than the BSD socket one you dont appear to need a lot of silicon, in fact the worst case is the wildcard. Alan From XXX Fri Feb 25 17:14:25 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 09:33:33 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id JAA19409 for ; Fri, 25 Feb 2000 09:33:32 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA23653 for ips-outgoing; Fri, 25 Feb 2000 09:33:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from the-village.bc.nu (lightning.swansea.uk.linux.org [194.168.151.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA23647 for ; Fri, 25 Feb 2000 09:33:19 -0500 (EST) Received: from alan by the-village.bc.nu with local (Exim 2.12 #1) id 12OLlj-0007hq-00; Fri, 25 Feb 2000 14:30:59 +0000 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: brian@hursley.ibm.com (Brian E Carpenter) Date: Fri, 25 Feb 2000 14:30:58 +0000 (GMT) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), julian_satran@il.ibm.com, David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: <38B68F5E.CCCB9A5C@hursley.ibm.com> from "Brian E Carpenter" at Feb 25, 2000 08:19:10 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-ips@ece.cmu.edu Precedence: bulk > As I recall, Van's solution required an unconventional I/O system where DMA went > straight into user memory right off the LAN chip. Not too many people make > computers like that. Actually it doesn't, although every PC today is your 'unconventional' system. Many 10Mbit and almost all the 100Mbit cards do scatter gather DMA. To do zero copy means landing the buffer into memory that can become accessible to the user, the rest of the benefits for single copy/checksum/user come with DMA landing in purely kernel controlled space From XXX Fri Feb 25 17:14:25 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 11:36:02 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA01235 for ; Fri, 25 Feb 2000 11:36:01 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA28915 for ips-outgoing; Fri, 25 Feb 2000 11:35:36 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from calcite.rhyolite.com (calcite.rhyolite.com [38.159.140.3]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA28910 for ; Fri, 25 Feb 2000 11:35:31 -0500 (EST) Received: (from vjs@localhost) by calcite.rhyolite.com (8.9.3/calcite) id JAA08403 env-from ; Fri, 25 Feb 2000 09:35:30 -0700 (MST) Date: Fri, 25 Feb 2000 09:35:30 -0700 (MST) From: Vernon Schryver Message-Id: <200002251635.JAA08403@calcite.rhyolite.com> To: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Sender: owner-ips@ece.cmu.edu Precedence: bulk ] From: Charles Esson ] I must have missed something. ] ] If we don't have this, you can take the destination port, convert to a ] table address, use the sequence number, ] do some calculations and come up with a buffer address and an offset. If ] you want to mess up the layering ] of your stack, they are all things you can do now. Standards committees don't like hashing. It looks complicated and insufficiently deterministic on an overhead projector. ] or using RDMA ] ... ] --->An attacker plays with the data so returned. ] ... That's a very good point. The tagging in RDMA cannot be used until after it has been validated by the receiver. The validating consists of looking at sequence numbers, RPC/XDR headers, etc. to figure out where the data can and should go, and then checking that the sender guessed right. Why not skip the last part and ignore the RDMA tag? Then why send the RDMA tag? ....... > From: Pete Zaitcev > Very well, but what about its companion document (SCOT)? > http://search.ietf.org/internet-drafts/draft-satran-scot-00.txt > It is published, isn't it? It was somewhat disturbing to see the > notice, but on the other hand it was honest. IBM could just as > easily come up silently with a silly software patent for RDMA option > or for SCSI over TCP idea as such. The IETF's protections against patent games are well intended, but nothing to worry about if you want to play them and nothing to rely upon if you don't. The history of IETF patent games demonstrates that the IETF is powerless to limit them (or worse), and that they're harder to play than the players hope. (E.g. PPP CCP and PPP 48-bit FCS, respectively) ...... } From: "Justin T. Gibbs" } ... } >Can you elaborate on this? Suppose TCP "blindly" does zero copy everything to } >an app's buffer (for example, to a web browser's receive buffer) without } >RDMA. Then the browser app looks at the data and displays it. What is the } >difference RDMA makes in this case? Yes, RDMA can separate different messages } >in the buffer. But this can also be done by the browser app, not by TCP. } } You seem to be saying that in the common case zero copy is achievable. } Most implementations I've seen require the network driver to make } a guess about where the payload will be in an incoming packet so the header } can be stripped off and the payload dmaed to an aligned area. A page } flip is then performed to get the data where the user wants it, } imposing the restriction that your payload be page sized so you don't } leave gaps in the user's destination buffer. That is required only if you stick to the current API. Obvious, minor changes in the direction of some operating systems that existed before UNIX are sufficient to relax the page boundary requirement. To use RDMA, you have to change the API. } Certainly, with a more } intelligent network adapter that knows every protocol you can determine } exactly where the data is in each packet. If you add connection tracking } and sequence number sniffing to the nic with a mechanism to register user } buffers to connections, you can get zero copy every time*. Unfortunately } this is not very general purpose solution. Only standards committees and some academics care about "every protocol" or optimizing absolutely every application. The rest of us (including academics) only care about optimizing the important stuff. Also as you say, looking at sequence numbers in the interface and relaxing the sockets API rules about not touching any bytes in the buffer except those that are actually received lets you avoid copies all of the time. I don't see why that is not a general purpose solution, if you want one. } The point of RDMA seems to be } to allow nic manufacturers to add support for a single tcp option that, at } the very least, allows the nic to align the payload for you. Add RID } registration with the nic and you get the payload exactly where you want it } too. All without too much state information kept by the nic. I've been hearing since the mid-1980's proposals to do TSP lookups in the network interface instead of software because it is so incredibly difficult to find the right TSP quickly in software. I think those ideas are similar to the RDMA idea. They assume facts not in evidence, that there is a problem that needs to be solved, and that the solution is not worse than the nominal problem. There are reasons why such proposals appear in standards committees before implementations. ...... ] From: Lloyd Wood ] Note the mentions of SCSI and SCSI/TCP and the tie-in with the ] proposed IP Storage efforts (recent ietf general list discussion). ] ] I'd still like to know _why_. ] ... ] SCSI DMA over TCP? What _is_ all this aiming for - trying to build ] distributed RAID arrays with really poor performance that are subject ] to WAN outages and DoS attacks? Why put SCSI over an protocol that measures RTT's, worries about congestion in routers, and that expects the error rates that come with 5000 miles of wire and 20 routers in the path? Does anyone really think that TCP/IP or even IP with it's 64K bit packet limit are remotely close to the right protocol, particularly given the existing and commercially available alternatives? A standards committee is the venue of first and last resort for such ideas, especially a committee that is related to currently trendy things like the SuperInfoHypeWay. .... ) From: julian_satran@il.ibm.com ) That is not completely accurate. You will need appreciably more silicon to ) do what you suggest. And you can do it only with information that "passes ) through the protocol" . Significantly silicon more than what to do what? Since the comment was addressed to me, I'll assume one 'what' was looking at sequence numbers, port numbers, and so forth to page flip. Clearly it takes more silicon to support page flipping in hardware than to not support page flipping in hardware. I will not agree that the required silicon is a big deal, not because I have a clue about floor plans and so forth (I don't), but because at a previous employeer I fought to keep the hardware guys from throwing in gates to do it. They had the silicon to spare and had heard so much about the wonderfulness of page flipping that they wanted to get in on the fun. Doing things in hardware is ok only if you absolutely must. Software is always better when it is good enough, because it is soft. ) The good thing about the proposal is that it can TAG whatever the ) application wants (and that can be several layers away from the protocol). ) You can't "page-flip" to buffers that you are not aware of. And page ) flipping wherever is applicable assumes also page boundaries for buffers. That's important only if you stick close to the sockets or UNIX read() API. If you are not ultra-conservative, and if you know a little of the history of file and device I/O API's, or of you think about such things for 10 seconds, then RDMA tagging becomes less interesting. To use RDMA tagging, you must abandone the UNIX read() API. If you change the API, then you may as well think about the whole problem instead of only a corner. If you let the operating system tell the application where the incoming data arrived, then you don't need elaborate hints from the sender to the receivers hardware to say where the receiving software will want the data. ) Vernon Schryver on 25/02/2000 04:23:47 ) ) Please respond to Vernon Schryver I did not write that! ..... ) From: Alan Cox ) ) > flip is then performed to get the data where the user wants it, ) > imposing the restriction that your payload be page sized so you don't ) > leave gaps in the user's destination buffer. Certainly, with a more ) ) Perhaps its about time the world put together an official, sane, ring buffer ) style mmap socket api. A lot of the requirement to align data is coming ) from the existing socket API. The IETF should not get involved in API's. There are plenty of other standards committees in that arena, as well as big commercial outfits including one in the U.S. Pacific Northwest. In other words, do you think the IETF would be more successful arguing with Microsoft about winsock than the IETF has been in dealing with Microsoft's obviously completely stupid and wrong PPP ideas? If you do get involved in standardizing such things, then *PLEASE* don't limit yourself to #$%$#@! ring buffers! The ancient Execelan and preceding (I've a mental block against the name starting with 'I') ring buffer notion was ok as an initial hack, but WRONG for something to go fast. To start, you don't need pointers or indeces that must be written by both the interface and the host. Vernon Schryver vjs@rhyolite.com From XXX Fri Feb 25 17:14:25 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 12:03:48 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA24318 for ; Fri, 25 Feb 2000 12:03:46 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA00051 for ips-outgoing; Fri, 25 Feb 2000 12:03:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pneumatic-tube.sgi.com (pneumatic-tube.sgi.com [204.94.214.22]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA00026 for ; Fri, 25 Feb 2000 12:03:11 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id JAA04975; Fri, 25 Feb 2000 09:05:59 -0800 (PST) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (clock.engr.sgi.com [163.154.34.45]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id JAA00610; Fri, 25 Feb 2000 09:02:52 -0800 (PST) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (localhost [127.0.0.1]) by clock.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id JAA73948; Fri, 25 Feb 2000 09:06:56 -0800 (PST) Message-Id: <200002251706.JAA73948@clock.engr.sgi.com> X-Mailer: exmh version 2.0.2 2/24/98 To: Charles Esson Cc: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. From: Zachary Amsden In-Reply-To: Your message of "Fri, 25 Feb 2000 19:55:31 +1100." <38B64383.7F0A213@cvs.com.au> Date: Fri, 25 Feb 2000 09:06:55 -0800 Sender: owner-ips@ece.cmu.edu Precedence: bulk > I must have missed something. > > If we don't have this, you can take the destination port, convert to a > table address, use the sequence number, > do some calculations and come up with a buffer address and an offset. If > you want to mess up the layering > of your stack, they are all things you can do now. Yes, you can do that, and it isn't that tricky for simple protocols. For any bulk data protocol higher than TCP with fixed headers, determining the payload offset is pretty straigtforward for a single transfer per connection. Once you start with multiple transfers per connection, variable headers, or many different protocols, it becomes harder to do all this work in silicon. The advantage I see in RDMA is giving a generic payload pointer for the NIC to separate protocol data and payload. I wonder whether it would be better to do this at the IP layer to enable it to be used for UDP and other protocols as well. Of course RDMA is only going to help in cases where you have receive bandwidth issues, and such a scenario isn't likely to be the case for web/file servers or desktop clients. The scenario that it gives the most benefit for is a "middleman" server that needs to do lots of I/O to a network storage device while servicing requests. In this case, any network I/O running locally may well be over a different protocol like UDP or ST. (ST however, has no need for RDMA acceleration, as it already has a buffer transfer design). -- Zachary Amsden zamsden@engr.sgi.com (650) 933-6919 09U-510 Core Protocols From XXX Fri Feb 25 17:14:25 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 09:22:34 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id JAA19299 for ; Fri, 25 Feb 2000 09:22:34 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA23278 for ips-outgoing; Fri, 25 Feb 2000 09:22:29 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail-gw.hursley.ibm.com (mail-gw.hursley.ibm.com [194.196.110.15]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA23269 for ; Fri, 25 Feb 2000 09:22:23 -0500 (EST) Received: from sp3at21.hursley.ibm.com (sp3at21.hursley.ibm.com [9.20.45.21]) by mail-gw.hursley.ibm.com (AIX4.3/UCB 8.8.8/8.8.8) with ESMTP id OAA277522; Fri, 25 Feb 2000 14:20:25 GMT Received: from hursley.ibm.com ([9.14.4.75]) by sp3at21.hursley.ibm.com (AIX4.2/UCB 8.7/8.7.3) with ESMTP id OAA12410; Fri, 25 Feb 2000 14:20:23 GMT Message-ID: <38B68F5E.CCCB9A5C@hursley.ibm.com> Date: Fri, 25 Feb 2000 08:19:10 -0600 From: Brian E Carpenter Organization: IBM X-Mailer: Mozilla 4.61 [en] (Win98; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: Alan Cox CC: julian_satran@il.ibm.com, David Robinson , ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk As I recall, Van's solution required an unconventional I/O system where DMA went straight into user memory right off the LAN chip. Not too many people make computers like that. Brian Alan Cox wrote: > > > Gb/s it requires some innovation and lots of silicon. The RDMA option makes > > it possible at a far lower price. And the zero copy it enables might go > > deep into the application space as it is only an annotation on packets. > > I am not convinced the amount of silicon changes between the two. The > RDMA id make be faked by an attacker so must still be verified. > > Va Jacobson proposed and to an extent implemented a system where the user > context does all the TCP work. In that sort of situation and with a more > sensible API than the BSD socket one you dont appear to need a lot of silicon, > in fact the worst case is the wildcard. > > Alan From XXX Fri Feb 25 18:55:32 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 13:51:21 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA17246 for ; Fri, 25 Feb 2000 13:51:20 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA04132 for ips-outgoing; Fri, 25 Feb 2000 13:51:15 -0500 (EST) Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA04124 for ; Fri, 25 Feb 2000 13:51:08 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id KAA08982; Fri, 25 Feb 2000 10:51:00 -0800 (PST) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id KAA19441; Fri, 25 Feb 2000 10:50:32 -0800 (PST) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Fri, 25 Feb 2000 10:50:32 -0800 (PST) From: Costa Sapuntzakis To: "David S. Miller" cc: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. In-Reply-To: <200002250101.RAA17054@pizda.ninka.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, I don't think it's really a matter of SACK vs. RDMA. I think they are both useful and somewhat orthogonal. There is a concern that both RDMA and SACK are large options and there is only 40 bytes of TCP options space. Perhaps there is a way of expanding the TCP options space (yet another TCP option!) that will enable both to co-exist peacefully. -Costa On Thu, 24 Feb 2000, David S. Miller wrote: > Date: Thu, 24 Feb 2000 17:50:52 -0700 > From: "Justin T. Gibbs" > > The performance impact of RDMA is quite a bit larger than SACK, > > It depends who you are. > > For someone over a satellite link, I think SACK benefits them > much more than RDMA. > > Later, > David S. Miller > davem@redhat.com > > > From XXX Fri Feb 25 18:55:32 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 13:46:36 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA16915 for ; Fri, 25 Feb 2000 13:46:36 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA03917 for ips-outgoing; Fri, 25 Feb 2000 13:46:20 -0500 (EST) Received: from kickme.cisco.com (kickme.cisco.com [198.92.30.42]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA03904 for ; Fri, 25 Feb 2000 13:46:14 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by kickme.cisco.com (8.9.1a/8.9.1) with ESMTP id KAA16466; Fri, 25 Feb 2000 10:35:05 -0800 (PST) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id KAA19437; Fri, 25 Feb 2000 10:45:34 -0800 (PST) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Fri, 25 Feb 2000 10:45:34 -0800 (PST) From: Costa Sapuntzakis To: "David S. Miller" cc: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. In-Reply-To: <200002242357.PAA16973@pizda.ninka.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, Your input on this is very much appreciated. I'd like to clarify a couple things, though. Some of these things may not need clarification. :) The TCP RDMA option is not about accelerating sending of data. It's about speeding up the receiving of bulk data. The TCP RDMA option isn't proposing to only about accelerating servers. It's about accelerating the receiver of data. In the case of NFS READ RPCs, that's the NFS client. I agree that for specific problem domains, SCSI, NFS, HTTP, people may want to build specialized server hardware. Perhaps explaining why I decided to explore this space will help. Today, you have specialized silicon that for simple bus protocols (SCSI parallel interface and ATA) will directly take transfer blocks between the device and the buffer cache. This is not currently done with TCP, to the best of my knowledge. The best TCP implementations do zero copy to a TCP receive buffer. However, in the case of most storage protocols, you don't want the data in the receive buffer. You want it in the buffer cache, so there is a copy to the buffer cache. So, NFS has a CPU overhead hit as compared to optimized storage host bus adapters. The goal was to eliminate part of this hit, by getting rid of an extra copy. Now, this proposal doesn't fix the interrupt overhead problem. Optimized FC/SCSI NICs have one interrupt/transfer or less. -Costa On Thu, 24 Feb 2000, David S. Miller wrote: > Date: Thu, 24 Feb 2000 14:59:38 -0800 (PST) > From: Erik Nordmark > > As an aside I think the RDMA proposal has a lot of holes too. For > example, there are in-kernel HTTP accelerators that do the complete > client header parse and initial packet response in the hw interrupt > handler. There are no user buffers involved, and static response > data is DMA'd directly from the filesystem page cache. > > > A draft describing the TCP RDMA option can be found at: > > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt > > There is no DNS entry for ftpeng.cisco.com so I can't access the > document. > > Here is what I get: > > ? host -a ftpeng.cisco.com > Trying null domain > rcode = 0 (Success), ancount=1 > The following answer is not authoritative: > The following answer is not verified as authentic by the server: > ftpeng.cisco.com 84574 IN CNAME ftp-eng.cisco.com > For authoritative answers, see: > cisco.com 38435 IN NS NS1.cisco.com > cisco.com 38435 IN NS NS2.cisco.com > Additional information: > NS1.cisco.com 78995 IN A 192.31.7.92 > NS2.cisco.com 67536 IN A 192.135.250.69 > rcode = 0 (Success), ancount=1 > The following answer is not authoritative: > The following answer is not verified as authentic by the server: > ftp-eng.cisco.com 84574 IN A 198.92.30.33 > For authoritative answers, see: > CISCO.com 38435 IN NS NS1.CISCO.com > CISCO.com 38435 IN NS NS2.CISCO.com > Additional information: > NS1.CISCO.com 78995 IN A 192.31.7.92 > NS2.CISCO.com 67536 IN A 192.135.250.69 > > > From XXX Fri Feb 25 19:09:02 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 14:08:09 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id OAA07724 for ; Fri, 25 Feb 2000 14:08:08 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA04768 for ips-outgoing; Fri, 25 Feb 2000 14:08:03 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from the-village.bc.nu (lightning.swansea.uk.linux.org [194.168.151.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA04764 for ; Fri, 25 Feb 2000 14:07:58 -0500 (EST) Received: from alan by the-village.bc.nu with local (Exim 2.12 #1) id 12OPyF-00085a-00; Fri, 25 Feb 2000 19:00:11 +0000 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: zamsden@cthulhu.engr.sgi.com (Zachary Amsden) Date: Fri, 25 Feb 2000 19:00:09 +0000 (GMT) Cc: charlese@cvs.com.au (Charles Esson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com In-Reply-To: <200002251706.JAA73948@clock.engr.sgi.com> from "Zachary Amsden" at Feb 25, 2000 09:06:55 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-ips@ece.cmu.edu Precedence: bulk > I wonder whether it would be better to do this at the IP layer to enable it to > be used for UDP and other protocols as well. Congratulations. You are probably the first person in the world to find a sane use for the IPv6 flow id 8) Alan From XXX Fri Feb 25 19:59:46 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 14:55:36 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id OAA09304 for ; Fri, 25 Feb 2000 14:55:36 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA06619 for ips-outgoing; Fri, 25 Feb 2000 14:54:59 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from deliverator.sgi.com (deliverator.sgi.com [204.94.214.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA06612 for ; Fri, 25 Feb 2000 14:54:55 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by deliverator.sgi.com (980309.SGI.8.8.8-aspam-6.2/980310.SGI-aspam) via ESMTP id LAA07647; Fri, 25 Feb 2000 11:50:12 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id LAA90761; Fri, 25 Feb 2000 11:54:39 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id LAA29821; Fri, 25 Feb 2000 11:52:53 -0800 (PST) Message-ID: <38B6DD95.19C9BCAD@engr.sgi.com> Date: Fri, 25 Feb 2000 11:52:53 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: Zachary Amsden CC: Charles Esson , ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: <200002251706.JAA73948@clock.engr.sgi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > In this case, any network I/O running locally may well be over a different > protocol like UDP or ST. (ST however, has no need for RDMA acceleration, as > it already has a buffer transfer design). Thats IMO an excellent point. If you're going to have to change your client as well as the server, and add an additional formatted header (through the TCP RDMA option), why do you want to constraint yourself to doing it in the context of TCP only! Why not do it using STP? a protocol that has been designed to do RDMAs - addressing various issues which come up when designing RDMA s/w, firmware, h/w.. The BDS fileserver using STP does something very similar to what you're proposing to do with NFS. There's even a prototype doing SCSI using RDMAs of the very same kind using STP. thanks, :a From XXX Fri Feb 25 20:09:55 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 15:06:32 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id PAA32167 for ; Fri, 25 Feb 2000 15:06:32 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA07081 for ips-outgoing; Fri, 25 Feb 2000 15:06:20 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from the-village.bc.nu (lightning.swansea.uk.linux.org [194.168.151.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA07072 for ; Fri, 25 Feb 2000 15:06:13 -0500 (EST) Received: from alan by the-village.bc.nu with local (Exim 2.12 #1) id 12OQyR-0008Cq-00; Fri, 25 Feb 2000 20:04:27 +0000 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: csapuntz@cisco.com (Costa Sapuntzakis) Date: Fri, 25 Feb 2000 20:04:25 +0000 (GMT) Cc: davem@redhat.com (David S. Miller), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: from "Costa Sapuntzakis" at Feb 25, 2000 10:45:34 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-ips@ece.cmu.edu Precedence: bulk > However, in the case of most storage protocols, you don't want > the data in the receive buffer. You want it in the buffer cache, so > there is a copy to the buffer cache. That assumes you require a copy to move from the network layer to the buffer/page cache. Is that a warranted assumption. I think not. > Now, this proposal doesn't fix the interrupt overhead problem. > Optimized FC/SCSI NICs have one interrupt/transfer or less. Interrupt overhead is old hat. Mitigation schemes are well understood Alan From XXX Fri Feb 25 20:36:25 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 15:17:15 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id PAA32313 for ; Fri, 25 Feb 2000 15:17:15 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA07502 for ips-outgoing; Fri, 25 Feb 2000 15:16:50 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA07496 for ; Fri, 25 Feb 2000 15:16:46 -0500 (EST) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by atlrel2.hp.com (Postfix) with ESMTP id B2788618; Fri, 25 Feb 2000 15:17:02 -0500 (EST) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com with ESMTP (8.7.6/8.7.3 TIS 5.0.1) id MAA24980; Fri, 25 Feb 2000 12:23:28 -0800 (PST) Message-Id: <4.2.2.20000225115214.00b73280@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 25 Feb 2000 12:03:38 -0800 To: "David S. Miller" , gibbs@freebsd.org From: Michael Krause Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Cc: Erik.Nordmark@eng.sun.com, csapuntz@cisco.com, ips@ece.cmu.edu, tcp-impl@grc.nasa.gov, gibbs@freebsd.org, zaitcev@metabyte.com, drich@fjst.com In-Reply-To: <200002250010.QAA16997@pizda.ninka.net> References: <200002250009.RAA01066@caspian.plutotech.com> <200002250009.RAA01066@caspian.plutotech.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:10 PM 2/24/00 -0800, David S. Miller wrote: > Date: Thu, 24 Feb 2000 17:09:51 -0700 > From: "Justin T. Gibbs" > > In the case of a server response, RDMA benefits the client, not the > server, so I fail to see why your example is problematic. Zero > copy send is not what this standard addresses. > >With client memory bus bandwidth in the multi-gigabyte per second >range, who needs to avoid the single copy? How much NFS and web >surfing does one need to do before this is would really come into >play? > >And the bus speeds will just be faster by the time something like >this could be deployed widely. It ain't free and there are plenty of reasons to avoid copying data since many applications do not always touch the data being moved. Also, http and web traffic are probably not the best examples to illustrate the value of RDMA Read and Write operations. Storage which was the context that started this effort certainly benefits from a combination of send for control messages and RDMA for data movement - doesn't matter whether this in user-space / kernel-space, client or server. Those buffers may be only used by the server to reflect a data set to a set of users without ever touching the buffers themselves. Also, one could use this technology with storage devices to bypass the server and send data to one or more NICs for remote access - RDMA is still quite good for this type of operation and does not involve touching the data. Mike From XXX Fri Feb 25 22:51:49 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Feb 25 17:51:23 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id RAA18731 for ; Fri, 25 Feb 2000 17:51:22 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA12824 for ips-outgoing; Fri, 25 Feb 2000 17:50:58 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from calcite.rhyolite.com (calcite.rhyolite.com [38.159.140.3]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA12818 for ; Fri, 25 Feb 2000 17:50:51 -0500 (EST) Received: (from vjs@localhost) by calcite.rhyolite.com (8.9.3/calcite) id PAA15085 env-from ; Fri, 25 Feb 2000 15:50:49 -0700 (MST) Date: Fri, 25 Feb 2000 15:50:49 -0700 (MST) From: Vernon Schryver Message-Id: <200002252250.PAA15085@calcite.rhyolite.com> To: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: Costa Sapuntzakis > ... > Today, you have specialized silicon that for simple bus protocols > (SCSI parallel interface and ATA) will directly take transfer blocks > between the device and the buffer cache. This is not currently done > with TCP, to the best of my knowledge. ... It might be good to investigate the history of Protocol Engines Inc., including its goals, the reasons for its failure as a business, and what it achieved technically. A skewed history might be: 1. founded to make silicon for XTP, a nominally faster protocol than TCP. 2. when XTP protocol and the XTP chips got bogged down, shifted to making chips to help TCP go wire speed over FDDI. 3. other people made TCP go wire speed over FDDI without any special silicon or new to protocols. That took some wind out of XTP's sails, and tore the sails driving PEI's TCP acclerator chips. 4. standard standards committee problems with XTP didn't help PEI's other sails. If you ask me, SCSI/IP and RDMA have striking parallels to #1 and #2. I bet you'll meet parallels to #3 before any real deployment. You've started to see #4 in some of the suggested improvements to RDMA today. It's not that the suggestions are not good ideas. That problem is that committees cannot say no to good ideas, while the one thing that matters above all in any design task is saying no to almost everything. Protocol Engines and XTP were based on the unexamined assumption that TCP is very difficult to implement and an unavoidably slow protocol. Most people just knew those "facts" 15 years ago. I think RDMA suffers a similar problem. Instead of starting by assuming that a new protocol is needed for a new goal, if you actually look within the existing boundaries, you'll often find a solution. Often the inside solution is better than any possible extension of the protocol. Protocol extensions require more bandwidth and more processing on both sender and receiver. They also have problems gaining enough marketshare to survive. Please don't misunderstand me. Greg didn't include my name among the authors on one of the XTP specs because I said XTP was a stupid idea. I still like lots of XTP. I also think that many of the XTP ideas can be *and have been* applied to TCP implementations. > However, in the case of most storage protocols, you don't want > the data in the receive buffer. You want it in the buffer cache, so > there is a copy to the buffer cache. Which NFS implementation written in the last 10 or at least 5 years and intended to be fast doesn't move data between the buffer cache near the disk and the buffer cache near the application with zero (0) copies? Page flipping to and from buffer caches is especially easy, because buffer caches tend to be page aligned, and file systems like to move data in page-sized or larger chunks. > So, NFS has a CPU overhead hit as compared to optimized storage host bus > adapters. The goal was to eliminate part of this hit, by getting rid of an > extra copy. How can you have fewer than zero copies? > Now, this proposal doesn't fix the interrupt overhead problem. > Optimized FC/SCSI NICs have one interrupt/transfer or less. Interrupts are killers, and so for that last 5 or 10 years, a competetive NFS system has had about 0.1 interrupts per packet. The trick is not reducing the ratio of interrupts/packet, but reducing it only so far that things don't slow down, and increasing the ratio when the total system (client & server) moves into a regime that requires more interrupts. ] From: Michael Krause ] It ain't free and there are plenty of reasons to avoid copying data since ] ... ] touching the buffers themselves. Also, one could use this technology with ] storage devices to bypass the server and send data to one or more NICs for ] remote access - RDMA is still quite good for this type of operation and ] does not involve touching the data. There are other, much easier ways to separate data and control information in the receiver than being forced to parse optional new bits in TCP or IP headers. For 10 years, network interfaces in commercial UNIX systems have been putting the headers (including RPC/XDR) of incoming NFS traffic in one place (a "small mbuf") and the data in another place (the buffer cache) without extra copies, and without parsing any headers, not to mention new header bits with the nasty problems of TCP or IP options. And this despite the fact that the RPC/XDR stuff is between variable length (recall the NFS group list) and a hard to predict length. Vernon Schryver vjs@rhyolite.com From XXX Sun Feb 27 12:58:35 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Feb 27 03:32:08 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id DAA21282 for ; Sun, 27 Feb 2000 03:32:08 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA24508 for ips-outgoing; Sun, 27 Feb 2000 03:30:01 -0500 (EST) Received: from kickme.cisco.com (kickme.cisco.com [198.92.30.42]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA24504 for ; Sun, 27 Feb 2000 03:29:57 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by kickme.cisco.com (8.9.1a/8.9.1) with ESMTP id AAA14708; Sun, 27 Feb 2000 00:18:45 -0800 (PST) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id AAA19701; Sun, 27 Feb 2000 00:29:25 -0800 (PST) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Sun, 27 Feb 2000 00:29:25 -0800 (PST) From: Costa Sapuntzakis To: Aman Singla cc: "Mailing List, IP Storage" Subject: Scheduled Transfer Protocol (ST) In-Reply-To: <38B300DD.3E4CF760@engr.sgi.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks for bringing the ST protocol to our attention. I'm not sure it fits all the requirements, though, because one of the goals of IPS is to generate a WAN-friendly protocol. Is STP designed for the WAN? I see no provision in the ST spec for TCP-style congestion control or even exponential back-off. In addition, the Clear_To_Send mechanism as currently speced seems to add a round-trip time to every transfer, increasing latency. As for the SCSI/ST spec: Is there any mechanism for ordering tasks? For example, if you want to enter several ordered tasks into a SCSI target, can you issue them simultaneously, i.e. without waiting for an ACK?. Thanks, -Costa On Tue, 22 Feb 2000, Aman Singla wrote: > Hi, > > Talking in context of the 3 layer protocol model, SI->Transport->IP, > I'd like to bring to attention an ANSI draft standard protocol - > Scheduled Transfer Protocol (STP). > > The design of STP has been driven by requirements very similar to > those outlined in the Storage of IP Design Framework, namely > low-latency, high bandwidth, low CPU utilization, a measure of > congestion/flow control built in, etc. > STP can very easily fit in as the transport layer in the 3 layer > protocol model. STP can be IP encapsulated, and a draft specification > for SCSI over STP also exists. > > STP has existed on Irix as part of the GSN infrastructure; SGI along > with Genroco demonstrated a SCSI over STP solution interfacing a > Origin 2000 server with GSN to a fibre channel raid network via a > Genroco bridge, at CERN in October 1999. > > STP is truly media independent, with support for GbE in the works. > We recently made an alpha release of STP on Linux to the open-source > community (http://oss.sgi.com/projects/stp), and going forward intend > to fully support it on a variety of interfaces, starting with GbE. > > More information about STP and the various drafts can be accessed > off http://www.hippi.org/cDOCS.html. > If there is interest in learning more about this protocol, we > could arrange for a presentation/demo at one of the meetings for > the proposed working group. > (Note: STP is also refered to as ST in the marketing literature > of various vendors - Genroco, Essential, PMR..) > > thanks, > > Aman Singla > MTS, SGI > > > From XXX Sun Feb 27 12:58:35 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Feb 27 05:11:56 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id FAA10075 for ; Sun, 27 Feb 2000 05:11:55 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA25759 for ips-outgoing; Sun, 27 Feb 2000 05:11:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA25754 for ; Sun, 27 Feb 2000 05:11:37 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA31802; Sun, 27 Feb 2000 11:11:01 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id LAA62354; Sun, 27 Feb 2000 11:11:01 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256892.0037EE2F ; Sun, 27 Feb 2000 11:10:54 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Alan Cox cc: David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Message-ID: Date: Sun, 27 Feb 2000 11:57:13 +0200 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk True. You can do everything by processing headers... but the you need to understand all the protocols that are amased over TCP. To do it in silicon it will probably make sense for a subset. RDMA is a general purpose solution and the user decides what to do with it. You can look at it as a simple way to enable the protocl stack and the application to completely separate the protocol state machine (defined by headers and/or trailers) from the payload. As for the vulnerability to attacks with a good size RDMAID and some imagination you can get the same level of protection as with the TCP sequence number (even a bit better because sequence numbers can be guessed from context). Julo Alan Cox on 25/02/2000 15:17:05 Please respond to Alan Cox To: julian_satran%ibmil.RSCS@STUTVM1.DE.IBM.COM cc: David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov (bcc: Julian Satran/Haifa/IBM) Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. > Gb/s it requires some innovation and lots of silicon. The RDMA option makes > it possible at a far lower price. And the zero copy it enables might go > deep into the application space as it is only an annotation on packets. I am not convinced the amount of silicon changes between the two. The RDMA id make be faked by an attacker so must still be verified. Va Jacobson proposed and to an extent implemented a system where the user context does all the TCP work. In that sort of situation and with a more sensible API than the BSD socket one you dont appear to need a lot of silicon, in fact the worst case is the wildcard. Alan From XXX Sun Feb 27 17:48:27 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Feb 27 11:10:08 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA24666 for ; Sun, 27 Feb 2000 11:10:07 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA08257 for ips-outgoing; Sun, 27 Feb 2000 11:09:12 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from the-village.bc.nu (lightning.swansea.uk.linux.org [194.168.151.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA08248 for ; Sun, 27 Feb 2000 11:08:52 -0500 (EST) Received: from alan by the-village.bc.nu with local (Exim 2.12 #1) id 12P6GM-0000Qe-00; Sun, 27 Feb 2000 16:09:42 +0000 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: julian_satran@il.ibm.com Date: Sun, 27 Feb 2000 16:09:39 +0000 (GMT) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: from "julian_satran@il.ibm.com" at Feb 27, 2000 11:57:13 AM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-ips@ece.cmu.edu Precedence: bulk > understand all the protocols that are amased over TCP. To do it in silicon > it will probably make sense for a subset. You dont want to do it in silicon. Forget doing all this in silicon. We have this funky stuff called software. The silicon needs no RDMA support to do sensible work in the API and the underlying OS are sensibly designed. > You can look at it as a simple way to enable the protocl stack and the > application to completely separate the protocol state machine (defined by > headers and/or trailers) from the payload. The two are tied together. You have to parse the TCP option stream to get the ident in the first place. You can't act on the RDMAID until you have checked the packet is syntactically valid and you've processed the options including handling the SACK data mixed in with it. It might also be fragmented of course. > As for the vulnerability to attacks with a good size RDMAID and some > imagination you can get the same level of protection as with the TCP > sequence number (even a bit better because sequence numbers can be guessed > from context). The tcp sequence number protects against ordering errors not against DMAing crap into the wrong buffer. Different game, different cost if you lose. Alan From XXX Sun Feb 27 21:52:39 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Feb 27 16:35:01 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id QAA15056 for ; Sun, 27 Feb 2000 16:35:01 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA14261 for ips-outgoing; Sun, 27 Feb 2000 16:34:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from kitty.netconnect.com.au (ren.netconnect.com.au [203.7.198.1]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id QAA14252 for ; Sun, 27 Feb 2000 16:34:23 -0500 (EST) Received: (qmail 27305 invoked from network); 27 Feb 2000 21:34:31 -0000 Received: from unknown (HELO cvs.com.au) (203.87.14.203) by mail.netconnect.com.au with SMTP; 27 Feb 2000 21:34:31 -0000 Message-ID: <38B95A08.EF0D9124@cvs.com.au> Date: Mon, 28 Feb 2000 04:08:24 +1100 From: Charles Esson X-Mailer: Mozilla 4.5 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: Alan Cox , David Robinson , ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > True. You can do everything by processing headers... but the you need to > understand all the protocols that are amased over TCP. To do it in silicon > it will probably make sense for a subset. > RDMA is a general purpose solution and the user decides what to do with it. > You can look at it as a simple way to enable the protocl stack and the > application to completely separate the protocol state machine (defined by > headers and/or trailers) from the payload. > > As for the vulnerability to attacks with a good size RDMAID and some > imagination Why not apply the imagination to the writing of a zero copy TCP stack based on the current standards? It would be a lot more general and a lot more useful as it would work against unmodified servers. > you can get the same level of protection as with the TCP > sequence number (even a bit better because sequence numbers can be guessed > from context). Are you suggesting that the sequence number should no longer be used? Are you suggesting that the RDMAID and sequence number should be unrelated? If the sequence number is still used then the RDMAIN is simple additional data that can be used by the attacker. Just another farmyard of potential bugs to be investigated and exploited? If you are not going to use the sequence number, how does a system that doesn't support this option receive the data? If your aim is to design a system that isn't backward compatible, why would anyone in there right mind support it? Aiming to keep the intellectual property rights, not backward compatible, dubious technical merit. ummm? Regards. > > > Julo > > Alan Cox on 25/02/2000 15:17:05 > > Please respond to Alan Cox > > To: julian_satran%ibmil.RSCS@STUTVM1.DE.IBM.COM > cc: David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, > tcp-impl@grc.nasa.gov (bcc: Julian Satran/Haifa/IBM) > Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. > > > Gb/s it requires some innovation and lots of silicon. The RDMA option > makes > > it possible at a far lower price. And the zero copy it enables might go > > deep into the application space as it is only an annotation on packets. > > I am not convinced the amount of silicon changes between the two. The > RDMA id make be faked by an attacker so must still be verified. > > Va Jacobson proposed and to an extent implemented a system where the user > context does all the TCP work. In that sort of situation and with a more > sensible API than the BSD socket one you dont appear to need a lot of > silicon, > in fact the worst case is the wildcard. > > Alan From XXX Sun Feb 27 22:50:03 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Feb 27 17:24:51 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id RAA06688 for ; Sun, 27 Feb 2000 17:24:51 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA15417 for ips-outgoing; Sun, 27 Feb 2000 17:24:37 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ruby.cisco.com (ruby.cisco.com [171.69.198.43]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA15411 for ; Sun, 27 Feb 2000 17:24:33 -0500 (EST) Received: from alfonso. ([10.19.129.228]) by ruby.cisco.com (8.8.8-Cisco List Logging/8.8.8) with SMTP id OAA20322; Sun, 27 Feb 2000 14:23:18 -0800 (PST) Message-Id: <3.0.3.32.20000227141724.006bd2d4@ruby.cisco.com> X-Sender: cheriton@ruby.cisco.com X-Mailer: QUALCOMM Windows Eudora Pro Version 3.0.3 (32) Date: Sun, 27 Feb 2000 14:17:24 -0800 To: Alan Cox , julian_satran@il.ibm.com From: "David R. Cheriton" Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:09 PM 2/27/00 +0000, Alan Cox wrote: >> understand all the protocols that are amased over TCP. To do it in silicon >> it will probably make sense for a subset. > >You dont want to do it in silicon. Forget doing all this in silicon. We have >this funky stuff called software. The silicon needs no RDMA support to do >sensible work in the API and the underlying OS are sensibly designed. > I think it would be useful to get more quantitative. I like to think all of us in this discussion are well aware that some level of performance can be achieved in software. I'm not seeing where "sensibly designed" leads us WRT to OSs, because, if they arent now by whatever definition you are using, it seems pretty academic, as they say. Clearly, data is being received from hardware and software does not get to touch it until it has been stored to some memory. My assumption is that the storage system memory is arranged in fixed size pages of disk/file pages. Without hardware RDMA to the storage level, I believe one requires an extra copy, from whatever the hardware delivers to what the storage system expects. Either you use twice the bandwidth in the storage system memory system or or else you have a separate memory system for the network, and have software/processor power adequate to copy between at wire speed (with all the associated support facilities for this processor.) Unless there is something wrong with this reasoning, it seems like a cost issue of providing the above hardware resources vs. providing a NIC chip that can RDMA. My guessitimate is that the software-only approach would be easily 10 times more expensive here at the higher speed rates, of 10 Gbps. If there is serious doubt about the merits of real hardware support, we should try to quantify costs further at these speed ranges, IMHO. >> You can look at it as a simple way to enable the protocl stack and the >> application to completely separate the protocol state machine (defined by >> headers and/or trailers) from the payload. > >The two are tied together. You have to parse the TCP option stream to get >the ident in the first place. You can't act on the RDMAID until you >have checked the packet is syntactically valid and you've processed >the options including handling the SACK data mixed in with it. > >It might also be fragmented of course. What you need to do >>in the common case<< before processing the RDMA option is relatively simple and a very small portion of the overall protocol state machines, so I presume your comment is asking for more careful wording? Or do you really disagree with the fundamental point? > > >> As for the vulnerability to attacks with a good size RDMAID and some >> imagination you can get the same level of protection as with the TCP >> sequence number (even a bit better because sequence numbers can be guessed >> from context). > >The tcp sequence number protects against ordering errors not against DMAing >crap into the wrong buffer. > >Different game, different cost if you lose. > It would help me to have a more careful definition of the types of attacks you have in mind. In an unsecure network with intruders, presumably I can end up with bad data in the right buffer or right data in the wrong buffer without using RDMA. Do you view we have made things worse, and if so, how? or are you objecting to us not making things better? >Alan > David Cheriton From XXX Sun Feb 27 22:50:03 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Feb 27 17:40:34 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id RAA06888 for ; Sun, 27 Feb 2000 17:40:33 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA15683 for ips-outgoing; Sun, 27 Feb 2000 17:40:14 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from the-village.bc.nu (lightning.swansea.uk.linux.org [194.168.151.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA15679 for ; Sun, 27 Feb 2000 17:40:09 -0500 (EST) Received: from alan by the-village.bc.nu with local (Exim 2.12 #1) id 12PCN6-0000vD-00; Sun, 27 Feb 2000 22:41:04 +0000 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: cheriton@cisco.com (David R. Cheriton) Date: Sun, 27 Feb 2000 22:41:01 +0000 (GMT) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), julian_satran@il.ibm.com, David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: <3.0.3.32.20000227141724.006bd2d4@ruby.cisco.com> from "David R. Cheriton" at Feb 27, 2000 02:17:24 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-ips@ece.cmu.edu Precedence: bulk > it seems like a cost issue of providing the above hardware resources > vs. providing a NIC chip that can RDMA. or a NIC chip that has more memory on the chip and sends you the header then you asynchronously give a DMA location for the rest of the buffer. You've now got rid of RDMA, instead your little bit of extra silicon is generic, and will work with stuff like appletalk even. What does the extra RAM and PCI glue cost you - probably not a lot. That is why I prefer software solutions > >It might also be fragmented of course. > What you need to do >>in the common case<< before processing the RDMA > option is relatively simple and a very small portion of the > overall protocol state machines, so I presume your comment is > asking for more careful wording? Or do you really disagree with the > fundamental point? Yes The sequence of operations you must execute is complex. You have to Check the packet is long enough Check the header is IPv4 Check the IPV4 IHL is valid Check the protocol field Check the source/dest addresses are legal Work out if dest is for us or another node Perform IP fragmentation (checks at least cannot be deferred defrag can) Walk the IP options (I guess you could defer this) Get the tcp header Check the tcp header lengths are legal and fit the packet length Check the tcp ports to figure out the socket Parse the tcp options Check the RDMA identifier, arbitarily aligned of course Check the RDMA identifier, port and addresses match Perform sequence space checks Then you can deliver the packet. Alan From XXX Mon Feb 28 00:48:25 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Feb 27 19:44:49 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA23031 for ; Sun, 27 Feb 2000 19:44:48 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA18345 for ips-outgoing; Sun, 27 Feb 2000 19:44:21 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from calcite.rhyolite.com (calcite.rhyolite.com [38.159.140.3]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA18335 for ; Sun, 27 Feb 2000 19:44:14 -0500 (EST) Received: (from vjs@localhost) by calcite.rhyolite.com (8.9.3/calcite) id RAA02693 env-from ; Sun, 27 Feb 2000 17:44:12 -0700 (MST) Date: Sun, 27 Feb 2000 17:44:12 -0700 (MST) From: Vernon Schryver Message-Id: <200002280044.RAA02693@calcite.rhyolite.com> To: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: "David R. Cheriton" > ... > Clearly, data is being received from hardware and software does not > get to touch it until it has been stored to some memory. My > assumption is that the storage system memory is arranged in fixed > size pages of disk/file pages. Without hardware RDMA to the storage > level, I believe one requires an extra copy, from whatever the > hardware delivers to what the storage system expects. Either > you use twice the bandwidth in the storage system memory system or > or else you have a separate memory system for the network, and > have software/processor power adequate to copy between at wire > speed (with all the associated support facilities for this processor.) > Unless there is something wrong with this reasoning, > it seems like a cost issue of providing the above hardware resources > vs. providing a NIC chip that can RDMA. Depending on how you are counting copies, that reasoning has been wrong in commercial UNIX systems for more than 10 years. Do you use the RDMA bits before IP checksum, the TCP checksum, and the medium FCS or checksum have been checked? If not, if you receive the entire link layer frame into some kind of temporary buffer or FIFO, probably in the "network interface card/controller," to check the trailing FCS and before using the RDMA bits, then commercial UNIX systems have been doing as you say to save copies since the late 1980's. As I said before, such systems were a part of what killed Protocol Engines Inc. If you do use the RDMA bits in the TCP header after 50-60 bytes of the frame have arrived, but before the frame FCS, aren't you worried about bit rot in the RDMA? > My guessitimate is that the software-only approach would be easily > 10 times more expensive here at the higher speed rates, of 10 Gbps. > If there is serious doubt about the merits of real hardware support, > we should try to quantify costs further at these speed ranges, IMHO. By "expensive," are you talking about dollars or bits/second? Regardless, if you look at the number of CPU cycles or gates in custom silicon required to support incoming page flipping in old, existing implementations, I bet you'll find that they are less "expensive" than any likely RDMA implementation. Power of 2 modular arithmetic is awfully cheap compared to parsing and validating TCP options. > ... > It would help me to have a more careful definition of the types of > attacks you have in mind. In an unsecure network with intruders, > presumably I can end up with bad data in the right buffer > or right data in the wrong buffer without using RDMA. > Do you view we have made things worse, and if so, how? > or are you objecting to us not making things better? Is it possible for a bad guy to use RDMA to put bad data into memory that is not a buffer? If the RID does no more than choose from a safe list of buffers, then how does RDMA usefully differ from the old FDDI, ATM, and HIPPI implementations that put incoming page-flippable data in buffers that get into user space with the data having been seen on the system bus the absolute minimum number of times for any scheme, including RDMA, once? Systems I've worked on have done mbuf allocation in the network interface hardware, including putting page-flippable payloads into page-mbufs that can eventually be flipped into user space. And of course, take care of the TCP or UDP checksum. Given the recently described extensions to readv(), absolutely all data received by a system like that would be page-flippable, and without needing the silicon or CPU cycles to parse RDMA options or requiring the sender to send RDMA options or even know that the receiver is being fast. Vernon Schryver vjs@rhyolite.com From XXX Mon Feb 28 02:04:26 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Feb 27 20:57:54 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA15201 for ; Sun, 27 Feb 2000 20:57:54 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA19782 for ips-outgoing; Sun, 27 Feb 2000 20:57:28 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ruby.cisco.com (ruby.cisco.com [171.69.198.43]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA19778 for ; Sun, 27 Feb 2000 20:57:23 -0500 (EST) Received: from alfonso. ([10.19.129.228]) by ruby.cisco.com (8.8.8-Cisco List Logging/8.8.8) with SMTP id RAA23480; Sun, 27 Feb 2000 17:56:10 -0800 (PST) Message-Id: <3.0.3.32.20000227175017.006baa14@ruby.cisco.com> X-Sender: cheriton@ruby.cisco.com X-Mailer: QUALCOMM Windows Eudora Pro Version 3.0.3 (32) Date: Sun, 27 Feb 2000 17:50:17 -0800 To: Alan Cox From: "David R. Cheriton" Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), julian_satran@il.ibm.com, David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: References: <3.0.3.32.20000227141724.006bd2d4@ruby.cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Alan, Given you are concerned about Appletalk and PCI, perhaps we just need to agree you are addressing a different performance level of storage system. I see no reason for hardware support at Appletalk and PCI speeds either. The RDMA option, even implemented in a hardware NIC, would not preclude software processing of Appletalk or anything else. I think it would be only fair for this group to expect you to provide a more fully worked out design and evaluation of your proposed NIC if this is really to be taken as a serious alternative to an RDMA option. I for one dont find it a competitive design (it is also one that was previously considered.) Regarding the complexity of TCP processing in hardware, people are doing this, so I regard this as a done deal. (I'd hate to be a high-speed NIC vendor that cant do this.) The only issue is how to handle the next level of protocols that have high performance requirements, i.e. storage. Also, NICs with hardware enhancements for protocol support, such as the Intel Gigabit Ethernet NIC have been very successful both in performance and market. So, expect more there. Finally, the RDMA option is targeted to allow use of standard Internet protocols for SANs in place of specialized protocols and networks such as FibreChannel, especially for non-local access. I.e. an extra option rather than an extra protocol stack and network. I hope the discussion can recognize that broader issue in considering alternatives and the general need for the RDMA option. I.e. can SCSI over TCP really compete with FC without the RDMA option? (In this sense, the subject line should have put SCSI first.) DRC At 10:41 PM 2/27/00 +0000, Alan Cox wrote: >> it seems like a cost issue of providing the above hardware resources >> vs. providing a NIC chip that can RDMA. > >or a NIC chip that has more memory on the chip and sends you the header then >you asynchronously give a DMA location for the rest of the buffer. > >You've now got rid of RDMA, instead your little bit of extra silicon is >generic, and will work with stuff like appletalk even. What does the >extra RAM and PCI glue cost you - probably not a lot. > >That is why I prefer software solutions > >> >It might also be fragmented of course. >> What you need to do >>in the common case<< before processing the RDMA >> option is relatively simple and a very small portion of the >> overall protocol state machines, so I presume your comment is >> asking for more careful wording? Or do you really disagree with the >> fundamental point? > >Yes > >The sequence of operations you must execute is complex. > >You have to > >Check the packet is long enough >Check the header is IPv4 >Check the IPV4 IHL is valid >Check the protocol field >Check the source/dest addresses are legal >Work out if dest is for us or another node >Perform IP fragmentation (checks at least cannot be deferred defrag can) >Walk the IP options (I guess you could defer this) >Get the tcp header >Check the tcp header lengths are legal and fit the packet length >Check the tcp ports to figure out the socket >Parse the tcp options >Check the RDMA identifier, arbitarily aligned of course >Check the RDMA identifier, port and addresses match >Perform sequence space checks > >Then you can deliver the packet. > >Alan > > > > From XXX Mon Feb 28 12:59:17 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Mon Feb 28 07:51:52 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id HAA25759 for ; Mon, 28 Feb 2000 07:51:51 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA08511 for ips-outgoing; Mon, 28 Feb 2000 07:51:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from the-village.bc.nu (lightning.swansea.uk.linux.org [194.168.151.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA08505 for ; Mon, 28 Feb 2000 07:51:34 -0500 (EST) Received: from alan by the-village.bc.nu with local (Exim 2.12 #1) id 12PPeo-0001qn-00; Mon, 28 Feb 2000 12:52:14 +0000 Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: cheriton@cisco.com (David R. Cheriton) Date: Mon, 28 Feb 2000 12:52:11 +0000 (GMT) Cc: alan@lxorguk.ukuu.org.uk (Alan Cox), julian_satran@il.ibm.com, David.Robinson@EBay.Sun.COM (David Robinson), ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: <3.0.3.32.20000227175017.006baa14@ruby.cisco.com> from "David R. Cheriton" at Feb 27, 2000 05:50:17 PM X-Mailer: ELM [version 2.5 PL1] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: From: Alan Cox Sender: owner-ips@ece.cmu.edu Precedence: bulk > Given you are concerned about Appletalk and PCI, perhaps we just > need to agree you are addressing a different performance level of > storage system. I see no reason for hardware support at Appletalk No we are addressing the same level of performance in a general purpose OS the difference is my case doesn't need weird protocol hacks. When working with a dedicated storage system RDMA becomes even less interesting because you code the stack to the needs of the filer. You don't even need an MMU on such kit > proposed NIC if this is really to be taken as a serious alternative > to an RDMA option. I for one dont find it a competitive design > (it is also one that was previously considered.) I don't find RDMA a credible useful solution at the protocol level. It doesn't offer any visible advantage, it complicates the stack futher thus punishing the majority in the interest of the few. I'm interested in why you think such a NIC wouldnt work, Providing you have interrupt mitigation I see no reason for it to fail. I'd be interested to know what the flaws in that technique were in your eyes. > doing this, so I regard this as a done deal. (I'd hate > to be a high-speed NIC vendor that cant do this.) But can you do it at $6 a part in volume ? Thats what the other 99.9% of the people care about. > The only issue is how to handle the next level of > protocols that have high performance requirements, i.e. storage. ST already has this sort of stuff figured out. > Also, NICs with hardware enhancements for protocol support, > such as the Intel Gigabit Ethernet NIC have been very successful > both in performance and market. So, expect more there. Intel provide no useful documentation so that is hard to evaluate. > Finally, the RDMA option is targeted to allow use of standard > Internet protocols for SANs in place of specialized protocols > and networks such as FibreChannel, especially for non-local > access. So is ST and ST is proven technology. If you are going to break the protocol to add hacks to it you might as well design the protocol properly based on the past twenty years of learning where TCP is hard to get right. You need IP to be compatible you don't need TCP. If you want to solve the generic problem then you don't need RDMA anyway Alan From XXX Mon Feb 28 12:59:17 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Mon Feb 28 03:43:38 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id DAA20701 for ; Mon, 28 Feb 2000 03:43:37 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA27170 for ips-outgoing; Mon, 28 Feb 2000 03:43:03 -0500 (EST) Received: from kickme.cisco.com (kickme.cisco.com [198.92.30.42]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA27166 for ; Mon, 28 Feb 2000 03:43:00 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by kickme.cisco.com (8.9.1a/8.9.1) with ESMTP id AAA29259; Mon, 28 Feb 2000 00:31:42 -0800 (PST) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id AAA19859; Mon, 28 Feb 2000 00:42:27 -0800 (PST) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Mon, 28 Feb 2000 00:42:27 -0800 (PST) From: Costa Sapuntzakis To: Vernon Schryver cc: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: NFS Header/data parsing and RDMA In-Reply-To: <200002280044.RAA02693@calcite.rhyolite.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Ok, so doing NFSv2/v3 header/data splitting is easy on an in-order TCP stream because NFS has fixed-length trailers. Here's a little technique: 1) Assume that all READ/WRITE transfers are powers of 2 2) Assume all RPCs larger than 4k are WRITE RPCs and all responses larger than 4k are READ responses 3) Take the message size from the first 4 bytes of the RPC/TCP encapsulation 4) Round the message size down to the nearest power of 2 (call this quantity data_size) 5) The data is the last data_size bytes of the message. Put the last data_size bytes in a separate aligned buffer. Note, to do this with NFS/TCP, your NIC has to do some primitive level of TCP processing (at least keep track of flows). It also needs to understand RPC/TCP message boundaries. Are there significantly simpler approaches than this? NFSv4 doesn't seem to have fixed length trailers and neither does CIFS in all cases. And it looks like it will be costly to parse NFSv4 headers. RDMA still has the following features: - Per-packet (Works with arbitrary out-of-order reception of TCP segments) - Fixed header that's generic across all protocols (NFSv4, v5, AFS, DFS, CIFS, etc..) - No page flipping necessary on solicited transfers - Message boundary bit (which is admittedly orthogonal to RDMA) allows out-of-order processing on TCP receive buffer. Decreases parsing latency, esp. in the face of packet drops. The following measures should improve security/safety: - NIC should ascertain that TCP segment is in receive window - NIC needs to check that the RID is valid for a given TCP conn for safety/security reasons If the NIC does header/data splitting, it needs to keep track of per-flow information because most file block transfesr will span multiple TCP segments. So the NIC will probably have a notion of a TCP flow #. The brute force approach way to check RID validity is to use a CAM to map from (RID, TCP flow #) -> buffer address. If that's too expensive, then RIDs can be hashed and the flow # and buffer address stored in the bucket. The flow # is verified before using the buffer address. -Costa From XXX Mon Feb 28 17:56:22 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Mon Feb 28 12:43:45 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA16186 for ; Mon, 28 Feb 2000 12:43:44 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA19352 for ips-outgoing; Mon, 28 Feb 2000 12:43:11 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from calcite.rhyolite.com (calcite.rhyolite.com [38.159.140.3]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA19347 for ; Mon, 28 Feb 2000 12:43:06 -0500 (EST) Received: (from vjs@localhost) by calcite.rhyolite.com (8.9.3/calcite) id KAA14348 env-from ; Mon, 28 Feb 2000 10:43:01 -0700 (MST) Date: Mon, 28 Feb 2000 10:43:01 -0700 (MST) From: Vernon Schryver Message-Id: <200002281743.KAA14348@calcite.rhyolite.com> To: ips@ece.cmu.edu, tcp-impl@grc.nasa.gov Subject: Re: NFS Header/data parsing and RDMA Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: Costa Sapuntzakis > Ok, so doing NFSv2/v3 header/data splitting is easy on an in-order > TCP stream because NFS has fixed-length trailers. Here's a little > technique: > ... > Note, to do this with NFS/TCP, your NIC has to do some primitive > level of TCP processing (at least keep track of flows). It also > needs to understand RPC/TCP message boundaries. Do I understand correctly that you're applying the familiar NFS/UDP page flipping tactic to NFS/TCP? > Are there significantly simpler approaches than this? 1. How about using NFS/UDP instead of NFS/TCP? It's well known in the NFS community that NFSv2-3/TCP is no faster or otherwise better than NFSv2-3/UDP except over very narrow or at least rather long pipes. (Recall also the congestion control and avoidance mechanisms in some NFSv2-3/UDP implementations.) 2. Use NFS/TCP, but send every RPC/XDR transaction in a single TCP segment, and use IP fragmentation to fit the MTU. This tactic was used for 10+ years ago in the FDDI adapters of some super computers. It does have the problems of IP fragmentation, but those problems are rarely encountered where NFS is used. > NFSv4 doesn't seem to have fixed length trailers and neither > does CIFS in all cases. And it looks like it will be costly to parse > NFSv4 headers. I've not been paying attention to NFSv4. A quick skim of the draft suggests that it will not displace NFSv2/3 in the environments where NFS is currently popular. NFSv4 certainly has nothing to do with anything like SCSI over IP. I'm also far from convinced that NFSv4 has got some of the extensions close enough to the underlying real filesystems to be popular. Even if I'm wrong, it will be years before NFSv4 is widely used While I think there are ways to page flip NFSv4 without special hardware, I don't think they are worth talking about yet. Even if I'm also wrong about that, it is years early to be modifying TCP/IP to support NFSv4. No one can see what NFSv4 will be like when it is popular enough to justify modifying TCP today, if NFSv4 ever is popular. > RDMA still has the following features: > > - Per-packet (Works with arbitrary out-of-order reception of TCP > segments) > - Fixed header that's generic across all protocols (NFSv4, v5, AFS, > DFS, CIFS, etc..) > - No page flipping necessary on solicited transfers > - Message boundary bit (which is admittedly orthogonal to RDMA) allows > out-of-order processing on TCP receive buffer. Decreases parsing latency, > esp. in the face of packet drops. > ... Knowing to which buffer an out-of-order TCP segment belongs is something that I don't see how to do without something like RDMA. However, out-of-order TCP segments are both very rare and very bad for TCP performance, regardless of whether RDMA is present. Out of order TCP segments must be even more rare in storage networks. Talk about NFSv5 or even AFS/DFS does the opposite of make me think there might be something good in RDMA. And as I've said, it's years too early to justifiy RDMA with NFSv4. With existing techniques, if you don't want to page flip, you don't need to. If you are able to provide enough distinct application buffer streams to the NIC for RDMA, then you could do the same for other techniques. What's that about "parsing latency" and what does it have to do with lost segments? Are you proposing to deliver TCP data to applications out of order? I trust not! .... ] From: Zachary Amsden ] ... ]No, that situation doesn't require any hardware support. However, a zero-copy ] receive path is not the only element of RDMA - RDMA was designed (I suppose ] from the discussion here) specifically to address header/payload issues for ] storage protocols. Clearly one can do zero-copy receive with changes to the ] API and no hardware/firmware modifications. But with no special hardware ] support, flipping the payload into some page with alignment constraints will ] require another copy. What about the many systems that have been page flipping NFS in and out of buffer caches for more than 10 years, with no changes to APIs or special silicon? ]There is one exception to my last statement that I know of: If you pre-adjust ]the hardware receive buffers to make the payload align on a page boundary, you ] can flip the page into the buffer cache for (hopefully) the common case. ] However, this requires the ability to tune these header offsets and will only ] work for one protocol at a time (mostly). The page flipping systems I've worked on did not tune header offsets and worked on more than one protocol. (Given your email address, it might be interesting to check the old IRIX source trees. Besides the NFS kernel code and the HIPPI, ATM, and FDDI drivers and firmware, check cmd/rcp and cmd/rsh.) UDP page flipping is trivial on protocols that have no trailers. It requires trivial smarts in the NIC and much simpler buffer allocation by the NIC than RDMA requires. (I suspect RDMA needs pools of buffers for every stream, while the classic tactic needs only two pools, "little" and "pages"....well, for tiny improvements I've also done it with "little", "medium" and "pages".) ] Realistically, who is going to be running a storage system that requires so ] much bandwidth that avoiding receive copies is necessary, and runs on generic ] NICs with no firmware/ASIC modifications possible? So I think using modified ] hardware is completely reasonable in those circumstances. ] ... Even more reasonable than special hardware are modified API's and protocols and other steps, including ensuring that out-of-order packets are very rare, and with header offsets are few, fixed, known, and friendly. How would you have out-of-order arrival on a storage network, other than due to bit rot in the wires, and what storage network is going to have significant bit rot? Vernon Schryver vjs@rhyolite.com From XXX Mon Feb 28 18:45:31 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Mon Feb 28 13:28:01 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA07611 for ; Mon, 28 Feb 2000 13:28:00 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA21022 for ips-outgoing; Mon, 28 Feb 2000 13:27:49 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail-gw.hursley.ibm.com (mail-gw.hursley.ibm.com [194.196.110.15]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA21015 for ; Mon, 28 Feb 2000 13:27:45 -0500 (EST) Received: from sp3at21.hursley.ibm.com (sp3at21.hursley.ibm.com [9.20.45.21]) by mail-gw.hursley.ibm.com (AIX4.3/UCB 8.8.8/8.8.8) with ESMTP id SAA41482; Mon, 28 Feb 2000 18:27:12 GMT Received: from hursley.ibm.com (gsine02.us.sine.ibm.com [9.14.6.42]) by sp3at21.hursley.ibm.com (AIX4.2/UCB 8.7/8.7.3) with ESMTP id SAA12432; Mon, 28 Feb 2000 18:27:12 GMT Message-ID: <38BABD1D.98766626@hursley.ibm.com> Date: Mon, 28 Feb 2000 12:23:25 -0600 From: Brian E Carpenter Organization: IBM X-Mailer: Mozilla 4.61 [en] (Win98; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: Costa Sapuntzakis CC: Aman Singla , "Mailing List, IP Storage" Subject: Re: Scheduled Transfer Protocol (ST) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I hope everybody realises that in the IETF, ST has been used for years to refer to the experimental Internet Stream protocol (RFC 1819), which is also IPv5. ANSI is creating confusion by re-using the acronym. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Brian E Carpenter (IAB Chair) Program Director, Internet Standards & Technology, IBM On assignment for IBM at http://www.iCAIR.org Attend INET 2000: http://www.isoc.org/inet2000 Non-IBM email: brian@icair.org From XXX Mon Feb 28 20:29:42 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Mon Feb 28 15:20:30 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id PAA25035 for ; Mon, 28 Feb 2000 15:20:30 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA25896 for ips-outgoing; Mon, 28 Feb 2000 15:20:06 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pneumatic-tube.sgi.com (pneumatic-tube.sgi.com [204.94.214.22]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA25888 for ; Mon, 28 Feb 2000 15:20:01 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id MAA06550; Mon, 28 Feb 2000 12:22:47 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id MAA47704; Mon, 28 Feb 2000 12:19:34 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id MAA37310; Mon, 28 Feb 2000 12:17:41 -0800 (PST) Message-ID: <38BAD7E5.6499FEAE@engr.sgi.com> Date: Mon, 28 Feb 2000 12:17:41 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: Costa Sapuntzakis CC: "Mailing List, IP Storage" , Jim Pinkerton Subject: Re: Scheduled Transfer Protocol (ST) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > Thanks for bringing the ST protocol to our attention. I'm not sure > it fits all the requirements, though, because one of the goals of IPS > is to generate a WAN-friendly protocol. > > Is STP designed for the WAN? I see no provision in the ST spec for > TCP-style congestion control or even exponential back-off. In addition, You are right in pointing out that STP as a protocol is still quite immature when it comes to applying it to a WAN scenario. What I would say is that STP has various useful primitives for congestion control etc. built into it - the number of outstanding CTSs, the block size to use etc. are all tunable parameter and can be updated in the STP header by switches/router/entities enroute; there are primitives to timeout and regenerate requests. But the mechanisms to trigger timeouts, and determine the various parameters correctly for a WAN scenario are missing; and require work/input/design based on the requirements. > the Clear_To_Send mechanism as currently speced seems to add > a round-trip time to every transfer, increasing latency. For significantly sized transfers, the CTSs can be and are pipelined with the data-transfer; so the only observable CTS latency is for the first block of data - which ammortized over the length of the transfer still gives you great b/w. For latency sensitive short message exchanges, persistently mapped memory regions act as data buffers and can be incredibly fast by not requiring CTSs for the data. > As for the SCSI/ST spec: > > Is there any mechanism for ordering tasks? For example, if you > want to enter several ordered tasks into a SCSI target, can you > issue them simultaneously, i.e. without waiting for an ACK?. I'm not very sure about the point you're trying to make here. STP will just be the transfer medium for the SCSI commands and the data - in a way like SCSI fibre channel is.. Yeah! a SCSI stack can issue multiple outstanding requests to a SCSI target using STP - their ordering requirement, if any, would be determined by the SCSI header - and their ordering, if any, would be determined by the SCSI target. My apologies if I haven't grasped your question correctly.. thanks, :a From XXX Mon Feb 28 20:29:42 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Mon Feb 28 15:24:35 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id PAA25052 for ; Mon, 28 Feb 2000 15:24:34 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA26087 for ips-outgoing; Mon, 28 Feb 2000 15:24:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pneumatic-tube.sgi.com (pneumatic-tube.sgi.com [204.94.214.22]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA26083 for ; Mon, 28 Feb 2000 15:24:26 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id MAA09195; Mon, 28 Feb 2000 12:27:02 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id MAA38449; Mon, 28 Feb 2000 12:23:52 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id MAA37470; Mon, 28 Feb 2000 12:21:04 -0800 (PST) Message-ID: <38BAD8B0.B3E32A0C@engr.sgi.com> Date: Mon, 28 Feb 2000 12:21:04 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: Brian E Carpenter CC: Costa Sapuntzakis , Aman Singla , "Mailing List, IP Storage" Subject: Re: Scheduled Transfer Protocol (ST) References: <38BABD1D.98766626@hursley.ibm.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks for pointing that out Brian. We've been aware of this issue for sometime now, and we at SGI are making a conscious effort to start using STP for the scheduled transfer protocol instead of the acronym ST. The IANA registered protocol number 118 is also registered as IPPROTO_STP. thanks, :a Brian E Carpenter wrote: > > I hope everybody realises that in the IETF, ST has been used for > years to refer to the experimental Internet Stream protocol > (RFC 1819), which is also IPv5. ANSI is creating confusion by re-using > the acronym. > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > Brian E Carpenter (IAB Chair) > Program Director, Internet Standards & Technology, IBM > On assignment for IBM at http://www.iCAIR.org > Attend INET 2000: http://www.isoc.org/inet2000 > Non-IBM email: brian@icair.org From XXX Fri Mar 03 16:27:10 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Mar 2 20:45:53 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA18846 for ; Thu, 2 Mar 2000 20:45:52 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA10011 for ips-outgoing; Thu, 2 Mar 2000 20:44:41 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA10006 for ; Thu, 2 Mar 2000 20:44:35 -0500 (EST) Received: from sunmail1.Sun.COM ([129.145.1.2]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id SAA17013; Thu, 2 Mar 2000 18:44:12 -0700 (MST) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.85.31]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id RAA24990; Thu, 2 Mar 2000 17:44:10 -0800 (PST) Received: from lillen (awe174-18.AWE.Sun.COM [192.29.174.18]) by jurassic.eng.sun.com (8.10.0.Gamma0+Sun/8.10.0.Gamma0) with SMTP id e231i8L11845; Thu, 2 Mar 2000 17:44:08 -0800 (PST) Date: Thu, 2 Mar 2000 16:20:10 -0800 (PST) From: Erik Nordmark Reply-To: Erik Nordmark Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc. To: "David R. Cheriton" Cc: Alan Cox , julian_satran@il.ibm.com, David Robinson , ips@ece.cmu.edu, tcp-impl@grc.nasa.gov In-Reply-To: "Your message with ID" <3.0.3.32.20000227175017.006baa14@ruby.cisco.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > I think it would be only fair for this group to expect you to > provide a more fully worked out design and evaluation of your > proposed NIC if this is really to be taken as a serious alternative > to an RDMA option. I for one dont find it a competitive design > (it is also one that was previously considered.) David, I suspect that it is the proponents of the RDMA option in TCP that have to make the case that RDMA is useful and not the other way around. Erik From XXX Sat Mar 11 03:42:15 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Mar 10 20:29:13 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA19209 for ; Fri, 10 Mar 2000 20:29:13 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA20838 for ips-outgoing; Fri, 10 Mar 2000 20:27:10 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.lss.emc.com (mercury.eng.emc.com [168.159.40.77]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA20828 for ; Fri, 10 Mar 2000 20:27:04 -0500 (EST) Received: by mercury.eng.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 10 Mar 2000 20:32:04 -0500 Message-ID: <8A7271ACBE5BD311A4D50090273BF02302E43BBB@montana.lss.emc.com> From: "harwood, jack" To: ips@ece.cmu.edu Subject: Comments on the current iSCSI draft Date: Fri, 10 Mar 2000 20:26:05 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Comments on the internet-SCSI (iSCSI) Document 3/7/00 ------------------------------------------------------------ The following are comments compiled from EMC engineering with respect to the internet-SCSI document drafted by IBM and Cisco. In general this document has provided a good start to mapping SCSI over TCP, in particular with mapping to TCP and not UDP. Specifically the comments cover * Architecture - Issues with the overall document that appear to pose obstacles to widespread deployment. * Conceptual - Issues with document that need addressing to enable effective implementations. * Specifics - Issues with bits and bytes in the command descriptions * Questions - Questions about decisions made. Architectural ------------- * There is an issue with the separation of the Control and Data Channel. NAT (address translation), firewall, or load balancing products will not support iSCSI without changes which in turn is a barrier to adoption for large networks. If the goal is to provide interleaving of control commands with large data transfers we feel this can be accomplished in other ways. - Use smaller data frames to allow better interleaving of control and data on a single connection - Use multiple connections between the same source and destination pair where each connection is independent of other connections (i.e., data/control are combined on each connection). Separation of control and data also adds new failure modes where one channel closes but the other does not. * The use of DNS addressing in the protocol as described in sections 3.13, Open Data Connection, and section 3.17, Third Party Copy, will force all parties to depend on DNS in order for the protocol to work. While system and network administrators should be free to make this choice (and invest the effort in making DNS suitably robust), this protocol design should NOT be based on the assumption that DNS is a robust highly available service. The protocol should be based on IP addresses. Conceptual ---------- * The iSCSI protocol requires a strong authentication mechanism. In its current form, without an implementation and corresponding specification, it is impossible to write an interoperable authentication implementation from the document as it stands, hence at least one strong authentication mechanism must be mapped onto the protocol, possibly in a separate document or documents. * The parameter negotiation, described in sections 3.9-12, is very general. The free-form text/value format will cost code to parse and may not be justified. * The action of killing all outstanding IOs on a login or operation timeout seems too severe for this process and provides an opening for a denial of service attack. Also there is no other rationale in the document as to why this semantic is useful. * A general mapping of error recovery for iSCSI is needed, i.e. what parts need definition versus what will use TCP error recovery mechanisms. * In section 3.17, Third party copy needs a much better explanation about authentication, login and how the entire process works. Specifics --------- * It should be stated specifically in sections 2.4 and 3.8 that iSCSI data segments cannot overlap. * The expected data length and flags, i.e. command direction, should be described in the SCB itself and not as separate fields in the SCSI command, see section 3.3. * Using the task tag and TCP connection 4-tuple (source and destination IP addresses and ports) we should have a fully qualified identifier and should not need LUN number in the response and task management response, see section 3.3 and 3.6. * The LUN number should be embedded in the data for the AEN, see section 3.4. * In section 5.1 a recommendation is made to use 8k as the upper limit for small TCP segments. Depending on the MTU size this recommendation may cause fragmentation. More detail and analysis are needed to justify this recommendation. * A standard CRC should be required, see section 6.1. * The target should not gets its name from the initiator, see section 10.1. * Section 10.3 needs to provide details on how to prevent reply/reuse. Also this text seems to allow passwords in the clear which is not acceptable. * In section 10.5 it states "Once AllowNoRTT has been set to 'yes', it cannot be set back to no". It should clarify this is for the open connection and closing this connection and opening a new connection will clear this condition. Questions -------- * What value does the ability to do an iSCSI ping add to the existing ability to do an ICMP ECHO? If little or none, this should be omitted, see section 3.15. TCP-RDMA -------- Although the premise of TCP acceleration is quite useful the concept of RDMA does not apply for our application of internet SCSI. We will handle the moving of data as implementation specific and not as generic design such as RDMA. From XXX Sat Mar 11 08:05:15 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sat Mar 11 01:23:08 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id BAA11197 for ; Sat, 11 Mar 2000 01:23:07 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA25993 for ips-outgoing; Sat, 11 Mar 2000 01:21:19 -0500 (EST) Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25989 for ; Sat, 11 Mar 2000 01:21:15 -0500 (EST) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [171.69.199.29]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id WAA16083; Fri, 10 Mar 2000 22:20:43 -0800 (PST) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id WAA23150; Fri, 10 Mar 2000 22:20:44 -0800 (PST) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Fri, 10 Mar 2000 22:20:44 -0800 (PST) From: Costa Sapuntzakis To: "harwood, jack" cc: ips@ece.cmu.edu Subject: Re: Comments on the current iSCSI draft In-Reply-To: <8A7271ACBE5BD311A4D50090273BF02302E43BBB@montana.lss.emc.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Thank you for the excellent, detailed feedback. There are at least a couple points where the iSCSI draft did not explain itself well. > * The use of DNS addressing in the protocol as described in sections 3.13, > Open Data Connection, and section 3.17, Third Party Copy, will force all > parties to depend on DNS in order for the protocol to work. While system and > network administrators should be free to make this choice (and invest the > effort in making DNS suitably robust), this protocol design should NOT be > based on the assumption that DNS is a robust highly available service. The > protocol should be based on IP addresses. Parties who wish to use IP addresses for extra reliability may encode the IP addresses in the target names (e.g. 10.0.4.5/dvd). After all, domain names can represent IP addresses. These type of domain names do not require DNS for resolution. Also, domain name resolution will only be required in parties that act as either initiators or application-level gateways/proxies. Domain name resolution does not necessarily imply DNS, though a directory service like DNS is a compelling method of domain name resolution. UNIX machines have also used the /etc/hosts file and YP/NIS for resolution. On the plus side, domain names decouple the iSCSI protocol from the underlying addressing architecture. The potential deployment of IPv6 will not require any changes to the iSCSI protocol. > * The parameter negotiation, described in sections 3.9-12, is very general. > The free-form text/value format will cost code to parse and may not be > justified. An implementation can ignore all free-form text/value pairs and still operate just fine. This needs to be clarified as a design goal. The code impact should be minimal. > * The expected data length and flags, i.e. command direction, should be > described in the SCB itself and not as separate fields in the SCSI command, > see section 3.3. The motivation was that SCSI/PI and ATAPI bridges would not be required to parse SCBs. > * A standard CRC should be required, see section 6.1. A CRC at the TCP layer or higher? > * The target should not gets its name from the initiator, see section 10.1. I think there's a misunderstanding here. This is used by to direct the TCP connection to a specific target. We will clarify this in the draft. > Questions > -------- > * What value does the ability to do an iSCSI ping add to the existing > ability to do an ICMP ECHO? If little or none, this should be omitted, see > section 3.15. It makes sure the iSCSI device server is still alive and kicking. -Costa From XXX Sun Mar 12 18:13:06 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Mar 12 12:43:00 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA05914 for ; Sun, 12 Mar 2000 12:43:00 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA10895 for ips-outgoing; Sun, 12 Mar 2000 12:40:22 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA10890 for ; Sun, 12 Mar 2000 12:40:18 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA70560; Sun, 12 Mar 2000 18:39:47 +0100 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id SAA167738; Sun, 12 Mar 2000 18:39:41 +0100 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568A0.00610350 ; Sun, 12 Mar 2000 18:39:38 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Jack Harwood cc: ips@ece.cmu.edu Message-ID: Date: Sun, 12 Mar 2000 19:39:30 +0200 Subject: Comments to Comments! Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jack, Thanks for your attention and detailed comments. I sincerely hope that we could work all together to get to a better standard. And here are our thoughts as expressed by several authors: --- start forwarded message by harwood, jack --- > From: "harwood, jack" > To: ips@ece.cmu.edu > Subject: Comments on the current iSCSI draft > Date: Fri, 10 Mar 2000 20:26:05 -0500 ... > Architectural > ------------- > * There is an issue with the separation of the Control and Data Channel. NAT > (address translation), firewall, or load balancing products will not support > iSCSI without changes which in turn is a barrier to adoption for large > networks. If the goal is to provide interleaving of control commands with > large data transfers we feel this can be accomplished in other ways. > - Use smaller data frames to allow better interleaving of control and > data on a single connection > - Use multiple connections between the same source and destination pair > where each connection is independent of other connections > (i.e., data/control are combined on each connection). > Separation of control and data also adds new failure modes where one channel > closes but the other does not. True, separating data from control introduces some new problems that could be avoided if we interleave. We briefly considered such a design aiming at one TCP connection per LUN. But this is inordinately expensive. If we multiplex LUNs (as we do in the current draft) keeping to a short TCP frame will leave as open to all sorts of troubles (possible deadlocks) due to the limited TCP window and our lack of control over the data source and sink. Separating the control and data stream we could resort to selective resets to get out of trouble - while with a common connection we might have to resort to radical means (e.g., closing connections). In addition in a "permissive" environment (like a video server) we might require CRC on the control connection while leaving the data connections up to the user. It is a bit more difficult to implement but worth the trouble. > * The use of DNS addressing in the protocol as described in sections 3.13, > Open Data Connection, and section 3.17, Third Party Copy, will force all > parties to depend on DNS in order for the protocol to work. While system and > network administrators should be free to make this choice (and invest the > effort in making DNS suitably robust), this protocol design should NOT be > based on the assumption that DNS is a robust highly available service. The > protocol should be based on IP addresses. It is true that the system recommends using DNS. However, the administrator is free to choose names such as "123.45.67.89" and the initiators and targets will interpret that as IPv4 (or IPv6) as necessary. It was felt that we should be completely independent of IP addresses because of firewall and IP masquerading issues with setting up new TCP connections. IP addresses /can/ be used, but only in dotted decimal notation. Note that no addresses need be provided for simple systems, and all key:value pairs can be safely ignored by the target. > Conceptual > ---------- > * The iSCSI protocol requires a strong authentication mechanism. In its > current form, without an implementation and corresponding specification, it > is impossible to write an interoperable authentication implementation from > the document as it stands, hence at least one strong authentication > mechanism must be mapped onto the protocol, possibly in a separate document > or documents. Correct. We decided to make a flexible framework for authentication, rather than specify a particular method. Specific authentication schemes could be described in other documents. We briefly considered (and are not outright rejecting) other schemes - most notably the one used in SST (SCSI over ST) in which in fact the connection can go through 3 stages - Idle - Authenticating - Active. 1 bit in the login indicates if the authentication is required and gets the state machines in either the Authenticating stage or the Active stage. The standard does not address how you go from authenticating to Active. This design enables non-authenticating machines to interoperate and leaves open the whole authentication process to other standards. We felt that we have to have a minimal authentication specifies at least to avoid "good faith" mistakes but we are open to discuss this in the working group at some length. > * The parameter negotiation, described in sections 3.9-12, is very general. > The free-form text/value format will cost code to parse and may not be > justified. We designed the system so that any non-responses to TEXT commands are considered as not supported. On targets or initiators where text:value is too complex, a set of defaults should be chosen and no TEXT commands supported. For targets, the MODE SELECT can set SCSI-like things. The TEXT command covers Network-like things. > * The action of killing all outstanding IOs on a login or operation timeout > seems too severe for this process and provides an opening for a denial of > service attack. Also there is no other rationale in the document as to why > this semantic is useful. I assume you are referring to what is written in the section on Error Handling (section 4.0). Denial of service is a problem inherent in all IP based protocols, and we cannot completely solve it. The initiator can wait a long time before it determines that it has timed out. TCP ensures ordered delivery as long at there is a connection. What other alternative is there other than to completely clean up, once it has been decided that we have a connection problem? > * A general mapping of error recovery for iSCSI is needed, i.e. what parts > need definition versus what will use TCP error recovery mechanisms. Did you have a particular situation in mind that iSCSI does not cover? > * In section 3.17, Third party copy needs a much better explanation about > authentication, login and how the entire process works. Again, this is a framework. When devices start offering third party commands that go beyond the provisions of iSCSI, we will extend it. We know about and we think we covered the extended copy commands considered by the SCSI working group. > Specifics > --------- > * It should be stated specifically in sections 2.4 and 3.8 that iSCSI data > segments cannot overlap. We agree that the iSCSI should state that data segments should not overlap (and will do this in the next version). However we would be reluctant to require that receiver implementations check for this type of error and report it in the status. Is this acceptable? > * The expected data length and flags, i.e. command direction, should be > described in the SCB itself and not as separate fields in the SCSI command, > see section 3.3. As stated by SAM the SCB contains only the number of data blocks not the transfer length. SAM also mandates that the "execution request" include the data length and CAM (as well as other standard software interfaces) require a residual count report with reference to the length. It make all the implementations "more compliant" to include the length. For all hardware bridge providers it makes also more sense to have the length and direction in a "common" header than to scan SCBs. > * Using the task tag and TCP connection 4-tuple (source and destination IP > addresses and ports) we should have a fully qualified identifier and should > not need LUN number in the response and task management response, see > section 3.3 and 3.6. You are right - it was so many times on and off! It ended up being there to make all controls "target-to-initiator" identical. The last reasoning behind getting it in was a "proxy LUN" - i.e. the work was done by a "third party". If the returned LUN disagrees with the transmitted LUN then it may mean that a proxy satisfied the request. However, we have not specified what action should be taken and I cannot at present think of anything useful to do with any proxy-LUN information. We (the working group, including you hopefully, in its infinite wisdom!) might decide to remove it. > * The LUN number should be embedded in the data for the AEN, see section > 3.4. We do not specify what goes in the data that is sent in an Asynchronous Error Notification. I think SAM-2 requires LUN to be specified (as a parameter). We want to be independent of whatever data is packaged, and we therefore have to specify the LUN in the header. > * In section 5.1 a recommendation is made to use 8k as the upper limit for > small TCP segments. Depending on the MTU size this recommendation may cause > fragmentation. More detail and analysis are needed to justify this > recommendation. 8k is an upper limit. If MTU size is smaller, then a smaller data size should be used, as implied by the note to the implementer. 8k is also an upper limit for good CRC algorithms (perhaps 8k is too big for this also). We welcome a more detailed analysis to provide a better recommendation. > * A standard CRC should be required, see section 6.1. A agree that a good CRC is a thing to have. I think that a TCP-CRC should be mandated for the control channel. This should be set when opening the TCP connection for the control channel. There are cases where CRC is not desirable for the data connection, as when transferring transient voice or video . Hence there ought to be some kind of negotiation as to whether CRC will be used for the data channel (like a parameter for open). Let's talk some more about it. > * The target should not gets its name from the initiator, see section 10.1. The target can ignore any key:value pairs sent by the initiator, so it need not receive its name from the initiator. This feature is useful in case the target is actually a front end for many machines and/or disks, in which case the initiator can specify to which target it really wants to interact with. > * Section 10.3 needs to provide details on how to prevent reply/reuse. Also > this text seems to allow passwords in the clear which is not acceptable. The example given is conceptual. You can use encryption if you the initiator and target can agree on it, or if it automatically provided by the TCP layer. But we are ready to work some more on it. > * In section 10.5 it states "Once AllowNoRTT has been set to 'yes', it > cannot be set back to no". It should clarify this is for the open > connection and closing this connection and opening a new connection will > clear this condition. This was the intention. We will clarify. > Questions > -------- > * What value does the ability to do an iSCSI ping add to the existing > ability to do an ICMP ECHO? If little or none, this should be omitted, see > section 3.15. This is very valuable. First, ICMP may be blocked by a firewall. Second, it is very useful to test certain pathological data sets over particular networks. Third, when a TCP link is not being used, no data is sent. This makes it almost impossible to detect if the connection has been broken. Having a ping command allows the TCP connection to be tested periodically. And it tests more than just the TCP/IP stack - a valuable add-on in many settings. > TCP-RDMA > -------- > Although the premise of TCP acceleration is quite useful the concept of RDMA > does not apply for our application of internet SCSI. We will handle the > moving of data as implementation specific and not as generic design such as > RDMA. As they say - we all leave in free world... I would agree that you have a strong case for a controller but I am not that confident about a general purpose host adapter - like a NIC card (not SCSI specific) > --- end forwarded message by harwood, jack --- Regards, Julo Julian Satran (on behalf of all my colleagues), IBM Research at Haifa From XXX Sun Mar 12 20:51:08 2000 Return-Path: julian_satran@il.ibm.com Delivery-Date: Sun Mar 12 13:47:05 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA30600 for ; Sun, 12 Mar 2000 13:47:04 -0500 From: julian_satran@il.ibm.com Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA12117 for ; Sun, 12 Mar 2000 13:47:03 -0500 (EST) Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id TAA15044; Sun, 12 Mar 2000 19:45:25 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id TAA56878; Sun, 12 Mar 2000 19:45:21 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568A0.006704BC ; Sun, 12 Mar 2000 19:45:14 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Dave_Lee@3com.com cc: "Bradley, Mark" , "VonStamwitz, Paul" , "Wilson, Andrew" , "'Costa Sapuntzakis'" , bassoon@ece.cmu.edu, alan@orca.com, harwood_jack@emc.com Message-ID: Date: Sun, 12 Mar 2000 20:42:46 +0200 Subject: RE: IPS Issues document Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Hi, Good job. I had only an hour to glance over it so I might come later with some more but here are some observations: Performance - has two aspects - related but not entirely equivalent - latency and bandwidth. It is obvious that bandwidth is not a differentiating factor for any type of interconnect as they use the same physical infrastructure in a similar fashion. Latency on the other hand is a more difficult factor as it relates to hardware, software and protocol. CPU utilization by itself is arguably not a limiting factor - as it can be solved by simply adding more CPUs or waiting until the CPUs get faster. All the elements you mention under CPU utilization refer mostly to the memory subsystem utilization - and there are no known ways to improve on that as the memory does not get faster as fast as the CPUs do (no Moore law for DRAMs!). Protocol stack processing reduction is the mainly improving your "placement" techniques and reducing copy operations. Storage subsystems fare better (and so does FCP) since the do scatter-gather DMA (and DMA is times faster than copy due to an architectural "feature" of DRAMs that use one address setup for several memory operations. This is the reason why we think that giving the protocols a chance to do well DMA is so important Adoption of the solution by the device vendors is important but not critical. Adoption of a "last half-meter" protocol - like the serial-ATA pushed by Intel, with all its drawbacks could offer a cheap alternative for fiercely competitive environments like desktops with SCSI as a more robust and functional cousin. For clusters and RAIDs both can be considered and with IP getting adopted by pervasive computing may get IP networks to the point at which the price differential will not exist anymore You do not mention the SBP although, I recall it as the first serial SCSI standardized On the performance section I think that solution scalability for the low-end to the high-end is essential as it is the only way to gain "mass" and survive the continuous shift in performance Management should include both storage and interconnect. Any solution should be able to use for interconnect management whatever is good for network management with only slight concern about the device specifics. A factor that I have difficulty expressing is - skills and tools availability. By this I mean planers, administrators and (why not!) even salesman and their tools. Any solution that we leverage an existing skill (and tools) base will have a distinct advantage Regards, Julo Julian Satran - IBM Research at Haifa From XXX Tue Mar 14 13:42:08 2000 Return-Path: paulv@corp.adaptec.com Delivery-Date: Tue Mar 14 00:52:13 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id AAA27173 for ; Tue, 14 Mar 2000 00:52:12 -0500 Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA09767; Tue, 14 Mar 2000 00:52:10 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id VAA20775; Mon, 13 Mar 2000 21:51:29 -0800 (PST) Received: from aimexc03.corp.adaptec.com (aimexc03.corp.adaptec.com [162.62.190.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id VAA18070; Mon, 13 Mar 2000 21:51:28 -0800 (PST) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Mon, 13 Mar 2000 21:51:28 -0800 Message-ID: <00C2F5C6AE05D311BD3700204840376703A10E01@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: "'ips@ece.cmu.edu'" , "'Scott Bradner'" , "'bassoon@ece.cmu.edu'" Subject: IPS Issues/Requirements Document Date: Mon, 13 Mar 2000 21:51:27 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Appended is a first draft of the IPS Issues/Requirements document. It was submitted to the IETF as an Internet Draft on March 10, 2:42 PST Unfortunately, it missed the cutoff time of 5:00 EST by 42 minutes, so it was NOT processed. All comments and suggestions for improvements all welcome by responding to the IPS list, or to me directly. The timeline for the final draft is March 20, so the more feedback I can have this week, the better. Dave Nagle, If you feel that it is appropriate, please cross post it to other storage groups (SNIA, NSIC, T10, etc.) to get them involved and post it to the web site. Thank you, Paul von Stamwitz ----------------------------- Internet Draft P. von Stamwitz D. Wilson Expires 10 September 2000 Adaptec C. Sapuntzakis Cisco Systems D. Lee 3Com March 2000 IP Storage Issues and Requirements This document is an Internet-Draft and is NOT offered in accordance with Section 10 of RFC2026, and the author does not provide the IETF with any rights other than to publish as an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Status of this Memo This document specifies issues and requirements for the IP Storage BOF and any future working group. It is a first draft and has not been reviewed by the Internet community. A subsequent revision is likely after discussion and suggestions for improvements have been submitted. The next revision is planned to be submitted by March 22, 2000. Please submit all comments to the ips@ece.cmu.edu or paulv@corp.adaptec.com. Distribution of this memo is unlimited. Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 1] IP Storage Issues and Requirements March 2000 Table of Contents Abstract ............................................................. 2 1. Introduction and Motivation ....................................... 3 2. Requirements and issues for storage ............................... 3 2.1. Data Integrity and Reliability ................................. 3 2.2. Ability to span large distances ................................ 3 2.3. Performance .................................................... 4 2.3.1. Performance vs. Connectivity Solution ....................... 4 2.3.2 Competitive with existing architectures ...................... 4 2.3.3 CPU Utilization .............................................. 5 2.3.4 Storage Stack vs. Network Stack Processing ................... 5 2.3.5. Flow control at the Storage Protocol ........................ 5 2.4. Storage Management ............................................. 6 2.4.1. Sharing Storage vs. Data .................................... 6 2.5. Security ....................................................... 6 2.6. Device Discovery and Configuration ............................. 6 3. Optimum Criteria for Solutions .................................... 7 4. Candidates for Solutions .......................................... 8 4.1. Transport ...................................................... 8 4.1.1. TCP ......................................................... 8 4.1.2. UDP ......................................................... 8 4.1.3. Scheduled Transfer (ST) ..................................... 8 4.1.4. Other? ...................................................... 8 4.2 Storage Protocol ................................................ 8 4.2.1. SCSI ........................................................ 8 4.2.2. FCP ......................................................... 9 4.2.3. ATA ......................................................... 9 4.2.4. NFS ......................................................... 9 4.2.5. Other? ...................................................... 9 5. An example solution ............................................... 9 6. Conclusion ........................................................ 9 Abstract Recently, there has been increasing interest in investigating ways on how to deliver enhancements to existing networking technology for storage solutions. To facilitate exploration of the issues, a Birds of a Feather (BoF) session has been scheduled during the IETF Meetings of March, 2000. The BoF session will be held on March 29, 2000 at 3:30PM in Adelaide, Australia. The intent of the memo is to facilitate discussion and achieve consensus on the best approach and the key issues that need to be investigated before storage over IP becomes a reality. Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 2] IP Storage Issues and Requirements March 2000 1. Introduction and Motivation Recently, the trend toward merging networking and storage interconnect technologies has accelerated. This merging has given rise to new technologies such as Storage Area Networks (SANs) and Network Attached Storage (NAS.) Other technologies, such as Object Based Storage Device (OBSD) are being discussed. Due to the increased usage of the Internet, storage demands will continue to increase. In light of this, it is reasonable to assume that the merging of networking and storage will continue. While file storage protocols like NFS and AFS have long operated over IP-based networks, block storage protocols such as SCSI and ATA have yet to make the leap from non-IP based interconnects to IP-based networks. With the emergence of gigabit and 10-gigabit IP networks, transporting block storage over IP networks is increasingly attractive. While block storage based SAN implementations have applied networking concepts to storage technology, another perspective is to apply storage principles and concepts to existing networking technology. The IP Storage architecture offers a framework within which existing network infrastructures can be used as a high-quality storage subsystem interconnect. The goal is to provide current and future storage requirements while leveraging the broad installed base as well as the inherent manageability, scalability, and availability of the network. 2. Requirements and issues for storage 2.1. Data Integrity and Reliability This is an absolute requirement of storage. Whatever solution is used, it must be shown empirically that it is safe and reliable. 2.2. Ability to span large distances Block storage has generally been relegated to the subnet. As storage demands increase, the need for remote backup and data mirroring for storage consolidation and disaster recovery is becoming increasingly important. The ability to transfer storage over an existing network infrastructure makes IP-based storage very attractive. Another attractive feature of IP Storage is the ability to bridge subnets via a high speed interconnect. While information between subnets are shared today through file transfers over the communications network, Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 3] IP Storage Issues and Requirements March 2000 the ability to directly access data resident in other subnets may be useful in some applications. The bridging of subnets raises the issue of supporting legacy installations. For example, one application of IP Storage could be the bridging of multiple FibreChannel-based SANs. The IP Storage architecture should consider ways that easily adapt to legacy installations. Adequate liaisons need to be in place with other standards bodies such as ANSI T10, T11, and T13 (SCSI, FC, and ATA respectively) to ensure interoperability. In the future, it may be highly desirable for the subnets themselves to be based on IP Storage. However, optimal solutions for the subnet raise a different, though overlapping, set of issues and requirements. Also, the IETF may not be the appropriate venue for optimizing IP Storage for the subnet. In order to keep the issues clear and relevant to the Internet community and the goals reasonably specific, a decision must be made early on as to whether issues related to LANs or SANs are to be addressed. 2.3. Performance In the storage market, speed sells. Projected access times and sequential access rates for disk drives, assuming historical annual improvement rates, will approach 4 milliseconds and 100 megabytes/second respectively within the next 5 years. But, performance requirements vary greatly depending on the application. For applications which access data sequentially, or in very large data blocks, low latencies and high bandwidth will be required. On the other hand, transaction processing applications will still be limited by disk access time to about 350 IOPS per drive, or one to five megabytes a second depending on request size. Furthermore, the caching of data at various points within the system can effectively eliminate the physical limitations of rotating media, resulting in memory to memory speeds. Before performance requirements can be discussed, certain issues need to be addressed. 2.3.1. Performance vs. Connectivity Solution There needs to be an agreement on which problem IP Storage is solving. While performance is always important, connectivity solutions tend to have less severe performance requirements while having a greater emphasis on ease-of-use or solving data access problems. 2.3.2 Competitive with existing architectures Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 4] IP Storage Issues and Requirements March 2000 If IP Storage is to provide storage solutions that are currently provided by other architectures, then IP Storage must be competitive in a cost/performance basis. Since it is assumed that the use of off- the-shelf networking switches, routers, and wiring will produce substantial cost savings due to the large volumes associated with the use of such equipment for networks, the performance requirements need only to meet those of other architectures in typical applications. 2.3.3 CPU Utilization One of the benefits of the IP Storage architecture is the ability to scale servers and storage devices independently. If bandwidth is exceeded, host bus adapters can be added. But, if the CPU is exceeded, then additional servers are required. Traditionally, CPU utilization of storage architectures has been significantly lower than that of networking. Methods for the reduction of network stack processing overhead at the storage clients and servers need to be considered and possibly incorporated into the protocol. 2.3.4 Storage Stack vs. Network Stack Processing In the server, the storage stack has generally been more efficient than the network stack for various reasons. The transport protocol is simpler than that of networking, thus more readily able to be incorporated in hardware. The server acts as the initiator, and rarely (if at all) receives unsolicited commands. Read memory is locked and the host bus adapter is able to transfer data directly into destination memory via a host-supplied scatter/gather list. This direct memory access is simplified by the fact that data is received from the target in order. For IP Storage, it is probably unreasonable to require in order delivery. However, a comparison of the two stacks might be useful in developing ways of increasing the efficiency of the protocol processing. 2.3.5. Flow control at the Storage Protocol Whereas the server, in the typical storage stack, has read memory pre-allocated by the operating system, the storage client has a finite amount of memory to store write data. This requires some level of flow control to avoid overrunning memory resources. For example, parallel SCSI uses disconnects and Fibrechannel uses transfer ready as methods of flow control. The flow control in the transport protocol could handle this situation if each command had a unique session. However, if a session could support multiple concurrent commands (i.e. tagged queuing), then the transport's flow control would effectively pause data flow for all commands, even if there are available resources for some of the commands. It needs to be Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 5] IP Storage Issues and Requirements March 2000 determined whether some flow control is necessary at the storage protocol layer. If so, a control mechanism such as transfer ready may produce undesirable latencies, especially in long distance configurations; and, if possible, a safe mechanism could be found to reduce this latency. 2.4. Storage Management The ability to consolidate storage allows for many heterogeneous servers to share the same storage pool. This also presents new problems of management of that storage pool. By leveraging the inherent manageability of IP as well as the experiences of the Fibrechannel community, virtual LANs can be established using IP- based tools to segment and allocate storage at the port, LUN, and/or partition level. SNMP MIBs and/or CIM object models for storage devices may be necessary. 2.4.1. Sharing Storage vs. Data One advantage that NFS has over block storage protocol is the ability to share files among multiple diverse server environments. Currently, sharing of data using block storage is only done in a clustered environment. Ongoing discussions in the Storage Networking Industry Association (SNIA) and the ANSI committees regarding object-based storage should be followed and appropriate liaisons established. 2.5. Security Assigning IP addresses to storage clients can be alarming to those tasked with ensuring data is protected from unwanted access. Work is being done at the storage protocol to increase the level of access control. It must be noted that these methods for restricting access may assume a protected environment. In other words, the protocol may guard against accidental access, but may not be a sufficient safeguard against malicious access. Leveraging IPsec will provide greater security in more open environments. 2.6. Device Discovery and Configuration Communication networks, which assume that all attached devices are peers and that any given device will want to communicate with only a small subset of other devices. On the other hand, storage networks are hierarchical, with a few Initiators in communication with many targets, and with targets in communication only with initiators, or with other targets under the control of an initiator. This storage viewpoint requires that initiators be able to locate all the targets, and be able to communicate with many or all of them. A decision must Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 6] IP Storage Issues and Requirements March 2000 be made on which method to use for discovery: peer-to-peer or master- slave. Any discovery protocol should be scalable up to systems of hundreds of initiators and thousands of targets. A negotiation protocol also needs to be defined to allow for auto- configuration of devices. Adequate liaisons need to be in place with other working groups in the IETF that are relevant to these areas (e.g. zeroconf, LDAP, Service Location Protocol). 2.7. Quality of Service Investigation should be done on how QoS can enhance storage solutions running over networks. A strong partnership with IPv6 will ensure that storage needs will be considered. 2.8 Storage Subsystem vs. Device Many of the issues discussed can be addressed by the subsystem vendors, but may be prohibitively expensive and/or complex in the highly competitive disk drive market. Assuming the IP Storage architecture is successful, it is highly likely that it will be adopted by the drive manufacturers. A decision needs to be made on whether integration at the drive level needs to be considered at this time, or at some later date (or by some other organization.) 3. Optimum Criteria for Solutions The following are some criteria that, though not necessary, would increase the likelihood of success and aid in the quick adoption of the IP Storage architecture in the marketplace: - Leverage existing technology wherever possible, thus increasing the likelihood of interoperability. - The solution requires little or no modification to existing operating systems, ensuring early deployment. - Existing software tools and applications are compatible, for both networking and storage. This allows for a high degree of application specific testing prior to deployment. - Deployment of the solution requires little or no additional training. - The solution has appeal in multiple segments of the market. Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 7] IP Storage Issues and Requirements March 2000 4. Candidates for Solutions 4.1. Transport 4.1.2. TCP TCP is ubiquitous on the Internet. It has many years of development and enhancements. There is a strong comfort level with IS administrators. It has proven interoperability in a wide range of configurations. TCP has a great deal of support among the networking community. TCP also has the reputation of being large, software intensive, and overly complicated. There are those in the storage community that are skeptical that TCP can provide the required level of performance. In addition, the checksum may not be adequate for ensuring data integrity. A detailed performance study should be done. A key factor is the network stack processing overhead. New development of hardware accelerated TCP could be pivotal. 4.1.3. UDP UDP is used heavily in the LAN environment, but will have to win over skeptics that it can run safely over the Internet/WAN. 4.1.4. Scheduled Transfer (ST) Developed in the HIPPI organization, it provides high bandwidth, low latency, and low CPU utilization. A SCSI encapsulation protocol has already been proposed in ANSI T10. Will have to win over skeptics that it can run safely over the Internet/WAN. 4.1.5. Other? Any other proposed transport protocol should be considered. 4.2 Storage Protocol 4.2.1. SCSI SCSI has two meanings. Logical SCSI refers to the command protocol. Most storage stacks are based on the SCSI Architectural Model (SAM). It is this logical command protocol that would need to be encapsulated and transported over IP. Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 8] IP Storage Issues and Requirements March 2000 SCSI also is equated with devices incorporating the SCSI Parallel Interface (SPI). These drives have the largest deployment in the server market. Since it is a parallel interface, bridging would usually be done at the storage subsystem. Its interlocked protocol is dissimilar in behavior to networking, but the newly approved Information Unit phase may make the conversion simpler. 4.2.2. FCP This is Fibrechannel's version of encapsulated SCSI command protocol. Bridging can be done at either the storage subsystem or router. 4.2.3. ATA There is interest here due to the cost differential between ATA and SCSI drives. A serial version of ATA is also being developed. The storage protocol would still be based on the SCSI model. This interface accounts for approximately 85% of all disk drives sold world-wide. 4.2.4. NFS NFS is the current solution for accessing storage over the network. Its strengths are ease-of-use, file sharing, and security. The general assumption is that a block storage protocol should outperform NFS in some applications, but that assumption needs to be tested. Also, some applications (e.g. tape library) could benefit from a block storage protocol. In order for a block-based solution to be accepted, clear benefits over NFS (e.g. performance, direct access) must be proven. Even then, NFS would still be the solution of choice for those requiring ease-of-use and file sharing. 4.2.5. Other? There may be other mappings of SCSI over serial interconnects. A more general approach could also be explored. A class-based (disk, tape, enclosure, etc.) message could be transported to the storage client where it was then translated to the native interface(SCSI, FCP, etc.) 5. An example solution A proposed solution which transports SCSI over TCP is available as an internet draft under the name 6. Conclusion Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 9] IP Storage Issues and Requirements March 2000 The critical success factors are: - Data integrity and reliability are empirically demonstrated. - Issues of security are satisfactorily addressed. - Any block storage protocol must show clear differentiation from NFS. Expires 10 September 2000 IP Storage Issues and Requirements March 2000 Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 5] Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 1] From XXX Tue Mar 14 13:42:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Mar 14 00:53:01 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id AAA27188 for ; Tue, 14 Mar 2000 00:53:00 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA09734 for ips-outgoing; Tue, 14 Mar 2000 00:50:43 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA09728 for ; Tue, 14 Mar 2000 00:50:38 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id VAA20731; Mon, 13 Mar 2000 21:50:07 -0800 (PST) Received: from aimexc03.corp.adaptec.com (aimexc03.corp.adaptec.com [162.62.190.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id VAA18018; Mon, 13 Mar 2000 21:50:06 -0800 (PST) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Mon, 13 Mar 2000 21:50:06 -0800 Message-ID: <5B0C98C89640D311A319002048406A1C03FBE577@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: FW: IPS Issues document Date: Mon, 13 Mar 2000 21:50:05 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I took the liberty of forwarding your comments to the reflector. To get the discussion going... Paul von Stamwitz -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Sunday, March 12, 2000 10:43 AM To: Dave_Lee@3com.com Cc: Bradley, Mark; VonStamwitz, Paul; Wilson, Andrew; 'Costa Sapuntzakis'; bassoon@ece.cmu.edu; alan@orca.com; harwood_jack@emc.com Subject: RE: IPS Issues document Hi, Good job. I had only an hour to glance over it so I might come later with some more but here are some observations: Performance - has two aspects - related but not entirely equivalent - latency and bandwidth. It is obvious that bandwidth is not a differentiating factor for any type of interconnect as they use the same physical infrastructure in a similar fashion. Latency on the other hand is a more difficult factor as it relates to hardware, software and protocol. CPU utilization by itself is arguably not a limiting factor - as it can be solved by simply adding more CPUs or waiting until the CPUs get faster. All the elements you mention under CPU utilization refer mostly to the memory subsystem utilization - and there are no known ways to improve on that as the memory does not get faster as fast as the CPUs do (no Moore law for DRAMs!). Protocol stack processing reduction is the mainly improving your "placement" techniques and reducing copy operations. Storage subsystems fare better (and so does FCP) since the do scatter-gather DMA (and DMA is times faster than copy due to an architectural "feature" of DRAMs that use one address setup for several memory operations. This is the reason why we think that giving the protocols a chance to do well DMA is so important Adoption of the solution by the device vendors is important but not critical. Adoption of a "last half-meter" protocol - like the serial-ATA pushed by Intel, with all its drawbacks could offer a cheap alternative for fiercely competitive environments like desktops with SCSI as a more robust and functional cousin. For clusters and RAIDs both can be considered and with IP getting adopted by pervasive computing may get IP networks to the point at which the price differential will not exist anymore You do not mention the SBP although, I recall it as the first serial SCSI standardized On the performance section I think that solution scalability for the low-end to the high-end is essential as it is the only way to gain "mass" and survive the continuous shift in performance Management should include both storage and interconnect. Any solution should be able to use for interconnect management whatever is good for network management with only slight concern about the device specifics. A factor that I have difficulty expressing is - skills and tools availability. By this I mean planers, administrators and (why not!) even salesman and their tools. Any solution that we leverage an existing skill (and tools) base will have a distinct advantage Regards, Julo Julian Satran - IBM Research at Haifa From XXX Tue Mar 14 13:42:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Mar 14 00:53:02 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id AAA27192 for ; Tue, 14 Mar 2000 00:53:01 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA09776 for ips-outgoing; Tue, 14 Mar 2000 00:52:16 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA09767; Tue, 14 Mar 2000 00:52:10 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id VAA20775; Mon, 13 Mar 2000 21:51:29 -0800 (PST) Received: from aimexc03.corp.adaptec.com (aimexc03.corp.adaptec.com [162.62.190.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id VAA18070; Mon, 13 Mar 2000 21:51:28 -0800 (PST) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Mon, 13 Mar 2000 21:51:28 -0800 Message-ID: <00C2F5C6AE05D311BD3700204840376703A10E01@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: "'ips@ece.cmu.edu'" , "'Scott Bradner'" , "'bassoon@ece.cmu.edu'" Subject: IPS Issues/Requirements Document Date: Mon, 13 Mar 2000 21:51:27 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Appended is a first draft of the IPS Issues/Requirements document. It was submitted to the IETF as an Internet Draft on March 10, 2:42 PST Unfortunately, it missed the cutoff time of 5:00 EST by 42 minutes, so it was NOT processed. All comments and suggestions for improvements all welcome by responding to the IPS list, or to me directly. The timeline for the final draft is March 20, so the more feedback I can have this week, the better. Dave Nagle, If you feel that it is appropriate, please cross post it to other storage groups (SNIA, NSIC, T10, etc.) to get them involved and post it to the web site. Thank you, Paul von Stamwitz ----------------------------- Internet Draft P. von Stamwitz D. Wilson Expires 10 September 2000 Adaptec C. Sapuntzakis Cisco Systems D. Lee 3Com March 2000 IP Storage Issues and Requirements This document is an Internet-Draft and is NOT offered in accordance with Section 10 of RFC2026, and the author does not provide the IETF with any rights other than to publish as an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Status of this Memo This document specifies issues and requirements for the IP Storage BOF and any future working group. It is a first draft and has not been reviewed by the Internet community. A subsequent revision is likely after discussion and suggestions for improvements have been submitted. The next revision is planned to be submitted by March 22, 2000. Please submit all comments to the ips@ece.cmu.edu or paulv@corp.adaptec.com. Distribution of this memo is unlimited. Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 1] IP Storage Issues and Requirements March 2000 Table of Contents Abstract ............................................................. 2 1. Introduction and Motivation ....................................... 3 2. Requirements and issues for storage ............................... 3 2.1. Data Integrity and Reliability ................................. 3 2.2. Ability to span large distances ................................ 3 2.3. Performance .................................................... 4 2.3.1. Performance vs. Connectivity Solution ....................... 4 2.3.2 Competitive with existing architectures ...................... 4 2.3.3 CPU Utilization .............................................. 5 2.3.4 Storage Stack vs. Network Stack Processing ................... 5 2.3.5. Flow control at the Storage Protocol ........................ 5 2.4. Storage Management ............................................. 6 2.4.1. Sharing Storage vs. Data .................................... 6 2.5. Security ....................................................... 6 2.6. Device Discovery and Configuration ............................. 6 3. Optimum Criteria for Solutions .................................... 7 4. Candidates for Solutions .......................................... 8 4.1. Transport ...................................................... 8 4.1.1. TCP ......................................................... 8 4.1.2. UDP ......................................................... 8 4.1.3. Scheduled Transfer (ST) ..................................... 8 4.1.4. Other? ...................................................... 8 4.2 Storage Protocol ................................................ 8 4.2.1. SCSI ........................................................ 8 4.2.2. FCP ......................................................... 9 4.2.3. ATA ......................................................... 9 4.2.4. NFS ......................................................... 9 4.2.5. Other? ...................................................... 9 5. An example solution ............................................... 9 6. Conclusion ........................................................ 9 Abstract Recently, there has been increasing interest in investigating ways on how to deliver enhancements to existing networking technology for storage solutions. To facilitate exploration of the issues, a Birds of a Feather (BoF) session has been scheduled during the IETF Meetings of March, 2000. The BoF session will be held on March 29, 2000 at 3:30PM in Adelaide, Australia. The intent of the memo is to facilitate discussion and achieve consensus on the best approach and the key issues that need to be investigated before storage over IP becomes a reality. Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 2] IP Storage Issues and Requirements March 2000 1. Introduction and Motivation Recently, the trend toward merging networking and storage interconnect technologies has accelerated. This merging has given rise to new technologies such as Storage Area Networks (SANs) and Network Attached Storage (NAS.) Other technologies, such as Object Based Storage Device (OBSD) are being discussed. Due to the increased usage of the Internet, storage demands will continue to increase. In light of this, it is reasonable to assume that the merging of networking and storage will continue. While file storage protocols like NFS and AFS have long operated over IP-based networks, block storage protocols such as SCSI and ATA have yet to make the leap from non-IP based interconnects to IP-based networks. With the emergence of gigabit and 10-gigabit IP networks, transporting block storage over IP networks is increasingly attractive. While block storage based SAN implementations have applied networking concepts to storage technology, another perspective is to apply storage principles and concepts to existing networking technology. The IP Storage architecture offers a framework within which existing network infrastructures can be used as a high-quality storage subsystem interconnect. The goal is to provide current and future storage requirements while leveraging the broad installed base as well as the inherent manageability, scalability, and availability of the network. 2. Requirements and issues for storage 2.1. Data Integrity and Reliability This is an absolute requirement of storage. Whatever solution is used, it must be shown empirically that it is safe and reliable. 2.2. Ability to span large distances Block storage has generally been relegated to the subnet. As storage demands increase, the need for remote backup and data mirroring for storage consolidation and disaster recovery is becoming increasingly important. The ability to transfer storage over an existing network infrastructure makes IP-based storage very attractive. Another attractive feature of IP Storage is the ability to bridge subnets via a high speed interconnect. While information between subnets are shared today through file transfers over the communications network, Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 3] IP Storage Issues and Requirements March 2000 the ability to directly access data resident in other subnets may be useful in some applications. The bridging of subnets raises the issue of supporting legacy installations. For example, one application of IP Storage could be the bridging of multiple FibreChannel-based SANs. The IP Storage architecture should consider ways that easily adapt to legacy installations. Adequate liaisons need to be in place with other standards bodies such as ANSI T10, T11, and T13 (SCSI, FC, and ATA respectively) to ensure interoperability. In the future, it may be highly desirable for the subnets themselves to be based on IP Storage. However, optimal solutions for the subnet raise a different, though overlapping, set of issues and requirements. Also, the IETF may not be the appropriate venue for optimizing IP Storage for the subnet. In order to keep the issues clear and relevant to the Internet community and the goals reasonably specific, a decision must be made early on as to whether issues related to LANs or SANs are to be addressed. 2.3. Performance In the storage market, speed sells. Projected access times and sequential access rates for disk drives, assuming historical annual improvement rates, will approach 4 milliseconds and 100 megabytes/second respectively within the next 5 years. But, performance requirements vary greatly depending on the application. For applications which access data sequentially, or in very large data blocks, low latencies and high bandwidth will be required. On the other hand, transaction processing applications will still be limited by disk access time to about 350 IOPS per drive, or one to five megabytes a second depending on request size. Furthermore, the caching of data at various points within the system can effectively eliminate the physical limitations of rotating media, resulting in memory to memory speeds. Before performance requirements can be discussed, certain issues need to be addressed. 2.3.1. Performance vs. Connectivity Solution There needs to be an agreement on which problem IP Storage is solving. While performance is always important, connectivity solutions tend to have less severe performance requirements while having a greater emphasis on ease-of-use or solving data access problems. 2.3.2 Competitive with existing architectures Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 4] IP Storage Issues and Requirements March 2000 If IP Storage is to provide storage solutions that are currently provided by other architectures, then IP Storage must be competitive in a cost/performance basis. Since it is assumed that the use of off- the-shelf networking switches, routers, and wiring will produce substantial cost savings due to the large volumes associated with the use of such equipment for networks, the performance requirements need only to meet those of other architectures in typical applications. 2.3.3 CPU Utilization One of the benefits of the IP Storage architecture is the ability to scale servers and storage devices independently. If bandwidth is exceeded, host bus adapters can be added. But, if the CPU is exceeded, then additional servers are required. Traditionally, CPU utilization of storage architectures has been significantly lower than that of networking. Methods for the reduction of network stack processing overhead at the storage clients and servers need to be considered and possibly incorporated into the protocol. 2.3.4 Storage Stack vs. Network Stack Processing In the server, the storage stack has generally been more efficient than the network stack for various reasons. The transport protocol is simpler than that of networking, thus more readily able to be incorporated in hardware. The server acts as the initiator, and rarely (if at all) receives unsolicited commands. Read memory is locked and the host bus adapter is able to transfer data directly into destination memory via a host-supplied scatter/gather list. This direct memory access is simplified by the fact that data is received from the target in order. For IP Storage, it is probably unreasonable to require in order delivery. However, a comparison of the two stacks might be useful in developing ways of increasing the efficiency of the protocol processing. 2.3.5. Flow control at the Storage Protocol Whereas the server, in the typical storage stack, has read memory pre-allocated by the operating system, the storage client has a finite amount of memory to store write data. This requires some level of flow control to avoid overrunning memory resources. For example, parallel SCSI uses disconnects and Fibrechannel uses transfer ready as methods of flow control. The flow control in the transport protocol could handle this situation if each command had a unique session. However, if a session could support multiple concurrent commands (i.e. tagged queuing), then the transport's flow control would effectively pause data flow for all commands, even if there are available resources for some of the commands. It needs to be Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 5] IP Storage Issues and Requirements March 2000 determined whether some flow control is necessary at the storage protocol layer. If so, a control mechanism such as transfer ready may produce undesirable latencies, especially in long distance configurations; and, if possible, a safe mechanism could be found to reduce this latency. 2.4. Storage Management The ability to consolidate storage allows for many heterogeneous servers to share the same storage pool. This also presents new problems of management of that storage pool. By leveraging the inherent manageability of IP as well as the experiences of the Fibrechannel community, virtual LANs can be established using IP- based tools to segment and allocate storage at the port, LUN, and/or partition level. SNMP MIBs and/or CIM object models for storage devices may be necessary. 2.4.1. Sharing Storage vs. Data One advantage that NFS has over block storage protocol is the ability to share files among multiple diverse server environments. Currently, sharing of data using block storage is only done in a clustered environment. Ongoing discussions in the Storage Networking Industry Association (SNIA) and the ANSI committees regarding object-based storage should be followed and appropriate liaisons established. 2.5. Security Assigning IP addresses to storage clients can be alarming to those tasked with ensuring data is protected from unwanted access. Work is being done at the storage protocol to increase the level of access control. It must be noted that these methods for restricting access may assume a protected environment. In other words, the protocol may guard against accidental access, but may not be a sufficient safeguard against malicious access. Leveraging IPsec will provide greater security in more open environments. 2.6. Device Discovery and Configuration Communication networks, which assume that all attached devices are peers and that any given device will want to communicate with only a small subset of other devices. On the other hand, storage networks are hierarchical, with a few Initiators in communication with many targets, and with targets in communication only with initiators, or with other targets under the control of an initiator. This storage viewpoint requires that initiators be able to locate all the targets, and be able to communicate with many or all of them. A decision must Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 6] IP Storage Issues and Requirements March 2000 be made on which method to use for discovery: peer-to-peer or master- slave. Any discovery protocol should be scalable up to systems of hundreds of initiators and thousands of targets. A negotiation protocol also needs to be defined to allow for auto- configuration of devices. Adequate liaisons need to be in place with other working groups in the IETF that are relevant to these areas (e.g. zeroconf, LDAP, Service Location Protocol). 2.7. Quality of Service Investigation should be done on how QoS can enhance storage solutions running over networks. A strong partnership with IPv6 will ensure that storage needs will be considered. 2.8 Storage Subsystem vs. Device Many of the issues discussed can be addressed by the subsystem vendors, but may be prohibitively expensive and/or complex in the highly competitive disk drive market. Assuming the IP Storage architecture is successful, it is highly likely that it will be adopted by the drive manufacturers. A decision needs to be made on whether integration at the drive level needs to be considered at this time, or at some later date (or by some other organization.) 3. Optimum Criteria for Solutions The following are some criteria that, though not necessary, would increase the likelihood of success and aid in the quick adoption of the IP Storage architecture in the marketplace: - Leverage existing technology wherever possible, thus increasing the likelihood of interoperability. - The solution requires little or no modification to existing operating systems, ensuring early deployment. - Existing software tools and applications are compatible, for both networking and storage. This allows for a high degree of application specific testing prior to deployment. - Deployment of the solution requires little or no additional training. - The solution has appeal in multiple segments of the market. Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 7] IP Storage Issues and Requirements March 2000 4. Candidates for Solutions 4.1. Transport 4.1.2. TCP TCP is ubiquitous on the Internet. It has many years of development and enhancements. There is a strong comfort level with IS administrators. It has proven interoperability in a wide range of configurations. TCP has a great deal of support among the networking community. TCP also has the reputation of being large, software intensive, and overly complicated. There are those in the storage community that are skeptical that TCP can provide the required level of performance. In addition, the checksum may not be adequate for ensuring data integrity. A detailed performance study should be done. A key factor is the network stack processing overhead. New development of hardware accelerated TCP could be pivotal. 4.1.3. UDP UDP is used heavily in the LAN environment, but will have to win over skeptics that it can run safely over the Internet/WAN. 4.1.4. Scheduled Transfer (ST) Developed in the HIPPI organization, it provides high bandwidth, low latency, and low CPU utilization. A SCSI encapsulation protocol has already been proposed in ANSI T10. Will have to win over skeptics that it can run safely over the Internet/WAN. 4.1.5. Other? Any other proposed transport protocol should be considered. 4.2 Storage Protocol 4.2.1. SCSI SCSI has two meanings. Logical SCSI refers to the command protocol. Most storage stacks are based on the SCSI Architectural Model (SAM). It is this logical command protocol that would need to be encapsulated and transported over IP. Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 8] IP Storage Issues and Requirements March 2000 SCSI also is equated with devices incorporating the SCSI Parallel Interface (SPI). These drives have the largest deployment in the server market. Since it is a parallel interface, bridging would usually be done at the storage subsystem. Its interlocked protocol is dissimilar in behavior to networking, but the newly approved Information Unit phase may make the conversion simpler. 4.2.2. FCP This is Fibrechannel's version of encapsulated SCSI command protocol. Bridging can be done at either the storage subsystem or router. 4.2.3. ATA There is interest here due to the cost differential between ATA and SCSI drives. A serial version of ATA is also being developed. The storage protocol would still be based on the SCSI model. This interface accounts for approximately 85% of all disk drives sold world-wide. 4.2.4. NFS NFS is the current solution for accessing storage over the network. Its strengths are ease-of-use, file sharing, and security. The general assumption is that a block storage protocol should outperform NFS in some applications, but that assumption needs to be tested. Also, some applications (e.g. tape library) could benefit from a block storage protocol. In order for a block-based solution to be accepted, clear benefits over NFS (e.g. performance, direct access) must be proven. Even then, NFS would still be the solution of choice for those requiring ease-of-use and file sharing. 4.2.5. Other? There may be other mappings of SCSI over serial interconnects. A more general approach could also be explored. A class-based (disk, tape, enclosure, etc.) message could be transported to the storage client where it was then translated to the native interface(SCSI, FCP, etc.) 5. An example solution A proposed solution which transports SCSI over TCP is available as an internet draft under the name 6. Conclusion Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 9] IP Storage Issues and Requirements March 2000 The critical success factors are: - Data integrity and reliability are empirically demonstrated. - Issues of security are satisfactorily addressed. - Any block storage protocol must show clear differentiation from NFS. Expires 10 September 2000 IP Storage Issues and Requirements March 2000 Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 5] Von Stamwitz, Wilson, Sapuntzakis, Lee [Page 1] From XXX Tue Mar 14 19:12:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Mar 14 13:48:22 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA31725 for ; Tue, 14 Mar 2000 13:48:20 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06259 for ips-outgoing; Tue, 14 Mar 2000 13:47:27 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pneumatic-tube.sgi.com (pneumatic-tube.sgi.com [204.94.214.22]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06239 for ; Tue, 14 Mar 2000 13:47:21 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id KAA06013; Tue, 14 Mar 2000 10:50:40 -0800 (PST) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (clock.engr.sgi.com [163.154.34.45]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id KAA01654; Tue, 14 Mar 2000 10:47:15 -0800 (PST) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (localhost [127.0.0.1]) by clock.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id KAA70632; Tue, 14 Mar 2000 10:53:42 -0800 (PST) Message-Id: <200003141853.KAA70632@clock.engr.sgi.com> X-Mailer: exmh version 2.0.2 2/24/98 To: "Bradley, Mark" Cc: ips@ece.cmu.edu Subject: Re: IPS Issues document From: Zachary Amsden In-Reply-To: Your message of "Tue, 14 Mar 2000 10:18:12 MST." <3CFAB2A312A9D311BB300000D11ABE920D30E1@btcexc01.btc.adaptec.com> Date: Tue, 14 Mar 2000 10:53:41 -0800 Sender: owner-ips@ece.cmu.edu Precedence: bulk > Julo, good comments. My observation is that faster CPU's > don't solve the problem. We have been getting faster CPU's > since the beginning of time and the only semi-constant is > that we find that the CPU continues to get utilized. Large > MP machines are bought because the processing horsepower > is needed for some appl.(s) that consumes it. Therefore they > don't have 'extra' cycles to spare. Most generic OS's > don't support dedicating a processor to specific jobs, > either. Putting a small processor someplace (an IOP) else > typically doesn't scale as system processing speed increases, > so those often become bottle-necks after some time. Remember > those 186-based networking cards? > > Bottom line is that % CPU utilization has been, is, and will > continue to be an important consideration for some time. > It seems we never have enough CPU, memory or disk. We seem > always to find ways to use it up. (I remember years ago > thinking "Wow! A megabyte on a single platter! We'll have > enough disk space to last 3 or 4 years!!!" I also recall > thinking that a 68020 made for a pretty fast workstation. > Now I complain that my dual 450 MHz PC with 128 MB of memory > is not fast enough to be a decent office machine next year... > :{) ) I agree completely with that - protocol processing and interrupt time alone for a 1 Gbps connection can easily saturate a single CPU, even with interrupt coalescing and many other tricks. When you move to a 10 Gbps connection, you can't just add more CPUs to deal with the problem, because of both cost and the fact that you don't get a perfect speedup as you add more CPUs. This makes techniques that reduce CPU and bus utilization more valuable as network speeds increase. -- Zachary Amsden zamsden@engr.sgi.com 3-6919 31-2-510 Core Protocols From XXX Tue Mar 14 19:12:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Mar 14 12:18:11 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA04747 for ; Tue, 14 Mar 2000 12:18:10 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA02897 for ips-outgoing; Tue, 14 Mar 2000 12:17:18 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA02891 for ; Tue, 14 Mar 2000 12:17:14 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id JAA29163 for ; Tue, 14 Mar 2000 09:15:57 -0800 (PST) Received: from btcexc01.btc.adaptec.com (btcexc01.btc.adaptec.com [162.62.147.10]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id JAA01600 for ; Tue, 14 Mar 2000 09:15:57 -0800 (PST) Received: by btcexc01.btc.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Tue, 14 Mar 2000 10:18:13 -0700 Message-ID: <3CFAB2A312A9D311BB300000D11ABE920D30E1@btcexc01.btc.adaptec.com> From: "Bradley, Mark" To: ips@ece.cmu.edu Subject: RE: IPS Issues document Date: Tue, 14 Mar 2000 10:18:12 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, good comments. My observation is that faster CPU's don't solve the problem. We have been getting faster CPU's since the beginning of time and the only semi-constant is that we find that the CPU continues to get utilized. Large MP machines are bought because the processing horsepower is needed for some appl.(s) that consumes it. Therefore they don't have 'extra' cycles to spare. Most generic OS's don't support dedicating a processor to specific jobs, either. Putting a small processor someplace (an IOP) else typically doesn't scale as system processing speed increases, so those often become bottle-necks after some time. Remember those 186-based networking cards? Bottom line is that % CPU utilization has been, is, and will continue to be an important consideration for some time. It seems we never have enough CPU, memory or disk. We seem always to find ways to use it up. (I remember years ago thinking "Wow! A megabyte on a single platter! We'll have enough disk space to last 3 or 4 years!!!" I also recall thinking that a 68020 made for a pretty fast workstation. Now I complain that my dual 450 MHz PC with 128 MB of memory is not fast enough to be a decent office machine next year... :{) ) -- markb > -----Original Message----- > From: VonStamwitz, Paul [mailto:paulv@corp.adaptec.com] > Sent: Monday, March 13, 2000 10:50 PM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: FW: IPS Issues document > > > Julian, > > I took the liberty of forwarding your comments to the reflector. > > To get the discussion going... > > Paul von Stamwitz > > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Sunday, March 12, 2000 10:43 AM > To: Dave_Lee@3com.com > Cc: Bradley, Mark; VonStamwitz, Paul; Wilson, Andrew; 'Costa > Sapuntzakis'; bassoon@ece.cmu.edu; alan@orca.com; harwood_jack@emc.com > Subject: RE: IPS Issues document > > > > > Hi, > > Good job. I had only an hour to glance over it so I might > come later with > some more but here are some observations: > > Performance - has two aspects - related but not entirely > equivalent - > latency and bandwidth. It is obvious that bandwidth is not a > differentiating factor for any type of interconnect as > they use the same > physical infrastructure in a similar fashion. Latency on > the other hand > is a more difficult factor as it relates to hardware, software and > protocol. CPU utilization by itself is arguably not a > limiting factor - > as it can be solved by simply adding more CPUs or waiting > until the CPUs > get faster. All the elements you mention under CPU > utilization refer > mostly to the memory subsystem utilization - and there are > no known ways > to improve on that as the memory does not get faster as > fast as the > CPUs do (no Moore law for DRAMs!). Protocol stack > processing reduction > is the mainly improving your "placement" techniques and > reducing copy > operations. Storage subsystems fare better (and so does > FCP) since the > do scatter-gather DMA (and DMA is times faster than copy due to an > architectural "feature" of DRAMs that use one address > setup for several > memory operations. This is the reason why we think that giving the > protocols a chance to do well DMA is so important > Adoption of the solution by the device vendors is important but not > critical. Adoption of a "last half-meter" protocol - like > the serial-ATA > pushed by Intel, with all its drawbacks could offer a > cheap alternative > for fiercely competitive environments like desktops with > SCSI as a more > robust and functional cousin. For clusters and RAIDs both can be > considered and with IP getting adopted by pervasive > computing may get IP > networks to the point at which the price differential will > not exist > anymore > You do not mention the SBP although, I recall it as the > first serial > SCSI standardized > On the performance section I think that solution > scalability for the > low-end to the high-end is essential as it is the only way to gain > "mass" and survive the continuous shift in performance > Management should include both storage and interconnect. > Any solution > should be able to use for interconnect management whatever > is good for > network management with only slight concern about the > device specifics. > A factor that I have difficulty expressing is - skills and tools > availability. By this I mean planers, administrators and > (why not!) even > salesman and their tools. Any solution that we leverage an existing > skill (and tools) base will have a distinct advantage > > Regards, > Julo > > Julian Satran - IBM Research at Haifa > From XXX Wed Mar 15 00:55:49 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Mar 14 15:46:43 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id PAA18293 for ; Tue, 14 Mar 2000 15:46:42 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA11508 for ips-outgoing; Tue, 14 Mar 2000 15:46:31 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail-out2.apple.com (mail-out2.apple.com [17.254.0.51]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA11501 for ; Tue, 14 Mar 2000 15:46:26 -0500 (EST) Received: from mailgate2.apple.com (A17-129-100-225.apple.com [17.129.100.225]) by mail-out2.apple.com (8.9.3/8.9.3) with ESMTP id MAA13123 for ; Tue, 14 Mar 2000 12:46:25 -0800 (PST) Received: from scv2.apple.com (scv2.apple.com) by mailgate2.apple.com (Content Technologies SMTPRS 2.0.15) with ESMTP id for ; Tue, 14 Mar 2000 12:43:46 -0800 Received: from [17.219.108.18] (rara1ppp10.apple.com [17.219.108.18]) by scv2.apple.com (8.9.3/8.9.3) with SMTP id MAA06776 for ; Tue, 14 Mar 2000 12:43:45 -0800 (PST) Message-Id: <200003142043.MAA06776@scv2.apple.com> Subject: Re: IPS Issues document Date: Tue, 14 Mar 2000 12:43:46 -0800 x-sender: russb@mail.apple.com x-mailer: Claris Emailer 2.0v3, January 22, 1998 From: Russell Berkoff To: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, Sorry, if I may have missed something here. Are we saying that using the CPU 90% of available cpu cycles to manage a communication device is OK, and that applying more CPU power is the answer? I really have to question the scalability of any protocol (network or otherwise) that requires the CPU manage 40-80K packets/second. Someone please show me a solution that offloads some of this stuff to hardware! Regards, Russell Berkoff Apple Computer russb@apple.com From XXX Wed Mar 15 08:56:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Mar 15 01:15:47 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id BAA07672 for ; Wed, 15 Mar 2000 01:15:47 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA28115 for ips-outgoing; Wed, 15 Mar 2000 01:14:33 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA28109 for ; Wed, 15 Mar 2000 01:14:28 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA91942; Wed, 15 Mar 2000 07:13:56 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id HAA151256; Wed, 15 Mar 2000 07:13:53 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568A3.00223943 ; Wed, 15 Mar 2000 07:13:48 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Zachary Amsden cc: "Bradley, Mark" , ips@ece.cmu.edu Message-ID: Date: Wed, 15 Mar 2000 08:13:36 +0200 Subject: Re: IPS Issues document Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I would like to clarify my comment. I am not contesting the fact that CPU will get saturated. My concern is that CPU load decrease does not justify specialized silicon for protocol processing (as it gets soon out-dated like the 186 NICS) and you have the nightmare of supporting 2 protocol stacks. Memory load on the other hand can't be solved any other way than by off-loading and reducing copies to a minimum. Regards, Julo Zachary Amsden on 14/03/2000 20:53:41 Please respond to Zachary Amsden To: "Bradley, Mark" cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: IPS Issues document > Julo, good comments. My observation is that faster CPU's > don't solve the problem. We have been getting faster CPU's > since the beginning of time and the only semi-constant is > that we find that the CPU continues to get utilized. Large > MP machines are bought because the processing horsepower > is needed for some appl.(s) that consumes it. Therefore they > don't have 'extra' cycles to spare. Most generic OS's > don't support dedicating a processor to specific jobs, > either. Putting a small processor someplace (an IOP) else > typically doesn't scale as system processing speed increases, > so those often become bottle-necks after some time. Remember > those 186-based networking cards? > > Bottom line is that % CPU utilization has been, is, and will > continue to be an important consideration for some time. > It seems we never have enough CPU, memory or disk. We seem > always to find ways to use it up. (I remember years ago > thinking "Wow! A megabyte on a single platter! We'll have > enough disk space to last 3 or 4 years!!!" I also recall > thinking that a 68020 made for a pretty fast workstation. > Now I complain that my dual 450 MHz PC with 128 MB of memory > is not fast enough to be a decent office machine next year... > :{) ) I agree completely with that - protocol processing and interrupt time alone for a 1 Gbps connection can easily saturate a single CPU, even with interrupt coalescing and many other tricks. When you move to a 10 Gbps connection, you can't just add more CPUs to deal with the problem, because of both cost and the fact that you don't get a perfect speedup as you add more CPUs. This makes techniques that reduce CPU and bus utilization more valuable as network speeds increase. -- Zachary Amsden zamsden@engr.sgi.com 3-6919 31-2-510 Core Protocols From XXX Wed Mar 15 15:06:00 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Mar 15 09:57:15 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id JAA18584 for ; Wed, 15 Mar 2000 09:57:14 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA17354 for ips-outgoing; Wed, 15 Mar 2000 09:56:51 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA17350 for ; Wed, 15 Mar 2000 09:56:47 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id PAA17840 for ; Wed, 15 Mar 2000 15:56:14 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id PAA173666 for ; Wed, 15 Mar 2000 15:56:13 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568A3.00520B48 ; Wed, 15 Mar 2000 15:56:08 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: Russell Berkoff cc: ips@ece.cmu.edu Message-ID: Date: Wed, 15 Mar 2000 16:55:52 +0200 Subject: Re: IPS Issues document Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk This can be certainly done (and has to) but not because of the CPU load. If it where only for CPU load reasons you would want to consider all other engineering elements (there is a thing out there called SMP or MP!). Regards, Julo Russell Berkoff on 14/03/2000 22:43:46 Please respond to Russell Berkoff To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: IPS Issues document Hi, Sorry, if I may have missed something here. Are we saying that using the CPU 90% of available cpu cycles to manage a communication device is OK, and that applying more CPU power is the answer? I really have to question the scalability of any protocol (network or otherwise) that requires the CPU manage 40-80K packets/second. Someone please show me a solution that offloads some of this stuff to hardware! Regards, Russell Berkoff Apple Computer russb@apple.com From XXX Wed Mar 15 19:35:56 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Mar 15 14:32:07 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id OAA12800 for ; Wed, 15 Mar 2000 14:32:07 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA28600 for ips-outgoing; Wed, 15 Mar 2000 14:30:32 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA28592 for ; Wed, 15 Mar 2000 14:30:27 -0500 (EST) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id LAA05779; Wed, 15 Mar 2000 11:28:36 -0800 (PST) Received: from btcexc01.btc.adaptec.com (btcexc01.btc.adaptec.com [162.62.147.10]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id LAA12456; Wed, 15 Mar 2000 11:28:35 -0800 (PST) Received: by btcexc01.btc.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Wed, 15 Mar 2000 12:30:53 -0700 Message-ID: <3CFAB2A312A9D311BB300000D11ABE924334C4@btcexc01.btc.adaptec.com> From: "Bradley, Mark" To: "'julian_satran@il.ibm.com'" , Zachary Amsden Cc: "Bradley, Mark" , ips@ece.cmu.edu Subject: RE: IPS Issues document Date: Wed, 15 Mar 2000 12:30:52 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is sort of an interesting thread. It sound like the goals covered so far are: 1)Reduce CPU utilization 2)Minimize copies 3)Acheive low latency 4)Avoid hardware obsolescence/bottlenecking (...along with, as previously stated of course, using IP as the transport). What approaches can meet the above? BTW, this sounds like a lot of what was in the Framework draft that was sent out. -- markb > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, March 14, 2000 11:14 PM > To: Zachary Amsden > Cc: Bradley, Mark; ips@ece.cmu.edu > Subject: Re: IPS Issues document > > > > > I would like to clarify my comment. I am not contesting the > fact that CPU > will get saturated. > My concern is that CPU load decrease does not justify > specialized silicon > for protocol processing > (as it gets soon out-dated like the 186 NICS) and you have > the nightmare of > supporting 2 protocol stacks. Memory load on the other hand can't be > solved any other way than by off-loading and reducing copies > to a minimum. > > Regards, > Julo > > Zachary Amsden on 14/03/2000 20:53:41 > > Please respond to Zachary Amsden > > To: "Bradley, Mark" > cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) > Subject: Re: IPS Issues document > > > > > > Julo, good comments. My observation is that faster CPU's > > don't solve the problem. We have been getting faster CPU's > > since the beginning of time and the only semi-constant is > > that we find that the CPU continues to get utilized. Large > > MP machines are bought because the processing horsepower > > is needed for some appl.(s) that consumes it. Therefore they > > don't have 'extra' cycles to spare. Most generic OS's > > don't support dedicating a processor to specific jobs, > > either. Putting a small processor someplace (an IOP) else > > typically doesn't scale as system processing speed increases, > > so those often become bottle-necks after some time. Remember > > those 186-based networking cards? > > > > Bottom line is that % CPU utilization has been, is, and will > > continue to be an important consideration for some time. > > It seems we never have enough CPU, memory or disk. We seem > > always to find ways to use it up. (I remember years ago > > thinking "Wow! A megabyte on a single platter! We'll have > > enough disk space to last 3 or 4 years!!!" I also recall > > thinking that a 68020 made for a pretty fast workstation. > > Now I complain that my dual 450 MHz PC with 128 MB of memory > > is not fast enough to be a decent office machine next year... > > :{) ) > > I agree completely with that - protocol processing and > interrupt time alone > for a 1 Gbps connection can easily saturate a single CPU, even with > interrupt > coalescing and many other tricks. When you move to a 10 Gbps > connection, > you > can't just add more CPUs to deal with the problem, because of > both cost and > the fact that you don't get a perfect speedup as you add more > CPUs. This > makes techniques that reduce CPU and bus utilization more valuable as > network > speeds increase. > > -- > Zachary Amsden zamsden@engr.sgi.com 3-6919 31-2-510 Core Protocols > > > > > From XXX Wed Mar 15 19:35:56 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Mar 15 13:45:28 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA20507 for ; Wed, 15 Mar 2000 13:45:27 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA26550 for ips-outgoing; Wed, 15 Mar 2000 13:44:54 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pneumatic-tube.sgi.com (pneumatic-tube.sgi.com [204.94.214.22]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA26542 for ; Wed, 15 Mar 2000 13:44:48 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id KAA08126; Wed, 15 Mar 2000 10:48:10 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id KAA14884; Wed, 15 Mar 2000 10:44:44 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id KAA44418; Wed, 15 Mar 2000 10:44:43 -0800 (PST) Message-ID: <38CFDA1B.B53AC4FF@engr.sgi.com> Date: Wed, 15 Mar 2000 10:44:43 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: Zachary Amsden CC: "Bradley, Mark" , ips@ece.cmu.edu Subject: Re: IPS Issues document References: <200003141853.KAA70632@clock.engr.sgi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > I agree completely with that - protocol processing and interrupt time alone > for a 1 Gbps connection can easily saturate a single CPU, even with interrupt > coalescing and many other tricks. When you move to a 10 Gbps connection, you > can't just add more CPUs to deal with the problem, because of both cost and > the fact that you don't get a perfect speedup as you add more CPUs. This > makes techniques that reduce CPU and bus utilization more valuable as network > speeds increase. And add to that the fact that indiscriminate interrupt coalescing trades-off against latency, making a latency sensitive application suffer in mixed traffic. :a From XXX Wed Mar 15 20:37:15 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Mar 15 15:37:00 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id PAA05323 for ; Wed, 15 Mar 2000 15:36:59 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA01355 for ips-outgoing; Wed, 15 Mar 2000 15:36:42 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA01349 for ; Wed, 15 Mar 2000 15:36:37 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id VAA141234; Wed, 15 Mar 2000 21:36:05 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id VAA141438; Wed, 15 Mar 2000 21:36:04 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568A3.0071299A ; Wed, 15 Mar 2000 21:36:02 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: "Bradley, Mark" cc: "'julian_satran@il.ibm.com'" , Zachary Amsden , "Bradley, Mark" , ips@ece.cmu.edu Message-ID: Date: Wed, 15 Mar 2000 22:35:46 +0200 Subject: RE: IPS Issues document Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk You are right. My only concern was that the "reducing the CPU load" has to be clearly qualified as having memory load as a target and as such being inevitable. CPU load in the sense of processor load (and that is what it sounds like) will raise a lot of alarms and will recall past failures (the nightmare of maintaining 2 protocol stacks by software or system providers for off-load adapters and plain vanilla adapters and the fact that in later machines the load dissipated!). Regards, Julo "Bradley, Mark" on 15/03/2000 21:30:52 Please respond to "Bradley, Mark" To: "'julian_satran@il.ibm.com'" , Zachary Amsden cc: "Bradley, Mark" , ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: IPS Issues document This is sort of an interesting thread. It sound like the goals covered so far are: 1)Reduce CPU utilization 2)Minimize copies 3)Acheive low latency 4)Avoid hardware obsolescence/bottlenecking (...along with, as previously stated of course, using IP as the transport). What approaches can meet the above? BTW, this sounds like a lot of what was in the Framework draft that was sent out. -- markb > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, March 14, 2000 11:14 PM > To: Zachary Amsden > Cc: Bradley, Mark; ips@ece.cmu.edu > Subject: Re: IPS Issues document > > > > > I would like to clarify my comment. I am not contesting the > fact that CPU > will get saturated. > My concern is that CPU load decrease does not justify > specialized silicon > for protocol processing > (as it gets soon out-dated like the 186 NICS) and you have > the nightmare of > supporting 2 protocol stacks. Memory load on the other hand can't be > solved any other way than by off-loading and reducing copies > to a minimum. > > Regards, > Julo > > Zachary Amsden on 14/03/2000 20:53:41 > > Please respond to Zachary Amsden > > To: "Bradley, Mark" > cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) > Subject: Re: IPS Issues document > > > > > > Julo, good comments. My observation is that faster CPU's > > don't solve the problem. We have been getting faster CPU's > > since the beginning of time and the only semi-constant is > > that we find that the CPU continues to get utilized. Large > > MP machines are bought because the processing horsepower > > is needed for some appl.(s) that consumes it. Therefore they > > don't have 'extra' cycles to spare. Most generic OS's > > don't support dedicating a processor to specific jobs, > > either. Putting a small processor someplace (an IOP) else > > typically doesn't scale as system processing speed increases, > > so those often become bottle-necks after some time. Remember > > those 186-based networking cards? > > > > Bottom line is that % CPU utilization has been, is, and will > > continue to be an important consideration for some time. > > It seems we never have enough CPU, memory or disk. We seem > > always to find ways to use it up. (I remember years ago > > thinking "Wow! A megabyte on a single platter! We'll have > > enough disk space to last 3 or 4 years!!!" I also recall > > thinking that a 68020 made for a pretty fast workstation. > > Now I complain that my dual 450 MHz PC with 128 MB of memory > > is not fast enough to be a decent office machine next year... > > :{) ) > > I agree completely with that - protocol processing and > interrupt time alone > for a 1 Gbps connection can easily saturate a single CPU, even with > interrupt > coalescing and many other tricks. When you move to a 10 Gbps > connection, > you > can't just add more CPUs to deal with the problem, because of > both cost and > the fact that you don't get a perfect speedup as you add more > CPUs. This > makes techniques that reduce CPU and bus utilization more valuable as > network > speeds increase. > > -- > Zachary Amsden zamsden@engr.sgi.com 3-6919 31-2-510 Core Protocols > > > > > From XXX Sat Mar 18 02:57:22 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Mar 17 20:58:47 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA27602 for ; Fri, 17 Mar 2000 20:58:47 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA13407 for ips-outgoing; Fri, 17 Mar 2000 20:56:24 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from manganese.network-alchemy.com (NAT-10-net.Network-Alchemy.COM [199.46.17.205]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA13394 for ; Fri, 17 Mar 2000 20:56:17 -0500 (EST) Received: from manganese.network-alchemy.com (localhost [127.0.0.1]) by manganese.network-alchemy.com (8.9.3/8.8.8) with ESMTP id RAA10016 for ; Fri, 17 Mar 2000 17:56:06 -0800 (PST) (envelope-from rdv@manganese.network-alchemy.com) Message-Id: <200003180156.RAA10016@manganese.network-alchemy.com> X-Mailer: exmh version 2.0.2 2/24/98 To: ips@ece.cmu.edu Reply-To: rdv@Network-Alchemy.COM Subject: transport protocols Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 17 Mar 2000 17:56:06 -0800 From: Rod Van Meter Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry to be so late getting in on this conversation. There was a discussion of transport protocols a couple of weeks ago, and I wanted to contribute my $0.02. When I was at Quantum, we investigated heavily a number of transport protocols for possible use over IP: * FC classes 2 & 3 * TCP * T/TCP * XTP * VMTP * ST * lightweight reliability over UDP I no longer have my notes (they stayed at Quantum), but if memory serves: * FC 2 & 3: forget it. Too dependent on a reliable lower layer. All 2 really gets you is faster notification of errors -- you still fail out to app-level recovery on dropped packets. * TCP: the best choice. The problem is efficiently multiplexing multiple commands and finding message boundaries -- issues the TCP option draft here addressed. * T/TCP: I like it. There are apparently some security concerns. For me the show-stopper, though, is that it's designed for single-threaded RPC. A T/TCP connection starts, does its thing, shuts down, then is ready for the next one. If you want two concurrent I/Os, you need multiple T/TCP streams. * XTP: another good option, with a few advantages over TCP (simplicity was an XTP goal, partially reached), but not compelling enough. Inadequate congestion control; by the time you add it, you're probably close to TCP in complexity. Not firewall friendly? * VMTP: a little long in the tooth -- would need a serious update. Rate-based flow control not useful over today's Internet infrastructure, could still be made to work in a LAN. Forwarded RPCs (command to node A, response from node B) is quite intriguing for more complex storage systems, but not really needed for SCSI over IP. Not firewall friendly. * ST: a nice, lightweight protocol, but I'm concerned about a) assumptions about a reliable link layer, b) latency sensitivity, c) congestion control. Probably fine in a LAN, not useful in a WAN. I've got a long list of references on the above, plus more, if anyone's interested. --Rod From XXX Sat Mar 18 20:05:40 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sat Mar 18 11:12:05 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA16614 for ; Sat, 18 Mar 2000 11:12:04 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03396 for ips-outgoing; Sat, 18 Mar 2000 11:11:15 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03392 for ; Sat, 18 Mar 2000 11:11:09 -0500 (EST) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA60324 for ; Sat, 18 Mar 2000 17:10:36 +0100 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id RAA49266 for ; Sat, 18 Mar 2000 17:10:37 +0100 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568A6.0058DB3A ; Sat, 18 Mar 2000 17:10:32 +0100 X-Lotus-FromDomain: IBMIL@IBMDE To: rdv@Network-Alchemy.COM cc: ips@ece.cmu.edu Message-ID: Date: Sat, 18 Mar 2000 18:10:16 +0200 Subject: Re: transport protocols Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Rod, Thanks for your note. Obviously we went through this exercise too and decided for TCP. I still have the lingering feeling that we did not get through thoroughly enough with the T/TCP option. As it is specified today it is sort of clumsy and not very easy to implement or to assist in hardware. However there might be other venues for reaching the same goal - some related to the RDMA option suggested by Costa and others somewhat different and more far-reaching with a similar effect. Will you participate in a working group. will your former colleagues from Quantum? Regards, Julo Rod Van Meter on 18/03/2000 03:56:06 Please respond to rdv@Network-Alchemy.COM To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: transport protocols Sorry to be so late getting in on this conversation. There was a discussion of transport protocols a couple of weeks ago, and I wanted to contribute my $0.02. When I was at Quantum, we investigated heavily a number of transport protocols for possible use over IP: * FC classes 2 & 3 * TCP * T/TCP * XTP * VMTP * ST * lightweight reliability over UDP I no longer have my notes (they stayed at Quantum), but if memory serves: * FC 2 & 3: forget it. Too dependent on a reliable lower layer. All 2 really gets you is faster notification of errors -- you still fail out to app-level recovery on dropped packets. * TCP: the best choice. The problem is efficiently multiplexing multiple commands and finding message boundaries -- issues the TCP option draft here addressed. * T/TCP: I like it. There are apparently some security concerns. For me the show-stopper, though, is that it's designed for single-threaded RPC. A T/TCP connection starts, does its thing, shuts down, then is ready for the next one. If you want two concurrent I/Os, you need multiple T/TCP streams. * XTP: another good option, with a few advantages over TCP (simplicity was an XTP goal, partially reached), but not compelling enough. Inadequate congestion control; by the time you add it, you're probably close to TCP in complexity. Not firewall friendly? * VMTP: a little long in the tooth -- would need a serious update. Rate-based flow control not useful over today's Internet infrastructure, could still be made to work in a LAN. Forwarded RPCs (command to node A, response from node B) is quite intriguing for more complex storage systems, but not really needed for SCSI over IP. Not firewall friendly. * ST: a nice, lightweight protocol, but I'm concerned about a) assumptions about a reliable link layer, b) latency sensitivity, c) congestion control. Probably fine in a LAN, not useful in a WAN. I've got a long list of references on the above, plus more, if anyone's interested. --Rod From XXX Wed Mar 22 22:00:28 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Mar 22 12:35:54 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA20084 for ; Wed, 22 Mar 2000 12:35:53 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA13385 for ips-outgoing; Wed, 22 Mar 2000 12:34:06 -0500 (EST) Received: from rogue.network-alchemy.com (Rogue.Network-Alchemy.COM [199.46.17.170]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA13380 for ; Wed, 22 Mar 2000 12:33:59 -0500 (EST) Received: (from rdv@localhost) by rogue.network-alchemy.com (8.8.8/8.8.8) id JAA00700; Wed, 22 Mar 2000 09:42:31 -0800 (PST) (envelope-from rdv@rogue.network-alchemy.com) Date: Wed, 22 Mar 2000 09:42:31 -0800 (PST) Message-Id: <200003221742.JAA00700@rogue.network-alchemy.com> X-Authentication-Warning: rogue.network-alchemy.com: rdv set sender to rdv@rogue.network-alchemy.com using -f From: Rod Van Meter To: ips@ece.cmu.edu Subject: bibliography Sender: owner-ips@ece.cmu.edu Precedence: bulk Okay, here are some bibliographic entries for some useful papers on transport protocols and other topics. For reasons probably related to my job change but hopefully not meaning I've lost a bunch of it, my current bibliography appears incomplete. I know I'm missing a survey of lightweight transport protocols (from IEEE ToN, I think) and a few others (my total bibliography is over 400 papers, but for some reason, I'm always missing whatever today's most important one is :-(). I'll immodestly start with the non-transport papers: All of the Netstation papers are available at http://www.isi.edu/netstation/ @InProceedings{van-meter:visa, author = "Rodney Van{ }Meter and Greg Finn and Steve Hotz", title = "{VISA}: Netstation's Virtual Internet {SCSI} Adapter", crossref = "asplos98" } @InProceedings{van-meter:ip-for-naps, author = "Rodney Van{ }Meter and Greg Finn and Steve Hotz", title = "Internet Protocols for Network-Attached Peripherals", crossref = "kobler:gsfc-msst-98" } A one-pager on why IP: @InProceedings{van-meter:ip-naps, author = "Rodney Van{ }Meter and Steve Hotz and Gregory G. Finn", title = "Task Force on Network Storage Architecture: Internet-attached storage devices", crossref = "hicss-30", note = "White paper for the task force that met in conjunction with the conference", URL = {http://www.isi.edu/netstation/hicss_isi.ftp.ps} } This one is now quite dated: @ARTICLE{van-meter:nap-overview, title = {A Brief Survey of Current Work on Network Attached Peripherals (extended abstract)}, author = {Rodney Van{ }Meter}, journal = {ACM Operating Systems Review}, year = 1996, month = jan, pages = {63--70} } Garth's stuff (these are the two best, IMHO): @TechReport{gibson:nasd-case, author = {Garth Gibson and others}, title = {A Case for Network-Attached Secure Disks}, institution = {CMU}, year = 1996, number = {CMU-CS-96-142}, month = jun, comment = {Good paper presenting a taxonomy of NAP interfaces, and numbers evaluating potential performance improvements for NFS and AFS, varing from a few percent to a factor of ten in CPU cycles.}, read = {1996/7/24} } @InProceedings{gibson:nasd-file-server-scaling, author = "Garth A. Gibson and others", title = "File Server Scaling with Network-Attached Secure Disks", crossref = "sigmetrics97", comment = {Presents their NASD taxonomy again, with more detailed analysis of the results of using their NASD model. Suggests total server workload reduction of 30 percent or so for the ``NetSCSI'' approach (which they don't recommend) and factors of 5 and 10 for AFS and NFS using their particular NASD model. Replays UCB NFS traces and their own AFS traces against an emulated server to test response time. Disparages UDP performance. Great references.}, read = {1997/10/26}, URL = {http://www.cs.cmu.edu/Web/Groups/NASD/}, location = { rdv : folder : naps } } Okay, now transport protocols: @Article{baguette92:_xtp_conn_compare, author = {Yves Baguette and Andre Danthine}, title = {Comparison of {TP4}, {TCP} and {XTP} -- Part 1: Connection Management Mechanisms}, journal = {European Transactions on Telecommunications}, year = 1992, volume = 3, number = 5, month = {Sept.-Oct.}, comment = {English is a little rough, but good info. Concentrates mostly on TP4 v. XTP. TP4, like TCP, uses a 3-way handshake. XTP requires an explicit connection open packet (different format than data packets), but data packets can be pipelined with it, cutting out a round trip. Connection close can also be reduced to 2-way handshake.}, location = { rdv : folder : XTP }, read = {1998/4/13}, URL = {http://www-run.montefiore.ulg.ac.be/publications/papers/abstract-R92-03_A.html} } @Article{baguette92:_xtp_data_compare, author = {Yves Baguette and Andre Danthine}, title = {Comparison of {TP4}, {TCP} and {XTP} -- Part 2: Data Transfer Mechanisms}, journal = {European Transactions on Telecommunications}, year = 1992, volume = 3, number = 5, month = {Sept.-Oct.}, comment = {Spends a lot of time on the differences between XTP 3.5 and 3.6 (current version is 4.0). Provides some support for ALF, in the form of an end-of-message bit. No segmentation/reassembly allowed in the net. Checksums are in trailers, and are optional. Uses a timeout to decide a packet has been lost, but the details of that setting aren't covered. Error handling is tricky in standard internets because it assumes in-order delivery (which requires XTP-aware routers under some conditions). Uses a window mechanism for flow control. Supports optional FASTNACKs (again depends on in-order delivery). ACKs are piggybacked. Congestion control is done via path rate control, which again requires XTP-aware routers. XTP supports both reliable and unreliable multicast.}, location = { rdv : folder : XTP }, read = {1998/4/13}, URL = {http://www-run.montefiore.ulg.ac.be/publications/papers/abstract-R92-03_B.html} } @INPROCEEDINGS{cheriton:vmtp, author = {Cheriton, David R.}, title = {Exploiting Recursion to Simplify RPC Communication Architectures}, booktitle = {Proc. ACM SigComm (Stanford, CA August 1988)}, year = 1988, month = aug, pages = {76--87}} @MISC{cheriton:vmtp-rfc, author = {Cheriton, David R.}, title = {VMTP: Versatile Message Transaction Protocol / Protocol Specification}, number = {Internet RFC 1045}, organization = {USC/ISI}, year = 1988, month = feb} @Manual{hippi-st, title = {Information Technology - Scheduled Transfer Protocol {(ST)} {T11.1/Project 1245-D}}, organization = {NCITS}, year = 1998, month = jan, note = {rev 1.45}, comment = {Scheduled transfers for HIPPI, as a transport protocol. Also claimed to work for FC and ethernet, but I dpn't think anybody's really using that yet. Separate control and data connections. Begins with a setup negotiation, in which the destination indicates how much data it's prepared to receive. Mentions that this could be adjusted downward by intermediate nodes, but no discussion of how. Assumes very reliable in-order delivery for efficiency, though should work on packet loss due to ACKs (apparently, one ACK per data pkt is required). Targetted to minimize the work on receive. Transfers consist of blocks which consist of STUs, which I think correspond to packets. Must be a power of two in size. Some support for striping, including fan-in and fan-out, but still looks incomplete. Headers indicate when the upper-layer system should be interrupted. Some support for negotiated out-of-order delivery, but I'm not sure how that works with the max received index in the ACK packets (ACK pkts are primarily indicators of buffer availability for flow control, really CTS). Says it assumes low latency. No discussion of how packets might overlap in transmission; diagrams show a series of data pkts then a series of CTSes. Thinks it ld be used to carry TCP/IP traffic, but I view it more as a transport protocol and would be curious to see ST over IP. Despite not being an Internet protocol, bows to the IANA for selection of things like well-known port numbers for services.}, location = { rdv : folder : hippi }, read = {1998/1/30}, URL = {http://www.cic-5.lanl.gov/~det/dST145.pdf} } There are a number of other papers on XTP, as well as the spec, available at http://www.ca.sandia.gov/xtp/. Hope this helps. From XXX Thu Mar 23 04:01:55 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Mar 22 22:43:36 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id WAA03153 for ; Wed, 22 Mar 2000 22:43:35 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA01242 for ips-outgoing; Wed, 22 Mar 2000 22:43:13 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from deliverator.sgi.com (deliverator.sgi.com [204.94.214.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA01235 for ; Wed, 22 Mar 2000 22:43:09 -0500 (EST) Received: from cthulhu.engr.sgi.com (gate3-relay.engr.sgi.com [130.62.1.234]) by deliverator.sgi.com (980309.SGI.8.8.8-aspam-6.2/980310.SGI-aspam) via ESMTP id TAA18675; Wed, 22 Mar 2000 19:38:16 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from dsl-lhotse.corp.sgi.com ([192.132.126.170]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via SMTP id TAA54630; Wed, 22 Mar 2000 19:42:52 -0800 (PST) mail_from (aman@engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by dsl-lhotse.corp.sgi.com (950413.SGI.8.6.12/970903.SGI.AUTOCF) via ESMTP id DAA27317; Thu, 23 Mar 2000 03:41:05 GMT Message-ID: <38D99251.1F04F2EA@engr.sgi.com> Date: Wed, 22 Mar 2000 19:41:05 -0800 From: Aman Singla Organization: Silicon Graphics, Inc. X-Mailer: Mozilla 4.51C-SGI [en] (X11; I; IRIX 6.2 IP22) X-Accept-Language: en MIME-Version: 1.0 To: Rod Van Meter CC: ips@ece.cmu.edu Subject: Re: bibliography References: <200003221742.JAA00700@rogue.network-alchemy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > @Manual{hippi-st, > title = {Information Technology - Scheduled Transfer Protocol > {(ST)} {T11.1/Project 1245-D}}, > organization = {NCITS}, > year = 1998, > month = jan, > note = {rev 1.45}, > comment = {Scheduled transfers for HIPPI, as a transport > protocol. Also claimed to work for FC and ethernet, > but I dpn't think anybody's really using that yet. > Separate control and data connections. Begins with > a setup negotiation, in which the destination > indicates how much data it's prepared to receive. > Mentions that this could be adjusted downward by > intermediate nodes, but no discussion of how. > Assumes very reliable in-order delivery for > efficiency, though should work on packet loss due to > ACKs (apparently, one ACK per data pkt is required). No there isn't an assumption of reliable in-order delivery. Packet loss is handled by timeout on the receiver side (or skip in block sequence if out of order is not enabled) trigerring the resend of a CTS => as all timeout schemes, has impact on performance when there is packet loss. > Targetted to minimize the work on receive. > Transfers consist of blocks which consist of STUs, > which I think correspond to packets. Must be a > power of two in size. Some support for striping, > including fan-in and fan-out, but still looks incomplete. > Headers indicate when the upper-layer system should > be interrupted. Some support for negotiated > out-of-order delivery, but I'm not sure how that > works with the max received index in the ACK packets > (ACK pkts are primarily indicators of buffer > availability for flow control, really CTS). Says it CTSs are an indication of buffer availability. Out-of-order delivery is simply handled by each incoming data block having an address of where the data is supposed to go, and the protocol doing book-keeping at the block level. The ACK, if by ACK you mean the RSR, is normally for the end of the transfer (transfer->blocks->stus) to signify receipt of all blocks to the sender. RSRs with max index recd. are also used for Persistent Memory short message Put and Get sequences (which are different from the Read/Write sequences for Long Messages/Transfers which are being discussed here) as the sliding window ack mechanism over unreliable media. > assumes low latency. No discussion of how packets > might overlap in transmission; diagrams show a > series of data pkts then a series of CTSes. Thinks Basically multiple outstanding CTSs are allowed => CTSs from the receiver to the sender are in a pipeline with the data flowing from the sender to the receiver. This thus hides the CTS latency for all blocks except the first one. > it ld be used to carry TCP/IP traffic, but I view it > more as a transport protocol and would be curious to > see ST over IP. Despite not being an Internet That is a true observation. We have ST working over IP over Gigabit ethernet working on Irix and Linux. We've been releasing the linux code to the open-source community. > protocol, bows to the IANA for selection of things > like well-known port numbers for services.}, > location = { rdv : folder : hippi }, > read = {1998/1/30}, > URL = {http://www.cic-5.lanl.gov/~det/dST145.pdf} Thanks, :a From XXX Thu Mar 23 04:01:55 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Mar 22 22:27:12 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id WAA02947 for ; Wed, 22 Mar 2000 22:27:12 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA00860 for ips-outgoing; Wed, 22 Mar 2000 22:26:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA00856 for ; Wed, 22 Mar 2000 22:26:30 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id TAA04880; Wed, 22 Mar 2000 19:26:26 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from dsl-lhotse.corp.sgi.com ([192.132.126.170]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via SMTP id TAA65123; Wed, 22 Mar 2000 19:26:24 -0800 (PST) mail_from (aman@engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by dsl-lhotse.corp.sgi.com (950413.SGI.8.6.12/970903.SGI.AUTOCF) via ESMTP id DAA27307; Thu, 23 Mar 2000 03:24:37 GMT Message-ID: <38D98E74.242E32E7@engr.sgi.com> Date: Wed, 22 Mar 2000 19:24:37 -0800 From: Aman Singla Organization: Silicon Graphics, Inc. X-Mailer: Mozilla 4.51C-SGI [en] (X11; I; IRIX 6.2 IP22) X-Accept-Language: en MIME-Version: 1.0 To: rdv@Network-Alchemy.COM CC: ips@ece.cmu.edu Subject: Re: transport protocols References: <200003180156.RAA10016@manganese.network-alchemy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > * ST: a nice, lightweight protocol, but I'm concerned about a) > assumptions about a reliable link layer, b) latency sensitivity, c) > congestion control. Probably fine in a LAN, not useful in a WAN. ST doesn't require reliable link layer. What do you mean by being concerned about latency sensitivity? :a From XXX Thu Mar 23 18:37:05 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Mar 23 13:36:39 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA14406 for ; Thu, 23 Mar 2000 13:36:38 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA00785 for ips-outgoing; Thu, 23 Mar 2000 13:35:35 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from manganese.network-alchemy.com (NAT-10-net.Network-Alchemy.COM [199.46.17.205]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA00778 for ; Thu, 23 Mar 2000 13:35:29 -0500 (EST) Received: from manganese.network-alchemy.com (localhost [127.0.0.1]) by manganese.network-alchemy.com (8.9.3/8.8.8) with ESMTP id KAA32196; Thu, 23 Mar 2000 10:35:19 -0800 (PST) (envelope-from rdv@manganese.network-alchemy.com) Message-Id: <200003231835.KAA32196@manganese.network-alchemy.com> X-Mailer: exmh version 2.0.2 2/24/98 To: Aman Singla cc: Rod Van Meter , ips@ece.cmu.edu Subject: Re: bibliography In-reply-to: aman's message of Wed, 22 Mar 2000 19:41:05 -0800. <38D99251.1F04F2EA@engr.sgi.com> Reply-To: rdv@Network-Alchemy.COM Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 23 Mar 2000 10:35:19 -0800 From: Rod Van Meter Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks for the additional info on ST. The difference between a careful reading and actually implementing the stuff :-). I no longer recall why (this was two years ago), but there was something in the retransmit behavior on error that made me think, "Oh, this is going to suck on packet loss." The explanation in the draft I read also was not clear on the amount of pipelining allowed, but I felt that there were going to be occasional round-trip pauses. Is this not true? If so, I'm happy about that. Is the Linux code in the main kernel distributions? I haven't seen it, but didn't know to go looking. If not, got a URL? --Rod From XXX Thu Mar 23 19:13:42 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Mar 23 13:52:51 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA15048 for ; Thu, 23 Mar 2000 13:52:50 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA01542 for ips-outgoing; Thu, 23 Mar 2000 13:52:44 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sgi.com (sgi.SGI.COM [192.48.153.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA01536 for ; Thu, 23 Mar 2000 13:52:39 -0500 (EST) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by sgi.com (980327.SGI.8.8.8-aspam/980304.SGI-aspam: SGI does not authorize the use of its proprietary systems or networks for unsolicited or bulk email from the Internet.) via ESMTP id KAA06011; Thu, 23 Mar 2000 10:52:31 -0800 (PST) mail_from (aman@cthulhu.engr.sgi.com) Received: from lhotse.engr.sgi.com (lhotse.engr.sgi.com [163.154.35.41]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id KAA41076; Thu, 23 Mar 2000 10:52:28 -0800 (PST) mail_from (aman@engr.sgi.com) Received: from engr.sgi.com (localhost [127.0.0.1]) by lhotse.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id KAA25030; Thu, 23 Mar 2000 10:52:27 -0800 (PST) Message-ID: <38DA67EB.D8675DE4@engr.sgi.com> Date: Thu, 23 Mar 2000 10:52:27 -0800 From: Aman Singla Organization: SGI X-Mailer: Mozilla 4.7C-SGI [en] (X11; I; IRIX 6.5 IP32) X-Accept-Language: en MIME-Version: 1.0 To: rdv@Network-Alchemy.COM CC: Aman Singla , ips@ece.cmu.edu Subject: Re: bibliography References: <200003231835.KAA32196@manganese.network-alchemy.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > I no longer recall why (this was two years ago), but there was something in > the retransmit behavior on error that made me think, "Oh, this is going to > suck on packet loss." Retransmit behvaior is always an artifact of how intelligent can you make your timeout stuff; ST spec doesn't talk much about how to best do the timeout for the retransmission. Sophistication in schemes there can come out of experience with more and more networks only... something that TCP has; hopefully some of the experience from TCP can apply to the ST case also. There was an issue with packet loss/out of order delivery related to the last block of the transfer not being very efficient and relying on timeouts for the final ack to successfully complete the transfer. But that was addressed in a more efficient manner by the spec sometime in the last two years. > The explanation in the draft I read also was not clear on the amount of > pipelining allowed, but I felt that there were going to be occasional > round-trip pauses. Is this not true? If so, I'm happy about that. That is not true. The amount of pipelining is controlled by the 'advised outstanding number of CTSs' used by the stack on a per transfer basis. Its best to have the exact number of outstanding CTSs to keep the pipeline full and flowing all the time - and again the onus is on the sophistication of the implementation to figure out that number; defaults values can work only so well. > Is the Linux code in the main kernel distributions? I haven't seen it, but > didn't know to go looking. If not, got a URL? The linux code is still not feature complete and alpha quality. I made the first release of the long message stack last month as a patch to the kernel. I'll be making the next release with some additional support related to Gigabit Ethernet tomorrow or early next week. These are development releases; we're still quite some ways from the complete implementation of the ST feature set and as such it might be a while before it gets into the mailine kernel. The project page is http://oss.sgi.com/projects/stp :a From XXX Fri Mar 24 03:00:04 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Mar 23 20:05:04 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA05549 for ; Thu, 23 Mar 2000 20:05:03 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA13969 for ips-outgoing; Thu, 23 Mar 2000 20:04:34 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh3.isus.emc.com ([168.159.129.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA13965 for ; Thu, 23 Mar 2000 20:04:31 -0500 (EST) From: Black_David@emc.com Received: by mxbh3.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 24 Mar 2000 08:50:53 -0500 Message-ID: <0F31E5C394DAD311B60C00E029101A07071191@mxclsk.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: Comments to Comments! Date: Thu, 23 Mar 2000 20:03:59 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk My turn to reply to both Julian and Costa - apologies for the delay involved. I hope folks have a chance to read this prior to Adelaide, and I apologize for the fact that both the timing (last week of the calendar quarter) and distance from here make it impossible for me to attend the BOF in Adelaide. This email includes input from EMC folks in addition to yours truly. -- DNS: > Parties who wish to use IP addresses for extra reliability may encode the > IP addresses in the target names (e.g. 10.0.4.5/dvd). After all, domain > names can represent IP addresses. These type of domain names do not > require DNS for resolution. Keep in mind that the storage side of this is a closed box that may not be fully configurable. Allowing the storage to use hostnames may force the initiator to perform host name resolution whether it wants to or not, and v.v. Host name resolution issues should be confined to hosts, with the storage kept out of this. This objection is primarily about Open Data Connection; the use of DNS names may be defensible for third party copy. Moving to a combined data/control connection model would go a long way towards resolving this. > It was felt that we should be completely independent of IP addresses > because of firewall and IP masquerading issues with setting up new TCP > connections. IP addresses /can/ be used, but only in dotted decimal > notation. But this doesn't solve the fundamental problem with NAT/NAPT (of which IP masquerade is an example) because there's no way of knowing whether the namespaces and conventions for name resolution match on both sides of the NAT/NAPT. For example, if I pass "foo" as an identifier, it may resolve to foo.emc.com here and foo.eng.cisco.com there - FQDNs eliminate this simple example but cause other problems because DNS need not be a globally uniform namespace - in general one is now at the mercy of all sorts of peculiar DNS configuration oddities. Non-DNS resolution mechanisms are not a magic cure - different YP/NIS domains and out of sync /etc/hosts files are capable of causing problems. OTOH, the use of a combined control/data connection, and avoiding passing endpoint addresses in the payload makes this problem vanish, except for Third Party Copy, which is a much more involved story that probably requires a discussion of ALGs and some serious SHOULD NOTs. The notion of passing IP addresses as text strings reproduces one of the most irritating (in 20/20 hindsight) design mistakes of ftp. The problem is that remapping an IP address may change the length of the text string, causing all sorts of complications (e.g., what if the packet is at MTU and the string gets longer?). This won't work in a NAT/NAPT, and makes writing an ALG to put in such a box unnecessarily painful. > On the plus side, domain names decouple the iSCSI protocol from the > underlying addressing architecture. The potential deployment of IPv6 will > not require any changes to the iSCSI protocol. Good Grief! Costa can't be serious about this as a reason. Designing a variable length address field that accommodates IPv4 and IPv6 addresses is so easy, .. and besides TCP requires no changes for IPv6 and it has no clue about host names. -- Parameter negotiation > An implementation can ignore all free-form text/value pairs and still operate just fine. Provided that all the defaults are acceptable. For example, Section 3.7 says: In order to allow write operations without RTT, the initiator and target must have agreed to do so by both sending the AllowNoRTT:yes key-pair attribute to each other (either during Login or through the Text Command/Response mechanism). In this case, the default (RTT required on write) appears to be correct, the bad news is that implementations that want to negotiate it away buy into the text processing by comparison to the small number of bits that are negotiated in FCP. One of the more important defaults that must be accepted is No Authentication, but I think the key:value pairing is an ok way to support authentication - I'm concerned that it's overkill for the small number of bits that SCSI needs to negotiate. In general, this sort of arbitrary extensibility can be both a virtue and a vice because while extensions don't require on-the-wire format changes, they do require complicated rules about what key:value pairs are supposed to be in which message and how to deal with situations in which some of them are missing. -- CRC > A CRC at the TCP layer or higher? Higher. The read and write data need to be covered by a real CRC. TCP and IP checksums are too weak to be acceptable, and routers strip/regenerate layer 2 checksums leaving no CRC to cover corruption in the router. Restricting the CRC to data only avoids any requirement that an ALG recalculate it, as CRCs are much more difficult to adjust than the 1's complement TCP and IP checksums. -- Ping > > * What value does the ability to do an iSCSI ping add to the existing > > ability to do an ICMP ECHO? If little or none, this should be omitted, see > > section 3.15. > > It makes sure the iSCSI device server is still alive and kicking. I'm not sure about this one, as a SCSI Inquiry command seems to do about the same thing, and verifies that the iSCSI engine can actually do something SCSI, as opposed to just answer a ping. One complication is that Inquiry can return a check condition that requires further action, in contrast to a self-contained ping. -- Combined control and data connections > If we multiplex LUNs (as we do in the current draft) keeping to a short TCP > frame will leave as open to all sorts of troubles (possible deadlocks) due > to the limited TCP window and our lack of control over the data source and > sink. Separating the control and data stream we could resort to selective > resets to get out of trouble - while with a common connection we might have > to resort to radical means (e.g., closing connections). Multiplexing LUNs is the right decision to avoid massive proliferation of TCP session state. Separating the control and data streams is not the only way out of "all sorts of troubles". With combined control and data connections, holding another control connection open works for selective resets, with the possible exception of Abort Task (which may have to be issued on the connection that the task was initiated on). > In addition in a "permissive" environment (like a video server) we might > require CRC on the control connection while leaving the data connections up > to the user. But there's more than enough flexibility to negotiate this behavior. In any case, EMC lives in a part of the world where omitting CRCs is a generally bad idea, even for video data. -- Killing all I/Os There are two important cases here. In the first case, if the control connection times out and closes (or closes for any other reason), then clearly all I/Os have to be killed. The problem of concern is in the second case: opening up a new control connection CAUSES the old one to be closed as a side effect. That seems unnecessary, especially because resets of various forms can be issued down the new connection to cause the device to clean up and get into a known state. This interacts with the issue above about combining data/control onto one connection and allowing multiple connections between an initiator and responder pair. -- Target Name from Initiator > The target can ignore any key:value pairs sent by the initiator, so it need > not receive its name from the initiator. This feature is useful in case the > target is actually a front end for many machines and/or disks, in which > case the initiator can specify to which target it really wants to interact > with. I wonder if going there is a good idea, vs. something like a front end simply exporting each machine and/or disk on a different TCP port. The problem with handing the target its address inband is that the connection address no longer fully specifies what the initiator is talking to, and that seems wrong. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From XXX Fri Mar 31 21:41:48 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Mar 31 16:35:50 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id QAA16939 for ; Fri, 31 Mar 2000 16:35:49 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04671 for ips-outgoing; Fri, 31 Mar 2000 16:31:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from manganese.network-alchemy.com (NAT-10-net.Network-Alchemy.COM [199.46.17.205]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA04666 for ; Fri, 31 Mar 2000 16:31:23 -0500 (EST) Received: from manganese.network-alchemy.com (localhost [127.0.0.1]) by manganese.network-alchemy.com (8.9.3/8.8.8) with ESMTP id NAA82663 for ; Fri, 31 Mar 2000 13:31:24 -0800 (PST) (envelope-from rdv@manganese.network-alchemy.com) Message-Id: <200003312131.NAA82663@manganese.network-alchemy.com> X-Mailer: exmh version 2.0.2 2/24/98 To: ips@ece.cmu.edu Subject: SCSI over ST Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 31 Mar 2000 13:31:24 -0800 From: Rod Van Meter Sender: owner-ips@ece.cmu.edu Precedence: bulk Not to derail the SCSI over TCP momentum here, but FYI, for those who don't follow the HiPPI reflector. Meeting sounds good, sorry I wasn't there. --Rod ------- Forwarded Message Return-Path: rdv Return-Path: Received: from SMTP.Network-Alchemy.COM (Hydrogen.Network-Alchemy.COM [199.46.17.130]) by Dysprosium.Network-Alchemy.COM (8.9.3/8.9.3) with SMTP id MAA15422 for ; Fri, 31 Mar 2000 12:42:13 -0800 (PST) (envelope-from hippi-owner@storage.network.com) Received: from tnt.isi.edu ([128.9.128.128]) by Hydrogen.Network-Alchemy.COM via INTERNET ; Fri, 31 Mar 2000 12:42:30 PST Received: from storage.network.com (storage.network.com [129.191.1.4]) by tnt.isi.edu (8.8.7/8.8.6) with ESMTP id MAA07393 for ; Fri, 31 Mar 2000 12:42:29 -0800 (PST) Received: (from majordom@localhost) by storage.network.com (8.9.3+Sun/8.9.3) id OAA13693 for hippi-outgoing; Fri, 31 Mar 2000 14:17:11 -0600 (CST) X-Authentication-Warning: storage.network.com: majordom set sender to owner-hippi@network.com using -f Message-ID: <013901bf9b4d$361eabb0$1b67aacf@parrot> From: "Roger Ronald" To: Subject: New release of SCSI on ST at web site Date: Fri, 31 Mar 2000 14:10:50 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2314.1300 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Sender: owner-hippi@storage.network.com Precedence: bulk Reply-To: "Roger Ronald" Status: * * From the hippi reflector, posted by: * "Roger Ronald" * www.hippi.org contains two locations where there's a hot new version of SCSI over ST. You can pick it off the "what's new" page or get it on the SCSI over ST document page. In any event, we'll be reviewing this document on 4/5 (next Wednesday) at 9 AM. Bring your copy, if you're coming! RR ------- End of Forwarded Message From XXX Fri Mar 31 21:54:54 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Mar 31 16:54:35 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id QAA17576 for ; Fri, 31 Mar 2000 16:54:35 -0500 Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA05362 for ips-outgoing; Fri, 31 Mar 2000 16:51:16 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA05277 for ips-outgoing; Fri, 31 Mar 2000 16:49:40 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA05219 for ips-outgoing; Fri, 31 Mar 2000 16:48:07 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA05138 for ips-outgoing; Fri, 31 Mar 2000 16:46:34 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA05088 for ips-outgoing; Fri, 31 Mar 2000 16:44:54 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA05022 for ips-outgoing; Fri, 31 Mar 2000 16:43:15 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04979 for ips-outgoing; Fri, 31 Mar 2000 16:41:40 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04923 for ips-outgoing; Fri, 31 Mar 2000 16:40:06 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04879 for ips-outgoing; Fri, 31 Mar 2000 16:38:32 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04828 for ips-outgoing; Fri, 31 Mar 2000 16:36:58 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04772 for ips-outgoing; Fri, 31 Mar 2000 16:35:25 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04719 for ips-outgoing; Fri, 31 Mar 2000 16:33:53 -0500 (EST) Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04671 for ips-outgoing; Fri, 31 Mar 2000 16:31:30 -0500 (EST) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from manganese.network-alchemy.com (NAT-10-net.Network-Alchemy.COM [199.46.17.205]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA04666 for ; Fri, 31 Mar 2000 16:31:23 -0500 (EST) Received: from manganese.network-alchemy.com (localhost [127.0.0.1]) by manganese.network-alchemy.com (8.9.3/8.8.8) with ESMTP id NAA82663 for ; Fri, 31 Mar 2000 13:31:24 -0800 (PST) (envelope-from rdv@manganese.network-alchemy.com) Message-Id: <200003312131.NAA82663@manganese.network-alchemy.com> X-Mailer: exmh version 2.0.2 2/24/98 To: ips@ece.cmu.edu Subject: SCSI over ST Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 31 Mar 2000 13:31:24 -0800 From: Rod Van Meter Sender: owner-ips@ece.cmu.edu Precedence: bulk Not to derail the SCSI over TCP momentum here, but FYI, for those who don't follow the HiPPI reflector. Meeting sounds good, sorry I wasn't there. --Rod ------- Forwarded Message Return-Path: rdv Return-Path: Received: from SMTP.Network-Alchemy.COM (Hydrogen.Network-Alchemy.COM [199.46.17.130]) by Dysprosium.Network-Alchemy.COM (8.9.3/8.9.3) with SMTP id MAA15422 for ; Fri, 31 Mar 2000 12:42:13 -0800 (PST) (envelope-from hippi-owner@storage.network.com) Received: from tnt.isi.edu ([128.9.128.128]) by Hydrogen.Network-Alchemy.COM via INTERNET ; Fri, 31 Mar 2000 12:42:30 PST Received: from storage.network.com (storage.network.com [129.191.1.4]) by tnt.isi.edu (8.8.7/8.8.6) with ESMTP id MAA07393 for ; Fri, 31 Mar 2000 12:42:29 -0800 (PST) Received: (from majordom@localhost) by storage.network.com (8.9.3+Sun/8.9.3) id OAA13693 for hippi-outgoing; Fri, 31 Mar 2000 14:17:11 -0600 (CST) X-Authentication-Warning: storage.network.com: majordom set sender to owner-hippi@network.com using -f Message-ID: <013901bf9b4d$361eabb0$1b67aacf@parrot> From: "Roger Ronald" To: Subject: New release of SCSI on ST at web site Date: Fri, 31 Mar 2000 14:10:50 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2314.1300 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Sender: owner-hippi@storage.network.com Precedence: bulk Reply-To: "Roger Ronald" Status: * * From the hippi reflector, posted by: * "Roger Ronald" * www.hippi.org contains two locations where there's a hot new version of SCSI over ST. You can pick it off the "what's new" page or get it on the SCSI over ST document page. In any event, we'll be reviewing this document on 4/5 (next Wednesday) at 9 AM. Bring your copy, if you're coming! RR ------- End of Forwarded Message From XXX Sun Apr 16 14:23:17 2000 Return-Path: bassoon@ece.cmu.edu Delivery-Date: Sun Apr 16 05:22:32 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id FAA10951 for ; Sun, 16 Apr 2000 05:22:31 -0400 Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA15571 for ; Sun, 16 Apr 2000 05:22:30 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA63536 for ; Sun, 16 Apr 2000 11:21:58 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id LAA131932 for ; Sun, 16 Apr 2000 11:21:30 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568C3.00336467 ; Sun, 16 Apr 2000 11:21:20 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: undisclosed-recipients:; To: ips@ece.cmu.edu Message-ID: Date: Sun, 16 Apr 2000 11:21:14 +0200 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline After the BOF meeting at IETF-47 in Adelaide, expecting that a working group will be formed soon, several of us started working at a draft that should answer to some of the concerns raised at the meeting and should includes ideas developed in several organizations. The Internet Draft iSCSI is the starting point - but we expect the outcome to depart from it and include the insights gained by the teams that presented at IETF-47 as well as any other team that had some work in this domain and is willing to share it's thoughts with us. We will publish new versions regularly on this list (or at least point to them) for comments. To expedite the process the authors plan to held regular phone conferences. If any of you would like to participate - and that implies reading the document in detail, and commenting in writing in advance - please let us know (write to Daniel Smith and/or Julian Satran). We need to know this in advance as phone conferences are paid for a specific number of participants. Julo Julian Satran - IBM Research at Haifa From XXX Tue Apr 25 13:47:20 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Apr 25 09:46:14 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id JAA12184 for ; Tue, 25 Apr 2000 09:46:14 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA19343 for ips-outgoing; Tue, 25 Apr 2000 09:38:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: (from jmck@localhost) by ece.cmu.edu (8.9.2/8.8.8) id JAA19187 for ips; Tue, 25 Apr 2000 09:34:00 -0400 (EDT) Date: Tue, 25 Apr 2000 09:34:00 -0400 (EDT) From: Jim McKinney Message-Id: <200004251334.JAA19187@ece.cmu.edu> To: ips@ece.cmu.edu Subject: loop check message Sender: owner-ips@ece.cmu.edu Precedence: bulk As some of you know the ips@ece.cmu.edu list has sent multiple copies of messages to certain subscribers in the past. We believe that we have tracked down the source of this problem, but to check this we are sending this test message. Please accept our apologies for this extra mail. If you have any problems with this list in the future please feel free to contact me via gripe@ece.cmu.edu or jmck@ece.cmu.edu thanx -jmck -- Jim McKinney jmck@ece.cmu.edu System Administrator Manager, ECE Office: HH1304 Carnegie Mellon University (412) 268-5141 I quite often remember to forget such things. - Winnie the Pooh From XXX Fri Apr 28 12:32:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Apr 28 02:53:03 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id CAA09951 for ; Fri, 28 Apr 2000 02:53:02 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA29089 for ips-outgoing; Fri, 28 Apr 2000 02:49:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from seattle.3com.com (seattle.3com.com [129.213.128.97]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA29084 for ; Fri, 28 Apr 2000 02:49:35 -0400 (EDT) From: Dave_Lee@3com.com Received: from new-york.3com.com (new-york.3com.com [129.213.157.12]) by seattle.3com.com (8.8.8/8.8.8) with ESMTP id XAA01645; Thu, 27 Apr 2000 23:49:28 -0700 (PDT) Received: from hqoutbound.ops.3com.com (hqoutbound.OPS.3Com.COM [139.87.48.104]) by new-york.3com.com (8.8.8/8.8.8) with SMTP id XAA01780; Thu, 27 Apr 2000 23:49:25 -0700 (PDT) Received: by hqoutbound.ops.3com.com(Lotus SMTP MTA v4.6.3 (778.2 1-4-1999)) id 882568CE.005E4F9B ; Thu, 27 Apr 2000 10:10:07 -0700 X-Lotus-FromDomain: 3COM To: "Jon William Toigo" cc: ietf@ietf.org, ips@ece.cmu.edu Message-ID: <882568CE.005E4DE7.00@hqoutbound.ops.3com.com> Date: Thu, 27 Apr 2000 10:10:48 -0700 Subject: Re: SCSI over IP (fwd) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jon - Keep in mind that there is no standards track document - the document you have is merely a proposal by one set of parties. This effort is pre-WG (chairs, charters, and final approvals are not complete to my knowledge). There are a number of parties that would like to see a proposed standard in the shortest amount of time but it is a large area. Thus, it's too early to speculate on products (standards compatible products, at any rate) but I'd imagine that it'd be *at least* towards 1Q 2001before we get to proposed standard. Only one IP networking vendor (Cisco) that I'm aware of has made any public direction statements in this area, though there is certainly interest among others - more IP traffic is good for business, after all. Obviously, the storage networking industry is interested as well. Dave "Jon William Toigo" on 04/26/2000 08:33:18 AM Sent by: "Jon William Toigo" To: Dave Lee/HQ/3Com cc: ietf@ietf.org Subject: Re: SCSI over IP (fwd) Thanks for the referral, Dave. Now that I have the SCSI over IP RFC text, I am wondering how soon, if ever, we will likely see a standard and/or product based on the specification? To your knowledge, is SCSI over IP being pursued in earnest by IP networking equipment companies at present? JWT From XXX Thu May 04 02:05:13 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed May 3 06:47:59 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id GAA11116 for ; Wed, 3 May 2000 06:47:59 -0400 From: owner-ips@ece.cmu.edu Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA02497; Wed, 3 May 2000 06:47:58 -0400 (EDT) Date: Wed, 3 May 2000 06:47:58 -0400 (EDT) Message-Id: <200005031047.GAA02497@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f To: owner-ips@ece.cmu.edu Subject: BOUNCE ips@ece.cmu.edu: Admin request of type /\bsubscribe\b/i at line 8 >>From owner-ips@ece.cmu.edu Wed May 3 06:47:53 2000 Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA02492 for ; Wed, 3 May 2000 06:47:52 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id MAA31952; Wed, 3 May 2000 12:47:19 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id MAA33870; Wed, 3 May 2000 12:47:17 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568D4.003B4075 ; Wed, 3 May 2000 12:47:11 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Dave Thompson cc: ips@ece.cmu.edu Message-ID: Date: Wed, 3 May 2000 13:47:03 +0300 Subject: Re: IPS Working Group Meetings Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Dave, The working group has not been formally approved (or commissioned - I don't know what is the proper term for that) but we assume it will be. We will post soon an update of the document on the ips mailing list. If you want to spend more on this you can also subscribe to the scsi-tcp@external.cisco,com - a list mean more for the authors/editors but certainly open to the public (mail to mailer@cisco.com with subscribe scsi-tcp in the body. Regards, Julo Dave Thompson on 03/05/2000 13:16:33 Please respond to Dave Thompson To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: IPS Working Group Meetings Hi Julian, My name is Dave Thompson and I saw your e-mail on the IPS mailing list. I would like to get involved with the IPS Working Group and was wondering how that might be possible. I have signed up with the IPS mailing list but haven't seen much activity on it. I've also read the draft-satran-iSCSI-02 document. Has the working group officially been approved by the IETF yet? Thanks for your time, Dave Thompson NuSpeed, Inc. dave.thompson@nuspeed.com From XXX Thu May 04 02:26:58 2000 Replied: Wed, 03 May 2000 22:26:58 -0400 Replied: "julian_satran@il.ibm.com " Return-Path: bassoon@ece.cmu.edu Delivery-Date: Sun Apr 16 05:22:32 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id FAA10951 for ; Sun, 16 Apr 2000 05:22:31 -0400 Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA15571 for ; Sun, 16 Apr 2000 05:22:30 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA63536 for ; Sun, 16 Apr 2000 11:21:58 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m2/NCO v2.06) with SMTP id LAA131932 for ; Sun, 16 Apr 2000 11:21:30 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568C3.00336467 ; Sun, 16 Apr 2000 11:21:20 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: undisclosed-recipients:; To: ips@ece.cmu.edu Subject: IPS Working Group Meetings Message-ID: Date: Sun, 16 Apr 2000 11:21:14 +0200 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline After the BOF meeting at IETF-47 in Adelaide, expecting that a working group will be formed soon, several of us started working at a draft that should answer to some of the concerns raised at the meeting and should includes ideas developed in several organizations. The Internet Draft iSCSI is the starting point - but we expect the outcome to depart from it and include the insights gained by the teams that presented at IETF-47 as well as any other team that had some work in this domain and is willing to share it's thoughts with us. We will publish new versions regularly on this list (or at least point to them) for comments. To expedite the process the authors plan to held regular phone conferences. If any of you would like to participate - and that implies reading the document in detail, and commenting in writing in advance - please let us know (write to Daniel Smith and/or Julian Satran). We need to know this in advance as phone conferences are paid for a specific number of participants. Julo Julian Satran - IBM Research at Haifa From XXX Fri May 12 00:24:02 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu May 11 11:43:00 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA03288 for ; Thu, 11 May 2000 11:43:00 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA29280 for ips-outgoing; Thu, 11 May 2000 11:42:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA29271 for ; Thu, 11 May 2000 11:41:56 -0400 (EDT) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id IAA29666 for ; Thu, 11 May 2000 08:41:25 -0700 (PDT) Received: from aimexc01.corp.adaptec.com ([162.62.62.41]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id IAA07883 for ; Thu, 11 May 2000 08:41:24 -0700 (PDT) Received: by aimexc01.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Thu, 11 May 2000 08:41:27 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C15AEF3@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: "'ips@ece.cmu.edu'" Subject: RE: Regarding Adaptec's storage over Ethernet demonstration at N+ I Date: Thu, 11 May 2000 08:41:01 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The link to the SEP draft is http://www.ietf.org/internet-drafts/draft-wilson-sep-00.txt > > > Some of you may be wondering about Adaptec's announcement > regarding its > technology demonstration of storage over ethernet at N+I this week. > > http://www.adaptec.com/adaptec/press/release000504.html > > The demo shows a Gb ethernet controller transporting SCSI > traffic to several > targets through an off-the-shelf 100TX switch with a Gb > uplink. The targets > are ethernet to U160 SCSI bridges with one or more SCSI > drives attached. The > host controller runs under NT4.0 at appears to the OS as a > SCSI host bus > adapter. > > The architecture is based on Adaptec's SCSI Encapsulation > Protocol (SEP). > SEP is mapped on top of TCP/IP or a light-weight transport protocol > specifically designed for SANs. > > An SEP overview was presented at the IPS BOF in Adelaide last > month and an > internet draft on SEP was submitted to IETF this week. I will > forward the > link as soon as it becomes available. This draft is > informational only and > intended to aid in this group's work toward an industry standard SCSI > transport protocol over IP networks. > > I will continue to keep this group informed of any more > information as soon > as it becomes publicly available. > > Regards, > > Paul von Stamwitz > Adaptec, Inc. > paulv@corp.adaptec.com > From XXX Fri May 26 12:33:07 2000 Return-Path: owner-extdom.scsi-tcp@sj-msg-core-crit.cisco.com Delivery-Date: Fri May 26 01:02:58 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id BAA25900 for ; Fri, 26 May 2000 01:02:57 -0400 Received: from sj-msg-core-crit.cisco.com (sj-msg-core-crit.cisco.com [171.71.163.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA28847 for ; Fri, 26 May 2000 01:02:56 -0400 (EDT) Received: from proxy2.cisco.com (proxy2.cisco.com [192.31.7.89]) by sj-msg-core-crit.cisco.com (8.9.3/8.9.1) with SMTP id VAA02083 for ; Thu, 25 May 2000 21:57:16 -0700 (PDT) Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by proxy2.cisco.com with SMTP (MailShield v1.5); Thu, 25 May 2000 21:57:19 -0700 Received: from omgw1.boi.hp.com (omgw1.boi.hp.com [15.56.8.101]) by palrel3.hp.com (Postfix) with ESMTP id A57ABA35; Thu, 25 May 2000 21:56:47 -0700 (PDT) Received: from xboibrg1.boi.hp.com (xboibrg1.boi.hp.com [15.56.8.167]) by omgw1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id WAA20192; Thu, 25 May 2000 22:56:46 -0600 (MDT) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2650.21) id ; Thu, 25 May 2000 22:56:46 -0600 Message-ID: <91A1374F62F6D21198D400A0C9F2D42806EB730B@xrose01.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "'Dave Nagle'" Cc: "Scsi-Tcp (E-mail)" Subject: RE: IETF mailing list question on Storage over Ethernet/IP Date: Thu, 25 May 2000 22:56:44 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFC6CE.CA10E250" X-SMTP-HELO: palrel3.hp.com X-SMTP-MAIL-FROM: randy_haagens@hp.com X-SMAP-Received-From: outside X-SMTP-PEER-INFO: palrel3.hp.com [156.153.255.226] This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFC6CE.CA10E250 Content-Type: text/plain; charset="iso-8859-1" David, A few comments about this one. 1. FC does not provide reliable transmission. It provides for error detection, but escalates recovery to "upper level protocol". FCP-2 has improved this situation, but is not widely implemented yet. One of the advantages of using a transport such as TCP is that link errors will be corrected in a manner that is transparent to the application protocol (SCSI). 2. Jumbo frames will not be necessary when TCP is implemented in hardware. Most FC implementations use 1024 byte frames, and performance is very adequate, given hardware implementation of FCP. 3. The cost of using different transport protocols in the LAN and WAN is that the two will not interoperate. Many of us believe that TCP has proven itself in both the LAN and WAN. I bet your PC or UN*X workstation is using TCP for all its protocol needs. 4. The IPS working group is mapping SCSI to TCP. Another working group is mapping FC to IP. These are very different approaches. The first (ours) preserves SCSI, but does not include any vestige of Fibre Channel. It is intended for use in the LAN, MAN and WAN. Its best use is for connecting hosts computers to storage controllers using Ethernet and IP WAN technology. It will be possible, but non-trivial, to translate between SCSI over TCP/IP and SCSI over Fibrechannel. The second is a tunneling scheme for extending Fibre Channel over the IP WAN. It does not contemplate Ethernet-based hosts or storage controllers. 5. Just about any reliable transport will do nicely for transporting SCSI commands. We chose TCP because its implementation and behavior are well-known, and it is well-supported with load-balancing, QoS and security features. While another protocol (such as reliable datagram) might be arguably better suited to storage transport applications, we'll use TCP "because it's there". We'll have the benefit of all the other investment that's going into improving TCP for internet uses. Randy Haagens Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 > -----Original Message----- > From: Dave Nagle [mailto:bassoon@yogi.ece.cmu.edu] > Sent: Thursday, May 25, 2000 4:29 PM > To: SCSI-over-TCP List > Subject: IETF mailing list question on Storage over Ethernet/IP > > > > ------- Forwarded Message > > Date: Thu, 25 May 2000 19:27:02 -0400 > From: Dave Nagle > To: "Jon William Toigo" > cc: ietf@ietf.org > Subject: Re: Storage over Ethernet/IP > > Jon, > > Original Message > - ---------------- > >> I am seeking a few points of clarification: > >> > >> 1. Fibre Channel folks have attempted to explain to me > why TCP/IP could = > >> NEVER be a viable interconnect for block level storage > operations. They = > >> claim: > >> a. TCP is too CPU intensive and creates too much latency > for storage = > >> I/O operations. > >> b. The IP stack is too top heavy and processing packet > headers is too = > >> slow to support storage I/O operations. > > There is a lot of work to show that this is not true. Check out Van > Meter's 1998 ASPLOS paper "VISA - Netstations virtual internet SCSI > adaptor." > > Perhaps more importantly, there are many companies that are building > TCP in silicon ASICs. This should make TCP's performance comparable > to Fibre Channel. Both TCP/IP and FC provide about the same > functionality ... reliable, in-order transmission. > > The bottom line is that FC is done in hardware while TCP has > traditionally been done in software. Therefore, previous performance > numbers are not going to be fair. Once TCP is in silicon, its > performance should be roughly equal to FC. > > >> c. The maximum throughput of a GE TCP/IP connection is > 768 Mps, which = > >> is too slow to support storage I/O operations. > > I believe there are higher numbers (especially with Jumbo > Frames). Alteon's web site show's 920 Mbps. Microsoft and Duke > University have both shown TCP performance o 1Gb+/s performance over > other networks. > > BTW, why is 768 Mbps too slow for storage. Many apps (e.g., > transaction workloads) are I/O's per second bound, not bandwidth > bound. Also, even if storage over IP/ether is a bit slower than FC, > the benefits of leveraging IP's infrastructure (i.e., routers, > switches, NICs, network management, networking people) is a huge > advantage. > > There is also the issue of SCSI over TCP/IP in the SAN vs. the > LAN/WAN. Some companies, focusing on the SAN, are building > SCSI/lightweight transport/IP while others, focusing on the WAN, > propose SCSI/TCP/IP. It may be the case that SAN and WAN traffic use > different transport protocols to gain a bit of extra performance in > the SAN. > > >> Is any of this true? > >> > >> 2. Adaptec has posited a replacement for TCP called STP > for use as a = > >> transport for storage. Does anyone know anything about this? > > From Paul von Stamwitz's posting to the ips mailing list ... > > The link to the SEP draft is > http://www.ietf.org/internet-drafts/draft-wilson-sep-00.txt > > The press release is at: > http://www.adaptec.com/adaptec/press/release000504.html > > The demo shows a Gb ethernet controller transporting SCSI > traffic to several > targets through an off-the-shelf 100TX switch with a Gb > uplink. The targets > are ethernet to U160 SCSI bridges with one or more SCSI > drives attached. The > host controller runs under NT4.0 at appears to the OS as > a SCSI host bus > adapter. > > The architecture is based on Adaptec's SCSI Encapsulation Protocol > (SEP). SEP is mapped on top of TCP/IP or a light-weight transport > protocol specifically designed for SANs. > > An SEP overview was presented at the IPS BOF in Adelaide > last month and an > internet draft on SEP was submitted to IETF this week. I > will forward the > link as soon as it becomes available. This draft is informational > only and intended to aid in this group's work toward an industry > standard SCSI transport protocol over IP networks. > > > >> 3. Current discussions of the SCSI over IP protocol seem > to ignore the = > >> issue of TCP or any other transport protocol. Does anyone know = > >> definitively what transport is being suggested by the > IBM/Cisco crowd? > > Current SCSI over IP discussions are not ignoring TCP ... they are > definitely considering TCP as the primary transport. See the ips > web site at: > http://www.ece.cmu.edu/~ips >> >> 4. Another storage company is looking at Reliable UDP as a substitute = >> for TCP in storage data transfers. Where can I learn more about this = >> protocol, which I am told was introduced many years ago by Cisco? Companies to look at include: nishansystems.com interprophet.com san.com arkresearch.com Also, I believe that the IETF IP over FC working group is now looking at FC over IP. dave........... David Nagle Director, Parallel Data Lab Senior Reseach Computer Scientist School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 412-268-3898 (office) 412-268-3890 (fax) http://www.ece.cmu.edu/~bassoon ------- End of Forwarded Message ------_=_NextPart_000_01BFC6CE.CA10E250 Content-Type: application/octet-stream; name="Haagens, Randy (E-mail).vcf" Content-Disposition: attachment; filename="Haagens, Randy (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture TITLE:Manager, Storage Network Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-3586 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:19990528T091551Z END:VCARD ------_=_NextPart_000_01BFC6CE.CA10E250-- From XXX Fri May 26 12:51:26 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 08:50:14 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id IAA11571 for ; Fri, 26 May 2000 08:50:14 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA10704 for ips-outgoing; Fri, 26 May 2000 08:50:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA10700 for ; Fri, 26 May 2000 08:50:05 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id IAA11563 for ; Fri, 26 May 2000 08:50:04 -0400 Message-Id: <200005261250.IAA11563@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: "Jon William Toigo": Storage over Ethernet/IP Date: Fri, 26 May 2000 08:50:04 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk FYI ------- Forwarded Message Date: Thu, 25 May 2000 17:52:14 -0400 From: "Jon William Toigo" To: Subject: Storage over Ethernet/IP This is a multi-part message in MIME format. - ------=_NextPart_000_06CD_01BFC671.F56982C0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I am seeking a few points of clarification: 1. Fibre Channel folks have attempted to explain to me why TCP/IP could = NEVER be a viable interconnect for block level storage operations. They = claim: a. TCP is too CPU intensive and creates too much latency for storage = I/O operations. b. The IP stack is too top heavy and processing packet headers is too = slow to support storage I/O operations. c. The maximum throughput of a GE TCP/IP connection is 768 Mps, which = is too slow to support storage I/O operations. Is any of this true? 2. Adaptec has posited a replacement for TCP called STP for use as a = transport for storage. Does anyone know anything about this? 3. Current discussions of the SCSI over IP protocol seem to ignore the = issue of TCP or any other transport protocol. Does anyone know = definitively what transport is being suggested by the IBM/Cisco crowd? 4. Another storage company is looking at Reliable UDP as a substitute = for TCP in storage data transfers. Where can I learn more about this = protocol, which I am told was introduced many years ago by Cisco? Thanks in advance for your assistance. Jon William Toigo Independent Consultant and Author jtoigo@intnet.net - ------=_NextPart_000_06CD_01BFC671.F56982C0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
I am seeking a few points of=20 clarification:
 
1.  Fibre Channel folks have = attempted to=20 explain to me why TCP/IP could NEVER be a viable interconnect for block = level=20 storage operations.  They claim:
 
a.  TCP is too CPU intensive and = creates too=20 much latency for storage I/O operations.
 
b.  The IP stack is too top heavy = and=20 processing packet headers is too slow to support storage I/O=20 operations.
 
c.  The maximum throughput of a GE = TCP/IP=20 connection is 768 Mps, which is too slow to support storage I/O=20 operations.
 
Is any of this true?
 
2.  Adaptec has posited a = replacement for TCP=20 called STP for use as a transport for storage.  Does anyone know = anything=20 about this?
 
3.  Current discussions = of the SCSI over=20 IP protocol seem to ignore the issue of TCP or any other transport=20 protocol.  Does anyone know definitively what transport is being = suggested=20 by the IBM/Cisco crowd?
 
4.  Another storage company is = looking at=20 Reliable UDP as a substitute for TCP in storage data transfers.  = Where can=20 I learn more about this protocol, which I am told was introduced many = years ago=20 by Cisco?
 
Thanks in advance for your = assistance.
 
Jon William Toigo
Independent Consultant and = Author
jtoigo@intnet.net
 
 
- ------=_NextPart_000_06CD_01BFC671.F56982C0-- ------- End of Forwarded Message From XXX Fri May 26 12:53:23 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 08:51:36 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id IAA11598 for ; Fri, 26 May 2000 08:51:36 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA10729 for ips-outgoing; Fri, 26 May 2000 08:51:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA10724 for ; Fri, 26 May 2000 08:51:25 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id IAA11590 for ; Fri, 26 May 2000 08:51:24 -0400 Message-Id: <200005261251.IAA11590@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: RE: IETF mailing list question on Storage over Ethernet/IP Date: Fri, 26 May 2000 08:51:24 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk Date: Thu, 25 May 2000 21:38:11 -0700 From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "'Dave Nagle'" cc: "Scsi-Tcp (E-mail)" Subject: RE: IETF mailing list question on Storage over Ethernet/IP Comments - - -------- 1. I agree with your comments about TCP's being implemented in hardware. It will be as fast as any other protocol implemented in hardware. 2. Adaptec should speak for themselves; but I believe that the reference to STP is a misunderstanding. At the N+I conference, Adaptec demoed a software prototype of their SCSI Encapsulation Protocol (SEP). SEP allows SCSI to be transported over a lightweight protocol of Adaptec's own design for the the local area, or over TCP for the wide area. 3. The IP Storage Working Group (IBM, Cisco, HP, Adaptec, Quantum, EMC, and others) are working on a mapping of SCSI to TCP, for use both in the WAN and in the LAN. All of us agree on the use of TCP as the transport for the WAN and LAN, while a minority would probably favor using a lighter-weight transport for the LAN. In summary, TCP is suitable as the transport for the WAN and LAN, and it will be as fast as any protocol when implemented in hardware. Using a single transport for the WAN and LAN removes the artificial barrier between these two environments, and means that applications (like mirroring) can be designed to scale seamlessly from the local to the wide area. Randy Haagens Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 > -----Original Message----- > From: Dave Nagle [mailto:bassoon@yogi.ece.cmu.edu] > Sent: Thursday, May 25, 2000 4:28 PM > To: SCSI-over-TCP List > Subject: IETF mailing list question on Storage over Ethernet/IP > > > > > ------- Forwarded Message > > Date: Thu, 25 May 2000 22:55:49 -0000 > From: Mike Fisk > To: Jon William Toigo > cc: ietf@ietf.org > Subject: Re: Storage over Ethernet/IP > > On Thu, 25 May 2000, Jon William Toigo wrote: > > > I am seeking a few points of clarification: > > > > 1. Fibre Channel folks have attempted to explain to me why TCP/IP > > could NEVER be a viable interconnect for block level storage > > operations. They claim: > > > > a. TCP is too CPU intensive and creates too much latency > for storage > > I/O operations. > > > > b. The IP stack is too top heavy and processing packet > headers is too > > slow to support storage I/O operations. > > > > c. The maximum throughput of a GE TCP/IP connection is 768 > Mps, which > > is too slow to support storage I/O operations. > > This is not a theoretical limitation, but is in the ballpark > reported by > many general-purpose operating systems with commodity hardware. > > >Is any of this true? > > I don't believe that TCP/IP implementations couldn't be optimized to > support full link rate and low latency. If you're building a hardware > adapter that can do SCSI and RAID fast, adding TCP shouldn't be > prohibitively hard. > > > 2. Adaptec has posited a replacement for TCP called STP > for use as a > > transport for storage. Does anyone know anything about this? > > STP is the Scheduled Transfer protocol being standardized by > the ANSI T11 > folks. ST was designed to run on top of GSN (a.k.a. > HIPPI-6400). In my > opinion, it is as heavy-weight as TCP with respect to most of > the things > stated above. It does have the potential advantage of being > designed from > scratch to support zero-copy access to user space using specialized > interface cards. > > > 3. Current discussions of the SCSI over IP protocol seem to ignore > > the issue of TCP or any other transport protocol. Does anyone know > > definitively what transport is being suggested by the > IBM/Cisco crowd? > > I believe the assumption is that you will have a local network with no > packet loss or significant bit error rate. Basically, you assume that > your ethernet is as reliable as your SCSI cable or > fiber-channel network. > For a well engineered, fully-switched LAN, that may be a reasonable > assumption. > > - -- Mike Fisk, RADIANT Team, Network Engineering Group, Los > Alamos National > Lab See http://home.lanl.gov/mfisk/ for contact information > > > ------- End of Forwarded Message > > ------- End of Forwarded Message From XXX Fri May 26 14:29:53 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 10:21:48 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id KAA23930 for ; Fri, 26 May 2000 10:21:48 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA12409 for ips-outgoing; Fri, 26 May 2000 10:21:43 -0400 (EDT) Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA12404 for ; Fri, 26 May 2000 10:21:40 -0400 (EDT) Received: from mwebster.rose.hp.com (mwebster.rose.hp.com [15.3.107.68]) by atlrel2.hp.com (Postfix) with ESMTP id B9F03941 for ; Fri, 26 May 2000 10:21:39 -0400 (EDT) Received: from localhost (mwebster@localhost) by mwebster.rose.hp.com (8.9.3/8.9.3) with ESMTP id GAA17167 for ; Fri, 26 May 2000 06:21:00 -0700 X-Authentication-Warning: mwebster.rose.hp.com: mwebster owned process doing -bs Date: Fri, 26 May 2000 06:21:00 -0700 (PDT) From: Matt Webster X-Sender: mwebster@mwebster.rose.hp.com To: ips@ece.cmu.edu Subject: Re: IETF mailing list question on Storage over Ethernet/IP In-Reply-To: <200005261242.IAA11351@yogi.ece.cmu.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk I take issue with FC not providing reliable transmissions. FC establishes a dedicated link between the host and target, avoiding congestion once the channel is established. TCP/IP expects packet dropping... this in storage? Matt Webster On Fri, 26 May 2000, Dave Nagle wrote: > > A few comments about this one. > > 1. FC does not provide reliable transmission. It provides for error > detection, but escalates recovery to "upper level protocol". FCP-2 has > improved this situation, but is not widely implemented yet. One of the > advantages of using a transport such as TCP is that link errors will be > corrected in a manner that is transparent to the application protocol > (SCSI). > > . > . > . > > Randy Haagens > Networked Storage Architecture > Storage Organization > Hewlett-Packard Co. > e-mail: Randy_Haagens@hp.com > tel: +1 916 785 4578 > fax: +1 916 785 1911 From XXX Fri May 26 14:42:12 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 10:31:50 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id KAA24104 for ; Fri, 26 May 2000 10:31:50 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA12601 for ips-outgoing; Fri, 26 May 2000 10:31:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from c014.sfo.cp.net (c014-h014.c014.sfo.cp.net [209.228.12.78]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id KAA12596 for ; Fri, 26 May 2000 10:31:30 -0400 (EDT) From: narakamath@lightel.com Received: (cpmta 20698 invoked from network); 26 May 2000 07:30:59 -0700 Date: 26 May 2000 07:30:59 -0700 Message-ID: <20000526143059.20697.cpmta@c014.sfo.cp.net> X-Sent: 26 May 2000 14:30:59 GMT Received: from [198.102.173.112] by mail.lightel.com with HTTP; 26 May 2000 07:30:58 PDT Content-Type: text/plain Content-Disposition: inline Mime-Version: 1.0 To: bassoon@yogi.ece.cmu.edu Cc: jtoigo@INTNET.NET, ietf@ietf.org, ips@ece.cmu.edu X-Mailer: Web Mail 3.6.3.1 Subject: Re: IETF mailing list question on Storage over Ethernet/IP Sender: owner-ips@ece.cmu.edu Precedence: bulk Thank you all for some excellent discussions on the subject. For all the reasons mentioned, I am mapping TCP/IP to DWDM to create very high performance SANs. I hope to share more on this as we progress and launch products. You can see more on this at www.lightel.com. On Fri, 26 May 2000, Dave Nagle wrote: > > > A few comments about this one. > > 1. FC does not provide reliable transmission. It provides for error > detection, but escalates recovery to "upper level protocol". FCP-2 has > improved this situation, but is not widely implemented yet. One of the > advantages of using a transport such as TCP is that link errors will be > corrected in a manner that is transparent to the application protocol > (SCSI). > > 2. Jumbo frames will not be necessary when TCP is implemented in hardware. > Most FC implementations use 1024 byte frames, and performance is very > adequate, given hardware implementation of FCP. > > 3. The cost of using different transport protocols in the LAN and WAN is > that the two will not interoperate. Many of us believe that TCP has proven > itself in both the LAN and WAN. I bet your PC or UN*X workstation is using > TCP for all its protocol needs. > > 4. The IPS working group is mapping SCSI to TCP. Another working group is > mapping FC to IP. These are very different approaches. The first (ours) > preserves SCSI, but does not include any vestige of Fibre Channel. It is > intended for use in the LAN, MAN and WAN. Its best use is for connecting > hosts computers to storage controllers using Ethernet and IP WAN technology. > It will be possible, but non-trivial, to translate between SCSI over TCP/IP > and SCSI over Fibrechannel. The second is a tunneling scheme for extending > Fibre Channel over the IP WAN. It does not contemplate Ethernet-based hosts > or storage controllers. > > 5. Just about any reliable transport will do nicely for transporting SCSI > commands. We chose TCP because its implementation and behavior are > well-known, and it is well-supported with load-balancing, QoS and security > features. While another protocol (such as reliable datagram) might be > arguably better suited to storage transport applications, we'll use TCP > "because it's there". We'll have the benefit of all the other investment > that's going into improving TCP for internet uses. > > Randy Haagens > Networked Storage Architecture > Storage Organization > Hewlett-Packard Co. > e-mail: Randy_Haagens@hp.com > tel: +1 916 785 4578 > fax: +1 916 785 1911 > > > > -----Original Message----- > > From: Dave Nagle [mailto:bassoon@yogi.ece.cmu.edu] > > Sent: Thursday, May 25, 2000 4:29 PM > > To: SCSI-over-TCP List > > Subject: IETF mailing list question on Storage over Ethernet/IP > > > > > > > > ------- Forwarded Message > > > > Date: Thu, 25 May 2000 19:27:02 -0400 > > From: Dave Nagle > > To: "Jon William Toigo" > > cc: ietf@ietf.org > > Subject: Re: Storage over Ethernet/IP > > > > Jon, > > > > Original Message > > - ---------------- > > >> I am seeking a few points of clarification: > > >> > > >> 1. Fibre Channel folks have attempted to explain to me > > why TCP/IP could = > > >> NEVER be a viable interconnect for block level storage > > operations. They = > > >> claim: > > >> a. TCP is too CPU intensive and creates too much latency > > for storage = > > >> I/O operations. > > >> b. The IP stack is too top heavy and processing packet > > headers is too = > > >> slow to support storage I/O operations. > > > > There is a lot of work to show that this is not true. Check out Van > > Meter's 1998 ASPLOS paper "VISA - Netstations virtual internet SCSI > > adaptor." > > > > Perhaps more importantly, there are many companies that are building > > TCP in silicon ASICs. This should make TCP's performance comparable > > to Fibre Channel. Both TCP/IP and FC provide about the same > > functionality ... reliable, in-order transmission. > > > > The bottom line is that FC is done in hardware while TCP has > > traditionally been done in software. Therefore, previous performance > > numbers are not going to be fair. Once TCP is in silicon, its > > performance should be roughly equal to FC. > > > > >> c. The maximum throughput of a GE TCP/IP connection is > > 768 Mps, which = > > >> is too slow to support storage I/O operations. > > > > I believe there are higher numbers (especially with Jumbo > > Frames). Alteon's web site show's 920 Mbps. Microsoft and Duke > > University have both shown TCP performance o 1Gb+/s performance over > > other networks. > > > > BTW, why is 768 Mbps too slow for storage. Many apps (e.g., > > transaction workloads) are I/O's per second bound, not bandwidth > > bound. Also, even if storage over IP/ether is a bit slower than FC, > > the benefits of leveraging IP's infrastructure (i.e., routers, > > switches, NICs, network management, networking people) is a huge > > advantage. > > > > There is also the issue of SCSI over TCP/IP in the SAN vs. the > > LAN/WAN. Some companies, focusing on the SAN, are building > > SCSI/lightweight transport/IP while others, focusing on the WAN, > > propose SCSI/TCP/IP. It may be the case that SAN and WAN traffic use > > different transport protocols to gain a bit of extra performance in > > the SAN. > > > > >> Is any of this true? > > >> > > >> 2. Adaptec has posited a replacement for TCP called STP > > for use as a = > > >> transport for storage. Does anyone know anything about this? > > > > From Paul von Stamwitz's posting to the ips mailing list ... > > > > The link to the SEP draft is > > http://www.ietf.org/internet-drafts/draft-wilson-sep-00.txt > > > > The press release is at: > > http://www.adaptec.com/adaptec/press/release000504.html > > > > The demo shows a Gb ethernet controller transporting SCSI > > traffic to several > > targets through an off-the-shelf 100TX switch with a Gb > > uplink. The targets > > are ethernet to U160 SCSI bridges with one or more SCSI > > drives attached. The > > host controller runs under NT4.0 at appears to the OS as > > a SCSI host bus > > adapter. > > > > The architecture is based on Adaptec's SCSI Encapsulation Protocol > > (SEP). SEP is mapped on top of TCP/IP or a light-weight transport > > protocol specifically designed for SANs. > > > > An SEP overview was presented at the IPS BOF in Adelaide > > last month and an > > internet draft on SEP was submitted to IETF this week. I > > will forward the > > link as soon as it becomes available. This draft is informational > > only and intended to aid in this group's work toward an industry > > standard SCSI transport protocol over IP networks. > > > > > > >> 3. Current discussions of the SCSI over IP protocol seem > > to ignore the = > > >> issue of TCP or any other transport protocol. Does anyone know = > > >> definitively what transport is being suggested by the > > IBM/Cisco crowd? > > > > Current SCSI over IP discussions are not ignoring TCP ... they are > > definitely considering TCP as the primary transport. See the ips > > web site at: > > > http://www.ece.cmu.edu/~ips > > >> > >> 4. Another storage company is looking at Reliable UDP as a substitute = > >> for TCP in storage data transfers. Where can I learn more about this = > >> protocol, which I am told was introduced many years ago by Cisco? > > Companies to look at include: > > nishansystems.com > interprophet.com > san.com > arkresearch.com > > Also, I believe that the IETF IP over FC working group is now > looking at FC over IP. > > > > dave........... > > David Nagle > Director, Parallel Data Lab > Senior Reseach Computer Scientist > School of Computer Science > Carnegie Mellon University > Pittsburgh, PA 15213 > 412-268-3898 (office) > 412-268-3890 (fax) > http://www.ece.cmu.edu/~bassoon > > > - ------- End of Forwarded Message From XXX Fri May 26 14:49:13 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 10:48:47 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id KAA24357 for ; Fri, 26 May 2000 10:48:47 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA13000 for ips-outgoing; Fri, 26 May 2000 10:48:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA12995 for ; Fri, 26 May 2000 10:48:38 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA33774 for ; Fri, 26 May 2000 16:48:04 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id QAA209806 for ; Fri, 26 May 2000 16:48:00 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568EB.00514970 ; Fri, 26 May 2000 16:47:52 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 26 May 2000 17:47:20 +0300 Subject: Re: IETF mailing list question on Storage over Ethernet/IP Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, FC is a class of protocols. FCP provides reliable communication and so does TCP. The congestion control mechanisms of TCP are adequate (not the best) for LAN and good for WAN. FCP congestion control is good for lan and quite bad for WAN. The only FCP to FCP solutions we see out there are through a TCP extender. But for the basic application (SAN) they are technologically equivalent. The question raised usually is that if FCP is not superior than why have it? Regards, Julo Matt Webster on 26/05/2000 16:21:00 Please respond to Matt Webster To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: IETF mailing list question on Storage over Ethernet/IP I take issue with FC not providing reliable transmissions. FC establishes a dedicated link between the host and target, avoiding congestion once the channel is established. TCP/IP expects packet dropping... this in storage? Matt Webster On Fri, 26 May 2000, Dave Nagle wrote: > > A few comments about this one. > > 1. FC does not provide reliable transmission. It provides for error > detection, but escalates recovery to "upper level protocol". FCP-2 has > improved this situation, but is not widely implemented yet. One of the > advantages of using a transport such as TCP is that link errors will be > corrected in a manner that is transparent to the application protocol > (SCSI). > > . > . > . > > Randy Haagens > Networked Storage Architecture > Storage Organization > Hewlett-Packard Co. > e-mail: Randy_Haagens@hp.com > tel: +1 916 785 4578 > fax: +1 916 785 1911 From XXX Fri May 26 14:49:13 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 10:47:40 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id KAA24341 for ; Fri, 26 May 2000 10:47:39 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA12953 for ips-outgoing; Fri, 26 May 2000 10:47:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA12946 for ; Fri, 26 May 2000 10:47:13 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id KAA24335 for ; Fri, 26 May 2000 10:47:12 -0400 Message-Id: <200005261447.KAA24335@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: RE: IETF mailing list question on Storage over Ethernet/IP Date: Fri, 26 May 2000 10:47:12 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk Date: Fri, 26 May 2000 09:43:56 -0500 From: "Don Woelz" To: "Dave Nagle" , "SCSI-over-TCP List" Subject: RE: IETF mailing list question on Storage over Ethernet/IP > > 2. Adaptec has posited a replacement for TCP called STP for use as a > > transport for storage. Does anyone know anything about this? > > STP is the Scheduled Transfer protocol being standardized by the ANSI T11 > folks. ST was designed to run on top of GSN (a.k.a. HIPPI-6400). In my > opinion, it is as heavy-weight as TCP with respect to most of the things > stated above. It does have the potential advantage of being designed from > scratch to support zero-copy access to user space using specialized > interface cards. I believe Adaptec's entry is called SEP. Dave, I'd like to clarify your STP comments. STP runs on top of 10Base-T, 100Base-T, Gigabit Ethernet, HIPPI-800, GSN, and will probably be mapped to more physical variants in the near future. It was designed for GSN only because a more efficient protocol was required to run at GSN speeds. It was designed to be used on almost any network media. Although a very rich protocol, it is also quite performant with very low latency and high bandwidth characteristics while maintaining low CPU overhead. Interestingly, STP can be transported on IP... There is also a draft ANSI standard for mapping SCSI on Scheduled Transfer called SST. The most recent copy can be found at http://www.t10.org/drafts.htm. SST has a demonstrated ability to transfer storage data on a GSN link at close to 720 megaBYTES per second. Data rates approaching those of Fibre Channel have been obtained running SST on GigE. Don Donald D. Woelz Vice Chair of ANSI T11.1 GENROCO, Inc. 255 Info Hwy Slinger, WI 53086 Tel: 262-644-2505 ------- End of Forwarded Message From XXX Fri May 26 16:47:10 2000 Return-Path: owner-ietf-outbound@ietf.org Delivery-Date: Fri May 26 11:19:14 2000 Return-Path: Received: from ietf.org (odin.ietf.org [132.151.1.176]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA30949 for ; Fri, 26 May 2000 11:19:14 -0400 Received: by ietf.org (8.9.1a/8.9.1a) id KAA16672 for ietf-outbound.06@ietf.org; Fri, 26 May 2000 10:40:01 -0400 (EDT) Received: from c014.sfo.cp.net (c014-h014.c014.sfo.cp.net [209.228.12.78]) by ietf.org (8.9.1a/8.9.1a) with SMTP id KAA16432 for ; Fri, 26 May 2000 10:31:29 -0400 (EDT) From: narakamath@lightel.com Received: (cpmta 20698 invoked from network); 26 May 2000 07:30:59 -0700 Date: 26 May 2000 07:30:59 -0700 Message-ID: <20000526143059.20697.cpmta@c014.sfo.cp.net> X-Sent: 26 May 2000 14:30:59 GMT Received: from [198.102.173.112] by mail.lightel.com with HTTP; 26 May 2000 07:30:58 PDT Content-Type: text/plain Content-Disposition: inline Mime-Version: 1.0 To: bassoon@YOGI.ECE.CMU.EDU Cc: jtoigo@INTNET.NET, ietf@ietf.org, ips@ece.cmu.edu X-Mailer: Web Mail 3.6.3.1 Subject: Re: IETF mailing list question on Storage over Ethernet/IP X-Loop: ietf@ietf.org Thank you all for some excellent discussions on the subject. For all the reasons mentioned, I am mapping TCP/IP to DWDM to create very high performance SANs. I hope to share more on this as we progress and launch products. You can see more on this at www.lightel.com. On Fri, 26 May 2000, Dave Nagle wrote: > > > A few comments about this one. > > 1. FC does not provide reliable transmission. It provides for error > detection, but escalates recovery to "upper level protocol". FCP-2 has > improved this situation, but is not widely implemented yet. One of the > advantages of using a transport such as TCP is that link errors will be > corrected in a manner that is transparent to the application protocol > (SCSI). > > 2. Jumbo frames will not be necessary when TCP is implemented in hardware. > Most FC implementations use 1024 byte frames, and performance is very > adequate, given hardware implementation of FCP. > > 3. The cost of using different transport protocols in the LAN and WAN is > that the two will not interoperate. Many of us believe that TCP has proven > itself in both the LAN and WAN. I bet your PC or UN*X workstation is using > TCP for all its protocol needs. > > 4. The IPS working group is mapping SCSI to TCP. Another working group is > mapping FC to IP. These are very different approaches. The first (ours) > preserves SCSI, but does not include any vestige of Fibre Channel. It is > intended for use in the LAN, MAN and WAN. Its best use is for connecting > hosts computers to storage controllers using Ethernet and IP WAN technology. > It will be possible, but non-trivial, to translate between SCSI over TCP/IP > and SCSI over Fibrechannel. The second is a tunneling scheme for extending > Fibre Channel over the IP WAN. It does not contemplate Ethernet-based hosts > or storage controllers. > > 5. Just about any reliable transport will do nicely for transporting SCSI > commands. We chose TCP because its implementation and behavior are > well-known, and it is well-supported with load-balancing, QoS and security > features. While another protocol (such as reliable datagram) might be > arguably better suited to storage transport applications, we'll use TCP > "because it's there". We'll have the benefit of all the other investment > that's going into improving TCP for internet uses. > > Randy Haagens > Networked Storage Architecture > Storage Organization > Hewlett-Packard Co. > e-mail: Randy_Haagens@hp.com > tel: +1 916 785 4578 > fax: +1 916 785 1911 > > > > -----Original Message----- > > From: Dave Nagle [mailto:bassoon@yogi.ece.cmu.edu] > > Sent: Thursday, May 25, 2000 4:29 PM > > To: SCSI-over-TCP List > > Subject: IETF mailing list question on Storage over Ethernet/IP > > > > > > > > ------- Forwarded Message > > > > Date: Thu, 25 May 2000 19:27:02 -0400 > > From: Dave Nagle > > To: "Jon William Toigo" > > cc: ietf@ietf.org > > Subject: Re: Storage over Ethernet/IP > > > > Jon, > > > > Original Message > > - ---------------- > > >> I am seeking a few points of clarification: > > >> > > >> 1. Fibre Channel folks have attempted to explain to me > > why TCP/IP could = > > >> NEVER be a viable interconnect for block level storage > > operations. They = > > >> claim: > > >> a. TCP is too CPU intensive and creates too much latency > > for storage = > > >> I/O operations. > > >> b. The IP stack is too top heavy and processing packet > > headers is too = > > >> slow to support storage I/O operations. > > > > There is a lot of work to show that this is not true. Check out Van > > Meter's 1998 ASPLOS paper "VISA - Netstations virtual internet SCSI > > adaptor." > > > > Perhaps more importantly, there are many companies that are building > > TCP in silicon ASICs. This should make TCP's performance comparable > > to Fibre Channel. Both TCP/IP and FC provide about the same > > functionality ... reliable, in-order transmission. > > > > The bottom line is that FC is done in hardware while TCP has > > traditionally been done in software. Therefore, previous performance > > numbers are not going to be fair. Once TCP is in silicon, its > > performance should be roughly equal to FC. > > > > >> c. The maximum throughput of a GE TCP/IP connection is > > 768 Mps, which = > > >> is too slow to support storage I/O operations. > > > > I believe there are higher numbers (especially with Jumbo > > Frames). Alteon's web site show's 920 Mbps. Microsoft and Duke > > University have both shown TCP performance o 1Gb+/s performance over > > other networks. > > > > BTW, why is 768 Mbps too slow for storage. Many apps (e.g., > > transaction workloads) are I/O's per second bound, not bandwidth > > bound. Also, even if storage over IP/ether is a bit slower than FC, > > the benefits of leveraging IP's infrastructure (i.e., routers, > > switches, NICs, network management, networking people) is a huge > > advantage. > > > > There is also the issue of SCSI over TCP/IP in the SAN vs. the > > LAN/WAN. Some companies, focusing on the SAN, are building > > SCSI/lightweight transport/IP while others, focusing on the WAN, > > propose SCSI/TCP/IP. It may be the case that SAN and WAN traffic use > > different transport protocols to gain a bit of extra performance in > > the SAN. > > > > >> Is any of this true? > > >> > > >> 2. Adaptec has posited a replacement for TCP called STP > > for use as a = > > >> transport for storage. Does anyone know anything about this? > > > > From Paul von Stamwitz's posting to the ips mailing list ... > > > > The link to the SEP draft is > > http://www.ietf.org/internet-drafts/draft-wilson-sep-00.txt > > > > The press release is at: > > http://www.adaptec.com/adaptec/press/release000504.html > > > > The demo shows a Gb ethernet controller transporting SCSI > > traffic to several > > targets through an off-the-shelf 100TX switch with a Gb > > uplink. The targets > > are ethernet to U160 SCSI bridges with one or more SCSI > > drives attached. The > > host controller runs under NT4.0 at appears to the OS as > > a SCSI host bus > > adapter. > > > > The architecture is based on Adaptec's SCSI Encapsulation Protocol > > (SEP). SEP is mapped on top of TCP/IP or a light-weight transport > > protocol specifically designed for SANs. > > > > An SEP overview was presented at the IPS BOF in Adelaide > > last month and an > > internet draft on SEP was submitted to IETF this week. I > > will forward the > > link as soon as it becomes available. This draft is informational > > only and intended to aid in this group's work toward an industry > > standard SCSI transport protocol over IP networks. > > > > > > >> 3. Current discussions of the SCSI over IP protocol seem > > to ignore the = > > >> issue of TCP or any other transport protocol. Does anyone know = > > >> definitively what transport is being suggested by the > > IBM/Cisco crowd? > > > > Current SCSI over IP discussions are not ignoring TCP ... they are > > definitely considering TCP as the primary transport. See the ips > > web site at: > > > http://www.ece.cmu.edu/~ips > > >> > >> 4. Another storage company is looking at Reliable UDP as a substitute = > >> for TCP in storage data transfers. Where can I learn more about this = > >> protocol, which I am told was introduced many years ago by Cisco? > > Companies to look at include: > > nishansystems.com > interprophet.com > san.com > arkresearch.com > > Also, I believe that the IETF IP over FC working group is now > looking at FC over IP. > > > > dave........... > > David Nagle > Director, Parallel Data Lab > Senior Reseach Computer Scientist > School of Computer Science > Carnegie Mellon University > Pittsburgh, PA 15213 > 412-268-3898 (office) > 412-268-3890 (fax) > http://www.ece.cmu.edu/~bassoon > > > - ------- End of Forwarded Message From XXX Fri May 26 16:49:27 2000 To: ips@ece.cmu.edu Subject: Keith Moore: Re: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:49:27 -0400 From: Dave Nagle ------- Forwarded Message Date: Fri, 26 May 2000 10:23:06 -0400 From: Keith Moore To: "Jon William Toigo" cc: ietf@ietf.org Subject: Re: Storage over Ethernet/IP None of the cited limitations of TCP performance are true. they are also missing the point. If you're going to run storage access over IP then you are potentially allowing it to be run over the global Internet. If you do that you need good authentication and privacy, and (if you try to do them in software) authentication and encyption will eat far more in performance than anything inherent to TCP or IP. and no, it's not acceptable to assume that the storage device will be behind a firewall. Keith ------- End of Forwarded Message From XXX Fri May 26 16:50:01 2000 To: ips@ece.cmu.edu Subject: Brian.Rubarts@born.com: RE: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:50:01 -0400 From: Dave Nagle ------- Forwarded Message Date: Fri, 26 May 2000 09:40:28 -0500 From: Brian.Rubarts@born.com To: moore@cs.utk.edu cc: ietf@ietf.org Subject: RE: Storage over Ethernet/IP Encryption will be offloaded to the network interface. ASICs on the NICs will greatly improve encryption and authentication performance. It won't run over the Internet because of latencies inherent on the public network. It will run over incredibly fast Packet over SONET Wide Area Networks--behind firewalls. OC-192 and soon-to-come OC-768 (terrabit) switches will be the backbone of WANs and MANs of large networks in a few years. Also, IPv6 has significantly improved authentication capabilities that will help ensure security on the Storage WAN (SWAN?) Brian - -----Original Message----- From: Keith Moore [mailto:moore@cs.utk.edu] Sent: Friday, May 26, 2000 9:23 AM To: Jon William Toigo Cc: ietf@ietf.org Subject: Re: Storage over Ethernet/IP None of the cited limitations of TCP performance are true. they are also missing the point. If you're going to run storage access over IP then you are potentially allowing it to be run over the global Internet. If you do that you need good authentication and privacy, and (if you try to do them in software) authentication and encyption will eat far more in performance than anything inherent to TCP or IP. and no, it's not acceptable to assume that the storage device will be behind a firewall. Keith ------- End of Forwarded Message From XXX Fri May 26 16:51:19 2000 To: ips@ece.cmu.edu Subject: Keith Moore: Re: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:51:19 -0400 From: Dave Nagle ------- Forwarded Message Date: Fri, 26 May 2000 11:01:21 -0400 From: Keith Moore To: Brian.Rubarts@born.com cc: moore@cs.utk.edu, ietf@ietf.org Subject: Re: Storage over Ethernet/IP > Encryption will be offloaded to the network interface. ASICs on the NICs > will greatly improve encryption and authentication performance. all well and good, provided that this encryption and authentication are actually compatible with that specified by higher level protocols and the authentication actually meets the needs of users. (if your network interface needs to use and verify users' credentials, as opposed to the host's credentials, it might be a stretch.) > It won't run over the Internet because of latencies inherent on the > public network. at least for some storage applications, latency is not as important as bandwidth. e.g. you can do backups over a high-latency medium as long as your bandwidth is adequate (though recovery from write errors gets a bit tricky). > It will run over incredibly fast Packet over SONET Wide Area > Networks--behind firewalls. I'm sure it will be used behind firewalls in some cases but it's inappropriate to assume that it will always be used behind firewalls. Firewalls don't help with the majority of security threats, and it's less and less the case that network topologies reflect trust domains. Keith ------- End of Forwarded Message From XXX Fri May 26 16:51:35 2000 To: ips@ece.cmu.edu Subject: Brian.Rubarts@born.com: RE: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:51:35 -0400 From: Dave Nagle ------- Forwarded Message Date: Fri, 26 May 2000 10:14:03 -0500 From: Brian.Rubarts@born.com To: moore@cs.utk.edu cc: ietf@ietf.org Subject: RE: Storage over Ethernet/IP >> Encryption will be offloaded to the network interface. ASICs on the NICs >> will greatly improve encryption and authentication performance. >all well and good, provided that this encryption and authentication >are actually compatible with that specified by higher level protocols >and the authentication actually meets the needs of users. >(if your network interface needs to use and verify users' credentials, >as opposed to the host's credentials, it might be a stretch.) A network server will still authenticate user requests. Only the host needs to be authenticated with the disk/disks. >> It won't run over the Internet because of latencies inherent on the >> public network. >at least for some storage applications, latency is not as important >as bandwidth. e.g. you can do backups over a high-latency medium >as long as your bandwidth is adequate (though recovery from write >errors gets a bit tricky). Backups could go through VPNs, I suppose. Good point. That would free your WAN of the backup jobs. I wasn't thinking of backups when I ruled out the Internet as a disk I/O medium. I suppose infrequently used and low priority files could also be accessed over the 'net. >> It will run over incredibly fast Packet over SONET Wide Area >> Networks--behind firewalls. >...it's >inappropriate to assume that it will always be used behind firewalls... If the larger network that is employing this technology doesn't hire a decent consultant, you might be right. If they do, it will ALWAYS be behind a firewall :-) >Firewalls don't help with the majority of security threats... True, but whether the server accesses the disks via SCSI over TCP or SCSI over Fibre Channel, the SERVER is still the weak link. The transport protocol doesn't create any inherent weaknesses of the type you are refering to--e-mail borne viruses, internal hackers, etc. The server would still be the attack point. Why goodness, the server and storage devices could be in a VLAN or something to deny direct hack attempts against the storage device, but the chink in the armor is how hardened is your OS? Brian ------- End of Forwarded Message From XXX Fri May 26 16:52:46 2000 To: ips@ece.cmu.edu Subject: Karl Auerbach: Re: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:52:46 -0400 From: Dave Nagle ------- Forwarded Message Date: Fri, 26 May 2000 08:32:05 -0700 From: Karl Auerbach To: Jon William Toigo cc: ietf@ietf.org Subject: Re: Storage over Ethernet/IP > a. TCP is too CPU intensive and creates too much latency for storage I/O operations. > > b. The IP stack is too top heavy and processing packet headers is too > slow to support storage I/O operations. There were some papers published duing the late '80's or early '90s by John Romkey and I belive Dave Clark and Van Jacobson about the length of instruction sequences to handle TCP. I'm not sure that those ever became RFCs. Those papers came up with figures indicating that if one structures code "correctly" and if the net path is "clean" (i.e. not a lot of packet loss, reordering, replication, etc) than the per-packet instruction sequences (sans IP checksum calculation) were potantially very short. Does anyone have the references to these papers? --karl-- ------- End of Forwarded Message From XXX Fri May 26 16:52:47 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 12:51:27 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA06019 for ; Fri, 26 May 2000 12:51:27 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16368 for ips-outgoing; Fri, 26 May 2000 12:51:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16363 for ; Fri, 26 May 2000 12:51:20 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA06013 for ; Fri, 26 May 2000 12:51:19 -0400 Message-Id: <200005261651.MAA06013@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Keith Moore: Re: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:51:19 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 11:01:21 -0400 From: Keith Moore To: Brian.Rubarts@born.com cc: moore@cs.utk.edu, ietf@ietf.org Subject: Re: Storage over Ethernet/IP > Encryption will be offloaded to the network interface. ASICs on the NICs > will greatly improve encryption and authentication performance. all well and good, provided that this encryption and authentication are actually compatible with that specified by higher level protocols and the authentication actually meets the needs of users. (if your network interface needs to use and verify users' credentials, as opposed to the host's credentials, it might be a stretch.) > It won't run over the Internet because of latencies inherent on the > public network. at least for some storage applications, latency is not as important as bandwidth. e.g. you can do backups over a high-latency medium as long as your bandwidth is adequate (though recovery from write errors gets a bit tricky). > It will run over incredibly fast Packet over SONET Wide Area > Networks--behind firewalls. I'm sure it will be used behind firewalls in some cases but it's inappropriate to assume that it will always be used behind firewalls. Firewalls don't help with the majority of security threats, and it's less and less the case that network topologies reflect trust domains. Keith ------- End of Forwarded Message From XXX Fri May 26 16:52:47 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 12:50:12 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA05976 for ; Fri, 26 May 2000 12:50:12 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16318 for ips-outgoing; Fri, 26 May 2000 12:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16310 for ; Fri, 26 May 2000 12:50:02 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA05918 for ; Fri, 26 May 2000 12:50:01 -0400 Message-Id: <200005261650.MAA05918@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Brian.Rubarts@born.com: RE: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:50:01 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 09:40:28 -0500 From: Brian.Rubarts@born.com To: moore@cs.utk.edu cc: ietf@ietf.org Subject: RE: Storage over Ethernet/IP Encryption will be offloaded to the network interface. ASICs on the NICs will greatly improve encryption and authentication performance. It won't run over the Internet because of latencies inherent on the public network. It will run over incredibly fast Packet over SONET Wide Area Networks--behind firewalls. OC-192 and soon-to-come OC-768 (terrabit) switches will be the backbone of WANs and MANs of large networks in a few years. Also, IPv6 has significantly improved authentication capabilities that will help ensure security on the Storage WAN (SWAN?) Brian - -----Original Message----- From: Keith Moore [mailto:moore@cs.utk.edu] Sent: Friday, May 26, 2000 9:23 AM To: Jon William Toigo Cc: ietf@ietf.org Subject: Re: Storage over Ethernet/IP None of the cited limitations of TCP performance are true. they are also missing the point. If you're going to run storage access over IP then you are potentially allowing it to be run over the global Internet. If you do that you need good authentication and privacy, and (if you try to do them in software) authentication and encyption will eat far more in performance than anything inherent to TCP or IP. and no, it's not acceptable to assume that the storage device will be behind a firewall. Keith ------- End of Forwarded Message From XXX Fri May 26 16:52:47 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 12:51:44 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA06037 for ; Fri, 26 May 2000 12:51:44 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16380 for ips-outgoing; Fri, 26 May 2000 12:51:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16376 for ; Fri, 26 May 2000 12:51:36 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA06031 for ; Fri, 26 May 2000 12:51:35 -0400 Message-Id: <200005261651.MAA06031@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Brian.Rubarts@born.com: RE: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:51:35 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 10:14:03 -0500 From: Brian.Rubarts@born.com To: moore@cs.utk.edu cc: ietf@ietf.org Subject: RE: Storage over Ethernet/IP >> Encryption will be offloaded to the network interface. ASICs on the NICs >> will greatly improve encryption and authentication performance. >all well and good, provided that this encryption and authentication >are actually compatible with that specified by higher level protocols >and the authentication actually meets the needs of users. >(if your network interface needs to use and verify users' credentials, >as opposed to the host's credentials, it might be a stretch.) A network server will still authenticate user requests. Only the host needs to be authenticated with the disk/disks. >> It won't run over the Internet because of latencies inherent on the >> public network. >at least for some storage applications, latency is not as important >as bandwidth. e.g. you can do backups over a high-latency medium >as long as your bandwidth is adequate (though recovery from write >errors gets a bit tricky). Backups could go through VPNs, I suppose. Good point. That would free your WAN of the backup jobs. I wasn't thinking of backups when I ruled out the Internet as a disk I/O medium. I suppose infrequently used and low priority files could also be accessed over the 'net. >> It will run over incredibly fast Packet over SONET Wide Area >> Networks--behind firewalls. >...it's >inappropriate to assume that it will always be used behind firewalls... If the larger network that is employing this technology doesn't hire a decent consultant, you might be right. If they do, it will ALWAYS be behind a firewall :-) >Firewalls don't help with the majority of security threats... True, but whether the server accesses the disks via SCSI over TCP or SCSI over Fibre Channel, the SERVER is still the weak link. The transport protocol doesn't create any inherent weaknesses of the type you are refering to--e-mail borne viruses, internal hackers, etc. The server would still be the attack point. Why goodness, the server and storage devices could be in a VLAN or something to deny direct hack attempts against the storage device, but the chink in the armor is how hardened is your OS? Brian ------- End of Forwarded Message From XXX Fri May 26 16:52:47 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 12:50:01 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA05900 for ; Fri, 26 May 2000 12:50:01 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16296 for ips-outgoing; Fri, 26 May 2000 12:49:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16292 for ; Fri, 26 May 2000 12:49:28 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA05858 for ; Fri, 26 May 2000 12:49:27 -0400 Message-Id: <200005261649.MAA05858@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Keith Moore: Re: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:49:27 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 10:23:06 -0400 From: Keith Moore To: "Jon William Toigo" cc: ietf@ietf.org Subject: Re: Storage over Ethernet/IP None of the cited limitations of TCP performance are true. they are also missing the point. If you're going to run storage access over IP then you are potentially allowing it to be run over the global Internet. If you do that you need good authentication and privacy, and (if you try to do them in software) authentication and encyption will eat far more in performance than anything inherent to TCP or IP. and no, it's not acceptable to assume that the storage device will be behind a firewall. Keith ------- End of Forwarded Message From XXX Fri May 26 17:00:08 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 12:53:02 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA06094 for ; Fri, 26 May 2000 12:53:02 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16449 for ips-outgoing; Fri, 26 May 2000 12:52:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16445 for ; Fri, 26 May 2000 12:52:47 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA06069 for ; Fri, 26 May 2000 12:52:46 -0400 Message-Id: <200005261652.MAA06069@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Karl Auerbach: Re: Storage over Ethernet/IP Date: Fri, 26 May 2000 12:52:46 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 08:32:05 -0700 From: Karl Auerbach To: Jon William Toigo cc: ietf@ietf.org Subject: Re: Storage over Ethernet/IP > a. TCP is too CPU intensive and creates too much latency for storage I/O operations. > > b. The IP stack is too top heavy and processing packet headers is too > slow to support storage I/O operations. There were some papers published duing the late '80's or early '90s by John Romkey and I belive Dave Clark and Van Jacobson about the length of instruction sequences to handle TCP. I'm not sure that those ever became RFCs. Those papers came up with figures indicating that if one structures code "correctly" and if the net path is "clean" (i.e. not a lot of packet loss, reordering, replication, etc) than the per-packet instruction sequences (sans IP checksum calculation) were potantially very short. Does anyone have the references to these papers? --karl-- ------- End of Forwarded Message From XXX Fri May 26 17:10:34 2000 Replied: Fri, 26 May 2000 13:10:34 -0400 Replied: "Ray Hoare " Return-Path: hoare+@pitt.edu Delivery-Date: Fri May 26 12:22:50 2000 Return-Path: Received: from mb2i0.ns.pitt.edu (mb2i0.ns.pitt.edu [136.142.186.36]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA05475 for ; Fri, 26 May 2000 12:22:50 -0400 Received: from hoare.pitt.edu ("port 3364"@[136.142.87.77]) by mb2i0.ns.pitt.edu (PMDF V5.2-32 #41462) with ESMTP id <01JPUTCNIB4800055X@mb2i0.ns.pitt.edu> for bassoon@yogi.ece.cmu.edu; Fri, 26 May 2000 12:22:11 EST Date: Fri, 26 May 2000 12:29:09 -0400 From: Ray Hoare Subject: Re: "Jon William Toigo": Storage over Ethernet/IP In-reply-to: <200005261250.IAA11563@yogi.ece.cmu.edu> X-Sender: hoare@imap.pitt.edu To: Dave Nagle , ips@ece.cmu.edu Message-id: <4.3.1.2.20000526122411.00c6fa40@imap.pitt.edu> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.1 Content-type: text/plain; charset="us-ascii"; format=flowed At 08:50 AM 5/26/00 -0400, Dave Nagle wrote: >FYI >------- Forwarded Message > >I am seeking a few points of clarification: > >1. Fibre Channel folks have attempted to explain to me why TCP/IP could = >NEVER be a viable interconnect for block level storage operations. They = >claim: > >a. TCP is too CPU intensive and creates too much latency for storage = >I/O operations. Approximately 40% of sending TCP/IP messages is software overhead. There is a 400ns access time for just touching a PCI card within a *PC* >b. The IP stack is too top heavy and processing packet headers is too = >slow to support storage I/O operations. This may be true for a PC/Workstation. Direct hardware support is needed. >c. The maximum throughput of a GE TCP/IP connection is 768 Mps, which = >is too slow to support storage I/O operations. The maximum throughput of a PCI bus is 33MHz*32bits/cycle = 1056 Mbps (132 MBytes/s) *PEAK* The disk in my computer works just fine. Ray --------------------------------------------------------- Raymond R. Hoare, Assistant Professor 348 Benedum Hall Department of Electrical Engineering University of Pittsburgh Pittsburgh, PA 15261, USA Email: hoare@pitt.edu Voice: (412) 624-5836 Fax: (412) 624-8003 http://www.pitt.edu/~hoare From XXX Fri May 26 17:18:12 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 13:12:20 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA12021 for ; Fri, 26 May 2000 13:12:20 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA17044 for ips-outgoing; Fri, 26 May 2000 13:12:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA17036 for ; Fri, 26 May 2000 13:12:10 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA12011 for ; Fri, 26 May 2000 13:12:09 -0400 Message-Id: <200005261712.NAA12011@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: RJ Atkinson: RE: Storage over Ethernet/IP Date: Fri, 26 May 2000 13:12:09 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 11:53:30 +0100 From: RJ Atkinson To: Brian.Rubarts@born.com cc: ietf@ietf.org Subject: RE: Storage over Ethernet/IP At 15:40 26-05-00 , Brian.Rubarts@born.com wrote: >It will run over incredibly fast Packet over SONET Wide Area >Networks--behind firewalls. OC-192 and soon-to-come OC-768 (terrabit) >switches will be the backbone of WANs and MANs of large networks in a few >years. I'll note that at least one vendor has already demonstrated 10 Gig Ethernet switch interfaces at Interop/LV earlier this month. Such 10 GE boxes are generally much less expensive than OC-192 POS boxes. Another post indicated that these systems will use frame sizes of 1024 bytes, which is well within the abilities of any Ethernet interface. >Also, IPv6 has significantly improved authentication capabilities >that will help ensure security on the Storage WAN (SWAN?) IPv6 has NO authentication capability not already shipping for IPv4, speaking as the person who designed both AH and ESP. Marketing aside, there is nothing in IPv6 that makes it more easily secured than IPv4. Both support AH and ESP. Deployed ISAKMP/IKE support IPv4, but might not support IPv6. Note that I have no axe to grind for or against IPv6, but the disinformation campaign that "IPv6 is secure and IPv4 isn't" is highly annoying and completely wrong. Ran rja@inet.org ------- End of Forwarded Message From XXX Fri May 26 17:18:12 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 13:08:46 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA11854 for ; Fri, 26 May 2000 13:08:45 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA16912 for ips-outgoing; Fri, 26 May 2000 13:08:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA16908 for ; Fri, 26 May 2000 13:08:35 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA11848 for ; Fri, 26 May 2000 13:08:34 -0400 Message-Id: <200005261708.NAA11848@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Keith Moore: Re: Storage over Ethernet/IP Date: Fri, 26 May 2000 13:08:34 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 11:33:17 -0400 From: Keith Moore To: Brian.Rubarts@born.com cc: moore@cs.utk.edu, ietf@ietf.org Subject: Re: Storage over Ethernet/IP > >> It won't run over the Internet because of latencies inherent on the > >> public network. > > >at least for some storage applications, latency is not as important > >as bandwidth. e.g. you can do backups over a high-latency medium > >as long as your bandwidth is adequate (though recovery from write > >errors gets a bit tricky). > > Backups could go through VPNs, I suppose. except that you can't assume the presence of a VPN either. you need authenticity and privacy specified as part of the storage access protocol. > I suppose infrequently used and low > priority files could also be accessed over the 'net. yes, but file access protocols are better for this purpose. I don't see wanting to mount a raw disk drive across the public Internet very often. (except perhaps read-only... virtual cdrom, anyone?) > >> It will run over incredibly fast Packet over SONET Wide Area > >> Networks--behind firewalls. > > >...it's > >inappropriate to assume that it will always be used behind firewalls... > > If the larger network that is employing this technology doesn't hire a > decent consultant, you might be right. If they do, it will ALWAYS > be behind a firewall :-) any consultant who pretends that firewalls provide security cannot be described as 'decent'. > >Firewalls don't help with the majority of security threats... > > True, but whether the server accesses the disks via SCSI over TCP or SCSI > over Fibre Channel, the SERVER is still the weak link. un, no. SCSI has some inherent length/delay/number-of-stations limitations. but if the disk is accessible using TCP, there is a significant probability that it will be accessible from the global Internet and/or from local threats who have physical access to the transmission medium, and the storage access protocol needs to assume that this is the case. > The transport protocol doesn't create any inherent weaknesses of > the type you are refering to--e-mail borne viruses, internal hackers, etc. you're assuming a different threat model than I am. I am indeed assuming that storage devices will be targed, in addition to servers. > The server would still be the attack point. Why goodness, > the server and storage devices could be in a VLAN or something to deny > direct hack attempts against the storage device yes, they *could* be. but you cannot assume that they *will* be. > but the chink in the armor is how hardened is your OS? there's more than one chink in the armor. IP-based protocols need to be able to work in the global Internet. Keith ------- End of Forwarded Message From XXX Fri May 26 17:18:12 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 13:14:53 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA12060 for ; Fri, 26 May 2000 13:14:53 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA17118 for ips-outgoing; Fri, 26 May 2000 13:14:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA17114 for ; Fri, 26 May 2000 13:14:44 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA12054 for ; Fri, 26 May 2000 13:14:43 -0400 Message-Id: <200005261714.NAA12054@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Brian.Rubarts@born.com: RE: Storage over Ethernet/IP Date: Fri, 26 May 2000 13:14:43 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 10:55:29 -0500 From: Brian.Rubarts@born.com To: Valdis.Kletnieks@vt.edu cc: ietf@ietf.org Subject: RE: Storage over Ethernet/IP >Odd.. I thought we had a clue about security. The guys at SANS just >gave us a 'Technology Leadership Award'. I just walked across the hallway, >and I didn't see any firewall in our router swamp. >I guess because we don't have a firewall, we don't have a clue. Or because >we don't have a firewall, we can't deploy this technology. Somehow, that >doesn't smell right. >If your OS is hardened enough, a firewall may not be appropriate. I am not saying that you don't have a clue if you don't utilize a firewall. I AM saying that if you have Internet access to your network, a firewall is extremely important. It isn't complete, in and of itself. OS hardening is still very important, as are other technologies (as necessary to facilitate application needs). I understand your point that if your OS is perfectly hardened, then a firewall isn't going to add any *extra* protection. You miss the point, though. You can prevent unnecessary processor and bandwidth utilization on the server by filtering it out at the perimeter of your network. You might not get a security advantage if you are an OS hardening god, but you would CERTAINLY get performance increases on your LAN. If you are utilizing pure access lists on routers for perimeter security, then you are assuming that this technology is as adept at securing a network as port filters combined with Network Address Translation or cicuit proxying. Don't make that assumption. Brian ------- End of Forwarded Message From XXX Fri May 26 17:18:12 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 13:11:21 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA11989 for ; Fri, 26 May 2000 13:11:21 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA17012 for ips-outgoing; Fri, 26 May 2000 13:11:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA17007 for ; Fri, 26 May 2000 13:11:13 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id NAA11983 for ; Fri, 26 May 2000 13:11:12 -0400 Message-Id: <200005261711.NAA11983@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Valdis.Kletnieks@vt.edu: Re: Storage over Ethernet/IP Date: Fri, 26 May 2000 13:11:12 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 11:35:54 -0400 From: Valdis.Kletnieks@vt.edu To: Brian.Rubarts@born.com cc: moore@CS.UTK.EDU, ietf@ietf.org Subject: Re: Storage over Ethernet/IP On Fri, 26 May 2000 10:14:03 CDT, Brian.Rubarts@born.com said: > A network server will still authenticate user requests. Only the host > needs to be authenticated with the disk/disks. Hmm. Isn't this security model the cause of most grumbling regarding NFS security? > If the larger network that is employing this technology doesn't hire a decent > consultant, you might be right. If they do, it will ALWAYS be behind a firewall :-) Double Hmm.. Odd.. I thought we had a clue about security. The guys at SANS just gave us a 'Technology Leadership Award'. I just walked across the hallway, and I didn't see any firewall in our router swamp. I guess because we don't have a firewall, we don't have a clue. Or because we don't have a firewall, we can't deploy this technology. Somehow, that doesn't smell right. > the server and storage devices could be in a VLAN or something to deny direct hack > attempts against the storage device, but the chink in the armor is how hardened is > your OS? If your OS is hardened enough, a firewall may not be appropriate. "New from Kellogs - Firewalls cereal - part of this *COMPLETE* and *BALANCED* security breakfast". - -- Valdis Kletnieks Operating Systems Analyst Virginia Tech ------- End of Forwarded Message From XXX Fri May 26 18:51:30 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 14:48:03 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id OAA19293 for ; Fri, 26 May 2000 14:48:03 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA19451 for ips-outgoing; Fri, 26 May 2000 14:47:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA19447 for ; Fri, 26 May 2000 14:47:41 -0400 (EDT) Received: from yogi.ece.cmu.edu (bassoon@localhost [127.0.0.1]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id OAA19283 for ; Fri, 26 May 2000 14:47:40 -0400 Message-Id: <200005261847.OAA19283@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Peter Johansson: Re: IETF mailing list question on Storage over Ethernet/IP Date: Fri, 26 May 2000 14:47:40 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk ------- Forwarded Message Date: Fri, 26 May 2000 10:39:58 -0700 From: Peter Johansson To: IP Storage Subject: Re: IETF mailing list question on Storage over Ethernet/IP Some time ago, Randy_Haagens@hp.com wrote: >Just about any reliable transport will do nicely for transporting SCSI >commands. Very true---which is why it's difficult to discuss these issues generically. Many questions can't be answered until you look at the particular needs of the particular environment. In the home network, IEEE 1394 is a strong contender, with speeds that range from 400 Mbps to 1.6 Gbps. TCP does add a lot of overhead; it's appropriate to consider it only when other considerations (extensibility to the WAN, for example) predominate. In an environment like the home network, a "native" solution may be better. I mention all of this because I think it's worth your time to look at NCITS 325-1998, Serial Bus Protocol 2 (SBP-2). SBP-2 describes the encapsulation of SCSI commands and a DMA architecture for the efficient transfer of data over IEEE 1394. OK, you say, "So what? FCP does this. SSA did this." and so on. Well, the point is that Fibre Channel is not a serious contender for use as a network. In some situations, such as the home or small office, IEEE 1394 is. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org ------- End of Forwarded Message From XXX Fri May 26 22:03:31 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri May 26 18:02:33 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id SAA05421 for ; Fri, 26 May 2000 18:02:33 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23543 for ips-outgoing; Fri, 26 May 2000 18:01:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pneumatic-tube.sgi.com (pneumatic-tube.sgi.com [204.94.214.22]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23539 for ; Fri, 26 May 2000 18:01:42 -0400 (EDT) Received: from cthulhu.engr.sgi.com (cthulhu.engr.sgi.com [192.26.80.2]) by pneumatic-tube.sgi.com (980327.SGI.8.8.8-aspam/980310.SGI-aspam) via ESMTP id PAA05800; Fri, 26 May 2000 15:06:07 -0700 (PDT) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (clock.engr.sgi.com [163.154.34.45]) by cthulhu.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id PAA35928; Fri, 26 May 2000 15:01:20 -0700 (PDT) mail_from (zamsden@clock.engr.sgi.com) Received: from clock.engr.sgi.com (localhost [127.0.0.1]) by clock.engr.sgi.com (980427.SGI.8.8.8/970903.SGI.AUTOCF) via ESMTP id PAA18202; Fri, 26 May 2000 15:16:14 -0700 (PDT) Message-Id: <200005262216.PAA18202@clock.engr.sgi.com> X-Mailer: exmh version 2.0.2 2/24/98 To: Dave Nagle Cc: ips@ece.cmu.edu Subject: Fwd: TCP in 30 instructions (Was Re: Karl Auerbach: Re: Storage over Ethernet/IP) From: Zachary Amsden In-Reply-To: Your message of "Fri, 26 May 2000 12:52:46 EDT." <200005261652.MAA06069@yogi.ece.cmu.edu> Date: Fri, 26 May 2000 15:16:14 -0700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Received: from rx7.ee.lbl.gov by uu2.psi.com (5.65b/4.0.071791-PSI/PSINet) via SMTP; id AA12583 for popbbn; Wed, 8 Sep 93 01:29:46 -0400 Received: by rx7.ee.lbl.gov for craig@aland.bbn.com (5.65/1.44r) id AA05271; Tue, 7 Sep 93 22:30:15 -0700 Message-Id: <9309080530.AA05271@rx7.ee.lbl.gov> To: Craig Partridge Cc: David Clark Subject: Re: query about TCP header on tcp-ip In-Reply-To: Your message of Tue, 07 Sep 93 09:48:00 PDT. Date: Tue, 07 Sep 93 22:30:14 PDT From: Van Jacobson Craig, As you probably remember from the "High Speed TCP" CNRI meeting, my kernel looks nothing at all like any version of BSD. Mbufs no longer exist, for example, and `netipl' and all the protocol processing that used to be done at netipl interrupt level are gone. TCP receive packet processing in the new kernel really is about 30 instructions on a RISC (33 on a sparc but three of those are compiler braindamage). Attached is the C code & the associated sparc assembler. A brief recap of the architecture: Packets go in 'pbufs' which are, in general, the property of a particular device. There is exactly one, contiguous, packet per pbuf (none of that mbuf chain stupidity). On the packet input interrupt, the device driver upcalls through the protocol stack (no more of that queue packet on the netipl software interrupt bs). The upcalls percolate up the stack until the packet is completely serviced (e.g., NFS requests that can be satisfied from in-memory data & data structures) or they reach a routine that knows the rest of the processing must be done in some process's context in which case the packet is laid on some appropriate queue and the process is unblocked. In the case of TCP, the upcalls almost always go two levels: IP finds the datagram is for this host & it upcalls a TCP demuxer which hashes the ports + SYN to find a PCB, lays the packet on the tail of the PCB's queue and wakes up any process sleeping on the PCB. The IP processing is about 25 instructions & the demuxer is about 10. As Dave noted, the two processing paths that need the most tuning are the data packet send & receive (since at most every other packet is acked, there will be at least twice as many data packets as ack packets). In the new system, the receiving process calls 'tcp_usrrecv' (the protocol specific part of the 'recv' syscall) or is already blocked there waiting for new data. So the following code is embedded in a loop at the start of tcp_usrrecv that spins taking packets off the pcb queue until there's no room for the next packet in the user buffer or the queue is empty. The TCP protocol processing is done as we remove packets from the queue & copy their data to user space (and since we're in process context, it's possible to do a checksum-and-copy). Throughout this code, 'tp' points to the pcb and 'ti' points to the tcp header of the first packet on the queue (the ip header was stripped as part of interrupt level ip processing). The header info (excluding the ports which are implicit in the pcb) are sucked out of the packet into registers [this is to minimize cache thrashing and possibly to take advantage of 64 bit or longer loads]. Then the header checksum is computed (tp->ph_sum is the precomputed pseudo-header checksum + src & dst ports). int tcp_usrrecv(struct uio* uio, struct socket* so) { struct tcpcb *tp = (struct tcpcb *)so->so_pcb; register struct pbuf* pb; while ((pb = tp->tp_inq) != 0) { register int len = pb->len; struct tcphdr *ti = (struct tcphdr *)pb->dat; u_long seq = ((u_long*)ti)[1]; u_long ack = ((u_long*)ti)[2]; u_long flg = ((u_long*)ti)[3]; u_long sum = ((u_long*)ti)[4]; u_long cksum = tp->ph_sum; /* NB - ocadd is an inline gcc assembler function */ cksum = ocadd(ocadd(ocadd(ocadd(cksum, seq), ack), flg), sum); Next is the header prediction check which is probably the most opaque part of the code. tp->pred_flags contains snd_wnd (the window we expect in incoming packets) in the bottom 16 bits and 0x4x10 in the top 16 bits. The 'x' is normally 0 but will be set non-zero if header prediction shouldn't be done (e.g., if not in established state, if retransmitting, if hole in seq space, etc.). So, the first term of the 'if' checks four different things simultaneously: - that the window is what we expect - that there are no tcp options - that the packet has ACK set & doesn't have SYN, FIN, RST or URG set - that the connection is in the right state and the 2nd term of the if checks that the packet is in sequence: #define FMASK (((0xf000 | TH_SYN|TH_FIN|TH_RST|TH_URG|TH_ACK) << 16) | 0xffff) if ((flg & FMASK) == tp->pred_flags && seq == tp->rcv_nxt) { The next few lines are pretty obvious -- we subtract the header length from the total length and if it's less than zero the packet was malformed, if it's zero we must have a pure ack packet & we do the ack stuff otherwise if the ack field didn't move we have a pure data packet which we copy to the user's buffer, checksumming as we go, then update the pcb state if everything checks: len -= 20; if (len <= 0) { if (len < 0) { /* packet malformed */ } else { /* do pure ack things */ } } else if (ack == tp->snd_una) { cksum = in_uiomove((u_char*)ti + 20, len, uio, cksum); if (cksum != 0) { /* packet or user buffer errors */ } seq += len; tp->rcv_nxt = seq; if ((int)(seq - tp->rcv_acked) >= 0) { /* send ack */ } else { /* free pbuf */ } continue; } } /* header prediction failed -- take long path */ ... That's it. On the normal receive data path we execute 16 lines of C which turn into 33 instructions on a sparc (it would be 30 if I could trick gcc into generating double word loads for the header & my carefully aligned pcb fields). I think you could get it down around 25 on a cray or big-endian alpha since the loads, checksum calc and most of the compares can be done on 64 bit quantities (e.g., you can combine the seq & ack tests into one). Attached is the sparc assembler for the above receive path. Hope this explains Dave's '30 instruction' assertion. Feel free to forward this to tcp-ip or anyone that might be interested. - Van ---------------- ld [%i0+4],%l3 ! load packet tcp header fields ld [%i0+8],%l4 ld [%i0+12],%l2 ld [%i0+16],%l0 ld [%i1+72],%o0 ! compute header checksum addcc %l3,%o0,%o3 addxcc %l4,%o3,%o3 addxcc %l2,%o3,%o3 addxcc %l0,%o3,%o3 sethi %hi(268369920),%o1 ! check if hdr. pred possible andn %l2,%o1,%o1 ld [%i1+60],%o2 cmp %o1,%o2 bne L1 ld [%i1+68],%o0 cmp %l3,%o0 bne L1 addcc %i2,-20,%i2 bne,a L3 ld [%i1+36],%o0 ! packet error or ack processing ... L3: cmp %l4,%o0 bne L1 add %i0,20,%o0 mov %i2,%o1 call _in_uiomove,0 mov %i3,%o2 cmp %o0,0 be L6 add %l3,%i2,%l3 ! checksum error or user buffer error ... L6: ld [%i1+96],%o0 subcc %l3,%o0,%g0 bneg L7 st %l3,[%i1+68] ! send ack ... br L8 L7: ! free pbuf ... L8: ! done with this packet - continue ... L1: ! hdr pred. failed - do it the hard way From XXX Sat May 27 01:38:16 2000 Return-Path: owner-ietf-outbound@ietf.org Delivery-Date: Fri May 26 18:51:23 2000 Return-Path: Received: from ietf.org (odin.ietf.org [132.151.1.176]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id SAA12976 for ; Fri, 26 May 2000 18:51:23 -0400 Received: by ietf.org (8.9.1a/8.9.1a) id SAA27153 for ietf-outbound.06@ietf.org; Fri, 26 May 2000 18:40:01 -0400 (EDT) Received: from dokka.maxware.no (dokka.maxware.no [195.139.236.69]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA27118 for ; Fri, 26 May 2000 18:35:16 -0400 (EDT) Received: from langfjella.Alvestrand.no ([10.128.167.143]) by dokka.maxware.no (8.9.3/8.9.3) with ESMTP id AAA05599; Sat, 27 May 2000 00:35:00 +0200 Message-Id: <4.3.1.2.20000527001328.047fb8b0@dokka.kvatro.no> X-Sender: hta@dokka.maxware.no X-Mailer: QUALCOMM Windows Eudora Version 4.3.1 Date: Sat, 27 May 2000 00:27:01 +0200 To: Brian.Rubarts@born.com From: Harald Tveit Alvestrand Subject: RE: Storage over Ethernet/IP Cc: ietf@ietf.org In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Loop: ietf@ietf.org At 10:14 26.05.2000 -0500, Brian.Rubarts@born.com wrote: >True, but whether the server accesses the disks via SCSI over TCP or SCSI >over Fibre Channel, the SERVER is still the weak link. The transport >protocol doesn't create any inherent weaknesses of the type you are >refering to--e-mail borne viruses, internal hackers, etc. The server >would still be the attack point. Why goodness, the server and storage >devices could be in a VLAN or something to deny direct hack attempts >against the storage device, but the chink in the armor is how hardened is >your OS? did you hear the story about the MIT students who broke encryption in Netscape by replacing the page of the binary containing the crypto verification code (sniffing the NFS request and replying faster than the real fileserver) while it was being transferred over the network? Replacing a dedicated medium (such as a SCSI bus) with a shared medium (such as an Ethernet cable plant) always opens new chinks. The point being made, remade and made again here is: - Any IP technology will be used in contexts where there are security threats - Any protocol that offers no means of countering such security threats is broken, and should not be considered for standardization. It is perfectly possible that after conducting a threat and modality analysis, one ends up with saying that hardware-accelerated IPsec using host identities is adequate for the scenarios involving otherwise-unprotected Internet links, and that a mode with no protection is adequate when the media is physically secured. But the analysis MUST BE DONE. Harald -- Harald Tveit Alvestrand, EDB Maxware, Norway Harald.Alvestrand@edb.maxware.no From XXX Sat May 27 15:21:42 2000 Return-Path: owner-ietf-outbound@ietf.org Delivery-Date: Sat May 27 10:40:04 2000 Return-Path: Received: from ietf.org (odin.ietf.org [132.151.1.176]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id KAA23955 for ; Sat, 27 May 2000 10:40:04 -0400 Received: by ietf.org (8.9.1a/8.9.1a) id KAA18510 for ietf-outbound.06@ietf.org; Sat, 27 May 2000 10:30:02 -0400 (EDT) Received: from mail-blue.research.att.com (mail-blue.research.att.com [135.207.30.102]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA18338 for ; Sat, 27 May 2000 10:12:10 -0400 (EDT) Received: from postal.research.att.com (postal.research.att.com [135.207.23.30]) by mail-blue.research.att.com (Postfix) with ESMTP id 775FD4CE08; Sat, 27 May 2000 10:12:11 -0400 (EDT) Received: from smb.research.att.com (postal.research.att.com [135.207.23.30]) by postal.research.att.com (8.8.7/8.8.7) with ESMTP id KAA20435; Sat, 27 May 2000 10:12:09 -0400 (EDT) Received: from smb.research.att.com (localhost.research.att.com [127.0.0.1]) by smb.research.att.com (Postfix) with ESMTP id 504B435DC2; Sat, 27 May 2000 10:12:09 -0400 (EDT) X-Mailer: exmh version 2.1.1 10/15/1999 From: "Steven M. Bellovin" To: Brian.Rubarts@born.com Cc: moore@cs.utk.edu, ietf@ietf.org Subject: Re: Storage over Ethernet/IP Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 27 May 2000 10:11:39 -0400 Sender: smb@research.att.com Message-Id: <20000527141209.504B435DC2@smb.research.att.com> X-Loop: ietf@ietf.org In message , Brian.Rubarts@born .com writes: > >>> Encryption will be offloaded to the network interface. ASICs on the NICs >>> will greatly improve encryption and authentication performance. > >>all well and good, provided that this encryption and authentication >>are actually compatible with that specified by higher level protocols >>and the authentication actually meets the needs of users. >>(if your network interface needs to use and verify users' credentials, >>as opposed to the host's credentials, it might be a stretch.) > >A network server will still authenticate user requests. Only the host >needs to be authenticated with the disk/disks. > Up to a point. Yes, there are NICs available today with IPsec on-card. But given the prevalence of -- how shall I put this? -- single-user computers with user physical access, no OS protection and crufty software, you really need user-granularity protection of the file access requests. NFS-style protection with host authentication works if and only if the server trusts the remote system to authenticate its users. That's demonstrably not true today. Yes, IPsec does, in theory, support user-granularity protection. That's very hard to do when you're using outboard IPsec implementations, since you then need some way to pass the user's credentials (generally a certificate, not a user-id) back to the host, and tie every received packet to that identity. It can be done, but (speaking as one of the primary participants in the IPsec development effort) I'm not impressed with its applicability in this case. > >>> It will run over incredibly fast Packet over SONET Wide Area >>> Networks--behind firewalls. > >>...it's >>inappropriate to assume that it will always be used behind firewalls... > >If the larger network that is employing this technology doesn't hire a >decent >consultant, you might be right. If they do, it will ALWAYS be behind a >firewall :-) > Speaking as someone whose firewall credentials are more or less beyond reproach, you're wrong -- period. *Many* such uses will be behind firewalls. Others won't. The large corporate firewall is a dinosaur, because of extranets, telecommuters, unofficial links through or around the firewall, etc. Comprehensive firewalls generally can't protect a network larger than one run by a single systems administrator (or, in some cases, a systems administration group); otherwise, they don't know where the links are. And even when one sysadmin runs the net, what does he or she do when word comes down from the pointy-haired layer of the stack that there *will* be a VPN link to a joint venture partner? Like it says on the (U.S.) toothpaste tubes -- firewalls can be an effective security measure when used as part of a program including good network hygiene and decent authentication. But they're not magic security pixie dust, and they're not a substitute for authentication in the protocol. --Steve Bellovin From XXX Sun May 28 03:58:19 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sat May 27 22:00:01 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id WAA29947 for ; Sat, 27 May 2000 22:00:00 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA15188 for ips-outgoing; Sat, 27 May 2000 21:59:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA15184 for ; Sat, 27 May 2000 21:59:52 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id DAA193956; Sun, 28 May 2000 03:55:27 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id DAA05960; Sun, 28 May 2000 03:55:27 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568ED.000A8E78 ; Sun, 28 May 2000 03:55:18 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: List SCSI-over-TCP Message-ID: Date: Sat, 27 May 2000 23:54:48 +0300 Subject: Re: Peter Johansson: Re: IETF mailing list question on Storage over Ethernet/IP Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk True - 1394 is a strong contender for the SOHO and home. There is an consumer-products oriented consortia trying to standardize its use for Audio-Video. I am just curios if those companies will ever come to an agreement and use it (how many compatible IR remote controls have you seen?). It would be also an interesting exercise to see how much overhead would TCP add to the stack ? (and eliminate the distance limitations inherent to SBP) Julo Julian Satran - IBM Research at Haifa Dave Nagle on 26/05/2000 21:47:40 Please respond to Dave Nagle To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Peter Johansson: Re: IETF mailing list question on Storage over Ethernet/IP ------- Forwarded Message Date: Fri, 26 May 2000 10:39:58 -0700 From: Peter Johansson To: IP Storage Subject: Re: IETF mailing list question on Storage over Ethernet/IP Some time ago, Randy_Haagens@hp.com wrote: >Just about any reliable transport will do nicely for transporting SCSI >commands. Very true---which is why it's difficult to discuss these issues generically. Many questions can't be answered until you look at the particular needs of the particular environment. In the home network, IEEE 1394 is a strong contender, with speeds that range from 400 Mbps to 1.6 Gbps. TCP does add a lot of overhead; it's appropriate to consider it only when other considerations (extensibility to the WAN, for example) predominate. In an environment like the home network, a "native" solution may be better. I mention all of this because I think it's worth your time to look at NCITS 325-1998, Serial Bus Protocol 2 (SBP-2). SBP-2 describes the encapsulation of SCSI commands and a DMA architecture for the efficient transfer of data over IEEE 1394. OK, you say, "So what? FCP does this. SSA did this." and so on. Well, the point is that Fibre Channel is not a serious contender for use as a network. In some situations, such as the home or small office, IEEE 1394 is. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org ------- End of Forwarded Message From XXX Wed May 31 01:26:16 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue May 30 21:22:21 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id VAA06185 for ; Tue, 30 May 2000 21:22:21 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA28839 for ips-outgoing; Tue, 30 May 2000 21:21:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ludwig.troikanetworks.com (host03.troikanetworks.com [12.31.172.3] (may be forged)) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA28835 for ; Tue, 30 May 2000 21:21:51 -0400 (EDT) Received: by host03.troikanetworks.com with Internet Mail Service (5.5.2650.21) id ; Tue, 30 May 2000 18:21:30 -0700 Message-ID: From: Wayland Jeong To: "'Michael Krause'" , Wayland Jeong , ips@ece.cmu.edu Subject: RE: IETF mailing list question on Storage over Ethernet/IP Date: Tue, 30 May 2000 18:21:27 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk At 10:49 AM 5/26/00 -0700, Wayland Jeong wrote: > > >I would argue, though, that congestion control in Fibre Channel networks is > >better (of course, FC is not completely immune from losing frames) than an > >Ethernet network. The link-level buffer-to-buffer credit mechanisms and > >the hardware end-to-end flow-control offered in Class-2 services help > >ensure that packet loss is at a minimum. Traffic patterns in storage > >networks tend to be more congestion oriented than normal peer-to-peer LAN > >traffic patterns. Most SAN's have many initiators (hosts) and very few > >targets (storage devices, like RAID's). Thus, congestion can occur quite > >often due to the funneling of traffic to shared storage. Flow-control in > >FC helps take care of this situation. > > This problem will be true of any application which shares a > common point of > contention including FC. As links evolve to 10 Gbps speeds, > the number of > links into a given device may be reduced and thus increase > contention and > fabric backpressure. As such, the contention may result in > packet loss as > fabric forward progress timers and other mechanisms such as > WRED kick in to > alleviate the congestion via packet drops. Note: I believe Cisco and > others have numerous papers on how WRED can smooth out > traffic flow so that > congestion is managed fairly well even under heavy load. > This problem is not true of any network. Since FC supports both link-level and end-to-end flow-control, the likelihood of packets being dropped under congestion situations is mitigated. Packets can still expire within congested switches but only after long time-outs occur (RA_TOV). Momentary bursts of congestion can be handled by applying backpressure all the way to the producers. Cisco's WRED (Weighted Random Early Detection) as I understand it, is a mechanism to start dropping packets before congestion becomes critical. The dropping of packets will trigger back-off by TCP. My point here is that in a streaming storage environment, packet loss is very bad. We want to prevent, at all costs, packet loss since losing a packet means triggering an I/O level retry of a very large chunk of data. I agree, bandwidth is always a solution, but are we going to see a ubiquitous deployment of 10Gbs Ethernet any time soon? > >I'm curious about how IPS will behave in these > configurations. Is the > >windowing mechanism in TCP sufficient? Will there need to be > XON/XOFF > >flow-control to help reduce packet loss? > > XON/XOFF are link-level constructs and not transport-level. > As such, they > do not really apply. TCP windows provides both sides with an > understanding > of how much buffering is available for that connection. To deal with > bandwidth management issues, there are numerous techniques such as > transparent window adjustments that are possible to > implement, i.e. traffic > shaping. It is also possible to use the IP-based QoS to > adjust the network > arbitration policies such that traffic can be segregated. > This will allow > the network to determine which traffic gets serviced first and thus > provides a degree of control for reducing the applications view of > congestion and the oscillations that may result in reaction to that > congestion. > > Note: As with many congestion-related problem and given the continual > downward spiral of costs for many links, e.g. ethernet, many > customers will > simply through bandwidth at the problem within the fabric > itself and then > adjust the arbitration and traffic shaping parameters (either > directly > within their applications or transparently within the fabric). > Okay. I am not familiar enough with today's LAN products to comment on their ability to provide near guaranteed QoS (i.e. fractional bandwidth). So, QoS applied correctly or proper configuration of networks (i.e. matching bandwidth requirements) could alleviate most of the problems. I am still curious though how windowing works in the SIP model. As I understand it, the proposal calls for a single command connection (a target implements a well-known TCP server port). After authentication of the client (host), a data channel is allocated for that connection. Windowing applies to that channel which provides the target, the ability to manage buffers with that host. Thus, if a data channel communicates with server A, the target will advertise its window size which would correspond to its available buffer space for that host. But, typically, many hosts will login with one target. Thus, each host will have its own data channel and hence its own advertised window size. If, say 10 hosts connect to a given target and each is allocated a 64KB window size, then the total buffer space available at the target must be 640KB. Now, a host will have no problem allocating this space, but the congestion point of interest is not host memory, but in the target adapter itself (in fact, these may be one in the same on a low-cost drive). Now, RTT is one way to coordinate access to the local buffers in the target interface which may be acceptable. But, the equivalent in FCP is XFER_READY and the intention of this protocol is to both pace FCP_WRITES and also give the producer and indication of what data is the best data to send. It really has no mapping to physical buffers, only cache state. It seems to me that protocol-level mechanisms for handling flow control, like windowing and RTT, are better suited for gross-level congestion management. In my opinion providing near zero packet loss is not best handled at the protocol-level. > >I certainly don't argue with both the merits and the > technical feasibility > >of putting block storage over a TCP transport. It seems > quite doable. The > >question is, how to get good performance out of real > implementations. > >Certainly, proposals such as RDMA are trying to address that > very concern > >(reduce system overhead due to datagram re-assembly). > > RDMA only defines how to place the data on the remote for > WRITEs or on the > requester for READs. It does not define a SAR solution. TCP > attempts to > avoid SAR operations (IP fragmentation) by only transmitting > MSS sized packets. > Yes, I think I understand how RDMA works. I was only making a comment that the thrust of the IETF work is geared towards creating an architecture which can yield acceptable performance. I think the mapping is fairly straightforward. Making implementations which achieve good performance and are cost effective is the real challenge. Now, I saw a comment in the RDMA proposal which said that the MSS size should be no more than 8KB to avoid fragmentation. How does an MSS of 8KB avoid fragmentation on a 1.5KB MTU Ethernet network? I'm sure I'm just missing something here. > > >Even though, TCP compensates for a lossy network, to achieve 100MBs > >streaming throughput, you don't want to have TCP, even in > hardware, performing > >datagram retry. If you look at SLIC from Alacritech, they > don't perform > >retry in hardware. Only the datapath is implemented in > hardware. Retry, > >connection management and IP fragmentation are all still > handled in software. > > This is implementation specific and not a function of the > architecture. It > is possible to perform these types of operations within > hardware and it has > been done for quite some time, albeit simplistic in many > cases, for those > solutions using tftp or discless devices (e.g. workstations). > The solution > does not require software - that is an implementation option. > Please note > that efforts such as InfiniBand, which may be argued as at > least as complex > as TCP/IP, does not mandate a hardware-only implementation > but many will > implement one for most operations: SAR, packet > retransmissions, most error > operations / recovery, etc. > I don't know anyone contemplating an Infiniband implementation that is not in either hardware or firmware. I would argue, as well, that error recovery is not something that is typically handled by the hardware. The assumption is that the network is reliable. In other words, lost data is the exception rather than the norm. > > Note: HP has been shipping TCP running over GbE running at > link rate (i.e. > 940 Mbps of data payload) using standard MTU packets for quite some > time. I have seen one netperf on a GSN implementation report > over 4 Gbps > of data payload. In the GbE case, the CPU was not fully > consumed; I do not > have the specifics (MTU, CPU consumption) on the GSN but > given most of its > data transmission and recovery is implemented in hardware, > clearly it is > possible to achieve very high bandwidth using standard as > well as high-end > link types while running the TCP/IP protocol. > I would be interested to see some information on this implementation. Is there some public whitepapers or such on this product? I would assume that it was on HP-UX. I'm not that familiar with GSN. Is that a HIPPI-based network? > > > > 2. Jumbo frames will not be necessary when TCP is > implemented in hardware. > > > Most FC implementations use 1024 byte frames, and > performance is very > > > adequate, given hardware implementation of FCP. > > > > >This is untrue. Most FC implementations use full 2KB frames > and are capable > >of very large sequences (i.e. greater than the 64KB > limitation imposed by > >the IPS > >proposal), unless there is something that I don't know about > HP's Tachyon > >chips ;-) > > I do not see what the issue is here. In any solution, the > application > posts a SGL to process and then the underlying hardware > performs the SAR > operations and transmits / receives the data. There should be no > architectural limit (in most cases the limit is very large > and is really a > function of the hardware implementation) on how large a SGL > is only how > much of it can be transmitted in a single window at a time and then > determining whether the window is sufficiently large to support the > distance and application requirements. An implementation > could take a > large SGL transaction and then slice it into an appropriate > window for the > environment it is operating in without violating the > transport capabilities. > Yeah, I guess the bottom-line is that I'm not arguing about limitations in the architecture. I'm more interested in actual implementations. How will one implement this architecture and what kind of performance might one expect? > > Mike > > Thanks for the feedback. -Wayland From XXX Wed May 31 14:17:15 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed May 31 09:39:11 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id JAA27691 for ; Wed, 31 May 2000 09:39:11 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA18130 for ips-outgoing; Wed, 31 May 2000 09:37:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA18124 for ; Wed, 31 May 2000 09:37:56 -0400 (EDT) Received: from l3107mxr.atl.hp.com (l3107mxr.atl.hp.com [15.19.254.19]) by atlrel1.hp.com (Postfix) with ESMTP id 25B60BDF; Wed, 31 May 2000 09:36:55 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by l3107mxr.atl.hp.com (Postfix) with ESMTP id A3C744FD88; Wed, 31 May 2000 08:36:53 -0400 (EDT) Received: from mk731912 (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com with ESMTP (8.7.6/8.7.3 TIS 5.0.1) id GAA22581; Wed, 31 May 2000 06:45:38 -0700 (PDT) Message-Id: <4.2.2.20000531060300.00bae190@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 31 May 2000 06:25:19 -0700 To: , ips@ece.cmu.edu From: Michael Krause Subject: RE: IETF mailing list question on Storage over Ethernet/IP In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 06:21 PM 5/30/00 -0700, Wayland Jeong wrote: >This problem is not true of any network. Since FC supports both >link-level and end-to-end flow-control, the likelihood of packets being >dropped under congestion situations is mitigated. Packet drops are bad for many protocols. However, just is bad is having fabric efficiency drop due to head-of-line blocking with long congestion timeouts within routing elements. This limits the effective throughput of the fabric and results in higher level application timeouts or additional resources within the endnodes to deal with application resource shortages. >Packets can still expire within congested switches but only after long >time-outs occur (RA_TOV). Momentary bursts of congestion can be handled by >applying backpressure all the way to the producers. Cisco's WRED (Weighted >Random Early Detection) as I understand it, is a mechanism to start >dropping packets before congestion becomes critical. The dropping of >packets will trigger back-off by TCP. WRED provides a number of advantages. It can be invoked when a threshold is hit such that one does not just drop the moment congestion is hit. When invoked, it can apply a filter mechanism based on packet attributes, e.g. class of service. In general, WRED has been shown to improve the overall efficiency of the fabric while reducing oscillations for applications thus delivering overall smoother operation. >My point here is that in a streaming storage environment, packet loss is >very bad. We want to prevent, at all costs, packet loss since losing a >packet means triggering an I/O level retry of a very large chunk of data. This is true for nearly any application I can think of which uses a network. >I agree, bandwidth is always a solution, but are we going to see a >ubiquitous deployment of 10Gbs Ethernet any time soon? I believe 10 GbE will be available before 10 Gb FC and that will be about the same time as this spec becomes sufficiently solid to begin building product. So, the answer is yes. Also, most workloads can operate quite well with GbE if they operate with FC today and one can always aggregate GbE links (802.1ad) to provide a fatter pipe while waiting for 10 GbE. >Okay. I am not familiar enough with today's LAN products to comment >on their ability to provide near guaranteed QoS (i.e. fractional bandwidth). >So, QoS applied correctly or proper configuration of networks (i.e. >matching bandwidth requirements) could alleviate most of the problems. Yes. >I am still curious though how windowing works in the SIP model. As I >understand it, the proposal calls for a single command connection >(a target implements a well-known TCP server port). After authentication >of the client (host), a data channel is allocated for that connection. >Windowing applies to that channel which provides the target, the >ability to manage buffers with that host. Thus, if a data channel >communicates with server A, the target will advertise its window size >which would correspond to its available buffer space for that host. >But, typically, many hosts will login with one target. Thus, each >host will have its own data channel and hence its own advertised >window size. If, say 10 hosts connect to a given target and each >is allocated a 64KB window size, then the total buffer space >available at the target must be 640KB. Now, a host will have no >problem allocating this space, but the congestion point of interest >is not host memory, but in the target adapter itself (in fact, these >may be one in the same on a low-cost drive). 1 MB of DRAM costs about $1. I can buy very low cost adapters today with 1 MB of DRAM without much problem - a variety of GbE adapters ship with at least this much and they are much cheaper than FC adapters. There are a number of adapters out there that support 8, 16, or 32 MB of memory for a slight cost delta more and I've seen a few adapters that can support 256 MB of memory. >Now, RTT is one way to coordinate access to the local buffers in >the target interface which may be acceptable. But, the equivalent >in FCP is XFER_READY and the intention of this protocol is to >both pace FCP_WRITES and also give the producer and indication >of what data is the best data to send. It really has no mapping >to physical buffers, only cache state. End-to-end flow control is implemented in both protocols though in two slightly different manners. It is possible to implement a similar credit scheme on top of TCP with little difficulty. I have some ideas on how this could be implemented within this spec but am still bouncing them around within HP to see if they are in alignment with the overall architecture. >It seems to me that protocol-level mechanisms for handling >flow control, like windowing and RTT, are better suited for gross-level >congestion management. In my opinion providing near zero >packet loss is not best handled at the protocol-level. I'd be interested in any data or modeling that suggests one mechanism over the other if you have it available. >Yes, I think I understand how RDMA works. I was only making >a comment that the thrust of the IETF work is geared towards >creating an architecture which can yield acceptable performance. >I think the mapping is fairly straightforward. Making implementations >which achieve good performance and are cost effective is the real >challenge. > >Now, I saw a comment in the RDMA proposal which said that the >MSS size should be no more than 8KB to avoid fragmentation. How >does an MSS of 8KB avoid fragmentation on a 1.5KB MTU Ethernet >network? I'm sure I'm just missing something here. The RDMA proposal has some problems in its current form. Again, RDMA is about packet placement and not fragmentation avoidance and thus the proposal needs to be fixed. >I would be interested to see some information on this implementation. >Is there some public whitepapers or such on this product? I would >assume that it was on HP-UX. It is on HP-UX. There is a whitepaper that will be coming out quite soon on the performance - should be out in June if I recall. >I'm not that familiar with GSN. Is that a HIPPI-based network? GSN is another name for HIPPI6400. >Yeah, I guess the bottom-line is that I'm not arguing about limitations in >the architecture. I'm more interested in actual implementations. How will >one implement this architecture and what kind of performance might one expect? I don't think the implementation is all that difficult. Most people might leverage some of the SGL algorithms and driver work that is used with say a Tachyon-TL implementation and merge this into a good GbE implementation. HP has been looking into how this would be done and have not seen anything insurmountable yet. Mike From XXX Wed May 31 19:56:36 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed May 31 14:00:51 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id OAA20980 for ; Wed, 31 May 2000 14:00:50 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA26277 for ips-outgoing; Wed, 31 May 2000 13:58:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from seattle.3com.com (seattle.3com.com [129.213.128.97]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA26273 for ; Wed, 31 May 2000 13:58:42 -0400 (EDT) From: Dave_Lee@3com.com Received: from new-york.3com.com (new-york.3com.com [129.213.157.12]) by seattle.3com.com (8.8.8/8.8.8) with ESMTP id KAA18726; Wed, 31 May 2000 10:58:36 -0700 (PDT) Received: from hqoutbound.ops.3com.com (hqoutbound.OPS.3Com.COM [139.87.48.104]) by new-york.3com.com (8.8.8/8.8.8) with SMTP id KAA16321; Wed, 31 May 2000 10:58:30 -0700 (PDT) Received: by hqoutbound.ops.3com.com(Lotus SMTP MTA v4.6.7 (934.1 12-30-1999)) id 882568F0.00629C62 ; Wed, 31 May 2000 10:57:05 -0700 X-Lotus-FromDomain: 3COM To: Michael Krause cc: wayland@troikanetworks.com, ips@ece.cmu.edu Message-ID: <882568F0.00629290.00@hqoutbound.ops.3com.com> Date: Wed, 31 May 2000 10:56:26 -0700 Subject: RE: IETF mailing list question on Storage over Ethernet/IP Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk FYI - Cisco demonstrated 10 GE at N+I a few weeks ago. Michael Krause on 05/31/2000 06:25:19 AM Sent by: Michael Krause To: wayland@troikanetworks.com, ips@ece.cmu.edu cc: (Dave Lee/HQ/3Com) Subject: RE: IETF mailing list question on Storage over Ethernet/IP At 06:21 PM 5/30/00 -0700, Wayland Jeong wrote: >This problem is not true of any network. Since FC supports both >link-level and end-to-end flow-control, the likelihood of packets being >dropped under congestion situations is mitigated. Packet drops are bad for many protocols. However, just is bad is having fabric efficiency drop due to head-of-line blocking with long congestion timeouts within routing elements. This limits the effective throughput of the fabric and results in higher level application timeouts or additional resources within the endnodes to deal with application resource shortages. >Packets can still expire within congested switches but only after long >time-outs occur (RA_TOV). Momentary bursts of congestion can be handled by >applying backpressure all the way to the producers. Cisco's WRED (Weighted >Random Early Detection) as I understand it, is a mechanism to start >dropping packets before congestion becomes critical. The dropping of >packets will trigger back-off by TCP. WRED provides a number of advantages. It can be invoked when a threshold is hit such that one does not just drop the moment congestion is hit. When invoked, it can apply a filter mechanism based on packet attributes, e.g. class of service. In general, WRED has been shown to improve the overall efficiency of the fabric while reducing oscillations for applications thus delivering overall smoother operation. >My point here is that in a streaming storage environment, packet loss is >very bad. We want to prevent, at all costs, packet loss since losing a >packet means triggering an I/O level retry of a very large chunk of data. This is true for nearly any application I can think of which uses a network. >I agree, bandwidth is always a solution, but are we going to see a >ubiquitous deployment of 10Gbs Ethernet any time soon? I believe 10 GbE will be available before 10 Gb FC and that will be about the same time as this spec becomes sufficiently solid to begin building product. So, the answer is yes. Also, most workloads can operate quite well with GbE if they operate with FC today and one can always aggregate GbE links (802.1ad) to provide a fatter pipe while waiting for 10 GbE. >Okay. I am not familiar enough with today's LAN products to comment >on their ability to provide near guaranteed QoS (i.e. fractional bandwidth). >So, QoS applied correctly or proper configuration of networks (i.e. >matching bandwidth requirements) could alleviate most of the problems. Yes. >I am still curious though how windowing works in the SIP model. As I >understand it, the proposal calls for a single command connection >(a target implements a well-known TCP server port). After authentication >of the client (host), a data channel is allocated for that connection. >Windowing applies to that channel which provides the target, the >ability to manage buffers with that host. Thus, if a data channel >communicates with server A, the target will advertise its window size >which would correspond to its available buffer space for that host. >But, typically, many hosts will login with one target. Thus, each >host will have its own data channel and hence its own advertised >window size. If, say 10 hosts connect to a given target and each >is allocated a 64KB window size, then the total buffer space >available at the target must be 640KB. Now, a host will have no >problem allocating this space, but the congestion point of interest >is not host memory, but in the target adapter itself (in fact, these >may be one in the same on a low-cost drive). 1 MB of DRAM costs about $1. I can buy very low cost adapters today with 1 MB of DRAM without much problem - a variety of GbE adapters ship with at least this much and they are much cheaper than FC adapters. There are a number of adapters out there that support 8, 16, or 32 MB of memory for a slight cost delta more and I've seen a few adapters that can support 256 MB of memory. >Now, RTT is one way to coordinate access to the local buffers in >the target interface which may be acceptable. But, the equivalent >in FCP is XFER_READY and the intention of this protocol is to >both pace FCP_WRITES and also give the producer and indication >of what data is the best data to send. It really has no mapping >to physical buffers, only cache state. End-to-end flow control is implemented in both protocols though in two slightly different manners. It is possible to implement a similar credit scheme on top of TCP with little difficulty. I have some ideas on how this could be implemented within this spec but am still bouncing them around within HP to see if they are in alignment with the overall architecture. >It seems to me that protocol-level mechanisms for handling >flow control, like windowing and RTT, are better suited for gross-level >congestion management. In my opinion providing near zero >packet loss is not best handled at the protocol-level. I'd be interested in any data or modeling that suggests one mechanism over the other if you have it available. >Yes, I think I understand how RDMA works. I was only making >a comment that the thrust of the IETF work is geared towards >creating an architecture which can yield acceptable performance. >I think the mapping is fairly straightforward. Making implementations >which achieve good performance and are cost effective is the real >challenge. > >Now, I saw a comment in the RDMA proposal which said that the >MSS size should be no more than 8KB to avoid fragmentation. How >does an MSS of 8KB avoid fragmentation on a 1.5KB MTU Ethernet >network? I'm sure I'm just missing something here. The RDMA proposal has some problems in its current form. Again, RDMA is about packet placement and not fragmentation avoidance and thus the proposal needs to be fixed. >I would be interested to see some information on this implementation. >Is there some public whitepapers or such on this product? I would >assume that it was on HP-UX. It is on HP-UX. There is a whitepaper that will be coming out quite soon on the performance - should be out in June if I recall. >I'm not that familiar with GSN. Is that a HIPPI-based network? GSN is another name for HIPPI6400. >Yeah, I guess the bottom-line is that I'm not arguing about limitations in >the architecture. I'm more interested in actual implementations. How will >one implement this architecture and what kind of performance might one expect? I don't think the implementation is all that difficult. Most people might leverage some of the SGL algorithms and driver work that is used with say a Tachyon-TL implementation and merge this into a good GbE implementation. HP has been looking into how this would be done and have not seen anything insurmountable yet. Mike From XXX Thu Jun 01 03:35:00 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed May 31 22:50:24 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id WAA14334 for ; Wed, 31 May 2000 22:50:24 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA08114 for ips-outgoing; Wed, 31 May 2000 22:49:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA08110 for ; Wed, 31 May 2000 22:48:56 -0400 (EDT) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id TAA19952 for ; Wed, 31 May 2000 19:48:25 -0700 (PDT) Received: from aimexc01.corp.adaptec.com ([162.62.62.41]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id TAA18665 for ; Wed, 31 May 2000 19:48:24 -0700 (PDT) Received: by aimexc01.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Wed, 31 May 2000 19:48:24 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C04643440@aimexc03.corp.adaptec.com> From: "Monia, Charles" To: "'Michael Krause'" , wayland@troikanetworks.com, ips@ece.cmu.edu Subject: RE: IETF mailing list question on Storage over Ethernet/IP Date: Wed, 31 May 2000 13:18:10 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFCB73.DA6A10C8" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFCB73.DA6A10C8 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Wednesday, May 31, 2000 6:25 AM > To: wayland@troikanetworks.com; ips@ece.cmu.edu > Subject: RE: IETF mailing list question on Storage over Ethernet/IP >=20 >=20 <....Lots of stuff deleted.....> >=20 > 1 MB of DRAM costs about $1. I can buy very low cost=20 > adapters today with 1=20 > MB of DRAM without much problem - a variety of GbE adapters=20 > ship with at=20 > least this much and they are much cheaper than FC adapters. =20 Cost comparisons must account for the fact that, unlike GbE, Fibre = Channel adapters implement performance assists for the storage and transport protocol layers in silicon. These assists aren't free and account for = part of the cost differential. Charles Charles Monia Adaptec Inc. 691 South Milpitas Blvd Milpitas, CA 95035 M/S 135 =A0 email: cmonia@corp.adaptec.com =20 voice: (408) 957-6977 Fax:=A0 (408) 945-6736 =A0 ------_=_NextPart_000_01BFCB73.DA6A10C8 Content-Type: application/octet-stream; name="Charles Monia.vcf" Content-Disposition: attachment; filename="Charles Monia.vcf" BEGIN:VCARD VERSION:2.1 N:Monia;Charles;; FN:Charles Monia ORG:Adaptec Inc.;Advanced Technology Group TITLE:Senior Software Engineer TEL;WORK;VOICE:408-957-6977 TEL;WORK;FAX:408-945-6736 ADR;WORK;ENCODING=QUOTED-PRINTABLE:;M/S 135;691 South Milpitas Blvd.=0D=0A;Milpitas;CA;95035;USA LABEL;WORK;ENCODING=QUOTED-PRINTABLE:M/S 135=0D=0A691 South Milpitas Blvd.=0D=0A=0D=0AMilpitas, CA 95035=0D=0AUSA URL: URL:http://www.adaptec.com EMAIL;PREF;INTERNET:cmonia@corp.adaptec.com REV:19980515T153408Z END:VCARD ------_=_NextPart_000_01BFCB73.DA6A10C8-- From XXX Thu Jun 15 14:11:24 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Jun 15 10:10:49 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id KAA21632 for ; Thu, 15 Jun 2000 10:10:49 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA17553 for ips-outgoing; Thu, 15 Jun 2000 10:08:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA17548 for ; Thu, 15 Jun 2000 10:08:35 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA83842; Thu, 15 Jun 2000 16:07:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id QAA111152; Thu, 15 Jun 2000 16:07:47 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C12568FF.004D91F6 ; Thu, 15 Jun 2000 16:07:16 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu, List SCSI-over-TCP Message-ID: Date: Thu, 15 Jun 2000 16:15:36 +0300 Subject: Re: new working version of the iSCSI ID Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=9Y9GvtN5f8kQTgvW2CmfSX5NGi8FufevdN5tMllV7hDYdPwFhWZauBVR" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=9Y9GvtN5f8kQTgvW2CmfSX5NGi8FufevdN5tMllV7hDYdPwFhWZauBVR Content-type: text/plain; charset=us-ascii Content-Disposition: inline Dear colleagues, Please discard the iSCSI version received earlier today and find here a new one. By mistake we sent an incomplete copy. Sorry for the inconvenience. Julo Julian Satran - IBM Research Laboratory in Haifa (See attached file: iscsi000615.txt) Steve DeGroote on 15/06/2000 15:32:34 Please respond to Steve DeGroote To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: new working version of the iSCSI ID julian_satran@il.ibm.com wrote: > Dear colleagues, > > Here is the latest iSCSI draft. > > It is work-in-progress and will be reviewed by the design team during the > meeting > in Haifa. > > Julo > > Julian Satran - IBM Research Laboratory at Haifa > > (See attached file: iscsi000615.txt) > > ------------------------------------------------------------------------ > Name: iscsi000615.txt > iscsi000615.txt Type: Plain Text (text/plain) > Encoding: base64 > Description: Text - character set unknown Julian, I only received up to page 26. Was your attachment truncated? Steve. --0__=9Y9GvtN5f8kQTgvW2CmfSX5NGi8FufevdN5tMllV7hDYdPwFhWZauBVR Content-type: application/octet-stream; name="iscsi000615.txt" Content-Disposition: attachment; filename="iscsi000615.txt" Content-transfer-encoding: base64 Content-Description: Text - character set unknown DQoNCg0KDQoNCg0KSW50ZXJuZXQtRHJhZnQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgSi4gU2F0cmFuDQo8ZHJhZnQtc2F0cmFuLWlTQ1NJLTAzLnR4dD4g ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgRC4gU21pdGgNCkV4cGlyZXMgRGVj ZW1iZXIgMTUsIDIwMDAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgSy4g TWV0aA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIE0uIFRvbGVkYW5vDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBJQk0NCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBDLiBTYXB1bnR6YWtpcw0K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBDaXNjbyBTeXN0ZW1zDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIEp1bmUgMTUsIDIwMDANCg0KICAgICAgICAgICAgICAgICAgICAg ICAgIGlTQ1NJIChJbnRlcm5ldCBTQ1NJKQ0KDQoNClN0YXR1cyBvZiB0aGlzIE1lbW8NCg0KICAg ICBUaGlzIGRvY3VtZW50IGlzIGFuIEludGVybmV0LURyYWZ0IGFuZCBpcyBpbiBmdWxsIGNvbmZv cm1hbmNlIHdpdGgNCiAgICAgYWxsIHByb3Zpc2lvbnMgb2YgU2VjdGlvbiAxMCBvZiBSRkMyMDI2 Lg0KDQogICAgIEludGVybmV0LURyYWZ0cyBhcmUgd29ya2luZyBkb2N1bWVudHMgb2YgdGhlIElu dGVybmV0IEVuZ2luZWVyaW5nDQogICAgIFRhc2sgRm9yY2UgKElFVEYpLCBpdHMgYXJlYXMsIGFu ZCBpdHMgd29ya2luZyBncm91cHMuIE5vdGUgdGhhdA0KICAgICBvdGhlciBncm91cHMgbWF5IGFs c28gZGlzdHJpYnV0ZSB3b3JraW5nIGRvY3VtZW50cyBhcyBJbnRlcm5ldC0NCiAgICAgRHJhZnRz Lg0KDQogICAgIEludGVybmV0LURyYWZ0cyBhcmUgZHJhZnQgZG9jdW1lbnRzIHZhbGlkIGZvciBh IG1heGltdW0gb2Ygc2l4DQogICAgIG1vbnRocyBhbmQgbWF5IGJlIHVwZGF0ZWQsIHJlcGxhY2Vk LCBvciBvYnNvbGV0ZWQgYnkgb3RoZXIgZG9jdS0NCiAgICAgbWVudHMgYXQgYW55IHRpbWUuIEl0 IGlzIGluYXBwcm9wcmlhdGUgdG8gdXNlIEludGVybmV0LURyYWZ0cyBhcw0KICAgICByZWZlcmVu Y2UgbWF0ZXJpYWwgb3IgdG8gY2l0ZSB0aGVtIG90aGVyIHRoYW4gYXMgIndvcmsgaW4gcHJvLQ0K ICAgICBncmVzcy4iDQoNCiAgICAgVGhlIGxpc3Qgb2YgY3VycmVudCBJbnRlcm5ldC1EcmFmdHMg Y2FuIGJlIGFjY2Vzc2VkIGF0DQogICAgIGh0dHA6Ly93d3cuaWV0Zi5vcmcvaWV0Zi8xaWQtYWJz dHJhY3RzLnR4dA0KDQogICAgIFRoZSBsaXN0IG9mIEludGVybmV0LURyYWZ0IFNoYWRvdyBEaXJl Y3RvcmllcyBjYW4gYmUgYWNjZXNzZWQgYXQNCiAgICAgaHR0cDovL3d3dy5pZXRmLm9yZy9zaGFk b3cuaHRtbC4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNClNhdHJhbiwgU21p dGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFn ZSAxXQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoNCg0KVGFi bGUgb2YgQ29udGVudHMNCg0KICAgICAgICAxLiAgQWJzdHJhY3QNCiAgICAgICAgMi4gIE92ZXJ2 aWV3DQogICAgICAgIDIuMS4gIFNDU0kgQ29uY2VwdHMNCiAgICAgICAgMi4yLiAgaVNDU0kgRnVu Y3Rpb25hbCBPdmVydmlldw0KICAgICAgICAyLjMuICBpU0NTSSBMb2dpbg0KICAgICAgICAyLjQu ICBpU0NTSSBGdWxsIEZlYXR1cmUgUGhhc2UNCiAgICAgICAgMi41LiAgaVNDU0kgQ29ubmVjdGlv biBUZXJtaW5hdGlvbg0KICAgICAgICAyLjYuICBOYW1pbmcNCiAgICAgICAgMy4gIE1lc3NhZ2Ug Rm9ybWF0cw0KICAgICAgICAzLjEuICBUZW1wbGF0ZSBIZWFkZXINCiAgICAgICAgMy4yLiAgU0NT SSBDb21tYW5kDQogICAgICAgIDMuMy4gIFNDU0kgUmVzcG9uc2UNCiAgICAgICAgMy40LiAgQXN5 bmNocm9ub3VzIEV2ZW50DQogICAgICAgIDMuNS4gIFNDU0kgVGFzayBNYW5hZ2VtZW50IE1lc3Nh Z2UNCiAgICAgICAgMy42LiAgU0NTSSBUYXNrIE1hbmFnZW1lbnQgUmVzcG9uc2UNCiAgICAgICAg My43LiAgUmVhZHkgVG8gVHJhbnNmZXIgKFJUVCkNCiAgICAgICAgMy44LiAgU0NTSSBEYXRhDQog ICAgICAgIDMuOS4gIFRleHQgQ29tbWFuZA0KICAgICAgICAzLjEwLiAgVGV4dCBSZXNwb25zZQ0K ICAgICAgICAzLjExLiAgTG9naW4gQ29tbWFuZA0KICAgICAgICAzLjEyLiAgTG9naW4gUmVzcG9u c2UNCiAgICAgICAgMy4xMy4gIFBpbmcgQ29tbWFuZA0KICAgICAgICAzLjE0LiAgUGluZyBSZXNw b25zZQ0KICAgICAgICAzLjE1LiAgVGhpcmQgUGFydHkgQ29tbWFuZHMNCiAgICAgICAgMy4xNi4g IE9wY29kZSBOb3QgVW5kZXJzdG9vZA0KICAgICAgICA0LiAgRXJyb3IgSGFuZGxpbmcgaVNDU0kN CiAgICAgICAgNS4gIE5vdGVzIHRvIEltcGxlbWVudG9ycw0KICAgICAgICA1LjEuICBTbWFsbCBU Q1AgU2VnbWVudHMNCiAgICAgICAgNS4yLiAgTXVsdGlwbGUgTmV0d29yayBBZGFwdGVycw0KICAg ICAgICA1LjMuICBBdXRvc2Vuc2UNCiAgICAgICAgNS40LiAgVENQIFJETUEgb3B0aW9uDQogICAg ICAgIDUuNS4gIERhdGEgQ29ubmVjdGlvbnMgT3B0aW9ucw0KICAgICAgICA2LiAgU2VjdXJpdHkg Q29uc2lkZXJhdGlvbnMNCiAgICAgICAgNi4xLiAgRGF0YSBJbnRlZ3JpdHkNCiAgICAgICAgNi4y LiAgTG9naW4gUHJvY2Vzcw0KICAgICAgICA2LjMuICBJQU5BIENvbnNpZGVyYXRpb25zDQogICAg ICAgIDcuICBBdXRob3JzJyBBZGRyZXNzZXMNCiAgICAgICAgOC4gIFJlZmVyZW5jZXMgYW5kIEJp Ymxpb2dyYXBoeQ0KICAgICAgICA5LiAgQXBwZW5kaXggQSAtIEV4YW1wbGVzDQogICAgICAgIDku MS4gIFJlYWQgb3BlcmF0aW9uIGV4YW1wbGUNCiAgICAgICAgOS4yLiAgV3JpdGUgb3BlcmF0aW9u IGV4YW1wbGUNCiAgICAgICAgMTAuICBBcHBlbmRpeCBCIC0gTG9naW4vVGV4dCBrZXlzDQoNCg0K DQoNCg0KDQoNClNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBbUGFnZSAyXQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAg ICAgSnVuZSAyMDAwDQoNCg0KMS4gIEFic3RyYWN0DQoNCiAgICAgVGhlIFNtYWxsIENvbXB1dGVy IFN5c3RlbXMgSW50ZXJmYWNlIChTQ1NJKSBpcyBhIHBvcHVsYXIgZmFtaWx5IG9mDQogICAgIHBy b3RvY29scyBmb3IgY29tbXVuaWNhdGluZyB3aXRoIEkvTyBkZXZpY2VzLCBlc3BlY2lhbGx5IHN0 b3JhZ2UNCiAgICAgZGV2aWNlcy4NCg0KICAgICBUaGlzIG1lbW8gZGVzY3JpYmVzIGEgdHJhbnNw b3J0IHByb3RvY29sIGZvciBTQ1NJIHRoYXQgb3BlcmF0ZXMgb24NCiAgICAgdG9wIG9mIFRDUC4N Cg0KICAgICBUaGUgaVNDU0kgcHJvdG9jb2wgYWltcyB0byBiZSBmdWxseSBjb21wbGlhbnQgd2l0 aCB0aGUgcmVxdWlyZW1lbnRzDQogICAgIGxhaWQgb3V0IGluIHRoZSBTQ1NJIEFyY2hpdGVjdHVy ZSBNb2RlbCAtIDIgW1NBTTJdIGRvY3VtZW50Lg0KDQoyLiAgT3ZlcnZpZXcNCg0KMi4xLiAgU0NT SSBDb25jZXB0cw0KDQogICAgIFRoZSBlbmRwb2ludCBvZiBtb3N0IFNDU0kgY29tbWFuZHMgaXMg YSAibG9naWNhbCB1bml0IiAoTFVOKS4gRXhhbS0NCiAgICAgcGxlcyBvZiBsb2dpY2FsIHVuaXRz IGluY2x1ZGUgaGFyZCBkcml2ZXMsIHRhcGUgZHJpdmVzLCBDRCBhbmQgRFZEDQogICAgIGRyaXZl cywgcHJpbnRlcnMgYW5kIHByb2Nlc3NvcnMuIFdpdGhpbiB0aGUgbG9naWNhbCB1bml0IHRoZQ0K ICAgICBhYnN0cmFjdCBlbnRpdHkgdGhhdCBleGVjdXRlcyB0aGUgU0NTSSBjb21tYW5kcyBpcyBu YW1lZCB0aGUNCiAgICAgZGV2aWNlLXNlcnZlci4NCg0KICAgICBBICJ0YXJnZXQiIGlzIGEgY29s bGVjdGlvbiBvZiBsb2dpY2FsIHVuaXRzLCBpbiBnZW5lcmFsIG9mIHRoZSBzYW1lDQogICAgIGtp bmQsIGFuZCBpcyBkaXJlY3RseSBhZGRyZXNzYWJsZSBvbiB0aGUgbmV0d29yay4gSW4gbGFyZ2Ug aW5zdGFsLQ0KICAgICBsYXRpb25zIGEgdGFyZ2V0IGlzIGtub3duIGFsc28gYXMgYSAiY29udHJv bCB1bml0Ii4gIFRoZSB0YXJnZXQNCiAgICAgY29ycmVzcG9uZHMgdG8gdGhlIHNlcnZlciBpbiB0 aGUgYWJzdHJhY3QgU0FNIGNsaWVudC1zZXJ2ZXIgbW9kZWwuDQoNCiAgICAgQW4gImluaXRpYXRv ciIgY3JlYXRlcyBhbmQgc2VuZHMgU0NTSSBjb21tYW5kcyB0byB0aGUgdGFyZ2V0LiBUaGUNCiAg ICAgaW5pdGlhdG9yIGNvcnJlc3BvbmRzIHRvIHRoZSBjbGllbnQgaW4gdGhlIGFic3RyYWN0IFNB TSBjbGllbnQtDQogICAgIHNlcnZlciBtb2RlbC4NCg0KICAgICBBICJ0YXNrIiBpcyBhIGxpbmtl ZCBzZXQgb2YgU0NTSSBjb21tYW5kcy4gU29tZSBMVU5zIHN1cHBvcnQgbXVsdGktDQogICAgIHBs ZSBwZW5kaW5nIChxdWV1ZWQpIHRhc2tzLiBUaGUgdGFyZ2V0IHVzZXMgYSAidGFzayB0YWciIHRv IGRpc3Rpbi0NCiAgICAgZ3Vpc2ggYmV0d2VlbiB0YXNrcy4gT25seSBvbmUgY29tbWFuZCBpbiBh IHRhc2sgY2FuIGJlIG91dHN0YW5kaW5nDQogICAgIGF0IGFueSBnaXZlbiB0aW1lLg0KDQogICAg IEEgU0NTSSBjb21tYW5kIHJlc3VsdHMgaW4gYW4gb3B0aW9uYWwgZGF0YSBwaGFzZSBhbmQgYSBy ZXNwb25zZQ0KICAgICBwaGFzZS4gSW4gdGhlIGRhdGEgcGhhc2UsIGluZm9ybWF0aW9uIHRyYXZl bHMgZWl0aGVyIGZyb20gdGhlIGluaS0NCiAgICAgdGlhdG9yIHRvIHRoZSB0YXJnZXQsIGFzIGlu IGEgV1JJVEUgY29tbWFuZCwgb3IgZnJvbSB0YXJnZXQgdG8gaW5pLQ0KICAgICB0aWF0b3IsIGFz IGluIGEgUkVBRCBjb21tYW5kLiBJbiB0aGUgcmVzcG9uc2UgcGhhc2UsIHRoZSB0YXJnZXQNCiAg ICAgcmV0dXJucyB0aGUgZmluYWwgc3RhdHVzIG9mIHRoZSBvcGVyYXRpb24sIGluY2x1ZGluZyBh bnkgZXJyb3JzLiBBDQogICAgIHJlc3BvbnNlIHRlcm1pbmF0ZXMgYSBTQ1NJIGNvbW1hbmQuDQoN CjIuMi4gIGlTQ1NJIEZ1bmN0aW9uYWwgT3ZlcnZpZXcNCg0KICAgICBDb21tdW5pY2F0aW9uIGJl dHdlZW4gaW5pdGlhdG9yIGFuZCB0YXJnZXQgb2NjdXJzIG92ZXIgb25lIG9yIG1vcmUNCiAgICAg VENQIGNvbm5lY3Rpb25zLiAgVGhlIFRDUCBjb25uZWN0aW9ucyBhcmUgdXNlZCBmb3Igc2VuZGlu ZyBjb250cm9sDQogICAgIG1lc3NhZ2VzLCBTQ1NJIGNvbW1hbmRzLCBwYXJhbWV0ZXJzIGFuZCBk YXRhLg0KDQoNCg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIFtQYWdlIDNdDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAg ICAgICBKdW5lIDIwMDANCg0KDQoyLjMuICBpU0NTSSBMb2dpbg0KDQogICAgIFRoZSBwdXJwb3Nl IG9mIGlTQ1NJIGxvZ2luIGlzIHRvIGVuYWJsZSBhIFRDUCBjb25uZWN0aW9uIGZvciBpU0NTSQ0K ICAgICB1c2UsIGF1dGhlbnRpY2F0ZSB0aGUgcGFydGllcywgYXV0aG9yaXplIHRoZSBpbml0aWF0 b3IgdG8gc2VuZCBTQ1NJDQogICAgIGNvbW1hbmRzIGFuZCBtYXJrIHRoZSBjb25uZWN0aW9uIGFz IGJlbG9uZ2luZyB0byBhIGlTQ1NJIHNlc3Npb24uDQogICAgIEEgc2Vzc2lvbiBpcyB1c2VkIHRv IGlkZW50aWZ5IHRvIGEgdGFyZ2V0IGFsbCB0aGUgY29ubmVjdGlvbnMgd2l0aA0KICAgICBhIGdp dmVuIGluaXRpYXRvci4NCg0KICAgICBUaGUgdGFyZ2V0cyBsaXN0ZW4gb24gYSB3ZWxsLWtub3du IFRDUCBwb3J0IGZvciBpbmNvbWluZyBjb25uZWMtDQogICAgIHRpb25zLiAgVGhlIGluaXRpYXRv ciBiZWdpbnMgdGhlIGxvZ2luIHByb2Nlc3MgYnkgY29ubmVjdGluZyB0bw0KICAgICB0aGF0IHdl bGwta25vd24gVENQIHBvcnQuDQoNCiAgICAgQXMgcGFydCBvZiB0aGUgbG9naW4gcHJvY2Vzcywg dGhlIGluaXRpYXRvciBhbmQgdGFyZ2V0IE1BWSB3aXNoIHRvDQogICAgIGF1dGhlbnRpY2F0ZSBl YWNoIG90aGVyLiBUaGlzIGNhbiBvY2N1ciBpbiBtYW55IGRpZmZlcmVudCB3YXlzLiBGb3INCiAg ICAgZXhhbXBsZSwgdGhlIGVuZHBvaW50cyBtYXkgd2lzaCB0byBjaGVjayB0aGUgSVAgYWRkcmVz cyBvZiB0aGUNCiAgICAgb3RoZXIgcGFydHkuIElmIHRoZSBUQ1AgY29ubmVjdGlvbiB1c2VzIHRy YW5zcG9ydCBsYXllciBzZWN1cml0eQ0KICAgICBbVExTXSwgY2VydGlmaWNhdGVzIG1heSBiZSB1 c2VkIHRvIGlkZW50aWZ5IHRoZSBlbmRwb2ludHMuIEFsc28sDQogICAgIGlTQ1NJIGluY2x1ZGVz IGNvbW1hbmRzIGZvciBpZGVudGlmeWluZyB0aGUgaW5pdGlhdG9yIGFuZCBwYXNzaW5nDQogICAg IGFuIGF1dGhlbnRpY2F0b3IgdG8gdGhlIHRhcmdldCAoc2VlIEFwcGVuZGl4IEIpLg0KDQogICAg IE9uY2Ugc3VpdGFibGUgYXV0aGVudGljYXRpb24gaGFzIG9jY3VycmVkLCB0aGUgdGFyZ2V0IE1B WSBhdXRob3JpemUNCiAgICAgdGhlIGluaXRpYXRvciB0byBzZW5kIFNDU0kgY29tbWFuZHMuIEhv dyB0aGUgdGFyZ2V0IGNob29zZXMgdG8NCiAgICAgYXV0aG9yaXplIGFuIGluaXRpYXRvciBpcyBi ZXlvbmQgdGhlIHNjb3BlIG9mIHRoaXMgZG9jdW1lbnQuDQoNCiAgICAgVGhlIHRhcmdldCBpbmRp Y2F0ZXMgYSBzdWNjZXNzZnVsIGF1dGhlbnRpY2F0aW9uIGFuZCBhdXRob3JpemF0aW9uDQogICAg IGJ5IHNlbmRpbmcgYSBsb2dpbiByZXNwb25zZSB3aXRoICJhY2NlcHQgbG9naW4iLg0KDQogICAg IFRoZSBsb2dpbiBtZXNzYWdlIGluY2x1ZGVzIGEgc2Vzc2lvbiBJRCAtIGNvbXBvc2VkIHdpdGgg YW4gaW5pdGlhLQ0KICAgICB0b3IgcGFydCBJU0lEIGFuZCBhIHRhcmdldCBwYXJ0IFRTSUQuIEZv ciBhIG5ldyBzZXNzaW9uIHRoZSBUU0lEIGlzDQogICAgIG51bGwuICBBcyBwYXJ0IG9mIHRoZSBy ZXNwb25zZSB0aGUgdGFyZ2V0IHdpbGwgZ2VuZXJhdGUgYSBUU0lELg0KDQogICAgIFNlc3Npb24g c3BlY2lmaWMgcGFyYW1ldGVycyBjYW4gYmUgc3BlY2lmaWVkIG9ubHkgZm9yIHRoZSBmaXJzdA0K ICAgICBsb2dpbiBvZiBhIHNlc3Npb24gKFRTSUQgbnVsbCkoZS5nIHRoZSBtYXhpbXVtIG51bWJl ciBvZiBjb25uZWMtDQogICAgIHRpb25zIHRoYXQgY2FuIGJlIHVzZWQgZm9yIHRoaXMgc2Vzc2lv bikuIENvbm5lY3Rpb24gc3BlY2lmaWMNCiAgICAgcGFyYW1ldGVycyAoaWYgYW55KSBjYW4gYmUg c3BlY2lmaWVkIGZvciBhbnkgbG9naW4uIFRodXMgYSBzZXNzaW9uDQogICAgIGlzIG9wZXJhdGlv bmFsIG9uY2UgaXQgaGFzIGF0IGxlYXN0IG9uZSBjb25uZWN0aW9uIGFuZCBhIHBlbmRpbmcNCiAg ICAgbG9naW4gY2FuJ3QgYWZmZWN0IGEgd2hvbGUgc2Vzc2lvbi4NCg0KICAgICBBZnRlciBhdXRo ZW50aWNhdGlvbiBhbmQgYXV0aG9yaXphdGlvbiwgb3RoZXIgcGFyYW1ldGVycyBtYXkgYmUNCiAg ICAgbmVnb3RpYXRlZCB1c2luZyB0aGUgaGlnaGx5IGV4dGVuc2libGUgVGV4dCBDb21tYW5kIG1l c3NhZ2UgdGhhdA0KICAgICBhbGxvd3MgYXJiaXRyYXJ5IGtleTp2YWx1ZSBwYWlycyB0byBiZSBw YXNzZWQuDQoNCiAgICAgQW55IG1lc3NhZ2Ugc2VudCBvbiBhIFRDUCBjb25uZWN0aW9uIGJlZm9y ZSB0aGlzIGNvbm5lY3Rpb24gZ2V0cw0KICAgICBpbnRvIGZ1bGwgZmVhdHVyZSBwaGFzZSBhdCB0 aGUgaW5pdGlhdG9yIHNob3VsZCBiZSByZWplY3RlZCBieSB0aGUNCiAgICAgaW5pdGlhdG9yLiBB IG1lc3NhZ2UgcmVhY2hpbmcgYSB0YXJnZXQgb24gYSBUQ1AgY29ubmVjdGlvbiBiZWZvcmUNCiAg ICAgdGhlIGZ1bGwgZmVhdHVyZSBwaGFzZSB3aWxsIGJlIHJlamVjdCB3aXRoIGFuIGlTQ1NJIGNo ZWNrIGNvbmRpdGlvbg0KICAgICBiaXQuDQoNCg0KDQoNClNhdHJhbiwgU21pdGgsIFNhcHVudHph a2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSA0XQ0KDA0KaVND U0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoNCg0KMi40LiAgaVNDU0kgRnVs bCBGZWF0dXJlIFBoYXNlDQoNCiAgICAgT25jZSB0aGUgaW5pdGlhdG9yIGlzIGF1dGhvcml6ZWQg dG8gZG8gc28sIHRoZSBpU0NTSSBzZXNzaW9uIGlzIGluDQogICAgIGlTQ1NJIGZ1bGwgZmVhdHVy ZSBwaGFzZS4gVGhlIGluaXRpYXRvciBtYXkgc2VuZCBTQ1NJIGNvbW1hbmRzIGFuZA0KICAgICBk YXRhIHRvIHRoZSB2YXJpb3VzIExVTnMgb24gdGhlIHRhcmdldCBieSBtYXBwaW5nIHRoZW0gaW4g aVNDU0kNCiAgICAgbWVzc2FnZXMgdGhhdCBnbyBvdmVyIHRoZSBlc3RhYmxpc2hlZCBpU0NTSSBz ZXNzaW9uLg0KDQogICAgIEZvciBTQ1NJIGNvbW1hbmRzIHRoYXQgcmVxdWlyZSBkYXRhIGFuZC9v ciBwYXJhbWV0ZXIgdHJhbnNmZXIsIHRoZQ0KICAgICAob3B0aW9uYWwpZGF0YSBhbmQgdGhlIHN0 YXR1cyBmb3IgYSBjb21tYW5kIG11c3QgYmUgc2VudCBvdmVyIHRoZQ0KICAgICBzYW1lIFRDUCBj b25uZWN0aW9uIHRoYXQgd2FzIHVzZWQgdG8gZGVsaXZlciB0aGUgU0NTSSBjb21tYW5kIChjb24t DQogICAgIG5lY3Rpb24gYWxsZWdpYW5jZSkuICBUaHVzIGlmIGFuIGluaXRpYXRvciBpc3N1ZXMg YSBSRUFEIGNvbW1hbmQsDQogICAgIHRoZSB0YXJnZXQgbXVzdCBzZW5kIHRoZSByZXF1ZXN0ZWQg ZGF0YSBmb2xsb3dlZCBieSB0aGUgc3RhdHVzIHRvDQogICAgIHRoZSBpbml0aWF0b3Igb3ZlciB0 aGUgc2FtZSBUQ1AgY29ubmVjdGlvbiB0aGF0IHdhcyB1c2VkIHRvIGRlbGl2ZXINCiAgICAgdGhl IFNDU0kgY29tbWFuZC4gIElmIGFuIGluaXRpYXRvciBpc3N1ZXMgYSBXUklURSBjb21tYW5kLCB0 aGUgaW5pLQ0KICAgICB0aWF0b3IgbXVzdCBzZW5kIHRoZSBkYXRhIGZvciB0aGF0IGNvbW1hbmQg YW5kIHRoZSB0YXJnZXQgbXVzdA0KICAgICByZXR1cm4gdGhlIHN0YXR1cyBvdmVyIHRoZSBzYW1l IFRDUCBjb25uZWN0aW9uIHRoYXQgd2FzIHVzZWQgdG8NCiAgICAgZGVsaXZlciB0aGUgU0NTSSBj b21tYW5kLg0KDQogICAgIER1cmluZyBpU0NTSSBGdWxsIEZlYXR1cmUgUGhhc2UsIHRoZSBpbml0 aWF0b3IgYW5kIHRhcmdldCBtYXkNCiAgICAgaW50ZXJsZWF2ZSB1bnJlbGF0ZWQgU0NTSSBjb21t YW5kcywgdGhlaXIgU0NTSSBEYXRhIGFuZCByZXNwb25zZXMsDQogICAgIG92ZXIgdGhlIHNlc3Np b24uDQoNCiAgICAgU0NTSSBEYXRhIHBhY2tldHMgYXJlIG1hdGNoZWQgdG8gdGhlaXIgY29ycmVz cG9uZGluZyBTQ1NJIGNvbW1hbmRzDQogICAgIGJ5IHVzaW5nIFRhZ3MgdGhhdCBhcmUgc3BlY2lm aWVkIGluIHRoZSBwcm90b2NvbC4gIEluaXRpYXRvciB0YWdzDQogICAgIGZvciBwZW5kaW5nIGNv bW1hbmRzIGFyZSB1bmlxdWUgaW5pdGlhdG9yLXdpZGUgZm9yIGEgc2Vzc2lvbi4gIFRhci0NCiAg ICAgZ2V0IHRhZ3MgZm9yIHBlbmRpbmcgY29tbWFuZHMgYXJlIHVuaXF1ZSB0YXJnZXQtd2lkZSBm b3IgdGhlIHNlcy0NCiAgICAgc2lvbi4NCg0KICAgICBFYWNoIGlTQ1NJIHNlc3Npb24gdG8gYSB0 YXJnZXQgaXMgdHJlYXRlZCBhcyBpZiBpdCBvcmlnaW5hdGVkIGZyb20NCiAgICAgYSBkaWZmZXJl bnQgaW5pdGlhdG9yLg0KDQoyLjUuICBpU0NTSSBDb25uZWN0aW9uIFRlcm1pbmF0aW9uDQoNCiAg ICAgR3JhY2VmdWwgVENQIGNvbm5lY3Rpb24gc2h1dGRvd25zIGFyZSBkb25lIGJ5IHNlbmRpbmcg VENQIEZJTnMuDQogICAgIEdyYWNlZnVsIGNvbm5lY3Rpb24gc2h1dGRvd25zIE1VU1Qgb25seSBv Y2N1ciB3aGVuIHRoZXJlIGFyZSBubw0KICAgICBvdXRzdGFuZGluZyB0YXNrcyBvbiB0aGUgY29u bmVjdGlvbi4gQSB0YXJnZXQgU0hPVUxEIHJlc3BvbmQNCiAgICAgcmFwaWRseSB0byBhIEZJTiBm cm9tIHRoZSBpbml0aWF0b3IgYnkgY2xvc2luZyBpdHMgaGFsZiBvZiB0aGUgY29uLQ0KICAgICBu ZWN0aW9uIGFzIHNvb24gYXMgaXQgaGFzIGZpbmlzaGVkIGFsbCBvdXRzdGFuZGluZyB0YXNrcyB0 aGF0IGhhdmUNCiAgICAgYWxsZWdpYW5jZSB0byB0aGUgY29ubmVjdGlvbi4NCg0KICAgICBUaGUg Y2xvc2luZyBvZiBvbmUgVENQIGNvbm5lY3Rpb24gbWF5IGFmZmVjdCB0aGUgZXhlY3V0aW9uIGNv bW1hbmRzDQogICAgIGRlbGl2ZXJlZCBvdmVyIGRpZmZlcmVudCBUQ1AgY29ubmVjdGlvbnMuIElm IGEgVENQIGNvbm5lY3Rpb24gaXMNCiAgICAgY2xvc2VkLCB0aGUgdGFyZ2V0IHNob3VsZCBjbGVh biB1cCBhbGwgb2YgaXRzIHN0YXRlIGFzc29jaWF0ZWQgd2l0aA0KICAgICBjb21tYW5kcyB0aGF0 IHdlcmUgc2VudCBvdmVyIHRoYXQgY29ubmVjdGlvbjsgYWxsIG91dHN0YW5kaW5nIHRhc2tzDQog ICAgIGFyZSBjYW5jZWxsZWQgYW5kIGFsbCByZXNvdXJjZXMgdGhhdCB3ZXJlIGFsbG9jYXRlZCBm b3IgdGhvc2UgdGFza3MNCiAgICAgY2FuIGJlIGZyZWVkLiBXaGVuIGRldGVjdGluZyBhIGNsb3Nl ZCBjb25uZWN0aW9uIGFuIGluaXRpYXRvcg0KICAgICBzaG91bGQgZW5kIGFsbCBvdXRzdGFuZGlu ZyBjb21tYW5kcyBmb3IgdGhlIGNvbm5lY3Rpb24gd2l0aCBhIGNoZWNrDQogICAgIGNvbmRpdGlv biwgbWFyayBhbGwgdGhlIGxvZ2ljYWwgdW5pdHMgdGhhdCBoYWQgcGVuZGluZyBjb21tYW5kcyBh bmQNCg0KDQoNClNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBbUGFnZSA1XQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAg ICAgSnVuZSAyMDAwDQoNCg0KICAgICBzZW5kIExVTnMgcmVzZXQgY29tbWFuZHMgdG8gYWxsIHRo b3NlIHVuaXRzIHVudGlsIGEgIHN1YnNlcXVlbnQNCiAgICAgZW5xdWlyeSBzaG93cyBlbXB0eSBx dWV1ZXMuICBBIHNpbXBsZXIgYnUgbW9yZSByYWRpY2FsIGJlaGF2aW91cg0KICAgICBpbnZvbHZp bmcgYSB0YXJnZXQgcmVzZXQoPykgaW52b2x2aW5nIGFsbCBpdCdzIExVTnMgaXMgYWxzbyBhY2Nl cHQtDQogICAgIGFibGUgaW4gY2FzZSBvZiBjb25uZWN0aW9uIGZhaWx1cmUgd2hpbGUgY29tbWFu ZHMgd2hlcmUgcGVuZGluZy4NCg0KMi42LiAgTmFtaW5nDQoNCiAgICAgVGFyZ2V0cyBhcmUgbmFt ZWQgdXNpbmcgYW4gVVJMIHR5cGUgbmFtZSBvZiB0aGUgZm9ybWF0Og0KDQogICAgICAgICAgICAg c2NzaTovLzxkb21haW4tbmFtZT5bL21vZGlmaWVyXQ0KDQoNCiAgICAgVGhlIG5hbWUgdXNlZCB0 byBjb25uZWN0IHdpbGwgYmUgb3B0aW9uYWxseSBpbmNsdWRlZCBpbiB0aGUgbG9naW4NCiAgICAg aW4gb3JkZXIgdG8gZW5hYmxlIHRoZSB0YXJnZXQgdG8gcHJlc2VudCBkaWZmZXJlbnQgdmlld3Mu IFRoaXMgaXMNCiAgICAgdGhlIFRhcmdldCBBY3F1aXJlZCBOYW1lIChUQU4pLiAgV2Ugd2lsbCBu b3QgYXR0ZW1wdCB0byBkZWZpbmUNCiAgICAgd2hpY2ggY29tcG9uZW50cyBvZiB0aGUgbmFtZSB3 aWxsIHBhcnRpY2lwYXRlIGluIHRoZSBuYW1lIHJlc29sdS0NCiAgICAgdGlvbiBwcm9jZXNzIGFu ZCB3aGljaCBvbmVzIHdpbGwgYmUgdXNlZCBvbmx5IGZvciAidmlldyIgZGVmaW5pLQ0KICAgICB0 aW9uLiBUaGUgc3ludGFjdGljIHN1Z2FyIGluY2x1ZGVkIG1pZ2h0IGJlIHVzZWQgdG8gaW50cm9k dWNlDQogICAgIHN0cnVjdHVyZSBmb3IgbWFuYWdlbWVudCBwdXJwb3NlcyBidXQgaGFzIG5vIHNw ZWNpZmljIHNpZ25pZmljYW5jZQ0KICAgICBmb3IgdGhpcyBzdGFuZGFyZC4NCg0KICAgICBFeGFt cGxlOg0KDQogICAgICAgICAgICAgc2NzaTovL2Rpc2tmYXJtMS5hY21lLmNvbQ0KICAgICAgICAg ICAgIHNjc2k6Ly9jb21wdXRpbmdjZW50ZXIuYWNtZS5jb20vcGVyaXBoZXJhbHMvZGlza2Zhcm0x DQoNCg0KDQogICAgIFdoZW4gYSB0YXJnZXQgaGFzIHRvIGFjdCBhcyBhbiBpbml0aWF0b3IgZm9y IGEgdGhpcmQgcGFydHkgY29tbWFuZA0KICAgICBpdCB3aWxsIHVzZSB0aGUgVEFOIGR1cmluZyBs b2dpbiBhcyByZXF1aXJlZCBieSB0aGUgYXV0aGVudGhpY2F0aW9uDQogICAgIG1lY2hhbmlzbS4N Cg0KICAgICBBIGRvbWFpbiBuYW1lIHRoYXQgY29udGFpbnMgZXhhY3RseSBmb3VyIG51bWJlcnMg c2VwYXJhdGVkIGJ5IGRvdHMNCiAgICAgKC4pLCB3aGVyZSBlYWNoIG51bWJlciBpcyBpbiB0aGUg cmFuZ2UgMCB0aHJvdWdoIDI1NSwgd2lsbCBiZQ0KICAgICBpbnRlcnByZXRlZCBhcyBhbiBJUHY0 IGFkZHJlc3MuICBFeGFtcGxlczoNCg0KICAgICAgICAgICAgIDEwLjAuMC4xL3RhcGVmYXJtMQ0K ICAgICAgICAgICAgIDEwLjAuMC4yDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNClNhdHJhbiwg U21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBb UGFnZSA2XQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoNCg0K My4gIE1lc3NhZ2UgRm9ybWF0cw0KDQogICAgIEFsbCBtdWx0aS1ieXRlIGludGVnZXJzIHNwZWNp ZmllZCBpbiBmb3JtYXRzIGRlZmluZWQgaW4gdGhpcyBkb2N1LQ0KICAgICBtZW50IGFyZSB0byBi ZSByZXByZXNlbnRlZCBpbiBuZXR3b3JrIGJ5dGUgb3JkZXIgKGkuZS4sIGJpZw0KICAgICBlbmRp YW4pLg0KDQozLjEuICBUZW1wbGF0ZSBIZWFkZXIgYW5kIE9wY29kZXMNCg0KICAgICBBbGwgaVND U0kgbWVzc2FnZXMgYW5kIHJlc3BvbnNlcyBoYXZlIGEgaGVhZGVyIG9mIHRoZSBzYW1lIGxlbmd0 aA0KICAgICAoNDAgYnl0ZXMpLiBBZGRpdGlvbmFsIGRhdGEgbWF5IGJlIGFkZGVkLCBhcyBuZWNl c3NhcnksIGJlZ2lubmluZw0KICAgICB3aXRoIGJ5dGUgNDAuIFRoZSBmaWVsZHMgb2YgT3Bjb2Rl IGFuZCBMZW5ndGggYXBwZWFyIGluIGFsbCBtZXNzYWdlDQogICAgIGFuZCByZXNwb25zZSBoZWFk ZXJzLiBUaGUgb3RoZXIgbW9zdCBjb21tb25seSB1c2VkIGZpZWxkcyBhcmUgSW5pLQ0KICAgICB0 aWF0b3IgVGFzayBUYWcsIExvZ2ljYWwgVW5pdCBOdW1iZXIsIGFuZCBGbGFncywgd2hpY2gsIHdo ZW4gdXNlZCwNCiAgICAgYWx3YXlzIGFwcGVhciBpbiB0aGUgc2FtZSBsb2NhdGlvbiBvZiB0aGUg aGVhZGVyLg0KDQogICAgIEJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAg IDIgICAgICAgfCAgICAgICAzICAgICAgIHwNCiAgICAgICAgLyAgICAgICAgICAgICAgfCAgICAg ICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfA0KICAgICAgIHw3IDYg NSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAx IDB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDB8IE9wY29kZSAgICAgICAgfCAgICAgT3Bjb2Rl LXNwZWNpZmljIGZpZWxkcyAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQog ICAgICA0fCBMZW5ndGggb2YgRGF0YSAoYWZ0ZXIgNDAgYnl0ZSBIZWFkZXIpICAgICAgICAgICAg ICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgOHwgTFVOIG9yIE9wY29kZS1z cGVjaWZpYyBmaWVsZHMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAg KyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICsNCiAgICAgMTJ8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDE2fCBJbml0 aWF0b3IgVGFzayBUYWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAyMC8gT3Bjb2RlLXNwZWNpZmljIGZpZWxkcyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICArLyAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKw0KICAgICA0MA0KDQoNCjMuMS4xLiAgT3Bjb2RlDQoNCiAgICAgVGhlIE9w Y29kZSBpbmRpY2F0ZXMgd2hpY2ggaVNDU0kgdHlwZSBvZiBtZXNzYWdlIG9yIHJlc3BvbnNlIGlz DQogICAgIGVuY2Fwc3VsYXRlZCBieSB0aGUgaGVhZGVyLiAgVmFsaWQgb3Bjb2RlcyBmb3IgbWVz c2FnZXMgKHNlbnQgYnkNCiAgICAgaW5pdGlhdG9yIHRvIHRhcmdldCkgYXJlOg0KICAgICAgIDB4 MDAgUGluZyBDb21tYW5kIChmcm9tIGluaXRpYXRvciB0byB0YXJnZXQpDQogICAgICAgMHgwMSBT Q1NJIENvbW1hbmQgKGVuY2Fwc3VsYXRlcyBhIFNDU0kgQ29tbWFuZCBEZXNjcmlwdG9yIEJsb2Nr KQ0KICAgICAgIDB4MDIgU0NTSSBUYXNrIE1hbmFnZW1lbnQgTWVzc2FnZQ0KICAgICAgIDB4MDMg TG9naW4gQ29tbWFuZA0KICAgICAgIDB4MDQgVGV4dCBDb21tYW5kDQogICAgICAgMHgwNSBTQ1NJ IERhdGEgKGZvciBXUklURSBvcGVyYXRpb24pDQoNCiAgICAgVmFsaWQgb3Bjb2RlcyBmb3IgcmVz cG9uc2VzIChzZW50IGJ5IHRhcmdldCB0byBpbml0aWF0b3IpIGFyZToNCg0KDQoNClNhdHJhbiwg U21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBb UGFnZSA3XQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoNCg0K ICAgICAgIDB4ODAgUGluZyBSZXNwb25zZSAoZnJvbSB0YXJnZXQgdG8gaW5pdGlhdG9yKQ0KICAg ICAgIDB4ODEgU0NTSSBSZXNwb25zZSAoY29udGFpbnMgU0NTSSBzdGF0dXMgYW5kIHBvc3NpYmx5 IHNlbnNlDQogICAgICAgICAgICBpbmZvcm1hdGlvbiBvciBvdGhlciByZXNwb25zZSBpbmZvcm1h dGlvbikNCiAgICAgICAweDgyIFNDU0kgVGFzayBNYW5hZ2VtZW50IFJlc3BvbnNlDQogICAgICAg MHg4MyBMb2dpbiBSZXNwb25zZQ0KICAgICAgIDB4ODQgVGV4dCBSZXNwb25zZQ0KICAgICAgIDB4 ODUgU0NTSSBEYXRhIChmb3IgUkVBRCBvcGVyYXRpb24pDQogICAgICAgMHg4NiBSZWFkeSBUbyBU cmFuc2ZlciAoUlRUIC0gc2VudCBieSB0YXJnZXQgdG8gaW5pdGlhdG9yIHdoZW4NCiAgICAgICAg ICAgIGl0IGlzIHJlYWR5IHRvIHJlY2VpdmUgZGF0YSBmcm9tIGluaXRpYXRvcikNCiAgICAgICAw eDg3IEFzeW5jaHJvbm91cyBFdmVudCAoc2VudCBieSB0YXJnZXQgdG8gaW5pdGlhdG9yIHRvIGlu ZGljYXRlDQogICAgICAgY2VydGFpbiBzcGVjaWFsIGNvbmRpdGlvbnMpDQogICAgICAgMHg4OCBP cGNvZGUgTm90IFVuZGVyc3Rvb2QNCiAgICAgICAweDg5IE9wZW4gRGF0YSBDb25uZWN0aW9ucyBS ZXNwb25zZSAob3B0aW9uYWwpDQoNCjMuMS4yLiAgTGVuZ3RoDQoNCiAgICAgVGhlIExlbmd0aCBm aWVsZCBpbmRpY2F0ZXMgdGhlIG51bWJlciBvZiBieXRlcywgYmV5b25kIHRoZSBmaXJzdCA0MA0K ICAgICBieXRlcywgdGhhdCBhcmUgYmVpbmcgc2VudCB0b2dldGhlciB3aXRoIHRoaXMgbWVzc2Fn ZSBoZWFkZXIuIEl0IGlzDQogICAgIGFudGljaXBhdGVkIHRoYXQgbW9zdCBpU0NTSSBtZXNzYWdl cyBhbmQgcmVzcG9uc2VzIChub3QgY291bnRpbmcNCiAgICAgZGF0YSB0cmFuc2ZlciBtZXNzYWdl cykgd2lsbCBub3QgbmVlZCBtb3JlIHRoYW4gdGhlIDQwIGJ5dGUgaGVhZGVyLA0KICAgICBhbmQg aGVuY2UgdGhlIExlbmd0aCBmaWVsZCB3aWxsIGNvbnRhaW4gdGhlIHZhbHVlIDAuICBJdCBpcw0K ICAgICBleHBlY3RlZCB0aGF0IGxhcmdlciB0aGFuIDE2IGJ5dGUgQ0RCcyBhbmQgcGFyYW1ldGVy IGRhdGEgd2lsbCBmb2wtDQogICAgIGxvdyB0aGUgaGVhZGVyLg0KDQozLjEuMy4gIExVTg0KDQog ICAgIFRoZSBMVU4gc3BlY2lmaWVzIHRoZSBMb2dpY2FsIFVuaXQgZm9yIHdoaWNoIHRoZSBjb21t YW5kIGlzIHRhci0NCiAgICAgZ2V0ZWQuICBJZiB0aGUgY29tbWFuZCBkb2VzIG5vdCByZWxhdGUg dG8gYSBMb2dpY2FsIFVuaXQsIHRoaXMNCiAgICAgZmllbGQgaXMgZWl0aGVyIGlnbm9yZWQgb3Ig bWF5IGJlIHVzZWQgZm9yIHNvbWUgb3RoZXIgcHVycG9zZS4NCiAgICAgQWNjb3JkaW5nIHRvIFtT QU0yXSwgYSBMb2dpY2FsIFVuaXQgTnVtYmVyIGNhbiB0YWtlIHVwIHRvIGEgNjQtYml0DQogICAg IGZpZWxkIHRoYXQgaWRlbnRpZmllcyB0aGUgTG9naWNhbCBVbml0IHdpdGhpbiBhIHRhcmdldCBk ZXZpY2UuIFRoZQ0KICAgICBleGFjdCBmb3JtYXQgb2YgdGhpcyBmaWVsZCBjYW4gYmUgZm91bmQg aW4gdGhlIFtTQU0yXSBkb2N1bWVudC4NCg0KMy4xLjQuICBJbml0aWF0b3IgVGFzayBUYWcNCg0K ICAgICBUaGUgaW5pdGlhdG9yIGFzc2lnbnMgYSBUYXNrIElkIChvciB0YWcpIHRvIGVhY2ggU0NT SSB0YXNrIHRoYXQgaXQNCiAgICAgaXNzdWVzLiAgKFJlY2FsbCB0aGF0IGEgdGFzayBpcyBhIGxp bmtlZCBzZXQgb2YgU0NTSSBjb21tYW5kcy4pDQogICAgIFRoaXMgVGFnIGlzIGEgaW5pdGlhdG9y LXdpZGUgdW5pcXVlIGlkZW50aWZpZXIgdGhhdCBjYW4gYmUgdXNlZCB0bw0KICAgICB1bmlxdWVs eSBpZGVudGlmeSB0aGUgVGFzay4NCg0KMy4xLjUuICBPcGNvZGUtc3BlY2lmaWMgZmllbGRzDQoN CiAgICAgVGhlc2UgZmllbGQgaGF2ZSBkaWZmZXJlbnQgbWVhbmluZ3MgZm9yIGRpZmZlcmVudCBt ZXNzYWdlcy4NCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0 aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgOF0NCgwNCmlTQ1NJICAgICAg ICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuMi4gIFNDU0kgQ29tbWFuZA0KDQoN CiAgICAgQnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwgICAgICAgMiAgICAgICB8 ICAgICAgIDMgICAgICAgfA0KICAgICAgICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAg fCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8DQogICAgICAgfDcgNiA1IDQgMyAyIDEg MHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwNCiAgICAg ICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKw0KICAgICAgMHwgT3Bjb2RlICgweDAxKSB8ICAgIENtZFJOICAgICAgfEF8IFdM Q0RCLTQgfFJ8V3xSc3J2ZCAoMCl8QVRUUiB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDR8IExl bmd0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICA4fCBMb2dpY2FsIFVuaXQgTnVtYmVyIChMVU4p ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKw0K ICAgICAxMnwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMTZ8IEluaXRpYXRvciBUYXNr IFRhZyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAg ICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rDQogICAgIDIwfCBFeHBlY3RlZCBEYXRhIFRyYW5zZmVyIExlbmd0aCAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAyNHwgU0NT SSBDb21tYW5kIERlc2NyaXB0b3IgQmxvY2sgKENEQikgICAgICAgICAgICAgICAgICAgICAgICAg ICB8DQogICAgICAgKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICsNCiAgICAgMjh8ICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICsgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICArDQog ICAgIDMyfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNCiAgICAgICArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKw0KICAgICAzNnwgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNCiAgICAgNDAvIEFkZGl0aW9uYWwgRGF0YSAoQ29tbWFuZCBEZXBlbmRlbnQpICAg ICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgKy8gICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICAgKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNCg0KDQozLjIuMS4gIENvbW1hbmQgUmVmZXJlbmNlIE51bWJlciAoQ21kUk4pDQoNCiAgICAg VGhlIENvbW1hbmQgUmVmZXJlbmNlIE51bWJlciAoQ21kUk4pIGlzIHByb3ZpZGVkIGJ5IHRoZSBp bml0aWF0b3INCiAgICAgdG8gYXNzaXN0IGluIHBlcmZvcm1pbmcgb3JkZXJlZCBkZWxpdmVyeSBm b3IgaVNDU0kgY29tbWFuZHMuICBJZg0KICAgICBwcmVjaXNlIGRlbGl2ZXJ5IGlzIGVuYWJsZWQg dGhyb3VnaCBhIFRleHQgY29tbWFuZCwgYSBub256ZXJvIHZhbHVlDQogICAgIGZvciBDbWRSTiBp cyB0byBiZSB0cmVhdGVkIGFzIGEgY29tbWFuZCByZWZlcmVuY2UgbnVtYmVyIGluIGRldGVyLQ0K ICAgICBtaW5pbmcgdGhlIHJlY2VpcHQgYW5kIG9yZGVyaW5nIG9mIGNvbW1hbmRzIGZyb20gdGhl IGluaXRpYXRvciB0bw0KICAgICB0aGF0IGxvZ2ljYWwgdW5pdC4gQSB6ZXJvIHZhbHVlIG9mIENt ZFJOIGlzIHRvIGJlIGlnbm9yZWQgYW5kIHRoYXQNCiAgICAgY29tbWFuZCBzaGFsbCBub3QgYmUg dmVyaWZpZWQgZm9yIHByZWNpc2UgZGVsaXZlcnkuDQoNCjMuMi4yLiAgRmxhZ3MgJiBNaXNjLg0K DQogICAgIFRoZSBGbGFncyBmaWVsZCBmb3IgYSBTQ1NJIENvbW1hbmQgY29uc2lzdHMgb2YgdHdv IGJ5dGVzLiAoSW4gZ2VuLQ0KICAgICBlcmFsLCBvbmUgYnl0ZSBpcyB1c2VkIGZvciBkYXRhIGZs b3cgaW5mb3JtYXRpb24gd2hpbGUgdGhlIG90aGVyDQogICAgIGJ5dGUgaXMgdXNlZCBmb3IgVGFz ayBBdHRyaWJ1dGVzIGluZm9ybWF0aW9uLikNCiAgICAgICAgICAgICAgIEJ5dGUgMg0KDQoNCg0K U2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIFtQYWdlIDldDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIw MDANCg0KDQogICAgICAgICAgYjAgICAoVykgc2V0IHdoZW4gZGF0YSBpcyBleHBlY3RlZCB0byBm bG93IGZyb20gaW5pdGlhdG9yIHRvDQogICAgICAgICAgICAgICB0YXJnZXQgKHdyaXRlKS4NCiAg ICAgICAgICBiMSAgIChSKSBzZXQgd2hlbiBkYXRhIGlzIGV4cGVjdGVkIHRvIGZsb3cgZnJvbSB0 YXJnZXQgdG8gaW5pLQ0KICAgICAgICAgICAgICAgdGlhdG9yIChyZWFkKS4NCiAgICAgICAgICBi MiAgIChBKSBzZXQgdG8gdHVybiBvZmYgQXV0b3NlbnNlIGZvciB0aGlzIGNvbW1hbmQgKHNlZQ0K ICAgICAgICAgICAgICAgW1NBTTJdKS4NCiAgICAgICAgICBiMy03IGluZGljYXRlIHRoZSBsZW5n dGggb2YgQ0RCIGluIHdvcmRzIG92ZXIgNCAoZm9yIGEgc3Rhbi0NCiAgICAgICAgICAgICAgIGRh cmQgQ0RCIGl0IGlzIDApDQoNCiAgICAgQXV0b3NlbnNlIHJlZmVycyB0byB0aGUgYXV0b21hdGlj IHJldHVybiBvZiBzZW5zZSBkYXRhIHRvIHRoZSBpbmktDQogICAgIHRpYXRvciBpbiBjYXNlIGEg Y29tbWFuZCBkaWQgbm90IGNvbXBsZXRlIHN1Y2Nlc3NmdWxseS4gSWYNCiAgICAgYXV0b3NlbnNl IGlzIHR1cm5lZCBvZmYsIHRoZSBpbml0aWF0b3IgbXVzdCBleHBsaWNpdGx5IHJlcXVlc3QgdGhh dA0KICAgICBzZW5zZSBkYXRhIGJlIHNlbnQgdG8gaXQgYWZ0ZXIgc29tZSBjb21tYW5kIGhhcyBj b21wbGV0ZWQgd2l0aCBhDQogICAgIENIRUNLIENPTkRJVElPTiBzdGF0dXMuDQogICAgICAgICAg ICAgICBCeXRlIDMNCiAgICAgICAgICBiMC0yIHVzZWQgdG8gaW5kaWNhdGUgVGFzayBBdHRyaWJ1 dGVzLg0KICAgICAgICAgIGIzLTcgbm90IHVzZWQgKHNob3VsZCBiZSBzZXQgdG8gMCkuDQoNCjMu Mi4zLiAgVGFzayBBdHRyaWJ1dGVzDQoNCiAgICAgVGhlIFRhc2sgQXR0cmlidXRlIGZpZWxkIChB VFRSKSBjYW4gaGF2ZSBvbmUgb2YgdGhlIGZvbGxvd2luZw0KICAgICBpbnRlZ2VyIHZhbHVlcyAo c2VlIFtTQU0yXSBmb3IgZGV0YWlscyk6DQoNCiAgICAgICAgICAgICAwICAgIFVudGFnZ2VkDQog ICAgICAgICAgICAgMSAgICBTaW1wbGUNCiAgICAgICAgICAgICAyICAgIE9yZGVyZWQNCiAgICAg ICAgICAgICAzICAgIEhlYWQgb2YgUXVldWUNCiAgICAgICAgICAgICA0ICAgIEFDQQ0KDQoNCjMu Mi40LiAgRXhwZWN0ZWQgRGF0YSBUcmFuc2ZlciBMZW5ndGgNCg0KICAgICBUaGUgRXhwZWN0ZWQg RGF0YSBUcmFuc2ZlciBMZW5ndGggZmllbGQgc3RhdGVzIHRoZSBudW1iZXIgb2YgYnl0ZXMNCiAg ICAgb2YgZGF0YSB0aGF0IHRoZSBpbml0aWF0b3IgZXhwZWN0cyB3aWxsIGJlIHNlbnQgZm9yIHRo aXMgKFJFQUQgb3INCiAgICAgV1JJVEUpIFNDU0kgb3BlcmF0aW9uIGluIFNDU0kgRGF0YSBwYWNr ZXRzLg0KDQogICAgIEZvciBhIFdSSVRFIG9wZXJhdGlvbiwgdGhlIGluaXRpYXRvciB1c2VzIHRo aXMgZmllbGQgdG8gc3BlY2lmeSB0aGUNCiAgICAgbnVtYmVyIG9mIGJ5dGVzIG9mIGRhdGEgaXQg ZXhwZWN0cyB0byB0cmFuc2ZlciBmb3IgdGhpcyBvcGVyYXRpb24NCiAgICAgKG5vdCBjb3VudGlu ZyBkYXRhIGhlYWRlcnMpLg0KDQogICAgIEZvciBhIFJFQUQgb3BlcmF0aW9uLCB0aGUgaW5pdGlh dG9yIHVzZXMgdGhpcyBmaWVsZCB0byBzcGVjaWZ5IHRoZQ0KICAgICBudW1iZXIgb2YgYnl0ZXMg b2YgZGF0YSBpdCBleHBlY3RzIHRoZSB0YXJnZXQgdG8gdHJhbnNmZXIgdG8gdGhlDQogICAgIGlu aXRpYXRvciAobm90IGNvdW50aW5nIGRhdGEgaGVhZGVycykuDQoNCiAgICAgSWYgbm8gZGF0YSB3 aWxsIGJlIHRyYW5zZmVycmVkIGluIFNDU0kgRGF0YSBwYWNrZXRzIGZvciB0aGlzIFNDU0kNCiAg ICAgb3BlcmF0aW9uLCB0aGlzIGZpZWxkIHNob3VsZCBiZSBzZXQgdG8gMC4NCg0KICAgICBXcml0 ZSBkYXRhIHdpbGwgYmUgc2VudCBpbiBTQ1NJIERhdGEgcGFja2V0cyAtIG5vdCB0b2dldGhlciB3 aXRoDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBbUGFnZSAxMF0NCgwNCmlTQ1NJICAgICAgICAgICAgICAgICAgICAg ICAgIEp1bmUgMjAwMA0KDQoNCiAgICAgdGhlIGNvbW1hbmQuDQoNCiAgICAgQ29tbWFuZCBwYXJh bWV0ZXJzIGFyZSB0byBiZSBzZW50IHRvZ2V0aGVyIHdpdGggdGhlIFNDU0kgY29tbWFuZA0KICAg ICAoYXMgb3Bwb3NlZCB0byBpbiBhIHNlcGFyYXRlIFNDU0kgRGF0YSBwYWNrZXQpLg0KDQogICAg IFRoZSB0YXJnZXQga25vd3MgdG8gZXhwZWN0IGRhdGEgZm9yIHRoaXMgU0NTSSBjb21tYW5kIGlu IGEgc2VwYXJhdGUNCiAgICAgU0NTSSBEYXRhIHBhY2tldCBieSByZWNlaXZpbmcgYSBub24temVy byB2YWx1ZSBpbiB0aGUgRXhwZWN0ZWQgRGF0YQ0KICAgICBUcmFuc2ZlciBMZW5ndGggZmllbGQu DQoNCiAgICAgVXBvbiBjb21wbGV0aW9uIG9mIGEgZGF0YSB0cmFuc2ZlciwgdGhlIHRhcmdldCB3 aWxsIGluZm9ybSB0aGUgaW5pLQ0KICAgICB0aWF0b3Igb2YgaG93IG1hbnkgYnl0ZXMgd2VyZSBh Y3R1YWxseSBwcm9jZXNzZWQgKHNlbnQgb3IgcmVjZWl2ZWQpDQogICAgIGJ5IHRoZSB0YXJnZXQu DQoNCjMuMi41LiAgU0NTSSBDb21tYW5kIERlc2NyaXB0b3IgQmxvY2sgKENEQikNCg0KICAgICBU aGVyZSBhcmUgMTYgYnl0ZXMgaW4gdGhlIENEQiBmaWVsZCwgZGVzaWduZWQgdG8gYWNjb21tb2Rh dGUgdGhlDQogICAgIGxhcmdlc3QgY3VycmVudGx5IGRlZmluZWQgQ0RCLg0KDQogICAgIElmLCBp biB0aGUgZnV0dXJlLCBsYXJnZXIgQ0RCJ3MgYXJlIGFsbG93ZWQsIHRoZSBzcGlsbC1vdmVyIG9m IHRoZQ0KICAgICBDREIgbWF5IGV4dGVuZCBiZXlvbmQgdGhlIDQwLWJ5dGUgYm91bmRhcnksIGZv bGxvd2VkIGJ5IHRoZQ0KICAgICBwYXJhbWV0ZXItZGF0YS4NCg0KMy4yLjYuICBDb21tYW5kLURh dGENCg0KICAgICBTb21lIFNDU0kgY29tbWFuZHMgcmVxdWlyZSBhZGRpdGlvbmFsIHBhcmFtZXRl cnMgZGF0YSB0byBhY2NvbXBhbnkNCiAgICAgdGhlIFNDU0kgY29tbWFuZC4gVGhpcyBkYXRhIG1h eSBiZSBwbGFjZWQgYmV5b25kIHRoZSA0MC1ieXRlIGJvdW4tDQogICAgIGRhcnkgb2YgdGhlIGlT Q1NJIGhlYWRlci4gVGhlIExlbmd0aCBmaWVsZCBpcyBzZXQgdG8gdGhlIGxlbmd0aCBvZg0KICAg ICB0aGlzIGRhdGEgYmV5b25kIHRoZSA0MC1ieXRlIGhlYWRlciAoaS5lLiBpbmNsdWRlcyB0aGUg Q0RCIGV4dGVuLQ0KICAgICBzaW9uIGlmIHByZXNlbnQpLg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAxMV0NCgwNCmlTQ1NJICAgICAgICAg ICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuMy4gIFNDU0kgUmVzcG9uc2UNCg0KDQog ICAgIEJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAg ICAgICAzICAgICAgIHwNCiAgICAgICAgLyAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwg ICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfA0KICAgICAgIHw3IDYgNSA0IDMgMiAxIDB8 NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8DQogICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNCiAgICAgIDB8IE9wY29kZSAoMHg4MSkgfCAgQ21kUk4gICAgICAgIHwgUnNydmQg KDApIHxPfFV8IFJlc2VydmVkICgwKSAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICA0fCBMZW5n dGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgOHwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKyAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICsNCiAg ICAgMTJ8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDE2fCBJbml0aWF0b3IgVGFzayBU YWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKw0KICAgICAyMHwgUmVzaWR1YWwgQ291bnQgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMjR8IENvbW1h bmQgU3RhdHVzfGlTQ1NJIFN0YXR1cyAgIHwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAg fA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDI4LyBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAg ICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rDQogICAgIDQwLyBSZXNwb25zZSBvciBTZW5zZSBEYXRhIChvcHRpb25hbCkg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rDQoNCg0KMy4zLjEuICBGbGFncw0KDQogICAgIFRoZSBTQ1NJIFJlc3BvbnNlIGhhcyBp dHMgb3duIHNldCBvZiBmbGFncywgdGhhdCBkaWZmZXJzIGZyb20gdGhlDQogICAgIGZsYWdzIGZv ciBhIFNDU0kgQ29tbWFuZC4NCiAgICAgICAgICAgICAgIEJ5dGUgMg0KICAgICAgICAgIGIwICAg KFUpIHNldCBmb3IgUmVzaWR1YWwgVW5kZXJmbG93LiBJbiB0aGlzIGNhc2UsIHRoZSBSZXNpLQ0K ICAgICAgICAgICAgICAgZHVhbCBDb3VudCBpbmRpY2F0ZXMgaG93IG1hbnkgYnl0ZXMgd2VyZSBu b3QgdHJhbnNmZXJyZWQNCiAgICAgICAgICAgICAgIG91dCBvZiB0aG9zZSBleHBlY3RlZCB0byBi ZSB0cmFuc2ZlcnJlZC4NCiAgICAgICAgICBiMSAgIChPKSBzZXQgZm9yIFJlc2lkdWFsIE92ZXJm bG93LiBJbiB0aGlzIGNhc2UsIHRoZSBSZXNpZHVhbA0KICAgICAgICAgICAgICAgQ291bnQgaW5k aWNhdGVzIGhvdyBtYW55IGJ5dGVzIGNvdWxkIG5vdCBiZSB0cmFuc2ZlcnJlZA0KICAgICAgICAg ICAgICAgYmVjYXVzZSB0aGUgaW5pdGlhdG9yJ3MgRXhwZWN0ZWQgRGF0YSBUcmFuc2ZlciBMZW5n dGggd2FzDQogICAgICAgICAgICAgICB0b28gc21hbGwuDQogICAgICAgICAgYjItNyBub3QgdXNl ZCAoc2hvdWxkIGJlIHNldCB0byAwKS4NCiAgICAgQml0cyAwIGFuZCAxIGFyZSBtdXR1YWxseSBl eGNsdXNpdmUuDQoNCjMuMy4yLiAgUmVzaWR1YWwgQ291bnQNCg0KICAgICBUaGUgUmVzaWR1YWwg Q291bnQgZmllbGQgaXMgdmFsaWQgb25seSBpbiBjYXNlIGVpdGhlciB0aGUgUmVzaWR1YWwNCiAg ICAgVW5kZXJmbG93IGJpdCBvciBSZXNpZHVhbCBPdmVyZmxvdyBiaXQgaXMgc2V0LiBJZiBuZWl0 aGVyIGJpdCBpcw0KDQoNCg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMTJdDQoMDQppU0NTSSAgICAgICAgICAgICAg ICAgICAgICAgICBKdW5lIDIwMDANCg0KDQogICAgIHNldCwgdGhlIFJlc2lkdWFsIENvdW50IGZp ZWxkIHNob3VsZCBiZSAwLg0KDQogICAgIElmIHRoZSBSZXNpZHVhbCBVbmRlcmZsb3cgYml0IGlz IHNldCwgdGhlIFJlc2lkdWFsIENvdW50IGluZGljYXRlcw0KICAgICBob3cgbWFueSBieXRlcyB3 ZXJlIG5vdCB0cmFuc2ZlcnJlZCBvdXQgb2YgdGhvc2UgZXhwZWN0ZWQgdG8gYmUNCiAgICAgdHJh bnNmZXJyZWQuDQoNCiAgICAgSWYgdGhlIFJlc2lkdWFsIE92ZXJmbG93IGJpdCBpcyBzZXQsIHRo ZSBSZXNpZHVhbCBDb3VudCBpbmRpY2F0ZXMNCiAgICAgaG93IG1hbnkgYnl0ZXMgY291bGQgbm90 IGJlIHRyYW5zZmVycmVkIGJlY2F1c2UgdGhlIGluaXRpYXRvcidzDQogICAgIEV4cGVjdGVkIERh dGEgVHJhbnNmZXIgTGVuZ3RoIHdhcyB0b28gc21hbGwuDQoNCjMuMy4zLiAgQ29tbWFuZCBTdGF0 dXMNCg0KICAgICBUaGUgQ29tbWFuZCBTdGF0dXMgZmllbGQgaXMgdXNlZCB0byByZXBvcnQgdGhl IFNDU0kgc3RhdHVzIG9mIHRoZQ0KICAgICBjb21tYW5kIChhcyBzcGVjaWZpZWQgaW4gW1NBTTJd KS4NCg0KMy4zLjQuICBpU0NTSSBTdGF0dXMNCg0KICAgICBUaGUgaVNDU0kgU3RhdHVzIGZpZWxk IGlzIHVzZWQgdG8gcmVwb3J0IHRoZSBzdGF0dXMgb2YgdGhlIGNvbW1hbmQNCiAgICAgYmVmb3Jl IGl0IHdhcyBzZW50IGJ5IHRoZSB0YXJnZXQgdG8gdGhlIExVTi4gVGhlIHZhbHVlcyBhcmUgZ2l2 ZW4NCiAgICAgYmVsb3cuDQoNCiAgICAgICAgICAgICAwIEdvb2Qgc3RhdHVzDQogICAgICAgICAg ICAgMSBpU0NTSSBjaGVjaw0KDQogICAgIElmIHRoZSBpU0NTSSBmaWVsZCBpcyBub3QgMCB0aGUg Y29tbWFuZCBzdGF0dXMgd2lsbCBpbmRpY2F0ZSBDSEVDSw0KICAgICBDT05ESVRJT04gKHdvdWxk bid0IGl0IGJlIGdyZWF0IHRvIGdpdmUgdXAgdGhlIGlTQ1NJIGZpZWxkIGVudGlyZWx5DQogICAg IGFuZCBrZWVwIHJlbHlpbmcgb25seSBvbiBzZW5zZT8gZG9lcyBpdCBoYXZlIG5hc3R5IEFDQSBp bXBsaWNhLQ0KICAgICB0aW9ucz8pDQoNCjMuMy41LiAgUmVzcG9uc2Ugb3IgU2Vuc2UgRGF0YQ0K DQogICAgIElmIEF1dG9zZW5zZSB3YXMgbm90IGRpc2FibGVkIGluIHRoZSBvcmlnaW5hdGluZyBD REIgYW5kIHRoZSBDb20tDQogICAgIG1hbmQgU3RhdHVzIHdhcyBDSEVDSyBDT05ESVRJT04gKDB4 MDIpLCB0aGVuIHRoZSBSZXNwb25zZSBEYXRhDQogICAgIGZpZWxkIHdpbGwgY29udGFpbiBzZW5z ZSBkYXRhIGZvciB0aGUgZmFpbGVkIGNvbW1hbmQuDQoNCiAgICAgSWYgdGhlIENvbW1hbmQgU3Rh dHVzIGlzIEdvb2QgKDB4MDApIHRoZW4gdGhlIFJlc3BvbnNlIERhdGEgZmllbGQNCiAgICAgd2ls bCBjb250YWluIGRhdGEgZnJvbSB0aGUgZGF0YSBwaGFzZSBvZiB0aGUgQ0RCLg0KDQogICAgIFRo ZSBMZW5ndGggcGFyYW1ldGVyIHNwZWNpZmllcyB0aGUgbnVtYmVyIG9mIGJ5dGVzIGluIHRoaXMg ZmllbGQuDQogICAgIElmIG5vIGVycm9yIG9jY3VycmVkLCBhbmQgbm8gZGF0YSBpcyBuZWVkZWQg Zm9yIHRoZSByZXNwb25zZSB0byB0aGUNCiAgICAgU0NTSSBDb21tYW5kIHRoZSBMZW5ndGggZmll bGQgaXMgMC4NCg0KICAgICBOb3RlIHRoYXQgaWYgdGhlIENvbW1hbmQgU3RhdHVzIHdhcyBDSEVD SyBDT05ESVRJT04gYnV0IEF1dG9zZW5zZQ0KICAgICB3YXMgZGlzYWJsZWQsIHRoZW4gc2Vuc2Ug ZGF0YSBtdXN0IGJlIGV4cGxpY2l0bHkgcmVxdWVzdGVkIGJ5IHRoZQ0KICAgICBpbml0aWF0b3Ig d2l0aCBhIG5ldyBTQ1NJIGNvbW1hbmQuDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1 bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAxM10NCgwN CmlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuNC4gIEFzeW5j aHJvbm91cyBFdmVudA0KDQogICAgIEFuIEFzeW5jaHJvbm91cyBFdmVudCBtYXkgYmUgc2VudCBm cm9tIHRoZSB0YXJnZXQgdG8gdGhlIGluaXRpYXRvcg0KICAgICB3aXRob3V0IGNvcnJlc3BvbmRp bmcgdG8gYSBwYXJ0aWN1bGFyIGNvbW1hbmQuIFRoZSB0YXJnZXQgc3BlY2lmaWVzDQogICAgIHRo ZSBzdGF0dXMgZm9yIHRoZSBldmVudCBhbmQgc2Vuc2UgZGF0YS4NCg0KICAgICBCeXRlIC8gICAg MCAgICAgICB8ICAgICAgIDEgICAgICAgfCAgICAgICAyICAgICAgIHwgICAgICAgMyAgICAgICB8 DQogICAgICAgIC8gICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAg fCAgICAgICAgICAgICAgIHwNCiAgICAgICB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEg MHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfA0KICAgICAgICstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAg ICAwfCBPcGNvZGUgKDB4ODcpIHwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgNHwgTGVuZ3RoICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSsNCiAgICAgIDh8IExvZ2ljYWwgVW5pdCBOdW1iZXIgKExVTikgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICsgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICArDQogICAgIDEyfCAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwN CiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKw0KICAgICAxNnwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKyAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICsNCiAgICAg MjB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDI0fCBDb21tYW5kIFN0YXR1c3xpU0NT SSBTdGF0dXN8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Kw0KICAgICAyOHxFdmVudCBJbmRpY2F0b3J8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMzJ8IFJlc2VydmVkICgw KSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAg ICAgICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICArDQogICAgIDM2fCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICA0MC8g U2Vuc2UgRGF0YSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAvDQogICAgICArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KDQoNCjMuNC4xLiAgaVND U0kgU3RhdHVzDQoNCiAgICAgU29tZSBBc3luY2hyb25vdXMgRXZlbnRzIGFyZSBzdHJpY3RseSBy ZWxhdGVkIHRvIGlTQ1NJIHdoaWxlIG90aGVycw0KICAgICBhcmUgcmVsYXRlZCB0byBTQU0tMi4g IFRoZSBjb2RlcyByZXR1cm5lZCBmb3IgaVNDU0kgQXN5bmNocm9ub3VzDQogICAgIEV2ZW50cyBh cmU6DQoNCiAgICAgICAgICAgICAyIFRhcmdldCBpcyBiZWluZyByZXNldC4NCg0KDQozLjQuMi4g IEV2ZW50IEluZGljYXRvcg0KDQogICAgIFRoZSBmb2xsb3dpbmcgdmFsdWVzIGFyZSBkZWZpbmVk LiAgKFNlZSBbU0FNMl0gZm9yIGRldGFpbHMuKQ0KICAgICAgICAgIDEgICAgQW4gZXJyb3IgY29u ZGl0aW9uIHdhcyBlbmNvdW50ZXJlZCBhZnRlciBjb21tYW5kDQoNCg0KDQpTYXRyYW4sIFNtaXRo LCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAx NF0NCgwNCmlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCiAgICAg ICAgICAgICAgIGNvbXBsZXRpb24uDQogICAgICAgICAgMiAgICBBIG5ld2x5IGluaXRpYWxpemVk IGRldmljZSBpcyBhdmFpbGFibGUuDQogICAgICAgICAgMyAgICBTb21lIG90aGVyIHR5cGUgb2Yg dW5pdCBhdHRlbnRpb24gY29uZGl0aW9uIGhhcyBvY2N1cnJlZC4NCiAgICAgICAgICA0ICAgIEFu IGFzeW5jaHJvbm91cyBldmVudCBoYXMgb2NjdXJyZWQuDQogICAgIFNlbnNlIERhdGEgYWNjb21w YW55aW5nIHRoZSByZXBvcnQgaWRlbnRpZmllcyB0aGUgY29uZGl0aW9uLiAgVGhlDQogICAgIExl bmd0aCBwYXJhbWV0ZXIgaXMgc2V0IHRvIHRoZSBsZW5ndGggb2YgdGhlIFNlbnNlIERhdGEuDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0K DQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtp cywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAxNV0NCgwNCmlTQ1NJ ICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuNS4gIFNDU0kgVGFzayBN YW5hZ2VtZW50IE1lc3NhZ2UNCg0KDQogICAgIEJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAg ICAgICB8ICAgICAgIDIgICAgICAgfCAgICAgICAzICAgICAgIHwNCiAgICAgICAgLyAgICAgICAg ICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfA0K ICAgICAgIHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3 IDYgNSA0IDMgMiAxIDB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDB8IE9wY29kZSAoMHgwMikg fCAgUmVzZXJ2ZWQgKDApIHwgTXNnIGluZGljYXRvciB8IFJlc2VydmVkICgwKSAgfA0KICAgICAg ICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rDQogICAgICA0fCBMZW5ndGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgOHwgTG9n aWNhbCBVbml0IE51bWJlciAoTFVOKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB8DQogICAgICAgKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICsNCiAgICAgMTJ8ICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQog ICAgIDE2fCBJbml0aWF0b3IgVGFzayBUYWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAyMC8gUmVzZXJ2ZWQgKDApICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICAr LyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIC8NCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICA0MA0KDQoNCjMuNS4xLiAgTXNnIEluZGlj YXRvcg0KDQogICAgIFRoZSBUYXNrIE1hbmFnZW1lbnQgZnVuY3Rpb25zIHByb3ZpZGUgYW4gaW5p dGlhdG9yIHdpdGggYSB3YXkgdG8NCiAgICAgZXhwbGljaXRseSBjb250cm9sIHRoZSBleGVjdXRp b24gb2Ygb25lIG9yIG1vcmUgVGFza3MuIFRoZSBUYXNrDQogICAgIE1hbmFnZW1lbnQgZnVuY3Rp b25zIGFyZSBzdW1tYXJpemVkIGFzIGZvbGxvd3MgKGZvciBhIG1vcmUgZGV0YWlsZWQNCiAgICAg ZGVzY3JpcHRpb24gc2VlIHRoZSBbU0FNMl0gZG9jdW1lbnQpOg0KICAgICAgICAgIDEgICAgQWJv cnQgVGFzay0tLWFib3J0cyB0aGUgdGFzayBpZGVudGlmaWVkIGJ5IHRoZSBUYXNrIFRhZw0KICAg ICAgICAgICAgICAgZmllbGQuDQogICAgICAgICAgMiAgICBBYm9ydCBUYXNrIFNldC0tLWFib3J0 cyBhbGwgVGFza3MgaXNzdWVkIGJ5IHRoaXMgaW5pdGlhLQ0KICAgICAgICAgICAgICAgdG9yIG9u IHRoZSBMb2dpY2FsIFVuaXQuDQogICAgICAgICAgMyAgICBDbGVhciBBQ0EtLS1jbGVhcnMgdGhl IEF1dG8gQ29udGluZ2VudCBBbGxlZ2lhbmNlIGNvbmRpLQ0KICAgICAgICAgICAgICAgdGlvbi4N CiAgICAgICAgICA0ICAgIENsZWFyIFRhc2sgU2V0LS0tQWJvcnRzIGFsbCBUYXNrcyAoZnJvbSBh bGwgaW5pdGlhdG9ycykNCiAgICAgICAgICAgICAgIGZvciB0aGUgTG9naWNhbCBVbml0Lg0KICAg ICAgICAgIDUgICAgTG9naWNhbCBVbml0IFJlc2V0Lg0KICAgICAgICAgIDYgICAgVGFyZ2V0IFJl c2V0Lg0KICAgICBGb3IgdGhlIGZ1bmN0aW9ucyBhYm92ZSBleGNlcHQgPFRhcmdldCBSZXNldD4s IGEgU0NTSSBUYXNrIE1hbmFnZS0NCiAgICAgbWVudCBSZXNwb25zZSBpcyByZXR1cm5lZCwgdXNp bmcgdGhlIEluaXRpYXRvciBUYXNrIFRhZyB0byBpZGVudGlmeQ0KICAgICB0aGUgb3BlcmF0aW9u IGZvciB3aGljaCBpdCBpcyByZXNwb25kaW5nLg0KDQogICAgIEZvciB0aGUgPFRhcmdldCBSZXNl dD4gZnVuY3Rpb24sIHRoZSB0YXJnZXQgY2FuY2VscyBhbGwgcGVuZGluZw0KICAgICBvcGVyYXRp b25zLiBUaGUgdGFyZ2V0IG1heSBzZW5kIGFuIEFzeW5jaHJvbm91cyBFdmVudCB0byBhbGwNCiAg ICAgYXR0YWNoZWQgaW5pdGlhdG9ycyBub3RpZnlpbmcgdGhlbSB0aGF0IHRoZSB0YXJnZXQgaXMg YmVpbmcgcmVzZXQuDQogICAgIFRoZSB0YXJnZXQgdGhlbiBjbG9zZXMgYWxsIG9mIGl0cyBUQ1Ag Y29ubmVjdGlvbnMgdG8gYWxsIGluaXRpYXRvcnMNCiAgICAgKGFsbCBzZXNzaW9ucyBhcmUgdGVy bWluYXRlZCkuDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAxNl0NCgwNCmlTQ1NJICAgICAgICAgICAgICAg ICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuNi4gIFNDU0kgVGFzayBNYW5hZ2VtZW50IFJlc3Bv bnNlDQoNCg0KICAgICBCeXRlIC8gICAgMCAgICAgICB8ICAgICAgIDEgICAgICAgfCAgICAgICAy ICAgICAgIHwgICAgICAgMyAgICAgICB8DQogICAgICAgIC8gICAgICAgICAgICAgIHwgICAgICAg ICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwNCiAgICAgICB8NyA2IDUg NCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAw fA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rDQogICAgICAwfCBPcGNvZGUgKDB4ODIpIHwgUmVzZXJ2ZWQgKDAp ICB8IE1zZyBpbmRpY2F0b3IgfCBSZXNlcnZlZCAoMCkgIHwNCiAgICAgICArLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAg ICAgNHwgTGVuZ3RoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDh8IExvZ2ljYWwgVW5pdCBOdW1i ZXIgKExVTikgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICsg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICArDQogICAgIDEyfCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAxNnwgSW5pdGlh dG9yIFRhc2sgVGFnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMjB8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAg IDI0fCBSZXNwb25zZSAgICAgIHwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAyOC8gUmVzZXJ2ZWQgKDApICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICArLyAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIC8NCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICA0MA0KDQogICAgIEZvciB0aGUgZnVuY3Rpb25z IDxBYm9ydCBUYXNrLCBBYm9ydCBUYXNrIFNldCwgQ2xlYXIgQUNBLCBDbGVhcg0KICAgICBUYXNr IFNldCwgTG9naWNhbCBVbml0IHJlc2V0PiwgdGhlIHRhcmdldCBwZXJmb3JtcyB0aGUgcmVxdWVz dGVkDQogICAgIFRhc2sgTWFuYWdlbWVudCBmdW5jdGlvbiBhbmQgc2VuZHMgYSBTQ1NJIFRhc2sg TWFuYWdlbWVudCBSZXNwb25zZQ0KICAgICBiYWNrIHRvIHRoZSBpbml0aWF0b3IuICBUaGUgdGFy Z2V0IGluY2x1ZGVzIGFsbCBvZiB0aGUgaW5mb3JtYXRpb24NCiAgICAgdGhlIGluaXRpYXRvciBw cm92aWRlZCBpbiB0aGUgU0NTSSBUYXNrIE1hbmFnZW1lbnQgTWVzc2FnZSwgc28gdGhlDQogICAg IGluaXRpYXRvciBjYW4ga25vdyBleGFjdGx5IHdoaWNoIFNDU0kgVGFzayBNYW5hZ2VtZW50IE1l c3NhZ2Ugd2FzDQogICAgIHNlcnZpY2VkLiAgSW4gYWRkaXRpb24sIHRoZSB0YXJnZXQgcHJvdmlk ZXMgYSBSZXNwb25zZSBpbmRpY2F0aW9uDQogICAgIHdoaWNoIG1heSB0YWtlIG9uIHRoZSBmb2xs b3dpbmcgdmFsdWVzOg0KICAgICAgICAgIDAgICAgRnVuY3Rpb24gQ29tcGxldGUNCiAgICAgICAg ICAxICAgIEZ1bmN0aW9uIFJlamVjdGVkDQogICAgIEZvciB0aGUgPFRhcmdldCBSZXNldD4gZnVu Y3Rpb24sIHRoZSB0YXJnZXQgY2FuY2VscyBhbGwgcGVuZGluZw0KICAgICBvcGVyYXRpb25zLiBU aGUgdGFyZ2V0IG1heSBzZW5kIGFuIEFzeW5jaHJvbm91cyBFdmVudCB0byBhbGwNCiAgICAgYXR0 YWNoZWQgaW5pdGlhdG9ycyBub3RpZnlpbmcgdGhlbSB0aGF0IHRoZSB0YXJnZXQgaXMgYmVpbmcg cmVzZXQuDQogICAgIFRoZSB0YXJnZXQgdGhlbiBjbG9zZXMgYWxsIG9mIGl0cyBUQ1AgY29ubmVj dGlvbnMgdG8gYWxsIGluaXRpYXRvcnMNCiAgICAgKHRlcm1pbmF0ZXMgYWxsIHNlc3Npb25zKS4N Cg0KDQoNCg0KDQoNCg0KDQoNCg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMTddDQoMDQppU0NTSSAgICAgICAgICAg ICAgICAgICAgICAgICBKdW5lIDIwMDANCg0KDQozLjcuICBSZWFkeSBUbyBUcmFuc2ZlciAoUlRU KQ0KDQogICAgIFdoZW4gYW4gaW5pdGlhdG9yIGhhcyBzdWJtaXR0ZWQgYSBTQ1NJIENvbW1hbmQg d2l0aCBkYXRhIHBhc3NpbmcNCiAgICAgZnJvbSB0aGUgaW5pdGlhdG9yIHRvIHRoZSB0YXJnZXQg KFdSSVRFKSwgdGhlIHRhcmdldCBtYXkgc3BlY2lmeQ0KICAgICB3aGljaCBibG9ja3Mgb2YgZGF0 YSBpdCBpcyByZWFkeSB0byByZWNlaXZlLiBJbiBnZW5lcmFsLCB0aGUgdGFyZ2V0DQogICAgIG1h eSByZXF1ZXN0IHRoYXQgdGhlIGRhdGEgYmxvY2tzIGJlIGRlbGl2ZXJlZCBpbiB3aGF0ZXZlciBv cmRlciBpcw0KICAgICBjb252ZW5pZW50IGZvciB0aGUgdGFyZ2V0IGF0IHRoYXQgcGFydGljdWxh ciBpbnN0YW50LiBUaGlzIGluZm9ybWEtDQogICAgIHRpb24gaXMgcGFzc2VkIGZyb20gdGhlIHRh cmdldCB0byB0aGUgaW5pdGlhdG9yIGluIHRoZSBSZWFkeSBUbw0KICAgICBUcmFuc2ZlciAoUlRU KSBtZXNzYWdlLg0KDQogICAgIEluIG9yZGVyIHRvIGFsbG93IHdyaXRlIG9wZXJhdGlvbnMgd2l0 aG91dCBSVFQsIHRoZSBpbml0aWF0b3IgYW5kDQogICAgIHRhcmdldCBtdXN0IGhhdmUgYWdyZWVk IHRvIGRvIHNvIGJ5IGJvdGggc2VuZGluZyB0aGUgQWxsb3dOb1JUVDp5ZXMNCiAgICAga2V5LXBh aXIgYXR0cmlidXRlIHRvIGVhY2ggb3RoZXIgKGVpdGhlciBkdXJpbmcgTG9naW4gb3IgdGhyb3Vn aA0KICAgICB0aGUgVGV4dCBDb21tYW5kL1Jlc3BvbnNlIG1lY2hhbmlzbSkuDQoNCiAgICAgQnl0 ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwgICAgICAgMiAgICAgICB8ICAgICAgIDMg ICAgICAgfA0KICAgICAgICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAg ICAgICAgIHwgICAgICAgICAgICAgICB8DQogICAgICAgfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0 IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwNCiAgICAgICArLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Kw0KICAgICAgMHwgT3Bjb2RlICgweDg2KSB8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDR8IExlbmd0aCAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAg ICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rDQogICAgICA4fCBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKw0KICAgICAxMnwg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMTZ8IEluaXRpYXRvciBUYXNrIFRhZyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQogICAgIDIwfCBEZXNpcmVkIERhdGEgVHJhbnNmZXIgTGVuZ3RoICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAyNHwgRGF0YSBPZmZzZXQg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAg ICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSsNCiAgICAgMjh8IFRhcmdldCBUcmFuc2ZlciBUYWcgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDMyfCBS ZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHwNCiAgICAgICArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgKw0KICAgICAzNnwgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsN CiAgICAgNDANCg0KDQozLjcuMS4gIERlc2lyZWQgRGF0YSBUcmFuc2ZlciBMZW5ndGggYW5kIERh dGEgT2Zmc2V0DQoNCiAgICAgVGhlIHRhcmdldCBzcGVjaWZpZXMgaG93IG1hbnkgYnl0ZXMgaXQg d2FudHMgdGhlIGluaXRpYXRvciB0byBzZW5kDQogICAgIGFzIGEgcmVzdWx0IG9mIHRoaXMgUlRU IG1lc3NhZ2UuICBUaGUgdGFyZ2V0IG1heSByZXF1ZXN0IHRoZSBkYXRhDQogICAgIGZyb20gdGhl IGluaXRpYXRvciBpbiBzZXZlcmFsIGNodW5rcywgbm90IG5lY2Vzc2FyaWx5IGluIHRoZSBvcmln aS0NCiAgICAgbmFsIG9yZGVyIG9mIHRoZSBkYXRhLiAgVGhlIHRhcmdldCwgdGhlcmVmb3JlLCBh bHNvIHNwZWNpZmllcyBhDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAxOF0NCgwNCmlTQ1NJICAgICAgICAg ICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCiAgICAgRGF0YSBPZmZzZXQgaW5kaWNhdGlu ZyB0aGUgcG9pbnQgYXQgd2hpY2ggdGhlIGRhdGEgdHJhbnNmZXIgc2hvdWxkDQogICAgIGJlZ2lu LCByZWxhdGl2ZSB0byB0aGUgYmVnaW5uaW5nIG9mIHRoZSB0b3RhbCBkYXRhIHRyYW5zZmVyLg0K DQozLjcuMi4gIFRhcmdldCBUcmFuc2ZlciBUYWcNCg0KICAgICBUaGUgdGFyZ2V0IGFzc2lnbnMg aXRzIG93biB0YWcgdG8gZWFjaCBSVFQgcmVxdWVzdCB0aGF0IGl0IHNlbmRzIHRvDQogICAgIHRo ZSBpbml0aWF0b3IuIFRoaXMgY2FuIGJlIHVzZWQgYnkgdGhlIHRhcmdldCB0byBlYXNpbHkgaWRl bnRpZnkNCiAgICAgZGF0YSBpdCByZWNlaXZlcywgYW5kIGNhbiBhbHNvIGJlIHVzZWQgYXMgYW4g UkRNQSB0YWcgW1JETUFdLg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNt aXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFn ZSAxOV0NCgwNCmlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMu OC4gIFNDU0kgRGF0YQ0KDQogICAgIFRoZSB0eXBpY2FsIGRhdGEgdHJhbnNmZXIgc3BlY2lmaWVz IHRoZSBsZW5ndGggb2YgdGhlIGRhdGEgcGF5bG9hZCwNCiAgICAgdGhlIFRyYW5zZmVyIFRhZyBw cm92aWRlZCBieSB0aGUgcmVjZWl2ZXIgZm9yIHRoaXMgZGF0YSB0cmFuc2ZlciwNCiAgICAgYW5k IGEgYnVmZmVyIG9mZnNldC4NCg0KICAgICBUaGUgdHlwaWNhbCBTQ1NJIERhdGEgcGFja2V0IGZv ciBXUklURSAoZnJvbSBpbml0aWF0b3IgdG8gdGFyZ2V0KQ0KICAgICBoYXMgdGhlIGZvbGxvd2lu ZyBmb3JtYXQ6DQoNCiAgICAgQnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwgICAg ICAgMiAgICAgICB8ICAgICAgIDMgICAgICAgfA0KICAgICAgICAvICAgICAgICAgICAgICB8ICAg ICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8DQogICAgICAgfDcg NiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAy IDEgMHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgMHwgT3Bjb2RlICgweDA1KSB8IFJlc2VydmVk ICgwKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsN CiAgICAgIDR8IExlbmd0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICA4fCBUYXJnZXQgVGFzayBU YWcgb3IgMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAg ICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKw0KICAgICAxMnwgRGF0YSBPZmZzZXQgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMTZ8IElu aXRpYXRvciBUYXNrIFRhZyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDIwLyBSZXNlcnZlZCAoMCkgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0K ICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rDQogICAgIDQwLyBQYXlsb2FkICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAg ICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rDQoNCiAgICAgVGhlIHR5cGljYWwgU0NTSSBEYXRhIHBhY2tldCBmb3IgUkVBRCAo ZnJvbSB0YXJnZXQgdG8gaW5pdGlhdG9yKQ0KICAgICBoYXMgdGhlIGZvbGxvd2luZyBmb3JtYXQ6 DQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNClNhdHJhbiwgU21pdGgsIFNh cHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDIwXQ0K DA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoNCg0KDQogICAgIEJ5 dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAgICAgICAz ICAgICAgIHwNCiAgICAgICAgLyAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAg ICAgICAgICB8ICAgICAgICAgICAgICAgfA0KICAgICAgIHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUg NCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8DQogICAgICAgKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNCiAgICAgIDB8IE9wY29kZSAoMHg4NSkgfCBSZXNlcnZlZCAoMCkgIHwgKDApICAgICB8U3xP fFV8IFJlc2VydmVkICgwKSAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICA0fCBMZW5ndGggICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKw0KICAgICAgOHwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMTJ8 IERhdGEgT2Zmc2V0ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDE2fCBJbml0aWF0b3IgVGFzayBUYWcgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Kw0KICAgICAyMHwgUmVzaWR1YWwgQ291bnQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMjR8IENvbW1hbmQgU3Rh dHVzfGlTQ1NJIFN0YXR1c3wgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgfA0KICAg ICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rDQogICAgIDI4LyBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rDQogICAgIDQwLyBQYXlsb2FkICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQoNCg0KMy44LjEuICBMZW5ndGgNCg0KICAgICBUaGUgbGVuZ3RoIGZpZWxkIHNwZWNpZmllcyB0 aGUgdG90YWwgbnVtYmVyIG9mIGJ5dGVzIGluIHRoZSBmb2xsb3ctDQogICAgIGluZyBwYXlsb2Fk Lg0KDQozLjguMi4gIFRyYW5zZmVyIFRhZw0KDQogICAgIFRoZSBUcmFuc2ZlciBUYWcgaWRlbnRp ZmllcyB0aGUgb3BlcmF0aW9uIHRvIHdoaWNoIHRoaXMgZGF0YQ0KICAgICB0cmFuc2ZlciBiZWxv bmdzLg0KDQogICAgIFdoZW4gdGhlIHRyYW5zZmVyIGlzIGZyb20gdGhlIHRhcmdldCB0byB0aGUg aW5pdGlhdG9yLCB0aGUgVHJhbnNmZXINCiAgICAgVGFnIGlzIHRoZSBJbml0aWF0b3IgVGFzayBU YWcgdGhhdCB3YXMgc2VudCB3aXRoIHRoZSBTQ1NJIGNvbW1hbmQuDQoNCiAgICAgV2hlbiB0aGUg dHJhbnNmZXIgaXMgZnJvbSB0aGUgaW5pdGlhdG9yIHRvIHRoZSB0YXJnZXQsIHRoZSBUcmFuc2Zl cg0KICAgICBUYWcgaXMgdGhlIFRhcmdldCBUcmFuc2ZlciBUYWcgd2hlbiBSVFQgaXMgZW5hYmxl ZCwgb3IgdGhlIEluaXRpYS0NCiAgICAgdG9yIFRhc2sgVGFnIHdoZW4gUlRUIGlzIGRpc2FibGVk Lg0KDQozLjguMy4gIEJ1ZmZlciBPZmZzZXQNCg0KICAgICBUaGUgQnVmZmVyIE9mZnNldCBmaWVs ZCBjb250YWlucyB0aGUgb2Zmc2V0IG9mIHRoZSBmb2xsb3dpbmcgZGF0YQ0KICAgICBhZ2FpbnN0 IHRoZSBjb21wbGV0ZSBkYXRhIHRyYW5zZmVyLiBUaGUgc3VtIG9mIHRoZSBidWZmZXIgb2Zmc2V0 DQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBbUGFnZSAyMV0NCgwNCmlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAg IEp1bmUgMjAwMA0KDQoNCiAgICAgYW5kIGxlbmd0aCBzaG91bGQgbm90IGV4Y2VlZCB0aGUgZXhw ZWN0ZWQgdHJhbnNmZXIgbGVuZ3RoIGZvciB0aGUNCiAgICAgY29tbWFuZC4NCg0KMy44LjQuICBG bGFncw0KDQogICAgIFRoZSBsYXN0IFNDU0kgRGF0YSBwYWNrZXQgc2VudCBmcm9tIGEgdGFyZ2V0 IHRvIGFuIGluaXRpYXRvciBmb3IgYQ0KICAgICBwYXJ0aWN1bGFyIFNDU0kgY29tbWFuZCB0aGF0 IGNvbXBsZXRlZCBzdWNjZXNzZnVsbHkgbWF5IG9wdGlvbmFsbHkNCiAgICAgYWxzbyBjb250YWlu IHRoZSBDb21tYW5kIFN0YXR1cyBmb3IgdGhlIGRhdGEgdHJhbnNmZXIuICBJbiB0aGlzDQogICAg IGNhc2UgU2Vuc2UgRGF0YSBjYW5ub3QgYmUgc2VudCB0b2dldGhlciB3aXRoIHRoZSBDb21tYW5k IFN0YXR1cy4NCg0KICAgICBJZiB0aGUgY29tbWFuZCBjb21wbGV0ZWQgd2l0aCBhbiBlcnJvciwg dGhlbiB0aGUgcmVzcG9uc2UgYW5kIHNlbnNlDQogICAgIGRhdGEgbXVzdCBiZSBzZW50IGluIGEg U0NTSSBSZXNwb25zZSBwYWNrZXQgYW5kIG11c3Qgbm90IGJlIHNlbnQgaW4NCiAgICAgYSBTQ1NJ IERhdGEgcGFja2V0Lg0KICAgICAgICAgICAgICAgQnl0ZSAyDQogICAgICAgICAgYjAtMSBhcyBp biBhbiBvcmRpbmFyeSBTQ1NJIFJlc3BvbnNlDQogICAgICAgICAgYjIgICAoUykgc2V0IHRvIGlu ZGljYXRlIHRoYXQgdGhlIENvbW1hbmQgU3RhdHVzIGZpZWxkIGNvbi0NCiAgICAgICAgICAgICAg IHRhaW5zIHN0YXR1cy4NCiAgICAgICAgICBiMy03IG5vdCB1c2VkIChzaG91bGQgYmUgc2V0IHRv IDApLg0KICAgICBJZiB0aGUgKFMpIGJpdCBpcyBzZXQsIHRoZW4gdGhlcmUgaXMgbWVhbmluZyB0 byB0aGUgZXh0cmEgZmllbGRzIGluDQogICAgIHRoZSBTQ1NJIERhdGEgcGFja2V0IChJbml0aWF0 b3IgVGFzayBUYWcsIENvbW1hbmQgU3RhdHVzLCBSZXNpZHVhbA0KICAgICBDb3VudCwgZXRjKQ0K DQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgW1BhZ2UgMjJdDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5l IDIwMDANCg0KDQozLjkuICBUZXh0IENvbW1hbmQNCg0KICAgICBUaGUgVGV4dCBDb21tYW5kIGlz IHByb3ZpZGVkIHRvIGFsbG93IHRoZSBleGNoYW5nZSBvZiBpbmZvcm1hdGlvbg0KICAgICBhbmQg Zm9yIGZ1dHVyZSBleHRlbnNpb25zLiBJdCBwZXJtaXRzIHRoZSBpbml0aWF0b3IgdG8gaW5mb3Jt IGENCiAgICAgdGFyZ2V0IG9mIGl0cyBjYXBhYmlsaXRpZXMgb3IgdG8gcmVxdWVzdCBzb21lIHNw ZWNpYWwgb3BlcmF0aW9ucy4NCg0KICAgICBCeXRlIC8gICAgMCAgICAgICB8ICAgICAgIDEgICAg ICAgfCAgICAgICAyICAgICAgIHwgICAgICAgMyAgICAgICB8DQogICAgICAgIC8gICAgICAgICAg ICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwNCiAg ICAgICB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2 IDUgNCAzIDIgMSAwfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICAwfCBPcGNvZGUgKDB4MDQpIHwg UmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKw0KICAgICAgNHwgTGVuZ3RoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDgvIFJlc2Vy dmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg Lw0KICAgICAgKy8gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAvDQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMTZ8IEluaXRpYXRvciBU YXNrIFRhZyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAg ICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rDQogICAgIDIwLyBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rDQogICAgIDQwLyBUZXh0ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQoNCg0KMy45LjEuICBMZW5ndGgNCg0KICAgICBUaGUgbGVuZ3RoLCBpbiBieXRlcywgb2YgdGhl IFRleHQgZmllbGQuDQoNCjMuOS4yLiAgSW5pdGlhdG9yIFRhc2sgVGFnDQoNCiAgICAgVGhlIGlu aXRpYXRvciBhc3NpZ25lZCBpZGVudGlmaWVyIGZvciB0aGlzIFRleHQgQ29tbWFuZC4NCg0KMy45 LjMuICBUZXh0DQoNCiAgICAgVGhlIGluaXRpYXRvciBzZW5kcyB0aGUgdGFyZ2V0IGEgc2V0IG9m IGtleTp2YWx1ZSBwYWlycyBpbiBVVEYtOA0KICAgICB1bmljb2RlIGZvcm1hdC4gVGhlIGtleSBh bmQgdmFsdWUgYXJlIHNlcGFyYXRlZCBieSBhICc6JyAoMHgzQSkNCiAgICAgZGVsaW1pdGVyLiBN YW55IGtleTp2YWx1ZSBwYWlycyBjYW4gYmUgaW5jbHVkZWQgaW4gdGhlIFRleHQgYmxvY2sNCiAg ICAgYnkgc2VwYXJhdGluZyB0aGVtIHdpdGggbnVsICcgJyAoMHgwMCkgZGVsaW1pdGVycy4gU29t ZSBiYXNpYw0KICAgICBrZXk6dmFsdWUgcGFpcnMgYXJlIGRlc2NyaWJlZCBpbiBBcHBlbmRpeCBC Lg0KDQogICAgIFRoZSB0YXJnZXQgcmVzcG9uZHMgYnkgc2VuZGluZyBpdHMgcmVzcG9uc2UgYmFj ayB0byB0aGUgaW5pdGlhdG9yLg0KICAgICBUaGUgdGFyZ2V0IGFuZCBpbml0aWF0b3IgY2FuIHRo ZW4gcGVyZm9ybSBzb21lIGFkdmFuY2VkIG9wZXJhdGlvbnMNCiAgICAgYmFzZWQgb24gdGhlaXIg Y29tbW9uIGNhcGFiYWxpdGllcy4NCg0KICAgICBNYW51ZmFjdHVyZXJzIG1heSBpbnRyb2R1Y2Ug bmV3IGtleXMgYnkgcHJlZml4aW5nIHRoZW0gd2l0aCB0aGVpcg0KDQoNCg0KU2F0cmFuLCBTbWl0 aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2Ug MjNdDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDANCg0KDQogICAg IChyZXZlcnNlZCkgZG9tYWluIG5hbWUsIGZvciBleGFtcGxlLA0KDQogICAgICAgICAgICAgY29t LmZvby5iYXIuZG9fc29tZXRoaW5nOjAwMDAwMDAwMDAwMDAwMDMNCg0KICAgICBBbnkga2V5IHRo YXQgdGhlIHRhcmdldCBkb2VzIG5vdCB1bmRlcnN0YW5kIG1heSBiZSBpZ25vcmVkIHdpdGhvdXQN CiAgICAgYWZmZWN0aW5nIGJhc2ljIGZ1bmN0aW9uLiBPbmNlIHRoZSB0YXJnZXQgaGFzIHByb2Nl c3NlZCBhbGwgdGhlDQogICAgIGtleTp2YWx1ZSBwYWlycywgaXQgcmVzcG9uZHMgd2l0aCB0aGUg VGV4dCBSZXNwb25zZSBjb21tYW5kLCBsaXN0LQ0KICAgICBpbmcgdGhlIHBhcmFtZXRlcnMgdGhh dCBpdCBzdXBwb3J0cy4gSXQgaXMgcmVjb21tZW5kZWQgdGhhdCBUZXh0DQogICAgIG9wZXJhdGlv bnMgdGhhdCB3aWxsIHRha2UgYSBsb25nIHRpbWUgc2hvdWxkIGJlIHBsYWNlZCBpbiB0aGVpciBv d24NCiAgICAgVGV4dCBjb21tYW5kLg0KDQogICAgIElmIHRoZSBUZXh0IFJlc3BvbnNlIGRvZXMg bm90IGNvbnRhaW4gYSBrZXkgdGhhdCB3YXMgcmVxdWVzdGVkLCB0aGUNCiAgICAgaW5pdGlhdG9y IG11c3QgYXNzdW1lIHRoYXQgdGhlIGtleSB3YXMgbm90IHVuZGVyc3Rvb2QgYnkgdGhlIHRhci0N CiAgICAgZ2V0Lg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywg TWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAyNF0NCgwNCmlTQ1NJICAg ICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuMTAuICBUZXh0IFJlc3BvbnNl DQoNCiAgICAgVGhlIFRleHQgUmVzcG9uc2UgbWVzc2FnZSBjb250YWlucyB0aGUgcmVzcG9uc2Vz IG9mIHRoZSB0YXJnZXQgdG8NCiAgICAgdGhlIGluaXRpYXRvcidzIFRleHQgQ29tbWFuZC4gVGhl IGZvcm1hdCBvZiB0aGUgVGV4dCBmaWVsZCBtYXRjaGVzDQogICAgIHRoYXQgb2YgdGhlIFRleHQg Q29tbWFuZC4NCg0KICAgICBCeXRlIC8gICAgMCAgICAgICB8ICAgICAgIDEgICAgICAgfCAgICAg ICAyICAgICAgIHwgICAgICAgMyAgICAgICB8DQogICAgICAgIC8gICAgICAgICAgICAgIHwgICAg ICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwNCiAgICAgICB8NyA2 IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIg MSAwfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICAwfCBPcGNvZGUgKDB4ODQpIHwgUmVzZXJ2ZWQg KDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0K ICAgICAgNHwgTGVuZ3RoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDgvIFJlc2VydmVkICgwKSAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAg Ky8gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAvDQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMTZ8IEluaXRpYXRvciBUYXNrIFRhZyAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rDQogICAgIDIwLyBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQog ICAgIDQwLyBUZXh0IFJlc3BvbnNlICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQoNCg0KMy4x MC4xLiAgTGVuZ3RoDQoNCiAgICAgVGhlIGxlbmd0aCwgaW4gYnl0ZXMsIG9mIHRoZSBUZXh0IFJl c3BvbnNlIGZpZWxkLg0KDQozLjEwLjIuICBJbml0aWF0b3IgVGFzayBUYWcNCg0KICAgICBUaGUg SW5pdGlhdG9yIFRhc2sgVGFnIG1hdGNoZXMgdGhlIHRhZyB1c2VkIGluIHRoZSBpbml0aWFsIFRl eHQNCiAgICAgQ29tbWFuZCBhbmQgaXMgdXNlZCBieSB0aGUgaW5pdGlhdG9yIHRvIHJlbGF0ZSB0 aGUgVGV4dCBDb21tYW5kcw0KICAgICB3aXRoIHRoZSBhcHByb3ByaWF0ZSBUZXh0IFJlc3BvbnNl cy4NCg0KMy4xMC4zLiAgVGV4dCBSZXNwb25zZQ0KDQogICAgIFRoZSBUZXh0IFJlc3BvbnNlIGZp ZWxkIGNvbnRhaW5zIHJlc3BvbnNlcyBpbiB0aGUgc2FtZSBrZXk6dmFsdWUNCiAgICAgZm9ybWF0 IGFzIHRoZSBUZXh0IENvbW1hbmQuIEFwcGVuZGl4IEIgbGlzdHMgc29tZSBiYXNpYyBUZXh0IENv bS0NCiAgICAgbWFuZHMgYW5kIHRoZWlyIFJlc3BvbnNlcy4NCg0KICAgICBJZiB0aGUgVGV4dCBS ZXNwb25zZSBkb2VzIG5vdCBjb250YWluIGEga2V5IHRoYXQgd2FzIHJlcXVlc3RlZCwgdGhlDQog ICAgIGluaXRpYXRvciBtdXN0IGFzc3VtZSB0aGF0IHRoZSBrZXkgd2FzIG5vdCB1bmRlcnN0b29k IGJ5IHRoZSB0YXItDQogICAgIGdldC4NCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6 YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAyNV0NCgwNCmlT Q1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuMTEuICBMb2dpbiBD b21tYW5kDQoNCiAgICAgQWZ0ZXIgZXN0YWJsaXNoaW5nIGEgVENQIGNvbm5lY3Rpb24gYmV0d2Vl biBhbiBpbml0aWF0b3IgYW5kIGEgdGFyLQ0KICAgICBnZXQsIHRoZSBpbml0aWF0b3Igc2hvdWxk IGlzc3VlIGEgTG9naW4gQ29tbWFuZCB0byBnYWluIGZ1cnRoZXINCiAgICAgYWNjZXNzIHRvIHRo ZSB0YXJnZXQncyByZXNvdXJjZXMuDQoNCiAgICAgQnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAx ICAgICAgIHwgICAgICAgMiAgICAgICB8ICAgICAgIDMgICAgICAgfA0KICAgICAgICAvICAgICAg ICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8 DQogICAgICAgfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAw fDcgNiA1IDQgMyAyIDEgMHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgMHwgT3Bjb2RlICgweDAz KSB8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAg ICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSsNCiAgICAgIDR8IExlbmd0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICA4LyBS ZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDE2fCBJU0lEICAg ICAgICAgICAgICAgICAgICAgICAgICB8VFNJRCAgICAgICAgICAgICAgICAgICAgICAgICAgIHwN CiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKw0KICAgICAyMC8gUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICArLyAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAg ICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKw0KICAgICA0MC8gTG9naW4gUGFyYW1ldGVycyBpbiBUZXh0IENvbW1hbmQgRm9y bWF0ICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICArLyAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICArLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKw0KDQogICAgIFRoZSBpbml0aWF0b3IgbWF5IHByb3ZpZGUgc29tZSBiYXNpYyBwYXJhbWV0 ZXJzIGluIG9yZGVyIHRvIGVuYWJsZQ0KICAgICB0aGUgdGFyZ2V0IHRvIGRldGVybWluZSBpZiB0 aGUgaW5pdGlhdG9yIG1heSBpbiBmYWN0IHVzZSB0aGUNCiAgICAgdGFyZ2V0J3MgcmVzb3VyY2Vz Lg0KDQogICAgIFRoZSBmb3JtYXQgb2YgdGhlIHBhcmFtZXRlcnMgaXMgYXMgc3BlY2lmaWVkIGZv ciB0aGUgVGV4dCBDb21tYW5kLg0KDQogICAgIFRhcmdldHMgbWF5IHJlcXVpcmUga2V5cyB0byBp bmRpY2F0ZSB0aGUgRG9tYWluIE5hbWUgb2YgdGhlIGluaXRpYS0NCiAgICAgdG9yIGFuZCB0aGUg dGFyZ2V0LCBhbmQgcGVyaGFwcyBhbHNvIGFuIEF1dGhlbnRpY2F0b3Iga2V5Lg0KDQogICAgIFRo ZSBpbml0aWF0b3IgbWF5IGFsc28gcHJvdmlkZSBhZGRpdGlvbmFsIHBhcmFtZXRlcnMgdG8gdGhl IHRhcmdldA0KICAgICBpbiBUZXh0IENvbW1hbmQgZm9ybWF0LCBpZiB0aGUgaW5pdGlhdG9yIHNv IGRlc2lyZXMuDQoNCiAgICAgS2V5cyBhbmQgdGhlaXIgZXhwbGFuYXRpb25zIGFyZSBsaXN0ZWQg aW4gQXBwZW5kaXggQi4NCg0KICAgICBXaGVuZXZlciBkZXNpcmVkIGFuIGluaXRpYXRvciB3aWxs IGlkZW50aWZ5IGl0cyB2aWV3IG9mIHRoZSB0YXJnZXQNCiAgICAgYXMgaW46DQoNCiAgICAgICAg ICAgICBUYXJnZXQ6PGRvbWFpbi1uYW1lPlsvbW9kaWZpZXJdDQoNCiAgICAgaW1wbHlpbmcgdGhh dCB0aGUgdGFyZ2V0IGlzIGtub3duIGFzOg0KDQogICAgICAgICAgICAgc2NzaTovLzxkb21haW4t bmFtZT5bL21vZGlmaWVyXQ0KDQoNCg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGgg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMjZdDQoMDQppU0NTSSAgICAgICAg ICAgICAgICAgICAgICAgICBKdW5lIDIwMDANCg0KDQogICAgIEluaXRpYXRvcnMgY2FuIHVzZSB0 aGUgc2FtZSB0eXBlIG9mIG5hbWluZyBpbXBseWluZyBtYWNoaW5lIG5hbWluZw0KICAgICByYXRo ZXIgdGhhbiBwcmluY2lwYWwuDQoNCiAgICAgVGh1cyB0aGUgcGFyYW1ldGVycyBwYXNzZWQgZm9y IGEgcGxhaW4tdGV4dCBwYXNzd29yZCBhdXRoZW50aWNhdGlvbg0KICAgICBhcmU6DQoNCiAgICAg ICAgICAgICBJbml0aWF0b3I6PGRvbWFpbi1uYW1lPlsvbW9kaWZpZXJdDQogICAgICAgICAgICAg VGFyZ2V0Og0KICAgICAgICAgICAgIEF1dGhlbnRpY2F0b3I6b3Blbi1zZXNhbWUNCg0KDQogICAg IElTSUQgYW5kIFRTSUQgZm9ybSBjb2xsZWN0aXZlbHkgdGhlIFNTSUQgKHNlc3Npb24gaWQpLiBB IFRTSUQgb2YgMA0KICAgICBpbmRpY2F0ZXMgYSBsZWFkaW5nIGNvbm5lY3Rpb24uIE9ubHkgYSBs ZWFkaW5nIGNvbm5lY3Rpb24gbG9naW4gY2FuDQogICAgIGNhcnJ5IHNlc3Npb24gc3BlY2lmaWMg cGFyYW1ldGVycywgZS5nLiBtYXgtY29ubmVjdGlvbnMtcmVxdWVzdGVkOngNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0K DQoNCg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgW1BhZ2UgMjddDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBK dW5lIDIwMDANCg0KDQozLjEyLiAgTG9naW4gUmVzcG9uc2UNCg0KICAgICBUaGUgdGFyZ2V0IHJl c3BvbmRzIHRvIHRoZSBMb2dpbiBDb21tYW5kIHdpdGggYSBMb2dpbiBSZXNwb25zZS4gIEl0DQog ICAgIGlzIHN1ZmZpY2llbnQgZm9yIHRoZSB0YXJnZXQgdG8gcmVzcG9uZCB3aXRoIGEgU3RhdHVz IGluZGljYXRpbmcNCiAgICAgdGhhdCB0aGUgTG9naW4gaXMgYWNjZXB0ZWQuDQoNCiAgICAgSW4g ZmFjdCwgdGhlIHRhcmdldCBtYXkgY29tcGxldGVseSBpZ25vcmUgdGhlIHBhcmFtZXRlcnMgdGhh dCB3ZXJlDQogICAgIHNlbnQgdG8gaXQgYW5kIG1heSBwcm92aWRlIHNlcnZpY2UgdG8gYW55IGlu aXRpYXRvciB0aGF0IGNvbm5lY3RzDQogICAgIHRvIGl0LiBUaGUgdGFyZ2V0IG1heSBhbHNvIHJl dHVybiBwYXJhbWV0ZXJzIHVzaW5nIHRoZSBmb3JtYXQgb2YNCiAgICAgdGhlIFRleHQgUmVzcG9u c2Ugb3Bjb2RlLCBpZiBpdCBzbyBkZXNpcmVzLg0KDQogICAgIEluIHBhcnRpY3VsYXIsIHRoZSB0 YXJnZXQgbWF5IHdhbnQgdG8gcHJvdmlkZSBpdHMgQXV0aGVudGljYXRvcg0KICAgICBrZXksIHNv IHRoYXQgdGhlIGluaXRpYXRvciBjYW4gYmUgc3VyZSB0aGF0IGl0IGlzIGluIGZhY3QgdGFsa2lu Zw0KICAgICB3aXRoIHRoZSBjb3JyZWN0IHRhcmdldC4NCg0KICAgICBUaGUgaW5pdGlhdG9yIGNh biByZXF1ZXN0IHRoYXQgdGhlIHRhcmdldCBwcm92aWRlIHRoZSBBdXRoZW50aWNhdG9yDQogICAg IHBhcmFtZXRlciBieSBzcGVjaWZ5aW5nIHRoZSBTZW5kQXV0aGVudGljYXRvcjp5ZXMga2V5OnZh bHVlIHBhaXIuDQoNCiAgICAgQnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwgICAg ICAgMiAgICAgICB8ICAgICAgIDMgICAgICAgfA0KICAgICAgICAvICAgICAgICAgICAgICB8ICAg ICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8DQogICAgICAgfDcg NiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAy IDEgMHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgMHwgT3Bjb2RlICgweDgzKSB8IFJlc2VydmVk ICgwKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsN CiAgICAgIDR8IExlbmd0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICA4LyBSZXNlcnZlZCAoMCkg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAg ICsvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgLw0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDE2fCBJU0lEICAgICAgICAgICAgICAg ICAgICAgICAgICB8IFRTSUQgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKw0KICAgICAyMHwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgMjR8IFN0YXR1cyAg ICAgICAgfCBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0K ICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rDQogICAgIDI4LyBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAg ICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rDQogICAgIDQwLyBMb2dpbiBQYXJhbWV0ZXJzIGluIFRleHQgQ29tbWFuZCBGb3Jt YXQgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgICstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rDQoNCiAgICAgVGhlIGZvcm1hdCBvZiB0aGUgTG9naW4gUmVzcG9uc2UgaXMgdGhlIHNhbWUg YXMgdGhlIFRleHQgUmVzcG9uc2UsDQogICAgIHdpdGggdGhlIGFkZGl0aW9uIG9mIG9uZSBmaWVs ZC4NCg0KMy4xMi4xLiAgU3RhdHVzDQoNCiAgICAgVGhlIFN0YXR1cyByZXR1cm5lZCBpbiBhIExv Z2luIFJlc3BvbnNlIGlzIG9uZSBvZiB0aGUgZm9sbG93aW5nOg0KDQoNCg0KU2F0cmFuLCBTbWl0 aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2Ug MjhdDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDANCg0KDQogICAg ICAgICAgMCAgICBhY2NlcHQgbG9naW4gICAod2lsbCBub3cgYWNjZXB0IFNDU0kgY29tbWFuZHMp DQogICAgICAgICAgMSAgICByZWplY3QgbG9naW4NCiAgICAgICAgICAyICAgIGFkZGl0aW9uYWwg YXV0aGVudGljYXRpb24gcmVxdWlyZWQNCiAgICAgSW4gdGhlIGNhc2UgdGhhdCB0aGUgU3RhdHVz IGlzICJhY2NlcHQgbG9naW4iIHRoZSBpbml0aWF0b3IgbWF5DQogICAgIHByb2NlZWQgdG8gaXNz dWUgU0NTSSBjb21tYW5kcy4gIEluIHRoZSBjYXNlIHRoYXQgdGhlIFN0YXR1cyBpcw0KICAgICAi cmVqZWN0IGxvZ2luIiB0aGUgaW5pdGlhdG9yIHNob3VsZCBpbW1lZGlhdGVseSBjbG9zZSBkb3du IGl0cyBlbmQNCiAgICAgb2YgdGhlIFRDUCBjb25uZWN0aW9uLCB0aHVzIGZyZWVpbmcgdXAgdGhl IHRhcmdldCdzIHBvcnQgZm9yIHNvbWUNCiAgICAgb3RoZXIgY29ubmVjdGlvbi4gVGhlIHRhcmdl dCBhbHNvIGhhcyB0aGUgb3B0aW9uIG9mIGltbWVkaWF0ZWx5DQogICAgIGNsb3NpbmcgZG93biBp dHMgZW5kIG9mIHRoZSBUQ1AgY29ubmVjdGlvbi4NCg0KICAgICBJbiB0aGUgY2FzZSB0aGF0IHRo ZSBTdGF0dXMgaXMgImFkZGl0aW9uYWwgYXV0aGVudGljYXRpb24gcmVxdWlyZWQiDQogICAgIHRo ZSBpbml0aWF0b3IgbXVzdCBwcm92aWRlIGFkZGl0aW9uYWwgYXV0aGVudGljYXRpb24gaW5mb3Jt YXRpb24gYnkNCiAgICAgaXNzdWluZyB0aGUgVGV4dCBDb21tYW5kIHdpdGggdGhlIGFwcHJvcHJp YXRlIGtleTp2YWx1ZSBwYWlycy4NCiAgICAgKFRoaXMgbWF5IGJlIHJlcXVpcmVkIGlmIHRoZSBh dXRoZW50aWNhdGlvbiBtZXRob2QgaXMgYmFzZWQgb24gYQ0KICAgICBjaGFsbGVuZ2UvcmVzcG9u c2UgYWxnb3JpdGhtLikgIFVwb24gcmVjZWlwdCBvZiB0aGUgbmVjZXNzYXJ5DQogICAgIGF1dGhl bnRpY2F0aW9uLCB0aGUgdGFyZ2V0IHdpbGwgaXNzdWUgYSBMb2dpbiBSZXNwb25zZSB3aXRoIHRo ZQ0KICAgICAiYWNjZXB0IGxvZ2luIiBTdGF0dXMuICBTQ1NJIENvbW1hbmRzIHdpbGwgbm90IGJl IGFjY2VwdGVkIHVudGlsDQogICAgIHRoZSB0YXJnZXQgcHJvdmlkZXMgYSBMb2dpbiBSZXNwb25z ZSB3aXRoIHRoZSAiYWNjZXB0IGxvZ2luIg0KICAgICBTdGF0dXMuDQoNCiAgICAgVGhlIFRTSUQg aXMgYW4gaW5pdGlhdG9yIGlkZW50aWZ5aW5nIHRhZyBzZXQgYnkgdGhlIHRhcmdldC4gIEEgMCBp bg0KICAgICB0aGUgcmV0dXJuZWQgVFNJRCBpbmRpY2F0ZXMgdGhhdCBlaXRoZXIgdGhlIHRhcmdl dCBzdXBwb3J0cyBvbmx5IGENCiAgICAgc2luZ2xlIGNvbm5lY3Rpb24gb3IgdGhhdCB0aGUgSVNJ RCBoYXMgYWxyZWFkeSBiZWVuIHVzZWQgYXMgYSBsZWFkLQ0KICAgICBpbmcgSVNJRC4gSW4gYm90 aCBjYXNlcyB0aGUgdGFyZ2V0IGlzIHJlamVjdGluZyB0aGUgbG9naW4uDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBT YXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAyOV0N CgwNCmlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuMTMuICBQ aW5nIENvbW1hbmQNCg0KDQogICAgIEJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8 ICAgICAgIDIgICAgICAgfCAgICAgICAzICAgICAgIHwNCiAgICAgICAgLyAgICAgICAgICAgICAg fCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfA0KICAgICAg IHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0 IDMgMiAxIDB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDB8IE9wY29kZSAoMHgwKSAgfCBSZXNl cnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rDQogICAgICA0fCBMZW5ndGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgOC8gUmVzZXJ2ZWQg KDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQog ICAgICArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIC8NCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAxNnwgSW5pdGlhdG9yIFRhc2sg VGFnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNCiAgICAgMjAvIFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgLw0KICAgICAgKy8gICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICAgKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNCiAgICAgNDAvIFBpbmcgRGF0YSAob3B0aW9uYWwpICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgLw0KICAgICAgKy8gICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICAgKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCg0K ICAgICBUaGUgUGluZyBDb21tYW5kIGNhbiBiZSB1c2VkIHRvIHZlcmlmeSB0aGF0IGEgY29ubmVj dGlvbiBpcyBzdGlsbA0KICAgICBhY3RpdmUuIFRoaXMgY29tbWFuZCBtYXkgYmUgdXNlZnVsIGlu IHRoZSBjYXNlIHdoZXJlIGFuIGluaXRpYXRvcg0KICAgICBoYXMgYmVlbiB3YWl0aW5nIGEgbG9u ZyB0aW1lIGZvciB0aGUgcmVzcG9uc2UgdG8gc29tZSBjb21tYW5kLCBhbmQNCiAgICAgdGhlIGlu aXRpYXRvciBzdXNwZWN0cyB0aGF0IHRoZXJlIGlzIHNvbWUgcHJvYmxlbSB3aXRoIHRoZSBjb25u ZWMtDQogICAgIHRpb24uDQoNCiAgICAgV2hlbiBhIHRhcmdldCByZWNlaXZlcyB0aGUgUGluZyBD b21tYW5kLCBpdCBzaG91bGQgcmVzcG9uZCB3aXRoIGENCiAgICAgUGluZyBSZXNwb25zZSwgZHVw bGljYXRpbmcgYXMgbXVjaCBvZiB0aGUgZGF0YSBhcyBwb3NzaWJsZSB0aGF0IHdhcw0KICAgICBw cm92aWRlZCBpbiB0aGUgUGluZyBDb21tYW5kIChpZiBzdWNoIGRhdGEgd2FzIHByZXNlbnQpLg0K DQogICAgIElmIHRoZSBpbml0aWF0b3IgZG9lcyBub3QgcmVjZWl2ZSB0aGUgUGluZyBSZXNwb25z ZSB3aXRoaW4gc29tZQ0KICAgICBwZXJpb2Qgb2YgdGltZSAoZGV0ZXJtaW5lZCBieSB0aGUgaW5p dGlhdG9yKSwgb3IgaWYgdGhlIGRhdGENCiAgICAgcmV0dXJuZWQgYnkgdGhlIFBpbmcgUmVzcG9u c2UgaXMgZGlmZmVyZW50IGZyb20gdGhlIGRhdGEgdGhhdCB3YXMNCiAgICAgaW4gdGhlIFBpbmcg Q29tbWFuZCwgdGhlIGluaXRpYXRvciBtYXkgY29uY2x1ZGUgdGhhdCB0aGVyZSBpcyBhDQogICAg IHByb2JsZW0gd2l0aCB0aGUgY29ubmVjdGlvbi4gVGhlIGluaXRpYXRvciB3aWxsIHRoZW4gY2xv c2UgdGhlIGNvbi0NCiAgICAgbmVjdGlvbiBhbmQgbWF5IHRyeSB0byBlc3RhYmxpc2ggYSBuZXcg Y29ubmVjdGlvbi4NCg0KMy4xMy4xLiAgTGVuZ3RoDQoNCiAgICAgVGhlIGxlbmd0aCBvZiB0aGUg b3B0aW9uYWwgUGluZyBEYXRhLg0KDQozLjEzLjIuICBJbml0aWF0b3IgVGFzayBUYWcNCg0KICAg ICBBbiBpbml0aWF0b3IgYXNzaWduZWQgaWRlbnRpZmllciBmb3IgdGhlIG9wZXJhdGlvbi4NCg0K DQoNCg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgW1BhZ2UgMzBdDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBK dW5lIDIwMDANCg0KDQozLjEzLjMuICBQaW5nIERhdGENCg0KICAgICBCaW5hcnkgZGF0YSB0aGF0 IHdpbGwgYmUgcmVmbGVjdGVkIGluIHRoZSBQaW5nIFJlc3BvbnNlLg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0K DQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGgg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMzFdDQoMDQppU0NTSSAgICAgICAg ICAgICAgICAgICAgICAgICBKdW5lIDIwMDANCg0KDQozLjE0LiAgUGluZyBSZXNwb25zZQ0KDQoN CiAgICAgQnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwgICAgICAgMiAgICAgICB8 ICAgICAgIDMgICAgICAgfA0KICAgICAgICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAg fCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8DQogICAgICAgfDcgNiA1IDQgMyAyIDEg MHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwNCiAgICAg ICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKw0KICAgICAgMHwgT3Bjb2RlICgweDgwKSB8IFJlc2VydmVkICgwKSAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDR8IExl bmd0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgICA4LyBSZXNlcnZlZCAoMCkgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICsvICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0K ICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rDQogICAgIDE2fCBJbml0aWF0b3IgVGFzayBUYWcgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAy MC8gUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAvDQogICAgICArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICA0MC8gUmV0 dXJuIFBpbmcgRGF0YSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAvDQogICAgICArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIC8NCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KDQogICAgIFdoZW4gYSB0YXJn ZXQgcmVjZWl2ZXMgdGhlIFBpbmcgQ29tbWFuZCwgaXQgc2hvdWxkIHJlc3BvbmQgd2l0aCBhDQog ICAgIFBpbmcgUmVzcG9uc2UsIGR1cGxpY2F0aW5nIHRoZSBkYXRhIGFuZCBJbml0aWF0b3IgVGFz ayBUYWcgdGhhdCB3YXMNCiAgICAgcHJvdmlkZWQgaW4gdGhlIFBpbmcgQ29tbWFuZCwgaWYgcHJl c2VudC4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0K U2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgW1BhZ2UgMzJdDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIw MDANCg0KDQozLjE1LiAgVGhpcmQgUGFydHkgQ29tbWFuZHMNCg0KICAgICBUaGVyZSBhcmUgc29t ZSB0aGlyZC1wYXJ0eSBTQ1NJIGNvbW1hbmRzLCBzdWNoIGFzIENPUFkgYW5kIEVYVEVOREVEDQog ICAgIENPUFksIHRoYXQgcmVxdWlyZSBvbmUgdGFyZ2V0IChUYXJnZXQgQSkgdG8gYWN0IGFzIGFu IGluaXRpYXRvciB0bw0KICAgICBvdGhlciB0YXJnZXRzIChlLmcuLCBUYXJnZXQgQikuICBTb21l IHN1Y2ggY29tbWFuZHMgY2FuIGJlIGV4dGVuZGVkDQogICAgIGluIGEgc3RyYWlnaHRmb3J3YXJk IHdheSB0byBhY2NvbW1vZGF0ZSBuZXcgZm9ybXMgb2YgYWRkcmVzc2luZywNCiAgICAgYW5kIHRo aXMgc2hvdWxkIGJlIGRvbmUgdG8gYWRkcmVzcyB0YXJnZXRzIHVzaW5nIGlTQ1NJLiBUaGVzZQ0K ICAgICBleHRlbnNpb25zIGFyZSBub3Qgc3RyYWlnaHRmb3J3YXJkIGZvciBhbGwgY29tbWFuZHMs IGFuZCB0aGV5IG1heQ0KICAgICBub3QgYmUgYWJsZSB0byBlbmNvbXBhc3MgdGhlIGZ1bGwgbmFt ZSBzcGFjZSBhbmQgYXV0aGVudGljYXRpb24NCiAgICAgaW5mb3JtYXRpb24gbmVlZGVkIGZvciBp U0NTSSBpbiBzb21lIGNvbnRleHRzLiBUaHVzIGlTQ1NJIGFsc28gcHJvLQ0KICAgICB2aWRlcyBh IGZhY2lsaXR5IGZvciBhc3NpZ25pbmcgbG9jYWwgc2hvcnQtZm9ybSBhbGlhc2VzIHRvIGZ1bGwN CiAgICAgYWRkcmVzc2luZy9hdXRob3JpemF0aW9uIGluZm9ybWF0aW9uIGZvciB0YXJnZXRzLCBh bmQgdGhlIGFsaWFzZXMNCiAgICAgY2FuIGJlIHVzZWQgaW4gdGhlIFNDU0kgY29tbWFuZHMgYW5k IHBhcmFtZXRlciBkYXRhLiAgVGhlIGFsaWFzDQogICAgIGluZm9ybWF0aW9uIGlzIHNwZWNpZmll ZCBhcyBUZXh0IGZvbGxvd2luZyB0aGUgaGVhZGVyIG9mIHRoZSBTQ1NJDQogICAgIGNvbW1hbmQg c3BlY2lmeWluZyB0aGUgdGhpcmQtcGFydHkgY29tbWFuZC4gIFRoZSBoZWFkZXIgd2lsbCB0aHVz DQogICAgIGFwcGVhciBhcyBmb2xsb3dzOg0KDQogICAgIEJ5dGUgLyAgICAwICAgICAgIHwgICAg ICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAgICAgICAzICAgICAgIHwNCiAgICAgICAgLyAg ICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAg ICAgfA0KICAgICAgIHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAy IDEgMHw3IDYgNSA0IDMgMiAxIDB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDB8IE9wY29kZSAo MHgwMSkgfCAgIENtZFJOICAgICAgIHxBfCBXTENELTQgIHwwIDB8VGFzayBBdHRyaWJ1dGVzfA0K ICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rDQogICAgICA0fCBMZW5ndGggKCE9IDApICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAg OHwgTG9naWNhbCBVbml0IE51bWJlciAoTFVOKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICB8DQogICAgICAgKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICsNCiAgICAgMTJ8ICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rDQogICAgIDE2fCBJbml0aWF0b3IgVGFzayBUYWcgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAyMHwgRXhwZWN0ZWQg RGF0YSBUcmFuc2ZlciBMZW5ndGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQog ICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSsNCiAgICAgMjR8IFNDU0kgQ29tbWFuZCBEZXNjcmlwdG9yIEJsb2NrIChD REIpICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICsgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICArDQogICAgIDI4 fCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIHwNCiAgICAgICArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgKw0KICAgICAzMnwgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQogICAgICAgKyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICsNCiAgICAgMzZ8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQogICAgIDQwLyBFeHRlbmRlZCBD REIgaWYgYW55ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NCiAg ICAgICsvIFBhcmFtZXRlcnMgbmVlZGVkIGZvciBUYXJnZXQgQiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgLw0KICAgICAgIC8gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCg0KICAgICBU aGUgTGVuZ3RoIGZpZWxkIHdpbGwgbm90IGJlIHplcm8uIFJhdGhlciwgaXQgd2lsbCBjb250YWlu IHRoZQ0KICAgICBsZW5ndGggb2YgZXh0ZW5kIENEQiBpZiBhbnkgYW4gdGhlIGxlbmd0aCBvZiB0 aGUgYWxpYXMgaW5mb3JtYXRpb24NCg0KDQoNClNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBN ZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDMzXQ0KDA0KaVNDU0kgICAg ICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoNCg0KICAgICB3aGljaCBtYXkgaW5jbHVk ZSB0aGUgbmFtZSBvZiBUYXJnZXQgQiBhbmQgYW4gQXV0aGVudGljYXRpb24ga2V5IGluDQogICAg IFRleHQgQ29tbWFuZCBmb3JtYXQuIEFuIGV4YW1wbGUgb2YgdGhlIGRhdGEgZm9yIHRoaXMgY29t bWFuZCBtaWdodA0KICAgICBiZToNCg0KICAgICAgICAgICAgIExvY2FsTmFtZTpUYXJnZXRCDQog ICAgICAgICAgICAgRnVsbE5hbWU6ZGlzazIuc2ouZm9vLmNvbQ0KICAgICAgICAgICAgIE9yaWdp bmFsQXV0aGVudGljYXRvcjpvcGVuLXNlc2FtZQ0KDQoNCiAgICAgVXBvbiByZWNlaXZpbmcgYSB0 aGlyZC1wYXJ0eSBjb21tYW5kLCBUYXJnZXQgQSB3aWxsIHBlcmZvcm0gbG9naW4NCiAgICAgb3Bl cmF0aW9ucyB3aXRoIHRoZSBpZGVudGlmaWVkIHRhcmdldHMuICBJbiBlZmZlY3QsIFRhcmdldCBB IHdpbGwNCiAgICAgYmVjb21lIGFuIGluaXRpYXRvciB0byBUYXJnZXQgQi4gIEFtb25nIHRoZSBw YXJhbWV0ZXJzIHByb3ZpZGVkIHRvDQogICAgIFRhcmdldCBCLCBUYXJnZXQgQSBtYXkgc3BlY2lm eSB0aGUgYXV0aGVudGljYXRpb24gaW5mb3JtYXRpb24gZnJvbQ0KICAgICB0aGUgaW5pdGlhdG9y LiAgVGhlIFRleHQgcHJvdmlkZWQgYnkgVGFyZ2V0IEEgd2hlbiBpdCBwZXJmb3JtcyB0aGUNCiAg ICAgTG9naW4gY29tbWFuZCB0byBUYXJnZXQgQiBtYXkgY29udGFpbiB0aGUga2V5cyBUYXJnZXQg KHJlZmVycmluZyB0bw0KICAgICBUYXJnZXQgQikgYW5kIEluaXRpYXRvciAocmVmZXJyaW5nIHRv IFRhcmdldCBBKSwgYW5kIGl0IG1heSBhbHNvDQogICAgIGNvbnRhaW4gdGhlIGtleXMgQXV0aGVu dGljYXRvciAob2YgVGFyZ2V0IEEpLCBPcmlnaW5hbEluaXRpYXRvciBhbmQNCiAgICAgT3JpZ2lu YWxBdXRoZW50aWNhdG9yIChyZWZlcnJpbmcgdG8gYXV0aGVudGljYXRvciBvZiB0aGUgb3JpZ2lu YWwNCiAgICAgaW5pdGlhdG9yKS4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywg TWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAzNF0NCgwNCmlTQ1NJICAg ICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjMuMTYuICBPcGNvZGUgTm90IFVu ZGVyc3Rvb2QNCg0KDQogICAgIEJ5dGUgLyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAg ICAgIDIgICAgICAgfCAgICAgICAzICAgICAgIHwNCiAgICAgICAgLyAgICAgICAgICAgICAgfCAg ICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfA0KICAgICAgIHw3 IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMg MiAxIDB8DQogICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNCiAgICAgIDB8IE9wY29kZSAoMHg4OCkgfCBSZXNlcnZl ZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KICAgICAgICstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQogICAgICA0fCBMZW5ndGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwNCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICAgOC8gUmVzZXJ2ZWQgKDAp ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAg ICArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIC8NCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICA0MC8gSGVhZGVyIG9mIEJhZCBNZXNz YWdlICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQogICAgICArLyAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIC8NCiAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0KICAgICA4MA0KDQogICAgIEl0IG1heSBoYXBwZW4gdGhh dCBhIHRhcmdldCByZWNlaXZlcyBhIG1lc3NhZ2Ugd2l0aCBhbiBPcGNvZGUgdGhhdA0KICAgICBp dCBkb2Vzbid0IHJlY29nbml6ZS4gVGhpcyBtYXkgb2NjdXIgYmVjYXVzZSBvZiBhIG5ldyB2ZXJz aW9uIG9mDQogICAgIHRoZSBwcm90b2NvbCB0aGF0IGRlZmluZXMgYSBuZXcgT3Bjb2RlLCBvciBi ZWNhdXNlIG9mIHNvbWUgY29ycnVwLQ0KICAgICB0aW9uIG9mIGEgbWVzc2FnZSBoZWFkZXIuDQoN CiAgICAgVGhlIHRhcmdldCByZXR1cm5zIHRoZSBoZWFkZXIgb2YgdGhlIG1lc3NhZ2Ugd2l0aCB0 aGUgdW5yZWNvZ25pemVkDQogICAgIG9wY29kZSBhcyB0aGUgZGF0YSBvZiB0aGUgcmVzcG9uc2Uu DQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNClNhdHJh biwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IFtQYWdlIDM1XQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoN Cg0KNC4gIEVycm9yIEhhbmRsaW5nIGlTQ1NJIC0gdGhpcyBzaG91bGQgYmUgY2xlYW5lZCBzb29u IQ0KDQogICAgIFRoZSBmb2xsb3dpbmcgZXJyb3JzIG1pZ2h0IGJlIGRldGVjdGVkIGJ5IHRoZSBp bml0aWF0b3IgYW5kIHRhci0NCiAgICAgZ2V0czoNCiAgICAgICAgLSBUQ1AgY29ubmVjdGlvbiB0 ZXJtaW5hdGlvbiB3aXRoIG91dHN0YW5kaW5nIG9wZXJhdGlvbnMgcGVuZGluZw0KICAgICAgICAt IE9wZXJhdGlvbiB0aW1lZCBvdXQNCiAgICAgICAgLSBJbGxlZ2FsIGZpZWxkIGZvcm1hdC0tLWNv dWxkIGJlIGFuIGltcGxlbWVudGF0aW9uIGVycm9yIG9yIGENCiAgICAgICAgICAgIGRhdGEgc3Ry ZWFtIHN5bmNocm9uaXphdGlvbiBlcnJvci4NCiAgICAgSW5pdGlhdG9ycyB0aGF0IGRldGVjdCBv bmUgb2YgdGhlIGFib3ZlIGVycm9ycyB3aWxsIHRha2UgdGhlIGZvbC0NCiAgICAgbG93aW5nIGFj dGlvbnM6DQogICAgICAgIC0gUmVzZXQgdGhlIFRDUCBjb25uZWN0aW9uLg0KICAgICAgICAtIFRl cm1pbmF0ZSBhbGwgb3V0c3RhbmRpbmcgcmVxdWVzdHMgb24gYmVoYWxmIG9mIHRoZSBMVU5zIG9u DQogICAgICAgICAgdGhlIHRhcmdldCB3aXRoIHRoZSBlcnJvci4NCiAgICAgICAgLSBSZWVzdGFi bGlzaCB0aGUgc2Vzc2lvbiBieToNCiAgICAgICAgICAgICAtIE9wZW5pbmcgYSBuZXcgVENQIGNv bm5lY3Rpb24uDQogICAgICAgICAgICAgIC0gUGVyZm9ybWluZyB0aGUgbG9naW4gcHJvY2Vzcy4N Cg0KICAgICBBIHRhcmdldCB0aGF0IGRldGVjdHMgb25lIG9mIHRoZSBhYm92ZSBlcnJvcnMgd2ls bCB0YWtlIHRoZSBmb2xsb3ctDQogICAgIGluZyBhY3Rpb25zOg0KICAgICAgICAtIFJlc2V0IHRo ZSBUQ1AgY29ubmVjdGlvbi4NCiAgICAgICAgLSBBYm9ydCBhbGwgVGFza3MgaW4gdGhlIHRhc2sg c2V0IGZvciB0aGUgY29ycmVzcG9uZGluZyBpbml0aWEtDQogICAgICAgICAgdG9yLg0KDQoNCg0K DQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNClNhdHJh biwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IFtQYWdlIDM2XQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoN Cg0KNS4gIE5vdGVzIHRvIEltcGxlbWVudGVycw0KDQogICAgIFRoaXMgcHJvdG9jb2wgd2FzIGRl c2lnbmVkIHRvIHRha2UgYWR2YW50YWdlIG9mIHRoZSBSZW1vdGUgRE1BIFRDUA0KICAgICBvcHRp b25zIFtSRE1BXSwgYWx0aG91Z2ggaXQgY2FuIHN0aWxsIG9wZXJhdGUgZWZmZWN0aXZlbHkgd2l0 aG91dA0KICAgICB0aGlzIFRDUCBleHRlbnNpb24uIFRoaXMgc2VjdGlvbiBub3RlcyBzb21lIG9m IHRoZSBwZXJmb3JtYW5jZSBjb24tDQogICAgIHNpZGVyYXRpb25zIG9mIHRoZSBpU0NTSSBwcm90 b2NvbC4NCg0KNS4xLiAgU21hbGwgVENQIFNlZ21lbnRzDQoNCiAgICAgSXQgaXMgcmVjb21tZW5k ZWQgdGhhdCBUQ1Agc2VnbWVudHMgYmUgbGltaXRlZCBpbiBzaXplIHRvIG5vIG1vcmUNCiAgICAg dGhhbiA4SyBieXRlcy4gT25lIHJlYXNvbiBpcyB0byBlbnN1cmUgdGhhdCBzZWdtZW50cyB3b24n dCBnZXQgYnJvLQ0KICAgICBrZW4gaW50byBzbWFsbGVyIHBhY2tldHMsIHRoZXJlYnkgcG9zc2li bHkgYnJlYWtpbmcgdGhlIGFzc3VtcHRpb25zDQogICAgIGZvciBSRE1BIGFuZCB0aGUgaW5mb3Jt YXRpb24gaW4gdGhlIFJETUEgaGVhZGVyLiBBbm90aGVyIHJlYXNvbiB3ZQ0KICAgICByZWNvbW1l bmQgc21hbGwgc2VnbWVudHMgaXMgdG8gYWxsb3cgYSBzdHJvbmdlciB0eXBlIG9mIGNoZWNrc3Vt LA0KICAgICBwb3NzaWJseSB1dGlsaXppbmcgQ1JDLCB3aGljaCBpcyBwcmFjdGljYWwgb25seSBm b3Igc21hbGxlciBzZWctDQogICAgIG1lbnRzLg0KDQo1LjIuICBNdWx0aXBsZSBOZXR3b3JrIEFk YXB0ZXJzDQoNCiAgICAgVGhlIGlTQ1NJIHByb3RvY29sIGFzc3VtZXMgdGhhdCB0aGUgVGFzayBU YWdzIHdpbGwgYWxzbyBzZXJ2ZSBhcw0KICAgICBSRE1BIHRhZ3MuIFRoZSBpU0NTSSBwcm90b2Nv bCBhbGxvd3MgbXVsdGlwbGUgY29ubmVjdGlvbnMsIG5vdCBhbGwNCiAgICAgb2Ygd2hpY2ggbmVl ZCBnbyBvdmVyIHRoZSBzYW1lIG5ldHdvcmsgYWRhcHRlci4gSWYgbXVsdGlwbGUgbmV0d29yaw0K ICAgICBjb25uZWN0aW9ucyBhcmUgdG8gYmUgdXRpbGl6ZWQgd2l0aCBSRE1BLCB0aGUgaVNDU0kg cHJvdG9jb2wNCiAgICAgY29tbWFuZC1kYXRhLXN0YXR1cyBhbGxlZ2lhbmNlIHRvIG9uZSBUQ1Ag Y29ubmVjdGlvbiBpbnN1cmUgdGhhdA0KICAgICB0aGVyZSBpcyBubyBuZWVkIHRvIHJlcGxpY2F0 ZSBpbmZvcm1hdGlvbiBhY3Jvc3MgbmV0d29yayBhZGFwdGVycw0KICAgICBvciBvdGhlcndpc2Ug cmVxdWlyZSB0aGVtIHRvIGNvb3BlcmF0ZS4NCg0KNS4zLiAgQXV0b3NlbnNlDQoNCiAgICAgQXV0 b3NlbnNlIHJlZmVycyB0byB0aGUgYXV0b21hdGljIHJldHVybiBvZiBzZW5zZSBkYXRhIHRvIHRo ZSBpbmktDQogICAgIHRpYXRvciBpbiBjYXNlIGEgY29tbWFuZCBkaWQgbm90IGNvbXBsZXRlIHN1 Y2Nlc3NmdWxseS4gSWYNCiAgICAgYXV0b3NlbnNlIGlzIHR1cm5lZCBvZmYsIHRoZSBpbml0aWF0 b3IgbXVzdCBleHBsaWNpdGx5IHJlcXVlc3QgdGhhdA0KICAgICBzZW5zZSBkYXRhIGJlIHNlbnQg dG8gaXQgYWZ0ZXIgc29tZSBjb21tYW5kIGhhcyBjb21wbGV0ZWQgd2l0aCBhDQogICAgIENIRUNL IENPTkRJVElPTiBzdGF0dXMuICBUaGUgZGVmYXVsdCBmb3IgaVNDU0kgaXMgdG8gd29yayB3aXRo DQogICAgIEF1dG9zZW5zZSBlbmFibGVkLg0KDQogICAgIE5vdGUgdGhhdCBldmVuIGlmIGEgU0NT SSB0YXJnZXQvTFVOIGRvZXMgbm90IHN1cHBvcnQgQXV0b3NlbnNlLCBpdA0KICAgICBtYXkgc3Rp bGwgYmUgcG9zc2libGUgZm9yIGlTQ1NJIHRvIHdvcmsgd2l0aCBBdXRvc2Vuc2UuICBUaGlzIGNh bg0KICAgICBiZSBhY2NvbXBsaXNoZWQgYXMgZm9sbG93cy4gV2hlbmV2ZXIgYSBDSEVDSyBDT05E SVRJT04gc3RhdHVzIGlzDQogICAgIGFib3V0IHRvIGJlIHJldHVybmVkLCB0aGUgaVNDU0kgY29t cG9uZW50IG9uIHRoZSB0YXJnZXQgaW1tZWRpYXRlbHkNCiAgICAgcXVlcmllcyB0aGUgdGFyZ2V0 L0xVTiBmb3IgdGhlIHNlbnNlIGRhdGEuIGlTQ1NJIGNhbiB0aGVuIHJldHVybg0KICAgICB0aGUg c2Vuc2UgZGF0YSB0byB0aGUgaW5pdGlhdG9yIHRvZ2V0aGVyIHdpdGggdGhlIENIRUNLIENPTkRJ VElPTg0KICAgICBzdGF0dXMuICBJdCBpcyBub3QgbmVjZXNzYXJ5IGZvciBpU0NTSSB0byB3YWl0 IGZvciB0aGUgaW5pdGlhdG9yIHRvDQogICAgIGV4cGxpY2l0bHkgcmVxdWVzdCB0aGUgc2Vuc2Ug ZGF0YTsgdGhlIHRhcmdldCBpU0NTSSBjb2RlIGNhbiBwZXItDQogICAgIGZvcm0gdGhpcyBvcGVy YXRpb24gYXV0b21hdGljYWxseSwgZXZlbiBmb3IgZGV2aWNlcy9MVU5zIHRoYXQgZG8NCiAgICAg bm90IG9yZGluYXJpbHkgcHJvdmlkZSBhdXRvbWF0aWMgc2Vuc2UgZGF0YS4NCg0KDQoNCg0KDQpT YXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBbUGFnZSAzN10NCgwNCmlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAw MA0KDQoNCjUuNC4gIFRDUCBSRE1BIG9wdGlvbg0KDQogICAgIFRoZSBUQ1AgUkRNQSBvcHRpb24g W1JETUFdIGlzIGFuIGFubm90YXRpb24gb24gaW5kaXZpZHVhbCBUQ1Agc2VnLQ0KICAgICBtZW50 cyB0aGF0IGNhbiByZWR1Y2UgdGhlIG51bWJlciBvZiBjb3BpZXMgbmVjZXNzYXJ5IGF0IHRoZQ0K ICAgICByZWNlaXZlci4gVGhlIFJETUEgb3B0aW9uIHN1Y2NpbmN0bHkgZGVzY3JpYmVzIHRoZSBw b3J0aW9uIG9mIGEgVENQDQogICAgIHBheWxvYWQgdGhhdCBob2xkcyBidWxrIGRhdGEuDQoNCg0K DQo1LjUuICBUQ1AgQ29ubmVjdGlvbiBPcHRpb25zDQoNCiAgICAgU29tZSB0YXJnZXRzIG1heSB3 YW50IHRvIGluZm9ybSAob3IgbmVnb3RpYXRlIHdpdGgpIGFuIGluaXRpYXRvcg0KICAgICBjb25j ZXJuaW5nIHNvbWUgcGFyYW1ldGVycyByZWxhdGVkIHRvIGJhbmR3aWR0aCwgUXVhbGl0eSBvZiBT ZXItDQogICAgIHZpY2UsIG9yIHNvbWUgb3RoZXIgYXZhaWxhYmxlIGZlYXR1cmVzIG9uIGl0cyB2 YXJpb3VzIG5ldHdvcmsgY29uLQ0KICAgICBuZWN0aW9ucy4gIFRoZXNlIGFyZSBleGNoYW5nZWQg YmV0d2VlbiB0aGUgaW5pdGlhdG9yIGFuZCB0aGUgdGFyZ2V0DQogICAgIHVzaW5nIFRleHQgQ29t bWFuZHMgYW5kIFJlc3BvbnNlcy4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6 YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAzOF0NCgwNCmlT Q1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjYuICBTZWN1cml0eSBD b25zaWRlcmF0aW9ucw0KDQoNCjYuMS4gIERhdGEgSW50ZWdyaXR5DQoNCiAgICAgV2UgYXNzdW1l IHRoYXQgZW5kLXRvLWVuZCBkYXRhIGludGVncml0eSBjYW4gYmUgYXNzdXJlZCBieSBUQ1AsIGJ5 DQogICAgIGFkZGluZyBhIG1vcmUgcG93ZXJmdWwgY2hlY2tzdW0gb3B0aW9uIHdoZW5ldmVyIHRo aXMgaXMgY29uc2lkZXJlZA0KICAgICBpbXBvcnRhbnQsIG9yIHJlcGxhY2luZyB0aGUgY2hlY2tz dW0gYnkgYSB3ZWFrZXIgb25lIChvciBldmVuICJudWwtDQogICAgIGxpZnlpbmcgaXQiKSBmb3Ig YXBwbGljYXRpb25zIGluIHdoaWNoIGRhdGEgaW50ZWdyaXR5IGlzIG5vdCBpbXBvci0NCiAgICAg dGFudCBhbmQgcmVjb3ZlcnkgZnJvbSBkYXRhIGVycm9ycyBjb3VsZCBiZSBoYXJtZnVsIChlLmcu LCBhdWRpbyBvcg0KICAgICB2aWRlbyBkaXN0cmlidXRpb24gc3RyZWFtcykuDQoNCjYuMi4gIExv Z2luIFByb2Nlc3MNCg0KICAgICBJbiBzb21lIGVudmlyb25tZW50cywgYSB0YXJnZXQgd2lsbCBu b3QgYmUgaW50ZXJlc3RlZCBpbiBhdXRoZW50aS0NCiAgICAgY2F0aW5nIHRoZSBpbml0aWF0b3Iu IEluIHRoaXMgY2FzZSwgdGhlIHRhcmdldCBjYW4gc2ltcGx5IGlnbm9yZQ0KICAgICBzb21lIG9y IGFsbCBvZiB0aGUgcGFyYW1ldGVycyBzZW50IGluIGEgTG9naW4gQ29tbWFuZCwgYW5kIHRoZSB0 YXItDQogICAgIGdldCBjYW4gc2ltcGx5IHJlcGx5IHdpdGggYSBiYXNpYyBMb2dpbiBSZXNwb25z ZSBpbmRpY2F0aW5nIGEgc3VjLQ0KICAgICBjZXNzZnVsIGxvZ2luLg0KDQogICAgIFNvbWUgdGFy Z2V0cyBtYXkgd2FudCB0byBwZXJmb3JtIHNvbWUga2luZCBvZiBhdXRoZW50aWNhdGlvbi4gIFRo ZQ0KICAgICBBdXRoZW50aWNhdG9yIGtleSBpcyBkZWZpbmVkIGZvciB0aGlzIHB1cnBvc2UuICBW YXJpb3VzIGF1dGhlbnRpY2EtDQogICAgIHRpb24gc2NoZW1lcyBjYW4gYmUgdXNlZCwgaW5jbHVk aW5nIGVuY3J5cHRlZCBwYXNzd29yZHMgYW5kIHRydXN0ZWQNCiAgICAgY2VydGlmaWNhdGUgYXV0 aG9yaXRpZXMuDQoNCiAgICAgT25jZSB0aGUgaW5pdGlhdG9yIGFuZCB0YXJnZXQgYXJlIGNvbmZp ZGVudCBvZiB0aGUgaWRlbnRpdHkgb2YgdGhlDQogICAgIGF0dGFjaGVkIHBhcnR5LCB0aGUgZXN0 YWJsaXNoZWQgY2hhbm5lbCBpcyBjb25zaWRlcmVkIHNlY3VyZS4NCg0KICAgICBJdCBpcyBhbnRp Y2lwYXRlZCB0aGF0IG1vc3QgdGFyZ2V0IGRldmljZXMgd2lsbCBub3QgYm90aGVyIHdpdGggYWxs DQogICAgIG9mIHRoZSBwb3NzaWJsZSBjaGVja3MsIGJ1dCB0aGUgcHJvdG9jb2wgcHJvdmlkZXMg c3VmZmljaWVudCBtZWFucw0KICAgICB0byBwZXJmb3JtIHRoZSBjaGVja3MsIGlmIHJlcXVpcmVk IGJ5IHRoZSB0YXJnZXQuDQoNCjYuMy4gIElBTkEgQ29uc2lkZXJhdGlvbnMNCg0KICAgICBUaGVy ZSB3aWxsIGJlIGEgd2VsbCBrbm93biBwb3J0IGZvciBpU0NTSSBjb25uZWN0aW9ucy4gIFRoZXNl IHdlbGwNCiAgICAga25vd24gcG9ydHMgd2lsbCBoYXZlIHRvIGJlIHJlZ2lzdGVyZWQgd2l0aCBJ QU5BLg0KDQogICAgIEEgY2hlY2tzdW0gdHlwZSB3aWxsIGFsc28gaGF2ZSB0byBiZSByZWdpc3Rl cmVkIHdpdGggSUFOQS4NCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KU2F0cmFuLCBTbWl0aCwg U2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMzld DQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDANCg0KDQo3LiAgQXV0 aG9ycycgQWRkcmVzc2VzDQoNCg0KICAgICBKdWxpYW4gU2F0cmFuDQogICAgIEthbG1hbiBNZXRo DQogICAgIE1laXIgVG9sZWRhbm8NCiAgICAgSUJNLCBIYWlmYSBSZXNlYXJjaCBMYWINCiAgICAg TUFUQU0gLSBBZHZhbmNlZCBUZWNobm9sb2d5IENlbnRlcg0KICAgICBIYWlmYSAzMTkwNSwgSXNy YWVsDQogICAgIFBob25lICs5NzIgNCA4MjkgNjIxMQ0KICAgICBFbWFpbDogSnVsaWFuX1NhdHJh bkB2bmV0LmlibS5jb20gbWV0aEBpbC5pYm0uY29tDQogICAgICAgICB0b2xlZGFub0BpbC5pYm0u Y29tDQoNCg0KICAgICBEYW5pZWwgRi4gU21pdGgNCiAgICAgSUJNIEFsbWFkZW4gUmVzZWFyY2gg Q2VudGVyDQogICAgIDY1MCBIYXJyeSBSb2FkDQogICAgIFNhbiBKb3NlLCBDQSA5NTEyMC02MDk5 LCBVU0ENCiAgICAgUGhvbmU6ICsxIDQwOCA5MjcgMjA3Mg0KICAgICBFbWFpbDogZGZzbWl0aEBh bG1hZGVuLmlibS5jb20NCg0KDQogICAgIENvc3RhIFNhcHVudHpha2lzDQogICAgIENpc2NvIFN5 c3RlbXMsIEluYy4NCiAgICAgMTcwIFcuIFRhc21hbiBEcml2ZQ0KICAgICBTYW4gSm9zZSwgQ0Eg OTUxMzQsIFVTQQ0KICAgICBQaG9uZTogKzEgNDA4IDUyNSA1NDk3DQogICAgIEVtYWlsOiBjc2Fw dW50ekBjaXNjby5jb20NCg0KDQogICAgIENvbW1lbnRzIG1heSBiZSBzZW50IHRvIEp1bGlhbiBT YXRyYW4sIERhbmllbCBTbWl0aCwgQ29zdGENCiAgICAgU2FwdW50emFraXMsIG9yIEthbG1hbiBN ZXRoLg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQpTYXRyYW4sIFNtaXRo LCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSA0 MF0NCgwNCmlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0KDQoNCjguICBS ZWZlcmVuY2VzIGFuZCBCaWJsaW9ncmFwaHkNCg0KICAgICBbUkRNQV0gICAgSW50ZXJuZXQgRHJh ZnQ6IFRDUCBSRE1BIG9wdGlvbiAod29yayBpbiBwcm9ncmVzcykNCiAgICAgW1NBTTJdICAgIEFO U0kgWDMuMjcwLTE5OTgsIFNDU0ktMyBBcmNoaXRlY3R1cmUgTW9kZWwgKFNBTS0yKQ0KICAgICBb VExTXSAgICAgVGhlIFRMUyBQcm90b2NvbCwgUkZDIDIyNDYsIFQuIERpZXJrcyBldCBhbC4NCg0K ICAgICBbQUxUQ10gICAgSW50ZXJuZXQgRHJhZnQ6IEFsdGVybmF0aXZlIGNoZWNrc3VtcyAod29y ayBpbiBwcm9ncmVzcykNCiAgICAgW0NBTV0gICAgIEFOU0kgWDMuMjMyLTE5OVgsIENvbW1vbiBB Y2Nlc3MgTWV0aG9kLTMgKENhbS0zKQ0KICAgICBbQ1JDXSAgICAgSVNPIDMzMDksIEhpZ2gtTGV2 ZWwgRGF0YSBMaW5rIENvbnRyb2wgKENSQyAzMikNCiAgICAgW1JGQzc5M10gIFRyYW5zbWlzc2lv biBDb250cm9sIFByb3RvY29sLCBSRkMgNzkzDQogICAgIFtSRkMxMTIyXSBSZXF1aXJlbWVudHMg Zm9yIEludGVybmV0IEhvc3RzIC0tIENvbW11bmljYXRpb24gTGF5ZXIsDQogICAgICAgICAgICAg ICBSRkMgMTEyMiwgUi4gQnJhZGVuIChlZGl0b3IpDQogICAgIFtTQkNdICAgICBBTlNJIFgzLjMw Ni0xOTlYLCBTQ1NJLTMgQmxvY2sgQ29tbWFuZHMgKFNCQykNCiAgICAgW1NDU0kyXSAgIEFOU0kg WDMuMTMxLTE5OTQsIFNDU0ktMg0KICAgICBbU1BDXSAgICAgQU5TSSBYMy4zMDEtMTk5WCwgU0NT SS0zIFByaW1hcnkgQ29tbWFuZHMgKFNQQykNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNClNhdHJhbiwgU21pdGgs IFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDQx XQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoNCg0KOS4gIEFw cGVuZGl4IEEgLSBFeGFtcGxlcw0KDQoNCjkuMS4gIFJlYWQgb3BlcmF0aW9uIGV4YW1wbGUNCg0K DQogICAgIHxJbml0aWF0b3IgRnVuY3Rpb258ICAgIE1lc3NhZ2UgVHlwZSAgICAgICB8ICBUYXJn ZXQgRnVuY3Rpb24gICAgIHwNCiAgICAgKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KICAgICB8ICBDb21tYW5kIHJlcXVl c3QgfFNDU0kgQ29tbWFuZCAoUkVBRCk+Pj4gfCAgICAgICAgICAgICAgICAgICAgICB8DQogICAg IHwgIChyZWFkKSAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAg ICAgICAgICAgIHwNCiAgICAgKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KICAgICB8ICAgICAgICAgICAgICAgICAgfCAg ICAgICAgICAgICAgICAgICAgICAgfCBQcmVwYXJlIERhdGEgVHJhbnNmZXJ8DQogICAgICstLS0t LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0t LS0tLSsNCiAgICAgfCAgIFJlY2VpdmUgRGF0YSAgIHwgICA8PDwgU0NTSSBEYXRhICAgICAgIHwg ICBTZW5kIERhdGEgICAgICAgICAgfA0KICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rDQogICAgIHwgICBSZWNlaXZl IERhdGEgICB8ICAgPDw8IFNDU0kgRGF0YSAgICAgICB8ICAgU2VuZCBEYXRhICAgICAgICAgIHwN CiAgICAgKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0tLS0tLS0tKw0KICAgICB8ICAgUmVjZWl2ZSBEYXRhICAgfCAgIDw8PCBTQ1NJIERh dGEgICAgICAgfCAgIFNlbmQgRGF0YSAgICAgICAgICB8DQogICAgICstLS0tLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsNCiAgICAg fCAgICAgICAgICAgICAgICAgIHwgICA8PDwgU0NTSSBSZXNwb25zZSAgIHxTZW5kIFN0YXR1cyBh bmQgU2Vuc2UgfA0KICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rDQogICAgIHwgQ29tbWFuZCBDb21wbGV0ZSB8ICAg ICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgIHwNCiAgICAgKy0tLS0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0t LS0tKw0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoN Cg0KDQpTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBbUGFnZSA0Ml0NCgwNCmlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1 bmUgMjAwMA0KDQoNCjkuMi4gIFdyaXRlIG9wZXJhdGlvbiBleGFtcGxlDQoNCg0KICAgICArLS0t LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0t LS0tLSsNCiAgICAgfEluaXRpYXRvciBGdW5jdGlvbnwgICAgTWVzc2FnZSBUeXBlICAgICAgIHwg IFRhcmdldCBGdW5jdGlvbiAgICB8DQogICAgICstLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KICAgICB8ICBDb21tYW5kIHJl cXVlc3QgfFNDU0kgQ29tbWFuZCAoV1JJVEUpPj4+fCBSZWNlaXZlIGNvbW1hbmQgICAgIHwNCiAg ICAgfCAgKHdyaXRlKSAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgIHwgYW5kIHF1ZXVl IGl0ICAgICAgICB8DQogICAgICstLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KICAgICB8ICAgICAgICAgICAgICAgICAgfCAg ICAgICAgICAgICAgICAgICAgICAgfCBQcm9jZXNzIG9sZCBjb21tYW5kc3wNCiAgICAgKy0tLS0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0t LS0rDQogICAgIHwgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICB8IFJl YWR5IHRvIHByb2Nlc3MgICAgfA0KICAgICB8ICAgICAgICAgICAgICAgICAgfCAgIDw8PCBSVFQg ICAgICAgICAgICAgfCBXUklURSBjb21tYW5kICAgICAgIHwNCiAgICAgKy0tLS0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0rDQogICAg IHwgICBTZW5kIERhdGEgICAgICB8ICAgU0NTSSBEYXRhID4+PiAgICAgICB8ICAgUmVjZWl2ZSBE YXRhICAgICAgfA0KICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsNCiAgICAgfCAgIFNlbmQgRGF0YSAgICAgIHwgICBT Q1NJIERhdGEgPj4+ICAgICAgIHwgICBSZWNlaXZlIERhdGEgICAgICB8DQogICAgICstLS0tLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0t Kw0KICAgICB8ICAgICAgICAgICAgICAgICAgfCAgIDw8PCBSVFQgICAgICAgICAgICAgfCAgICAg ICAgICAgICAgICAgICAgIHwNCiAgICAgKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0rDQogICAgIHwgICBTZW5kIERhdGEgICAg ICB8ICAgU0NTSSBEYXRhID4+PiAgICAgICB8ICAgUmVjZWl2ZSBEYXRhICAgICAgfA0KICAgICAr LS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LS0tLS0tLSsNCiAgICAgfCAgICAgICAgICAgICAgICAgIHwgICA8PDwgU0NTSSBSZXNwb25zZSAg IHxTZW5kIFN0YXR1cyBhbmQgU2Vuc2V8DQogICAgICstLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KICAgICB8IENvbW1hbmQg Q29tcGxldGUgfCAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgIHwN CiAgICAgKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0tLS0tLS0rDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0K DQoNCg0KDQoNClNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIFtQYWdlIDQzXQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAg ICAgSnVuZSAyMDAwDQoNCg0KMTAuICBBcHBlbmRpeCBCIC0gTG9naW4vVGV4dCBrZXlzDQoNCg0K MTAuMS4gIFRhcmdldA0KDQogICAgIFRhcmdldDpkb21haW5uYW1lWy9tb2RpZmllcl0NCg0KICAg ICBFeGFtcGxlczoNCg0KICAgICAgICAgICAgIFRhcmdldDpkaXNrLWFycmF5LnNqLWJsZGctaC5j aXNjby5jb20NCiAgICAgICAgICAgICBUYXJnZXQ6ZGlzay1hcnJheS5zai1ibGRnLWguY2lzY28u Y29tL2Rpc2szDQoNCiAgICAgVGhpcyBrZXkgaXMgcHJvdmlkZWQgYnkgdGhlIGluaXRpYXRvciBv ZiB0aGUgVENQIGNvbm5lY3Rpb24gdG8gdGhlDQogICAgIHJlbW90ZSBlbmRwb2ludC4gVGhlIFRh cmdldCBrZXkgc3BlY2lmaWVzIHRoZSBkb21haW4gbmFtZSBvZiB0aGUNCiAgICAgdGFyZ2V0LCBz aW5jZSB0aGF0IGluZm9ybWF0aW9uIGlzIG5vdCBhdmFpbGFibGUgZnJvbSB0aGUgVENQIGxheWVy Lg0KICAgICBUaGUgdGFyZ2V0IGlzIG5vdCByZXF1aXJlZCB0byBzdXBwb3J0IHRoaXMga2V5Lg0K DQogICAgIFRoZSBpbml0aWF0b3Igc2hvdWxkIHNlbmQgdGhpcyBrZXkgaW4gdGhlIGZpcnN0IGxv Z2luIG1lc3NhZ2UuIFRoZQ0KICAgICBUYXJnZXQga2V5IG1pZ2h0IGJlIHVzZWQgYnkgdGhlIHRh cmdldCB0byBsZWFybiB0aGUgaW50ZW5kZWQgaW5pLQ0KICAgICB0aWF0b3IgdmlldyBvZiB0aGUg dGFyZ2V0Lg0KDQoNCjEwLjIuICBJbml0aWF0b3INCg0KDQogICAgIEluaXRpYXRvcjpbZG9tYWlu bmFtZVsvbW9kaWZpZXJdXQ0KDQogICAgIEV4YW1wbGVzOg0KDQogICAgICAgICAgICAgSW5pdGlh dG9yOnNhbXBsZS5mb29iYXIub3JnDQogICAgICAgICAgICAgSW5pdGlhdG9yOmNsdXN0ZXIuZm9v YmFyLm9yZy9tYWNoaW5lMQ0KICAgICAgICAgICAgIEluaXRpYXRvcjoNCg0KICAgICBUaGUgSW5p dGlhdG9yIGtleSBlbmFibGVzIHRoZSBpbml0aWF0b3IgdG8gaWRlbnRpZnkgaXRzZWxmIHRvIHRo ZQ0KICAgICByZW1vdGUgZW5kcG9pbnQuIFRoZSBkb21haW4gbmFtZSBzaG91bGQgYmUgdGhhdCBv ZiB0aGUgaW5pdGlhdG9yLg0KICAgICBBIHplcm8tbGVuZ3RoIGRvbWFpbiBuYW1lIGlzIGludGVy cHJldGVkIGFzICJvdGhlciBzaWRlIG9mIFRDUCBjb24tDQogICAgIG5lY3Rpb24iLiBUaGUgdGFy Z2V0IG1heSBzaWxlbnRseSBpZ25vcmUgdGhpcyBrZXkgaWYgaXQgZG9lcyBub3QNCiAgICAgc3Vw cG9ydCBpdC4NCg0KICAgICBGb3IgbW9yZSBzZWN1cml0eSwgYSBjZXJ0aWZpY2F0ZS1iYXNlZCBw cm90b2NvbCBbVExTXSBtYXkgYmUgdXNlZA0KICAgICBvbiB0aGUgY2hhbm5lbCBhbmQgdGFrZSBw cmVjZWRlbmNlIG92ZXIgdGhpcyBwcm90b2NvbC4NCg0KDQoxMC4zLiAgQXV0aGVudGljYXRvcg0K DQoNCiAgICAgQXV0aGVudGljYXRvcjo8VVRGOC1TdHJpbmc+DQoNCg0KDQoNClNhdHJhbiwgU21p dGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdl IDQ0XQ0KDA0KaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQoNCg0KICAg ICBFeGFtcGxlczoNCg0KICAgICAgICAgICAgIEF1dGhlbnRpY2F0b3I6b3Blbi1zZXNhbWUNCg0K ICAgICBUaGUgYXV0aGVudGljYXRvciBpcyBhIHNlY3JldCB0aGF0IHRoZSBpbml0aWF0b3IgdXNl cyB0byBnYWluDQogICAgIGFjY2VzcyB0byB0aGUgdGFyZ2V0J3MgTFVOcy4NCg0KDQoNCjEwLjQu ICBTZW5kQXV0aGVudGljYXRvcg0KDQoNCiAgICAgU2VuZEF1dGhlbnRpY2F0b3I6eWVzIFJlc3Bv bnNlOiBBdXRoZW50aWNhdG9yOjxVVEY4LVN0cmluZz4NCg0KICAgICBFeGFtcGxlczoNCg0KICAg ICAgICAgICAgIFNlbmRBdXRoZW50aWNhdG9yOnllcw0KICAgICAgICAgICAgIC0+IEF1dGhlbnRp Y2F0b3I6YWxha2F6YW0NCg0KICAgICBUaGUgU2VuZEF1dGhlbnRpY2F0b3Iga2V5IGlzIHVzZWQg dG8gcmVxdWVzdCBmcm9tIHRoZSBwYXJ0eSBvbiB0aGUNCiAgICAgb3RoZXIgc2lkZSBvZiB0aGUg VENQIGNvbm5lY3Rpb24gdG8gc2VuZCBpdHMgQXV0aGVudGljYXRvci4gIGlTQ1NJDQogICAgIGRl dmljZXMgbWF5IHJlZnVzZSB0byBncmFudCBhY2Nlc3MgdW50aWwgcHJvcGVyIGF1dGhlbnRpY2F0 aW9uIGhhcw0KICAgICBiZWVuIHBlcmZvcm1lZCBieSB0aGUgcGFydGllcyBpbnZvbHZlZC4NCg0K DQoxMC41LiAgQWxsb3dOb1JUVA0KDQoNCiAgICAgQWxsb3dOb1JUVDo8eWVzfG5vPiBSZXNwb25z ZTogQWxsb3dOb1JUVDo8eWVzfG5vPg0KDQogICAgIEV4YW1wbGVzOg0KDQogICAgICAgICAgICAg QWxsb3dOb1JUVDp5ZXMNCiAgICAgICAgICAgICAtPiBBbGxvd05vUlRUOnllcw0KDQogICAgIFRo ZSBBbGxvd05vUlRUIGtleSBpcyB1c2VkIHRvIGFsbG93IGFuIGluaXRpYXRvciB0byBzZW5kIGRh dGEgdG8gYQ0KICAgICB0YXJnZXQgd2l0aG91dCB0aGUgdGFyZ2V0IGhhdmluZyBzZW50IGFuIFJU VCB0byB0aGUgaW5pdGlhdG9yLiAgVGhlDQogICAgIGRlZmF1bHQgYWN0aW9uIGlzIHRoYXQgUlRU IGlzIHJlcXVpcmVkLCB1bmxlc3MgYm90aCB0aGUgaW5pdGlhdG9yDQogICAgIGFuZCB0aGUgdGFy Z2V0IHNlbmQgdGhpcyBrZXktcGFpciBhdHRyaWJ1dGUgc3BlY2lmeWluZw0KICAgICBBbGxvd05v UlRUOnllcy4gIE9uY2UgQWxsb3dOb1JUVCBoYXMgYmVlbiBzZXQgdG8gJ3llcycsIGl0IGNhbm5v dA0KICAgICBiZSBzZXQgYmFjayB0byAnbm8nLg0KDQoNCg0KMTAuNi4gIE9yaWdpbmFsSW5pdGlh dG9yDQoNCg0KICAgICBPcmlnaW5hbEluaXRpYXRvcjpbZG9tYWlubmFtZVsvbW9kaWZpZXJdXQ0K DQoNCg0KU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgW1BhZ2UgNDVdDQoMDQppU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBK dW5lIDIwMDANCg0KDQogICAgIEV4YW1wbGVzOg0KDQogICAgICAgICAgICAgT3JpZ2luYWxJbml0 aWF0b3I6c2FtcGxlLmZvb2Jhci5vcmcNCg0KICAgICBUaGUgT3JpZ2luYWxJbml0aWF0b3Iga2V5 IGlzIHVzZWQgdG8gcGVyZm9ybSBhIHByb3h5IGxvZ2luIGZyb20gb25lDQogICAgIHRhcmdldCB0 byBhbm90aGVyIHRhcmdldCBpbiBvcmRlciB0byBwZXJmb3JtIGEgdGhpcmQtcGFydHkgb3BlcmEt DQogICAgIHRpb24gKGxpa2UgQ09QWSkgZm9yIHNvbWUgaW5pdGlhdG9yLiBUaGUgZmlyc3QgdGFy Z2V0IGFjdHMgYXMgdGhlDQogICAgIGluaXRpYXRvciBmb3IgdGhlIHNlY29uZCB0YXJnZXQsIGJ1 dCBpdCBtdXN0IHByb3ZpZGUgdGhlIGF1dGhvcml6YS0NCiAgICAgdGlvbiBpbmZvcm1hdGlvbiBv ZiB0aGUgb3JpZ2luYWwgaW5pdGlhdG9yLg0KDQoNCg0KMTAuNy4gIFRhcmdldDINCg0KDQogICAg IFRhcmdldDI6ZG9tYWlubmFtZVsvbW9kaWZpZXJdDQoNCiAgICAgRXhhbXBsZXM6DQoNCiAgICAg ICAgICAgICBUYXJnZXQyOnNhbXBsZS5mb29iYXIub3JnDQogICAgICAgICAgICAgVGFyZ2V0Mjpz YW1wbGUuZm9vYmFyLm9yZy9yYWlkMg0KDQogICAgIFRoZSBUYXJnZXQyIGtleSBpcyB1c2VkIGlu IGEgdGhpcmQtcGFydHkgU0NTSSBjb21tYW5kIChsaWtlIENPUFkpDQogICAgIGJldHdlZW4gdGFy Z2V0cyB0aGF0IGRvIG5vdCBsaWUgb24gdGhlIHNhbWUgU0NTSSBmYWJyaWMuIFRoZSBpbmktDQog ICAgIHRpYXRvciBtdXN0IHNwZWNpZnkgdGhlIG5hbWUgb2YgdGhlIGRpc3RhbnQgdGFyZ2V0IHRv IHRoZSBvcmlnaW5hbA0KICAgICB0YXJnZXQsIHNvIHRoYXQgdGhlIG9yaWdpbmFsIHRhcmdldCBj YW4gTG9naW4gdG8gdGhlIGRpc3RhbnQgdGFyZ2V0DQogICAgIGFuZCB0aGVuIHBlcmZvcm0gdGhl IHRoaXJkLXBhcnR5IGNvbW1hbmQuDQoNCg0KDQogICAgIEV4cGlyZXMgMTUgRGVjZW1iZXIgMjAw MA0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNCg0KDQoNClNhdHJhbiwgU21pdGgs IFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDQ2 XQ0KDA0K --0__=9Y9GvtN5f8kQTgvW2CmfSX5NGi8FufevdN5tMllV7hDYdPwFhWZauBVR-- From XXX Thu Jun 15 23:40:34 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Jun 15 19:38:45 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA15763 for ; Thu, 15 Jun 2000 19:38:45 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA05658 for ips-outgoing; Thu, 15 Jun 2000 19:36:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from calliope1.fm.intel.com (calliope1.fm.intel.com [132.233.247.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA05653 for ; Thu, 15 Jun 2000 19:36:46 -0400 (EDT) Received: from fmsmsx28.FM.INTEL.COM (fmsmsx28.fm.intel.com [132.233.42.28]) by calliope1.fm.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.30 2000/06/08 18:25:35 dmccart Exp $) with ESMTP id XAA24547 for ; Thu, 15 Jun 2000 23:36:44 GMT Received: by fmsmsx28.fm.intel.com with Internet Mail Service (5.5.2448.0) id ; Thu, 15 Jun 2000 16:36:42 -0700 Message-ID: From: "Cameron, Don" To: ips@ece.cmu.edu Subject: Who is doing what Date: Thu, 15 Jun 2000 16:36:41 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I am trying to figure out who the players are in the storage over IP arena. >>From following the discussions here, and reading the trade rags, I have come up with the following. Please let me know where I am wrong: The IP Storage Working Group (IBM, Cisco, HP, Adaptec, Quantum, EMC, and others) Mapping of SCSI to TCP. All agree that TCP is appropriate for WAN. A majority advocate TCP for both LAN and WAN, a minority advocate a lighter-weight transport protocol for LAN only. Adaptec: EtherStorage Mapping of SCSI to light-weight transport protocol specifically designed for LAN only. Uses SEP (SCSI Encapsulation Protocol). Nishan: SoIP (Storage Over IP) Can't find much detail here. Gadzoox and Lucent: Fibre Channel over IP. Proposal is to use IP to bridge between FC SANs for MANs (and WANs?). SAN, Ltd: SSCOP (Service Specific Connection Oriented Protocol) Route storage data over IP (UDP?) using the data link protocol from ATM that allows selective re-transmission. From XXX Fri Jun 16 02:52:03 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Jun 15 20:03:32 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA03028 for ; Thu, 15 Jun 2000 20:03:32 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA06177 for ips-outgoing; Thu, 15 Jun 2000 20:02:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA06173 for ; Thu, 15 Jun 2000 20:01:58 -0400 (EDT) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id RAA06565; Thu, 15 Jun 2000 17:00:44 -0700 (PDT) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id RAA21534; Thu, 15 Jun 2000 17:00:44 -0700 (PDT) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Thu, 15 Jun 2000 17:00:44 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C05B31C04@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: "'Cameron, Don'" , ips@ece.cmu.edu Subject: RE: Who is doing what Date: Thu, 15 Jun 2000 17:00:43 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Don, Adaptec's EtherStorage also supports TCP, as well as a light-weight transport protocol (STP). There has been some confusion over acronyms. STP is SAN Transport Protocol (an acronym within an acronym). It has been confused with Scheduled Transfer, which it is not. For more information on EtherStorage, see http://adaptec.com/etherstorage Paul von Stamwitz > -----Original Message----- > From: Cameron, Don [mailto:don.cameron@intel.com] > Sent: Thursday, June 15, 2000 4:37 PM > To: ips@ece.cmu.edu > Subject: Who is doing what > > > I am trying to figure out who the players are in the storage > over IP arena. > From following the discussions here, and reading the trade > rags, I have come > up with the following. Please let me know where I am wrong: > > The IP Storage Working Group (IBM, Cisco, HP, Adaptec, > Quantum, EMC, and > others) > Mapping of SCSI to TCP. All agree that TCP is > appropriate for WAN. A > majority advocate TCP for both LAN and WAN, a minority advocate a > lighter-weight transport protocol for LAN only. > > Adaptec: EtherStorage > Mapping of SCSI to light-weight transport protocol specifically > designed for LAN only. Uses SEP (SCSI Encapsulation Protocol). > > Nishan: SoIP (Storage Over IP) > Can't find much detail here. > > Gadzoox and Lucent: > Fibre Channel over IP. Proposal is to use IP to bridge > between FC > SANs for MANs (and WANs?). > > SAN, Ltd: SSCOP (Service Specific Connection Oriented Protocol) > Route storage data over IP (UDP?) using the data link > protocol from > ATM that allows selective re-transmission. > From XXX Fri Jun 16 02:52:03 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Thu Jun 15 19:54:51 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id TAA16157 for ; Thu, 15 Jun 2000 19:54:51 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA06000 for ips-outgoing; Thu, 15 Jun 2000 19:53:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from seattle.3com.com (seattle.3com.com [129.213.128.97]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA05995 for ; Thu, 15 Jun 2000 19:53:10 -0400 (EDT) From: Dave_Lee@3com.com Received: from new-york.3com.com (new-york.3com.com [129.213.157.12]) by seattle.3com.com (8.8.8/8.8.8) with ESMTP id QAA04898; Thu, 15 Jun 2000 16:52:39 -0700 (PDT) Received: from hqoutbound.ops.3com.com (hqoutbound.OPS.3Com.COM [139.87.48.104]) by new-york.3com.com (8.8.8/8.8.8) with SMTP id QAA24786; Thu, 15 Jun 2000 16:52:28 -0700 (PDT) Received: by hqoutbound.ops.3com.com(Lotus SMTP MTA v4.6.7 (934.1 12-30-1999)) id 882568FF.0082F6D7 ; Thu, 15 Jun 2000 16:50:28 -0700 X-Lotus-FromDomain: 3COM To: "Cameron, Don" cc: ips@ece.cmu.edu Message-ID: <882568FF.0082F647.00@hqoutbound.ops.3com.com> Date: Thu, 15 Jun 2000 16:50:32 -0700 Subject: Re: Who is doing what Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The IP Storage WG hasn't officially been created yet and, thus, there are no WG documents. So the IBM/Cisco/etc. group is a "design team" and does not speak for the WG. There are a number of parties in support of Scheduled Transfer. "Cameron, Don" on 06/15/2000 04:36:41 PM Sent by: "Cameron, Don" To: ips@ece.cmu.edu cc: (Dave Lee/HQ/3Com) Subject: Who is doing what I am trying to figure out who the players are in the storage over IP arena. >>From following the discussions here, and reading the trade rags, I have come up with the following. Please let me know where I am wrong: The IP Storage Working Group (IBM, Cisco, HP, Adaptec, Quantum, EMC, and others) Mapping of SCSI to TCP. All agree that TCP is appropriate for WAN. A majority advocate TCP for both LAN and WAN, a minority advocate a lighter-weight transport protocol for LAN only. Adaptec: EtherStorage Mapping of SCSI to light-weight transport protocol specifically designed for LAN only. Uses SEP (SCSI Encapsulation Protocol). Nishan: SoIP (Storage Over IP) Can't find much detail here. Gadzoox and Lucent: Fibre Channel over IP. Proposal is to use IP to bridge between FC SANs for MANs (and WANs?). SAN, Ltd: SSCOP (Service Specific Connection Oriented Protocol) Route storage data over IP (UDP?) using the data link protocol from ATM that allows selective re-transmission. From XXX Fri Jun 16 21:54:06 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Fri Jun 16 15:50:26 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id PAA09959 for ; Fri, 16 Jun 2000 15:50:26 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA10423 for ips-outgoing; Fri, 16 Jun 2000 15:50:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gifw.genroco.com (genroco.com [205.254.195.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA10417 for ; Fri, 16 Jun 2000 15:50:07 -0400 (EDT) Received: from gi2.genroco.com (gi2.genroco.com [192.133.120.3]) by gifw.genroco.com (8.9.3/8.9.3) with ESMTP id OAA05305; Fri, 16 Jun 2000 14:50:01 -0500 Received: from don (don.genroco.com [192.133.120.101]) by gi2.genroco.com (8.9.3/8.9.3) with SMTP id OAA04896; Fri, 16 Jun 2000 14:50:01 -0500 From: "Don Woelz" To: Cc: "Cameron, Don" Subject: RE: Who is doing what Date: Fri, 16 Jun 2000 14:50:00 -0500 Message-ID: <002601bfd7cc$0e9c7ca0$657885c0@genroco.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk GENROCO has mapped SCSI on Scheduled Transfer, currently a draft ANSI T10 standard. The Schedule Transfer Protocol, also a draft ANSI standard (passed public review) is defined on IP (and implemented as such by SGI on Linux). So there is a standards-based solution already existing for SCSI on STP/IP. DonDonald D. Woelz Tel: 262-644-2505 V. P. of Sales and Marketing Mobile: 414-732-7705 GENROCO, Inc. Toll Free: 800-243-6762 255 Info Hwy Fax: 262-644-6667 Slinger, WI 53086 > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Cameron, Don > Sent: Thursday, June 15, 2000 6:37 PM > To: ips@ece.cmu.edu > Subject: Who is doing what > > > I am trying to figure out who the players are in the storage over IP arena. > From following the discussions here, and reading the trade rags, I have come > up with the following. Please let me know where I am wrong: > > The IP Storage Working Group (IBM, Cisco, HP, Adaptec, Quantum, EMC, and > others) > Mapping of SCSI to TCP. All agree that TCP is appropriate for WAN. A > majority advocate TCP for both LAN and WAN, a minority advocate a > lighter-weight transport protocol for LAN only. > > Adaptec: EtherStorage > Mapping of SCSI to light-weight transport protocol specifically > designed for LAN only. Uses SEP (SCSI Encapsulation Protocol). > > Nishan: SoIP (Storage Over IP) > Can't find much detail here. > > Gadzoox and Lucent: > Fibre Channel over IP. Proposal is to use IP to bridge between FC > SANs for MANs (and WANs?). > > SAN, Ltd: SSCOP (Service Specific Connection Oriented Protocol) > Route storage data over IP (UDP?) using the data link protocol from > ATM that allows selective re-transmission. > > From XXX Sun Jun 18 17:44:56 2000 Return-Path: owner-extdom.scsi-tcp@sj-msg-core-2.cisco.com Delivery-Date: Sun Jun 18 12:52:18 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA20316 for ; Sun, 18 Jun 2000 12:52:18 -0400 Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA06839 for ; Sun, 18 Jun 2000 12:52:16 -0400 (EDT) Received: from sj-msg-av-1.cisco.com (sj-msg-av-1.cisco.com [171.69.11.130]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id JAA08310 for ; Sun, 18 Jun 2000 09:46:39 -0700 (PDT) From: julian_satran@il.ibm.com Received: from proxy1.cisco.com (localhost [127.0.0.1]) by sj-msg-av-1.cisco.com (8.10.1/8.10.1) with SMTP id e5IGkSF09436 for ; Sun, 18 Jun 2000 09:46:28 -0700 (PDT) Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by proxy1.cisco.com with SMTP (MailShield v1.5); Sun, 18 Jun 2000 09:46:25 -0700 Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA19380; Sun, 18 Jun 2000 18:44:36 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA05882; Sun, 18 Jun 2000 18:44:36 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256902.005BF7A5 ; Sun, 18 Jun 2000 18:44:31 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: List SCSI-over-TCP , ips@ece.cmu.edu Message-ID: Date: Sun, 18 Jun 2000 19:43:42 +0300 Subject: a tentative agenda with some more details Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=MYoG3PRjZvobQjksSqYM2uAa4YW8KRAV83QOlrT8IFR9OJveCwESqzx0" Content-Disposition: inline X-SMTP-HELO: d12lmsgate-3.de.ibm.com X-SMTP-MAIL-FROM: julian_satran@il.ibm.com X-SMAP-Received-From: outside X-SMTP-PEER-INFO: d12lmsgate-3.de.ibm.com [195.212.91.201] --0__=MYoG3PRjZvobQjksSqYM2uAa4YW8KRAV83QOlrT8IFR9OJveCwESqzx0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline (See attached file: Haifa-Session-AgendaDraft.htm) --0__=MYoG3PRjZvobQjksSqYM2uAa4YW8KRAV83QOlrT8IFR9OJveCwESqzx0 Content-type: text/html; name="Haifa-Session-AgendaDraft.htm" Content-Disposition: attachment; filename="Haifa-Session-AgendaDraft.htm" Content-transfer-encoding: base64 Content-Description: Internet HTML PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIg0K eG1sbnM6dz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCINCnhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCg0KPGhlYWQ+DQo8bWV0YSBodHRwLWVx dWl2PUNvbnRlbnQtVHlwZSBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9d2luZG93cy0xMjU1 Ij4NCjxtZXRhIG5hbWU9UHJvZ0lkIGNvbnRlbnQ9V29yZC5Eb2N1bWVudD4NCjxtZXRhIG5hbWU9 R2VuZXJhdG9yIGNvbnRlbnQ9Ik1pY3Jvc29mdCBXb3JkIDkiPg0KPG1ldGEgbmFtZT1PcmlnaW5h dG9yIGNvbnRlbnQ9Ik1pY3Jvc29mdCBXb3JkIDkiPg0KPGxpbmsgcmVsPUZpbGUtTGlzdCBocmVm PSIuL0hhaWZhLVNlc3Npb24tQWdlbmRhRHJhZnRfZmlsZXMvZmlsZWxpc3QueG1sIj4NCjx0aXRs ZT5JUCBTdG9yYWdlIChJUFMpIFdvcmtpbmcgR3JvdXAgUHJvcG9zZWQgQ2hhcnRlciAtIERSQUZU PC90aXRsZT4NCjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPg0KIDxvOkRvY3VtZW50UHJvcGVydGll cz4NCiAgPG86QXV0aG9yPkp1bGlhbiBTYXRyYW48L286QXV0aG9yPg0KICA8bzpMYXN0QXV0aG9y Pkp1bGlhbiBTYXRyYW48L286TGFzdEF1dGhvcj4NCiAgPG86UmV2aXNpb24+MjwvbzpSZXZpc2lv bj4NCiAgPG86VG90YWxUaW1lPjI8L286VG90YWxUaW1lPg0KICA8bzpDcmVhdGVkPjIwMDAtMDYt MThUMTY6NDE6MDBaPC9vOkNyZWF0ZWQ+DQogIDxvOkxhc3RTYXZlZD4yMDAwLTA2LTE4VDE2OjQx OjAwWjwvbzpMYXN0U2F2ZWQ+DQogIDxvOlBhZ2VzPjE8L286UGFnZXM+DQogIDxvOldvcmRzPjE0 MTwvbzpXb3Jkcz4NCiAgPG86Q2hhcmFjdGVycz44MDQ8L286Q2hhcmFjdGVycz4NCiAgPG86Q29t cGFueT5JQk0gSGFpZmEgUmVzZWFyY2ggTGFib3JhdG9yeTwvbzpDb21wYW55Pg0KICA8bzpMaW5l cz42PC9vOkxpbmVzPg0KICA8bzpQYXJhZ3JhcGhzPjE8L286UGFyYWdyYXBocz4NCiAgPG86Q2hh cmFjdGVyc1dpdGhTcGFjZXM+OTg3PC9vOkNoYXJhY3RlcnNXaXRoU3BhY2VzPg0KICA8bzpWZXJz aW9uPjkuMzgyMTwvbzpWZXJzaW9uPg0KIDwvbzpEb2N1bWVudFByb3BlcnRpZXM+DQo8L3htbD48 IVtlbmRpZl0tLT4NCjxzdHlsZT4NCjwhLS0NCiAvKiBGb250IERlZmluaXRpb25zICovDQpAZm9u dC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkNvdXJpZXI7DQoJcGFub3NlLTE6MCAwIDAgMCAwIDAgMCAw IDAgMDsNCgltc28tZm9udC1hbHQ6IkNvdXJpZXIgTmV3IjsNCgltc28tZm9udC1jaGFyc2V0OjA7 DQoJbXNvLWdlbmVyaWMtZm9udC1mYW1pbHk6bW9kZXJuOw0KCW1zby1mb250LWZvcm1hdDpvdGhl cjsNCgltc28tZm9udC1waXRjaDpmaXhlZDsNCgltc28tZm9udC1zaWduYXR1cmU6MyAwIDAgMCAx IDA7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseTpXaW5nZGluZ3M7DQoJcGFub3NlLTE6NSAw IDAgMCAwIDAgMCAwIDAgMDsNCgltc28tZm9udC1jaGFyc2V0OjI7DQoJbXNvLWdlbmVyaWMtZm9u dC1mYW1pbHk6YXV0bzsNCgltc28tZm9udC1waXRjaDp2YXJpYWJsZTsNCgltc28tZm9udC1zaWdu YXR1cmU6MCAyNjg0MzU0NTYgMCAwIC0yMTQ3NDgzNjQ4IDA7fQ0KIC8qIFN0eWxlIERlZmluaXRp b25zICovDQpwLk1zb05vcm1hbCwgbGkuTXNvTm9ybWFsLCBkaXYuTXNvTm9ybWFsDQoJe21zby1z dHlsZS1wYXJlbnQ6IiI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJ bXNvLXBhZ2luYXRpb246d2lkb3ctb3JwaGFuOw0KCWZvbnQtc2l6ZToxMi4wcHQ7DQoJZm9udC1m YW1pbHk6IlRpbWVzIE5ldyBSb21hbiI7DQoJbXNvLWZhcmVhc3QtZm9udC1mYW1pbHk6IlRpbWVz IE5ldyBSb21hbiI7fQ0KQHBhZ2UgU2VjdGlvbjENCgl7c2l6ZTo1OTUuM3B0IDg0MS45cHQ7DQoJ bWFyZ2luOjcyLjBwdCA5MC4wcHQgNzIuMHB0IDkwLjBwdDsNCgltc28taGVhZGVyLW1hcmdpbjoz NS40cHQ7DQoJbXNvLWZvb3Rlci1tYXJnaW46MzUuNHB0Ow0KCW1zby1wYXBlci1zb3VyY2U6MDt9 DQpkaXYuU2VjdGlvbjENCgl7cGFnZTpTZWN0aW9uMTt9DQogLyogTGlzdCBEZWZpbml0aW9ucyAq Lw0KQGxpc3QgbDANCgl7bXNvLWxpc3QtaWQ6Mjk4NTM5MjYyOw0KCW1zby1saXN0LXR5cGU6aHli cmlkOw0KCW1zby1saXN0LXRlbXBsYXRlLWlkczotMjAzNzQ3ODU1MiA2NzY5ODY4OSA2NzY5ODY5 MSA2NzY5ODY5MyA2NzY5ODY4OSA2NzY5ODY5MSA2NzY5ODY5MyA2NzY5ODY4OSA2NzY5ODY5MSA2 NzY5ODY5Mzt9DQpAbGlzdCBsMDpsZXZlbDENCgl7bXNvLWxldmVsLW51bWJlci1mb3JtYXQ6YnVs bGV0Ow0KCW1zby1sZXZlbC10ZXh0OlxGMEI3Ow0KCW1zby1sZXZlbC10YWItc3RvcDozNi4wcHQ7 DQoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlvbjpsZWZ0Ow0KCXRleHQtaW5kZW50Oi0xOC4wcHQ7 DQoJZm9udC1mYW1pbHk6U3ltYm9sO30NCkBsaXN0IGwxDQoJe21zby1saXN0LWlkOjM1MTEwNTQz ODsNCgltc28tbGlzdC10eXBlOmh5YnJpZDsNCgltc28tbGlzdC10ZW1wbGF0ZS1pZHM6Mjc1OTIw ODA4IDY3Njk4Njg5IDY3Njk4NjkxIDY3Njk4NjkzIDY3Njk4Njg5IDY3Njk4NjkxIDY3Njk4Njkz IDY3Njk4Njg5IDY3Njk4NjkxIDY3Njk4NjkzO30NCkBsaXN0IGwxOmxldmVsMQ0KCXttc28tbGV2 ZWwtbnVtYmVyLWZvcm1hdDpidWxsZXQ7DQoJbXNvLWxldmVsLXRleHQ6XEYwQjc7DQoJbXNvLWxl dmVsLXRhYi1zdG9wOjM2LjBwdDsNCgltc28tbGV2ZWwtbnVtYmVyLXBvc2l0aW9uOmxlZnQ7DQoJ dGV4dC1pbmRlbnQ6LTE4LjBwdDsNCglmb250LWZhbWlseTpTeW1ib2w7fQ0KQGxpc3QgbDINCgl7 bXNvLWxpc3QtaWQ6ODU1MDc1NDkyOw0KCW1zby1saXN0LXR5cGU6aHlicmlkOw0KCW1zby1saXN0 LXRlbXBsYXRlLWlkczotNzQzNzg5NTMwIDY3Njk4Njg5IDY3Njk4NjkxIDY3Njk4NjkzIDY3Njk4 Njg5IDY3Njk4NjkxIDY3Njk4NjkzIDY3Njk4Njg5IDY3Njk4NjkxIDY3Njk4NjkzO30NCkBsaXN0 IGwyOmxldmVsMQ0KCXttc28tbGV2ZWwtbnVtYmVyLWZvcm1hdDpidWxsZXQ7DQoJbXNvLWxldmVs LXRleHQ6XEYwQjc7DQoJbXNvLWxldmVsLXRhYi1zdG9wOjM2LjBwdDsNCgltc28tbGV2ZWwtbnVt YmVyLXBvc2l0aW9uOmxlZnQ7DQoJdGV4dC1pbmRlbnQ6LTE4LjBwdDsNCglmb250LWZhbWlseTpT eW1ib2w7fQ0KQGxpc3QgbDMNCgl7bXNvLWxpc3QtaWQ6MTI2NzczNDEzNjsNCgltc28tbGlzdC10 eXBlOmh5YnJpZDsNCgltc28tbGlzdC10ZW1wbGF0ZS1pZHM6MjA0MTA4NTc4MiA2NzY5ODY4OSA2 NzY5ODY5MSA2NzY5ODY5MyA2NzY5ODY4OSA2NzY5ODY5MSA2NzY5ODY5MyA2NzY5ODY4OSA2NzY5 ODY5MSA2NzY5ODY5Mzt9DQpAbGlzdCBsMzpsZXZlbDENCgl7bXNvLWxldmVsLW51bWJlci1mb3Jt YXQ6YnVsbGV0Ow0KCW1zby1sZXZlbC10ZXh0OlxGMEI3Ow0KCW1zby1sZXZlbC10YWItc3RvcDoz Ni4wcHQ7DQoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlvbjpsZWZ0Ow0KCXRleHQtaW5kZW50Oi0x OC4wcHQ7DQoJZm9udC1mYW1pbHk6U3ltYm9sO30NCkBsaXN0IGw0DQoJe21zby1saXN0LWlkOjEz ODEwNTQyMzc7DQoJbXNvLWxpc3QtdHlwZTpoeWJyaWQ7DQoJbXNvLWxpc3QtdGVtcGxhdGUtaWRz OjcyNjQyOTIzNiA2NzY5ODY4OSA2NzY5ODY5MSA2NzY5ODY5MyA2NzY5ODY4OSA2NzY5ODY5MSA2 NzY5ODY5MyA2NzY5ODY4OSA2NzY5ODY5MSA2NzY5ODY5Mzt9DQpAbGlzdCBsNDpsZXZlbDENCgl7 bXNvLWxldmVsLW51bWJlci1mb3JtYXQ6YnVsbGV0Ow0KCW1zby1sZXZlbC10ZXh0OlxGMEI3Ow0K CW1zby1sZXZlbC10YWItc3RvcDozNi4wcHQ7DQoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlvbjps ZWZ0Ow0KCXRleHQtaW5kZW50Oi0xOC4wcHQ7DQoJZm9udC1mYW1pbHk6U3ltYm9sO30NCm9sDQoJ e21hcmdpbi1ib3R0b206MGNtO30NCnVsDQoJe21hcmdpbi1ib3R0b206MGNtO30NCi0tPg0KPC9z dHlsZT4NCjwvaGVhZD4NCg0KPGJvZHkgbGFuZz1FTi1VUyBzdHlsZT0ndGFiLWludGVydmFsOjM2 LjBwdCc+DQoNCjxkaXYgY2xhc3M9U2VjdGlvbjE+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHls ZT0nbGluZS1oZWlnaHQ6MTIuMHB0O21zby1sYXlvdXQtZ3JpZC1hbGlnbjpub25lOw0KdGV4dC1h dXRvc3BhY2U6bm9uZSc+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZTo4LjBwdDtmb250LWZhbWlseTpD b3VyaWVyOw0KbXNvLWJpZGktZm9udC1mYW1pbHk6Q291cmllcjtjb2xvcjpibGFjayc+SGFpZmEg liBEZXNpZ24gVGVhbSBNZWV0aW5nIJYgRFJBRlQNCkFnZW5kYTxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdsaW5lLWhlaWdodDoxMi4wcHQ7bXNvLWxh eW91dC1ncmlkLWFsaWduOm5vbmU7DQp0ZXh0LWF1dG9zcGFjZTpub25lJz48c3BhbiBzdHlsZT0n Zm9udC1zaXplOjguMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7DQptc28tYmlkaS1mb250LWZhbWls eTpDb3VyaWVyO2NvbG9yOmJsYWNrJz5KdW5lIDE5LCAyMDAwPG86cD48L286cD48L3NwYW4+PC9w Pg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J2xpbmUtaGVpZ2h0OjEyLjBwdDttc28tbGF5 b3V0LWdyaWQtYWxpZ246bm9uZTsNCnRleHQtYXV0b3NwYWNlOm5vbmUnPjxzcGFuIHN0eWxlPSdm b250LXNpemU6OC4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjsNCm1zby1iaWRpLWZvbnQtZmFtaWx5 OkNvdXJpZXI7Y29sb3I6YmxhY2snPjwhW2lmICFzdXBwb3J0RW1wdHlQYXJhc10+Jm5ic3A7PCFb ZW5kaWZdPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxl PSdsaW5lLWhlaWdodDoxMi4wcHQ7bXNvLWxheW91dC1ncmlkLWFsaWduOm5vbmU7DQp0ZXh0LWF1 dG9zcGFjZTpub25lJz48c3BhbiBzdHlsZT0nZm9udC1zaXplOjguMHB0O2ZvbnQtZmFtaWx5OkNv dXJpZXI7DQptc28tYmlkaS1mb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJsYWNrJz5UaGVtZXM6 PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J2xpbmUt aGVpZ2h0OjEyLjBwdDttc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTsNCnRleHQtYXV0b3NwYWNl Om5vbmUnPjxzcGFuIHN0eWxlPSdmb250LXNpemU6OC4wcHQ7Zm9udC1mYW1pbHk6Q291cmllcjsN Cm1zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2snPjwhW2lmICFzdXBwb3J0 RW1wdHlQYXJhc10+Jm5ic3A7PCFbZW5kaWZdPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAg Y2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdsaW5lLWhlaWdodDoxMi4wcHQ7bXNvLWxheW91dC1ncmlk LWFsaWduOm5vbmU7DQp0ZXh0LWF1dG9zcGFjZTpub25lJz48c3BhbiBzdHlsZT0nZm9udC1zaXpl OjguMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7DQptc28tYmlkaS1mb250LWZhbWlseTpDb3VyaWVy O2NvbG9yOmJsYWNrJz5JRDAgliBJRCBPdmVydmlldzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0K PHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdsaW5lLWhlaWdodDoxMi4wcHQ7bXNvLWxheW91dC1n cmlkLWFsaWduOm5vbmU7DQp0ZXh0LWF1dG9zcGFjZTpub25lJz48c3BhbiBzdHlsZT0nZm9udC1z aXplOjguMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7DQptc28tYmlkaS1mb250LWZhbWlseTpDb3Vy aWVyO2NvbG9yOmJsYWNrJz5JRDEgliBJRCBTZWN1cml0eTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N Cg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdsaW5lLWhlaWdodDoxMi4wcHQ7bXNvLWxheW91 dC1ncmlkLWFsaWduOm5vbmU7DQp0ZXh0LWF1dG9zcGFjZTpub25lJz48c3BhbiBzdHlsZT0nZm9u dC1zaXplOjguMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7DQptc28tYmlkaS1mb250LWZhbWlseTpD b3VyaWVyO2NvbG9yOmJsYWNrJz5JRDIgliBJRCBFcnJvciBSZXBvcnRpbmcgJmFtcDsNClJlY292 ZXJ5PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J2xp bmUtaGVpZ2h0OjEyLjBwdDttc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTsNCnRleHQtYXV0b3Nw YWNlOm5vbmUnPjxzcGFuIHN0eWxlPSdmb250LXNpemU6OC4wcHQ7Zm9udC1mYW1pbHk6Q291cmll cjsNCm1zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2snPklEMyCWIElEIE1l c3NhZ2UgRm9ybWF0cyAocmV2aWV3DQpiYXNlZCBvbiBlZmZlY3RpdmVuZXNzICZhbXA7IGVhc2Ug b2YgdHJhbnNsaXRlcmF0aW9uKTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNv Tm9ybWFsIHN0eWxlPSdsaW5lLWhlaWdodDoxMi4wcHQ7bXNvLWxheW91dC1ncmlkLWFsaWduOm5v bmU7DQp0ZXh0LWF1dG9zcGFjZTpub25lJz48c3BhbiBzdHlsZT0nZm9udC1zaXplOjguMHB0O2Zv bnQtZmFtaWx5OkNvdXJpZXI7DQptc28tYmlkaS1mb250LWZhbWlseTpDb3VyaWVyO2NvbG9yOmJs YWNrJz5JRDQgliBJRCBNaW5pbWFsIE5hbWluZyBSZXF1aXJlbWVudHM8bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbGluZS1oZWlnaHQ6MTIuMHB0O21z by1sYXlvdXQtZ3JpZC1hbGlnbjpub25lOw0KdGV4dC1hdXRvc3BhY2U6bm9uZSc+PHNwYW4gc3R5 bGU9J2ZvbnQtc2l6ZTo4LjBwdDtmb250LWZhbWlseTpDb3VyaWVyOw0KbXNvLWJpZGktZm9udC1m YW1pbHk6Q291cmllcjtjb2xvcjpibGFjayc+UkQwIJYgUmVxdWlyZW1lbnRzIGFuZCBBcHBsaWNh YmlsaXR5PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9 J2xpbmUtaGVpZ2h0OjEyLjBwdDttc28tbGF5b3V0LWdyaWQtYWxpZ246bm9uZTsNCnRleHQtYXV0 b3NwYWNlOm5vbmUnPjxzcGFuIHN0eWxlPSdmb250LXNpemU6OC4wcHQ7Zm9udC1mYW1pbHk6Q291 cmllcjsNCm1zby1iaWRpLWZvbnQtZmFtaWx5OkNvdXJpZXI7Y29sb3I6YmxhY2snPjwhW2lmICFz dXBwb3J0RW1wdHlQYXJhc10+Jm5ic3A7PCFbZW5kaWZdPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N Cg0KPHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdsaW5lLWhlaWdodDoxMi4wcHQ7bXNvLWxheW91 dC1ncmlkLWFsaWduOm5vbmU7DQp0ZXh0LWF1dG9zcGFjZTpub25lJz48c3BhbiBzdHlsZT0nZm9u dC1zaXplOjguMHB0O2ZvbnQtZmFtaWx5OkNvdXJpZXI7DQptc28tYmlkaS1mb250LWZhbWlseTpD b3VyaWVyO2NvbG9yOmJsYWNrJz48IVtpZiAhc3VwcG9ydEVtcHR5UGFyYXNdPiZuYnNwOzwhW2Vu ZGlmXT48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbD5Qcm9wb3Nl ZCBTY2hlZHVsZTwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsPjwhW2lmICFzdXBwb3J0RW1wdHlQ YXJhc10+Jm5ic3A7PCFbZW5kaWZdPjxvOnA+PC9vOnA+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3Jt YWw+SnVuZSAxOTwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsPjwhW2lmICFzdXBwb3J0RW1wdHlQ YXJhc10+Jm5ic3A7PCFbZW5kaWZdPjxvOnA+PC9vOnA+PC9wPg0KDQo8cCBjbGFzcz1Nc29Ob3Jt YWw+PCFbaWYgIXN1cHBvcnRFbXB0eVBhcmFzXT4mbmJzcDs8IVtlbmRpZl0+PG86cD48L286cD48 L3A+DQoNCjx1bCBzdHlsZT0nbWFyZ2luLXRvcDowY20nIHR5cGU9ZGlzYz4NCiA8bGkgY2xhc3M9 TXNvTm9ybWFsIHN0eWxlPSdtc28tbGlzdDpsMSBsZXZlbDEgbGZvMTt0YWItc3RvcHM6bGlzdCAz Ni4wcHQnPjxzcGFuDQogICAgIHN0eWxlPSdmb250LXNpemU6MTAuMHB0Jz45OjMwLTEwOjMwIEdl dCB0b2dldGhlciAmYW1wOyBkaXNjdXNzIGFnZW5kYTwvc3Bhbj48L2xpPg0KIDxsaSBjbGFzcz1N c29Ob3JtYWwgc3R5bGU9J21zby1saXN0OmwxIGxldmVsMSBsZm8xO3RhYi1zdG9wczpsaXN0IDM2 LjBwdCc+PHNwYW4NCiAgICAgc3R5bGU9J2ZvbnQtc2l6ZToxMC4wcHQnPjEwOjMwIC0gMTE6MzAg RHJhZnQgb3ZlcnZpZXcgJmFtcDsgbGF1bmRyeSBsaXN0PC9zcGFuPjwvbGk+DQogPGxpIGNsYXNz PU1zb05vcm1hbCBzdHlsZT0nbXNvLWxpc3Q6bDEgbGV2ZWwxIGxmbzE7dGFiLXN0b3BzOmxpc3Qg MzYuMHB0Jz48c3Bhbg0KICAgICBzdHlsZT0nZm9udC1zaXplOjEwLjBwdCc+MTE6MzAgLSAxMjoz MCBSZXF1aXJlbWVudHMgliBwYXNzIDE8L3NwYW4+PC9saT4NCiA8bGkgY2xhc3M9TXNvTm9ybWFs IHN0eWxlPSdtc28tbGlzdDpsMSBsZXZlbDEgbGZvMTt0YWItc3RvcHM6bGlzdCAzNi4wcHQnPjxz cGFuDQogICAgIHN0eWxlPSdmb250LXNpemU6MTAuMHB0Jz4xMjozMCAtIDEzOjMwIEx1bmNoPC9z cGFuPjwvbGk+DQogPGxpIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbXNvLWxpc3Q6bDEgbGV2ZWwx IGxmbzE7dGFiLXN0b3BzOmxpc3QgMzYuMHB0Jz48c3Bhbg0KICAgICBzdHlsZT0nZm9udC1zaXpl OjEwLjBwdCc+MTM6MzAgLSAxNTozMCBTZWN1cml0eSCWIHBhc3MgMTwvc3Bhbj48L2xpPg0KIDxs aSBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1saXN0OmwxIGxldmVsMSBsZm8xO3RhYi1zdG9w czpsaXN0IDM2LjBwdCc+PHNwYW4NCiAgICAgc3R5bGU9J2ZvbnQtc2l6ZToxMC4wcHQnPjE1OjMw IC0gMTc6MzAgTmFtaW5nIGZpcnN0ICZhbXA7IGxhc3QgKGF0dGVtcHQpPC9zcGFuPjwvbGk+DQo8 L3VsPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWw+PCFbaWYgIXN1cHBvcnRFbXB0eVBhcmFzXT4mbmJz cDs8IVtlbmRpZl0+PG86cD48L286cD48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbD48IVtpZiAh c3VwcG9ydEVtcHR5UGFyYXNdPiZuYnNwOzwhW2VuZGlmXT48bzpwPjwvbzpwPjwvcD4NCg0KPHAg Y2xhc3M9TXNvTm9ybWFsPkp1bmUgMjA8L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbD48IVtpZiAh c3VwcG9ydEVtcHR5UGFyYXNdPiZuYnNwOzwhW2VuZGlmXT48bzpwPjwvbzpwPjwvcD4NCg0KPHVs IHN0eWxlPSdtYXJnaW4tdG9wOjBjbScgdHlwZT1kaXNjPg0KIDxsaSBjbGFzcz1Nc29Ob3JtYWwg c3R5bGU9J21zby1saXN0OmwzIGxldmVsMSBsZm8yO3RhYi1zdG9wczpsaXN0IDM2LjBwdCc+PHNw YW4NCiAgICAgc3R5bGU9J2ZvbnQtc2l6ZToxMC4wcHQnPjk6MzAgLSAxMDozMCBSZXF1aXJlbWVu dHMgliBwYXNzIDIgYW5kIGZpbmFsPG86cD48L286cD48L3NwYW4+PC9saT4NCiA8bGkgY2xhc3M9 TXNvTm9ybWFsIHN0eWxlPSdtc28tbGlzdDpsMyBsZXZlbDEgbGZvMjt0YWItc3RvcHM6bGlzdCAz Ni4wcHQnPjxzcGFuDQogICAgIHN0eWxlPSdmb250LXNpemU6MTAuMHB0Jz4xMDozMCAtIDEyOjMw IFNlY3VyaXR5IJYgcGFzcyAyPG86cD48L286cD48L3NwYW4+PC9saT4NCiA8bGkgY2xhc3M9TXNv Tm9ybWFsIHN0eWxlPSdtc28tbGlzdDpsMyBsZXZlbDEgbGZvMjt0YWItc3RvcHM6bGlzdCAzNi4w cHQnPjxzcGFuDQogICAgIHN0eWxlPSdmb250LXNpemU6MTAuMHB0Jz4xMjozMCAtIDEzOjMwIEx1 bmNoPG86cD48L286cD48L3NwYW4+PC9saT4NCiA8bGkgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdt c28tbGlzdDpsMyBsZXZlbDEgbGZvMjt0YWItc3RvcHM6bGlzdCAzNi4wcHQnPjxzcGFuDQogICAg IHN0eWxlPSdmb250LXNpemU6MTAuMHB0Jz4xMzozMCAtIDE2OjMwIFNlY3VyaXR5IJYgYXR0ZW1w dCB0byBjbG9zZTxvOnA+PC9vOnA+PC9zcGFuPjwvbGk+DQogPGxpIGNsYXNzPU1zb05vcm1hbCBz dHlsZT0nbXNvLWxpc3Q6bDMgbGV2ZWwxIGxmbzI7dGFiLXN0b3BzOmxpc3QgMzYuMHB0Jz48c3Bh bg0KICAgICBzdHlsZT0nZm9udC1zaXplOjEwLjBwdCc+MTY6MzAgliAxNzozMCBNaXNjLiBMYXVu ZHJ5IExpc3QgaXRlbXM8bzpwPjwvbzpwPjwvc3Bhbj48L2xpPg0KPC91bD4NCg0KPHAgY2xhc3M9 TXNvTm9ybWFsPjxzcGFuIHN0eWxlPSdmb250LXNpemU6MTAuMHB0Jz48IVtpZiAhc3VwcG9ydEVt cHR5UGFyYXNdPiZuYnNwOzwhW2VuZGlmXT48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQoNCjxwIGNs YXNzPU1zb05vcm1hbD48c3BhbiBzdHlsZT0nZm9udC1zaXplOjEwLjBwdCc+PCFbaWYgIXN1cHBv cnRFbXB0eVBhcmFzXT4mbmJzcDs8IVtlbmRpZl0+PG86cD48L286cD48L3NwYW4+PC9wPg0KDQo8 cCBjbGFzcz1Nc29Ob3JtYWw+SnVuZSAyMTwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFsPjwhW2lm ICFzdXBwb3J0RW1wdHlQYXJhc10+Jm5ic3A7PCFbZW5kaWZdPjxvOnA+PC9vOnA+PC9wPg0KDQo8 cCBjbGFzcz1Nc29Ob3JtYWw+PCFbaWYgIXN1cHBvcnRFbXB0eVBhcmFzXT4mbmJzcDs8IVtlbmRp Zl0+PG86cD48L286cD48L3A+DQoNCjx1bCBzdHlsZT0nbWFyZ2luLXRvcDowY20nIHR5cGU9ZGlz Yz4NCiA8bGkgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGlzdDpsNCBsZXZlbDEgbGZvNDt0 YWItc3RvcHM6bGlzdCAzNi4wcHQnPjxzcGFuDQogICAgIHN0eWxlPSdmb250LXNpemU6MTAuMHB0 Jz45OjMwIC0gMTE6MzAgRXJyb3IgcmVjb3ZlcnkgliBtb2RlbCAmYW1wOw0KICAgICBkaXNjdXNz aW9uPG86cD48L286cD48L3NwYW4+PC9saT4NCiA8bGkgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdt c28tbGlzdDpsNCBsZXZlbDEgbGZvNDt0YWItc3RvcHM6bGlzdCAzNi4wcHQnPjxzcGFuDQogICAg IHN0eWxlPSdmb250LXNpemU6MTAuMHB0Jz4xMTozMCAtIDEyOjMwIEZvcm1hdCByZXZpZXc8bzpw PjwvbzpwPjwvc3Bhbj48L2xpPg0KIDxsaSBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1saXN0 Omw0IGxldmVsMSBsZm80O3RhYi1zdG9wczpsaXN0IDM2LjBwdCc+PHNwYW4NCiAgICAgc3R5bGU9 J2ZvbnQtc2l6ZToxMC4wcHQnPjEyOjMwIC0gMTM6MzAgTHVuY2g8bzpwPjwvbzpwPjwvc3Bhbj48 L2xpPg0KIDxsaSBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21zby1saXN0Omw0IGxldmVsMSBsZm80 O3RhYi1zdG9wczpsaXN0IDM2LjBwdCc+PHNwYW4NCiAgICAgc3R5bGU9J2ZvbnQtc2l6ZToxMC4w cHQnPjEzOjMwIC0gMTc6MzAgRm9ybWF0IHJldmlldzxvOnA+PC9vOnA+PC9zcGFuPjwvbGk+DQo8 L3VsPg0KDQo8cCBjbGFzcz1Nc29Ob3JtYWwgc3R5bGU9J21hcmdpbi1sZWZ0OjE4LjBwdCc+PHNw YW4gc3R5bGU9J2ZvbnQtc2l6ZToxMC4wcHQnPjwhW2lmICFzdXBwb3J0RW1wdHlQYXJhc10+Jm5i c3A7PCFbZW5kaWZdPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCg0KPHAgY2xhc3M9TXNvTm9ybWFs IHN0eWxlPSdtYXJnaW4tbGVmdDoxOC4wcHQnPjxzcGFuIHN0eWxlPSdmb250LXNpemU6MTAuMHB0 Jz48IVtpZiAhc3VwcG9ydEVtcHR5UGFyYXNdPiZuYnNwOzwhW2VuZGlmXT48bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQoNCjxwIGNsYXNzPU1zb05vcm1hbD5KdW5lIDIyIChzcGlsbG92ZXIpPC9wPg0K DQo8cCBjbGFzcz1Nc29Ob3JtYWw+PCFbaWYgIXN1cHBvcnRFbXB0eVBhcmFzXT4mbmJzcDs8IVtl bmRpZl0+PG86cD48L286cD48L3A+DQoNCjx1bCBzdHlsZT0nbWFyZ2luLXRvcDowY20nIHR5cGU9 ZGlzYz4NCiA8bGkgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGlzdDpsMCBsZXZlbDEgbGZv NTt0YWItc3RvcHM6bGlzdCAzNi4wcHQnPjxzcGFuDQogICAgIHN0eWxlPSdmb250LXNpemU6MTAu MHB0Jz45OjMwIJYgMTE6MzAgRXJyb3IgUmVjb3ZlcnkgJmFtcDsgUmVwb3J0aW5nPG86cD48L286 cD48L3NwYW4+PC9saT4NCiA8bGkgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtc28tbGlzdDpsMCBs ZXZlbDEgbGZvNTt0YWItc3RvcHM6bGlzdCAzNi4wcHQnPjxzcGFuDQogICAgIHN0eWxlPSdmb250 LXNpemU6MTAuMHB0Jz4xMTMwICZhbXA7IE9OIFNwaWxsb3ZlciBJdGVtcyAoJmFtcDsgbGF1bmRy eQ0KICAgICBMaXN0KTxvOnA+PC9vOnA+PC9zcGFuPjwvbGk+DQo8L3VsPg0KDQo8L2Rpdj4NCg0K PC9ib2R5Pg0KDQo8L2h0bWw+DQo= --0__=MYoG3PRjZvobQjksSqYM2uAa4YW8KRAV83QOlrT8IFR9OJveCwESqzx0-- From XXX Mon Jun 19 00:18:33 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Sun Jun 18 20:16:16 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id UAA05942 for ; Sun, 18 Jun 2000 20:16:16 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA14155 for ips-outgoing; Sun, 18 Jun 2000 20:14:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ophidian.com (ophidian.com [209.136.122.245] (may be forged)) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA14150 for ; Sun, 18 Jun 2000 20:14:19 -0400 (EDT) Received: from boa [209.136.121.227] by ophidian.com (SMTPD32-5.05) id A89E1F601E4; Sun, 18 Jun 2000 18:26:06 -0600 Message-ID: <002b01bfd983$b4b66a20$0302010a@boa> From: "Edward A. Gardner" To: , "List SCSI-over-TCP" , Subject: Re: a tentative agenda with some more details Date: Sun, 18 Jun 2000 18:17:07 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The following item needs to be added to the formal agenda, simply because it is too important to risk it being forgotten. You need to draw up a formal list of action items and the planned schedule from now through the Pittsburg IETF meeting, and perhaps for the month following that meeting. Writing assignments with names and completion dates, any/all planned meetings or phone conferences or planned reflector discussions, plans for the IETF week, any plans for the weeks that follow assuming you get approved that week. This needs to published asap, without waiting for complete minutes of this week's meeting. It affects things like travel planning (e.g. whether to come to Pittsburg) that cannot be deferred. Edward A. Gardner eag@ophidian.com Ophidian Designs 719 593-8866 voice 1262 Hofstead Terrace 719 593-8989 fax Colorado Springs, CO 80907 719 210-7200 cell - -----Original Message----- From: julian_satran@il.ibm.com To: List SCSI-over-TCP ; ips@ece.cmu.edu Date: Sunday, June 18, 2000 11:34 AM Subject: a tentative agenda with some more details > > >(See attached file: Haifa-Session-AgendaDraft.htm) > -----BEGIN PGP SIGNATURE----- Version: PGP Personal Privacy 6.0.2 iQA/AwUBOU1mg5sv4MZNZL9GEQJ8nwCgxny3GDm43qGy4GZQugZ1mzbRcrYAoPjj gVXJJpaKnUCTQnUCYFi6nCoc =bILz -----END PGP SIGNATURE----- From XXX Mon Jun 19 12:11:23 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Mon Jun 19 01:23:45 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id BAA05631 for ; Mon, 19 Jun 2000 01:23:45 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA18947 for ips-outgoing; Mon, 19 Jun 2000 01:21:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA18943 for ; Mon, 19 Jun 2000 01:21:51 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA67846; Mon, 19 Jun 2000 07:17:28 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA262048; Mon, 19 Jun 2000 07:17:28 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256903.001D0F37 ; Mon, 19 Jun 2000 07:17:24 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu, List SCSI-over-TCP Message-ID: Date: Mon, 19 Jun 2000 08:16:38 +0300 Subject: Re: a tentative agenda with some more details Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=EZs3P83zPnGvrx8XtK1CNAIoJqbFxwSMaYJoYBK9HoVqWe1I2cKjZ6pb" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=EZs3P83zPnGvrx8XtK1CNAIoJqbFxwSMaYJoYBK9HoVqWe1I2cKjZ6pb Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-transfer-encoding: quoted-printable Sorry for the inconvenience - and let's keep to the IETF rules. Here is a tentative agenda in text only. Please consider although the thoughtfull note from Ed Gardner. Regards, Julo Haifa ? Design Team Meeting ? DRAFT Agenda June 19, 2000 Themes: ID0 ? ID Overview ID1 ? ID Security ID2 ? ID Error Reporting & Recovery ID3 ? ID Message Formats (review based on effectiveness & ease of transliteration) ID4 ? ID Minimal Naming Requirements RD0 ? Requirements and Applicability Proposed Schedule June 19 =B7 9:30-10:30 Get together & discuss agenda =B7 10:30 - 11:30 Draft overview & laundry list =B7 11:30 - 12:30 Requirements ? pass 1 =B7 12:30 - 13:30 Lunch =B7 13:30 - 15:30 Security ? pass 1 =B7 15:30 - 17:30 Naming first & last (attempt) June 20 =B7 9:30 - 10:30 Requirements ? pass 2 and final =B7 10:30 - 12:30 Security ? pass 2 =B7 12:30 - 13:30 Lunch =B7 13:30 - 16:30 Security ? attempt to close =B7 16:30 ? 17:30 Misc. Laundry List items June 21 =B7 9:30 - 11:30 Error recovery ? model & discussion =B7 11:30 - 12:30 Format review =B7 12:30 - 13:30 Lunch =B7 13:30 - 17:30 Format review June 22 (spillover) =B7 9:30 ? 11:30 Error Recovery & Reporting =B7 1130 & ON Spillover Items (& laundry List) Scott Bradner on 18/06/2000 19:51:12 Please respond to Scott Bradner To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: a tentative agenda with some more details = --0__=EZs3P83zPnGvrx8XtK1CNAIoJqbFxwSMaYJoYBK9HoVqWe1I2cKjZ6pb Content-type: text/plain; charset=us-ascii Content-Disposition: inline please use plain text on IETF email lists Scott --0__=EZs3P83zPnGvrx8XtK1CNAIoJqbFxwSMaYJoYBK9HoVqWe1I2cKjZ6pb-- From XXX Mon Jun 19 17:23:05 2000 Forwarded: Mon, 19 Jun 2000 13:23:05 -0400 Forwarded: "nitin " Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Mon Jun 19 12:21:50 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id MAA27999 for ; Mon, 19 Jun 2000 12:21:50 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA10641 for ips-outgoing; Mon, 19 Jun 2000 12:21:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA10635 for ; Mon, 19 Jun 2000 12:21:00 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA121588; Mon, 19 Jun 2000 18:17:47 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA72492; Mon, 19 Jun 2000 18:17:39 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256903.0059807A ; Mon, 19 Jun 2000 18:17:35 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu, scsi-tcp@external.cisco.com Message-ID: Date: Mon, 19 Jun 2000 19:17:31 +0300 Subject: minutes of iSCSI meeting 19 June 2000 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk iSCSI design team meeting Monday, 19 June 2000 Haifa, Israel Attendees: AA Alaan Azagury (IBM) JD John Dowdy (IBM) SDG Steve De Grate (NuSpeed) RH Randy Haagens (HP) GH Gabi Hecht (Gadzooks) JH John Hufferd (IBM) SL Steve Legg (IBM) JM John Matze (Veritas) KM Kalman Meth (IBM) NN Nelson Nahum (Storage) LDO Luciano Dalle Ore (Quantumm) CS Costa Sapuntzakis (Cisco) JS Julian Satran (IBM) MS Mark Shifardt (NuSpeed) MT Meir Toledano (IBM) MW Matt Wakeley (Agilent) EZ Efri Zeidner (SanGate) Disclaimer: Rough paraphrase of some of what was said. Some comments may be incorrectly attributed. Comments on proposed agenda: JS: Ed Gardner wrote to add action items and produce schedule for next month. Paul missed his connection, so we'll push off discussion of security until tomorrow. We'll discuss Error Recovery today instead. JH: When will we discuss Discovery? JS: After the Pittsburgh IETF meeting. JS: Can Luciano please provide us with all that is written on Security. LDO: Will send out what he has. Overview of Requirements draft: RH: We first have an Applicablity Statement. Discussion on Applicability Section, paragraph by paragraph: JD: Is iSCSI a "mapping" or an "encapsulation?" JS: It is a mapping. SAM defines an RPC model. It is somewhat abstract. It is not simply a command that can be unwrapped and delivered. JM: SCSI is sector based rather than block based. Applicability statement uses the term "block." Are we abstracting that out? Also there is also no mention of tapes and other devices in the applicability statement. RH: Yes, tapes and other devices semantics were meant to be incuded. JH: Tapes are relevant. We don't want to assume it is a controller. Can have remote tape. RH: One of the things wrong with FC is that they are totally disk drive oriented. If we keep in mind the model of conneting to large SCSI controllers, we won't tie ourselves into a too narrow applicability. MW: Whatever can be done over SCSI, we want to do over iSCSI. It doesn't matter if you use the term "block" or "sector." RH: We'll adjust the language. CS: Applicability section should be aimed at people who are not convinced of the advantages of iSCSI. For example, people who cannot imagine placing storage directly on the network. CS: We should also explain why we choose SCSI for accessing the devices. Why not some other block storage protocol? JS: Because it is ubiquitous. And this is already said in the Applicability statement. CS: In the IETF, anybody can get up and say they don't like it. We have to justify it. We need to simply add a paragraph that there is a large installed base that uses SCSI and we want to leverage this base. This will defuse the argument against iSCSI. JS: Also SCSI is a living protocol. LDO: Also mention the timeframe since customers are already asking for it, and we can't spend the time to invent new protocols. RH: The first paragraph of the Applicability statement already hints to these things. JH: Let's add another sentence (at xxx location) that explicitly says what Costa raised to defelect the objections. JS: In applications section, can add clusters (in addition to consolidatin and pooling). RH: Do we have to include desktop? Will iSCSI take over IDE disk interface? Several: No. EZ: Yes, we do. This is what will take over for the local bus. Similar to the Infiniband idea. JH: Isn't this included in "Local storage access?" CS: Should include in applications section: shared DVD players, CD burners, etc. JH: Scanners? JS: We should then also add something about QoS. JH: Can we simply put a bullet mentioning these things in the Applicability statement, and then not discuss it further. RH: We are puttng SCSI over xxxx. We should therefore support all that SCSI supports. We are not out to support everything. Just once we support SCSI, we should aim to support all that is supported by SCSI. We'll add some language that this protocol aims to support the various SCSI command sets. How successful we'll be depends on how well layered things are. JS: Under topology, simply reference LAN. Delete reference to Ethernet. We aren't limiting ourselves to any particular technology. LdO: Could also add storage over general internet using encryption. JH: Isn't that then a VPN? LdO: The IETF is about general connectivity over the internet. So this is an importatn point. RH: We'll adjust language to "Private and Public networks .." CS: TCP adaptive retransmission is not limited to local area. RH: The point is that even in the LAN, there are advantages of TCP for error recovery over others (like FC). JS: We should state that explicitly. JH: This might poke the FC people in the eye. Do we want to say this? RH: The way it is written, the point is made without poking in the eye. CS: With an Ethernet switch, there can be congestion even in a LAN. So TCP is advantageous there also. LdO: We should say that we want something that works and get it going fast. We therefore have to use what we have today: SCSI and TCP. This will defelct most of the dissenters. "The full realization ..." CS: Are we saying that this can't be done in software. JH: The "full" realization..." Without hardware support, iSCSI will never get into servers. JM&KM: Let's say "While iSCSI can be implemented totally in software, the FULL realization will involve ....." What will go on these new NICs? Discussion. A key goal is to not require modifications to existing protocols. AA: iSCSI also enables device sharing. Won't this have an affect on T10 and existing SCSI protocols, since we now enable a totally new application of their protocols? Shouldn't we same something here about the possible affect on these protocols? RH: We'll add a few sentences about the possible sresses on these protocols as iSCSI develops. JD: Such stresses already exist from video and other things that affect evolving of TCP. RH: We won't add requirements to these protocols, but we might push them to some new features. Paragraph on security: separate networks for storage traffic. RH: Perhaps can add a firewall to allow only the storage traffic through to ensure the security. JS: We have to address on our own the security needs of iSCSI (as required by IETF). Enterprise LAN ----------------------------------- | ----- Storage Management Firewall | ----------------------------------- Storage LAN Must ensure that IP packets cannot be routed to the Storage LAN. Routing will have to be turned off to disable any packets from getting to the Storage LAN except through the Storage Management Firewall. JM: FC is unroutable and therefore get the security. For IP we'll have to do something to prevent routing. RH: In the requirements sections, there are contributions from others included. They will be noted in the references. Overview of iSCSI draft: JS: initiator, target, TCP connections, session, affinity to a single connection, why several connections per session, evolution of proposals for multiple channels, some special iSCSI messages (Login, Ping, asynchronous event, task management, text). JH: How can an initiator figure out how many connections to use? CS: This may be an implementatin issue. We just have to provide the infrastructure to do it. JH: Every scenario raised requires a simulation to determine what is optimal. Is there a better way to determine an almost-optimal number of connections? Clarification of multiple tcp connections per single iSCSI session. Can have multiple active tasks per iSCSI session. A separate iSCSI session defines a separate initiator. Picture on board. x x x active tasks \ | / \ | / \ | / \ | / ------------------------------ iSCSI layer x iSCSI session / | \ / | \ | | | | | | tcp connection group | | | same or different IP address | | | \ | / \ | / x iSCSI session ------------------------------ iSCSI layer / | \ / | \ | | | device servers We need a more full discussion of iSCSI sessions in the document. JM: CDBs can have tacked on to them some vendor specific data (parameters), which then messes up our assumption of fixed sized headers. MW: Should not have any data sent in command phase. CS: In parallel SCSI, the parameters get sent in the data phase. MW: Therefore, the parameters shoud be sent only in an iSCSI data phase and not in a command phase. We now have 2 ways of sending parameters: either tacked on to a command or as data. This may cause confusion. What is the maximum length of CDB? Do we limit it artificially? We don't want to parse the CDB itself to determine how long it is. CS&RH: If we get rid of the parameters and have only a CDB, then we are OK. We then insist that the parameters get sent in a data phase. But then we have to perform another read operation to get the parameters. MW: What about WRITE without RTT. It would be nice to have the data appended to the command. JS: The Length field can be split. JM: Have an offset field to specify where the data begins. MW: Need a version number. Several: Put it only in Login. No need to have it in each packet. Lunch RH: Discussion on RTT. Don't want more than one round-trip delay. How long is max SCSI CDB? What is the "right" way to communicate command parameters? If data will follow command without RTT, can we include it with the command packet? It would be nice if we could avoid requiring a non-RTT WRITE to have to go in a separate command and data packet, since this would cause complications on the receiving end to connect the packets back together. JM: Add another length field in the header to state where the data starts. Further discussion. How many fields must we delineate? We'll come back to this tomorrow. Seems to be concensus that there are 3 fields: iSCSI header, variable length CDB (including CDB extension), data. (1) D | C | H -----> <----- H | S (2) D | H C | H -----> <----- H | S (3) C | H -----> <----- H | RTT D | H -----> <----- H | S RH: We want to enable (1). (2) causes problems for the target to implement since it has to match up Initiator tags between command and data, with other commands possibly having inerleaved. MW: Either send all data with header or send all data in separate iSCSI Data packets. Have a bit to indicate that we have immediate data. Discussion of fields in i SCSI Command packet. Picture on board of packet header. Has variable length CDB possibly extending beyond byte 40, followed by immediate data. Also have an "I" bit/flag to indicate that we have immediate data. Have the other fields currently specified in SCSI command header: "Length" field at byte 4, "Expected" field at byte 20, CDB begins at byte 24. Costa's proposal for specifying legths. If (I == 1) { /* immediate bit is set */ length(CDB) = Length - Expected + 16 length(immediate data) = Expected } if (I == 0) { /* no immediate data */ length(CDB) = Length - 24 length(immediate data) = 0; } The meaning of Length and Expected are essentially unchanged from what is currently written in the draft. Should we also send the CDB length explicitly in the header? What must be in a header? CS: (1) must contain all necesary information. (2) Should allow simple implementation. (3) Should be as short as possible. There are tradeoffs between these. JS: We also don't want to have multiple fields that may conflict with one another, thereby requiring consistency checks. RH: There are also some symmetry considerations to have consistent headers. We'll come back to all this tomorrow or Wednesday. Discussion of error recovery: JS: We should not attempt to re-do a failed SCSI command. We should report that the command may have started and we should report to the best of our ability what happened and what is the current state. What should we do if an iSCSI connection breaks? RH: We should differentiate between what happens at the SCSI layer and what happens at the iSCSI layer. Let SCSI do its own recovery. Let's concentrate on being a good transport. i.e. What do we do when an iSCSI connection fails? RH: One possibiliy (1) is to simply let everything die by timeout, the upper layer then forces session cancellation from above and cleanup, and then create new sessions. This removes from iSCSI almost all responsibility. (We would have to add a means to cancel an existing session.) JH: Why do we have to blow away the entire session? Can't we just deal with the commands on the broken connection. KM&CS: Commands are sequenced. So a failure on one connection will block the execution of commands on another connection, thereby causing backup on the entire session. KM: This is also dependent on whether sequencing is across the entire session (RH's view) or is per LUN (JS's view). LdO: Even more basically, if we have a single connection for the session, and the connection fails, do we want to try to recover? MT: Instead of simply letting the session hang, we can detect that the session is broken, and we can let the upper layer know. It can then cancel a task, etc and try to start recovery. This is possiblility (2): hang and notify. Another possibility (3) iSCSI recovers from TCP errors. The session stays alive as long as one connection of the session still exists. How do we recover from a command on the failed connection? Most extreme possibility (4) iSCSI session will stay up no matter what (by some magic). Discussion. Arguments. What do we want to do? MW: FC has methods to determine what commands have actually been delivered and then continue from that point. JS: Let's have a minimum action. All commands that we know about that went over a failed connection, we can purge from the target, and inform the upper layer. The upper layer can then reset a task set, a LUN, or a target reset. CS: Please write up details of proposal and we'll discuss it tomorrow or Wednesday. Additional discussion. Can target clean up all of its state when a session fails? Do we want to try to recover session level failures? We can make an attempt to allow application to recover by reporting it, etc. LdO: Do we want to be more reliable than TCP? Why do we think we can do better? LdO: Hang silently (1) and hang and notify (2) are the same as far as iSCSI is concerned. They are different only with regard to implementation. RH: SAM seems to imply a notification, but this is not explicitly stated. When a target sees an error, should it completely clean up all state? or wait for the initiator to tell it what it should do? If a session died with some pending commands, should the iSCSI layer try to re-establish the session transparaently to the application? What state is cleaned up on the target when the session fails? Can cancel all outstanding tasks SDG: In their prototype, they abort all pending tasks in the SCSI devices. iSCSI layer cleans up all state. The initiator must then check the device and see where it is up to and what the state is. The application layer can do all the necessary recovery operations. iSCSI simply reports failed commands, and the upper level performs its recovery operations. CS: In order to perform recovery at the iSCSI level, you'll have to save a lot of information (at the target) in order to be able to recover in the case of failure. It may take 3-5 minutes to know that a TCP connection failed. That can be a lot of information to hold on to. Should we add timeout mechanisms to iSCSI to detect failed connections? CS: Timeouts should be at the highest level possbile. If the application already has a timeout mechanism, we need not add our own. RH: We expect TCP connections to not fail very often; certainly less often than FC. TCP may be even more reliable than SCSI. MW: If one physical link drops and we still have other links, do we want to abort the entire job, or continue transparently to the application with degraded performance? If we want the session to be able to carry on, then we must define the recovery mechanism. As in FC, the initiator can query the target as to what state it arrived at, and then continue from that point. There are applications that would entirely fail if we report an error on some command (like backup to tape which would rewind the tape and eject). JS: If the data was not acknowledged on the target, then the data was saved somewhere in the TCP layer. There is a way to recover a TCP flow without going to an upper layer (IP takeover, TCP splicing). MW: But then you need a way for iSCSI to get the lost information from the TCP layer. And what do you do if TCP is implemented in hardware? JS: This would now impose a requirement on another Working Group. JS: The right layer for this recovery is at the TCP layer rather than at the iSCSI layer. We are also not likely to do this recovery any better than TCP can. Let's have someone write up the different possibilities and then discuss further. RH: TCP already handles most of the problems that FC experiences that instigated FCP-2; dropped packets, disconnected wire for short periods, congestion, etc. The cases where TCP actually fails will be very rare so that we can claim that the QoS demanded by SAM is achieved, and we can fail the command on those exceptional cases where TCP fails, and perform hang and notify. JS: We should still write up a page or so of exactly what the target does to clean up in such a case. Conclusion: At a minimum we must support hang and notify. We still have a question as to whether TCP can be considered reliable enough to satisfy the QoS transport SAM requirement. JM: If necessary, the IP connections will be made more reliable using hardware. CS: as in IP telephony. Still we can't do anything against a link that is physically cut. Laundry List: Need better discussion of iSCSI session in the document. RTT used to communicate X_ID. Version # - associate to a session. How long is max SCSI CDB? What is the "right" way to communicate command parameters? If data will follow command without RTT, can we include it with the command packet? Action items: JS and CS will write up a page on details for error recovery for Wednesday. CS will present some thoughts on how to perform session recovery. We will all think a little more about the length fields in the message headers before deciding on Wednesday. LdO will send us whatever he has in writing on security. From XXX Tue Jun 20 06:40:41 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Tue Jun 20 01:38:52 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id BAA13643 for ; Tue, 20 Jun 2000 01:38:52 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA10026 for ips-outgoing; Tue, 20 Jun 2000 01:27:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09982 for ; Tue, 20 Jun 2000 01:26:14 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA79112 for ; Tue, 20 Jun 2000 07:25:43 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA181698 for ; Tue, 20 Jun 2000 07:25:43 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256904.001DCE8B ; Tue, 20 Jun 2000 07:25:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 20 Jun 2000 08:24:44 +0300 Subject: updated agenda Mime-Version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id BAA10023 Sender: owner-ips@ece.cmu.edu Precedence: bulk Proposed Schedule June 19 · 9:30-10:30 Get together & discuss agenda · 10:30 - 11:30 Requirements ? pass 1 · 11:30 - 12:30 Draft overview & laundry list · 12:30 - 13:30 Lunch · 13:30 - 17:30 Error Recovery & Reporting ? pass 1 June 20 · 9:30 - 10:30 Naming attempt to finalize · 10:30 - 12:30 Security ? Pass 1 · 12:30 - 13:30 Lunch · 13:30 - 16:30 Security ? Pass 2 · 16:30 ? 17:30 Requirements - pass 2 June 21 · 9:30 - 11:30 Security - Pass 3 and final · 11:30 - 12:30 Format review · 12:30 - 13:30 Lunch · 13:30 - 17:30 Format review June 22 (spillover) · 9:30 ? 11:30 Error Recovery & Reporting · 1130 & ON Spillover Items (& laundry List) From XXX Wed Jun 21 15:47:17 2000 Return-Path: owner-ips@ece.cmu.edu Delivery-Date: Wed Jun 21 11:43:06 2000 Return-Path: Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by yogi.ece.cmu.edu (8.8.7/8.8.7) with ESMTP id LAA28526 for ; Wed, 21 Jun 2000 11:43:06 -0400 Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA19802 for ips-outgoing; Wed, 21 Jun 2000 11:41:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA19798 for ; Wed, 21 Jun 2000 11:41:32 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA203208; Wed, 21 Jun 2000 17:38:24 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id RAA264794; Wed, 21 Jun 2000 17:38:24 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256905.0055E8D4 ; Wed, 21 Jun 2000 17:38:21 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu, scsi-tcp@external.cisco.com Message-ID: Date: Wed, 21 Jun 2000 18:38:08 +0300 Subject: minutes of iSCSI meeting 20 June 2000 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk iSCSI design team meeting Tuesday, 20 June 2000 Haifa, Israel Attendees: AA Alaan Azagury (IBM) SDG Steve De Grate (NuSpeed) JD John Dowdy (IBM) RH Randy Haagens (HP) GH Gabi Hecht (Gadzoox) JH John Hufferd (IBM) SL Steve Legg (IBM) JM John Matze (Veritas) KM Kalman Meth (IBM) LDO Luciano Dalle Ore (Quantumm) CS Costa Sapuntzakis (Cisco) JS Julian Satran (IBM) MS Mark Shrandt (NuSpeed) PvS Paul von Stamwitz (Adaptec) MT Meir Toledano (IBM) MW Matt Wakeley (Agilent) Disclaimer: Rough paraphrase of some of what was said. Some comments may be incorrectly attributed. Naming RH - Need to have a better understanding of the SAM-2 naming KM - Domain names are not necessarily global. CS- There is nothing better than DNS JS - Using URL syntax -- does it need to say syntax and semantics KM - Do we require applications to change since today they do not use URLs. CS - How to associate in Linus /dev/sdd1 to a specific SCSI device. This is part of discovery so this is beyond the scope of this document. RH - Naming is critical to identify third party objects RH - Use of domain names does not seem to be controversial. The rest of the URL structure might be more debatable (e.g. to identify LUNs) CS - Domain names can also directly code an IPv4 and IPv6 address, where you don't want to require a DNS RH - Issue with naming LUNs - different users refer to the same LUN using different names. Do we need a central authority that will translate names to be able for one user to pass a reference to a LUN to a different user. KM - Using the VPD with the URL gives you a unique identifier to the LUN. PvS - VPD may refere to a virtual LUN,not a physical device. Need to make sure that the VPD satisfies our requirements. JS - Is it too early to resolve, since we don't have a discovery mechanism in place yet. We should restrict ourselves to the basic naming service. JH - An entity could have a unique name based on the domain + VPD. A directory could translate this name to a human-readbale name. A host can remap the name to the VPD and map this address to his local LUN number and access the device through this LUN. JS - Could we not use the LUN number at all? JS - Naming allows exporting only the devices that an iSCSI target wants to export. JH - Initiator can query the target what devices are "attached" to it. The target can return whatever devices the initiator can access. JS - The current naming mechanism allows different naming for the same target that allows mapping different views to the same target. We should not enter the details of the LUN naming. Views are maps to "real" things that the initiator is allowed to see. LDO - SAM-2 has unfriendly name. Why are we trying to build a mapping from user-friendly names to the SAM-2 names? Names need to map well onto the SAM-2 naming. RH - SAM-2 allows dfferent, maybe overlapping views of LUs through different SDPs. These views are defined by the controller. JH - In FC, the views are defined by initiators (host worldwide name). RH- Jim Hafner has "equated" the worldwide name to an accessid. In this world, the controller does not present different views, but the view is determined by the accessid. JS - Do we know enough to name LUs. Julian thinks not. RH - We should support the SCSI naming architecture. LDO - Are we trying to do too much in iSCSI layer? Should the query of LUN 0 be left to SCSI? SC - The URL just identifies the target (with a certain view). JS - View = Virtual target. A domain name defines a virtual target. LDO - Can the name be: scsi://a.b.c.d/accessid JS - Acessid has been accepted by FC Security LDO - Two security models: (1) authentication/authorization and (2) fully encrypted link. (1) is resistent to interception (listener cannot reuse authentication to authenticate itself). Use the SCSI access control proposal. User ID is stored in the access control database. Security key is to be stored in a parallel db using a separate exchange protocol. CS - Can many machines/processses have the same user id. LDO - Can the User ID correspond to AccessID? RH - Do we want to separate principals and views? CS - What is the scope of the AccessID? Host/Target? Only host? CS - scsi:/a.b.c.d/cdrom -> cdrom is the target, not the accessid. JS - We are talking about authenticating machines, not "users". PvS - The Hafner proposal assumes a trusted environment. LDO - Until there is a good reason, we will assume that user id and access id is the same. SC - Access ids allow you to have different views within a SDP (target). You can also use different targets (SDPs) to present different views. SC - a tuple (domain_name/path, accessid) defines the actual view. Lunch LDO: Have User id (== access id). SCSI doesn't provide place to save this info. So we'll need a separate database for these. LDO continues presentation: Fully Encrypted Link ---------------------- Use IPSec as security mechanism use Login with null authentication scheme - use IPSec authentication Discard SSL IPSec better hardware support possible use of SSL on Internet? Challenge by JH. LDO: At present, we don't see a good reason to use SSL vs IPSec. CS: Let's write down on the board the advantages and disadvantages of each. IPSec: + simplet hardware ? requires an IPSec supportive TCP stack. - no App API for key mgmt. + more secure (no TCP DOS) - NAT boxes don't exist yet. + hardware support readily available ? it encrypts everything above the IP layer. all or none SSL + APP layer library (rapid deployment) - APP layer library (not easily available in kernel) + mature + goes wherever internet goes - lower bandwidth? (due to lack of current hardware support) + internet JS: Now that IPSec is included in Windows 2000, it is available enough. CS: IPSec doesn't work so well through firewalls, whereas SSL works fine. Once we decide what we want, we could probably get either one in hardware. RH: IPSec will make it worse for my Traffic Monitoring software. LDO: That's why you do encryption! Discussion on what happens in NAT box and why IPSec won't work with it, since the checksum is encrypted and the NAT tries to change an IP address, thereby requiring changing the encrypted checksum. JS: It is reasonable to say that use either IPSec or the data will have to be encrypted at the storage level. If you have a VPN, then no need to impose another level of security. SSL (in the short term) is going everywhere the internet goes; it will be more widely deployed. LDO: SSL is currently very tied into html. JH: IPSec will work with lot's of protocols. Will SSL work with so many? JH: If we choose SSL, then vendors will support it in hardware, so there won't be lower bandwidth for SSL. LDO: We should put this question out to the Security community. LDO: Most people won't use security unless they are big companies. So we should look at what big companies are going to use. RH:And most big companies are going to use IPSec infrastructures. CS: IPSec is not quite mature yet. LDO: What about in 2 years time? PvS: Who is going to ask for block storage over the internet? What applications are going to use this? Then choose SSL or IPSec depending on which is more appropriate for the application. JS: We don't want to choose one method that may not be accepted by the broader community in n years time. LDO: Essentially we are saying that we will support both. KM: A target MAY support one or the other or both. If a connection comes in and tries to talk IPSec, the target can reject the request and insist on the other. JH: If a vendor comes to you and asks which one you want implemented in hardware, what do we tell him? RH: We will figure out what we need to put into iSCSI to support both. JS: And then we'll see the mechanics to see which one is more apropriate. RH: So what commands do we need in iSCSI to use IPSec and/or SSL. LDO: Use your identity for IPSec as your name for iSCSI, and then iSCSI simply needs to pass this name to the IPSec layer to perform the authentication. JH: How does this now connect with out discussion on AccessId? RH: Can you please give an explanation of how IPSec comes up? LDO: In IPSec one of the authentication parameters is the user id from the operating system. JH: We can make it our Access id. LDO: We can try to define it later... we think there is a way to implement it. LDO: There is also a security descriptor. LDO: Fundamental question: does IPSec allow you to use a user id as authentication? JS: Win2K allows configuring IPSec for all TCP/IP connections: it allows configuring for Server request, client (respond only), secure server etc. KM: When do you set the IPSec connection, does it need to be called by application or is it at the IP level? LDO: It depends how you implement the protocol stack on the machine. KM: The application has no control of whether to use IPSec. JS: The only thing that can be done is to set a policy. KM: iSCSI has no code to write to use IPSec. JS: Let's go through the mechanisms and then come back to this issue. LDO: If we use IPSec, we use the access id as the user id. RH: The PAM guys might not agree to use the access id as the IPSec user id. LDO: Do we need a separate port for SSL? KM: We can do by trial and failure: if the SSL handshake fails, we can retry with non-SSL LDO: (1) iSCSI negotiation to start SSL, (2) SSL negotiation, (3)encrypted iSCSI LDO: In http, there are two different ports. JS: How will the mechanism work at login? LDO: Two options: (1) single port and need to upgrade to SSL; (2) two ports. LDO: Need to specify the interaction (user id vs access id) JS: IPSec requires an administration server. We need to answer those questions. LDO: SSL provides encryption and authentication of the server. CS: SSL provides the means also to authenticate the client to the server (client certificates). CS: Do we need to spec at least two security mechanisms? LDO: Do we all agree that we do IPSec and SSL. KM: Do we need to authenticate as part of the login? LDO: In SSL you need it if you are not using client certificates. LDO: If we only want authentication, we do not want to require encryption. JS: Challenge/response is good enough for authentication? All agree JS: For Pittsburgh we need the draft ready in 2 weeks. RH: Are the secuirty models mutually exclusive 0. No security - generic login with ACL 0.5. Login name + password (clear) 1. challenge/response authentication 2a. IP Sec. IP sec authentication is used 2b. SSL - authentication thru client certificates; privacy + 1. CS: Challenge/response is very easy to implement. We should require it instead of 0.5. Doesn't like to send password in the clear. JM: It becomes an administrator headache. No security is not enough. CS: Most recent CIFS does 1. JM: Agrees to drop requirement for 0.5. LDO: IPSec is more adequate for enterprise; SSL is more adequate for Internet. JH: When going thru NAT you probably do not want to use IPSec. JM. In what environments do you need 2? CS: Marketing guys said do not release this without security. JS: SSL suits short-lived transactions. IPSec can live with long transaction, change keys regularly. Julian bets that IPsec is the right approach. JS: Do we need any security hardware in the adaptor? If we need something, it is probably IPSec. CS: SSL allows to renegotiate keys. RH: IPsec runs on the network... Network administrators prefer it because they are "securing the network". JH: Is it sufficient to say 0, 1 and 2.b? 2a is implicit and should not concern iSCSI. RH: Does not agree. JM: CIFS uses 0, 1 and 2b. Requirements (pass 2) PvS: Do not need an exclusivity statement that this runs only on TCP. RH: Non-goal: iSCSI does not need to run on anything other than TCP. JS: Reliable, in-order delivery and congestion control are the minimal requirements of the transport. JS: About 3.2, the number of 100 microsecond is not too realistic. Replace with delay can be significant. SL: Reinforce that not more than one interrupt is required per I/O operation JS: Regarding low delay, add "we will handle well infrastructures that have a good bandwith-latency product." CS: Are there any implications on the implementation on cost comptetiveness. JS: The fact that we did not consider implmeneting FCP on top of IP. PvS: Will any SCSI "drive" need to export a LUN 0? RH: Including good support for individual disk might imply significant simplifications in iSCSI. MW: What part of FC will hijack ... MW: Need to require that this runs on reliable protocol CS: Support SCSI SMA-2 architecture: add "where it is applicable" KM: Is task queuing a requirement? PvS: We need to make it a requirement. KM: Can't assume it supports task queueing just because it runs on top of iSCSI. PvS: Could require task queueing, but specify max tags is 1. MT: SCSI3 compatibility -- what about CAM? KM: Why do we need the "forward compatibility" requirement? RH: The main point is to have clean layering. JH&PvS: Isn't that already included in SAM? So we can drop this requirement. RH: MS: We can never cover ourselves from all possible evolutions of SAM. This statement should move to a Front Matter section. MT: If you say SPC-2, then you should also list other similar protocols. MW: FCP needs (FCP-CONF) confirmation only for error reovery. For TCP we shouldn't need this. RH: We'll remove this from the requirements, but keep it in mind for implementation. KM: Do we want to require that iSCSI can be used for a gateway between FCP networks? Might we not be somewhat dependent on FCP to be able to run over iSCSI? JH&RH: Without this kind of gateway, iSCSI won't sell. We absolutely need this to be achievable by iSCSI. KM: Do we want to make this a requirement on ourselves? or simply state it as a goal? ....... ---------------------------------------------------------- command protocol -> command sets ---------- Topic: Requirements opoint on translating gateways Add that we think it's redundant but we'd like to call it out CS: Process check How do we speed this process up? Options: Just raise issues explicitly Going through it one-by-one was valuable Next hour we'll raise issues explicitly Clarification of 3.4.1 - doesn't preclude trunking Point 3.4.3 brings up ordering of commands per LUN or ordering of commands per session? MW - reorder requirements - second before the first one RH - does anybody disagree with 3.4.3? CS - No. It basically says that you support SCSI queuing JH - In the wide area, a method of pipelining commands and responses is required CS - the requirement is more complex than saying you just support SCSI queing. RH - delivering commands in order never hurts MT - Why keep order between logical units in commands? RH - SAM-2 does not require order between LUNs. However, it may make target implemetnation easier. ------------------ JH - why do we require that pipeline commands and responses? RH - change required -> needed --------- 3.5.1 agreed 3.5.2. agreed 3.5.3. Strike we reject the idea... Hufferd - how could the storage not respect bandwidth management or congestion avoidance algorithms? RH - By not using TCP MW - Are we going to say anythign about IPv6? CS - Definitely, we should say we support it. RH - Should we require that implemetntaions hafve dual stack? CS - No. Just the protocol should support it. MW - You don't mention the IPv6 literal naming in the iSCSI spec 3.5.4 Link indpendent 3.5.5 LAN, MAN, WAN capable RH - mention insufficient memory for spanning large distances LO - it might degrade performance, but it doesn' tlimit capability CS - Other aspect of this is firewalls and NAT. This is common in WAN RH - we need to think of these things in the WAN. Agreed 3.5.6 Framing - some method iSCSI protocol units within the TCP stream must be defined RH - this might not need to be here. It has to know when an iSCSI protcol unit starts. The transmitter must ensure that it gets transmitted and not buffered. LO - buffering is a specific TCP implementation and not a generic problem JVeritas - RH - how do we establish frame syncrhonization? LO - we can ignroe this JC - TCP is a stream of bytes RH - it just comes as a byte range. How does it identify? LO - never make a mistake. MW - Proper message forming not just parsing RH - framing issue and pushing issue - connections are long-lived - cumulative effect of length fields LO - overwritten by CRC - any error in synchronization is major RH - badly formed packets due to software error Answers - should fail checksum RH - target fibrechannel should be robust against anything sent MW - can somebody jam data into somebody's TCP connection? Answer - yes... LO - CRC is sufficient RH - number of techniques one uses for reliablity. Framing of packets - length fields - most robust systems use several techniques (SOF, CRCs, length field) - could compensate for TCP's lack of framing LO - known bit-error rate of TCP is ... RH - mainly worried about software and hardware problems - don't know what errors we're open to when we're relying on stuff from time 0 LO - use flag byte at beginning of packet RH - same as byte stuffing? LO - this is just error detection JH - CRC strength is not strong enough - recommendation was to use another CRC - use the CRC stuff we actually use on the disk JVeritas - CRC is overkill JH - stronger thing that can be used MW - where are you going to drop the bits - covered by 32 bit CRC CRC discussion -------------- MW - TCP checksum and MAC layer doesn't fix PCI CS - can be fixed with end-to-end CRCs JS - any decent storage system is end-to-end - only do it in two says (SCSI/iSCSI layer) - could protect against errors in protocol stack - could protect against erros in PCI - do it in TCP stack - could protect against errors in TCP segments MW - solving the problem halfway - TCP checksum doesn't deal with stuff at higher layer RH - this end-to-end principle - recovery should be done at highest layer (most correct) - put stuff lower in the stack for optimization purposes only JH - protected against most probably problems - high value in and of itself LO - verify framing by putting a well known bytes - at the beginning of each packet - ------------ Large discussion on iSCSI framing and CRCs Main issues on CRCs and reliability - Do nothing - TCP-layer CRC (CRC on datagram) - iSCSI layer CRC Higher layer CRCs were seen as being able to detect the most kinds of errors. However, The iSCSI layer CRC was proposed in a couple different forms: covering iSCSI messages, every n kilobytes in the iSCSI stream, or covering only iSCSI data. LO pointed out that IPSec provides a very strong CRC with its authentication header (MD5 checksum). A well-known key (say 0) could be used to avoid the need to do key exchange for authentication. Consensus: IPSec AH for extra integrity ------ Some discussion on having the common cases in iSCSI being that 1 TCP segment contains an iSCSI message as payload (optimization). This would be especially valuable for data transfers, as each segment would have enough information to place the data in memory at the remote end. From owner-ips@ECE.cmu.edu Sat Jun 24 15:53:26 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02873; 22 Jun 2000 15:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA20944 for ips-outgoing; Thu, 22 Jun 2000 13:33:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA20940 for ; Thu, 22 Jun 2000 13:33:28 -0400 (EDT) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id LAA29019; Thu, 22 Jun 2000 11:33:22 -0600 (MDT) Received: from swanaba.central (swanaba.Central.Sun.COM [129.147.30.5]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA23400; Thu, 22 Jun 2000 11:33:18 -0600 (MDT) Received: from ebay.sun.com (salespilot.Central.Sun.COM [129.147.30.18]) by swanaba.central (8.8.8+Sun/8.8.8) with ESMTP id LAA26857; Thu, 22 Jun 2000 11:12:23 -0600 (MDT) Message-ID: <39524D1D.E21339B9@ebay.sun.com> Date: Thu, 22 Jun 2000 10:30:05 -0700 From: David Robinson X-Mailer: Mozilla 4.72 [en] (Win98; I) X-Accept-Language: en MIME-Version: 1.0 To: meth@il.ibm.com CC: ips@ece.cmu.edu, scsi-tcp@external.cisco.com Subject: Re: summary of iSCSI meeting 22 June 2000 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk meth@il.ibm.com wrote: > Further discussion of what happens when TCP packets get lost, especially if > they contain an iSCSI header. > How well can iSCSI compete with FC if we are so dependent on TCP, with its > dropped packets. > > In the LAN, TCP packets are not generally lost and we should be comparable > to FC. > Over WAN, can have packet loss and resulting complications, but that is no > longer competing with FC > (which doesn't exist at all in the WAN). Huh? TCP packets can never get lost, you either get the packet or the connection is dropped. There may be some delay as TCP performs a retransmission which will be rare on LANs and not so rare on WANs. I don't see how this is a FC vs TCP issue. -David From owner-ips@ECE.cmu.edu Sat Jun 24 15:53:26 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02790; 22 Jun 2000 14:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA19621 for ips-outgoing; Thu, 22 Jun 2000 12:53:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from charon.Exabyte.COM ([206.104.178.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA19617 for ; Thu, 22 Jun 2000 12:53:38 -0400 (EDT) From: JoeBre@exabyte.com Received: from prime1.Exabyte.COM (prime1.Exabyte.COM [161.81.18.231]) by charon.Exabyte.COM (8.9.1a/8.9.1) with ESMTP id KAA18102; Thu, 22 Jun 2000 10:52:20 -0600 (MDT) Received: from bouldermta.exabyte.com (bouldermta.Exabyte.COM [161.81.192.16]) by prime1.Exabyte.COM (8.6.12/8.6.10) with ESMTP id KAA04783; Thu, 22 Jun 2000 10:52:20 -0600 Received: by bouldermta.Exabyte.COM with Internet Mail Service (5.5.2448.0) id ; Thu, 22 Jun 2000 10:51:03 -0600 Message-ID: <118A72FD3F02D211AD1000805F9AA58B012B7A80@boulderp03.Exabyte.COM> To: meth@il.ibm.com, ips@ece.cmu.edu, scsi-tcp@external.cisco.com Subject: RE: summary of iSCSI meeting 22 June 2000 Date: Thu, 22 Jun 2000 10:51:08 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01BFDC6A.0C55BB94" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01BFDC6A.0C55BB94 Content-Type: text/plain; charset="iso-8859-1" Comments inline > ... > > Further discussion of what happens when TCP packets get lost, > especially if > they contain an iSCSI header. > How well can iSCSI compete with FC if we are so dependent on > TCP, with its > dropped packets. All error handling for dropped packets should exist below the SCSI layer. If SCSI would be required to be involved in error handling for dropped packets, it would be disastrous for SCSI Stream Devices (ex: tape backup session). This is due to the fact that SCSI Stream Devices have stateful behavior. > > In the LAN, TCP packets are not generally lost and we should > be comparable > to FC. > Over WAN, can have packet loss and resulting complications, > but that is no > longer competing with FC > (which doesn't exist at all in the WAN). The people working on the FC-BB specification would probably refute the statement that FC does not exist at all in the WAN. > ... > Naming: > > What have most of us agreed to in the meantime? > We do not presently attempt to name LUNs. > We name targets with URL syntax > scsi://host_name[/virtual_target_name] This may be more appropriately scsi://host_name[/virtual_device_name] > > Multiple views can also be achieved by using different > host_names for the > same target. Perhaps same device, rather than same target. > > Further rehashing and discussion about virtual_target_name (= > view, = VSDP > == Virtual Service Delivery Port). Careful about the terminology here - under SAM and SAM-2, the term Target is not synonymous with Service Delivery Port. Indeed, a SCSI Device consists of (1 Target, or 1 Initiator, or both), and (a nonzero number of Service Delivery Ports). Joe Breher Exabyte Corp ------_=_NextPart_001_01BFDC6A.0C55BB94 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: summary of iSCSI meeting 22 June 2000

Comments inline

> ...
>
> Further discussion of what happens when TCP = packets get lost,
> especially if
>  they contain an iSCSI header.
> How well can iSCSI compete with FC if we are so = dependent on
> TCP, with its
> dropped packets.

All error handling for dropped packets should exist = below the SCSI layer. If SCSI would be required to be involved in error = handling for dropped packets, it would be disastrous for SCSI Stream = Devices (ex: tape backup session). This is due to the fact that SCSI = Stream Devices have stateful behavior.

>
> In the LAN, TCP packets are not generally lost = and we should
> be comparable
> to FC.
> Over WAN, can have packet loss and resulting = complications,
> but that is no
> longer competing with FC
> (which doesn't exist at all in the WAN).

The people working on the FC-BB specification would = probably refute the statement that FC does not exist at all in the = WAN.

> ...

> Naming:
>
> What have most of us agreed to in the = meantime?
> We do not presently attempt to name = LUNs.
> We name targets with URL syntax
> scsi://host_name[/virtual_target_name]

This may be more appropriately = scsi://host_name[/virtual_device_name]

>
> Multiple views can also be achieved by using = different
> host_names for the
> same target.

Perhaps same device, rather than same target.

>
> Further rehashing and discussion about = virtual_target_name (=3D
> view, =3D VSDP
> =3D=3D Virtual Service Delivery Port).

Careful about the terminology here - under SAM and = SAM-2, the term Target is not synonymous with Service Delivery Port. = Indeed, a SCSI Device consists of (1 Target, or 1 Initiator, or both), = and (a nonzero number of Service Delivery Ports).

Joe Breher
Exabyte Corp

------_=_NextPart_001_01BFDC6A.0C55BB94-- From owner-ips@ECE.cmu.edu Sat Jun 24 15:53:26 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07843; 24 Jun 2000 11:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA06220 for ips-outgoing; Sat, 24 Jun 2000 10:23:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA06216 for ; Sat, 24 Jun 2000 10:23:06 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA110080 for ; Sat, 24 Jun 2000 16:22:33 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id QAA71588 for ; Sat, 24 Jun 2000 16:22:35 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256908.004EF57C ; Sat, 24 Jun 2000 16:22:26 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 24 Jun 2000 17:21:34 +0300 Subject: WG status and actions for IETF 48, status Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk FYI ---------------------- Forwarded by Julian Satran/Haifa/IBM on 24/06/2000 17:18 --------------------------- Julian Satran 23/06/2000 16:09 To: Scott Bradner cc: Allison Mankin From: Julian Satran/Haifa/IBM@IBMIL Subject: status Scott, I just wanted to inform you that meeting in Haifa went well and we have made considerable progress - at least in convincing ourselves that there is no major blunder in what we are doing. As you have probably seen the attendance was covering many companies (and many more are getting active on the mailing list). As expected there was a considerable level of anxiety related to the fact that our formal status is still unknown (to the extent that we don't know even if we have a session at IETF-48). As you are aware - at current speeds - the protocols we are talking about - will have to have some silicon support - and silicon builders are reluctant to build on "informal" standards. That is the main reason some of us where so anxious to move the transport path to solid ground as soon as possible and then proceed to integrate related activities (discovery, management etc.) with the rest of the IETF world. I would appreciate if you could let us know where we are with regard to the formal IETF process. We will have a revised draft ready for public review before the IETF-48 deadline. Thanks, Julo Scott Bradner on 24/06/2000 15:00:13 Please respond to Scott Bradner To: Julian Satran/Haifa/IBM@IBMIL, sob@harvard.edu cc: mankin@east.isi.edu Subject: Re: status sorry to be so confusing - there will be a working group and it will meet durng IETF-48 the silicon builders should be reluctant at this point since the result of IETF delebration may have quite significant changes from what the process starts with we will be back with you soon but you can announce that the wg is about to be formed Scott From owner-ips@ECE.cmu.edu Sat Jun 24 15:53:26 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04969; 23 Jun 2000 9:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA24488 for ips-outgoing; Fri, 23 Jun 2000 08:14:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA24484 for ; Fri, 23 Jun 2000 08:14:18 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id OAA309166 for ; Fri, 23 Jun 2000 14:13:45 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id OAA21612 for ; Fri, 23 Jun 2000 14:13:45 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256907.00432C31 ; Fri, 23 Jun 2000 14:13:42 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Jun 2000 14:46:30 +0300 Subject: agenda history Mime-Version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-Disposition: inline X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id IAA24485 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id IAB24488 Thanks to all and everyoane of the participants at design team meeting in Haifa for their hard work, patience and enthusiasm and see you all at IETF-48 in Pittsburgh. Julo Julian Satran - IBM - Research Laboratory in Haifa Haifa ? Design Team Meeting ? Real Agenda - History June 19, 2000 Themes: ID0 ? ID Overview ID1 ? ID Security ID2 ? ID Error Reporting & Recovery ID3 ? ID Message Formats (review based on effectiveness & ease of transliteration) ID4 ? ID Minimal Naming Requirements RD0 ? Requirements and Applicability Proposed Schedule June 19 =B7 9:30-10:30 Get together & discuss agenda =B7 10:30 - 11:30 Requirements ? pass 1 =B7 11:30 - 12:30 Draft overview & laundry list =B7 12:30 - 13:30 Lunch =B7 13:30 - 17:30 Error Recovery & Reporting ? pass 1 =B7 17:30 - 19:15 More unplanned error Recovery & Reporting =B7 19:30 - 22:00 Working dinner - No Error Recovery & Reporting! June 20 =B7 9:30 - 10:30 Naming attempt to finalize =B7 10:30 - 12:30 Security ? Pass 1 =B7 12:30 - 13:30 Lunch =B7 13:30 - 16:30 Security ? Pass 2 =B7 16:30 ? 17:30 Requirements - pass 2 =B7 17:30 ? 21:30 More Reurements... June 21 =B7 9:30 - 11:30 Error Recovery & Reporting =B7 11:30 - 12:30 Sequencing & Flow Control =B7 12:30 - 13:30 Lunch =B7 13:30 - 17:30 Sequencing & Flow Control =B7 17:30 ? 21:30 Session model June 22 (spillover) =B7 9:30 ? 11:30 Security - Pass 3 and final =B7 11:30 ? 12:30 Naming - new attempt to finalize =B7 12:30 - 13:30 Lunch =B7 13:30 - 14:30 IETF Plans From owner-ips@ECE.cmu.edu Sat Jun 24 15:53:26 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06953; 24 Jun 2000 3:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA21805 for ips-outgoing; Sat, 24 Jun 2000 02:20:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA21800 for ; Sat, 24 Jun 2000 02:19:58 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA33518 for ; Sat, 24 Jun 2000 08:19:25 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id IAA102860 for ; Sat, 24 Jun 2000 08:19:23 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256908.0022B973 ; Sat, 24 Jun 2000 08:19:16 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 23 Jun 2000 15:29:31 +0300 Subject: Re: summary of iSCSI meeting 22 June 2000 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk For all those concerned about the recovery discussion here are some clarifications: 1. The whole discussion thread was related to an attempt to recover from one TCP connection failure in a session that has multiple TCP connections in order to fully exploit the fault tolerance level users are expecting when using several connections 2. As we are aware that stateful devices and operation idempotency are hard to handle in general terms we are currently contemplating mostly recovery mechanisms that are "target-centric" (mostly target initiated). Obviously we would love to be able to recreate a TCP connection in exactly the state it got lost but we are not aware of any such magic being available... Julo Julian Satran - IBM Research Laboratory at Haifa David Robinson on 22/06/2000 20:30:05 Please respond to David Robinson To: Kalman Meth/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu, scsi-tcp@external.cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: Re: summary of iSCSI meeting 22 June 2000 meth@il.ibm.com wrote: > Further discussion of what happens when TCP packets get lost, especially if > they contain an iSCSI header. > How well can iSCSI compete with FC if we are so dependent on TCP, with its > dropped packets. > > In the LAN, TCP packets are not generally lost and we should be comparable > to FC. > Over WAN, can have packet loss and resulting complications, but that is no > longer competing with FC > (which doesn't exist at all in the WAN). Huh? TCP packets can never get lost, you either get the packet or the connection is dropped. There may be some delay as TCP performs a retransmission which will be rare on LANs and not so rare on WANs. I don't see how this is a FC vs TCP issue. -David From owner-ips@ECE.cmu.edu Sat Jun 24 15:53:26 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02349; 22 Jun 2000 10:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA04409 for ips-outgoing; Thu, 22 Jun 2000 08:39:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA04398 for ; Thu, 22 Jun 2000 08:39:52 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id OAA211166 for ; Thu, 22 Jun 2000 14:39:07 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id OAA20052 for ; Thu, 22 Jun 2000 14:39:08 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256906.00457DC4 ; Thu, 22 Jun 2000 14:39:01 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 22 Jun 2000 15:38:13 +0300 Subject: agenda history Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks to all and everyoane of the participants at design team meeting in Haifa for their hard work, patience and enthusiasm and see you all at IETF-48 in Pittsburgh. Julo Julian Satran - IBM - Research Laboratory in Haifa From owner-ips@ECE.cmu.edu Sat Jun 24 15:53:26 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02356; 22 Jun 2000 10:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA04189 for ips-outgoing; Thu, 22 Jun 2000 08:29:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA04185 for ; Thu, 22 Jun 2000 08:29:12 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA180898; Thu, 22 Jun 2000 14:28:37 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id OAA93634; Thu, 22 Jun 2000 14:28:33 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256906.004485A3 ; Thu, 22 Jun 2000 14:28:26 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu, scsi-tcp@external.cisco.com Message-ID: Date: Thu, 22 Jun 2000 15:14:02 +0300 Subject: summary of iSCSI meeting 22 June 2000 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk iSCSI design team meeting Thursday, 22 June 2000 Haifa, Israel Attendees: SDG Steve De Grate (NuSpeed) JD John Dowdy (IBM) RH Randy Haagens (HP) GH Gabi Hecht (Gadzoox) JH John Hufferd (IBM) NN Nelson Nachum (Storage) JM John Matze (Veritas) KM Kalman Meth (IBM) LDO Luciano Dalle Ore (Quantumm) JS Julian Satran (IBM) MS Mark Shrandt (NuSpeed) PvS Paul von Stamwitz (Adaptec) MT Meir Toledano (IBM) MW Matt Wakeley (Agilent) Overview of last night's brainstorming session. Problem of spreading the same set of sequence numbers across multiple iSCSI NICs. Only commands need seq numbers. If assign the seq in the host, then have multiple interrupts on the host and contesting of some semaphore by multiple iSCSI NICs. If even data packets and RTT etc are numbered, this helps in recovery at the iSCSI level. Have thin layer of iSCSI in host and the rest in the NIC. The part on the host gives seq numbering and perhaps some PDU formatting. The SCSI layer must be well enough behaved to not saturate the iSCSI level. perhaps via max MTU size or max number of commands pending, etc. i.e. have some level of flow control imposed to the SCSI level above iSCSI level (by limiting the number of pending SCSI commands). This then obviates need of communication between host iSCSI levels on each command, since almost all can be done in the iSCSI NICs. The communication between the software iSCSI layers can be so low that it doesn't affect the overall flow. When needed, (for task management emergencies) the software iSCSI layers can communicate via regular tcp/ip, not using the high bandwidth iSCSI NICs. This obviates the need to use flow control on the other high bandwidth lines. They can all saturate, while the auxiliary line is always kept open for urgent messages. Only SCSI command PDUs need be numbered The connections between initiator and target with the iSCSI NICs on both ends can do much of their work independently. (RTT and send data). Discussion. JS has a scheme to allow seq numbering per LUN all without using excessive resoures on initiator side. JS will send out a page describing his ideas. Discussion of LDO's method from yesterday to use TCP back pressure to avoid need for separate iSCSI flow control. KM put a proposal on the table to go back to the model with one control channel and multiple data channels, with several additional features to overcome objections raised earlier. JS asked KM to put the idea in writing. LDO suggested we come up with several implementations to see which of the suggested models works best. Further discussion of what happens when TCP packets get lost, especially if they contain an iSCSI header. How well can iSCSI compete with FC if we are so dependent on TCP, with its dropped packets. In the LAN, TCP packets are not generally lost and we should be comparable to FC. Over WAN, can have packet loss and resulting complications, but that is no longer competing with FC (which doesn't exist at all in the WAN). Security: Open Issues: need text to go into the internet-draft. We still have question of SSL vs IPSec. In the meantime we proposed to describe how to use both, with a recommendation to use IPSec. This may have to be left to the WG to decide. In the meantime, we have to write something concrete. We have to write the security details according the guidelines on how to write them up and include it the draft. See http://www.ietf.org/internet-drafts/draft-rescorla-sec-cons-01.txt. Suggestion: Start with the description on how to use iSCSI with IPSec. But IPSec is still (after 3 years) only an Internet-Draft; not an internet standard. And it may never become an internet standard, which may delay the acceptance of iSCSI as a standard. We can start with IPSec, and if that prevents us from becoming a standard, then we can change that section of the draft to no longer depend on IPSec. The problem with IPSec was going past a NAT (Network Address Translation). Suggestion to place IPSec at the gateway to the internet, whereas on the local LANs (SANs) we can go in the clear. The NATs would be behind the gateways to the LANs. In this case, have an IPSec tunnel (when use NATs). Alternatively, use IPSec end-to-end (with no NATs). Check out SASL to see if we can leverage it. Naming: What have most of us agreed to in the meantime? We do not presently attempt to name LUNs. We name targets with URL syntax scsi://host_name[/virtual_target_name] Multiple views can also be achieved by using different host_names for the same target. Further rehashing and discussion about virtual_target_name (= view, = VSDP == Virtual Service Delivery Port). Each of these virtual_targets (views) will have a virtual LUN 0 that can report what is visibile on that virtual_target (view). There will be an administrative view with name: iSCSI This view has no LUNs and is only to establish iSCSI sessions that perform text dialogs to discover what is available on that machine. How to specify the Access ID? Lunch What do we want to accomplish at the upcoming IETF meeting? Assuming that we have a Working Group formally estalished ...... Do we want to make iSCSI into an RFC proposed standard, implying that we commit to having 2 different implementations within 6 months? Are we stable enough to have compatible implementations? We are also dependent on the opinions of the WG Chairmen and Area Directors if this can progress. RH: We are still at a preliminary stage and we haven't nailed down all the issues. We will still have to make some changes. JS: There will also be other bright people who can give us some ideas. So we aren't really ready to go straight to a proposed standard. We still need some input. JS: It is important that as many of us show up at the IETF. We need to counter the dissenters who may show up. We need to clean up the 2 documents: Requirements and Protocol. What presentations do we want to prepare for the IETF meeting? RH will take a number of the pictures we had on the board and prepare slides. We should have a presentations on: High level goals and requirements Security issues Design principles Protocol details? Implementation implications We also need time for feedback from others. Can we ask for 2 sessions? One for high level stuff and one for details and feedback? JS will ask Scott Bradner if we can have 2 sessions. Action Items: Place updated Requirements and Protocol documents on the mailing list by next Wednesday. LDO and PvS will provide their stuff on security: login process and discussion of using IPSec. Schedule: 28 June 2000 - updated Requirements and Protocol documents 29 June 2000 - iSCSI design team phone conference call 8:00-13:00 Pacific time 5 July 2000 - updated documents ready for IETF 30 July 2000 - IETF meeting From owner-ips@ECE.cmu.edu Sat Jun 24 15:53:26 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07847; 24 Jun 2000 11:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA06157 for ips-outgoing; Sat, 24 Jun 2000 10:18:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA06153 for ; Sat, 24 Jun 2000 10:18:26 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA198026 for ; Sat, 24 Jun 2000 16:17:53 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id QAA256128 for ; Sat, 24 Jun 2000 16:17:54 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256908.004E89C1 ; Sat, 24 Jun 2000 16:17:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 24 Jun 2000 17:16:59 +0300 Subject: Re: mailing list - new name/old name/content Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, On behalf of the entire team - thanks. Regards, Julo Dave Nagle on 24/06/2000 09:29:10 Please respond to Dave Nagle To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: mailing list Julian, Original Message ---------------- >> Are the archives updated automatically? (I don't see the latest traffic) Ooops ... The pages will be updates automatically ... just as soon as the script get's fixed (by monday). >> Can we rename the list to iSCSI without changing the URL (or keeping it as >> an alias) - that was the main concern at the meeting. I'll make a second url ... and move everything to iSCSI ... but maintain a pointer from ips to iSCSI. >> Can we make it have 3 areas: >> >> - draft documents (keep all versions) sure ... >> - memos to consider ( a proposal for a specific area) - should be kept at >> least as long as they are under consideration sure .... >> - mailing archive >> sure ... I'll try to get this done by Monday (except for the renaming ... which may take a few more days). Once I've got the web pages redone, I'll ping you to make sure all of the revelent documents are up. dave........ From owner-ips@ECE.cmu.edu Sun Jun 25 05:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09120; 25 Jun 2000 1:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA17775 for ips-outgoing; Sat, 24 Jun 2000 23:51:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA17770 for ; Sat, 24 Jun 2000 23:51:33 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id FAA122098; Sun, 25 Jun 2000 05:51:00 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id FAA134764; Sun, 25 Jun 2000 05:51:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256909.0015239E ; Sun, 25 Jun 2000 05:50:53 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Allison Mankin cc: ips@ece.cmu.edu Message-ID: Date: Sun, 25 Jun 2000 06:44:26 +0300 Subject: Re: status Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Allison, Thanks. We will have new draft submitted in time. Julo Julian Satran - IBM Research Lab. at Haifa ---------------------- Forwarded by Julian Satran/Haifa/IBM on 25/06/2000 06:41 --------------------------- Allison Mankin on 24/06/2000 18:33:09 Please respond to mankin@east.isi.edu To: Julian Satran/Haifa/IBM@IBMIL cc: Scott Bradner Subject: Re: status Hi, Just to add to what Scott said, it would be very timely to put the draft you all worked on in Haifa out as an internet-draft - the deadline for drafts is July 7. We do apologize for the long time with not too specific info. Allison From owner-ips@ECE.cmu.edu Tue Jun 27 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13859; 26 Jun 2000 23:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA23167 for ips-outgoing; Mon, 26 Jun 2000 22:13:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA23163 for ; Mon, 26 Jun 2000 22:13:03 -0400 (EDT) Received: from rtl.rose.hp.com (rtl.rose.hp.com [15.43.208.243]) by palrel1.hp.com (Postfix) with ESMTP id 29C582D4 for ; Mon, 26 Jun 2000 19:13:02 -0700 (PDT) Received: from agilent.com ([15.3.252.210]) by rtl.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id TAA21577 for ; Mon, 26 Jun 2000 19:12:59 -0700 (PDT) Message-ID: <3957791B.C8CAF987@agilent.com> Date: Mon, 26 Jun 2000 17:39:07 +0200 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.72 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: summary of iSCSI meeting 22 June 2000 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > For all those concerned about the recovery discussion here are some > clarifications: > > 1. The whole discussion thread was related to an attempt to recover from > one TCP connection failure > in a session that has multiple TCP connections in order to fully exploit > the fault tolerance level users are expecting when using several > connections Agreed. > 2. As we are aware that stateful devices and operation idempotency are hard > to handle in general terms > we are currently contemplating mostly recovery mechanisms that are > "target-centric" (mostly target initiated). That's not the way I understand it. Targets are (generically) dumb devices and only do what they're told. It is the initiator's responsibility to recover the device/connection. Not the target's responsibility to recover the initiator. This is the way FCP-2 performs error recovery. The only thing the target does is "logout" an initiator if a master timeout expires (see RR_TOV). -Matt > Obviously we would love to be able to recreate a TCP connection > in exactly the state it got lost > but we are not aware of any such magic being available... > > Julo > > Julian Satran - IBM Research Laboratory at Haifa > > David Robinson on 22/06/2000 20:30:05 > > Please respond to David Robinson > > To: Kalman Meth/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu, scsi-tcp@external.cisco.com (bcc: Julian > Satran/Haifa/IBM) > Subject: Re: summary of iSCSI meeting 22 June 2000 > > meth@il.ibm.com wrote: > > > Further discussion of what happens when TCP packets get lost, especially > if > > they contain an iSCSI header. > > How well can iSCSI compete with FC if we are so dependent on TCP, with > its > > dropped packets. > > > > In the LAN, TCP packets are not generally lost and we should be > comparable > > to FC. > > Over WAN, can have packet loss and resulting complications, but that is > no > > longer competing with FC > > (which doesn't exist at all in the WAN). > > Huh? TCP packets can never get lost, you either get the packet > or the connection is dropped. There may be some delay as TCP > performs a retransmission which will be rare on LANs and not > so rare on WANs. I don't see how this is a FC vs TCP issue. > > -David From owner-ips@ECE.cmu.edu Tue Jun 27 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14095; 27 Jun 2000 1:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA25348 for ips-outgoing; Tue, 27 Jun 2000 00:11:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA25340 for ; Tue, 27 Jun 2000 00:11:34 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id GAA38702 for ; Tue, 27 Jun 2000 06:10:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id GAA20378 for ; Tue, 27 Jun 2000 06:11:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690B.0016FA79 ; Tue, 27 Jun 2000 06:10:59 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 27 Jun 2000 07:04:44 +0300 Subject: Re: another issue with error recovery... Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk As far as I know that is a recognized issue in the TCP world. Let's hope it will we fixed. Julo Matt Wakeley on 26/06/2000 21:03:43 Please respond to Matt Wakeley To: SCSI-TCP cc: (bcc: Julian Satran/Haifa/IBM) Subject: another issue with error recovery... Here is another issue that may affect error recovery... Any comments on this? -Matt IMAP://cassin.rose.hp.com?fetch>UID>/INBOX>44400&part=1.2 Path: news.cup.hp.com!fc.hp.com!col.hp.com!agilent.com!sdd.hp.com!enews.sgi.com!nntp.primenet.com!nntp.gctr.net!newsfeed.direct.ca!news.he.net!newsfeed.concentric.net!global-news-master From: "Eric A. Hall" Newsgroups: comp.protocols.tcp-ip Subject: Re: Sequence number of TCP. Date: 14 Jun 2000 14:27:11 EDT Organization: EHS Company Message-ID: <3947CE7D.E3949D43@ehsco.com> References: <0eb98234.3d28ff1a@usw-ex0102-016.remarq.com> <8i7gs0$17f$1@s2.feed.news.oleane.net> <8i8fq0$ppi$1@web1.cup.hp.com> NNTP-Posting-Host: 209.31.7.42 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en Xref: news.cup.hp.com comp.protocols.tcp-ip:11799 > How well that happens today when one uses a gigabit interface without > enabling the large window extensions I'm not really sure... sending > data at roughly 900mbit/s is about 88 MB/s into a 4096 MB sequence > space, or wrap in ~46 seconds if I've done my math correctly. The theoretical wrap-time with Gigabit Ethernet is ~17 seconds. I haven't done this math in a long time so don't ask to see my work :) It's also a theoretical boundary and not a practical one. Yet. Terrabit and Petabit networks will change this again. Giving a fast enough frame rate it's probably going to get so that window wrap-around happens in the millisecond range. This is only a few years away at best. -- Eric A. Hall http://www.ehsco.com/ Internet Core Protocols http://www.oreilly.com/catalog/coreprot/ From owner-ips@ECE.cmu.edu Tue Jun 27 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14097; 27 Jun 2000 1:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA25347 for ips-outgoing; Tue, 27 Jun 2000 00:11:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA25339 for ; Tue, 27 Jun 2000 00:11:34 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id GAA150536 for ; Tue, 27 Jun 2000 06:10:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id GAA20372 for ; Tue, 27 Jun 2000 06:11:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690B.0016FAB4 ; Tue, 27 Jun 2000 06:10:59 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 27 Jun 2000 07:02:22 +0300 Subject: Re: summary of iSCSI meeting 22 June 2000 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, You are correct on the connection recovery. The initiator will do it. However data recovery, for data that has been discarded voluntarily or not and command recovery are up to the target iSCSI (and to do it the target can't be completely dump). If the target is completely dumb the only thing it can do is notify). Julo Matt Wakeley on 26/06/2000 18:39:07 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: summary of iSCSI meeting 22 June 2000 julian_satran@il.ibm.com wrote: > For all those concerned about the recovery discussion here are some > clarifications: > > 1. The whole discussion thread was related to an attempt to recover from > one TCP connection failure > in a session that has multiple TCP connections in order to fully exploit > the fault tolerance level users are expecting when using several > connections Agreed. > 2. As we are aware that stateful devices and operation idempotency are hard > to handle in general terms > we are currently contemplating mostly recovery mechanisms that are > "target-centric" (mostly target initiated). That's not the way I understand it. Targets are (generically) dumb devices and only do what they're told. It is the initiator's responsibility to recover the device/connection. Not the target's responsibility to recover the initiator. This is the way FCP-2 performs error recovery. The only thing the target does is "logout" an initiator if a master timeout expires (see RR_TOV). -Matt > Obviously we would love to be able to recreate a TCP connection > in exactly the state it got lost > but we are not aware of any such magic being available... > > Julo > > Julian Satran - IBM Research Laboratory at Haifa > > David Robinson on 22/06/2000 20:30:05 > > Please respond to David Robinson > > To: Kalman Meth/Haifa/IBM@IBMIL > cc: ips@ece.cmu.edu, scsi-tcp@external.cisco.com (bcc: Julian > Satran/Haifa/IBM) > Subject: Re: summary of iSCSI meeting 22 June 2000 > > meth@il.ibm.com wrote: > > > Further discussion of what happens when TCP packets get lost, especially > if > > they contain an iSCSI header. > > How well can iSCSI compete with FC if we are so dependent on TCP, with > its > > dropped packets. > > > > In the LAN, TCP packets are not generally lost and we should be > comparable > > to FC. > > Over WAN, can have packet loss and resulting complications, but that is > no > > longer competing with FC > > (which doesn't exist at all in the WAN). > > Huh? TCP packets can never get lost, you either get the packet > or the connection is dropped. There may be some delay as TCP > performs a retransmission which will be rare on LANs and not > so rare on WANs. I don't see how this is a FC vs TCP issue. > > -David From owner-ips@ECE.cmu.edu Tue Jun 27 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14776; 27 Jun 2000 6:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA29241 for ips-outgoing; Tue, 27 Jun 2000 04:53:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA29236 for ; Tue, 27 Jun 2000 04:53:09 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA292108; Tue, 27 Jun 2000 10:49:51 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id KAA224522; Tue, 27 Jun 2000 10:50:01 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690B.0030843F ; Tue, 27 Jun 2000 10:49:55 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu, scsi-tcp@external.cisco.com Message-ID: Date: Tue, 27 Jun 2000 11:49:42 +0300 Subject: single vs multiple channels for iSCSI commands Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Proposal to support single Control Channel with multiple Data Channels in the iSCSI protocol. by Kalman Meth 27 June 2000 In our discussions on the iSCSI protocol, we came to the conclusion that we needed to send data over multiple channels in order to make best use of the available network resources. We also were inclined to have all of the channels acting in a symmetric manner so as to simplify the protocol by not having to deal differently with some channels. This allows vendors to introduce uniform iSCSI NICs for all of the network connections that will be exploited by iSCSI. We decided on allowing commands to be sent over any of the multiple connections, with the command's data and status being sent in the same channel that was used to issue the command. The use of multiple channels to pass commands introduced a complication of servicing the commands on the receiving end in the original order that the commands were issued. We had a further complication when one of the connections failed; how do we determine which command got lost on a broken connection, and what actions are required to recover from the failed connection. The solution we found to these problems (introducing a Command Reference Number and placing the commands back in order on the receiver's end) introduced flow control problems, such as maintaining a window on commands to ensure that we don't overrun the reference count, and that we don't block up all of the channels just because one channel failed and its lost command causes us to fill up the command queue on the target (while we wait for the lost command to arrive). I would like us to go back and consider a variation of the model we originally proposed with one Command Channel and multiple Data Channels. Some ideas that came up during our discussions are included below and also apply to the symmetric model. Session establishment: as in existing draft. Naming: as in existing draft with adjustments from design discussions. security: as decided in design discussions. (0) none (1) challenge/response (2) IPSec or SSL Normal case: An iSCSI session between and initiator and a target consists of a number of TCP connections. Each TCP connection between initiator and target requires an iSCSI login. The first established connection of a session between initiator and target (numbered 0) is the Control Connection (also called Control Channel). Subsequent connections between the same initiator and target can be added to an existing session upon request of the initiator during login. These connections are numbered 1,2,3, etc, and are called Data Connections (also called Data Channels). An initiator may establish several sessions with the same target, each session having its own Control Channel and its own set of Data Channels. All SCSI commands and task management messages will go over the Control Connection. Order is maintained within a single session by virtue of all commands going through the same TCP connection. The iSCSI packets for RTT and Data may go over any of the channels. iSCSI Login must be performed on each of the connections. iSCSI Ping may be performed over any of the connections. It is recommended that large data transfers be performed on the Data Channels (rather than the Control Channel) so as to ensure that the Control Channel is always free. It is permissible, however, to establish a single connection and perform all iSCSI operations on that single channel. On a READ or WRITE command, the initiator specifies on which channel it expects to perform the data transfer. This gives the initiator and target a chance to set up buffers for DMA ahead of time. Once a data transfer for a particular SCSI command begins on a particular Data Channel, all subsequent data that is transferred for the same SCSI command is to be transferred over the same Data Channel. On RTT, the target confirms on which channel it is expecting the data transfer. An RTT request will be sent over the same channel as the expected data transfer (as was specified by the initiator). If the target decides (for whatever reason) that it wants to receive the data transfer on another channel, it sends the RTT over the Control Channel with an indication as to which Data Channel it wants to use. It is understood that this may entail a performance cost on the initiator's side to now move the data transfer to another Data Channel (which may be another NIC, thus requiring DMA to be set up all over again). A target will usually change the connection for a data transfer only in case of some problem it has with the originally specified connection (unresponsive connection, or couldn't handle large amount of data on specified connection, etc). Commands may be sent with immediate data (in the Control Channel) if the immediate data is small (say less than 8K), thereby avoiding the need to later match up the data with the corresponding command. A bit in the iSCSI command header indicates that there is immediate data. An initiator may also send unsolicited data (no RTT) over the Data Channels, in case the initiator and target have agreed (during login on the Control Channel) to not use RTT. The initiator and target may renegotiate the use (or non-use) of RTT between commands, using an iSCSI Text command. The initiator sends the request to the target and does not send any other commands to the target until the target has responded. The change in using RTT will take affect with the command following the response of the target. The status of a READ command is sent with the last data packet, thus allowing hardware implementations to perform a single interrupt when the entire data transfer has completed. Similarly, a flag in a data packet sent from initiator to target indicates the last data buffer in an unsolicited WRITE operation. If the initiator sends unsolicited data for a WRITE operation (i.e. without an RTT) over one of the Data Channels, it is possible that the data will arrive before the command arrived on the Control Channel. It is also possible that the target will not have enough buffers to receive the unsolicited data. The target has the option of placing the unsolicited data in reserve buffers or of completely discarding the data. If the target discards the data, the target will later issue an RTT to instruct the initiator to resend the data. Multiple iSCSI NICs: One argument to support the symmetric model was to allow having identical iSCSI NICs to handle all iSCSI connections. In the symmetric model, since all channels look alike, all of the (identical) NICs can be fully utilized. We argue that even in the model with one Control connection and many Data Connections that we can still utilize the NICs to their maximum. The main operations to be implemented by iSCSI NICs will be to send data packets and RTTs. Data Channels can be spread across these iSCSI NICs. The less frequent iSCSI operations (and especially recovery) can be performed in software in a device driver. Note also that a Control Channel and a Data Channel can go over the same wire (NIC) even if they are different TCP connections. In order to handle additional iSCSI operations in hardware, vendors can introduce fancier NICs that also handle some other iSCSI operations. A target may use one NIC to handle the Control channel from one initiator, and another NIC to handle the Control channel from another initiator. Thus, even if all NICs can handle the entire iSCSI set, they can still be utilized to the maximum by using each NIC for the Control Channel of a different session. Similarly, if an initiator has devices on several targets, it can use each NIC to handle the Control Connection of a different session. An initiator can also open multiple sessions with the same target using a different NIC for the Control Channel of the different sessions. Recovery: An initiator must hold on to data it has sent via a WRITE operation until it has received the status for the corresponding command. Even if the initiator sends immediate data (in the Control Channel) or unsolicited data (in one of the Data Channels), the target may discard the data in case it didn't have the resources to handle the data at that instant. The target may then request that the data be resent with an RTT. A target need not keep a copy of the data buffers it has sent, if such data can be regenerated from the storage device. However, the target must keep around the status information until it has been acknowledged by the initiator. The initiator sends Status Ack info (a new iSCSI message type) over the Control Channel. If strict ordering between commands is needed (such as reading and writing of the same device) then the application must perform the proper synchronization by not issuing the second command until it has received the status of the first command (as in linked commands). If it seems that a connection has stopped functioning, then either the initiator or the target may issue an iSCSI Ping command to determine if the connection is still alive. (A bit in the Ping header determines which side initiated the ping operation.) If the Ping operation times out, then it may be assumed that the connection is not functioning properly. When a Ping operation fails, the connection should immediately be closed. Note: It is not required to support iSCSI level recovery. It is sufficient for the initiator to report failure for the commands that did not complete and let the upper layer protocol handle the recovery. In this case, all channels of the session should be closed, all data structures should be cleaned up, and a new session may be established between the initiator and target. There is an advanced recovery mechanism that MAY be implemented by the initiator and target, as described below. Data that was sent over a failed Data Connection will have to be resent over another Data Connection. On a WRITE operation, the target will eventually issue an RTT over the Control Connection to inform the initiator as to which other Data Connection to use. (Is it OK to wait for the target to figure out that the connection is down? Can the initiator somehow bring this to the target's attention?) On a READ operation, the initiator will indicate to the target which data it wants resent from the failed data transfer. This is done using an RTT (sent from the initiator to the target over the Control Channel) to resend the data from some preivous READ operation. (This is the only time an RTT is sent from the initiator to a target.) (Since the status of that READ operation did not arrive at the initiator and it was never acknowledged, the target will have kept the relevant information about the corresponding command.) If the Control Channel stops working properly (agin, determined by a time out on an iSCSI Ping operation) then the initiator must know which commands made it to the target and were not lost, which commands were completed whose status got lost, and which commands never made it to the target. Upon setting up a new session (by establishing a Control Channel), the initiator may specify whether it is in fact starting a new session or taking over an existing session. When taking over an existing session, the initiator must specify the identifiers of the session to be taken over. The target then stops transmitting on the old Control Channel, and transfers all of the old session resources to the new Control Channel. The target returns to the initiator the Initiator Task Tag of the last command it received on the old Control Channel. For each command that was sent to the target before the specified Initiator Task Tag, the initiator queries (new iSCSI Query command) about the status of that command. (The information about the status of those commands will not have been discarded since the target never received an Ack about them from the initiator.) Some of the commands may have had incomplete data transfers (use special iSCSI status code), and the target and initiator will re-issue RTTs to recover the data from those commands. Once the initiator has received (and acknowledged) the status of all pending commands, the initiator sends an iSCSI Sync message to the target to inform it that they are back in sync, and that all commands before the specified Initiator Task Tag have been satisfactorily accounted for. From owner-ips@ECE.cmu.edu Tue Jun 27 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15238; 27 Jun 2000 11:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA22753 for ips-outgoing; Tue, 27 Jun 2000 08:46:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pecos2.iphase.com (firewall-user@pecos2.iphase.com [157.175.3.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA22748 for ; Tue, 27 Jun 2000 08:46:06 -0400 (EDT) Received: by pecos2.iphase.com; id HAA06303; Tue, 27 Jun 2000 07:56:20 -0500 (CDT) Received: from nodnsquery(157.175.85.9) by pecos2.iphase.com via smap (V5.0) id xma006298; Tue, 27 Jun 00 07:55:50 -0500 Received: from localhost (kquick@localhost) by hera.Iphase.COM (8.8.8+Sun/8.8.7) with SMTP id HAA12537; Tue, 27 Jun 2000 07:45:34 -0500 (CDT) Date: Tue, 27 Jun 2000 07:45:34 -0500 (CDT) From: Kevin Quick X-Sender: kquick@hera To: meth@il.ibm.com cc: ips@ece.cmu.edu, scsi-tcp@external.cisco.com Subject: Re: single vs multiple channels for iSCSI commands In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk I wanted to respond with a few concerns I had regarding the following proposal, but I must begin with two warnings: (1) I did not attend the meeting in Haifa, so I may be unaware of substansive discussions in this area, and (2) I am leaving on vacation shortly and will be unable to sustain a dialog based on my comments until I return on July 10th. That said, my primary concern is that the focus on an iSCSI-enabled NIC seems to be diverging from "legacy" implementation focus. More precisely, the focus of this Proposal and related discussions is in identifying physical paths on the basis of a TCP connection to allow iSCSI-NIC offloads at the TCP-level, but this path/virtual-circuit association is not established or required by normal TCP functionality. Stated another way, it is entirely possible for a single TCP connection to be distributed across multiple physical NICs in existing networking implementations. As a result of this, I see the following constraints, summarized here and also noted in the Proposal text below: 1) In the iSCSI work it is imperative that a Channel be a virtual entity, tightly bound to the TCP virtual circuit concept. A Channel should not be confused with a physical connection. 2) The specification should enable offloading and traffic distribution, but not require it. Each end must be capable of operating in full legacy mode, regardless of the configuration of the other end. a) It is not possible for the recipient to pre-configure DMA buffers for a Data Channel transfer (as stated in the Proposal) unless that DMA configuration is applicable across all recipient NICs or duplicated across those NICs. MAC-level load balancing will insure that the packets comprising a Data Transfer are spread across receiver ports. b) Recipient offload functionality will be limited by this effect. The recipient host system (/embedded controller/whatever) will need to be involved to successfully collate all the information needed to reconstruct and manage a Data Transfer. 3) Whilst a separate Control Channel sounds like a good idea from the perspective of preventing data transfers from perturbing management capabilities, I have become less convinced of the usefulness on further inspection, and I can forsee a large number of problems for the recipient due to this separation (note that the recipient is the target in a WRITE or the initiator in a READ). a) What happens to the recipient from the Data perspective if a Command is sent over the Control Channel, subsequently cancelled via the Control Channel, and then a new Command is issued? The recipient must have some way of determining if the Data (read from a separate TCP stream on a separate controller with separate buffering and data servicing) is associated with the cancelled Command or the new Command. A CRN, Exchange ID, or similar identification tag will help, but not entirely. Imagining multiple Gigabit Ethernet connections and keeping in mind a target of 30000 - 40000 SCSI operations per second and the aforementioned Gigabit Ethernet wraparound duration of just a few seconds, it's very conceivable that lots and lots of management operations have happened before the data manages to work it's way into the processor's attention. Fibre Channel can deal with most of this (although there are still issues and active, current discussions on FC reflectors about those issues) because it has a tightly-coupled low-latency configuration. It's usually very possible for the initiator to know whether a target has a SCSI operation or is in the process of a Data Transfer on behalf of that specific SCSI operation, so cancelling or otherwise affecting that SCSI operation is fairly deterministic. For IP, and especially WAN considerations, with large NIC and socket buffers, the hysteresis window becomes very large and it becomes very difficult to deterministically control individual SCSI operations. b) I'm not sure the Command/Data channel focus addresses the correct issues. In an implementation, an iSCSI packet should not sit for a long period of time on inbound NIC queues or sockets unless the receiving host processor is becoming overloaded. If the receiver is becoming overloaded, expediting Command operation handling will probably only increase the overload (i.e. the recipient will get further and further behind). Instead, standard network flow control seems appropriate to indicate the recipient's overload to the originator. A Task Management Channel does seem appropriate to allow operations like Target Reset and such to be performed independently of command processing, with expedited handling of Task Management Channel communications. This is fairly infrequent traffic, however. I'll be glad to address any responses when I return on the 10th of July. Regards, Kevin ________________________________________________________________________ Kevin Quick Interphase Product Development Project UDI kquick@iphase.com Dallas, Texas Chairman +1 214 654 5173 www.iphase.com www.projectudi.org On Tue, 27 Jun 2000 meth@il.ibm.com wrote: : Date: Tue, 27 Jun 2000 11:49:42 +0300 : From: meth@il.ibm.com : To: ips@ece.cmu.edu, scsi-tcp@external.cisco.com : Subject: single vs multiple channels for iSCSI commands : : : : : : Proposal to support single Control Channel with multiple Data Channels : in the iSCSI protocol. : : by Kalman Meth : 27 June 2000 : : In our discussions on the iSCSI protocol, we came to the conclusion that : we needed to send data over multiple channels in order to make best use : of the available network resources. We also were inclined to : have all of the channels acting in a symmetric manner so as to simplify : the protocol by not having to deal differently with some channels. : This allows vendors to introduce uniform iSCSI NICs for all of the : network connections that will be exploited by iSCSI. : : We decided on allowing commands to be sent over any of the multiple : connections, with the command's data and status being sent in the same : channel that was used to issue the command. I like this model. Each command is wholly contained within a Channel, therefore ordering and management sequencing is preserved. I'm not sure if this was discussed in Haifa, but I would think that the best focus would be to associate a Channel with a LU. All SCSI operations to that LU must be performed over that Channel; operations for other LU's might share this channel or use a unique channel. This preserves command ordering for the target LU. : The use of multiple channels to pass commands introduced a complication : of servicing the commands on the receiving end in the original order : that the commands were issued. We had a further complication when one : of the connections failed; how do we determine which command got lost : on a broken connection, and what actions are required to recover from : the failed connection. The solution we found to these problems : (introducing a Command Reference Number and placing the commands back : in order on the receiver's end) introduced flow control problems, : such as maintaining a window on commands to ensure that we don't overrun : the reference count, and that we don't block up all of the channels : just because one channel failed and its lost command causes us to fill : up the command queue on the target (while we wait for the lost command : to arrive). : : I would like us to go back and consider a variation of the model we : originally proposed with one Command Channel and multiple Data Channels. : Some ideas that came up during our discussions are included below and : also apply to the symmetric model. : : Session establishment: as in existing draft. : Naming: as in existing draft with adjustments from design discussions. : security: as decided in design discussions. : (0) none : (1) challenge/response : (2) IPSec or SSL : : : : Normal case: : : An iSCSI session between and initiator and a target consists of a : number of TCP connections. Each TCP connection between initiator and : target requires an iSCSI login. The first established connection of a : session between initiator and target (numbered 0) is the Control : Connection (also called Control Channel). : Subsequent connections between the same initiator and target can be : added to an existing session upon request of the initiator during login. : These connections are numbered 1,2,3, etc, and are called Data : Connections (also called Data Channels). : An initiator may establish several sessions with the same target, each : session having its own Control Channel and its own set of Data Channels. : : All SCSI commands and task management messages will go over the Control : Connection. Order is maintained within a single session by virtue of all : commands going through the same TCP connection. : The iSCSI packets for RTT and Data may go over any of the channels. : iSCSI Login must be performed on each of the connections. : iSCSI Ping may be performed over any of the connections. : : It is recommended that large data transfers be performed on the Data : Channels (rather than the Control Channel) so as to ensure that the : Control Channel is always free. It is permissible, however, to : establish a single connection and perform all iSCSI operations on that : single channel. : : On a READ or WRITE command, the initiator specifies on which channel it : expects to perform the data transfer. This gives the initiator and : target a chance to set up buffers for DMA ahead of time. Legacy networking considerations would seem to prevent this unless the DMA setup was replicated across all recipient NICs or unless the platform DMA characteristics allow a single mapping for multiple NICs. : Once a data transfer for a particular SCSI command begins on a : particular Data Channel, all subsequent data that is transferred for the : same SCSI command is to be transferred over the same Data Channel. For this model: why? As long as the response isn't sent until all the data is confirmed as received, this doesn't seem necessary as a requirement, although it may be desirable as an option (e.g. for an iSCSI-NIC). : On RTT, the target confirms on which channel it is expecting the data : transfer. An RTT request will be sent over the same channel as the : expected data transfer (as was specified by the initiator). In a READ operation, I would think the initiator would want to control the data channel use. : If the target decides (for whatever reason) that it wants to receive the : data transfer on another channel, it sends the RTT over the Control : Channel with an indication as to which Data Channel it wants to use. : It is understood that this may entail a performance : cost on the initiator's side to now move the data transfer to another : Data Channel (which may be another NIC, thus requiring DMA to be set : up all over again). A target will usually change the connection for : a data transfer only in case of some problem it has with the originally : specified connection (unresponsive connection, or couldn't handle : large amount of data on specified connection, etc). I'm not sure I understand the purposes for changing a connection, especially from a recipient's perspective. Generally, the sender is much more aware of any network difficulties than the recipient (designing a NIC that can't handle a large amount of data transfer is an implementation weakness IMO). The sender is usually aware of connectivity or responsiveness problems, so as long as the receiver (or the specification/protocol) *doesn't* impose any unnecessary restrictions on the data transfer, it seems like it should be the sender's prerogative to determine how the data is sent. : : Commands may be sent with immediate data (in the Control Channel) if the : immediate data is small (say less than 8K), thereby avoiding the need to : later match up the data with the corresponding command. A bit in the : iSCSI command header indicates that there is immediate data. : An initiator may also send unsolicited data (no RTT) over the Data : Channels, in case the initiator and target have agreed (during login : on the Control Channel) to not use RTT. SCSI commands have a desireable attribute of being small. They can usually be received in a single packet and receiver buffer, even for WAN. Sending an entire 8K of data along with the command imposes significant resource requirements on the recipient and is at cross-purposes with the flow control inherent in the SCSI XFER-READY phase. Again, don't forget WAN issues. : : The initiator and target may renegotiate the use (or non-use) of RTT : between commands, using an iSCSI Text command. : The initiator sends the request to the target and does not send any : other commands to the target until the target has responded. : The change in using RTT will take affect with the command following the : response of the target. I'm not sure I fully understand what an RTT is, but I don't much like the statement above. It sounds like all SCSI operations handling is stalled while this RTT renegotiation is performed. Viewing one end of the spectrum as a couple of GE connections between two machines that are about 3 feet apart and probably crunching through 30000-40000 SCSI operations/second, a full command stall sounds pretty expensive. >From the other end of the spectrum, with a 24Kbaud WAN connection halfway across the globe, anything that isn't queued and requires a complete round-trip before continuing sounds pretty expensive. Maybe this RTT renegotiation isn't too frequent, but the "between commands" text makes it sound like it is. : : The status of a READ command is sent with the last data packet, : thus allowing hardware implementations to perform a single interrupt : when the entire data transfer has completed. : Similarly, a flag in a data packet sent from initiator to target : indicates the last data buffer in an unsolicited WRITE operation. SCSI is a command/response protocol, so I'm not sure what an unsolicited WRITE is. Perhaps another RTT issue that I'm not aware of? Are you doing away with the SCSI Status packet? I would think that an iSCSI-NIC would be capable of generating a single interrupt on receipt of the Status packet, regardless of the number of intervening data packets. This is how most Fibre Channel cards operate. : If the initiator sends unsolicited data for a WRITE operation : (i.e. without an RTT) over one of the Data Channels, it is possible : that the data will arrive before the command arrived on the Control : Channel. It is also possible that the target will not have enough : buffers to receive the unsolicited data. The target has the option of : placing the unsolicited data in reserve buffers or of completely : discarding the data. If the target discards the data, the target will : later issue an RTT to instruct the initiator to resend the data. OK, I'm surmising that RTT = Ready_To_Transfer and is different from SCSI's XFER_RDY operation in that RTT applies across multiple commands (requiring command stalling as above). I'm still not clear as to why RTT would be issued except as a response to a Command. An unsolicited WRITE sounds like a SCSI Write with "Auto-XFER_RDY" mode, which is much easier to handle (and grant) from a target's perspective if the data follows the command in the same channel. If separate channels occur, you have the possible data-lead problem you noted above... to me this is yet another complication of separating Command and Data channels. : : : Multiple iSCSI NICs: : : One argument to support the symmetric model was to allow having : identical iSCSI NICs to handle all iSCSI connections. In the : symmetric model, since all channels look alike, all of the (identical) : NICs can be fully utilized. : : We argue that even in the model with one Control connection and many : Data Connections that we can still utilize the NICs to their maximum. : : The main operations to be implemented by iSCSI NICs will be to send : data packets and RTTs. Data Channels can be spread across these iSCSI : NICs. The less frequent iSCSI operations (and especially recovery) : can be performed in software in a device driver. : Note also that a Control Channel and a Data Channel can : go over the same wire (NIC) even if they are different TCP connections. : In order to handle additional iSCSI operations in hardware, : vendors can introduce fancier NICs that also handle some other iSCSI : operations. : : A target may use one NIC to handle the Control channel from one : initiator, and another NIC to handle the Control channel from another : initiator. Thus, even if all NICs can handle the entire iSCSI set, I'm not sure how the target would be capable of directing initiators in this way. : they can still be utilized to the maximum by using each NIC for the : Control Channel of a different session. Similarly, if an initiator has : devices on several targets, it can use each NIC to handle the Control : Connection of a different session. : An initiator can also open multiple sessions with the same target : using a different NIC for the Control Channel of the different sessions. Your method would require NIC-to-NIC communications to complete individual iSCSI operations if full host offload is desired. OTOH, because of MAC-level load-balancing causing inbound traffic distribution across multiple NICs, offloading in a system with multiple NICs is problematic under either model. : : Recovery: : : An initiator must hold on to data it has sent via a WRITE operation : until it has received the status for the corresponding command. : Even if the initiator sends immediate data (in the Control Channel) or : unsolicited data (in one of the Data Channels), the target may discard : the data in case it didn't have the resources to handle the data at that : instant. The target may then request that the data be resent with an : RTT. : A target need not keep a copy of the data buffers it has sent, if : such data can be regenerated from the storage device. : However, the target must keep around the status information until it has : been acknowledged by the initiator. The initiator sends Status Ack info : (a new iSCSI message type) over the Control Channel. : If strict ordering between commands is needed (such as reading and : writing of the same device) then the application must perform the : proper synchronization by not issuing the second command until it has : received the status of the first command (as in linked commands). Changing heuristics for applications will cause compatibility issues. : : If it seems that a connection has stopped functioning, then either : the initiator or the target may issue an iSCSI Ping command to determine : if the connection is still alive. (A bit in the Ping header determines : which side initiated the ping operation.) If the Ping operation times : out, then it may be assumed that the connection is not functioning : properly. When a Ping operation fails, the connection should immediately : be closed. : : Note: It is not required to support iSCSI level recovery. : It is sufficient for the initiator to report failure for the commands : that did not complete and let the upper layer protocol handle the : recovery. : In this case, all channels of the session should be closed, all data : structures should be cleaned up, and a new session may be established : between the initiator and target. : : There is an advanced recovery mechanism that MAY be implemented by : the initiator and target, as described below. [elided] From owner-ips@ECE.cmu.edu Tue Jun 27 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15292; 27 Jun 2000 11:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA25877 for ips-outgoing; Tue, 27 Jun 2000 10:25:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA25864 for ; Tue, 27 Jun 2000 10:25:26 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id C9CD5159F for ; Tue, 27 Jun 2000 07:25:25 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA07397 for ; Tue, 27 Jun 2000 07:34:32 -0700 (PDT) Message-Id: <4.2.2.20000627065052.00ae08f0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 27 Jun 2000 06:58:34 -0700 To: ips@ece.cmu.edu From: Michael Krause Subject: Re: another issue with error recovery... In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:04 AM 6/27/00 +0300, julian_satran@il.ibm.com wrote: >As far as I know that is a recognized issue in the TCP world. Let's hope >it will we fixed. The same problem exists in other technologies as well, e.g. InfiniBand. At 3 GB/s, the sequence number space gets used up in a couple hundred milliseconds using the smallest packet size. However, is this really a realistic problem for either technology? If one is only pounding on a single connection and has an application that can generate the workload, then it is a problem and one needs to expand the sliding window to much larger values (e.g. a TCP jumbo frame). However, if one is using multiple connections sharing the same physical links, then the problem is further mitigated because there are multiple windows being operated upon and neither is being consumed as quickly. Note: The problem will only get worse. 10 GbE will be here second half of next year; 100 GbE will be here 5+ years after that. All of the transports including those still be developed, will not work well at those operating speeds. One would need to expand the windows (however implemented) to accommodate much larger values - measured in GBs instead of MBs - to have any measure of success. Mike From owner-ips@ECE.cmu.edu Tue Jun 27 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15297; 27 Jun 2000 11:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA25888 for ips-outgoing; Tue, 27 Jun 2000 10:25:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA25879 for ; Tue, 27 Jun 2000 10:25:38 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 19A6618AF; Tue, 27 Jun 2000 07:25:37 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA07408; Tue, 27 Jun 2000 07:34:47 -0700 (PDT) Message-Id: <4.2.2.20000627071046.00aec6e0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 27 Jun 2000 07:20:49 -0700 To: JoeBre@exabyte.com, meth@il.ibm.com, ips@ece.cmu.edu, scsi-tcp@external.cisco.com From: Michael Krause Subject: RE: summary of iSCSI meeting 22 June 2000 In-Reply-To: <118A72FD3F02D211AD1000805F9AA58B012B7A80@boulderp03.Exabyt e.COM> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_3973434==_.ALT" Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_3973434==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed At 10:51 AM 6/22/00 -0600, JoeBre@exabyte.com wrote: >Comments inline > > > ... > > > > Further discussion of what happens when TCP packets get lost, > > especially if > > they contain an iSCSI header. > > How well can iSCSI compete with FC if we are so dependent on > > TCP, with its > > dropped packets. > >All error handling for dropped packets should exist below the SCSI layer. >If SCSI would be required to be involved in error handling for dropped >packets, it would be disastrous for SCSI Stream Devices (ex: tape backup >session). This is due to the fact that SCSI Stream Devices have stateful >behavior. iSCSI can compete quite well with TCP especially if the TCP implementation uses SACK for selective retransmissions. The fact that TCP does all of the reliability for packets (not transactions which are different) is a benefit available to any layered architecture approach - TCP can take on and solve a class of problems without impacting iSCSI architecture / device designs. Note: iSCSI should be responsible for its level of transaction recovery. For example, all transactions should be numbered with iSCSI implementing a SACK scheme. This would allow iSCSI to recover transactions that may have been lost due to a failure within the fabric, e.g. a session which is striping across multiple TCP connections may loose a TCP connection and should recover. > > In the LAN, TCP packets are not generally lost and we should be > comparable > > to FC. Over WAN, can have packet loss and resulting > complications, but that is no > > longer competing with FC (which doesn't exist at all in the WAN). > >The people working on the FC-BB specification would probably refute the >statement that FC does not exist at all in the WAN. The general Internet has packet loss in the 1-in-100 range today - it is getting better as higher bandwidth backbones are used to accommodate the increased workload (most packet drops are due to congestion). For a dark fiber solution which is properly provisioned, one can insure that congestion is a rare event and thus the packet loss will be very low. There are many dark fiber providers in the world delivering solutions today so one can deploy a system (many companies are doing just this) without major problems. It should be noted that many companies and backbone providers will be moving to OC192 and deploying DWDM with the ability to deliver Tbps over the next couple of years. So, perhaps the congestion problem will really go away and one will be left with the much harder problems to worry about. Mike --=====================_3973434==_.ALT Content-Type: text/html; charset="us-ascii" At 10:51 AM 6/22/00 -0600, JoeBre@exabyte.com wrote:

Comments inline

> ...
>
> Further discussion of what happens when TCP packets get lost,
> especially if
>  they contain an iSCSI header.
> How well can iSCSI compete with FC if we are so dependent on
> TCP, with its
> dropped packets.

All error handling for dropped packets should exist below the SCSI layer. If SCSI would be required to be involved in error handling for dropped packets, it would be disastrous for SCSI Stream Devices (ex: tape backup session). This is due to the fact that SCSI Stream Devices have stateful behavior.

iSCSI can compete quite well with TCP especially if the TCP implementation uses SACK for selective retransmissions.  The fact that TCP does all of the reliability for packets (not transactions which are different) is a benefit available to any layered architecture approach - TCP can take on and solve a class of problems without impacting iSCSI architecture / device designs.

Note: iSCSI should be responsible for its level of transaction recovery.   For example, all transactions should be numbered with iSCSI implementing a SACK scheme.  This would allow iSCSI to recover transactions that may have been lost due to a failure within the fabric, e.g. a session which is striping across multiple TCP connections may loose a TCP connection and should recover. 

> In the LAN, TCP packets are not generally lost and we should  be comparable
> to FC. Over WAN, can have packet loss and resulting complications,  but that is no
> longer competing with FC (which doesn't exist at all in the WAN).

The people working on the FC-BB specification would probably refute the statement that FC does not exist at all in the WAN.

The general Internet has packet loss in the 1-in-100 range today - it is getting better as higher bandwidth backbones are used to accommodate the increased workload (most packet drops are due to congestion).  For a dark fiber solution which is properly provisioned, one can insure that congestion is a rare event and thus the packet loss will be very low.  There are many dark fiber providers in the world delivering solutions today so one can deploy a system (many companies are doing just this) without major problems.  It should be noted that many companies and backbone providers will be moving to OC192 and deploying DWDM with the ability to deliver Tbps over the next couple of years.  So, perhaps the congestion problem will really go away and one will be left with the much harder problems to worry about.

Mike --=====================_3973434==_.ALT-- From owner-ips@ECE.cmu.edu Tue Jun 27 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16044; 27 Jun 2000 19:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA11530 for ips-outgoing; Tue, 27 Jun 2000 18:07:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA11521 for ; Tue, 27 Jun 2000 18:07:41 -0400 (EDT) Received: from rtl.rose.hp.com (rtl.rose.hp.com [15.43.208.243]) by atlrel2.hp.com (Postfix) with ESMTP id 0190F981 for ; Tue, 27 Jun 2000 18:07:40 -0400 (EDT) Received: from agilent.com (ros54259wak.rose.hp.com [15.43.208.219]) by rtl.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id PAA16036 for ; Tue, 27 Jun 2000 15:07:40 -0700 (PDT) Message-ID: <39592002.B0D6E60B@agilent.com> Date: Tue, 27 Jun 2000 14:43:30 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.72 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: summary of iSCSI meeting 22 June 2000 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > Matt, > > You are correct on the connection recovery. The initiator will do it. > However data recovery, for data that has been discarded voluntarily or not > and command > recovery are up to the target iSCSI (and to do it the target can't be > completely dump). > If the target is completely dumb the only thing it can do is notify). Julian, I don't see why the target is to perform command and data recovery. The initiator knows what commands it sent. It should ask the target what commands it received. This is how FCP-2 performs error recovery. Maybe I'm just missing something, so please explain what you have in mind. -Matt > > > Julo > > Matt Wakeley on 26/06/2000 18:39:07 > > Please respond to Matt Wakeley > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: summary of iSCSI meeting 22 June 2000 > > julian_satran@il.ibm.com wrote: > > > For all those concerned about the recovery discussion here are some > > clarifications: > > > > 1. The whole discussion thread was related to an attempt to recover from > > one TCP connection failure > > in a session that has multiple TCP connections in order to fully exploit > > the fault tolerance level users are expecting when using several > > connections > > Agreed. > > > 2. As we are aware that stateful devices and operation idempotency are > hard > > to handle in general terms > > we are currently contemplating mostly recovery mechanisms that are > > "target-centric" (mostly target initiated). > > That's not the way I understand it. Targets are (generically) dumb devices > and > only do what they're told. It is the initiator's responsibility to recover > the > device/connection. Not the target's responsibility to recover the > initiator. > > This is the way FCP-2 performs error recovery. The only thing the target > does > is "logout" an initiator if a master timeout expires (see RR_TOV). > > -Matt > > > Obviously we would love to be able to recreate a TCP connection > > in exactly the state it got lost > > but we are not aware of any such magic being available... > > > > Julo > > > > Julian Satran - IBM Research Laboratory at Haifa > > > > David Robinson on 22/06/2000 20:30:05 > > > > Please respond to David Robinson > > > > To: Kalman Meth/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu, scsi-tcp@external.cisco.com (bcc: Julian > > Satran/Haifa/IBM) > > Subject: Re: summary of iSCSI meeting 22 June 2000 > > > > meth@il.ibm.com wrote: > > > > > Further discussion of what happens when TCP packets get lost, > especially > > if > > > they contain an iSCSI header. > > > How well can iSCSI compete with FC if we are so dependent on TCP, with > > its > > > dropped packets. > > > > > > In the LAN, TCP packets are not generally lost and we should be > > comparable > > > to FC. > > > Over WAN, can have packet loss and resulting complications, but that is > > no > > > longer competing with FC > > > (which doesn't exist at all in the WAN). > > > > Huh? TCP packets can never get lost, you either get the packet > > or the connection is dropped. There may be some delay as TCP > > performs a retransmission which will be rare on LANs and not > > so rare on WANs. I don't see how this is a FC vs TCP issue. > > > > -David From owner-ips@ECE.cmu.edu Wed Jun 28 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16593; 28 Jun 2000 0:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA18501 for ips-outgoing; Tue, 27 Jun 2000 23:30:30 -0400 (EDT) Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA18493 for ; Tue, 27 Jun 2000 23:30:25 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id UAA04086; Tue, 27 Jun 2000 20:30:05 -0700 (PDT) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id UAA02249; Tue, 27 Jun 2000 20:26:27 -0700 (PDT) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Tue, 27 Jun 2000 20:26:27 -0700 (PDT) From: Costa Sapuntzakis To: scsi-tcp@external.cisco.com, ips@ece.cmu.edu Subject: Re: single vs multiple channels for iSCSI commands In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk I would like to stake out a position against any approach that involves using multiple simultaneous TCP connections per iSCSI session. The design of iSCSI (and the hardware that implements it) becomes significantly simpler when there is just 1 TCP connection/session. Parallel paths in the network can still be exploited by setting up multiple sessions with multiple TCP connections. So far, I have heard of two applications that allegedly require strict ordering: tape backup and remote asynchronous mirroring. New SCSI commands or even higher-layer techniques can be used to satisfy these applications. For example, I have heard that some high-bandwidth tape applications blast self-describing blocks of data to tape, making the order in which data gets read/written to tape less relevant. For the case of high bandiwdth remote asynchronous mirroring, a special SCSI remote asynchrnous mirroring command set could be introduced. I believe remote asynchronous mirroring principally consists of one command: WRITE, so the command set would probably be small. Ditching multiple simultaneous connections/session would help convince us of the correctness of iSCSI and allow us to move onto other important issues and help the time to market. -Costa App A. Possible remote asynch mirroring command sets 1) 1 CDB: ORDERED WRITE 2) 3 CDBs: WRITE UNCOMMITTED, COMMIT, ABORT From owner-ips@ECE.cmu.edu Wed Jun 28 06:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16750; 28 Jun 2000 1:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA19979 for ips-outgoing; Wed, 28 Jun 2000 00:27:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from storage.network.com (storage.network.com [129.191.1.4]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA19971 for ; Wed, 28 Jun 2000 00:26:59 -0400 (EDT) Received: from nsc-bridge.network.com (nsc-bridge.network.com [129.191.215.11]) by storage.network.com (8.9.3+Sun/8.9.3) with ESMTP id XAA17679; Tue, 27 Jun 2000 23:23:10 -0500 (CDT) Received: by nsc-bridge.network.com with Internet Mail Service (5.5.2650.21) id ; Tue, 27 Jun 2000 23:23:10 -0500 Message-ID: From: "Hughes, James P. (MRDC)" To: scsi-tcp@external.cisco.com, ips@ece.cmu.edu Subject: RE: single vs multiple channels for iSCSI commands Date: Tue, 27 Jun 2000 23:23:08 -0500 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree. One TCP connection with command and data mixed. Keep it simple. The need to go out of order is so small that we should not let this "wag the dog". In addition, we need to have centralized control of devices. That is, a tape drive needs to be given out by a controlling authority, and this needs to be stronger than just cookies. IPSEC and SSL need another layer to determine who owns a peripheral. Jim -----Original Message----- From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] Sent: Tuesday, June 27, 2000 10:26 PM To: scsi-tcp@external.cisco.com; ips@ece.cmu.edu Subject: Re: single vs multiple channels for iSCSI commands I would like to stake out a position against any approach that involves using multiple simultaneous TCP connections per iSCSI session. The design of iSCSI (and the hardware that implements it) becomes significantly simpler when there is just 1 TCP connection/session. Parallel paths in the network can still be exploited by setting up multiple sessions with multiple TCP connections. So far, I have heard of two applications that allegedly require strict ordering: tape backup and remote asynchronous mirroring. New SCSI commands or even higher-layer techniques can be used to satisfy these applications. For example, I have heard that some high-bandwidth tape applications blast self-describing blocks of data to tape, making the order in which data gets read/written to tape less relevant. For the case of high bandiwdth remote asynchronous mirroring, a special SCSI remote asynchrnous mirroring command set could be introduced. I believe remote asynchronous mirroring principally consists of one command: WRITE, so the command set would probably be small. Ditching multiple simultaneous connections/session would help convince us of the correctness of iSCSI and allow us to move onto other important issues and help the time to market. -Costa App A. Possible remote asynch mirroring command sets 1) 1 CDB: ORDERED WRITE 2) 3 CDBs: WRITE UNCOMMITTED, COMMIT, ABORT From owner-ips@ECE.cmu.edu Wed Jun 28 07:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16864; 28 Jun 2000 2:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA21307 for ips-outgoing; Wed, 28 Jun 2000 01:51:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA21303 for ; Wed, 28 Jun 2000 01:51:32 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA290044 for ; Wed, 28 Jun 2000 07:50:53 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA169816 for ; Wed, 28 Jun 2000 07:51:01 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690C.00201E3F ; Wed, 28 Jun 2000 07:50:48 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 28 Jun 2000 08:49:56 +0300 Subject: Re: summary of iSCSI meeting 22 June 2000 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk After a connection is restarted (open+login) all outstanding commands are resent. The target intterprets the "duplicates" as it needs - restarts running commands, starts new ones etc. Wait for the draft - it is already there in some incipient form! Julo Matt Wakeley on 28/06/2000 00:43:30 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: summary of iSCSI meeting 22 June 2000 julian_satran@il.ibm.com wrote: > Matt, > > You are correct on the connection recovery. The initiator will do it. > However data recovery, for data that has been discarded voluntarily or not > and command > recovery are up to the target iSCSI (and to do it the target can't be > completely dump). > If the target is completely dumb the only thing it can do is notify). Julian, I don't see why the target is to perform command and data recovery. The initiator knows what commands it sent. It should ask the target what commands it received. This is how FCP-2 performs error recovery. Maybe I'm just missing something, so please explain what you have in mind. -Matt > > > Julo > > Matt Wakeley on 26/06/2000 18:39:07 > > Please respond to Matt Wakeley > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: summary of iSCSI meeting 22 June 2000 > > julian_satran@il.ibm.com wrote: > > > For all those concerned about the recovery discussion here are some > > clarifications: > > > > 1. The whole discussion thread was related to an attempt to recover from > > one TCP connection failure > > in a session that has multiple TCP connections in order to fully exploit > > the fault tolerance level users are expecting when using several > > connections > > Agreed. > > > 2. As we are aware that stateful devices and operation idempotency are > hard > > to handle in general terms > > we are currently contemplating mostly recovery mechanisms that are > > "target-centric" (mostly target initiated). > > That's not the way I understand it. Targets are (generically) dumb devices > and > only do what they're told. It is the initiator's responsibility to recover > the > device/connection. Not the target's responsibility to recover the > initiator. > > This is the way FCP-2 performs error recovery. The only thing the target > does > is "logout" an initiator if a master timeout expires (see RR_TOV). > > -Matt > > > Obviously we would love to be able to recreate a TCP connection > > in exactly the state it got lost > > but we are not aware of any such magic being available... > > > > Julo > > > > Julian Satran - IBM Research Laboratory at Haifa > > > > David Robinson on 22/06/2000 20:30:05 > > > > Please respond to David Robinson > > > > To: Kalman Meth/Haifa/IBM@IBMIL > > cc: ips@ece.cmu.edu, scsi-tcp@external.cisco.com (bcc: Julian > > Satran/Haifa/IBM) > > Subject: Re: summary of iSCSI meeting 22 June 2000 > > > > meth@il.ibm.com wrote: > > > > > Further discussion of what happens when TCP packets get lost, > especially > > if > > > they contain an iSCSI header. > > > How well can iSCSI compete with FC if we are so dependent on TCP, with > > its > > > dropped packets. > > > > > > In the LAN, TCP packets are not generally lost and we should be > > comparable > > > to FC. > > > Over WAN, can have packet loss and resulting complications, but that is > > no > > > longer competing with FC > > > (which doesn't exist at all in the WAN). > > > > Huh? TCP packets can never get lost, you either get the packet > > or the connection is dropped. There may be some delay as TCP > > performs a retransmission which will be rare on LANs and not > > so rare on WANs. I don't see how this is a FC vs TCP issue. > > > > -David From owner-ips@ECE.cmu.edu Wed Jun 28 07:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16869; 28 Jun 2000 2:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA21663 for ips-outgoing; Wed, 28 Jun 2000 02:14:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA21659 for ; Wed, 28 Jun 2000 02:14:54 -0400 (EDT) Received: from rtl.rose.hp.com (rtl.rose.hp.com [15.43.208.243]) by palrel1.hp.com (Postfix) with ESMTP id 349BAFF; Tue, 27 Jun 2000 23:14:53 -0700 (PDT) Received: from agilent.com (wak7854259b.rose.hp.com [15.26.30.187]) by rtl.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id XAA05826; Tue, 27 Jun 2000 23:14:51 -0700 (PDT) Message-ID: <395997DA.BC51434A@agilent.com> Date: Tue, 27 Jun 2000 23:14:50 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: summary of iSCSI meeting 22 June 2000 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Ok, I'll wait, but be prepared for me to disagree with it. The initiator is the master, and the target is the slave, so the master should be the instigator of any error recovery, rather than the slave second guessing the master.... -Matt julian_satran@il.ibm.com wrote: > After a connection is restarted (open+login) all outstanding commands are > resent. > The target intterprets the "duplicates" as it needs - restarts running > commands, starts > new ones etc. Wait for the draft - it is already there in some incipient > form! > Julo > > Matt Wakeley on 28/06/2000 00:43:30 > > Please respond to Matt Wakeley > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: summary of iSCSI meeting 22 June 2000 > > julian_satran@il.ibm.com wrote: > > > Matt, > > > > You are correct on the connection recovery. The initiator will do it. > > However data recovery, for data that has been discarded voluntarily or > not > > and command > > recovery are up to the target iSCSI (and to do it the target can't be > > completely dump). > > If the target is completely dumb the only thing it can do is notify). > > Julian, > > I don't see why the target is to perform command and data recovery. > The initiator knows what commands it sent. It should ask the target what > commands it received. This is how FCP-2 performs error recovery. Maybe I'm > just missing something, so please explain what you have in mind. > > -Matt > > > > > > > Julo > > > > Matt Wakeley on 26/06/2000 18:39:07 > > > > Please respond to Matt Wakeley > > > > To: ips@ece.cmu.edu > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: Re: summary of iSCSI meeting 22 June 2000 > > > > julian_satran@il.ibm.com wrote: > > > > > For all those concerned about the recovery discussion here are some > > > clarifications: > > > > > > 1. The whole discussion thread was related to an attempt to recover > from > > > one TCP connection failure > > > in a session that has multiple TCP connections in order to fully > exploit > > > the fault tolerance level users are expecting when using several > > > connections > > > > Agreed. > > > > > 2. As we are aware that stateful devices and operation idempotency are > > hard > > > to handle in general terms > > > we are currently contemplating mostly recovery mechanisms that are > > > "target-centric" (mostly target initiated). > > > > That's not the way I understand it. Targets are (generically) dumb > devices > > and > > only do what they're told. It is the initiator's responsibility to > recover > > the > > device/connection. Not the target's responsibility to recover the > > initiator. > > > > This is the way FCP-2 performs error recovery. The only thing the target > > does > > is "logout" an initiator if a master timeout expires (see RR_TOV). > > > > -Matt > > > > > Obviously we would love to be able to recreate a TCP connection > > > in exactly the state it got lost > > > but we are not aware of any such magic being available... > > > > > > Julo > > > > > > Julian Satran - IBM Research Laboratory at Haifa > > > > > > David Robinson on 22/06/2000 20:30:05 > > > > > > Please respond to David Robinson > > > > > > To: Kalman Meth/Haifa/IBM@IBMIL > > > cc: ips@ece.cmu.edu, scsi-tcp@external.cisco.com (bcc: Julian > > > Satran/Haifa/IBM) > > > Subject: Re: summary of iSCSI meeting 22 June 2000 > > > > > > meth@il.ibm.com wrote: > > > > > > > Further discussion of what happens when TCP packets get lost, > > especially > > > if > > > > they contain an iSCSI header. > > > > How well can iSCSI compete with FC if we are so dependent on TCP, > with > > > its > > > > dropped packets. > > > > > > > > In the LAN, TCP packets are not generally lost and we should be > > > comparable > > > > to FC. > > > > Over WAN, can have packet loss and resulting complications, but that > is > > > no > > > > longer competing with FC > > > > (which doesn't exist at all in the WAN). > > > > > > Huh? TCP packets can never get lost, you either get the packet > > > or the connection is dropped. There may be some delay as TCP > > > performs a retransmission which will be rare on LANs and not > > > so rare on WANs. I don't see how this is a FC vs TCP issue. > > > > > > -David From owner-ips@ECE.cmu.edu Wed Jun 28 07:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16862; 28 Jun 2000 2:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA21081 for ips-outgoing; Wed, 28 Jun 2000 01:35:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA21077 for ; Wed, 28 Jun 2000 01:35:13 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA317242 for ; Wed, 28 Jun 2000 07:34:25 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA253850 for ; Wed, 28 Jun 2000 07:34:38 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690C.001EA1C5 ; Wed, 28 Jun 2000 07:34:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 28 Jun 2000 08:33:41 +0300 Subject: RE: summary of iSCSI meeting 22 June 2000 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with Mike. In a decently layered approach as you add new applications at the upper layers it is to be expected that lower layers will also adjust to better cope with new demands. However in the current scheme of things a message oriented approach has the distinct advantage that messages can be processed out of order while in a stream approach a packet loss can affect the whole stack. A clever TCP option (like the RDMA option forwarded by Costa) can help by enabling TCP segments to be processed independently (it will not help if IP fragments are lost and this should be avoided anyhow - I mean IP fragmentation!). And again Mike is right - iSCSI packets numbered across connections go a long way to enable recovery as you will see in the draft I sending for designer-review today. Regards, Julo Michael Krause on 27/06/2000 17:20:49 Please respond to Michael Krause To: JoeBre@exabyte.com, Kalman Meth/Haifa/IBM@IBMIL, ips@ece.cmu.edu, scsi-tcp@external.cisco.com cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: summary of iSCSI meeting 22 June 2000 At 10:51 AM 6/22/00 -0600, JoeBre@exabyte.com wrote: >Comments inline > > > ... > > > > Further discussion of what happens when TCP packets get lost, > > especially if > > they contain an iSCSI header. > > How well can iSCSI compete with FC if we are so dependent on > > TCP, with its > > dropped packets. > >All error handling for dropped packets should exist below the SCSI layer. >If SCSI would be required to be involved in error handling for dropped >packets, it would be disastrous for SCSI Stream Devices (ex: tape backup >session). This is due to the fact that SCSI Stream Devices have stateful >behavior. iSCSI can compete quite well with TCP especially if the TCP implementation uses SACK for selective retransmissions. The fact that TCP does all of the reliability for packets (not transactions which are different) is a benefit available to any layered architecture approach - TCP can take on and solve a class of problems without impacting iSCSI architecture / device designs. Note: iSCSI should be responsible for its level of transaction recovery. For example, all transactions should be numbered with iSCSI implementing a SACK scheme. This would allow iSCSI to recover transactions that may have been lost due to a failure within the fabric, e.g. a session which is striping across multiple TCP connections may loose a TCP connection and should recover. > > In the LAN, TCP packets are not generally lost and we should be > comparable > > to FC. Over WAN, can have packet loss and resulting > complications, but that is no > > longer competing with FC (which doesn't exist at all in the WAN). > >The people working on the FC-BB specification would probably refute the >statement that FC does not exist at all in the WAN. The general Internet has packet loss in the 1-in-100 range today - it is getting better as higher bandwidth backbones are used to accommodate the increased workload (most packet drops are due to congestion). For a dark fiber solution which is properly provisioned, one can insure that congestion is a rare event and thus the packet loss will be very low. There are many dark fiber providers in the world delivering solutions today so one can deploy a system (many companies are doing just this) without major problems. It should be noted that many companies and backbone providers will be moving to OC192 and deploying DWDM with the ability to deliver Tbps over the next couple of years. So, perhaps the congestion problem will really go away and one will be left with the much harder problems to worry about. Mike - att1.htm From owner-ips@ECE.cmu.edu Wed Jun 28 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17945; 28 Jun 2000 11:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA24008 for ips-outgoing; Wed, 28 Jun 2000 10:28:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA24003 for ; Wed, 28 Jun 2000 10:28:23 -0400 (EDT) Received: from rtl.rose.hp.com (rtl.rose.hp.com [15.43.208.243]) by atlrel2.hp.com (Postfix) with ESMTP id 04D60445; Wed, 28 Jun 2000 10:28:23 -0400 (EDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by rtl.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id HAA22180; Wed, 28 Jun 2000 07:28:20 -0700 (PDT) Message-ID: <395A0B83.5F6E70A4@agilent.com> Date: Wed, 28 Jun 2000 07:28:19 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: scsi-tcp@external.cisco.com, ips@ece.cmu.edu Subject: Re: single vs multiple channels for iSCSI commands References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Costa, Costa Sapuntzakis wrote: > I would like to stake out a position against any approach that involves > using multiple simultaneous TCP connections per iSCSI session. > > The design of iSCSI (and the hardware that implements it) becomes > significantly simpler when there is just 1 TCP connection/session. I disagree. An upper layer multiplexing entity receives SCSI I/Os and distributes each to a HW adaptor. A mirror image of this entity at the other end gathers the I/Os from all the adaptors and presents them to the scsi layer (in the order that the initiator received them from its scsi layer). Since multiple adaptors may be involved, it must be software that is performing these functions. > Parallel paths in the network can still be exploited by setting up > multiple sessions with multiple TCP connections. This will still need the multiplexing entity I described above, so what's the difference? In order to talk to a device using multiple paths (for increased bandwidth), multiple TCP connections must be set up and the I/Os distributed amoung them and gathered together again at the target. And if the target requires ordering, then either you can't use multiple TCP connections (and not enable higher bandwidth than a single link) or invent some protocols to enable ordering. Having multiple TCP connections per session provides the protocol to deliver to the target scsi layer the commands in the same order the initiator scsi layer issued them. > So far, I have heard of two applications that allegedly require strict > ordering: tape backup and remote asynchronous mirroring. New SCSI commands > or even higher-layer techniques can be used to satisfy these applications. As you say, "new" commands, etc. One of the objects of iSCSI is to enable existing applications. > For example, I have heard that some high-bandwidth tape applications blast > self-describing blocks of data to tape, making the order in which data > gets read/written to tape less relevant. "Some" applications does not cover "all" applications. > For the case of high bandiwdth remote asynchronous mirroring, a special > SCSI remote asynchrnous mirroring command set could be introduced. > I believe remote asynchronous mirroring principally consists of one > command: WRITE, so the command set would probably be small. This would require a change to be made by T10. > Ditching multiple simultaneous connections/session would help convince us > of the correctness of iSCSI and allow us to move onto other important > issues and help the time to market. You can prototype with 1 TCP connection per session and build on that later. -Matt From owner-ips@ECE.cmu.edu Wed Jun 28 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17943; 28 Jun 2000 11:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA24147 for ips-outgoing; Wed, 28 Jun 2000 10:32:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch11.nsmg.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA24143 for ; Wed, 28 Jun 2000 10:32:24 -0400 (EDT) Received: by lmoxch11.nsmg.veritas.com with Internet Mail Service (5.5.2448.0) id ; Wed, 28 Jun 2000 10:32:59 -0400 Message-ID: From: John Matze To: scsi-tcp@external.cisco.com, ips@ece.cmu.edu Subject: RE: single vs multiple channels for iSCSI commands Date: Wed, 28 Jun 2000 10:32:58 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk As in Haifa, I again endorse the single connection approach. Especially for this version of the spec. Multiple channels may increase performance but this could also be handled with additional hardware or programming. At this point we don't need to supply any ammunition to shoot holes in the spec. I believe this issue alone will cause enough dialog to slow down and possible suspend the spec from being approved. John -----Original Message----- From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] Sent: Tuesday, June 27, 2000 8:26 PM To: scsi-tcp@external.cisco.com; ips@ece.cmu.edu Subject: Re: single vs multiple channels for iSCSI commands I would like to stake out a position against any approach that involves using multiple simultaneous TCP connections per iSCSI session. The design of iSCSI (and the hardware that implements it) becomes significantly simpler when there is just 1 TCP connection/session. Parallel paths in the network can still be exploited by setting up multiple sessions with multiple TCP connections. So far, I have heard of two applications that allegedly require strict ordering: tape backup and remote asynchronous mirroring. New SCSI commands or even higher-layer techniques can be used to satisfy these applications. For example, I have heard that some high-bandwidth tape applications blast self-describing blocks of data to tape, making the order in which data gets read/written to tape less relevant. For the case of high bandiwdth remote asynchronous mirroring, a special SCSI remote asynchrnous mirroring command set could be introduced. I believe remote asynchronous mirroring principally consists of one command: WRITE, so the command set would probably be small. Ditching multiple simultaneous connections/session would help convince us of the correctness of iSCSI and allow us to move onto other important issues and help the time to market. -Costa App A. Possible remote asynch mirroring command sets 1) 1 CDB: ORDERED WRITE 2) 3 CDBs: WRITE UNCOMMITTED, COMMIT, ABORT From owner-ips@ECE.cmu.edu Wed Jun 28 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18031; 28 Jun 2000 12:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA26821 for ips-outgoing; Wed, 28 Jun 2000 11:48:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from charon.Exabyte.COM ([206.104.178.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA26812 for ; Wed, 28 Jun 2000 11:48:29 -0400 (EDT) From: JoeBre@exabyte.com Received: from prime1.Exabyte.COM (prime1.Exabyte.COM [161.81.18.231]) by charon.Exabyte.COM (8.9.1a/8.9.1) with ESMTP id JAA06581; Wed, 28 Jun 2000 09:47:58 -0600 (MDT) Received: from bouldermta.exabyte.com (bouldermta.Exabyte.COM [161.81.192.16]) by prime1.Exabyte.COM (8.6.12/8.6.10) with ESMTP id JAA10626; Wed, 28 Jun 2000 09:47:57 -0600 Received: by bouldermta.Exabyte.COM with Internet Mail Service (5.5.2448.0) id ; Wed, 28 Jun 2000 09:47:27 -0600 Message-ID: <118A72FD3F02D211AD1000805F9AA58B012B7A8B@boulderp03.Exabyte.COM> To: HugheJP@nsc-bridge.network.com, scsi-tcp@external.cisco.com, ips@ece.cmu.edu Subject: RE: single vs multiple channels for iSCSI commands Date: Wed, 28 Jun 2000 09:47:31 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01BFE118.28FEB8B0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01BFE118.28FEB8B0 Content-Type: text/plain; charset="iso-8859-1" > -----Original Message----- > From: Hughes, James P. (MRDC) [mailto:HugheJP@nsc-bridge.network.com] > Sent: Tuesday, June 27, 2000 10:23 PM > To: scsi-tcp@external.cisco.com; ips@ece.cmu.edu > Subject: RE: single vs multiple channels for iSCSI commands > > > I agree. One TCP connection with command and data mixed. Keep > it simple. The > need to go out of order is so small that we should not let > this "wag the > dog". By this, do you mean that the requirement to execute CDBs in the order that they were issued, is so small as to be negligible? If so, I would ask you to re-evaluate your position. > > In addition, we need to have centralized control of devices. > That is, a tape > drive needs to be given out by a controlling authority, and > this needs to be > stronger than just cookies. IPSEC and SSL need another layer > to determine > who owns a peripheral. SCSI contains commands for this purpose. See the RESERVE, RELEASE, PERSISTANT RESERVE, and PERSISTANT RELEASE CDBs. They allow a given initiator to claim exclusive access to a target. This is enforced by the target. This can be administrated by a third party SCSI device. Would this fulfill your requirement? Joe Breher Exbyate Corp ------_=_NextPart_001_01BFE118.28FEB8B0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: single vs multiple channels for iSCSI commands

> -----Original Message-----
> From: Hughes, James P. (MRDC) [mailto:HugheJP@nsc-bridge= .network.com]
> Sent: Tuesday, June 27, 2000 10:23 PM
> To: scsi-tcp@external.cisco.com; = ips@ece.cmu.edu
> Subject: RE: single vs multiple channels for = iSCSI commands
>
>
> I agree. One TCP connection with command and = data mixed. Keep
> it simple. The
> need to go out of order is so small that we = should not let
> this "wag the
> dog".

By this, do you mean that the requirement to execute = CDBs in the order that they were issued, is so small as to be = negligible? If so, I would ask you to re-evaluate your = position.

>
> In addition, we need to have centralized = control of devices.
> That is, a tape
> drive needs to be given out by a controlling = authority, and
> this needs to be
> stronger than just cookies. IPSEC and SSL need = another layer
> to determine
> who owns a peripheral.
 
SCSI contains commands for this purpose. See the = RESERVE, RELEASE, PERSISTANT RESERVE, and PERSISTANT RELEASE CDBs. They = allow a given initiator to claim exclusive access to a target. This is = enforced by the target. This can be administrated by a third party SCSI = device. Would this fulfill your requirement?

Joe Breher
Exbyate Corp

------_=_NextPart_001_01BFE118.28FEB8B0-- From owner-ips@ECE.cmu.edu Wed Jun 28 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18116; 28 Jun 2000 13:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA28408 for ips-outgoing; Wed, 28 Jun 2000 12:30:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA28395 for ; Wed, 28 Jun 2000 12:30:25 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA135476 for ; Wed, 28 Jun 2000 18:29:40 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA183526 for ; Wed, 28 Jun 2000 18:29:53 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690C.005A9BE1 ; Wed, 28 Jun 2000 18:29:41 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 28 Jun 2000 19:28:47 +0300 Subject: a working version of the draft Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, Except for the stuff that will be coming from Paul and Luciano I have now (attached) a working version of the draft - that is supposed to contain - in broad terms - the thinks we worked on in Haifa: - an ordering scheme for the session for command AND status delivery - a recovery scheme - a component structure easier to parse But bear with me! I think I will spend the better part of my time until next Friday polishing it before delivery. I just wanted you all to have it for tomorrows call of the design team. Tomorrow we will probably want to discuss both the requirement document and the draft in detail. An administrative issue - write only to one of the lists (preferably the ips or iSCSI?). With David Nagle's help it is up and collecting mail from both ips and scsi-tcp (many of our mail items appear twice). And let's all of us wish good luck to the new chairmen. Julo --------------------------- Internet-Draft J. Satran D. Smith Expires December 28, 2000 K. Meth IBM C. Sapuntzakis Cisco Systems Randy Haagens Hewlett-Packard Co. Efri Zeidner SANGate Paul Von Stamwitz Adaptec Luciano Dalle Ore Quantum June 28, 2000 iSCSI (Internet SCSI) Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are work- ing documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also dis- tribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet- Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 0.0.1. Acknowledgements A large group of people contributed through their review, comments and valuable insights to the creation of this document - too many to mention them all. Nevertheless, we are grateful to all of them. We are especially grateful to those that found the time and pati- ence to participate in our weekly phone conferences and intermedi- ate meetings in Almaden and Haifa and thus helped shape this docu- ment: Matt Wakeley (Agilent), Jim Hafner, John Hufferd, Prasenjit Sarkar, Meir Toledano, John Dowdy, Steve Legg, Alain Azagury (IBM), Dave Nagle (CMU), David Black (EMC), John Matze (Veritas), Mark Bakke, Steve DeGroote, Mark Shrandt (NuSpeed), Gabi Hecht Satran, Smith, Sapuntzakis, Meth [Page 1] iSCSI June 2000 (Gadzoox), Robert Snively (Brocade), Nelson Nachum (StorAge) Satran, Smith, Sapuntzakis, Meth [Page 2] iSCSI June 2000 Table of Contents 1. Abstract 2. Overview 2.1. SCSI Concepts 2.2. iSCSI Concepts & Functional Overview 2.3. iSCSI Login 2.4. iSCSI Full Feature Phase 2.5. iSCSI Connection Termination 2.6. Naming 3. Message Formats 3.1. Template Header 3.2. SCSI Command 3.3. SCSI Response 3.4. Asynchronous Event 3.5. SCSI Task Management Message 3.6. SCSI Task Management Response 3.7. Ready To Transfer (RTT) 3.8. SCSI Data 3.9. Text Command 3.10. Text Response 3.11. Login Command 3.12. Login Response 3.13. Ping Command 3.14. Ping Response 3.15. Third Party Commands 3.16. Opcode Not Understood 4. Error Handling iSCSI 5. Notes to Implementors 5.1. Small TCP Segments 5.2. Multiple Network Adapters 5.3. Autosense 5.4. TCP RDMA option 5.5. Data Connections Options 6. Security Considerations 6.1. Data Integrity 6.2. Login Process 6.3. IANA Considerations 7. Authors' Addresses 8. References and Bibliography 9. Appendix A - Examples 9.1. Read operation example 9.2. Write operation example 10. Appendix B - Login/Text keys Satran, Smith, Sapuntzakis, Meth [Page 3] iSCSI June 2000 1. Abstract The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. This memo describes a transport protocol for SCSI that operates on top of TCP. The iSCSI protocol aims to be fully com- pliant with the requirements laid out in the SCSI Architecture Model - 2 [SAM2] document. 2. Overview 2.1. SCSI Concepts The endpoint of most SCSI commands is a "logical unit" (LUN). Exam- ples of logical units include hard drives, tape drives, CD and DVD drives, printers and processors. Within the logical unit the abstract entity that executes the SCSI commands is named the device-server. A "target" is a collection of logical units, in general of the same kind, and is directly addressable on the net- work. In large installations a target is known also as a "control unit". The target corresponds to the server in the abstract SAM client-server model. An "initiator" creates and sends SCSI com- mands to the target. The initiator corresponds to the client in the abstract SAM client-server model. A "task" is a linked set of SCSI commands. Some LUNs support multiple pending (queued) tasks. The target uses a "task tag" to distinguish between tasks. Only one command in a task can be outstanding at any given time. A SCSI command results in an optional data phase and a response phase. In the data phase, information travels either from the initiator to the target, as in a WRITE command, or from target to initiator, as in a READ command. In the response phase, the target returns the final status of the operation, including any errors. A response terminates a SCSI command. 2.2. iSCSI Concepts & Functional Overview The following conceptual layering model is used in this document to specify Initiator and target actions and how those relate to transmitted and received Protocol Data Units: - SCSI layer builds/receives SCSI CDB (Command Data Blocks) and relays/receives them with the remaining command execute parameters (cf. SAM-2) to/from the - iSCSI layer that is building/receiving iSCSI PDUs and relaying/receiving them to/from - one or more TCP connections that form an initiator-target "session" Communication between initiator and target occurs over one or more TCP connections. The TCP connections are used for sending control messages, SCSI commands, parameters and data within iSCSI protocol Satran, Smith, Sapuntzakis, Meth [Page 4] iSCSI June 2000 data units (iSCSI PDU) The group of TCP connections linking an ini- tiator with a target form a session (loosely equivalent to a SCSI nexus); a session is defined by a session ID (composed of a initia- tor part and a target part). TCP connections can be added and removed from a session. iSCSI supports ordered command delivery within a session and limited command and data recovery. All SCSI commands presented to iSCSI get a "command reference number" and this number can be used by a receiving target for ordered delivery. A sliding window mechanism is used to limit the number of outstand- ing commands. For descriptive purposes it is assumed that the iSCSI layer is implementing the sliding window mechanism. 2.3. iSCSI Login The purpose of iSCSI login is to enable a TCP connection for iSCSI use, authenticate the parties, authorize the initiator to send SCSI commands and mark the connection as belonging to a iSCSI session. A session is used to identify to a target all the connections with a given initiator. The targets listen on a well-known TCP port for incoming connections. The initiator begins the login process by connecting to that well-known TCP port. As part of the login pro- cess, the initiator and target MAY wish to authenticate each other. This can occur in many different ways. For example, the endpoints may wish to check the IP address of the other party. If the TCP connection uses transport layer security [TLS], certificates may be used to identify the endpoints. Also, iSCSI includes commands for identifying the initiator and passing an authenticator to the tar- get (see Appendix B). Once suitable authentication has occurred, the target MAY authorize the initiator to send SCSI commands. How the target chooses to authorize an initiator is beyond the scope of this document. The target indicates a successful authentication and authorization by sending a login response with "accept login". The login message includes a session ID - composed with an initia- tor part ISID and a target part TSID. For a new session the TSID is null. As part of the response the target will generate a TSID. Session specific parameters can be specified only for the first login of a session (TSID null)(e.g the maximum number of connec- tions that can be used for this session). Connection specific parameters (if any) can be specified for any login. Thus a session is operational once it has at least one connection and a pending login can't affect a whole session. After authentication and authorization, other parameters may be negotiated using the highly extensible Text Command message that allows arbitrary key:value pairs to be passed. Any message sent on a TCP connection before this connection gets into full feature phase at the initiator should be rejected by the initiator. A message reaching a target on a TCP connection before the full feature phase will be reject with an iSCSI check condition bit. Satran, Smith, Sapuntzakis, Meth [Page 5] iSCSI June 2000 2.4. iSCSI Full Feature Phase Once the initiator is authorized to do so, the iSCSI session is in iSCSI full feature phase. The initiator may send SCSI commands and data to the various LUNs on the target by mapping them in iSCSI messages that go over the established iSCSI session. For SCSI com- mands that require data and/or parameter transfer, the (optional)data and the status for a command must be sent over the same TCP connection that was used to deliver the SCSI command (con- nection allegiance). Thus if an initiator issues a READ command, the target must send the requested data followed by the status to the initiator over the same TCP connection that was used to deliver the SCSI command. If an initiator issues a WRITE command, the ini- tiator must send the data for that command and the target must return the status over the same TCP connection that was used to deliver the SCSI command. During iSCSI Full Feature Phase, the initiator and target may interleave unrelated SCSI commands, their SCSI Data and responses, over the session. Outgoing SCSI data (initiator to target - user data or command parameters)is sent as either unsolicited data or solicited data. Unsolicited data can be part of an iSCSI command PDU ("immediate data") or an iSCSI data PDU. Solicited data are sent in response to Ready To Transfer PDUs. Targets are operating in either solicited (RTT) data mode or unsolicited (non RTT) data mode. An initiator must always honor an RTT data request. It is considered an error for an initiator to send unsolicited data PDUs to a target operating in RTT mode (only solicited data). By default, immediate data is limited to 64Kbytes and an initiator is allowed to send immediate data (subject to lim- itations specified somewhere else in this document)even to targets working in RTT mode. An initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response. A target is allowed to silently discard data and request retransmis- sion through RTT. Initiators will not perform any scoreboarding for data and the residual count calculation is to be performed by the targets. Incoming data is allways solicited. However an ini- tiator will be able to request retransmission of all or part of the target data. SCSI Data packets are matched to their corresponding SCSI commands by using Tags that are specified in the protocol. Initiator tags for pending commands are unique initiator-wide for a session. Target tags for pending commands are unique target-wide for the session. Although the above mechanisms are designed to accomplish efficient data delivery and a large degree of control over the data flow it is recognized that some specific sequences involving ordered execution and a mix of solicited and immediate data can result in deadlocks. It is for this reason that discarding data by a target is considered a legitimate action. Examples of such sequences are presented in appendix C together recovery Satran, Smith, Sapuntzakis, Meth [Page 6] iSCSI June 2000 scenarios. Outgoing commands are numbered by iSCSI (CmdRN) and response PDUs (target to initiator) will continuously update the initiator about the maximum command number that can be sent(sliding window). Each iSCSI session to a target is treated as if it ori- ginated from a different initiator. 2.5. iSCSI Connection Termination Connection termination is assumed to be an exceptional event. Graceful TCP connection shutdowns are done by sending TCP FINs. Graceful connection shutdowns MUST only occur when there are no outstanding tasks that have allegiance to the connection. A target SHOULD respond rapidly to a FIN from the initiator by closing its half of the connection as soon as it has finished all outstanding tasks that have allegiance to the connection. Closing a connection that has outstanding tasks may require recovery actions and will Be described elsewhere in this document. 2.6. Naming Targets are named using an URL type name of the format: scsi://[/modifier] The name used to connect will be optionally included in the login in order to enable the target to present different views. This is the Target Acquired Name (TAN). We will not attempt to define which components of the name will participate in the name resolu- tion process and which ones will be used only for "view" defini- tion. The syntactic sugar included might be used to introduce structure for management purposes but has no specific significance for this standard. Example: scsi://diskfarm1.acme.com scsi://computingcenter.acme.com/peripherals/diskfarm1 When a target has to act as an initiator for a third party command it will use the TAN during login as required by the authentication mechanism. A domain name that contains exactly four numbers separated by dots (.), where each number is in the range 0 through 255, will be interpreted as an IPv4 address. Examples: 10.0.0.1/tapefarm1 10.0.0.2 Satran, Smith, Sapuntzakis, Meth [Page 7] iSCSI June 2000 3. Message Formats All multi-byte integers specified in formats defined in this docu- ment are to be represented in network byte order (i.e., big endian). 3.1. Template Header and Opcodes All iSCSI messages and responses have a header of the same length (40 bytes). Additional data may be added, as necessary, beginning with byte 40. The fields of Opcode and Length appear in all message and response headers. The other most commonly used fields are Ini- tiator Task Tag, Logical Unit Number, and Flags, which, when used, always appear in the same location of the header. Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode | Opcode-specific fields | +---------------+---------------+---------------+---------------+ 4| Length of Data (after 40 byte Header) | +---------------+---------------+---------------+---------------+ 8| LUN or Opcode-specific fields | + + 12| | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20/ Opcode-specific fields / +/ / +---------------+---------------+---------------+---------------+ 40 3.1.1. Opcode The Opcode indicates which iSCSI type of message or response is encapsulated by the header. Valid opcodes for messages (sent by initiator to target) are: 0x00 Ping Command (from initiator to target) 0x01 SCSI Command (encapsulates a SCSI Command Descriptor Block) 0x02 SCSI Task Management Message 0x03 Login Command 0x04 Text Command 0x05 SCSI Data (for WRITE operation) Valid opcodes for responses (sent by target to initiator) are: Satran, Smith, Sapuntzakis, Meth [Page 8] iSCSI June 2000 0x80 Ping Response (from target to initiator) 0x81 SCSI Response (contains SCSI status and possibly sense information or other response information) 0x82 SCSI Task Management Response 0x83 Login Response 0x84 Text Response 0x85 SCSI Data (for READ operation) 0x86 Ready To Transfer (RTT - sent by target to initiator when it is ready to receive data from initiator) 0x87 Asynchronous Event (sent by target to initiator to indicate certain special conditions) 0x88 Opcode Not Understood 0x89 Open Data Connections Response (optional) 3.1.2. Length The Length field indicates the number of bytes, beyond the first 40 bytes, that are being sent together with this message header. It is anticipated that most iSCSI messages and responses (not counting data transfer messages) will not need more than the 40 byte header, and hence the Length field will contain the value 0. It is expected that larger than 16 byte CDBs and parameter data will fol- low the header. 3.1.3. LUN The LUN specifies the Logical Unit for which the command is tar- geted. If the command does not relate to a Logical Unit, this field is either ignored or may be used for some other purpose. According to [SAM2], a Logical Unit Number can take up to a 64-bit field that identifies the Logical Unit within a target device. The exact format of this field can be found in the [SAM2] document. 3.1.4. Initiator Task Tag The initiator assigns a Task Id (or tag) to each SCSI task that it issues. (Recall that a task is a linked set of SCSI commands.) This Tag is a initiator-wide unique identifier that can be used to uniquely identify the Task. 3.1.5. Opcode-specific fields These field have different meanings for different messages. Satran, Smith, Sapuntzakis, Meth [Page 9] iSCSI June 2000 3.2. SCSI Command Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x01) |I|R|A|Rsv|ATTR | CmdRN +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| Logical Unit Number (LUN) | + + 12| | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20| Expected Data Transfer Length | +---------------+---------------+---------------+---------------+ 24| SCSI Command Descriptor Block (CDB) | + + 28| | + + 32| | + + 36| | +---------------+---------------+---------------+---------------+ 40/ Additional Data (Command Dependent) / +/ / +---------------+---------------+---------------+---------------+ 3.2.1. Flags. The Flags field for a SCSI Command consists of on byte. Byte 2 b0 (I) Immediate Data from initiator to target (write/control). b1 (r) set when data is expected to flow from target to ini- tiator (read). b2 (A) set to turn off Autosense for this command (see [SAM2]). b3-4 Reserved (should be 0) b5-7 used to indicate Task Attributes. Autosense refers to the automatic return of sense data to the ini- tiator in case a command did not complete successfully. If autosense is turned off, the initiator must explicitly request that Satran, Smith, Sapuntzakis, Meth [Page 10] iSCSI June 2000 sense data be sent to it after some command has completed with a CHECK CONDITION status. 3.2.2. Task Attributes The Task Attribute field (ATTR) can have one of the following integer values (see [SAM2] for details): 0 Untagged 1 Simple 2 Ordered 3 Head of Queue 4 ACA 3.2.3. Command Reference Number (CmdRN) The Command Reference Number (CmdRN) is provided by the initiator to assist in performing ordered delivery for iSCSI commands. CmdRN is reflecting the value of a 16 bit counter maintained by the ini- tiator and increased by 1 for every command received by the iSCSI delivery mechanism. The counter is set to an initial value at ses- sion initiation (default is 0) and when sending target resets (0). 3.2.4. Expected Data Transfer Length The Expected Data Transfer Length field states the number of bytes of data that the initiator expects will be sent for this (READ or WRITE) SCSI operation in SCSI Data packets. For a WRITE operation, the initiator uses this field to specify the number of bytes of data it expects to transfer for this operation (not counting data headers). For a READ operation, the initiator uses this field to specify the number of bytes of data it expects the target to transfer to the initiator (not counting data headers). If no data will be transferred in SCSI Data packets for this SCSI operation, this field should be set to 0. Upon completion of a data transfer, the target will inform the ini- tiator of how many bytes were actually processed (sent or received) by the target. 3.2.5. SCSI Command Descriptor Block (CDB) There are 16 bytes in the CDB field, designed to accommodate the largest currently defined CDB. If, in the future, larger CDBs are allowed, the spill-over of the CDB may extend beyond the 40-byte. Satran, Smith, Sapuntzakis, Meth [Page 11] iSCSI June 2000 3.2.6. Command-Data Some SCSI commands require additional parameter data to accompany the SCSI command. This data may be placed beyond the 40-byte boun- dary of the iSCSI header. Alternatively user data can be placed in the the same PDU (in both cases we talk about immediate data). The Length field is set to the length of this data beyond the 40-byte header (i.e. includes the CDB extension if present). The CDB length is: Length + 16 - I*ExpectedDataTransferLength Satran, Smith, Sapuntzakis, Meth [Page 12] iSCSI June 2000 3.3. SCSI Response Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x81) | Rsvd(0) |O|U| MaxCmdRN | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| Reserved (0) | + + 12| | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20| Residual Count | +---------------+---------------+---------------+---------------+ 24| Command Status|iSCSI Status | StatRN | +---------------+---------------+---------------+---------------+ 28/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Response or Sense Data (optional) / +/ / +---------------+---------------+---------------+---------------+ 3.3.1. Flags The SCSI Response has its own set of flags, that differs from the flags for a SCSI Command. Byte 2 b0 (U) set for Residual Underflow. In this case, the Resi- dual Count indicates how many bytes were not transferred out of those expected to be transferred. b1 (O) set for Residual Overflow. In this case, the Residual Count indicates how many bytes could not be transferred because the initiator's Expected Data Transfer Length was too small. b2-7 not used (should be set to 0). Bits 0 and 1 are mutually exclusive. 3.3.2. MaxCmdRN Indicates the maximum CmdRN the initiator should send. It will set an internal limit register. The initiator will refrain from sending Satran, Smith, Sapuntzakis, Meth [Page 13] iSCSI June 2000 commands numbered past MaxCmdRN (considering also wrap-around). Attention should be paid to the fact that response PDUs can arrive in "wrong order". The internal limit register can only be advanced by incoming responses (considering also wraparounds). It is assumed that any target will accept less than 64k outstanding com- mands. 3.3.3. Residual Count The Residual Count field is valid only in case either the Residual Underflow bit or Residual Overflow bit is set. If neither bit is set, the Residual Count field should be 0. If the Residual Under- flow bit is set, the Residual Count indicates how many bytes were not transferred out of those expected to be transferred. If the Residual Overflow bit is set, the Residual Count indicates how many bytes could not be transferred because the initiator's Expected Data Transfer Length was too small. 3.3.4. Command Status The Command Status field is used to report the SCSI status of the command (as specified in [SAM2]). 3.3.5. iSCSI Status The iSCSI Status field is used to report the status of the command before it was sent by the target to the LUN. The values are given below. 0 Good status 1 iSCSI check If the iSCSI field is not 0 the command status will indicate CHECK CONDITION 3.3.6. Response or Sense Data If Autosense was not disabled in the originating CDB and the Com- mand Status was CHECK CONDITION (0x02), then the Response Data field will contain sense data for the failed command. Some sense codes will relate to iSCSI check conditions (e.g. excessive number of outstanding commands, immediate data blocks too large etc.). If the Command Status is Good (0x00) then the Response Data field will contain data from the data phase of the CDB. The Length parameter specifies the number of bytes in this field. If no error occurred, and no data is needed for the response to the SCSI Command the Length field is 0. Note that if the Command Status was CHECK CON- DITION but Autosense was disabled, then sense data must be Satran, Smith, Sapuntzakis, Meth [Page 14] iSCSI June 2000 explicitly requested by the initiator with a new SCSI command. 3.3.7. StatRN - Status Reference Number StatRN is a reference number that the target iSCSI layer generates whenever it issues a response by incrementing an internal counter. A gap in StatRN indicates a lost status (possible due to connection failure) and be recovered by reissuing the outstanding command with the original TaskID and CmdRN. Satran, Smith, Sapuntzakis, Meth [Page 15] iSCSI June 2000 3.4. Asynchronous Event An Asynchronous Event may be sent from the target to the initiator without corresponding to a particular command. The target specifies the status for the event and sense data. Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x87) | Reserved (0) | MaxCmdRN | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| Logical Unit Number (LUN) | + + 12| | +---------------+---------------+---------------+---------------+ 16| Reserved (0) | + + 20| | +---------------+---------------+---------------+---------------+ 24| Command Status|iSCSI Status| Reserved (0) | +---------------+---------------+---------------+---------------+ 28|Event Indicator| Reserved (0) | +---------------+---------------+---------------+---------------+ 32| Reserved (0) | + + 36| | +---------------+---------------+---------------+---------------+ 40/ Sense Data / +/ / +---------------+---------------+---------------+---------------+ 3.4.1. iSCSI Status Some Asynchronous Events are strictly related to iSCSI while others are related to SAM-2. The codes returned for iSCSI Asynchronous Events are: 2 Target is being reset. 3.4.2. Event Indicator The following values are defined. (See [SAM2] for details.) 1 An error condition was encountered after command Satran, Smith, Sapuntzakis, Meth [Page 16] iSCSI June 2000 completion. 2 A newly initialized device is available. 3 Some other type of unit attention condition has occurred. 4 An asynchronous event has occurred. Sense Data accompanying the report identifies the condition. The Length parameter is set to the length of the Sense Data. 3.4.3. MaxCmdRN - inform about this value other initiators after a tar- get Reset Satran, Smith, Sapuntzakis, Meth [Page 17] iSCSI June 2000 3.5. SCSI Task Management Message Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x02) | Msg indicator | Reserved (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| Logical Unit Number (LUN) | + + 12| | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40 3.5.1. Msg Indicator The Task Management functions provide an initiator with a way to explicitly control the execution of one or more Tasks. The Task Management functions are summarized as follows (for a more detailed description see the [SAM2] document): 1 Abort Task---aborts the task identified by the Task Tag field. 2 Abort Task Set---aborts all Tasks issued by this initia- tor on the Logical Unit. .ti -5 3 Clear ACA---clears the Auto Contingent Allegiance condition. 4 Clear Task Set---Aborts all Tasks (from all initiators) for the Logical Unit. 5 Logical Unit Reset. 6 Target Reset. For the functions above except , a SCSI Task Manage- ment Response is returned, using the Initiator Task Tag to identify the operation for which it is responding. For the function, the target cancels all pending operations. The target may send an Asynchronous Event to all attached initiators notifying them that the target is being reset. The target then closes all of its TCP connections to all initiators (all sessions are ter- minated). Satran, Smith, Sapuntzakis, Meth [Page 18] iSCSI June 2000 3.6. SCSI Task Management Response Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x82) | Msg indicator | Reserved (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| Logical Unit Number (LUN) | + + 12| | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20| Reserved (0) | +---------------+---------------+---------------+---------------+ 24| Response | Reserved (0) | +---------------+---------------+---------------+---------------+ 28/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40 For the functions , the target performs the requested Task Management function and sends a SCSI Task Management Response back to the initiator. The target includes all of the information the initiator provided in the SCSI Task Management Message, so the initiator can know exactly which SCSI Task Management Message was serviced. In addition, the target provides a Response indication which may take on the following values: 0 Function Complete 1 Function Rejected .RE .RE For the func- tion, the target cancels all pending operations. The tar- get may send an Asynchronous Event to all attached ini- tiators notifying them that the target is being reset. The target then closes all of its TCP connections to all initiators (terminates all sessions). 3.6.1. MaxCmdRN - maximum CmdRN the target will accept Satran, Smith, Sapuntzakis, Meth [Page 19] iSCSI June 2000 3.7. Ready To Transfer (RTT) When an initiator has submitted a SCSI Command with data passing from the initiator to the target (WRITE), the target may specify which blocks of data it is ready to receive. In general, the target may request that the data blocks be delivered in whatever order is convenient for the target at that particular instant. This information is passed from the target to the initiator in the Ready To Transfer (RTT) message. In order to allow write operations without RTT, the initiator and target must have agreed to do so by both sending the AllowNoRTT:yes key-pair attribute to each other (either during Login or through the Text Command/Response mechanism). Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x86) | Reserved (0) | MaxCmdRN | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| Reserved (0) | + + 12| | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20| Desired Data Transfer Length | +---------------+---------------+---------------+---------------+ 24| Data Offset | +---------------+---------------+---------------+---------------+ 28| Target Transfer Tag | +---------------+---------------+---------------+---------------+ 32| Reserved (0) | + + 36| | +---------------+---------------+---------------+---------------+ 40 3.7.1. MaxCmdRN - maximum CmdRN the target will accept 3.7.2. Desired Data Transfer Length and Data Offset The target specifies how many bytes it wants the initia- tor to send as a result of this RTT message. The target Satran, Smith, Sapuntzakis, Meth [Page 20] iSCSI June 2000 may request the data from the initiator in several chunks, not necessarily in the original order of the data. The target, therefore, also specifies a Data Offset indicating the point at which the data transfer should begin, relative to the beginning of the total data transfer. 3.7.3. Target Transfer Tag The target assigns its own tag to each RTT request that it sends to the initiator. This can be used by the target to easily identify data it receives, and can also be used as an RDMA tag [RDMA]. Satran, Smith, Sapuntzakis, Meth [Page 21] iSCSI June 2000 3.8. SCSI Data The typical data transfer specifies the length of the data payload, the Transfer Tag provided by the receiver for this data transfer, and a buffer offset. The typical SCSI Data packet for WRITE (from initiator to target) has the following format: Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x05) | Reserved (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| Transfer Tag | +---------------+---------------+---------------+---------------+ 12| Data Offset | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Payload / +/ / +---------------+---------------+---------------+---------------+ The typical SCSI Data packet for READ (from target to initiator) has the following format: Satran, Smith, Sapuntzakis, Meth [Page 22] iSCSI June 2000 Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x85) | (0) |S|O|U| MaxCmdRN or (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| Transfer Tag | +---------------+---------------+---------------+---------------+ 12| Data Offset | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20| Residual Count | +---------------+---------------+---------------+---------------+ 24| Command Status|iSCSI Status | StatRN | +---------------+---------------+---------------+---------------+ 28/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Payload / +/ / +---------------+---------------+---------------+---------------+ 3.8.1. Length The length field specifies the total number of bytes in the following payload. 3.8.2. Transfer Tag The Transfer Tag identifies the operation to which this data transfer belongs. When the transfer is from the target to the initiator, the Transfer Tag is the Initia- tor Task Tag that was sent with the SCSI command. When the transfer is from the initiator to the target, the Transfer Tag is the Target Transfer Tag when RTT is enabled, or the Initiator Task Tag when RTT is disabled. 3.8.3. Buffer Offset The Buffer Offset field contains the offset of the fol- lowing data against the complete data transfer. The sum of the buffer offset and length should not exceed the expected transfer length for the command. Satran, Smith, Sapuntzakis, Meth [Page 23] iSCSI June 2000 3.8.4. Flags The last SCSI Data packet sent from a target to an ini- tiator for a particular SCSI command that completed suc- cessfully may optionally also contain the Command Status for the data transfer. In this case Sense Data cannot be sent together with the Command Status. If the command completed with an error, then the response and sense data must be sent in a SCSI Response packet and must not be sent in a SCSI Data packet. Byte 2 b0-1 as in an ordinary SCSI Response b2 (S) set to indicate that the Command Status field contains status. b3-7 not used (should be set to 0). If the (S) bit is set, then there is meaning to the extra fields in the SCSI Data packet (MaxCmdRN, Command Status, Residual Count, StatRN) Satran, Smith, Sapuntzakis, Meth [Page 24] iSCSI June 2000 3.9. Text Command The Text Command is provided to allow the exchange of information and for future extensions. It permits the initiator to inform a target of its capabilities or to request some special operations. Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x04) | Reserved (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Text / +/ / +---------------+---------------+---------------+---------------+ 3.9.1. Length The length, in bytes, of the Text field. 3.9.2. Initiator Task Tag The initiator assigned identifier for this Text Command. 3.9.3. Text The initiator sends the target a set of key:value pairs in UTF-8 Unicode format. The key and value are separated by a ':' (0x3A) delimiter. Many key:value pairs can be included in the Text block by separating them with null ' ' (0x00) delimiters. Some basic key:value pairs are described in Appendix B. The target responds by sending its response back to the initiator. The target and ini- tiator can then perform some advanced operations based on their common capabilities. Manufacturers may introduce new keys by prefixing them with their (reversed) domain Satran, Smith, Sapuntzakis, Meth [Page 25] iSCSI June 2000 name, for example, com.foo.bar.do_something:0000000000000003 Any key that the target does not understand may be ignored without affecting basic function. Once the target has processed all the key:value pairs, it responds with the Text Response command, listing the parameters that it supports. It is recommended that Text operations that will take a long time should be placed in their own Text command. If the Text Response does not contain a key that was requested, the initiator must assume that the key was not understood by the target. Satran, Smith, Sapuntzakis, Meth [Page 26] iSCSI June 2000 3.10. Text Response The Text Response message contains the responses of the target to the initiator's Text Command. The format of the Text field matches that of the Text Command. Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x84) | Reserved (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Text Response / +/ / +---------------+---------------+---------------+---------------+ 3.10.1. Length The length, in bytes, of the Text Response field. 3.10.2. Initiator Task Tag The Initiator Task Tag matches the tag used in the ini- tial Text Command and is used by the initiator to relate the Text Commands with the appropriate Text Responses. 3.10.3. Text Response The Text Response field contains responses in the same key:value format as the Text Command. Appendix B lists some basic Text Commands and their Responses. If the Text Response does not contain a key that was requested, the initiator must assume that the key was not understood by the target. Satran, Smith, Sapuntzakis, Meth [Page 27] iSCSI June 2000 3.11. Login Command After establishing a TCP connection between an initiator and a target, the initiator should issue a Login Command to gain further access to the target's resources. Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x03) | Reserved (0) | CmdRN or Reserved (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| CID | RecoverCID or 0 | +---------------+---------------+---------------+---------------+ 12| Reserved (0) | +---------------+---------------+---------------+---------------+ 16| ISID |TSID | +---------------+---------------+---------------+---------------+ 20/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Login Parameters in Text Command Format / +/ / +---------------+---------------+---------------+---------------+ 3.11.1. CID a unique id for this connection within the session 3.11.2. For a connection used to recover a lost TCP connection (see later the ID of the failed connection) the initiator provides the CID of the failed connection. A simple tar- get may reject recovery. In this case the initiator will terminate all outstanding commands with a check condi- tions and reset the target. The initiator may provide some basic parameters in order to enable the target to determine if the initiator may in fact use the target's resources. The format of the parameters is as specified for the Text Command. Targets may require keys to indicate the Domain Name of the ini- tiator and the target, and perhaps also an Authenticator key. The initiator may also provide additional Satran, Smith, Sapuntzakis, Meth [Page 28] iSCSI June 2000 parameters to the target in Text Command format, if the initiator so desires. Keys and their explanations are listed in Appendix B. Whenever desired an initiator will identify its view of the target as in: Target:[/modifier][:port] implying that the target is known as: scsi://[/modifier] and it should be connected through port "port" (the default well known port has an IANA defined value of xx) Initiators can use the same type of naming implying machine and optional a principal (e.g. operating system image) as in: Initiator:[/modifier] implying that the initiator is known as: iSCSI://[/modifier] Thus the parameters passed for a plain-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame The modifier iSCSI-SYS is reserved for administrative functions. ISID and TSID form collectively the SSID (session id). A TSID of 0 indicates a leading connection. Only a leading connection login can carry session specific parameters, e.g. max-connections-requested, the maximum immediate data length requested, etc.. CmdRN is significant only if TSID is 0 and indicates the starting Command reference number for this session; it should be 0 for all other Instances. Satran, Smith, Sapuntzakis, Meth [Page 29] iSCSI June 2000 3.12. Login Response The target responds to the Login Command with a Login Response. It is sufficient for the target to respond with a Status indicating that the Login is accepted. In fact, the target may completely ignore the parameters that were sent to it and may provide service to any ini- tiator that connects to it. The target may also return parameters using the format of the Text Response opcode, if it so desires. In particular, the target may want to provide its Authenticator key, so that the initiator can be sure that it is in fact talking with the correct tar- get. The initiator can request that the target provide the Authenticator parameter by specifying the SendAuthenticator:yes key:value pair. Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x83) | Reserved (0) | MaxCmdRN or Reserved (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 16| ISID | TSID | +---------------+---------------+---------------+---------------+ 20| Reserved (0) | +---------------+---------------+---------------+---------------+ 24| Status | Reserved (0) | +---------------+---------------+---------------+---------------+ 28/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Login Parameters in Text Command Format / +/ / +---------------+---------------+---------------+---------------+ The format of the Login Response is the same as the Text Response, with the addition of one field. 3.12.1. Status The Status returned in a Login Response is one of the following: 0 accept login (will now accept SCSI commands) Satran, Smith, Sapuntzakis, Meth [Page 30] iSCSI June 2000 1 reject login 2 additional authentication required 3 reject recovery In the case that the Status is "accept login" the initia- tor may proceed to issue SCSI commands. In the case that the Status is "reject login" the initiator should immedi- ately close down its end of the TCP connection, thus freeing up the target's port for some other connection. The target also has the option of immediately closing down its end of the TCP connection. In the case that the Status is "additional authentication required" the ini- tiator must provide additional authentication information by issuing the Text Command with the appropriate key:value pairs. (This may be required if the authenti- cation method is based on a challenge/response algo- rithm.) Upon receipt of the necessary authentication, the target will issue a Login Response with the "accept login" Status. SCSI Commands will not be accepted until the target provides a Login Response with the "accept login" Status. The TSID is an initiator identifying tag set by the target. A 0 in the returned TSID indicates that either the target supports only a single connection or that the ISID has already been used as a leading ISID. In both cases the target is rejecting the login. MaxCmdRN indicates the maximum CmdRN the initiator should send. When reaching this number (considering number wrap-around) the initiator should refrain from sending further commands until the initiator does receive a new MaxCmdRN that advanced past the old value. Satran, Smith, Sapuntzakis, Meth [Page 31] iSCSI June 2000 3.13. Ping Command Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x0) | Reserved (0) | MaxStatRN | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Ping Data (optional) / +/ / +---------------+---------------+---------------+---------------+ The Ping Command can be used to verify that a connection is still active. It may be useful in the case where an initiator has been waiting a long time for the response to some command, and the initiator suspects that there is some problem with the connection. When a target receives the Ping Command, it should respond with a Ping Response, duplicating as much of the data as possible that was pro- vided in the Ping Command (if such data was present). If the initiator does not receive the Ping Response within some period of time (determined by the initiator), or if the data returned by the Ping Response is different from the data that was in the Ping Command, the initiator may conclude that there is a problem with the connection. The initiator will then close the connection and may try to establish a new connection. 3.13.1. MaxStatRN - the next StatRN expected 3.13.2. Length The length of the optional Ping Data. 3.13.3. Initiator Task Tag An initiator assigned identifier for the operation. Satran, Smith, Sapuntzakis, Meth [Page 32] iSCSI June 2000 3.13.4. Ping Data Binary data that will be reflected in the Ping Response. Satran, Smith, Sapuntzakis, Meth [Page 33] iSCSI June 2000 3.14. Ping Response Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x80) | Reserved (0) | MaxCmdRN | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Return Ping Data / +/ / +---------------+---------------+---------------+---------------+ When a target receives the Ping Command, it should respond with a Ping Response, duplicating the data and Initiator Task Tag that was provided in the Ping Command, if present. Satran, Smith, Sapuntzakis, Meth [Page 34] iSCSI June 2000 3.15. Third Party Commands -INCOMPLETE! There are some third-party SCSI commands, such as COPY and EXTENDED COPY, that require one target (Target A) to act as an initiator to other targets (e.g., Target B). Some such commands can be extended in a straightforward way to accommodate new forms of addressing, and this should be done to address targets using iSCSI. These extensions are not straightforward for all commands, and they may not be able to encompass the full name space and authentication information needed for iSCSI in some con- texts. Thus iSCSI also provides a facility for assigning local short-form aliases to full addressing/authorization information for targets, and the aliases can be used in the SCSI commands and parameter data. The alias informa- tion is specified as Text following the header of the SCSI command specifying the third-party command. The header will thus appear as follows: Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x01) |1|0|A|Rsv|ATTR | CmdRN | +---------------+---------------+---------------+---------------+ 4| Length (!= 0) | +---------------+---------------+---------------+---------------+ 8| Logical Unit Number (LUN) | + + 12| | +---------------+---------------+---------------+---------------+ 16| Initiator Task Tag | +---------------+---------------+---------------+---------------+ 20| Expected Data Transfer Length | +---------------+---------------+---------------+---------------+ 24| SCSI Command Descriptor Block (CDB) | + + 28| | + + 32| | + + 36| | +---------------+---------------+---------------+---------------+ 40/ Extended CDB if any / +/ Parameters needed for Target B / / / +---------------+---------------+---------------+---------------+ Satran, Smith, Sapuntzakis, Meth [Page 35] iSCSI June 2000 The Length field will not be zero. Rather, it will con- tain the length of extend CDB if any an the length of the alias information which may include the name of Target B and an Authentication key in Text Command format. An example of the data for this command might be: LocalName:TargetB FullName:sj.foo.com/controller1 OriginalAuthenticator:open-sesame Upon receiving a third-party command, Target A will per- form login operations with the identified targets. In effect, Target A will become an initiator to Target B. Among the parameters provided to Target B, Target A may specify the authentication information from the initia- tor. The Text provided by Target A when it performs the Login command to Target B may contain the keys Target (referring to Target B) and Initiator (referring to Tar- get A), and it may also contain the keys Authenticator (of Target A), OriginalInitiator and OriginalAuthentica- tor (referring to authenticator of the original initia- tor). Satran, Smith, Sapuntzakis, Meth [Page 36] iSCSI June 2000 3.16. Opcode Not Understood Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x88) | Reserved (0) | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8/ Reserved (0) / +/ / +---------------+---------------+---------------+---------------+ 40/ Header of Bad Message / +/ / +---------------+---------------+---------------+---------------+ 80 It may happen that a target receives a message with an Opcode that it doesn't recognize. This may occur because of a new version of the protocol that defines a new Opcode, or because of some corruption of a message header. The target returns the header of the message with the unrecognized opcode as the data of the response. Satran, Smith, Sapuntzakis, Meth [Page 37] iSCSI June 2000 4. iSCSI Error Handling 4.1. Communications Errors For any outstanding SCSI command it is assumed that iSCSI in conjunction with SCSI at the initiator is able to keep enough information to be able to rebuild the command PDU, that outgoing data is available (in host memory) for retransmission while the command is outstanding and that at target iSCSI and specialized TCP implementations are able to recover unacknowledged data packets from a clos- ing connection or, alternatively the target has means to re-read data from a device-server. It is further assumed that a target will keep a the "status & sense" for com- mands it has executed while the total number of outstand- ing commands and executed commands Does not exceed it's limit. A target will sequentially number the delivered responses and thus enable initiators to tell when a response is missing and what response they miss. Under those conditions iSCSI will be able to keep a ses- sion in operation provided that it has at is able to keep/establish at least one TCP connection between the initiator and target in a timely fashion. Unfortunately the maximum admissible recovery time is a function of the target and for some devices and communications networks recovery may be complex and may percolate to upper software layers. It is assumed that targets and/or ini- tiators will recognize a failing connection by either transport level means (TCP) or by a gap in the command stream that does not get filled for a long time, or by failing iSCSI ping (the later should be used by periodi- cally by highly reliable implementations). The recovery involves the following steps: -abort offending TCP connection(s) (target & initia- tor) and recover at target all unacknowledged read- data .ti -2 -create one or more new TCP connections (within the same session) and associate all out- standing commands from the failed connection(s) to the FIRST new connection at both initiator and tar- get -the initiator will reissue all outstanding commands with their original CmdRN and TaskID -upon receiving the new/restarting commands the target will resume command execution; for write commands it means requesting data retransmission through RTT, for reads retransmitting recovered data and for "terminated" commands retransmitting the status & Satran, Smith, Sapuntzakis, Meth [Page 38] iSCSI June 2000 sense while retaining the original StatRN. If data recovery is not possible the target will either pro- vide data from the media or redo the operation (if the operation is not idem-potent the device server may fail the operation). 4.2. Protocol Errors The authors recognize that mapping framed messages over a "stream" connection (like TCP) makes the proposed mechan- isms vulnerable to simple software framing errors and introducing framing mechanisms may be onerous for perfor- mance and bandwidth. Command reference numbers and the above mechanisms for connection drop and reestablishment will help handle this type of mapping errors. 4.3. Session Errors If all the connections of a session fail and can't be reestablished in a short time or if initiators detect protocol errors repeatedly an initiator may choose to terminate a session an establish a new session (indicat- ing old session termination?). It will terminate all out- standing request with an iSCSI error indication before initiating a new session. A target that detects one of the above errors will take the following actions: - Reset the TCP connections (close the session). - Abort all Tasks in the task set for the correspond- ing initiator. Satran, Smith, Sapuntzakis, Meth [Page 39] iSCSI June 2000 5. Notes to Implementers This section notes some of the performance and reliabil- ity considerations of the iSCSI protocol. This protocol was designed to take advantage of the a generic Remote DMA TCP options [RDMA], although it can still operate effectively without this TCP extension. 5.1. Small TCP Segments It is recommended that TCP segments be limited in size to no more than 8K bytes. One reason is to ensure that seg- ments won't get broken into smaller packets, thereby pos- sibly breaking the assumptions for RDMA and the informa- tion in the RDMA header. Another reason we recommend small segments is to allow a stronger type of checksum, possibly utilizing CRC, which is practical only for smaller segments. 5.2. Multiple Network Adapters The iSCSI protocol assumes that the Task Tags will also serve as RDMA tags. The iSCSI protocol allows multiple connections, not all of which need go over the same net- work adapter. If multiple network connections are to be utilized with RDMA, the iSCSI protocol command-data- status allegiance to one TCP connection insure that there is no need to replicate information across network adapters or otherwise require them to cooperate. 5.3. Autosense Autosense refers to the automatic return of sense data to the initiator in case a command did not complete success- fully. If autosense is turned off, the initiator must explicitly request that sense data be sent to it after some command has completed with a CHECK CONDITION status. The default for iSCSI is to work with Autosense enabled. Note that even if a SCSI target/LUN does not support Autosense, it may still be possible for iSCSI to work with Autosense. This can be accomplished as follows. Whenever a CHECK CONDITION status is about to be returned, the iSCSI component on the target immediately queries the target/LUN for the sense data. iSCSI can then return the sense data to the initiator together with the CHECK CONDITION status. It is not necessary for iSCSI to wait for the initiator to explicitly request the sense data; the target iSCSI code can perform this operation Satran, Smith, Sapuntzakis, Meth [Page 40] iSCSI June 2000 automatically, even for devices/LUNs that do not ordi- narily provide automatic sense data. 5.4. TCP RDMA option The TCP RDMA option [RDMA] is an annotation on individual TCP segments that can reduce the number of copies neces- sary at the receiver. The RDMA option succinctly describes the portion of a TCP payload that holds bulk data. 5.5. TCP Connection Options Some targets may want to inform (or negotiate with) an initiator concerning some parameters related to bandwidth, Quality of Service, or some other available features on its various network connections. These are exchanged between the initiator and the target using Text Commands and Responses. Satran, Smith, Sapuntzakis, Meth [Page 41] iSCSI June 2000 6. Security Considerations 6.1. Data Integrity We assume that end-to-end data integrity can be assured by TCP, by adding a more powerful checksum option when- ever this is considered important, or replacing the checksum by a weaker one (or even "nullifying it") for applications in which data integrity is not important and recovery from data errors could be harmful (e.g., audio or video distribution streams). 6.2. Login Process In some environments, a target will not be interested in authenticating the initiator. In this case, the target can simply ignore some or all of the parameters sent in a Login Command, and the target can simply reply with a basic Login Response indicating a successful login. Some targets may want to perform some kind of authentication. The Authenticator key is defined for this purpose. Vari- ous authentication schemes can be used, including encrypted passwords and trusted certificate authorities. Once the initiator and target are confident of the iden- tity of the attached party, the established channel is considered secure. It is anticipated that most target devices will not bother with all of the possible checks, but the protocol provides sufficient means to perform the checks, if required by the target. 6.3. IANA Considerations There will be a well known port for iSCSI connections. These well known ports will have to be registered with IANA. A checksum type will also have to be registered with IANA. Satran, Smith, Sapuntzakis, Meth [Page 42] iSCSI June 2000 7. Authors' Addresses Julian Satran Kalman Meth IBM, Haifa Research Lab MATAM - Advanced Technology Center Haifa 31905, Israel Phone +972 4 829 6211 Email: Julian_Satran@vnet.ibm.com meth@il.ibm.com Daniel F. Smith IBM Almaden Research Center 650 Harry Road San Jose, CA 95120-6099, USA Phone: +1 408 927 2072 Email: dfsmith@almaden.ibm.com Costa Sapuntzakis Cisco Systems, Inc. 170 W. Tasman Drive San Jose, CA 95134, USA Phone: +1 408 525 5497 Email: csapuntz@cisco.com Efri Zeidner SANGate Israel efri@sangate.com Comments may be sent to Julian Satran, Daniel Smith, Costa Sapuntzakis, or Kalman Meth. Satran, Smith, Sapuntzakis, Meth [Page 43] iSCSI June 2000 8. References and Bibliography [RDMA] Internet Draft: TCP RDMA option (work in pro- gress) [SAM2] ANSI X3.270-1998, SCSI-3 Architecture Model (SAM-2) [TLS] The TLS Protocol, RFC 2246, T. Dierks et al. [ALTC] Internet Draft: Alternative checksums (work in progress) [CAM] ANSI X3.232-199X, Common Access Method-3 (Cam- 3) [CRC] ISO 3309, High-Level Data Link Control (CRC 32) [RFC793] Transmission Control Protocol, RFC 793 [RFC1122] Requirements for Internet Hosts-Communication Layer, RFC 1122, R. Braden (editor) [SBC] ANSI X3.306-199X, SCSI-3 Block Commands (SBC) [SCSI2] ANSI X3.131-1994, SCSI-2 [SPC] ANSI X3.301-199X, SCSI-3 Primary Commands (SPC) Satran, Smith, Sapuntzakis, Meth [Page 44] iSCSI June 2000 9. Appendix A - Examples 9.1. Read operation example |Initiator Function| Message Type | Target Function | +------------------+-----------------------+----------------------+ | Command request |SCSI Command (READ)>>> | | | (read) | | | +------------------+-----------------------+----------------------+ | | | Prepare Data Transfer| +------------------+-----------------------+----------------------+ | Receive Data | <<< SCSI Data | Send Data | +------------------+-----------------------+----------------------+ | Receive Data | <<< SCSI Data | Send Data | +------------------+-----------------------+----------------------+ | Receive Data | <<< SCSI Data | Send Data | +------------------+-----------------------+----------------------+ | | <<< SCSI Response |Send Status and Sense | +------------------+-----------------------+----------------------+ | Command Complete | | | +------------------+-----------------------+----------------------+ Satran, Smith, Sapuntzakis, Meth [Page 45] iSCSI June 2000 9.2. Write operation example +------------------+-----------------------+---------------------+ |Initiator Function| Message Type | Target Function | +------------------+-----------------------+---------------------+ | Command request |SCSI Command (WRITE)>>>| Receive command | | (write) | | and queue it | +------------------+-----------------------+---------------------+ | | | Process old commands| +------------------+-----------------------+---------------------+ | | | Ready to process | | | <<< RTT | WRITE command | +------------------+-----------------------+---------------------+ | Send Data | SCSI Data >>> | Receive Data | +------------------+-----------------------+---------------------+ | Send Data | SCSI Data >>> | Receive Data | +------------------+-----------------------+---------------------+ | | <<< RTT | | +------------------+-----------------------+---------------------+ | Send Data | SCSI Data >>> | Receive Data | +------------------+-----------------------+---------------------+ | | <<< SCSI Response |Send Status and Sense| +------------------+-----------------------+---------------------+ | Command Complete | | | +------------------+-----------------------+---------------------+ Satran, Smith, Sapuntzakis, Meth [Page 46] iSCSI June 2000 10. Appendix B - Login/Text keys 10.1. Target Target:domainname[/modifier] Examples: Target:disk-array.sj-bldg-h.cisco.com Target:disk-array.sj-bldg-h.cisco.com/disk3 This key is provided by the initiator of the TCP connec- tion to the remote endpoint. The Target key specifies the domain name of the target, since that information is not available from the TCP layer. The target is not required to support this key. The initiator should send this key in the first login message. The Target key might be used by the target to learn the intended initiator view of the target. 10.2. Initiator Initiator:[domainname[/modifier]] Examples: Initiator:sample.foobar.org Initiator:cluster.foobar.org/machine1 Initiator: The Initiator key enables the initiator to identify itself to the remote endpoint. The domain name should be that of the initiator. A zero-length domain name is interpreted as "other side of TCP connection". The target may silently ignore this key if it does not support it. For more security, a certificate-based protocol [TLS] may be used on the channel and take precedence over this pro- tocol. 10.3. Authenticator Authenticator: Examples: Authenticator:open-sesame The authenticator is a secret that the initiator uses to gain access to the target's LUNs. Satran, Smith, Sapuntzakis, Meth [Page 47] iSCSI June 2000 10.4. SendAuthenticator SendAuthenticator:yes Response: Authenticator: Examples: SendAuthenticator:yes -> Authenticator:alakazam The SendAuthenticator key is used to request from the party on the other side of the TCP connection to send its Authenticator. iSCSI devices may refuse to grant access until proper authentication has been performed by the parties involved. 10.5. AllowNoRTT AllowNoRTT: Response: AllowNoRTT: Exam- ples: AllowNoRTT:yes -> AllowNoRTT:yes The AllowNoRTT key is used to allow an initiator to send data to a target without the target having sent an RTT to the initiator. The default action is that RTT is required, unless both the initiator and the target send this key-pair attribute specifying AllowNoRTT:yes. Once AllowNoRTT has been set to 'yes', it cannot be set back to 'no'. 10.6. OriginalInitiator OriginalInitiator:[domainname[/modifier]] Examples: OriginalInitiator:sample.foobar.org The OriginalInitiator key is used to perform a proxy login from one target to another target in order to per- form a third-party operation (like COPY) for some initia- tor. The first target acts as the initiator for the second target, but it must provide the authorization information of the original initiator. Satran, Smith, Sapuntzakis, Meth [Page 48] iSCSI June 2000 10.7. Target2 Target2:domainname[/modifier] Examples: Target2:sample.foobar.org Target2:sample.foobar.org/raid2 The Target2 key is used in a third-party SCSI command (like COPY) between targets that do not lie on the same SCSI fabric. The initiator must specify the name of the distant target to the original target, so that the origi- nal target can Login to the distant target and then per- form the third-party command. Expires 15 December 2000 Satran, Smith, Sapuntzakis, Meth [Page 49] From owner-ips@ECE.cmu.edu Wed Jun 28 21:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18394; 28 Jun 2000 16:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA05073 for ips-outgoing; Wed, 28 Jun 2000 15:42:21 -0400 (EDT) Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA05069 for ; Wed, 28 Jun 2000 15:42:18 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id MAA12325; Wed, 28 Jun 2000 12:42:00 -0700 (PDT) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id MAA02533; Wed, 28 Jun 2000 12:38:19 -0700 (PDT) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Wed, 28 Jun 2000 12:38:19 -0700 (PDT) From: Costa Sapuntzakis To: julian_satran@il.ibm.com cc: ips@ece.cmu.edu Subject: 06/28 draft: MaxCmdRN scheme In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk I propose to remove MaxCmdRN from the iSCSI PDUs in which it currently appears and placing it in a separate iSCSI PDU. We need some way of ordering updates to MaxCmdRN. I propose the following mechanism: Update MaxCmdRN Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x8A) | 0 | MaxCmdRN | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| UpdateRN | 0 | +---------------------------------------------------------------+ MaxCmdRN: the maximum CmdRN that can be accepted by the client UpdateRN: the next CmdRN that is expect at the target Algorithm at initiator to update MaxCmdRN: The initiator keeps the following state for each session: session.maxCmdRN session.updateRN if packet.updateRN > session.updateRN || (packet.updateRN == session.updateRN && packet.maxCmdRN > session.maxCmdRN) session.maxCmdRN = packet.maxCmdRN (all arithmetic is done in sequence space) Solicit MaxCmdRN Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x0f) | 0 | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ Used to probe closed windows/get initial MaxCmdRN. Also, something needs to be said about shrinking windows. Is it allowed? -Costa From owner-ips@ECE.cmu.edu Wed Jun 28 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18581; 28 Jun 2000 19:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA08974 for ips-outgoing; Wed, 28 Jun 2000 17:43:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA08967 for ; Wed, 28 Jun 2000 17:43:05 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id OAA10093; Wed, 28 Jun 2000 14:42:59 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Wed, 28 Jun 2000 14:42:56 -0700 Message-ID: From: Jim McGrath To: "'Costa Sapuntzakis'" , scsi-tcp@external.cisco.com, ips@ece.cmu.edu Subject: RE: single vs multiple channels for iSCSI commands Date: Wed, 28 Jun 2000 14:42:55 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk The problem is that while such a solution may allow you to close on a protocol faster, it does not insure that you get products to market faster. Specifically, requiring the tape people to add new commands introduces more development time, and perhaps more importantly creates a division of interest. That is, the people responsible for supporting tape commands are not the same people doing iSCSI, so you might create a classic chicken and egg problem (no tape support for iSCSI until it gets to be important - and iSCSI market development is hindered by the lack of tape support). Mind, I am not thrilled about the alternative either. But solving the problem by creating new work for people "not in the room" may result in a pyrrhic victory. Jim PS in my opinion the only way to really solve this sort of problem is to identify your application space (or spaces) to allow for a concrete discussion of tradeoffs. For instance, initially FC ignored tape, focusing on disk, and got FC done more quickly - but eventually had to retrofit tape support. This may have been the best overall path, since the disk drive application was more important to the market than the tape drive application. However, for iSCSI (especially operated over a WAN) tape may be more important - this is the sort of conversation you have to have before making decisions that trade off the speed of adoption in different markets. -----Original Message----- From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] Sent: Tuesday, June 27, 2000 8:26 PM To: scsi-tcp@external.cisco.com; ips@ece.cmu.edu Subject: Re: single vs multiple channels for iSCSI commands I would like to stake out a position against any approach that involves using multiple simultaneous TCP connections per iSCSI session. The design of iSCSI (and the hardware that implements it) becomes significantly simpler when there is just 1 TCP connection/session. Parallel paths in the network can still be exploited by setting up multiple sessions with multiple TCP connections. So far, I have heard of two applications that allegedly require strict ordering: tape backup and remote asynchronous mirroring. New SCSI commands or even higher-layer techniques can be used to satisfy these applications. For example, I have heard that some high-bandwidth tape applications blast self-describing blocks of data to tape, making the order in which data gets read/written to tape less relevant. For the case of high bandiwdth remote asynchronous mirroring, a special SCSI remote asynchrnous mirroring command set could be introduced. I believe remote asynchronous mirroring principally consists of one command: WRITE, so the command set would probably be small. Ditching multiple simultaneous connections/session would help convince us of the correctness of iSCSI and allow us to move onto other important issues and help the time to market. -Costa App A. Possible remote asynch mirroring command sets 1) 1 CDB: ORDERED WRITE 2) 3 CDBs: WRITE UNCOMMITTED, COMMIT, ABORT From owner-ips@ECE.cmu.edu Thu Jun 29 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18760; 28 Jun 2000 20:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA11358 for ips-outgoing; Wed, 28 Jun 2000 19:27:59 -0400 (EDT) Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA11353 for ; Wed, 28 Jun 2000 19:27:56 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id QAA04547 for ; Wed, 28 Jun 2000 16:27:36 -0700 (PDT) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id QAA02596 for ; Wed, 28 Jun 2000 16:23:56 -0700 (PDT) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Wed, 28 Jun 2000 16:23:56 -0700 (PDT) From: Costa Sapuntzakis To: ips@ece.cmu.edu Subject: 16-bit CmdRN too small? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Is 16-bit CmdRN too small? A 16-bit CmdRN with a sliding window gives you the ability to issue max 32768 commands simulatenously. If each of these is an 8K write: 32768 * 8K = 256MB Which, on a 10GE connection, is about 200ms. If the round-trip time is greater than 200ms, then the initiator can't fill the pipe. -Costa From owner-ips@ECE.cmu.edu Thu Jun 29 07:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20170; 29 Jun 2000 3:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA17570 for ips-outgoing; Thu, 29 Jun 2000 01:51:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA17566 for ; Thu, 29 Jun 2000 01:51:09 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA78930 for ; Thu, 29 Jun 2000 07:50:24 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA117410 for ; Thu, 29 Jun 2000 07:50:28 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690D.00201248 ; Thu, 29 Jun 2000 07:50:18 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 29 Jun 2000 08:49:19 +0300 Subject: MaxCmdRN and 16 bit Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Making a separate PDU was on our mind too. It allows a cleaner implementation and handled two-way (an initiator to target PDU) it may solve also the StatRN thing (that left for piggyback will strain the header). As for the 16 bit - it means 150k I/Os per second in the first stage of the pipeline! Anyhow if we extend it to 4 byte then we may want to go for a 48 byte header. We should do a tally on that on today's call. Julo ---------------------- Forwarded by Julian Satran/Haifa/IBM on 29/06/2000 08:28 --------------------------- Costa Sapuntzakis on 28/06/2000 22:38:19 Please respond to Costa Sapuntzakis To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: 06/28 draft: MaxCmdRN scheme I propose to remove MaxCmdRN from the iSCSI PDUs in which it currently appears and placing it in a separate iSCSI PDU. We need some way of ordering updates to MaxCmdRN. I propose the following mechanism: Update MaxCmdRN Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x8A) | 0 | MaxCmdRN | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ 8| UpdateRN | 0 | +---------------------------------------------------------------+ MaxCmdRN: the maximum CmdRN that can be accepted by the client UpdateRN: the next CmdRN that is expect at the target Algorithm at initiator to update MaxCmdRN: The initiator keeps the following state for each session: session.maxCmdRN session.updateRN if packet.updateRN > session.updateRN || (packet.updateRN == session.updateRN && packet.maxCmdRN > session.maxCmdRN) session.maxCmdRN = packet.maxCmdRN (all arithmetic is done in sequence space) Solicit MaxCmdRN Byte / 0 | 1 | 2 | 3 | / | | | | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ 0| Opcode (0x0f) | 0 | +---------------+---------------+---------------+---------------+ 4| Length | +---------------+---------------+---------------+---------------+ Used to probe closed windows/get initial MaxCmdRN. Also, something needs to be said about shrinking windows. Is it allowed? -Costa Costa Sapuntzakis on 29/06/2000 02:23:56 Please respond to Costa Sapuntzakis To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: 16-bit CmdRN too small? Is 16-bit CmdRN too small? A 16-bit CmdRN with a sliding window gives you the ability to issue max 32768 commands simulatenously. If each of these is an 8K write: 32768 * 8K = 256MB Which, on a 10GE connection, is about 200ms. If the round-trip time is greater than 200ms, then the initiator can't fill the pipe. -Costa From owner-ips@ECE.cmu.edu Thu Jun 29 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20617; 29 Jun 2000 4:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA18718 for ips-outgoing; Thu, 29 Jun 2000 03:08:43 -0400 (EDT) Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA18714 for ; Thu, 29 Jun 2000 03:08:40 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id AAA00253 for ; Thu, 29 Jun 2000 00:08:21 -0700 (PDT) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id AAA02990 for ; Thu, 29 Jun 2000 00:04:40 -0700 (PDT) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Thu, 29 Jun 2000 00:04:40 -0700 (PDT) From: Costa Sapuntzakis To: ips@ece.cmu.edu Subject: Notes of 06/21 meeting Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Action items: Action item: get rid of command reference number in SCSI response Action item: get rid of immediate data in SCSI response Unresolved question: Should we allow multiple RTTs to be pending simultaneously? If so, how many? -------------- The day started of with a discussion of SCSI error recovery. The main motivation behind iSCSI layer error recovery was tape drives. Many tape backup applications cannot recover from aborted commands. It was pointed out that the command reference number as spec'ed was not long-lived enough to provide error recovery. However, the task tag could be used to do error recovery. -------------- There was then a discussion about whether the command reference number should be per LU or per session. The per-LU advocates pointed out that there are no inter-LU ordering constraints in SCSI SAM. Per-LU provides greater parallelism and less head-of-line blocking by providing more lines. The per-session folks pointed out that LUs are not an iSCSI layer concept. What's more, adding command reference numbers per LU would necessitate adding state per LU into the iSCSI drivers at both ends. Unfortunately, there was no way of bounding or reclaiming this state. ---------------- There was a lot of talk about whether we want to support multiple TCP connections/session. John Hufferd pointed out that SCSI load balancers already exist that take advantage of multiple sessions (multiple SCSI busses) to stripe commands to a target. He argued that multiple TCP connections are unnecessary. He also argued that no applications make effective use of SCSI ORDERED attribute, because the interface are not there. However, they have to stop and wait for ordered commands. One application where stop and wait hurts is tape (where all writes are ordered), so some tape applications write self-describing blocks to tape which can be written in any order. Remote asynchronous mirroring can also be done with ordered writes. Hufferd argued that remote asynchronous mirroring must be solved at a higher layer and is being solved today. Most of those arguing for multiple TCP connection said that - it isn't that hard - it would make iSCSI better than other SCSI transports - it would make high-perf apps easier to write ---------------- Can a single initiator have multiple sessions? The answer was yes. An initiator is named by the initiator ID which we think will be the same as the Access ID in Hafner's proposal. Multiple sessions can be used to separate different classes of traffic (bulk vs. latency sensitive) or even multiple processes ------------- Deadlock: Luciano pointed out that it is possible to run out of buffers and deadlock with multiple TCP connections. The source of the problem is 1) receive too many out-of-order commands 2) receiving too much unsolicited (immediate) data The solution to 1) is to either - limit the number of out-of-order commands that are read from each TCP pipe to 1 (requires NIC to know that command is out-of-order) and then stop reading from the connections (deskewing) - have a windowing mechanism on the command ordering queue in target - have a separate TCP pipe for emergency recovery commands - Nuspeed aborts command with SCSI status TASK QUEUE FULL The consensus seems to have resulted in windowing being adopted. The consensus solution to 2) was to allow the target to drop immediate data and request it be retransmited via ready-to-transmit (RTT). -------------- Should task management commands be ordered with respect to tasks? Those against feared that ordering task mangement commands would prevent their timely delivery. Those for feared that not ordering task management commands would lead to surprising behaviors (like ABORT TASK SET overtaking and not aborting all previously issued tasks). ---------------- Can a single iSCSI TCP connection use multiple paths in the network simultaneously? Answer: Most networks keep a flow on one path to help ensure minimal re-ordering, so no in that case. Of course, this being IP, people could design a network that sprays packets of a flow across multiple paths and it would still work... From owner-ips@ECE.cmu.edu Thu Jun 29 12:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22051; 29 Jun 2000 8:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA29671 for ips-outgoing; Thu, 29 Jun 2000 06:54:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA29666 for ; Thu, 29 Jun 2000 06:54:48 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id MAA140730 for ; Thu, 29 Jun 2000 12:54:12 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id MAA129522 for ; Thu, 29 Jun 2000 12:54:16 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690D.003BE54D ; Thu, 29 Jun 2000 12:54:13 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 29 Jun 2000 13:47:57 +0300 Subject: ordered command retrieval by LUN Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Looking back at my Haifa meeting notes I found that I owe you all a scheme that will allow ordered command retrieval by LUN (with the net effect that missing commands for one LU will not affect other LUs. And all this without having the iSCSI layer at the initiator number commands per LU. Assume that for any issued command within the window the initiator keeps a record of the LUN it was issued for (i.e. an array of LUN indexed by the command number - sort of). A simple error detection, that will allow operation in cases where gaps are at most one command, requires only to transmit with every command the LUN of the previous command; the target can deliver the commands to LU even with gaps if the gaps are of a single command. To cope with larger gaps the target has to issue an enquiry about the missing commands to the initiator and get their LUNs. The scheme is simple for 1 entry gaps but not very interesting for multiple entry gaps. However the sent commands array might be considered for a faster recovery from connection failures. Julo From owner-ips@ECE.cmu.edu Thu Jun 29 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22460; 29 Jun 2000 10:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA03056 for ips-outgoing; Thu, 29 Jun 2000 09:40:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA03048 for ; Thu, 29 Jun 2000 09:40:43 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id PAA58662 for ; Thu, 29 Jun 2000 15:40:07 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id PAA58656 for ; Thu, 29 Jun 2000 15:40:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690D.004B13A4 ; Thu, 29 Jun 2000 15:40:02 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 29 Jun 2000 16:38:50 +0300 Subject: Re: Proposed Connection Recovery Additions for Draft 03 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, Excellent job. It described mostly what I had in mind. With your permission I will use even your wording whenever possible. I thought I had the AckStatRN (MaxStatRN) but I could not piggyback it on the right messages - the headers are too short. Also if you think about a layered implementation - Costa's proposal to use a separate PDU makes sense as those are generated by iSCSI alone and convey values of iSCSI registers. I would like to hear what everybody has to say about it. The alternative looks like increasing the header (48?) I would also like to clarify for implementors that device-recovery is to be based on Initiator Tag as it it is the only identifier unchanged for the life of the task while the two numberings are only for the delivery mechanism (iSCSI). Regards, Julo From owner-ips@ECE.cmu.edu Thu Jun 29 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22475; 29 Jun 2000 10:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA03556 for ips-outgoing; Thu, 29 Jun 2000 09:58:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA03551 for ; Thu, 29 Jun 2000 09:57:57 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id PAA97128 for ; Thu, 29 Jun 2000 15:57:09 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id PAA183510 for ; Thu, 29 Jun 2000 15:57:23 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690D.004CA883 ; Thu, 29 Jun 2000 15:57:18 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 29 Jun 2000 16:56:17 +0300 Subject: Re: Digest Login, iSCSI Discovery, and Other Draft 03 Comments Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk 1. version is already in my working draft! 2. we will have to merge it with whatever comes from Luciano & Paul 3.that assumption was made for SSL 1. anything related to discovery might be a bit premature. What you propose look like a neat thing but I don't know how it will merge with the management scheme, administrative access. I though that reserving the name is a thing we should do now but not more 2. Queue full - Is a SCSI layer thing. I think that passing over MaxCmdRN should have the command discarded with iSCSI protocol error (or something of this nature) 2. I can reinstate the bit. But control commands can have data and in this sense are writes. R/W was meant only to indicate the meaning of ExpectedDataLength (to avoid scaning the CDB) Regards, Julo Mark Bakke on 29/06/2000 16:13:32 Please respond to Mark Bakke To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Digest Login, iSCSI Discovery, and Other Draft 03 Comments Here are some non-connection-recovery-related thoughts on Draft 03; mainly concerning digest logins and iSCSI target discovery. Logins for iSCSI draft 03 might benefit from the following additions: 1. Version number text fields should be placed in both login requests and responses. the request could either specify the version of the protocol the initiator wishes to run, or perhaps a list of versions that the initiator is capable of running. Version: 1.0 -OR- Version: 1.0 1.1 1.2 The target would then choose the highest version that it supported in common with those requested, and return it in the response: Version: 1.1 2. Login with username and password. It seems that the scheme already proposed (a digest scheme) would work quite well; we can re-use the text headers defined for HTTP/1.1, with some simplification (see RFC 2617). The initiator would send the initial login message, which would be failed by the target with status == 2 (additional authentication required; just like HTTP 401). The following text fields would be added to the response by the target: Authenticate: Digest Nonce: text-string (hex or uuencode of random value) The Nonce is used to provide a temporary value with which to hash the username and password, preventing replay of the password on another TCP connection. The nonce is randomly generated by the target, and should be thrown out after being used to validate the login for a particular connection. The initiator would then concatenate the nonce, user-name, and password, and compute their MD5 hash. It would send a new login request to the target: Authenticator: Digest user-name hex-hashed-response Nonce: (echo of hex-string) Where hex-hashed-response = md5(concat(nonce, ":", user-name, ":", password)) Note that if a given nonce value is only used on a single connection, we really don't have to echo it here as they do in HTTP/1.1. 3. Underlying encryption mechanisms could be either negotiated by name at login, or could just be assumed on different TCP ports; CIFS and telnet are done over SSL without modification to their protocols; perhaps IPSec is similar? For the most part, I think that any of these mechanisms supported by the target and initiator, or transparently by network security devices would work with the protocol, and that there shouldn't be a lot to specify here. However, it might be worth thinking about how user-names and passwords are tied in to those lower levels. -------------------------------------------------------- iSCSI Target and LUN Discovery 1. The target named "iSCSI-SYS" in Draft 03 looks good. 2. We would like to add the sendTargets text command - this is a value-add command for NuSpeed, but would be a good addition to the spec, as it eliminates most of the need to configure the iSCSI driver in the initiator. Targets that do not wish to report this information can deny the command. Normally, only the targets to which an initiator should have access are returned. Here's the text command we use now. We log into iSCSI-SYS, use sendTargets to find the targets, then log into each target. The command: com.nuspeed.sendTargets:please The response is a list of "target:" target:jbod_one target:tape_three ... We would rename this command to just sendTargets. 3. Async notification for a new LUN in an existing, connected target is in the draft, but there's no notification for a newly-created target. This notification would work with our sendTargets mechanism, and would probably just define an event type to ask the initiator to re-request the target list. -------------------------------------------------------- Flow Control 1. Right now we are sending Queue Full back to handle flow control from the iSCSI target; someone indicated that this may break some host SCSI implementations. Is this correct? Anyway, will the MaxCmdRN scheme handle this effectively? 2. MaxCmdRN has found its way into quite a few response headers. This probably does not need to be updated that often, most likely when watermarks are reached. Costa's separate message looks like would be cleaner. -------------------------------------------------------- Data and Command CRC (or other checksum/hash/whatever) - More to come; we would like to see these negotiated at login, and have separate CRCs for iSCSI headers and actual data. -------------------------------------------------------- Some additional notes on iSCSI draft 03. 1. Immediate Data - this is a nice optimization for small data messages, since they cut down on data headers, and eliminate another message to parse on the target. However, using this for large data can get around a target that wants to use RTT. Should RTT be required to used immediate data, or should the length of immediate data be limited to some fixed (or negotiated) maximum (a few kbytes)? 2. The Write bit is gone from the header; clearing both the Read and Write bits implied a Control message before. Without the Write bit, I assume that an expected data length of zero would indicate a Control message instead. -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Thu Jun 29 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22462; 29 Jun 2000 10:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA02149 for ips-outgoing; Thu, 29 Jun 2000 09:10:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA02145 for ; Thu, 29 Jun 2000 09:10:25 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 2E7BE837F3 for ; Thu, 29 Jun 2000 08:08:38 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N5PCKWYA; Thu, 29 Jun 2000 08:16:08 -0500 Message-ID: <395B4ACC.136F050@nuspeed.com> Date: Thu, 29 Jun 2000 08:10:36 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Proposed Connection Recovery Additions for Draft 03 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Draft 03 seems to be pretty close as far as dealing with connection recovery. Here are a few additions that we (NuSpeed) think will help complete the picture, along with what we believe are some of the requirements. Other than hopefully clarifying initiator and target behavior, this scheme adds one field to the Command Request, and one event type to the Asynchronous Event message. I've attempted to include some of our reasoning behind this. Assumptions - iSCSI is only a transport for SCSI. Its recovery scheme does not attempt to retry failed commands. However, it is a reliable transport for SCSI, and must deliver commands within a session in order to the target. - TCP handles any losses from a given connection. The target end of the byte stream is either valid, or the connection is lost. Within a connection, there is no such thing as losing a packet. We will, however, need to deal with stronger error checking over the SCSI data, but that's (mostly) orthogonal to connection recovery. - This scheme will work with either single or multiple connections per session. It is up to the implementation of the initiator and target whether either one supports multiple connections. The initiator can simply not use more than one connection per session; the target can deny the login if the client requests more than one connection per session. - It is also up to the initiator to determine whether to multiplex access to targets and luns over a single session, or to use a session for each, or some combination. If the initiator chooses to use multiple sessions to the same device, it must be prepared to deal with multipath command ordering issues itself. - We can be fairly optimistic about the longevity of TCP connections; if the network is so slow, overloaded, or poorly designed as to lose connections regularly, it is not likely a good candidate for storage access. Connection recovery should still handle these cases, especially if the problems are transient, but need not be optimized for these cases. If a connection fails, it should be acceptable to re-send write data with the re-sent command, and to re-send read data with the re-sent status. There may, however, be simple optimizations to avoid this, too, especially when transporting larger blocks, such as tape reads and writes. - Some commands, such as FORMAT UNIT and REWIND, may take several minutes or more to complete. Thousands of operations may complete before status is returned for these commands. (Are 16-bit reference numbers enough?) - If multiple connections are used, they are symmetrical (no special control or data connections). Command-Data-Status connection allegiance is also assumed, and CmdRN and StatRN are used to ensure in-order delivery. - CmdRN and StatRN are implemented as in Draft 03, as 16-bit, per-session incrementing counters. Requirements for Connection Recovery - Protocol fields associated with the connection recovery scheme will work with either a single connection per session, or multiple connections per session. - iSCSI must preserve ordered delivery within a session. - The transport may re-send commands, data, and status at any time, but must not attempt to re-try the actual command at the target without involving the upper (SCSI) layer for recovery. This means that, as in section 4.1 of draft 03, the client should keep sufficient information handy to re-send commands and data until status is received. - We can generally make an exception to the above for commands issued to a block device (disk); reads and writes are idempotent, as long as the commands are re-issued in the original order. - Commands must be issued at the target end of an iSCSI session in-order, but status may, of course, be returned from the iSCSI target to the initiator in any order. - Either the initiator or target may decide to terminate a connection. It is the responsibility of the initiator to reconnect if it so chooses. - A connection must be recoverable quickly. At most, a connection must fail, be detected as failed, be restarted, have commands reissued, and get status back (except on high-latency commands) within a portion of the normal SCSI timeout window (30 seconds). The actual time for this depends on the network, the commands issued, etc. At any rate, connection recovery must be as transparent as possible to the end user or application. - Connection recovery should work for target reboot or failover. - Basically, we have to handle the following steps for each connection: 1. Detection - deciding when a connection is down, or should be. 2. Disconnection - terminating a connection. 3. Reconnection - re-connecting to the target. 4. Resend - re-sending commands to the target. Besides these procedures, normal mechanisms such as reference numbers and response caching will be in place to support these procedures when they are needed. Support Mechanisms The initiator and target must keep some state around in order to support connection recovery and resending of commands, data, and status that may have been lost. Their responsibilities are outlined in section 4.1 of Draft 03. Basically, an Initiator must: - Increment CmdRN for each new command request sent. - Keep information required to rebuild and resend each command with its data until the matching command response is received from the target. - Acknowledge command responses soon after they are received from the target. A Target must: - Increment StatRN for each new status response sent. - Keep a cache of responses (status & sense data) until the StatRN is acknowledged by the initiator. - For non-disk devices, keep data response (read data) along with the cached command response (although this might be difficult with large-block devices). Reclaiming Cached Responses - section 4.1 already mentioned most of the above; however, there was no mechanism for notifying the target that its cached responses were no longer needed. In this scheme, an AckStatRN is sent from the initiator to the target, as the highest (honoring wrap) consecutive value received for StatRN in a response on any connection in the session. All cached responses up to and including this StatRN value may be safely de-allocated. Detecting Connection Failure During an initiator, target, or intervening network outage, whether temporary or permanent, TCP connections will normally be retried for much longer than most SCSI drivers can handle. In many cases, new connections can be made and started long before the old connection times out. For this reason, we have to detect connections that have gone away. Both the initiator and the target may detect these conditions, and should detect them in a timely manner (let's say 5 seconds for now, but we need to think about this). From the initiator's point of view, the connection can fail for several reasons (temporary or permanent): - Target powered down or removed from network - Target reboot or failover - Lost network route - Backed-off (slow) tcp connection - Unexpected message fields received (software error on target)? If no responses are being received from the target, and there are outstanding commands, the initiator will periodically send a ping request, and expect a ping response within a small amount of time. If no ping response is received, the connection is considered to have failed. This is mentioned in section 4.1 as well. From the target's point of view, the connection can fail for several reasons (temporary or permanent): - Initiator powered down or removed from network - Initiator reboot or failover - Lost network route - Backed-off (slow) tcp connection - Unexpected message fields received (software error on initiator)? Since the target does not send requests, it could do one of two things: 1. During the login phase, negotiate a maximum inactivity time for the incoming target connection. If this time will be exceeded, the client promises to send an iSCSI ping request on the connection to keep it alive. If the inactivity timer expires on the target, the connection is assumed to have failed. 2. Add an asychronous event requesting that the initiator ping the target. Send this when approaching the target's maximum inactivity time; if the timer expires anyway, the connection is assumed to have failed. In any case, the target must detect connection failure to avoid having connections from powered-down clients hang around for long periods of time. Disconnecting When a connection fails, the initiator, target, or both will close it. The initiator can generally not wait around for the close to complete before starting a new connection; the target will need to accept a new (recovered) connection from an initiator, even if the target has not realized the original connection's failure. These are implementation issues. The initiator may disconnect for reasons other than failure: - Normal host shutdown (reboot or power off) - Application (and disk) failover to another host (e.g. using HP, Veritas, or other application failover software). The target may also disconnect for reasons other than failure: - If the target is to be rebooted or failed over to another physical unit, it may wish to gracefully shut down the connection before restarting another. To make target reboot or failover more graceful, a target should attempt to send an asynchronous event "connection shutdown", to the initiator on each connection. This new event contains two values: - MaxUpTime - the number of seconds (can be zero) before this connection is expected to cease functioning. The initiator should not attempt to issue more commands than can be expected to complete and receive status within this amount of time. The target will wait this amount of time before it shuts its connections down. - MinHoldTime - the number of seconds (can also be zero) after MaxUpTime before this entity will be available for re-connection. After this, the initiator has a good chance of reconnecting to the target. This should be set to the amount of time the server is expected to take to fail over, reboot, etc. We should probably define a value (-1?) for "never". Note that an initiator could just reconnect right away, however, it could either connect to the running server just before it reboots, or it could lose several SYN segments while waiting for the server, causing exponential backoff to make the ultimate connection take longer. Reconnection The initiator always handles reconnection. During the new connection's login phase, the initiator specifies that it is replacing a failed connection by including the non-zero CID of the old connection in the RecoverCID field. If a target supports stateful recovery (meaning it still has the cached responses for the session), it accepts the login. If the target does not support stateful recovery, or the target has rebooted and lost its state, or the target has dropped the cached responses due to an excessive amount of time passing (perhaps 60 seconds), it rejects the login with a "reject recovery" status. The initiator then performs a new login, and does stateless recovery. 1. Stateful Recovery In a stateful recovery, the initiator resends all commands for which it has not received status. If a command has already completed, the cached response is returned. If a command has already been issued and is in progress, it is not re-issued; and will just be queued somewhere to wait for status. If a command had not been received by the target (or incompletely received and thrown away), it will be issued as normal. 2. Stateless Recovery By default, stateless recovery means that all outstanding commands are terminated (to the SCSI layer) (check condition?); higher layers must perform recovery. Non-Recovery Let's face it; there are times when things just can't be recovered at this level. However, there are many higher-level entities that may recover for us: - Tape backup software (reload into a different tape drive) - Volume managers (break mirrors) - Multipath SCSI drivers (find alternate path or controller) - Host application clusters (move app to host with connectivity) This should be handled as specified in section 4.3. Optimizations: 1. If RTT is in use, and a write request is re-sent to a target, and the target has already written the data, the target could send the Command Response back instead of the RTT. The initiator would just accept this as the final status, and would not have to send the write data again. iSCSI Draft 03 Message Modifications: 1. Remove StatRN from the Data Response, or make it equal the StatRN for the matching Command Response. There should be no need for it to increment separatly from the Command Response, since this scheme assumes that if the response was not received, the data will be re-sent anyway. The current draft does not specify how StatRN is used in a Data Response. 2. Add an AckStatRN field to Command Request, to acknowledge the highest (honoring wrap) consecutive StatRN received for the session. 3. Add a new event (Event Indicator 5) specifying that the connection will be closed by the target. This event sends two parameters (using some of the reserved fields): - MaxUpTime - the number of seconds the target intends to keep the connection alive. - MinHoldTime - the number of seconds the initiator should wait before establishing a new connection. Alternative Implementations 1. We considered a separate message to send the AckStatRN, but since this is generally done for every command, it seemed simpler to just piggyback it on the next command request. 2. CmdRN and StatRN are assumed to be per-session. If they were made per-LUN for any reason, the initiator and target would simply have to demux requests and responses based on LUN + RN. A Few Alternatives that Didn't Quite Work Here are some alternatives we went through, and why we did not choose them: 1. We considered just tossing out StatRN, and acking the CmdRN (or the ITT) matching the status last received instead. However, the target needed to keep track of status in the order in which it was sent (and NOT in the order in which the original command was received) to avoid trouble with commands which incur a long response delay (REWIND et al). Keeping StatRN and acking it is much simpler, and makes it easier to preserve response ordering if multiple connections are used (and if response ordering is required). 2. Target Retries - if one just assumes disk, the target could avoid caching responses, and the initiator could avoid acking them; the target could just retry any requests, in order, sent over the new connection. However, this excludes most of the SCSI Peripheral Device Types, and will likely not work in every case for this either. In these cases, ALL connection recovery would be pushed up to SCSI to handle. By caching status, we remain a truer transport, and will work better for these devices (especially tape). 3. Selective StatRN acks - one could individually acknowledge each response, to free its resources on the target. However, if a selective ack is lost during a connection recovery, its resources would then hang around forever on the target (unless, of course, we wanted to ack the ack). Cumulative acks may be lost at the end of a connection; the next command sent will just re-ack everything anyway. -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Thu Jun 29 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22458; 29 Jun 2000 10:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA02216 for ips-outgoing; Thu, 29 Jun 2000 09:13:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA02210 for ; Thu, 29 Jun 2000 09:13:21 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id EB06B837F3 for ; Thu, 29 Jun 2000 08:11:33 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N5PCKWYD; Thu, 29 Jun 2000 08:19:04 -0500 Message-ID: <395B4B7C.4FCFEB85@nuspeed.com> Date: Thu, 29 Jun 2000 08:13:32 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Digest Login, iSCSI Discovery, and Other Draft 03 Comments Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Here are some non-connection-recovery-related thoughts on Draft 03; mainly concerning digest logins and iSCSI target discovery. Logins for iSCSI draft 03 might benefit from the following additions: 1. Version number text fields should be placed in both login requests and responses. the request could either specify the version of the protocol the initiator wishes to run, or perhaps a list of versions that the initiator is capable of running. Version: 1.0 -OR- Version: 1.0 1.1 1.2 The target would then choose the highest version that it supported in common with those requested, and return it in the response: Version: 1.1 2. Login with username and password. It seems that the scheme already proposed (a digest scheme) would work quite well; we can re-use the text headers defined for HTTP/1.1, with some simplification (see RFC 2617). The initiator would send the initial login message, which would be failed by the target with status == 2 (additional authentication required; just like HTTP 401). The following text fields would be added to the response by the target: Authenticate: Digest Nonce: text-string (hex or uuencode of random value) The Nonce is used to provide a temporary value with which to hash the username and password, preventing replay of the password on another TCP connection. The nonce is randomly generated by the target, and should be thrown out after being used to validate the login for a particular connection. The initiator would then concatenate the nonce, user-name, and password, and compute their MD5 hash. It would send a new login request to the target: Authenticator: Digest user-name hex-hashed-response Nonce: (echo of hex-string) Where hex-hashed-response = md5(concat(nonce, ":", user-name, ":", password)) Note that if a given nonce value is only used on a single connection, we really don't have to echo it here as they do in HTTP/1.1. 3. Underlying encryption mechanisms could be either negotiated by name at login, or could just be assumed on different TCP ports; CIFS and telnet are done over SSL without modification to their protocols; perhaps IPSec is similar? For the most part, I think that any of these mechanisms supported by the target and initiator, or transparently by network security devices would work with the protocol, and that there shouldn't be a lot to specify here. However, it might be worth thinking about how user-names and passwords are tied in to those lower levels. -------------------------------------------------------- iSCSI Target and LUN Discovery 1. The target named "iSCSI-SYS" in Draft 03 looks good. 2. We would like to add the sendTargets text command - this is a value-add command for NuSpeed, but would be a good addition to the spec, as it eliminates most of the need to configure the iSCSI driver in the initiator. Targets that do not wish to report this information can deny the command. Normally, only the targets to which an initiator should have access are returned. Here's the text command we use now. We log into iSCSI-SYS, use sendTargets to find the targets, then log into each target. The command: com.nuspeed.sendTargets:please The response is a list of "target:" target:jbod_one target:tape_three ... We would rename this command to just sendTargets. 3. Async notification for a new LUN in an existing, connected target is in the draft, but there's no notification for a newly-created target. This notification would work with our sendTargets mechanism, and would probably just define an event type to ask the initiator to re-request the target list. -------------------------------------------------------- Flow Control 1. Right now we are sending Queue Full back to handle flow control from the iSCSI target; someone indicated that this may break some host SCSI implementations. Is this correct? Anyway, will the MaxCmdRN scheme handle this effectively? 2. MaxCmdRN has found its way into quite a few response headers. This probably does not need to be updated that often, most likely when watermarks are reached. Costa's separate message looks like would be cleaner. -------------------------------------------------------- Data and Command CRC (or other checksum/hash/whatever) - More to come; we would like to see these negotiated at login, and have separate CRCs for iSCSI headers and actual data. -------------------------------------------------------- Some additional notes on iSCSI draft 03. 1. Immediate Data - this is a nice optimization for small data messages, since they cut down on data headers, and eliminate another message to parse on the target. However, using this for large data can get around a target that wants to use RTT. Should RTT be required to used immediate data, or should the length of immediate data be limited to some fixed (or negotiated) maximum (a few kbytes)? 2. The Write bit is gone from the header; clearing both the Read and Write bits implied a Control message before. Without the Write bit, I assume that an expected data length of zero would indicate a Control message instead. -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Thu Jun 29 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22656; 29 Jun 2000 12:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA05293 for ips-outgoing; Thu, 29 Jun 2000 10:56:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA05289 for ; Thu, 29 Jun 2000 10:56:47 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 2662F17CD for ; Thu, 29 Jun 2000 07:56:46 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id IAA07111 for ; Thu, 29 Jun 2000 08:06:00 -0700 (PDT) Message-Id: <4.2.2.20000629070252.00a6cb80@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Thu, 29 Jun 2000 07:23:06 -0700 To: ips@ece.cmu.edu From: Michael Krause Subject: Re: Notes of 06/21 meeting In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 12:04 AM 6/29/00 -0700, Costa Sapuntzakis wrote: >It was pointed out that the command reference number >as spec'ed was not long-lived enough to provide error >recovery. However, the task tag could be used to do >error recovery. This is why it should be at least a 32-bit value and possibly a 48-bit value. It should also be done on all commands to simplify the problems later described, i.e. multiple TCP op support, the ability to deal with overflows (only receive what is within the window of support ops and let a SACK-like error recovery deal anything lost), simplifies the hardware (always present and can be used to retain ordering without much overhead), simplifies mirroring since one can immediately forward the ops in the order the initiator wanted without stalls, etc. >-------------- > >There was then a discussion about whether the command >reference number should be per LU or per session. Large value per session and then it does not matter. >There was a lot of talk about whether we want >to support multiple TCP connections/session. > >John Hufferd pointed out that SCSI load balancers already exist >that take advantage of multiple sessions (multiple SCSI busses) >to stripe commands to a target. He argued that multiple >TCP connections are unnecessary. He also argued that no applications >make effective use of SCSI ORDERED attribute, because the >interface are not there. Very simple implementation can be built with multiple TCP connections / per session. With the command reference numbers always sent on operation, the start / stop problem is mitigated because one is receiving / processing the operations in the order they were received. In addition, one can develop the hooks that separate specs provide for arbitration policies, QoS, etc. to deal with different link bandwidth / etc. attributes. >However, they have to stop and wait for ordered commands. >One application where stop and wait hurts is tape (where >all writes are ordered), so some tape applications write >self-describing blocks to tape which can be written in any order. > >Remote asynchronous mirroring can also be done with ordered >writes. Hufferd argued that remote asynchronous mirroring must >be solved at a higher layer and is being solved today. Not that difficult to do with what I described above. >Most of those arguing for multiple TCP connection said that > - it isn't that hard > - it would make iSCSI better than other SCSI transports > - it would make high-perf apps easier to write Add in - Multi-path support is much easier to implement. - Higher performance can be achieved - Implementations are fairly simple - minimal state - Application transparent ability to take advantage / recover from hot-plug / removal of fabric components >------------- >Deadlock: > >Luciano pointed out that it is possible to run out of >buffers and deadlock with multiple TCP connections. > >The source of the problem is > 1) receive too many out-of-order commands > 2) receiving too much unsolicited (immediate) > data > >The solution to 1) is to either > - limit the number of out-of-order commands that > are read from each TCP pipe to 1 (requires NIC > to know that command is out-of-order) and then > stop reading from the connections (deskewing) > - have a windowing mechanism on the command > ordering queue in target > - have a separate TCP pipe for emergency > recovery commands > - Nuspeed aborts command with SCSI status TASK QUEUE FULL > >The consensus seems to have resulted in windowing >being adopted. The NIC does not have to track this per se. If the NIC has the SGL for the target buffer it can perform the DMA. If the SGL does not exist, then it can drop the message without issuing a TCP ACK (Issue is whether one wants to slow this down at the TCP level or allow it to complete but have the NIC still drop the buffers w.r.t. the DMA targeting - preference is to complete from TCP point of view but drop the DMA operation). The operation target and buffers are locally posted so the rate can be controlled quite easily. The windowing proposal will work well as a control point for SGL posting to individual commands - again with minimal if any complexity. If the command reference number is always present, life can be further simplified. >The consensus solution to 2) was to allow the >target to drop immediate data and request it be >retransmited via ready-to-transmit (RTT). > >-------------- > >Should task management commands be ordered with respect to tasks? > >Those against feared that ordering task mangement commands >would prevent their timely delivery. > >Those for feared that not ordering task management commands >would lead to surprising behaviors (like ABORT TASK SET >overtaking and not aborting all previously issued tasks). > >---------------- > >Can a single iSCSI TCP connection use multiple paths in the network >simultaneously? > >Answer: Most networks keep a flow on one path to help ensure >minimal re-ordering, so no in that case. Of course, this being IP, >people could design a network that sprays packets of a flow across >multiple paths and it would still work... Most of us would prefer to not have a single connection flow through different paths - the complexity to the hardware for what is nominally a rare event would be increased. A well-behaved environment is possible to implement but then one is asking for IP to do this and creating additional specification work. Mike From owner-ips@ECE.cmu.edu Thu Jun 29 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22652; 29 Jun 2000 12:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA05120 for ips-outgoing; Thu, 29 Jun 2000 10:50:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA05115 for ; Thu, 29 Jun 2000 10:50:44 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 0C9C1837F3; Thu, 29 Jun 2000 09:48:56 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N5PCKW53; Thu, 29 Jun 2000 09:56:26 -0500 Message-ID: <395B624E.1B675639@nuspeed.com> Date: Thu, 29 Jun 2000 09:50:54 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: Proposed Connection Recovery Additions for Draft 03 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian- Permission is granted to use anything we've sent. Anyway, increasing the command request header to 48 is probably necessary to fit everything in, especially given Costa's thinking on 32-bit CmdRN and StatRN in the presense of commands that can take a long time. Talk to you in 10 minutes, Mark julian_satran@il.ibm.com wrote: > > Mark, > > Excellent job. It described mostly what I had in mind. > > With your permission I will use even your wording whenever possible. > > I thought I had the AckStatRN (MaxStatRN) but I could not piggyback it on > the right messages > - the headers are too short. > > Also if you think about a layered implementation - Costa's proposal to use > a separate > PDU makes sense as those are generated by iSCSI alone and convey values of > iSCSI registers. > > I would like to hear what everybody has to say about it. The alternative > looks like increasing the > header (48?) > > I would also like to clarify for implementors that device-recovery is to be > based on Initiator Tag as it > it is the only identifier unchanged for the life of the task while the two > numberings are only for > the delivery mechanism (iSCSI). > > Regards, > Julo -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Thu Jun 29 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22789; 29 Jun 2000 13:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07737 for ips-outgoing; Thu, 29 Jun 2000 12:07:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07733 for ; Thu, 29 Jun 2000 12:07:48 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 88DCA1F3C for ; Thu, 29 Jun 2000 09:07:47 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id JAA08466 for ; Thu, 29 Jun 2000 09:17:03 -0700 (PDT) Message-Id: <4.2.2.20000629090236.00ac9ef0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Thu, 29 Jun 2000 09:03:11 -0700 To: ips@ece.cmu.edu From: Michael Krause Subject: Re: 16-bit CmdRN too small? In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:23 PM 6/28/00 -0700, Costa Sapuntzakis wrote: >Is 16-bit CmdRN too small? > >A 16-bit CmdRN with a sliding window gives you the ability to issue >max 32768 commands simulatenously. > >If each of these is an 8K write: > >32768 * 8K = 256MB > >Which, on a 10GE connection, is about 200ms. If the round-trip time >is greater than 200ms, then the initiator can't fill the pipe. Definitely too small. 32-bit or even 48-bit with epochs to deal with some types of ghost I/O would be prefered. Mike From owner-ips@ECE.cmu.edu Thu Jun 29 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23272; 29 Jun 2000 17:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15584 for ips-outgoing; Thu, 29 Jun 2000 15:42:26 -0400 (EDT) Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15561 for ; Thu, 29 Jun 2000 15:42:07 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id MAA13969 for ; Thu, 29 Jun 2000 12:41:48 -0700 (PDT) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id MAA03408 for ; Thu, 29 Jun 2000 12:38:05 -0700 (PDT) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Thu, 29 Jun 2000 12:38:05 -0700 (PDT) From: Costa Sapuntzakis To: ips@ece.cmu.edu Subject: 06/28 Draft with change bars Message-ID: MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-1804928587-962307485=:3404" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. ---559023410-1804928587-962307485=:3404 Content-Type: TEXT/PLAIN; charset=US-ASCII I have included the Perl script I used to generate the change bars. The draft was generated as follows: perl create-change-bar.pl iSCSI000628.txt iscsi00615.txt > iSCSI000628.2.txt Hope it helps! -Costa ---559023410-1804928587-962307485=:3404 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="iSCSI000628.2.txt" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="iSCSI000628.2.txt" IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIEludGVybmV0LURyYWZ0ICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEou IFNhdHJhbg0NCiA8ZHJhZnQtc2F0cmFuLWlTQ1NJLTAzLnR4dD4gICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgRC4gU21pdGgNDQp8RXhw aXJlcyBEZWNlbWJlciAyOCwgMjAwMCAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBLLiBNZXRoDQ0KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIElCTQ0NCiANDQogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEMuIFNhcHVudHpha2lz DQ0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgQ2lzY28gU3lzdGVtcw0NCiANDQp8ICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBSYW5keSBIYWFnZW5zDQ0KfCAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgSGV3bGV0dC1QYWNr YXJkIENvLg0NCiANDQp8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgRWZyaSBaZWlkbmVyDQ0K fCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgU0FOR2F0ZQ0NCiANDQp8ICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IFBhdWwgVm9uIFN0YW13aXR6DQ0KfCAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgQWRh cHRlYw0NCiANDQp8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIEx1Y2lhbm8gRGFsbGUgT3JlDQ0KfCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgUXVhbnR1bQ0NCiANDQp8ICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBKdW5lIDI4LCAyMDAwDQ0KIA0NCiAgICAgICAgICAgICAgICAgICAgICAg ICAgaVNDU0kgKEludGVybmV0IFNDU0kpDQ0KIA0NCiANDQogU3RhdHVzIG9m IHRoaXMgTWVtbw0NCiANDQogICAgICBUaGlzIGRvY3VtZW50IGlzIGFuIElu dGVybmV0LURyYWZ0IGFuZCBpcyBpbiBmdWxsIGNvbmZvcm1hbmNlIHdpdGgN DQp8ICAgICBhbGwgcHJvdmlzaW9ucyBvZiBTZWN0aW9uIDEwIG9mIFJGQzIw MjYuICBJbnRlcm5ldC1EcmFmdHMgYXJlIHdvcmstDQ0KfCAgICAgaW5nIGRv Y3VtZW50cyBvZiB0aGUgSW50ZXJuZXQgRW5naW5lZXJpbmcgVGFzayBGb3Jj ZSAoSUVURiksIGl0cw0NCnwgICAgIGFyZWFzLCBhbmQgaXRzIHdvcmtpbmcg Z3JvdXBzLiBOb3RlIHRoYXQgb3RoZXIgZ3JvdXBzIG1heSBhbHNvIGRpcy0N DQp8ICAgICB0cmlidXRlIHdvcmtpbmcgZG9jdW1lbnRzIGFzIEludGVybmV0 LURyYWZ0cy4gIEludGVybmV0LURyYWZ0cyBhcmUNDQp8ICAgICBkcmFmdCBk b2N1bWVudHMgdmFsaWQgZm9yIGEgbWF4aW11bSBvZiBzaXggbW9udGhzIGFu ZCBtYXkgYmUNDQp8ICAgICB1cGRhdGVkLCByZXBsYWNlZCwgb3Igb2Jzb2xl dGVkIGJ5IG90aGVyIGRvY3VtZW50cyBhdCBhbnkgdGltZS4gSXQNDQp8ICAg ICBpcyBpbmFwcHJvcHJpYXRlIHRvIHVzZSBJbnRlcm5ldC1EcmFmdHMgYXMg cmVmZXJlbmNlIG1hdGVyaWFsIG9yIHRvDQ0KfCAgICAgY2l0ZSB0aGVtIG90 aGVyIHRoYW4gYXMgIndvcmsgaW4gcHJvZ3Jlc3MuIg0NCiANDQogICAgICBU aGUgbGlzdCBvZiBjdXJyZW50IEludGVybmV0LURyYWZ0cyBjYW4gYmUgYWNj ZXNzZWQgYXQNDQp8ICAgICBodHRwOi8vd3d3LmlldGYub3JnL2lldGYvMWlk LWFic3RyYWN0cy50eHQgVGhlIGxpc3Qgb2YgSW50ZXJuZXQtDQ0KfCAgICAg RHJhZnQgU2hhZG93IERpcmVjdG9yaWVzIGNhbiBiZSBhY2Nlc3NlZCBhdA0N CiAgICAgIGh0dHA6Ly93d3cuaWV0Zi5vcmcvc2hhZG93Lmh0bWwuDQ0KIA0N CiAwLjAuMS4gIEFja25vd2xlZGdlbWVudHMNDQogDQ0KICAgICAgQSBsYXJn ZSBncm91cCBvZiBwZW9wbGUgY29udHJpYnV0ZWQgdGhyb3VnaCB0aGVpciBy ZXZpZXcsIGNvbW1lbnRzDQ0KICAgICAgYW5kIHZhbHVhYmxlIGluc2lnaHRz IHRvIHRoZSBjcmVhdGlvbiBvZiB0aGlzIGRvY3VtZW50IC0gdG9vIG1hbnkN DQogICAgICB0byBtZW50aW9uIHRoZW0gYWxsLiBOZXZlcnRoZWxlc3MsIHdl IGFyZSBncmF0ZWZ1bCB0byBhbGwgb2YgdGhlbS4NDQogICAgICBXZSBhcmUg ZXNwZWNpYWxseSBncmF0ZWZ1bCB0byB0aG9zZSB0aGF0IGZvdW5kIHRoZSB0 aW1lIGFuZA0NCiANDQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6 YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1Bh Z2UgMV0NDQogDA0NCiBpU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBK dW5lIDIwMDANDQogDQ0KIA0NCiAgICAgIHBhdGllbmNlIHRvIHBhcnRpY2lw YXRlIGluIG91ciB3ZWVrbHkgcGhvbmUgY29uZmVyZW5jZXMgYW5kIGludGVy LQ0NCiAgICAgIG1lZGlhdGUgbWVldGluZ3MgaW4gQWxtYWRlbiBhbmQgSGFp ZmEgYW5kIHRodXMgaGVscGVkIHNoYXBlIHRoaXMNDQogICAgICBkb2N1bWVu dDogTWF0dCBXYWtlbGV5IChBZ2lsZW50KSwgSmltIEhhZm5lciwgSm9obiBI dWZmZXJkLCBQcmFzZW4tDQ0KICAgICAgaml0IFNhcmthciwgTWVpciBUb2xl ZGFubywgSm9obiBEb3dkeSwgU3RldmUgTGVnZywgQWxhaW4gQXphZ3VyeQ0N CiAgICAgIChJQk0pLCBEYXZlIE5hZ2xlIChDTVUpLCBEYXZpZCBCbGFjayAo RU1DKSwgSm9obiBNYXR6ZSAoVmVyaXRhcyksDQ0KICAgICAgTWFyayBCYWtr ZSwgU3RldmUgRGVHcm9vdGUsIE1hcmsgU2hyYW5kdCAoTnVTcGVlZCksIEdh YmkgSGVjaHQNDQogICAgICAoR2Fkem9veCksIFJvYmVydCBTbml2ZWx5IChC cm9jYWRlKSwgTmVsc29uIE5hY2h1bSAoU3RvckFnZSkNDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRo LCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgW1BhZ2UgMl0NDQogDA0NCiBpU0NTSSAgICAgICAgICAgICAgICAg ICAgICAgICBKdW5lIDIwMDANDQogDQ0KIA0NCiBUYWJsZSBvZiBDb250ZW50 cw0NCiANDQogICAgICAgICAxLiAgQWJzdHJhY3QNDQogICAgICAgICAyLiAg T3ZlcnZpZXcNDQogICAgICAgICAyLjEuICBTQ1NJIENvbmNlcHRzDQ0KfCAg ICAgICAgMi4yLiAgaVNDU0kgQ29uY2VwdHMgJiBGdW5jdGlvbmFsIE92ZXJ2 aWV3DQ0KICAgICAgICAgMi4zLiAgaVNDU0kgTG9naW4NDQogICAgICAgICAy LjQuICBpU0NTSSBGdWxsIEZlYXR1cmUgUGhhc2UNDQogICAgICAgICAyLjUu ICBpU0NTSSBDb25uZWN0aW9uIFRlcm1pbmF0aW9uDQ0KICAgICAgICAgMi42 LiAgTmFtaW5nDQ0KICAgICAgICAgMy4gIE1lc3NhZ2UgRm9ybWF0cw0NCiAg ICAgICAgIDMuMS4gIFRlbXBsYXRlIEhlYWRlcg0NCiAgICAgICAgIDMuMi4g IFNDU0kgQ29tbWFuZA0NCiAgICAgICAgIDMuMy4gIFNDU0kgUmVzcG9uc2UN DQogICAgICAgICAzLjQuICBBc3luY2hyb25vdXMgRXZlbnQNDQogICAgICAg ICAzLjUuICBTQ1NJIFRhc2sgTWFuYWdlbWVudCBNZXNzYWdlDQ0KICAgICAg ICAgMy42LiAgU0NTSSBUYXNrIE1hbmFnZW1lbnQgUmVzcG9uc2UNDQogICAg ICAgICAzLjcuICBSZWFkeSBUbyBUcmFuc2ZlciAoUlRUKQ0NCiAgICAgICAg IDMuOC4gIFNDU0kgRGF0YQ0NCiAgICAgICAgIDMuOS4gIFRleHQgQ29tbWFu ZA0NCiAgICAgICAgIDMuMTAuICBUZXh0IFJlc3BvbnNlDQ0KICAgICAgICAg My4xMS4gIExvZ2luIENvbW1hbmQNDQogICAgICAgICAzLjEyLiAgTG9naW4g UmVzcG9uc2UNDQogICAgICAgICAzLjEzLiAgUGluZyBDb21tYW5kDQ0KICAg ICAgICAgMy4xNC4gIFBpbmcgUmVzcG9uc2UNDQogICAgICAgICAzLjE1LiAg VGhpcmQgUGFydHkgQ29tbWFuZHMNDQogICAgICAgICAzLjE2LiAgT3Bjb2Rl IE5vdCBVbmRlcnN0b29kDQ0KICAgICAgICAgNC4gIEVycm9yIEhhbmRsaW5n IGlTQ1NJDQ0KICAgICAgICAgNS4gIE5vdGVzIHRvIEltcGxlbWVudG9ycw0N CiAgICAgICAgIDUuMS4gIFNtYWxsIFRDUCBTZWdtZW50cw0NCiAgICAgICAg IDUuMi4gIE11bHRpcGxlIE5ldHdvcmsgQWRhcHRlcnMNDQogICAgICAgICA1 LjMuICBBdXRvc2Vuc2UNDQogICAgICAgICA1LjQuICBUQ1AgUkRNQSBvcHRp b24NDQogICAgICAgICA1LjUuICBEYXRhIENvbm5lY3Rpb25zIE9wdGlvbnMN DQogICAgICAgICA2LiAgU2VjdXJpdHkgQ29uc2lkZXJhdGlvbnMNDQogICAg ICAgICA2LjEuICBEYXRhIEludGVncml0eQ0NCiAgICAgICAgIDYuMi4gIExv Z2luIFByb2Nlc3MNDQogICAgICAgICA2LjMuICBJQU5BIENvbnNpZGVyYXRp b25zDQ0KICAgICAgICAgNy4gIEF1dGhvcnMnIEFkZHJlc3Nlcw0NCiAgICAg ICAgIDguICBSZWZlcmVuY2VzIGFuZCBCaWJsaW9ncmFwaHkNDQogICAgICAg ICA5LiAgQXBwZW5kaXggQSAtIEV4YW1wbGVzDQ0KICAgICAgICAgOS4xLiAg UmVhZCBvcGVyYXRpb24gZXhhbXBsZQ0NCiAgICAgICAgIDkuMi4gIFdyaXRl IG9wZXJhdGlvbiBleGFtcGxlDQ0KICAgICAgICAgMTAuICBBcHBlbmRpeCBC IC0gTG9naW4vVGV4dCBrZXlzDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgM10NDQogDA0NCiBpU0NT SSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDANDQogDQ0KIA0N CiAxLiAgQWJzdHJhY3QNDQogDQ0KICAgICAgVGhlIFNtYWxsIENvbXB1dGVy IFN5c3RlbXMgSW50ZXJmYWNlIChTQ1NJKSBpcyBhIHBvcHVsYXIgZmFtaWx5 IG9mDQ0KICAgICAgcHJvdG9jb2xzIGZvciBjb21tdW5pY2F0aW5nIHdpdGgg SS9PIGRldmljZXMsIGVzcGVjaWFsbHkgc3RvcmFnZQ0NCnwgICAgIGRldmlj ZXMuICBUaGlzIG1lbW8gZGVzY3JpYmVzIGEgdHJhbnNwb3J0IHByb3RvY29s IGZvciBTQ1NJIHRoYXQNDQp8ICAgICBvcGVyYXRlcyBvbiB0b3Agb2YgVENQ LiAgVGhlIGlTQ1NJIHByb3RvY29sIGFpbXMgdG8gYmUgZnVsbHkgY29tLQ0N CnwgICAgIHBsaWFudCB3aXRoIHRoZSByZXF1aXJlbWVudHMgbGFpZCBvdXQg aW4gdGhlIFNDU0kgQXJjaGl0ZWN0dXJlDQ0KfCAgICAgTW9kZWwgLSAyIFtT QU0yXSBkb2N1bWVudC4NDQogDQ0KIDIuICBPdmVydmlldw0NCiANDQogMi4x LiAgU0NTSSBDb25jZXB0cw0NCiANDQogICAgICBUaGUgZW5kcG9pbnQgb2Yg bW9zdCBTQ1NJIGNvbW1hbmRzIGlzIGEgImxvZ2ljYWwgdW5pdCIgKExVTiku IEV4YW0tDQ0KICAgICAgcGxlcyBvZiBsb2dpY2FsIHVuaXRzIGluY2x1ZGUg aGFyZCBkcml2ZXMsIHRhcGUgZHJpdmVzLCBDRCBhbmQgRFZEDQ0KICAgICAg ZHJpdmVzLCBwcmludGVycyBhbmQgcHJvY2Vzc29ycy4gV2l0aGluIHRoZSBs b2dpY2FsIHVuaXQgdGhlDQ0KICAgICAgYWJzdHJhY3QgZW50aXR5IHRoYXQg ZXhlY3V0ZXMgdGhlIFNDU0kgY29tbWFuZHMgaXMgbmFtZWQgdGhlDQ0KfCAg ICAgZGV2aWNlLXNlcnZlci4gIEEgInRhcmdldCIgaXMgYSBjb2xsZWN0aW9u IG9mIGxvZ2ljYWwgdW5pdHMsIGluDQ0KfCAgICAgZ2VuZXJhbCBvZiB0aGUg c2FtZSBraW5kLCBhbmQgaXMgZGlyZWN0bHkgYWRkcmVzc2FibGUgb24gdGhl IG5ldC0NDQp8ICAgICB3b3JrLiBJbiBsYXJnZSBpbnN0YWxsYXRpb25zIGEg dGFyZ2V0IGlzIGtub3duIGFsc28gYXMgYSAiY29udHJvbA0NCnwgICAgIHVu aXQiLiAgVGhlIHRhcmdldCBjb3JyZXNwb25kcyB0byB0aGUgc2VydmVyIGlu IHRoZSBhYnN0cmFjdCBTQU0NDQp8ICAgICBjbGllbnQtc2VydmVyIG1vZGVs LiAgQW4gImluaXRpYXRvciIgY3JlYXRlcyBhbmQgc2VuZHMgU0NTSSBjb20t DQ0KfCAgICAgbWFuZHMgdG8gdGhlIHRhcmdldC4gVGhlIGluaXRpYXRvciBj b3JyZXNwb25kcyB0byB0aGUgY2xpZW50IGluIHRoZQ0NCnwgICAgIGFic3Ry YWN0IFNBTSBjbGllbnQtc2VydmVyIG1vZGVsLiAgQSAidGFzayIgaXMgYSBs aW5rZWQgc2V0IG9mIFNDU0kNDQp8ICAgICBjb21tYW5kcy4gU29tZSBMVU5z IHN1cHBvcnQgbXVsdGlwbGUgcGVuZGluZyAocXVldWVkKSB0YXNrcy4gVGhl DQ0KfCAgICAgdGFyZ2V0IHVzZXMgYSAidGFzayB0YWciIHRvIGRpc3Rpbmd1 aXNoIGJldHdlZW4gdGFza3MuIE9ubHkgb25lDQ0KfCAgICAgY29tbWFuZCBp biBhIHRhc2sgY2FuIGJlIG91dHN0YW5kaW5nIGF0IGFueSBnaXZlbiB0aW1l LiAgQSBTQ1NJDQ0KfCAgICAgY29tbWFuZCByZXN1bHRzIGluIGFuIG9wdGlv bmFsIGRhdGEgcGhhc2UgYW5kIGEgcmVzcG9uc2UgcGhhc2UuIEluDQ0KfCAg ICAgdGhlIGRhdGEgcGhhc2UsIGluZm9ybWF0aW9uIHRyYXZlbHMgZWl0aGVy IGZyb20gdGhlIGluaXRpYXRvciB0bw0NCnwgICAgIHRoZSB0YXJnZXQsIGFz IGluIGEgV1JJVEUgY29tbWFuZCwgb3IgZnJvbSB0YXJnZXQgdG8gaW5pdGlh dG9yLCBhcw0NCnwgICAgIGluIGEgUkVBRCBjb21tYW5kLiBJbiB0aGUgcmVz cG9uc2UgcGhhc2UsIHRoZSB0YXJnZXQgcmV0dXJucyB0aGUNDQp8ICAgICBm aW5hbCBzdGF0dXMgb2YgdGhlIG9wZXJhdGlvbiwgaW5jbHVkaW5nIGFueSBl cnJvcnMuIEEgcmVzcG9uc2UNDQp8ICAgICB0ZXJtaW5hdGVzIGEgU0NTSSBj b21tYW5kLg0NCiANDQp8Mi4yLiAgaVNDU0kgQ29uY2VwdHMgJiAgRnVuY3Rp b25hbCBPdmVydmlldw0NCiANDQp8ICAgICBUaGUgZm9sbG93aW5nIGNvbmNl cHR1YWwgbGF5ZXJpbmcgbW9kZWwgaXMgdXNlZCBpbiB0aGlzIGRvY3VtZW50 IHRvDQ0KfCAgICAgc3BlY2lmeSBJbml0aWF0b3IgYW5kIHRhcmdldCBhY3Rp b25zIGFuZCBob3cgdGhvc2UgcmVsYXRlIHRvDQ0KfCAgICAgdHJhbnNtaXR0 ZWQgYW5kIHJlY2VpdmVkIFByb3RvY29sIERhdGEgVW5pdHM6ICAtIFNDU0kg bGF5ZXINDQp8ICAgICBidWlsZHMvcmVjZWl2ZXMgU0NTSSBDREIgKENvbW1h bmQgRGF0YSBCbG9ja3MpIGFuZCByZWxheXMvcmVjZWl2ZXMNDQp8ICAgICB0 aGVtIHdpdGggdGhlIHJlbWFpbmluZyBjb21tYW5kIGV4ZWN1dGUgcGFyYW1l dGVycyAoY2YuIFNBTS0yKQ0NCnwgICAgIHRvL2Zyb20gdGhlIC0gaVNDU0kg bGF5ZXIgdGhhdCBpcyBidWlsZGluZy9yZWNlaXZpbmcgaVNDU0kgUERVcyBh bmQNDQp8ICAgICByZWxheWluZy9yZWNlaXZpbmcgdGhlbSB0by9mcm9tIC0g b25lIG9yIG1vcmUgVENQIGNvbm5lY3Rpb25zIHRoYXQNDQp8ICAgICBmb3Jt IGFuIGluaXRpYXRvci10YXJnZXQgInNlc3Npb24iDQ0KIA0NCiAgICAgIENv bW11bmljYXRpb24gYmV0d2VlbiBpbml0aWF0b3IgYW5kIHRhcmdldCBvY2N1 cnMgb3ZlciBvbmUgb3IgbW9yZQ0NCiAgICAgIFRDUCBjb25uZWN0aW9ucy4g IFRoZSBUQ1AgY29ubmVjdGlvbnMgYXJlIHVzZWQgZm9yIHNlbmRpbmcgY29u dHJvbA0NCnwgICAgIG1lc3NhZ2VzLCBTQ1NJIGNvbW1hbmRzLCBwYXJhbWV0 ZXJzIGFuZCBkYXRhIHdpdGhpbiBpU0NTSSBwcm90b2NvbA0NCiANDQogDQ0K IA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgNF0NDQogDA0NCiBpU0NT SSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDANDQogDQ0KIA0N CnwgICAgIGRhdGEgdW5pdHMgKGlTQ1NJIFBEVSkgVGhlIGdyb3VwIG9mIFRD UCBjb25uZWN0aW9ucyBsaW5raW5nIGFuIGluaS0NDQp8ICAgICB0aWF0b3Ig d2l0aCBhIHRhcmdldCBmb3JtIGEgc2Vzc2lvbiAobG9vc2VseSBlcXVpdmFs ZW50IHRvIGEgU0NTSQ0NCnwgICAgIG5leHVzKTsgYSBzZXNzaW9uIGlzIGRl ZmluZWQgYnkgYSBzZXNzaW9uIElEIChjb21wb3NlZCBvZiBhIGluaXRpYS0N DQp8ICAgICB0b3IgcGFydCBhbmQgYSB0YXJnZXQgcGFydCkuIFRDUCBjb25u ZWN0aW9ucyBjYW4gYmUgYWRkZWQgYW5kDQ0KfCAgICAgcmVtb3ZlZCBmcm9t IGEgc2Vzc2lvbi4gIGlTQ1NJIHN1cHBvcnRzIG9yZGVyZWQgY29tbWFuZCBk ZWxpdmVyeQ0NCnwgICAgIHdpdGhpbiBhIHNlc3Npb24gYW5kIGxpbWl0ZWQg Y29tbWFuZCBhbmQgZGF0YSByZWNvdmVyeS4gIEFsbCBTQ1NJDQ0KfCAgICAg Y29tbWFuZHMgcHJlc2VudGVkIHRvIGlTQ1NJIGdldCBhICJjb21tYW5kIHJl ZmVyZW5jZSBudW1iZXIiIGFuZA0NCnwgICAgIHRoaXMgbnVtYmVyIGNhbiBi ZSB1c2VkIGJ5IGEgcmVjZWl2aW5nIHRhcmdldCBmb3Igb3JkZXJlZCBkZWxp dmVyeS4NDQp8ICAgICBBIHNsaWRpbmcgd2luZG93IG1lY2hhbmlzbSBpcyB1 c2VkIHRvIGxpbWl0IHRoZSBudW1iZXIgb2Ygb3V0c3RhbmQtDQ0KfCAgICAg aW5nIGNvbW1hbmRzLiAgRm9yIGRlc2NyaXB0aXZlIHB1cnBvc2VzIGl0IGlz IGFzc3VtZWQgdGhhdCB0aGUNDQp8ICAgICBpU0NTSSBsYXllciBpcyBpbXBs ZW1lbnRpbmcgdGhlIHNsaWRpbmcgd2luZG93IG1lY2hhbmlzbS4NDQogDQ0K IDIuMy4gIGlTQ1NJIExvZ2luDQ0KIA0NCiAgICAgIFRoZSBwdXJwb3NlIG9m IGlTQ1NJIGxvZ2luIGlzIHRvIGVuYWJsZSBhIFRDUCBjb25uZWN0aW9uIGZv ciBpU0NTSQ0NCiAgICAgIHVzZSwgYXV0aGVudGljYXRlIHRoZSBwYXJ0aWVz LCBhdXRob3JpemUgdGhlIGluaXRpYXRvciB0byBzZW5kIFNDU0kNDQogICAg ICBjb21tYW5kcyBhbmQgbWFyayB0aGUgY29ubmVjdGlvbiBhcyBiZWxvbmdp bmcgdG8gYSBpU0NTSSBzZXNzaW9uLg0NCiAgICAgIEEgc2Vzc2lvbiBpcyB1 c2VkIHRvIGlkZW50aWZ5IHRvIGEgdGFyZ2V0IGFsbCB0aGUgY29ubmVjdGlv bnMgd2l0aA0NCnwgICAgIGEgZ2l2ZW4gaW5pdGlhdG9yLiAgVGhlIHRhcmdl dHMgbGlzdGVuIG9uIGEgd2VsbC1rbm93biBUQ1AgcG9ydCBmb3INDQp8ICAg ICBpbmNvbWluZyBjb25uZWN0aW9ucy4gIFRoZSBpbml0aWF0b3IgYmVnaW5z IHRoZSBsb2dpbiBwcm9jZXNzIGJ5DQ0KfCAgICAgY29ubmVjdGluZyB0byB0 aGF0IHdlbGwta25vd24gVENQIHBvcnQuICBBcyBwYXJ0IG9mIHRoZSBsb2dp biBwcm8tDQ0KfCAgICAgY2VzcywgdGhlIGluaXRpYXRvciBhbmQgdGFyZ2V0 IE1BWSB3aXNoIHRvIGF1dGhlbnRpY2F0ZSBlYWNoIG90aGVyLg0NCnwgICAg IFRoaXMgY2FuIG9jY3VyIGluIG1hbnkgZGlmZmVyZW50IHdheXMuIEZvciBl eGFtcGxlLCB0aGUgZW5kcG9pbnRzDQ0KfCAgICAgbWF5IHdpc2ggdG8gY2hl Y2sgdGhlIElQIGFkZHJlc3Mgb2YgdGhlIG90aGVyIHBhcnR5LiBJZiB0aGUg VENQDQ0KfCAgICAgY29ubmVjdGlvbiB1c2VzIHRyYW5zcG9ydCBsYXllciBz ZWN1cml0eSBbVExTXSwgY2VydGlmaWNhdGVzIG1heSBiZQ0NCnwgICAgIHVz ZWQgdG8gaWRlbnRpZnkgdGhlIGVuZHBvaW50cy4gQWxzbywgaVNDU0kgaW5j bHVkZXMgY29tbWFuZHMgZm9yDQ0KfCAgICAgaWRlbnRpZnlpbmcgdGhlIGlu aXRpYXRvciBhbmQgcGFzc2luZyBhbiBhdXRoZW50aWNhdG9yIHRvIHRoZSB0 YXItDQ0KfCAgICAgZ2V0IChzZWUgQXBwZW5kaXggQikuICBPbmNlIHN1aXRh YmxlIGF1dGhlbnRpY2F0aW9uIGhhcyBvY2N1cnJlZCwNDQp8ICAgICB0aGUg dGFyZ2V0IE1BWSBhdXRob3JpemUgdGhlIGluaXRpYXRvciB0byBzZW5kIFND U0kgY29tbWFuZHMuIEhvdw0NCnwgICAgIHRoZSB0YXJnZXQgY2hvb3NlcyB0 byBhdXRob3JpemUgYW4gaW5pdGlhdG9yIGlzIGJleW9uZCB0aGUgc2NvcGUg b2YNDQp8ICAgICB0aGlzIGRvY3VtZW50LiAgVGhlIHRhcmdldCBpbmRpY2F0 ZXMgYSBzdWNjZXNzZnVsIGF1dGhlbnRpY2F0aW9uDQ0KfCAgICAgYW5kIGF1 dGhvcml6YXRpb24gYnkgc2VuZGluZyBhIGxvZ2luIHJlc3BvbnNlIHdpdGgg ImFjY2VwdCBsb2dpbiIuDQ0KICAgICAgVGhlIGxvZ2luIG1lc3NhZ2UgaW5j bHVkZXMgYSBzZXNzaW9uIElEIC0gY29tcG9zZWQgd2l0aCBhbiBpbml0aWEt DQ0KICAgICAgdG9yIHBhcnQgSVNJRCBhbmQgYSB0YXJnZXQgcGFydCBUU0lE LiBGb3IgYSBuZXcgc2Vzc2lvbiB0aGUgVFNJRCBpcw0NCiAgICAgIG51bGwu ICBBcyBwYXJ0IG9mIHRoZSByZXNwb25zZSB0aGUgdGFyZ2V0IHdpbGwgZ2Vu ZXJhdGUgYSBUU0lELg0NCiAgICAgIFNlc3Npb24gc3BlY2lmaWMgcGFyYW1l dGVycyBjYW4gYmUgc3BlY2lmaWVkIG9ubHkgZm9yIHRoZSBmaXJzdA0NCiAg ICAgIGxvZ2luIG9mIGEgc2Vzc2lvbiAoVFNJRCBudWxsKShlLmcgdGhlIG1h eGltdW0gbnVtYmVyIG9mIGNvbm5lYy0NDQogICAgICB0aW9ucyB0aGF0IGNh biBiZSB1c2VkIGZvciB0aGlzIHNlc3Npb24pLiBDb25uZWN0aW9uIHNwZWNp ZmljDQ0KICAgICAgcGFyYW1ldGVycyAoaWYgYW55KSBjYW4gYmUgc3BlY2lm aWVkIGZvciBhbnkgbG9naW4uIFRodXMgYSBzZXNzaW9uDQ0KICAgICAgaXMg b3BlcmF0aW9uYWwgb25jZSBpdCBoYXMgYXQgbGVhc3Qgb25lIGNvbm5lY3Rp b24gYW5kIGEgcGVuZGluZw0NCnwgICAgIGxvZ2luIGNhbid0IGFmZmVjdCBh IHdob2xlIHNlc3Npb24uICBBZnRlciBhdXRoZW50aWNhdGlvbiBhbmQNDQp8 ICAgICBhdXRob3JpemF0aW9uLCBvdGhlciBwYXJhbWV0ZXJzIG1heSBiZSBu ZWdvdGlhdGVkIHVzaW5nIHRoZSBoaWdobHkNDQp8ICAgICBleHRlbnNpYmxl IFRleHQgQ29tbWFuZCBtZXNzYWdlIHRoYXQgYWxsb3dzIGFyYml0cmFyeSBr ZXk6dmFsdWUNDQp8ICAgICBwYWlycyB0byBiZSBwYXNzZWQuICBBbnkgbWVz c2FnZSBzZW50IG9uIGEgVENQIGNvbm5lY3Rpb24gYmVmb3JlDQ0KfCAgICAg dGhpcyBjb25uZWN0aW9uIGdldHMgaW50byBmdWxsIGZlYXR1cmUgcGhhc2Ug YXQgdGhlIGluaXRpYXRvcg0NCnwgICAgIHNob3VsZCBiZSByZWplY3RlZCBi eSB0aGUgaW5pdGlhdG9yLiAgQSBtZXNzYWdlIHJlYWNoaW5nIGEgdGFyZ2V0 DQ0KfCAgICAgb24gYSBUQ1AgY29ubmVjdGlvbiBiZWZvcmUgdGhlIGZ1bGwg ZmVhdHVyZSBwaGFzZSB3aWxsIGJlIHJlamVjdA0NCnwgICAgIHdpdGggYW4g aVNDU0kgY2hlY2sgY29uZGl0aW9uIGJpdC4NDQogDQ0KIA0NCiANDQogU2F0 cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIFtQYWdlIDVdDQ0KIAwNDQogaVNDU0kgICAgICAg ICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQogMi40LiAg aVNDU0kgRnVsbCBGZWF0dXJlIFBoYXNlDQ0KIA0NCiAgICAgIE9uY2UgdGhl IGluaXRpYXRvciBpcyBhdXRob3JpemVkIHRvIGRvIHNvLCB0aGUgaVNDU0kg c2Vzc2lvbiBpcyBpbg0NCiAgICAgIGlTQ1NJIGZ1bGwgZmVhdHVyZSBwaGFz ZS4gVGhlIGluaXRpYXRvciBtYXkgc2VuZCBTQ1NJIGNvbW1hbmRzIGFuZA0N CiAgICAgIGRhdGEgdG8gdGhlIHZhcmlvdXMgTFVOcyBvbiB0aGUgdGFyZ2V0 IGJ5IG1hcHBpbmcgdGhlbSBpbiBpU0NTSQ0NCnwgICAgIG1lc3NhZ2VzIHRo YXQgZ28gb3ZlciB0aGUgZXN0YWJsaXNoZWQgaVNDU0kgc2Vzc2lvbi4gIEZv ciBTQ1NJIGNvbS0NDQp8ICAgICBtYW5kcyB0aGF0IHJlcXVpcmUgZGF0YSBh bmQvb3IgcGFyYW1ldGVyIHRyYW5zZmVyLCB0aGUNDQogICAgICAob3B0aW9u YWwpZGF0YSBhbmQgdGhlIHN0YXR1cyBmb3IgYSBjb21tYW5kIG11c3QgYmUg c2VudCBvdmVyIHRoZQ0NCiAgICAgIHNhbWUgVENQIGNvbm5lY3Rpb24gdGhh dCB3YXMgdXNlZCB0byBkZWxpdmVyIHRoZSBTQ1NJIGNvbW1hbmQgKGNvbi0N DQogICAgICBuZWN0aW9uIGFsbGVnaWFuY2UpLiAgVGh1cyBpZiBhbiBpbml0 aWF0b3IgaXNzdWVzIGEgUkVBRCBjb21tYW5kLA0NCiAgICAgIHRoZSB0YXJn ZXQgbXVzdCBzZW5kIHRoZSByZXF1ZXN0ZWQgZGF0YSBmb2xsb3dlZCBieSB0 aGUgc3RhdHVzIHRvDQ0KICAgICAgdGhlIGluaXRpYXRvciBvdmVyIHRoZSBz YW1lIFRDUCBjb25uZWN0aW9uIHRoYXQgd2FzIHVzZWQgdG8gZGVsaXZlcg0N CiAgICAgIHRoZSBTQ1NJIGNvbW1hbmQuICBJZiBhbiBpbml0aWF0b3IgaXNz dWVzIGEgV1JJVEUgY29tbWFuZCwgdGhlIGluaS0NDQogICAgICB0aWF0b3Ig bXVzdCBzZW5kIHRoZSBkYXRhIGZvciB0aGF0IGNvbW1hbmQgYW5kIHRoZSB0 YXJnZXQgbXVzdA0NCiAgICAgIHJldHVybiB0aGUgc3RhdHVzIG92ZXIgdGhl IHNhbWUgVENQIGNvbm5lY3Rpb24gdGhhdCB3YXMgdXNlZCB0bw0NCnwgICAg IGRlbGl2ZXIgdGhlIFNDU0kgY29tbWFuZC4gIER1cmluZyBpU0NTSSBGdWxs IEZlYXR1cmUgUGhhc2UsIHRoZQ0NCnwgICAgIGluaXRpYXRvciBhbmQgdGFy Z2V0IG1heSBpbnRlcmxlYXZlIHVucmVsYXRlZCBTQ1NJIGNvbW1hbmRzLCB0 aGVpcg0NCnwgICAgIFNDU0kgRGF0YSBhbmQgcmVzcG9uc2VzLCBvdmVyIHRo ZSBzZXNzaW9uLiAgT3V0Z29pbmcgU0NTSSBkYXRhDQ0KfCAgICAgKGluaXRp YXRvciB0byB0YXJnZXQgLSB1c2VyIGRhdGEgb3IgY29tbWFuZCBwYXJhbWV0 ZXJzKWlzIHNlbnQgYXMNDQp8ICAgICBlaXRoZXIgdW5zb2xpY2l0ZWQgZGF0 YSBvciBzb2xpY2l0ZWQgZGF0YS4gIFVuc29saWNpdGVkIGRhdGEgY2FuIGJl DQ0KfCAgICAgcGFydCBvZiBhbiBpU0NTSSBjb21tYW5kIFBEVSAoImltbWVk aWF0ZSBkYXRhIikgb3IgYW4gaVNDU0kgZGF0YQ0NCnwgICAgIFBEVS4gIFNv bGljaXRlZCBkYXRhIGFyZSBzZW50IGluIHJlc3BvbnNlIHRvIFJlYWR5IFRv IFRyYW5zZmVyDQ0KfCAgICAgUERVcy4gIFRhcmdldHMgYXJlIG9wZXJhdGlu ZyBpbiBlaXRoZXIgc29saWNpdGVkIChSVFQpIGRhdGEgbW9kZSBvcg0NCnwg ICAgIHVuc29saWNpdGVkIChub24gUlRUKSBkYXRhIG1vZGUuICBBbiBpbml0 aWF0b3IgbXVzdCBhbHdheXMgaG9ub3IgYW4NDQp8ICAgICBSVFQgZGF0YSBy ZXF1ZXN0LiAgSXQgaXMgY29uc2lkZXJlZCBhbiBlcnJvciBmb3IgYW4gaW5p dGlhdG9yIHRvDQ0KfCAgICAgc2VuZCB1bnNvbGljaXRlZCBkYXRhIFBEVXMg dG8gYSB0YXJnZXQgb3BlcmF0aW5nIGluIFJUVCBtb2RlIChvbmx5DQ0KfCAg ICAgc29saWNpdGVkIGRhdGEpLiAgQnkgZGVmYXVsdCwgaW1tZWRpYXRlIGRh dGEgaXMgbGltaXRlZCB0byA2NEtieXRlcw0NCnwgICAgIGFuZCBhbiBpbml0 aWF0b3IgaXMgYWxsb3dlZCB0byBzZW5kIGltbWVkaWF0ZSBkYXRhIChzdWJq ZWN0IHRvIGxpbS0NDQp8ICAgICBpdGF0aW9ucyBzcGVjaWZpZWQgc29tZXdo ZXJlIGVsc2UgaW4gdGhpcyBkb2N1bWVudClldmVuIHRvIHRhcmdldHMNDQp8 ICAgICB3b3JraW5nIGluIFJUVCBtb2RlLiBBbiBpbml0aWF0b3IgbWF5IHJl cXVlc3QsIGF0IGxvZ2luLCB0byBzZW5kDQ0KfCAgICAgaW1tZWRpYXRlIGRh dGEgb2YgYW55IHNpemUgYW5kIGEgdGFyZ2V0IG1heSBpbmRpY2F0ZSB0aGUg c2l6ZSBvZg0NCnwgICAgIGltbWVkaWF0ZSBkYXRhIGJsb2NrcyBpdCBpcyBy ZWFkeSB0byBhY2NlcHQgaW4gaXRzIHJlc3BvbnNlLiAgQQ0NCnwgICAgIHRh cmdldCBpcyBhbGxvd2VkIHRvIHNpbGVudGx5IGRpc2NhcmQgZGF0YSBhbmQg cmVxdWVzdCByZXRyYW5zbWlzLQ0NCnwgICAgIHNpb24gdGhyb3VnaCBSVFQu ICBJbml0aWF0b3JzIHdpbGwgbm90IHBlcmZvcm0gYW55IHNjb3JlYm9hcmRp bmcNDQp8ICAgICBmb3IgZGF0YSBhbmQgdGhlIHJlc2lkdWFsIGNvdW50IGNh bGN1bGF0aW9uIGlzIHRvIGJlIHBlcmZvcm1lZCBieQ0NCnwgICAgIHRoZSB0 YXJnZXRzLiAgSW5jb21pbmcgZGF0YSBpcyBhbGx3YXlzIHNvbGljaXRlZC4g SG93ZXZlciBhbiBpbmktDQ0KfCAgICAgdGlhdG9yIHdpbGwgYmUgYWJsZSB0 byByZXF1ZXN0IHJldHJhbnNtaXNzaW9uIG9mIGFsbCBvciBwYXJ0IG9mIHRo ZQ0NCnwgICAgIHRhcmdldCBkYXRhLiAgU0NTSSBEYXRhIHBhY2tldHMgYXJl IG1hdGNoZWQgdG8gdGhlaXIgY29ycmVzcG9uZGluZw0NCnwgICAgIFNDU0kg Y29tbWFuZHMgYnkgdXNpbmcgVGFncyB0aGF0IGFyZSBzcGVjaWZpZWQgaW4g dGhlIHByb3RvY29sLg0NCnwgICAgIEluaXRpYXRvciB0YWdzIGZvciBwZW5k aW5nIGNvbW1hbmRzIGFyZSB1bmlxdWUgaW5pdGlhdG9yLXdpZGUgZm9yIGEN DQp8ICAgICBzZXNzaW9uLiAgVGFyZ2V0IHRhZ3MgZm9yIHBlbmRpbmcgY29t bWFuZHMgYXJlIHVuaXF1ZSB0YXJnZXQtd2lkZQ0NCnwgICAgIGZvciB0aGUg c2Vzc2lvbi4gIEFsdGhvdWdoIHRoZSBhYm92ZSBtZWNoYW5pc21zIGFyZSBk ZXNpZ25lZCB0bw0NCnwgICAgIGFjY29tcGxpc2ggZWZmaWNpZW50IGRhdGEg ZGVsaXZlcnkgYW5kIGEgbGFyZ2UgZGVncmVlIG9mIGNvbnRyb2wNDQp8ICAg ICBvdmVyIHRoZSBkYXRhIGZsb3cgaXQgaXMgcmVjb2duaXplZCB0aGF0IHNv bWUgc3BlY2lmaWMgc2VxdWVuY2VzDQ0KfCAgICAgaW52b2x2aW5nIG9yZGVy ZWQgZXhlY3V0aW9uIGFuZCBhIG1peCBvZiBzb2xpY2l0ZWQgYW5kIGltbWVk aWF0ZQ0NCnwgICAgIGRhdGEgY2FuIHJlc3VsdCBpbiBkZWFkbG9ja3MuIEl0 IGlzIGZvciB0aGlzIHJlYXNvbiB0aGF0IGRpc2NhcmRpbmcNDQp8ICAgICBk YXRhIGJ5IGEgdGFyZ2V0IGlzIGNvbnNpZGVyZWQgYSBsZWdpdGltYXRlIGFj dGlvbi4gIEV4YW1wbGVzIG9mDQ0KfCAgICAgc3VjaCBzZXF1ZW5jZXMgYXJl IHByZXNlbnRlZCBpbiBhcHBlbmRpeCBDIHRvZ2V0aGVyIHJlY292ZXJ5DQ0K IA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRo ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSA2XQ0NCiAM DQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0N CiANDQogDQ0KfCAgICAgc2NlbmFyaW9zLiAgT3V0Z29pbmcgY29tbWFuZHMg YXJlIG51bWJlcmVkIGJ5IGlTQ1NJIChDbWRSTikgYW5kDQ0KfCAgICAgcmVz cG9uc2UgUERVcyAodGFyZ2V0IHRvIGluaXRpYXRvcikgd2lsbCBjb250aW51 b3VzbHkgdXBkYXRlIHRoZQ0NCnwgICAgIGluaXRpYXRvciBhYm91dCB0aGUg bWF4aW11bSBjb21tYW5kIG51bWJlciB0aGF0IGNhbiBiZSBzZW50KHNsaWRp bmcNDQp8ICAgICB3aW5kb3cpLiAgRWFjaCBpU0NTSSBzZXNzaW9uIHRvIGEg dGFyZ2V0IGlzIHRyZWF0ZWQgYXMgaWYgaXQgb3JpLQ0NCnwgICAgIGdpbmF0 ZWQgZnJvbSBhIGRpZmZlcmVudCBpbml0aWF0b3IuDQ0KIA0NCiAyLjUuICBp U0NTSSBDb25uZWN0aW9uIFRlcm1pbmF0aW9uDQ0KIA0NCiAgICAgIENvbm5l Y3Rpb24gdGVybWluYXRpb24gaXMgYXNzdW1lZCB0byBiZSBhbiBleGNlcHRp b25hbCBldmVudC4NDQogICAgICBHcmFjZWZ1bCBUQ1AgY29ubmVjdGlvbiBz aHV0ZG93bnMgYXJlIGRvbmUgYnkgc2VuZGluZyBUQ1AgRklOcy4NDQogICAg ICBHcmFjZWZ1bCBjb25uZWN0aW9uIHNodXRkb3ducyBNVVNUIG9ubHkgb2Nj dXIgd2hlbiB0aGVyZSBhcmUgbm8NDQp8ICAgICBvdXRzdGFuZGluZyB0YXNr cyB0aGF0IGhhdmUgYWxsZWdpYW5jZSB0byB0aGUgY29ubmVjdGlvbi4gIEEg dGFyZ2V0DQ0KfCAgICAgU0hPVUxEIHJlc3BvbmQgcmFwaWRseSB0byBhIEZJ TiBmcm9tIHRoZSBpbml0aWF0b3IgYnkgY2xvc2luZyBpdHMNDQp8ICAgICBo YWxmIG9mIHRoZSBjb25uZWN0aW9uIGFzIHNvb24gYXMgaXQgaGFzIGZpbmlz aGVkIGFsbCBvdXRzdGFuZGluZw0NCnwgICAgIHRhc2tzIHRoYXQgaGF2ZSBh bGxlZ2lhbmNlIHRvIHRoZSBjb25uZWN0aW9uLiAgQ2xvc2luZyBhIGNvbm5l Y3Rpb24NDQp8ICAgICB0aGF0IGhhcyBvdXRzdGFuZGluZyB0YXNrcyBtYXkg cmVxdWlyZSByZWNvdmVyeSBhY3Rpb25zIGFuZCB3aWxsIEJlDQ0KfCAgICAg ZGVzY3JpYmVkIGVsc2V3aGVyZSBpbiB0aGlzIGRvY3VtZW50Lg0NCiANDQog Mi42LiAgTmFtaW5nDQ0KIA0NCiAgICAgIFRhcmdldHMgYXJlIG5hbWVkIHVz aW5nIGFuIFVSTCB0eXBlIG5hbWUgb2YgdGhlIGZvcm1hdDoNDQogDQ0KICAg ICAgICAgICAgICBzY3NpOi8vPGRvbWFpbi1uYW1lPlsvbW9kaWZpZXJdDQ0K IA0NCiANDQogICAgICBUaGUgbmFtZSB1c2VkIHRvIGNvbm5lY3Qgd2lsbCBi ZSBvcHRpb25hbGx5IGluY2x1ZGVkIGluIHRoZSBsb2dpbg0NCiAgICAgIGlu IG9yZGVyIHRvIGVuYWJsZSB0aGUgdGFyZ2V0IHRvIHByZXNlbnQgZGlmZmVy ZW50IHZpZXdzLiBUaGlzIGlzDQ0KICAgICAgdGhlIFRhcmdldCBBY3F1aXJl ZCBOYW1lIChUQU4pLiAgV2Ugd2lsbCBub3QgYXR0ZW1wdCB0byBkZWZpbmUN DQogICAgICB3aGljaCBjb21wb25lbnRzIG9mIHRoZSBuYW1lIHdpbGwgcGFy dGljaXBhdGUgaW4gdGhlIG5hbWUgcmVzb2x1LQ0NCiAgICAgIHRpb24gcHJv Y2VzcyBhbmQgd2hpY2ggb25lcyB3aWxsIGJlIHVzZWQgb25seSBmb3IgInZp ZXciIGRlZmluaS0NDQogICAgICB0aW9uLiBUaGUgc3ludGFjdGljIHN1Z2Fy IGluY2x1ZGVkIG1pZ2h0IGJlIHVzZWQgdG8gaW50cm9kdWNlDQ0KICAgICAg c3RydWN0dXJlIGZvciBtYW5hZ2VtZW50IHB1cnBvc2VzIGJ1dCBoYXMgbm8g c3BlY2lmaWMgc2lnbmlmaWNhbmNlDQ0KfCAgICAgZm9yIHRoaXMgc3RhbmRh cmQuICBFeGFtcGxlOg0NCiANDQogICAgICAgICAgICAgIHNjc2k6Ly9kaXNr ZmFybTEuYWNtZS5jb20NDQogICAgICAgICAgICAgIHNjc2k6Ly9jb21wdXRp bmdjZW50ZXIuYWNtZS5jb20vcGVyaXBoZXJhbHMvZGlza2Zhcm0xDQ0KIA0N CiANDQogDQ0KICAgICAgV2hlbiBhIHRhcmdldCBoYXMgdG8gYWN0IGFzIGFu IGluaXRpYXRvciBmb3IgYSB0aGlyZCBwYXJ0eSBjb21tYW5kDQ0KfCAgICAg aXQgd2lsbCB1c2UgdGhlIFRBTiBkdXJpbmcgbG9naW4gYXMgcmVxdWlyZWQg YnkgdGhlIGF1dGhlbnRpY2F0aW9uDQ0KfCAgICAgbWVjaGFuaXNtLiAgQSBk b21haW4gbmFtZSB0aGF0IGNvbnRhaW5zIGV4YWN0bHkgZm91ciBudW1iZXJz DQ0KfCAgICAgc2VwYXJhdGVkIGJ5IGRvdHMgKC4pLCB3aGVyZSBlYWNoIG51 bWJlciBpcyBpbiB0aGUgcmFuZ2UgMCB0aHJvdWdoDQ0KfCAgICAgMjU1LCB3 aWxsIGJlIGludGVycHJldGVkIGFzIGFuIElQdjQgYWRkcmVzcy4gIEV4YW1w bGVzOg0NCiANDQogICAgICAgICAgICAgIDEwLjAuMC4xL3RhcGVmYXJtMQ0N CiAgICAgICAgICAgICAgMTAuMC4wLjINDQogDQ0KIA0NCiANDQogDQ0KIFNh dHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBbUGFnZSA3XQ0NCiAMDQ0KIGlTQ1NJICAgICAg ICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0NCiANDQogDQ0KIDMuICBN ZXNzYWdlIEZvcm1hdHMNDQogDQ0KICAgICAgQWxsIG11bHRpLWJ5dGUgaW50 ZWdlcnMgc3BlY2lmaWVkIGluIGZvcm1hdHMgZGVmaW5lZCBpbiB0aGlzIGRv Y3UtDQ0KICAgICAgbWVudCBhcmUgdG8gYmUgcmVwcmVzZW50ZWQgaW4gbmV0 d29yayBieXRlIG9yZGVyIChpLmUuLCBiaWcNDQogICAgICBlbmRpYW4pLg0N CiANDQogMy4xLiAgVGVtcGxhdGUgSGVhZGVyIGFuZCBPcGNvZGVzDQ0KIA0N CiAgICAgIEFsbCBpU0NTSSBtZXNzYWdlcyBhbmQgcmVzcG9uc2VzIGhhdmUg YSBoZWFkZXIgb2YgdGhlIHNhbWUgbGVuZ3RoDQ0KICAgICAgKDQwIGJ5dGVz KS4gQWRkaXRpb25hbCBkYXRhIG1heSBiZSBhZGRlZCwgYXMgbmVjZXNzYXJ5 LCBiZWdpbm5pbmcNDQogICAgICB3aXRoIGJ5dGUgNDAuIFRoZSBmaWVsZHMg b2YgT3Bjb2RlIGFuZCBMZW5ndGggYXBwZWFyIGluIGFsbCBtZXNzYWdlDQ0K ICAgICAgYW5kIHJlc3BvbnNlIGhlYWRlcnMuIFRoZSBvdGhlciBtb3N0IGNv bW1vbmx5IHVzZWQgZmllbGRzIGFyZSBJbmktDQ0KICAgICAgdGlhdG9yIFRh c2sgVGFnLCBMb2dpY2FsIFVuaXQgTnVtYmVyLCBhbmQgRmxhZ3MsIHdoaWNo LCB3aGVuIHVzZWQsDQ0KICAgICAgYWx3YXlzIGFwcGVhciBpbiB0aGUgc2Ft ZSBsb2NhdGlvbiBvZiB0aGUgaGVhZGVyLg0NCiANDQogICAgICBCeXRlIC8g ICAgMCAgICAgICB8ICAgICAgIDEgICAgICAgfCAgICAgICAyICAgICAgIHwg ICAgICAgMyAgICAgICB8DQ0KICAgICAgICAgLyAgICAgICAgICAgICAgfCAg ICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAg fA0NCiAgICAgICAgfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8 NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwNDQogICAgICAgICst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgIDB8IE9wY29kZSAgICAgICAg fCAgICAgT3Bjb2RlLXNwZWNpZmljIGZpZWxkcyAgICAgICAgICAgICAgICAg ICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAg NHwgTGVuZ3RoIG9mIERhdGEgKGFmdGVyIDQwIGJ5dGUgSGVhZGVyKSAgICAg ICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKw0NCiAgICAgICA4fCBMVU4gb3IgT3Bjb2RlLXNwZWNpZmljIGZp ZWxkcyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAg ICAgICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICArDQ0KICAgICAgMTJ8ICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQog ICAgICAxNnwgSW5pdGlhdG9yIFRhc2sgVGFnICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKw0NCiAgICAgIDIwLyBPcGNvZGUtc3BlY2lmaWMgZmll bGRzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8N DQogICAgICAgKy8gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICArLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDQwDQ0KIA0NCiANDQogMy4xLjEu ICBPcGNvZGUNDQogDQ0KICAgICAgVGhlIE9wY29kZSBpbmRpY2F0ZXMgd2hp Y2ggaVNDU0kgdHlwZSBvZiBtZXNzYWdlIG9yIHJlc3BvbnNlIGlzDQ0KICAg ICAgZW5jYXBzdWxhdGVkIGJ5IHRoZSBoZWFkZXIuICBWYWxpZCBvcGNvZGVz IGZvciBtZXNzYWdlcyAoc2VudCBieQ0NCiAgICAgIGluaXRpYXRvciB0byB0 YXJnZXQpIGFyZToNDQogICAgICAgIDB4MDAgUGluZyBDb21tYW5kIChmcm9t IGluaXRpYXRvciB0byB0YXJnZXQpDQ0KICAgICAgICAweDAxIFNDU0kgQ29t bWFuZCAoZW5jYXBzdWxhdGVzIGEgU0NTSSBDb21tYW5kIERlc2NyaXB0b3Ig QmxvY2spDQ0KICAgICAgICAweDAyIFNDU0kgVGFzayBNYW5hZ2VtZW50IE1l c3NhZ2UNDQogICAgICAgIDB4MDMgTG9naW4gQ29tbWFuZA0NCiAgICAgICAg MHgwNCBUZXh0IENvbW1hbmQNDQogICAgICAgIDB4MDUgU0NTSSBEYXRhIChm b3IgV1JJVEUgb3BlcmF0aW9uKQ0NCiANDQogICAgICBWYWxpZCBvcGNvZGVz IGZvciByZXNwb25zZXMgKHNlbnQgYnkgdGFyZ2V0IHRvIGluaXRpYXRvcikg YXJlOg0NCiANDQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtp cywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2Ug OF0NDQogDA0NCiBpU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5l IDIwMDANDQogDQ0KIA0NCiAgICAgICAgMHg4MCBQaW5nIFJlc3BvbnNlIChm cm9tIHRhcmdldCB0byBpbml0aWF0b3IpDQ0KICAgICAgICAweDgxIFNDU0kg UmVzcG9uc2UgKGNvbnRhaW5zIFNDU0kgc3RhdHVzIGFuZCBwb3NzaWJseSBz ZW5zZQ0NCiAgICAgICAgICAgICBpbmZvcm1hdGlvbiBvciBvdGhlciByZXNw b25zZSBpbmZvcm1hdGlvbikNDQogICAgICAgIDB4ODIgU0NTSSBUYXNrIE1h bmFnZW1lbnQgUmVzcG9uc2UNDQogICAgICAgIDB4ODMgTG9naW4gUmVzcG9u c2UNDQogICAgICAgIDB4ODQgVGV4dCBSZXNwb25zZQ0NCiAgICAgICAgMHg4 NSBTQ1NJIERhdGEgKGZvciBSRUFEIG9wZXJhdGlvbikNDQogICAgICAgIDB4 ODYgUmVhZHkgVG8gVHJhbnNmZXIgKFJUVCAtIHNlbnQgYnkgdGFyZ2V0IHRv IGluaXRpYXRvciB3aGVuDQ0KICAgICAgICAgICAgIGl0IGlzIHJlYWR5IHRv IHJlY2VpdmUgZGF0YSBmcm9tIGluaXRpYXRvcikNDQogICAgICAgIDB4ODcg QXN5bmNocm9ub3VzIEV2ZW50IChzZW50IGJ5IHRhcmdldCB0byBpbml0aWF0 b3IgdG8gaW5kaWNhdGUNDQogICAgICAgIGNlcnRhaW4gc3BlY2lhbCBjb25k aXRpb25zKQ0NCiAgICAgICAgMHg4OCBPcGNvZGUgTm90IFVuZGVyc3Rvb2QN DQogICAgICAgIDB4ODkgT3BlbiBEYXRhIENvbm5lY3Rpb25zIFJlc3BvbnNl IChvcHRpb25hbCkNDQogDQ0KIDMuMS4yLiAgTGVuZ3RoDQ0KIA0NCiAgICAg IFRoZSBMZW5ndGggZmllbGQgaW5kaWNhdGVzIHRoZSBudW1iZXIgb2YgYnl0 ZXMsIGJleW9uZCB0aGUgZmlyc3QgNDANDQogICAgICBieXRlcywgdGhhdCBh cmUgYmVpbmcgc2VudCB0b2dldGhlciB3aXRoIHRoaXMgbWVzc2FnZSBoZWFk ZXIuIEl0IGlzDQ0KICAgICAgYW50aWNpcGF0ZWQgdGhhdCBtb3N0IGlTQ1NJ IG1lc3NhZ2VzIGFuZCByZXNwb25zZXMgKG5vdCBjb3VudGluZw0NCiAgICAg IGRhdGEgdHJhbnNmZXIgbWVzc2FnZXMpIHdpbGwgbm90IG5lZWQgbW9yZSB0 aGFuIHRoZSA0MCBieXRlIGhlYWRlciwNDQogICAgICBhbmQgaGVuY2UgdGhl IExlbmd0aCBmaWVsZCB3aWxsIGNvbnRhaW4gdGhlIHZhbHVlIDAuICBJdCBp cw0NCiAgICAgIGV4cGVjdGVkIHRoYXQgbGFyZ2VyIHRoYW4gMTYgYnl0ZSBD REJzIGFuZCBwYXJhbWV0ZXIgZGF0YSB3aWxsIGZvbC0NDQogICAgICBsb3cg dGhlIGhlYWRlci4NDQogDQ0KIDMuMS4zLiAgTFVODQ0KIA0NCiAgICAgIFRo ZSBMVU4gc3BlY2lmaWVzIHRoZSBMb2dpY2FsIFVuaXQgZm9yIHdoaWNoIHRo ZSBjb21tYW5kIGlzIHRhci0NDQogICAgICBnZXRlZC4gIElmIHRoZSBjb21t YW5kIGRvZXMgbm90IHJlbGF0ZSB0byBhIExvZ2ljYWwgVW5pdCwgdGhpcw0N CiAgICAgIGZpZWxkIGlzIGVpdGhlciBpZ25vcmVkIG9yIG1heSBiZSB1c2Vk IGZvciBzb21lIG90aGVyIHB1cnBvc2UuDQ0KICAgICAgQWNjb3JkaW5nIHRv IFtTQU0yXSwgYSBMb2dpY2FsIFVuaXQgTnVtYmVyIGNhbiB0YWtlIHVwIHRv IGEgNjQtYml0DQ0KICAgICAgZmllbGQgdGhhdCBpZGVudGlmaWVzIHRoZSBM b2dpY2FsIFVuaXQgd2l0aGluIGEgdGFyZ2V0IGRldmljZS4gVGhlDQ0KICAg ICAgZXhhY3QgZm9ybWF0IG9mIHRoaXMgZmllbGQgY2FuIGJlIGZvdW5kIGlu IHRoZSBbU0FNMl0gZG9jdW1lbnQuDQ0KIA0NCiAzLjEuNC4gIEluaXRpYXRv ciBUYXNrIFRhZw0NCiANDQogICAgICBUaGUgaW5pdGlhdG9yIGFzc2lnbnMg YSBUYXNrIElkIChvciB0YWcpIHRvIGVhY2ggU0NTSSB0YXNrIHRoYXQgaXQN DQogICAgICBpc3N1ZXMuICAoUmVjYWxsIHRoYXQgYSB0YXNrIGlzIGEgbGlu a2VkIHNldCBvZiBTQ1NJIGNvbW1hbmRzLikNDQogICAgICBUaGlzIFRhZyBp cyBhIGluaXRpYXRvci13aWRlIHVuaXF1ZSBpZGVudGlmaWVyIHRoYXQgY2Fu IGJlIHVzZWQgdG8NDQogICAgICB1bmlxdWVseSBpZGVudGlmeSB0aGUgVGFz ay4NDQogDQ0KIDMuMS41LiAgT3Bjb2RlLXNwZWNpZmljIGZpZWxkcw0NCiAN DQogICAgICBUaGVzZSBmaWVsZCBoYXZlIGRpZmZlcmVudCBtZWFuaW5ncyBm b3IgZGlmZmVyZW50IG1lc3NhZ2VzLg0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRo ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSA5XQ0NCiAM DQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0N CiANDQogDQ0KIDMuMi4gIFNDU0kgQ29tbWFuZA0NCiANDQogDQ0KICAgICAg Qnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwgICAgICAgMiAg ICAgICB8ICAgICAgIDMgICAgICAgfA0NCiAgICAgICAgIC8gICAgICAgICAg ICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAg ICAgICAgIHwNDQogICAgICAgIHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAz IDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8DQ0KICAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCnwgICAgICAwfCBPcGNvZGUg KDB4MDEpIHxJfFJ8QXxSc3Z8QVRUUiB8ICAgICAgICAgICBDbWRSTg0NCiAg ICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAgNHwgTGVuZ3Ro ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0N CiAgICAgICA4fCBMb2dpY2FsIFVuaXQgTnVtYmVyIChMVU4pICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAgICsgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICArDQ0KICAgICAgMTJ8ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg fA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAxNnwg SW5pdGlhdG9yIFRhc2sgVGFnICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKw0NCiAgICAgIDIwfCBFeHBlY3RlZCBEYXRhIFRyYW5zZmVyIExlbmd0 aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAg ICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgMjR8IFNDU0kgQ29tbWFu ZCBEZXNjcmlwdG9yIEJsb2NrIChDREIpICAgICAgICAgICAgICAgICAgICAg ICAgICAgfA0NCiAgICAgICAgKyAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICsNDQogICAg ICAyOHwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICArICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgKw0NCiAgICAgIDMyfCAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQog ICAgICAgICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICArDQ0KICAgICAgMzZ8ICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsN DQogICAgICA0MC8gQWRkaXRpb25hbCBEYXRhIChDb21tYW5kIERlcGVuZGVu dCkgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICsvICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgLw0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNDQogDQ0KIA0NCnwzLjIuMS4gIEZsYWdzLg0NCiANDQp8ICAgICBUaGUg RmxhZ3MgZmllbGQgZm9yIGEgU0NTSSBDb21tYW5kIGNvbnNpc3RzIG9mIG9u IGJ5dGUuDQ0KICAgICAgICAgICAgICAgIEJ5dGUgMg0NCnwgICAgICAgICAg YjAgICAoSSkgSW1tZWRpYXRlIERhdGEgZnJvbSBpbml0aWF0b3IgdG8gdGFy Z2V0DQ0KfCAgICAgICAgICAgICAgICh3cml0ZS9jb250cm9sKS4NDQp8ICAg ICAgICAgIGIxICAgKHIpIHNldCB3aGVuIGRhdGEgaXMgZXhwZWN0ZWQgdG8g ZmxvdyBmcm9tIHRhcmdldCB0byBpbmktDQ0KICAgICAgICAgICAgICAgIHRp YXRvciAocmVhZCkuDQ0KICAgICAgICAgICBiMiAgIChBKSBzZXQgdG8gdHVy biBvZmYgQXV0b3NlbnNlIGZvciB0aGlzIGNvbW1hbmQgKHNlZQ0NCiAgICAg ICAgICAgICAgICBbU0FNMl0pLg0NCnwgICAgICAgICAgYjMtNCBSZXNlcnZl ZCAoc2hvdWxkIGJlIDApDQ0KfCAgICAgICAgICBiNS03IHVzZWQgdG8gaW5k aWNhdGUgVGFzayBBdHRyaWJ1dGVzLg0NCiANDQogICAgICBBdXRvc2Vuc2Ug cmVmZXJzIHRvIHRoZSBhdXRvbWF0aWMgcmV0dXJuIG9mIHNlbnNlIGRhdGEg dG8gdGhlIGluaS0NDQogICAgICB0aWF0b3IgaW4gY2FzZSBhIGNvbW1hbmQg ZGlkIG5vdCBjb21wbGV0ZSBzdWNjZXNzZnVsbHkuIElmDQ0KICAgICAgYXV0 b3NlbnNlIGlzIHR1cm5lZCBvZmYsIHRoZSBpbml0aWF0b3IgbXVzdCBleHBs aWNpdGx5IHJlcXVlc3QgdGhhdA0NCiANDQogDQ0KIA0NCiBTYXRyYW4sIFNt aXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBbUGFnZSAxMF0NDQogDA0NCiBpU0NTSSAgICAgICAgICAgICAg ICAgICAgICAgICBKdW5lIDIwMDANDQogDQ0KIA0NCiAgICAgIHNlbnNlIGRh dGEgYmUgc2VudCB0byBpdCBhZnRlciBzb21lIGNvbW1hbmQgaGFzIGNvbXBs ZXRlZCB3aXRoIGENDQogICAgICBDSEVDSyBDT05ESVRJT04gc3RhdHVzLg0N CiANDQp8My4yLjIuICBUYXNrIEF0dHJpYnV0ZXMNDQogDQ0KICAgICAgVGhl IFRhc2sgQXR0cmlidXRlIGZpZWxkIChBVFRSKSBjYW4gaGF2ZSBvbmUgb2Yg dGhlIGZvbGxvd2luZw0NCiAgICAgIGludGVnZXIgdmFsdWVzIChzZWUgW1NB TTJdIGZvciBkZXRhaWxzKToNDQogDQ0KICAgICAgICAgICAgICAwICAgIFVu dGFnZ2VkDQ0KICAgICAgICAgICAgICAxICAgIFNpbXBsZQ0NCiAgICAgICAg ICAgICAgMiAgICBPcmRlcmVkDQ0KICAgICAgICAgICAgICAzICAgIEhlYWQg b2YgUXVldWUNDQogICAgICAgICAgICAgIDQgICAgQUNBDQ0KIA0NCiANDQog My4yLjMuICBDb21tYW5kIFJlZmVyZW5jZSBOdW1iZXIgKENtZFJOKQ0NCiAN DQogICAgICBUaGUgQ29tbWFuZCBSZWZlcmVuY2UgTnVtYmVyIChDbWRSTikg aXMgcHJvdmlkZWQgYnkgdGhlIGluaXRpYXRvcg0NCiAgICAgIHRvIGFzc2lz dCBpbiBwZXJmb3JtaW5nIG9yZGVyZWQgZGVsaXZlcnkgZm9yIGlTQ1NJIGNv bW1hbmRzLiAgQ21kUk4NDQogICAgICBpcyByZWZsZWN0aW5nIHRoZSB2YWx1 ZSBvZiBhIDE2IGJpdCBjb3VudGVyIG1haW50YWluZWQgYnkgdGhlIGluaS0N DQogICAgICB0aWF0b3IgYW5kIGluY3JlYXNlZCBieSAxIGZvciBldmVyeSBj b21tYW5kIHJlY2VpdmVkIGJ5IHRoZSBpU0NTSQ0NCiAgICAgIGRlbGl2ZXJ5 IG1lY2hhbmlzbS4gIFRoZSBjb3VudGVyIGlzIHNldCB0byBhbiBpbml0aWFs IHZhbHVlIGF0IHNlcy0NDQogICAgICBzaW9uIGluaXRpYXRpb24gKGRlZmF1 bHQgaXMgMCkgYW5kIHdoZW4gc2VuZGluZyB0YXJnZXQgcmVzZXRzICgwKS4N DQogDQ0KIDMuMi40LiAgRXhwZWN0ZWQgRGF0YSBUcmFuc2ZlciBMZW5ndGgN DQogDQ0KICAgICAgVGhlIEV4cGVjdGVkIERhdGEgVHJhbnNmZXIgTGVuZ3Ro IGZpZWxkIHN0YXRlcyB0aGUgbnVtYmVyIG9mIGJ5dGVzDQ0KICAgICAgb2Yg ZGF0YSB0aGF0IHRoZSBpbml0aWF0b3IgZXhwZWN0cyB3aWxsIGJlIHNlbnQg Zm9yIHRoaXMgKFJFQUQgb3INDQp8ICAgICBXUklURSkgU0NTSSBvcGVyYXRp b24gaW4gU0NTSSBEYXRhIHBhY2tldHMuICBGb3IgYSBXUklURSBvcGVyYXRp b24sDQ0KfCAgICAgdGhlIGluaXRpYXRvciB1c2VzIHRoaXMgZmllbGQgdG8g c3BlY2lmeSB0aGUgbnVtYmVyIG9mIGJ5dGVzIG9mDQ0KfCAgICAgZGF0YSBp dCBleHBlY3RzIHRvIHRyYW5zZmVyIGZvciB0aGlzIG9wZXJhdGlvbiAobm90 IGNvdW50aW5nIGRhdGENDQp8ICAgICBoZWFkZXJzKS4gIEZvciBhIFJFQUQg b3BlcmF0aW9uLCB0aGUgaW5pdGlhdG9yIHVzZXMgdGhpcyBmaWVsZCB0bw0N CnwgICAgIHNwZWNpZnkgdGhlIG51bWJlciBvZiBieXRlcyBvZiBkYXRhIGl0 IGV4cGVjdHMgdGhlIHRhcmdldCB0bw0NCnwgICAgIHRyYW5zZmVyIHRvIHRo ZSBpbml0aWF0b3IgKG5vdCBjb3VudGluZyBkYXRhIGhlYWRlcnMpLiAgSWYg bm8gZGF0YQ0NCnwgICAgIHdpbGwgYmUgdHJhbnNmZXJyZWQgaW4gU0NTSSBE YXRhIHBhY2tldHMgZm9yIHRoaXMgU0NTSSBvcGVyYXRpb24sDQ0KfCAgICAg dGhpcyBmaWVsZCBzaG91bGQgYmUgc2V0IHRvIDAuDQ0KIA0NCiAgICAgIFVw b24gY29tcGxldGlvbiBvZiBhIGRhdGEgdHJhbnNmZXIsIHRoZSB0YXJnZXQg d2lsbCBpbmZvcm0gdGhlIGluaS0NDQogICAgICB0aWF0b3Igb2YgaG93IG1h bnkgYnl0ZXMgd2VyZSBhY3R1YWxseSBwcm9jZXNzZWQgKHNlbnQgb3IgcmVj ZWl2ZWQpDQ0KICAgICAgYnkgdGhlIHRhcmdldC4NDQogDQ0KIDMuMi41LiAg U0NTSSBDb21tYW5kIERlc2NyaXB0b3IgQmxvY2sgKENEQikNDQogDQ0KICAg ICAgVGhlcmUgYXJlIDE2IGJ5dGVzIGluIHRoZSBDREIgZmllbGQsIGRlc2ln bmVkIHRvIGFjY29tbW9kYXRlIHRoZQ0NCnwgICAgIGxhcmdlc3QgY3VycmVu dGx5IGRlZmluZWQgQ0RCLiAgSWYsIGluIHRoZSBmdXR1cmUsIGxhcmdlciBD REJzIGFyZQ0NCnwgICAgIGFsbG93ZWQsIHRoZSBzcGlsbC1vdmVyIG9mIHRo ZSBDREIgbWF5IGV4dGVuZCBiZXlvbmQgdGhlIDQwLWJ5dGUuDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1l dGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMTFdDQ0K IAwNDQogaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAw DQ0KIA0NCiANDQogMy4yLjYuICBDb21tYW5kLURhdGENDQogDQ0KfCAgICAg U29tZSBTQ1NJIGNvbW1hbmRzIHJlcXVpcmUgYWRkaXRpb25hbCBwYXJhbWV0 ZXIgZGF0YSB0byBhY2NvbXBhbnkNDQogICAgICB0aGUgU0NTSSBjb21tYW5k LiBUaGlzIGRhdGEgbWF5IGJlIHBsYWNlZCBiZXlvbmQgdGhlIDQwLWJ5dGUg Ym91bi0NDQp8ICAgICBkYXJ5IG9mIHRoZSBpU0NTSSBoZWFkZXIuICBBbHRl cm5hdGl2ZWx5IHVzZXIgZGF0YSBjYW4gYmUgcGxhY2VkIGluDQ0KfCAgICAg dGhlIHRoZSBzYW1lIFBEVSAoaW4gYm90aCBjYXNlcyB3ZSB0YWxrIGFib3V0 IGltbWVkaWF0ZSBkYXRhKS4gVGhlDQ0KfCAgICAgTGVuZ3RoIGZpZWxkIGlz IHNldCB0byB0aGUgbGVuZ3RoIG9mIHRoaXMgZGF0YSBiZXlvbmQgdGhlIDQw LWJ5dGUNDQp8ICAgICBoZWFkZXIgKGkuZS4gaW5jbHVkZXMgdGhlIENEQiBl eHRlbnNpb24gaWYgcHJlc2VudCkuICBUaGUgQ0RCDQ0KfCAgICAgbGVuZ3Ro IGlzOg0NCnxMZW5ndGggKyAxNiAtIEkqRXhwZWN0ZWREYXRhVHJhbnNmZXJM ZW5ndGgNDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiBTYXRyYW4s IFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBbUGFnZSAxMl0NDQogDA0NCiBpU0NTSSAgICAgICAgICAg ICAgICAgICAgICAgICBKdW5lIDIwMDANDQogDQ0KIA0NCiAzLjMuICBTQ1NJ IFJlc3BvbnNlDQ0KIA0NCiANDQogICAgICBCeXRlIC8gICAgMCAgICAgICB8 ICAgICAgIDEgICAgICAgfCAgICAgICAyICAgICAgIHwgICAgICAgMyAgICAg ICB8DQ0KICAgICAgICAgLyAgICAgICAgICAgICAgfCAgICAgICAgICAgICAg IHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfA0NCiAgICAgICAg fDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIg MSAwfDcgNiA1IDQgMyAyIDEgMHwNDQogICAgICAgICstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rDQ0KfCAgICAgIDB8IE9wY29kZSAoMHg4MSkgfCAgUnN2ZCgwKSAg fE98VXwgICAgICAgTWF4Q21kUk4gICAgICAgICAgICAgICAgfA0NCiAgICAg ICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAgNHwgTGVuZ3RoICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAg ICAgICA4fCBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAgICsgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICArDQ0KICAgICAgMTJ8ICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0N CiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAxNnwgSW5p dGlhdG9yIFRhc2sgVGFnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Kw0NCiAgICAgIDIwfCBSZXNpZHVhbCBDb3VudCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAgICst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rDQ0KfCAgICAgMjR8IENvbW1hbmQgU3RhdHVz fGlTQ1NJIFN0YXR1cyAgIHwgICAgICAgU3RhdFJOICAgICAgICAgICAgICAg ICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAy OC8gUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgLw0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAg ICA0MC8gUmVzcG9uc2Ugb3IgU2Vuc2UgRGF0YSAob3B0aW9uYWwpICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICsvICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgLw0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQog DQ0KIA0NCiAzLjMuMS4gIEZsYWdzDQ0KIA0NCiAgICAgIFRoZSBTQ1NJIFJl c3BvbnNlIGhhcyBpdHMgb3duIHNldCBvZiBmbGFncywgdGhhdCBkaWZmZXJz IGZyb20gdGhlDQ0KICAgICAgZmxhZ3MgZm9yIGEgU0NTSSBDb21tYW5kLg0N CiAgICAgICAgICAgICAgICBCeXRlIDINDQogICAgICAgICAgIGIwICAgKFUp IHNldCBmb3IgUmVzaWR1YWwgVW5kZXJmbG93LiBJbiB0aGlzIGNhc2UsIHRo ZSBSZXNpLQ0NCiAgICAgICAgICAgICAgICBkdWFsIENvdW50IGluZGljYXRl cyBob3cgbWFueSBieXRlcyB3ZXJlIG5vdCB0cmFuc2ZlcnJlZA0NCiAgICAg ICAgICAgICAgICBvdXQgb2YgdGhvc2UgZXhwZWN0ZWQgdG8gYmUgdHJhbnNm ZXJyZWQuDQ0KICAgICAgICAgICBiMSAgIChPKSBzZXQgZm9yIFJlc2lkdWFs IE92ZXJmbG93LiBJbiB0aGlzIGNhc2UsIHRoZSBSZXNpZHVhbA0NCiAgICAg ICAgICAgICAgICBDb3VudCBpbmRpY2F0ZXMgaG93IG1hbnkgYnl0ZXMgY291 bGQgbm90IGJlIHRyYW5zZmVycmVkDQ0KICAgICAgICAgICAgICAgIGJlY2F1 c2UgdGhlIGluaXRpYXRvcidzIEV4cGVjdGVkIERhdGEgVHJhbnNmZXIgTGVu Z3RoIHdhcw0NCiAgICAgICAgICAgICAgICB0b28gc21hbGwuDQ0KICAgICAg ICAgICBiMi03IG5vdCB1c2VkIChzaG91bGQgYmUgc2V0IHRvIDApLg0NCiAg ICAgIEJpdHMgMCBhbmQgMSBhcmUgbXV0dWFsbHkgZXhjbHVzaXZlLg0NCiAN DQp8My4zLjIuICBNYXhDbWRSTg0NCiANDQp8ICAgICBJbmRpY2F0ZXMgdGhl IG1heGltdW0gQ21kUk4gdGhlIGluaXRpYXRvciBzaG91bGQgc2VuZC4gSXQg d2lsbCBzZXQNDQp8ICAgICBhbiBpbnRlcm5hbCBsaW1pdCByZWdpc3Rlci4g VGhlIGluaXRpYXRvciB3aWxsIHJlZnJhaW4gZnJvbSBzZW5kaW5nDQ0KIA0N CiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDEzXQ0NCiAMDQ0K IGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0NCiAN DQogDQ0KfCAgICAgY29tbWFuZHMgbnVtYmVyZWQgcGFzdCBNYXhDbWRSTiAo Y29uc2lkZXJpbmcgYWxzbyB3cmFwLWFyb3VuZCkuDQ0KfCAgICAgQXR0ZW50 aW9uIHNob3VsZCBiZSBwYWlkIHRvIHRoZSBmYWN0IHRoYXQgcmVzcG9uc2Ug UERVcyBjYW4gYXJyaXZlDQ0KfCAgICAgaW4gIndyb25nIG9yZGVyIi4gVGhl IGludGVybmFsIGxpbWl0IHJlZ2lzdGVyIGNhbiBvbmx5IGJlIGFkdmFuY2Vk DQ0KfCAgICAgYnkgaW5jb21pbmcgcmVzcG9uc2VzIChjb25zaWRlcmluZyBh bHNvIHdyYXBhcm91bmRzKS4gIEl0IGlzDQ0KfCAgICAgYXNzdW1lZCB0aGF0 IGFueSB0YXJnZXQgd2lsbCBhY2NlcHQgbGVzcyB0aGFuIDY0ayBvdXRzdGFu ZGluZyBjb20tDQ0KfCAgICAgbWFuZHMuDQ0KIA0NCnwzLjMuMy4gIFJlc2lk dWFsIENvdW50DQ0KIA0NCiAgICAgIFRoZSBSZXNpZHVhbCBDb3VudCBmaWVs ZCBpcyB2YWxpZCBvbmx5IGluIGNhc2UgZWl0aGVyIHRoZSBSZXNpZHVhbA0N CiAgICAgIFVuZGVyZmxvdyBiaXQgb3IgUmVzaWR1YWwgT3ZlcmZsb3cgYml0 IGlzIHNldC4gSWYgbmVpdGhlciBiaXQgaXMNDQp8ICAgICBzZXQsIHRoZSBS ZXNpZHVhbCBDb3VudCBmaWVsZCBzaG91bGQgYmUgMC4gIElmIHRoZSBSZXNp ZHVhbCBVbmRlci0NDQp8ICAgICBmbG93IGJpdCBpcyBzZXQsIHRoZSBSZXNp ZHVhbCBDb3VudCBpbmRpY2F0ZXMgaG93IG1hbnkgYnl0ZXMgd2VyZQ0NCnwg ICAgIG5vdCB0cmFuc2ZlcnJlZCBvdXQgb2YgdGhvc2UgZXhwZWN0ZWQgdG8g YmUgdHJhbnNmZXJyZWQuICBJZiB0aGUNDQp8ICAgICBSZXNpZHVhbCBPdmVy ZmxvdyBiaXQgaXMgc2V0LCB0aGUgUmVzaWR1YWwgQ291bnQgaW5kaWNhdGVz IGhvdyBtYW55DQ0KfCAgICAgYnl0ZXMgY291bGQgbm90IGJlIHRyYW5zZmVy cmVkIGJlY2F1c2UgdGhlIGluaXRpYXRvcidzIEV4cGVjdGVkDQ0KfCAgICAg RGF0YSBUcmFuc2ZlciBMZW5ndGggd2FzIHRvbyBzbWFsbC4NDQogDQ0KfDMu My40LiAgQ29tbWFuZCBTdGF0dXMNDQogDQ0KICAgICAgVGhlIENvbW1hbmQg U3RhdHVzIGZpZWxkIGlzIHVzZWQgdG8gcmVwb3J0IHRoZSBTQ1NJIHN0YXR1 cyBvZiB0aGUNDQogICAgICBjb21tYW5kIChhcyBzcGVjaWZpZWQgaW4gW1NB TTJdKS4NDQogDQ0KfDMuMy41LiAgaVNDU0kgU3RhdHVzDQ0KIA0NCiAgICAg IFRoZSBpU0NTSSBTdGF0dXMgZmllbGQgaXMgdXNlZCB0byByZXBvcnQgdGhl IHN0YXR1cyBvZiB0aGUgY29tbWFuZA0NCiAgICAgIGJlZm9yZSBpdCB3YXMg c2VudCBieSB0aGUgdGFyZ2V0IHRvIHRoZSBMVU4uIFRoZSB2YWx1ZXMgYXJl IGdpdmVuDQ0KICAgICAgYmVsb3cuDQ0KIA0NCiAgICAgICAgICAgICAgMCBH b29kIHN0YXR1cw0NCiAgICAgICAgICAgICAgMSBpU0NTSSBjaGVjaw0NCiAN DQogICAgICBJZiB0aGUgaVNDU0kgZmllbGQgaXMgbm90IDAgdGhlIGNvbW1h bmQgc3RhdHVzIHdpbGwgaW5kaWNhdGUgQ0hFQ0sNDQp8ICAgICBDT05ESVRJ T04NDQogDQ0KfDMuMy42LiAgUmVzcG9uc2Ugb3IgU2Vuc2UgRGF0YQ0NCiAN DQogICAgICBJZiBBdXRvc2Vuc2Ugd2FzIG5vdCBkaXNhYmxlZCBpbiB0aGUg b3JpZ2luYXRpbmcgQ0RCIGFuZCB0aGUgQ29tLQ0NCiAgICAgIG1hbmQgU3Rh dHVzIHdhcyBDSEVDSyBDT05ESVRJT04gKDB4MDIpLCB0aGVuIHRoZSBSZXNw b25zZSBEYXRhDQ0KfCAgICAgZmllbGQgd2lsbCBjb250YWluIHNlbnNlIGRh dGEgZm9yIHRoZSBmYWlsZWQgY29tbWFuZC4gIFNvbWUgc2Vuc2UNDQp8ICAg ICBjb2RlcyB3aWxsIHJlbGF0ZSB0byBpU0NTSSBjaGVjayBjb25kaXRpb25z IChlLmcuIGV4Y2Vzc2l2ZSBudW1iZXINDQp8ICAgICBvZiBvdXRzdGFuZGlu ZyBjb21tYW5kcywgaW1tZWRpYXRlIGRhdGEgYmxvY2tzIHRvbyBsYXJnZSBl dGMuKS4gIElmDQ0KfCAgICAgdGhlIENvbW1hbmQgU3RhdHVzIGlzIEdvb2Qg KDB4MDApIHRoZW4gdGhlIFJlc3BvbnNlIERhdGEgZmllbGQgd2lsbA0NCnwg ICAgIGNvbnRhaW4gZGF0YSBmcm9tIHRoZSBkYXRhIHBoYXNlIG9mIHRoZSBD REIuICBUaGUgTGVuZ3RoIHBhcmFtZXRlcg0NCnwgICAgIHNwZWNpZmllcyB0 aGUgbnVtYmVyIG9mIGJ5dGVzIGluIHRoaXMgZmllbGQuICBJZiBubyBlcnJv ciBvY2N1cnJlZCwNDQp8ICAgICBhbmQgbm8gZGF0YSBpcyBuZWVkZWQgZm9y IHRoZSByZXNwb25zZSB0byB0aGUgU0NTSSBDb21tYW5kIHRoZQ0NCnwgICAg IExlbmd0aCBmaWVsZCBpcyAwLiAgTm90ZSB0aGF0IGlmIHRoZSBDb21tYW5k IFN0YXR1cyB3YXMgQ0hFQ0sgQ09OLQ0NCnwgICAgIERJVElPTiBidXQgQXV0 b3NlbnNlIHdhcyBkaXNhYmxlZCwgdGhlbiBzZW5zZSBkYXRhIG11c3QgYmUN DQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1l dGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMTRdDQ0K IAwNDQogaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAw DQ0KIA0NCiANDQp8ICAgICBleHBsaWNpdGx5IHJlcXVlc3RlZCBieSB0aGUg aW5pdGlhdG9yIHdpdGggYSBuZXcgU0NTSSBjb21tYW5kLg0NCiANDQp8My4z LjcuICBTdGF0Uk4gLSBTdGF0dXMgUmVmZXJlbmNlIE51bWJlcg0NCiANDQp8 ICAgICBTdGF0Uk4gaXMgYSByZWZlcmVuY2UgbnVtYmVyIHRoYXQgdGhlIHRh cmdldCBpU0NTSSBsYXllciBnZW5lcmF0ZXMNDQp8ICAgICB3aGVuZXZlciBp dCBpc3N1ZXMgYSByZXNwb25zZSBieSBpbmNyZW1lbnRpbmcgYW4gaW50ZXJu YWwgY291bnRlci4NDQp8ICAgICBBIGdhcCBpbiBTdGF0Uk4gaW5kaWNhdGVz IGEgbG9zdCBzdGF0dXMgKHBvc3NpYmxlIGR1ZSB0byBjb25uZWN0aW9uDQ0K fCAgICAgZmFpbHVyZSkgYW5kIGJlIHJlY292ZXJlZCBieSByZWlzc3Vpbmcg dGhlIG91dHN0YW5kaW5nIGNvbW1hbmQgd2l0aA0NCnwgICAgIHRoZSBvcmln aW5hbCBUYXNrSUQgYW5kIENtZFJOLg0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0 aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAxNV0NDQog DA0NCiBpU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDAN DQogDQ0KIA0NCiAzLjQuICBBc3luY2hyb25vdXMgRXZlbnQNDQogDQ0KICAg ICAgQW4gQXN5bmNocm9ub3VzIEV2ZW50IG1heSBiZSBzZW50IGZyb20gdGhl IHRhcmdldCB0byB0aGUgaW5pdGlhdG9yDQ0KICAgICAgd2l0aG91dCBjb3Jy ZXNwb25kaW5nIHRvIGEgcGFydGljdWxhciBjb21tYW5kLiBUaGUgdGFyZ2V0 IHNwZWNpZmllcw0NCiAgICAgIHRoZSBzdGF0dXMgZm9yIHRoZSBldmVudCBh bmQgc2Vuc2UgZGF0YS4NDQogDQ0KICAgICAgQnl0ZSAvICAgIDAgICAgICAg fCAgICAgICAxICAgICAgIHwgICAgICAgMiAgICAgICB8ICAgICAgIDMgICAg ICAgfA0NCiAgICAgICAgIC8gICAgICAgICAgICAgIHwgICAgICAgICAgICAg ICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwNDQogICAgICAg IHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAy IDEgMHw3IDYgNSA0IDMgMiAxIDB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKw0NCnwgICAgICAwfCBPcGNvZGUgKDB4ODcpIHwgUmVzZXJ2ZWQg KDApICB8ICAgICAgTWF4Q21kUk4gICAgICAgICAgICAgICAgIHwNDQogICAg ICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgIDR8IExlbmd0aCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQog ICAgICAgOHwgTG9naWNhbCBVbml0IE51bWJlciAoTFVOKSAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICArICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgKw0NCiAgICAgIDEyfCAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwN DQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgMTZ8IFJl c2VydmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKyAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICsNDQogICAgICAyMHwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDI0fCBDb21tYW5kIFN0YXR1 c3xpU0NTSSBTdGF0dXN8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAg IHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgMjh8 RXZlbnQgSW5kaWNhdG9yfCBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSsNDQogICAgICAzMnwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAg ICArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgKw0NCiAgICAgIDM2fCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAg ICAgNDAvIFNlbnNlIERhdGEgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAgICArLyAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIC8NDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0K IA0NCiANDQogMy40LjEuICBpU0NTSSBTdGF0dXMNDQogDQ0KICAgICAgU29t ZSBBc3luY2hyb25vdXMgRXZlbnRzIGFyZSBzdHJpY3RseSByZWxhdGVkIHRv IGlTQ1NJIHdoaWxlIG90aGVycw0NCiAgICAgIGFyZSByZWxhdGVkIHRvIFNB TS0yLiAgVGhlIGNvZGVzIHJldHVybmVkIGZvciBpU0NTSSBBc3luY2hyb25v dXMNDQogICAgICBFdmVudHMgYXJlOg0NCiANDQogICAgICAgICAgICAgIDIg VGFyZ2V0IGlzIGJlaW5nIHJlc2V0Lg0NCiANDQogDQ0KIDMuNC4yLiAgRXZl bnQgSW5kaWNhdG9yDQ0KIA0NCiAgICAgIFRoZSBmb2xsb3dpbmcgdmFsdWVz IGFyZSBkZWZpbmVkLiAgKFNlZSBbU0FNMl0gZm9yIGRldGFpbHMuKQ0NCiAg ICAgICAgICAgMSAgICBBbiBlcnJvciBjb25kaXRpb24gd2FzIGVuY291bnRl cmVkIGFmdGVyIGNvbW1hbmQNDQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0 aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgW1BhZ2UgMTZdDQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAgICAg ICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQogICAgICAgICAgICAgICAg Y29tcGxldGlvbi4NDQogICAgICAgICAgIDIgICAgQSBuZXdseSBpbml0aWFs aXplZCBkZXZpY2UgaXMgYXZhaWxhYmxlLg0NCiAgICAgICAgICAgMyAgICBT b21lIG90aGVyIHR5cGUgb2YgdW5pdCBhdHRlbnRpb24gY29uZGl0aW9uIGhh cyBvY2N1cnJlZC4NDQogICAgICAgICAgIDQgICAgQW4gYXN5bmNocm9ub3Vz IGV2ZW50IGhhcyBvY2N1cnJlZC4NDQogICAgICBTZW5zZSBEYXRhIGFjY29t cGFueWluZyB0aGUgcmVwb3J0IGlkZW50aWZpZXMgdGhlIGNvbmRpdGlvbi4g IFRoZQ0NCiAgICAgIExlbmd0aCBwYXJhbWV0ZXIgaXMgc2V0IHRvIHRoZSBs ZW5ndGggb2YgdGhlIFNlbnNlIERhdGEuDQ0KIA0NCiAzLjQuMy4gIE1heENt ZFJOIC0gaW5mb3JtIGFib3V0IHRoaXMgdmFsdWUgb3RoZXIgaW5pdGlhdG9y cyBhZnRlciBhIHRhci0NDQogZ2V0IFJlc2V0DQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lz LCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDE3 XQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUg MjAwMA0NCiANDQogDQ0KIDMuNS4gIFNDU0kgVGFzayBNYW5hZ2VtZW50IE1l c3NhZ2UNDQogDQ0KIA0NCiAgICAgIEJ5dGUgLyAgICAwICAgICAgIHwgICAg ICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAgICAgICAzICAgICAgIHwN DQogICAgICAgICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAg ICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8DQ0KICAgICAgICB8NyA2 IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8 NyA2IDUgNCAzIDIgMSAwfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNDQp8ICAgICAgMHwgT3Bjb2RlICgweDAyKSB8IE1zZyBpbmRpY2F0b3Ig fCAgICAgICBSZXNlcnZlZCAoMCkgICAgICAgICAgICB8DQ0KICAgICAgICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgICA0fCBMZW5ndGggICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAg IDh8IExvZ2ljYWwgVW5pdCBOdW1iZXIgKExVTikgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKyAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICsNDQogICAgICAxMnwgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDE2fCBJbml0aWF0 b3IgVGFzayBUYWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0K ICAgICAgMjAvIFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAgICArLyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIC8NDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQ0KICAgICAgNDANDQogDQ0KIA0NCiAzLjUuMS4gIE1zZyBJbmRpY2F0b3IN DQogDQ0KICAgICAgVGhlIFRhc2sgTWFuYWdlbWVudCBmdW5jdGlvbnMgcHJv dmlkZSBhbiBpbml0aWF0b3Igd2l0aCBhIHdheSB0bw0NCiAgICAgIGV4cGxp Y2l0bHkgY29udHJvbCB0aGUgZXhlY3V0aW9uIG9mIG9uZSBvciBtb3JlIFRh c2tzLiBUaGUgVGFzaw0NCiAgICAgIE1hbmFnZW1lbnQgZnVuY3Rpb25zIGFy ZSBzdW1tYXJpemVkIGFzIGZvbGxvd3MgKGZvciBhIG1vcmUgZGV0YWlsZWQN DQogICAgICBkZXNjcmlwdGlvbiBzZWUgdGhlIFtTQU0yXSBkb2N1bWVudCk6 DQ0KICAgICAgICAgICAxICAgIEFib3J0IFRhc2stLS1hYm9ydHMgdGhlIHRh c2sgaWRlbnRpZmllZCBieSB0aGUgVGFzayBUYWcNDQogICAgICAgICAgICAg ICAgZmllbGQuDQ0KICAgICAgICAgICAyICAgIEFib3J0IFRhc2sgU2V0LS0t YWJvcnRzIGFsbCBUYXNrcyBpc3N1ZWQgYnkgdGhpcyBpbml0aWEtDQ0KfCAg ICAgICAgICAgICAgIHRvciBvbiB0aGUgTG9naWNhbCBVbml0LiAgLnRpIC01 IDMgICAgQ2xlYXIgQUNBLS0tY2xlYXJzDQ0KfCAgICAgICAgICAgICAgIHRo ZSBBdXRvIENvbnRpbmdlbnQgQWxsZWdpYW5jZSBjb25kaXRpb24uDQ0KICAg ICAgICAgICA0ICAgIENsZWFyIFRhc2sgU2V0LS0tQWJvcnRzIGFsbCBUYXNr cyAoZnJvbSBhbGwgaW5pdGlhdG9ycykNDQogICAgICAgICAgICAgICAgZm9y IHRoZSBMb2dpY2FsIFVuaXQuDQ0KICAgICAgICAgICA1ICAgIExvZ2ljYWwg VW5pdCBSZXNldC4NDQogICAgICAgICAgIDYgICAgVGFyZ2V0IFJlc2V0Lg0N CiAgICAgIEZvciB0aGUgZnVuY3Rpb25zIGFib3ZlIGV4Y2VwdCA8VGFyZ2V0 IFJlc2V0PiwgYSBTQ1NJIFRhc2sgTWFuYWdlLQ0NCiAgICAgIG1lbnQgUmVz cG9uc2UgaXMgcmV0dXJuZWQsIHVzaW5nIHRoZSBJbml0aWF0b3IgVGFzayBU YWcgdG8gaWRlbnRpZnkNDQp8ICAgICB0aGUgb3BlcmF0aW9uIGZvciB3aGlj aCBpdCBpcyByZXNwb25kaW5nLiAgRm9yIHRoZSA8VGFyZ2V0IFJlc2V0Pg0N CnwgICAgIGZ1bmN0aW9uLCB0aGUgdGFyZ2V0IGNhbmNlbHMgYWxsIHBlbmRp bmcgb3BlcmF0aW9ucy4gVGhlIHRhcmdldCBtYXkNDQp8ICAgICBzZW5kIGFu IEFzeW5jaHJvbm91cyBFdmVudCB0byBhbGwgYXR0YWNoZWQgaW5pdGlhdG9y cyBub3RpZnlpbmcNDQp8ICAgICB0aGVtIHRoYXQgdGhlIHRhcmdldCBpcyBi ZWluZyByZXNldC4gIFRoZSB0YXJnZXQgdGhlbiBjbG9zZXMgYWxsIG9mDQ0K fCAgICAgaXRzIFRDUCBjb25uZWN0aW9ucyB0byBhbGwgaW5pdGlhdG9ycyAo YWxsIHNlc3Npb25zIGFyZSB0ZXItDQ0KfCAgICAgbWluYXRlZCkuDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMs IE1ldGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMThd DQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAy MDAwDQ0KIA0NCiANDQogMy42LiAgU0NTSSBUYXNrIE1hbmFnZW1lbnQgUmVz cG9uc2UNDQogDQ0KIA0NCiAgICAgIEJ5dGUgLyAgICAwICAgICAgIHwgICAg ICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAgICAgICAzICAgICAgIHwN DQogICAgICAgICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAg ICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8DQ0KICAgICAgICB8NyA2 IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8 NyA2IDUgNCAzIDIgMSAwfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNDQp8ICAgICAgMHwgT3Bjb2RlICgweDgyKSB8IE1zZyBpbmRpY2F0b3Ig fCAgICAgICAgUmVzZXJ2ZWQgKDApICAgICAgICAgICB8DQ0KICAgICAgICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgICA0fCBMZW5ndGggICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAg IDh8IExvZ2ljYWwgVW5pdCBOdW1iZXIgKExVTikgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKyAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICsNDQogICAgICAxMnwgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDE2fCBJbml0aWF0 b3IgVGFzayBUYWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0K ICAgICAgMjB8IFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSsNDQogICAgICAyNHwgUmVzcG9uc2UgICAgICB8IFJl c2VydmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDI4LyBS ZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIC8NDQogICAgICAgKy8gICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAvDQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDQw DQ0KIA0NCiAgICAgIEZvciB0aGUgZnVuY3Rpb25zIDxBYm9ydCBUYXNrLCBB Ym9ydCBUYXNrIFNldCwgQ2xlYXIgQUNBLCBDbGVhcg0NCiAgICAgIFRhc2sg U2V0LCBMb2dpY2FsIFVuaXQgcmVzZXQ+LCB0aGUgdGFyZ2V0IHBlcmZvcm1z IHRoZSByZXF1ZXN0ZWQNDQogICAgICBUYXNrIE1hbmFnZW1lbnQgZnVuY3Rp b24gYW5kIHNlbmRzIGEgU0NTSSBUYXNrIE1hbmFnZW1lbnQgUmVzcG9uc2UN DQogICAgICBiYWNrIHRvIHRoZSBpbml0aWF0b3IuICBUaGUgdGFyZ2V0IGlu Y2x1ZGVzIGFsbCBvZiB0aGUgaW5mb3JtYXRpb24NDQogICAgICB0aGUgaW5p dGlhdG9yIHByb3ZpZGVkIGluIHRoZSBTQ1NJIFRhc2sgTWFuYWdlbWVudCBN ZXNzYWdlLCBzbyB0aGUNDQogICAgICBpbml0aWF0b3IgY2FuIGtub3cgZXhh Y3RseSB3aGljaCBTQ1NJIFRhc2sgTWFuYWdlbWVudCBNZXNzYWdlIHdhcw0N CiAgICAgIHNlcnZpY2VkLiAgSW4gYWRkaXRpb24sIHRoZSB0YXJnZXQgcHJv dmlkZXMgYSBSZXNwb25zZSBpbmRpY2F0aW9uDQ0KICAgICAgd2hpY2ggbWF5 IHRha2Ugb24gdGhlIGZvbGxvd2luZyB2YWx1ZXM6DQ0KICAgICAgICAgICAw ICAgIEZ1bmN0aW9uIENvbXBsZXRlDQ0KICAgICAgICAgICAxICAgIEZ1bmN0 aW9uIFJlamVjdGVkDQ0KICAgICAgRm9yIHRoZSA8VGFyZ2V0IFJlc2V0PiBm dW5jdGlvbiwgdGhlIHRhcmdldCBjYW5jZWxzIGFsbCBwZW5kaW5nDQ0KICAg ICAgb3BlcmF0aW9ucy4gVGhlIHRhcmdldCBtYXkgc2VuZCBhbiBBc3luY2hy b25vdXMgRXZlbnQgdG8gYWxsDQ0KICAgICAgYXR0YWNoZWQgaW5pdGlhdG9y cyBub3RpZnlpbmcgdGhlbSB0aGF0IHRoZSB0YXJnZXQgaXMgYmVpbmcgcmVz ZXQuDQ0KICAgICAgVGhlIHRhcmdldCB0aGVuIGNsb3NlcyBhbGwgb2YgaXRz IFRDUCBjb25uZWN0aW9ucyB0byBhbGwgaW5pdGlhdG9ycw0NCiAgICAgICh0 ZXJtaW5hdGVzIGFsbCBzZXNzaW9ucykuDQ0KIA0NCiAzLjYuMS4gIE1heENt ZFJOIC0gbWF4aW11bSBDbWRSTiB0aGUgdGFyZ2V0IHdpbGwgYWNjZXB0DQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0 aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgW1BhZ2UgMTldDQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAgICAg ICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQogMy43LiAgUmVhZHkgVG8g VHJhbnNmZXIgKFJUVCkNDQogDQ0KICAgICAgV2hlbiBhbiBpbml0aWF0b3Ig aGFzIHN1Ym1pdHRlZCBhIFNDU0kgQ29tbWFuZCB3aXRoIGRhdGEgcGFzc2lu Zw0NCiAgICAgIGZyb20gdGhlIGluaXRpYXRvciB0byB0aGUgdGFyZ2V0IChX UklURSksIHRoZSB0YXJnZXQgbWF5IHNwZWNpZnkNDQogICAgICB3aGljaCBi bG9ja3Mgb2YgZGF0YSBpdCBpcyByZWFkeSB0byByZWNlaXZlLiBJbiBnZW5l cmFsLCB0aGUgdGFyZ2V0DQ0KICAgICAgbWF5IHJlcXVlc3QgdGhhdCB0aGUg ZGF0YSBibG9ja3MgYmUgZGVsaXZlcmVkIGluIHdoYXRldmVyIG9yZGVyIGlz DQ0KICAgICAgY29udmVuaWVudCBmb3IgdGhlIHRhcmdldCBhdCB0aGF0IHBh cnRpY3VsYXIgaW5zdGFudC4gVGhpcyBpbmZvcm1hLQ0NCiAgICAgIHRpb24g aXMgcGFzc2VkIGZyb20gdGhlIHRhcmdldCB0byB0aGUgaW5pdGlhdG9yIGlu IHRoZSBSZWFkeSBUbw0NCnwgICAgIFRyYW5zZmVyIChSVFQpIG1lc3NhZ2Uu ICBJbiBvcmRlciB0byBhbGxvdyB3cml0ZSBvcGVyYXRpb25zIHdpdGhvdXQN DQp8ICAgICBSVFQsIHRoZSBpbml0aWF0b3IgYW5kIHRhcmdldCBtdXN0IGhh dmUgYWdyZWVkIHRvIGRvIHNvIGJ5IGJvdGgNDQp8ICAgICBzZW5kaW5nIHRo ZSBBbGxvd05vUlRUOnllcyBrZXktcGFpciBhdHRyaWJ1dGUgdG8gZWFjaCBv dGhlciAoZWl0aGVyDQ0KfCAgICAgZHVyaW5nIExvZ2luIG9yIHRocm91Z2gg dGhlIFRleHQgQ29tbWFuZC9SZXNwb25zZSBtZWNoYW5pc20pLg0NCiANDQog ICAgICBCeXRlIC8gICAgMCAgICAgICB8ICAgICAgIDEgICAgICAgfCAgICAg ICAyICAgICAgIHwgICAgICAgMyAgICAgICB8DQ0KICAgICAgICAgLyAgICAg ICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAg ICAgICAgICAgICAgfA0NCiAgICAgICAgfDcgNiA1IDQgMyAyIDEgMHw3IDYg NSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwN DQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KfCAgICAgIDB8IE9w Y29kZSAoMHg4NikgfCBSZXNlcnZlZCAoMCkgIHwgICAgTWF4Q21kUk4gICAg ICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNDQogICAgICAgNHwgTGVuZ3RoICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgICA4fCBSZXNlcnZlZCAoMCkg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHwNDQogICAgICAgICsgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICArDQ0KICAgICAg MTJ8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNDQogICAgICAxNnwgSW5pdGlhdG9yIFRhc2sgVGFnICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDIwfCBEZXNpcmVk IERhdGEgVHJhbnNmZXIgTGVuZ3RoICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0K ICAgICAgMjR8IERhdGEgT2Zmc2V0ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSsNDQogICAgICAyOHwgVGFyZ2V0IFRyYW5zZmVyIFRh ZyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDMyfCBS ZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwNDQogICAgICAgICsgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICArDQ0KICAgICAgMzZ8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICA0MA0NCiANDQogDQ0KfDMu Ny4xLiAgTWF4Q21kUk4gLSBtYXhpbXVtIENtZFJOIHRoZSB0YXJnZXQgd2ls bCBhY2NlcHQNDQogDQ0KfDMuNy4yLiAgRGVzaXJlZCBEYXRhIFRyYW5zZmVy IExlbmd0aCBhbmQgRGF0YSBPZmZzZXQNDQogDQ0KICAgICAgVGhlIHRhcmdl dCBzcGVjaWZpZXMgaG93IG1hbnkgYnl0ZXMgaXQgd2FudHMgdGhlIGluaXRp YXRvciB0byBzZW5kDQ0KICAgICAgYXMgYSByZXN1bHQgb2YgdGhpcyBSVFQg bWVzc2FnZS4gIFRoZSB0YXJnZXQgbWF5IHJlcXVlc3QgdGhlIGRhdGENDQog ICAgICBmcm9tIHRoZSBpbml0aWF0b3IgaW4gc2V2ZXJhbCBjaHVua3MsIG5v dCBuZWNlc3NhcmlseSBpbiB0aGUgb3JpZ2ktDQ0KICAgICAgbmFsIG9yZGVy IG9mIHRoZSBkYXRhLiAgVGhlIHRhcmdldCwgdGhlcmVmb3JlLCBhbHNvIHNw ZWNpZmllcyBhDQ0KIA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVu dHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQ YWdlIDIwXQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAg IEp1bmUgMjAwMA0NCiANDQogDQ0KICAgICAgRGF0YSBPZmZzZXQgaW5kaWNh dGluZyB0aGUgcG9pbnQgYXQgd2hpY2ggdGhlIGRhdGEgdHJhbnNmZXIgc2hv dWxkDQ0KICAgICAgYmVnaW4sIHJlbGF0aXZlIHRvIHRoZSBiZWdpbm5pbmcg b2YgdGhlIHRvdGFsIGRhdGEgdHJhbnNmZXIuDQ0KIA0NCnwzLjcuMy4gIFRh cmdldCBUcmFuc2ZlciBUYWcNDQogDQ0KICAgICAgVGhlIHRhcmdldCBhc3Np Z25zIGl0cyBvd24gdGFnIHRvIGVhY2ggUlRUIHJlcXVlc3QgdGhhdCBpdCBz ZW5kcyB0bw0NCiAgICAgIHRoZSBpbml0aWF0b3IuIFRoaXMgY2FuIGJlIHVz ZWQgYnkgdGhlIHRhcmdldCB0byBlYXNpbHkgaWRlbnRpZnkNDQogICAgICBk YXRhIGl0IHJlY2VpdmVzLCBhbmQgY2FuIGFsc28gYmUgdXNlZCBhcyBhbiBS RE1BIHRhZyBbUkRNQV0uDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAyMV0NDQogDA0N CiBpU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDANDQog DQ0KIA0NCiAzLjguICBTQ1NJIERhdGENDQogDQ0KICAgICAgVGhlIHR5cGlj YWwgZGF0YSB0cmFuc2ZlciBzcGVjaWZpZXMgdGhlIGxlbmd0aCBvZiB0aGUg ZGF0YSBwYXlsb2FkLA0NCiAgICAgIHRoZSBUcmFuc2ZlciBUYWcgcHJvdmlk ZWQgYnkgdGhlIHJlY2VpdmVyIGZvciB0aGlzIGRhdGEgdHJhbnNmZXIsDQ0K fCAgICAgYW5kIGEgYnVmZmVyIG9mZnNldC4gIFRoZSB0eXBpY2FsIFNDU0kg RGF0YSBwYWNrZXQgZm9yIFdSSVRFIChmcm9tDQ0KfCAgICAgaW5pdGlhdG9y IHRvIHRhcmdldCkgaGFzIHRoZSBmb2xsb3dpbmcgZm9ybWF0Og0NCiANDQog ICAgICBCeXRlIC8gICAgMCAgICAgICB8ICAgICAgIDEgICAgICAgfCAgICAg ICAyICAgICAgIHwgICAgICAgMyAgICAgICB8DQ0KICAgICAgICAgLyAgICAg ICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAg ICAgICAgICAgICAgfA0NCiAgICAgICAgfDcgNiA1IDQgMyAyIDEgMHw3IDYg NSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwN DQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgIDB8IE9w Y29kZSAoMHgwNSkgfCBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSsNDQogICAgICAgNHwgTGVuZ3RoICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKw0NCnwgICAgICA4fCBUcmFuc2ZlciBUYWcg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAg MTJ8IERhdGEgT2Zmc2V0ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNDQogICAgICAxNnwgSW5pdGlhdG9yIFRhc2sgVGFnICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDIwLyBSZXNlcnZl ZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIC8NDQogICAgICAgKy8gICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQ0K ICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDQwLyBQYXls b2FkICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIC8NDQogICAgICAgKy8gICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAv DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiANDQogICAgICBU aGUgdHlwaWNhbCBTQ1NJIERhdGEgcGFja2V0IGZvciBSRUFEIChmcm9tIHRh cmdldCB0byBpbml0aWF0b3IpDQ0KICAgICAgaGFzIHRoZSBmb2xsb3dpbmcg Zm9ybWF0Og0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAyMl0NDQogDA0NCiBpU0NT SSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDANDQogDQ0KIA0N CiANDQogICAgICBCeXRlIC8gICAgMCAgICAgICB8ICAgICAgIDEgICAgICAg fCAgICAgICAyICAgICAgIHwgICAgICAgMyAgICAgICB8DQ0KICAgICAgICAg LyAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAg ICB8ICAgICAgICAgICAgICAgfA0NCiAgICAgICAgfDcgNiA1IDQgMyAyIDEg MHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAy IDEgMHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KfCAgICAg IDB8IE9wY29kZSAoMHg4NSkgfCAoMCkgICAgIHxTfE98VXwgICAgICAgTWF4 Q21kUk4gIG9yICgwKSAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNDQogICAgICAgNHwgTGVuZ3RoICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCnwgICAgICA4fCBUcmFuc2Zl ciBUYWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0K ICAgICAgMTJ8IERhdGEgT2Zmc2V0ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSsNDQogICAgICAxNnwgSW5pdGlhdG9yIFRhc2sgVGFn ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDIwfCBS ZXNpZHVhbCBDb3VudCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rDQ0KfCAgICAgMjR8IENvbW1hbmQgU3RhdHVzfGlTQ1NJIFN0YXR1cyAg IHwgICAgIFN0YXRSTiAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAyOC8gUmVzZXJ2ZWQgKDAp ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAg ICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICA0MC8gUGF5bG9hZCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAg ICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogDQ0KIA0NCiAzLjguMS4g IExlbmd0aA0NCiANDQogICAgICBUaGUgbGVuZ3RoIGZpZWxkIHNwZWNpZmll cyB0aGUgdG90YWwgbnVtYmVyIG9mIGJ5dGVzIGluIHRoZSBmb2xsb3ctDQ0K ICAgICAgaW5nIHBheWxvYWQuDQ0KIA0NCiAzLjguMi4gIFRyYW5zZmVyIFRh Zw0NCiANDQogICAgICBUaGUgVHJhbnNmZXIgVGFnIGlkZW50aWZpZXMgdGhl IG9wZXJhdGlvbiB0byB3aGljaCB0aGlzIGRhdGENDQp8ICAgICB0cmFuc2Zl ciBiZWxvbmdzLiAgV2hlbiB0aGUgdHJhbnNmZXIgaXMgZnJvbSB0aGUgdGFy Z2V0IHRvIHRoZSBpbmktDQ0KfCAgICAgdGlhdG9yLCB0aGUgVHJhbnNmZXIg VGFnIGlzIHRoZSBJbml0aWF0b3IgVGFzayBUYWcgdGhhdCB3YXMgc2VudA0N CnwgICAgIHdpdGggdGhlIFNDU0kgY29tbWFuZC4gIFdoZW4gdGhlIHRyYW5z ZmVyIGlzIGZyb20gdGhlIGluaXRpYXRvciB0bw0NCnwgICAgIHRoZSB0YXJn ZXQsIHRoZSBUcmFuc2ZlciBUYWcgaXMgdGhlIFRhcmdldCBUcmFuc2ZlciBU YWcgd2hlbiBSVFQgaXMNDQp8ICAgICBlbmFibGVkLCBvciB0aGUgSW5pdGlh dG9yIFRhc2sgVGFnIHdoZW4gUlRUIGlzIGRpc2FibGVkLg0NCiANDQogMy44 LjMuICBCdWZmZXIgT2Zmc2V0DQ0KIA0NCiAgICAgIFRoZSBCdWZmZXIgT2Zm c2V0IGZpZWxkIGNvbnRhaW5zIHRoZSBvZmZzZXQgb2YgdGhlIGZvbGxvd2lu ZyBkYXRhDQ0KICAgICAgYWdhaW5zdCB0aGUgY29tcGxldGUgZGF0YSB0cmFu c2Zlci4gVGhlIHN1bSBvZiB0aGUgYnVmZmVyIG9mZnNldA0NCiAgICAgIGFu ZCBsZW5ndGggc2hvdWxkIG5vdCBleGNlZWQgdGhlIGV4cGVjdGVkIHRyYW5z ZmVyIGxlbmd0aCBmb3IgdGhlDQ0KICAgICAgY29tbWFuZC4NDQogDQ0KIA0N CiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDIzXQ0NCiAMDQ0K IGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0NCiAN DQogDQ0KIDMuOC40LiAgRmxhZ3MNDQogDQ0KICAgICAgVGhlIGxhc3QgU0NT SSBEYXRhIHBhY2tldCBzZW50IGZyb20gYSB0YXJnZXQgdG8gYW4gaW5pdGlh dG9yIGZvciBhDQ0KICAgICAgcGFydGljdWxhciBTQ1NJIGNvbW1hbmQgdGhh dCBjb21wbGV0ZWQgc3VjY2Vzc2Z1bGx5IG1heSBvcHRpb25hbGx5DQ0KICAg ICAgYWxzbyBjb250YWluIHRoZSBDb21tYW5kIFN0YXR1cyBmb3IgdGhlIGRh dGEgdHJhbnNmZXIuICBJbiB0aGlzDQ0KICAgICAgY2FzZSBTZW5zZSBEYXRh IGNhbm5vdCBiZSBzZW50IHRvZ2V0aGVyIHdpdGggdGhlIENvbW1hbmQgU3Rh dHVzLg0NCiAgICAgIElmIHRoZSBjb21tYW5kIGNvbXBsZXRlZCB3aXRoIGFu IGVycm9yLCB0aGVuIHRoZSByZXNwb25zZSBhbmQgc2Vuc2UNDQogICAgICBk YXRhIG11c3QgYmUgc2VudCBpbiBhIFNDU0kgUmVzcG9uc2UgcGFja2V0IGFu ZCBtdXN0IG5vdCBiZSBzZW50IGluDQ0KICAgICAgYSBTQ1NJIERhdGEgcGFj a2V0Lg0NCiAgICAgICAgICAgICAgICBCeXRlIDINDQogICAgICAgICAgIGIw LTEgYXMgaW4gYW4gb3JkaW5hcnkgU0NTSSBSZXNwb25zZQ0NCiAgICAgICAg ICAgYjIgICAoUykgc2V0IHRvIGluZGljYXRlIHRoYXQgdGhlIENvbW1hbmQg U3RhdHVzIGZpZWxkIGNvbi0NDQogICAgICAgICAgICAgICAgdGFpbnMgc3Rh dHVzLg0NCiAgICAgICAgICAgYjMtNyBub3QgdXNlZCAoc2hvdWxkIGJlIHNl dCB0byAwKS4NDQogICAgICBJZiB0aGUgKFMpIGJpdCBpcyBzZXQsIHRoZW4g dGhlcmUgaXMgbWVhbmluZyB0byB0aGUgZXh0cmEgZmllbGRzIGluDQ0KfCAg ICAgdGhlIFNDU0kgRGF0YSBwYWNrZXQgKE1heENtZFJOLCBDb21tYW5kIFN0 YXR1cywgUmVzaWR1YWwgQ291bnQsDQ0KfCAgICAgU3RhdFJOKQ0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogU2F0 cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgW1BhZ2UgMjRdDQ0KIAwNDQogaVNDU0kgICAgICAg ICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQogMy45LiAg VGV4dCBDb21tYW5kDQ0KIA0NCiAgICAgIFRoZSBUZXh0IENvbW1hbmQgaXMg cHJvdmlkZWQgdG8gYWxsb3cgdGhlIGV4Y2hhbmdlIG9mIGluZm9ybWF0aW9u DQ0KICAgICAgYW5kIGZvciBmdXR1cmUgZXh0ZW5zaW9ucy4gSXQgcGVybWl0 cyB0aGUgaW5pdGlhdG9yIHRvIGluZm9ybSBhDQ0KICAgICAgdGFyZ2V0IG9m IGl0cyBjYXBhYmlsaXRpZXMgb3IgdG8gcmVxdWVzdCBzb21lIHNwZWNpYWwg b3BlcmF0aW9ucy4NDQogDQ0KICAgICAgQnl0ZSAvICAgIDAgICAgICAgfCAg ICAgICAxICAgICAgIHwgICAgICAgMiAgICAgICB8ICAgICAgIDMgICAgICAg fA0NCiAgICAgICAgIC8gICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8 ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwNDQogICAgICAgIHw3 IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEg MHw3IDYgNSA0IDMgMiAxIDB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKw0NCiAgICAgICAwfCBPcGNvZGUgKDB4MDQpIHwgUmVzZXJ2ZWQgKDAp ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAg ICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgIDR8IExlbmd0aCAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAg ICAgOC8gUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICsvICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgLw0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQog ICAgICAxNnwgSW5pdGlhdG9yIFRhc2sgVGFnICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKw0NCiAgICAgIDIwLyBSZXNlcnZlZCAoMCkgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8N DQogICAgICAgKy8gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICArLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIDQwLyBUZXh0ICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IC8NDQogICAgICAgKy8gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICAr LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKw0NCiANDQogDQ0KIDMuOS4xLiAgTGVuZ3Ro DQ0KIA0NCiAgICAgIFRoZSBsZW5ndGgsIGluIGJ5dGVzLCBvZiB0aGUgVGV4 dCBmaWVsZC4NDQogDQ0KIDMuOS4yLiAgSW5pdGlhdG9yIFRhc2sgVGFnDQ0K IA0NCiAgICAgIFRoZSBpbml0aWF0b3IgYXNzaWduZWQgaWRlbnRpZmllciBm b3IgdGhpcyBUZXh0IENvbW1hbmQuDQ0KIA0NCiAzLjkuMy4gIFRleHQNDQog DQ0KICAgICAgVGhlIGluaXRpYXRvciBzZW5kcyB0aGUgdGFyZ2V0IGEgc2V0 IG9mIGtleTp2YWx1ZSBwYWlycyBpbiBVVEYtOA0NCnwgICAgIFVuaWNvZGUg Zm9ybWF0LiBUaGUga2V5IGFuZCB2YWx1ZSBhcmUgc2VwYXJhdGVkIGJ5IGEg JzonICgweDNBKQ0NCiAgICAgIGRlbGltaXRlci4gTWFueSBrZXk6dmFsdWUg cGFpcnMgY2FuIGJlIGluY2x1ZGVkIGluIHRoZSBUZXh0IGJsb2NrDQ0KfCAg ICAgYnkgc2VwYXJhdGluZyB0aGVtIHdpdGggbnVsbCAnICcgKDB4MDApIGRl bGltaXRlcnMuIFNvbWUgYmFzaWMNDQp8ICAgICBrZXk6dmFsdWUgcGFpcnMg YXJlIGRlc2NyaWJlZCBpbiBBcHBlbmRpeCBCLiAgVGhlIHRhcmdldCByZXNw b25kcw0NCnwgICAgIGJ5IHNlbmRpbmcgaXRzIHJlc3BvbnNlIGJhY2sgdG8g dGhlIGluaXRpYXRvci4gIFRoZSB0YXJnZXQgYW5kIGluaS0NDQp8ICAgICB0 aWF0b3IgY2FuIHRoZW4gcGVyZm9ybSBzb21lIGFkdmFuY2VkIG9wZXJhdGlv bnMgYmFzZWQgb24gdGhlaXINDQp8ICAgICBjb21tb24gY2FwYWJpbGl0aWVz LiAgTWFudWZhY3R1cmVycyBtYXkgaW50cm9kdWNlIG5ldyBrZXlzIGJ5IHBy ZS0NDQp8ICAgICBmaXhpbmcgdGhlbSB3aXRoIHRoZWlyIChyZXZlcnNlZCkg ZG9tYWluIG5hbWUsIGZvciBleGFtcGxlLA0NCiANDQogICAgICAgICAgICAg IGNvbS5mb28uYmFyLmRvX3NvbWV0aGluZzowMDAwMDAwMDAwMDAwMDAzDQ0K IA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRo ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDI1XQ0NCiAM DQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0N CiANDQogDQ0KICAgICAgQW55IGtleSB0aGF0IHRoZSB0YXJnZXQgZG9lcyBu b3QgdW5kZXJzdGFuZCBtYXkgYmUgaWdub3JlZCB3aXRob3V0DQ0KICAgICAg YWZmZWN0aW5nIGJhc2ljIGZ1bmN0aW9uLiBPbmNlIHRoZSB0YXJnZXQgaGFz IHByb2Nlc3NlZCBhbGwgdGhlDQ0KICAgICAga2V5OnZhbHVlIHBhaXJzLCBp dCByZXNwb25kcyB3aXRoIHRoZSBUZXh0IFJlc3BvbnNlIGNvbW1hbmQsIGxp c3QtDQ0KICAgICAgaW5nIHRoZSBwYXJhbWV0ZXJzIHRoYXQgaXQgc3VwcG9y dHMuIEl0IGlzIHJlY29tbWVuZGVkIHRoYXQgVGV4dA0NCiAgICAgIG9wZXJh dGlvbnMgdGhhdCB3aWxsIHRha2UgYSBsb25nIHRpbWUgc2hvdWxkIGJlIHBs YWNlZCBpbiB0aGVpciBvd24NDQp8ICAgICBUZXh0IGNvbW1hbmQuICBJZiB0 aGUgVGV4dCBSZXNwb25zZSBkb2VzIG5vdCBjb250YWluIGEga2V5IHRoYXQg d2FzDQ0KfCAgICAgcmVxdWVzdGVkLCB0aGUgaW5pdGlhdG9yIG11c3QgYXNz dW1lIHRoYXQgdGhlIGtleSB3YXMgbm90IHVuZGVyLQ0NCnwgICAgIHN0b29k IGJ5IHRoZSB0YXJnZXQuDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSAyNl0NDQogDA0N CiBpU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDANDQog DQ0KIA0NCiAzLjEwLiAgVGV4dCBSZXNwb25zZQ0NCiANDQogICAgICBUaGUg VGV4dCBSZXNwb25zZSBtZXNzYWdlIGNvbnRhaW5zIHRoZSByZXNwb25zZXMg b2YgdGhlIHRhcmdldCB0bw0NCiAgICAgIHRoZSBpbml0aWF0b3IncyBUZXh0 IENvbW1hbmQuIFRoZSBmb3JtYXQgb2YgdGhlIFRleHQgZmllbGQgbWF0Y2hl cw0NCiAgICAgIHRoYXQgb2YgdGhlIFRleHQgQ29tbWFuZC4NDQogDQ0KICAg ICAgQnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwgICAgICAg MiAgICAgICB8ICAgICAgIDMgICAgICAgfA0NCiAgICAgICAgIC8gICAgICAg ICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAg ICAgICAgICAgIHwNDQogICAgICAgIHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUg NCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8DQ0K ICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgICAwfCBPcGNv ZGUgKDB4ODQpIHwgUmVzZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQ0KICAgICAgIDR8IExlbmd0aCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAgOC8gUmVzZXJ2ZWQgKDApICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAxNnwgSW5pdGlhdG9yIFRh c2sgVGFnICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAg IDIwLyBSZXNlcnZlZCAoMCkgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIC8NDQogICAgICAgKy8gICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAvDQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAg ICAgIDQwLyBUZXh0IFJlc3BvbnNlICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIC8NDQogICAgICAgKy8gICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAvDQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0N CiANDQogDQ0KIDMuMTAuMS4gIExlbmd0aA0NCiANDQogICAgICBUaGUgbGVu Z3RoLCBpbiBieXRlcywgb2YgdGhlIFRleHQgUmVzcG9uc2UgZmllbGQuDQ0K IA0NCiAzLjEwLjIuICBJbml0aWF0b3IgVGFzayBUYWcNDQogDQ0KICAgICAg VGhlIEluaXRpYXRvciBUYXNrIFRhZyBtYXRjaGVzIHRoZSB0YWcgdXNlZCBp biB0aGUgaW5pdGlhbCBUZXh0DQ0KICAgICAgQ29tbWFuZCBhbmQgaXMgdXNl ZCBieSB0aGUgaW5pdGlhdG9yIHRvIHJlbGF0ZSB0aGUgVGV4dCBDb21tYW5k cw0NCiAgICAgIHdpdGggdGhlIGFwcHJvcHJpYXRlIFRleHQgUmVzcG9uc2Vz Lg0NCiANDQogMy4xMC4zLiAgVGV4dCBSZXNwb25zZQ0NCiANDQogICAgICBU aGUgVGV4dCBSZXNwb25zZSBmaWVsZCBjb250YWlucyByZXNwb25zZXMgaW4g dGhlIHNhbWUga2V5OnZhbHVlDQ0KICAgICAgZm9ybWF0IGFzIHRoZSBUZXh0 IENvbW1hbmQuIEFwcGVuZGl4IEIgbGlzdHMgc29tZSBiYXNpYyBUZXh0IENv bS0NDQp8ICAgICBtYW5kcyBhbmQgdGhlaXIgUmVzcG9uc2VzLiAgSWYgdGhl IFRleHQgUmVzcG9uc2UgZG9lcyBub3QgY29udGFpbiBhDQ0KfCAgICAga2V5 IHRoYXQgd2FzIHJlcXVlc3RlZCwgdGhlIGluaXRpYXRvciBtdXN0IGFzc3Vt ZSB0aGF0IHRoZSBrZXkgd2FzDQ0KfCAgICAgbm90IHVuZGVyc3Rvb2QgYnkg dGhlIHRhcmdldC4NDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIFNh dHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIFtQYWdlIDI3XQ0NCiAMDQ0KIGlTQ1NJICAgICAg ICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0NCiANDQogDQ0KIDMuMTEu ICBMb2dpbiBDb21tYW5kDQ0KIA0NCiAgICAgIEFmdGVyIGVzdGFibGlzaGlu ZyBhIFRDUCBjb25uZWN0aW9uIGJldHdlZW4gYW4gaW5pdGlhdG9yIGFuZCBh IHRhci0NDQogICAgICBnZXQsIHRoZSBpbml0aWF0b3Igc2hvdWxkIGlzc3Vl IGEgTG9naW4gQ29tbWFuZCB0byBnYWluIGZ1cnRoZXINDQogICAgICBhY2Nl c3MgdG8gdGhlIHRhcmdldCdzIHJlc291cmNlcy4NDQogDQ0KICAgICAgQnl0 ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAgICAgIHwgICAgICAgMiAgICAg ICB8ICAgICAgIDMgICAgICAgfA0NCiAgICAgICAgIC8gICAgICAgICAgICAg IHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfCAgICAgICAgICAg ICAgIHwNDQogICAgICAgIHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIg MSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8DQ0KICAgICAg ICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCnwgICAgICAwfCBPcGNvZGUgKDB4 MDMpIHwgUmVzZXJ2ZWQgKDApICB8ICAgQ21kUk4gb3IgUmVzZXJ2ZWQgKDAp ICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAg ICAgIDR8IExlbmd0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSsNDQp8ICAgICAgOHwgQ0lEICAgICAgICAgICAgICAgICAg ICAgICAgICAgfCAgIFJlY292ZXJDSUQgb3IgMCAgICAgICAgICAgICB8DQ0K fCAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCnwgICAgIDEyfCAgIFJl c2VydmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQ0KICAgICAgMTZ8IElTSUQgICAgICAgICAgICAgICAgICAgICAgICAgIHxU U0lEICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAyMC8gUmVzZXJ2ZWQgKDApICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICA0MC8gTG9naW4gUGFyYW1l dGVycyBpbiBUZXh0IENvbW1hbmQgRm9ybWF0ICAgICAgICAgICAgICAgICAg ICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAg ICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogDQ0KIA0NCiAzLjExLjEuDQ0K IA0NCiAgICAgIENJRCBhIHVuaXF1ZSBpZCBmb3IgdGhpcyBjb25uZWN0aW9u IHdpdGhpbiB0aGUgc2Vzc2lvbg0NCiANDQogMy4xMS4yLg0NCiANDQogICAg ICBGb3IgYSBjb25uZWN0aW9uIHVzZWQgdG8gcmVjb3ZlciBhIGxvc3QgVENQ IGNvbm5lY3Rpb24gKHNlZSBsYXRlcg0NCiAgICAgIHRoZSBJRCBvZiB0aGUg ZmFpbGVkIGNvbm5lY3Rpb24pIHRoZSBpbml0aWF0b3IgcHJvdmlkZXMgdGhl IENJRCBvZg0NCiAgICAgIHRoZSBmYWlsZWQgY29ubmVjdGlvbi4gIEEgc2lt cGxlIHRhcmdldCBtYXkgcmVqZWN0IHJlY292ZXJ5LiBJbg0NCiAgICAgIHRo aXMgY2FzZSB0aGUgaW5pdGlhdG9yIHdpbGwgdGVybWluYXRlIGFsbCBvdXRz dGFuZGluZyBjb21tYW5kcw0NCiAgICAgIHdpdGggYSBjaGVjayBjb25kaXRp b25zIGFuZCByZXNldCB0aGUgdGFyZ2V0Lg0NCiANDQogICAgICBUaGUgaW5p dGlhdG9yIG1heSBwcm92aWRlIHNvbWUgYmFzaWMgcGFyYW1ldGVycyBpbiBv cmRlciB0byBlbmFibGUNDQogICAgICB0aGUgdGFyZ2V0IHRvIGRldGVybWlu ZSBpZiB0aGUgaW5pdGlhdG9yIG1heSBpbiBmYWN0IHVzZSB0aGUNDQp8ICAg ICB0YXJnZXQncyByZXNvdXJjZXMuICBUaGUgZm9ybWF0IG9mIHRoZSBwYXJh bWV0ZXJzIGlzIGFzIHNwZWNpZmllZA0NCnwgICAgIGZvciB0aGUgVGV4dCBD b21tYW5kLiAgVGFyZ2V0cyBtYXkgcmVxdWlyZSBrZXlzIHRvIGluZGljYXRl IHRoZQ0NCnwgICAgIERvbWFpbiBOYW1lIG9mIHRoZSBpbml0aWF0b3IgYW5k IHRoZSB0YXJnZXQsIGFuZCBwZXJoYXBzIGFsc28gYW4NDQp8ICAgICBBdXRo ZW50aWNhdG9yIGtleS4gIFRoZSBpbml0aWF0b3IgbWF5IGFsc28gcHJvdmlk ZSBhZGRpdGlvbmFsDQ0KfCAgICAgcGFyYW1ldGVycyB0byB0aGUgdGFyZ2V0 IGluIFRleHQgQ29tbWFuZCBmb3JtYXQsIGlmIHRoZSBpbml0aWF0b3INDQp8 ICAgICBzbyBkZXNpcmVzLiAgS2V5cyBhbmQgdGhlaXIgZXhwbGFuYXRpb25z IGFyZSBsaXN0ZWQgaW4gQXBwZW5kaXggQi4NDQogDQ0KIA0NCiANDQogU2F0 cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgW1BhZ2UgMjhdDQ0KIAwNDQogaVNDU0kgICAgICAg ICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQogICAgICBX aGVuZXZlciBkZXNpcmVkIGFuIGluaXRpYXRvciB3aWxsIGlkZW50aWZ5IGl0 cyB2aWV3IG9mIHRoZSB0YXJnZXQNDQogICAgICBhcyBpbjoNDQogDQ0KfCAg ICAgICAgICAgICBUYXJnZXQ6PGRvbWFpbi1uYW1lPlsvbW9kaWZpZXJdWzpw b3J0XQ0NCiANDQogICAgICBpbXBseWluZyB0aGF0IHRoZSB0YXJnZXQgaXMg a25vd24gYXM6DQ0KIA0NCiAgICAgICAgICAgICAgc2NzaTovLzxkb21haW4t bmFtZT5bL21vZGlmaWVyXQ0NCiANDQp8ICAgICBhbmQgaXQgc2hvdWxkIGJl IGNvbm5lY3RlZCB0aHJvdWdoIHBvcnQgInBvcnQiICh0aGUgZGVmYXVsdCB3 ZWxsDQ0KfCAgICAga25vd24gcG9ydCBoYXMgYW4gSUFOQSBkZWZpbmVkIHZh bHVlIG9mIHh4KSBJbml0aWF0b3JzIGNhbiB1c2UgdGhlDQ0KfCAgICAgc2Ft ZSB0eXBlIG9mIG5hbWluZyBpbXBseWluZyBtYWNoaW5lIGFuZCBvcHRpb25h bCBhIHByaW5jaXBhbCAoZS5nLg0NCnwgICAgIG9wZXJhdGluZyBzeXN0ZW0g aW1hZ2UpIGFzIGluOg0NCiANDQp8ICAgICAgICAgICAgIEluaXRpYXRvcjo8 ZG9tYWluLW5hbWU+Wy9tb2RpZmllcl0NDQogDQ0KfCAgICAgaW1wbHlpbmcg dGhhdCB0aGUgaW5pdGlhdG9yIGlzIGtub3duIGFzOg0NCiANDQp8ICAgICAg ICAgICAgIGlTQ1NJOi8vPGRvbWFpbi1uYW1lPlsvbW9kaWZpZXJdDQ0KIA0N CiANDQogICAgICBUaHVzIHRoZSBwYXJhbWV0ZXJzIHBhc3NlZCBmb3IgYSBw bGFpbi10ZXh0IHBhc3N3b3JkIGF1dGhlbnRpY2F0aW9uDQ0KICAgICAgYXJl Og0NCiANDQogICAgICAgICAgICAgIEluaXRpYXRvcjo8ZG9tYWluLW5hbWU+ Wy9tb2RpZmllcl0NDQp8ICAgICAgICAgICAgIFRhcmdldDo8ZG9tYWluLW5h bWU+Wy9tb2RpZmllcl0NDQogICAgICAgICAgICAgIEF1dGhlbnRpY2F0b3I6 b3Blbi1zZXNhbWUNDQogDQ0KIA0NCiAgICAgIFRoZSBtb2RpZmllciBpU0NT SS1TWVMgaXMgcmVzZXJ2ZWQgZm9yIGFkbWluaXN0cmF0aXZlIGZ1bmN0aW9u cy4NDQogICAgICBJU0lEIGFuZCBUU0lEIGZvcm0gY29sbGVjdGl2ZWx5IHRo ZSBTU0lEIChzZXNzaW9uIGlkKS4gQSBUU0lEIG9mIDANDQogICAgICBpbmRp Y2F0ZXMgYSBsZWFkaW5nIGNvbm5lY3Rpb24uIE9ubHkgYSBsZWFkaW5nIGNv bm5lY3Rpb24gbG9naW4gY2FuDQ0KfCAgICAgY2Fycnkgc2Vzc2lvbiBzcGVj aWZpYyBwYXJhbWV0ZXJzLCBlLmcuIG1heC1jb25uZWN0aW9ucy1yZXF1ZXN0 ZWQsDQ0KfCAgICAgdGhlIG1heGltdW0gaW1tZWRpYXRlIGRhdGEgbGVuZ3Ro IHJlcXVlc3RlZCwgZXRjLi4gIENtZFJOIGlzIHNpZ25pLQ0NCnwgICAgIGZp Y2FudCBvbmx5IGlmIFRTSUQgaXMgMCBhbmQgaW5kaWNhdGVzIHRoZSBzdGFy dGluZyBDb21tYW5kIHJlZmVyLQ0NCnwgICAgIGVuY2UgbnVtYmVyIGZvciB0 aGlzIHNlc3Npb247IGl0IHNob3VsZCBiZSAwIGZvciBhbGwgb3RoZXINDQp8 ICAgICBJbnN0YW5jZXMuDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0aCwg U2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgW1BhZ2UgMjldDQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAgICAgICAg ICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQogMy4xMi4gIExvZ2luIFJlc3Bv bnNlDQ0KIA0NCiAgICAgIFRoZSB0YXJnZXQgcmVzcG9uZHMgdG8gdGhlIExv Z2luIENvbW1hbmQgd2l0aCBhIExvZ2luIFJlc3BvbnNlLiAgSXQNDQogICAg ICBpcyBzdWZmaWNpZW50IGZvciB0aGUgdGFyZ2V0IHRvIHJlc3BvbmQgd2l0 aCBhIFN0YXR1cyBpbmRpY2F0aW5nDQ0KfCAgICAgdGhhdCB0aGUgTG9naW4g aXMgYWNjZXB0ZWQuICBJbiBmYWN0LCB0aGUgdGFyZ2V0IG1heSBjb21wbGV0 ZWx5DQ0KfCAgICAgaWdub3JlIHRoZSBwYXJhbWV0ZXJzIHRoYXQgd2VyZSBz ZW50IHRvIGl0IGFuZCBtYXkgcHJvdmlkZSBzZXJ2aWNlDQ0KfCAgICAgdG8g YW55IGluaXRpYXRvciB0aGF0IGNvbm5lY3RzIHRvIGl0LiBUaGUgdGFyZ2V0 IG1heSBhbHNvIHJldHVybg0NCnwgICAgIHBhcmFtZXRlcnMgdXNpbmcgdGhl IGZvcm1hdCBvZiB0aGUgVGV4dCBSZXNwb25zZSBvcGNvZGUsIGlmIGl0IHNv DQ0KfCAgICAgZGVzaXJlcy4gIEluIHBhcnRpY3VsYXIsIHRoZSB0YXJnZXQg bWF5IHdhbnQgdG8gcHJvdmlkZSBpdHMgQXV0aGVuLQ0NCnwgICAgIHRpY2F0 b3Iga2V5LCBzbyB0aGF0IHRoZSBpbml0aWF0b3IgY2FuIGJlIHN1cmUgdGhh dCBpdCBpcyBpbiBmYWN0DQ0KfCAgICAgdGFsa2luZyB3aXRoIHRoZSBjb3Jy ZWN0IHRhcmdldC4gIFRoZSBpbml0aWF0b3IgY2FuIHJlcXVlc3QgdGhhdA0N CnwgICAgIHRoZSB0YXJnZXQgcHJvdmlkZSB0aGUgQXV0aGVudGljYXRvciBw YXJhbWV0ZXIgYnkgc3BlY2lmeWluZyB0aGUNDQp8ICAgICBTZW5kQXV0aGVu dGljYXRvcjp5ZXMga2V5OnZhbHVlIHBhaXIuDQ0KIA0NCiAgICAgIEJ5dGUg LyAgICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAgIDIgICAgICAg fCAgICAgICAzICAgICAgIHwNDQogICAgICAgICAvICAgICAgICAgICAgICB8 ICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAg ICB8DQ0KICAgICAgICB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEg MHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfA0NCiAgICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSsNDQp8ICAgICAgMHwgT3Bjb2RlICgweDgz KSB8IFJlc2VydmVkICgwKSAgfCAgICAgTWF4Q21kUk4gb3IgUmVzZXJ2ZWQg KDApICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAg ICA0fCBMZW5ndGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rDQ0KICAgICAgIDgvIFJlc2VydmVkICgwKSAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAg ICAgICArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIC8NDQogICAgICAgICstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rDQ0KICAgICAgMTZ8IElTSUQgICAgICAgICAgICAgICAg ICAgICAgICAgIHwgVFNJRCAgICAgICAgICAgICAgICAgICAgICAgICAgfA0N CiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAyMHwgUmVz ZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Kw0NCiAgICAgIDI0fCBTdGF0dXMgICAgICAgIHwgUmVzZXJ2ZWQgKDApICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAgICst LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgMjgvIFJlc2VydmVkICgwKSAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgLw0NCiAgICAgICArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NDQogICAgICAg ICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgNDAvIExvZ2luIFBhcmFt ZXRlcnMgaW4gVGV4dCBDb21tYW5kIEZvcm1hdCAgICAgICAgICAgICAgICAg ICAgICAgLw0NCiAgICAgICArLyAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIC8NDQogICAg ICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KIA0NCiAgICAgIFRoZSBmb3Jt YXQgb2YgdGhlIExvZ2luIFJlc3BvbnNlIGlzIHRoZSBzYW1lIGFzIHRoZSBU ZXh0IFJlc3BvbnNlLA0NCiAgICAgIHdpdGggdGhlIGFkZGl0aW9uIG9mIG9u ZSBmaWVsZC4NDQogDQ0KIDMuMTIuMS4gIFN0YXR1cw0NCiANDQogICAgICBU aGUgU3RhdHVzIHJldHVybmVkIGluIGEgTG9naW4gUmVzcG9uc2UgaXMgb25l IG9mIHRoZSBmb2xsb3dpbmc6DQ0KICAgICAgICAgICAwICAgIGFjY2VwdCBs b2dpbiAgICh3aWxsIG5vdyBhY2NlcHQgU0NTSSBjb21tYW5kcykNDQogICAg ICAgICAgIDEgICAgcmVqZWN0IGxvZ2luDQ0KICAgICAgICAgICAyIGFkZGl0 aW9uYWwgYXV0aGVudGljYXRpb24gcmVxdWlyZWQNDQogICAgICAgICAgIDMg ICByZWplY3QgcmVjb3ZlcnkNDQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0 aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgW1BhZ2UgMzBdDQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAgICAg ICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQogICAgICBJbiB0aGUgY2Fz ZSB0aGF0IHRoZSBTdGF0dXMgaXMgImFjY2VwdCBsb2dpbiIgdGhlIGluaXRp YXRvciBtYXkNDQogICAgICBwcm9jZWVkIHRvIGlzc3VlIFNDU0kgY29tbWFu ZHMuICBJbiB0aGUgY2FzZSB0aGF0IHRoZSBTdGF0dXMgaXMNDQogICAgICAi cmVqZWN0IGxvZ2luIiB0aGUgaW5pdGlhdG9yIHNob3VsZCBpbW1lZGlhdGVs eSBjbG9zZSBkb3duIGl0cyBlbmQNDQogICAgICBvZiB0aGUgVENQIGNvbm5l Y3Rpb24sIHRodXMgZnJlZWluZyB1cCB0aGUgdGFyZ2V0J3MgcG9ydCBmb3Ig c29tZQ0NCiAgICAgIG90aGVyIGNvbm5lY3Rpb24uIFRoZSB0YXJnZXQgYWxz byBoYXMgdGhlIG9wdGlvbiBvZiBpbW1lZGlhdGVseQ0NCnwgICAgIGNsb3Np bmcgZG93biBpdHMgZW5kIG9mIHRoZSBUQ1AgY29ubmVjdGlvbi4gIEluIHRo ZSBjYXNlIHRoYXQgdGhlDQ0KfCAgICAgU3RhdHVzIGlzICJhZGRpdGlvbmFs IGF1dGhlbnRpY2F0aW9uIHJlcXVpcmVkIiB0aGUgaW5pdGlhdG9yIG11c3QN DQp8ICAgICBwcm92aWRlIGFkZGl0aW9uYWwgYXV0aGVudGljYXRpb24gaW5m b3JtYXRpb24gYnkgaXNzdWluZyB0aGUgVGV4dA0NCnwgICAgIENvbW1hbmQg d2l0aCB0aGUgYXBwcm9wcmlhdGUga2V5OnZhbHVlIHBhaXJzLiAgKFRoaXMg bWF5IGJlDQ0KfCAgICAgcmVxdWlyZWQgaWYgdGhlIGF1dGhlbnRpY2F0aW9u IG1ldGhvZCBpcyBiYXNlZCBvbiBhDQ0KICAgICAgY2hhbGxlbmdlL3Jlc3Bv bnNlIGFsZ29yaXRobS4pICBVcG9uIHJlY2VpcHQgb2YgdGhlIG5lY2Vzc2Fy eQ0NCiAgICAgIGF1dGhlbnRpY2F0aW9uLCB0aGUgdGFyZ2V0IHdpbGwgaXNz dWUgYSBMb2dpbiBSZXNwb25zZSB3aXRoIHRoZQ0NCiAgICAgICJhY2NlcHQg bG9naW4iIFN0YXR1cy4gIFNDU0kgQ29tbWFuZHMgd2lsbCBub3QgYmUgYWNj ZXB0ZWQgdW50aWwNDQogICAgICB0aGUgdGFyZ2V0IHByb3ZpZGVzIGEgTG9n aW4gUmVzcG9uc2Ugd2l0aCB0aGUgImFjY2VwdCBsb2dpbiINDQp8ICAgICBT dGF0dXMuICBUaGUgVFNJRCBpcyBhbiBpbml0aWF0b3IgaWRlbnRpZnlpbmcg dGFnIHNldCBieSB0aGUgdGFyLQ0NCnwgICAgIGdldC4gIEEgMCBpbiB0aGUg cmV0dXJuZWQgVFNJRCBpbmRpY2F0ZXMgdGhhdCBlaXRoZXIgdGhlIHRhcmdl dA0NCnwgICAgIHN1cHBvcnRzIG9ubHkgYSBzaW5nbGUgY29ubmVjdGlvbiBv ciB0aGF0IHRoZSBJU0lEIGhhcyBhbHJlYWR5IGJlZW4NDQp8ICAgICB1c2Vk IGFzIGEgbGVhZGluZyBJU0lELiBJbiBib3RoIGNhc2VzIHRoZSB0YXJnZXQg aXMgcmVqZWN0aW5nIHRoZQ0NCnwgICAgIGxvZ2luLiAgTWF4Q21kUk4gaW5k aWNhdGVzIHRoZSBtYXhpbXVtIENtZFJOIHRoZSBpbml0aWF0b3Igc2hvdWxk DQ0KfCAgICAgc2VuZC4gIFdoZW4gcmVhY2hpbmcgdGhpcyBudW1iZXIgKGNv bnNpZGVyaW5nIG51bWJlciB3cmFwLWFyb3VuZCkNDQp8ICAgICB0aGUgaW5p dGlhdG9yIHNob3VsZCByZWZyYWluIGZyb20gc2VuZGluZyBmdXJ0aGVyIGNv bW1hbmRzIHVudGlsDQ0KfCAgICAgdGhlIGluaXRpYXRvciBkb2VzIHJlY2Vp dmUgYSBuZXcgTWF4Q21kUk4gdGhhdCBhZHZhbmNlZCBwYXN0IHRoZQ0NCnwg ICAgIG9sZCB2YWx1ZS4NDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIFNhdHJh biwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIFtQYWdlIDMxXQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAg ICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0NCiANDQogDQ0KIDMuMTMuICBQ aW5nIENvbW1hbmQNDQogDQ0KIA0NCiAgICAgIEJ5dGUgLyAgICAwICAgICAg IHwgICAgICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAgICAgICAzICAg ICAgIHwNDQogICAgICAgICAvICAgICAgICAgICAgICB8ICAgICAgICAgICAg ICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8DQ0KICAgICAg ICB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMg MiAxIDB8NyA2IDUgNCAzIDIgMSAwfA0NCiAgICAgICAgKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSsNDQp8ICAgICAgMHwgT3Bjb2RlICgweDApICB8IFJlc2VydmVk ICgwKSAgfCAgICAgTWF4U3RhdFJOICAgICAgICAgICAgICAgICB8DQ0KICAg ICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgICA0fCBMZW5ndGgg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0K ICAgICAgIDgvIFJlc2VydmVkICgwKSAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAgICArLyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIC8NDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r DQ0KICAgICAgMTZ8IEluaXRpYXRvciBUYXNrIFRhZyAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICAgKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAyMC8gUmVzZXJ2ZWQgKDApICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAgICAg Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICA0MC8gUGluZyBEYXRhIChv cHRpb25hbCkgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAg ICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogDQ0KICAgICAgVGhlIFBpbmcg Q29tbWFuZCBjYW4gYmUgdXNlZCB0byB2ZXJpZnkgdGhhdCBhIGNvbm5lY3Rp b24gaXMgc3RpbGwNDQp8ICAgICBhY3RpdmUuIEl0IG1heSBiZSB1c2VmdWwg aW4gdGhlIGNhc2Ugd2hlcmUgYW4gaW5pdGlhdG9yIGhhcyBiZWVuDQ0KfCAg ICAgd2FpdGluZyBhIGxvbmcgdGltZSBmb3IgdGhlIHJlc3BvbnNlIHRvIHNv bWUgY29tbWFuZCwgYW5kIHRoZSBpbmktDQ0KfCAgICAgdGlhdG9yIHN1c3Bl Y3RzIHRoYXQgdGhlcmUgaXMgc29tZSBwcm9ibGVtIHdpdGggdGhlIGNvbm5l Y3Rpb24uDQ0KICAgICAgV2hlbiBhIHRhcmdldCByZWNlaXZlcyB0aGUgUGlu ZyBDb21tYW5kLCBpdCBzaG91bGQgcmVzcG9uZCB3aXRoIGENDQogICAgICBQ aW5nIFJlc3BvbnNlLCBkdXBsaWNhdGluZyBhcyBtdWNoIG9mIHRoZSBkYXRh IGFzIHBvc3NpYmxlIHRoYXQgd2FzDQ0KfCAgICAgcHJvdmlkZWQgaW4gdGhl IFBpbmcgQ29tbWFuZCAoaWYgc3VjaCBkYXRhIHdhcyBwcmVzZW50KS4gIElm IHRoZQ0NCnwgICAgIGluaXRpYXRvciBkb2VzIG5vdCByZWNlaXZlIHRoZSBQ aW5nIFJlc3BvbnNlIHdpdGhpbiBzb21lIHBlcmlvZCBvZg0NCnwgICAgIHRp bWUgKGRldGVybWluZWQgYnkgdGhlIGluaXRpYXRvciksIG9yIGlmIHRoZSBk YXRhIHJldHVybmVkIGJ5IHRoZQ0NCnwgICAgIFBpbmcgUmVzcG9uc2UgaXMg ZGlmZmVyZW50IGZyb20gdGhlIGRhdGEgdGhhdCB3YXMgaW4gdGhlIFBpbmcg Q29tLQ0NCnwgICAgIG1hbmQsIHRoZSBpbml0aWF0b3IgbWF5IGNvbmNsdWRl IHRoYXQgdGhlcmUgaXMgYSBwcm9ibGVtIHdpdGggdGhlDQ0KfCAgICAgY29u bmVjdGlvbi4gVGhlIGluaXRpYXRvciB3aWxsIHRoZW4gY2xvc2UgdGhlIGNv bm5lY3Rpb24gYW5kIG1heQ0NCnwgICAgIHRyeSB0byBlc3RhYmxpc2ggYSBu ZXcgY29ubmVjdGlvbi4NDQogDQ0KfDMuMTMuMS4gIE1heFN0YXRSTiAtIHRo ZSBuZXh0IFN0YXRSTiBleHBlY3RlZA0NCiANDQp8My4xMy4yLiAgTGVuZ3Ro DQ0KIA0NCiAgICAgIFRoZSBsZW5ndGggb2YgdGhlIG9wdGlvbmFsIFBpbmcg RGF0YS4NDQogDQ0KfDMuMTMuMy4gIEluaXRpYXRvciBUYXNrIFRhZw0NCiAN DQogICAgICBBbiBpbml0aWF0b3IgYXNzaWduZWQgaWRlbnRpZmllciBmb3Ig dGhlIG9wZXJhdGlvbi4NDQogDQ0KIA0NCiANDQogDQ0KIA0NCiBTYXRyYW4s IFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0aCAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBbUGFnZSAzMl0NDQogDA0NCiBpU0NTSSAgICAgICAgICAg ICAgICAgICAgICAgICBKdW5lIDIwMDANDQogDQ0KIA0NCnwzLjEzLjQuICBQ aW5nIERhdGENDQogDQ0KICAgICAgQmluYXJ5IGRhdGEgdGhhdCB3aWxsIGJl IHJlZmxlY3RlZCBpbiB0aGUgUGluZyBSZXNwb25zZS4NDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMzNdDQ0KIAwNDQogaVNDU0kg ICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQog My4xNC4gIFBpbmcgUmVzcG9uc2UNDQogDQ0KIA0NCiAgICAgIEJ5dGUgLyAg ICAwICAgICAgIHwgICAgICAgMSAgICAgICB8ICAgICAgIDIgICAgICAgfCAg ICAgICAzICAgICAgIHwNDQogICAgICAgICAvICAgICAgICAgICAgICB8ICAg ICAgICAgICAgICAgfCAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8 DQ0KICAgICAgICB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3 IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAzIDIgMSAwfA0NCiAgICAgICAgKy0t LS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSsNDQp8ICAgICAgMHwgT3Bjb2RlICgweDgwKSB8 IFJlc2VydmVkICgwKSAgfCAgIE1heENtZFJOICAgICAgICAgICAgICAgICAg ICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgICA0 fCBMZW5ndGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0t LS0tLS0rDQ0KICAgICAgIDgvIFJlc2VydmVkICgwKSAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0NCiAgICAg ICArLyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIC8NDQogICAgICAgICstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rDQ0KICAgICAgMTZ8IEluaXRpYXRvciBUYXNrIFRhZyAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0NCiAg ICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAyMC8gUmVzZXJ2 ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLw0N CiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICA0MC8gUmV0 dXJuIFBpbmcgRGF0YSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg Lw0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogDQ0KICAgICAg V2hlbiBhIHRhcmdldCByZWNlaXZlcyB0aGUgUGluZyBDb21tYW5kLCBpdCBz aG91bGQgcmVzcG9uZCB3aXRoIGENDQogICAgICBQaW5nIFJlc3BvbnNlLCBk dXBsaWNhdGluZyB0aGUgZGF0YSBhbmQgSW5pdGlhdG9yIFRhc2sgVGFnIHRo YXQgd2FzDQ0KICAgICAgcHJvdmlkZWQgaW4gdGhlIFBpbmcgQ29tbWFuZCwg aWYgcHJlc2VudC4NDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVu dHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQ YWdlIDM0XQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAg IEp1bmUgMjAwMA0NCiANDQogDQ0KfDMuMTUuICBUaGlyZCBQYXJ0eSBDb21t YW5kcyAtSU5DT01QTEVURSENDQogDQ0KICAgICAgVGhlcmUgYXJlIHNvbWUg dGhpcmQtcGFydHkgU0NTSSBjb21tYW5kcywgc3VjaCBhcyBDT1BZIGFuZCBF WFRFTkRFRA0NCiAgICAgIENPUFksIHRoYXQgcmVxdWlyZSBvbmUgdGFyZ2V0 IChUYXJnZXQgQSkgdG8gYWN0IGFzIGFuIGluaXRpYXRvciB0bw0NCiAgICAg IG90aGVyIHRhcmdldHMgKGUuZy4sIFRhcmdldCBCKS4gIFNvbWUgc3VjaCBj b21tYW5kcyBjYW4gYmUgZXh0ZW5kZWQNDQogICAgICBpbiBhIHN0cmFpZ2h0 Zm9yd2FyZCB3YXkgdG8gYWNjb21tb2RhdGUgbmV3IGZvcm1zIG9mIGFkZHJl c3NpbmcsDQ0KICAgICAgYW5kIHRoaXMgc2hvdWxkIGJlIGRvbmUgdG8gYWRk cmVzcyB0YXJnZXRzIHVzaW5nIGlTQ1NJLiBUaGVzZQ0NCiAgICAgIGV4dGVu c2lvbnMgYXJlIG5vdCBzdHJhaWdodGZvcndhcmQgZm9yIGFsbCBjb21tYW5k cywgYW5kIHRoZXkgbWF5DQ0KICAgICAgbm90IGJlIGFibGUgdG8gZW5jb21w YXNzIHRoZSBmdWxsIG5hbWUgc3BhY2UgYW5kIGF1dGhlbnRpY2F0aW9uDQ0K ICAgICAgaW5mb3JtYXRpb24gbmVlZGVkIGZvciBpU0NTSSBpbiBzb21lIGNv bnRleHRzLiBUaHVzIGlTQ1NJIGFsc28gcHJvLQ0NCiAgICAgIHZpZGVzIGEg ZmFjaWxpdHkgZm9yIGFzc2lnbmluZyBsb2NhbCBzaG9ydC1mb3JtIGFsaWFz ZXMgdG8gZnVsbA0NCiAgICAgIGFkZHJlc3NpbmcvYXV0aG9yaXphdGlvbiBp bmZvcm1hdGlvbiBmb3IgdGFyZ2V0cywgYW5kIHRoZSBhbGlhc2VzDQ0KICAg ICAgY2FuIGJlIHVzZWQgaW4gdGhlIFNDU0kgY29tbWFuZHMgYW5kIHBhcmFt ZXRlciBkYXRhLiAgVGhlIGFsaWFzDQ0KICAgICAgaW5mb3JtYXRpb24gaXMg c3BlY2lmaWVkIGFzIFRleHQgZm9sbG93aW5nIHRoZSBoZWFkZXIgb2YgdGhl IFNDU0kNDQogICAgICBjb21tYW5kIHNwZWNpZnlpbmcgdGhlIHRoaXJkLXBh cnR5IGNvbW1hbmQuICBUaGUgaGVhZGVyIHdpbGwgdGh1cw0NCiAgICAgIGFw cGVhciBhcyBmb2xsb3dzOg0NCiANDQogICAgICBCeXRlIC8gICAgMCAgICAg ICB8ICAgICAgIDEgICAgICAgfCAgICAgICAyICAgICAgIHwgICAgICAgMyAg ICAgICB8DQ0KICAgICAgICAgLyAgICAgICAgICAgICAgfCAgICAgICAgICAg ICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgfA0NCiAgICAg ICAgfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0IDMgMiAxIDB8NyA2IDUgNCAz IDIgMSAwfDcgNiA1IDQgMyAyIDEgMHwNDQogICAgICAgICstLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0rDQ0KfCAgICAgIDB8IE9wY29kZSAoMHgwMSkgfDF8MHxBfFJz dnxBVFRSIHwgICAgICAgICAgIENtZFJOICAgICAgICAgICAgICAgfA0NCiAg ICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAgNHwgTGVuZ3Ro ICghPSAwKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0N CiAgICAgICA4fCBMb2dpY2FsIFVuaXQgTnVtYmVyIChMVU4pICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAgICsgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICArDQ0KICAgICAgMTJ8ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg fA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAxNnwg SW5pdGlhdG9yIFRhc2sgVGFnICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0t LS0tKw0NCiAgICAgIDIwfCBFeHBlY3RlZCBEYXRhIFRyYW5zZmVyIExlbmd0 aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAg ICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KICAgICAgMjR8IFNDU0kgQ29tbWFu ZCBEZXNjcmlwdG9yIEJsb2NrIChDREIpICAgICAgICAgICAgICAgICAgICAg ICAgICAgfA0NCiAgICAgICAgKyAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICsNDQogICAg ICAyOHwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB8DQ0KICAgICAgICArICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgKw0NCiAgICAgIDMyfCAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQog ICAgICAgICsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICArDQ0KICAgICAgMzZ8ICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgfA0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsN DQogICAgICA0MC8gRXh0ZW5kZWQgQ0RCIGlmIGFueSAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICsvIFBh cmFtZXRlcnMgbmVlZGVkIGZvciBUYXJnZXQgQiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgLw0NCiAgICAgICAgLyAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IC8NDQogICAgICAgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0t Ky0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rDQ0KIA0NCiAgICAg IFRoZSBMZW5ndGggZmllbGQgd2lsbCBub3QgYmUgemVyby4gUmF0aGVyLCBp dCB3aWxsIGNvbnRhaW4gdGhlDQ0KICAgICAgbGVuZ3RoIG9mIGV4dGVuZCBD REIgaWYgYW55IGFuIHRoZSBsZW5ndGggb2YgdGhlIGFsaWFzIGluZm9ybWF0 aW9uDQ0KIA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lz LCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDM1 XQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUg MjAwMA0NCiANDQogDQ0KICAgICAgd2hpY2ggbWF5IGluY2x1ZGUgdGhlIG5h bWUgb2YgVGFyZ2V0IEIgYW5kIGFuIEF1dGhlbnRpY2F0aW9uIGtleSBpbg0N CiAgICAgIFRleHQgQ29tbWFuZCBmb3JtYXQuIEFuIGV4YW1wbGUgb2YgdGhl IGRhdGEgZm9yIHRoaXMgY29tbWFuZCBtaWdodA0NCiAgICAgIGJlOg0NCiAN DQogICAgICAgICAgICAgIExvY2FsTmFtZTpUYXJnZXRCDQ0KfCAgICAgICAg ICAgICBGdWxsTmFtZTpzai5mb28uY29tL2NvbnRyb2xsZXIxDQ0KICAgICAg ICAgICAgICBPcmlnaW5hbEF1dGhlbnRpY2F0b3I6b3Blbi1zZXNhbWUNDQog DQ0KIA0NCiAgICAgIFVwb24gcmVjZWl2aW5nIGEgdGhpcmQtcGFydHkgY29t bWFuZCwgVGFyZ2V0IEEgd2lsbCBwZXJmb3JtIGxvZ2luDQ0KICAgICAgb3Bl cmF0aW9ucyB3aXRoIHRoZSBpZGVudGlmaWVkIHRhcmdldHMuICBJbiBlZmZl Y3QsIFRhcmdldCBBIHdpbGwNDQogICAgICBiZWNvbWUgYW4gaW5pdGlhdG9y IHRvIFRhcmdldCBCLiAgQW1vbmcgdGhlIHBhcmFtZXRlcnMgcHJvdmlkZWQg dG8NDQogICAgICBUYXJnZXQgQiwgVGFyZ2V0IEEgbWF5IHNwZWNpZnkgdGhl IGF1dGhlbnRpY2F0aW9uIGluZm9ybWF0aW9uIGZyb20NDQogICAgICB0aGUg aW5pdGlhdG9yLiAgVGhlIFRleHQgcHJvdmlkZWQgYnkgVGFyZ2V0IEEgd2hl biBpdCBwZXJmb3JtcyB0aGUNDQogICAgICBMb2dpbiBjb21tYW5kIHRvIFRh cmdldCBCIG1heSBjb250YWluIHRoZSBrZXlzIFRhcmdldCAocmVmZXJyaW5n IHRvDQ0KICAgICAgVGFyZ2V0IEIpIGFuZCBJbml0aWF0b3IgKHJlZmVycmlu ZyB0byBUYXJnZXQgQSksIGFuZCBpdCBtYXkgYWxzbw0NCiAgICAgIGNvbnRh aW4gdGhlIGtleXMgQXV0aGVudGljYXRvciAob2YgVGFyZ2V0IEEpLCBPcmln aW5hbEluaXRpYXRvciBhbmQNDQogICAgICBPcmlnaW5hbEF1dGhlbnRpY2F0 b3IgKHJlZmVycmluZyB0byBhdXRoZW50aWNhdG9yIG9mIHRoZSBvcmlnaW5h bA0NCiAgICAgIGluaXRpYXRvcikuDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMs IE1ldGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2UgMzZd DQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVuZSAy MDAwDQ0KIA0NCiANDQogMy4xNi4gIE9wY29kZSBOb3QgVW5kZXJzdG9vZA0N CiANDQogDQ0KICAgICAgQnl0ZSAvICAgIDAgICAgICAgfCAgICAgICAxICAg ICAgIHwgICAgICAgMiAgICAgICB8ICAgICAgIDMgICAgICAgfA0NCiAgICAg ICAgIC8gICAgICAgICAgICAgIHwgICAgICAgICAgICAgICB8ICAgICAgICAg ICAgICAgfCAgICAgICAgICAgICAgIHwNDQogICAgICAgIHw3IDYgNSA0IDMg MiAxIDB8NyA2IDUgNCAzIDIgMSAwfDcgNiA1IDQgMyAyIDEgMHw3IDYgNSA0 IDMgMiAxIDB8DQ0KICAgICAgICArLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKw0NCiAg ICAgICAwfCBPcGNvZGUgKDB4ODgpIHwgUmVzZXJ2ZWQgKDApICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICAgICstLS0tLS0t LS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSstLS0t LS0tLS0tLS0tLS0rDQ0KICAgICAgIDR8IExlbmd0aCAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0N CiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0rLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICAgOC8gUmVz ZXJ2ZWQgKDApICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg Lw0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICA0MC8g SGVhZGVyIG9mIEJhZCBNZXNzYWdlICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAvDQ0KICAgICAgICsvICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgLw0NCiAgICAgICAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0rLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLSsNDQogICAgICA4 MA0NCiANDQogICAgICBJdCBtYXkgaGFwcGVuIHRoYXQgYSB0YXJnZXQgcmVj ZWl2ZXMgYSBtZXNzYWdlIHdpdGggYW4gT3Bjb2RlIHRoYXQNDQogICAgICBp dCBkb2Vzbid0IHJlY29nbml6ZS4gVGhpcyBtYXkgb2NjdXIgYmVjYXVzZSBv ZiBhIG5ldyB2ZXJzaW9uIG9mDQ0KICAgICAgdGhlIHByb3RvY29sIHRoYXQg ZGVmaW5lcyBhIG5ldyBPcGNvZGUsIG9yIGJlY2F1c2Ugb2Ygc29tZSBjb3Jy dXAtDQ0KfCAgICAgdGlvbiBvZiBhIG1lc3NhZ2UgaGVhZGVyLiAgVGhlIHRh cmdldCByZXR1cm5zIHRoZSBoZWFkZXIgb2YgdGhlDQ0KfCAgICAgbWVzc2Fn ZSB3aXRoIHRoZSB1bnJlY29nbml6ZWQgb3Bjb2RlIGFzIHRoZSBkYXRhIG9m IHRoZSByZXNwb25zZS4NDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogU2F0cmFuLCBT bWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgW1BhZ2UgMzddDQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAg ICAgICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQp8NC4gIGlTQ1NJIEVy cm9yIEhhbmRsaW5nDQ0KIA0NCnw0LjEuICBDb21tdW5pY2F0aW9ucyBFcnJv cnMNDQogDQ0KfCAgICAgRm9yIGFueSBvdXRzdGFuZGluZyBTQ1NJIGNvbW1h bmQgaXQgaXMgYXNzdW1lZCB0aGF0IGlTQ1NJIGluIGNvbi0NDQp8ICAgICBq dW5jdGlvbiB3aXRoIFNDU0kgYXQgdGhlIGluaXRpYXRvciBpcyBhYmxlIHRv IGtlZXAgZW5vdWdoIGluZm9ybWEtDQ0KfCAgICAgdGlvbiB0byBiZSBhYmxl IHRvIHJlYnVpbGQgdGhlIGNvbW1hbmQgUERVLCB0aGF0IG91dGdvaW5nIGRh dGEgaXMNDQp8ICAgICBhdmFpbGFibGUgKGluIGhvc3QgbWVtb3J5KSBmb3Ig cmV0cmFuc21pc3Npb24gd2hpbGUgdGhlIGNvbW1hbmQgaXMNDQp8ICAgICBv dXRzdGFuZGluZyBhbmQgdGhhdCBhdCB0YXJnZXQgaVNDU0kgYW5kIHNwZWNp YWxpemVkIFRDUCBpbXBsZW1lbi0NDQp8ICAgICB0YXRpb25zIGFyZSBhYmxl IHRvIHJlY292ZXIgdW5hY2tub3dsZWRnZWQgZGF0YSBwYWNrZXRzIGZyb20g YQ0NCnwgICAgIGNsb3NpbmcgY29ubmVjdGlvbiBvciwgYWx0ZXJuYXRpdmVs eSB0aGUgdGFyZ2V0IGhhcyBtZWFucyB0byByZS0NDQp8ICAgICByZWFkIGRh dGEgZnJvbSBhIGRldmljZS1zZXJ2ZXIuICBJdCBpcyBmdXJ0aGVyIGFzc3Vt ZWQgdGhhdCBhIHRhci0NDQp8ICAgICBnZXQgd2lsbCBrZWVwIGEgdGhlICJz dGF0dXMgJiBzZW5zZSIgZm9yIGNvbW1hbmRzIGl0IGhhcyBleGVjdXRlZA0N CnwgICAgIHdoaWxlIHRoZSB0b3RhbCBudW1iZXIgb2Ygb3V0c3RhbmRpbmcg Y29tbWFuZHMgYW5kIGV4ZWN1dGVkIGNvbS0NDQp8ICAgICBtYW5kcyBEb2Vz IG5vdCBleGNlZWQgaXQncyBsaW1pdC4gQSB0YXJnZXQgd2lsbCBzZXF1ZW50 aWFsbHkgbnVtYmVyDQ0KfCAgICAgdGhlIGRlbGl2ZXJlZCByZXNwb25zZXMg YW5kIHRodXMgZW5hYmxlIGluaXRpYXRvcnMgdG8gdGVsbCB3aGVuIGENDQp8 ICAgICByZXNwb25zZSBpcyBtaXNzaW5nIGFuZCB3aGF0IHJlc3BvbnNlIHRo ZXkgbWlzcy4NDQogDQ0KfCAgICAgVW5kZXIgdGhvc2UgY29uZGl0aW9ucyBp U0NTSSB3aWxsIGJlIGFibGUgdG8ga2VlcCBhIHNlc3Npb24gaW4NDQp8ICAg ICBvcGVyYXRpb24gcHJvdmlkZWQgdGhhdCBpdCBoYXMgYXQgaXMgYWJsZSB0 byBrZWVwL2VzdGFibGlzaCBhdA0NCnwgICAgIGxlYXN0IG9uZSBUQ1AgY29u bmVjdGlvbiBiZXR3ZWVuIHRoZSBpbml0aWF0b3IgYW5kIHRhcmdldCBpbiBh DQ0KfCAgICAgdGltZWx5IGZhc2hpb24uICBVbmZvcnR1bmF0ZWx5IHRoZSBt YXhpbXVtIGFkbWlzc2libGUgcmVjb3ZlcnkgdGltZQ0NCnwgICAgIGlzIGEg ZnVuY3Rpb24gb2YgdGhlIHRhcmdldCBhbmQgZm9yIHNvbWUgZGV2aWNlcyBh bmQgY29tbXVuaWNhdGlvbnMNDQp8ICAgICBuZXR3b3JrcyByZWNvdmVyeSBt YXkgYmUgY29tcGxleCBhbmQgbWF5IHBlcmNvbGF0ZSB0byB1cHBlcg0NCnwg ICAgIHNvZnR3YXJlIGxheWVycy4gIEl0IGlzIGFzc3VtZWQgdGhhdCB0YXJn ZXRzIGFuZC9vciBpbml0aWF0b3JzIHdpbGwNDQp8ICAgICByZWNvZ25pemUg YSBmYWlsaW5nIGNvbm5lY3Rpb24gYnkgZWl0aGVyIHRyYW5zcG9ydCBsZXZl bCBtZWFucw0NCnwgICAgIChUQ1ApIG9yIGJ5IGEgZ2FwIGluIHRoZSBjb21t YW5kIHN0cmVhbSB0aGF0IGRvZXMgbm90IGdldCBmaWxsZWQNDQp8ICAgICBm b3IgYSBsb25nIHRpbWUsIG9yIGJ5IGZhaWxpbmcgaVNDU0kgcGluZyAodGhl IGxhdGVyIHNob3VsZCBiZSB1c2VkDQ0KfCAgICAgYnkgcGVyaW9kaWNhbGx5 IGJ5IGhpZ2hseSByZWxpYWJsZSBpbXBsZW1lbnRhdGlvbnMpLiAgVGhlIHJl Y292ZXJ5DQ0KfCAgICAgaW52b2x2ZXMgdGhlIGZvbGxvd2luZyBzdGVwczoN DQp8ICAgICAgICAtYWJvcnQgb2ZmZW5kaW5nIFRDUCBjb25uZWN0aW9uKHMp ICh0YXJnZXQgJiBpbml0aWF0b3IpIGFuZA0NCnwgICAgICAgICAgcmVjb3Zl ciBhdCB0YXJnZXQgYWxsIHVuYWNrbm93bGVkZ2VkIHJlYWQtZGF0YSAudGkg LTIgLWNyZWF0ZQ0NCnwgICAgICAgICAgb25lIG9yIG1vcmUgbmV3IFRDUCBj b25uZWN0aW9ucyAod2l0aGluIHRoZSBzYW1lIHNlc3Npb24pIGFuZA0NCnwg ICAgICAgICAgYXNzb2NpYXRlIGFsbCBvdXRzdGFuZGluZyBjb21tYW5kcyBm cm9tIHRoZSBmYWlsZWQNDQp8ICAgICAgICAgIGNvbm5lY3Rpb24ocykgdG8g dGhlIEZJUlNUIG5ldyBjb25uZWN0aW9uIGF0IGJvdGggaW5pdGlhdG9yDQ0K fCAgICAgICAgICBhbmQgdGFyZ2V0DQ0KfCAgICAgICAgLXRoZSBpbml0aWF0 b3Igd2lsbCByZWlzc3VlIGFsbCBvdXRzdGFuZGluZyBjb21tYW5kcyB3aXRo IHRoZWlyDQ0KfCAgICAgICAgICBvcmlnaW5hbCBDbWRSTiBhbmQgVGFza0lE DQ0KfCAgICAgICAgLXVwb24gcmVjZWl2aW5nIHRoZSBuZXcvcmVzdGFydGlu ZyBjb21tYW5kcyB0aGUgdGFyZ2V0IHdpbGwNDQp8ICAgICAgICAgIHJlc3Vt ZSBjb21tYW5kIGV4ZWN1dGlvbjsgZm9yIHdyaXRlIGNvbW1hbmRzIGl0IG1l YW5zIHJlcXVlc3QtDQ0KfCAgICAgICAgICBpbmcgZGF0YSByZXRyYW5zbWlz c2lvbiB0aHJvdWdoIFJUVCwgZm9yIHJlYWRzIHJldHJhbnNtaXR0aW5nDQ0K fCAgICAgICAgICByZWNvdmVyZWQgZGF0YSBhbmQgZm9yICJ0ZXJtaW5hdGVk IiBjb21tYW5kcyByZXRyYW5zbWl0dGluZw0NCnwgICAgICAgICAgdGhlIHN0 YXR1cyAmIHNlbnNlIHdoaWxlIHJldGFpbmluZyB0aGUgb3JpZ2luYWwgU3Rh dFJOLiBJZg0NCnwgICAgICAgICAgZGF0YSByZWNvdmVyeSBpcyBub3QgcG9z c2libGUgdGhlIHRhcmdldCB3aWxsIGVpdGhlciBwcm92aWRlDQ0KfCAgICAg ICAgICBkYXRhIGZyb20gdGhlIG1lZGlhIG9yIHJlZG8gdGhlIG9wZXJhdGlv biAoaWYgdGhlIG9wZXJhdGlvbiBpcw0NCnwgICAgICAgICAgbm90IGlkZW0t cG90ZW50IHRoZSBkZXZpY2Ugc2VydmVyIG1heSBmYWlsIHRoZSBvcGVyYXRp b24pLg0NCiANDQogDQ0KIA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNh cHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IFtQYWdlIDM4XQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAg ICAgIEp1bmUgMjAwMA0NCiANDQogDQ0KfDQuMi4gIFByb3RvY29sIEVycm9y cw0NCiANDQp8ICAgICBUaGUgYXV0aG9ycyByZWNvZ25pemUgdGhhdCBtYXBw aW5nIGZyYW1lZCBtZXNzYWdlcyBvdmVyIGEgInN0cmVhbSINDQp8ICAgICBj b25uZWN0aW9uIChsaWtlIFRDUCkgbWFrZXMgdGhlIHByb3Bvc2VkIG1lY2hh bmlzbXMgdnVsbmVyYWJsZSB0bw0NCnwgICAgIHNpbXBsZSBzb2Z0d2FyZSBm cmFtaW5nIGVycm9ycyBhbmQgaW50cm9kdWNpbmcgZnJhbWluZyBtZWNoYW5p c21zDQ0KfCAgICAgbWF5IGJlIG9uZXJvdXMgZm9yIHBlcmZvcm1hbmNlIGFu ZCBiYW5kd2lkdGguICBDb21tYW5kIHJlZmVyZW5jZQ0NCnwgICAgIG51bWJl cnMgYW5kIHRoZSBhYm92ZSBtZWNoYW5pc21zIGZvciBjb25uZWN0aW9uIGRy b3AgYW5kIHJlZXN0YWItDQ0KfCAgICAgbGlzaG1lbnQgd2lsbCBoZWxwIGhh bmRsZSB0aGlzIHR5cGUgb2YgbWFwcGluZyBlcnJvcnMuDQ0KIA0NCnw0LjMu ICBTZXNzaW9uIEVycm9ycw0NCiANDQp8ICAgICBJZiBhbGwgdGhlIGNvbm5l Y3Rpb25zIG9mIGEgc2Vzc2lvbiBmYWlsIGFuZCBjYW4ndCBiZSByZWVzdGFi bGlzaGVkDQ0KfCAgICAgaW4gYSBzaG9ydCB0aW1lIG9yIGlmIGluaXRpYXRv cnMgZGV0ZWN0IHByb3RvY29sIGVycm9ycyByZXBlYXRlZGx5DQ0KfCAgICAg YW4gaW5pdGlhdG9yIG1heSBjaG9vc2UgdG8gdGVybWluYXRlIGEgc2Vzc2lv biBhbiBlc3RhYmxpc2ggYSBuZXcNDQp8ICAgICBzZXNzaW9uIChpbmRpY2F0 aW5nIG9sZCBzZXNzaW9uIHRlcm1pbmF0aW9uPykuIEl0IHdpbGwgdGVybWlu YXRlDQ0KfCAgICAgYWxsIG91dHN0YW5kaW5nIHJlcXVlc3Qgd2l0aCBhbiBp U0NTSSBlcnJvciBpbmRpY2F0aW9uIGJlZm9yZSBpbmktDQ0KfCAgICAgdGlh dGluZyBhIG5ldyBzZXNzaW9uLiAgQSB0YXJnZXQgdGhhdCBkZXRlY3RzIG9u ZSBvZiB0aGUgYWJvdmUNDQp8ICAgICBlcnJvcnMgd2lsbCB0YWtlIHRoZSBm b2xsb3dpbmcgYWN0aW9uczoNDQp8ICAgICAgICAtIFJlc2V0IHRoZSBUQ1Ag Y29ubmVjdGlvbnMgKGNsb3NlIHRoZSBzZXNzaW9uKS4NDQogICAgICAgICAt IEFib3J0IGFsbCBUYXNrcyBpbiB0aGUgdGFzayBzZXQgZm9yIHRoZSBjb3Jy ZXNwb25kaW5nIGluaXRpYS0NDQogICAgICAgICAgIHRvci4NDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0aCwgU2FwdW50 emFraXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1Bh Z2UgMzldDQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAg SnVuZSAyMDAwDQ0KIA0NCiANDQogNS4gIE5vdGVzIHRvIEltcGxlbWVudGVy cw0NCiANDQp8ICAgICBUaGlzIHNlY3Rpb24gbm90ZXMgc29tZSBvZiB0aGUg cGVyZm9ybWFuY2UgYW5kIHJlbGlhYmlsaXR5IGNvbi0NDQp8ICAgICBzaWRl cmF0aW9ucyBvZiB0aGUgaVNDU0kgcHJvdG9jb2wuICBUaGlzIHByb3RvY29s IHdhcyBkZXNpZ25lZCB0bw0NCnwgICAgIHRha2UgYWR2YW50YWdlIG9mIHRo ZSBhIGdlbmVyaWMgUmVtb3RlIERNQSBUQ1Agb3B0aW9ucyBbUkRNQV0sDQ0K fCAgICAgYWx0aG91Z2ggaXQgY2FuIHN0aWxsIG9wZXJhdGUgZWZmZWN0aXZl bHkgd2l0aG91dCB0aGlzIFRDUCBleHRlbi0NDQp8ICAgICBzaW9uLg0NCiAN DQogNS4xLiAgU21hbGwgVENQIFNlZ21lbnRzDQ0KIA0NCiAgICAgIEl0IGlz IHJlY29tbWVuZGVkIHRoYXQgVENQIHNlZ21lbnRzIGJlIGxpbWl0ZWQgaW4g c2l6ZSB0byBubyBtb3JlDQ0KICAgICAgdGhhbiA4SyBieXRlcy4gT25lIHJl YXNvbiBpcyB0byBlbnN1cmUgdGhhdCBzZWdtZW50cyB3b24ndCBnZXQgYnJv LQ0NCiAgICAgIGtlbiBpbnRvIHNtYWxsZXIgcGFja2V0cywgdGhlcmVieSBw b3NzaWJseSBicmVha2luZyB0aGUgYXNzdW1wdGlvbnMNDQogICAgICBmb3Ig UkRNQSBhbmQgdGhlIGluZm9ybWF0aW9uIGluIHRoZSBSRE1BIGhlYWRlci4g QW5vdGhlciByZWFzb24gd2UNDQogICAgICByZWNvbW1lbmQgc21hbGwgc2Vn bWVudHMgaXMgdG8gYWxsb3cgYSBzdHJvbmdlciB0eXBlIG9mIGNoZWNrc3Vt LA0NCiAgICAgIHBvc3NpYmx5IHV0aWxpemluZyBDUkMsIHdoaWNoIGlzIHBy YWN0aWNhbCBvbmx5IGZvciBzbWFsbGVyIHNlZy0NDQogICAgICBtZW50cy4N DQogDQ0KIDUuMi4gIE11bHRpcGxlIE5ldHdvcmsgQWRhcHRlcnMNDQogDQ0K ICAgICAgVGhlIGlTQ1NJIHByb3RvY29sIGFzc3VtZXMgdGhhdCB0aGUgVGFz ayBUYWdzIHdpbGwgYWxzbyBzZXJ2ZSBhcw0NCiAgICAgIFJETUEgdGFncy4g VGhlIGlTQ1NJIHByb3RvY29sIGFsbG93cyBtdWx0aXBsZSBjb25uZWN0aW9u cywgbm90IGFsbA0NCiAgICAgIG9mIHdoaWNoIG5lZWQgZ28gb3ZlciB0aGUg c2FtZSBuZXR3b3JrIGFkYXB0ZXIuIElmIG11bHRpcGxlIG5ldHdvcmsNDQog ICAgICBjb25uZWN0aW9ucyBhcmUgdG8gYmUgdXRpbGl6ZWQgd2l0aCBSRE1B LCB0aGUgaVNDU0kgcHJvdG9jb2wNDQogICAgICBjb21tYW5kLWRhdGEtc3Rh dHVzIGFsbGVnaWFuY2UgdG8gb25lIFRDUCBjb25uZWN0aW9uIGluc3VyZSB0 aGF0DQ0KICAgICAgdGhlcmUgaXMgbm8gbmVlZCB0byByZXBsaWNhdGUgaW5m b3JtYXRpb24gYWNyb3NzIG5ldHdvcmsgYWRhcHRlcnMNDQogICAgICBvciBv dGhlcndpc2UgcmVxdWlyZSB0aGVtIHRvIGNvb3BlcmF0ZS4NDQogDQ0KIDUu My4gIEF1dG9zZW5zZQ0NCiANDQogICAgICBBdXRvc2Vuc2UgcmVmZXJzIHRv IHRoZSBhdXRvbWF0aWMgcmV0dXJuIG9mIHNlbnNlIGRhdGEgdG8gdGhlIGlu aS0NDQogICAgICB0aWF0b3IgaW4gY2FzZSBhIGNvbW1hbmQgZGlkIG5vdCBj b21wbGV0ZSBzdWNjZXNzZnVsbHkuIElmDQ0KICAgICAgYXV0b3NlbnNlIGlz IHR1cm5lZCBvZmYsIHRoZSBpbml0aWF0b3IgbXVzdCBleHBsaWNpdGx5IHJl cXVlc3QgdGhhdA0NCiAgICAgIHNlbnNlIGRhdGEgYmUgc2VudCB0byBpdCBh ZnRlciBzb21lIGNvbW1hbmQgaGFzIGNvbXBsZXRlZCB3aXRoIGENDQogICAg ICBDSEVDSyBDT05ESVRJT04gc3RhdHVzLiAgVGhlIGRlZmF1bHQgZm9yIGlT Q1NJIGlzIHRvIHdvcmsgd2l0aA0NCnwgICAgIEF1dG9zZW5zZSBlbmFibGVk LiAgTm90ZSB0aGF0IGV2ZW4gaWYgYSBTQ1NJIHRhcmdldC9MVU4gZG9lcyBu b3QNDQp8ICAgICBzdXBwb3J0IEF1dG9zZW5zZSwgaXQgbWF5IHN0aWxsIGJl IHBvc3NpYmxlIGZvciBpU0NTSSB0byB3b3JrIHdpdGgNDQp8ICAgICBBdXRv c2Vuc2UuICBUaGlzIGNhbiBiZSBhY2NvbXBsaXNoZWQgYXMgZm9sbG93cy4g V2hlbmV2ZXIgYSBDSEVDSw0NCnwgICAgIENPTkRJVElPTiBzdGF0dXMgaXMg YWJvdXQgdG8gYmUgcmV0dXJuZWQsIHRoZSBpU0NTSSBjb21wb25lbnQgb24N DQp8ICAgICB0aGUgdGFyZ2V0IGltbWVkaWF0ZWx5IHF1ZXJpZXMgdGhlIHRh cmdldC9MVU4gZm9yIHRoZSBzZW5zZSBkYXRhLg0NCnwgICAgIGlTQ1NJIGNh biB0aGVuIHJldHVybiB0aGUgc2Vuc2UgZGF0YSB0byB0aGUgaW5pdGlhdG9y IHRvZ2V0aGVyIHdpdGgNDQp8ICAgICB0aGUgQ0hFQ0sgQ09ORElUSU9OIHN0 YXR1cy4gIEl0IGlzIG5vdCBuZWNlc3NhcnkgZm9yIGlTQ1NJIHRvIHdhaXQN DQp8ICAgICBmb3IgdGhlIGluaXRpYXRvciB0byBleHBsaWNpdGx5IHJlcXVl c3QgdGhlIHNlbnNlIGRhdGE7IHRoZSB0YXJnZXQNDQp8ICAgICBpU0NTSSBj b2RlIGNhbiBwZXJmb3JtIHRoaXMgb3BlcmF0aW9uIGF1dG9tYXRpY2FsbHks IGV2ZW4gZm9yDQ0KfCAgICAgZGV2aWNlcy9MVU5zIHRoYXQgZG8gbm90IG9y ZGluYXJpbHkgcHJvdmlkZSBhdXRvbWF0aWMgc2Vuc2UgZGF0YS4NDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogU2F0cmFuLCBTbWl0aCwgU2FwdW50emFr aXMsIE1ldGggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgW1BhZ2Ug NDBdDQ0KIAwNDQogaVNDU0kgICAgICAgICAgICAgICAgICAgICAgICAgSnVu ZSAyMDAwDQ0KIA0NCiANDQogNS40LiAgVENQIFJETUEgb3B0aW9uDQ0KIA0N CiAgICAgIFRoZSBUQ1AgUkRNQSBvcHRpb24gW1JETUFdIGlzIGFuIGFubm90 YXRpb24gb24gaW5kaXZpZHVhbCBUQ1Agc2VnLQ0NCiAgICAgIG1lbnRzIHRo YXQgY2FuIHJlZHVjZSB0aGUgbnVtYmVyIG9mIGNvcGllcyBuZWNlc3Nhcnkg YXQgdGhlDQ0KICAgICAgcmVjZWl2ZXIuIFRoZSBSRE1BIG9wdGlvbiBzdWNj aW5jdGx5IGRlc2NyaWJlcyB0aGUgcG9ydGlvbiBvZiBhIFRDUA0NCiAgICAg IHBheWxvYWQgdGhhdCBob2xkcyBidWxrIGRhdGEuDQ0KIA0NCiANDQogNS41 LiAgVENQIENvbm5lY3Rpb24gT3B0aW9ucw0NCiANDQogICAgICBTb21lIHRh cmdldHMgbWF5IHdhbnQgdG8gaW5mb3JtIChvciBuZWdvdGlhdGUgd2l0aCkg YW4gaW5pdGlhdG9yDQ0KICAgICAgY29uY2VybmluZyBzb21lIHBhcmFtZXRl cnMgcmVsYXRlZCB0byBiYW5kd2lkdGgsIFF1YWxpdHkgb2YgU2VyLQ0NCiAg ICAgIHZpY2UsIG9yIHNvbWUgb3RoZXIgYXZhaWxhYmxlIGZlYXR1cmVzIG9u IGl0cyB2YXJpb3VzIG5ldHdvcmsgY29uLQ0NCiAgICAgIG5lY3Rpb25zLiAg VGhlc2UgYXJlIGV4Y2hhbmdlZCBiZXR3ZWVuIHRoZSBpbml0aWF0b3IgYW5k IHRoZSB0YXJnZXQNDQogICAgICB1c2luZyBUZXh0IENvbW1hbmRzIGFuZCBS ZXNwb25zZXMuDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHph a2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdl IDQxXQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1 bmUgMjAwMA0NCiANDQogDQ0KIDYuICBTZWN1cml0eSBDb25zaWRlcmF0aW9u cw0NCiANDQogDQ0KIDYuMS4gIERhdGEgSW50ZWdyaXR5DQ0KIA0NCiAgICAg IFdlIGFzc3VtZSB0aGF0IGVuZC10by1lbmQgZGF0YSBpbnRlZ3JpdHkgY2Fu IGJlIGFzc3VyZWQgYnkgVENQLCBieQ0NCiAgICAgIGFkZGluZyBhIG1vcmUg cG93ZXJmdWwgY2hlY2tzdW0gb3B0aW9uIHdoZW5ldmVyIHRoaXMgaXMgY29u c2lkZXJlZA0NCiAgICAgIGltcG9ydGFudCwgb3IgcmVwbGFjaW5nIHRoZSBj aGVja3N1bSBieSBhIHdlYWtlciBvbmUgKG9yIGV2ZW4gIm51bC0NDQogICAg ICBsaWZ5aW5nIGl0IikgZm9yIGFwcGxpY2F0aW9ucyBpbiB3aGljaCBkYXRh IGludGVncml0eSBpcyBub3QgaW1wb3ItDQ0KICAgICAgdGFudCBhbmQgcmVj b3ZlcnkgZnJvbSBkYXRhIGVycm9ycyBjb3VsZCBiZSBoYXJtZnVsIChlLmcu LCBhdWRpbyBvcg0NCiAgICAgIHZpZGVvIGRpc3RyaWJ1dGlvbiBzdHJlYW1z KS4NDQogDQ0KIDYuMi4gIExvZ2luIFByb2Nlc3MNDQogDQ0KICAgICAgSW4g c29tZSBlbnZpcm9ubWVudHMsIGEgdGFyZ2V0IHdpbGwgbm90IGJlIGludGVy ZXN0ZWQgaW4gYXV0aGVudGktDQ0KICAgICAgY2F0aW5nIHRoZSBpbml0aWF0 b3IuIEluIHRoaXMgY2FzZSwgdGhlIHRhcmdldCBjYW4gc2ltcGx5IGlnbm9y ZQ0NCiAgICAgIHNvbWUgb3IgYWxsIG9mIHRoZSBwYXJhbWV0ZXJzIHNlbnQg aW4gYSBMb2dpbiBDb21tYW5kLCBhbmQgdGhlIHRhci0NDQogICAgICBnZXQg Y2FuIHNpbXBseSByZXBseSB3aXRoIGEgYmFzaWMgTG9naW4gUmVzcG9uc2Ug aW5kaWNhdGluZyBhIHN1Yy0NDQp8ICAgICBjZXNzZnVsIGxvZ2luLiAgU29t ZSB0YXJnZXRzIG1heSB3YW50IHRvIHBlcmZvcm0gc29tZSBraW5kIG9mDQ0K fCAgICAgYXV0aGVudGljYXRpb24uICBUaGUgQXV0aGVudGljYXRvciBrZXkg aXMgZGVmaW5lZCBmb3IgdGhpcyBwdXJwb3NlLg0NCnwgICAgIFZhcmlvdXMg YXV0aGVudGljYXRpb24gc2NoZW1lcyBjYW4gYmUgdXNlZCwgaW5jbHVkaW5n IGVuY3J5cHRlZA0NCnwgICAgIHBhc3N3b3JkcyBhbmQgdHJ1c3RlZCBjZXJ0 aWZpY2F0ZSBhdXRob3JpdGllcy4gIE9uY2UgdGhlIGluaXRpYXRvcg0NCnwg ICAgIGFuZCB0YXJnZXQgYXJlIGNvbmZpZGVudCBvZiB0aGUgaWRlbnRpdHkg b2YgdGhlIGF0dGFjaGVkIHBhcnR5LCB0aGUNDQp8ICAgICBlc3RhYmxpc2hl ZCBjaGFubmVsIGlzIGNvbnNpZGVyZWQgc2VjdXJlLiAgSXQgaXMgYW50aWNp cGF0ZWQgdGhhdA0NCnwgICAgIG1vc3QgdGFyZ2V0IGRldmljZXMgd2lsbCBu b3QgYm90aGVyIHdpdGggYWxsIG9mIHRoZSBwb3NzaWJsZQ0NCnwgICAgIGNo ZWNrcywgYnV0IHRoZSBwcm90b2NvbCBwcm92aWRlcyBzdWZmaWNpZW50IG1l YW5zIHRvIHBlcmZvcm0gdGhlDQ0KfCAgICAgY2hlY2tzLCBpZiByZXF1aXJl ZCBieSB0aGUgdGFyZ2V0Lg0NCiANDQogNi4zLiAgSUFOQSBDb25zaWRlcmF0 aW9ucw0NCiANDQogICAgICBUaGVyZSB3aWxsIGJlIGEgd2VsbCBrbm93biBw b3J0IGZvciBpU0NTSSBjb25uZWN0aW9ucy4gIFRoZXNlIHdlbGwNDQogICAg ICBrbm93biBwb3J0cyB3aWxsIGhhdmUgdG8gYmUgcmVnaXN0ZXJlZCB3aXRo IElBTkEuDQ0KIA0NCiAgICAgIEEgY2hlY2tzdW0gdHlwZSB3aWxsIGFsc28g aGF2ZSB0byBiZSByZWdpc3RlcmVkIHdpdGggSUFOQS4NDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywgTWV0 aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSA0Ml0NDQog DA0NCiBpU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIwMDAN DQogDQ0KIA0NCiA3LiAgQXV0aG9ycycgQWRkcmVzc2VzDQ0KIA0NCiANDQog ICAgICBKdWxpYW4gU2F0cmFuDQ0KICAgICAgS2FsbWFuIE1ldGgNDQogICAg ICBJQk0sIEhhaWZhIFJlc2VhcmNoIExhYg0NCiAgICAgIE1BVEFNIC0gQWR2 YW5jZWQgVGVjaG5vbG9neSBDZW50ZXINDQogICAgICBIYWlmYSAzMTkwNSwg SXNyYWVsDQ0KICAgICAgUGhvbmUgKzk3MiA0IDgyOSA2MjExDQ0KICAgICAg RW1haWw6IEp1bGlhbl9TYXRyYW5Adm5ldC5pYm0uY29tIG1ldGhAaWwuaWJt LmNvbQ0NCiANDQogDQ0KIA0NCiAgICAgIERhbmllbCBGLiBTbWl0aA0NCiAg ICAgIElCTSBBbG1hZGVuIFJlc2VhcmNoIENlbnRlcg0NCiAgICAgIDY1MCBI YXJyeSBSb2FkDQ0KICAgICAgU2FuIEpvc2UsIENBIDk1MTIwLTYwOTksIFVT QQ0NCiAgICAgIFBob25lOiArMSA0MDggOTI3IDIwNzINDQogICAgICBFbWFp bDogZGZzbWl0aEBhbG1hZGVuLmlibS5jb20NDQogDQ0KIA0NCiAgICAgIENv c3RhIFNhcHVudHpha2lzDQ0KICAgICAgQ2lzY28gU3lzdGVtcywgSW5jLg0N CiAgICAgIDE3MCBXLiBUYXNtYW4gRHJpdmUNDQogICAgICBTYW4gSm9zZSwg Q0EgOTUxMzQsIFVTQQ0NCiAgICAgIFBob25lOiArMSA0MDggNTI1IDU0OTcN DQogICAgICBFbWFpbDogY3NhcHVudHpAY2lzY28uY29tDQ0KIA0NCiANDQog ICAgICBFZnJpIFplaWRuZXINDQogICAgICBTQU5HYXRlDQ0KICAgICAgSXNy YWVsDQ0KICAgICAgZWZyaUBzYW5nYXRlLmNvbQ0NCiANDQogDQ0KICAgICAg Q29tbWVudHMgbWF5IGJlIHNlbnQgdG8gSnVsaWFuIFNhdHJhbiwgRGFuaWVs IFNtaXRoLCBDb3N0YQ0NCiAgICAgIFNhcHVudHpha2lzLCBvciBLYWxtYW4g TWV0aC4NDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtp cywgTWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSA0 M10NDQogDA0NCiBpU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5l IDIwMDANDQogDQ0KIA0NCiA4LiAgUmVmZXJlbmNlcyBhbmQgQmlibGlvZ3Jh cGh5DQ0KIA0NCiAgICAgIFtSRE1BXSAgICBJbnRlcm5ldCBEcmFmdDogVENQ IFJETUEgb3B0aW9uICh3b3JrIGluIHByb2dyZXNzKQ0NCiAgICAgIFtTQU0y XSAgICBBTlNJIFgzLjI3MC0xOTk4LCBTQ1NJLTMgQXJjaGl0ZWN0dXJlIE1v ZGVsIChTQU0tMikNDQogICAgICBbVExTXSAgICAgVGhlIFRMUyBQcm90b2Nv bCwgUkZDIDIyNDYsIFQuIERpZXJrcyBldCBhbC4NDQogDQ0KICAgICAgW0FM VENdICAgIEludGVybmV0IERyYWZ0OiBBbHRlcm5hdGl2ZSBjaGVja3N1bXMg KHdvcmsgaW4gcHJvZ3Jlc3MpDQ0KICAgICAgW0NBTV0gICAgIEFOU0kgWDMu MjMyLTE5OVgsIENvbW1vbiBBY2Nlc3MgTWV0aG9kLTMgKENhbS0zKQ0NCiAg ICAgIFtDUkNdICAgICBJU08gMzMwOSwgSGlnaC1MZXZlbCBEYXRhIExpbmsg Q29udHJvbCAoQ1JDIDMyKQ0NCiAgICAgIFtSRkM3OTNdICBUcmFuc21pc3Np b24gQ29udHJvbCBQcm90b2NvbCwgUkZDIDc5Mw0NCnwgICAgIFtSRkMxMTIy XSBSZXF1aXJlbWVudHMgZm9yIEludGVybmV0IEhvc3RzLUNvbW11bmljYXRp b24gTGF5ZXIsIFJGQw0NCnwgICAgICAgICAgICAgICAxMTIyLCBSLiBCcmFk ZW4gKGVkaXRvcikNDQogICAgICBbU0JDXSAgICAgQU5TSSBYMy4zMDYtMTk5 WCwgU0NTSS0zIEJsb2NrIENvbW1hbmRzIChTQkMpDQ0KICAgICAgW1NDU0ky XSAgIEFOU0kgWDMuMTMxLTE5OTQsIFNDU0ktMg0NCiAgICAgIFtTUENdICAg ICBBTlNJIFgzLjMwMS0xOTlYLCBTQ1NJLTMgUHJpbWFyeSBDb21tYW5kcyAo U1BDKQ0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiBTYXRyYW4sIFNtaXRoLCBTYXB1bnR6YWtpcywg TWV0aCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBbUGFnZSA0NF0N DQogDA0NCiBpU0NTSSAgICAgICAgICAgICAgICAgICAgICAgICBKdW5lIDIw MDANDQogDQ0KIA0NCiA5LiAgQXBwZW5kaXggQSAtIEV4YW1wbGVzDQ0KIA0N CiANDQogOS4xLiAgUmVhZCBvcGVyYXRpb24gZXhhbXBsZQ0NCiANDQogDQ0K ICAgICAgfEluaXRpYXRvciBGdW5jdGlvbnwgICAgTWVzc2FnZSBUeXBlICAg ICAgIHwgIFRhcmdldCBGdW5jdGlvbiAgICAgfA0NCiAgICAgICstLS0tLS0t LS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tLS0tLS0tLS0tLSsNDQogICAgICB8ICBDb21tYW5kIHJlcXVlc3QgfFND U0kgQ29tbWFuZCAoUkVBRCk+Pj4gfCAgICAgICAgICAgICAgICAgICAgICB8 DQ0KICAgICAgfCAgKHJlYWQpICAgICAgICAgIHwgICAgICAgICAgICAgICAg ICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgfA0NCiAgICAgICstLS0t LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0t LS0tLS0tLS0tLS0tLS0tLSsNDQogICAgICB8ICAgICAgICAgICAgICAgICAg fCAgICAgICAgICAgICAgICAgICAgICAgfCBQcmVwYXJlIERhdGEgVHJhbnNm ZXJ8DQ0KICAgICAgKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0t LS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0NCiAgICAgIHwg ICBSZWNlaXZlIERhdGEgICB8ICAgPDw8IFNDU0kgRGF0YSAgICAgICB8ICAg U2VuZCBEYXRhICAgICAgICAgIHwNDQogICAgICArLS0tLS0tLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0t LS0tLS0rDQ0KICAgICAgfCAgIFJlY2VpdmUgRGF0YSAgIHwgICA8PDwgU0NT SSBEYXRhICAgICAgIHwgICBTZW5kIERhdGEgICAgICAgICAgfA0NCiAgICAg ICstLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0r LS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsNDQogICAgICB8ICAgUmVjZWl2ZSBE YXRhICAgfCAgIDw8PCBTQ1NJIERhdGEgICAgICAgfCAgIFNlbmQgRGF0YSAg ICAgICAgICB8DQ0KICAgICAgKy0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0NCiAg ICAgIHwgICAgICAgICAgICAgICAgICB8ICAgPDw8IFNDU0kgUmVzcG9uc2Ug ICB8U2VuZCBTdGF0dXMgYW5kIFNlbnNlIHwNDQogICAgICArLS0tLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0t LS0tLS0tLS0tLS0rDQ0KICAgICAgfCBDb21tYW5kIENvbXBsZXRlIHwgICAg ICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICAgfA0N CiAgICAgICstLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsNDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNhcHVudHpha2lzLCBNZXRoICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtQYWdlIDQ1XQ0NCiAMDQ0K IGlTQ1NJICAgICAgICAgICAgICAgICAgICAgICAgIEp1bmUgMjAwMA0NCiAN DQogDQ0KIDkuMi4gIFdyaXRlIG9wZXJhdGlvbiBleGFtcGxlDQ0KIA0NCiAN DQogICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsNDQogICAgICB8SW5pdGlh dG9yIEZ1bmN0aW9ufCAgICBNZXNzYWdlIFR5cGUgICAgICAgfCAgVGFyZ2V0 IEZ1bmN0aW9uICAgIHwNDQogICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsN DQogICAgICB8ICBDb21tYW5kIHJlcXVlc3QgfFNDU0kgQ29tbWFuZCAoV1JJ VEUpPj4+fCBSZWNlaXZlIGNvbW1hbmQgICAgIHwNDQogICAgICB8ICAod3Jp dGUpICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgICAgfCBhbmQgcXVl dWUgaXQgICAgICAgIHwNDQogICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsN DQogICAgICB8ICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAg ICAgICAgfCBQcm9jZXNzIG9sZCBjb21tYW5kc3wNDQogICAgICArLS0tLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLS0tLS0tLSsNDQogICAgICB8ICAgICAgICAgICAgICAgICAgfCAg ICAgICAgICAgICAgICAgICAgICAgfCBSZWFkeSB0byBwcm9jZXNzICAgIHwN DQogICAgICB8ICAgICAgICAgICAgICAgICAgfCAgIDw8PCBSVFQgICAgICAg ICAgICAgfCBXUklURSBjb21tYW5kICAgICAgIHwNDQogICAgICArLS0tLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLS0tLS0tLSsNDQogICAgICB8ICAgU2VuZCBEYXRhICAgICAgfCAg IFNDU0kgRGF0YSA+Pj4gICAgICAgfCAgIFJlY2VpdmUgRGF0YSAgICAgIHwN DQogICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsNDQogICAgICB8ICAgU2Vu ZCBEYXRhICAgICAgfCAgIFNDU0kgRGF0YSA+Pj4gICAgICAgfCAgIFJlY2Vp dmUgRGF0YSAgICAgIHwNDQogICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsN DQogICAgICB8ICAgICAgICAgICAgICAgICAgfCAgIDw8PCBSVFQgICAgICAg ICAgICAgfCAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICArLS0tLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLS0tLS0tLSsNDQogICAgICB8ICAgU2VuZCBEYXRhICAgICAgfCAg IFNDU0kgRGF0YSA+Pj4gICAgICAgfCAgIFJlY2VpdmUgRGF0YSAgICAgIHwN DQogICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsNDQogICAgICB8ICAgICAg ICAgICAgICAgICAgfCAgIDw8PCBTQ1NJIFJlc3BvbnNlICAgfFNlbmQgU3Rh dHVzIGFuZCBTZW5zZXwNDQogICAgICArLS0tLS0tLS0tLS0tLS0tLS0tKy0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLSsN DQogICAgICB8IENvbW1hbmQgQ29tcGxldGUgfCAgICAgICAgICAgICAgICAg ICAgICAgfCAgICAgICAgICAgICAgICAgICAgIHwNDQogICAgICArLS0tLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0t LS0tLS0tLS0tLS0tLSsNDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog DQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIFNhdHJhbiwgU21pdGgsIFNh cHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IFtQYWdlIDQ2XQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAgICAgICAgICAgICAg ICAgIEp1bmUgMjAwMA0NCiANDQogDQ0KIDEwLiAgQXBwZW5kaXggQiAtIExv Z2luL1RleHQga2V5cw0NCiANDQogDQ0KIDEwLjEuICBUYXJnZXQNDQogDQ0K ICAgICAgVGFyZ2V0OmRvbWFpbm5hbWVbL21vZGlmaWVyXQ0NCiANDQogICAg ICBFeGFtcGxlczoNDQogDQ0KICAgICAgICAgICAgICBUYXJnZXQ6ZGlzay1h cnJheS5zai1ibGRnLWguY2lzY28uY29tDQ0KICAgICAgICAgICAgICBUYXJn ZXQ6ZGlzay1hcnJheS5zai1ibGRnLWguY2lzY28uY29tL2Rpc2szDQ0KIA0N CiAgICAgIFRoaXMga2V5IGlzIHByb3ZpZGVkIGJ5IHRoZSBpbml0aWF0b3Ig b2YgdGhlIFRDUCBjb25uZWN0aW9uIHRvIHRoZQ0NCiAgICAgIHJlbW90ZSBl bmRwb2ludC4gVGhlIFRhcmdldCBrZXkgc3BlY2lmaWVzIHRoZSBkb21haW4g bmFtZSBvZiB0aGUNDQogICAgICB0YXJnZXQsIHNpbmNlIHRoYXQgaW5mb3Jt YXRpb24gaXMgbm90IGF2YWlsYWJsZSBmcm9tIHRoZSBUQ1AgbGF5ZXIuDQ0K fCAgICAgVGhlIHRhcmdldCBpcyBub3QgcmVxdWlyZWQgdG8gc3VwcG9ydCB0 aGlzIGtleS4gIFRoZSBpbml0aWF0b3INDQp8ICAgICBzaG91bGQgc2VuZCB0 aGlzIGtleSBpbiB0aGUgZmlyc3QgbG9naW4gbWVzc2FnZS4gVGhlIFRhcmdl dCBrZXkNDQp8ICAgICBtaWdodCBiZSB1c2VkIGJ5IHRoZSB0YXJnZXQgdG8g bGVhcm4gdGhlIGludGVuZGVkIGluaXRpYXRvciB2aWV3IG9mDQ0KfCAgICAg dGhlIHRhcmdldC4NDQogDQ0KIDEwLjIuICBJbml0aWF0b3INDQogDQ0KIA0N CnwgICAgIEluaXRpYXRvcjpbZG9tYWlubmFtZVsvbW9kaWZpZXJdXSBFeGFt cGxlczoNDQogDQ0KICAgICAgICAgICAgICBJbml0aWF0b3I6c2FtcGxlLmZv b2Jhci5vcmcNDQogICAgICAgICAgICAgIEluaXRpYXRvcjpjbHVzdGVyLmZv b2Jhci5vcmcvbWFjaGluZTENDQogICAgICAgICAgICAgIEluaXRpYXRvcjoN DQogDQ0KICAgICAgVGhlIEluaXRpYXRvciBrZXkgZW5hYmxlcyB0aGUgaW5p dGlhdG9yIHRvIGlkZW50aWZ5IGl0c2VsZiB0byB0aGUNDQogICAgICByZW1v dGUgZW5kcG9pbnQuIFRoZSBkb21haW4gbmFtZSBzaG91bGQgYmUgdGhhdCBv ZiB0aGUgaW5pdGlhdG9yLg0NCiAgICAgIEEgemVyby1sZW5ndGggZG9tYWlu IG5hbWUgaXMgaW50ZXJwcmV0ZWQgYXMgIm90aGVyIHNpZGUgb2YgVENQIGNv bi0NDQogICAgICBuZWN0aW9uIi4gVGhlIHRhcmdldCBtYXkgc2lsZW50bHkg aWdub3JlIHRoaXMga2V5IGlmIGl0IGRvZXMgbm90DQ0KfCAgICAgc3VwcG9y dCBpdC4gIEZvciBtb3JlIHNlY3VyaXR5LCBhIGNlcnRpZmljYXRlLWJhc2Vk IHByb3RvY29sIFtUTFNdDQ0KfCAgICAgbWF5IGJlIHVzZWQgb24gdGhlIGNo YW5uZWwgYW5kIHRha2UgcHJlY2VkZW5jZSBvdmVyIHRoaXMgcHJvdG9jb2wu DQ0KIA0NCiAxMC4zLiAgQXV0aGVudGljYXRvcg0NCiANDQogDQ0KfCAgICAg QXV0aGVudGljYXRvcjo8VVRGOC1TdHJpbmc+IEV4YW1wbGVzOg0NCiANDQog ICAgICAgICAgICAgIEF1dGhlbnRpY2F0b3I6b3Blbi1zZXNhbWUNDQogDQ0K ICAgICAgVGhlIGF1dGhlbnRpY2F0b3IgaXMgYSBzZWNyZXQgdGhhdCB0aGUg aW5pdGlhdG9yIHVzZXMgdG8gZ2Fpbg0NCiAgICAgIGFjY2VzcyB0byB0aGUg dGFyZ2V0J3MgTFVOcy4NDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogU2F0 cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgW1BhZ2UgNDddDQ0KIAwNDQogaVNDU0kgICAgICAg ICAgICAgICAgICAgICAgICAgSnVuZSAyMDAwDQ0KIA0NCiANDQogMTAuNC4g IFNlbmRBdXRoZW50aWNhdG9yDQ0KIA0NCiANDQp8ICAgICBTZW5kQXV0aGVu dGljYXRvcjp5ZXMgUmVzcG9uc2U6IEF1dGhlbnRpY2F0b3I6PFVURjgtU3Ry aW5nPiBFeGFtLQ0NCnwgICAgIHBsZXM6DQ0KIA0NCiAgICAgICAgICAgICAg U2VuZEF1dGhlbnRpY2F0b3I6eWVzDQ0KICAgICAgICAgICAgICAtPiBBdXRo ZW50aWNhdG9yOmFsYWthemFtDQ0KIA0NCiAgICAgIFRoZSBTZW5kQXV0aGVu dGljYXRvciBrZXkgaXMgdXNlZCB0byByZXF1ZXN0IGZyb20gdGhlIHBhcnR5 IG9uIHRoZQ0NCiAgICAgIG90aGVyIHNpZGUgb2YgdGhlIFRDUCBjb25uZWN0 aW9uIHRvIHNlbmQgaXRzIEF1dGhlbnRpY2F0b3IuICBpU0NTSQ0NCiAgICAg IGRldmljZXMgbWF5IHJlZnVzZSB0byBncmFudCBhY2Nlc3MgdW50aWwgcHJv cGVyIGF1dGhlbnRpY2F0aW9uIGhhcw0NCiAgICAgIGJlZW4gcGVyZm9ybWVk IGJ5IHRoZSBwYXJ0aWVzIGludm9sdmVkLg0NCiANDQogMTAuNS4gIEFsbG93 Tm9SVFQNDQogDQ0KIA0NCnwgICAgIEFsbG93Tm9SVFQ6PHllc3xubz4gUmVz cG9uc2U6IEFsbG93Tm9SVFQ6PHllc3xubz4gRXhhbXBsZXM6DQ0KIA0NCiAg ICAgICAgICAgICAgQWxsb3dOb1JUVDp5ZXMNDQogICAgICAgICAgICAgIC0+ IEFsbG93Tm9SVFQ6eWVzDQ0KIA0NCiAgICAgIFRoZSBBbGxvd05vUlRUIGtl eSBpcyB1c2VkIHRvIGFsbG93IGFuIGluaXRpYXRvciB0byBzZW5kIGRhdGEg dG8gYQ0NCiAgICAgIHRhcmdldCB3aXRob3V0IHRoZSB0YXJnZXQgaGF2aW5n IHNlbnQgYW4gUlRUIHRvIHRoZSBpbml0aWF0b3IuICBUaGUNDQogICAgICBk ZWZhdWx0IGFjdGlvbiBpcyB0aGF0IFJUVCBpcyByZXF1aXJlZCwgdW5sZXNz IGJvdGggdGhlIGluaXRpYXRvcg0NCiAgICAgIGFuZCB0aGUgdGFyZ2V0IHNl bmQgdGhpcyBrZXktcGFpciBhdHRyaWJ1dGUgc3BlY2lmeWluZw0NCiAgICAg IEFsbG93Tm9SVFQ6eWVzLiAgT25jZSBBbGxvd05vUlRUIGhhcyBiZWVuIHNl dCB0byAneWVzJywgaXQgY2Fubm90DQ0KICAgICAgYmUgc2V0IGJhY2sgdG8g J25vJy4NDQogDQ0KIA0NCiAxMC42LiAgT3JpZ2luYWxJbml0aWF0b3INDQog DQ0KIA0NCnwgICAgIE9yaWdpbmFsSW5pdGlhdG9yOltkb21haW5uYW1lWy9t b2RpZmllcl1dIEV4YW1wbGVzOg0NCiANDQogICAgICAgICAgICAgIE9yaWdp bmFsSW5pdGlhdG9yOnNhbXBsZS5mb29iYXIub3JnDQ0KIA0NCiAgICAgIFRo ZSBPcmlnaW5hbEluaXRpYXRvciBrZXkgaXMgdXNlZCB0byBwZXJmb3JtIGEg cHJveHkgbG9naW4gZnJvbSBvbmUNDQogICAgICB0YXJnZXQgdG8gYW5vdGhl ciB0YXJnZXQgaW4gb3JkZXIgdG8gcGVyZm9ybSBhIHRoaXJkLXBhcnR5IG9w ZXJhLQ0NCiAgICAgIHRpb24gKGxpa2UgQ09QWSkgZm9yIHNvbWUgaW5pdGlh dG9yLiBUaGUgZmlyc3QgdGFyZ2V0IGFjdHMgYXMgdGhlDQ0KICAgICAgaW5p dGlhdG9yIGZvciB0aGUgc2Vjb25kIHRhcmdldCwgYnV0IGl0IG11c3QgcHJv dmlkZSB0aGUgYXV0aG9yaXphLQ0NCiAgICAgIHRpb24gaW5mb3JtYXRpb24g b2YgdGhlIG9yaWdpbmFsIGluaXRpYXRvci4NDQogDQ0KIA0NCiAxMC43LiAg VGFyZ2V0Mg0NCiANDQogDQ0KfCAgICAgVGFyZ2V0Mjpkb21haW5uYW1lWy9t b2RpZmllcl0gRXhhbXBsZXM6DQ0KIA0NCiANDQogDQ0KIFNhdHJhbiwgU21p dGgsIFNhcHVudHpha2lzLCBNZXRoICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIFtQYWdlIDQ4XQ0NCiAMDQ0KIGlTQ1NJICAgICAgICAgICAgICAg ICAgICAgICAgIEp1bmUgMjAwMA0NCiANDQogDQ0KIA0NCiAgICAgICAgICAg ICAgVGFyZ2V0MjpzYW1wbGUuZm9vYmFyLm9yZw0NCiAgICAgICAgICAgICAg VGFyZ2V0MjpzYW1wbGUuZm9vYmFyLm9yZy9yYWlkMg0NCiANDQogICAgICBU aGUgVGFyZ2V0MiBrZXkgaXMgdXNlZCBpbiBhIHRoaXJkLXBhcnR5IFNDU0kg Y29tbWFuZCAobGlrZSBDT1BZKQ0NCiAgICAgIGJldHdlZW4gdGFyZ2V0cyB0 aGF0IGRvIG5vdCBsaWUgb24gdGhlIHNhbWUgU0NTSSBmYWJyaWMuIFRoZSBp bmktDQ0KICAgICAgdGlhdG9yIG11c3Qgc3BlY2lmeSB0aGUgbmFtZSBvZiB0 aGUgZGlzdGFudCB0YXJnZXQgdG8gdGhlIG9yaWdpbmFsDQ0KICAgICAgdGFy Z2V0LCBzbyB0aGF0IHRoZSBvcmlnaW5hbCB0YXJnZXQgY2FuIExvZ2luIHRv IHRoZSBkaXN0YW50IHRhcmdldA0NCiAgICAgIGFuZCB0aGVuIHBlcmZvcm0g dGhlIHRoaXJkLXBhcnR5IGNvbW1hbmQuDQ0KIA0NCiANDQogICAgICBFeHBp cmVzIDE1IERlY2VtYmVyIDIwMDANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiAN DQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0N CiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0K IA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQogDQ0KIA0NCiANDQog U2F0cmFuLCBTbWl0aCwgU2FwdW50emFraXMsIE1ldGggICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgW1BhZ2UgNDldDQ0KIAwNDQo= ---559023410-1804928587-962307485=:3404 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="create-change-bar.pl" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="create-change-bar.pl" IyENCg0KdXNlIHN0cmljdDsNCg0KbXkgJHNyYyA9ICRBUkdWWzBdOw0KbXkg JGRzdCA9ICRBUkdWWzFdOw0KDQpvcGVuKFNSQywgJHNyYyk7DQoNCiRzcmMg Lj0gIi50bXAiOw0KDQpvcGVuKFNSQzIsICI+JHNyYyIpOw0KDQpvcGVuKERT VCwgJGRzdCk7DQoNCiRkc3QgLj0gIi50bXAiOw0KDQpvcGVuKERTVDIsICI+ JGRzdCIpOw0KDQojDQojIFJlbW92ZSB1bm5lY2Vzc2FyeSBsaW5lcyBmcm9t IGVhY2ggYW5kIHN0b3JlIG1hcHBpbmcNCiMNCm15IEBsaW5lbWFwID0gKCk7 DQpteSBAY2hhbmdlcyA9ICgpOw0KDQpteSAkb2xkbGluZW5vID0gLTE7DQpt eSAkbmV3bGluZW5vID0gMDsNCm15ICRzdGF0ZSA9IDA7DQoNCndoaWxlKDxT UkM+KSB7DQogICAgJG9sZGxpbmVubysrOw0KICAgICRjaGFuZ2VzWyRvbGRs aW5lbm9dID0gMDsNCiAgICBpZiAoJHN0YXRlID09IDEpIHsNCgkkc3RhdGUg PSAwOw0KCW5leHQ7DQogICAgfQ0KDQogICAgaWYgKC9cMDE0Lykgew0KCSRz dGF0ZSA9IDE7DQoJbmV4dDsNCiAgICB9DQogICAgbmV4dCBpZiAoL1xbUGFn ZS8pOw0KICAgIG5leHQgaWYgKC9eXHMqJC8pOw0KDQogICAgJGxpbmVtYXBb JG5ld2xpbmVub10gPSAkb2xkbGluZW5vOw0KDQogICAgJG5ld2xpbmVubysr Ow0KICAgIHByaW50IFNSQzIgJF87DQp9DQoNCndoaWxlICg8RFNUPikgew0K ICAgIGlmICgkc3RhdGUgPT0gMSkgew0KCSRzdGF0ZSA9IDA7DQoJbmV4dDsN CiAgICB9DQoNCiAgICBpZiAoL1wwMTQvKSB7DQoJJHN0YXRlID0gMTsNCglu ZXh0Ow0KICAgIH0NCiAgICBuZXh0IGlmICgvXFtQYWdlLyk7DQogICAgbmV4 dCBpZiAoL15ccyokLyk7DQoNCiAgICBwcmludCBEU1QyICRfOw0KfQ0KDQpj bG9zZSBEU1QyOw0KY2xvc2UgRFNUOw0KY2xvc2UgU1JDMjsNCg0Kb3BlbiAo RElGRlMsICJkaWZmIC13ICRkc3QgJHNyYyB8Iik7DQoNCndoaWxlKDxESUZG Uz4pIHsNCiMgICAgcHJpbnQgOw0KDQogICAgbmV4dCBpZiAoIS9eXGQvKTsN Cg0KICAgIC9jKFxkKykoLChcZCspKT8vOw0KICAgIA0KIyAgICBwcmludCAk MSAuICIgIiAuICQzIC4gIlxuIjsNCg0KICAgIG5leHQgaWYgKCFkZWZpbmVk ICQxKTsNCg0KICAgIG15ICRzdGFydCA9ICQxOw0KICAgIG15ICRlbmQgPSAk MzsNCg0KICAgIGlmICghZGVmaW5lZCAkZW5kKSB7DQoJJGVuZCA9ICRzdGFy dDsNCiAgICB9DQoNCiAgICBmb3IgKG15ICRpZHggPSAkc3RhcnQgLSAxOyAk aWR4IDw9ICRlbmQgLSAxOyAkaWR4KyspIHsNCiMJcHJpbnQgJGlkeCAuICJc biI7DQoNCgkkY2hhbmdlc1skbGluZW1hcFskaWR4XV0gPSAxOw0KICAgIH0N Cn0NCg0Kc2VlayBTUkMsIDAsIDA7DQoNCiRvbGRsaW5lbm8gPSAwOw0KDQp3 aGlsZSAoPFNSQz4pIHsNCiAgICBpZiAoJGNoYW5nZXNbJG9sZGxpbmVub10p IHsNCglwcmludCAifCI7DQogICAgfSBlbHNlIHsNCglwcmludCAiICI7DQog ICAgfQ0KDQogICAgcHJpbnQ7DQoNCiAgICAkb2xkbGluZW5vKys7DQp9DQo= ---559023410-1804928587-962307485=:3404-- From owner-ips@ECE.cmu.edu Thu Jun 29 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23264; 29 Jun 2000 17:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA17325 for ips-outgoing; Thu, 29 Jun 2000 16:36:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA17319; Thu, 29 Jun 2000 16:36:46 -0400 (EDT) Received: from sj-msg-av-2.cisco.com (sj-msg-av-2.cisco.com [171.71.163.110]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id NAA19594 for ; Thu, 29 Jun 2000 13:31:07 -0700 (PDT) Received: from proxy2.cisco.com (localhost [127.0.0.1]) by sj-msg-av-2.cisco.com (8.10.1/8.10.1) with SMTP id e5TKUtm28997 for ; Thu, 29 Jun 2000 13:30:55 -0700 (PDT) Received: from gordan.pl.gadzoox.com (i175.gadzoox.com [216.52.31.175]) by proxy2.cisco.com with SMTP (MailShield v1.5); Thu, 29 Jun 2000 13:30:54 -0700 Received: from gadzoox.com (tickle.pl.gadzoox.com [209.95.220.48]) by gordan.pl.gadzoox.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N22S79B7; Thu, 29 Jun 2000 13:31:23 -0700 Message-ID: <395BB10C.928BE27E@gadzoox.com> Date: Thu, 29 Jun 2000 13:26:52 -0700 From: Lani X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: John Matze CC: scsi-tcp@external.cisco.com, ips@ece.cmu.edu Subject: Review of requirements document References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-SMTP-HELO: gordan.pl.gadzoox.com X-SMTP-MAIL-FROM: lani@gadzoox.com X-SMAP-Received-From: outside X-SMTP-PEER-INFO: i175.gadzoox.com [216.52.31.175] Sender: owner-ips@ece.cmu.edu Precedence: bulk John, In reviewing Randy's requirements document in our phone meeting today (6/29/00) a change was proposed/introduced (starting on first page under section APPLICABILITY) to define current storage controllers ( I assume SCSI) as volume/block/sector(VBS) oriented devices & I felt that this was confusing to add "sector" to this definition, as the SCSI protocols used on such storage devices are block (or stream) oriented and the specs never use the term "sector". We needed input from you as this change was initiated from your comment at the 6/19 Haifa meeting. Could you please elaborate on what you had in mind - should we attempt to describe the media mapping method underneath SCSI in the document? thanks - Lani From owner-ips@ECE.cmu.edu Thu Jun 29 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23268; 29 Jun 2000 17:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA17160 for ips-outgoing; Thu, 29 Jun 2000 16:30:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i175.gadzoox.com [216.52.31.175]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA17156 for ; Thu, 29 Jun 2000 16:30:33 -0400 (EDT) Received: from gadzoox.com (tickle.pl.gadzoox.com [209.95.220.48]) by gordan.pl.gadzoox.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N22S79B7; Thu, 29 Jun 2000 13:31:23 -0700 Message-ID: <395BB10C.928BE27E@gadzoox.com> Date: Thu, 29 Jun 2000 13:26:52 -0700 From: Lani X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: John Matze CC: scsi-tcp@external.cisco.com, ips@ece.cmu.edu Subject: Review of requirements document References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk John, In reviewing Randy's requirements document in our phone meeting today (6/29/00) a change was proposed/introduced (starting on first page under section APPLICABILITY) to define current storage controllers ( I assume SCSI) as volume/block/sector(VBS) oriented devices & I felt that this was confusing to add "sector" to this definition, as the SCSI protocols used on such storage devices are block (or stream) oriented and the specs never use the term "sector". We needed input from you as this change was initiated from your comment at the 6/19 Haifa meeting. Could you please elaborate on what you had in mind - should we attempt to describe the media mapping method underneath SCSI in the document? thanks - Lani From owner-ips@ECE.cmu.edu Fri Jun 30 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23651; 29 Jun 2000 21:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA26066 for ips-outgoing; Thu, 29 Jun 2000 19:48:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch11.nsmg.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA26061 for ; Thu, 29 Jun 2000 19:48:19 -0400 (EDT) Received: by lmoxch11.nsmg.veritas.com with Internet Mail Service (5.5.2448.0) id ; Thu, 29 Jun 2000 19:48:55 -0400 Message-ID: From: John Matze To: "'Lani'" Cc: ips@ece.cmu.edu Subject: RE: Review of requirements document Date: Thu, 29 Jun 2000 19:48:48 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The SCSI spec does use the sector representation in the Mode Select/Mode Sense pages for the format device page. However, my request was more for SCSI software developers who in many cases have treated the blocks as sectors. In some cases it maps easier with the upper layers of the operating system. In either case if this sounds too confusing, then I have no objection of dropping the term. This does bring up another objection if we do change the VBS. Not all SCSI devices are block and volume oriented. For instance tape libraries do not have a concept of blocks they use elements and I'm not sure about scanners and printers. I wouldn't want the spec to be too "disk/tape" specific. It may stop other devices from utilizing the spec. John -----Original Message----- From: Lani [mailto:lani@gadzoox.com] Sent: Thursday, June 29, 2000 1:27 PM To: John Matze Cc: scsi-tcp@external.cisco.com; ips@ece.cmu.edu Subject: Review of requirements document John, In reviewing Randy's requirements document in our phone meeting today (6/29/00) a change was proposed/introduced (starting on first page under section APPLICABILITY) to define current storage controllers ( I assume SCSI) as volume/block/sector(VBS) oriented devices & I felt that this was confusing to add "sector" to this definition, as the SCSI protocols used on such storage devices are block (or stream) oriented and the specs never use the term "sector". We needed input from you as this change was initiated from your comment at the 6/19 Haifa meeting. Could you please elaborate on what you had in mind - should we attempt to describe the media mapping method underneath SCSI in the document? thanks - Lani From owner-ips@ECE.cmu.edu Fri Jun 30 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23757; 29 Jun 2000 22:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA28548 for ips-outgoing; Thu, 29 Jun 2000 21:16:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gordan.pl.gadzoox.com (i175.gadzoox.com [216.52.31.175]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA28544 for ; Thu, 29 Jun 2000 21:16:48 -0400 (EDT) Received: from gadzoox.com (tickle.pl.gadzoox.com [209.95.220.48]) by gordan.pl.gadzoox.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N22S79HL; Thu, 29 Jun 2000 18:17:38 -0700 Message-ID: <395BF423.250BA55D@gadzoox.com> Date: Thu, 29 Jun 2000 18:13:07 -0700 From: Lani X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: John Matze CC: ips@ece.cmu.edu Subject: Re: Review of requirements document References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree that the spec appears very disk tape specific, and perhaps the confusing point here is that we are doing a transport for all SCSI devices, but tend to describe storage concerns in particular detail. I do not think that the VBS point is worth lengthy debate, and am willing to accept the paper as is, but will offer the following solution by avoiding the use of all terms such as block,sector,library, etc. In the first use of the term we are trying to distinguish file oriented storage from block or scsi oriented storage which may be a good focus point to keep, while the remaining two references in the spec could do with the general term of SCSI device... so here are the suggested sentence changes I would propose: was: Traditionally, volume block...VBS....through fibre channel. new: Traditionally, storage controllers utilizing the SCSI protocol have been attached to computers through the SCSI parallel bus or through Fibre Channel. was: The IP/Ethernet infrastructure...VBS....current approaches. new: The IP/Ethernet infrastructure offers compelling advantages for SCSI device attachment as compared to current approaches. was: The iSCSI standard ....VBS...such as Ethernet. new: The iSCSI standard will permit SCSI devices to be directly attached to IP networks such as the Ethernet. -Lani John Matze wrote: > > The SCSI spec does use the sector representation in the Mode Select/Mode > Sense pages for the format device page. However, my request was more for > SCSI software developers who in many cases have treated the blocks as > sectors. In some cases it maps easier with the upper layers of the > operating system. In either case if this sounds too confusing, then I have > no objection of dropping the term. > > This does bring up another objection if we do change the VBS. Not all SCSI > devices are block and volume oriented. For instance tape libraries do not > have a concept of blocks they use elements and I'm not sure about scanners > and printers. I wouldn't want the spec to be too "disk/tape" specific. It > may stop other devices from utilizing the spec. > > John > > -----Original Message----- > From: Lani [mailto:lani@gadzoox.com] > Sent: Thursday, June 29, 2000 1:27 PM > To: John Matze > Cc: scsi-tcp@external.cisco.com; ips@ece.cmu.edu > Subject: Review of requirements document > > John, > In reviewing Randy's requirements document in our phone meeting today > (6/29/00) a change was proposed/introduced (starting on first page under > section APPLICABILITY) to define current storage controllers ( I assume > SCSI) as volume/block/sector(VBS) oriented devices & I felt that this was > confusing to add "sector" to this definition, as the SCSI protocols used on > such storage devices are block (or stream) oriented and the specs never use > the term "sector". We needed input from you as this change was initiated > from your comment at the 6/19 Haifa meeting. Could you please elaborate on > what you had in mind - should we attempt to describe the media mapping > method underneath SCSI in the document? > thanks - Lani From owner-ips@ECE.cmu.edu Fri Jun 30 14:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25647; 30 Jun 2000 9:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA20602 for ips-outgoing; Fri, 30 Jun 2000 08:39:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA20598 for ; Fri, 30 Jun 2000 08:39:15 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id IAA16791 for ips@ece.cmu.edu; Fri, 30 Jun 2000 08:39:13 -0400 (EDT) Date: Fri, 30 Jun 2000 08:39:13 -0400 (EDT) From: Scott Bradner Message-Id: <200006301239.IAA16791@newdev.harvard.edu> To: ips@ece.cmu.edu Subject: ips working group Sender: owner-ips@ece.cmu.edu Precedence: bulk fyi The Transport Area Directors have started the process for chartering a working group on IP Storage, along the lines that have been recommended by the Adelaide BOF and discussions on this mailing list. We have asked David Black, of EMC, and Steve Bellovin, of AT&T, to co-chair the working group to be, based on a number of considerations and they will soon be airing the draft of the charter for comments. Allison & Scott From owner-ips@ECE.cmu.edu Fri Jun 30 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25849; 30 Jun 2000 12:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA24528 for ips-outgoing; Fri, 30 Jun 2000 10:53:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA24523 for ; Fri, 30 Jun 2000 10:53:29 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 5F615837F3; Fri, 30 Jun 2000 09:51:38 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N5PCKXJK; Fri, 30 Jun 2000 09:59:14 -0500 Message-ID: <395CB479.D4D1BA31@nuspeed.com> Date: Fri, 30 Jun 2000 09:53:45 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: Digest Login, iSCSI Discovery, and Other Draft 03 Comments References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian- Thanks for taking a look. My comments are embedded below. - Mark julian_satran@il.ibm.com wrote: > > 1. version is already in my working draft! Excellent > 2. we will have to merge it with whatever comes from Luciano & Paul Luciano & Paul- Any comments? > > 3.that assumption was made for SSL > > 1. anything related to discovery might be a bit premature. What you propose > look like a neat > thing but I don't know how it will merge with the management scheme, > administrative access. > I though that reserving the name is a thing we should do now but not more We'll leave our discovery scheme as an implementation enhancement for now, until we see how it plays out. > 2. Queue full - Is a SCSI layer thing. I think that passing over MaxCmdRN > should have the command discarded with iSCSI protocol error (or something > of this nature) It sounds like the MaxCmdRN should eliminate the need to send Queue Full most (if not all) of the time. This should take care of any concerns about the host's SCSI-layer implementations. > 2. I can reinstate the bit. But control commands can have data and in this > sense are writes. > R/W was meant only to indicate the meaning of ExpectedDataLength (to > avoid scaning the CDB) We shouldn't need to reinstate the bit; your description works just fine. > Regards, > Julo > > Mark Bakke on 29/06/2000 16:13:32 > > Please respond to Mark Bakke > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Digest Login, iSCSI Discovery, and Other Draft 03 Comments > > Here are some non-connection-recovery-related thoughts on Draft 03; > mainly concerning digest logins and iSCSI target discovery. > > Logins for iSCSI draft 03 might benefit from the following additions: > > 1. Version number text fields should be placed in both login requests > and responses. the request could either specify the version of the > protocol the initiator wishes to run, or perhaps a list of versions > that the initiator is capable of running. > > Version: 1.0 > > -OR- > > Version: 1.0 1.1 1.2 > > The target would then choose the highest version that it supported > in common with those requested, and return it in the response: > > Version: 1.1 > > 2. Login with username and password. It seems that the scheme already > proposed (a digest scheme) would work quite well; we can re-use the > text headers defined for HTTP/1.1, with some simplification (see > RFC 2617). > > The initiator would send the initial login message, which would be > failed by the target with status == 2 (additional authentication > required; just like HTTP 401). The following text fields would be > added to the response by the target: > > Authenticate: Digest > Nonce: text-string (hex or uuencode of random value) > > The Nonce is used to provide a temporary value with which to hash > the username and password, preventing replay of the password on > another TCP connection. The nonce is randomly generated by the > target, and should be thrown out after being used to validate the > login for a particular connection. > > The initiator would then concatenate the nonce, user-name, and > password, and compute their MD5 hash. It would send a new login > request to the target: > > Authenticator: Digest user-name hex-hashed-response > Nonce: (echo of hex-string) > > Where hex-hashed-response > = md5(concat(nonce, ":", user-name, ":", password)) > > Note that if a given nonce value is only used on a single > connection, we really don't have to echo it here as they > do in HTTP/1.1. > > 3. Underlying encryption mechanisms could be either negotiated by > name at login, or could just be assumed on different TCP ports; > CIFS and telnet are done over SSL without modification to their > protocols; perhaps IPSec is similar? > > For the most part, I think that any of these mechanisms supported > by the target and initiator, or transparently by network security > devices would work with the protocol, and that there shouldn't > be a lot to specify here. > > However, it might be worth thinking about how user-names and > passwords are tied in to those lower levels. > > -------------------------------------------------------- > > iSCSI Target and LUN Discovery > > 1. The target named "iSCSI-SYS" in Draft 03 looks good. > > 2. We would like to add the sendTargets text command - this is a > value-add command for NuSpeed, but would be a good addition to > the spec, as it eliminates most of the need to configure the > iSCSI driver in the initiator. Targets that do not wish to > report this information can deny the command. Normally, only > the targets to which an initiator should have access are > returned. Here's the text command we use now. We log into > iSCSI-SYS, use sendTargets to find the targets, then log into > each target. > > The command: > > com.nuspeed.sendTargets:please > > The response is a list of "target:" > > target:jbod_one > target:tape_three > ... > > We would rename this command to just sendTargets. > > 3. Async notification for a new LUN in an existing, connected target > is in the draft, but there's no notification for a newly-created > target. This notification would work with our sendTargets > mechanism, and would probably just define an event type to > ask the initiator to re-request the target list. > > -------------------------------------------------------- > > Flow Control > > 1. Right now we are sending Queue Full back to handle flow control > from the iSCSI target; someone indicated that this may break > some host SCSI implementations. Is this correct? Anyway, will > the MaxCmdRN scheme handle this effectively? > > 2. MaxCmdRN has found its way into quite a few response headers. This > probably does not need to be updated that often, most likely when > watermarks are reached. Costa's separate message looks like would > be cleaner. > > -------------------------------------------------------- > > Data and Command CRC (or other checksum/hash/whatever) > > - More to come; we would like to see these negotiated at login, and > have separate CRCs for iSCSI headers and actual data. > > -------------------------------------------------------- > > Some additional notes on iSCSI draft 03. > > 1. Immediate Data - this is a nice optimization for small data messages, > since they cut down on data headers, and eliminate another message > to parse on the target. However, using this for large data can get > around a target that wants to use RTT. Should RTT be required to > used immediate data, or should the length of immediate data be > limited to some fixed (or negotiated) maximum (a few kbytes)? > > 2. The Write bit is gone from the header; clearing both the Read and > Write bits implied a Control message before. Without the Write > bit, I assume that an expected data length of zero would indicate > a Control message instead. > > -- > Mark A. Bakke > NuSpeed, Inc. > mark.bakke@nuspeed.com > 763.398.1054 -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Fri Jun 30 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26174; 30 Jun 2000 15:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA01916 for ips-outgoing; Fri, 30 Jun 2000 14:41:36 -0400 (EDT) Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA01912 for ; Fri, 30 Jun 2000 14:41:33 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id LAA14997 for ; Fri, 30 Jun 2000 11:41:14 -0700 (PDT) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id LAA03943 for ; Fri, 30 Jun 2000 11:37:30 -0700 (PDT) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Fri, 30 Jun 2000 11:37:30 -0700 (PDT) From: Costa Sapuntzakis To: ips@ece.cmu.edu Subject: More notes from Haifa Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a rewrite of some previous notes that were sent out. I believe they corresponded to Tuesday, June 20th, discussion. -Costa There was a review of the requirements document. Randy has incorporated the feedback into the version 0.2 draft of the requirements. ---------- Action Item: Add pointer to IPv6 URL naming to iSCSI spec ---------- What does it take to make a iSCSI host WAN-happy? - need large buffers to support retransmissions, fill pipelines - smaller buffer degrade performance but not correctness - support NATs and not challenge firewalls ---------------------- There was some discussion about frame synchronization in the iSCSI protocol. Randy Haagens thought the mechanism of the length fields scheme was somewhat fragile on long-lived connections. There were two issues: detecting loss of synchronization and finding synchronization. Luciano Dalle Ore pointed out that a CRC or well known magic number in the packet could be used to confirm synchronization (or loss of it). As for resynchronizing, three proposals were discussed: - having each iSCSI PDU be a fixed length (never lose syncrhonization, in theory) - using special byte sequence to denote packet bounadry (with byte stuffing of course) - augment TCP to have message boundaries ------------ Large discussion on iSCSI framing and CRCs Main issues on CRCs and reliability - Do nothing - TCP-layer CRC (CRC on datagram) - iSCSI layer CRC Higher layer CRCs were seen as being able to detect the most kinds of errors. Unfortanately, any CRC higher than the TCP segment may require doing another pass after data has been put into memory, costing main memory bandwidth and performance. The iSCSI layer CRC was proposed in a couple different forms: covering iSCSI messages, every n kilobytes in the iSCSI stream, or covering only iSCSI data. Luciano Dalle Ore pointed out that IPSec provides a very strong CRC with its authentication header (MD5 checksum). A well-known key (say 0) could be used to avoid the need to do key exchange for authentication. Consensus: IPSec for extra integrity ------ An optimization to iSCSI was discussed. The suggestion was that each TCP segment be the start of a new iSCSI PDU. This would be especially valuable for data transfers, as each segment would have enough information to place the data in memory at the remote end. From owner-ips@ECE.cmu.edu Fri Jun 30 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26309; 30 Jun 2000 17:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA04952 for ips-outgoing; Fri, 30 Jun 2000 16:07:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA04948 for ; Fri, 30 Jun 2000 16:07:45 -0400 (EDT) Received: from rtl.rose.hp.com (rtl.rose.hp.com [15.43.208.243]) by atlrel1.hp.com (Postfix) with ESMTP id B5F02F7E; Fri, 30 Jun 2000 16:07:43 -0400 (EDT) Received: from agilent.com (ros54259wak.rose.hp.com [15.43.208.219]) by rtl.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id NAA27120; Fri, 30 Jun 2000 13:07:42 -0700 (PDT) Message-ID: <395CFDE6.47142021@agilent.com> Date: Fri, 30 Jun 2000 13:07:03 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 Subject: New contact info Content-Type: multipart/mixed; boundary="------------9AC48DE2E5EF6E3FC59EDE27" To: undisclosed-recipients:; Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------9AC48DE2E5EF6E3FC59EDE27 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Matt Wakeley Engineer/Architect Agilent Technologies Roseville Tachyon Lab 1101 Creekside Ridge Drive Suite 100 P.O. Box 619051 Roseville, CA 95661-9051 Work: 916-788-5670 Fax: 916-788-5907 email: matt_wakeley@agilent.com --------------9AC48DE2E5EF6E3FC59EDE27 Content-Type: text/x-vcard; charset=us-ascii; name="matt_wakeley.vcf" Content-Transfer-Encoding: 7bit Content-Description: Card for Matt Wakeley Content-Disposition: attachment; filename="matt_wakeley.vcf" begin:vcard n:Wakeley;Matt tel;fax:916-788-5907 tel;work:916-788-5670 x-mozilla-html:FALSE org:Agilent Technologies;Roseville Tachyon Lab version:2.1 email;internet:matt_wakeley@agilent.com title:Engineer/Architect adr;quoted-printable:;;1101 Creekside Ridge Drive Suite 100=0D=0AM/S ????=0D=0AP.O. Box 619051;Roseville;CA;95661-9051;USA x-mozilla-cpt:;27088 fn:Matt Wakeley end:vcard --------------9AC48DE2E5EF6E3FC59EDE27-- From owner-ips@ECE.cmu.edu Fri Jun 30 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26305; 30 Jun 2000 17:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA05052 for ips-outgoing; Fri, 30 Jun 2000 16:10:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA05047 for ; Fri, 30 Jun 2000 16:10:24 -0400 (EDT) Received: from sunmail1.Sun.COM ([129.145.1.2]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA25540 for ; Fri, 30 Jun 2000 13:10:23 -0700 (PDT) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.85.31]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id NAA08220 for ; Fri, 30 Jun 2000 13:10:23 -0700 (PDT) Received: from terra.eng.sun.com (terra.Eng.Sun.COM [129.146.86.99]) by jurassic.eng.sun.com (8.10.2+Sun/8.10.2) with ESMTP id e5UKALj653419 for ; Fri, 30 Jun 2000 13:10:21 -0700 (PDT) Received: (from brent@localhost) by terra.eng.sun.com (8.9.3+Sun/8.9.3) id NAA28279 for ips@ece.cmu.edu; Fri, 30 Jun 2000 13:10:15 -0700 (PDT) Date: Fri, 30 Jun 2000 13:10:15 -0700 (PDT) From: Brent Callaghan Message-Id: <200006302010.NAA28279@terra.eng.sun.com> To: ips@ece.cmu.edu Subject: Re: More notes from Haifa X-Sun-Charset: US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > An optimization to iSCSI was discussed. The suggestion > was that each TCP segment be the start of a new iSCSI PDU. > This would be especially valuable for data transfers, as > each segment would have enough information to place the data > in memory at the remote end. It's also good if you're trying to write an iSCSI protocol interpreter for a wire sniffer like tcpdump or snoop. The interpreter doesn't have to sync up with the stream - or maintain connection state. Brent From owner-ips@ECE.cmu.edu Fri Jun 30 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26318; 30 Jun 2000 17:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA05891 for ips-outgoing; Fri, 30 Jun 2000 16:37:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA05885 for ; Fri, 30 Jun 2000 16:37:31 -0400 (EDT) Received: from rtl.rose.hp.com (rtl.rose.hp.com [15.43.208.243]) by palrel3.hp.com (Postfix) with ESMTP id D19F23DB; Fri, 30 Jun 2000 13:37:30 -0700 (PDT) Received: from agilent.com (ros54259wak.rose.hp.com [15.43.208.219]) by rtl.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id NAA24198; Fri, 30 Jun 2000 13:37:30 -0700 (PDT) Message-ID: <395D04E2.A1E7B94E@agilent.com> Date: Fri, 30 Jun 2000 13:36:50 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Costa Sapuntzakis Cc: ips@ece.cmu.edu Subject: Re: More notes from Haifa References: Content-Type: multipart/mixed; boundary="------------79BE6CFD6589DFD04D622935" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------79BE6CFD6589DFD04D622935 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit > An optimization to iSCSI was discussed. The suggestion > was that each TCP segment be the start of a new iSCSI PDU. > This would be especially valuable for data transfers, as > each segment would have enough information to place the data > in memory at the remote end. You didn't note the results of this discussion. I thought I remember something that this could not be enforced upon TCP. Something about it could "send" a segment whenever it wants (thus that segment would not have a header) and that the push bit wouldn't work either, because tcp could accept a few more bytes after the push bit. -Matt --------------79BE6CFD6589DFD04D622935 Content-Type: text/x-vcard; charset=us-ascii; name="matt_wakeley.vcf" Content-Transfer-Encoding: 7bit Content-Description: Card for Matt Wakeley Content-Disposition: attachment; filename="matt_wakeley.vcf" begin:vcard n:Wakeley;Matt tel;fax:916-788-5907 tel;work:916-788-5670 x-mozilla-html:FALSE org:Agilent Technologies;Roseville Tachyon Lab version:2.1 email;internet:matt_wakeley@agilent.com title:Engineer/Architect adr;quoted-printable:;;1101 Creekside Ridge Drive Suite 100=0D=0AM/S ????=0D=0AP.O. Box 619051;Roseville;CA;95661-9051;USA x-mozilla-cpt:;27088 fn:Matt Wakeley end:vcard --------------79BE6CFD6589DFD04D622935-- From owner-ips@ECE.cmu.edu Fri Jun 30 23:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26472; 30 Jun 2000 18:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA07922 for ips-outgoing; Fri, 30 Jun 2000 17:51:02 -0400 (EDT) Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA07914 for ; Fri, 30 Jun 2000 17:50:58 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id OAA09763 for ; Fri, 30 Jun 2000 14:50:39 -0700 (PDT) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id OAA04015 for ; Fri, 30 Jun 2000 14:46:55 -0700 (PDT) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Fri, 30 Jun 2000 14:46:54 -0700 (PDT) From: Costa Sapuntzakis To: ips@ece.cmu.edu Subject: Re: More notes from Haifa In-Reply-To: <395D04E2.A1E7B94E@agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk On Fri, 30 Jun 2000, Matt Wakeley wrote: > > An optimization to iSCSI was discussed. The suggestion > > was that each TCP segment be the start of a new iSCSI PDU. > > This would be especially valuable for data transfers, as > > each segment would have enough information to place the data > > in memory at the remote end. > > You didn't note the results of this discussion. I thought I remember > something that this could not be enforced upon TCP. Something about it could > "send" a segment whenever it wants (thus that segment would not have a header) > and that the push bit wouldn't work either, because tcp could accept a few > more bytes after the push bit. Your recollections match mine. Thanks for filling in the gap. --------------------------- The motivation for the discussion are 1) to describe a simpler fast-path for iSCSI, 2) minimize data buffering needs in the presence of out-of-order reception. The bulk of the traffic on an iSCSI TCP connection will be SCSI data. SCSI data can be delivered to the buffer at each end in pretty much any order, just as long as it is all delivered. So, as an optimization, instead of keeping tons of SCSI data in a TCP receive queue, an iSCSI w/optimized TCP could parse out-of-order segments and deliver the SCSI data in them to the right SCSI buffer. This would decrease the amount of buffering needed for TCP receive. In the limit, this optimization would decrease memory requirements by up to a factor of 2. As you point out, implementations can't rely on alignment in a TCP stream - to be interoperable they would have to implement unaligned parsing too. However, it would be reasonable to implement aligned parsing in the fast-path and kick unaligned parsing to the slow path. TCP stacks don't know the alignment requirements of applications because most applications fail to communicate message boundaries to the TCP stacks. Historically, this is because the TCP stack ignores message boundaries. However, it is possible to communicate message boundaries through the sockets interface! With the current TCP sockets interface, the sendmsg command, coupled with the MSG_EOR (end of record) flag, could be used to communicate iSCSI PDU boundaries to the TCP stack. The getopt command could be used to get the current path MTU of the connection. To communicate message boundaries over the wire, it would be expedient to add a bit to the TCP header saying: this segment is also the start of a new higher-layer PDU. This could help out-of-order parsing of the TCP stream. As for the PSH bit, RFC 1122, section 4.2.2.2, states "The PSH bit is not a record marker..." There are many constraints that can be envisioned 1) say nothing about alignment 2) iSCSI PDU headers do not span segments 3) if more than one iSCSI PDU header appears in a segment, one iSCSI PDU header always appears at the start of the segment 4) zero or one iSCSI PDU header per segment and header always aligned with start of segment. - potentially creates lots of small segments 5) new segment always starts new iSCSI PDU - implies iSCSI MTU <= TCP path MTU - creates lots of small segments My preference is #3. -Costa From owner-ips@ECE.cmu.edu Sat Jul 01 01:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26812; 30 Jun 2000 21:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA10625 for ips-outgoing; Fri, 30 Jun 2000 20:11:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA10617 for ; Fri, 30 Jun 2000 20:10:58 -0400 (EDT) Received: from rtl.rose.hp.com (rtl.rose.hp.com [15.43.208.243]) by palrel3.hp.com (Postfix) with ESMTP id 5E39312AB for ; Fri, 30 Jun 2000 17:10:57 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by rtl.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id RAA27908 for ; Fri, 30 Jun 2000 17:10:56 -0700 (PDT) Message-ID: <395D370F.3155BD68@agilent.com> Date: Fri, 30 Jun 2000 17:10:55 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: More notes from Haifa References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa Sapuntzakis wrote: > On Fri, 30 Jun 2000, Matt Wakeley wrote: > > > > An optimization to iSCSI was discussed. The suggestion > > > was that each TCP segment be the start of a new iSCSI PDU. > > > This would be especially valuable for data transfers, as > > > each segment would have enough information to place the data > > > in memory at the remote end. > > > > You didn't note the results of this discussion. I thought I remember > > something that this could not be enforced upon TCP. Something about it could > > "send" a segment whenever it wants (thus that segment would not have a header) > > and that the push bit wouldn't work either, because tcp could accept a few > > more bytes after the push bit. > > Your recollections match mine. Thanks for filling in the gap. > > --------------------------- > > The motivation for the discussion are 1) to describe > a simpler fast-path for iSCSI, 2) minimize data buffering needs in > the presence of out-of-order reception. > > The bulk of the traffic on an iSCSI TCP connection will be SCSI data. > SCSI data can be delivered to the buffer at each end in pretty much > any order, just as long as it is all delivered. So, as an > optimization, instead of keeping tons of SCSI data in a TCP receive > queue, an iSCSI w/optimized TCP could parse out-of-order segments > and deliver the SCSI data in them to the right SCSI buffer. Yes - but only if there is a iSCSI header in the frame received. If there is no header, it can't be determined where to put the data. Yes, it's possible to detect that this tcp segment is part of an iSCSI message *if* the header was previously received by inspecting the length field in that header and placing the tcp segment in the proper buffer. But this does not cover *all* cases. So, in the case where the frame with the iSCSI header is lost (for example, a 64k data pdu), now all that 64K of data must be buffered and copied. It would be *so* much easier if each tcp segment contained an iSCSI header immediately after the TCP header, and each TCP segment fit into exactly one ethernet frame. This is basically how Fibre Channel works. A frame comes in, and from the information in the FC header, the payload can be placed into memory. There is also the issue of "getting out of sync". I know that this "should not happen" in a good implementation, but there are lots of people nervous about this issue. Also, as someone else posted, if every ethernet frame contained an iSCSI header, it would be real easy to implement a protocol analyzer. How is a protocol analyzer supposed to figure out what's in a TCP stream? These issues are a real detriment to the use of TCP as an iSCSI transport. The benefits of TCP are it's "reliable" delivery and congestion avoidance. The disadvantage is there is no "framing". I hate to say it, but if another transport was utilized (say UDP), the same proven algorithms that make TCPs reliablity and congestion control could be implemented in the iSCSI layer along with the benefits of UDP framing. (I wonder why all the proprietary start-ups seem to be using UDP...?) TCP is great for things like telnet and email, but I'm less and less convinced it's appropriate for mass storage. > This would > decrease the amount of buffering needed for TCP receive. In the > limit, this optimization would decrease memory requirements by up to a > factor of 2. > > As you point out, implementations can't rely on alignment in a TCP stream > - to be interoperable they would have to implement unaligned parsing > too. However, it would be reasonable to implement aligned parsing > in the fast-path and kick unaligned parsing to the slow path. > > TCP stacks don't know the alignment requirements of applications > because most applications fail to communicate message boundaries to the > TCP stacks. Historically, this is because the TCP stack ignores > message boundaries. > > However, it is possible to communicate message boundaries through > the sockets interface! With the current TCP sockets interface, the sendmsg > command, coupled with the MSG_EOR (end of record) flag, could be used to > communicate iSCSI PDU boundaries to the TCP stack. The getopt > command could be used to get the current path MTU of the connection. > > To communicate message boundaries over the wire, it > would be expedient to add a bit to the TCP header saying: > this segment is also the start of a new higher-layer PDU. This > could help out-of-order parsing of the TCP stream. Ok, but would this require a "change/enhancement" to the existing TCP standard? > > > As for the PSH bit, RFC 1122, section 4.2.2.2, states "The PSH bit is not > a record marker..." > > There are many constraints that can be envisioned > > 1) say nothing about alignment > 2) iSCSI PDU headers do not span segments With existing TCP implementations, how can this be enforced? > 3) if more than one iSCSI PDU header appears in a segment, one > iSCSI PDU header always appears at the start of the segment See #2. > 4) zero or one iSCSI PDU header per segment and header always > aligned with start of segment. > - potentially creates lots of small segments > > 5) new segment always starts new iSCSI PDU > - implies iSCSI MTU <= TCP path MTU > - creates lots of small segments > > My preference is #3. My preference is #5. But the problem is comment to #2. > > > -Costa -Matt From owner-ips@ECE.cmu.edu Sat Jul 01 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26904; 30 Jun 2000 22:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA11892 for ips-outgoing; Fri, 30 Jun 2000 21:21:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA11888 for ; Fri, 30 Jun 2000 21:21:42 -0400 (EDT) Received: from sunmail1.Sun.COM ([129.145.1.2]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id SAA16663 for ; Fri, 30 Jun 2000 18:21:41 -0700 (PDT) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.84.31]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id SAA10671 for ; Fri, 30 Jun 2000 18:21:42 -0700 (PDT) Received: from terra.eng.sun.com (terra.Eng.Sun.COM [129.146.86.99]) by jurassic.eng.sun.com (8.10.2+Sun/8.10.2) with ESMTP id e611Lej709461 for ; Fri, 30 Jun 2000 18:21:40 -0700 (PDT) Received: (from brent@localhost) by terra.eng.sun.com (8.9.3+Sun/8.9.3) id SAA28525 for ips@ece.cmu.edu; Fri, 30 Jun 2000 18:21:32 -0700 (PDT) Date: Fri, 30 Jun 2000 18:21:32 -0700 (PDT) From: Brent Callaghan Message-Id: <200007010121.SAA28525@terra.eng.sun.com> To: ips@ece.cmu.edu Subject: Re: More notes from Haifa X-Sun-Charset: US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt Wakely writes: > These issues are a real detriment to the use of TCP as an iSCSI > transport. The benefits of TCP are it's "reliable" delivery and > congestion avoidance. The disadvantage is there is no "framing". In practice, this needn't be a problem. The NFS protocol started out as a UDP protocol, but now more commonly runs over TCP connections because TCP is so much better at managing congestion and reliable delivery. NFS is built on an RPC protocol that assumes discrete messages being passed back and forth on the TCP streams. The RPC messages are mapped onto the stream with a simple record marking protocol (see RFC 1831 Section 10. "Record Marking Standard"). The record marking is simple, each message has a 32 bit value prepended that gives the length of the message. That's it. TCP's reliable delivery makes certain that there's never a sync problem. If the TCP connection is lost, for whatever reason, then you a bit of higher level code to retransmit any unacknowledged message. This works very well for NFS - clients and servers crash, connections are lost, but there's never a sync problem. Brent From owner-ips@ECE.cmu.edu Sat Jul 01 05:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27160; 1 Jul 2000 0:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA14035 for ips-outgoing; Fri, 30 Jun 2000 23:34:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA14031 for ; Fri, 30 Jun 2000 23:34:42 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id DD85D1505 for ; Fri, 30 Jun 2000 20:34:41 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id UAA12142; Fri, 30 Jun 2000 20:34:39 -0700 (PDT) Message-ID: <395D6723.AB31D18A@hp.com> Date: Fri, 30 Jun 2000 20:36:03 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: "sip@cup.hp.com" , ips@ece.cmu.edu Subject: Synchronization problem and TCP big window problem Content-Type: multipart/mixed; boundary="------------8FDD8B8CB93EFA0D4E972EC7" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------8FDD8B8CB93EFA0D4E972EC7 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit --------------8FDD8B8CB93EFA0D4E972EC7 Content-Type: text/plain; charset=us-ascii; name="mail" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="mail" Hello, This mail is about three problems outlined in the recent discussions : a) the lost of synchronization in the TCP byte stream b) in case of lost or out of order packet, the extra delay in the command completion on the initiator side. This leading to block the initiator because its command window is closed. c) the quantity of TCP dedicated storage needed to cope with a full TCP compliant implementation on a fast link with a long round trip time. About c) -------- The problem is that the quantity of memory needed is unlimited. More the link is fast, more the target is far from the initiator and more TCP dedicated storage is needed. Some calculations several did showed how big can be this memory and we don't know where we are going in the future the link being faster and faster. This TCP dedicated memory is needed to cope with the out of order or lost datagrams. To have good performance one want to use SACK. If a datagram is lost, the receive side have to store all the byte stream incoming through the TCP pipe until the send side re-transmit. This byte stream even if acknowledged with SACK has to be stored in a temporary TCP dedicated buffer. This is because iSCSI can't process it. iSCSI lost the synchronization due to the missing datagram. The quantity of memory needed depends on the RTT and on the link speed hence it can be very big and will get bigger and bigger in the future. Hence a large memory buffer will be needed just to handle error cases. Proposition to solve these problems. =================================== Add a "pad" command in iSCSI. This pad command is only one byte: the opcode. How does it works? ------------------ At the login time the initiator and the target agree on two synchronization periods (SPEs). One for each direction. A synchronization period is a number of bytes that separates two synchronization points (SPO). At each SPO the sender guarantees that it will put the beginning of an iSCSI header, eventually adding some padding before with the pad command. The SPE value is implementation dependent and could be determined based on the memory capacity of the receiver. Shorter the SPE is, and less memory the receiver needs to handle lost or out order datagrams. On the receiver side when a hole in the TCP data stream occurs (datagram lost), the receiver continues to SACK the incoming data stream and store it in a TCP dedicated buffer up to the next SPO. Then, from the SPO it can start again to interpret the data stream and process it. It stops copying in the TCP dedicated buffer. That means for example, in case of WRITE data, copy the data on disk in case of READ data, copy it into the host reception buffer, in case of command completion, do the cleanup and so on. When it receives the missing datagram it empties the TCP dedicated buffer. For example, if the receiver can store up to 5Mbytes of TCP dedicated memory per TCP connexion it could choose a SPE of 5Mbytes. In case of bad quality line, if its dedicated memory get full (because it got other holes in the data stream after the re-synchronization and the first holes have not been filled in by the sender), it drops everything new it receives till it gets the missing datagrams. Advantages of this proposal =========================== 1) Reduce the memory needed on the receive side while maintaining good performance 2) Cap the memory needed for TCP even with long RTT and increasing bandwidth 3) Allow synchronization check each SPE 4) Negligeable loss of bandwidth (padding) Regards, Pierre --------------8FDD8B8CB93EFA0D4E972EC7-- From owner-ips@ECE.cmu.edu Sat Jul 01 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27432; 1 Jul 2000 2:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA16127 for ips-outgoing; Sat, 1 Jul 2000 01:40:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA16122 for ; Sat, 1 Jul 2000 01:39:57 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA29114 for ; Sat, 1 Jul 2000 07:39:12 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA206362 for ; Sat, 1 Jul 2000 07:39:26 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690F.001F1237 ; Sat, 1 Jul 2000 07:39:22 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 1 Jul 2000 08:37:16 +0300 Subject: notes from the June 29 - phone conference Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Participants (I don't have all the names as it was a rather long call - 5:30 hours! - and people kept coming and leaving) according to the initial roll call: Gabi Hecht (GH) and Lani Brauer (spell?) - Gadzoox Luciano Dalle Ore (LO) - Quantum Matt Wakeley (MW) - Agilent Brian Berg (BB) - Berg Software Dave Nagle (DN) - CMU Howard Hall (HH) Pirrus Networks Kalman Meth (KM) IBM Alain Azagury (AA) IBM John Hufferd (JH) IBM Meir Toledano (MT) IBM Daniel Smith (DS) IBM Randy Haagens (RH) HP Costa Sapuntzakis (CS) CISCO Ed Gardner (EG) Ophidian Mark Bakke (MB) NuSpeed (together with two colleagues) Julian Satran (JS) IBM The following items where on the agenda: - review the working draft for iSCSI - review the requirements document - a short statement by JS urging all participants to consider Kalman Meth memo about asymmetric use of connections within a session - administrivia Review the working draft: JS outlined the major areas of change and the rationale behind them as follows: ordering per session as agreed during the meeting in Haifa. The document now considers 3 phases in the "life" of a command - emission, execution and termination; an initiator task tag is associated with the command for the whole life. During emission commands are sequentially numbered by the initiator and a complete sliding window mechanism is provided do enable a target to use ordered delivery (a target is not required though to do so). During execution commands are identified solely by the Initiator tag; the initiator can reuse a the reference numbers as soon as they are acked. On termination the responses are numbered by the target iSCSI and sliding window is used - except that the window size; window size for statuses is always the maximum allowed by the sequence numbers. Concern was expressed by many that the CmdRN current size - 16bit - might be two small for the future networks and data rates. After some debate a consensus was reached to use 32 bit counts; although that might be an overshoot for a long time it is hard to correct if we don't reserve the space. Another concern voiced was that if the counters are raised to 32 bits then a 32 bit tag is inadequate (JS, MW). The majority opinion was that as the 32bit counters will not get fully used for a long time 32bit tags will be adequate - and that is the current consensus. The counters will be carried by the headers (piggybacked); we will consider a new PDU only for the CmdRN window (if needed) and the headers will be increased adequately. Concern was raised about the need to support multiple connections where multiple session will suffice (JH, DS); it was pointed out that a target is free to reject any connection but the first. connection recovery - is based on the assumption that commands can be restarted; the target will request/provide the missing data. For the case that command reexecution will have to be communicated to the target SCSI layer JS thinks that SAM is vague about it is probably supported well by most interesting applications; interacting with T10 on a better specified mechanism could lead to further optimization. Remarkably little discussion around the CID - the connection identifier. RH expressed concern about the need to close explicitly a connection. That is being taken care by a CID login with a recovery option that can be rejected. RH also expressed concern about the fact that there is no clear interface for recovery between iSCSI and SCSI and we might have to invent the exchange mechanism. It was pointed out (?) that we have already the exchange mechanism. A personal note - SAM is vague about who does what in this area; one assumes that DMA's are done by the delivery mechanism and the rest by some SCSI mechanism. The issue might be more academic than practical - whetter the interface between SCSI and iSCSI can specify recovery. A consensus was reached to simplify command recovery by having the initiator indicate by a flag bit that the command is a resent command so that only for those will have the target to look in it's tables for an old match - if any. MB went over the way the numbering schemes will be used to achieve recovery both for commands that did not make it to the target; if AckCmdRN and MaxCmdRN are the lower and upper boundary of the current window then numbers are important for commands between AckCmdrN and MaxCmdRN during recovery - as the order for those might not have been established by the target. For commands outside this range the numbers are irrelevant as the target has already established the execution order for them. The security text will be sent by LO to JS to include in the draft (during weekend?) Review of the requirements draft: RH took us over the changes. JS expressed concern about explicitly excluding classes of devices beyond saying that resources are needed (CP and memory). It was also agreed to require the protocol to support multiple connections but simple targets can reject it and initiators do not have to use it. It was also agreed that since connection allegiance is part of the solution and not a requirement to remove it from requirements and allow an open discussion on the issue of allegiance, symmetry etc. Administrative etc. EG suggested that whatever presentation we make at IETF we should also take T10 through it to keep them informed and influence thinking - i.e. give a talk at their next meeting. He mentioned that T10 has already 2 liaisons to the not yet formed IETF IPS working group - Charles Monia and Gary Robinson (spelling?). JS mentioned that this is an issue our WG chairmen would like to handle. The only active mailing list is THIS ONE (scsi-tcp is decommissioned). We will publish working versions of the draft often this week and point you to them. We will have a (final) phone conference next Thursday 8AM PDT - 4-5 hours. RH will host it. Julo Julian Satran IBM Research Laboratory in Haifa From owner-ips@ECE.cmu.edu Sat Jul 01 09:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27982; 1 Jul 2000 5:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA17856 for ips-outgoing; Sat, 1 Jul 2000 04:08:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA17852 for ; Sat, 1 Jul 2000 04:08:14 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA93752 for ; Sat, 1 Jul 2000 10:07:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id KAA182556 for ; Sat, 1 Jul 2000 10:07:42 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125690F.002CA2AA ; Sat, 1 Jul 2000 10:07:32 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 1 Jul 2000 11:06:12 +0300 Subject: Re: More notes from Haifa Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree that it would be nice to have to have all the good things of both TCP and UDP. But there are good reasons why this is unlikely to happened even if we start redoing TCP over UDP. For a long time we had assumed that a TCP option (like and expanded RDMA) that adds message boundaries to bulk data can solve most of the issues we talking about on this thread - effects packet loss on memory requirements and throughput. At a certain point we even considered of making this option part of iSCSI - as a mandated segment header (i.e. build the TCP segment payload segment at a time). I wonder if this or another equivalent scheme might not fit what we are looking for. Julo Matt Wakeley on 01/07/2000 03:10:55 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: More notes from Haifa Costa Sapuntzakis wrote: > On Fri, 30 Jun 2000, Matt Wakeley wrote: > > > > An optimization to iSCSI was discussed. The suggestion > > > was that each TCP segment be the start of a new iSCSI PDU. > > > This would be especially valuable for data transfers, as > > > each segment would have enough information to place the data > > > in memory at the remote end. > > > > You didn't note the results of this discussion. I thought I remember > > something that this could not be enforced upon TCP. Something about it could > > "send" a segment whenever it wants (thus that segment would not have a header) > > and that the push bit wouldn't work either, because tcp could accept a few > > more bytes after the push bit. > > Your recollections match mine. Thanks for filling in the gap. > > --------------------------- > > The motivation for the discussion are 1) to describe > a simpler fast-path for iSCSI, 2) minimize data buffering needs in > the presence of out-of-order reception. > > The bulk of the traffic on an iSCSI TCP connection will be SCSI data. > SCSI data can be delivered to the buffer at each end in pretty much > any order, just as long as it is all delivered. So, as an > optimization, instead of keeping tons of SCSI data in a TCP receive > queue, an iSCSI w/optimized TCP could parse out-of-order segments > and deliver the SCSI data in them to the right SCSI buffer. Yes - but only if there is a iSCSI header in the frame received. If there is no header, it can't be determined where to put the data. Yes, it's possible to detect that this tcp segment is part of an iSCSI message *if* the header was previously received by inspecting the length field in that header and placing the tcp segment in the proper buffer. But this does not cover *all* cases. So, in the case where the frame with the iSCSI header is lost (for example, a 64k data pdu), now all that 64K of data must be buffered and copied. It would be *so* much easier if each tcp segment contained an iSCSI header immediately after the TCP header, and each TCP segment fit into exactly one ethernet frame. This is basically how Fibre Channel works. A frame comes in, and from the information in the FC header, the payload can be placed into memory. There is also the issue of "getting out of sync". I know that this "should not happen" in a good implementation, but there are lots of people nervous about this issue. Also, as someone else posted, if every ethernet frame contained an iSCSI header, it would be real easy to implement a protocol analyzer. How is a protocol analyzer supposed to figure out what's in a TCP stream? These issues are a real detriment to the use of TCP as an iSCSI transport. The benefits of TCP are it's "reliable" delivery and congestion avoidance. The disadvantage is there is no "framing". I hate to say it, but if another transport was utilized (say UDP), the same proven algorithms that make TCPs reliablity and congestion control could be implemented in the iSCSI layer along with the benefits of UDP framing. (I wonder why all the proprietary start-ups seem to be using UDP...?) TCP is great for things like telnet and email, but I'm less and less convinced it's appropriate for mass storage. > This would > decrease the amount of buffering needed for TCP receive. In the > limit, this optimization would decrease memory requirements by up to a > factor of 2. > > As you point out, implementations can't rely on alignment in a TCP stream > - to be interoperable they would have to implement unaligned parsing > too. However, it would be reasonable to implement aligned parsing > in the fast-path and kick unaligned parsing to the slow path. > > TCP stacks don't know the alignment requirements of applications > because most applications fail to communicate message boundaries to the > TCP stacks. Historically, this is because the TCP stack ignores > message boundaries. > > However, it is possible to communicate message boundaries through > the sockets interface! With the current TCP sockets interface, the sendmsg > command, coupled with the MSG_EOR (end of record) flag, could be used to > communicate iSCSI PDU boundaries to the TCP stack. The getopt > command could be used to get the current path MTU of the connection. > > To communicate message boundaries over the wire, it > would be expedient to add a bit to the TCP header saying: > this segment is also the start of a new higher-layer PDU. This > could help out-of-order parsing of the TCP stream. Ok, but would this require a "change/enhancement" to the existing TCP standard? > > > As for the PSH bit, RFC 1122, section 4.2.2.2, states "The PSH bit is not > a record marker..." > > There are many constraints that can be envisioned > > 1) say nothing about alignment > 2) iSCSI PDU headers do not span segments With existing TCP implementations, how can this be enforced? > 3) if more than one iSCSI PDU header appears in a segment, one > iSCSI PDU header always appears at the start of the segment See #2. > 4) zero or one iSCSI PDU header per segment and header always > aligned with start of segment. > - potentially creates lots of small segments > > 5) new segment always starts new iSCSI PDU > - implies iSCSI MTU <= TCP path MTU > - creates lots of small segments > > My preference is #3. My preference is #5. But the problem is comment to #2. > > > -Costa -Matt From owner-ips@ECE.cmu.edu Sat Jul 01 13:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28334; 1 Jul 2000 9:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA10359 for ips-outgoing; Sat, 1 Jul 2000 07:55:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA10355 for ; Sat, 1 Jul 2000 07:55:03 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id HAA00629; Sat, 1 Jul 2000 07:54:59 -0400 (EDT) Date: Sat, 1 Jul 2000 07:54:59 -0400 (EDT) From: Scott Bradner Message-Id: <200007011154.HAA00629@newdev.harvard.edu> To: ips@ece.cmu.edu Subject: level setting Cc: mankin@ISI.EDU Sender: owner-ips@ece.cmu.edu Precedence: bulk a word from one of the Area Directors to remind the working group of some basic rules IETF protocols must be able to operate in the Internet that means they can not assume they will always operate in some private network behind a firewall and with guaranteed overprovisioning that means that they must to deal with congestion in a TCP-friendly way that means that they must have good security Scott From owner-ips@ECE.cmu.edu Mon Jul 03 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01807; 3 Jul 2000 13:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06950 for ips-outgoing; Mon, 3 Jul 2000 12:12:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA06946 for ; Mon, 3 Jul 2000 12:12:17 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA44654 for ; Mon, 3 Jul 2000 18:11:18 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA98640 for ; Mon, 3 Jul 2000 18:11:45 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256911.0058F5A3 ; Mon, 3 Jul 2000 18:11:40 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 3 Jul 2000 19:10:44 +0300 Subject: working draft Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, A working version of the ID is on: http://www.haifa.il.ibm.com/satran/iSCSI-Working-Draft.txt All the 32 bit counters are in and the header (as predicted) is now 48 bytes. The text part is still pretty much in flux. The commands and all the related stuff are in a better shape. I haven't got the logon-security stuff yet and I am running out of time. PLEASE SEND IT! Before Thursday I will post at least one other version and will correct the dates. Please send comments. Regards, Julo Julian Satran - IBM Research Laboratory in Haifa From owner-ips@ECE.cmu.edu Mon Jul 03 22:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02288; 3 Jul 2000 18:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA14173 for ips-outgoing; Mon, 3 Jul 2000 17:12:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA14162 for ; Mon, 3 Jul 2000 17:12:02 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 31FF6837F3 for ; Mon, 3 Jul 2000 16:09:59 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N5PCKXZG; Mon, 3 Jul 2000 16:17:52 -0500 Message-ID: <396101C0.A3703C15@nuspeed.com> Date: Mon, 03 Jul 2000 16:12:32 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Some thoughts on synchronization Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk After watching the synchronization discussion for a few days, I'm wondering if this isn't starting to get too complicated for our own good. Basically, iSCSI uses TCP connections in almost exactly the same way as HTTP/1.1 and CIFS. The only ways I could come up with where sync would be lost are: 1. A bit error happens in a length field (or another field controlling the length of a header or data), and is not caught by a data-link layer CRC or by the TCP checksum. If an error is detected at either of these levels, the layer-2 frame or TCP packet is discarded, and allowed to retransmit, so detected errors do not cause data integrity or iSCSI framing problems. 2. The initiator is not formatting messages correctly, or is expecting incorrect message formats. 3. The target is not formatting messages correctly, or is expecting incorrect message formats. 2 and 3 have to be solved by interoperability testing anyway; detecting these problems should cause some sort of warning via the initiator's and/or target's management interfaces. An undetected error can happen in a few ways: - A set of bit errors within a layer-2 network happens that makes an L2 frame still have the same CRC, and at the same time makes the overall TCP segment have the same checksum. This is extremely unlikely, but possible. - Bit errors happen on a router or other device that strips and regenerates a layer-2 CRC, and at the same time keeps the same TCP segment checksum. This is probably more likely that the previous case, since only one of the checks has to fail in its duty. - Bit errors happen in a bus, switched fabric, or buffer memory in either the initiator or target, without being detected by whatever scheme (parity, ECC, etc.) is in use. This will likely have a probability similar to the bit-error-in-a-router case. These are all cases that should rarely happen, although their probabilities depend heavily on the equipment and services purchased by the end customer. However, they can happen, and in some cases it's important enough to deal with the possibility. If we end up with these types of bit errors, corrupting length fields, and messing up iSCSI framing, is probably the least (or at least just one of) our worries. For example, a read command could become a write, or the wrong data could be written to a database block, or the block address could be wrong that's read or written, or ... Anyway, ALL of these cases are equally bad. I see a few things that are reasonable to do. 1. If either the initiator or target detects a framing problem (length fields out-of-range, unexpected values in a particular field, etc), it should proceed to shut down the connection, and allow the initiator-based connection recovery to take place and initiate a new connection in its place. Since these types of errors should not happen more than once in a while (once per day seems like alot of undetected errors), connection recovery will make use of a mechanism we already have, instead of adding another one. If a framing problem is detected in this way, chances are that the previous iSCSI header was corrupted, and we skipped too many or too few bytes to find the next one. In this case, damage may have been done, and either end of the implementation detecting this should attempt to warn the user through its normal management interfaces. 2. A magic number added to the header could help detect framing problems. This makes error checking on iSCSI headers much stronger. Again, however, the liklihood is that the damage was actually to the previous iSCSI header, and other damage may have already been done as a result. Again, the implementation should warn like crazy. 3. If many of these errors are seen in a real customer environment, or if the application absolutely cannot tolerate a chance of error, then TCP stream resync is definitely too weak anyway. In this case, the implementation should provide a data integrity check, such as a CRC, on iSCSI headers, commands, and data, where CRC errors on these will also cause a connection recovery sequence. I believe that these three suggestions, taken together, would solve our synchronization problems without extra effort (other than connection recovery and data integrity check mechanisms, which we were going to do anyway). Any other thoughts? -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Tue Jul 04 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03523; 4 Jul 2000 4:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA22500 for ips-outgoing; Tue, 4 Jul 2000 02:47:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA22496 for ; Tue, 4 Jul 2000 02:47:42 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel1.hp.com (Postfix) with ESMTP id 29C59460; Mon, 3 Jul 2000 23:47:41 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id XAA05476; Mon, 3 Jul 2000 23:48:09 -0700 (PDT) Message-ID: <3961888C.BC1CF7C1@agilent.com> Date: Mon, 03 Jul 2000 23:47:41 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu, scsi-tcp@external.cisco.com Subject: Re: single vs multiple channels for iSCSI commands References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk meth@il.ibm.com wrote: > Proposal to support single Control Channel with multiple Data Channels > in the iSCSI protocol. How would the requirement to send immediate data with commands be supported? (This would be Randy H's requirement to send write data without a RTT). We already agreed that immediate write data would be part of the command message. This would mean that all immediate write commands (with immediate data) would be over the same channel. So, then how would the striping across multiple channels be utilized? Oh, well send the data on another channel you say - ok, but what happens when the data arrives before the command? From owner-ips@ECE.cmu.edu Tue Jul 04 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03525; 4 Jul 2000 4:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA22852 for ips-outgoing; Tue, 4 Jul 2000 03:16:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA22848 for ; Tue, 4 Jul 2000 03:16:06 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA302828; Tue, 4 Jul 2000 09:15:27 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id JAA73772; Tue, 4 Jul 2000 09:15:35 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256912.0027DCF4 ; Tue, 4 Jul 2000 09:15:24 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Matt Wakeley , ips@ece.cmu.edu Message-ID: Date: Tue, 4 Jul 2000 10:14:50 +0300 Subject: Re: single vs multiple channels for iSCSI commands Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, We have to distinguish between the cases when the target and initiator have agreed to use RTT or have agreed to not use RTT. When using RTT, the target will have prepared the buffers before issuing the RTT. If the target has agreed to not use RTT, that means the target agrees to provide spare buffers to store whatever data may arrive on any of its data channels before the corresponding commands arrive. When the command arrives on the control channel, the target will be informed over which channel the data is being sent for that command, and the target will be able to match up the corresponding data buffers (including the data buffers that arrived before the command arrived). If the target is very limited in resources, then it shouldn't agree to work without RTT. If for some reason the target agreed to work without RTT and it runs out of buffers, we allow the target to discard data from its early arrival buffers and request that data to be resent later on via an RTT. In any case, the initiator may send immediate data over the control channel in the same packet with the command. This should only be used for small amounts of data (say up to 8K). - Kalman Matt Wakeley on 04/07/2000 09:47:41 Please respond to Matt Wakeley To: ips@ece.cmu.edu, scsi-tcp@external.cisco.com cc: (bcc: Kalman Meth/Haifa/IBM) Subject: Re: single vs multiple channels for iSCSI commands meth@il.ibm.com wrote: > Proposal to support single Control Channel with multiple Data Channels > in the iSCSI protocol. How would the requirement to send immediate data with commands be supported? (This would be Randy H's requirement to send write data without a RTT). We already agreed that immediate write data would be part of the command message. This would mean that all immediate write commands (with immediate data) would be over the same channel. So, then how would the striping across multiple channels be utilized? Oh, well send the data on another channel you say - ok, but what happens when the data arrives before the command? From owner-ips@ECE.cmu.edu Wed Jul 05 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05958; 5 Jul 2000 2:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA18892 for ips-outgoing; Wed, 5 Jul 2000 01:27:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA18888 for ; Wed, 5 Jul 2000 01:27:32 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA71510 for ; Wed, 5 Jul 2000 07:26:51 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA60550 for ; Wed, 5 Jul 2000 07:27:00 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256913.001DEEB5 ; Wed, 5 Jul 2000 07:26:56 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 5 Jul 2000 08:23:28 +0300 Subject: Re: Some thoughts on synchronization & Other outstanding questions Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Very good points Mark. The only exception I would take is that if would be nice if we would have a way not to force things to pile up on an adapter because some router handling QoS flows has decided to drop a packet from an iSCSI stream. And yes there are many good solutions to it - some where already discussed here. To those I can add one only to illustrate how trivial it could be to add synch. to a stream - you can have a special frame issued in the stream every so many bytes or time or both and the framing for it will be escaped if it appears in the regular data stream (i.e. hardware or software will delete it before it reaches SCSI. It will contain the distance to the previous and next SCSI header. This is the textbook solution for resynch and it is very old. However ... I consider it a bad solution for our environment as it will not operate on plain vanilla TCP/IP adapters that have only TCP assists but nothing special for iSCSI. That is why a solution based on a simple TCP options (like the RDMA or even only a TCP option indicating how far it is until the next/previous header will help). RDMA s also great as it allows you to place the bulk data in memory regardless of order. With all those said I would like to remind you all that we are close to the end of the cycle and as we are running out of time - at least for the this IETF session. I suggest that we deal with the following three outstanding issues in this order: 1-ordered delivery & recovery over multiple connections 2-security (I am still missing the text!) 3-third party commands and the related naming and mapping issues 4-synchronization I think that the posted draft has 1 fixed (some minor corrections might reach it today) I am busy with 3 (hope to get it until tomorrow - as I promised) I am lost on 2 (any help appreciated). We have to get in the text the agreements reached in Haifa. I am not sure that we will be able to do something different from what we have on 4 until the IETF deadline. Regards, Julo Mark Bakke on 04/07/2000 00:12:32 Please respond to Mark Bakke To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Some thoughts on synchronization After watching the synchronization discussion for a few days, I'm wondering if this isn't starting to get too complicated for our own good. Basically, iSCSI uses TCP connections in almost exactly the same way as HTTP/1.1 and CIFS. The only ways I could come up with where sync would be lost are: 1. A bit error happens in a length field (or another field controlling the length of a header or data), and is not caught by a data-link layer CRC or by the TCP checksum. If an error is detected at either of these levels, the layer-2 frame or TCP packet is discarded, and allowed to retransmit, so detected errors do not cause data integrity or iSCSI framing problems. 2. The initiator is not formatting messages correctly, or is expecting incorrect message formats. 3. The target is not formatting messages correctly, or is expecting incorrect message formats. 2 and 3 have to be solved by interoperability testing anyway; detecting these problems should cause some sort of warning via the initiator's and/or target's management interfaces. An undetected error can happen in a few ways: - A set of bit errors within a layer-2 network happens that makes an L2 frame still have the same CRC, and at the same time makes the overall TCP segment have the same checksum. This is extremely unlikely, but possible. - Bit errors happen on a router or other device that strips and regenerates a layer-2 CRC, and at the same time keeps the same TCP segment checksum. This is probably more likely that the previous case, since only one of the checks has to fail in its duty. - Bit errors happen in a bus, switched fabric, or buffer memory in either the initiator or target, without being detected by whatever scheme (parity, ECC, etc.) is in use. This will likely have a probability similar to the bit-error-in-a-router case. These are all cases that should rarely happen, although their probabilities depend heavily on the equipment and services purchased by the end customer. However, they can happen, and in some cases it's important enough to deal with the possibility. If we end up with these types of bit errors, corrupting length fields, and messing up iSCSI framing, is probably the least (or at least just one of) our worries. For example, a read command could become a write, or the wrong data could be written to a database block, or the block address could be wrong that's read or written, or ... Anyway, ALL of these cases are equally bad. I see a few things that are reasonable to do. 1. If either the initiator or target detects a framing problem (length fields out-of-range, unexpected values in a particular field, etc), it should proceed to shut down the connection, and allow the initiator-based connection recovery to take place and initiate a new connection in its place. Since these types of errors should not happen more than once in a while (once per day seems like alot of undetected errors), connection recovery will make use of a mechanism we already have, instead of adding another one. If a framing problem is detected in this way, chances are that the previous iSCSI header was corrupted, and we skipped too many or too few bytes to find the next one. In this case, damage may have been done, and either end of the implementation detecting this should attempt to warn the user through its normal management interfaces. 2. A magic number added to the header could help detect framing problems. This makes error checking on iSCSI headers much stronger. Again, however, the liklihood is that the damage was actually to the previous iSCSI header, and other damage may have already been done as a result. Again, the implementation should warn like crazy. 3. If many of these errors are seen in a real customer environment, or if the application absolutely cannot tolerate a chance of error, then TCP stream resync is definitely too weak anyway. In this case, the implementation should provide a data integrity check, such as a CRC, on iSCSI headers, commands, and data, where CRC errors on these will also cause a connection recovery sequence. I believe that these three suggestions, taken together, would solve our synchronization problems without extra effort (other than connection recovery and data integrity check mechanisms, which we were going to do anyway). Any other thoughts? -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Wed Jul 05 08:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06264; 5 Jul 2000 4:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA20303 for ips-outgoing; Wed, 5 Jul 2000 03:19:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA20299 for ; Wed, 5 Jul 2000 03:19:01 -0400 (EDT) From: hufferd@us.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id CAA20780; Wed, 5 Jul 2000 02:12:19 -0500 Received: from d53mta08h.boulder.ibm.com (d53mta08h.boulder.ibm.com [9.99.142.8]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id BAA26966; Wed, 5 Jul 2000 01:19:00 -0600 Received: by d53mta08h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256913.00282CDE ; Wed, 5 Jul 2000 01:18:49 -0600 X-Lotus-FromDomain: IBMUS To: scsi-tcp@external.cisco.com, ips@ece.cmu.edu Message-ID: <87256913.00282C38.00@d53mta08h.boulder.ibm.com> Date: Wed, 5 Jul 2000 00:17:33 -0700 Subject: RE: single vs multiple channels for iSCSI commands Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I as I said in Haifa, agree with one iSCSI session per TCP connection (at least to start with, because it is simpler). However, I would not like this to get in the way of our IETF deadlines. As long as a Target can respond to a Login Message -- that it can only take one TCP connection per iSCSI session, then this may be an approprate subset. I, therefore, would like to see that type of a target response included in our spec. (The Initiator can always just send one connection per session, but the target needs to express its limits.) With this capability, the folks that think that multiple connections per session are most important for their boxes, can ensure that they have all the problems solved with that approach, while the others can go ahead with the simpler approach. As I also said in Haifa, the only required "in order" action is on a single (virtual perhaps) LU. This applies equally to Disk as well as Tape. Therefore, it is not required that any special commands be created for tape. It is just required that all the Commands that are needed to be delivered in order, be sent on the same Session. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 Jim McGrath on 06/28/2000 02:42:55 PM To: "'Costa Sapuntzakis'" , scsi-tcp@external.cisco.com, ips@ece.cmu.edu cc: Subject: RE: single vs multiple channels for iSCSI commands The problem is that while such a solution may allow you to close on a protocol faster, it does not insure that you get products to market faster. Specifically, requiring the tape people to add new commands introduces more development time, and perhaps more importantly creates a division of interest. That is, the people responsible for supporting tape commands are not the same people doing iSCSI, so you might create a classic chicken and egg problem (no tape support for iSCSI until it gets to be important - and iSCSI market development is hindered by the lack of tape support). Mind, I am not thrilled about the alternative either. But solving the problem by creating new work for people "not in the room" may result in a pyrrhic victory. Jim PS in my opinion the only way to really solve this sort of problem is to identify your application space (or spaces) to allow for a concrete discussion of tradeoffs. For instance, initially FC ignored tape, focusing on disk, and got FC done more quickly - but eventually had to retrofit tape support. This may have been the best overall path, since the disk drive application was more important to the market than the tape drive application. However, for iSCSI (especially operated over a WAN) tape may be more important - this is the sort of conversation you have to have before making decisions that trade off the speed of adoption in different markets. -----Original Message----- From: Costa Sapuntzakis [mailto:csapuntz@cisco.com] Sent: Tuesday, June 27, 2000 8:26 PM To: scsi-tcp@external.cisco.com; ips@ece.cmu.edu Subject: Re: single vs multiple channels for iSCSI commands I would like to stake out a position against any approach that involves using multiple simultaneous TCP connections per iSCSI session. The design of iSCSI (and the hardware that implements it) becomes significantly simpler when there is just 1 TCP connection/session. Parallel paths in the network can still be exploited by setting up multiple sessions with multiple TCP connections. So far, I have heard of two applications that allegedly require strict ordering: tape backup and remote asynchronous mirroring. New SCSI commands or even higher-layer techniques can be used to satisfy these applications. For example, I have heard that some high-bandwidth tape applications blast self-describing blocks of data to tape, making the order in which data gets read/written to tape less relevant. For the case of high bandiwdth remote asynchronous mirroring, a special SCSI remote asynchrnous mirroring command set could be introduced. I believe remote asynchronous mirroring principally consists of one command: WRITE, so the command set would probably be small. Ditching multiple simultaneous connections/session would help convince us of the correctness of iSCSI and allow us to move onto other important issues and help the time to market. -Costa App A. Possible remote asynch mirroring command sets 1) 1 CDB: ORDERED WRITE 2) 3 CDBs: WRITE UNCOMMITTED, COMMIT, ABORT From owner-ips@ECE.cmu.edu Wed Jul 05 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07048; 5 Jul 2000 11:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA03535 for ips-outgoing; Wed, 5 Jul 2000 09:45:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA03525 for ; Wed, 5 Jul 2000 09:45:23 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA116620 for ; Wed, 5 Jul 2000 15:44:31 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id PAA279424 for ; Wed, 5 Jul 2000 15:44:52 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256913.004B8340 ; Wed, 5 Jul 2000 15:44:48 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 5 Jul 2000 16:43:47 +0300 Subject: working draft Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, A slightly corrected working version of the ID is on: http://www.haifa.il.ibm.com/satran/iSCSI-Working-Draft0705.txt The "reply" bit was moved to the command code - thanks to Meir Toledano that reminded me that the original reason for having the response bit there is valid for the reply too : make it easy to read on a log or a logic analyzer. I haven't got the logon-security stuff yet and I am running out of time. PLEASE SEND IT! Before Thursday I will post at least one other version and will correct the dates. Please send comments. Regards, Julo Julian Satran - IBM Research Laboratory in Haifa From owner-ips@ECE.cmu.edu Wed Jul 05 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07368; 5 Jul 2000 14:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA09454 for ips-outgoing; Wed, 5 Jul 2000 12:57:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA09450 for ; Wed, 5 Jul 2000 12:57:36 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA274746 for ; Wed, 5 Jul 2000 18:56:31 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA121712 for ; Wed, 5 Jul 2000 18:57:05 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256913.005D1B44 ; Wed, 5 Jul 2000 18:56:58 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 5 Jul 2000 19:55:57 +0300 Subject: working draft Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, Now there are two versions of draft out there: the slightly corrected working version of the ID is still on: http://www.haifa.il.ibm.com/satran/iSCSI-Working-Draft0705.txt A newer version: http://www.haifa.il.ibm.com/satran/iSCSI-Working-Draft0706.txt is also out. It includes the third party commands mechanisms (mapping of protocol specific addresses to SCSI standard addresses and a description of how it works). I did not do much editing on it (warm of the press!). I haven't got the logon-security stuff yet and I am running out of time. PLEASE SEND IT! Please send comments. Regards, Julo Julian Satran - IBM Research Laboratory in Haifa From owner-ips@ECE.cmu.edu Wed Jul 05 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07705; 5 Jul 2000 16:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA22347 for ips-outgoing; Wed, 5 Jul 2000 15:34:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA22343 for ; Wed, 5 Jul 2000 15:34:43 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5712B1277 for ; Wed, 5 Jul 2000 12:32:09 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id NAA24056 for ; Wed, 5 Jul 2000 13:32:08 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Wed, 05 Jul 2000 13:32:08 -0600 (Mountain Daylight Time) Received: by xboibrg2.boi.hp.com with Internet Mail Service (5.5.2650.21) id ; Wed, 5 Jul 2000 13:32:07 -0600 Message-ID: <8F889D13E8D4D311BC510090278CE32D0E4525@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Tomorrow's iSCSI teleconf Date: Wed, 5 Jul 2000 13:32:06 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFE6B7.B4210610" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFE6B7.B4210610 Content-Type: text/plain; charset="iso-8859-1" Thursday 6 Jul, 8:00 - 13:00 Pacific Time Dial-in: +1 612 315 6802 (international) or +1 877 612 3185 ID: 854578 You will hear music only until I login. I'll try to be on time. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 <> ------_=_NextPart_000_01BFE6B7.B4210610 Content-Type: application/octet-stream; name="Haagens, Randy (E-mail).vcf" Content-Disposition: attachment; filename="Haagens, Randy (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000607T194543Z END:VCARD ------_=_NextPart_000_01BFE6B7.B4210610-- From owner-ips@ECE.cmu.edu Thu Jul 06 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08640; 6 Jul 2000 1:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA08204 for ips-outgoing; Thu, 6 Jul 2000 00:46:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA08199 for ; Thu, 6 Jul 2000 00:46:31 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel1.hp.com (Postfix) with ESMTP id 1CDC8EAA; Wed, 5 Jul 2000 21:46:30 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id VAA10907; Wed, 5 Jul 2000 21:46:58 -0700 (PDT) Message-ID: <39640F26.B3579F63@agilent.com> Date: Wed, 05 Jul 2000 21:46:30 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: working draft: comments to 0706 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Comments to draft 0706: Section 2.2.2: The "retry" bit is introduced here, but it's use is not really described anywhere else in the document. "NOOP" commands are introduced, but these are typically described as "NOP" or "NO-OP". Section 2.2.6: Don't forget to describe how an IPv6 address would be used. Section 3.2.6: You need to change "40-byte" to "48-byte". Section 3.2.7: The length of the CDB algorithm should be: if (I==0) then cdb_length = 16+ length else cdb_length = 16 This also means that the new "object oriented" CDBs will never be able to have immediate write data. Section 3.3 In the reserved area bytes 40-47, the "SNS_LEN" and "RSP_LEN" fields should be included, as they are in FCP-2. Section 3.3.3 I don't understand the text in this section. Typically, MaxCmdRN will be greater than the current value of MaxCmdRN, and certainly greater than ExpCmdRN. Was this a cut-and-paste error? Section 3.4 ExpStatRN should be at offset 28 to be consistent with the other message headers. Section 3.10 The Data Packet for READ should not have the Transfer Tag field (should be reserved). Section 3.17 The description and use of the "map command" needs a lot more work. Section 6.1 RFC1122 says that TCP checksum checking is *required*. So the "weakening" or "nullifying" of the TCP checksum text must be removed. This is another bummer with TCP - audio or video distribution streams will not be able to make use of iSCSI with TCP due to the retry mechanisms of TCP. That's all for now... -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Thu Jul 06 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08653; 6 Jul 2000 1:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA07532 for ips-outgoing; Thu, 6 Jul 2000 00:05:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA07528 for ; Thu, 6 Jul 2000 00:05:30 -0400 (EDT) Received: from amrelay1.boi.hp.com (amrelay1.boi.hp.com [15.56.8.24]) by atlrel2.hp.com (Postfix) with ESMTP id 8A997A2F for ; Thu, 6 Jul 2000 00:05:29 -0400 (EDT) Received: from xboibrg1.boi.hp.com (xboibrg1.boi.hp.com [15.56.8.167]) by amrelay1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id WAA24103 for ; Wed, 5 Jul 2000 22:05:27 -0600 (MDT) Received: from 15.56.8.167 by xboibrg1.boi.hp.com (InterScan E-Mail VirusWall NT); Wed, 05 Jul 2000 22:05:27 -0600 (Mountain Daylight Time) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2650.21) id ; Wed, 5 Jul 2000 22:05:27 -0600 Message-ID: <8F889D13E8D4D311BC510090278CE32D0E452A@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: iSCSI Requirements v0.3 05 Jul 00 Date: Wed, 5 Jul 2000 22:05:24 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFE6FF.695C2A40" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFE6FF.695C2A40 Content-Type: text/plain; charset="iso-8859-1" Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 <> <> ------_=_NextPart_000_01BFE6FF.695C2A40 Content-Type: application/octet-stream; name="Haagens, Randy (E-mail).vcf" Content-Disposition: attachment; filename="Haagens, Randy (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture TITLE:Manager, Storage Network Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-3586 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:19990528T091551Z END:VCARD ------_=_NextPart_000_01BFE6FF.695C2A40 Content-Type: application/octet-stream; name="iSCSI Requirements v0.3.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="iSCSI Requirements v0.3.pdf" JVBERi0xLjINJeLjz9MNCjU2IDAgb2JqDTw8IA0vTGluZWFyaXplZCAxIA0vTyA1OCANL0ggWyAx MTg0IDM1MCBdIA0vTCA4MTMyNSANL0UgMTE5MzEgDS9OIDExIA0vVCA4MDA4NyANPj4gDWVuZG9i ag0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB4cmVmDTU2IDM1IA0wMDAwMDAwMDE2IDAwMDAwIG4NCjAwMDAwMDEwNDcgMDAwMDAgbg0KMDAw MDAwMTUzNCAwMDAwMCBuDQowMDAwMDAxNzQxIDAwMDAwIG4NCjAwMDAwMDE5MjMgMDAwMDAgbg0K MDAwMDAwMjEyMSAwMDAwMCBuDQowMDAwMDAyNzgxIDAwMDAwIG4NCjAwMDAwMDI5NzIgMDAwMDAg bg0KMDAwMDAwMzQ1NSAwMDAwMCBuDQowMDAwMDAzNjM1IDAwMDAwIG4NCjAwMDAwMDM2NTYgMDAw MDAgbg0KMDAwMDAwNDU2NiAwMDAwMCBuDQowMDAwMDA0NTg3IDAwMDAwIG4NCjAwMDAwMDU1ODYg MDAwMDAgbg0KMDAwMDAwNTYwNyAwMDAwMCBuDQowMDAwMDA2NDc2IDAwMDAwIG4NCjAwMDAwMDY2 NTYgMDAwMDAgbg0KMDAwMDAwNjg0NCAwMDAwMCBuDQowMDAwMDA2ODY1IDAwMDAwIG4NCjAwMDAw MDc3MTQgMDAwMDAgbg0KMDAwMDAwNzczNSAwMDAwMCBuDQowMDAwMDA4Njk3IDAwMDAwIG4NCjAw MDAwMDg4MDcgMDAwMDAgbg0KMDAwMDAwOTExMSAwMDAwMCBuDQowMDAwMDA5MTk0IDAwMDAwIG4N CjAwMDAwMDkzNzUgMDAwMDAgbg0KMDAwMDAwOTU1MCAwMDAwMCBuDQowMDAwMDA5NTcxIDAwMDAw IG4NCjAwMDAwMTAzNzggMDAwMDAgbg0KMDAwMDAxMDM5OSAwMDAwMCBuDQowMDAwMDExMDk2IDAw MDAwIG4NCjAwMDAwMTExMTcgMDAwMDAgbg0KMDAwMDAxMTY4MyAwMDAwMCBuDQowMDAwMDAxMTg0 IDAwMDAwIG4NCjAwMDAwMDE1MTMgMDAwMDAgbg0KdHJhaWxlcg08PA0vU2l6ZSA5MQ0vSW5mbyA1 MiAwIFIgDS9Sb290IDU3IDAgUiANL1ByZXYgODAwNzcgDS9JRFs8NzNiZTc2Zjc1NWJhMzVlYTk1 NTYzOWQyYjUzZDg0MmU+PDczYmU3NmY3NTViYTM1ZWE5NTU2MzlkMmI1M2Q4NDJlPl0NPj4Nc3Rh cnR4cmVmDTANJSVFT0YNICAgICANNTcgMCBvYmoNPDwgDS9UeXBlIC9DYXRhbG9nIA0vUGFnZXMg NTQgMCBSIA0vT3V0bGluZXMgMzcgMCBSIA0vT3BlbkFjdGlvbiBbIDU4IDAgUiAvWFlaIG51bGwg bnVsbCBudWxsIF0gDS9QYWdlTW9kZSAvVXNlTm9uZSANPj4gDWVuZG9iag04OSAwIG9iag08PCAv UyAyMTAgL08gMjgxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTAgMCBSID4+IA1zdHJl YW0NCkiJYmBgYGZgYFnAwMrAwM/FwMuAALwMLAxsQMzxwImBQZGB4bj3Ab8zB47lG1yt0eOO3fiW geFG14ppOxoQOhyOcQT7VSgcEUn2q1B0EzDurWNv4glsafzcCxV0OCaiKur74IiEYpyAybFQh1aT 5008jm4wEmgUs2loaAMDCDMYA7EgiKEEYgEZxi4eDKwRQHsElUCWgqQbMB0FBMoMjNqXgbQ8A9jh QCDKwKXXJXSsXmvOoxqGew78HgxXOD/NYTjPwLcl0u3wo4ZZDIYKTEsYlASentA5APOOKgOjXgWQ 5mdgYLsFEGAAty9L6A1lbmRzdHJlYW0NZW5kb2JqDTkwIDAgb2JqDTIzNyANZW5kb2JqDTU4IDAg b2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1MyAwIFIgDS9SZXNvdXJjZXMgNTkgMCBSIA0v Q29udGVudHMgWyA2NiAwIFIgNjggMCBSIDcwIDAgUiA3NCAwIFIgNzYgMCBSIDgzIDAgUiA4NSAw IFIgODcgMCBSIF0gDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYx MiA3OTIgXSANL1JvdGF0ZSAwIA0+PiANZW5kb2JqDTU5IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9Q REYgL1RleHQgXSANL0ZvbnQgPDwgL0YyIDc3IDAgUiAvVFQyIDYxIDAgUiAvVFQ0IDYzIDAgUiAv VFQ2IDcxIDAgUiAvVFQ4IDgxIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDg4IDAgUiA+PiAN L0NvbG9yU3BhY2UgPDwgL0NzNSA2NCAwIFIgPj4gDT4+IA1lbmRvYmoNNjAgMCBvYmoNPDwgDS9U eXBlIC9Gb250RGVzY3JpcHRvciANL0FzY2VudCA4OTEgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQg LTIxNiANL0ZsYWdzIDM0IA0vRm9udEJCb3ggWyAtMTg0IC0zMDcgMTA4OSAxMDI2IF0gDS9Gb250 TmFtZSAvVGltZXNOZXdSb21hblBTLUJvbGRNVCANL0l0YWxpY0FuZ2xlIDAgDS9TdGVtViAxMzMg DT4+IA1lbmRvYmoNNjEgMCBvYmoNPDwgDS9UeXBlIC9Gb250IA0vU3VidHlwZSAvVHJ1ZVR5cGUg DS9GaXJzdENoYXIgMzIgDS9MYXN0Q2hhciAxODIgDS9XaWR0aHMgWyAyNTAgMCA0MDggMCAwIDAg MCAxODAgMzMzIDMzMyAwIDAgMjUwIDMzMyAyNTAgMjc4IDUwMCA1MDAgNTAwIDUwMCANNTAwIDUw MCA1MDAgNTAwIDUwMCA1MDAgMjc4IDI3OCA1NjQgNTY0IDU2NCA0NDQgMCA3MjIgNjY3IDY2NyA3 MjIgDTYxMSA1NTYgNzIyIDcyMiAzMzMgMzg5IDcyMiA2MTEgODg5IDcyMiA3MjIgNTU2IDcyMiA2 NjcgNTU2IDYxMSANNzIyIDcyMiA5NDQgNzIyIDAgMCAzMzMgMCAzMzMgMCAwIDAgNDQ0IDUwMCA0 NDQgNTAwIDQ0NCAzMzMgNTAwIA01MDAgMjc4IDI3OCA1MDAgMjc4IDc3OCA1MDAgNTAwIDUwMCA1 MDAgMzMzIDM4OSAyNzggNTAwIDUwMCA3MjIgDTUwMCA1MDAgNDQ0IDAgMCAwIDAgMCAwIDAgMCAw IDAgMTAwMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAzMzMgDTQ0NCA0NDQgMCA1MDAgMCAwIDAg MCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCA1MDAgMCAwIDAgMCAwIDAgMCANMCAwIDAgMCAwIDAg MCA0NTMgXSANL0VuY29kaW5nIC9XaW5BbnNpRW5jb2RpbmcgDS9CYXNlRm9udCAvVGltZXNOZXdS b21hblBTTVQgDS9Gb250RGVzY3JpcHRvciA2MiAwIFIgDT4+IA1lbmRvYmoNNjIgMCBvYmoNPDwg DS9UeXBlIC9Gb250RGVzY3JpcHRvciANL0FzY2VudCA4OTEgDS9DYXBIZWlnaHQgMCANL0Rlc2Nl bnQgLTIxNiANL0ZsYWdzIDM0IA0vRm9udEJCb3ggWyAtMTY3IC0zMDcgMTAwOSAxMDA3IF0gDS9G b250TmFtZSAvVGltZXNOZXdSb21hblBTTVQgDS9JdGFsaWNBbmdsZSAwIA0vU3RlbVYgMCANPj4g DWVuZG9iag02MyAwIG9iag08PCANL1R5cGUgL0ZvbnQgDS9TdWJ0eXBlIC9UcnVlVHlwZSANL0Zp cnN0Q2hhciAzMiANL0xhc3RDaGFyIDEyMSANL1dpZHRocyBbIDI1MCAwIDAgMCAwIDAgMCAwIDMz MyAzMzMgMCAwIDI1MCAwIDI1MCAyNzggNTAwIDUwMCA1MDAgNTAwIDUwMCA1MDAgDTUwMCA1MDAg NTAwIDUwMCAzMzMgMCAwIDAgMCAwIDAgNzIyIDAgNzIyIDcyMiAwIDAgNzc4IDAgMzg5IDAgMCAN NjY3IDk0NCA3MjIgNzc4IDYxMSAwIDcyMiA1NTYgNjY3IDcyMiAwIDAgMCAwIDAgMzMzIDAgMzMz IDAgMCAwIA01MDAgNTU2IDQ0NCA1NTYgNDQ0IDMzMyA1MDAgNTU2IDI3OCAwIDU1NiAyNzggODMz IDU1NiA1MDAgNTU2IDU1NiANNDQ0IDM4OSAzMzMgNTU2IDUwMCA3MjIgMCA1MDAgXSANL0VuY29k aW5nIC9XaW5BbnNpRW5jb2RpbmcgDS9CYXNlRm9udCAvVGltZXNOZXdSb21hblBTLUJvbGRNVCAN L0ZvbnREZXNjcmlwdG9yIDYwIDAgUiANPj4gDWVuZG9iag02NCAwIG9iag1bIA0vQ2FsUkdCIDw8 IC9XaGl0ZVBvaW50IFsgMC45NTA1IDEgMS4wODkgXSAvR2FtbWEgWyAyLjIyMjIxIDIuMjIyMjEg Mi4yMjIyMSBdIA0vTWF0cml4IFsgMC40MTI0IDAuMjEyNiAwLjAxOTMgMC4zNTc2IDAuNzE1MTkg MC4xMTkyIDAuMTgwNSAwLjA3MjIgMC45NTA1IF0gPj4gDQ1dDWVuZG9iag02NSAwIG9iag04MzIg DWVuZG9iag02NiAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDY1IDAgUiA+ PiANc3RyZWFtDQpIiYxUTW+cMBC98yvmaEuB2AbMIlU9NKmaRKoUJdyqHBCwG9oku2VJk/z7vvGY 3bbKoULCxp558+aD96lJTpvGkaVmndRZ7cngCZvS+Kx2pqR8lZXOwOIxOT3bl9Ttg5GhfZecfrm1 tNknJjOmoKZLUt7B9iVRu4ysbr4nn5vE0khJmdvMFJbKHHhksspRaiveT0OyTj41/xCoAVUYTxU4 IDYDu4pjYGddwTG+qcsnvcoqNQ+TbIZZp6vMq1RbkxXqXDs1tTq1NsvVWs5gkdsC6F7d6Eq12mQr Jd49LEuYvGnHh3TRthuBjRBittd3zVViQD+rypqacyHnIjljVkLuQz+1a41kENPhnd7D3SkBdRFU YjuAph5LGuxHndaZxVmJ+06WcfoZ6NXqEQ7AXDxeg7+8M50y8/l1/qhTZ3ObI52LJdKLxnepHgKb YZ4lioS81mmFpe1+CMmppzPtYb1FveCb/ZF0US1Jl4emO8n586u2Vu1GUK7UNOw1d4P6nr5ysBLc waTC4hiUnDHgloPFG/jmReFxym1QXDrruLcBin4x7qDZdML05CdkSj6iq2deHoLVG8PaAGveb5I/ NClOECdZq/vAUyalQEdSrtsmHA4YEO4515pg4BAgjFKuJs3oXCH0hXNz7A2qRz/maqmdBpIQIzLI 1UY+Hkae1ioGkrM5GAw9xbsnqeB8P75LoY3sUh6RGGIrPmEWQIeaWGK5HgRnoO5/k56GGOxBuHXR Y6Y+kuqC+bN8yFsSXUoTsxiXbCS3SIjaPlZnKxy6Y+9s6f1h2PLjtOXSPUy0F79Q7B86/KYoned8 TKhtOKK4Ttqq7bPsd0stMZQoCLTDL67hfnG1NeQkV4h5FT2jARkjvwVEtBARtS4oGBYHofMGAuZ5 KlasYenfI8g7TgIxC3Tvlv/nQp3F9ZJuhp+yfZYl2k3DoyjZ8CTn834h8a6SL0JartCXRUrLo1wv Wspjwl2+16G7I9rIElQfRu5FuxXih2L7WGzMVz+I5q51mDLxe5LLeBVQ2mO5XcESo1mMdrvosIlw 24izWN/qCqTOIFJV3F9qy/dz8ItsaGEvhten4fLy+kTbglH13W8BBgAJZIN+DWVuZHN0cmVhbQ1l bmRvYmoNNjcgMCBvYmoNOTIxIA1lbmRvYmoNNjggMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNv ZGUgL0xlbmd0aCA2NyAwIFIgPj4gDXN0cmVhbQ0KSIlsVctu2zoQ3esrZikCFUtSIiUt+wiKFChQ IALuoumClpVaiGMJtBq3f3+HM5QdJ4UBm495nJk5h+6+ZhoUfo59plvZ1rbFTfc5U1Ip00LXZ7g9 ZT/y/rdoZJuHMBxosYhSunwvikZW+V9hdA7iZ/c1u+ky07TorcE2jawNlGgBhZJVA2HIHrKPXYaZ HKWlRWlqWSnl0MFJA91TplZMRcThCAYuWobyeBCFNvkkClNJnZ+ElWXOZ+BFUSO6I94hLhhFES/v 8NKh6Sf8LWUd99HzFrcKr+WK3JZaqkqDUxqBqIi+0LW05oz8fddVoKF7eFVDi/iqWIJrpGuweKri 0r67HuG6fBYa0Q2UD0MZDmWkVe7S98pGRypdW3aXAP8NMAehjcTK5+mIQ9D5AAsWT63Q0uIEsAXb 4YGtRro6CCzRYkqD4wIPsU3o+SRMiZV7Op5n7JOTzWr8i39gSpFWpzvhsKnYxBIPeXMrVDSYw5SQ 9NOeFv9A1uEa+4oBjMIufH9PFrffKRBwRas9e+8YiOcdrPkbhHqV/piSBZ+gD9BPqRi+CgkWf3M3 wpHmUHD7Ma+uKncmf7OyThmewX0+h/FAyXsRCxhnvycB2CiAeALb8RgZ5/JHsgPP9ltY/DyAD8H/ pYMj13y+3o+b4MM48Pm9gD5dwQbdlsX3O9oOGGkCbBmyvMyBbQZCszCSkzCOWkFUX4Fw3HfJYeIJ Ln7zCn6a85fxF9l5CrwZF7hJ0XcvEoY1O091oWYq6qN15z5W5YXM5drIL/R4bG6wUE9jKrEHWvEK +Bbpy0RkllaIQ9A084jGMjdKGmUVsfCO7u9zrVKUTeRTnWMmuYou6ZeQxvfuSnVJrt+mZXz2iUAj UulasJWsq5oF+1amQldRakmhw2MSE/MwCe1JUM+Yh3s+wyGwRrfTvBq/kNDhtSj9v1T3Rtf0BqyC TvNNEVLgAJt9irJiZexvRTU9p1VADsLV08LGJ6FtJFh45BDc9IL79YoZxp4Vltr3QSBlUV39FLbj IXLQ8j9Mwwx1+TtRvCzkxA6Rj+58uCM/T7S0+TPtkkXPd1OUg07MjZ1taEDTJayxSHCSCfaHlNSS kt6kHgUpiUHuUiD+HWBN8If2I2dNtuyYisSXDQXt8GmLRLi7xZciQR2XoU9RFvL4TccRUyycXxCL 6on/Lzq+PE+iRTP+9lcVbrkehnHx9KnFfJ6KPhchfv4vwAAnJNc+DWVuZHN0cmVhbQ1lbmRvYmoN NjkgMCBvYmoNNzkxIA1lbmRvYmoNNzAgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xl bmd0aCA2OSAwIFIgPj4gDXN0cmVhbQ0KSIlUVEtv2zAMvudXED1JQ+1afsU5btklAwYUiHcqenBt NfGQ2JmtrOu/Hx+SgyBALIrkx48Psf6xqr+skjhJ8hzqdhXR0eDxY/Wi7D9tkjhXvS5io2ZdKaej Mt74i+Egaqh1ZEycqa2u4rV6fmLt7hkcHyad4H8ziDGhXMZJVHDiT6PTuFCfOqWPZWGKAb6NYnX0 Ye4kS0AWLpM3GtvxtNCEZrKApFKUP3S6jitPubMnyeBTUCLNNh3axqWyF4EYUcwwkztGHYQUuhu0 KdBM9OLq4T3uVRyGzk5SPUYfO0zul6YIdOuLKRQO+rVeWpKk1BI6pZV0xB11GacYkQhW6qzNRgU2 Z02RSYXAjbcY2H6G5t6q544Zrw6S6K46KhibBBhFetfr+BZqWFhgKKyCwfyD7h767x1fuXM9FZjw RH7UUYpKCMndU/UEZgdTc5FzjxO1oaaRAP4y2M39m5xObGWh6e4MvDmTKBZOgUMjYsBeKB34O+ko R2K2cRb5tCENboPHbZzPP9DoT324+uTmJoBlM3leQv1dmrsObU4yaTOWlF5dhrODvhXOxZGfFuwG Z6eBz9bhlLfH3tnWXflm8lP8yK+CHsSMrDPKLcEP5UZmHdg/4iARkDiOcY5ZpIanXR6zGLv4xnld bALlZKGcesq0AwpF5UJq0O+3+510JiNeGfIaRXLaYDAvtEQ5DdKJVdDTuFJNkU0jBh7Jf3q2G0Q4 cEjwEO/IGiH2Otrgd6sLhTycV4KQpNtnMeC2Z77ta2k7iq13kNtZ58S7xzKT6GSaM35VDzyrldrj DigVJe0mDzZ7tuNdFVzwgGYGH89KksI9JMeFFEWg9cbzltPyqpbpJKvbWBXlMlbFss9TadGe39NW bxBLzjt8Ili7/VfO8CdvrbVKoRvb65meNYLj0jO8udDwJVhytikOWgzZI6Cbeqgpb8P7F3V+xaPP 8ySoo8h+T1NVUwr+QPNdqFeZs6e6zsFA/b4yKST4w88GU8iTEvIM64d78Iyp0vBRUsro+jd5leKV xAU6YQFwHYki/y/AANzNjsYNZW5kc3RyZWFtDWVuZG9iag03MSAwIG9iag08PCANL1R5cGUgL0Zv bnQgDS9TdWJ0eXBlIC9UcnVlVHlwZSANL0ZpcnN0Q2hhciAzMiANL0xhc3RDaGFyIDMyIA0vV2lk dGhzIFsgMjc4IF0gDS9FbmNvZGluZyAvV2luQW5zaUVuY29kaW5nIA0vQmFzZUZvbnQgL0FyaWFs LUJvbGRNVCANL0ZvbnREZXNjcmlwdG9yIDcyIDAgUiANPj4gDWVuZG9iag03MiAwIG9iag08PCAN L1R5cGUgL0ZvbnREZXNjcmlwdG9yIA0vQXNjZW50IDkwNSANL0NhcEhlaWdodCAwIA0vRGVzY2Vu dCAtMjExIA0vRmxhZ3MgMzIgDS9Gb250QkJveCBbIC0xNjcgLTM3NiAxMDc1IDEwNDggXSANL0Zv bnROYW1lIC9BcmlhbC1Cb2xkTVQgDS9JdGFsaWNBbmdsZSAwIA0vU3RlbVYgMTMzIA0+PiANZW5k b2JqDTczIDAgb2JqDTc3MSANZW5kb2JqDTc0IDAgb2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2Rl IC9MZW5ndGggNzMgMCBSID4+IA1zdHJlYW0NCkiJpFTBcpswEL37K/YozRgsCYHtY5OmM+mh05kw vTQ5YExSJsR4ZJJM/r5PWmETt82lB2BhV6v3nt5Cmsr7mUpNTorKz4iUMvhWz36KT/t919YyT9ei 2rSdXCNoB7xr8Sbvyq+zRVkaCg2QKtBAUQjW6GJVQVZjraHyaaZ8R987o/IVrUuZILcUTiY2zUQl tULXrUwK7ND2/rmTBb4nGpt2HZ4adW9Sr1A3l0km6AUFoXLMPvuXJ1mgb8MLF5uQD1X1Y8CsA8xD PcMSk2cT1ssR4whyIZNVWogDOmaiqYfe0a34cSGTXNxIi8ytDC2vyplRJtXQzQKftZTn/p6o1K7I NbP72UV5JpHJl2lenGkUkQWdjEeTBDiG4SRepLWnbNKVcG2zw4kYMTRb8hBXYpCZz7vqISQaqvtY EhKu77rGcSmINCmXpXPatofHEFPlnNdcoeRNGi/qB03mNFT7ZpQgzzROHRJkyhNS6dJQosHS/EOC o0uUTW0QAGStPvLWmnl30Hol2nDf8Lm6ygGe8Yg9vGCeIRQAX3jyvZG+BnCX8MStJKzGOYlfvKIK 2Rd+4dKxglc8c2q/790w6bclfottGrqBEy/hSVjCh9cS+I0/L42qjCHT3vW8rK8jxjlVEfuWPoC0 aZpYRhV34HtVjwBGQH3cqvaBxShAorXYP0fwUYf36F3//BDDM1rhYBXOMNXWFuOY2Ox0QHFOwBk4 L+WK+SO+lgqEK1d1Mhf+8lRsiGgTVM2CxMCKocIvhTfNTmhgvC+hVYvkxjV0GT9Xux0Hp5apDEad LOAtkzXY4hQMC4O/jc81cf1whOXnx2PxX8L0cH+WMYMikxXeX1NKLtIY0fGmL1OIdIiM2UenXSe8 G/omvX2Yws1E+rw4Sf/nbBwNtGADOPLHYEcLol1w5F/Mx16YQzrvq//yYUjXnvlanCy55Yl1+HHG iUyWqBvn9r1jr7/TbrrZwMWv0uAHAlqP0bqHOJKj9anibWP66szBYfrDSGdi7M8lqbz7LcAAdFqY mQ1lbmRzdHJlYW0NZW5kb2JqDTc1IDAgb2JqDTg4NCANZW5kb2JqDTc2IDAgb2JqDTw8IC9GaWx0 ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzUgMCBSID4+IA1zdHJlYW0NCkiJvFXLbtw2FEW3+oq7 JIGIFql3dw3iAA7QB2AhmyALWuJM1IxHA448Qb+j+eDeB2VP0GYXFAbG5OXl4bnnHlLDu6yA3Jq2 7mF4kxWmKGwLwyijEoYv2Qc16NxW6pNuTasC3P2h885U6uZ2TaF4lP86L4xTK8wy3+ke06LOnWmU RwxnSnXWNf6u8YlTxrRF5z3mSCwGWHReYdZOtgpOQo9nGHHZNhh61K4wtTpJdjgcZsE5SmDPcOAn mV546gUmpazaIY7fb/yvKIIcm6hE0B+Hd5mFAv/OY+Zq03YlTZNqhagmgr1/rfNW3fOW2yErq8Z0 lYWyQdYdWMStAGngOIZsl70est70DWPzgBJdV9CgpPnwKGc4OuMDVoLLxKw2nToga2ssisfTR1zr uRSs6OYh5aTc8TNTKrYqalPb7qqIqqQD8uve59ryUQQXZ10TdI/jo7YFC0gREnGCM6rYc2SJKCmv h02DZ+WkNhLDma6DukJprsV4IVdaU5BqbcfimNahU1tTu++o1iPvikSrHQmcRHPdZmeSj0ry6+rH T4/YXuJeIOfjKqaivqOpLHoITVWiyD6GCdYFxifdoYAxhiMPVl3ibvCnU1wQjGPhrMmVP3PJN28d WBh2WWkaW9V0x9AvJSv97BT1kx7+zG6GoZNcVKFuO+mHgrSWcKwpq7566VXpnnu1FXZ3xJqcGmPw RKVT83HPETiFuKOG9NgarJMqwzod38oCE9PGAF5GE0o8PaXghOIIHlXdK5jifOG18M12eKB3AhP+ IgnxBiU+a9jOlHlYIYyLjJe0hdnMDD8mUDnyhRHfTqv+1ghVqjuypFX4FpHfYcPbeNH5cf8fJMfw 9dv+UGMKtpapyrL9cQ36t/Hug+fL6jaTHUKai3Pkl2q5iKGiRGY222ZQMSBwP2t6Xu3VYgKGwzL6 Axk3qfZFo8w2IU3YZ/QIiKe3M/BRr+ldpEbt5Qx66lvqZFyermxPvP4nDatnDRvR8JatlArF5kai mujjRZYZ+Av/9/PBP8wHWaPaLX1qOrEoqQeiG32rUE2rAN3lMM7u6hWXj5+Qya9+XY6MGb7iQ9fw w4HqX2a8NGlzlE0bZGIIo49x5laqV/Txwe7MAgWnGNLeEMUKjZIlRH1p3i9k+ZY/w1T6r7BEuP/9 t1vNvh70x38EGADs7MMiDWVuZHN0cmVhbQ1lbmRvYmoNNzcgMCBvYmoNPDwgDS9UeXBlIC9Gb250 IA0vU3VidHlwZSAvVHlwZTEgDS9FbmNvZGluZyA3OSAwIFIgDS9CYXNlRm9udCAvU3ltYm9sIA0v VG9Vbmljb2RlIDc4IDAgUiANPj4gDWVuZG9iag03OCAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURl Y29kZSAvTGVuZ3RoIDIzMCA+PiANc3RyZWFtDQpIiVRQu27EIBDs+YotL0oBxorSWBS5a1zkoThJ j2FtIR0PYVz47wMcOSkFI+0MMwxLz+NldCYB/YheTZhgMU5H3PweFcKMq3HQcdBGpTZVVFYGoNk8 HVtCO7rFwzAQ+pnFLcUDTtNhZ399ZA9A36PGaNwKp6/u+ycT0x7CFS26BAyEAI0LoedXGd6kxSw3 a+W79qDXuAWpMEq3IgysExl6Aej0f430N8e83MZ2tQBnnItM8EJUYJfnQvQtLBMvT4LkzD93iS9/ vbdTe4y5eF1I7VxaGof3nQUfSqlyyK8AAwAZuW/vCmVuZHN0cmVhbQ1lbmRvYmoNNzkgMCBvYmoN PDwgDS9UeXBlIC9FbmNvZGluZyANL0RpZmZlcmVuY2VzIFsgMSAvYnVsbGV0IC9tdWx0aXBseSAv bXUgXSANPj4gDWVuZG9iag04MCAwIG9iag08PCANL1R5cGUgL0ZvbnREZXNjcmlwdG9yIA0vQXNj ZW50IDkwNSANL0NhcEhlaWdodCAwIA0vRGVzY2VudCAtMjExIA0vRmxhZ3MgMzIgDS9Gb250QkJv eCBbIC0yMjIgLTMyNSAxMDcyIDEwMzcgXSANL0ZvbnROYW1lIC9BcmlhbE1UIA0vSXRhbGljQW5n bGUgMCANL1N0ZW1WIDAgDT4+IA1lbmRvYmoNODEgMCBvYmoNPDwgDS9UeXBlIC9Gb250IA0vU3Vi dHlwZSAvVHJ1ZVR5cGUgDS9GaXJzdENoYXIgMzIgDS9MYXN0Q2hhciAzMiANL1dpZHRocyBbIDI3 OCBdIA0vRW5jb2RpbmcgL1dpbkFuc2lFbmNvZGluZyANL0Jhc2VGb250IC9BcmlhbE1UIA0vRm9u dERlc2NyaXB0b3IgODAgMCBSIA0+PiANZW5kb2JqDTgyIDAgb2JqDTcyOSANZW5kb2JqDTgzIDAg b2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODIgMCBSID4+IA1zdHJlYW0NCkiJ rFRNb5xADFWv/Aofx1IgDAww9NoPqVEPbcOt6WEEkw0t2V0BadR/X3s8bLKRcsuF+fCbZ/vZprtK ckh1po2pofuY5FmeGw1dn6S01SVtH5Of6jBDh6nWWa1SpMUqffHypgQ6F2rBJmuUxyIr1PwXdZ5p NaIhQO8X/NVdJZefCyDe24T82rwp2L8pyyb4Z9c5O1XvsPudXHadFXCemaqxkDNKQbRFIp2VpjVi Cxk0Wwa5biSDb/NhxSoz6tCj5mgOUzguaLNSgdtzpI0aQhIU8j0WZVZR5AwaBgF7TFtCP2JhCeJm v6FvkRbipiPZZ7hnSYxygo/cjjSxaieHSBWA1XaKQAn0AiQ2H571gngQxCxxrZjWVIB/WDAEzt0N 8B1PIV2/rnyh31D53AYa2hWFCP+ppxAqaoY91iw7xdNI4i2FPvoF3DwumDZ82gt0F6BP4lqSaMaU C/LiPax3AXuqhHjxfhAmWKN3GMUSPa3Isrtp4srz/YaP8KMsntXTanarh7M0JjLkxPAPtc1aBcsY L84ymDys58BzXtiykRQhBi/G7eUjhsSi+yjDn1BPugnlI3xjmq0Gut1qoOP48qSWxHxHs9mSVrfI 2hGjIeJp4ihq4qa+riiYkdvIUCYM3oUvuONxGnu3jhtWrAu1YfWsUM9oZxivPyBDr5F9fwGel/4g D1dq6jpkGoeNO/YYuadgJM2H9+dtW2a1NhWna0tdvl3XFvWmWG5Esa80l5xJSUL0bqJR5MBWyqSk n6HbUb0ofHA9/dTEdoH0vzEqZMhGuT1M4xBVq0J3sOkEnR4ir5/HvXByP/I6wPFAj7fbG+UEyn0c bqg9LGl2FyMhLw56L0biu8HXJ95U9g0n3pwmvhbtfvjQbrXMaCGdy0HBILsRub9pEi31cc1KBB35 vIRMgedBbw9ncLCs24Fr0iq3Cw89PETycRplYFoaNTY9CfBfgAEAoQaNsQ1lbmRzdHJlYW0NZW5k b2JqDTg0IDAgb2JqDTYxOSANZW5kb2JqDTg1IDAgb2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2Rl IC9MZW5ndGggODQgMCBSID4+IA1zdHJlYW0NCkiJvFTBbtswDMWu+QodJWBWLcmJnd2KdQM2FMOA +Vb04NpKmy61DMVZsb8fKUqOE2S3bpc4Eskn8r0nMcXqzSJTsspLzeBbaFWy+maRs7rFn9cFfyfq 58VVXVcsJOeyWJYVyzGLsxjTFFPSFOuCYrnM88IgTIZ/lUGwO34rlJYldwIOK3nb7MSSs6YXKpeG d8xbka0h8CK0kSqljZBkhZYFZ/vflNq3T/THi5y7WI/pRlb8QMu9KOUa0AlyOqOZMOjTUvwS3iFC AK6GdmJXW2jHe+fx2z9SKnuwo8hK/irUUq64tREilRIQDuJ8E2ssa9NJGPAOydhNs/q9uK+/Lq4+ R3bnMhnzhjKZo0z5USZFMmmSCTVaTxJBz0EjTRphzmjZQ9P+PISsgfbmRXuxCgQUQID9+2D/2n+f fkVpkefDkXzUMtnnTLooKWYEiaNR2A+YSPOPIjMgFi2+gHnYGKGQAJI9njMMzmOQuQ3cAs5OvW79 44mh3MMznmZbLMmwIIIG39mZccguHbvssVPvus7uAvvZkshGyCJwfsevhYFBp4uCSE9nMJtZI8mu ibcZUdMU4PCIR4Nc4jl0PARFXKzbTmOlS+zPbsspeQMWN2G/+3DqLhhJFUucszLogjczV3k0V0nm +u6gb+g5tBmeLVigdPChRWDOpL0h5VOQdbT0gSrciBOXCTEFKM/1/+uF0GHW+QNxY7tt24wWXbcK kuAtCL6rsFF2ey1Uwb+9xxkoZbsXYDqgZdtTUrgHOc6JtBBxVOw8A2GVDF4x+CpPBoy48Vh8i5BQ AnyB16aY9noqHYGkPwIMADGZkU4NZW5kc3RyZWFtDWVuZG9iag04NiAwIG9iag00ODggDWVuZG9i ag04NyAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDg2IDAgUiA+PiANc3Ry ZWFtDQpIiaxTwW7bMAzFrv4KHkVgUi3ZjuVjgG7AhiAoMAM7DDu4idN6y7zCdtruO7YPHkVKHbq2 t14kkXx8pJ6o9mN29t6BhfaQaWt8XjugvSyKGtrzLId2F5a7TL3B9lt21rZewLkpq9pDHlAKYiwS WVOUTSmx3OS5XTFNOJWB64v6dI21Walu6veoS1Mq2Ky3b2HXTdMvjgyj+K/Ygg5QW2es+oGOKqsB dWNqdS/Rn4I9YGMKBcvURdJhGShkV6ZSo9RDnRPkmNg2a6S9UduQdJD8AR2tO7gRjqNUOnH+jJVx CuYlVpxQOwp2TJea7Z+S4df2BZ2dfSWddZB3VSWhm0qE3lBrOd10jaXxaktWaEujpXssqCty0l1c SX3rzxQtSas16prCW9mgR+3pcvdBuyolkdMFc5w5qQkvEslICvLDKZ7moBHHbU2A8Qo+cE9eXfDb 0PtN0ZGST/+KeFGZwzMkQPLAThDdY2Dah2dp9M1/LIn1NuL30Y55yYTfTxo/F4li+dj1Y+tBkMj2 5+VZeN0/5x7+nJdRuAgNNOHWBQ3EcIvc+tLDyKc+vG8d2uaBuUNumwfdp6TvjJx5lkCm3itJ3wsQ lmthk//j+eM4dSnGcdjx64dfJGlLP8XyCwvzrs3+CjAArxD/pQ1lbmRzdHJlYW0NZW5kb2JqDTg4 IDAgb2JqDTw8IA0vVHlwZSAvRXh0R1N0YXRlIA0vU0EgZmFsc2UgDS9TTSAwLjAyIA0vVFIgL0lk ZW50aXR5IA0+PiANZW5kb2JqDTEgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDUzIDAg UiANL1Jlc291cmNlcyAzIDAgUiANL0NvbnRlbnRzIDQgMCBSIA0vQW5ub3RzIFsgMiAwIFIgXSAN L01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0vUm90 YXRlIDAgDT4+IA1lbmRvYmoNMiAwIG9iag08PCANL0Rlc3QgWyAxIDAgUiAvRml0QiBdIA0vVHlw ZSAvQW5ub3QgDS9TdWJ0eXBlIC9MaW5rIA0vUmVjdCBbIDE2MSA1MjQgMTY2IDUzMiBdIA0vQyBb IDAgMCAwIF0gDS9Cb3JkZXIgWyAwIDAgMSBdIA0vSCAvSSANPj4gDWVuZG9iag0zIDAgb2JqDTw8 IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUMiA2MSAwIFIgL1RUNCA2MyAw IFIgL1RUNiA3MSAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA4OCAwIFIgPj4gDS9Db2xvclNw YWNlIDw8IC9DczUgNjQgMCBSID4+IA0+PiANZW5kb2JqDTQgMCBvYmoNPDwgL0xlbmd0aCA3MTQ2 IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJrFdLb+PIEb7rVzRyCRlYGrH5zp52 tLuBFgNjEAnJYScHimpbXNOiQtKeTH59vno0JXpmgyQIDFjs7uqqr97V7/eLd/u9NZHZPyzKVZmZ Nf74I11nq9KuUxMXq9SuQfG8eLcZUlMPTLQ2Q71496ddZB6HxXq1XidmXy+W9AXaz4vgsjI23P+6 +HG/iExjFmkcrdZJZNIY/Mx6lVuzjHL67t3iYfF+/wZACVbJOjN5xLKJMXGu+cuWJOOXYB8uo3SV BacwW0WBM81us9uaYaywH6/K4Mz7x3CZYkGbqyLodWmwtDj9HFpACpq2NRc5cUqIn2wVB89hVGC3 Gf2N3SZc5pAKUeHf9j9DP7UHsERZkWOx/0EQ54xYwP7lPa4FO74Cq9gkWkU2goJskahcJYmBAZPi Nyxis/Uqzd6YxJKAX4JXKGqDjvAXQQuc4B28yBL4RacYi3cHpVHa+onxTC5NV2lURFcNbO5tvo5F jWUYASTJimGcvnHn0R3N0b2GxSoPmjDGYe2GkExnxs4cnKnGsapPfE6kTe/qESAL2PBLaNeAQYTb j940k0FFZdhoDQtYkyarYmajK2yNrtxm/0VsZWVJrNSWNp1UTUXVMyCnMN0aQEfCWyBa2Ixd/zRw bBn5eWHKmg2fBidemUpJftS7pxt2vectN0aYzQYrY/bsJ+tpDWItw9Xdls1ug5jMDt51p6H/HJbY lv+VchPeRyW99N3Y1V1rPmGLuT4Ir0YxHM1BmbE3MnjDi9eI/160vg8pL3YS/VvJEnMvCLce+W7S IWIma2C9BdmMo3OfQg078IiSJPPhlkTXMhKJE2DxFCHVhImEdg4eLS8otBTgc2gtp7jF6eUClUYm IStxrkeWi0UE4BvASoKPMLbw3IV+j762ujuEOcJV2ZzIYFEwO5kkRxTAXrKQnB/lwp2pzvJ1nLOF p0quT3xYiT6d3iLQCUKKFzMcXp1PwUGPZd+crzdxrpujGEvBOyOsVLTidoybQpXOPoVGrIHIycQc rGZMWU/e/mmy1cdwmRGQ6kl1uHFoml0dWlwdWohDf0fZkBMr/N8gJiIwo289QGwg62E7eJ7y5yRI mzDlFCM/6xZsusywOTAi6Jty8oliNw6inKj4IjmIiMRB4NAhFyxlyU6Q7BjJlhEoQzYqbCSA5f6h d2ajkqrz+WrLErFJBKurOfK09NaIEl9iokyM8QFS2a8JDF5X7ZJRwEcEoHcVnIamMYZWvEtU6Esc 649wWQI9z/IrV+Bza6mnJVe21MjS4InJhhB6fSadSRWOhNb/kAwqBXRH2fnlC19u2tG8yAbxQf3g bwHCZa5k59zg6RXd1Dw/fC/C7/0GK+gRjTXvZnMmgxbGBIGoB8P/ovcUEYXYmCV9QXu80dQrfpwt ddWwjeoKHe/b4R7l4taRISBvB/8FMLH4LA7cnbndNcOJt6veHfXadTZJ2FGWFIzWgRCasa+Ushkb ZXUWHoQ3C1r2xnTByW4vNFdnvPB6QLmQCYF8ICtUrsGoDL09OjaoVFsphyRBeXrkB56aCjFrIo2K DobxjuarG379Q1gGVzvcclYthHUNc3l14bqZWPkeYI83drijpJVCy1rJrnht/wfp91MyrjUZ0a8D V79wNekb7tolFClEjzN/DFT3mqNjiX4ro9a9PfsyxPdOvHKm7ZDWBr6t7rQzxqh6EQ0EH6lA58Hv Qx5WiQ0UrS5j86p3ewfNzizymQtDQ0QCARMDSoKX2YhQoXJcvnpmOEUSZOR6qxXilhc0H4g8aGUe pkZCHaA3lVhjlHGDHGAjnjQ46Zp6ZtCvZsXqzEPf0fTVpTka1/fz4RFj44hZUAxJ3proXd3JQEm6 YBbXOfHODz6JIEmDi7Ci3KEhsmoRpsKlEjuChYEQbv5UPphcKBRGi8Z7nV11dL0t4Ek+jcNfzYjb jzIOmQ+kWnkdlRCY0jdp1PpryAPpdGr6TkZGGfxcr6Pirbc40ilM8DLBpCP0Qie6un8ICz9Ezse+ aYabEMpdkdj11aOe/9spVyZZCJvPsF8BJZ4IHwnzi+vZO8hPvTY2ZxEHzypapegELpT4dkH9qmPO SvEg1d1UNT050EgGGkkGhHF50zBN5evw191T+9nYtI32ry98gimECe+kA8umtJx5j1KGyl4oOt81 p15JoU0nip4Grv8H197pLS6c1pOMbt7rxFh9723QCEOxAZfKzJdKL8wD/W0J2jCrWnwgI8JlVhCy KVuSm4KQcYIP/GSlaEQQYJwiv/GW1CCNDFjCjNXFmVe+SHUn59cWepwUvTwQno/8n9xGhd/X40zr cc6Ri/JuJBUTScUyuPdVOQroIVNKUYZpnUJdCpqx0w/HEPRQIM6yoeRsiKXI5lxk0bErwd+hCo4e id4SVg45/VZxscdFlq5uqrb5p5v1sHXpLWz1ueT+Hi5Tbu0ZQyHXXlhleoChJ3P/ZX3PTDJCLEOT ez11af4Qsbo9MiNdUKj675YPuCynE89X/k9y0IWa2dEdBXbE5uXdQYAiAI6yQWEYjBUipG2qg+xJ cnLroLliYuKlKmQWeOMMetqIBfztEwsz5ys+f6Lk3Uz5JyHnnkMGlHuP022y4jB+W8FKlqoVZ1c8 +WXs5i7xp+ioVdtOqt56esol63OJWSdBdZHfUccFCOAZIUfdfZYprNGl+iIJhGgawWl+co9yRAZD lA9mVFrKGE0YelzeG5VLBqA4F+wEVwQnUOvbzx+rY8GeB/MTv6cch1/Cdsmp1aGDIMxlnotyxYrZ gNHEgVCbZkfFOgk2uy0/Z2QSpKrQGv95lhuvfF+vt68qdVSiOQpP1XrhM97K8FEAudbpOPcsvUSA esZkiDj4I+paJNc+hWbj2T7Lo+Oi4pzGUHyjLNOptubPktaxmqnp34jWlSKgJjjXj5xKg+q1xybJ 1GPtVETWWkSOXf3C/RmWOY8AznUbwWx2F9QhrlMygtKjlRs6xW4nVEaul3KbilgafAdIFox+cK9M 40KUju7Cs5wNvCzlJGxG06kgfswUnL7+bi8016eMiBkZkg43ODb32w21qULPr4qgvNAzzhz7RhH1 0+G3R5GEzbTkWaQUO2F0HsOUJNZdyx+o2TTmkP8A6iHk1gn0GceSLfiJhslXxh3sFmaNP/6IMjTY BImeWpQMa/bPAEGeIWGIov2vC8wnGV/gjyhD3hV8AeN4Rt57llnJTlNTIUjJ/rFYQr1A1YdfYsi6 Tk4u/HYrudVTOBPJyLFIuw/i3Lq7pbrITaTrQb5a52etQqpoJLMWhOlbsKA8oh/qKaj8utm2mIK/ kznJyvhRBECdTKipbCSrlCckYtc2T/IlBTOSNyearV5W7f4DVTmGiuBGY6J5YB+h9sV5nL6NBqpl mjS5mJhrLzkYk5m8rsgsKFOkLz2+HvnUyA9NMgnP7pbOeHTIAsmlnB+dnhFN3/pGi/mNJk9RecSg UtciRM9piA34tYS6cXC8VObOKXe5aPjsJEyQYvrFuHo2deRFjEagIL18csXzAcZfvJUnz4dYet9V rk5AxOCn5oDGv+HGcRIjzWlvWbe37TCdctFOk2UUBcd3Ib1SO0wxVV+1vJD/jh+78r2ktMyDwwtd GWRK3G1Cip0dt+ctz3ioY9qWESX0zKmYR0PZTNCmy//iu2p620aS6D2/om9LAc6s+E1hTzODJOvL rGcdIIfNhaLaMteKqCGpYD2/fqveq+aHYw8MWGSzur66+tWr17tdHE+X0ODiTpODS6FEoxRdqNQd uDNolc64At9LCiMdeeppCUpUXBiFC9pK+EXxC3veU6WnMtayXl5Y1TtchW+9N/OOyybau0/tEWZq ypnGdgQOw9gjvgc9dEqqpB7HmnPMzkS0P1V2wXa8+Dtc/Eomqxdhjtzi+vpiFiEXnJwmmniHb/SE nlpSyRoyWIKq0V5dzN+tvX6ymD7InPAxxP3GBFpNx2ilJuOtzpeC8j2GIl7bxE6nihofQDKNTAAB CQiNHDLzqP3OxKxGB9SWfkUm5PDq0+h5FQvTTUQRkbD/XjtjitkljoQMKQrpZZVRrOtHPt04Obu1 BSmpvXfU6RsZxt6TvIFQ6rHGUTD8zIQrDNnmENTJW9zf0PT8Kgujs1AfrQRe5R5pNV3kbbVIr2CA 5Tex/LL6d8zvpR43bAU7K/I4AznEPrf8OKX44Yr3E77xv0MD1yc0cCwNYFRc1dkWBlq80nzQR/U1 m9jBPJAerz3EVr2U12cA2i54Q/Wa+p1dDA1MDgdTwmDeKMbS+d+AObd6JLmccgWqoyrkQO08yyme ENUig3C2iCyRUy74eZ0RKHrPL6ssmJON/T69wY/imR/FEz9qEHEZ8b/mUfss2vWRPzw9uqQ1z8uT 0eUdOjXaKVu6Fti4eFbgVmoowM0sJTx+PRn9PjzRyo0yCzzVVEUj5kN35S+9NAUdLYe3Q631rM0V on/TIyvCDsurevGjP7bR7B/cx01hmWD7I7Wlc0EUpm0HI6Xb7p8bHbZ+AXMudSjLwezVi6naiqDO B5eThIl5X6JV4PVCH5r6xOVT+Lros1kxH2phnVabaIrWlzBHWTTDWFKxQbLvpvbfDVdS+YuiEpfq lcAjv5s+KjdBY9lpYNnBtuOq7axN2gxRxFtCYpzKLroTKNxwzoJUQ32250Dj5r+JHG0G0Yu4iz7q GJaDJuTRzUZT7ahaOUMafbyf5gyqxHntouDG57vXycIrOPizYYKl9mkFGM86Mc1NkWghydNzrQ0G CFqGFhNXWwjyehta2poMRn8QFri9J/L4uf0CSLoDPxuQ8KUxVG6DQkAiTZoba1trnA74QvN977nT vLy906TqwkF5yGIjsdYAdQyxA83ONGReWiz1EtwCQNJoA138Ym74Fz4PXGch/9AxZuR+g01MY0M8 jw3CJqKHtiFX48CS6jzKgSeT2lWkudNOXwnZ1MqJnPvNm6BGVUQ9SMP0evLDcOPA1yL74C7UHzao WTPZA05idnYjfqQcqbZb8pvSFg7c4wbf2PoV6307PuNBa4gy/ND23gSFKER+pUzYrvmg7oQqsxES jqUcINMVWdIjU17c9L4e3+AWyW4a0awfdXJquKz9eMXvGf+FTnu7nwOv78OERtpjey0nBelveuvK yKYxkGzs+iZHXkxvVDqGTXx11CzFij6qcnShV0qocnqlCXaKc8OEIyp0cL939y+9OP2leeXCZ2qx vqAj7iP9dvXh0KoAfessXnNSxjyMbxj0dm+VcpJP6c0XpYwrrNUsfPHUSXN5zyWGkksoS8qrMKPM /UDvwT+2IFAoUV+7ltuECJ8nlhmbkwnOS7L2NQra9cQkb/77SpgfHQKJ3Vb+huadlPq2kPlxqxER ekuEZNF07APOwrdd8U/btIi5CWnIpjRYg8R4m/G+oUlf8b4/SVJuz3i2aDJzsEA0cfR1s4DZGGng 52e9l/l0PVrT0ncH6pY7iuJgRcu7ZE5SE8mEp85/+LyIuhJ4Eq6WVrHTVDuJPKvkyr57mIPMUwlT hXbyW7ntT6XIaSYTCv7y+Z3wigLieNhJAjJJQFqKn5KHb+xqMyUMV9DaS2ygsBWXv7GX+fOop1ku Seo2NCsdS0N5VeGLlo5g9pwz9LZzO3Z9ez7CCnuGki5vK8gTRwnhR36WXH1w4yN+O7rkXXD2TG++ my51oO/OND1F8HqP306TZGKT5K2AxAC+m8NygXYXW+GmmKziHefHHAxNvwngHfhOfEcnLDAczqko jbgDECD8DBlnxnpelCM3rgRbWPlTASQ4hp9ho81ILyTFxrY+ncwBU74E6JgAbYHQQ1PYLRVKg7X3 4JtF6+lVL/G6YVz7XdNxnaMKo0UiGtIhNDSbzbC5FcKgChARRLHnN4t8mDK6OgvXynQhILATwqct 6/YNJCwnCEgXEFDqgdRCIHBdS8G4i7cnDSK1y59GXBwNhjOFs1YLKWUDEo6sR+7sq4ZjTDkP3SBX 6EAfKSKa9m3vmise+/6FIUwQ7lCPteOK3+h1GuXYskmvmXmiI87/D5KNv4yr+5kFH3tHSR+setd3 14XBlu7R4hH/54hD34cKylv/Ka3/yEDyH1zxDM2hMkW1hRZE+Ua5V8NWFDGrLVYaa49pyJzF4br+ aCZmc9zyZ7BqLbQ0xxf1P8Urd2DvzyvvgvmXmXz9ZAWKvHs061B55Ju7+D6cQY9rVoY99Sryxv+9 6ZapnfN+e6dEWR4UlucAL2sXXhZqHcr4dd6FmVEGRacqeeIFfS8ZagozkuPDErtTZLo0eYaQ4WAD uclMzi/kRmiSe9aYyCPNjb4Zr72/AbzGOrQhGMSmIcrFPChHqU/WjHWEbL/Tmgt2qLIGkUpC3sTc nqG1p4PFdaZWeusoTtesjouZv54tpC2qEttBhoSO90/4xv1SmLbj5+GNcSKexokdkWfQ+1cqbLbj NCuW1gn1F+xdON/vHR/uHQn5DnNHHNmytprKljAwVNGNmysWCJ8B4ffc4JHnkh1rF/YsG1IVdCs0 CD+xN7pVm5oT2wFctCXGI6m25bUhE2qpxgehc/AcoYVUoDHkkx8ojFQaQwnsyRbkIwRoGzFcxXqw 73UU4ZuWUMCmQACbiRTqabBXZlL1lUKk1ojgyTD2VzngApMALqUaUZ5YgPT+Yy6UwmpfpL4vX/qw 53G5KkOf+scml/DAhGCcWhN+Wgnb4jMWlfAUdu1SjLMq44aOAVz4Q5nWrIxtE7iIIMiVvzTt+2GR SaV3MxkIakx1t9pnZED0mbPeTAcHtbENQShk9GiBLQ7jx+mPww2nrv15NcDZCCblMXXdOCKm5tET RG0gsxmMbx0a880P4+IYxqy9X05UUrk2nOG1704vJ6+/HA8xqi0HwtdpZpZPpDu2wewzVCoJAz2S yaTX8haGdJG+UOHeuMa+1lJbpF+JOgkXE5LNfKnj9kNQ+1F81x4eJ5H0Ezzf4+dX0V1FfL7dYAr8 ohAHDe3GuGUMHoo194mMGrLCHrB4ET50e0ctX0j/KPZ1Q50HzyAaekzFe1t86fa+72ppzyJ4cGa1 ow/1aZMHkEfA0jBSdCVVe7m0SgWVCGDT8Q1ETidELpn8+1/vbw3TSjZcTYpg+xfOly2wJSWCpahz /RXuuOceLfLI9tcKqwVBXLQ1JttLD7tZzI5bRSeKnin6bMr72haGC3+7hZExdEM6ANhAD5BWOMzq OTAk0BrvjE1nJDhb1jnfqQhoIQBuTvve25Pk4F9XiPTOosdQscLgYvLw6XW6kc3lzoQ/KOAgsCRq roPoLVDMCvH67Do9wsixfEtUaRrd3ejdknVhBegEWjFSpMgV9/0bglT1x5WfOMVqM9oiAYRtWRnZ KWzroWuuci6JzSrbIINTbFeivecDP/Y00/DNB+cQ3NLR2nQdtJgzVkGC0S7h0FuCv2ncnSXILVdN P6TXHixYyAbt0iyNoRXKbq+zhvzKqHEe3xiS8gmPDZXGllcBvIykrohuFoefIltb5g+jSk6MFUAF YApu+Scsu3rJJJMgJkzjiO9Uc148920jqQbrFSyyESm37eGNG0ba0uafG3rL5Di+2GKycn/r8UUQ nU2gNHbC/oP7L8PnIk1QteXlQWGpkjuWIAUjVx8Zw/8Zr5bdxo0geNdXzJECbK84oijyGm+A+BIE WP0ATVESY1oSSBrw5uvT3VVDk7RXiX2g5tXTPf2q0tS396jLil3EI6RVwqPRV8P2qdJJ4Qej1hj7 oUZlcIVhQ49My6z354ZxPFJawtBdsdaix012clBiK0eNLQktLTpUHZXWUcplIqy/c12NDWJMz4vH 0DEf9Kj7zulzpjJ3QEt8g2g11hviUtFHm7QHUcT3uFQGYLVXmygQ6NJIzR4H6saKmVJduxjg17Ns ZepfmMxankWfNLV6tRmMe8F5g1lxBPuoFg9gNH1SWHOd4Er0FLjK4BB4r0axYLA6TJ1B8Trggm+7 XeKkJB4W4u+V/MsnF3GJwNQ4k7KdacF8VbnlIvLL3d96JMUR6XByQlJXdMVCEhY8V0yzBFTve3XQ ZmYsT2P2LHX8cu6CHh5nBa6npon9GHTZSjHypolIXGdDOV8xOB+BUCyeWi0+0qpr+5wl8zLt/0Tp Vtg2ERfxwQH8VpWM7WoAXLibH8qXCnsyqWE8LkgbK6WQKtG2ibBVYmkmBIqVNgrK8Wroo/X7wYcR b3SXFtKPNq5xB2QJONEVNhJgmTNOzxSkMLGwuF6bkfVl8YwhtBR2MH7ES7uv2kG7rztt/NFq2WtL lSjameJWssA4Pa2znDTsJR50e+zSl9hquciN3dlip9G/5hK2WZvWugtP2Tuj7WbWdvW0dTl749yo pB3oR7c8LBVLOzb7NOjSXnjvEZ9TGOPsJ80xzWvex1vv3Btuqj703Ff8VUMM9dLu6okvMo5ac94a ACIo7cbv5KamXrWiBcWK8F52zT8SvM8/g+9EujrsOxKc1cPd/JOLVhqjt7ao9N93C8ve2MWJRKKk d5I48XySiYsWh8Vvu0WqA811+zHkut88aM14XUSx1pVZRZDWldkuE8WK4LMBOHgaUf8wCPK4VFyJ 30+jLEgsTdfWau9Dx5U8ea5A3zaAHILpm4oUhhNheDbi0kvCVOpeZR48edZirXVcNyjq0a/CHpzA RVTlCy3BNCwz67LQIglBTcGbf9pYIYrUEYerL7jz0EMFWlhgjxSoX2ixMzVyKvAX4No3W3viyJ1H 1w3ifapYgHhb3Lr2m/V/85zrZ7bSL334WYIacNTIQue6vihfQBEMfG9DN97ayxoxUJu2Bjqlnvz5 ZEAqjh67uxlJqGW9N2R9ItMZDaQUUi4/2p7GFMVxdFgqOuAc6M+PJQDVvXoCgye4a0SASHzaGRGC DtSwoiVDdASZT3PbAVoSAy1pFAijho4suj8GVhQryBN3jW4kWaM6d+bB2DKsK6cFXfA3eWfHtwJF vE6p3PhhalIdo5dX3q+urPgaXApDeq6A+PojGLAwd+GJvpGcI2V8Iycsnj/CpnKqS+CE7nbU6U+6 iBtrsi8bHOGQ2fMf3sZaMnBnAVNgdqCq03fHJYd5rVzJN3NriWDBO9NyOS2EyUq2pFIJcwEAKWGa FsA0JJ1PkHQoIor/kkjhnxAoRRYJnWXcNdFWjEXu7F0Tll9sXj0m+jeclCfL9CVoXM9jGt0bRHvM pNHhYCfUHgyNFQ2nY0NXIRI365hPkuqWlT2IIij/ixcJDSRL2T8+gvrLYlQz9gQGdb0LAfkOj1aM hX4aSkUIu9el4q3GMUoQhToZsrud1guK46cJPrH0gB7qBEVc+7HEikxCHnqadO1YLVY8plYxSa2y epjFWJbpe/pMytTNCFtvZcfkQS2q8qHVEt+djGJZ70uiI0b3ij1SNVdpS9UeML5g3BoF1Raby2Rx tiNlNW552tU2BsKD9lBH1E9UuXi9VaVuBU5uOOL/x81WYK1HIk0ixw8ZtWJGVe+isVhXlb3Aqf5i oNOmaoHUJ/ulrVdcszffoGOGOTG0LKumakNzT6O5j8RWgUtxmj8kt40UL4phn4z8dwDnz0IrCmVu ZHN0cmVhbQ1lbmRvYmoNNSAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNTMgMCBSIA0v UmVzb3VyY2VzIDcgMCBSIA0vQ29udGVudHMgOCAwIFIgDS9Bbm5vdHMgWyA2IDAgUiBdIA0vTWVk aWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDS9Sb3RhdGUg MCANPj4gDWVuZG9iag02IDAgb2JqDTw8IA0vRGVzdCBbIDUgMCBSIC9GaXRCIF0gDS9UeXBlIC9B bm5vdCANL1N1YnR5cGUgL0xpbmsgDS9SZWN0IFsgMzU0IDM0MSAzNjEgMzUxIF0gDS9DIFsgMCAw IDAgXSANL0JvcmRlciBbIDAgMCAxIF0gDS9IIC9JIA0+PiANZW5kb2JqDTcgMCBvYmoNPDwgDS9Q cm9jU2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQyIDYxIDAgUiAvVFQ0IDYzIDAgUiAv VFQ2IDcxIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDg4IDAgUiA+PiANL0NvbG9yU3BhY2Ug PDwgL0NzNSA2NCAwIFIgPj4gDT4+IA1lbmRvYmoNOCAwIG9iag08PCAvTGVuZ3RoIDUxOTkgL0Zp bHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImMV81y20gOvusp+khWWQyb/9xbJt7aytZk aipWTo4PFEXLTGRSQ9Lx+InmNRfAh6Yo2ZmdcpWp7kYDaPx8AH7ZrN5tNpGxZnO/KoMyMyH9yY80 zIIyClMTF0EahUTxuHr3YUxNPQpRaMZ69e4/N9bsx1UYhGFiNvVqzb+I9nnlHQMT+5tvq39vVta0 ZpXGNggTa9KY+JkwyCOztjn/HprV/eqXzYUCJbFKwszkVmQz4yhnGfwrjFnGrXcc+t2TXwS5V7ed fPfy30z+uggy70FWjb+2Noi9Rz+yQeqZFoe4YLb9YWfufRsGiVfVjZleZP/YBMZcN/fKt/VjOle2 WPQ4Gf2MeBuSEQXWe/Zt4qmACeJ7OkqJHopO9F6V1vhhUHhD09WNMqkGkq9Kn3EkHjERDN+hs2Op jE53Kn2qylpaoR+cEOi9I0V+LN7mrCTyGiK+2/x3FZKTAptmmdlcw83lyc8lnNCTFhGJIaOnIo10 KtkGGXEdSQKZfNfXT2T+GN4o6VkdyHAnEGEUiwliUcTmSe6kUhyo56MEQn8lE7AF9vi0PqteV1gd zJeuncxX79cvX/1/Od4a53lQ5mlJYUa8+R3WxsrcJhpW5r25UMgGkc30EqsRIxQl0qdq2DcTx/pJ SBSkeRGf6G3pHmAtZKz9kh4++OuIbT+SGVKv3TWdn7NVTKPe13U7sdUS70WWRlcPsqomjmhek4VD Dt2Du/3os/8veEGWqcyu+dHWc5zp5X7XHDhEmHRnmj/lR1Mr0RNY+NGCrfK7+cDGL7ybj6bm+MyC SOU/SpQ5nggsjrWcc3sRXxJV6WypQuNrJEE5Me4mFhBRxpLR2BQknVYc8oQY9LqQlOElEx8lmsv5 e8DdFp8aREo7nZ31eqMzSnV+tXHinDa3tBHSjxt+s/fe5wD8RBpyYq4jZXbFmUiO+0vXsX4D7Nvz ZRpeXDvqgcn14E6Fvp04bEONu2iOu+KUOAm9ao8PJ04picMrTZzfniDukaCAPlvCB0uQwAnFWZ16 X357nVfkjdQW9hTzSXSCisgllm8lWLJk7ctDtq2fSETxf4PFjh5HyghQFACKSCladoll3LOy4j1S TNc9Lg4U2wc50o09WIC8rnBm5pCWw25Jo+rc3rynp6fkTuaiGkfgeWX+IqtQ/brybcxxQAUvBw9y DofipW/WzkLnLqIa67AthZVuPlAOXRM4rwUUSDhFbHNpb8JNKtILSCpmc4fqa+PeKGan5KCM90vx ekKcudixPVMCEt6tJtk2OB0pXam01H3XyWlT47jZGfzoGbs4AQ1IGwml4UICwYn8bCEJp0r6Igsz Psl3O2INbipM4/yRLZsh2eEtptypZNw/HvthWqpegcrcUE7m3gcuoaWHxUd2EYPg8ejUE33VBO3y gR0AS+3903po43JuSlDV+ro/yA8KJCq9BbuBUfETB3bG8cQwKvFkKZ7YV54EFUdU6eXFFQeVbJd3 IP5/+R7N+e5i4KYZBOWvm0P7oxlezO9kJcrkm+vfv/o+I9kbiRxzxs6RFc3VN0xOecwqSkC5jojq L3qIhJ8WcrvCT8s96W4iVDfpi9CB5Z4uG6aLyT73uMZd2BPuULBtX1CY0NPl0s1k3OqQ61pyWNuD jakPrZOvjEWpq1dKmrlwiQifuQ8gaYYr0w9aodQO7J4kmT0eFnO+KqpN1fhd/HsqpRzqlbyK4ydD rWE372XRYDGYqdcq0oCKbGHO7+2UYGimoaWHLK7T3h84fcL2OI0/vT0e5/KWzsdjMzqd76VRGzhJ UkeEd7DhbSgoIVLMKA7jwIY6HF67U3UkZhRqoHWEVKyLBeaODAaZqr3FVd17gWKUgIQBDTQI3u5F s9kThTZuN9IUSM3gT9+9UFcjqTj3xXIwat/8zGpZxibO98kHFgrJZQeYL6vb3PxRthNr6uOmyyQK iiROFsUwOxXDTJOIklCAuSREYKARQIgBCKScQEFmv/qAj9T7G2KtRgVgsnSbR4ESyo3y7lU1KoM4 s8kFfsTp4mkbX5ukiCODetxLpKA4S+w/wQnN0BvUgA/0mpL7xFeJeZbiFY9KddP+mJtYKwMFJ+0r VjVGopRitpRRj/9XZ3Cw49hOdP6hpdm1xH9Shvg8yVGt9x4UWP4xb+U19UZBqSE0WTTYfKEHKU16 h36/fD71JYI7sOplnYlP8aPB/gS07PDh+srTlBS/U6bxbIY2iDCO23khbmq9zPWfrqY8zMqG9DyS sp+01HKgIcxyjckSpDn2XJTRXRvi5FWwLcvU+vQcF2eo6jFDACmde9/JwvmrukSDUpIWi3CbG6dQ G6dFuxMLCsjs2W+/cVhJB+MonhkWSrE8t5Zy/CDkLRZ6d8Km+IqKzeHUNPJ2t7zB0XocpNZRs4yj 6YzbXnkuxTUnjUTnHgA8fAdtNTI4lh646mHdVhMVxvOLrapK7ZjGayaRGul/xKvTeQdOHJ2V2Tvo 5762x9OOpr9HOKrZL+Px1G26yeKAGVb+d8wsIkeSX7leRTJNimKxpJDoQhnB6FRxP6QdeMEDYsFp KoHIr5Z2KQfgpYjEyAFeWeASOiUKVBvege7tEEzyOQQVrTQEB5SAin2TsfrcFIy6rF19WDPatL0S ve7IzwcggOn6LELf80hVMKb0x2aQdhNDRyStMQcEVh1gFwTcL1ORqeolfQ+KTq6ZbTNhXH2GhKbR qq3n82nEb+iRFfnc10eCG9SG+qEbAWYdoJiKOyx0bcEBEvZnch70MU6LP2VdY7uaVROWuIk3cthZ drA5Y66bqsMgoV9iKoi86udWofjGoerjWnk3h52F9Ot5FXgZObwUqMilYZDpUmdG1bn0ZByRHJaq J+qVEv2xO8MCD2AzLgfTxmz1JrW8csdJaJy7rLqrPJ2R5x/UDsIb063TaFZNNX+YGYKNW0Mth/6p on/89tCLideG5RV+LZIPP8ytsrnsomIq2tRUL3IkX2ThNFBtZuBiW8XoPKgwSot/nmprx+jCh+E8 /0SKSqMkAiPko/QKo/bclINbfButbTzHUmLLsFl61WEB9pl2zOwqhlFLaD89yEal3HFMTC5ezGCU RItpSrtW1g6Tw+bbWd+YF/HSPkzsBXdKNTMlitDQpySihMbSJEsDxuRHNKderBcyRb8gBU/yyBkn cn56iVi33mfyAs1t4oU/ntpBfjTsjpytk0kq8V43SWc6XriHxspMNJQfs45JHuSR6MiuKmdXWbjq YycegmVLHvGmqarFvtbTPZkzCKYMYeGTbLEDvIObheAtAulBZgf0jXy/wylNSONU6d4kkOMo2O3y aneB+WD2SWQg4eJKbdu9NG0Hues47YUY/5mBp3ouinQk0WNlLgVPPMncfqasMZ+bA1d1vaZf0I/O BnjaiNZ3bHsQqdXas1HutYJC+3caRgnaSNGNfPDmzEUzzRy9t9fU6FE0LUvr23PAZnYimmJyFjpt cZO0xa6/R9/sunK2N36e9+CO+hnzxQN6apoYqmHbEowM7QGjBCEpV3KDTntAqvcQfqQmqu2oWa/J 8iDoh7aBvIBbAGpDzLnurSrmvqc2m9HS9fZU3vEON7uwDl3bYJOAvu/O9Zul/da/bXeCOpctpQO2 U5kpNA8wa3HTNLX1ae7IHMoVF1Rth/ZyLysOIYBjqeBYumsGuBc5YUecHlpE7HLe4Con3RR8Vrgr XIGExQM6ZLW69rXMg72TkRVf+MLJtlC1ml+7eM142eW9Bsb4f5RX23LkthF911fgkXRpJPEyHDJ5 iuVUoqokVlly5WHXDxSHKzGmhxMMR5vN16f7nAaH1GWz3q3SEEADaPTl9OlyAp1T5Io/l/iYTDD4 PkCuTrz3Q/QX8eVq6kuFmumLy6j/JiDMCgwNCPP8xDssXz78FGtw/CJtrhoE5COP9vvBS4ennnro 8Ts0v3JJkqSMuDb4+pGTrbv5URocif7h2Wa8u7l1OxvE6YnUSvqDung7UM/7BuOuRfaVbdPfb9sP Cibux6OXhOzGThqter/3Qy2EUYJA66tEQ3ElllhlOjoAphAHBbjrSkfu7vruRvBZetohvpI0HJ+E U/ViJQF9tUvr5FAN8eRKSD36rV5KyW7r7q9v2UlcapVLoxu74BY2kGaFCXnqHk5Ew7jiJ3YxA/fr 3aKb+GCHHwL/Z9ysRpY5B7ojH4e9/hXfXty2PhzjNbxycsiSVAiEXOHjUoMtj64HNUMmLlQdS3EJ /IOPbJ2bi3LJoZfxny5JxztRKllY5iFKkwmAkmoq18ooBVrBS42T78JI1fa1mLe7u1YvFdFdnEU3 J1TSeDtC+iC43ffqlg0rpQR0R4E9JrWWldMVXNEEVAY71qNWwhx340drtQT9aLNOmYF+8LKaqvXO lr1iNvsNUGHQACDcbNVkn/k4N90HA4inOd2YgrzI+5YCI7Q33pGCVOA54Rit1+XsmNqP81Jehts9 4vud+pDOeHRHMjq2vjGuuws/BbqZ0YCTlXtdTdW9OnWs1QyKUoGiv6ljUOISxfbPSvDFbNtWG47C eroErYu2tyaG+C3Cz3HXQbhBy1ZEHCFlErRslYFOdlEk+fr/8goFjWtQGqu8y6q7YBSdleTapnsj FqPgQEbUJDmpm0Y4AOz1otgPVuaV5dM/iyP2Ky7XvGhrjMVmjaqE67tRm1MflDQWZM8YAVRNuz3O aYuWTOlKxgVz2LdCHv/Es/f7vms0HXaHd6JkPaHW2louwznU3IJ1fQ3emUZ623LZkXOskYJpkO4h jfQVmSBqFnqO2S2s9HkeJqkmmb6z88exlyfPL7eBbewR0usoyO+4+ohJN3D0CZHqwpOeeHPnFfwp gWwWNOW+ydUbBY32oJUAC1tKu8MQPl4eYGZYnHNw26XOQeoLnHH/3UTfmonSqQu2QoKbsSfei1cT YFEZgkJcLL0OJw58717LBBJKy4Q5BKJ1nCDeK6kSF+SuKUxRoqGSZK2tgX3zooEXCC9+gKdSE2ts 0xHnANhK4FimXjzw2RxZIultnUkpDO343S5fcq5YrGqYzmYEXrNVWytHfZzbL5vhHIp5M/ROoClC wuixRcS/HbiPoIq4qZKrriyRT7Gp81qZgT325RH0JWhoZQbKsK4Arc/yMOF0MLc59ifWCUj1Sa6u HMxamiKNH3h7q81BZfCQmLIXqp75axPxomNc2Wpr0CPnNsqJqmjSa5VBCzXhiYVz14h74RBVcx6E 2RSEG5qSDYtyiSzyQ9+3/hDPeT2ekqCTyQ3DNhG3mKA4tkKgZdqHTdsU8XTP7guWHKX1GouUzPLH zuQ6/y7nWFQUlk2ZvQaIkINW4LLu9XpbGLhwUNqR4C71Y7XQV2eX7xhxBNVEYIq1ay5tTW07WCOB ik8JZXich0AoEQgLqyw98KqWdVanpmzRSBnF3/wOFn1aFLF5fylVyXpUFgjWM05JoemlMHBhscnm rP4I1NjJrGnOiqZ2ENik6THhTxbwpwz7g16h0WU1X71ZzvP1qe9Zz8hGKWTjryDnXawQ9Pikdwns PdQ7fm1xaUEOEsS2I35M+Fyy8pu3uPrxkWI+VgrY2qhWLr6OKMUt6l2ZslNfkZWvU6kf4gxUSuHq HpZSaK4AlqppRdQzfiUNWWYMSRSVIpWbsPsYkT5523NQfqj1L0ngWG0/vFEsHZ27v39/idFB6rbg e/Mxdvw84gB2lkFc0J8sjiJcIIxA1Ync8XbTZSAP7KdjPLe7bnazqLW3J/V8Em/qOOCBjW9rii+m ef3528xmXha2R8DnODhDTyD08xzincFpjQnE/ASX3PQHjMTUidjqRsGVahGOibZbqye/voJehaYI QlshNjU+HHGf1xBjp0zahEyyWkyKwSLTtH8UNVJR4+dX1aDFq8LW1aJQJqcANKbR116NKDDurY7o tx66kRghg69g7UTUBTKk1KfS8vpM+e1pr4lwB48GEuskh+D2Qhm68UlqASi1kIUs0idlZln1gNl2 o8fzQ9hneKg2T5gbAQsWiui4Bu7nQihcBT23O/LKBzHQ3DCvM3Ow1PlqUnBtz2aFQWwpazFPgfOl YCsPmo6f7zod7V8kFP+CKm1AlfS4j7GWpDwSVveP2FrSFC/Pgc/ctEiqqQaz96L8PKfdAlZaU+9O y2YaXYNN38UKlzfO3F8FQLLHOXuZKePD+DjXhuZxx9mzu/kKj/zv2z3hKdvz/NTHlFMfk5oHNaxS tACCMrV2HZJDXV8/cKFHPWhDx7FmB5FK6qqgMylNAdyqk1s0NGjc6PEktAba8engCXLnqqpi2IjY x+kH0+fkA8aVyh9Ga6PS6D+Y2M9VtI1y2H6hkxbetu/QrRTq00pt+lG+gP1HO2PswmnE6TWiyn70 Mgmrbjl/EIPnEl9OaNvIJ7XO3tny8mdTCho27dsQnOYTpSlmlEa7WYJcEtVG80DRjV3lklpmpc/a xm7AlzWwV9oFPEGk44DybrFkfUY+Y045mJMdGq6O1S8PvmtecZLqKuMjZjXEffWf8AJwva+J6CV/ vj8Twas8kXZA+GPhJIidpnEphOzs09n392eFDq7kPz4gLeaTX1Hpt7Moje//dVbpVpXBRyWRU4pM WVwod/ztHeO7Ww2VDXieEBszw+CB56kxtcJM2rSX14BNMNwC1BmgmrBpil5YUyuOQsa/jzR3zJSC G+ic/gtG5+6BIqNb+FAYvynAZTuwHtueYqDT1iGCv5q4KWCv8oKuUudrqPjiBfKAy2CD6flc1Nus 0oolggkEWv+pgV7Y6QJ6bbjOk/PnxuJFqSxdZy9TID/V3MRq7i3uTlC25EIthqCVFTlkoqaXBCyB tzn4JSF/A6gUbArbaSEPR2RE9VKsh0N6iNrB3Nfhr+2yu87Zq/J4W3IUfwppMjvC8YygqNboMIOt j1xw8zdO6n3TU0FeZ29FXUhtztn5w7DdBa+vqmkL/16ERPvfAOtYe1wKZW5kc3RyZWFtDWVuZG9i ag05IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1MyAwIFIgDS9SZXNvdXJjZXMgMTAg MCBSIA0vQ29udGVudHMgMTEgMCBSIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJv eCBbIDAgMCA2MTIgNzkyIF0gDS9Sb3RhdGUgMCANPj4gDWVuZG9iag0xMCAwIG9iag08PCANL1By b2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNjEgMCBSIC9UVDQgNjMgMCBSIC9U VDYgNzEgMCBSIC9UVDEwIDM1IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDg4IDAgUiA+PiAN L0NvbG9yU3BhY2UgPDwgL0NzNSA2NCAwIFIgPj4gDT4+IA1lbmRvYmoNMTEgMCBvYmoNPDwgL0xl bmd0aCA2NTM0IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJlFfbctzGEX3fr5jK E5DiQrhfnKeIshO6Elslbp5EPYDYIYkQXKwBLGX6N+wPzunLYLFLybFLJS5mpqe7py+nu99uVm82 m9hEZnO3qoIqNyH+8UcW5kEVh5lJyiCLQ1A8rd5cjplpRiYKzdis3vzjOjL34yoMwjA1m2a1pi/Q fl55+8Ck/ua/q283q8i0ZpUlURCmkckS8DNhUMRmHRX0PdjV3ert5kyBCqzSMDdFxLKVccMfMUn4 6H388Mn8y6+C1Ot94hV5n/0oDhLPXL731xXW/zEHv/AmPwoKrwVNiMOu/aWeWtzIvB0OL/x17Bn7 E62JuAYZiMzEFIb/Dv46Iba3tLA+Xll502QHMz3QDWJjGro8DHY3CzSTbZgZEdE28+qcrve0evEj 0jPwP22+XyVBHqUZzBIUWWU27+TRqXt1lOuz330y34FLGuTeYBr9evLjMIi9vazs1E4t2aDynv0C FmKt8R53Lsvhzifxjhk0yxyjWih2fLuxF2YSdg/CzrTX/rrEKy+vrxxT3I9BodwmPz6yboSbrjo+ Mk9+iZ8DMxwnU3dsmxxM2Y9kwaXMwVrz6B6D8xzWo5XRhx4twDxEiCPVQ7eUZ021CmiXuu7YGSH5 IUrT3DkirObwi8QRzYOfy3sz3G7hdDzUjhI69MBb+SWhHsK8RujZrex9Y268SD5vfBiTzPgEpXLY g3dhENY1guqIEQ1MLB9ZqPBsJ7J9JrrzrbFVoR2rY18pc+Drbad6mKs3P5paF7UyQ2iPFNtEWc96 6JoE7/hTb3XWON1k31j5ndrBLmyZ5WxLxog4Y2OKHenx/joPSs8wOTApFUxCMiMlAAnqg7yka/Db +EjYcgQvpEyZREfChOkuDFFFDq4QgVG5IDrHkzufAlqT3aUpWx+acapbZ4txqoeJbTc6XLnXLHcA coMMkBNNcjKu1zYI83kfIV4bxapLP8PeFeKEOLUMISOBjIIaZyeCtp0sIkbhqTlTk2V0doFv0GOh om7WgmEdRyOHBjN5YfMTYhdxkMJUeZEEYWmSJAwi4HWITYHruQQ4WM+LKshyh+vR13E9inMB9jyr AqoaT0sHwReF8wo5kdzS79gvgKB+gLnufLwlZXMgwOsGbxUC0+uRcTdMKx+w4nDgr/1EqJV7fyN/ Ck1X697kJ4Qtg+mbRqiHEXEM3FenwxGpp9d8emTiqeRWOLyWpgzqwZrbgxxvYfUbPzAGbvU2WiDo oFXi9otSOSihqvMQmxF2zyOye1LkwYmDXtk9C+MgTlHQj3bnHKjmRPTqUQu20IJ5VtBVoHT++87P S2L5Z5yfculXJc56h49e3fkpWZL/WgbnYQfnkgFSqeUFzPFMW9ZcI3Mq5A9MVHiyuKIAT6iApJLS tKqZxcgou+8HOTPKCwUipgTqVCYyhpIqnBd8We7UrJFThf5SGqYMgLMEs+3h5BsvRrY6V1NOP4jS e/pRxXph1Te9SDeqzIF6E+Gmyu7twGGROn3bhaHMXwhJMu8XO/Rrn4U1/f6FCCNPz8zJm5RJT3Ic 45fXkL1AbMrLTJ00NBS4GSFSwzlEGtMG52bpWQrkiFB4juQE23LJIiX0km5oKar3BHqollu5jqwn leNFQsTkYhUC4Uo3l6lSOg00mp4cUWYf7+PRlLrWD7kvqHDUM7ZKB5SA9wt1C3ChpHnm/cw8gTWd aPzCa8GZDDWf/GK2rUh7lMNpqHeyMWNUpo3e5q9SpKo57qm5IJveeMlJuJBxqEGBhAFRwo2Nhkmu YVJwmFBNQqBkHAr5/NYd3+6FQI844DWAgFG93LNSS1JYw53ILZHF8YLVz8zQ9HhwLfxa/qsiRJ5w JPMw8SQC3Useljp9SWnZU4I7CmSKf+aBEMq4OUW82mVg0e6D3BCFGj9WiWdU7pUHPpTXbY1cEoUb BJ8a5Sut4NyTx9qTU2+UShOWU+smFbfihoiaWwqsnHsvNNNSnVkcraWFy7lxYFvIVWFpOjRTfHzP 67PTwaJ8tc+8cL0eDFl3kxWRIqKenEJKOkogrl+NHDJVXR5foh1GIR0Gt9fZccBY8hT1F9aWqSvi yDvR+v+qZ0b3FO7G0HXe68Ec1wuRjruKPL35KCJ1FsO+6LBbOqzrnYn5R3R2Njp2o2iBqJLhxw2n WZFzDTvWUFRp7Uxz15kCObnl9IyeKLtwPmEw4NaWmmE/99APq2jtb0/L6Cw+K1GnXRVNhMHHDz7V MziRfwkIva6rb/uhZhhaJkRMRuDmJfHQEB2P/o6mA3s/UNEoUVNTqamEm+YHYXy1YdpCT41bRmAT ou7JttAqgxv/63Pu6/GK5lwZjNoRfd/OzT1xJrqniyYJVuMBAsS/EvAy7QipmXc6IHFUII1+Mx+v 6XExPRXdjvdviaVYyD4JGQr4W2HM81YrfO9lytEByWqypzr90PCkwYbAb1WuTka6okkjXygPtU9H KDedbRisSh4OYu+9aHwhOKfGWYxl4x8t3ft+slKQprbupH0tXWHco1650sYKUPGMQlOf1lktw1JG u24+djevvnV3v4Mf2Jb/dHX25O6zlH/+Hi7g6WXRrie7Bcr9JI0FDZRSkaVu292ESLu13Rlj1FAt xqc9BobZlkKFjHnNWl0tC3I8T4SuHo+we8GXUcbxyEWPEbHNIwYtWtED0GTseNEL+JBlK6moBFBi WQW+0vmXQCkS+lB2tbOKpLM68OdATseLaQhq+idfHEZ/nRrCcquSRXNRapIFPHDpYI9olqjActkW 8zQWE9agR3EQFcXHPr1KxT6f/STjJkyRfRPxbygrDUflB5FxUQg7YhInrozGuWMHe3HvgT4rSeWX 1zSiQp9UKzINQI5tCiCgWTVDc4RBqPrdSSUD4uXln5lUshCTVTmPS3MqRZpKiE1/TThz4I4BEbqV NVdj/qjhTT6zz9pVrHlMnXRbQJZ8FFFFP+fHMZdJDCWeW+6kcQqOKb8A0yg9Qui16JFKBBVUSmmg sKaeJtKMzyiSElGMhlremzDlJO4ZUPfqPaVSRiFeSEdIxBNpX3DieaeCHplsZORC4sSpdAUxIq9F KlkVc++48fQhvLdyhKa6lg1zkB1Z7U/k2MY6ejme3NooYb98kCMi4LsDiiSz2saxkZc9M13Llxr7 FWidq5YbXQl2MgYIgPLAyJpyalLdoL8BcYejtWCWStvKRfRGvVDb47xSaXdIbQlGkyMGK3sG7kLg RU/uDsxkkh/MWk4agJzrvnQGBCyPqtaXG0J6G7LEJX4ca2R98GmcRHjBShGCFcqhOdz3w+QLtKMT Ql991kLhiPjOXY80Lcy4EMZc094hU9aZtI/O9sdmKAuSKEmOTNJkVi9KhAvPHGQriqm9pW6ensj9 C0A5A/qvM66rVIivYfrYuzLjA02aCZAeFPTfjAfZ0YeRe9yRXDph4Iee2UJ1vkJSGusUES5nivzo E/ariAEjHo1m5E3ySz2oK/UygjWS6QV2aSTr9Kpw5cHOfSOI2J/klrKKz+NWuk0ZPUsXuDRg0VRU 6nNTBqeUwYaLzz0fml/9KNHRKXNq5npofSpFw29sB9oQBo0dv+EPnv5KGe5ilSmS3PyZyPzJWc5L UUmI0FvxFW6XqLTt+HBrlpxQ01QZ8Nwtbg+96Cx/VVVleONtRdWRQCAHevFb62FwoijzdLhNvQsj LGtmsremW5jrVl431IPeOb7tD6rzP8qrZblx5Aje5ysQeyIdQ5p4AyfHenbXoQiPDx6FLzM+QGRT gocjcEBKYX2Af8Af4+9zZWYBBCWObF1IoNFdXV1dlZX5Zb58TpWz8rVy/IU23lqO1uPSIp6U49je kvxlOeZURgku80VFFnVVneyMPMrdi4j7qIkFE9bg/+iPT5CVMVk6zVtukwrbnPMZJ+5VzjTDAFpY bQFuwJhn6J3snJixIhk/oWhcO49HhpUuFNFEjDlm4y4UnrNo8NSX0b0d3Zta0LM1lNlxPBD8C07c SxL31Ij7UIxlmtUvirE8FaOj4HHuDQ6kuEHSqLINQ1SZKAENOz2uiZaTVcab9f5pDmZsQOVPV5Es BXdYb/rVCiVrNtMGPl27NvoL70/bN5bd0yln83tXhXJwDTQk5OFtcDTS1DsO+gbRunH3Bie0Zn86 /9/m6ezPcykoHUz2AJSxTpEQV7dauuOynTxYaM7Z0ex6KyArt/16JgxeKqi+O3brzmTPeh12YU4I xGFByJ1R5+r1+XNibe02jyfVcqGeI1OlBaEB1XuEBkXms6o7r+5bfz8JMi5RqT8bbGVB5vRrNE7m 4CMgiYN2LjfbcuBGX1v3g3MOL+Al6rYidpEmCoU+kCLo+QoMnlOe0N5SVlhBvUtbGy181N/Gd2jc Vf96+Oon3vQ+Xe+GRxBCJo0GdZBYTScGQxmQu4rSVba0PveaMsgsTmXxFmWQ1tb0MyiD8WYv58qx sf5gLrvQdWH6xTXpM+mqVpGzSCpmQIrwnO4zEZ4j0/9ATphbt1DivYekmOxhHN+Y3/2wTQrBELVH kMZq5m5sfG7LfW41OVqH+2PoTTCOcounju3QFcKUZOmyyqYBfRGntDDgrM7itJrGKWbWrxSiP5GV J4JzQ4bG0R/s56x2jHiYCycR6cWDp2qoncOecaE15B6QpLbXrwB1VMP97XCsbGWgn4zHSis75Kun yjKzWLxyqmoUiO7OewIkSEVC+VJRh6D7a2y3M5lzqlZQP5NHg4Njeq5ybPeW9KwyTHbh+jIrb0LU 6rbtpvedZ8wmugnHIcmC55cnhSeOp1jUfhKUfrCV6C3A0CtLa5/2BL1KFoe8a9e8Un2gtV/PzTGh e+0Ujl/moG7+tgaeero2Gts8d+ZUXpNToTvbObZz8rfG2Pj/491FoRePlCauvQPg+moRjYoZG3YR 6VKnLz1sG1j91t6cDYTowx0x0c7C8Xu+BRnaGfWLoiuNOalIZlpgLLMblvDvVsN9pMPmA2GpLeva 4ySrIIMsqyL3JBBkymGybz1qR6JpaA64X+MEvmhnTX+jxwd35+juAHc0MmVbp+CsHfs1awjQdhLq LBtDnZ3a4VBDaNDw+m7upJ9/nVBvx4+dX/YtRV/qBL0eJrmSS0YCwV6VsVcV7FUVe1XBXhWgG8Y9 YBaibKYFYCb1uO9OGOO7UoOpq63YqLQEjYvvBwLAXj6dz93IXkCKa8BP0MwTasdTbBOQNwyu5YmH ZVjodg/CF05XV0mZOZwj01IgtR9FNnv5KP2xmAiQvB5vKMUNmer4q0WtoOowMBzUqNOCmY9+4Bw9 X81XNmzcBv3to+f2Aro2sVal60WYc1ANFD4UrZvtbe1ZDOyv2wRjCLMXWkmuUhaldPCKDhmuOe+3 UFSDYVgA7EW+PcKQcZT3VM62k4kRDHUaB1na+DfDlibadGt/df8C7qdUzFOeiVSGtLPW7Y3n43XF /glXOCy71R9Ty+P4wcMJXYVQ8hPM+xGsN8g2aJkfqA/3o6vTQ1y+DsOgvzB23QU8HCkwj5mIXVqW m8QAFXs1d9LqJHYqF69Io9huSf9Gpj0lc1ZcTmZuVFRD0lax8BTsmkKkYuCyQR4hJwhpEU4zLLjj xBDJKkRDyfSM/flqkl7g89p3v9d8Yw1Yjlt/5JOGN5AZ5KDYP6LCycetLmzCPVyB5LNUy1WgmQAc p/K/xm9FZ/S9op2HRd8Me2KWL4MyQf2cvA5Tz7yyTO3DK/FVAF+N7vvLXXJVD13SOBXvNnyfL+DG w7xQPzARwraReyPAPxujxe6Wk/wtNNbU0MwxBofb5kbT2117fNKw+iO0I83fXxoN0V7ruPvYqJGz RlsoTE4eHihs8bRxH/dnPrrHj5zS+hxzkv/yH4XK7WVnQfwbTuor7GTDE0PSyqB38dyuNLYejtsG 5zkMd2pNzDea7BcOxzA4C5FVMiUuttYqnnBuFH5tbn5kWi6QLpUV/mUsZek66v8CwGfDCNFnPZkp 5O5HJ4wLJG85Syxm1oj3y8icQiH+neX40zWPUyIdmbXm+FYrHrDCMs6wEcMgfvOaecJk9lmgG/Wp GYAaFm6ot9I0OmeE0BIGjC6dPY679E9zyVdL9gOetfJInmsVB8SMfJU+WapGxx414AOogIjvUej7 rl+8LITPg5u4TPSTYMW9Z5HGPzxKH74/0E944m6ZnrXxYd+TDx4yI+py1dpagYLRYvrmX2R7ugpf NwiyZSCPYSzjDtuvQzsJ1L9+Yg5bv/r86X/dbZJcvFbE5fp3AoNiFB+Zcg+xTq0gYYk0v4BDDH05 0/uZJqpJ9K3RrelF5q8WoU3oWy241RCoGx4OKAkTW31zZvZoXU23azja9CEKej42NzufqAHtEOji Jmp9hU+R//td0ENwr+4FEscGibsY+joW7h3U1m5gO1AdeLO2buun/XnuLG0avlF6KHgtejs5Blqc OYG19gq71uTtzA1HWD0nWq8J+kT5YzlEYo6jzrruAD6PugkWSd6NFaCN3bUHpOGexg6ATdMSc1z0 DZBYzrjB5XKJitEYzFqR4iS/v77OInN/+85StUiK3OQqS+Uwz3n3j0PNPob+KWKf8XWJ1lVEotWI RdmJRnhOSTYB0UCzZ+qnpXp54gm5SE0jVXn+vHEl+ZihuaytGynOoD/TU7pQSgJoRmSePJbuHNSm K8sJlYiZ6qqeJ04G23Qlq4yqZi5a+7NX7Y0kSWZrEI996I/DBkKSYpC3QJJK/bsfdptaIXLp8YHD N6yQ2l3yF2Eh+QfiFVvEV9Fh/S61TlDG8ekC4jFisUeMmJHgGrj01+t3WWbE03pOYoEpE/vDr91Z VllCvdu+++M1Ljhe6YZNMBbcjg9ZYeJltSpsVby0Cvj2I234+efNoJTCP13Z3dltQMSE0RNaGz3J TdcWU09WwzlzO+gqs4kW4yRa0WHEIxk9PncztnxeZeZmXBtNgptjyOBkno/+Jh6mLfGgt0Q1/fLN /K3AwT/OS3HvJbSQvgYS8NtGbyYcsPD+lmDb+WDQ//Dt3we+Wn75iEjcrbhm1O2PHNbHTqP3zY6v S7QRE0tDzHQ0C8XKZKHd3ipFGF5cX3Lp9tKkXubFNCyrkXlYaVz/Axto0mmDl+afGU2T5ao6MzqK UaVEqhAPEYoTcd3vDydhSRn5Y3kWpL/w3FA23UD1UV/py5lI3JBwQUs+AWWc/o9zwDLRRFytDYHV OWI7RoX8ioscp3s1G1eGCW/KxqpaJi/TMSlHmEtHEXfs1t2OyJCyO2YzQpN1vabfLBnL6D92otlv lqzGZHtvseXsUS2HSuu+DRpch/eiHwVbaKkwLHJQ7aZfa6w9hrVPeuAAuZI1gugbEbwzYDXeC0xS P05JDdn18F176oMQtFDzTcVPMqF0eo7S5kv/pK45wLN1q5vwPOPjyooxSuvMrunVe4nrEmF+y8WU xbIs0MmfXc0qG6VToatpJD0KSY9aEqKY+WCQPkQ7iNlH/POWUtMlUE0JlIEDsj8VFC6ZCPJgZwEQ kALJZj7tiS+WFIuKwsjqQNM5oq+HeaZ2AZT6ZsVh7R+KxVpTdM3elPr2RiCag9uilW9QHiRFJgxK EBWN32vb9nDqnqULN/Px3GNXkbbbofXDPjO7a7d+jufROLd0WPetR3XiyfOcKG24sqSw7MleT4rK krZ4U1IYAiTZhaTI4hO2OfnrtkDtChBPmImJU4ldLF9aNpSO5J4Dm2jTEdRMa/gQiAMxjr3Hilmf 9S4DgDEz+qAF+hImWDps1KO/JP9lvAp224Zh6K/otmRYitixJxs9DdthPWzYhgI97KQ5bmLEqAM1 GZC/Hx8fpaRrMOxkS6JIipTI91hIW0KtNo2eLjffvIqndORCuvtyiRf0z3h66d3Vf8RTUsNwVh6y qT+cW3BCKj9/zIGxQGNxH458SntFnTWglMS4JNLD/Ygm91F38f9urpjrXtFSqTnA48Db21Gd+z5H YzDlGpM6jQag6IYR0u6MD0Mkj6moFJ/6yiewdQ32fJoDhfIMyhtqGX7Ri65FECkupfQ98hctTPlf wYSVSk0Akgw8BU7tDE0yz5ZaO0apWUdmF7iCG869U9DiAgXNwNpR+URRG9FSpJaOo5GOncwuBffc 1oddsmveXzF85om1BhrHkEv/9AYT3my6dPA+gcXkWSfwh36oHBe3L8xanCYLAqnJ8hWHWOXm6pkj kJuW1M+D+gGpA80LuRjl2ITeKyDmzwTzDdkuaWel/XVl/1G6alD2hRuGeaSoyeTTzzgCOVTcv5Iw NdYeW3PFcWGiR4eL0MEoPJNWeXaMW5XBFMYzC2VUbTZKhQdEdVoHm9QeW81uRR9pjuh1wzVzamyE 3EC5LvaBvvaYQqDv35Jm5AZZWIMcctVZsuorPOEnohNJjRPzXlyRfnc2DuoKAovapBq2KuMeRARA 4gNFvrq7b0AWWji9gg4F0wdTcbZU1JISry2xFqa44QJN/E5bi1Lbilp6NmflNvJnQTqA3iuaH3XE uZEuDqaUrh5UtbPN5tcYzMgp+yI36wFXJssIAuudGQ+cioehO+qOUbWGaCZP6Ka1ZkZt9lH6yMKb 8eT5X1mSKp4rVZNeQZOKbJOLLGpgrQx1gUe84arjEkpRa6WoSaVIqsHx6p5Jq7U4uufyXq7wpQHu XUvTwLnKmYL/93z/eMypasjz+BU6ypjp5DRFrac2cpnD84m/5ku35TBOJjIpLlolDYyBW9OnfLKM cpSIcTHGKb5wfZNa5x8BBgDUWlemCmVuZHN0cmVhbQ1lbmRvYmoNMTIgMCBvYmoNPDwgDS9UeXBl IC9QYWdlIA0vUGFyZW50IDUzIDAgUiANL1Jlc291cmNlcyAxMyAwIFIgDS9Db250ZW50cyAxNCAw IFIgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSAN L1JvdGF0ZSAwIA0+PiANZW5kb2JqDTEzIDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQg XSANL0ZvbnQgPDwgL1RUMiA2MSAwIFIgL1RUNCA2MyAwIFIgL1RUNiA3MSAwIFIgL1RUMTAgMzUg MCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgODggMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1 IDY0IDAgUiA+PiANPj4gDWVuZG9iag0xNCAwIG9iag08PCAvTGVuZ3RoIDY3NDMgL0ZpbHRlciAv RmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImsV81y28gRvvMp5gikBBiDf9SeYjnraA+prZCHrVLp AIMjCRFEMgDkXb/GJg+c7q97QFBSnD2k7BIxMz3931/3fNxtPux2qbFmd79p4qY0Cf3DR5GUcZMm hcnquEgTonjefLieCtNNIErM1G0+fN5a8zBtkjhJcrPrNhF/Ee2vm+AUmyLc/WPzl93Gmt5siszG SW5NkRE/k8RVaiJb8ffoNvebj7tXCti0JPqkNJWFcOacpYuQJGUpt8Htpzvz00vYxGkwhWVcB3OY xU1gvriuXW+70MZZYPptGJFhwXVIooNtWMQ2uDGTUJ5Ox3EWcjO38vGEE/NPoXBhZBPalVUfRjXd P4RRSnsPQrk/Ormp27+HdKPxatH9lK48h2mqKrXK8iDX58d2pj8izbiLw70Rha/DKD9rTyT0Y4Tk uHemVxNGp1r3o9ub+Rje7X7aJIbJi7I0u0/waVqyT+HSXFy6PxrmQMzCKIvzwLTDxTo2JrRpsCPB GUluQwpmIBRPYR2X8Bb/OvmBmyqygRcPQqF7csu0cmaOp7k/yrdwHbzD7tmLWeDaWXiObHVFHjrq iSfcknPYLdubGPaSxqXNC7a6yitvNOeoGJ02mkd/vzPXZKCtKTi2plidmFETtHP/Rb4G54X8GlpL jujDBAkX0SoNHskWigOkWl8hVDxFTunOgqOVZJRIO5B9VCJLPRGbIhNiLSRVUnSk3G0Q/oIM3nLg C9IiCtmTGekYmI4tqJBgFW2SJQ1pyCmWkf+rYA8qtow9mBGbI3/PITsTn10Ycbrie8D2ZG63P68l i8CUKa7M9iMXVBVcXxk3d/HdKjly2n4koW50hjyW0tWeLGZvciGxGMNKQRZnK/++0AblKzRW9d2i /syXYd83uBnY4n1XFXFK+FLW1rCs0pCEvBZ0WTzsUahsqjip/wAMNYmgUFklcVF6IMpXaKdVs0Ma ZIHlfMyDxMwoTw4IpzZVfIb67LF9/40QgGqYqIBHj3LLGcpf9u6Wkimj2k44pRDVDPEEk2dOPy48 CkpJjsHVPSFP96JH7uJINJnCmkv3b2EmsYZaZzzy2kGd+xDbomvXCgNZHQ/MyLQUVAAMJMjRCHUJ a456X33wR42D6u8bp+oO4Kdn6kIX8t/xfXSTtuRxjuPUA1IbSa5UqrxEqlvJNFpx9dDqQHqS91rd JWwSYtm/YnwnndxvWHZOWc1GP5R6mgRAWIiI/oYLjMjYNgNJwteDCFSyR6xUC8EV6ta5dGtbn3Gi rM+YQn3riTHl3NephuvMnqnTegF8xRVOQgIqAeqCk4Gd7Ua1e/LqmrZDtBgEnqmzLKTHfTuvwTFH tlCjD/Yu4ggWwPMmGF1r9m5oFTK/UTrUnv/+BSvHgmTrxBBFbdDtlclRNu5lFQnAUxhh4OvIV0vk MzFz6B/YzRkcWwH0cvQeXu3liCRHjJyMV5oqpVJ2j0LSH+T3Afeky6yaTNF4DbL6PK3U6unbTyH3 zDuUIdAhJYPmMEciW4ZEWfAEwM4YaZqokBS8O/b46WQlC+VxwAW9rXuYFwibQx5T5PsGUSrJeRSB gtsGCBHXHHGFb5991FglYb1X0tN4VCHdccCHaBijGxojQNh4CwgnCCtCi/HnpFedsFybIaKEIcv4 Ip+ORyUkYYUqWHJMEIlJZg4Ze+x9BGiWPLASBA5cg6YYzE6HGPSnVDol2guTTKLMF5koHC4gJXJP 7/z9Ay7QeHnNAFqTs3n2u5HsaiS7LMcjYtDhPppSB48wy6R6eJJfKDbSbOAGL/pF1DH7syYs9Cu2 +5BLsHMTgfvNfEZ0rotXBhnR5ihs5gvLzL3MbPztb4kDeiUbnEIWA/uolk2qtR6MMiLOq1jk+RKL V2+EW+Rkg/zM9FvyM5f8xJRxzk4raVn5nzZM4XqbIDmRIzxFKvFXPnCj4SypvCiVQr76HGYYv5nH DBJM2xTalFMNczn5jf4oX+RZhm5GUyFuXORjxoDBFEIPOiN0R5gxcbeUV4bcRv0Q9pCS73oiwbS1 csTewSixqHOq2DnkiYY8keZolzNRGifillD7s2h3MbO+LaGlY6TaMRB0eQ5x0Dl1CkmBVFKgkAQs l5GUkQWvlme8vJA9tdRFqumEz2lo0WdxRYiMsFe+Z3bCX0C6QMUx1GF+kcWN2fo1Sg1wVyxwJ1yq s05izaIZCCbN+VTrM/VkWqHb6+2NlHWGsraLrHQZUfGSpZdvU/PMmeXUw74/nCY5z5o6nNr/PpwK Z5pOizqPubqez1FEmZX5Mhnc8mhfy8SPCnt+JktSBrPDXremNyrXJWvyWuO3ipQEa5eKoN6Lc8EX mjnj8bdvAumrt10qr0vuSCOGxlSGRgr3hOblNPVlSuEAcfbzyZV5kQ/dPwjVg/zIyFlj5MwBLtxl IpGYiUQ/gpYye6b+h6Gh9Pz2QjspsPlHB3Wxqil0uDpPYTGPYBevEziIHZrCX/T8qOvvJkHRZOzJ /50E/oVCBctE6vq3bY9fuD+eYZoAq2Cs46bfjr5BaWGkvn/4njT7JtAPve+BhI0NBzHNzw20nx+l 0/jYSuvyV2SFp8O6eU29F6tNxaie0pM8s6W5Ul8dOxHUz657xV1akHvde08XHcr3QMjXRce9T78H HFCb+LMYNs9O2OkrYGn5S0vsBuf75uH9UaSwSzn4oPCDpmZIqfEuwCvMhfx37HF0eJD3kL6pap2/ +AjzV4EBuv7+JGqXXNBU+HRneMDdoVNWVGtV0E+G/reYMRCWKri/B+Oe/ZKRd6tgmPnRxxS86kfH yCmNk9Ga9mdxCM3VHbjK0dcQRAlmpAMtEA2JBaVfN/eduzKTYDlz4u62aq4pbfr7HVoFaI4icc3L if7dbB6FwFZgQUma6/xlAdQZp5Gy9nKH5ffd8cWm3o9W53nK4kqgK6c4DBhhZQvaVpxAf2UlOFUg rPAnF2TjFZk1OqoW3hQ+pkXOgbO0xBKzeIWGqSLZdv6F9dYrMoOBLjoRcaGjGVquXu9VUqBXSQ/C /5JtL1K7/rRm4qaLCEnWNOpE6yU9ru0clIHxhk1iMSHCrNeEnlKRXz4Rf6tTZknw6N0Mfwt3hvHu c7ua8AXq3pvwlxp+ZwRfgQ7kf9jtbGJIyD11ybiy1NETr0W9bre3I5eYIzkF4RinXRxanizvPJ9U 2JAHy7oSLpFWqdiiPZOQjLKaH3dHzBusEA+G/J77UUNw/bN+vNGSRBbV/1VJOLxaHJ4JqxiQwk8Q uJpbhT5gMhkU/QiVCcbSbDX1es65wpA96wn6UKETYIUwZcEsXasCzldAawZmn4T/4kYtuZ8HOk9m epWHSuRXGej+A3j8W1KK1C2og76Ca7uYaDP/cqxkwE5YsIPgjLNf8ooX30AyAWCM/KDyMrwfGrwL wGSmxyLDfk5oVNJ8wvPHjZFDvTFxoCuySGYzcYwXdAClmdvhSb+OZs3wRrE9oqjmwS+hldcDk/Zg 2Hn9hfMP2Hyt8qLsugVjxF1z7XkYU/UWvZT7Sj2HnoCHz4UvnDh0RDB2f5L8Kpf8ysX5yIgcj96S n3UtOZCH74NsoK5znTQqlLu/InlTB0+g/IEShDRt9YJc3wulmfSKwRxTsuXoGQU5NcdDQnlFeKn9 wppbBrnSjxEUKa/DBHeUeDt6ccOTbnhBvedPpqgWqJZMLe2hbuftm8x/SK+W5caRI3ifr+gjGCHR QOPti2Ps8GEjPPaGNQ4fRnOAQIjCLodkgBB35zu8/+DfdGVlNQiSGsljH0ShX9XV9cjKuo9MhV8W SmafpvfLPgHSC33PFR2CnEedDqcOJ332/N/YioJgcMzteeanJT1DTMqFit/pyyu5HOGsxkL5vV0A qJCtClPYcQOKh+gelC0EEWH2qOHUcRG5jMmvYXVcxNGTLY76L2w57Pl/Lny8X+DfK/woK059ShEo kjyhjk48KdHapBDSL4gsFSBN+NDzfHpYxDQsEuQLGF0WRlvuG5kqvWlMMTayA/uN7qHETpvZFbBU XJdH2j5JEh3s3v1+N5jMR/IZbaUk7t2dAvt7fcqHWy76G/fwfKHJ7D0Tn/fVJGekck/h5p1pYte5 89fubdWs1VDw2XOXJ4JVZmVwQ1pPbkA9n9zgz9xQieBa6YeARw6er+whk6sGRDn6AF1fyGuavZgy U+fgt23G2Uie4ZWtYzt5tU6DV+PDrXVF+sCIK51xFDFOaVwiVi5RK5eAOjeIVgxXATI5tLsox/0E ZhMU5UFbeRio24r719NTfjd/4Y4ng1aCnRQidPPCSG5FeY8Ai+mfHgmv2ZqsF3sWn4Vsjy0zDgsE DjiUvBTlZiXtIyYFBStDDtxx1FEn6YC05tZNaHQqxViOeKxTpkVefrtQxVKwz8Ka0lyzAwyFv+Ea 08G2wonwnxJcHyvdxu+AtkMvfVHYjse5ZQkbiyw1JXoKUxBv7CQm5o9UEoAPkAByAx3yhWoVhCcF 2y3jvM6l1Snmran4gtdDcaC7HN437c+iZRV1I+mqOjFnSHmtbp4aKHcdVp1GQ4Zo+Cfu1rPdyvG4 HXMbjnaU7XodcvP5Ro6eKIbbBmfR45H0PuSSt5NUHc74Vbyg3w8dBTR27KgjStsNczyYwXJyguVk hgfCTt17vFb9rq0IMRhREbSoFDftnbypsuWefcWBXGvLkyFdv5BGPlMw91CevMF18wlKe5hfcCBx eqJMarCatUnQmTspwjaKOTczkXP9V+78yttau0A9xsxTRvWwW1GuaXgj1ueE7W052je8iL9fz99u Rv2r2lhqR/QDmZmBShEltEr8MlZk6ayzuIcAVHwJ/cK+f7hfLENfkYUeSnoK6VS8q0VEJiCTFhK7 gjpfIKl9F4kui48/4UhhR5Y+ZyMiinPFhMXTSsini9QiY7zrpPPY0c/CG6SA/aX52g0XHY/YuVDd 9GPSLpO4hXKJrh2Mo05sNTa2+unv2mB+BqoVSsQC8uSKOan9EjAyhZ6CMJaJ81bN2Ggp0VkJj4Ma cL/jDEeANjDroZnPNu3Yh5vm827Xts86HsA2KjnKZWHSRLokoq4/Ij8EZ9onXTfp/OWdG2J3GdmO MUi0q1bdCHomMisABB7YbwWBHhSUy+j8eKeW//PHd2rkRGwsa95lcb6U8oOCV4kF3ol/g83zNOHO QqLYxdgNppl77vvjx295LxX2ce0+Roo60luggLdLUo99Q2YsOHvjbNY1KyW2pOjENK2jGLSdswOg 2PmUW4exGawJzW3rmtJGm3zSYecuMyRbxqVPT3FdVFOaRWNz+NlywIfsKKs0sd2aQEtsmFk3FfBz SSE01b9u3KyEscy6ybetm/jCzAvnFlfmTSbrBhA/8blMo0B60gOYNHj6M0bsUwDGWnV0IJ1lNDpp SHbWVvyysP5VmZPZGS2rGJGW5qHJixL5/O8aQQDbscYEtjk7utKGAsgnrKhhesnMgZwNXi3Rf6JR UtWbA11uS5JOFx4Ue6ZVnv8vHlT0msDFSDEiJjea77WTLVFi132Iyo4zWzxFdtgGO9WEbKPbxM9i Y9EorZM3IyLN4d+38+0UEXGNHL6KiIlVhn5LrVwqZ7otxZJgZWLkbouCWoiTSq2cJNQle6qYfFvB h1VRN4m3jRIpEGaKXVhxNqJ0WLFW+qnirtJOeqY4Lf7/tJvMHFdLeW2SC+S9gWpJBaO9nXgB1ry4 rvTIsy+zQowaJCpMhj/T7g9Yik8+yfKyOr3Vp1M9K+mfU8aiQKFOFKwTldWJlHUijfiLOiF8391H 7e5svoXZo94m7xfCGyMkdbfuTUDbCXG9o2xedCqeKCe1/rFwliZWCyfKTy4evUFoSCKiiDoTChqV axKIoqrCs85zViPIRAm7jknVS+XWGbsEORWUDlBztlPSfq2rKuo4M0I4dAEKNbnTq/GF8JncFxNL vXhdAigDSXo1iuICAfRSsr5FcnwlTCoEUzLBd1KHDtBQG4R7szmZo1Jiie5TIZoAnUUz/us1ixPl v2wEKKa30VZHisQAgKl46o1Ma7i/ZFEAgXNcOxew1l+q0jnWmUKCCez2R/jHISpLA5MEUdlTZY7J UifTS9eWZLl2J1k55Ug+5Uh+XddSfRoIQA9OJG8K/xvHD4lEsIrddq3bBOzm8411lmSLwvW2I0lT ZWI1rvPofFIK0dY+mofdkZ/GI3PJgaDDpnscg23t5AI9r3SbzaTmfjd2PCiIukF5fuaDhivVbi5r ii+ASC4tC5SW18LUl+V/ReGmkuLz5EUON5UUbyVFyYNnKAIbxo3GSC1dT4GoPChPKwIBq5WAZUpm x1nIAmR6ifIH7sbblRxvumZAoNqZcyFkEbIdwS7x22pIFAI6BWuctIq93b+lOqMN9/OD7eHi3o3t MsXc/mx1GJve9LiQGjTtHBds2sJecbSwEplriawQBEFte+V2rnBQ6Mr1OcAjLdO32IQXCvp9/NIL aJYF69w5n/CzzlOLiaSxVxvWBhuVprZHdbnSWLoCidJMNr0OqR6NSfFdGnsvpfxUmdPk1JMmJ9S4 W+TR+4WAVBZ9sIbwlv2pV1QQwilP8WrzWBEQtJgr4mLmJkiN5CY8XRNShUaETeeAhiIufcAVx6RG D8oajUQ+ifl8HB37Oeccggl9LorAMj6DZYQfxmcmvLKMr6Vjq16wzInlgi9CjfZJi39jnGOrI/5q 7KVCL+BrW1/rClyeXpxp2WBO7iwvHw+So3cLOePdv3f/EiNmcEqszcUFOVzWaT3jS0U2M98YqInU cDt2Iom1xMx07Lq0uvM6dtRfpa11tLGl4wsVkwXPzWtYh3Z1vtO6IIgabqccCEahY+DJBDRV0rd+ I33RR2bfhdyJbg4OryeHJ8FwA5xYR91IXEvUkRLU7vCkCzO2sdWJHbeN7mAHupbPlUbtq+A/6pjk gy71w4ofV97MlhJ0845rKiZJbrX97k9CQlfdcXFbKIWJlahe+De/oHQv+Dd4AA/MCE5I82Hs2z5M jZPrnLVFCbvP1KhPP44n9+sZ879tXtkdQ9d2/ZEDsq7z7eccjK+5laKZVJIRkmH6mm/SnRkN8Amh IQVZ6Nx+2LXdimwBFFpZWNi16jau21ycDUPjG+Q0xkRAs1nHIV5rrhKg8cqPpcRZNcuv72jZDHfa CYvwQulDyOVSZUlZ9CwqRcanpFjeffiHU5hl74DekyNk2uA+dcu1Ti/liKMEfUWBpynRQ41lZ1PS FB71n0wB27mnGz6bb8QsaZKqb7Js8s0UZv5E0FG0y/OEKTVdJFnaoUOQNZcGLGd5AHG5P4O1gxL8 F0CtTObJkwc7+hAp7vmAeC2UZrC8rcGUiW063ir/7Bjll3ply7SSRDx5akqr8F7lQMOxb4V4dZv+ 2A1fhb8O44WuUr7zepaf1STILDfjfTB8r41IBfbUue5X/d5Ls6AdJKiwdhGVZhLG3A02yK+OPSiy +ADkALGqlezTn94KyWWuXUXiTrsDC4dmZHBgRnLcRtqrWhzOO1QaGSFlqUO62kAhceZ8u766DNsp dqNXrtyube0ed+wbiZ1V/2jiu6HjtVcZKTX+VGr1ZSleRo/VM4+hJ+sXafSK46Ygg5xsolEWAr+5 T3cqUyo2eMoHhbYcNAql2AuJkcT9t8uWiSgdxRgvS/Ytt1LRvCDHpRviKTy8hYcWGTkop+VfypH9 +6yoJJDxt0d2Zu2OC+D8FdwD+Dr8h/Zq6Y0aBsJ/JcdEaqPaecONIqTeKrUSB8oh23q7ESG7SrKg /nvmmxnnsVsVEOKSXdvxeByPvwfZsabjntH1Go/OsB4bfV0GA21tJ5fH29nxYDOQ/ajVZR7lxUEt h4hEOpXDqiVR9Y0hWK81SthVhGRaHMwBZOah5u08T8+fBEBZ/vbiiqb8k9s7F8GM7DFg/TKRu31y hK/o7ZutKOtz+DNxbtL/ADNXk7hDhAIr36uxumYrdQs/SGbLWM9zQihGn4Q4QBr8vYBeqBRwLNMS jXezXTPsQdGsO50dgT4nrWyY/0sWhrBFtfbJy7Qzn+WBylKXeYksYkwLtDzT1X27HmcfUooPeYqV s0gqFUl2drnO9P4HgL/h60JIFzQoqeSVY6LKzJJTSPEy/uc/gos/pSoUlqcEmrtr1Hga3hGt2vAm 0PjRFWPwZc7yDF1ySDSF4NrPrjcyRtAd9vXjuWhJ4tJUi/0kxVK09M9uXMsWG2ekW94kMehU1ol0 TBkd6cXER56utKksRNhtTXGqKs4FeiOi2wOXWQKXV+SBtnB6uQeuNBT9vgAuI8BlAFzU3TpWuBlr F4BnS9UVq2+jxHChPPYXM/jriX/5GKXh18+ORBY8BtU06vOHNBw3ICMtBNixk+4nqqzuSKILsLxx fbDXN4L9KFy9W87vqXIGPlJ5NhFKYtPyjwt0LentNcBxGWDQapIhTUKqYTZhNi7SBV2aYjrbRFUY Tg7r0L3LuewSvnbQ9QktyIOszC3bBdAyn3oRjt7zJZmNcwI1A6a0QZ7B8C2s3pmDS0lZFzndWENR S2/gzOQIjOq8x13NC2dsQf3TAbJSBiBkXYate+YBlKcN11PUU01us4jTNFswSJLMgM6fJHwXPITm IQpu+J4sLa2kjX0a+M6CbF76pqU1lm7F3zhac0XKoQQ/yCex012cpTgA/sjK+QDskQ6vuL6zHJTh dmwOrRNDIU6o1OmQfZkCtdWRR6eRxD4kovWyUDQeyPye16iIYwCrtyqtyZuNe6aaKtyTE5Noz/xc LQ18o+u8cU0noyRoZETS6ffEJ0/BhrEfcEm4eAmCbHhYJ2FDd7K05AEQLRhEkb3TPUlLnjLfb6ub MWBJH2byPEbZta11vy+RqUTEgOb791QfVmwvVcmnummPnEjv8OnQvZXtCSunzMqZsHIlKi5leizE RHk+XnYvBDQQpCHX5bPZgs6trkETg9XEluERBMph+BsWepFNuJFXB/5kbjzJd9S5O8lJ/IkhztCV ZYWR1BterodvPpoUwoNgCbsD7ncnEckl6LYkG03Do8gvAQYAIuA/TgplbmRzdHJlYW0NZW5kb2Jq DTE1IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1MyAwIFIgDS9SZXNvdXJjZXMgMTYg MCBSIA0vQ29udGVudHMgMTcgMCBSIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJv eCBbIDAgMCA2MTIgNzkyIF0gDS9Sb3RhdGUgMCANPj4gDWVuZG9iag0xNiAwIG9iag08PCANL1By b2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNjEgMCBSIC9UVDQgNjMgMCBSIC9U VDYgNzEgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgODggMCBSID4+IA0vQ29sb3JTcGFjZSA8 PCAvQ3M1IDY0IDAgUiA+PiANPj4gDWVuZG9iag0xNyAwIG9iag08PCAvTGVuZ3RoIDY1ODcgL0Zp bHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImMV0uT27gRvutX4EimZmgSfFdOtjdJObXZ uNaq2oPtA0VBEmOOpCU5Hju/Pt39NShRHtsplzUE0Oj348Or9erFem1NYta7VR3VhYnpn3zkcRHV Ns5NWkW5jYniYfXi9ZibdhSi2Izt6sU/3iVmP67iKI4zs25X9/xFtE+r4ByZIlz/Z/W39SoxnVnl aRLFWWLylPiZOCqtuU9K/h7card6tb5RILEF0ceFKRMRzpxtykL4Ky5ZyPtgOoRVlAbO7MIkDprw PkmiMuh6tzXt6Shn+HXt1J3oOI8q3fkQhvc2+Kv5EKQfQvMyTNKoCPrerIkqjbLgdVjQxtvwvqQr z3E7jmEe1YFpBmceZRsbJLsDmXFfmnbqw/s6+IqNib+9ziD3Sj+EdDWMSbKhHRsltGMT+uMJjip7 YJ0z0jlMbOC1VZ6kSjeGH9f/XMXk3ijJi8Ksf0GAriKUwXtHchrdZb+QqUaXjgyYSLVkPjjgoAlt lAeflUoWF1WT5HKBtACD+eJ9TdvK35y8IPPmrTmfBtCKOVtHgWSGnewdr2VtjZ55KUshZHl4T0kb vKPApcFr/HljzsNJrWlPvXzcmafQclQPYARZrQhRZp7XGFLsFyoNDsY8hZRLpfqkBzXY34Ncbw3s Ib0LO9SorUSu9He/hpYPIwnePeVikuUcwjIrfQSpGjX9bY4Avv/9o3nN7ihJo4ewjKz+cs6QjCP5 IGdJOcdq5FyugylMiGLoztjuZHnEYi8XKL/6EzaUkWexWez2DQS0WHYLJlxfKZdO0w7KbBwv+VJS cB6FsJ+8Kr2jVOMtb9I4mWZpg/51kPhZzmBBKwlArq3U7MkNUvA1haJin7RujIx5M5kRUppe6Xtv l7I1qs+sdqfnvUii7D49X2Pcltq5VXGE2MQymLq++6+7Nj6jE2KZztanbH17wudRbrWSIanyGAaH bY6f1zFFKCw3QlmOMCWFKbQvJkq6kuLKwStykJXKHKfRNCoRZFscmO2C42c560QJ8qjnxT7OPC8q TfB2Xk21QVdeZJhJI9QlWJhel42Xv1lYiqSDd2jZYbUHDcrnqnry2scmtXP/i62Wzy8fza/cuQrW yFImNFuzaVhCzRlFNqYkYC8rmlM8ZLjBy/4wkmBK7+kkwwEXkE+PspKA1RKDlOeEqEbzNsO8TYrI JkVN804VLCpJHh1szfgptMxs9Pd0TpOzi6q8XEvmlMPN64CU3H5SOJwKSvp0LTVpRhRLKpVP7is5 /6SCg27qmomDEAxULzxhMuJ0IBLMFxoUyJrAgAoyOFycJOyioWOaPZpZQqOpyvPbYrF2nuYVVMeQ tTJHq2Aamu44GYy/SqZhcNKKBt1efmWoSjbzyuGyu3V3HWV19iNns7ee8XUVX/naVrPK6m1y0B/O NORMwkmY2IXkxLbj/Mgk+OSibsMQgYzi0Si0B1k7UH2jLpmcZ+ShWXRulwrL7U8hB/VnWsf1nCKJ 5siHoEVKYGBgeJRSYlL8NYqfeGtPuNlGfVMpdAPVRd9Mzh/oX+E0NUgAS6VZl8XPu2WzbDzj1Pj+ hf6BhkHQx8+U9GamZErY+3tXzfWm25llL5IkLoJpvDM9BllK7r0IvfNiZIqWaKjd5Hs0mH+Pp2n+ /476fb201ZnlyZLXYCY9Jh9Bqc5BgJNgrP+CrjG3jaRW3+8lmfArvK0wtRRmbFKUWblMWuA996Zh J2l4wibGbYomA2BrpVb5Mhnm/gTdI7iBRS8KZwKkryNZKRnmI5/3MigynefV3Oi0k1kVBF15UnMX 6C+HchUV8zMzUq2Lb434ET57BmG//136CqE0+XtSzKdwWf80t7CQW8hp2LphgYL3Ck3n/v4cGAav s6Mx1n7ChUf8OSvjxgsyzfgVn7rTAtkqBB7oLULs/RvhUaDw4ERn68VOV7j25i2gcHk4PWtFBBBD aEzB/+Kuqjzq4Yby7sTy+T+4bYC13c1L5YzTs8PxzSviWcz2DC5oT/QYo6Djt5lRJQ/zLYDCZfg3 ZksFxkedzPnW3RkFEFs+P8inZwJKIAhuorVWWSwTjsUZv/yCwxZoYuJnJU+ORBVBhpjNV545F2bN RPoshF2ppVqfB4evluYS8Z15vMNb5GVIEz8P/iWTN+UHDc90+wNwZeenSZxfsNW/JYd5VucyqylH 2gZLjun1M5fhnfRrNrEKQDQ1iyVrKi2m4NdSLMnc9yfdoobMzw22lqpdxTlughDs6R4e4Cfw3AK/ +TOocuYOwM8RangboD7JeJnkjIC4w9aSA5yrlbxBVFEhUukTZDtzdF+m6Pn0S+bhnOhwfsXc0f7o scZMpBFNF39VpKoLY21OiTSnWlBTrmgoC3ALgA0HoJImxx2woyLpsNfhyhFHe9mLuKKIzvz9RL2l 8R2l1P5dB2dlpDy8ntRyT23Tm2ZwzZ3fVAGqyMIObwNHVLKi2bo/H7X3xpT2krbXk2tGDYmiBhHC bw1Sk5VK0dctgp4Jxi3l4WS5r+le33P14PtRzne7kLWkbc5S3vHnnt8lY2N+P2J3u6QZuYZKft5l 8qZhljtuGpTcbeeO2IcuPTruV0XlfTMo2N57i1TZ2ZLPOICswcymSxkwkuQyuJxvr+2bltJasSal NR+2frx6nf1dtVKxnA9CNgeh0CAM4rNM8JGwoO48jNILS8y1jCtWXfjrSx5fZfCbGc84cU5JRs4d Aw7IjoLhmgMf2FmKSTxN5MpBFuqtQsNWKM0wiuE19TFmuev63ugO7rkl/c4rT9j+FQcz1YPJdLh2 VHH+utr0BJO6kBdqjESCch7l0CwyyPrThTZqw/aqXWTZBTPPjdZqoxW4lwsmsgpAcwGTNtCFQENK NsGP0qp0nhXSqoq5PfqHgNVfQP9cKbfg5l8EN9sEDUaVpxrheHSj8RIZSSfciPjAOb3p/5puN7/n hPwAOjP2XrmnUJ6CVKF/MArPg5e/ebSeC1pnLkthk2rDMb5yhoEsygW3fb4xs8/F09Z7ugUYAupJ BVxwaxF+jzgRmER8ezQBamqcEYb0zqUN5lL6cnbADRNa1nPbgCnjtzK4mhU/HbL8DllLRGyAedNh MJnNPHt1A6NKp5uEMZE4ybDSWzrbQErNVwebDsHrUVlLngDcsCU6UPVcRXIp49NdEU0ypM1iuptt t1MpNLWUznjLXgteectRJ5/pUG+OCgyEdQ9ukEa1+7LX+etVhk17BQSN6uUBQ/89fFhd8GEFt6/F c3XAoD4J3rI63FEF9QEnVgLx6M0SZmKaw0cnv59BMIxoI838QpG2LLBuBmWEAOQO2BlcwRZ1FL35 wGlW0BN1QlUcVADNfqFUQTNaqPHYsTOKBBkO9lCDk70Wd15JRLtiVFbO9Diiya3CJdcp3G5p1o3P roeJLea81meTxLfQ+ArWoaKQ+JYSX8vxfcfAjRMwDfCLhM4CXNpymmeBAT1nXBoM3bkTaMjQsR1O OMOvploiqVYj1XJNtW9U0SuXxp8C5bCCNF6Ary2DANahn0yH6xyhSopfmPlyyoOf2qESzwudO7Fq o54RUMygxakmR1zdL5xdzc7W1477EnLd8EOD//JD46SfBuqlKP9cs+XO4Pgov33I3UQMhpYyGWfT YFSuvzDN6lUxzc6XWOlcXFNJm060weRKwb6vkQZcREKmd7nNgOfjLNXyi3ADEVxR/6O8WpYbR47g XV/R4RPokLTEGwgfHF7tHBSxnrBH2pjDzhwgEpTgoUgtCGpXv+EvdmVWdRMgNVrrQBDoR3V1PbKy SEV4rHUhlXaW/q7ktxzt9AhdIwXk1Uqc1gdcqK0UwzqXRGRppUDir1gN9P0a2QNetW2RVGTktTJy BBHyCB1sv33mcDdTGsBJocXuUYtJO/PdpKZyHZk0C8YVamThpfa4X20wUtiT+ZvZ7nvtOumujBZn XeJfeXTS0o6iqbhtIsqHx934SEmG3/ajG/VqBm8DXfTABVIDTyCYBi5iH7S1NUmrvUczobEAIala GS6WK4VKcJVNg7n1ZG4glciiF8a2GcwykxMP0A/lSra6v8hgTbtxc09rZBHlbsB3KMFL8tO22m9+ vZqfUmhU888gBwfauDA2uDEUZiwYeTTevOZgMwQWO9liNHRCXO+1HDwiNWptVETM0HmBLaEQ3P9K ArcEFAJNAIXYaIRen60R9PXOfYkO2o40aBdDNx2XNjD6MtPGMYsCaTe5rru5UtZ9Q3S7duwozCJ8 h4Bucs1D8NfqD0ZaYOivV/Y41J3Y6g4tkJHuy3b72uofmlcBEmefK0K1/+qRbaVaO1Ndpdq0SMUw as7M2Pim9mx06YYn+oW7oe/scDqutPl7Pi+Nwfo7s5PyGVwQaify48S8dTigb1X9oKhOL9oDRyhJ mEb63tndenH01Sd4KI4+EpHFl+MOqAards167fWxUe/YmI7NxbEGrb6nTII7jGf17aLtntmKUNMy XLJmh+KfjU5uuJJ3FN5ijUBsw+qpCp5KfXtWebE924HaT9pS5FmILIYS/D8V2w2dmriyQJYT3M3V zfV3NeXaiapu7YdfZnFwi2h1rjlxWDjVeu1t01vLFEcP3lgUpxd4lx38JVw37F6ve0nArbnh1lMr GigViyN9oiPQAppplS/sloDaFw63XNIrM/Llv2b5l0nHw3+4vc2cIP7qTG5Q5xIX81B+S6oBDaKh 6e+Fft7+BxsS3ZBc5mWVHtbPQ/Pk2zmhkB+tORTwHLTbbAb/1u286x+DV5B3C9LE3LrTxr42tgfl UJ0t4LCYzh63oK3NmvwnnV3PwHt8xysRQaa4tC5W9JPMcoPvch+DwwFR57Qapabz+Bjvkjp4zqro Xbto9oz8HTUUwGKz0j1zEI3BRWVMFEyhCjgDLHyEr0GNOb5UEW6yRRmB0w9Ifw5vvXvQo/tBd55r 4XWtX4Noib0gE7tSduLY2tVRACxAC+JWEs/iLAYfju0o201SWEZKovNwhGmuJ201Mawx8jcf41Rg 0LExaPRKJXslMoy+6TaDvjvNbLZoc63NcUG8vwBBuec2LVOVpW1CvM0RntqR1JFOdybS/w8oDblN tk5H/Xkr8uZWTbpTUgH6INfst3tu0c/2XAtxNamhEtiLQeDHHzn4Sd36pOrr0Vv9WC+9Xl4j7xrY PkWZgl3/ydEyuuApiZPugxKFJjRCFEwVb6eVN/yvUUebpdxoRVkq11L/Jcb8G66cGDYKbehfKCIW JJi73eKsIAc7wMgBRcTggJAPt2dxLcRDOGeaS84VrsQDpLWSwnm2Ovvx9kzIVEGJfEnmgk6JwGGa p5foYR+/g5RXyOI02hPjepTdSuli5RubWru4nD2UNDXibIn2Jf88ft7rduS8ZEBro4ztJPKfXjRj uaGUYTtpWNGFEplbHKXVOSftgh2v/8s9AQIpXGG9U0XthGbyOflQRT1glAoYsQIG6gIcI+ZW8wVz p3PhbNnY4HPvvDyNxb6yskhgZylZicN18+Q7nomTQhbCMxlXiWdCJDCDQ2MR+x4OYWRMwApvrUGF yt4sZmxLe1u2mxaJmvkRw5UxotA4dX3g1AU5dQZOXXtGqLw8tmfgy354J/4Q5sPSkjLnYq/l3qRL yai8rtoGBJ2lcZQGzC/H3q5vTVkFHH/O8DejCkCPSjmAxN+rAmyjkpojMecTNpiZxhJkR7YcbMKH gfpKnJtlcG5aJ/DYm2GQz7HkPWGQzk9iICMGsM+MUwsCYv8MhTsGQb7Xv8PFEmDLlvBciVU4u5b1 S9c8Pa3Ro0gaJ4rSORfChXh/0MWta032gNU4qweGqz0T9hsl67IYLea8SbFt5+Jz9B+92278+d85 dnSRZYdFz/qB8aUp32ifgTu4n2dE6F+Qyrlw/JQ9G7RxzXLZt16zg65LBB1t8XLizZRuTNO3ITRj Rh88RHekgbMVxtmAikCpQ8e3s+/djsBXszFEadlcelVUdlAlE0L1pi45KMZIl/moRliF0CUiMhG0 S5zY4O1ATet3RSk430mcJqE/mlt/9BlXr8ikJRSkklaSu6ikSo/QrTR33bojU4i1sqSsA2QECQuR 7Hja9kIrySC8NH3XZzdDdm83XOzRXDE+CUt4gn4M9qUL7nXbFrSiHhWUSvELQbMWnBSEXNCDVaS6 Nxvdr/qsVTR0v2vd9olUobKSJNNSvK8Hx7qWs/+8O8UVIdSIxDr70/Ii6V28y2FSsuCUY4+VwWMG LMo800ifnaRWFt3J1UlrM+WeKeqMQ08jFxQGM+DSfWtL9rpEBcA2ahRJTNm0eOAWFtgy0qeuWXOR G2/bIXpKr8pWV1+eWE2ul6QZrvumyYSuiQX+3Gb1XE2WVJKkBRqRY6OdsKZfP4mKZfTV6X+72D6P O0jU1cCa0L9NG0nQGE+K8I5fR1q0tc5z3fiOVFf1gXqnR1JwxJEMozZBVOhxqY9vcrHhydjWwjgS HRqbnEXDqj+WhRy189HyMb2z6GndeQFK7o5ooyG06ig+eJexvPPppFh8VMHjmcBR9bb35xmYm3m/ /D+8X1ZGlKe+L4LvM/U9olMxoLR3fcJqqVk/p8WKYH27YGnbevclsuq00mLX+32ktejS3MqXQezo 1u3SIiAWvlaIgf4F9pUD/fQsfZKr6Su712zURFEeE1q4unrPU/jMHyyo92V2eWz1smBTIUlXvs1/ kqp8Jw1OCjCmV7IuDlkX+6z76asTIhApb03JW0vYATDhfleOAKRFjIGlFBFarkwJHoJ+RbYvIcfy wY6uxM3BMLVdo0GweGjc2oSA56KQhCV7lSoApt9KcCUBB9f2vQ32O+ffeKrzOm24+36s2c4Itp+8 BE8Sue5HVMXUDhwmNJbNEhUeHMsb8pcHNe51C00V0Gfg9hg+Atokr1GXUuk934bapMxHrPeNZDt4 PatPUy0NqVaqwxGyTADp4lhupVozc0BAARCSHpt7nfgy+zsjWe4ZgiQl169IbnUVEQ+A3VG25Kzi 3IarvH31q22XdqZWM4BYTpqpgyq5s0+4IPILxGkxqYxIOalfSVaSoaUV6tObds1fr18/3N5mTjJm JTK5R/4CjiV02pQkCgUBTZR9he27THLZd/vTmVxaZ0ziPMzQE0noRPAKp9z2zWYHfnbuPrbD70Ch QtqTb67ZLGE/VBQxZDGPfhZjJ0rFxCGbbzSEnJToSd/BYRAuqv86Bnz66j7DvkXks+sbA3k3aoUy o4hxNGhiPHCJ07awiP7gZ7ezWV26UXGalz6jNEl2g5S71pLffZgI9dhi6SSs1Q7RAb/p2iehKf8P TeGPrrNzJ9DU7IZec35haKRH6ljfCh3obq4UG25Yd64dUUp12o5hau1V2IznBgmm38bLOmn2GtNF W6a5hN1lnBeFD4b0NBgEO5Nou2T5txJTsxLWyCsdRXbVAFhJow+SNTXLJ3rFfmMfFzHo38B97mHP 0TuU1So6d2SFCUAvA5NWobp28TARoDvQgX7GRSpYGXs+un6rQnVb29tKJLbJ6LyQ55GCh96gDvfa mAY6tofDKvsQYYv2aWjAn5/a3syx7QlFiRqroWC9+KI9d//e3tDat39VEKwCCJqJSZ0raWihMwB7 z+++o3nz6IWfijKSe3GWw29lXoccDiKT5FBIf9kh8nK2Q7SF8IHCklVJZYaS4z83nLSljV1IB5cq BtUO/ytAbC5/UO6Cz+F/hFdNb9NAFPwre4ylFsmJ26hHxIUixIEeOAAHY6+aSCa2Nk6l9Nfz3sys 4zQBLom9+769O2+ejGwg74W2+nFrT/zYUFdbneRpI4yxkUMaoFu56/qcBf6OjHDeKEtwgjobdQXr USsCxRJXHRtPhd9I3KoHvTyixSM7xnKQ02zsWJROVkMXFeHLPMVUP89fw5s6oCG7uf14c/3GVdV0 40rRT54rz6pPdqBWGDgNHBxwMI41MZwvtCHVw9Y6U2mP/JfikIyKQyZS43dReqBRmiOE/cOW3jXl d1ZcZxeyHDsW7kjVW8q2JDJx6CCVLRkbXmIW8d/WwDF8fk9rXwrfdyRS7N9AjAGYs31GFpvNbhpT 3c/JC5AcW4xPCe6LfFVY66qaan3J9DPf3pBZa+A1Cp3s3jr+8qA4wjyGpp7TcLFsoOwBK46vI3uA 7Bn0c3IYgeDWYmoJaJTQDNHLbaD0QNNGYeruZs5YzOvulqJndkXwKdLWqc2jRcz+GKHDlHMjKntj gvs3QcvhYZZsq4hSR9EjE949Xz/UlzzPLjjJFIlXT5LWcSYy1MHwZKw4hI95zQ4fRldxvhdyLVDF ZB9nPCH30k90LTlZCr9iaEXnnAMmQLZ3g2HoskwT01iD1okd9sOIV9ncvnr9HhaZPWZe6Gsip3PG 7uFC3cAo7iSgsGigySzVPpwdMgMPJjVrD0YKJ0AoBQipHwtvdn3Td3gA/yVOlL7uOAFWkTmNff0S zcc6JXU3XMtti2qIfbWYtJeAfCkDFaqFhEN/bgeyKVzGtjeysXbMrQDkkL6IwZm2i8WZt+xo9Iax PAnv5cDLJd+zet2tJhKsSt0zlnvBwO3VlnmlzN+/2teoFj/Dp/4seLq3sxFTzWduA4SmYtvY0vTW YfGynVWCKqdEfxRn8FpJ+Dzrv1TbYBQplo5Lbyr+rkAHUuGfLP8PyMefnDuuccVhzjulO9NhUmjK R+naJ2q4f+B2y23vaTkXRKXtlOLuv5W6jhYXsIy79CCQPEOncMI6IBJu8poN7A58gkA6ZjDzcnmh /g15zt3XTkSGpAYAJeGx4H2066QW0GqH+rVQsXPQqcMzkbq3rkfpMSY+2OzQbMfta+TrNDf+EWAA PJs42QplbmRzdHJlYW0NZW5kb2JqDTE4IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1 MyAwIFIgDS9SZXNvdXJjZXMgMjAgMCBSIA0vQ29udGVudHMgMjEgMCBSIA0vQW5ub3RzIFsgMTkg MCBSIF0gDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIg XSANL1JvdGF0ZSAwIA0+PiANZW5kb2JqDTE5IDAgb2JqDTw8IA0vRGVzdCBbIDE4IDAgUiAvRml0 QiBdIA0vVHlwZSAvQW5ub3QgDS9TdWJ0eXBlIC9MaW5rIA0vUmVjdCBbIDM4OSAyODEgMzk1IDI4 OSBdIA0vQyBbIDAgMCAwIF0gDS9Cb3JkZXIgWyAwIDAgMSBdIA0vSCAvSSANPj4gDWVuZG9iag0y MCAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9GMiA3NyAwIFIg L1RUMiA2MSAwIFIgL1RUNCA2MyAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA4OCAwIFIgPj4g DS9Db2xvclNwYWNlIDw8IC9DczUgNjQgMCBSID4+IA0+PiANZW5kb2JqDTIxIDAgb2JqDTw8IC9M ZW5ndGggNjMzOSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZxXS3PbRhK+81fM LcCWSHPwIIjaUyI7KW95s96IKR9cPkDgSMKKIrgASFv59dvdXw8wpBRXaktVImamp5/TX3f/tJm9 2WwSY83mblYuypVZ0p985MvVokyWuUnXizxZEsXT7M11n5u6F6Kl6evZm19urLnvZ8vFcpmZTT2b 8xfRfp1Fh4Up4s1/Zu82M2saM8tTu1hm1uQp8TPLRZGYuS34u3Ozu9lPmwsFbLIi+uXKFFaEK+da PhIW8Tn6/PaL+We1j+d59BzbcmEj0/Kii+fpIo3u4yKq4rlNoz19NbFdFNEf1dAIjdzq42yxigzR JHT5a0w/adTsdqZ+oBtCR/9ITRv1zgyyAREH/udiMriMumpwprm5vnlv+iFUoGINnGHpSjrE8yRZ ZCQqyejHE9ucFo9EpgpVvdepdyKpCiSds7NLEqSah7IDbsrqLl6TA/TYm/UUf9n8Y7akWCxsvlqZ zVt4uvCutil8PZBH6LprOjN01TaeZ+wqcSZ/7eVU1EyindFzXQ+VwbnTNXmBg0FeyEkHZcF+yGj3 UUj7OKe9K3NLuyvyjgR4Ta4xndIfhW6I2Z+uM2eK7OJ5SaY/y8LcxSWcLWK8P57iZLkQ14rzRgtk de/1ldUTuS7z2u8DuewLpbkjo0gkZDVyWi+M2bBVEZzX9KY6QIvDaDNfUtWqGnTefSpw15ygTdcH 0cqyMVrJGK3lFC0OtgtstQs8Zn5qfbwi9kOcsgnnJGyOFXes1R1W3CHsqtudM25/klUj17sWlE9i txtvyysbyPX0QDOJLewU+SqLgnlwnZK0TJMKp1cUqZ0539gaGOHqI951Q69qLTHnJTMGwcHVQwON 3dULJg2ITKX27ryKsLgF8/71LEnXI+gt1+p4Sjnxc0KBM7raVs8xP94fYsuPuqdoTn7HyhE6ldGh ogyXO1ibnykL2NO8dds5c03qWHpNzK2S5Of3yCv8x7Wd0J8LUFU6wiQhNP4CmJDzCIw5IwtaSSz4 YT4KEThcGU/8TXZrdwBTDrKdrO7U7AcV1N6e5AtWtEc1/9ada6BMQAUOSqm6qdUqc+s6kJ6UC0do 87cRs+oRxzgslJWCfPICV0QvyW9HHzGWpIKXJWlACkmWAwbSkerDj7+awVGSzhmnQNFisdNfXKD0 V+wolCkhwb9wwQhoCyKk7B3cIzjxW5z0K/IilpWyweUtqK9Mf2GI3vLsxNZ7z4BP9jhgzbQGwgb5 gY5ndXAldXAsHeWoyCk0uzN/zXPMwDtE2Xs7zwI3VnhbInDMTK1ba7lLwEEBHXGlzUesWBN2UIWb W+xOIEdVRBWVh8w0zZhIQnsvmyjlAf8K+86o9GbKHPl9wDW9NFAirnVPte+MJ5ValqAGeL+Q5v4u F+r1VFr8Bv43wnisLORBCKHacjjXl2CvdmcxzSSmt6NY1urCM95md8HNnURKtXPqpcE0+zB0STbW oFVQg7IoVCHVskFxcXOGfeIYcxHuXHV1QffAgumZkCpbt21qan0E86VsWSAZ+O+avXw98qlx32RR 1YL0RE5VBpo0XCDKCNewZQ4dOYmc+XNL4VGFmY2jtrTij3bP/68Mbkkdo1yonXK/gxVHKEuNYzsP vfJKhXDfAjQMsdnFIZ5T6EISUue/R2x0bstoIl5Q8POQC4In1innatcAOQkjOOMOXav05MijfB0O bfeXQPzfcDoOb6QqcYrxIUdlKmG1WgH+nYdz0eOZK3sZgIJIdJUe6xV+lzlHQexWEyDqIShyndur 5iPJq83Ry2Lw/iM/hHUEjBhB1bwbzhEYp25AKhRjAfF4rZ81yH0hoIPeBBBKveOInY8hf9MMU/Kt R0R1l2qF7aoNyPiBtmEpqm5VBYwqcrm+qBNK0qhtY48whIWjuj8/9dbEinZaqf68dr3eLmXpNCP6 PpVCx8+CsekgIafn0dLDSEa797GMOE46aqH/ym0UObx7xBE9VR57luzmkvSR6/KvoYN6MWlT5OXY M69GvMqmafITTVAKDoQBe/mRRjMR2Ek97DBANNhXnLZ0DfTY5sSiPGtPoFZ0WV3AYSlwSLEmgve4 PrgOH264Onsipcw881T7d68RNVJm2+69UqpzdSHGq/iMY1WmGugV6YmqDQHUM/9Jz5tNMYTbNnK/ iLg1TaOP0uMCtUjObcXh46dMpcqivcwEpqj5lrRPydlEEZShpcwdPvRocwm06Z8gpCdxIBmEl1E5 9N73IIUCOIW8VpUQSlOd+KdVVXCdh4wdlBImDHeAsEI5PfCOKgAJWoitHNIdUKv5DZjpqhUhTt43 05vQAVpwmJBLzvdryCVWhwCM5p8CqBUhgE+0/qpvGWGY+Giqsc8OER8rReNqD5qxXCxiaR7MJ+cL h3M6doRsfLd+KfJll6+3ldl57asMNG/bLWdJKa3gNCV1OkiorFptDkviHzpsYACcqiwlqFYdSiNf jMOGNFGUsKk2pDUjHXVXe/mp2fy175uYTSzPWGG3FNhFo8fOaG7x1ewa9IIWvWAaAaS4ibBZLlCV FT7lcjs+gEST7vNvpHUWfTG/k3mcuzLe4keczTB8LTQf31C1XlPRWxiB1UaWN3Gh5/z1XneVQUVt UTvITlu3O/mgOpYASLgdPBxwgZPNctvLcCvFm58FG59ymnCram4COXg1ypqfVvlCXXgicMSE2dnk COsd8TZOxRGfHtCJcbzXkgRrgYgkGGuSQt6MbBJk/oA6MggtmexqXDuC0xa9nmNM4adRKG+/31W4 id0Wu3vc1RznHgM8h0ChThRKPe0ZmxP2kH7aDCUobCNzKKOpuNZl2D7nAgZi5oPvsoWB537mF9VU DaiPoRBQSpbIbmdUW29PqNer7ni9iORTIHMEUsBxgopEoGJNz8H8QhFmtjltnrS1jRMZaXEjNNwT ynyRiOuYlLDV3B31Qi1bg9CBmgPMwGSw9IwrGgJxzlUfkvk14BNHneRHEuE/FOuuUI+IaGtA9uCv d64/4hs3vBrzlaaTSNbbjjqIO9ghDiU8vvSPgebKE/08c7z6/nCmIxO9eJmJjrLiSadhQbn0lDIg tTrhsEWyUe2oQ+FSqhSEauBx8uNYW1e7y+mOuySZpFjBoKW1eN0ySz1zi4p+fi3QWRLNQTV8Vumq hdKoFZ0fx3DHYXsw3hZvQ91OA+SVAVWL0Y3HxPBYRk2of2r297z6+5lLk9GlWpRFNU4Watpq1w1V g40dJ4kVAxKe4whoegFAh/MB59gzyoUaOW+V9O428gu9xIHh3EOHn6PUWPRvhH+e+ptQU+dOBEOo yMuCM+GstePIVPp688V8IHnSUGj9LIjXo/w370WFVDdlcCkxDKxR/MbVOQVWA/VQzQ0pF13fUCXq zTmDuwZkutTDCb3WCx2eSoYnmrh4vis5h4Jbg6pNRSBD5yT07K1Ereilc0OHkMjbClRltXaXtsvJ XBRoYsRGtv4fB6g5P3K2Z9FoBbjXuKtaP9GlkW2l9qG1KDnyE+Rm2Qi5aTF1kQWC+oGE5dGvMc0Q 6IpS6YqoHTafpqOddBjSUxHkPgpBj/mvoim0P8rOQcsRulCKEpl0E3N5PXCTSvNb3cixVBuMAJS4 FbjvwP2Z8YCbNHB6EM6OhLAgqNaCm9m6g1NllT9WKsWpPapoi0Pz7oyxKIpiXKjlDgTf6UGWpc+N sRfD3KiPzXKnZEuZYagL0g4vmGZXeAtydBpXUUcj2bjYeYK+D6pboliZjnyG2Mo01/LYSpJv8Ss8 ziXdybw3jZP/o7xqmttGjmhVjvoVcwtYJSnEN5GLa21nEx9247KU8mG9B5AciVhRJBcAvav8kPze 9OvXA4CUrGR1EDEzPT093T39XnPcsttDBQGbNcOH6gUFIetR7lX5A6dX9YHj2g4Fp218N8o/jPeB 4zk5uWRrFvbONNktzKj+ZeowTwf/WxoLNBfaseG/bM7s0qVNCUCp5lJdYsmeDsM11xyeGwBVGY8N JJSlYCPEtoC5OMaDlHSNI46o4kH/XyMRRZF7i2TIZJ8q7p1GVjLUdQdO+ZWgyxVgmytmnm2A9mDU brreO5vG/fiyJqPT25zeOFjQ+RC30uJWWtzMJR3amXBeK0h20onOiXl0O3IXyAIOT11DXxNr1hf/ C2WGlzSQeaCMVB7U5B8v3Q+aq0UUZqwyx1pDrcpKbn0Gd4pHqf9wXi+aWYLid+tRZmHPX25vMydn 3l2AKidx6ebfTq8bgNJ7/9W8JcFZ+aAloRaBvjhNRyXJgJ9zw0+Ae4nGR9IKZHChvB8sptJ3hyJ/ MNCubKz0QMBBsXEhfO8K5fRmlkm79hsuq1UZ+wUtlhrZern1IAbqdCFTZZpVz4BgEsPV/lHLHv8f d83K2JwigbyfasquM5m2VUmocUm4EWoLV6iq2dbtlhNPM6CYPH7J1F+VJ4CtViKfKm4IxV2Ld8PE ylMdiJDE2OEgUDQ7pVchFww/TBRyybe2f29LUO2FGOwUXUBWPrJYA1EUBNC5qKl2x30rKCcEeh65 3zbe1jaDcEuGe/lyfXrOnQjXgYe0BuIG3zaSd1OTC1yycXKnDGXSw2ZKwuKRCXnjK6QKxgdAuY03 uU6s9a9AWpIP6ZqPkNbcvJuhJ7lRRAQvm+GZBUvQMeQR5ySEOx0ql85BTjnWFFgo48yn/eacDU2q DY16ExBfu7Ud8sANS1myz3rFxIjtSPJk7a/qg+T8wbeo87uJVfLaPwir33Jkxw3XOLP/Zdu+EeKh m4qtm2q9lapHtDAJoyrJ6kllCyXgYjqIZhYuc4Vy+yWqUXMrWgjRVtvOMACmVIC4UnurwfkY9iy3 lXZHmPgyo/il+38NghFm1NiTza1jkFRvgx3Njh80RPtbfJg2OW88KC6AVd4tuVavHvhBow+245mC 8zv7KfJk8cBeA1jAc6JCioFSSkWtWHsAcDg8BAC55MbX0IpifORgqyIUVGorpP6eS19mkjj/mqG8 Un+jUjtqNyF3aw3nO+0hP7ogxFUq3nDkbVGqeJC/kW7yg/OMnW3CG9KP/drbxcaQ0CFGACebTLDn FezAmsefzXp3aPdc2a/2dIBZZlr4Q04u/lvyl8cGJ7coLwxzJoZNQvQczgUkgNPeXni/d8PMCCqp JVsRrZXn188owyJLMzw9jX4+1KpFqFWkap8QnyT62YjbP0AK1FvIst1aw6UFiXfS6YFcbxrO35vY RlhNrnRahxMtidglZufMm4xVWIV6zXyJunGM740cIE3yeWHsYADf6E+z218mHEIOzLNk5BBZMmZ8 wpsKTiIkFXMAZTXRXkhDU2mCp9GybVRmpaNOscdcKgWvmmda/MELvlnNUP2l24iQ45lCH27fdPJI f50+5ea8yoQhn3Y/5u9Ca5Zu7fpQGBQRxcsr4QoNS2DjQz0Y6nE1lLvWah3Nsd2tlKIuFBMXDPid BngrWsK+osaqi0mGOsUDmJgosEIm4wU5J8j8j6/dGbcKR/TXI1YkeTpEMR2jmNK777Tz0mJV8hkj TMpQG7FzzQWPMpbz7aKFke/phLcClJm6j+6AzVLtKBrPKef+PkPCLg+dPrzz1BRSWuXJH07NfLyU PULNl2Q+l4ZJXggupM5ZObU353Xx/HJJkzUHlHFJPnc/vFXDiutsIcfMnX6ki+p6vpCXkyykc8zm Ysvj1MoUVkpCFLpFP9JKkj7BllJe6MK2gNgUA63PaLEU+c981o3Q55DuIIzv9P1oByie7BD/XNs5 0Eq/05+V0O39dH5n7PRKiW7nllz0RsWLkYqnoFvvB7q1ElugImc7hzDvw8dG55vu0tmUssHFlJPH 2nVtgyz7uFy7r/xcznbX5weF86T1k+iYFK+dm23aY2Z6y9xuuUAx1d+nmT6ZN6/0elkxpkyoMp/k uEp2iZ8zqduart/rAa0Oas0lbXKK8FQKAnAhJQ3o69yNbtjbtR9x0VQh6wQ+Uw0FJnsq21DLfo1u KQ5VFL92Nq0yfS9a4GyWFrzjzweCbI4uAiCLD3fkBtve6CRlOvX4JEz5dTUITA09tcBxt62FWhBr LZDMdVRLIUHUcBd1w+gVvdqRgZ3uQC+59uaO04Ppw/Uf7iU+M5f8TK0XlDf6TbrewgbJOCPq2686 S+G1lHcSdqEPYxOUKw4oSWeDBrY17mrxHhbaKOmaqQjcv0ZVpwVNq6lTad+a2HaahX4yMewUjkP6 MZCcRTXQD20Z3sNRBfiHDkNFSZCmoBMxSYc83hV/9xo0YXJf8biU0/VwE+iQ/gwCNTdsOT16AQWm tr1Prj/d5db2MUyYAI2xMzpw7VRzOS4n4y5MPM60t5SSkWT64DH5hMlE+ZHMhtHBDjJzW/OEqQwK dyZ17+6CRLDwcTY32hqXg6HBa0sTsvH9ub5duBZn71EyxThJY9OZP1PZGb+xxWCOLQfPmgcukUni xTDN89Yvd4jJkCZz48I7fZp+Fkdfh692fIsVGiWZemikDeE5p4uN4nhn9TdVcT76B+pDt8tl5n7F 3M9sDy7Lco6luzsrK6vGmxoaSNVvbM65fwq/ogLq3HP/dj2pWZn6TKSPlPS2Y694ihRKlNFAZN/T qM1LThjE6hObuhMs44hmrExuYy4zyaUdLPw4FhJ40qTEA9m199v1SkzFJygnSQQQVG6vuQC2GGvT GytnhGBLQXfU2U7YglithLK03aST2nLpbJCUvCKbjiOSYAlS/0YvJua7T35b92TAmm/6ailO7VL4 MlzQRMyeSVnMCNX6jLzp3U0PpyVSf83Cug/Kam7ra1eveorVpmG7NW/IS0eRdME7vLvr23pHJXbK o3YdTf/sMjWNmb6aLCteIQnrvSc0GQD9eQaebDhFeK0Vk3b3xym8diQxm1OoDvBZP0PMpe021LMf wl1reDqSDOMTCqsG2hviqG/9m/FyUwKUjXfLAjDKYyujn913UFECg7W8VIEroKH5nR82raAdRwfc HOlF8hJrfSvUCpX2YCFLE1Jzk+ivOhpf0SI6Uvhw2GPkB6+k6pWSPEJ9kEf3psWfmNdTZ6M/aj7P VLaPVo/r62Mwq58uC4M2Pfe2TuVQRMFmKm6ylzP4+cxPa7Ohf16OCdvlANslnb9RlCoiglY5/HpO 7zqFDIOm0kC7GMYK1ZHpUNCJdmyuYu1GutVFeV2WcTx0TXGcDq3LT5E8v7jUDKuUk2Hn324vkrlk Vha7OI+l9XFSWsvCyQ2kG2r9xd3F29uzhicR5p2gu4kzoSxFaHfiio2FvrfQ9s2DZeKcKlmM/Zyc FJqjhOY1N3CvJMLNh5Ax6C8kG+o1wdK3AjjfMfv6WTy0K8TNVMmkzh4GWkJqEyvtgyAFBF964ago GdMir459UjlKr0wtD2nJ/UwrJZp/132jhVIs59QEpfCDRmlp9iOpJDe3pqLrLwU/zuR9iMsQVPpb ApSVCApy4SRAg4/zVCKokSwRF2kKE4fUyZNvRLIS/2cIZLKQj9NW9ycBh9f+JFhaLV4TsZvoMWJW OodZUn3P0qvAAEbpx2CUFChkyeNFlM5ufzk3XeKyUClRCu9YFj7vBZAxqfLxq5xwgZ8jg6kEIA1N wpYkIPQIa8pczRR4lhwFEdJ6v5aKotBfGJvnf1O/s4YASbRCpnBiEvKUwA8Et/ZhZGE84rDvHVdq tXJ7NzO2f6Ik2PVkStr90W4R+hJ/tmNPG1shVgLJg1DoUnhav8ej+3A/aZ3CPmaqhDJN8vScjT4n PbXh8jqQkMa4xC4AP7jANnCDRkl01xt8r3xYOBrN6QMVWdXGa7betfv/Ul72vA3CQBiW2q2/4sZk CBhjKKjqkqGdumXM0iDSoiAFESK1/773ZQNN1I8Jg31nG7/ne246aGAf17gqUjignU0W4WkLHdWd tGvEo36ECJlSmUPppxkqpRYBnKsgp6ZKSq3nnM8J0D2wlmQ1ijb0ufGmPcz+WCu2QS1+wR6AVsyF R0+a9CKf2uaNs8e7dE0SmMUw/QGM6in9IOnSAzyiKL9AJnEfP1lAZ/s7RHVMSnrzh+tlcUsxHW82 Ooquhjwd88NlNaOgX0EMB6lPUJgb3nzh2R6qpq/O0qvlBvN+LVRfCTJnjNyCZYO31L8olVDLJclx V4fIKENklFw0YVxAo+z/2iH6fHBXBA6vKG4aA1ImaZEB27Fw0VJH59Q6De8S1DTHlIsKlzo+EVsE nktciCk9EdFLzoKgqitTseQkll47z9wpQ7R3NfRNJ03wBiTMDIXJwE2pVc39Kg/soK69HdO8k3rD MZHQqG7mDfZ4FMniLB/bdrvEM0vkzaif55004hOe7Vw6iH9FUqYX2rm50E7m7KidLA3StakmAbdk yjS0LjwLY/SdtGQdH8Y31d4X6aVqf5t5nJiHUzCQgfeK+TmxfwiF4HTlCUnOPVNWOpGEckoq9O9w 8RgTlEQo/XC1Nt8MitlY99/NuHK8AfQOhzXES8p1UOzgESwFUbnIDLysOe1FPuV/CTAApcBIEwpl bmRzdHJlYW0NZW5kb2JqDTIyIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1MyAwIFIg DS9SZXNvdXJjZXMgMjQgMCBSIA0vQ29udGVudHMgMjUgMCBSIA0vQW5ub3RzIFsgMjMgMCBSIF0g DS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANL1Jv dGF0ZSAwIA0+PiANZW5kb2JqDTIzIDAgb2JqDTw8IA0vRGVzdCBbIDIyIDAgUiAvRml0QiBdIA0v VHlwZSAvQW5ub3QgDS9TdWJ0eXBlIC9MaW5rIA0vUmVjdCBbIDE4MSA1MzYgMTg3IDU0NCBdIA0v QyBbIDAgMCAwIF0gDS9Cb3JkZXIgWyAwIDAgMSBdIA0vSCAvSSANPj4gDWVuZG9iag0yNCAwIG9i ag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNjEgMCBSIC9UVDQg NjMgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgODggMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAv Q3M1IDY0IDAgUiA+PiANPj4gDWVuZG9iag0yNSAwIG9iag08PCAvTGVuZ3RoIDY5MDQgL0ZpbHRl ciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImcV9ty28gRfedXzFuAlAjjNgCYF1csOymldrdU a1b2wfYDRAwprCmCAUB5nc/wF6dPdw8IyrLjpFQCgZm+nOnp7jnzar14sV6nJjHr7WIVrQoT0x+/ 2LiIVmlsTVZFNo1J4mHx4nqwZjOwUGyGzeLF398mZjcs4iiOc7PeLJZ4I9lPi+AYmSpc/754s14k pjULmyVRnCfGZmTPxFGZmmVS4r13i+3i1foJgCQtSD4uTJmwc1jOsslJnMHLu2BTH8JVlAXy7MJl GpXBGOaBCZdJGiXBNkySoA2XlYo0ImIgc88jzsiMIw3CFfzBX2yD9d6GNkqD67CMCn2/8cbVQN24 PjLmbzypGPakj3/6TPKoCj6FtKByjmTHTwJCIx6JE+e1QnkIkzg4wowz9V33KFJX3v2nkEwXE/Av WGpPHyTd3UGrpX9x4xpesYIzmxpfG3FrUhubn1+FH9b/WMS0K1Fii8KsX0vMV+eYryTmHWJKfk0D K/RSh2lg7pwOsw8C4s4w05Ljm8IxFr8KaNeHE5bH+FOEOuf4LgsOdkVPBLuiYHdH16snkVIPYikK Y9gzNx7Vc1670541G6NStA9llKuJRuwZ8XHPYxN6EYGZP/hNhJ7g9VBVFwlx9Q0gLKFoZiHP8ynk SYqQ81slEW95hxPKh5TyIVwiofZhwrjgpKKZEl8H+kmD0dQ6LN+mExXVHFnzwmb7b68xtiqsqu/1 BXlMLUG/lrRBKzXU141otKw7kyKbyMs9BR3iarg3a64JS9ts6XlLBYAdbH9gdYK/noHknyHE831o xntxPM5KJCUfHteJp8W4oqZM/Je8yWRLiStvYtVcrmmvYfv8fLlQh5v6YCKbN7IZWMEqufLom2qP 7ZjaDGS45NSmBe1keM9flDMR18ay5HoouDJyfpuaEFeRlx3EWK1mKIBIty/YCCwIc0tUABpSYiPt T0E/igsPSkq88iVeocRpmpocXJ9E6k7sIXxk4uQ9QvAgH6JrZGmoFVQKoE+YGZ7YUeQytJtbkxku nllwGmo5pnGbrnHNFe/G+s8SeVv4AqoK2YMay7MEK6Yl08ZLpxix7ytaNEadiBg/qyoqo1/mqLMn 1tHJc7JZwnmvImqwlwV5862oeM27S+G9DHc6StkgLwPV4YWdR5TGCiKXq0BrzAiE4PPqMmc2outl Hh5qb2QKifesqJtn16JrOOxmFgXvoGBUUKPW+grS2eEyYBNmXf0E5ngJRkFsLkJ1OkPUFPBHlWRA qlVYb8aWuoUlr7sQYKjI8XOUsU5+DjI1cpGDIBR8FDQyi3BR9yHNmfDY7r+Sf2Cy0Ooo1aSE4dL/ W6wq5SZItX1D6R2CEaWBaMsTLrPJl+Iw7UBda+PaR/EHWH6qfiKpGGR00NW6zei8BHpz4dffG/8S LjMpFStthIZaMbKbBzplShYrGesOZCyjerUITF+3h1He34e0x3RUQ5MYoSeDhU08FUx+gAoWaR7l Fdhg4iko99wzLUyUFr77lVf1wdyMIRUdt7WMm1jO3QSV59xBho3+cosGe7Tcd2j12Ngy8NpsqiaQ DyBaX0mb12FGPn/miKfBX9FfUVFHiLSh9RreK7XY1I/JmZiQlWsGfmuOneCUJ9SJ0Vk6dlL2xYzG EsJjT7SQDUKmEXPOwKfiRpM8W+ld43dB4kq7QNWXm6woo7gyFEYKMW/ExPOn7cpWmPwfyHtByRtf 7hjvU7Gapc2257UzX0hB5hBfZSgVHyboFpR40VfIiSplGTx+FzaFLK5+INFWsaC2K+IlRXy+dxRn DlxMCYaIvtTy+iBMx7zpAT0L+q432N7GjVRrbQcSWTBVpYwwb5n34HCEpMzt+OlCyYZrMX/NT1k2 5VaSW/CN0q6+w87fIQ1TSn34l6yqJk7rv4Vg35qNTmw+hv4gtnR2LCdeMTFdmTIg4sycVbF/huXW SpHEpqlV1fWICkWE7VFYvsXozY5/dMyJOlGPmwvqrVYHIdTHY9erYUJA1ACF8pTR79WjUH+Ft5+u MJ8npt9/g9yl50aTasvjIks5gvxiwEeArQJ3vkYfAtGlbmqFea2CLJVbiJzzzIjfAOgq8Maon1Bm HPRL5t6HCIFa5k1IvHwtEogFf2+3voAwvGFr8s6EqwrUsqi9lDFjfqNO0enUn9C9SpUw3o9IPs5R GkEkgpTMOw0D9SUZk432YHTF6sXHoUf5UEvllOPLoXhaXhw105ke+zN9oAPJBvJ0nNp8n0EN3SHk oIpLEObPfAGRFBRxCuav/mIDwZPeaNDCncqcGTab7JpWrG3VHN02qvlVcSsVg3ZGUvUlIGf8wIG1 GhzhCIflI59ljs9gwV5XXHDi61ouKwL+mp+RhM+YN3pfehRZcUQ7hGZdgPHnuOCMGwWiJsNEQ/gc otPoLrh1OnFrq62QcnzJB821sNRkRYFQTlVJOJjv/PMnvOBgrMjwL1hqNoV4uG+Z6Xjd0X8exMgu BB0hGgEs01mCS0Se5fThK7Q8V2gp8K5MTSclk3XLYUzQqbvTTt/MTke6BmypYOqVolyPose5mMmh RF8HsQMuUanNlp/dYWD2Q0VkLnw5tHM089UUYSzGezmpwQtbSKGMFq8HXprlfJTZnE5/k+aEJP/u mWdtOjuqf4Rc2Sx9elT7A286qZkBFsEG9alroPsDh4g/ZXqs93tOiZLqDgsoGAr98UtSJWLXZiSd 8yELH0Eern9/iq6ijL1EF8/RnbMxzgUi1fVaOfM9Uyk3HYQYu2YKe6uXsY2KUPOu+EhBfp6E+j6E 2HWi2jxopC4s12YBnnzgF+4vWGfKueSUxclksxQGNnpl/XZhzDyQZcQb90ulChVThSVf2czNVuni KJ78ov7/BYkdXdbBs/+MK8Tzq6nAlGhRWhlb/Td+aCkvf4honZOOwjZnWknq9zOpZD/pTON+t91y s2o3rdMmd+DeN4Y4QK5wRYNYJz8zViKcQOgZbmCYHnuVU60Tm9o4M+j4AxMvr6bGPrJUyw7Fuaob 1RKIZqMA71kGe1H6DComVRwePI17YMVcDxlC90A1Wotzum7eeZAzJ1tt3SzTa88eQkZzp44+oziz mad2crVEu38bIoFuzL5WwKrgrb58ngXFU82lWnO36JBW7pC47rZ38tbu27F1w18YEfKTHCD3Rucv u6gntAxtHCGa3pWPNw4WsJxrqunfPOuEXD26R36Z008mEk76pmkEgCK6ApOkYmrH84kOsiq+h9Eo Xq99vNA+r2fP+OhE1pnzKrQPquJptho+v7kXCnY3OE6QbFoNEgS/bA3qzQWaYc6BnuH7nZLmS6p7 ZiSJdKmSWRBx4qswnrPyph4nAmyUGuuUTPTSiWpxM6hFYlKuMfqhIghGLtQhRdn8wheRG5mUG4fa VGp/3g3mTU7vDaqvQj0LWWXnV7MNt1yNOYd6difp5Eagl4VLF1jlR9E57PR6YmS5dDDLEhsFd9QQ aTS6R3/j8ZGW68ksgrONirOpSJSH8OaWnOkZWg0lYzsg4kmigzOsK1jmLCmRq16558pOpI5yvRjG T++EswP73esPKB3LF0taU90j+/jgQM/HhYtPAroVNa5v8bkDTVEkp/M0OgfkSYklqKkgz25upUYH TWWDU6YUMq6Dg5aRlC7jgAVF1Kjl89Cg1ZPNWu8WSIBuwyKj6LQsg23jm0QsSUM3J6MCwNiKPXOE if7YKVK+T34RJZWo972rG+6liTIXaX55zs3vP5xXTXPbyBG951fMEUyZDPENVA4px1tJ+bKlWmsr h9gHkIAk2DAgg6Ad78/IL073ez1DUnTs1R4kcGZ6unv68zXSLg9TSGUJ2KLno/boJHTX63uTaLS1 P37B2qtTQInupaHuD51dWvjRClGym+H4OHA/kO8uZbrj4Yk0L+agUZqGJUpPjUlzyxDTXZP2AXtP ZT95HiH+loMTDsYBRTDRqEzkIyCdBCGKDr1RjPen9LUSpzw9A/8VmKTqjCZAC0FmN8bQjDt7ydfD X81odvJgXEd7iC29/S8UOJkdgmmVYHX7Ltd9kKFQhlCwHFd28GwKTyJft8avNEuWwlcJoB0SKy7V MNABGqRUWSLSHin40vnLEHE4444NiQcZJipjjVPse9Gj9jT8ulRpZ9+BNYLCPrJAbAgUnftV4ULG UIpPCr8GpxvkiwSa6WzPcmemcLY3mZLje7A4jrZ/oS4pzr1U8OzcPop+kxhdZa0fr4STrlBa64A5 8C5vOjf6Z0OioylM708XzulP9u0uynpweZLS5c1F1TYQlLKHpwCLGcBaZr/3jWB8hBvHJVFrsTvA 9LkCQSIqX+ADGq83SVzUnHYZgVmIQINizr0Uk9Vm89QcnVkyWGSl+rQaYdFwvTMyT65WgGMyVZAJ XKB78Rzhmm3oj8Jz1ZBQtjc27hxs32SfWSo1fFMDXzX2697zaU1MTyH3gdRrdEdBxyevs2O7Ft7u vPoHDZcqulTeS/e87bZffvTnX523UufnpETcWMngk8l0KmNNUleb709IWZbK9zkTUiZYSKhkPoov Bt/geYXj6nlBTTGrcqqQrLPOqqMCUNWdvl0HqCOiceDhItFHGJsj88HjkTxIuGBK6FyzuIf+/gF7 ApkKT9UJDly43cA+BSbbjMjKkEON6TP2dwZBUa8Y9L8gB17h/ybMnzZ2ZprkmUtkrix+YNgUxnyO Ybf5lVXjPIyf+QlA/V3rX4IhUcqQorZbQ7o6s1Ua7pWWvGUmybBK2A1roArDryNhZMdvi4gspNXV 2n8wQOa26i5utDzSAq7fWauwzblaFBP6XBf7AMMzaCEqptHfjJlz0p0jr/gDIa+jCM+stx+mqq3u oJRrnvWALcob9vwDZGQwqtntTQEZfcBUbLpoSlemmQRbv1xFwzbd1IlLy/qH0ZBsfSiUvyMU0lqC 8ioYhI+fteLYiv3IXGoxA6iyWPoZoNL5YoM5Shfcco/zxF/dnl8dKEo/X5R+ecFxxiwlIYfVPVPX zhymneQk3JTqINsOMR5JLMwf7JLHYka8t9uNWr1GXd8o9j/j1L5wO1PAv2Q615iP3A0UqLOlPv1g V3bvue8fbe+bKIjBwKfa+03jBytToB5ot6eBkNaoiWmVwcffC4RtIp59RlVIq2Szra7LbRHKrTXa SYucprcb+vEDf0+z68N2aHZJbM1Ov1oGUxbpDABDYYrYTscQGUns0kvtkDVKbxqxukqhFd33XU9U 9hlbtqDMe0r2krjZOn6ntnuhWQdGe+rbK+r0vE2yFuJaivI6Dwt/RJHdSlWde3QVjmBpeJ0OUsmZ wvtpFp0XKv54ldBppb7MtSN+z4lpHatPnuPFcrvJrzM6CbPb1hJ6wjPEapLAjY6Eaq0a/Y11Mdbx AD7UoVano4ZnHT8vvHW4POL6HuaQSYTM7AaH0sx2O0FVZOSCEo3Wiv9INMReAxXJKMlhf7k3a5g9 djPZNI/GvHE/v1yhtt9CIww8LBde4H7i6/hf/NL7B9qre+NlT+IuYjWOrp1X1OqytKrUOd/1nxTs /HlZmNcaFD8GPR2hRru2OuWBzNoXsi3aGWgIPVDUgGukw9AldUT44i0EkN5dYJnG26r04Ga6O1Vj ZfFFU1Ywp+uCuHNNlimodM7XlAqC+V9rJpj2tq04ykdnFj3RWg9FdQ++sMS79ReCoHh6xfVviLje aJEuo9fuhj9+Wmmg/3rl67xU/6R19gN8mxbVM/FtmpXfwrdpFZqvlmP19dsIcVloOuqn08aRRfyv r99KJN+vanQM7SFvV2dFODGkv0WvUiK3E1Ar/7Qx5RH/86LgE+n0StP22GZ798cyfejTMGUKN4Wy lQj5RX1ZR1z4rp9bA8wjcrI75H6Fd9OsAN5NpHP+wNB58fvwbr01O6dbNbI3qCVUYij3l3fuTTdo SfgscSEGNIhbA+LGnEelFc3dMjcjSA4aMeoMxWv94WBTbRrxmG+T4I6zXDTclFmpYyuAVXkCVqWH 2asseuf+udKZT2HAZ82WEkikikYuNIgVcj7YmRtAOo33dt523Gmo7VduH8QhqWKyHAnmWQWwIe3L /QvZmmrjLaKfRYsXSASt0RW6HO+C2ktTC2mPrKMbRzJqSx3U7fnm6XPUgNiYV4qQm/HwcWWTbo5Y VjaH3t5lgkIQg/JILUhKZhrJXB955fFxmnlElVq3+wp/bNUVcV4U3hdpccozq6kaqKlUhlz8+kq8 W9vv1y/EhqgSGr2AO610owXkiGvRBJmmzzkjtMVHjxiU/rfOLj6ApHO7hsQt4i7XepoZDkyjdiEL ErvAELIemz3yOovIEaU5tmjFzqxFsIxMghaM5KTNmoh27ZF5YjInU4SXGM7rs3jO6xDP2SmeM59Q KwVm79w/JBkYP9LIO43LQp4m+V0xSguES2z/tbKoqzU6YrtllxAMbCu4tFZoBtCda2tAMMKcIOtY f/T+tJ/I/L8rAypS+o4k09/EH3riXt+4pm3nTrf1jwowvTJMgNQhSHuFV97IYMCTmSKdER6prPgp FbPuuOpmaBBgU+V1noBzysh0o9L2Xo1w0/KpLdFlde7KgymvIh2+KYj+6KC7GU9QZ68LjqyVVpoW etVq4LhCq5C79XU1O3n/ejT490/vnOgKIJYaDJQsMHAsjtk1u8Ew/9WYgEANEB5jwiWqbhbXDIMg BpSs3HD+aMh+4UTQ+/Fk7j4deUQudm4sD4rzK1NwIuHQojEyDpSK/6lZKx2T60fq1u0vJ4F9AMRf OMPw3jBQ7pq0H8501spRMuFTgNNv1ilpeSHHDLvfGuB5BUxzA5Oj7mVoPzGmXtkLvoh975Cx14bS oCtUQEbW7BsxWkBszb6ygZRhV0c2rGrYJRorJT6+hJQR4FiN5xAlXnCxC9RBSuiu4a0O7Fo3+Vvc Vk8LqyM3UcySaN9ZPkqOSOMjJTn2ZD9ePJWSjZXnf0p4FsGTEvxtJgiCznyTZafM2pY+BVLr561m lfJaK+46LP240vZqMBqZhg0noE55/uX2NnPi17s/SdGvZNYL/akMSRstzXwvFf32vZInJJe0L6v0 RL8NI15iYaIoMEa8ohryMzN6FzdixD1yV3pCLVrvuOpmq6sZUqTADHUwBkesjY344m3Uk1W/9M3i pYjsPyJXEtzLcQdKJrhKTO7Q2qm/IqfnwZwBjMie5h2q6Rq59gG3R88xqYXjFy2rieWdBI4kXXqe fVIrfZ7iBVqHaAt5Mbb0yUOwlALGhIAxiVCm0ETOJNvkedreaJjIA/wLXlIle2QbpFxcavC+PyoW nwMguTMLPDHxKiYagZ2ljJ77vZ8759UC3R1t1M2dsV7+b6R8OyBg/ds/M+brEyarGcQKQhAuBgR1 pttxpf1Ui5eEIIqMeGp2R17QpC+iTyQ8YqtDQMTG0a/Igv/frrTCCAR2O15ZwEVsyuWAJbrE9Nh5 Plq3GlOAF3RvRr8oMQbx9kydSPIVW67lXmPKUMB0bwprOQQZ7zzY7p4ALbGX8HCeBlOWZA0PLx9L ku8iuvzUbfIzRFcJoruZDpjw+L9H2O0GfDqMFYwk9QiAT+qXmAFy1FdRUEdKpW1sSUYzx4aEAK8i wCtCat8xUhUeJoaXDF5JKUVrylnbZWRdk/akkdLaJnXwlFTvvHHnm8petlCPB5JOR37J1D00FGkv 4w3CxTyyUcdr60edxMgOGvj1hRX/R3m19LQNBOF7fsUejQSRH7t+qKeCeuDSA0XqpRfXXsCqG0eO QeXfd76ZWcdJaKAciPc5s/P45ptBFaK4e3hYKjrKLqMCdJeqK0JUQX/QLAVZbav+OUfj3Ezj3Fs0 LmWW5ITGQRf9Fa4mNMxxa4IwYTqXMR2pZWW6wB37dcDUtvfy4edLyCi0F3VSL2z9VHc9VaJbnTgW +A3VHiwIfvpGoW2jWyF2YDIlp3XOdsoWkkTupfF1I+8ywifziOlcoJuwqD8Q2Br5HVqvFvgRjgzj KT9g0870IM7EtP4PNw61thFsLQdzBPc98foAc2Ssf8b62yjwXjsf0uEzn5BNqKAbHrdmeXgY60ee BnpsJgKL3qNGP6k2MfM37Df/Vg2WE6Je6reqOWr/8sq3fQJ6skqGOXeub/ISAsJ+adWPLzKvAmX6 YI/oHOTLHJv/TaYs/Qjxnujr7Q3XnsvA85Vid1OjDYYQf6HoQusB/gmktYOZ+TlXKArB77gmNCkw IW+YmPh7vVIbmW1fT8RQw5HPIvkrI3Aa2gAR2dRkXAKLkSBC9IRZM2Qaj2SXtBcnbU037gVXCzWC ajBRQvQwNrtmldEVWWEX7HLfVcRKF5XzpVppkj2kEK/KEfiophUwKeeAuyr5QRD05X5lXbmOy8Qk jjq/1LgE/1FOS9J59bCKgy4uS9boahJHbawhg9I2pFkq+67vV1Syct7NHxUpaePcJKQEXXb/G6R4 bmfN2T9CaRT0s1tUfxZDWqWVoRReak4asWRoxB+zRuRal8J+v/emjix4+tETCIBKHIhLGsqBt/FB wK6ImtEf5qHfTEQ1bygworuLgkCvQCx8BBO09XAaG3P8FprxXetHzX2d7yUHJZ2Pk7h5OkCwjUCA ogtjAxGkV01xvbfbCKgW+5hi5LQMHUqA5LEMMZd7FJvPkdJymyTAMW5Mx36MizW1UzYm058PQ+IQ 1RyHxftxWKFTOHI6OzObnam9IJuKer/NI//COnbO4AwZ3KKVwDesXEWy37ObaesdvVfqG0jbFWOP kaJsF3uxfCcnZIASjSK4BM5M4AyoPIeNVusEYYMlUYuiJoJiUEiV6ya9UGxe6PzUgelcodqm/LSf eu2rHJvrBgup9ewveTxfL4YhtNzWIkHlDC/yOjSE/aFtiD7W6yNnV5yWNi3W2TvOjgnI/gd0ypx+ T12dzK7WNuUa9qeC0tTPypGd5gOoz4bnmmEEy+2lU2ObccO27cjceNYCbx2zypRsqdt7Hhi5aRAK OZmH7lHkjT5IYWspaUuk6FbMHVkFhZUKTom1M0jWlS4DYlQHRJKVsMp1EAInD4e3ooZoMLGCtUoX FpdElwjbSjkV1BkJE66QxnKqD/s5FEj7G5E4Cz50dGnhHEvxlJdnHV05+O3jfi7sKZJzExQcnYqf 2X7AqaTCa+V1ZHmC5R2F8ctFESb7rsWsafjAEy345tdFseholOg56hUnXt9RGzXWvH9D410zdlus M4A7njN+HOXM7jgJCkIFa9LYvZcEJarWqW3+DgC80ry/CmVuZHN0cmVhbQ1lbmRvYmoNMjYgMCBv YmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDUzIDAgUiANL1Jlc291cmNlcyAyNyAwIFIgDS9D b250ZW50cyAyOCAwIFIgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAw IDYxMiA3OTIgXSANL1JvdGF0ZSAwIA0+PiANZW5kb2JqDTI3IDAgb2JqDTw8IA0vUHJvY1NldCBb IC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL0YyIDc3IDAgUiAvVFQyIDYxIDAgUiAvVFQ0IDYzIDAg UiAvVFQ2IDcxIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDg4IDAgUiA+PiANL0NvbG9yU3Bh Y2UgPDwgL0NzNSA2NCAwIFIgPj4gDT4+IA1lbmRvYmoNMjggMCBvYmoNPDwgL0xlbmd0aCA2NDQ5 IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJnFfbcuO4Ea1K3vQVeAuYsmgCJEGy UnmYS2ZqtnYnW7GSffDMAyXRNjOyqKUoz06+Pn26AYqSL5lsuSySQKPR6Mvpg9eL2eViYZVRi5tZ FVdOJfTHL3ni4somuUrLOLcJSdzPLt/sc7Xas1Ci9qvZ5fsro273syROkkwtVrM53kj260zvYlVF i3/P/raYGdWqWZ6aOMmMylPSp5K4sGpuCrz3zexm9npxZoCxjuQTpwqTxMZhfxP2xS5pPm5IVtKO 13pV76MiNrpRQ5TFVjeRiZ3u+0aGZbBv+VFv+KH67hBVca63/LuO5iajNfPIGJo9WbHzWttoXsSl vo+sjVO/h/qk/xFhdEEKLAmF56coIhNLrUSHGKJq0aTEhqae2qdskiiv3B9nRerXjQjXpDghS79F Fhv/PZpXJHgju/wIs6tg4S0f6Y5/RfcHtqryh30X5XGmxbBl06vo8+IHDhd73pDjbexKldJGNlO0 QVaOsbp892TWZFkR2/IkaAnClHBK/MHng0iNG+RIgjP1z2Rlltu4TE71UwJY3iOk3rVW63ZPh3N0 bBsXut5GpV41f1F5OKLoGS3Icmw+MWHM8JC1hU1Czlp443/4YUxeVxVURqfJe3TIH0OBBI8nZEJG DnGP/P2MQwzZ58qTjbgkUoddrvW+iebkX72KUGxOX0ZzJMAXpI2jHE6RjmPgWdloRkop9Z1xN84C JaZm8DHHkIvA5ITfHXLjynNXMtxUR7ypfNSz5ILAQH2JKDkqFGhKwVd/9RVlUC/7iOCMijacmJWP ZpXVY7tOrbE5QZZ7ZI2tQgaGFFx9imKlXkUmYxxJARJGvV/u9pEjCy7UcEemVIRVXJNGfxXRLpoj Jr2IDQqYhreGDBp6VL+h6S2v3Uts7yPEsvVfIt7yjp3IqeXh5gZOcYAr3aupzH8awqZ5SUggo8oE 58hZR+fYjBS8XCRJERfuyTp5rjzKHOlrHqG7cWOVXOv3y0uUswUSnoOUK8vYIlW/F6JM7gSipnuf JSuLBN3/V64agacT3chWc8zWgFEjzleM88XJ6VhRsCCzUP5dp7NJRiF46nSnECxyv+OIltLosftw xPR4xNQf0by+jHJdLlGDyL9U55TAKMqfXlNmOh2PucZqgz1UlbZ8uQ6L0tfhuRlHWmA8Lego9y13 1lz2LdGvqRgp/0u9jLif7oAMhvq16vpRXKYSWkULZNQkT667UPJsGtXU/YaztZWcpRZEJGQ8KBuO gxLPoHDhuC/VlKvyCV/6rpoqMnjvvKLydFJR5JFUd90QOYISQpQUuEN7aIw0dL6MPCHBGS1KCe+K gt4XbwXzihHzQrwVaE+qgWyUzn3DoJPprm8uVDsoAFWmDzwLsEkZGDOttt2glo3ayrpVw1AWZOoI 1KaHZhoj3sN8yqNm2AGomRKyDX5vAjXRNjRh5YPMbNvtLb/57bp1I/uooRuBMiMn8K5qXQ/1mKWj Q0f0KVKgT/VyCMsUAfEhNM9T3irxEST8KZjcXOtr4pX6s3pDv/6Q24n/zg4nrhoYz104cCT8U5YF v02D0CjSbMAF/dl9GLHnz6CSNFLL8rXyS1ipvMtvy5HstnNRsZFOlVK8jkb0atWJnhDpoQ1bT08V nyWSj9RBzi9ZtJQPoq1986ucpu2bYJ8s7A6bdYiyiA9+flVvGp/dhEeE9y5k9RP4sZZC7slcftlE c0en+QbsdnrSv3MaFdlBRO4EI5Ss819jx9/K9wGQiC7OV46lfNG5RtjaHzzU7Lp+mGDKWtoFIXUm SA1H5kUpJTo/grJA/4//BPYfcZ2iU+b5RFg4o97HkENaWVtW8E+RV88X/fXbz0qRpfpx0jwRbZ+F Yzq1VyKLZVcRrl4fxhSTBHoqyU5SeYok9UCF78v5FCGkNFqCZq4NSUvV/MbPXdhsmoXtg2dGzg+r I5bc4/aVe8FH5eN39kJBt+RtQKozOCNXedCTXBbhZj1J0iw7Jqk9JqmVMCw7XPEqSTJKmm4rL3O5 EqoajL/UfnTN2YWJGy+wknm/vGVdO+QiN0t/m/0aWVwiGpgr2zWyDFxb7zY8VPsZ9S8KSU53UvDO K/KUwf0zwYWDL6hwEBN1v25UxYPNib2isqa0T/27WNiJhV7qUyT6CT98CF6hKHO/RNXHyPAeomMb jjy16cDxqcLnRYTaPCn1nKuQog/yVlLH+eI9P0wt87MHqd0nEWfalylqleYUBXJaUH3Sx593EXy6 PXBWEf3wCIGmTeHgDASkVPwPqOJVLWtEnhJSMVJWgpPU7OoVdLacn9Qs4HPecCq2rLF+7dVT/B2l JcqQindimBIxtW4E+EfjV/wFkbqvV1RArdgytDKzZ58s/iyeOFLIgC7LBrvIxmjrtChhRKng+jt+ dLjTStVWmsXllY2ieWkKmAIVqb2KW57jVOY3zPbdZtP0Ii1Nn4rVGeoJRxSUlkxH4KaMSxYpzKQG kBw7saYe2uXm9HLnkHBioBiuqGj924efpWM88GcmOo4HOJ12Mi0WPjIwtK7VbNrEANO/QAkDEsPU lsynLwKZecZdqvMvNSDBaT8epmu1aSmA9UbdkBV8U8V4fw8II5o2HBF3Os22Cx3Bdi5oG7VjtMeJ qIj2+2YPqoIx5TUcREL5BXA3MihsK4dpA1zwyrHB4qvmrzwsCKc/MbaNAG/NRvB2pHkpSVS2HDmv MccMLQPpJXNNgRge5NlTK9ugIe5P+TOFrEyN13Vsy3G4dAY5wnm60RiV54RHuOnE7kV6mYODZoFe FlN6OaEGlpfRI5DMPC3Hi5OmJPLkwAVyYD01IE/JjNeUjDPsBsut6Fp/rH2PYMQED6JogdbNHZWa 5ylP3ihHgyg8lX3hOkl1V3LdfST8cZo7GrfilOII+BYoT2k/fsQR2q5Sv9zJdyMLtidfD/6rV7sO WA2wLLg35HpJ9ziNqws90MAs0ZTSNzMLouIV15DD/5GSFNxDeNbTNmaBOpiivBVoaRlzP3+CW1X3 Ky+FfbGI0bwI+vooAWUHO9Tq6hX74qc5viw1vrf83bDaByiooQEvYBiFh0VW9Ay58A3Jkz04/sA0 ZyvMuemHumXsloGBGwgawn6CdsCPdrNRSyE1EJlwZTpvRY9aRtenMkHdb6x+J5ObdtUOFzDdseU8 KH2vYOwt+D+s9aZd9s2+8wrY5IM32Q+KGCqYPx/kQSjnN3gBYW01EmFzRNgPuORS5yu03H8ME03y ub/dVuPtVsgj0Ml4Mphz0+Rr6K7vHli6XTeEuwd+l/Utx/dXbshWy0zQHfYEeSw0c9fML7sVMyaU OSfKZ/iiVHmKKguRPGQFUwsndJUsVMzTUQD4vfpwfvWwZWzKpPjdVw8TvOm8N+kWMgdlDCard+0S Fcvewo2BbKaaesPdHA3VigNyQA7EtiywQdvAZycPHtwrygru7raKiYml55QsGW85Nj2WAChGxmU5 Z99w1gBtMIXmCvT3j49UpHt/j0XVM3vpmz/JlY4NQOZmUgpySQB5bL3StTxagrCMbqs+Y6UIC+aE bIIoptZ7OtD4fY0sSuShrpqVnKNvg/y3yJSgqWPl2kyclftLCsrgi2x6rG7Ltb06O8NOVlGp1pzQ 4FQwv1GDmBUcJ25Uu3GXVDINX0uvRFbeB3rIWeImTHkQuTvyc89qmQcay8zd6tu78wz9L+nVtty2 kUTf8xXzCFSJXOJCXKpScWWt3SpW2S6VaW8eLD2AICRhFyYZEFSsD8gPbPLB2326BxiAtJNoH0QB mJ6eS3efc5r2k0ZuzgUZ2t03H9+Nk5POuMjinvMXUT6o0lw5v60gxFmFkRCvJSN3LClzT6MezEXm cXmqqUhPSs2dfG+1CyPQ21bcNgC4cR4VLKHcGrxjzuhF1hvamhPerWGsOBEDJ7DglRPn1CpnqqgG xls1RnsTqoetmNDUkzgu9RBytkIPfWlf0wCQXEkW4Z+EiFlEEBWH4VlpJn1pxhKMHRoKSTnNxVjq KXIIh4974pyJvWMntDRYbUevJHKR+Jl1VvddB69UqTXF+aceitzFuIhQIEXXVf22Uktm1AJAai8d FaoL2UP8RxZSt3bbPgenMjr13s9Q0zcMbEsCZrSvM3wN1buuhwpLPZ34BOf1sdb3neveiFsdas9i GM6TNH4JzCNiWc+ZoRbSalvtuhpksxTGYiwkIei1R23AYkmbiUqJ00Efqky59f7lc7BwxfhldZWx MsNltPstJFTqlTJstoVrZw7Fg4xX8p5Fj/JuPq1xaTevqZFlqUVzQonOlTnMDcTXIrqTEXPrzwdp 5WqGvitbjLuypZ468cDkaMpYB5B88RPBY9YHzZMMOom2pJt6lK/cWAJzst5by8g7miLu1R+xirwD hxYcfjX7bRJ3Iok8WKYOKKZD2LuifaDVJyC6pG5nsF/0cilUgv/d1Ec6YFNxPuaa+iIqcxBH4mmZ EPnX69csBzKS3hyZFQtdHaOGsZBJZMW1QCJlvVJ6JxLJ4ymEnEs30UOse+RJfkVq7VVqiY4i0ttZ IcdlzOA7vqqM+qPoL10UC5ChoBZ9kkSaJK+AyxQl85pOyrli6s4cT3jc8L3wjvnyqp/lTYYqnSdz Op8D3CgzPYufYisT9F/LTVtGV3ocgCkBfbB10+nk+iDmTTU9PN1UzgnfHz8Opsc/Ts+fJ5EzIcoG ts0UJAqmlczbnq02z5fE4i/UnHFoLzpVNDoytqIOWkEi828Eny+Uz03EUagq3ajhaVfKl0dFq2i+ yJOzlHOVC9daBD01PQ3JjujsNH9CPMO7d3zFdsTWcZLnE+SJouFWoxH0xIA7FluGbzmy0MOQhjaE v6ENkWFzwj95gThgYXvUqo1V46TWoIY+iARhAlxo5omLYY6jLGNoWHglVXtWXCSIEib+gVL6IEq2 eOuqfarLylxXTf1Utc/mZt9OSi6X0IiPIWGMWulSi0kdYyEmmbXPsHN9c+vbzV0oZUtyZU98yGSC vHqHkhJNTHzesxwAc5I3kqPCcpqkLMtybw1eeA30lucVQ1OHweqLoLm8zX10FmZ1JA1CrCYOZIzz mUtrp/C/ujGVPm8Pe7Hcuc5eib0xP/ooDCNOmTA5ovwsvzJ3D1TvffxV509i1MrwSWwLrNXI/reV 2lDIb72pWiFRkS/il7HWV/pSijo0H8UP3WbHyRrjIvk2pL+EtkIT2qNCkGTpVMOwgi17Vcvey4LJ hdxtKg4nnuWX6AaF+QoYTNxo1j8Kjryd+RDkIcUiAfEL9LPVZ593wd9tjVFpVz+f4LKAaYfSxqLr 6xs+3KjVi9CBMLlAX4jaiL0LpUlIktF9vVginKsjpidO8K3VQfJ75c944+Y3lluJ96sfgt5c9WOK w4H7UCgckQKkv5NgedZO9IuGumjZ1NUO/N8ByiihpWZpxaookdBL71E+6ECtll1diGQhKjViUuiE J5UYs0CSR15NsRHzuqntcs/i2jqS5oTX4DI+lvpZ/bXWlV2nk2pv1OxB98cfy6IZOsUEwkA8d3RC XVwP2u9P7ddvP15oBeIw+D8kodTWbNBhFO+2AodF53GK+jip2i/KknRKjftLCQ5I/Ri+lIhP6xVO 5yd9H9l09UGfzoRLPm1VcoexPxCiMe7MuAxwsCnw50zCi6/XNfocFE5XSxndS9FKaYFAvXMAiyfi KOzhiPW0KNclCBict7WcdyDOO042mQv0DMosGaglsdj2q88E/Dv1O4Dg3HvLOKY9TyQ9T8w9DzU9 AZ8gsd/UIqAmiNf9xwfqGQg7CDajmNq50IQLOj4JIxIHIen+7+6/+/sH2lTOW1oYPOS0mZiuLCIs iYkyP8tG42GjsWqX93SXqXdnPvqMT+85MpH3hrfA/JN51B+sOvC04Ve0cdAmOe24wYD8MtAe9mzD fzW+bWSoMuKBOUw6M9fJZz+MQN8MPmb9GhtaI/9WPkeWqUlmiNeykom8DiMuhraXFp+xLnr2Q3ie oiw1QUm6fKnmTfoEym3rQ4JuRvdN0KLtQs01wrQjBfTA1EZb/kjvnLjv+Toy740tMDvtmcQHJpJ5 yBfwhWE6IPkBJjTmg8/9GAcD3yv2emT2XMqz2dmZn3VMgFsaqqkwOk8KN9DTED/I436jI1RqJzHd 2Rjh1u99nElD3vIRbaRvfSdqx5MbWsmTAqnQJ49ez8SjORTjeJM+hieD73ZWn7AhYFpWFRfq30mm DQESdv+LH0KM6aDOu5iczI9wpMlZdM6bbtbOL11LSfidmKPUEKbFpOEI+kQLcluzd+aaPad0owGD SsGJQPewI3gN+qPLW4G3nOEb8hKWxzmySJuEFNd82aV2Z+r53eANVuquJzeCTxjWEJHX79YU6yMr oUiXKZqGO77YmXuS16OPBmorbyBtPJBi0cfmSXfa2S1/D6oO1ffeddSNNl70p8fuf+CwELcfZIZO bBn8SEHX+q5n1rf7ITqu8jyXPT3jizSA18xrBhmi/0UgVFf2XQ0Hug24tzvJoGXcJTPu6kbSklUd vm1HHojMKw7wT3zxVmCpEiO1Ke+66NFqpd1I++jiVmRJDLC6mqtoOg6nGlQC3eGlXX9js1e9TNL1 OYYc4G/MccLxcjRjAe+i1pOLFiN80BJWKpxi0f1Jx0uV0F+DgBFxUon8IWnZUsZs9aWbOFSt3ccI X0dYV1ZUggc9ZSvLVQPaRPMkiJcTyIkGQWNl16drnwvnztCC3oceW/ks3DVKxxo6WRByL1eRWk6R VEIQArQw7KQ/EhdFBx/jjxx99CspKDWFRmB/6qEc++VwnpwVqMeTncmwrCAO2lJn6uojPw+yeMGR 4M08i/dCV23gZ797GH8e2jyBNl7GbPe7//rUPUa6Om255UZlf9ItXUzhc80rgV4g0HSzgUjoQNIq Q5Cp3P+5b031RftR27TC+NAQyoyazgod7mCWI9HRC/KAPHaMTAGrYl2d+Y/qMPLWq6maYjbIspe3 MF+p20Eo/E2KqT1rnuYxeXqBirvYOPGqCYfu1us2W6ocvprMO+GCZKjzgdKUbeib8E1+a4xsmkrl VjqPF6TF/zC8ertcw0JGuHONnamKUmJ2Fo1OY6lRROBszegr4shPUu0zp9zjtL+FbLiFzGkLAir3 a+mqCk77pZPmIWd9LQTTYewRNyUlbN8qU+6hMiJBhMQ7sEwNPXFX47csOudtv+OK0tPyxHuuoCU7 jh2/PS7mSuvM9b9wpxWoTCM1go5uxTIDeZ0qIuVeqRNOmN7KyrrzZ7530h1FWz7aAf6tSjXQSeK2 Mp/yfCZbDONlm2HwTga/AbFOuvcAy53XnbkpgP2sLDtVMhBGvuUB2WiE+0o5XUL5bLrhgz4z7dEW D+3+i3W6A9oxwS+QsnCF+Q9wUgg4ytTa3UCp6zKXyPEJT6k3Q4DGrmSzW7Oxq3ZWA2I/DqlJ7CRy kQQp0emyD1nNbgRerNNBKPFR9tuqgdXc5x5p5d6JHgazZQkzWvwiDidhH6MolCBREi9lIeboK5/Z lrMt4GivkXQrJNgUpehwaeQAzyLsVaNW3BvoqIyigH81VS7laSFvjfm4qzvz7vR5w/qBCKaFULKt 3YBw+XwZh6GDcAMYEqYpGuqmaNM5JeIAnQGAgFDz3RSsg0UWD07DrIewcEDNmBo7hrClUnMA9Z0p WA7CIADl8vEOByZptSnOTFwX8lthWitctgSXZSQezEP9pOP/67wKdpwGYuidr8gxkXajziSdSY4I OOwZJM6BZncrSlulLYi/573nSdI2iMNeqmlqN7bHfu/ZfCycETqvl0F+O5jONyMjYSw+05TegHWY Mk2shCJa5RUmJNlAkBsGjMcOjclOfBhfHKYXvy/gg/v8bhHWerlPn0+sWsw/gm667LKX4xbCtCkN weOinRzaL1TtfCMx/pf02hiurpkoZEk1aaOcSt9xRhqitGMbl6YZtCvqUl4vp4f0LJmi9uxRn/9i auzN/X20ICnv4tso2rlqmkLoBYv2NwcONQIBOkNCJwzkGXG1vPuFTIjRubfpE708xLFobUxhsFq6 8D+F7pSLLcpIrDayBW24xi1WlDkjlzKahG6b2ycRsJ4AFH+9wXqQP1PuY/Z7iSKqyvUiUQiV0MTF UI/FXk62X62vrE3656eSdiTglo/uuCvOCcRJL3A5+Ix+DoJsVOF4PAzcVBC9Pf8AGzs9FdR0WaXY MYXUb6tMB0RfhhrSyNXAMk96/GkDeNU0w4bhodmCHHVwsS5X5oiow+xYzeGuUriPdmESI1XeDSKW YNSPlj5AcIt8azGg9i6fNglbP6pRUNVl49rqTlBN4D5x+7w3VaIjSpgfhQGWz4UbIK0twZ0MtZac eyQ+XWQrj0Iqsc+Gftede7PKzgeiYStypSXgQ9DJbbAxWm4VOqmlGfndy3p7Fqm3InVJGi9Of1ae SBobY9L88R+a3wdL/m3a281VShR4whLzlVmrLsK9va1y/aZgpZltOqQHye4sPiQKAZXB9MkNWGBK vCn9Cr11P4iz+nduJHi0bqPx8yY3vMQDV49NdpDUwduPWvusFaJ6BAz1kmR5J22BU2dcffOfGTaY 15ufk/HG1P+uqGfbb30KQXobYrO/cUEYw1n22ysvUwSfvrz7C3QbujUKZW5kc3RyZWFtDWVuZG9i ag0yOSAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNTMgMCBSIA0vUmVzb3VyY2VzIDMw IDAgUiANL0NvbnRlbnRzIDMxIDAgUiANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BC b3ggWyAwIDAgNjEyIDc5MiBdIA0vUm90YXRlIDAgDT4+IA1lbmRvYmoNMzAgMCBvYmoNPDwgDS9Q cm9jU2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQyIDYxIDAgUiAvVFQ0IDYzIDAgUiAv VFQ2IDcxIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDg4IDAgUiA+PiANL0NvbG9yU3BhY2Ug PDwgL0NzNSA2NCAwIFIgPj4gDT4+IA1lbmRvYmoNMzEgMCBvYmoNPDwgL0xlbmd0aCA1NDM0IC9G aWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJlFfbbuPIEX3XV/QjGVgcNsWbnoIdOwic LJDBSEgexn6gqbbFHZn0kJQHzmfki1NVp5qUfMluYFjsS3XVqeq69eft4tN2mxhrtveLdbTOTUx/ MshiG5VlnJtVGWVJTBSPi0+XQ2bqQYhiM9SLT3/dWPMwLOIojtPSbOvFkoeWhj8XwVNkbBxuf1v8 ZbuwpjGLbGWjOLUmWxFLE0dFYpa24HHvFveLz4ImBRqbiBD6rIllSkCKuGRghCNmSbHIWEUFS6Bz uZ6LkozOba8WgdEd5RhPOwJylTOXb8EmXBZRGrj6iEHfhMssKoPxJbzd/u1D+3hU+ZokCijiGWeC jEYkiuB9C759vTW/HMM8yoIxXNqYGO9l5tqzxSYkNkFdjU3H4rMA21G4TOhjDJElkQ1+Ca1lbEJt Kr/8GBY0B48W5zHBBmlGZs51dqE8G9J8TauXm2vzhEMVEbIJxsYNZmB7rBVudTi8JaobvzQ60wAx gRoaVQrylObgPIM+XK5YZRyYmYDvYTA8IhcJ1G577DvVjOSemokvKSY/imyW53y7crnryRnjNW7i qe/GMCV9u7o7yCAKyXhrNq5Ngi3JtHI7dL1kPSYYCD9p9PakwQ5oK6zpDk62bGMbPAtBhwkIsORO 745slQf+HKSLJ+oO314qdylT092zE9DAiZmtHqp7YZgFL2EiRmfmwA3eCgPUYuP58M60p1BVP2Xi ajBQRY2qHSZBv1NsP0Ob0UfWnFGAyqw/uaI0xRVNAfyNlSXqFatXkv1JJtuSkkPA+rEVch6TswRj JSty+WwrvsKJd5Gtp9jOch+HZa5xyL6+Dr6yj62CW8zMZ3G0FSP1iOVLUZnS/qDfHuQVX4tYRYj8 twFR+0DBWrJ9zDXo9bQZ9nrygAX9mDtl4LBrnl6hGAZl7Qn1oKdX/p7ctLrx8EoN5aIgxnMm6oV7 voac1h/2uqNKm3Ohz+eYdZNSBaUjL/Y+FB9SzoOS1vgc6WpTz1yRjSr7RfaMn+5lqrDb9/ZeKfHo lWj94PGomB4v9HLO1SdQlnPgvlUoP0gTqikK072fXdJkLnUJ/IsDDVdBPmzunNk5iQKkEkLEwZsQ w4TcbffWbSVTFVP1WM3V46oaKyPVokCMZzR4mYIlI36aDVcBqKRorMgPcVR9/Pc5WE46ICOflW+l hyn734GS2UW4bJ6+j6AC0eHCnzrK8mh+cqrdz5xmYAZFg4sU3UjmGR9Ci3sWUi4o6auDo9uB9ELV /l3oSn+GWOtXcVZUBUg1STsl7z/wDDt7hvV1x3E2ZdOzK5QSzeyBO3PHqZoRYWMTcjGmn+uLeat3 CJV7VH52IM6x4k82qJ0Z5WwHpqbDdI9tpTYAgK0K/JTBrup32P0glXoH//Y1pEYwuDWX8u3EKhRg YSJOwNgqBAEc/g5rhxDRJjejhpVikSndyIFoAZjUvf6CcmsLX23XqEYl+ZCQ+GMnfLHPHZxQ9A14 Kmux5Goq4+uZMc4ZDVQN3t6AShFR14ILLAOnQYx9I4YsfWTvzP3Rb4tYD8bBsKsot2km1k0L7y52 Cng18tWt+RcdZ1eNRUYbFoFvtHybRa7Evhv0soght1QrIpDtB/onSzI+zr+1uSFtClAOUsPHJd0i cUbU2wBTJZIVpbsJgYFjJjCbza+ftlLyi2AD9r9SM0B5+Sbw65cq19RsVgupzIL/XY1MIZObEKgN QWR3IpMVdHloiVd0b5wVOOwY5x9t89Dx7LXLQuNCUHO6CvbchCbc3V+/19VJ0HHIaQPjQoSatFfu 4k3PVGmD2D192GeBIjpt4LlR2oFGYZ6JMeMpF/OAvqzbGYD0GASqnNRuEedBfhZpie8nz/pU1wLa UQ5Q03YTeHXOG1MQ1Cp2Lxex/dNUpuqpdLH1iUXG8p86DPDbhKvg7uDIl6iXL8VPvRCydEwieJHu fa1DgoOTTspwLUW88JsNPc14QDVWN55lDuJD9yQljp8fiZUkCKIWUhQhQRCBAOQkXCQbVKDbma2k uJTcJiczfDGHimyOrkBaGGVUV2f8ufarPs2BBTT/1kwv/UuzueQ6kpIbct98jeSwPMkOcz+QrCfz 2ikJ84vq1ghYvn9qr0XNIuC4KYnxgGfYEV8nzwV5N9w3PUzBfpHkfA06pRKpRB1HpJWKIaeJf61E rNQa4cliSKGTHg8vOUbjXCuEDchaMHyQtTeZcNbVJl5XW8658Hr8jxwcpHJpi8CemcRcdLTw8aeH nJGTleRgqYaj7hp6vnbaEnILgnfq5ILAayUOZLH0VBXn4YINNLG6w9dhh+6a5QEgdyJCMxHrFyLu DzpVrp66Ag+KDdwpEKgi/rGLvlbWDNipyn5PjOJ18rzMOTcvETNFsztTiFIVMjk/b5mqGd7PvW+D v/Gh9X74z3lFw1K7+5PgUQbisifhLVPQa3Cy6uTy7QNiznPBtGo47k45jubqPAswpOogL5ZM7ofl bVyPRNLUnA6vug1Xv4GLh1VGlYBAfEt0azq5kADiKxGzavoRWxyct4XwWTLdSOJ7sbJVZoqT1ilb VL2rToyepvnHKWEKb0mtXPPUzAhDjmCNWHJpTjld/12DccmxZLjhyOQ+aFIhkgde4togHqYJRgiU F2Qhh85z9z9zzI2mCS773NQbZaIEzxPe2VHWUn9KrT9W6g/PzIOPAHkdUF/Ucz2vnYefS3+pGaiU uliiLsnlJN4YkkBfJ78PWvu517Daa+waurU8GLgJZnfkG+YEXAqGEhh4ifzUSj931+noiH1Tdy0W 9OPIA2o0UKMwhwjpTa0nog7lx1HpB5ApuxHy9tisPP2uMyDTDckL65nhzuig2zlwQqr+tN2mhhzt fkH5OaY/+qzJAmmcmzRfRZS0H8lS7JJskWAVleH2Nz6W67EoyegYmZBuFTvKMJ520L7N2WQlrLbd U3foHl7MVTPU3bPrX/R8gvPUlOQCSQYTqJRqayqotCfU/trOReUff7+Y+6Mk1YchdaFVs0MonLRP iWZkdgyuRXngaQ6ueg5zMSY6VjY7z5vhtLLkeGeymx3YTym/fuZL4vdMLqnpXNge1C/0+AzmUM7x PuGd0ejgycft0lNwDgAiBclFjXyjVgUgceDegRy0UT44++f3n39Z6k1YpmpC9pwi+AqPvMWMS7M+ ClbE+3JzreuDiOYHaYzmtUDmZGL9eC33nAn4MfHsQGbaTqI+ReXPiZZ6Um7JnnRd+dX4jMrPn2pD JEY9Cwietwo9Dn7Bn7rHm1J5zrzwffZMBCkkqFw1wXTCq+PVVJN8kQxEsdq6CbbgIwcop9M9iL/z TcqCt8DwCtjzqVZcf/gN+8K+VIjqqWRubyYBtPcIFfEPnR/PdOObI1Yf9GziHLn3jfXkGvJyu2IH L9g3ZMqay+ALy6PQYZ3kVZJx5ROFxL1pkdQBXGR3vAPlpliZDMFaoCBQv7OXQ62y+iGzI1jQ7sBv qFTag1OGyuEk8Arp6mWVbsuqjzLLJTg9A50unoCEKhG3Asb8E3q/EqfgOgXHvx5Yq1xGFfoznCn9 8e9TOawESqvsH4DsEZ9WrWOeToFW5+zvJ84Ji38MpTUcTgOQyZ/fGEh5kZyX4f9uTHbN/DzotMr3 aN1IxrU+ZUy12/Vn9ZtrM0YX2qpg9m5DUJ11l8V58X+vE+i7I3cNdVVr7wLm0R/qh9YSJ3Zqut40 Qv7xNboZp5z1gCiggh9HbXlAoQ837UwUM7087tx/Oa+a3saNJHrPr+gjtbC95qeoYzLZDQzsDIKx F3vI5ECRPTITwVRo2on//Va9V002R/YkyMEym+yurq9X9ep1QnJdz90to8u1vteYRQrMIkp0LnSg rUjzdb1qN9Ih5IKoW4GxSNjKZH1I5ie2ncy2sdXI1IAA7jTz/PrIif/tlHaiIHTC86O1zTKhJPFh a5vjbc1aIdff8jppMY6v7JDd69mXHlm3Ll8dNrPZc9fmuZ8+yqwj3UzEv8PT7UYr441D5CuEVVQU LwjNfMai7zwSsjDCnyMvSswpkl/xghQ3Q/DrhIdcOAWYlDrtSD6iW1ZM4oo5UNoxIqY2Cl4GlSZJ fSKsEEyAw1E/P8okQXQxOPqyEwbc81FmDb3WVs7UaNrJ2bWmZ2Mb1DNSvneza/heGOMbLQKeLmdP lwsDk9RL7mBqBjPKBAakRmzxwuaIkr7Ik7B8wGdTks6Uc79w9bQJiqewL8Mwemxaz2uGYwcG9oOn GI+ON0pcef5lk6FkmXBuMifOlSyFnyv6WWQwaLtwGMWqtNKgn70L1hnQuGL8q8U+3LygU1lTY0a/ UBEpqWeVQKYQTRxNN0HZQ8Aw1QcZO8yArVmTgq26PHnHQp0l4UPY94a1klWdf7Z71MWtF4+aiwZe yYP85ab9Fx6e7oNTwv/O/PEU1BwWlU/sDFKUaWD7tta3H6xvvv8Rzrr7B6e3ZXhjHgqOJyTy0A5H PDiMSwKRx42aIvVLqknb45tcxq6ii9a9v/nOPevmZqPRGvm+2VOQ3zDKKuVKu4S7gWTHC3UGM0nc I02ID/7C3Uqv3SUfaNP7H91qX3TeyyhpBvAjn8fm5CLLtJVIplEtidK3G5QcrqdI2REKNnzVs2bA QH/k4kUXF3+597PNeano7HFHNLxOddt7RxLQbNRztsGa3x3yv060CGdJoAYnbBnYQcfJUfrcjdHw 91z50SRJ+iwVtZxpQVDM2mvgKOvuLZD1f8TMgTInEz1zjfuomR/CWbvwkpJpuDfag/YxacY58fWr XT3bzs7M6UxPiiIJehrE+E/JxKVWjNpePuANPJLT1pIekcIyRtMuCs/TsRNisjR+YLK2glQAgNu4 EhUkL8JkZ0XMik8bSaq7AQVd4mqVCLf3VMl3HHhLE2uFu0r4WTD3wocWVEvUvYhhm+WzN7b0xiMA Ql1rmprCFUWyUEBxdfPAnVpfyojM6r1yi++kjoUcMSm9snj7ztqTo7+iCG/tCwjIFilaiRnaDDNp hv5h6ic9D2cuuoTbUcH0yVOMiD2TzD5TW7POl2atM2Ke/E1G8z/yItIFowHKGbzRGNCA1gem84S3 J80qIwAxsa6YYzJgTP3p6P+cX1AGemKWnPElSDTCtATvT3jT64RpF3KuXmgTNA2syRywqNwO3M5f 3059UMycNA2n4TgcXt4AajG7vKLLtZMlgGLBBPsaMarm08VCjL7dgElYcCoEJ9fgRM7JzS0FU227 ohUZ4lcxfutT/GBxQ7iqJVzaPy5rNBMVgngtobeLNChA8PSX9KFcVO1CTkl9GH1D+QO/ScNEn9+b Dh33oiicdZqfknZAxhSsbAXjvLV7mhC+0l5YouRW3LdAgGTbq7arm6ma6GNiDtxPo+3w6FaXiu6N 3cayUKIskCOXGB8qm0lSayOZ/TK0a5WlVP57GNl7qtmtZrSE7ULZrAa2HWLFx+F4FM0+Jc2vdvD1 hD2rEa3NEA8GAYBubgsEjiJMmbPVhU8bqdRj5wy7AaEvRIzJs5fPkdQwXwSIPcXCXVRtpnmuacPk sVey8xRhtSOyJVQHu4IXBowrxSeT2IW6Z6WHRWkYGzs4FwF+ME+uK95iY8wns4VQZqEzbTGCgJYC gJvCLE+tZS3fR0VYyiTVr5qkW1AKfPYU5t0iRjXFN65HsLpdwl9lcZIIPHUR4TbDaJGhhUCMCTFt 7GUTyxpsnvl9g9T8UjNBK7aZoI6CwjssDjM1j+XZ21ibIBOw2qJmRCaajpL05g2cfOSij7fYscWb dzY4vcNUsJoEzgkWc2BrlGRpBIW9ITGJa14hSaGUqZM5xIhM139GihdKHx+M60Rg3SLFCksxirxa gBq1hiJdUisNvQFnfyaBF5RIJdVg1cmRFBmRzqJWsA08PtM0ViCc1FWMMV355fYh3oA0TREI+pr8 sKCrU8a2Dmdsdc9vdi+lgvLbpaZJRC1LowNFuJA7O8X8+obOI7ZBGVPD7GgiyGYGh0yBuwuJGoy0 FKlVIN+02DtS7N50tzMHMxYofLWoFvkSLEunaSMVRpiD/juwBSE8SoNSTSToteUdOYeiZz73cgaA kjTUumE4lblOI97pZzt1mHtbSbcrVd2UwawdzCrUrDI56Qd9MGEmwnk1NokkLKUPd4+b60Rabpqq g3Xkld4EHUbBiPxTC80ytxwA2sHMc+bJ5XaW7/T0LfrfO/67eWOkPOP9x0CWjWCT8mvX0/ThdFMy 6Xegy+Tvonm7DAs7/K1pN5rJNpFR5iv8+jVMfgTR+9mJIagzhVomBpHq6ojEBG4HJvUlE69jQvuT tyTrwgP3W9779v4hwEnXv8UosdR1j4a4uRgU1i+fuME7y3ZD0eg+m4RxhVjiBdEtwLQXYBCdz7Zv RH7kc73Ze2LM1qsS0b1WGzQFdClsaeU1R92bIC5W62s0+nzuERr9fc9u34J6DMZDwqyABOJgCMYx Z8g8kZxsxgm84g/yEAlp63mIv1cWax1wtRuRo0zulhPGO0xgtzcEOCaTzs4bHbGLfNsbPbJpLMCQ 5GUYfo0HoAWiRqhyxi4MXmHg4RQU+M9cc7OKgcrJQuASyr9wTZjimuPxDRKZzQ6v6fDfdWbhxJav uH+mQ5DS+25QX5dQamz4iWv0R7C2DFwMYVCQ1vjNUsNqbQI7EyNjxGU1S4ExddL2zSQXLs0lh0O1 H0yaVnRnloQLe740kRbcEn7WXwRXPrgfvFkDC0d1DQ4ijxaTZzMQ/R0GASl2NyvWuFQROg+A2wJw FQCXJugFAWkFkZXO+uk+/joeFcqVAZ7qhox8IOcL2x4UVOTb2eioNhLjeDlAuuTKR9RHD62Mh+p0 p/b88+6ucGLD529SsbXKd+5aM4SmEZNqYPKf/37Y3P2i2zNuF1nXdcHdIYvaObNAo5kHW8BRJyr+ NkhV7cwVi2gZhjU82AtFrMbCj9K3drD0EjDhsQgGhi+c6lsTfo9dZyZmV3VZFpGJ7PrBxLWF0uJl d7Q5r4KJCiOa2OjFpeJvC7rCzlNc5Wmen/XCM8LadN3oSSH5Sza6P3onc1PjDuCbPTjjM0ktYJQb of3SwN2VxCRfgjJjPA2TTSmu9ONz33ohGMf+2Y8vCylabK9RmxfTy1n12tr4lVYl0cDdPu2tDQuW MxT7S03N3+ztk6oMmiS5qVxDUkL7AnkUhUx4eQ9c2nGH3Qqea4gwpFySV6XmCtKva8whl1ok5IfC Xtxg+z5vJK1MQsN/rR29P8sQKcxFWf+NDIG70znAOxwQUtP7Cz30r7tv/j8AuHmbygplbmRzdHJl YW0NZW5kb2JqDTMyIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1NSAwIFIgDS9SZXNv dXJjZXMgMzMgMCBSIA0vQ29udGVudHMgMzQgMCBSIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBd IA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDS9Sb3RhdGUgMCANPj4gDWVuZG9iag0zMyAwIG9i ag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNjEgMCBSIC9UVDQg NjMgMCBSIC9UVDYgNzEgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgODggMCBSID4+IA0vQ29s b3JTcGFjZSA8PCAvQ3M1IDY0IDAgUiA+PiANPj4gDWVuZG9iag0zNCAwIG9iag08PCAvTGVuZ3Ro IDM0NjIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImcV8ty28oR3esrZjlIiTAG b1RWsewkTMWO6pLJXdheQOBIgk2TDAhKV3+f7j49ICnLj0qpRMyjp5/Tp3teLy9eLZepcWZ5e9HE TWkS+pNBkbi4rpPSZHVcpAlRfL14dbUvTLcXosTsu4tXf1s4c7e/SOIkyWuz7C5mPHQ0fLywu9g4 Fy0/X7xdXjjTm4sic3GSO1NkxNIkcZWamat4PPiL24vXy2c6uLQketKhciJfxLijGMdiPthN5JLY 2Taa0Sn7NUpT+vgojVP7MTJ9NKtotI+quLJm1Ud5XOusE5rtA877wa/MTTRzuX1iHqkltVo9oDLu 8DH/iWY1McDeKL+tMFvL2OyG7eoA2g7bZtWe0pldq7ygqKmze8xjY14H+d1W1FGTdjShTzsElWQa VIo+Lf/B0cwRzSyJyzKtyJXLN3BXnrHnEgnNP//NcTnG3sVZXRQnxFkhxDQqC3jZzNmRtX1DYss4 s/vLaFbGhTXjvY9mRZyTiRyBgrwyS9nHHZbJhtTFpX0gQwurtANx4Wg9idqzLKVA13wdYleUJavB stMsaJFU0IIc2/l9VNBR/Jqu3UQ16WNuyREsu7FYIG0yDvl4L9N2NK0hqamEkVf6KCOleERKOUe0 yumZK2mrbOoa3mFlquqHnmyq8kjrqmCCy9SRPWtOYtqOTSEnVhYrPXkotzdrzzrzBR224uzcHmR6 F81yIsNesOUr6RwI1hyA0o79jnjsWuUC5jE87bI4acqcPV3l1aRlPmlZQssPbz6Z31m5kr2TcKA3 xC233q9YjdKaLb5YXj9FjoMf1HoAMY4O/e2TzM2IJL2XGYWEVBQm/d4skFNXpG1lF/NT+yhLOj0i q0FoD27kxIqTJOGcmdZIF9k4QMJt1BAf/LKjKTOVF2YeLEfKv7+rZTt81IjbAwxdQwAMuuQkLCer HyP6VJPHzlf79TqsQJaKGb/x7LiVYCXf5MMUJQSpPUS1vWMPiUyX8g0mfuJKR67kRCVXwtG19fAO rUUZmXDFuVjpZG7W7RPTRAwnQ/w8CVLBZfo0CUC5LBwnJpD5mBDESzOiDBmRKrKQQ7CjPJNpR23q yKZ37aa9E5RIKaaO75Cg3RgUerFcTUplWZzmU7FIj8Ui1Xv9G3mrsJ/M/HoWOQZQhnzGZa4JZP7K 0E3KUE0U99Mwu8PMYyZ0/oyOIb4R+Mdo223XMgDzOHJ8Rc1c6bB6rzJAiV9zQwm8xT5+e1m/wbY3 H+3N4UymarCFNbo4+P8qFY4PfDHprq+oMKqGoDdKpk74oXeIsJEylv3EPULoQTi+fKOzZopQ0kwQ z6qVwXmlKFXAWwWlM2cfwQF/TKt7C0KN2r6nm5zad9eGagJvrwxHm1bI7wXdcOkFcotNqq4YfMbq AbMOwnuowKXMhQPHdC7Eo3IM5KqOrkGbK3zmjCiimFRz+j5GaSVVJyWYbOAjO1wyOPJp8KL4Dx7W eciHJMOJngiUkdty+nB9CpoOSjTpyLM1+MKZBsyUXslXXtmcechLzJZ/QnzKqRg3WslGzlK2nfYl E9jXR9x2XO/paqS8uwkDzm0ZfNUDG2VjWswHsa9keBY6Zd+DSon1bIcPLVIfWdpDOBt0ud2KNtnE 1CibBSNtzgjo8jCZB5qNP2f7SLXkOaMvOt+zHyutrnK9q6IJtxst1+y0+gv+lIQ/CwLnlBSoRRse z6Pk1HtOcixHjikgVcgxhZ0KZB5kI1c8SZ4CqVNI4jSCOhlQR7OoAeZkjDmF5f8j3jR6qqfVG96i bkTt567jgIOg8d+zQhrXDCAwGSMaqzUrsAObYDTdwTQIGo6eSLP4//LFR+uV4hIeWJzreQSkPJ8A yTVTjdUi+3YjSnbrLTdShbRagqQsiTzgB0nrQloe8hv608JemsVb9cICDimlpNGlrqSrKbWbQTvQ iE9KwWA+zgi9PXECn2rJt6ANJ+8QB/Z5v8CtvRLzuKgT9lBmq1oRtzqd13OQ8fMiX5SOn2nPS7xL ntf46pdqfFqiyM85QpKnpTwGSumGWMPd6VIrSzf9ugf50y/1AEWe8VR7gHRqwJNjClbcAIxUyxrp nemhwFeau6WR/JZP6VBO672sc2/O7ww0b3xqMPfSfm2/ZYIxw5pzUiuZbiURweORifi2CNnI0WMw 5T1q2HrMoSL188oFcw9mI/fgrpDEqNH/1+Imui31M5ZqjT9JNjbDt0p1EKoB/WxQCp8dbPOd6qRH YQC1BD2kn8vzht38V9STq6j6flfrjg/TD7bn+00e9IN0xxWacG2pSzu0o/8O1h47CddMgW4oFYC1 NZAFKINCwleMCmUpHQyFUtpjXhTMIdFrvaSzRiEZijSSstLd4F7yaYx6+QX6pYHIYHUvnafcKx7g F1s3KikE54CDgUSU2kD9O+XJ3YuMHkKf9a1eL2rDykKcWUoxo05F/HT9Slbn14ZxGJVE2jDqJexW j0AlNbv7ApaISBaXLi+eh+UYlfCwjPgyfTJ/4RYlk66kstL6ZFKW7PbANmJsJryq1JIccJhyNyoP NwMOYe8RbGeI1k7QOQ19EpGFufLg7EzENZSKXD5XBxmPwhRiT1D4eORJ1EMULKilM0PfK5cnl0zG ljkRZFrs4Vc1PbdQteUOpMEtLaTuSY/n0WdmQSEVdSbebyAYRzDug6NwF6gsvvzKLCbELBAzEp/F eBpqA8r99B9yFXa+w+uSG9WjpxpJKqb/zKhQAyALO/SjwBPxAXYpnKQoYOyDK8nWRSQ9mYaEuXcj w5FjIJl0IA8wKu7WgVlgDoWgwbDSTbVDVyXcI7dKpu06v/YELv12IwdjhhtKVkmFpSYQePb773QO buocFH/o9cXg1UjL4Gw/oPo2HCPprzfYHumd1q7XkowpX6xG3KHzvUSNDGIeAoM16z5ucdjsdbB+ AAXDsBx8jOSiB7odvgPHIQ2rN3o2chLfM/X21EE57FMzck34cjzMkpTFbcThNeczgXE3wXhjVXrQ v1XBUtrP7L5kmCN1+eFZTc6Q/qkKmikzCteRjXI5cnvxbhduwqPUTZe7lGrGjzWpChxpx1hr5CHQ 0E2X6SgkrbzlNpgohJYCobm9NmAjBaRBAamtcpECUktmHvlgjFPbzV5K55/B+qNN2e8Dh5E2Qamf JxaQsiMSae5E6w3UvMPnEuh4fLSW8raUA36Qrp4YqaVeFacULNSWBdXsuT4oWLA8KSS28oGq5hba YW3A4hjoZuh4xewf+sKLL1bhvfkyCvVITQMUeIYeI1DmeYYfQNxv7nBWyRSusOkZxR+wspkQACzW xwJUyg1gPanuYTqhxDlCTHwFXwLe7QCAunkSl0z8x8fX0G4tE7Puv4CdLj/BhBvfBdsV/RSXz20z vh2mgyd+dfkEU2WAqduoCdhCCO05d7kz5Qg2Eqw0wWWjO8RlKwW19FG6GaY4OzKSKjlW6MHxC8+N nHhl5bPnRn7+1iAdf/bUkPzO9K3xG5mHHPVs2qzSrvtXnhJ5QjaFl0QGfh+4MjWW2pecfP9OC9EM ZTelIFNvo8FVmvfksMzi2JzhjTrr91fS586XQlrpbswPOfO7v/HDJffPUvM5P9e7+4hA8V/xpfGr ftwOMRcmY3AMvM4FqOwB0+7+f4xXwW7bMAy95yt0dIAksyTbsY9dMmAdkGFYA2yH9mA0HprCW4o0 39Lv3Xsk5bjA2uWS1rJMkXxPj+T+hEJNPkSUoyatd25z2HW9G7sP4flviLfThbKevnvYy1EgrLzQ 7gFLj3oaj2WuP20noSkXHrpb5LCHxJdQ+uCAVlGjBE5+TT5u3wAiNn4REhDlWcOVwd6Xy/lUfFlP 6fJiyuQcMWl5XDUfZ4gkcyG4TctOCj2UCznf5fm/p4kYhzOGsfEGUoUumapWS8or6d2YEJ58pymv ADduTZnpdsJhaFcJ7cLeAW0iZXjTfzx9l51EvVDUuTjCnZjzY7U4PsQ8IwLZXlpDn7VHUSbPgkGL iZmrUbdZS8+oP9KlyqXluTuNjIoF8r24INR4NwtKDJ+r399ks3h0kBVy4jUjgGvtXVzyovkyEvSL CFEhjOptRoRYvceI2hjhwQippVgMeWJDkso4SOXSOLDSWexKZ8iNjmRzkD3qmDq++kG4QOG5uXZf 7RKtdMbaSuZBBGRrjd6vfZ65H7SNLr3v963dLBFXuLYW/PWIvXZ+y4xcuP4jVx8CnqVVqQ0+fapC XJgOR21YeHJ3/yALumx7ehXtGj2EPEv7hTy96KNbpfe/zXx6NuNX91TWOVu/Z7fpTg9j4+Y/jBn4 MS9kLowFCmMFMKqL0Y9AbNCDYnkeMg2oCKaijSEMAlMDH+faD0dVqNupUPXnlCvCeWbGZz6Xnl/b LnbP8u6R2FTC3JIJaRpdz8GubGHE4oKLM/ke3Ny0R/03ba4XKXKJA4HHgL+uqBnxRXGz7A9hn6M2 djaNDZOhKI/6X33nPrc2TcrQx8ryZW+TI6oIZ5yBshwy3bo7tfu+26mgeDKKOnV4OuiM2vbODMoo eXT2oWDPDfp7lppaOWBTp86ix0Ov26yO0JzPP4wi4N7SEmujq35hY6wYc/XMhSp58KT+4CLzXT6k 2yN/FYkG5rPuQDuWr+rOXwEGABvbx68KZW5kc3RyZWFtDWVuZG9iag0zNSAwIG9iag08PCANL1R5 cGUgL0ZvbnQgDS9TdWJ0eXBlIC9UcnVlVHlwZSANL0ZpcnN0Q2hhciAzMiANL0xhc3RDaGFyIDE0 NiANL1dpZHRocyBbIDI1MCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAzMzMgMjUwIDAgMCAwIDUw MCAwIDAgMCAwIDAgMCAwIDAgMCAwIA0wIDAgMCAwIDYxMSAwIDAgMCAwIDAgMCAwIDAgMCAwIDAg ODMzIDAgMCAwIDAgMCA1MDAgMCAwIDAgMCAwIDAgDTAgMzg5IDAgMzg5IDAgMCAwIDUwMCA1MDAg MCA1MDAgNDQ0IDI3OCA1MDAgNTAwIDI3OCAwIDAgMjc4IDcyMiANNTAwIDUwMCA1MDAgMCAzODkg Mzg5IDI3OCAwIDAgMCA0NDQgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIA0wIDAgMCAwIDAg MCAwIDAgMCAwIDAgMzMzIF0gDS9FbmNvZGluZyAvV2luQW5zaUVuY29kaW5nIA0vQmFzZUZvbnQg L1RpbWVzTmV3Um9tYW5QUy1JdGFsaWNNVCANL0ZvbnREZXNjcmlwdG9yIDM2IDAgUiANPj4gDWVu ZG9iag0zNiAwIG9iag08PCANL1R5cGUgL0ZvbnREZXNjcmlwdG9yIA0vQXNjZW50IDg5MSANL0Nh cEhlaWdodCAwIA0vRGVzY2VudCAtMjE2IA0vRmxhZ3MgOTggDS9Gb250QkJveCBbIC0xODkgLTMw NyAxMTIwIDEwMjMgXSANL0ZvbnROYW1lIC9UaW1lc05ld1JvbWFuUFMtSXRhbGljTVQgDS9JdGFs aWNBbmdsZSAtMTUgDS9TdGVtViAwIA0+PiANZW5kb2JqDTM3IDAgb2JqDTw8IA0vQ291bnQgMTQg DS9GaXJzdCAzOCAwIFIgDS9MYXN0IDM5IDAgUiANPj4gDWVuZG9iag0zOCAwIG9iag08PCANL1Rp dGxlIChBcHBsaWNhYmlsaXR5KQ0vRGVzdCBbIDU4IDAgUiAvRml0QiBdIA0vUGFyZW50IDM3IDAg UiANL05leHQgNDEgMCBSIA0+PiANZW5kb2JqDTM5IDAgb2JqDTw8IA0vVGl0bGUgKFJlZmVyZW5j ZXMpDS9EZXN0IFsgMzIgMCBSIC9GaXRCIF0gDS9QYXJlbnQgMzcgMCBSIA0vUHJldiA0MCAwIFIg DT4+IA1lbmRvYmoNNDAgMCBvYmoNPDwgDS9UaXRsZSAoUmVxdWlyZW1lbnRzKQ0vRGVzdCBbIDUg MCBSIC9GaXRCIF0gDS9QYXJlbnQgMzcgMCBSIA0vUHJldiA0MSAwIFIgDS9OZXh0IDM5IDAgUiAN L0ZpcnN0IDQyIDAgUiANL0xhc3QgNDMgMCBSIA0vQ291bnQgMTAgDT4+IA1lbmRvYmoNNDEgMCBv YmoNPDwgDS9UaXRsZSAoRGVmaW5pdGlvbnMpDS9EZXN0IFsgMSAwIFIgL0ZpdEIgXSANL1BhcmVu dCAzNyAwIFIgDS9QcmV2IDM4IDAgUiANL05leHQgNDAgMCBSIA0+PiANZW5kb2JqDTQyIDAgb2Jq DTw8IA0vVGl0bGUgKEdlbmVyYWwpDS9EZXN0IFsgNSAwIFIgL0ZpdEIgXSANL1BhcmVudCA0MCAw IFIgDS9OZXh0IDUxIDAgUiANPj4gDWVuZG9iag00MyAwIG9iag08PCANL1RpdGxlIChJbnRlcm9w ZXJhYmlsaXR5KQ0vRGVzdCBbIDMyIDAgUiAvRml0QiBdIA0vUGFyZW50IDQwIDAgUiANL1ByZXYg NDQgMCBSIA0+PiANZW5kb2JqDTQ0IDAgb2JqDTw8IA0vVGl0bGUgKE1hbmFnZW1lbnQpDS9EZXN0 IFsgMzIgMCBSIC9GaXRCIF0gDS9QYXJlbnQgNDAgMCBSIA0vUHJldiA0NSAwIFIgDS9OZXh0IDQz IDAgUiANPj4gDWVuZG9iag00NSAwIG9iag08PCANL1RpdGxlIChUb3BvbG9neSBEaXNjb3Zlcnkp DS9EZXN0IFsgMjkgMCBSIC9GaXRCIF0gDS9QYXJlbnQgNDAgMCBSIA0vUHJldiA0NiAwIFIgDS9O ZXh0IDQ0IDAgUiANPj4gDWVuZG9iag00NiAwIG9iag08PCANL1RpdGxlIChTZWN1cml0eSkNL0Rl c3QgWyAyOSAwIFIgL0ZpdEIgXSANL1BhcmVudCA0MCAwIFIgDS9QcmV2IDQ3IDAgUiANL05leHQg NDUgMCBSIA0+PiANZW5kb2JqDTQ3IDAgb2JqDTw8IA0vVGl0bGUgKE5hbWluZykNL0Rlc3QgWyAy NiAwIFIgL0ZpdEIgXSANL1BhcmVudCA0MCAwIFIgDS9QcmV2IDQ4IDAgUiANL05leHQgNDYgMCBS IA0+PiANZW5kb2JqDTQ4IDAgb2JqDTw8IA0vVGl0bGUgKFRyYW5zcG9ydCwgTmV0d29yayBhbmQg TGluaykNL0Rlc3QgWyAxNSAwIFIgL0ZpdEIgXSANL1BhcmVudCA0MCAwIFIgDS9QcmV2IDQ5IDAg UiANL05leHQgNDcgMCBSIA0+PiANZW5kb2JqDTQ5IDAgb2JqDTw8IA0vVGl0bGUgKGlTQ1NJIFNl c3Npb24gTGF5ZXIpDS9EZXN0IFsgMTIgMCBSIC9GaXRCIF0gDS9QYXJlbnQgNDAgMCBSIA0vUHJl diA1MCAwIFIgDS9OZXh0IDQ4IDAgUiANPj4gDWVuZG9iag01MCAwIG9iag08PCANL1RpdGxlIChT Q1NJKQ0vRGVzdCBbIDkgMCBSIC9GaXRCIF0gDS9QYXJlbnQgNDAgMCBSIA0vUHJldiA1MSAwIFIg DS9OZXh0IDQ5IDAgUiANPj4gDWVuZG9iag01MSAwIG9iag08PCANL1RpdGxlIChbRF0gT3VyIGlu aXRpYWwgYXBwcm9hY2ggdXNlcyBTQ1NJIGZvciB0aGUgYmxvY2sgc3RvcmFnZSBwcm90b2NvbCwg YW5kIFRcDUNQL0lQIGZvciB0aGUgbmV0d29yayB0cmFuc3BvcnQuUGVyZm9ybWFuY2UvQ29zdCkN L0Rlc3QgWyA1IDAgUiAvRml0QiBdIA0vUGFyZW50IDQwIDAgUiANL1ByZXYgNDIgMCBSIA0vTmV4 dCA1MCAwIFIgDT4+IA1lbmRvYmoNNTIgMCBvYmoNPDwgDS9Qcm9kdWNlciAoQWNyb2JhdCBEaXN0 aWxsZXIgNC4wIGZvciBXaW5kb3dzKQ0vQ3JlYXRvciAoTWljcm9zb2Z0IFdvcmQgOC4wKQ0vTW9k RGF0ZSAoRDoyMDAwMDcwNTIwNTgwNy0wNycwMCcpDS9BdXRob3IgKFJhbmRvbHBoIEIuIEhhYWdl bnMpDS9UaXRsZSAoU0NTSSBvbiBUQ1AgUmVxdWlyZW1lbnRzIERvY3VtZW50KQ0vQ3JlYXRpb25E YXRlIChEOjIwMDAwNzA1MjA1NjE1KQ0+PiANZW5kb2JqDTUzIDAgb2JqDTw8IA0vVHlwZSAvUGFn ZXMgDS9LaWRzIFsgNTggMCBSIDEgMCBSIDUgMCBSIDkgMCBSIDEyIDAgUiAxNSAwIFIgMTggMCBS IDIyIDAgUiAyNiAwIFIgMjkgMCBSIA1dIA0vQ291bnQgMTAgDS9QYXJlbnQgNTQgMCBSIA0+PiAN ZW5kb2JqDTU0IDAgb2JqDTw8IA0vVHlwZSAvUGFnZXMgDS9LaWRzIFsgNTMgMCBSIDU1IDAgUiBd IA0vQ291bnQgMTEgDT4+IA1lbmRvYmoNNTUgMCBvYmoNPDwgDS9UeXBlIC9QYWdlcyANL0tpZHMg WyAzMiAwIFIgXSANL0NvdW50IDEgDS9QYXJlbnQgNTQgMCBSIA0+PiANZW5kb2JqDXhyZWYNMCA1 NiANMDAwMDAwMDAwMCA2NTUzNSBmDQowMDAwMDExNzYxIDAwMDAwIG4NCjAwMDAwMTE5MzEgMDAw MDAgbg0KMDAwMDAxMjA3NCAwMDAwMCBuDQowMDAwMDEyMjMyIDAwMDAwIG4NCjAwMDAwMTk0NTIg MDAwMDAgbg0KMDAwMDAxOTYyMiAwMDAwMCBuDQowMDAwMDE5NzY1IDAwMDAwIG4NCjAwMDAwMTk5 MjMgMDAwMDAgbg0KMDAwMDAyNTE5NiAwMDAwMCBuDQowMDAwMDI1MzQ5IDAwMDAwIG4NCjAwMDAw MjU1MjEgMDAwMDAgbg0KMDAwMDAzMjEzMCAwMDAwMCBuDQowMDAwMDMyMjg0IDAwMDAwIG4NCjAw MDAwMzI0NTYgMDAwMDAgbg0KMDAwMDAzOTI3NCAwMDAwMCBuDQowMDAwMDM5NDI4IDAwMDAwIG4N CjAwMDAwMzk1ODcgMDAwMDAgbg0KMDAwMDA0NjI0OSAwMDAwMCBuDQowMDAwMDQ2NDIzIDAwMDAw IG4NCjAwMDAwNDY1NjggMDAwMDAgbg0KMDAwMDA0NjcyNiAwMDAwMCBuDQowMDAwMDUzMTQwIDAw MDAwIG4NCjAwMDAwNTMzMTQgMDAwMDAgbg0KMDAwMDA1MzQ1OSAwMDAwMCBuDQowMDAwMDUzNjA2 IDAwMDAwIG4NCjAwMDAwNjA1ODUgMDAwMDAgbg0KMDAwMDA2MDczOSAwMDAwMCBuDQowMDAwMDYw OTA5IDAwMDAwIG4NCjAwMDAwNjc0MzMgMDAwMDAgbg0KMDAwMDA2NzU4NyAwMDAwMCBuDQowMDAw MDY3NzQ2IDAwMDAwIG4NCjAwMDAwNzMyNTUgMDAwMDAgbg0KMDAwMDA3MzQwOSAwMDAwMCBuDQow MDAwMDczNTY4IDAwMDAwIG4NCjAwMDAwNzcxMDUgMDAwMDAgbg0KMDAwMDA3NzU4MiAwMDAwMCBu DQowMDAwMDc3NzgyIDAwMDAwIG4NCjAwMDAwNzc4NDYgMDAwMDAgbg0KMDAwMDA3Nzk0NyAwMDAw MCBuDQowMDAwMDc4MDQ1IDAwMDAwIG4NCjAwMDAwNzgxOTggMDAwMDAgbg0KMDAwMDA3ODMxMCAw MDAwMCBuDQowMDAwMDc4NDA0IDAwMDAwIG4NCjAwMDAwNzg1MDggMDAwMDAgbg0KMDAwMDA3ODYy MCAwMDAwMCBuDQowMDAwMDc4NzQwIDAwMDAwIG4NCjAwMDAwNzg4NTAgMDAwMDAgbg0KMDAwMDA3 ODk1OCAwMDAwMCBuDQowMDAwMDc5MDg3IDAwMDAwIG4NCjAwMDAwNzkyMDggMDAwMDAgbg0KMDAw MDA3OTMxMyAwMDAwMCBuDQowMDAwMDc5NTM2IDAwMDAwIG4NCjAwMDAwNzk3NzcgMDAwMDAgbg0K MDAwMDA3OTkyMSAwMDAwMCBuDQowMDAwMDc5OTk1IDAwMDAwIG4NCnRyYWlsZXINPDwNL1NpemUg NTYNL0lEWzw3M2JlNzZmNzU1YmEzNWVhOTU1NjM5ZDJiNTNkODQyZT48NzNiZTc2Zjc1NWJhMzVl YTk1NTYzOWQyYjUzZDg0MmU+XQ0+Pg1zdGFydHhyZWYNMTczDSUlRU9GDQ== ------_=_NextPart_000_01BFE6FF.695C2A40-- From owner-ips@ECE.cmu.edu Thu Jul 06 06:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08824; 6 Jul 2000 2:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA08787 for ips-outgoing; Thu, 6 Jul 2000 01:25:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA08782 for ; Thu, 6 Jul 2000 01:25:13 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id 193631BE7; Thu, 6 Jul 2000 01:25:13 -0400 (EDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id WAA15743; Wed, 5 Jul 2000 22:25:41 -0700 (PDT) Message-ID: <39641839.A0A6F3@agilent.com> Date: Wed, 05 Jul 2000 22:25:13 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: iSCSI Subject: Re: iSCSI Requirements v0.3 05 Jul 00 References: <8F889D13E8D4D311BC510090278CE32D0E452A@xrose03.rose.hp.com> Content-Type: text/plain; charset=iso-8859-1 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id BAB08787 Comments on iSCSI Requirements v0.3: Section 3.2: Due to the buffering requirements at each node as link rates increase (ne= eds to buffer frames when a frame is lost), the lack of TCP framing, the retry behavior= of TCP to the detriment of audio and video streaming, do we really want to restrict= iSCSI to TCP? The benefits of TCP are it's recovery and congestion algorithms, bu= t those could just as easily be implemented in a UDP type of transport. Section 3.3, first [R]: "a stated requirement (below) is that iSCSI shall have no impact on T10 a= rchitecture or command sets. Collaboration with T10 will be required to achieve this = requirement." These two sentences seem to be mutually exclusive. If iSCSI is to have n= o impact on T10, why must there be collaboration with T10? Section 3.5: "[R] It has been noted that a remote DMA option for TCP poss= ibly could provide the desired framing." Not if the iSCSI header is lost. If the iSCSI header is lost, RDMA won't= help, because the RDMA will not know which I/O this iSCSI message is for, and w= ill not be able to DMA the data into the correct buffers described for the I/O (Init= iator read for example). Section 3.5 "Selective TCP retransmission.": "[D] Given the long delays in the WAN, using TCP selective retransmission= must be supported by iSCSI, in order to minimize the bandwidth impact of retransm= ission." How can iSCSI support how the underlying TCP transport performs it's retr= ansmission? Section 3.5 "Firewall friendly. The protocol=92s use of IP addressing and= TCP port numbers should be firewall friendly." Why? "real" implementations will use their own dedicated lines to reach = long distances, and will not transport data over the "public internet", especi= ally due to the "information super highway traffic jam". From owner-ips@ECE.cmu.edu Thu Jul 06 06:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08834; 6 Jul 2000 2:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA08862 for ips-outgoing; Thu, 6 Jul 2000 01:29:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA08856 for ; Thu, 6 Jul 2000 01:29:34 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by atlrel2.hp.com (Postfix) with ESMTP id ECF43433 for ; Thu, 6 Jul 2000 01:29:32 -0400 (EDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA10506 for ; Wed, 5 Jul 2000 23:29:31 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Wed, 05 Jul 2000 23:29:31 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id <3L5VXXC7>; Wed, 5 Jul 2000 23:29:30 -0600 Message-ID: <8F889D13E8D4D311BC510090278CE32D0E452B@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: iSCSI Dwgs v0.2 05 Jul 2000 Date: Wed, 5 Jul 2000 23:29:29 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFE70B.28332440" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFE70B.28332440 Content-Type: text/plain; charset="iso-8859-1" Here's a revision of the drawings that I'm developing to explain iSCSI to the world (and to ourselves). A lot of this is SCSI-oriented, but some of the new stuff is iSCSI-oriented; hence the name change. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 <> <> ------_=_NextPart_000_01BFE70B.28332440 Content-Type: application/octet-stream; name="Haagens, Randy (E-mail).vcf" Content-Disposition: attachment; filename="Haagens, Randy (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture TITLE:Manager, Storage Network Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-3586 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:19990528T091551Z END:VCARD ------_=_NextPart_000_01BFE70B.28332440 Content-Type: application/octet-stream; name="iSCSI Dwgs v0.2.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="iSCSI Dwgs v0.2.pdf" JVBERi0xLjINJeLjz9MNCjM0IDAgb2JqDTw8IA0vTGluZWFyaXplZCAxIA0vTyAzNiANL0ggWyA2 MjIgMjQ0IF0gDS9MIDMxNjUyIA0vRSAyNjk5IA0vTiA5IA0vVCAzMDg1NCANPj4gDWVuZG9iag0g ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB4cmVmDTM0IDExIA0wMDAwMDAwMDE2IDAwMDAwIG4NCjAwMDAwMDA1NjcgMDAwMDAgbg0KMDAw MDAwMDg2NiAwMDAwMCBuDQowMDAwMDAxMDIyIDAwMDAwIG4NCjAwMDAwMDExNTcgMDAwMDAgbg0K MDAwMDAwMTMzNyAwMDAwMCBuDQowMDAwMDAxNjg2IDAwMDAwIG4NCjAwMDAwMDE4NjcgMDAwMDAg bg0KMDAwMDAwMjQ2OCAwMDAwMCBuDQowMDAwMDAwNjIyIDAwMDAwIG4NCjAwMDAwMDA4NDUgMDAw MDAgbg0KdHJhaWxlcg08PA0vU2l6ZSA0NQ0vSW5mbyAzMyAwIFIgDS9Sb290IDM1IDAgUiANL1By ZXYgMzA4NDQgDS9JRFs8MmQ4N2YxYmJmOTJiMGI1ZjJkNTBjMDQ2MjdjODMyZGU+PDJkODdmMWJi ZjkyYjBiNWYyZDUwYzA0NjI3YzgzMmRlPl0NPj4Nc3RhcnR4cmVmDTANJSVFT0YNICAgICANMzUg MCBvYmoNPDwgDS9UeXBlIC9DYXRhbG9nIA0vUGFnZXMgMzIgMCBSIA0+PiANZW5kb2JqDTQzIDAg b2JqDTw8IC9TIDEyNyAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQ0IDAgUiA+PiANc3Ry ZWFtDQpIiWJgYGBmYGDKA5GsGgy8DAjAy8AChhwNQA63y7+oTWkPGEQdS5om8Wd4TnFxL2BgCPJY U5vkeqd6mqsaBIboHk91ulPGKMHAwCjYwODiGsHAAYXMxg0MShpo5gBNlmRgyC8E0uxAzAe22I+B S7h54kx+yQcbGBTypzgeyGPxM2BgAAgwAHgkIvQNZW5kc3RyZWFtDWVuZG9iag00NCAwIG9iag0x MzggDWVuZG9iag0zNiAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgMzIgMCBSIA0vUmVz b3VyY2VzIDM3IDAgUiANL0NvbnRlbnRzIDM5IDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBb IDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0zNyAw IG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNDEgMCBSID4+ IA0vRXh0R1N0YXRlIDw8IC9HUzEgNDIgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDM4IDAg UiA+PiANPj4gDWVuZG9iag0zOCAwIG9iag1bIA0vQ2FsUkdCIDw8IC9XaGl0ZVBvaW50IFsgMC45 NTA1IDEgMS4wODkgXSAvR2FtbWEgWyAyLjIyMjIxIDIuMjIyMjEgMi4yMjIyMSBdIA0vTWF0cml4 IFsgMC40MTI0IDAuMjEyNiAwLjAxOTMgMC4zNTc2IDAuNzE1MTkgMC4xMTkyIDAuMTgwNSAwLjA3 MjIgMC45NTA1IF0gPj4gDQ1dDWVuZG9iag0zOSAwIG9iag08PCAvTGVuZ3RoIDI3NSAvRmlsdGVy IC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiUSPTU/DMAyG7/kVPqaHZHa+2h5hIMROSM0Ncai6 tASNgdpO+/sk6QBZsl/FzuPXu/1iYViASiwD2z11BNPCCCIw7aRpoLalWIMgaoUwBzaye8923qv0 yY8sNRqJBkpCMFpqBQ1JB/4zNXMkNEpEC34TWIO/slf+0leNbPlUCcdDkUDVmz8wQUrWmjSkaTIt +AcmCoFuCMJ2Q2AlyEniNlWpOBwut4dT4hEH9TuAWMjJrUbQ2ZVWTbZqqc4lmS0rsKwokopNHrt9 9wx38/Ae1zCslzlU/iODFOaTt4yglJKtA5up7h9n/nBqc9x9hyGOcejX+HUWczj1azhWJh9/nPtr PE9Lcfro2Y8AAwDtNF2JCmVuZHN0cmVhbQ1lbmRvYmoNNDAgMCBvYmoNPDwgDS9UeXBlIC9Gb250 RGVzY3JpcHRvciANL0FzY2VudCA5MDUgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQgLTIxMSANL0Zs YWdzIDMyIA0vRm9udEJCb3ggWyAtMjIyIC0zMjUgMTA3MiAxMDM3IF0gDS9Gb250TmFtZSAvQXJp YWxNVCANL0l0YWxpY0FuZ2xlIDAgDS9TdGVtViAwIA0+PiANZW5kb2JqDTQxIDAgb2JqDTw8IA0v VHlwZSAvRm9udCANL1N1YnR5cGUgL1RydWVUeXBlIA0vRmlyc3RDaGFyIDMyIA0vTGFzdENoYXIg MTY3IA0vV2lkdGhzIFsgMjc4IDAgMCAwIDAgMCA2NjcgMCAzMzMgMzMzIDAgMCAyNzggMzMzIDI3 OCAyNzggNTU2IDU1NiA1NTYgNTU2IDU1NiANNTU2IDU1NiA1NTYgNTU2IDU1NiAyNzggMjc4IDAg MCAwIDU1NiAwIDY2NyA2NjcgNzIyIDcyMiA2NjcgNjExIA0wIDcyMiAyNzggNTAwIDAgNTU2IDgz MyA3MjIgNzc4IDY2NyAwIDcyMiA2NjcgNjExIDcyMiA2NjcgOTQ0IDAgDTY2NyAwIDI3OCAwIDI3 OCAwIDU1NiAwIDU1NiA1NTYgNTAwIDU1NiA1NTYgMjc4IDU1NiA1NTYgMjIyIDIyMiANNTAwIDIy MiA4MzMgNTU2IDU1NiA1NTYgNTU2IDMzMyA1MDAgMjc4IDU1NiA1MDAgNzIyIDUwMCA1MDAgMCAw IA0wIDAgMCAwIDAgMCAwIDAgMCAxMDAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDIyMiAzMzMg MzMzIDAgMCAwIA0wIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCA1NTYgXSANL0VuY29kaW5n IC9XaW5BbnNpRW5jb2RpbmcgDS9CYXNlRm9udCAvQXJpYWxNVCANL0ZvbnREZXNjcmlwdG9yIDQw IDAgUiANPj4gDWVuZG9iag00MiAwIG9iag08PCANL1R5cGUgL0V4dEdTdGF0ZSANL1NBIGZhbHNl IA0vU00gMC4wMiANL1RSIC9JZGVudGl0eSANPj4gDWVuZG9iag0xIDAgb2JqDTw8IA0vVHlwZSAv UGFnZSANL1BhcmVudCAzMiAwIFIgDS9SZXNvdXJjZXMgMiAwIFIgDS9Db250ZW50cyAzIDAgUiAN L1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYx MiA3OTIgXSANPj4gDWVuZG9iag0yIDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSAN L0ZvbnQgPDwgL0YyIDI1IDAgUiAvVFQyIDQxIDAgUiAvVFQ0IDI2IDAgUiA+PiANL0V4dEdTdGF0 ZSA8PCAvR1MxIDQyIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSAzOCAwIFIgPj4gDT4+IA1l bmRvYmoNMyAwIG9iag08PCAvTGVuZ3RoIDYwNDkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0 cmVhbQ0KSInUV01vG0kOxe5Rv6JOi+6B1a6vru7GnhJPsuMgwWhHws4hyEGxZccZWcpIcgL/ovmb +8iqarVlizEWGGAWARxVs4pkkY98rNOzba0utsrwv+3F6PRfU6OutyOjbtTIhcq3qqn5v9prNW6s VpvF6Gr0cjY6nc0sDs2uRhC0lfaK/2jlXeW9Mq2rWq9mtyNdaf6uycCYV62aXdB3o4OafRu9L2w5 NnXVFm2JDV2h3tylDytldfqpdflh9maksypjq9CGgMXsRzZSJ61aN1HrZF620HZdjkOx4J/Kso5x cFUTjI1no0/doU+/rje/qRucrUKxgg+F+rJZsy+s0hSbRVpttxV+6aoplCqNx5fXeeOG1vje7/2a NC6+qfkqfbtUF3n/bdlUNv3NR1Y71h4Ktc4nlriNK+6xspUv/ll6rNT2Lkk/fk420vqiRKiKXkt0 MS+z5fT5Iun8RDecZw9ocZ0WixhCgzQ72w5DaHMCui6G8Jd0RVtcUmKjx4bi8VNpQlUnA7aYX6cf i3xgy1ZeAWeE0bOpinmfngGbb/Djs0K8rfqmjFbv1PsPWl2OuqbSreq6KgA1prJe1Z0n8AKzU8Is kGoDoTT+1apGCFxQjba0jcBKV+DbEJy+jQpfASazTwtYnp6r2/XlYqmu1ht1ebPdbW4+3u0WlL3b 27vVzcV8d7NebcvZZ/LbubqqrfJNQxYMLHZBdZr+wh2L68N9jw91UNgKd4Knwhn42vER+qOV9YZ2 1s6TOngaQ+445DHa0zPErqlMMT2PKUJ6G9+osal0Y12fJ8uFgvLYEEzaYr1bI+st0rEukZ0a6ErH dQsDdJz+x3HYWKRDX+MJoLlDti5Kh8WiT5rxuF7orwdgWfl6pjHIWo1LOqSO7gf/Pt1HI1vWzqZc kVyd9062TVcPnYx3NPGO56td9Nizj12xXq0WF7vYSarWdO5oeJ57VUAR6YT7aJLWUlZvR6bl9Kb1 Ele2uqUg1KalwNxmAPQ7shw4QcSWrBHh6uVXP0Bpx50YW2CLjAyNLns5TLQtrxlzQxWPA++1IyWh IaUx8jEKuR9qawfwqlG8qIMXX74sOR5NzEdXzHclNMYgxaaCrkEhQmV1noJsue/nIDc9cIu383uE GlUzdpVtGqOoR5iWunjeHbIzpnvgjCdnIo47xvF6GVGi3s4Jy75gCHErpP68yaBJdg6c8r2ZxCDz 6rdqXqlkzZK1GbEJW2yK+Wr7ZZ1+J0w95EJbt1SuAXzThRhdZqph2b6fviihPBTv0KbrYlxa6sNg RIq1Yq3gWh+5FskyiEHkPFJS/EGRG5BxVSPhg67s+kR26VLKVDFLxLpk60N2/bAi8SPUTe/6oDyS 70+VqD0sUVvZUD8RbdNH20dtR4pVRYCkeqPWn5urtcjtoLcC5vveSh0TyH5+b7UtsvD/0lrz7Z7b Wm0HcAWptT7K21+qtdqG58m+tab1oLXa0FHf61trvyPLwcSh3bfWXr5vrbRl0Fn7Hb1Y79vq8Ph7 S5ZtZYLiQcQDl4Cns0zWt3ntGwt6Y589RMSMtqH5ot/hTBd3YLSio0m+HGoM0SbtgE8et64f2OB5 Y8kWQi9fDn3QbVRhgR70PEftljkprn3LxcWhdWiVQBmGvc7udzgdolvWtaQtyZdDjT7baCtYbW2i vWQiDgZLNuCzeDn0wCYn42hnHAAy4FfLNLqnvrjeJwzbdTsEiYEfRIaZf1MZ7TUkeeZfAgU+tX6P KzoS/NBIlGdc0ZH8kLIdZMp7y0gD//Xd6eqJEqX4wV0PGkD2eQxNj5zvtxJ0aVM3j7qrdvtJNrVX VN1X0uWo3oi4iWAWiuv4CnTpIpvThsW+BNOFTMuB83XNUCdGR0OMs/VTV7IYqQEJ7/O8qh88AR83 /9SSDLckHiQQwdiS8BBZqgfthj9yHPA4QdeJH3Lv6eDhnxEPi8QjVcpiSg3kd25r/cPH1En1i3f8 jsJra766VGfr27S8pTkELXZ1mYYj9PFjzJhU7bB5Hq+K+xtCwGVcbjMCTB0CnjaV14h9T/z9NGOb oVe4BIaMcYPb0QAAdSdqOiGCC+jBB5IcUdTNIzed6w2EZOBlVGOimjhhIGEn6h8XcS6rMrU0zncK e+D5gB/EoZMqoYGLac4jRbi61fX/EsA0mfCYeRi5x3Pg9HxCQcJpjcudj+PIZ0lpQJDG/KZ9TZNh nhQneU8OIQw8HoOd7VHZ9TGMZroYwzrG0FGS0vdpiWsXs9gWYJymRsX4wnzyn8lJDjGg6o4lLdnC 6Ik6ztmC4sJNTg++zuJ0/Ra3pRH65GAihVIMQqAJUAimIu4VlWvcg6xw2aW54Yb0kTEMLcXZNBo5 Py3hf1fMzvgiTTE5PZ9kS3m07TsIol5rRvzDZ8Kf11SIJYDO4ZWeg66DujxoRCapmJyfqNfx4pj/ Jz+NUxG5EuM5Mo+0DOQJ+DxYPaPLvZ6e9vodVeaLtxmSsQaPv8ww47169YpR5wtDd3SF63wPMD73 6HyfA5eeG4wneiqOYzOd4J3jip/Syj4DUKQYM8IjW2boawvGIR9NYU/ptzsKHxoOSJ07aD9PRO98 xdywKzn29JuAU1LZ79YbdX76c0ngUdP7eJtt/G9XUsgWkUNuE4K6pn4cbjOsxzO8XltGabFYJeDh Ir5DaL73VOV36TzSS3qSXi926rzk7J3GHvVz7O7sbovP290ie8eReALfB6N6eojRwzG+Dlz0r79P 2ztmHmgwpAHjdZGX6pfF73cLCqylsJFoR16a/Oh9nRLXEBSGb152rfgbPXxxDBPug7AWfyfBIO+Y B5p0fJybQ3RR5/ZO+DQFOxMoesQ47K5nd2kJd7fxy5f1Km5M60Uqd1zKe9Q7Rtq2rn0u+LrpURXy 5IGk4Bg1vs3XmwtCbZPW6seyTj+XN1+Hks19LNvHLf0B29193Mbb3Kf/c4ppQFMUh8DDNU2+mF5p MksMfgD9MzRojJ2UEgsMYzzGzbclsfjNjkOFTK2vFE0VJBxbMCAA7wnA1A6vq7hbocR56lD1iQrx 4AkXiGqypDtRNkpCNNiv27hWeSLV9Cg0Oj5+MIojpqblCbzjCRXz6HT0+34HRDSu9vP3rz+o1WCs RdYgrUNLk7emV81wVDe25leCIKZQHhUjgqIU34+L+ZkhiZ3kGT9JJbGk29DsLom9dFrzS0oQe8kz fVxmOnqYSmIt+EXcLPhl+Cl7XExAkqRG8oxfjpLYSMprgrcklkx7+i2JJfwTgUmeORH/KGBZKuEf rU/CP00jkmdaxD+R4nGp7kT8QyzhX7ci/iGW8C9hUDci/iGW8K+DiH+IRdu1hH9IJfxrL+IfYgn/ 2on4h1gybSX8/3v0csbMSCnjP1pRORLo644Ckggy7GfDZk/ioOUFyM8mGqe5Nq6JxnshE/letilt cZ9fFZRvkdAn600cM3aZARO1Weu/Q37Wts8hP4e+KZCfJEbxHxcT+UlSfD8uZvKTxE7yjMlPEku6 mfwksZdOM/lJYi95po/LIvlJYi34FclPEmvBLyY/SWokz5j8JLGRlDP5SWLJNJOfJJbwH8lPEkv4 Z/KTpBL+I/lJYgn/kfwksaA7kp8klvAfyU8SS/iXMBjJTxJL+I/kJ4lF27WE/0h+x8VexH8kv+Ni J+Kfye+41Er4F8jPASN/TfKj55+vHc9QrkWJ3fZL76icve949POBSsj7/5Jedrtx3EgURnI5T9GX 3cGOzL9ukpeOswGy2ACBNYu9EHxhKCPHzkgjS7IX+0T7mjlVJHsmtnk22YUBqzmnu1gsflUsztqs ldH1xs9RG94yPmCsPWoZenzqmp060DnwXRkZjc5hHZYsVhtBjmAZ6AR1dL2pDtTxYVPdq+PmejG0 jnSW681j3aAkaOt/Zgh+1r44OKk+ZYOsxuxqfD4tCPT9hLqWx/vDhEPWjW8n7P6IDcIJML5+wtZ4 /KbScTLjnUbWXtgcZC/kT9uKkIvZF4dpixqOr5Lu4xZFN493654YoOWFHiSok8bCoyeRUBg95BdM NWyX1qDcnBoQI5vpkxPUXNBzd9EILLpYvPztN/0QuIyUN8ZqEAb5JyZbNGQFzrjC0d+PJSZv6sJb NLD+g4Zn+Mck4CMapgXnE+RQ7VF+XK7IlaE3VnwPqMe6Xu07Qi2RZYRoZF1THYOdbJWaMvYpn1mq ozKNBLIMwYXRT9vYaI0uhhyIkvpTpqkjgFfcqGOAV5ys47aeYmkd6TxfBs/DJ6ljcNS1whDntTDY VEL93QRbQgmmC+NHBBMFZ7yu4+GyPjxIUbAjasM87h9aqH1M4lyLch222MTlPG6x1M01ynNpYdYo z7PGqL49h3NLZbRGuQ7XKLdxiXIx1KJcplmjXN1Yo1ydbG/X9VRLbcSivJTiA0dxXp2nt0Z5rkDv QG0Gv0r1Y6H51wmbOA4/TuiObdPu6t83+/J3DfWVxsWJ86/M8DNwsYXDKK7e6ljKlh4/B1wrPNJt LifDkuoLszsFPOAY9Kt+OOlJ5jhsbr5pr4Rsfj9HLY11hiofzlwIoRj4Qi0wGkypsqfzCrWoBczn FUvUYcFSgiRYwm3FUoN2OUlw6uhBQuWFTTS2KAeIiFQMGeyHl2LIt1fffyh/949Ff8Ie5HHQID/7 3qHM7eQYvsjwE96ispr1eBu/mnbvNlv7ae0dv5bfn+129XN8nWZbvtY1IQt3pdI5W1e3x4oQFysr 2tpYM608PazaZMsSmvJ4f6xPd9PWYRmPGHoNREvIZSl5U6nwy/wJFYsr7UulYv2gpeGSS0NYqVj1 ExXyyjkVbY5GxWJ/R8XJhT4VfokyXZit/Pk8i/z/mUW3Rd3r+am/lHNV6EAnMdbXhA7lqtqWl9xn dCDwydkeHYijM/nP4DHPDQ+7Vgs5uF9jcxfZ5F+xOjnP2w937eHNvj7cqt5GOOmFUnj+cv94X388 3jVz+1Mf8G640mKSvBwVATslrRDaGeyCdJazdp6Xm/ftjSgXS7i5dqX//Ga4OzUIeZZPZrQDX+pf oWK3u2oQbIiKetFV4R0TLXHKycKIaollxZWoZFp9JKrrO4WmIvadgur6TiU5Qono+06hSUzEqShF oasu2uP3VWJYj3OiBvJtkGJG1ECc6pv90kUMjsAWCpjVNkAzG911yWxj53oPeyHd+Hj5w4AjQKq8 9Ptea70O1tws+eZwdi00Ix32+r/lo7damjv52FVDaR+6qjRmPVXykYiWOKX5SFRLLGs+EpVMq49E dX2nSj4S1fWd0nwkou87VfKRqJ44pflIVGK49Ol9NZBvQ2kuumogTvXNdvPx1Jb/0dTDuR2cjWj6 sWtYZzukXW6p7Nx5KuOuBIvP7+8PaNoyDmG1hiblSXuV+qO0beg8dAI3l7P/f7ZQr7hXZrASaz3A a78W0cYFbevKGDEK0sU5YyTVV9nmLJvY5MvPbxdzuYa5IBTdtnGIToiUL0JIyoGLshvrG97m8kaY F82ooh/OLbbbBd5I0nVk2aOzOZRcfIE7mVn1w7kPeu1bO8nzbQ+4C8Kt4MHS+b7/6Sjbi2znKH2c MwjSysG8dnxX408P01bax+PTUc2hy5rQeM3jYWUpzGnYOnORTPyjMInVCCDF6nG1tBgnlw5zMdtk T5bsesKE6tMv/y63h0e92uhVIpXVoj0+DD/cPe3V7dB+VJ/RM9/t6w9PDbP3yhGCj7KLnciyoXY2 n58zHisEOi7oeyU/64s3G0ECLPZF19U03fsiItITo5Tkvuj6/mBDieb7/sxCc1/0fX+CJl9X7Fv1 wnpfDP0vnaRLXwx9f/oTWmk2+qLpeaPlGxekUC9In9wNmztWFttaKyytXpqMreBLCtvxYTK4O0k+ KtX7nzWxwvjhYT+0Z1yUgPV4M0k+WeSIpMhdFd9KckgSYFfa4NjEx+GXfX2Evatq4BJTj89xWczj jyV3t274z1xf/Iu+5calzNqGsQxfnWr6mkyllcaCUw2Vj7KXJZKutNJ9NfTF0kr31UC+La10Xw3E KWLW642xrxryrdMbY181xCV7QT61emPsqUZvjH3V9i2XUkjU/rSlGBKVRLmUQ6I64pQURCJ64pSW RKISzEtRJCox7BnmpS52VccwL6Wor/YlyzAvxbGrGoY5VIK5yQRziMRlkxjmUAnmJjLMoZJpF4Y5 VIK5mRnmZmaYm0Awh0gwN55hDpVgbhzDXPr1vmgZ5sYyzHEfIJgbwzA3pic5XCv6mIvax9zlRDAX tY+5y7GPuYh9zF1eCOai9jF3eSaYi0qmDQRzUfuYu+wJ5qL2MXfo5KnYx9xlSzAXtY+5w02zj7mo fcMpE8xF7WPuUiKYi9rH3BF/U2SYQyWYp4VhDpVgnmaCOUSCeQoMc6gE8+QZ5lDJtI5hDpVgnizD HCrBPBmCOUSCecwMc6gE85gY5lCJ4cgwj5FhHheGOVSCOQlxnBnmUAnmMTDMoRLMoyeYQySYR8cw h0owj5ZhDpVMaxjmUAnmS2aYQyWYL4lgDpFgvkSG+cKuoG5hV1BRiWF2BRWVYL6wK6ioBHNCxcKu oKISzBd2BRW1j/m3u82z3c4NdtjdbLaawQb/ypNcIzErmvtl2N1ujEqP1xs0+9bNw+56s5XHgMd/ ba7Gnyb0OOPDhJtcHo9PE65H83i8nnBfwd8Jt7Q8Hia5SIzDpb67Lz8+fCzvvC1i/WI/vdr9bYNL iQXVW/xd8rD7Tqc0XiaXuZMvcw8v5SOPj7Zy3xvff2hPj0+DGnr2fV0lumgLA0aMGbUjJsavpt27 zRZfiLhFjs7YD7wyfi3CKUjoWROSwZx8ic2Z3OKwzRoIcWCWSGwtAnJ9PEywH2XxSRePfF3f+jht 0VAhBPrKdRmVEGDTIlJXQpAW1+a1cwsC7iM1CC+O09aJ7TtYhaPjDexg98a3mMZJxG/r39f1haf2 wmTG08cy6193GzvIP2y4d5ow0poBnM+bcacZ05c1ZZiMnCEy0TRrmGzY15o3TDbML1RqqlrmmZEj gsmWGLdZDgkmk6ltkmOCySziSO7IPItyUvRl1GyqeubZLIcFkz3zLMhxwWRm21P+ITP+UZkY/5AZ /wwjayn/kBn/Um2ZX4bybzLjHypz3CTKP2TGv4mUf8hs6oXyD5nxb2bKP2TGvwmMf6iMf+Mp/5AZ /8ZR/iEz25byD5nxbwzlHzLj35iu5nJm/ItM+Hc5Mf5FJvy7HAn/ohL+XV4Y/yIT/l2eGf8is6kD 419kwr/LnvEvMuFffucq4d9ly/gXmfDvsmH8i0xsp8z4F5nw71Ji/ItM+HfM6xQp/5AZ/2mh/ENm /KeZ8Q+V8Z8C5R8y4z95yj9kNrWj/ENm/CdL+YfM+E+G8Q+V8R8z5R8y4z8myj9kZjtS/mOk/MeF 8g+Z8c/CHWfKP2TGfwyUf8iM/+gZ/1AZ/9FR/iEz/qOl/ENmUxvKP2TG/5Ip/5AZ/0ti/ENl/C+R 8g+Z8b8slH/IzPZM+YfM+P+t7yrYaRiGoYJjviLHBClamjZpe0WABCfEcps4oG6TQB1FGxq/z7PT prdo0pbsxX4vluPEoSnmP+BS/pfyJNTF/Adcyv/givkPuJT/oSrlP9BS/gdbyv/7KAzXDItPGlFT S2rg1zW2kvEkLMOXQaBPqZyXcRCGhg2Gf2KnXjXeYOqs0ZP2avrVaPG8mgaNfgu/Gt1mr0ZNTZCS W157SH+er2nNZwJni4N+jy/gCnXbS4POqgtOxgfmtK4nemLvXGKXb2TWwMygM23V5Wf6XoYHyb42 T05iJ0fhkzNL7iw7Ih/qRscvCoSzAYwoCd4zo7olYBPjbI7ndYdTZ7Oaul3U9EsoTM+xIAGegmEq xGSYRm1IEfbf8f5RG/KqqzZ47CEKvGRIsyUKXdvXHAVIy7SBaLH1Z9opDpTaI4RVMnawUR+avokd TyI1g4gL+XyMYvdu5V7UNTcE3jk67yfhPI4uT1G3RrEVtWtp6Cu+0k4w4GtgNhgz7gKVyZEdoqZl /HgHnz3l/ryEOPhezyQLnihozk+pjMPFLJMqPfJzkUlTJskyUVFXjfPqcQVR1FaJGV4lphVZYfaf YbvKyyDM/wFI6F7SCmVuZHN0cmVhbQ1lbmRvYmoNNCAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9Q YXJlbnQgMzIgMCBSIA0vUmVzb3VyY2VzIDUgMCBSIA0vQ29udGVudHMgNiAwIFIgDS9Sb3RhdGUg LTkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0g DT4+IA1lbmRvYmoNNSAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8 IC9GMiAyNSAwIFIgL1RUMiA0MSAwIFIgL1RUNiAyNyAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dT MSA0MiAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgMzggMCBSID4+IA0+PiANZW5kb2JqDTYg MCBvYmoNPDwgL0xlbmd0aCA0MzUzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJ jJdLcxNJFoX3+hW5rCJGRb6zMnoFppkx0cR4RoroBbBwCwFiZImW3Cb493NuZlVZYOvQ4QirMk9V 3nx895FPL45BrY7KlL/javb0nwujPh5nRm3UzMXO9yqF8hO8VvNktTqsZx9mz5ezp8ulxUfLDzMI fae9Kv+08q7zXpnedb1Xy5uZ7nTp12JgLq2Az1bSb3RWy6+zN41u5yZ2pgn47WyjXv01dGzbHv+V HV/Qun23fDXT43jGdrGPEY3li2IpDENrnerQV9cYIjcf23ls1uVRuTLGPLouRWPrt2ViPk8Ti/Xr 3/eH/6kNvu1is8McGvXlsMdvqEOa5rAeWsdjhyfdpUap1nj0vBxfPEgb/dO7d8OI66/qejf0vVer 8f2bNmET6v/xk91tGT02aj9+IXvjmm9o2c43v7QeLXX8a1D/+DzYGNqrFlvVTKPUKY7N0fLQvRrG /CQrvB5nII2PQ2Ndt9DgrJ3tT7fQjgeQh7P977BE27xv8UqdsZH9+FeLMw2DAdtcfxwe1uMHx2Ll 1yXG66NTp/8FVqH3YqEqDIsLUPsKD5/xRrLqqzJavVZv3mn1fma0LRBn7FEEmwnUqNzj/IXn508E aGDstHIyVsD8nFWpfiUMj7wVspwscX4K2eLZ67lVi/XhbrNaqxdrnEVotpu7+nD4pq729el2XNHk clpMfFUgOXfWK59cZ6KKoYMPwb77foa5y1GVf1qZpDvsfQjhwRRdmKYoByJTfHkxL6eaG9uahLmo l4eh47p23AgwudnsWhjHSRdHw5yMEqfUshPVx4ydRjf5dHQ5M9OWH3VRTlnQwoJw9KUX3AdY2pW+ asF0+hETri8Y1cHfYvh2npu58BOal4sSFN62015in3DiDqvAgVowhcjktfxg7xaPbR3WbZXXeDXK 5pW12LJpiBifQGgPQ0dMM8iccTLiE9J3vX1wgtYGOUMx3+MnFsRg3p89OYtg6nKJlz+enC172ywv rtrl5+/MmGRGG9Z10f/EhklZQHLZyrunZiZyYedyMhPkqCcDWnZyMvDoFiIvIMD74AYfqTzEKf4O ZPQ4nNZIVLfqt98u1FP16+2n9UFAQExZ3yLkWJzmS+mwjSDZC5ASfm6ET9l/iXO7VmLHxwe7P7j2 4Dc2/dRrTI+jh9PISz/ufugnsmMYyYY7GGGvRErdmtz1jbr6VB++HTeyOtOsrhGvU1PxwAjR+HBK dd11cx86hgxzubtdHz5gxa4GXslya/V28CiJj1eXJ6hP0EUj2z8y30u8umf+ceqSlXjhtZbo9+PS YzrxuM0Ci2ou2tgsJOpfPtx1RFKYGnbdabHwk423xtZwlR7jnsYU92hMuWltSZbFPWvXbggpDrv/ YPOrfw8rXAiCILi5q349ebm4/Pr4qJPHKTp7iRs/W2/w8m6I+lE/vyfNhdP1wrUbL4kRkC0uFoBK 9l9mNZcoaHRzNU2upoyRAfOzoOeCPCBhS7Scot5Y85TQM3ksvmmcel0m5Jr1+821elZCYG6GHyks UPU82CkHMxKAe0R2hEMUMQhCXmrC053CeZfSy/kse+mDGULx93ks3lPhx1TbFkd8PW+lvLGYaWr+ 0RrEC+wXfLVvFpdVc21x1dq5l+QhVZUo9f910Xfvy89JrVGqh3EdQeJ0j8nGspVlUViIxQnhvQge cQJBjhhUuLObjxwnC4yIO7aEgWH7vTuFUg5YDnxxOYWR5JOQrJN1P5CMTDUwvL/dD+lpL8Eo3Ech 3cOAfC6/+PwM+O4h+D4KVMP6UBRavj5kJ/hHjFaYx+L+XhrFDPuUw+kM6wJNXWAJjjJdXyaYm/1u t17dDqVJb7I7uzd/d52oF3GYRkJirFeXm5np6+HW9hYLtrovW+CyJJub6fjHN0bdJ4lx2zKi9/f6 hycYtHwrr1Qbpza3kwwLsTQLb6cDPNx0n+oh9ZLSRqJOy1M/5hisXri6lcRa9kJaSDpyJdgf1OXT f7exeNA32TSHMys/t0i6Hp6OIJDgNLJniLkp5IeZLZ9wfLEVA6nawRVkN5W9f96X4iVlIXxlLYkM WP3+RO1OMgwcDmrsSxmjO0SL+fTqB8ihLv28jIM4L/uOq5jbeblcbZns2MysVFhMZmObghmRPfu6 5GkmezYzfV5DLRTIvCBrMi+UZIHMy5QIfF5OHfs4Sdd5OUp5zGTDBg/Fh4nMTPsSQojM+BeXZDNz lH+EAa4y/o2h/BtD+Uf2ZvwbzfjXmfIPmfEvQZrMDDLjnzGoE+VfJ8p/vYcymdoOjH+ojH/tKf/a U/61o/xDZqYt4/8/j97dar5LhQIpFuXicp/v0lBG1Qpx3fZyo7zbrCTruaGtXqAcHMXt5u5UO7S2 +TZkuSTn/WOWc+Yky13tD5LffDNlNx9cIShk2dSbqR1dKbxR6gr5MZWK14fCam2hbMa9yE7tLdrF RWtTqmQ7jjM0qhF8WJs2iant1HRC3DCIKW4xWqit1WycQW1vZ+P8anucex1oahUrq9lxOJ1exPIP 1QjKYIkLciDlaMxQxT0rtcWXFhEtN1+2LdKrRW1QyzFTysHbNpS+Iu1bPd3kcNGUU5Cf8RB8rsNK lSFl/AZn58p1BPeb5r7W0FLa9U52KfT16gNPAq+C0XhN+nN8o9YjgPBMPZID8/AcmINnz/wbKnHv 7Ih3QyTOnS3z7WyZa2fDPBsqMauZX+ORpDV0k6yGbpLUUAVTkaS0PrGMBpUkNBTqJJ9BJQMHls36 wJJZ71kug0pSGVnrY2EXE8FY6f6KYW0Jg3gy4eTmGnBvVS/W5cIlzgw4633PwjvLdQ+VgYcnl+Jc Ivh4Z8vjeDLy/Xi4QmDEZ1++bMvtLdbRsoQLTK0ZOvd4dYgX5UI3OKyz9dhi8YK6UFeiaVmos/Xc iOyJWk+OyJ59Xc+OyJ7NjI3sSpQismZf2xKniKzZvEzHPjYlVJ2VdYlVRDZkcKSsSGZmavY8p/Yl XhGZ7TjuF4nNLJWQdVaOHVcdm9mQb8/LjH9kY8Y/ZDa2o/xDZvyj5GL8Q2b8M4xw92D8Q2b8o2xk /ENm/OPyQT6GyiaOuwfjHzLjH/cLxj9kZjpS/iEz/nG/YPzrQPnH/YKrjH9cLxj/kBn/uH4w/iGz sQ3lHzLjX2vKP2TGv9ZnNZsz419kwr9FDUz4F5nwb3Mi/ItK+Lc5Mv5FJvxblN+Ef5GZac/4F5nw b1GCE/5FJvzbbAn/ohL+Lapwwr/IhH8rRR+zrQn/qN4Y/yIT/i2KccK/yIR/y2aNUp3xD5nxj2Kd 8Q+Z8Y8ymX0cKP8o2Bn/kBn/codlM3OM/95S/iEz/ntD+YfM+O814x8q4z9lyj9kxn/qKf+Q2diJ 8g+Z8Z8i5R8y459tdwqUf8iM/+Qp/5AZ/8kx/qEy/pOl/ENm/CdD+YfMTGvKP2TGf8yUf8iM/9gz /qEy/mOi/ENm/MdI+YfMxg6Uf8iM/+gp/5AZ/4yT6Cj/kBn/6Gf8o5/w/3w5mxe/1virT3JthTln 5BiXNzNdtONqhouEsUEtV7O5PHo8fp29aa5aFEnNocWlMTf72xZ3sNDsVy0uRPhtcR3MzbaVW0qj FuXdde083NV3NlUcvli375avZrjyGPA9x2/MavmimNROjIvt/3NebbttG0EU7aO+Yh9XQaVweWff UqdFDSRAUAkoiuRFtihbqSwqEWXDH5AvaD+458zsUjdbkQsDFsmdnbN75l4mim3+4KYEmwacK+2X TXhat0YUvf4tNs6MZz105Q4KIiqLRA9V2B/648+9AXZwcYCAzWAKiNgfufB6PPbb0fyWiIloe5Yi HKYKPAwqIYIHyMjEwIGQ62bRh/6Cly/l8ojcTuq+P0ArBgpE5FrflALYqkAMk4IyjwOuywIJmHE8 CRdNfxBT9xJacVA7g55omNs5YGIyfud/J16gDQL9yG43E/XXcc+Zz6aXJNrPpwl95a4XZ/lQXxEj i96ol0AhM1DCAfIO8prEVX7RLacoKnxNNCGF5dkraKzEcUWCAFqTA0JYFv181S4oLENBhHPKIR9M z0XSDzBzpfjPoyRogenkIzo5nLuic1fq6w5eA2k06EUeOXq52DRNOseAPX8HWSTtry0x/EMkxHFG XKIRKGfKSqTOAO+XV08AxmkkcFmscLtBFcUxYe344gN9bhfIFS6gxAkD+DSKQ/eAisMm2h0CyQUj cVp72QFlYpYAEdECHcSTxNENsEFmlx3mGARQ/O7dxeEV4lwafrUMckOUdpZ5mimkYeBz6ovLJ8jK 5Qof7XzUd85e9HM7gp3sZWekuApXwn932g3gtBX54hBIIMQXzif6379hJiuA8Jz1kQvPNn12lEpP WN0DOLahW4gTlBEhKof7GEEzRLMCCZMJp4jjbTZ1pc8kkXPq8n8yhZXMRiVyU3trrm8ny2XNHBsj h8tXc9Usp/rFyy1vzGuz0k8NspvkwBw/zID4MZObG/+lvpmoXOv3Npp1mJ0SVBeTOv3tzhiHM0a+ 3KAjG5ZUUtpKUqnxiX48zjVVJ8j0SbKTqo8vuq6/3s+v/cXMlNk3lczs7GJ+X399xH3kW6M/qG7A a/USPwc8Xxock3HudgC7Q7tQp96YaY1cD26v+b8eUFWiREn5IvRaBPo4v53Wy9YIRwUSswNhtZed kdTETlSP4RWc3cjWemquKPvIF9PehtXJarVQVYo+ab3mhtJLIx+DBIB/wmG5ogfW7wrHr0ax9Mj+ UPfdKgsbhhDroRsvYNqJbPtbze1wx7SS4hZHcHUQ95FFSjqACVXHvLZDqZrc6M2hsm1M7b+23cfJ cmr8c60bW/YThT8J2gshXiqvLB9If1GFm3qtUu36CZXrVbNcBwkz84vNnZC8DupFUYC9hx0rZa8S 9rYioFfY9uSWXjZc6dHDrDdXXvVjOFsdOBrSCzqW0VpI4D6KnmWzfAxyDTyDjI9fadPTVQF65cOc gAUD/Z9V87X913xieCHv2WHuPvW1DaGRsjxPtuFUde4dq6IU6fMbqNg9TW79qxJCH0nxtnUntoth x1vIpNL5RAhAiuT0KRWJ1Mvo19RgPqj6plOfi9HpS9Y0V591e33dGr9Wr8LDGt69Vr0wnSqoze3E r08fwlNNJ/CH8xuQ2uys3ZFobycqEbD9wZX7xK4WdXgkrPjKap+TZk9B41lpFust9SE+tonlo6WD 6s4uRUg0HicJ8VnkbInN9kFoRqTWGvhPRnJqgxbd24WwTxNMTx5WrrV/BKUV7IwuPNjo0ryt91JJ IFdz1OWyDZmDMILSaNXRl1YvY0YaDxoNjIXhjoOywnUOutvGdfWisK7Pnve4XGyzt3BcdIGSpKoD Y4u9kNbZyfMlSEDwvae5CrvhzTHjtPMBoqjPS7Rms5y3h4WiGhYu2p0hjiMpFAq6wTwEybGn3U6C F99tFup1fsn7oPczv8YYCm78fFTmh1Hp01Ruu4ikbN1Fo55g1exrlPRb2XDAT1bFA3woSmKXCklG Vs2Bki509bPfvGpCUHZB3PiCAvumxXPxomGYqqUcLaV5d+XfvTeLS0ryEnPTZeC8sMdeDTRzFuf5 LMTAXvhoyOid+ZbywrxqQeWVyuqlEzscDo+bS98uo7l8QbOcIdCeb5Z9JDzbMR9ga9/ZoSffh3ec UffAz0ErMhw6oKFw4X8MKz2PxukGU0qWocE/uOwJPD8ZxuCxPH80xDSKcWt/MDw5E8qoBpD0RZNa mp8/pKny82c0bMvykzOaH5U8NefNShlMFFzt5LikvLsyfhHvUfU/eCfIy3iP0rN598rP5x2TWJZ/ l/VAzHmsI6Wexfp/AwBNux8cCmVuZHN0cmVhbQ1lbmRvYmoNNyAwIG9iag08PCANL1R5cGUgL1Bh Z2UgDS9QYXJlbnQgMzIgMCBSIA0vUmVzb3VyY2VzIDggMCBSIA0vQ29udGVudHMgOSAwIFIgDS9S b3RhdGUgLTkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIg NzkyIF0gDT4+IA1lbmRvYmoNOCAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9G b250IDw8IC9UVDIgNDEgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgNDIgMCBSID4+IA0vQ29s b3JTcGFjZSA8PCAvQ3M1IDM4IDAgUiA+PiANPj4gDWVuZG9iag05IDAgb2JqDTw8IC9MZW5ndGgg MjYwNSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiaRX32+jSBJ+91/Rj7CKGbqB Bvaespm9u4x2tNHG0uk0uw/EYMcTB7IBJ5r//qq6qjB2MjbcKFIwTXf9+qq/qvpw1SZq2Srt/trl 7MO/brVatzOtNmoW2SDOVJq4RxKHap6aUD1Xs9Xsl8Xsw2Jh4NBiNYMPWRDGyv0LVRwFcax0FgVZ rBaPszAI3XqICub4lsCxJa7rMFeL19kXL/Tn2gbaS+AZGE992vHC1s/gvzKyIQz9vxafZqHI0yaw mbXwsvjoNCUsOgxTEn1TgIjcW/tz61Xup4qdjLmNgtRqQ2edYXHeG2bp9H+a5we1gbOB9WqwwVNP zw08ExKpveeK39o2gF9hkHpK+TqGlX/Kxmd8h/V+7wtLrF5VUfNaqZay/9FPIQj0X47UnZNuPdXI CYxN5H2DNxPE3j/8GN5Uu+Ovd19ZB78vfQiV10shE+VVNPPykmXeo4eFWIAva36pKIQasI5MNgyh EQByxvYPdtF4pQ9byGKN8fi3D5gmrMB4xZp/VHKgZbDnUagiRDyBg5FRaWgwJyG5nM5QYAsjjTq9 y/X6uVoX3aap1eW2q55r+P1Stf7i6+zXxWyf7pj/V7eK0un2Cr58gh9fVYjiX5UO1Wf15a9QlbM4 cZptmmFa6zzIjTJRYGO8EL/8hDfi+B4kWRArm2dBYtw9kJxFQzVdAYrQ4sq3ENMb5+3QvjiF2xA7 pblVaZBY1AnivqszNUEIdzbU+P5GaYpKv3ibW18b7wryK/Fu/bkB3dff1R1nwRTVNoV8ykJ9oHyo Mj6vUk9UGZ9R6byM3lUIaPb6bKaQtLIT+nJgDOdibk5rO3AwDhPMJ8mdEBOpz53b9xSFqVMEuQMP 4VBG7wauTQha/vvWnQgIwfZ6DCbNmRyNNW61eYycfZwvidPoXd/wvYmNRZEiX0N6n3bDQLyMcyOU UkCJ/8X77Teimqs+SHDjs4Fsbc8IB9RiND1Hn5kKtGHxny9dwchYPpzN0XT3D4GMUL4FNWZPIzoT GtGaa8czcqj2mqem9ecpiKsAZcdkIZalVcMbsERAEgHyBpC5ur0OlLoCC2LHre7Io+PU/bOoS/rA kqESlApXtKwgOcNb0e1akp+IfJZMb9fqseIT/ChI8LrqRbOZbL0SZ/7e8Uq95BUxgfXVZfUkW8qq 7rbfaN+FGFSzxEKJBBcZXmYPQDuAZcSuLR098gkuTY7eCF4ATYgc/xFwkJg1TjjHbedeWBiphaj/ 3t1XB6DsFVyr1U4c7XzdO9nUEifynbcUtbqTgJWV6FnTStFBqDqGVoHOo4CU/PpCsjY+EJlX7grx /Ym2SWSag3g1bEGzVUepsHw4ADp4SwFcpoC1IXgTypTN8EZMKlNCa6xrPK1Z1DiZ1VjNBFazMZSH o7rgeC17n9dYw2hes+m+7pxntl76WGaz+Tlaew95PRX56ATy3wddTwU9+r9A15NBjxD00aVMT4Q8 mgS5ngp5NAnyCEazxHD/Aoif618imHgQO3uiJWXEs3e60Si1kmGgLsFW7Yy+TL/XLI1tRSPMDTto m+IzIEWQJ1k8uW2KaGIVPRHmz+lci2KLTo1sm6LIuG6jb21MdsYPyOFsZNsUGT2haYro67imaQBB T/FjIZhG8YIAq5mAwFiKFwx6Eh6JwUiKdyiMJnhBYRTBCwR6KgTTCFcg0JMhGEe4AoCeCMBIwnUA jKZbAeA83R4NDhHcNDehHgwOpp8bIu6U/HkC/fkVEloKTAoyUmhHoWuDJpOa2Rh6vrl0s9SJpt7K PbENx0XoL1WxXrtuNqLOE/tO1bxUvNbL2pGsLYlx3WLkbZ74vaKv6vpGYQvulaXIdBa19AjolFJs /T2dcidYYq+vKuoWut6iI4E1K6hghMEG2GvKSi2hgb4vXqo3Rnbs7dOB0WIk9eDXVH4WP0E0i729 KJtsxn98pL1ws1OHqZ150Irvww9JApVMbcQEigicfIRhpug2Ta3YmNaPYYq5E2NEGoOkeL0pCJm7 ghcK5+5yYI0qltTVx66rH5gLAcOFSoyBaeW2WqMl9Hl1oFIg7g4cGuZTP63AmbquekMEfjaZxNWD eLJofnDOvZAP6F2z61SzYvkNbyv7pAPUwSi3cwhDn5XFAa4SYBeoTvasyMaCIgcI3oHOIzf30GB0 XKLdvZsybg6DmWrXGyYhBqBwZMrYiqzPVHdCcHKYVodh3zmtm3otVnBUW/b57507VJMDlPWc4KL8 Tpxtg33wb3it6VPt8ee3CbtEQJdyUWBAVQIa3CvUuDuwVr7VnLUs+pvzvHQRWQkMMLfe4cI3TsmK cf7T42hckwzOsmLAUuWB0gvmhBvJkO4CGGZ/WS8GX2jrnz4HN57zugtiT1PBMfdUjBI5fpAORDOM 5nH8WLzcUdr1DeDoYaBbI1cDKayiu4OUcP0RqbfnnGHKERtWB4m/HmYc3mb13g1L6YYNb8NakrFW 5Wa1qvYwOqqmmJ+k6pumq+oBQW8OPJYb+DQI4yDv2OKfCW/1uz9PYUP9hm6A1teb+g2Pt91BFFj+ C/AoW8GsyDF3lTJKpVRypUQexZ7G2yr5tXHPxKsfWkrB4E2zYpIY6+/owcdY7cr2qcEn5NL9venH pK7PGT/9mNSNIT8wAJkwcwPX6AHIwB47ef4xMURHHHON3xnH4hh6Mtd7R2ZE720MIrQfUs45AZ1i YkeOP0ZzkEV4eE66QQNGD0ACwegBiCEYNf9wYPq5YWRgRs4kEppe/MjQjJpKJC6jpxKOy6ihhOOi J8Zl5KggcdET4zJ5WDBOy+GkoPtRISa+AaIvm6p1hRiKiwZLueWHWrJ2LeoFtpuF6zSAZ7El6XhD sVpteBmmAvpRkRDqbRLpEom7E+Ju43Eb7KpGLIMCU7nhbhdrVQRv9QMJbqE8U4XNQcyBErG3puVO 8QI0khX/bNlMMueV5TS7bUlDQskN2t6jY1MhR6ilQTFPorI99hpXORTS861Z61MhcWOhvdvu8Hbf Hqyk3391JU5aHRJTSDdf7fuoV/CEhMEg4tZxImnUcLpzx7jxgNWXt8VTYRM2aJjupWjvXI9xry7/ kBan76WaeksmNLW0ciV1Cd2mLrrjKQAb5c9cyC99AxKka3q/4VAraeJUodbSyVT1dzsU6k+Oa4y2 Zny51sCeuf2xcq0zHQyqNZD9GZ2ZRTr4gWqtgQm0HVRr4MeTvKI1cejEcq2B0PMp9VonBuM4ul5r JNJYFDj+6hW860ccurqRSuROkK82HGap1xCx00GCWqDN+IItKPQFeywKoyq2mN/X1LHmjyqqYrue avuoqiq266m2Ty58GtIlNce1T+d97TN0eX+FmWeJU0dO9J4TU1qkRlyskNnwh6uObUFFDEey0MNi iVzX8o7irnmpLtQrS+ju4QztgLpalpuOPzR1seXqYWQYsm7Q7KtkhPqKTm06PAnTnOJdLKKmzQ+s uayeWFFdVvWSTe4p8H8DADWCeCgKZW5kc3RyZWFtDWVuZG9iag0xMCAwIG9iag08PCANL1R5cGUg L1BhZ2UgDS9QYXJlbnQgMzIgMCBSIA0vUmVzb3VyY2VzIDExIDAgUiANL0NvbnRlbnRzIDEyIDAg UiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAw IDYxMiA3OTIgXSANPj4gDWVuZG9iag0xMSAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0 IF0gDS9Gb250IDw8IC9UVDIgNDEgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgNDIgMCBSID4+ IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDM4IDAgUiA+PiANPj4gDWVuZG9iag0xMiAwIG9iag08PCAv TGVuZ3RoIDk1NSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZRV227bOBB911fM I7lYMyQlUdTuU+p2dxu0gFELKBZpHxSbtpU4ciEpMfob/eLO8OK6SYGgECBxhpxzhnPTxXwsYTWC 8s+4yi7+XSrYjpmCDrLciMJCVfpPWUiYVVrC4LJN9qrJLppGo1GzyXDDClmAf0koclEUoGwubAHN fSaF9HpJBDOSSjRbkV7JGppjds0knykjFCvxKzSDq4eo2HOLb9DpgJT8c3OVyYSntDDWGBSa156p jNBSVgF60SJEzbZ8ZpjzSyg9xszkojJKB1vvWFGfHDPB+uNhuIMObYVhPfrA4MtwwG8ZIBUbXJTG UeBKiooBcFWg5p90cCAZ9aezjxHRHaHto24Nq3T+nlcYhPBOJv3k0Q2DQ7Kg2OTsK0paFOxvXqAE 40PcvbmNHFFecQwVO6EEF5OYmKN6FTF3dMM2eUDCNgouhFBhrnNtz0OoUwLqmNsP8YqarTkeCR4r isd/HHNaRgLN2m1cuGQwxmTPcgk5ZbxEw1xDJTXVJBaX55QpbTJXxMkut9vBbdupO/RwuZ/c0OP6 0Y3wielPnDe32Zsm+1H01AXzJYSiWs5x5woXtyCJ5AhKwnu4/ixhnRXKCgsG3wZULWoNVOaWmuLV H9QVT3tBl7RtjBWl9s2QCpe8VaEPQpiaOTfCsoW/8rl7RW5FpRMphqV4iZQ6EEwlyb9nnBVxXrNu yZVmc6yxki35TGNO3ybqvELRIIKlFsYCwX7WuTAFUS5/wZhX3i1To6fm1PORaYFplsjy//ObyZII 4s085ws3kzWl3xgsdS3Vs8uVlijZ20VMcV4X5HskUOIscr+8Ro0dh+hVCpyIKbpm796FzpifYmR9 kBOyfAnalgRqDF7ApMJVOqK/v/TjzZ7BxxQUlmrwN1KAlVa8mIGErn4XPY/oTzHxeE2I/kUVKAnM YGC0Td7QSI0/AJWHmr/kFNORz0ocNjBymldf3KrbcBosrHNruPnqtWCx5XPKgWHtWgAsvOnAZzjE 2eFm78LePUdHLPsLNkMb1y7gwuYwwLRzOOI0/kjG0zbOxZJBMw+IOIX7ANW71dRxGnY4RY7dfg9T G3buHAGFNZwBBcW+6+8CY9dDG+l+KG8e+vXewzocR+MhxQ9DJWmmxTmqiqc/smnXThEN2XGIVhS0 I/5KLI7KPX3of5nTJdqBYlqx+OnC5mOwcZwSF6HiVh90h2iwdnHxJxx38ciK0/V2EMXRi9AfJjzS Tt9+0q7dGI9FHMzkLmE6HMF+IVI9fhdgAOIo3GgKZW5kc3RyZWFtDWVuZG9iag0xMyAwIG9iag08 PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgMzIgMCBSIA0vUmVzb3VyY2VzIDE0IDAgUiANL0NvbnRl bnRzIDE1IDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9w Qm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0xNCAwIG9iag08PCANL1Byb2NTZXQgWyAv UERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNDEgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEg NDIgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDM4IDAgUiA+PiANPj4gDWVuZG9iag0xNSAw IG9iag08PCAvTGVuZ3RoIDIzMTQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImk V9tu40YSfddX9CM5iDjsK5vYp0TJbjxIgkEkYLFw5kGWaI9nJNEw5Rjz96lbUxxblikvDFgsdled 7qpT3YfvZ51Xq05p+utWk/f/mWt10020ulUTGwoXVeXpx7tSTStTqvtmcj35aTF5v1gYcFpcT2Ag FqVT9K9UzhbOKR1tEZ1abCdlUdL7EgGmZEW1WOF7XXq1eJxcZh8e8qn2Rcw2eSxc9g0sU9hM+R/g qSxCpkwpM8oy/7T4MClTRJgYYghgLH4mLC/By7Li4B+XELTObvJpyBp6VIFiTIMtqqAN+/LS6n5p gb3/295/VbfgC6vYwRoydXff0loopM7uG7G6rqDVVplSuXbw5t9p4j3a8L6f+7dEbB7Vcifv1mqV 5m/zqjDyP7ns9ikXbfLAbFnJlsv+lTvMWfcgo1dfBEPsVQ6pyvoovMRkJmR5vZKYn3GHy7QCNG7E aDiFGqptTRym0KQC1DWn8E/ZosnWOUzhFWvMx6+5DoUXAJMtb+ShSQ6dFHtqS2Wx4h4crVFVaZCV QC/CxGJJ0SNiZrfz2fxCzZuuu213atbuVs3dPl98mfyymBzYjvSfzRVzaT6DkQ/w8EWVGPtR6VL9 ri4/lWo9sdYXOihfVoUNytjCA/nrojbYDz+9w4aYCv9TO1gPiQQXA8uilfaUxXVqTSvmBC1meQBq f6TNDleYYA0QISIs/K8KH06hQiRoV+sKbUr9HLdC3EtIUK5NNgOG+WyeTw1U9OI5fPCFG6BX4TX0 WBWVOY7+DNMeQ6zPRHT6/0WEFAOKDw7j4KmFKQ7x1CYjVtRXGlf2EqB7AiildEEjd4RBJXIZgObH SwkdAA41nqfpIJXyfYTOKWFT/3uJMq52kEiBMUjEE5QpKQMl7f0pYTwhZhcfpXtS/GhTdA1AL+/B YcO4mnbNm2DqX2a//cZnzexpiiCx0R2i63AyRRHWUhbGpcNAG4n/+49wiGBfzY4SG4If2jm4V9s5 eqwDtrN5oZufN7LgHIqOOKeLDjBAPqy6kOtZ1d2xqgtQqjrivFL1qDFpY6su8bnqGP1k1aPDCmLV g3u16hK6rzpHP1X1GPB4wLLb8FLZq2Nlr88uu9NvKXt9btkR5nnZw7Ds9ljZ63PL7vRZZa/PKbvT 55S9PrfsTo8o+9FulyO+As0gJzzMPH3EV3CdKI+TnyUpdeN8xmfwfHCpoHaAa+QR1HNV4ZbwXglq O3GYcLimIrJgAxu0oI68rWSQ7ju8xCAXG7Ad8RSktDZkW6QFesPPZnL97ml4NMGyNW6Owlckw70z uHWeMIRI7Q/jHiEkgCsZ8QgC3BKGdyAbcMRVWCSOOkoohq8onIgfS522IW/cEXg7x/H78gwUV7oC LCQ2jpdZsCNrjhL6pQ59el1ayLBxIxVWRes7JuqeiaunKuCJtDrAjpVWR5DPEFaIp02Pd6oFRFi9 GY97zgQ618bJqgopY6BL5fQ4Q1bZECSRo2QV7MhW7iBJxusqa6nBRwsr2LZ1tJ6RwspC04c4VltB 11hvDgLxNW0lmx+vrWywuN/ztVXfwuOkFSTozBZOOH3lR2kraLGzKy9AfeXHiSsbRldeAPrKj9FX dRhbeYneV36svjpd+hP66ozSo+6R0lfhXH11RukR5y2lr88uPQqsE6WPRwXW+NKjxhpf+vrc0ovG ekPXy3nv6tEKCy4HFAWlfqPCQtXUX6FbUjR4t1WBJQgoFFCKQI9IKnybBI2pyAElkEdugoPoIfwm AO96ILEwFz0A2Cjnq4ivk8gCVhnAqY8jcNBIOd8MI/ikslg4ycWMIRymZ7gL+CZAllce03oEBDwq BmETheZwH5gngMevERNFbeJVwItDCNSXhk4VWkOaADFIrAKl0+Bm6D3I0zC+2IP4uKZB/DQhxYeZ h/gD7/Bdig7xJWWD+KhyhwAyowdg9XnAOERIGMaRHDaVYBg9sK2OqDE9TIoDS6MF1GfbGVHrvU1a 2mpaRC+n7MBC3+9H2VMOxWQZtHocdBnAYEQ8QGm9fPLKZthYcaND39XICPpXEt8A0Guypc9LTR+c +FTWavGIJ28+9QUqryksAg5gCGIzdQdqDJTYPZwGMWvZ+DufAi+z2xz622brHD9YGx7qeEj9odhD Xm94qngsrzaNuvrGU/c5fqbCRIyiuvQSFlkjKvRt1iy3Ke7+s7xraf4Dh78h4zOdHLB7iFeCWlr8 LFt1/VYr3ur+c6N26BoI12f7R1gtZJJ3GBi4yr4iqsvU9fJK3tAOdLZKh5SIXvkY2oJtBjaeevQh ZYqDYSLWYTURG7lE3O5t4pLxFVIO2QIN793BAM/vxsSNL4ve0kiYHsRCkTcHQ/S6SccRnjm8EbGO M8m4kDrF9zdGaWxKL3IK0ztvt5BO6FPILkxxxCJ9MCGX0JTZFBkCb7u9vIA6i19yuJcpLZsr5k/T 8W93iyTV2e5GrZY7ddXwLLVud8AvNr7xVLXcUckW716iv3BzniP+bM70v1DIB53RurJ2/QDE/Stj BvK7RpjZLW8add0Ou2UL1NLZcr+X0LsbHv0rp+GCOS4I6g9ioIOtDtdpeu5q4S7lAKdNkbhbzljM HnaSjNVyL0/tDlMATbNMk1pcAr3qOAr0GSZpufrKtrxWt+lh2cnTg+BJpIbNNYcr+ju7tig/PHzY 4TWq6/Jg6kgfhT5EvEF7y/HpKLaD86zGI6+3SQJouL8NM95gJDOw0Pf7UfbUrE3EKmv86XHwSnXx gGPw5oXZvGBsBjTswTjeEbrS+OADqa6+I/qaWa7Zn8RYzrZl+sthCJzeK3lC6lk8f4EHUawL1bFv l7vk0e7Uhp+WwB1iONdDp9DqTh7aVfKGf7e5JRaqfYvdYZKT2jXNmuy1ancbiqce7tq+Y+DApHgu W+46ZoDnxdTMGA0/Fgx6hzRT1LB1do07BhrNeVc1yDu1Yg+ZkcKl3+VuLR2BTwJLCzXZqhGAO7b3 qr1G1QgtT/0a+fKKiCLr2vMClgwm1oMM9rS9hEqixv5UqjXoAVIQeIRHJLFBmRaTvUnjeFJC6ftx sTe9vgDSkxYF1YtUE7sflwO3HxcbNc4/AwB+sIENCmVuZHN0cmVhbQ1lbmRvYmoNMTYgMCBvYmoN PDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDMyIDAgUiANL1Jlc291cmNlcyAxNyAwIFIgDS9Db250 ZW50cyAxOCAwIFIgDS9Sb3RhdGUgLTkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3Jv cEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMTcgMCBvYmoNPDwgDS9Qcm9jU2V0IFsg L1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQyIDQxIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1Mx IDQyIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSAzOCAwIFIgPj4gDT4+IA1lbmRvYmoNMTgg MCBvYmoNPDwgL0xlbmd0aCAzOTM0IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJ 1FfbbhtHEn3nV/RjTwCO+z49yMMikbO7NmwnWFHIg50HmqJk2hYpkJQNf0f2H/ybe6r6MqQuNJUE u1gImpkaTnXd61Q9Odl4MdsIzX+b2ejJP061uNyMtFiIkQ2ti6LzfPNOiXFnlFjPRxejHyejJ5OJ AdPkYoQfYquc4IsSzrbOCR1tG52YXI1Uq/i9IgFjojzYZvReq15MPo9eS9WMdWi19Li3RornN/nF xybiKkz5QKnmt8nzkSrnKWHaYAKOecqCbMxHK5NOXpw22oHxpOlbL08bKBDlM/FDervG8V7OmrFt O/mu6XBdbBtt5bwZd1Bk1mjdWrm9SeS6MTAOP5IOGpJjCGK8r4EvCqguafDLFEJ6edmMAzjpUXR8 wDhAatAGloCXPeP66pmQuH9drT+IBXjbIJdwghTX6xXuPh0JE+aZ2mxaPCmYIARZ18u/lw/XRON9 /fZTPnH+WUyX+d25mJXvrxqyNl0Ly3LLpwcpVoWDgmPlF1AGXvm+caDE5ib/+vZ9lpHpWQNHyXpK UrGQRXJ+PctnviMLp0UDIi4zkWIw1oiHNXHXhaYEoM/J9a9sopHnFP6ksSZ//LNBUvkswMjpZX6Y F4ZNzraxVcJSynnKByM6ZagokN0sU5WwKauTzNOT02fi5c3H7eJ6tab0gqliMl1fzrfibIkco3N/ QhVRBZ6cipTQpyeovOd4eC8QRyM+C63ES/H6NyXOR870LZLF9V0bnDCoEyeColcoyVMqSejZt30Q fKFC9PTgHTxqBl0D65rUfLq6Il/AA5TmC8pyCz9n5WpbsJFyXXgbyfhbon/87h7Ztoev0AJCVJqb QClYyJx/SomRxc0aX2sKQi1KyfnYxigsK37QRttzm/HKtz5UGy03GERhviYZTkIinEcSe5Q/ZWLN IJxujRfoO6oztlRx6NIJT5tUtEF+XHxqkJNdJtEKcO6XckboXMdnRGdrKlY1bt5ukgJf8n07v6r2 6o5s9QqZfJTFzlBqeA8JnIGPNjOryGbGaib4yLKuGBozSYb6bCh6Z6/Drq/27PxllfWo6W0NekIQ TnN4TCRkCJ7sfdA6RV+6Dh/VcJp8/rPlliwFHsE2KqnVcjmfbbNqxtsHNTscAasooX0EIkBJQwoc DgHqAD0nwB4fUn6zNN0naZNm7LmtKMnqGomyH4rqvSi17A3X8tXIoQ/hIdMfKx244EArR+A60Mfx u8h1VPkrzUWWiCsQiuza+ZFpbykjQXokw0AdwekcB7qwDmRuI8kKYu/aOBiVyfvFHsMaNEew8Fby dEQddWSiojMCIowm5rmPmwNhNj1/EzouOK60hyOLzOs6re9kILw+yWOPsqnpPjsHvgC4AIaLNFdc 8B0FuBZvZHDpx7foyV6+afYSJ+dqNg1+0JEsGCzPNBoo+wXdvjxnL7iuI+iyIVBlHgMjqAu4wSnO rZLr1EMqjJy+BIrQhHG2n+VZVM+ikKUBFplCfyz0g1leNE7JbpSiqjse+IzrKO/2ge+xmGccp/Sx mGcibDN/BPN0ds7RmGcUxfX/GPIMQLvfsRgOPox5xg+V+Ccw7zF2/lHE05qCczzg6S7u5up/B/Eo WSnFj0Y8a/VOyj0a7ozlCq5wlekKV6bjWquNYKCP49c9l1DlrzTXVyIqaO38mCoc664agGcgj+DV nlKs8g5k7iHJjoJa1axMPiT4GGarOYSFeSBThDNdcWL4PdPQP7oMFJU4sm8/GK6Ct8gWnOdUTwWO uojucJkbF+hX51kP6qAENHlEnUwxo3Zy86HUdi3M3f5TcvIlL5CWk3JJxY1+f4lSSusclkASCCMM 9WeIV5bo3us4IDfjhXxx9qqZvKdUNoAV1KiNmpGTR9px8hkMuhigIxtuk8nV8APQkVHGolUot4cd O/LvHQAMppO4M+tk2uFAziLjOeKVPpLfepdyIfNX+nZimc4OWUXEtyTvcCZJA/NhyftTXHHu0WOc xUphwyOa15+Y6PQREx20jOQYvkBLY2lURgnpkFNf6yrIZ0FfcRLgUraYH6zsBsJIBwu+bwx8LMXX AFM0qtLvcRgZa1uGF1FZjlaYWpusygNe1L3jvIoxZ2jyYpoCc7WQhi/OxCseBrW8SaDK0xbub3P5 EWOPbn0Llq2t1rp01Bv5gpVGAUAVchz5MhtLV/iThqmdETlZhTTkcWG/8A5YZXVPDXt/Usyz7aHR DR8HZe40IcwcwywWMJXgBPhZPtR/uraLwO2h/7A/usEf2SFogR6KoAX28Kc4bSL0oDcD2mJWhyEG ABTcoRaVPaVRG/qOp+5tUTr2Lc3UPK3tT7ff0OuOvWPbelgscHfODx1Xu+p3ebbcTi8v5+fU+3IB 3ppX7wcGcgKNVDR89WQ+nIB2MU5AzU6AO4IVu1dyCGCXUMqaDFZ9GMDqfofAyZbEoMxuT/w75vxh S+5JEyQZpprWhb6/g3ilkUFUwrvznKLIUPdo3wGz3Y7rSv4ccp323L+r6w7lUnKd7pFS8V7X/bVG 7UwjGU4AP5iAgtuBm0S7YKgwQCuGjYE+jt8q7j2Vf6CP4zeGK7LyV7r0tkRfjXTHK+nwe6aDTXBL +ySFo9LH8evIUaz8lS7tcw+0PC8veZgs7dPpAgp9TI3r5yUwwBMcitVaXKE/rPKL9VxgwNlUwO01 ZUqIIeSoJhBMPeFrxzWEOqBpFWVguloGriCRpnzaEWqLULBlsZbEbvGogPvgdkPJQQfkfskoHweo z6duCMpxJiWmkdtMimtuffntutHwixQJeV+kl2eNJvR/RbcAONb4QPwMrTwG1MRcVQAsDYb5alge Aq7ZCE/SFK2r4vcF9tZOLvP7RLF9kLtIU8200Kv1v0Uuoo6LkauOVlRF1C1/st/fyL8BY9MQqsRz QXPY61TsNK38psR5Gcs8FsWYpzLf5ZZ7YCrz0VNW3mkAdY2d5+1106AxkzF9DiNMWZZ5Arnd3ZnT TJrff1qeX68Wy20Z4pOeDtlOI0JRND68lGRFabK6Mz3+sN1OZzQX9PJdalDz86IU9p5wu0WFYhZl CE0RUW42iXGxArHE/xuJMRHvd8Ya+Jx62Gv2tKUCIE+jAnizM/n5Y3rGXsgNh1a6gfgGk1LUfDJT IU6/Fe28thzvRRM7Orq48VCU+78gyhqDYIyP0c+Y+5aE/2GYDcBVlZBlIsXMhD51ZorZQHyTTZm0 bzFbIe7r7rrnYHmUSkVpaopdNm47Rd8Kqdu53AqxfFysG3I4hY9uVzSkRykUf9xSz9LofNwHHT6i t1sxvdN/095jbi8Fv/6aOu6rltYKHERNT74E4vfUzDX1ui+CN5AeK8ednroPKa9Je8/NlxkATdQr E3F9ne6IdV6h6mGx935Pyc8c7cDBTkzia3qgvaxNe5nGo9pdx7QgGJgu07tzwSQdY+UvfD1pNO0P Y1IySHOvNar6iF/ymhgb3sDaxkEjNxAeBL5EOC6YWiUKEaO2mnw5FeeLBtsoUHFGsevkzSbd843w hVj51GU+L3s8v71gTSffccJ0w2an85b4lIKoyVdU4Z8W5B8vZ5kWz87Tw3yZ7tsFrxMX5A3CNyLy t4ifkWV+8IL8hKZXCrGPg2+S5CmQt5cf+IpnR+sKheIkuZgILZ/R+6dNQVnWMe2aZPt/GK+C3baR GHrfr9BRBqrAo5E10qmHAAUMFEEAb5HLXryxHBtwrNR2UvQ32v3g5XvkSM7aGSwCxObMmMMhH8lH lz+qkECXAsMa9pw3GIo6i/ZpMqO+Vv6vFR8qGIAOapBwl4lHm4c9wEInIBJjGjqi5UBKobvOYCIt GhJIfNvCtcjOQOoB/65xKTR6uralZ1GEBcY8WHHtJuhHGWsWmsMfKLMSWV816OWebaKubQAYqm3W sqbM0JC8fIxtwCiOQUMsa+RZb+gC8rhCkRAZYtmw2NZjjL2Wo/mKI8IsIlIw41Bg19tH0qtlXO7j MSVBvqnDhcoxhlPDzcsSP6vzpy77K2/0u9+cF+//P2P4xr2bMQZ5Yf70nDnMn6X70Jm+xeBVTWsU 8ejN6RDycqa2/xYKiqz8dpc9PNz9887kGDs1QcaCmefdo0kuwx/mvFqvk9QNGEvFpEJM9LXOhtiu qsR2g1ab2pYLE9uJvYCZNbU9TdlVY4ZJbU9Td0uSJnddyrKKI2di26WU+5ukV6QbJ3ZL5GFqu0xZ 5rCe2i5Tlkkifrh7hYeIMcNl55PCyBCIe0P74l5quZOeWUrWSc+U8tWgjIFfNirdoNqhMarkL+eh wiM0rhEbgquroTBUQ25ZbX8g89hM0A+WoIjCZrQ6ZNsjmJGUUFvY6PZ/l+1XB+GETxNtK6bmSneZ VoMFzgr6djVp9FdSBkAtRUDxg471tqN84OuzdW/SWXcea50rxwEzkE4FORpYO0O+23X4ny1uRUeb L+bZUo/tZXn1TmMzOEn1ybXQciBvyL7cIjKz/P7z+MDKC16G0bMdJt5IF/qOs23LAQGu1ceiW1f5 7QLzwkzaLCbckGevh92F7/jAaYiaawPLsyguHQlgge56IoUUK08b25CrJEyVcQx5YmSjI4XsvpM4 vSq33WrDPExgS/eseqXr48iJTODKw6kzDJHw1nXm6+zPpbitZgS9AKSoGeuGAMHzjLXNV1zr9MBe DxAFraGgtrhfkGwf/T1tg0EKT6jlyWAD2VIoUoHfS14h8qS5c/hcNv9+NeZ3vE5OR+5XWTCXJELC h8g6hRAddUI4fhJORteTmm70lNIl4YQ/hP7ExZ9XYmFsbzA/0Hzp879UOsHgBtAh4zFpmb0cejrw RaNHoDF4tUXrJymI2AVyrmfXSbifNoQ707gEvyQlNlGT4A3cRNjfo3525/rKdkxIF+lyQ4VA0m77 FqfKmtRdmh9srBCN+x6Rcly1QuQMbtKKOLUIs5crRtAN1kemM+ek6pkCuHYbZRV7IKzMf8AIOqXS yQuL+6slSysMn2YQ2C7A/yVzRWMjgCrg33nGwggMI3ZFxaTjinC2YMpj0Ntot5uZ0iMyFhSPg2eL dKS4W9mXbL21UJ5YTWKWT8mqz5S7Mpyl97PoE6ehyBUNj6q4U78Wws6rKoCWCpqbmft4DtDJbqMu 61/1c8fEtuGAKDeSH11rW/3L+dyQ6YfpuhxEitgh1PHBoCm1F269ZwnP9lbalhMQ4ucJq6f4CKPZ /umdu4c26yJMHnuWSgn6G9UErXE1k6vKWaHbfDiEGHuMnQWbX6/yIfv67e74OTLSfwcAdZA8Pgpl bmRzdHJlYW0NZW5kb2JqDTE5IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCAzMiAwIFIg DS9SZXNvdXJjZXMgMjAgMCBSIA0vQ29udGVudHMgMjEgMCBSIA0vUm90YXRlIC05MCANL01lZGlh Qm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2Jq DTIwIDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL0YyIDI1IDAg UiAvVFQyIDQxIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDQyIDAgUiA+PiANL0NvbG9yU3Bh Y2UgPDwgL0NzNSAzOCAwIFIgPj4gDT4+IA1lbmRvYmoNMjEgMCBvYmoNPDwgL0xlbmd0aCAxNjAx IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJrFdbr9NGEH7om3/FPK6r2mdvXtvt Q0UPpQ0CikiOeABU5SQ+wXCSIMeHI/4Gf4K/2ZnZSwykXCQUKd7Z2Z3duX0ze3Z+qGB1AMW/wyo7 +2uuYHPIFPSQGVfaBuqKP5WVUNRawtBlV9kfi+xssdC4aXGVIaMppQX+k2BNaS2oxpSNhcU2k6Xk eUkHFEw1sFjRvJIOFrfZM6HzQlVlI5ocF7QC7t+EiR1oGYZS5i8W9zMZRSldusY5JBZ3+ZAqSJWy 9lIfL/MGpW3ywomOh9CwjMKZsnZK+73+Tu2nd3q6H15Dj3tLJ3Z4BwFvhj3fhUUqMXSBOhxKHMmy FgC5sjhzLy4ciMb5tPZtkNjdwnIX5tawiuu3eV3q8B+37EaW7gTs445r1MaId0jp0orfcosUHG4C 9/JVOCPQqxxNJZIUf8VIxpPD9CrIfEkaLuMNiNgEovMmVOhmo5upCXV0QNt6Ez4JKmqxJsf6Gyuy x9+5cmUVDtBiuQmDLm44BGcXRoIhj1e40WiopaZwxLjiMzGywpma3Sbm5/MZzB9ewGI5bLoRZutu N/ZX/Wo59vsd7K+QcXh9gHzxiqRrVbYO/H86Q1UYb41UMXorH69ep+dinrtSiTs5LnbiYV5oVKjI K1RHoyOceJ7Hu2OEcZBZh5ZuwNq21E2UeoxXFeKVboYXRvE1Wpn+0RoGzUYK+IkhiS5Vmyzv9OSC HyxFY4M+bHFAX82b0L3oIZSnddpYJZfVNtwhL3AdOQajfNOt6QtpUqFfCnLPa/+BGR5VI4usnBc0 6q/Ct/N3Ldh7IUVVUln74y52GOpKjEuO+M0mBD7NrXlMZzcUITzH9vHsHbO9ZYgecJ12GN7/PAE+ +E/EKMK38zl4zJifI67dR8x6BZirGm5BSXgIz15IWGcWo15DJdHzsA2UlY7cdh1I03CYRK6uXakt caumVC56l9htiYataknLE9/UlgAy8TVmALHnhKaTYDEGMwXF6aZ0CUBVcpQK2LbPycoDRpzmXPa+ 8GazbELDNrLJRnYaPfprwYPgMI2exc8hYpILa+O3XL6jWFDsKUVARGEybGgzgsXIu9FVFWDipkNV hFtpg0KzXT/2y5FlKEQlkjRQNqBwTAdO4m6qgPqaBnVQQPl4OLsX6pUpMfV8SNI+8RNhwaSclVV1 jFjDuf9MOJsrKS5DSKPjPs7AbzNL+ZlhmhLXTw1jE6C5UIceXJBnCc/XPhU41YimVONv9z2O9SUQ 96G+fjU7hNEIIWgTkPGEhU9hyw83LiJli+3C5J/aBjyzVlBwPiabxzZCNgm7uKS0aBMCf7oqp0UT 8AnJHmHUiqvJuAsbTiLrsaY5G/H/bveW8FCLfuW/HZ3gnYRIiG5uvY8U0X3e8HFxzIuHSY34jkjW P9zYsZnC2tJiHFZH6x7jUKc4fHSzZYy57Aag9gGN8eDi0ffp0nyig6Z69C06fBSIqESRtECktwjc KmLqNjNWE2laQ/h7kqfbmmD2f9nSMjob7HNMJLeZtjV1t7pV1N2eYGMogXKV5+rWUgmwuuaqolus O5HUTU3lwuqKikOiFEX4Kgs01QwWFEksKXga9dZgXENzuqYjA0Ebp7ywDe9DBSdSmk5MZ2hnqRpe H2lFhgmX1Vi8iGBFPLH6rGRh3aVaZbWhVb5k2RrTF41KSZw6/5Sy6ksp+57zkjsNzKvw7XJ5nAyf ntojbGF3/3KS3w2dqZo0HCY9CkJPeud67FLp9CUSbzD2bzse/RpeIDPP4Jq056Ff/z43OD58AW8J jwJWzHzaPib9sIVZr7FJwYDR4hfPgDf7ODNGiS2ZsPX9qMZWgOyK5q2CXWtmHN9S+thShUOfcmOv qINHB1DL3VAZ6g9hsPAVdenZg0ctapVzbGuU8A0zrSXc4ldNoLq446g7P+ViSU+WDhXwzXIYYZ8r Q/hH3T+ML3tqDmu0XyV+B5iN74807PbjR63GxHehFO46bEgV9dvdGm7pWoJF4jvRT+MJHYzLgVR0 +GKR/Api1kh3cKL8tDc3dUMPCmfRDDb1/KejN3XsdR3b1/WenGvo3UQvgJucDLb18F/7i155HffM Gvh9hdqOfvZluOOJeLJuAqDbvQ8ZFv1F0FOaXi/cWP43AJz4b74KZW5kc3RyZWFtDWVuZG9iag0y MiAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgMzIgMCBSIA0vUmVzb3VyY2VzIDIzIDAg UiANL0NvbnRlbnRzIDI0IDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzky IF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0yMyAwIG9iag08PCANL1By b2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9GMiAyNSAwIFIgL1RUMiA0MSAwIFIgPj4g DS9FeHRHU3RhdGUgPDwgL0dTMSA0MiAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgMzggMCBS ID4+IA0+PiANZW5kb2JqDTI0IDAgb2JqDTw8IC9MZW5ndGggMTQzNiAvRmlsdGVyIC9GbGF0ZURl Y29kZSA+PiANc3RyZWFtDQpIiaxX227bRhB96Bu/Yh6XRbneG5dk+xCkTi8KmqCoVOQhzoMs0QoT SzJIJm5+wz/R3+zM7C4pOw7aAIUBcWcvM7tnzlx8dj6UsBlA89+wyc5+WWrYDZmGDjLrpauhKvlT OgVFZRT0bXaV/bjKzlYrg4dWVxku1FI54B8FzkrnQNdW1g5W+0xJxfOKDBQs1bDa0LxWHla32Wth 8kKXshZ1jhsaAc8/xIkDGBWHSuVvVs8zlVRpI33tPQqrZ2ykjFqVqoLW39d5jdp2eeFFy0NoWEfh ray8NuFsuFPz8E6vjv176PCs9OKAdxBw0x/5LqxSi76N0jBIHClZCYBcO5z5OW3sScb5ae/HqLG9 hfUhzm1hk/bv80qa+JuOHEbW7gUc04lrfI0Vn1Ay0okfcocSDB/i6uW7aCPKmxyhEpOWcMUkJstx ehN1vqUXrtMNSNhFoQ0QanSzNfUphCY5oGkChH/EJxqxJceGG2vC49dce1lGA0asd3HQpgNDdHZh FVjyeIkHrYFKGaIj8optzk437HSxPF8uYPniT1gcurFbj8ceFtv2MHZX3WY9dscDHK9gtR7eD5Cv 3pEBXSNZZwPaOWmc0rMJxyZY+4VYPn1RmIs8ncX9xCKHpPcObFNJNV/OT4BQJBAgq7woEVx6thK7 XbvlAUzTCt+NrjPiffjA07xoGD4lwt4+LyzhFPdNMGlpOA6Cnb8dMbJGnQ0O6Gt54+rbECCJ7Erb cOASGafRPRx9ATqeGY/86fm0kUZDgSyxs89d0uRj2PyG0Af6bckyuXQMcncVv20/31o3k64I18MX 1Ly5kRiXaTvfnD0vVkid6IuAwImqx3DQwOrOfo7Jy0qkkIrQob5vSNtJbpNlOWcYWwe13uXoisvw CukbTG8nv5Tm0KavKC8iK9Klynqiw8wG4kKDEdHvWorIWhB6tdgyaih2OaZfcXUybuOBB55/EII+ Mu5CPGs/5ug0I7pN+LZkIDjJoZM0xTRbI7nLa7aWxry5R8LP1vS/eayKWJv/HeuU+720gHlgvslM QzPR8OWHPaUbcdn2wLFzgbMvv+4p9YMnYAiY//SEezTENxTTI37C2kl193wJ4T3Lc6y3z1HBOyRQ ZeAWtIIX8PqNgm3mMBsbsLaWpYE91uSKqivJDq6/sGqskpiZrzNaqHzcvc+Ma6RFscFy+uVlU5XE 2utsSWX+JMnZEhdqcI0j7bGy64lxOhZdyhgW6UlUG7luLtgNW5wn9tLqgceUlFsefVVCwLIV+fVo CATIL8QiZEgsKJWk2MFdtRjDZJTWUTz2d5hPnXgS9kaKoKMioK7kQN7fx/tzhEyDTYbBSsK4BoRc hV412lNymDqgKRXoz1KBn1PBHcf7QMUBA3bZxlH8dFQJuH433KFQfQWKFU4gzx5F1EyJv4zu6vY3 1x3VIU9l+fITDO0wYJl8DNp7/thE6EKeIs/+FWbGCTzjiE+lsgE8axTxLsoWDyFoJdZzrLqTpGht k0XZVSognWRfU1dpsUVAzQ69w5pMGtLB+0vxmD8x4ry7b8SkeEiy493hus5YiC+h4eZLYeGRFGau /baakI7N0NOA8TVRryQihmbKoNAf1sQ8n2a7j+33YUStAXKuQW5QV4bUoI3T8bvhoZvZtibbRar0 ZHyBvTBGNwZh+jWif/0d3PDoSN0ZlRRPJZ+bgDdJcSMbD/yD/LYNIVd6TTLzu+KFubc3esoHsQK9 4kZTU0eJgUAtIKIsoBvigJufishPy30oSxgCyGWD66F7o71UmLjLjlKbTsxU5X8tIluni6jY49ys +xGOubZU4DhaxrddiPghL8UTgMV4R3LFMhyOY2qaitQrRY2x1zm0GDl4y0q0W7ilawlWif+3hGm0 0MK47umJHjtoxV05L410By9kipZ/BgCjxO+dCmVuZHN0cmVhbQ1lbmRvYmoNMjUgMCBvYmoNPDwg DS9UeXBlIC9Gb250IA0vU3VidHlwZSAvVHlwZTEgDS9FbmNvZGluZyAzMCAwIFIgDS9CYXNlRm9u dCAvU3ltYm9sIA0vVG9Vbmljb2RlIDMxIDAgUiANPj4gDWVuZG9iag0yNiAwIG9iag08PCANL1R5 cGUgL0ZvbnQgDS9TdWJ0eXBlIC9UcnVlVHlwZSANL0ZpcnN0Q2hhciAzMiANL0xhc3RDaGFyIDE2 NyANL1dpZHRocyBbIDI3OCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMjc4IDMzMyAyNzggMjc4IDU1 NiAwIDU1NiA1NTYgMCA1NTYgNTU2IA01NTYgNTU2IDAgMCAwIDAgMCAwIDAgMCA2NjcgMCA3MjIg MCAwIDAgMCAwIDI3OCAwIDAgMCA4MzMgMCAwIDY2NyANMCAwIDY2NyA2MTEgMCAwIDAgMCAwIDAg Mjc4IDAgMjc4IDAgMCAwIDAgMCA1MDAgNTU2IDU1NiAyNzggMCA1NTYgDTIyMiAwIDAgMCAwIDU1 NiA1NTYgMCAwIDMzMyA1MDAgMjc4IDU1NiAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCANMCAwIDAg MCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAw IA0wIDAgMCAwIDAgNTU2IF0gDS9FbmNvZGluZyAvV2luQW5zaUVuY29kaW5nIA0vQmFzZUZvbnQg L0FyaWFsLUl0YWxpY01UIA0vRm9udERlc2NyaXB0b3IgMjggMCBSIA0+PiANZW5kb2JqDTI3IDAg b2JqDTw8IA0vVHlwZSAvRm9udCANL1N1YnR5cGUgL1RydWVUeXBlIA0vRmlyc3RDaGFyIDMyIA0v TGFzdENoYXIgMTIxIA0vV2lkdGhzIFsgMjc4IDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDMzMyAw IDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAzMzMgMCAwIDAgDTAgMCAwIDAgMCA3MjIgMCAwIDAgMCAw IDI3OCAwIDAgMCA4MzMgMCAwIDAgMCAwIDY2NyAwIDAgMCAwIDAgMCANMCAwIDAgMCAwIDAgMCAw IDAgNTU2IDYxMSA1NTYgMCAwIDAgMjc4IDAgMCAyNzggMCA2MTEgNjExIDYxMSAwIA0zODkgNTU2 IDMzMyA2MTEgNTU2IDAgMCA1NTYgXSANL0VuY29kaW5nIC9XaW5BbnNpRW5jb2RpbmcgDS9CYXNl Rm9udCAvQXJpYWwtQm9sZE1UIA0vRm9udERlc2NyaXB0b3IgMjkgMCBSIA0+PiANZW5kb2JqDTI4 IDAgb2JqDTw8IA0vVHlwZSAvRm9udERlc2NyaXB0b3IgDS9Bc2NlbnQgOTA1IA0vQ2FwSGVpZ2h0 IDAgDS9EZXNjZW50IC0yMTEgDS9GbGFncyA5NiANL0ZvbnRCQm94IFsgLTI2MiAtMzI1IDEwODIg MTAyNSBdIA0vRm9udE5hbWUgL0FyaWFsLUl0YWxpY01UIA0vSXRhbGljQW5nbGUgLTE1IA0vU3Rl bVYgMCANPj4gDWVuZG9iag0yOSAwIG9iag08PCANL1R5cGUgL0ZvbnREZXNjcmlwdG9yIA0vQXNj ZW50IDkwNSANL0NhcEhlaWdodCAwIA0vRGVzY2VudCAtMjExIA0vRmxhZ3MgMzIgDS9Gb250QkJv eCBbIC0xNjcgLTM3NiAxMDc1IDEwNDggXSANL0ZvbnROYW1lIC9BcmlhbC1Cb2xkTVQgDS9JdGFs aWNBbmdsZSAwIA0vU3RlbVYgMTMzIA0+PiANZW5kb2JqDTMwIDAgb2JqDTw8IA0vVHlwZSAvRW5j b2RpbmcgDS9EaWZmZXJlbmNlcyBbIDEgL2Fycm93cmlnaHQgL2Fycm93bGVmdCAvbGVzc2VxdWFs IF0gDT4+IA1lbmRvYmoNMzEgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAy MzEgPj4gDXN0cmVhbQ0KSIlUULtuxCAQ7PmKLS9KgcFRpEgWzV3jIndRnKTHsLaQjocwLvz3AZuc lIKRdoYZhqXn/tI7k4B+RK8GTDAZpyMufo0KYcTZOGActFGpTjsqKwPQbB62JaHt3eSh6wj9zOKS 4ganYbOjvz83T0BvUWM0bobTF/v+ycSwhnBHiy5BA0KAxonQ87sMV2kxy9W686w+6DUuQSqM0s0I XcNEhlYAOv1fI+3hGKdjrFcLcPbGRSZ4IfhBNIVoa1jH+euLIDnzz13iy18f7dQaYy6+L2TvXFoa h4+dBR9KqXLIrwADABUOb98KZW5kc3RyZWFtDWVuZG9iag0zMiAwIG9iag08PCANL1R5cGUgL1Bh Z2VzIA0vS2lkcyBbIDM2IDAgUiAxIDAgUiA0IDAgUiA3IDAgUiAxMCAwIFIgMTMgMCBSIDE2IDAg UiAxOSAwIFIgMjIgMCBSIF0gDS9Db3VudCA5IA0+PiANZW5kb2JqDTMzIDAgb2JqDTw8IA0vQ3Jl YXRpb25EYXRlIChEOjIwMDAwNzA1MjIyMDQyKQ0vUHJvZHVjZXIgKEFjcm9iYXQgRGlzdGlsbGVy IDQuMCBmb3IgV2luZG93cykNL01vZERhdGUgKEQ6MjAwMDA3MDUyMjIzMTEtMDcnMDAnKQ0+PiAN ZW5kb2JqDXhyZWYNMCAzNCANMDAwMDAwMDAwMCA2NTUzNSBmDQowMDAwMDAyNTQ2IDAwMDAwIG4N CjAwMDAwMDI2OTkgMDAwMDAgbg0KMDAwMDAwMjg1NiAwMDAwMCBuDQowMDAwMDA4OTc5IDAwMDAw IG4NCjAwMDAwMDkxMzIgMDAwMDAgbg0KMDAwMDAwOTI4OSAwMDAwMCBuDQowMDAwMDEzNzE2IDAw MDAwIG4NCjAwMDAwMTM4NjkgMDAwMDAgbg0KMDAwMDAxNDAwMyAwMDAwMCBuDQowMDAwMDE2Njgy IDAwMDAwIG4NCjAwMDAwMTY4MzggMDAwMDAgbg0KMDAwMDAxNjk3MyAwMDAwMCBuDQowMDAwMDE4 MDAyIDAwMDAwIG4NCjAwMDAwMTgxNTggMDAwMDAgbg0KMDAwMDAxODI5MyAwMDAwMCBuDQowMDAw MDIwNjgyIDAwMDAwIG4NCjAwMDAwMjA4MzggMDAwMDAgbg0KMDAwMDAyMDk3MyAwMDAwMCBuDQow MDAwMDI0OTgyIDAwMDAwIG4NCjAwMDAwMjUxMzggMDAwMDAgbg0KMDAwMDAyNTI4NCAwMDAwMCBu DQowMDAwMDI2OTYwIDAwMDAwIG4NCjAwMDAwMjcxMTYgMDAwMDAgbg0KMDAwMDAyNzI2MiAwMDAw MCBuDQowMDAwMDI4NzczIDAwMDAwIG4NCjAwMDAwMjg4ODMgMDAwMDAgbg0KMDAwMDAyOTQwNyAw MDAwMCBuDQowMDAwMDI5ODA5IDAwMDAwIG4NCjAwMDAwMjk5OTkgMDAwMDAgbg0KMDAwMDAzMDE4 NyAwMDAwMCBuDQowMDAwMDMwMjgyIDAwMDAwIG4NCjAwMDAwMzA1ODcgMDAwMDAgbg0KMDAwMDAz MDcwNiAwMDAwMCBuDQp0cmFpbGVyDTw8DS9TaXplIDM0DS9JRFs8MmQ4N2YxYmJmOTJiMGI1ZjJk NTBjMDQ2MjdjODMyZGU+PDJkODdmMWJiZjkyYjBiNWYyZDUwYzA0NjI3YzgzMmRlPl0NPj4Nc3Rh cnR4cmVmDTE3Mw0lJUVPRg0= ------_=_NextPart_000_01BFE70B.28332440-- From owner-ips@ECE.cmu.edu Thu Jul 06 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09046; 6 Jul 2000 4:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA10110 for ips-outgoing; Thu, 6 Jul 2000 03:03:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA10106 for ; Thu, 6 Jul 2000 03:03:15 -0400 (EDT) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id AAA05754; Thu, 6 Jul 2000 00:02:42 -0700 (PDT) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id AAA15371; Thu, 6 Jul 2000 00:02:42 -0700 (PDT) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Thu, 6 Jul 2000 00:02:36 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C05B31C34@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: ips@ece.cmu.edu, "'julian_satran@il.ibm.com'" Subject: Security Considerations Date: Thu, 6 Jul 2000 00:02:34 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk In keeping with the "Guidelines for Writing RFC Text on Security Considerations" by E. Rescorla and B. Korver, the following is my pass at describing the kind of security threats our protocol could encounter. It does not yet include specific mechanisms that address these threats. Julian, I am sorry for the tardiness. I tried to format this in such a way that you could easily cut and place into your document as you see fit. I hope this is useful. I spoke with Luciano today, and he said he would be sending his contributions to you today as well. It is my belief that he will provide the detailed mechanisms by modifying the document directly. Unfortunately, I will be off-site and will not be able to join the conference call tomorrow. Paul von Stamwitz ------------------------ Security Considerations Historically, native storage systems have not had to consider security due to the fact that their environments offered minimal security risks. That is, these environments consisted of storage devices either directly attached to hosts or connected via a subnet distinctly separate from the communications network. The use of storage protocols, such as SCSI, over IP networks requires that security concerns be addressed. First, a threat model needs to be developed which describes the kind of attacks that can be made on the protocol. It needs to be determined which of these attacks are out of scope and why. Then, mechanisms need to be applied to the protocol to protect against those attacks that are in-scope. Threat Model Attacks fall into three main areas; passive, active, and denial of service. Passive Attacks In general, data transfers will be made through a switched fabric, making sniffing difficult. Also, the nature of the data (block transfers), even if sniffed, would not necessarily be readily understandable to the attacker. That being said, a determined attacker could, by capturing of content and analyzing traffic over time, could replicate enough of a drive to make the captured data meaningful. Certain storage operations which are mostly uni-directional, such as writing to a tape or reading from a CD-ROM, are even more susceptible to passive attacks since the listener will be able to replicate most if not all of the operation. Passive attacks by traffic analysis alone is deemed out of scope since it is unlikely that the listener will be able to guess any pertinent information without knowing the content of the messages. It is also out of scope to detect passive attacks. The protocol must be able to prevent passive attacks by masking the contents of messages through some form of encryption. Finally, it is assumed that a strong authentication mechanism will be neccessary. Therefore, any long-lived passwords or private keys must never be sent in the clear. Active Attacks Whereas passive attacks involve SNIFFING, active attacks will generally involve SPOOFING. If an attacker can successfully masquerade as a client, he will have total read/write access to those storage resources assigned to that client. Spoofing as a server is more difficult, since many operations involve client reads of some expected or otherwise understandable data. Most likely, many of the sessions will be long-lived. This feature has a dual effect of making these sessions more vulnerable to attack (hijacking TCP connections, crytographic attacks), while at the same time providing mechanisms to detect attacks. An attempt to open a session while one is already active can be treated as a possible attack. Both the transport and session layer protocols will have sequencing that would need to be adhered to be the attacker to avoid generating errors that could also be treated as a possible attack. Message modification can be a significant threat to an environment reliant on the integrity of the data. Message replay, insertion, or deletion will generally produce errors (such as data overruns/underruns) that can be recovered successfully, they can have the effect of reducing performance, and as such can act as a denial of service. It is possible that an attacker can modify a message in such a way the the session becomes out of sync, resulting in a tear down of the session. Security Model No Security This mode does not authenticate nor encrypt data. This mode should only be used in environments where there is minimal security risk and little chance for configuration errors. End-to-End Authentication This mode protects against an unauthorized access to storage resources either through an active attack (SPOOFING) or configuration errors. Once the client is authenticated, all messages are sent and received in the clear. This mode should only be used when there is minimal risk to man-in-the-middle attacks, eavesdropping, message insertion, deletion, and modification. For example, this mode can be used when IPsec is used in security gateways. Encryption This mode provides for the end-to-end encryption (e.g. IPsec). It addition to authenticating the client, it provides end-to-end data integritya and protects against man-in-the-middle attacks, eavesdropping, message insertion, deletion, and modification. Other Considerations Due to long-lived sessions, is there a need for periodic authentication after the session is established? For example, should the client be challenged during key-alive exchanges in addition to login? Due to long-lived sessions with encryption, is there a higher level of vulnerability to crytographic attacks? From owner-ips@ECE.cmu.edu Thu Jul 06 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09914; 6 Jul 2000 11:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA26358 for ips-outgoing; Thu, 6 Jul 2000 10:31:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA26353 for ; Thu, 6 Jul 2000 10:31:35 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id 986BF1B7; Thu, 6 Jul 2000 10:31:34 -0400 (EDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id HAA22276; Thu, 6 Jul 2000 07:32:03 -0700 (PDT) Message-ID: <39649847.71AC5523@agilent.com> Date: Thu, 06 Jul 2000 07:31:35 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI Requirements v0.3 05 Jul 00 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > Section 3.5: "[R] It has been noted that a remote DMA option for TCP > possibly could provide the desired framing." > > Not if the iSCSI header is lost. If the iSCSI header is lost, RDMA won't > help, > because the RDMA will not know which I/O this iSCSI message is for, and > will not be able to DMA the data into the correct buffers described for the > I/O > (Initiator read for example). > > ++++RDMA knows what to do with data on a TCP segment base - It does not > need the ISCSI header - that is the whole idea! Ok, I went back and read the TCP RDMA proposal, and I agree. This is essentially adding the framing and other fields to handle the data in each received ethernet frame - but what is the likelyhood (and when) that this will be approved? > Section 3.5 "Selective TCP retransmission.": > > "[D] Given the long delays in the WAN, using TCP selective retransmission > must be supported by iSCSI, in order to minimize the bandwidth impact of > retransmission." > > How can iSCSI support how the underlying TCP transport performs it's > retransmission? > > ++++ isn't SACK an option SACK is a TCP option. The only thing iSCSI has to do to "support" this is turn it on! The wording makes it sound like iSCSI must do more than that. > Section 3.5 "Firewall friendly. The protocols use of IP addressing and TCP > port numbers should be firewall friendly." > > Why? "real" implementations will use their own dedicated lines to reach > long distances, and will not transport data over the "public internet", > especially due to the "information super highway traffic jam". > > ++++ firewall support will be deemed essential by everybody as it includes > handling threats beyond what we will handle in iSCSI My point is, it only has to be "firewall friendly" if it is going *through* the firewall. Now, how many real implementations are going to do storage through a firewall? -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Thu Jul 06 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09912; 6 Jul 2000 11:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA25677 for ips-outgoing; Thu, 6 Jul 2000 10:11:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA25666 for ; Thu, 6 Jul 2000 10:11:43 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 35FB1837F3 for ; Thu, 6 Jul 2000 09:09:31 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N5PCKYPG; Thu, 6 Jul 2000 09:17:38 -0500 Message-ID: <396493CC.443B7817@nuspeed.com> Date: Thu, 06 Jul 2000 09:12:28 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: A Data Integrity Negotiation Scheme for iSCSI Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Here are some thoughts on data integrity, and negotiating an iSCSI-level scheme. In some networks, iSCSI-level data integrity is not a requirement, but in others, it will be essential. Providing an extensible mechanism to negotiate appropriate data integrity will make it easier to bring iSCSI into new environments, without rewriting the original RFC. A proposal for adding a data integrity mechanism at the iSCSI level. This method is independent of transport-level data integrity mechanisms, such as TCP-CRC or IPSec or other tunneling security protocols. It attempts to first specify an extensible mechanism that is independent of the actual technologies that could be used, such as checksums, CRCs, or hashes. It then begins to specify an actual mechanism, to be used until it is deemed insufficient for some application. The point is that if there is a negotiable, extensible mechanism, future RFCs can define new mechanisms, and we don't have to make too many predictions about the future of data integrity right now. Assumptions - This method is meant to protect data from accidential harm, caused by undetected TCP checksum errors, or intervening devices that may deliver iSCSI headers and/or data between TCP connections. - This method is NOT meant to be cryptographically secure. An active agent could strip and (or just strip) this mechanism when modifying data purposefully. This mechanism assumes that in cases where this may be a problem, that other security mechanisms, such as the transport-level mechanisms mentioned above, or higher-level (file or database) cryptographic integrity mechanisms are used. Requirements - Data integrity checks should be done on data separately from commands and headers; this allows data integrity checks on data to be stored in an intelligent storage device (more end-to-end protection), or forwarded by a gateway. - Different integrity checks may be used on headers and commands vs. data. These are negotiated at login. Different checks may have different and more or less desirable properties, such as the ability to cover larger ranges of data, or be incrementally updated, or whatever. The login mechanism allows this. - No iSCSI-level checks are added unless they are specified at login. Initiators and targets not supporting data integrity checks at this level must interoperate with those that do. So here's the scheme: 1. At login, data integrity schemes are negotiated separately for use on data and commands. For the purposes of this scheme, data is defined as the payloads in the SCSI Data Request and Response. Commands are defined as everything else, including iSCSI headers. 2. Most schemes will add the command integrity check of their choice to: - The end of an iSCSI non-data request or response (command integrity check). - Immediately after the iSCSI header in a data request or response, if length > 0. - Immediately after the iSCSI header in a Command Request if the Immediate (I) bit is set. 3. Most schemes would add the data integrity check of their choice to: - The end of the data for an Immediate Command Request, Data Request, or Data Response. - OR - - After Every xxxx bytes (perhaps 8k) of data in an Immediate Command Request, Data Request, or Data Response. - OR - - Both of the above. Granularity Some checksum schemes may weaken significantly if used on continuous data of over a certain size. These schemes should define a granularity, or data length, after which individual integrity checks are inserted. Login Request If an initiator wishes to use an iSCSI-level data integrity scheme, it will include one or both of the following fields: dataIntegrity: ... commandIntegrity: ... dataIntegrity specifies the scheme names, in order of preference, that the initiator would like to support. The target must choose the highest-preference scheme that it wishes to allow. commandIntegrity specifies scheme names it would like to use for commands and headers. If unspecified, the default for both values is "none". Example: dataIntegrity: CRC32 commandIntegrity: Checksum32 none This specifies that the initiator will only accept the CRC32 scheme for data, and would prefer the Checksum32 for commands, but will accept no data integrity for commands if Checksum32 is not available. Login Response When a target receives a login request, it checks to see if it supports, and is willing to support, one of the schemes requested by the initiator. If any are supported, it returns the one most preferred by the initiator. If none are supported, it fails the login with a new error code. Data and command integrity schemes are negotiated separately. Note that even if a scheme, such as "none" is supported by a target, it may refuse to use it for a particular target, due to the user configuring the iSCSI target in a more protective or paranoid manner. Example: dataIntegrity: CRC32 commandIntegrity: none The login response is not protected by the negotiated command integrity check. All subsequent requests from the host are The CRC32 Scheme Other than "none", "CRC32" is the scheme specified in this document. Other schemes that may be more applicable should be proposed as well. If the CRC32 scheme is negotiated for headers, a 32-bit CRC is at the end of an iSCSI request or response, or between the request and its data, as described above. If the CRC32 scheme is negotiated for data, a 32-bit CRC is added to the end of all data, covering ONLY the data (headers and commands sold separately). The big open question here is whether to support a CRC at the end of the data request, or at the end of every [8k?] "block" of data, or both. The advantage of a CRC over just the block is simplicity; this works fine for most block device accesses, since file systems and databases usually write in 2, 4, or 8k clusters. However, if the data is of sufficient size as to make the check too weak (tape backup software and specialized file systems for large block data can write 64k .. 1MB or more at a time), we would need to add the CRC more often. [ WORK - Insert the appropriate algorithm or polynomial here ] There's more to do here, but I wanted to at least float something out for comments, and I think that the negotiation scheme is more important than nailing down the exact scheme right now, especially since data integrity requirements will change depending upon the end user's application and network. Notes All of the above schemes negotiate insertion of commands and data outside the normal headers and data, and therefore do not specify modifications to either. If iSCSI is used entirely over a layer-2 network (no l3 routers in the middle), the data-link layer CRC is kept end-to-end, and is not regenerated by bridges or switches. iSCSI transfers, regardless of their size, are broken up into 576, 1500, or at worst 9k (GbE Jumbo Frame) packets, each with their own 32-bit CRC. In this case, the only likely place for data integrity problems is within the iSCSI host and device, which will have the exact same characteristics as if Fibre Channel, parallel SCSI, or any other storage interface were used. The strength of the TCP checksum in this case should not be an issue. -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Thu Jul 06 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10007; 6 Jul 2000 12:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA27735 for ips-outgoing; Thu, 6 Jul 2000 11:11:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA27731 for ; Thu, 6 Jul 2000 11:11:23 -0400 (EDT) From: hafner@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA55002 for ; Thu, 6 Jul 2000 11:08:52 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id JAA74060 for ; Thu, 6 Jul 2000 09:11:21 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256914.00536F41 ; Thu, 6 Jul 2000 09:11:19 -0600 X-Lotus-FromDomain: IBMUS To: ips@ece.cmu.edu Message-ID: <87256914.0052FFA4.00@d53mta03h.boulder.ibm.com> Date: Thu, 6 Jul 2000 08:06:31 -0700 Subject: Long CDBs Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk In a recent posting, someone wrote: >This also means that the new "object oriented" CDBs will never be able to have >immediate write data. I just want to remind folks on this list that the "long CDBs" are by no means limited to the Object-based stuff. In fact, the first use of them will be for read/write on block devices with greater than 2TB of data. Many old CDBs didn't have enough room for large addresses (>4bytes). T10 has approved a number of related proposals to address this limitation. Some of these proposals use long cdbs. Also keep in mind that these are approved, not just under discussion. Jim Hafner From owner-ips@ECE.cmu.edu Thu Jul 06 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10123; 6 Jul 2000 13:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA00034 for ips-outgoing; Thu, 6 Jul 2000 12:21:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA00029 for ; Thu, 6 Jul 2000 12:21:01 -0400 (EDT) From: hafner@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id LAA125646; Thu, 6 Jul 2000 11:09:08 -0500 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id KAA146486; Thu, 6 Jul 2000 10:15:49 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256914.005951B8 ; Thu, 6 Jul 2000 10:15:36 -0600 X-Lotus-FromDomain: IBMUS To: "IPS (E-mail)" cc: "HAAGENS,RANDY (HP-Roseville,ex1)" Message-ID: <87256914.005928CF.00@d53mta03h.boulder.ibm.com> Date: Thu, 6 Jul 2000 09:13:50 -0700 Subject: Re: iSCSI Requirements and Dwgs Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk A couple of things (some big, some little - you decide which is which :-{)...) 1) In the requirements document, I (personally) would not use the word "security" when referring to my proposal for access controls; "access controls" is more appropriate and carries less baggage. Related note: my (approved) proposal's latest revision is 99-245r9, and (I expect) there will be a followup for editorial reasons. Soon it will be incorporated into an SPC-3 draft, and that would be the correct place to reference (when it comes out). 2) Many of the terms from SAM-2 used in both documents a rapidly being phased out or clarified. E.g., SMU, I believe, is being expunged. I think a lot of the questions in both documents about "target", "port", naming, etc. will be settled in the near future. If would recommend following at least some of the stuff going on t10@t10.org to get the latest on what's happening in SCSI. 3) I wouldn't be so reluctant to "avoid impact on T10". Many changes where made to SCSI for FCP. The T10 group is very willing to make changes (particularly ones that don't impact existing things). So, for example, a proposal to enable third party addressing that *extends* existing stuff in SPC-2 would not be a problem. (see (6) below). 4) I again (in spite of the long discussion with David) object to the wording under Naming: "It may be necessary to provide a naming scheme for SCSI LUs. *Fibre Channel does so using WWNs.*..." I haven't found anything in FCP which says this. There is a paragraph in 5.2.3 (under the 5.2 section on SCSI address format for FCP) which says that targets (i.e., FC ports) need WWN for both port and node (but that's an FC requirement and doesn't have anything to do with SCSI (yet)). Node and Port names must be different. Now we get to the SCSI requirements. FCP devices *must* support INQUIRY EVPD page 83h. A device with only one LU *may* use the FC WWNodeName for the INQUIRY WWname. The document makes no requirement or suggestion for multi-LU device, other than each LU must have a SCSI unique name in EVPD data. If I missed something (in FCP), please correct me and provide the specific reference in the requirements doc, so its not so confusing. 5) I would recommend going back to Bob Snively's posting on naming. I found it clear and informed and answers most of the questions. 6) Suggestion for third party naming in iSCSI. I'll post this in another note as it's too long for this one. Jim Hafner From owner-ips@ECE.cmu.edu Thu Jul 06 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10212; 6 Jul 2000 14:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA01173 for ips-outgoing; Thu, 6 Jul 2000 12:57:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA01168 for ; Thu, 6 Jul 2000 12:57:00 -0400 (EDT) From: hafner@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA25014 for ; Thu, 6 Jul 2000 12:54:29 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id KAA148872 for ; Thu, 6 Jul 2000 10:56:58 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256914.005D182A ; Thu, 6 Jul 2000 10:56:50 -0600 X-Lotus-FromDomain: IBMUS To: "IPS (E-mail)" Message-ID: <87256914.005D16E2.00@d53mta03h.boulder.ibm.com> Date: Thu, 6 Jul 2000 09:56:45 -0700 Subject: Third party naming vis-a-vis iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Here's my promised suggestion for third-party naming. There are two approaches, one fully integrated into SCSI (and so independent of iSCSI, though iSCSI can best take advantage of it). The other approach is more attuned to iSCSI, though could be used in other (probably future) transports. The real problem in this space is not in the higher application layers which can do naming in anyway they want, so long as there is an application that can translate from one form to another. (So a URL can get "resolved" by different application layers into an ethernet MAC and IP port, etc.) The problem is that an arbitrary naming scheme doesn't map well into existing SCSI paramter data where third party addresses are used (e.g., third party reservations, RAID commands like XDWRITE, extended copy, etc.). The biggest problem in that space is the fact that SCSI has only (at the moment) set aside fixed sized fields for this purpose (and we know iSCSI is going to need much longer names). In some cases, this field is 8 bytes (e.g., third party reservations) and in others 16 bytes (bytes 12-27 in extended copy target descriptors; I'm referring here to the identifier of the "target device" and not the logical unit within that device). The first suggestion, then is to use the existing fixed field (and only 8 bytes of that) of third party addressing as an "alias" field. The requesting initiator will (through a mechanism described later) define for the target a translation mapping of this alias to the transport specific identifier of the target (e.g., the portion of the url, the FC WWN, the parallel SCSI address). A flag bit or field (TBD for each usage) will indicate that this is an alias field and not a specific address identifier. Using aliases in this way, we don't need to change the size of the fields currently dedicated to third party addresses and we don't need to change their existing usage. Here are the two approaches for an initiator to define the alias<->thirdparty mapping. Note, in all cases, (without a completely different protocol) the mapping would be initiator specific, not a global aliasing. 1) Within iSCSI, a Text command could define the mapping. Advantages: a) you'd only need to change SPC-2 to allow for alias-type usage, and leave the mapping to "the transport". b) it would apply to all third-party addressing contexts. Disadvantages: a) there's a disassociation between the actual mapping and the use of the mapping (that is, state must be maintained) and so all the issues about state (persistence, etc) have to be addressed. b) it's not clear how other transports could take advantage of this. 2)an additional "chunk" of the parameter data could be used to define the alias. For example, in extended copy, an additional "page" of parameter data can define the mapping for that instance of the command. Advantages: a) can be used in all transports and future transports b) is a general extensible mechanism c) has state only for the context of the command itself d) can handle whatever size of device identifier you want (e.g., arbitrarily long ) - that's because you can define the parameter data in this way. Disadvantages a) would require more changes to SPC-2 (both alias usage and the definition of mapping), and would (might?) affect any command where third-party addressing is used. As for how logical units are referenced within a target, the existing mechanisms of LUN and Proxy Token (from access controls) are sufficient. In all cases, the harder part is finding the device. SCSI already knows how to find the logical unit within the device. But I don't see a fundamental problem with adding this logical unit identifier (in whatever form you like, LUN, Proxy Token, EVPD data) as part of the alias mapping. The subtle issue is that in some cases there is no logical unit to reference (e.g., third party reservations reference an initiator, not a target). Political comment: After having worked with the T10 guys over the last year or so, I find them very amenable to additions/changes to SCSI so long as it doesn't break anything and that it meets someones perceived need. I think the issues raised here meet both those requirements. So, I wouldn't be reluctant (if I were more active in this iSCSI thing) to take specific proposals to T10 to enhance what iSCSI can do. Jim Hafner From owner-ips@ECE.cmu.edu Thu Jul 06 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10439; 6 Jul 2000 16:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA06925 for ips-outgoing; Thu, 6 Jul 2000 15:31:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA06920 for ; Thu, 6 Jul 2000 15:31:06 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 1933A837F3; Thu, 6 Jul 2000 14:28:54 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N5PCKYX6; Thu, 6 Jul 2000 14:37:02 -0500 Message-ID: <3964DEA9.5CAA44BD@nuspeed.com> Date: Thu, 06 Jul 2000 14:31:53 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: Robert Elliott Cc: ips@ece.cmu.edu Subject: Re: A Data Integrity Negotiation Scheme for iSCSI References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Rob- Good point. Perhaps if we were to negotiate the scheme, we could negotiate the maximum "block" size covered by a data integrity check as well. The login might look like: Login Request: dataIntegrity: CRC32 dataIntegrityBlock: 8192 commandIntegrity: CRC32 Login Response: dataIntegrity: CRC32 dataIntegrityBlock: 512 commandIntegrity: CRC32 The initiator would specify the largest block of data over which any CRC could be used; the target could further limit the size but could not request a larger size. Both would implicitly agree on the size sent by the target. If the initiator did not support the size specified by the target, it could either give up, or try to log in using a different data integrity scheme. Another note - someone mentioned having data check tags every once in a while to just make sure that the data stream was still in sync; this is implicitly done when using a CRC every so many bytes. If one wanted a scheme without CRC overhead, but with a sync check, one could negotiate a scheme of SYNC32, and place a magic number instead of the CRC; if the magic number was wrong, data errors affecting length fields would be assumed, and the connection would be re-started. Request: dataIntegrity: SYNC32 dataIntegrityBlock: 4096 dataIntegrityMagic: 0x00c0ffee Response: dataIntegrity: SYNC32 dataIntegrityBlock: 4096 dataIntegrityMagic: 0x00c0ffee Subsequent data would include the magic number "0x00c0ffee" after every 4k bytes for both read and write requests. The above could also be done by just doing CRC on commands, and no CRC on the data, but a sync check would allow this do be done on a block-by-block basis, which as you said, may be implemented in a device with relatively minimal buffering. At any rate, the data integrity negotiation scheme should be extensible enough to add these things. -- Mark Robert Elliott wrote: > > Mark A Bakke wrote: > > The CRC32 Scheme > > > > Other than "none", "CRC32" is the scheme specified in this > > document. Other schemes that may be more applicable should > > be proposed as well. > > > > If the CRC32 scheme is negotiated for headers, a 32-bit CRC > > is at the end of an iSCSI request or response, or between > > the request and its data, as described above. > > > > If the CRC32 scheme is negotiated for data, a 32-bit CRC > > is added to the end of all data, covering ONLY the data (headers > > and commands sold separately). > > > > The big open question here is whether to support a CRC at the > > end of the data request, or at the end of every [8k?] "block" > > of data, or both. The advantage of a CRC over just the block > > is simplicity; this works fine for most block device accesses, > > since file systems and databases usually write in 2, 4, or 8k > > clusters. However, if the data is of sufficient size as to make > > the check too weak (tape backup software and specialized file > > systems for large block data can write 64k .. 1MB or more at > > a time), we would need to add the CRC more often. > > Storage devices cannot commit data to storage until the CRC has been > checked, so larger intervals imply larger temporary storage buffers. > In parallel SCSI (Ultra 3) the target was given the ability to choose > how often CRC is transferred for both writes and reads. This keeps > it from needing infinite buffers for writes - if it has a 512 byte > write buffer, it can ask for a CRC every 512 bytes. For reads, the > initiator can just redo the read if an error is detected - nothing > disastrous happens if bad data enters its destination buffer before > the command is marked complete. The target can send CRC when it > chooses. > > -- > Rob Elliott UNIX mailto:relliott@hobbit.eng.hou.compaq.com > Houston, TX PC mailto:Robert.Elliott@compaq.com -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Thu Jul 06 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10445; 6 Jul 2000 16:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA07316 for ips-outgoing; Thu, 6 Jul 2000 15:42:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA07312 for ; Thu, 6 Jul 2000 15:42:36 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id MAA06429; Thu, 6 Jul 2000 12:42:25 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id <3MXWCDTM>; Thu, 6 Jul 2000 12:42:22 -0700 Message-ID: From: Jim McGrath To: "'VonStamwitz, Paul'" , ips@ece.cmu.edu, "'julian_satran@il.ibm.com'" Subject: RE: Security Considerations Date: Thu, 6 Jul 2000 12:42:24 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk For reference, what security is present today for NAS class products (e.g. those typically using a higher level protocol like NFS rather than SCSI)? My impression (which I admit could be quite mistaken) is that we may be on a path of requiring a higher level of security for iSCSI than is warranted by alternative protocols. Of course, enhancing security has its value, but since SCSI starts off with essentially no security, iSCSI is a poor protocol upon which to require lots of security. Thoughts? Jim -----Original Message----- From: VonStamwitz, Paul [mailto:paulv@corp.adaptec.com] Sent: Thursday, July 06, 2000 12:03 AM To: ips@ece.cmu.edu; 'julian_satran@il.ibm.com' Subject: Security Considerations In keeping with the "Guidelines for Writing RFC Text on Security Considerations" by E. Rescorla and B. Korver, the following is my pass at describing the kind of security threats our protocol could encounter. It does not yet include specific mechanisms that address these threats. Julian, I am sorry for the tardiness. I tried to format this in such a way that you could easily cut and place into your document as you see fit. I hope this is useful. I spoke with Luciano today, and he said he would be sending his contributions to you today as well. It is my belief that he will provide the detailed mechanisms by modifying the document directly. Unfortunately, I will be off-site and will not be able to join the conference call tomorrow. Paul von Stamwitz ------------------------ Security Considerations Historically, native storage systems have not had to consider security due to the fact that their environments offered minimal security risks. That is, these environments consisted of storage devices either directly attached to hosts or connected via a subnet distinctly separate from the communications network. The use of storage protocols, such as SCSI, over IP networks requires that security concerns be addressed. First, a threat model needs to be developed which describes the kind of attacks that can be made on the protocol. It needs to be determined which of these attacks are out of scope and why. Then, mechanisms need to be applied to the protocol to protect against those attacks that are in-scope. Threat Model Attacks fall into three main areas; passive, active, and denial of service. Passive Attacks In general, data transfers will be made through a switched fabric, making sniffing difficult. Also, the nature of the data (block transfers), even if sniffed, would not necessarily be readily understandable to the attacker. That being said, a determined attacker could, by capturing of content and analyzing traffic over time, could replicate enough of a drive to make the captured data meaningful. Certain storage operations which are mostly uni-directional, such as writing to a tape or reading from a CD-ROM, are even more susceptible to passive attacks since the listener will be able to replicate most if not all of the operation. Passive attacks by traffic analysis alone is deemed out of scope since it is unlikely that the listener will be able to guess any pertinent information without knowing the content of the messages. It is also out of scope to detect passive attacks. The protocol must be able to prevent passive attacks by masking the contents of messages through some form of encryption. Finally, it is assumed that a strong authentication mechanism will be neccessary. Therefore, any long-lived passwords or private keys must never be sent in the clear. Active Attacks Whereas passive attacks involve SNIFFING, active attacks will generally involve SPOOFING. If an attacker can successfully masquerade as a client, he will have total read/write access to those storage resources assigned to that client. Spoofing as a server is more difficult, since many operations involve client reads of some expected or otherwise understandable data. Most likely, many of the sessions will be long-lived. This feature has a dual effect of making these sessions more vulnerable to attack (hijacking TCP connections, crytographic attacks), while at the same time providing mechanisms to detect attacks. An attempt to open a session while one is already active can be treated as a possible attack. Both the transport and session layer protocols will have sequencing that would need to be adhered to be the attacker to avoid generating errors that could also be treated as a possible attack. Message modification can be a significant threat to an environment reliant on the integrity of the data. Message replay, insertion, or deletion will generally produce errors (such as data overruns/underruns) that can be recovered successfully, they can have the effect of reducing performance, and as such can act as a denial of service. It is possible that an attacker can modify a message in such a way the the session becomes out of sync, resulting in a tear down of the session. Security Model No Security This mode does not authenticate nor encrypt data. This mode should only be used in environments where there is minimal security risk and little chance for configuration errors. End-to-End Authentication This mode protects against an unauthorized access to storage resources either through an active attack (SPOOFING) or configuration errors. Once the client is authenticated, all messages are sent and received in the clear. This mode should only be used when there is minimal risk to man-in-the-middle attacks, eavesdropping, message insertion, deletion, and modification. For example, this mode can be used when IPsec is used in security gateways. Encryption This mode provides for the end-to-end encryption (e.g. IPsec). It addition to authenticating the client, it provides end-to-end data integritya and protects against man-in-the-middle attacks, eavesdropping, message insertion, deletion, and modification. Other Considerations Due to long-lived sessions, is there a need for periodic authentication after the session is established? For example, should the client be challenged during key-alive exchanges in addition to login? Due to long-lived sessions with encryption, is there a higher level of vulnerability to crytographic attacks? From owner-ips@ECE.cmu.edu Thu Jul 06 20:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10464; 6 Jul 2000 16:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA08426 for ips-outgoing; Thu, 6 Jul 2000 16:12:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA08421 for ; Thu, 6 Jul 2000 16:12:13 -0400 (EDT) Received: from sunmail1.Sun.COM ([129.145.1.2]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA19448 for ; Thu, 6 Jul 2000 13:12:11 -0700 (PDT) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.83.130]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id NAA08212 for ; Thu, 6 Jul 2000 13:12:12 -0700 (PDT) Received: from terra.eng.sun.com (terra.Eng.Sun.COM [129.146.86.99]) by jurassic.eng.sun.com (8.10.2+Sun/8.10.2) with ESMTP id e66KC9j367547 for ; Thu, 6 Jul 2000 13:12:09 -0700 (PDT) Received: (from brent@localhost) by terra.eng.sun.com (8.9.3+Sun/8.9.3) id NAA03076 for ips@ece.cmu.edu; Thu, 6 Jul 2000 13:12:02 -0700 (PDT) Date: Thu, 6 Jul 2000 13:12:02 -0700 (PDT) From: Brent Callaghan Message-Id: <200007062012.NAA03076@terra.eng.sun.com> To: ips@ece.cmu.edu Subject: RE: Security Considerations X-Sun-Charset: US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim McGrath writes: > For reference, what security is present today for NAS class products (e.g. > those typically using a higher level protocol like NFS rather than SCSI)? NFS uses security mechanisms defined in the underlying RPC layer. Most NFS users use UNIX "trusted host" security because it's easy to administer and almost zero overhead - though it also provides almost zero security. There's also mechanisms that use Kerberos or Diffie-Hellman public keys to exchange DES keys. For details, see RFC's 2695 and 2623. A unique feature of these higher-level protocols is that they authenticate end-users, so it's done per request rather than per connection. An NFS server can have requests coming in from multiple users over a single TCP connection from a multi-user client. However, iSCSI will be authenticating nodes rather than users, so connection-oriented security would be more appropriate, e.g. IPSEC, SKIP, TLS, etc. Brent From owner-ips@ECE.cmu.edu Thu Jul 06 20:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10462; 6 Jul 2000 16:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA08065 for ips-outgoing; Thu, 6 Jul 2000 16:02:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA08057 for ; Thu, 6 Jul 2000 16:02:14 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id NAA09693; Thu, 6 Jul 2000 13:02:10 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id <3MXWC12F>; Thu, 6 Jul 2000 13:02:08 -0700 Message-ID: From: Jim McGrath To: "'relliott@hobbit.eng.hou.compaq.com'" , mark.bakke@nuspeed.com, ips@ece.cmu.edu Subject: RE: A Data Integrity Negotiation Scheme for iSCSI Date: Thu, 6 Jul 2000 13:02:09 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with Rob. This is where your intended application gets in the way again. If this is the sort of traditional, server to server communications networks employ today, then the original proposal is OK - the server is responsible for buffering and breaking down the long transmissions to block size units the data storage devices understand. But if you are going directly to the storage device, then your buffering and processing power is severely limited. Frankly, I don't know why we need multiple levels of CRC/ECC at all. Yes, TCP can be run on layer 2 networks that do not have any error checking. But is that a real concern (for LANs probably not - maybe for WANs)? Is all the TCP checksum doing is insuring frames do not get lost? And if so, aren't there other (more effective) mechanisms? Jim PS note that you can still use TCP as is in the protocol, you could just drop the requirement that the destination device check the checksum, allowing it to process blocks one at a time as normal (actually, ethernet/fibre channel/infiniband frame as a time - allowing for reassembly of course). -----Original Message----- From: relliott@hobbit.eng.hou.compaq.com [mailto:relliott@hobbit.eng.hou.compaq.com] Sent: Thursday, July 06, 2000 7:59 AM To: mark.bakke@nuspeed.com; ips@ece.cmu.edu Subject: Re: A Data Integrity Negotiation Scheme for iSCSI Mark A Bakke wrote: > The CRC32 Scheme > > Other than "none", "CRC32" is the scheme specified in this > document. Other schemes that may be more applicable should > be proposed as well. > > If the CRC32 scheme is negotiated for headers, a 32-bit CRC > is at the end of an iSCSI request or response, or between > the request and its data, as described above. > > If the CRC32 scheme is negotiated for data, a 32-bit CRC > is added to the end of all data, covering ONLY the data (headers > and commands sold separately). > > The big open question here is whether to support a CRC at the > end of the data request, or at the end of every [8k?] "block" > of data, or both. The advantage of a CRC over just the block > is simplicity; this works fine for most block device accesses, > since file systems and databases usually write in 2, 4, or 8k > clusters. However, if the data is of sufficient size as to make > the check too weak (tape backup software and specialized file > systems for large block data can write 64k .. 1MB or more at > a time), we would need to add the CRC more often. Storage devices cannot commit data to storage until the CRC has been checked, so larger intervals imply larger temporary storage buffers. In parallel SCSI (Ultra 3) the target was given the ability to choose how often CRC is transferred for both writes and reads. This keeps it from needing infinite buffers for writes - if it has a 512 byte write buffer, it can ask for a CRC every 512 bytes. For reads, the initiator can just redo the read if an error is detected - nothing disastrous happens if bad data enters its destination buffer before the command is marked complete. The target can send CRC when it chooses. -- Rob Elliott UNIX mailto:relliott@hobbit.eng.hou.compaq.com Houston, TX PC mailto:Robert.Elliott@compaq.com From owner-ips@ECE.cmu.edu Thu Jul 06 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10656; 6 Jul 2000 18:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA10399 for ips-outgoing; Thu, 6 Jul 2000 17:11:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA10395 for ; Thu, 6 Jul 2000 17:11:14 -0400 (EDT) Received: from centralmail2.Central.Sun.COM ([129.147.62.11]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id PAA01481 for ; Thu, 6 Jul 2000 15:11:09 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail2.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA12259; Thu, 6 Jul 2000 15:10:30 -0600 (MDT) Received: from sun.com (hobo9.Japan.Sun.COM [129.158.86.109]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id OAA23371; Thu, 6 Jul 2000 14:57:31 -0600 (MDT) Message-ID: <3964F6EF.40137D2@sun.com> Date: Thu, 06 Jul 2000 15:15:27 -0600 From: "Mark A. Carlson" X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: SCSI over IP Working Group Subject: Asymetric Assumptions Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk In many of the discussions that I have seen on this list, there seems to be an unwritten assumption running through the threads that initiators are hosts with lots of memory and resources and targets are storage devices with restrained resources such as memory that need negotiation of the protocol parameters as a result. I would caution against making the protocol too asymmetric in this regard as we would like this protocol to work between storage devices as well as between hosts and storage devices. This would be used, for example, in a hostless back scheme or in a situation where one storage device handles mirroring to other storage devices itself. In one case, for example, an initiator's read might actually be committing the blocks to storage such as in a recovery from tape situation. I think that the requirements document adequately describes the requirements in this area, but we just need to be careful in our thinking during protocol discussions that the assumptions that we make do not cause the protocol to become asymmetric with regards to initiators and targets. So far this hasn't been the case. Thanks, -- mark From owner-ips@ECE.cmu.edu Fri Jul 07 00:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10885; 6 Jul 2000 20:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA15035 for ips-outgoing; Thu, 6 Jul 2000 19:11:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA15023 for ; Thu, 6 Jul 2000 19:11:17 -0400 (EDT) From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA18412 for ; Thu, 6 Jul 2000 19:05:48 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id RAA46892 for ; Thu, 6 Jul 2000 17:11:15 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256914.007F5D1C ; Thu, 6 Jul 2000 17:11:09 -0600 X-Lotus-FromDomain: IBMUS To: ips@ece.cmu.edu Message-ID: <87256914.007EBAD5.00@d53mta03h.boulder.ibm.com> Date: Thu, 6 Jul 2000 16:03:50 -0700 Subject: draft q. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk In the latest draft p35, in the picture of the login command layout, byte offset 24 follows 16 - should it be 20? prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose From owner-ips@ECE.cmu.edu Fri Jul 07 01:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10978; 6 Jul 2000 21:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA17386 for ips-outgoing; Thu, 6 Jul 2000 20:06:32 -0400 (EDT) Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA17382 for ; Thu, 6 Jul 2000 20:06:28 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id RAA01665; Thu, 6 Jul 2000 17:06:12 -0700 (PDT) Received: from localhost (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) with ESMTP id RAA06959; Thu, 6 Jul 2000 17:02:13 -0700 (PDT) X-Authentication-Warning: csapuntz-u1.cisco.com: csapuntz owned process doing -bs Date: Thu, 6 Jul 2000 17:02:13 -0700 (PDT) From: Costa Sapuntzakis To: Matt Wakeley cc: ips@ece.cmu.edu Subject: Re: change bars In-Reply-To: <3963FF5F.D2F1AFF1@agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-851401618-962928133=:6952" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. ---559023410-851401618-962928133=:6952 Content-Type: TEXT/PLAIN; charset=US-ASCII My original script had bugs in it. Here is a newer version with better revision tracking and hopefully fewer bugs. :-) -Costa On Wed, 5 Jul 2000, Matt Wakeley wrote: > Hi Costa, > > Thanks for the perl script. It really helps. > > But I noticed that it did not catch the "difference" between 0628 and 0706 of the > changing of the "ping" and the new "map" commands in section 3.1.1 (these are "new" > lines that it did not flag). > > -Matt > > Costa Sapuntzakis wrote: > > > I have included the Perl script I used to generate the change bars. > > The draft was generated as follows: > > > > perl create-change-bar.pl iSCSI000628.txt iscsi00615.txt > iSCSI000628.2.txt > > > > Hope it helps! > > > > -Costa > > > ---559023410-851401618-962928133=:6952 Content-Type: TEXT/PLAIN; charset=US-ASCII; name="create-change-bar.pl" Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename="create-change-bar.pl" IyENCg0KIw0KIyBUaGUgc2NyaXB0IG9wZXJhdGVzIGluIHRoZSBmb2xsb3dp bmcgZmFzaGlvbi4gRmlyc3QsIGl0DQojIHN0cmlwcyBhbGwgYmxhbmsgbGlu ZXMsIGhlYWRlcnMsIGFuZCBmb290ZXJzIGZyb20gYm90aA0KIyBkb2N1bWVu dC4gDQojDQojIEl0IHRoZW4gcnVucyBkaWZmIG9uIHRoZSBzdHJpcHBlZCBm aWxlcyB0byBmaW5kIHRoZQ0KIyBkaWZmZXJlbmNlcy4NCiMNCiMgRmluYWxs eSwgaXQgdGFrZXMgdGhlIG91dHB1dCBvZiBkaWZmIGFuZCBnZW5lcmF0ZXMg dGhlDQojIGNoYW5nZSBiYXJzIGF0IHRoZSBkZXN0aW5hdGlvbi4NCiMNCiMg Tm90ZTogdGhlIGNoYW5nZSBiYXJzIG9ubHkgbWFyayBhZGRpdGlvbnMgYW5k IGNoYW5nZXMNCiMNCiMgLUNvc3RhIChjc2FwdW50ekBhbHVtLm1pdC5lZHUp IDA2LzI4LzAwDQoNCnVzZSBzdHJpY3Q7DQoNCnVzZSBHZXRvcHQ6Okxvbmc7 DQoNCm15ICRsaW5lc19mbGFnID0gMDsNCg0Kc3ViIHVzYWdlIHsNCiAgICBw cmludCBTVERFUlIgIlVzYWdlOlxucGVybCBjcmVhdGUtY2hhbmdlLWJhci5w bCBbLS1saW5lc10gPGZpbGUxPiA8ZmlsZTI+XG4iOw0KICAgIHByaW50IFNU REVSUiAiZmlsZTEgd2lsbCBiZSBwcmludGVkIHdpdGggY2hhbmdlIGJhcnNc blxuIjsNCiAgICBwcmludCBTVERFUlIgIi0tbGluZXMgaXMgYSBsZXNzIGV4 YWN0IG1hdGNoIHRoYXQgbWF5IHdvcmsgYmV0dGVyIGZvclxuIjsNCiAgICBw cmludCBTVERFUlIgIiAgICAgICAgbGFyZ2UgZGlmZnNcbiI7DQogICAgZXhp dCAxOw0KfQ0KDQomR2V0T3B0aW9ucyhsaW5lcyA9PiBcJGxpbmVzX2ZsYWcp Ow0KDQomdXNhZ2UgaWYgKCQjQVJHViAhPSAxKTsNCg0KbXkgJHNyYyA9ICRB UkdWWzBdOw0KbXkgJGRzdCA9ICRBUkdWWzFdOw0KDQomdXNhZ2UgaWYgKCFk ZWZpbmVkICRzcmMgfHwgIWRlZmluZWQgJGRzdCk7DQoNCm9wZW4oU1JDLCAk c3JjKSBvciBkaWUgIlVuYWJsZWQgdG8gb3BlbiAkc3JjOiAkISI7DQoNCiRz cmMgLj0gIi50bXAiOw0KDQppZiAoLXIgJHNyYykgew0KICAgIHByaW50IFNU REVSUiAiRXJyb3I6IHRlbXBvcmFyeSBmaWxlICRzcmMgYWxyZWFkeSBleGlz dHNcbiI7DQogICAgZXhpdCAxOw0KfQ0KDQpvcGVuKFNSQzIsICI+JHNyYyIp ICBvciBkaWUgIlVuYWJsZWQgdG8gb3BlbiAkc3JjOiAkISI7Ow0KDQpvcGVu KERTVCwgJGRzdCkgIG9yIGRpZSAiVW5hYmxlZCB0byBvcGVuICRkc3Q6ICQh Ijs7DQoNCiRkc3QgLj0gIi50bXAiOw0KDQppZiAoLXIgJGRzdCkgew0KICAg IHByaW50IFNUREVSUiAiRXJyb3I6IHRlbXBvcmFyeSBmaWxlICRkc3QgYWxy ZWFkeSBleGlzdHNcbiI7DQogICAgZXhpdCAxOw0KfQ0KDQpvcGVuKERTVDIs ICI+JGRzdCIpICBvciBkaWUgIlVuYWJsZWQgdG8gb3BlbiAkZHN0OiAkISI7 Ow0KDQoNCnN1YiBwcm9jZXNzX2ZpbGUgew0KICAgIG15ICgkbGluZW1hcCwg JGZoMSwgJGZoMikgPSBAXzsNCg0KICAgICMgU3RyaXAgbGVhZGluZyBjaGFu Z2UgYmFycw0KICAgIHMvXnwvLzsNCiAgICANCg0KICAgIG15ICRvbGRsaW5l bm8gPSAtMTsNCiAgICBteSAkbmV3bGluZW5vID0gMDsNCiAgICBteSAkc3Rh dGUgPSAwOw0KDQogICAgbXkgJHNhdmV3b3JkIDsNCiAgICBteSAkc2F2ZXdv cmRsaW5lbm8gPSAwOw0KICAgIA0KICAgIHdoaWxlKDwkZmgxPikgew0KCWNo b21wOw0KCQ0KCSRvbGRsaW5lbm8rKzsNCg0KCWlmICgkc3RhdGUgPT0gMSkg ew0KCSAgICAkc3RhdGUgPSAwOw0KCSAgICBuZXh0Ow0KCX0NCgkNCglpZiAo L15cMDE0Lykgew0KCSAgICAkc3RhdGUgPSAxOw0KCSAgICBuZXh0Ow0KCX0N CgluZXh0IGlmICgvXFtQYWdlLyk7DQoJbmV4dCBpZiAoL15ccyokLyk7DQoJ DQoJaWYgKCAkbGluZXNfZmxhZyApIHsNCgkgICAgJCRsaW5lbWFwWyRuZXds aW5lbm9dID0gJG9sZGxpbmVubzsNCgkgICAgJG5ld2xpbmVubysrOw0KCSAg ICBwcmludCAkZmgyICRfIC4gIlxuIjsNCgl9IGVsc2Ugew0KCSAgICBzL1tc dFxuXHJdLy9nOw0KCSAgICBteSBAd29yZHMgPSBzcGxpdDsNCgkgICAgDQoJ ICAgIGlmICgkc2F2ZXdvcmQpIHsNCgkJcHJpbnQgJGZoMiAkc2F2ZXdvcmQ7 DQoJCXVuZGVmICRzYXZld29yZDsNCgkgICAgfQ0KCSAgICANCgkgICAgaWYg KCQjd29yZHMgPj0gMCAmJg0KCQkkd29yZHNbJCN3b3Jkc10gPX4gLy0kLykg ew0KCQkkc2F2ZXdvcmQgPSBzdWJzdHIocG9wIEB3b3JkcywgMCwgLTEpOw0K CQkkc2F2ZXdvcmRsaW5lbm8gPSAkb2xkbGluZW5vOw0KCSAgICB9DQoJICAg IA0KCSAgICBmb3JlYWNoIG15ICR3b3JkIChAd29yZHMpIHsNCiMJICAgIHBy aW50ICR3b3JkIC4gIlxuIjsNCgkJcHJpbnQgJGZoMiAkd29yZCAuICJcbiI7 DQoJCQ0KCQkkJGxpbmVtYXBbJG5ld2xpbmVub10gPSAkb2xkbGluZW5vOw0K CQkkbmV3bGluZW5vKys7DQoJICAgIH0NCgl9DQogICAgfQ0KDQp9DQoNCiMN CiMgU3RvcmUgbWFwcGluZyBiZXR3ZWVuIHN0cmlwcGVkIGxpbmUgbnVtYmVy cyBhbmQgb3JpZ2luYWwNCiMgbGluZSBudW1iZXJzDQojDQpteSBAbGluZW1h cCA9ICgpOw0KbXkgQGNoYW5nZXMgPSAoKTsNCg0KJnByb2Nlc3NfZmlsZShc QGxpbmVtYXAsIFwqU1JDLCBcKlNSQzIpOw0KJnByb2Nlc3NfZmlsZShbXSwg XCpEU1QsIFwqRFNUMik7DQoNCmNsb3NlIERTVDI7DQpjbG9zZSBEU1Q7DQpj bG9zZSBTUkMyOw0KDQojDQojIGRpZmYgdGhlIHR3byBkb2N1bWVudHMNCiMN Cm9wZW4gKERJRkZTLCAiZGlmZiAtdyAkZHN0ICRzcmMgfCIpIG9yIGRpZSAi VW5hYmxlZCB0byBydW4gZGlmZjogJCEiOw0KDQp3aGlsZSg8RElGRlM+KSB7 DQojICAgIHByaW50IDsNCg0KICAgIG5leHQgaWYgKCEvXlxkLyk7DQoNCiAg ICAvW2FjZF0oXGQrKSgsKFxkKykpPy87DQogICAgDQogICAgbmV4dCBpZiAo IWRlZmluZWQgJDEpOw0KDQogICAgbXkgJHN0YXJ0ID0gJDE7DQogICAgbXkg JGVuZCA9ICQzOw0KDQogICAgaWYgKCFkZWZpbmVkICRlbmQpIHsNCgkkZW5k ID0gJHN0YXJ0Ow0KICAgIH0NCg0KICAgIGZvciAobXkgJGlkeCA9ICRzdGFy dCAtIDE7ICRpZHggPD0gJGVuZCAtIDE7ICRpZHgrKykgew0KCSRjaGFuZ2Vz WyRsaW5lbWFwWyRpZHhdXSA9IDE7DQogICAgfQ0KfQ0KDQojDQojIFByaW50 IG91dCBkb2N1bWVudCwgbWFya2luZyBjaGFuZ2VkIGxpbmVzDQojDQpzZWVr IFNSQywgMCwgMDsNCg0KbXkgJG9sZGxpbmVubyA9IDA7DQoNCndoaWxlICg8 U1JDPikgew0KICAgICMgU3RyaXAgbGVhZGluZyBjaGFuZ2UgYmFycw0KICAg IHMvXnwvLzsNCg0KICAgIGlmICgkY2hhbmdlc1skb2xkbGluZW5vXSkgew0K CXByaW50ICJ8IjsNCiAgICB9IGVsc2Ugew0KCXByaW50ICIgIjsNCiAgICB9 DQoNCiAgICBwcmludDsNCg0KICAgICRvbGRsaW5lbm8rKzsNCn0NCg0KIw0K IyBDbGVhbnVwDQojDQoNCnVubGluayAkc3JjOw0KdW5saW5rICRkc3Q7DQoN Cg0KDQo= ---559023410-851401618-962928133=:6952-- From owner-ips@ECE.cmu.edu Fri Jul 07 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11559; 7 Jul 2000 0:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA21491 for ips-outgoing; Thu, 6 Jul 2000 23:15:33 -0400 (EDT) Message-Id: <200007070315.XAA21491@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Experiment in Requirements Formatting Date: Thu, 6 Jul 2000 19:14:22 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFE7B9.10EC4970" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFE7B9.10EC4970 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id XAB21491 Attached is version 0.3 of the Requirements document. As I send it, it i= s in rich text appended to this message. I'm curious to see if the ips reflector/archive will assist us at all in converting it to ascii. That = is, will it convert to ascii at all, and if so, how readable will the convers= ion be? R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 <>=20 Internet-Draft Randy Haagens Hewlett-Packard Co. Expires dd Mmm 200y draft ver 0.3, 05 July 2000 Changes from ver 0.1 are highlighted in this fashion. These changes refl= ect discussion in the ad hoc working group meeting 19-23 Jun 00. iSCSI Requirements This work defines a mapping of SCSI to TCP/IP, known as iSCSI. Scope. We propose to define a mapping of SCSI protocol to TCP/IP so that SCSI storage controllers (principally disk and tape arrays and libraries) can be attached to IP networks, notably Gigabit Ethernet (GbE) and 10 Gigabit Ethernet (10 GbE). Motivation. We seek timely adoption of a protocol mapping for block stor= age over IP networks. Accordingly, we have chosen to work with the existing SCSI architecture and commands and also the existing TCP/IP transport lay= er. Both these protocols are widely-deployed and well-understood. Using them means a minimum of new invention, the most rapid possible adoption, and t= he greatest compatibility with Internet architecture, protocols, and equipme= nt. The iSCSI protocol is a mapping of SCSI to TCP, and constitutes a "SCSI transport" as defined by the SCSI SAM-2 document [SAM2, p. 3, "Transport Protocols"]. 1 Applicability Traditionally, volume/block-oriented storage controllers (e.g., disk arra= y controllers, tape library controllers) have supported the SCSI-3 protocol= , and have been attached to computers through the SCSI parallel bus or thro= ugh Fibre Channel. File-oriented storage controllers have supported the NFS and/or CIFS protocols, and have been attached directly to IP networks suc= h as Ethernet. The IP/Ethernet infrastructure offers compelling advantages for volume/block-oriented storage attachment compared to current approaches: * Increasing performance and reduced cost driven by Internet economics and "IP convergence" * Seamless conversion from local to wide area using IP routers * Emerging availability of "IP datatone" service from carriers, in preference to ATM or SONET or T-1, T-3 services * Protocols and middleware for management, security and QoS * Economies arising from the need to install and operate only single type of network The following applications for iSCSI are contemplated: * Local storage access, consolidation, clustering and pooling (as in the data center) * Remote disk access (as for a storage utility) * Local and remote synchronous and asynchronous mirroring between storage controllers * Local and remote backup and restore * Evolution with SCSI to support of emerging object-oriented storage model And the following connection topologies are contemplated: * Point-to-point direct connections * Dedicated storage LAN, consisting of one or more LAN segments * Shared LAN, carrying a mix of traditional LAN traffic plus storage traffic * LAN-to-WAN extension using IP routers or carrier-provided "IP Datatone" * Private networks and the public Internet The iSCSI standard will permit SCSI volume/block-oriented devices to be attached directly to IP networks such as Ethernet. The SCSI-3 command protocol (defined by the ANSI NCITS T10 committee) will be mapped to TCP. iSCSI is this mapping, and is analogous to (but not the same as) SCSI-FCP (aka "FCP"), which is the mapping of SCSI to Fibre Channel. Local-area storage networks will be built using Ethernet LAN switches. These networks may be dedicated to storage, or shared with traditional Ethernet uses, as determined by cost, performance, administration, and security considerations. In the local area, TCP's adaptive retransmissio= n timers will provide for automatic and rapid error detection and recovery, compared to alternative technologies. IP LAN-WAN routers will be used to extend the IP storage network to the w= ide area, permitting remote disk access (as for a storage utility), synchrono= us and asynchronous remote mirroring, and remote backup and restore (as for tape vaulting). In the WAN, TCP end-to-end will avoid the need for specialized equipment for protocol conversion, ensure data reliability, c= ope with network congestion, and automatically adapt retransmission strategie= s to WAN delays. The full realization of iSCSI will involve the following elements: (1) Completion of Requirements (this document) and Specification documents; = (2) Development of Ethernet storage NICs and related driver and protocol software; (3) Development of compatible storage controllers; and (4) The likely development of translating gateways to provide connectivity betwee= n the Ethernet storage network and the Fibre Channel and/or parallel-bus SC= SI domains. Products will initially be offered for Gigabit Ethernet attachment, with rapid migration to 10 GbE. For performance competitive with alternative SCSI transports, it will be necessary to implement the performance path o= f the full protocol stack in hardware. These new storage NICs will perform full-stack processing of a complete SCSI task, analogous to today's SCSI = and Fibre Channel HBAs. They typically also will support all host protocols that use TCP, including NFS, CIFS and HTTP. A key goal is not to require modifications to the current IP and Ethernet infrastructure to support storage traffic over TCP. Nevertheless, the performance and security requirements of storage will create opportunitie= s for improvement in security protocols and QoS implementations. The addit= ion of storage traffic to local- and wide-area internets (and even to the pub= lic Internet) may introduce increased requirements for traffic monitoring and engineering in those environments. It is contemplated that many organizations initially will choose to opera= te storage networks based on iSCSI that are independent of (isolated from) their current data networks except for secure routing of storage manageme= nt traffic. These organizations will benefit from the high performance/cost= of IP equipment and a unified management architecture, compared to alternati= ve means of building storage networks. As security and QoS evolve, it will become more reasonable to build combined networks with shared infrastructure; nevertheless, it is likely that sophisticated users will choose to keep their storage subnetworks isolated, for the best control o= f security and QoS. The proposed charter of the IETF IP SCSI Working Group (IPSWG) describes = the broad goal of mapping SCSI to IP. Within that broad charter, many transp= ort alternatives may be considered. Our initial work focuses on TCP, and thi= s Requirements document is restricted to that domain of interest. At the current time, we do not seek a more generic requirements statement that would justify the choice of TCP (or another protocol) as transport, since the merits of using TCP are readily evident to the working group participants. 2 Definitions Certain definitions are offered here, with references to the original document where applicable, in order to clarify the discussion of requirements. Throughout the text, use of defined terms is emphasized by producing them in bold face type. Definitions without references are the work of the authors and reviewers of this document. Logical Unit (LU): A target-resident entity that implements a device mode= l and executes SCSI commands sent by an application client [SAM-2, =A73.1.5= 0, p. 7]. Logical Unit Number (LUN): A 64-bit identifier for a logical unit [SAM-2, =A73.1.52, p. 7]. SCSI Device: A device that is connected to a service delivery subsystem = and supports an SCSI application protocol [SAM-2, =A73.1.78, p. 9]. Service Delivery Port (SDP): A device-resident interface used by the application client, device server, or task manager to enter and retrieve requests and responses from the service delivery subsystem. Synonymous w= ith port (SAM-2 =A73.1.61) [SAM-2, =A73.1.89, p. 9]. Target: An SCSI device that receives SCSI command and directs such comman= ds to one or more logical units for execution [SAM-2 =A73.1.97, p. 10]. Task: An object within the logical unit representing the work associated with a command or a group of linked commands [SAM-2, =A73.1.98, p. 10]. Transaction: A cooperative interaction between two objects, involving the exchange of information or the execution of some service by one object on behalf of the other [SAM-2, =A73.1.109, p. 10]. [A transaction seems to = be a small unit than a task.] 3 Requirements In the attached, actual requirements statements are flagged with [R]. Related discussion is flagged with [D]. The requirements are somewhat arbitrarily grouped into categories. This = is for convenience only. No semantic meaning is to be implied from the category names. 3.1 General [R] Support block storage IO over IP networks. 3.2 [D] Our initial approach uses SCSI for the block storage protocol, and TCP/IP for the network transport.Performance/Cost In general, iSCSI must allow implementations to equal or improve on the current state of the art for SCSI interconnects. [R] Low delay communication. [D] Conventional storage access is of a stop-and-wait or remote procedure call type. Applications typically employ very little pipelining of their storage accesses, and so storage access delay directl= y impacts performance. The delay imposed by current storage interconnects, including protocol processing, is generally in the range of 100 microseconds. The use of caching in storage controllers means that many storage accesses complete almost instantly, and so the delay of the interconnect can have a high relative impact on overall performance. [R] High bandwidth, bandwidth aggregation. [D] The bandwidth (transfer rate, MB/sec) supported by storage controllers is rapidly increasing, due to several factors: (1) Increase in disk spindle and controller performance; (2) Use of ever-larg= er caches, and improved caching algorithms; (3) Increased scale of storage controllers (number of supported spindles, speed of interconnects). Not only must the iSCSI provide for full utilization of available link bandwidth, it also must exploit parallelism (multiple connections) at the device interfaces and within the interconnect fabric. [R] Low CPU utilization, equal to or better than current technology. [D] For competitive performance, the iSCSI protocol must allow three key implementation choices to be realized: (1) iSCSI must mak= e it possible to build I/O adapters that handle an entire SCSI task, as alternative SCSI transport implementations do. (2) The protocol must per= mit "zero-copy" memory architectures, where the I/O adapter reads or writes h= ost memory exactly once per disk transaction. (3) The protocol must not impos= e complex operations on the host software, which would increase host instruction path length relative to alternatives. [R] Cost competitive with alternative storage network technologies. 3.3 SCSI [R] Collaboration with ANSI NCITS T10 (SCSI) [D] iSCSI is a new SCSI "transport" [SAM2]. Being the intersection of SCSI and TCP, iSCSI has potential impact on T10 as well a= s on IETF. However, a stated requirement (below) is that iSCSI shall have = no impact on T10 architecture or command sets. Collaboration with T10 will = be required to achieve this requirement. [D] Storage attachment to IP networks will engender an unprecedented potential for device sharing. This alone may impact future T10 work. [R] Supported SCSI Device types. iSCSI shall support all SCSI device typ= es. Our primary focus is on supporting "larger" devices: host computers and storage controllers (disk arrays, tape library controllers). [D] Supported SCSI Devices will typically have adequate memory to implement the TCP transport and required iSCSI session state, a= nd a cost structure that can support VLSI for full-stack protocol accelerati= on. Generally, a controller will be interposed between the iSCSI (typically Ethernet) connections and the drive interface (typically parallel SCSI or Fibre Channel). In the longer term, it will become feasible, due to the march of technology, to support iSCSI economically in disk spindle and ta= pe mechanism controllers. [R] Support SCSI SAM-2 architecture model. [D] It would be helpful to produce a document discussing iSCSI with reference to SAM-2. No promises. [R] Reliable Transport. The iSCSI mapping provides the SCSI-3 command la= yer with a reliable transport, equal to or greater in reliability than the parallel SCSI bus, and providing in-order delivery, as suggested by SAM-2. [D] See [SAM-2, p. 17.] "The function of the service delivery subsystem is to transport an error-free copy of the request or response between the sender and the receiver..." [SAM-2, p. 22] "The mann= er in which ordering constraints are established is implementation-specific. An implementation may choose to delegate this responsibility...to the service delivery port. In some cases, in-order delivery may be an intrin= sic property of the transport subsystem or a requirement established by the S= CSI protocol standard. =B6For convenience, the SCSI architecture model assum= es in-order delivery to be a property of the service delivery subsystem. Th= is assumption is made to simplify the description of behavior and does not constitute a requirement. [R] Support for SCSI Task Queuing. [D] SAM-2 defines task queuing, and so strictly speaking, we don't need to call this out specifically. However, task queuing is not widely implemented today; and it will increase in importance with WAN IP networks, given speed-of-light delays. We are particularly interested in supporting task queuing of pipelined remote backup and asynchronous disk mirroring [D] Just because iSCSI supports task queuing doesn't mean that the end SCSI node is required to do so also. Task queuing is an optional feature of SCSI.=20 [R] Compatible with all SCSI-3 command protocols [SPC-2, SBC, etc.]. The= re will be no requirement by T10 to modify the SCSI command documents. No modifications are required of the SCSI command layer implementation, exce= pt possibly to lengthen task timers to accommodate wide-area delays due to speed-of-light and switching. [D] Note the restriction to SCSI-3 command protocols. There are potential problems with gateways between iSCSI and SCSI-2 paral= lel bus devices. It may not be feasible to transport SCSI-2 commands over iSCSI. Gateways that wish to support older SCSI-2 devices may have to pr= oxy for those devices, using SCSI-3 commands. [R] Forward compatibility with future revisions of SCSI architecture and protocol. Attention to clean layering of protocols. [D] This is a difficult requirement to achieve in practice, since we cannot predict how SCSI will evolve. However, careful attention= to protocol layering principles will help ensure this result. [R] Gateways to parallel SCSI [ref.] and to SCSI-FCP [ref.]. It will be possible to construct "translating" gateways so that iSCSI hosts can talk= to SCSI-X devices; so that SCSI-X devices can talk to each other over a iSCS= I network; and so that SCSI-X hosts can talk to iSCSI devices (where SCSI-X refers to parallel SCSI, SCSI-FCP, or SCSI over any other transport). [D] This requirement is implied by support for SAM-2, but is worthy of emphasis. [D] These are true application protocol gateways, and not just bridge/routers. The different standards have only the SCSI-3 comman= d protocol layer in common. These gateways are not mere packet forwarders. We need to look into their remote proxy behavior. [D] Adequate liaison must be established with related standards bodies, principally ANSI T10 (SCSI). 3.4 iSCSI Session Layer [R] SCSI command, data, and response transactions occur in a TCP channel that is determined by the initiator, in advance of starting the task. [D] This requirement allows the initiator to assign the data transfer phase of a task to a given data transfer engine, at initiation o= f the task.=20 [R?] TCP channel (connection) allegiance. SCSI commands, data and status information for a given task shall flow within the same single TCP connection. [D] This is a stronger statement than the one above, and is left here as a potential requirement, mostly so that it will be clear tha= t the discussion topics below pertain to the notion of channel allegiance. [D] SAM-2 seems to require this channel allegiance: "A task involving one initiator-target pair shall not specify a third SCSI device= to participate in transmitting and receiving the remote procedure model elements for that task. Thus, an SMU initiator [e.g., a host computer] shall not create a task using one service delivery port with the expectat= ion that the data transfer or status return for that task would occur via a different service delivery port" [SAM-2, =A7 4.10.7, p.33]. Of course, interpretation of this clause depends on the definition of service delive= ry port. If a service delivery port is a TCP connection, then channel allegiance is pretty clearly required. But if a service delivery port is= an iSCSI session or an abstract target device, then the interpretation of th= is clause is less clear. [D]We have found a number of other possible virtues in channel allegiance: (1) It supports multiple instances of the TCP protoco= l engine being controlled by a single iSCSI session layer; (2) Failure of a TCP connection will affect only a subset of the extant tasks (those that = use the failed connection); (3) All TCP connections are used in exactly the s= ame manner; (4) There is no need to have more than one IP port defined for th= e iSCSI protocol, which is firewall-friendly. [R] Command striping (load balancing) across multiple host and device interfaces. It shall be possible to utilize multiple concurrent paths between hosts and devices for the purpose of load balancing. [D] Load balancing refers to concurrent tasks from a single initiator. There is no ordering constraint among these tasks. We aim to distribute these tasks (commands and their related data and status) acros= s multiple host ports, links, switch ports and device ports, in order to achieve aggregate performance equal to a multiple of single link performance. [R] Command ordering for tape backup and asynchronous remote mirroring. = It must be possible to pipeline commands to a device, and to have them execu= ted in order by that device, as prescribed by SAM-2. [D] Ordering can be maintained by allowing each command to complete before issuing the next. But that means there is no pipelining. For tape backup in the local area, this may be adequate, as the tape controller buffer can be made sufficiently large to cover the lower duty cycle of data transfers, and LAN speeds are fast enough to burst-fill the buffer. But in the wide area, a method of pipelining commands and respon= ses is needed if the slower WAN link is to be filled continuously with data. [D] This brings up an issue, if commands are sent in different TCP channels. Although a single TCP channel delivers an ordere= d byte stream, there is no ordering constraint between TCP channels. So command striping across TCP channels will result in the commands possibly being executed out of order, unless the commands themselves are numbered, and can be put back into order. SCSI does not provide a means for puttin= g commands back in order, but requires that functionality of the "transport= ". [D] We contemplate bonding multiple TCP channels (connections) into an iSCSI session for the purpose of ordered command striping. A command reference number (CRN) will allow iSCSI to receive commands in order from the initiator SCSI command layer, and deliver them= in order to its peer command layer in the target. Note that this mechanism = can be employed at all times, because delivering commands in order never hurt= s, even if the SCSI layer imposes no ordering constraints among them. This = is the safest route, in fact, as it upholds the SAM-2 expectation of in-orde= r delivery. We expect the ability to support a session consisting of multi= ple channels to be optional. It will be possible for a target to refuse to a= dd a channel to a session. [R] Recovery at the session layer. The session layer specification shall explicitly address recovery at the session layer (from a failed TCP connection, for example). [D] TCP will recover from data loss due to bit errors or congestion. But what if a TCP connection fails (hangs)? The specificati= on needs to address this issue. 3.5 Transport, Network and Link [R] Works with existing installed Ethernet and IP WAN infrastructure. iS= CSI should not require any modification to Ethernet hubs, switches or WAN routers to achieve minimum acceptable performance, QoS and security. [D] Using existing and off-the-shelf technology will allow iSCSI to fully leverage the cost, performance and rapid improvement of widely-deployed IP LAN and WAN technologies. Therefore, iSCSI cannot require the installation of special, non-standard features in the underly= ing technology. However, it may be desirable to apply certain optimizations that will enhance storage protocol performance, or the performance of oth= er protocols in the presence of the storage protocol. [R] Joint operation (coexistence) with other IP protocols. iSCSI shall n= ot preclude concurrent operation with any of the protocols in the IP protoco= l suite, and shall be a good Internet citizen. [D] Many organizations will choose to operate iSCSI storage networks as separate networks from their traditional data networks, by a router only for management traffic. This approach delivers the most manageable environment from a performance and security perspective, and i= s analogous to today's separate Fibre Channel storage networks, except for = the obvious benefits that derive from using LAN technologies. On the other hand, some organizations will favor using fewer networks, and mixing stor= age with other types of traffic. This practice will be more prevalent in the wide-area, where dedicated storage links exact a high price. For these reasons, graceful co-existence is required. Over time, improved support = for the QoS and security features inherent in IP and Ethernet protocols will make it more and more reasonable to combine storage with other types of network traffic. [D] When storage is transported over the wider Internet, it must be done in a way that respects TCP's bandwidth management and congestion avoidance algorithms. This is one of the reasons for selectin= g TCP as the transport. We feel that TCP itself is a good Internet citizen= , and our best chance for compatibility. [R] Uses TCP/IP. iSCSI is a protocol mapping from SCSI to TCP. [D] While we don't preclude consideration of alternative transports, we have focused our attention on TCP. Given wide-area functio= ns in a storage controller, and the resulting need for TCP support, inclusio= n of an alternative local-area transport may imply an increment of cost, no= t a cost savings; and it certainly represents an increment of complexity. [R] Link Independent. iSCSI is defined for all IP networks, and is link-independent. All IP-compatible LAN and WAN links are supported. Specifically, there are no dependencies on Ethernet. [D] We may nevertheless want to benefit from certain link capabilities like Ethernet port aggregation and PPP multi-link. But the spec should not depend on these capabilities for its viability. [R] LAN, MAN and WAN -capable. SCSI Devices that implement iSCSI will be capable of communicating with similarly-equipped devices and host compute= rs over any IP network, whether local, metropolitan, or wide-area in scale. [D] iSCSI is used not only for local area disk block access and tape operations. It also is used for remote disk access (as for a storage utility), remote disk mirroring, and remote backup and restore (a= s for tape vaulting). Using TCP in the iSCSI end nodes means that the protocol is scalable from the local to the wide area. [R] Handles high bandwidth * delay fabrics. [D] This requirement must be clarified further, as an extension of the WAN requirement. Consider that the TCP pipe at 10 Gbps = * 200 msec holds 250 MB. Will TCP sequence counts be up to this, or will t= hey wrap too frequently? [R?] Framing. Some method of framing iSCSI protocol units within the TCP stream must be defined. [D] We are unresolved as to whether this is a requirment. [D] The conventional way to do this is simply by parsing from the beginning of the stream, and never making a mistake. Is this sufficient? Or, should we use some other means such as byte stuffing or = use of the push bit? Related, how do we ensure that data actually is transmitted, and doesn't languish in a TCP buffer somewhere? [D] As an example of the problem: suppose a TCP segment is lost due to congestion, and it happens to contain an iSCSI header. At th= at point, stream synchronization will be lost, as we cannot find the next iS= CSI header. Following the example above, we're obliged to catch 250 MB of da= ta before we can resume iSCSI operation. If we could find the next iSCSI header, we could implement an optimization (non-traditional for TCP implementations) that would require us only to catch a single iSCSI message's-worth of data. Subsequent iSCSI messages could be decoded, and the data put where it belongs (even though command ordering constraints would preclude acting upon the data until the missing SCSI command is received and inspected for ordering constraints). [R] It has been noted that a remote DMA option for TCP possibly could provide the desired framing. [R?] Error detection. Stronger CRC. [D] The TCP checksum is rather weak as error detection goes. It is supported by the link layer check codes (CRC-32 for Ethernet). Is that sufficient? We don't have strong protection from re-assembly errors. Routers modify the frame and recompute the CRC. Even switches recompute CRCs (for VLAN shifting), although good implementations do the CRC recomputation incrementally. The TCP checksum is our only end-to-end protection. If the TCP checksum is not sufficient, do we introduce some kind of check on the SCSI data buffers by the iSCSI layer? Possibilities= : byte count, CRC. Whatever we do, it must be possible to compute these ch= eck codes on the fly, as data is transferred from NIC to memory, without maki= ng a second pass over the data once it is in memory. [D] We are considering using the IPsec messsage digest function for this purpose. It's already defined, and it could be used as= a check code (only) using well-known keys; hence, without introducing the k= ey distribution problem. Using IPsec in conjunction with TCP would not requ= ire a modification to TCP. A concern about using the IPsec message digest function is that it may be more difficult to compute at high speed than a simpler CRC. [D] But is TCP truly an end-to-end protocol? The notion of an end-to-end error check is that it and the data it protects pass throug= h the network unchanged, but possibly subject to errors while on a link or = in a memory. At the receiving end node, checking the CRC verifies the corre= ct receipt of data. In some cases, such as the use of a proxy server or perhaps a NAT, the connection is not end-to-end, but it the concatenation= of two end-to-end connections. In these cases, the iSCSI PDU (message) may = be a better candidate for CRC protection. [R] Selective TCP retransmission. [D] Given the long delays in the WAN, using TCP selective retransmission must be supported by iSCSI, in order to minimize the bandwidth impact of retransmission. [R] Firewall friendly. The protocol's use of IP addressing and TCP port numbers should be firewall friendly. [D] This probably means that all connection requests should be addressed a specific, well-known TCP port. That way, firewalls can filter based on source and destination IP addresses, and destination (target) port number. The source (initiator) port number also should be well-known for the initial TCP connection. Additional TCP connections wo= uld require different source port numbers (for uniqueness), but could be open= ed after a security dialogue on the control channel. [R] Possible to move data directly from end-to-end, without having retransmission buffers in the middle. [D] This is an important implementation detail. In an iSCSI system, each of the end nodes (for example host computer and storage controller) has ample memory; but the intervening nodes (NIC, switches) d= o not. We contemplate a WAN-scale retransmission requirementof 25 MB (1 Gb= ps) or 250 MB (10 Gbps, see earlier footnote). Therefore, it must not be necessary for thee intervening nodes to buffer data. [R] Conservative in use of TCP and session-layer connections. The number required should not scale directly with the number of supported LUs. [D] TCP connection and iSCSI session state is fairly expensive in terms of memory consumed both on- and off-chip (we contempla= te VLSI implementation). At a minimum, we seek to support only the number o= f connections required to achieve required bandwidth and delay characterist= ics between hosts and storage controllers. [R] Compatible with both IPv4 and IPv6. [D] We need to add a literal format for IPv6 addresses in our domain name field in urls. 3.6 Naming [R] Naming. Whenever possible, iSCSI shall support the naming architectu= re of SAM-2. Deviations and uncertainties will be made explicit, and comment/resolution invited. [D] It may be necessary to provide a unique naming scheme for SCSI LUs. Fibre Channel does so using WWNs. There's some indication that the T10 Security work will complicate this problem through LUN renumbering. The manner of determining a unique, worldwide, unchanging L= U name must be determined. We will attempt to make use of SPC-2 provisions for LU Identifiers (Vital product data page 83h [SPC-2, p. 203] ). [D] We need to resolve whether the notion of "target" is relevant to iSCSI. Does an iSCSI session connect to a target? Can it subsequently address multiple targets and LUs or just a bunch of LUs? [D] We need to provide an understanding of just what a Service Delivery Port (SDP) is in the iSCSI context. Is it an IP endpoin= t? A session endpoint? A virtual device (target) that a session can be connected to? SAM-2 seems to equate an SDP with a target address, "...th= e application clients in each initiator have the ability to discover that logical units in the SMU target are accessible via multiple Target Identifiers (service delivery ports)..." [SAM-2, pp. 12-13] [R] URLs. It shall be possible to name SCSI devices and possibly LUs usi= ng a URL syntax. These names shall be global (uniform) and suitable for passing as handles between SCSI application clients. [R] Domain names. The Domain Name Service (DNS) shall be used to resolve the portion of the url to one, or multiple IP addresses. When= a hostname resolves to multiple addresses, these addresses shall be equival= ent for functional (possibly not performance) purposes. [D] This means that the addresses can be used interchangeably as long as we don't care about performance. For example, the same set of SCSI targets and/or LUs (tbd) must be accessible from eac= h of these addresses. [R] Deal with the complications of the new SCSI security architecture [99-245r8]. [D] Pay attention to the proxy naming architecture defined by the new security model. In this new model, SCSI Logical Unit Numbers (LUNs) can be mapped in a manner that gives each host (more correctly, ea= ch AccessID) a unique LU map. Thus, a given LU within a target may be addressed by different LUNs. [R] Support SCSI 3rd-party operations. [D] The key issue here relates to the naming architecture for SCSI LUs. We need to determine a method of passing a name or handle between parties=20 3.7 Security [R] Authentication. At a minimum, iSCSI parties shall participate in a simple principals authentication protocol. This protocol shall involve a minimum of encryption and no special hardware for implementation. [R] Bootstrapping. It shall be possible to negotiate higher levels of security than the minimum, technique to be defined. [R] Data encryption. Data encryption shall be optional, but when implemented, shall be done in a manner prescribed by iSCSI, by reference = to other standards. [R] Compatible with IP protocol suite security protocols for the present = and future. [D] We anticipate incorporating IPsec (host-to-host) and SSL/TSL (TCP connection) security into the iSCSI protocol by reference, a= nd as options. Adherence to good layering will ensure (as much as possible) that future security developments at the IP and TCP layers can be utilize= d by iSCSI. [R] Permits use of firewall for security screening. [D] It's important to allow a firewall to be used to offload authentication from the end node. This is a possible means of defending against Denial of Service (DoS) assaults, from a less-trusted area of the network. We assume that the firewall(s) have much greater processing pow= er for dismissing bogus connection requests than do the end nodes. 3.8 Topology Discovery [D] OK, we said we'd leave this for later. But why not open the discussi= on? [R] iSCSI shall have no impact on the use of conventional IP network discovery techniques. [D] IP discovery techniques are well-evolved. Various network management platforms have ways of discovering IP addresses, such = a mining router caches. We assume that these techniques will be used, and will find all of the IP end points that contain iSCSI nodes. [R] iSCSI shall provide some means of determining that a discovered IP en= d point in fact is an iSCSI node. [D] This requirement is just a placeholder. Generally in IP discovery, there is some way of determining the type of the discovered device. Possibly this is due to the presence of the SNMP protocol and specific MIB variables. In this case, SNMP is the bootstrap protocol. Alternatively, one could probe various TCP port numbers to determine if there exists a higher-level protocol at each port (the port number would tell you which protocol). To be determined. But in any case, some means= is needed to determine that an iSCSI entity is present at an IP end point. [R] When a device supports multiple IP end points, some means of determin= ing the IP connection topology is needed. [D] A device may support multiple end points, yet it may not be reasonable to bind any combination of the end points together into an iSCSI session. For example, a port controller (aka channel group) card m= ay have four ports that can be bound together. The storage controller may support four of these port controllers, yet not allow the binding togethe= r into a session of TCP connections made on different port controllers. [D] A really simple solution to this problem would be to define a means of describing port topology, and provide for reading that description either from a MIB or directly from the iSCSI layer (with a command). [R] SCSI protocol-dependent techniques shall be use for further discovery beyond the iSCSI layer. [D] Discovery is a complex process. But SCSI provides specific hooks for doing the work, and all we need to do is transport the commands associated with this process. Generally the SCSI discovery proc= ess involves using the Report LUNs command to determine which LUs are addressable at a given service delivery port. Subsequently, the true identity of each LU (ie, name) is discovered by reading Vital product dat= a page 83h. By comparing LU IDs, the discovery process can find that a giv= en LU is accessible through multiple paths. [D] We need only verify that this SCSI mechanism is sufficient. Hopefully, we will not need to augment SCSI at the iSCSI lay= er. 3.9 Management [R] IP-based management protocols. It shall be possible (but not require= d) to use IP-based management protocols such as SNMP and RMI in conjunction with iSCSI. However, the present effort will not define the management architecture for iSCSI networks. [R] SCSI management protocols. It shall be possible to use SCSI commands for management (eg, SCSI Enclosure Services, SES commands) to manage iSCS= I devices. 3.10 Interoperability [R] It must be possible for hosts and devices that implement only those features specified in the RFC to interoperate. [R] Software implementation is possible using conventional TCP/IP protoco= l stack. [D] Although some low-performance products may contemplate an all-software implementation, we expect the majority of iSCSI products = to employ hardware protocol acceleration. This requirement really is here t= o solve two problems (1) Proof of interoperability, by compatibility with extant TCP implementations; (2) Prototyping, where the iSCSI protocol is first implemented in software using these conventional stacks. These prototypes will likely become the early reference implementations. 4 References [SAM-2] ANSI NCITS. Weber, Ralph O., editor. SCSI Architecture Model -2 (SAM-2). T10 Project 1157-D. rev 13, 22 Mar 2000. [SPC-2] ANSI NCITS. Weber, Ralph O., editor. SCSI Primary Commands - 2 (SPC-2). T10 Project 1236-D. rev 18, 21 May 2000. [CAM-3] ANSI NCITS. Dallas, William D., editor. Information Technology = - Common Access Method - 3 (CAM-3)). X3T10 Project 990D. rev 3, 16 Mar 19= 98. [99-245r8] Hafner, Jim. A Detailed Proposal for Access Controls. T10/99-245 revision 8, 26 Apr 2000. ------_=_NextPart_000_01BFE7B9.10EC4970 Content-Type: application/octet-stream; name="Haagens, Randy (E-mail).vcf" Content-Disposition: attachment; filename="Haagens, Randy (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000607T194543Z END:VCARD ------_=_NextPart_000_01BFE7B9.10EC4970-- From owner-ips@ECE.cmu.edu Fri Jul 07 14:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13488; 7 Jul 2000 10:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA08774 for ips-outgoing; Fri, 7 Jul 2000 07:46:16 -0400 (EDT) From: owner-ips@ece.cmu.edu X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-ID: Date: Fri, 7 Jul 2000 09:25:16 +0300 Subject: Re: draft q. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Numbering fiels as many others - I've attempted to put them at the same offset whenever they are used. Julo psarkar@almaden.ibm.com on 07/07/2000 02:03:50 Please respond to psarkar@almaden.ibm.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: draft q. In the latest draft p35, in the picture of the login command layout, byte offset 24 follows 16 - should it be 20? prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose From owner-ips@ECE.cmu.edu Fri Jul 07 14:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13485; 7 Jul 2000 10:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA08759 for ips-outgoing; Fri, 7 Jul 2000 07:45:57 -0400 (EDT) From: owner-ips@ece.cmu.edu X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-ID: Date: Fri, 7 Jul 2000 08:59:45 +0300 Subject: Re: Third party naming vis-a-vis iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, Did you have a chance to took at the current mapping commands I've set is iSCSI? It would be nice if they become part of SPC and fix the hole of "protocol specific" to general purpose addresses. In fact the result in copy managers having a de-facto local DNS for mapping for the length of the session and that is more effective than having them handle all the time mapping. Regards, Julo hafner@almaden.ibm.com on 06/07/2000 19:56:45 Please respond to hafner@almaden.ibm.com To: "IPS (E-mail)" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Third party naming vis-a-vis iSCSI Here's my promised suggestion for third-party naming. There are two approaches, one fully integrated into SCSI (and so independent of iSCSI, though iSCSI can best take advantage of it). The other approach is more attuned to iSCSI, though could be used in other (probably future) transports. The real problem in this space is not in the higher application layers which can do naming in anyway they want, so long as there is an application that can translate from one form to another. (So a URL can get "resolved" by different application layers into an ethernet MAC and IP port, etc.) The problem is that an arbitrary naming scheme doesn't map well into existing SCSI paramter data where third party addresses are used (e.g., third party reservations, RAID commands like XDWRITE, extended copy, etc.). The biggest problem in that space is the fact that SCSI has only (at the moment) set aside fixed sized fields for this purpose (and we know iSCSI is going to need much longer names). In some cases, this field is 8 bytes (e.g., third party reservations) and in others 16 bytes (bytes 12-27 in extended copy target descriptors; I'm referring here to the identifier of the "target device" and not the logical unit within that device). The first suggestion, then is to use the existing fixed field (and only 8 bytes of that) of third party addressing as an "alias" field. The requesting initiator will (through a mechanism described later) define for the target a translation mapping of this alias to the transport specific identifier of the target (e.g., the portion of the url, the FC WWN, the parallel SCSI address). A flag bit or field (TBD for each usage) will indicate that this is an alias field and not a specific address identifier. Using aliases in this way, we don't need to change the size of the fields currently dedicated to third party addresses and we don't need to change their existing usage. Here are the two approaches for an initiator to define the alias<->thirdparty mapping. Note, in all cases, (without a completely different protocol) the mapping would be initiator specific, not a global aliasing. 1) Within iSCSI, a Text command could define the mapping. Advantages: a) you'd only need to change SPC-2 to allow for alias-type usage, and leave the mapping to "the transport". b) it would apply to all third-party addressing contexts. Disadvantages: a) there's a disassociation between the actual mapping and the use of the mapping (that is, state must be maintained) and so all the issues about state (persistence, etc) have to be addressed. b) it's not clear how other transports could take advantage of this. 2)an additional "chunk" of the parameter data could be used to define the alias. For example, in extended copy, an additional "page" of parameter data can define the mapping for that instance of the command. Advantages: a) can be used in all transports and future transports b) is a general extensible mechanism c) has state only for the context of the command itself d) can handle whatever size of device identifier you want (e.g., arbitrarily long ) - that's because you can define the parameter data in this way. Disadvantages a) would require more changes to SPC-2 (both alias usage and the definition of mapping), and would (might?) affect any command where third-party addressing is used. As for how logical units are referenced within a target, the existing mechanisms of LUN and Proxy Token (from access controls) are sufficient. In all cases, the harder part is finding the device. SCSI already knows how to find the logical unit within the device. But I don't see a fundamental problem with adding this logical unit identifier (in whatever form you like, LUN, Proxy Token, EVPD data) as part of the alias mapping. The subtle issue is that in some cases there is no logical unit to reference (e.g., third party reservations reference an initiator, not a target). Political comment: After having worked with the T10 guys over the last year or so, I find them very amenable to additions/changes to SCSI so long as it doesn't break anything and that it meets someones perceived need. I think the issues raised here meet both those requirements. So, I wouldn't be reluctant (if I were more active in this iSCSI thing) to take specific proposals to T10 to enhance what iSCSI can do. Jim Hafner From owner-ips@ECE.cmu.edu Fri Jul 07 14:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13498; 7 Jul 2000 10:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA08736 for ips-outgoing; Fri, 7 Jul 2000 07:44:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-ID: <396526B0.A2942C39@ebay.sun.com> Date: Thu, 06 Jul 2000 17:39:12 -0700 From: David Robinson Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.73 [en] (Win95; U) X-Accept-Language: en MIME-Version: 1.0 To: Jim McGrath CC: "'VonStamwitz, Paul'" , ips@ece.cmu.edu, "'julian_satran@il.ibm.com'" Subject: Re: Security Considerations References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim McGrath wrote: > > For reference, what security is present today for NAS class products (e.g. > those typically using a higher level protocol like NFS rather than SCSI)? > > My impression (which I admit could be quite mistaken) is that we may be on a > path of requiring a higher level of security for iSCSI than is warranted by > alternative protocols. Of course, enhancing security has its value, but > since SCSI starts off with essentially no security, iSCSI is a poor protocol > upon which to require lots of security. The benchmark that iSCSI will have to meet is what is being done in the NFSv4 WG. Using the ONCRPC WG mechanisms based on GSSAPI and RPCSEC_GSS they provide authentication, integrity, and privacy using one of two manditory to implement mechanisms, Kerberos V5 and LIPKEY. I suspect that if iSCSI doesn't address security to provide similar capabilities it will not pass muster. Of course the security should always be able to be negotiated to the desired levels. -David From owner-ips@ECE.cmu.edu Fri Jul 07 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13779; 7 Jul 2000 11:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA12814 for ips-outgoing; Fri, 7 Jul 2000 10:21:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-ID: <3965E568.6A78C472@gis.net> Date: Fri, 07 Jul 2000 10:12:57 -0400 From: Bill Main Reply-To: wmain@gis.net X-Mailer: Mozilla 4.5 [en]C-CCK-MCD compaq (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ipSCSI list Subject: iSCSI has already been done before, sort of... Content-Type: text/plain; charset=iso-8859-1 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id KAB12814 Folks- If this seems late in the cycle, I apologize but I just recently joined this project and I realize there is an IETF deadline looming but.... How does the adage go? Those who fail to learn from history are doomed to repeat it. That I believe applies here. Many protocols have been put forward or mentioned but one seems distinctly absent from this discussion. Right up front, this is **NOT** proposed as a protocol for adoption but it does offer to us some valuable insights in how to handle some of the questions at hand. The important thing is that it does everything iSCSI is trying to do and it has been running for years at performance levels that iSCSI will struggle to match. Here are some of its features: - Been around a long time (not well known even though has an ANSI spec) - Specs revised and improved 1986, 92, 95 and latest twik to spec in 97 - Up and running and supported by IBM, Compaq(Digital), NEC, Cray and SGI & probably others. - Connects large storage farms to hosts over switched network - Deals well with congestion delays - Handles tapes - works with all common unix host based file systems. - priority classifications for what now is called QoS. - adaptable block size on transmit - allows coalescing of data for transfer (sorts transmit queue to optimize transfers) - Designed and built to function on full duplex 100 MB/s networks albeit LAN only. - Optional multiple path capability for load balancing or striping for both hosts and targets. - Optional out of band command and response channel (usually Ethernet). - Header layout done so receiver has a priori knowledge of pay load and can segregate address data such that payload goes to page flip memory (Does exactly what RDMA wants to do). - Has a transmit credit system. - Sports a well thought out error and recovery system (this could be very useful) - Handles partition, spanning and volume work. - Performance numbers are commonly in the range of 85MB/s raw and 50 MB/s with a file system (WRITE) on single path with in line command and response using 8KB file blocks. The 50 is a file system limitation regardless of storage system class. - relatively easy to implement, only about 1.5 man-years to get it designed, coded and working into production envirnoment - has stages of operation that are very SMP (or ASIC) friendly It is called IPI-3 and runs currently only over Hippi networks which are as close to being a SAN as any FC layout. Specs are available from the T11 group (http://www.t11.org/index.html) Grab the IPI-3DR and IPI-3TR versions (disk and tape respectively). Having been the original architect on one implementation of IPI-3, there is a lot of knowledge in there that has been stepwise improved on over the 14 years of its existence, but it does exactly what iSCSI is trying to do. Now the bad news: Does not use IP or TCP but IPI-3 is very efficient. Note though, the Hippi-PH can be replaced by IP if one really wanted to without much effort. First, from my perspective it appears that iSCSI is a re-invention of the wheel =96 Not that a redesigned wheel can=92t be made better. But it always seemed better to study the original while doing it. Second, when you pull the specs, you will notice that they are weighty. Having designed an implementation of a network disk transfer system (ala iSCSI) for IPI-3, I assure you there is very little excess in there. Eventually every issue that generated paragraphs in the IPI-3 spec will show up in the iSCSI implementations as well. Regards, Bill Main From owner-ips@ECE.cmu.edu Fri Jul 07 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14473; 7 Jul 2000 15:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA23466 for ips-outgoing; Fri, 7 Jul 2000 14:21:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA23462 for ; Fri, 7 Jul 2000 14:21:32 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id OAA10684; Fri, 7 Jul 2000 14:21:32 -0400 Message-Id: <200007071821.OAA10684@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Date: Fri, 07 Jul 2000 14:21:32 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA12336 for ; Fri, 7 Jul 2000 14:05:39 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id MAA56816 for ; Fri, 7 Jul 2000 12:11:06 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256915.0063E3B5 ; Fri, 7 Jul 2000 12:11:03 -0600 X-Lotus-FromDomain: IBMUS To: ips@ece.cmu.edu Message-ID: <87256915.00634A02.00@d53mta03h.boulder.ibm.com> Date: Fri, 7 Jul 2000 11:04:05 -0700 Subject: clarification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline 3.2.5. Expected Data Transfer Length The Expected Data Transfer Length field states the number of bytes of data that the initiator expects will be sent for this (READ or WRITE) SCSI operation in SCSI Data packets. For a WRITE operation, the initiator uses this field to specify the number of bytes of data it expects to transfer for this operation (not counting data headers). For a READ operation, the initiator uses this field to specify the number of bytes of data it expects the target to transfer to the initiator (not counting data headers). If no data will be transferred in SCSI Data packets or Immediate data for this SCSI operation, this field should be set to 0. The first sentence and last sentence are mutually confusing. I think the Exepcted Data Transfer Length field should states the number of bytes to be sent in SCSI Data packets *or immediate data*. Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose From owner-ips@ECE.cmu.edu Fri Jul 07 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14471; 7 Jul 2000 15:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA24612 for ips-outgoing; Fri, 7 Jul 2000 14:50:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA24608 for ; Fri, 7 Jul 2000 14:50:38 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id OAA12888; Fri, 7 Jul 2000 14:50:38 -0400 Message-Id: <200007071850.OAA12888@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Date: Fri, 07 Jul 2000 14:50:38 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk Date: Fri, 07 Jul 2000 11:40:58 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: clarification References: <200007071821.OAA10684@yogi.ece.cmu.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit > From: psarkar@almaden.ibm.com > Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) > by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA12336 > for ; Fri, 7 Jul 2000 14:05:39 -0400 > Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) > by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id MAA56816 > for ; Fri, 7 Jul 2000 12:11:06 -0600 > Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256915.0063E3B5 ; Fri, 7 Jul 2000 12:11:03 -0600 > X-Lotus-FromDomain: IBMUS > To: ips@ece.cmu.edu > Message-ID: <87256915.00634A02.00@d53mta03h.boulder.ibm.com> > Date: Fri, 7 Jul 2000 11:04:05 -0700 > Subject: clarification > Mime-Version: 1.0 > Content-type: text/plain; charset=us-ascii > Content-Disposition: inline > > 3.2.5. Expected Data Transfer Length > > The Expected Data Transfer Length field states the number of bytes > of data that the initiator expects will be sent for this (READ or > WRITE) SCSI operation in SCSI Data packets. For a WRITE operation, > the initiator uses this field to specify the number of bytes of > data it expects to transfer for this operation (not counting data > headers). For a READ operation, the initiator uses this field to > specify the number of bytes of data it expects the target to > transfer to the initiator (not counting data headers). If no data > will be transferred in SCSI Data packets or Immediate data for this > SCSI operation, this field should be set to 0. > > > The first sentence and last sentence are mutually confusing. I think the > Exepcted Data Transfer Length field should > states the number of bytes to be sent in SCSI Data packets *or immediate > data*. > > Prasenjit Sarkar > Research Staff Member > IBM Almaden Research > San Jose I agree. I think the last sentence should be deleted entirely. Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Fri Jul 07 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14475; 7 Jul 2000 15:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA23388 for ips-outgoing; Fri, 7 Jul 2000 14:19:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA23383 for ; Fri, 7 Jul 2000 14:19:11 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id OAA10554; Fri, 7 Jul 2000 14:19:11 -0400 Message-Id: <200007071819.OAA10554@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Date: Fri, 07 Jul 2000 14:19:11 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk Sender: plabat@cup.hp.com Message-ID: <396616B2.84603BDD@hp.com> Date: Fri, 07 Jul 2000 10:43:14 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Recovery from a dropped TCP connexion Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hello, In the last teleconf it has been stated that : -> in case of a droped TCP connexion, a command that was on the flight must be replayed on a new TCP connexion going through the same service delivery port. This, because SAM-2 requires that all that concern one command must flow through the same delivery port. But, because the recovery is done at the command level, why not abort this command on the target (the target will abort/cancel locally all the commands on the flight for the dropped TCP connexion), and replay the command on another link (other service delivery port). As the command will be totally restarted we don't need some intermediate command state that would need to be maintained per port. We can cleanup what was ongoing for this command on the initial port and restart drom scratch on the alternate port. It doesn't hurt SAM-2 because the replay of the command is a whole new command. Of course iSCSI must change the LUN in the headers to take into account the new service delivery port. The big advantage is that it can speed up the recovery. 1) It is likely that if the TCP connexion dropped through a service delivery port, the attempt to open a new one through the same SDP will fail. 2) Replaying the command through an alternate link is faster than waiting for the establishment of new TCP connexion. Regards, Pierre From owner-ips@ECE.cmu.edu Fri Jul 07 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14485; 7 Jul 2000 15:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA24762 for ips-outgoing; Fri, 7 Jul 2000 14:53:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f From: Black_David@emc.com Message-ID: <0F31E5C394DAD311B60C00E029101A070148FADC@corpmx9.isus.emc.com> To: wmain@gis.net, ips@ece.cmu.edu Subject: RE: iSCSI has already been done before, sort of... Date: Fri, 7 Jul 2000 14:48:52 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk It's not clear to me what the point of this message is, since the author says: > Right up front, this is **NOT** proposed as a protocol for adoption but it > does offer to us some valuable insights in how to handle some of the > questions at hand. but then has very little to say in terms of exactly what the insights are. The bullet list of IPI-3 features isn't particularly helpful; is the intent to put them forward as requirements? Could I suggest writing an Internet-Draft on the topic of what the author thinks is wrong with iSCSI, why, and how to fix it as a productive alternative to making disparaging comments on the efforts of others? --David (future WG co-chair) --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Bill Main [SMTP:wmain@gis.net] > Sent: Friday, July 07, 2000 10:13 AM > To: ipSCSI list > Subject: iSCSI has already been done before, sort of... > > Folks- > If this seems late in the cycle, I apologize but I just recently > joined this project and I realize there is an IETF deadline looming > but.... > > How does the adage go? Those who fail to learn from history are > doomed to repeat it. > > That I believe applies here. Many protocols have been put forward or > mentioned but one seems distinctly absent from this discussion. Right up > front, this is **NOT** proposed as a protocol for adoption but it does > offer to us some valuable insights in how to handle some of the > questions at hand. > > The important thing is that it does everything iSCSI is trying to do and > it has been running for years at performance levels that iSCSI will > struggle to match. > > Here are some of its features: > > - Been around a long time (not well known even though has an ANSI spec) > - Specs revised and improved 1986, 92, 95 and latest twik to spec in 97 > - Up and running and supported by IBM, Compaq(Digital), NEC, Cray and > SGI & probably others. > - Connects large storage farms to hosts over switched network > - Deals well with congestion delays > - Handles tapes > - works with all common unix host based file systems. > - priority classifications for what now is called QoS. > - adaptable block size on transmit > - allows coalescing of data for transfer (sorts transmit queue to > optimize transfers) > - Designed and built to function on full duplex 100 MB/s networks albeit > LAN only. > - Optional multiple path capability for load balancing or striping for > both hosts and targets. > - Optional out of band command and response channel (usually Ethernet). > - Header layout done so receiver has a priori knowledge of pay load and > can segregate address data such that payload goes to page flip memory > (Does exactly what RDMA wants to do). > - Has a transmit credit system. > - Sports a well thought out error and recovery system (this could be > very useful) > - Handles partition, spanning and volume work. > - Performance numbers are commonly in the range of 85MB/s raw and 50 > MB/s with a file system (WRITE) on single path with in line command and > response using 8KB file blocks. The 50 is a file system limitation > regardless of storage system class. > - relatively easy to implement, only about 1.5 man-years to get it > designed, coded and working into production envirnoment > - has stages of operation that are very SMP (or ASIC) friendly > > > It is called IPI-3 and runs currently only over Hippi networks which are > as close to being a SAN as any FC layout. Specs are available from the > T11 group (http://www.t11.org/index.html) > Grab the IPI-3DR and IPI-3TR versions (disk and tape respectively). > > Having been the original architect on one implementation of IPI-3, > there is a lot of knowledge in there that has been stepwise improved on > over the 14 years of its existence, but it does exactly what iSCSI is > trying to do. > > Now the bad news: Does not use IP or TCP but IPI-3 is very efficient. > Note though, the Hippi-PH can be replaced by IP if one really wanted to > without much effort. > > First, from my perspective it appears that iSCSI is a re-invention of > the wheel - Not that a redesigned wheel can't be made better. But it > always seemed better to study the original while doing it. > > Second, when you pull the specs, you will notice that they are weighty. > Having designed an implementation of a network disk transfer system (ala > iSCSI) for IPI-3, I assure you there is very little excess in there. > Eventually every issue that generated paragraphs in the IPI-3 spec will > show up in the iSCSI implementations as well. > > Regards, > Bill Main > From owner-ips@ECE.cmu.edu Fri Jul 07 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14489; 7 Jul 2000 15:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA24545 for ips-outgoing; Fri, 7 Jul 2000 14:49:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA24535 for ; Fri, 7 Jul 2000 14:49:12 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id OAA12781; Fri, 7 Jul 2000 14:49:12 -0400 Message-Id: <200007071849.OAA12781@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Date: Fri, 07 Jul 2000 14:49:12 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk Message-ID: <39662389.91F0D45@agilent.com> Date: Fri, 07 Jul 2000 11:38:01 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu, Pierre Labat Subject: Re: Recovery from a dropped TCP connexion References: <200007071819.OAA10554@yogi.ece.cmu.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit > Sender: plabat@cup.hp.com > Message-ID: <396616B2.84603BDD@hp.com> > Date: Fri, 07 Jul 2000 10:43:14 -0700 > From: Pierre Labat > Organization: Hewlett Packard ATM-SISL > X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) > X-Accept-Language: en > MIME-Version: 1.0 > To: ips@ece.cmu.edu > Subject: Recovery from a dropped TCP connexion > Content-Type: text/plain; charset=us-ascii > Content-Transfer-Encoding: 7bit > > Hello, > > In the last teleconf it has been stated that : > -> in case of a droped TCP connexion, a command that was on the flight > must be replayed on a new TCP connexion going through the same > service delivery port. This, because SAM-2 requires that all that > concern > one command must flow through the same delivery port. > > But, because the recovery is done at the command level, why not > abort this command on the target (the target will abort/cancel locally > all the commands on the flight for the dropped TCP connexion), > and replay the command on another link (other service delivery port). The command cannot be aborted and replayed because some devices (tapes for example) cannot abort a command and then replay it. That is the whole reason FCP-2 was created for Fibre Channel. > > As the command will be totally restarted we don't need some intermediate > > command state that would need to be maintained per port. > We can cleanup what was ongoing for this command on the initial port > and restart drom scratch on the alternate port. > > It doesn't hurt SAM-2 because the replay of the command is a whole > new command. > Of course iSCSI must change the LUN in the headers to take into account > the new service delivery port. But iSCSI does not have visibility to LUN mapping. That is at a layer above iSCSI. > > > The big advantage is that it can speed up the recovery. > 1) It is likely that if the TCP connexion dropped through a service > delivery port, the attempt to open a new one through the same > SDP will fail. > > 2) Replaying the command through an alternate link is faster > than waiting for the establishment of new TCP connexion. > > Regards, > > Pierre Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Fri Jul 07 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14481; 7 Jul 2000 15:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA23280 for ips-outgoing; Fri, 7 Jul 2000 14:16:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Date: Fri, 7 Jul 2000 10:20:19 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: Bill Main cc: iSCSI list Subject: Re: iSCSI has already been done before, sort of... In-Reply-To: <3965E568.6A78C472@gis.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk [ emerging from lurking - I'll probably regret it .. ] You know, I actually was seeing this myself also. In fact, we implemented much the same thing at NASA/Ames with IPI-3 over TCP/IP (at first) and then planned for a frame mode HIPPI implementation. While it's true that IPI-3's symmetric request/response model might be better suited to a network transport that iSCSI is becoming, one of the main things that iSCSI should be able to do is to (re)capture existing platform implementations- i.e., use all of this (supposed) investment in SCSI target device drivers. An IPI-3 implementation The trouble with IPI-3 is that the development of command sets for it stopped around 1995 or so, so it has not kept up with a lot of the nifty features of SCSI since then (SES && WWNs, to be precise). -matt From owner-ips@ECE.cmu.edu Fri Jul 07 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14464; 7 Jul 2000 15:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA23586 for ips-outgoing; Fri, 7 Jul 2000 14:24:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-Id: <200007071822.OAA12334@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Tao of the IETF Date: Fri, 07 Jul 2000 14:22:42 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk For those who are interested in how the IETF works ... dave........... -- cut here ---------------------- A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the User Services Working Group of the IETF. Title : The Tao of IETF Author(s) : S. Harris Filename : draft-ietf-uswg-tao-00.txt Pages : Date : 06-Jul-00 In recent years, the attendance at Internet Engineering Task Force (IETF) face-to-face meetings has grown phenomenally. Many of the attendees are new to the IETF at each meeting, and many of those go on to become regular attendees. When the meetings were smaller, it wasn't very difficult for a newcomer to get into the swing of things. Today, however, a newcomer meets many more new people, some previously known only as the authors of documents or thought-provoking e-mail messages. Many IETF participants don't go to the face-to-face meetings. Instead, they are active on the mailing list of the IETF Working Groups. The methods and mechanisms of Working Groups are often difficult for newcomers to comprehend. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-uswg-tao-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-uswg-tao-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-uswg-tao-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. - --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" - --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20000706152420.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-uswg-tao-00.txt - --OtherAccess Content-Type: Message/External-body; name="draft-ietf-uswg-tao-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20000706152420.I-D@ietf.org> - --OtherAccess-- - --NextPart-- ------- End of Forwarded Message From owner-ips@ECE.cmu.edu Fri Jul 07 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14706; 7 Jul 2000 16:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA26108 for ips-outgoing; Fri, 7 Jul 2000 15:28:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-Id: <200007071927.PAA19472@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Date: Fri, 07 Jul 2000 15:27:45 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: Dave Nagle Subject: Re: References: <200007071849.OAA12781@yogi.ece.cmu.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit > > > Hello, > > > > In the last teleconf it has been stated that : > > -> in case of a droped TCP connexion, a command that was on the flight > > must be replayed on a new TCP connexion going through the same > > service delivery port. This, because SAM-2 requires that all that > > concern > > one command must flow through the same delivery port. > > > > But, because the recovery is done at the command level, why not > > abort this command on the target (the target will abort/cancel locally > > all the commands on the flight for the dropped TCP connexion), > > and replay the command on another link (other service delivery port). > > The command cannot be aborted and replayed because some devices (tapes for > example) cannot abort a command and then replay it. That is the whole reason > FCP-2 was created for Fibre Channel. But this will happen only when the TCP connexion drops, that will be very seldom. Most of the time the problems are handled by TCP itself. Hence as the probability of TCP drop is very low why not restart the whole operation for the tapes and have a quick recovery for other devices switching on another port? Anyway trying to open a new TCP connexion on the same port doesn't guarantee that the tape operation will recover. If the TCP connexion drops it can be because of a link or adapter failure, you have better way to try another path. > > > > > > As the command will be totally restarted we don't need some intermediate > > > > command state that would need to be maintained per port. > > We can cleanup what was ongoing for this command on the initial port > > and restart drom scratch on the alternate port. > > > > It doesn't hurt SAM-2 because the replay of the command is a whole > > new command. > > Of course iSCSI must change the LUN in the headers to take into account > > the new service delivery port. > > But iSCSI does not have visibility to LUN mapping. That is at a layer above > iSCSI. Ok, let say iSCSI layer warns the layer above and the layer above do the work. Regards, Pierre > > > > > > > > The big advantage is that it can speed up the recovery. > > 1) It is likely that if the TCP connexion dropped through a service > > delivery port, the attempt to open a new one through the same > > SDP will fail. > > > > 2) Replaying the command through an alternate link is faster > > than waiting for the establishment of new TCP connexion. > > > > Regards, > > > > Pierre > > Matt Wakeley > Agilent Technologies From owner-ips@ECE.cmu.edu Fri Jul 07 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14716; 7 Jul 2000 16:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA26050 for ips-outgoing; Fri, 7 Jul 2000 15:26:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-ID: <8F889D13E8D4D311BC510090278CE32D0E4537@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: iSCSI Requirements v0.4 7 Jul 00 Date: Fri, 7 Jul 2000 13:18:29 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFE848.21FFBA00" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFE848.21FFBA00 Content-Type: text/plain; charset="iso-8859-1" Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 <> <> ------_=_NextPart_000_01BFE848.21FFBA00 Content-Type: application/octet-stream; name="Haagens, Randy (E-mail).vcf" Content-Disposition: attachment; filename="Haagens, Randy (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture TITLE:Manager, Storage Network Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-3586 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:19990528T091551Z END:VCARD ------_=_NextPart_000_01BFE848.21FFBA00 Content-Type: application/octet-stream; name="iSCSI Requirements v0.4.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="iSCSI Requirements v0.4.pdf" JVBERi0xLjINJeLjz9MNCjU5IDAgb2JqDTw8IA0vTGluZWFyaXplZCAxIA0vTyA2MSANL0ggWyAx MjA0IDM1OCBdIA0vTCA4ODE1MCANL0UgMTI4MDggDS9OIDEyIA0vVCA4Njg1MiANPj4gDWVuZG9i ag0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB4cmVmDTU5IDM2IA0wMDAwMDAwMDE2IDAwMDAwIG4NCjAwMDAwMDEwNjcgMDAwMDAgbg0KMDAw MDAwMTU2MiAwMDAwMCBuDQowMDAwMDAxNzg5IDAwMDAwIG4NCjAwMDAwMDE5NzcgMDAwMDAgbg0K MDAwMDAwMjE1OSAwMDAwMCBuDQowMDAwMDAyMzU3IDAwMDAwIG4NCjAwMDAwMDMwMTcgMDAwMDAg bg0KMDAwMDAwMzIwOCAwMDAwMCBuDQowMDAwMDAzNjg1IDAwMDAwIG4NCjAwMDAwMDM4NjUgMDAw MDAgbg0KMDAwMDAwMzg4NiAwMDAwMCBuDQowMDAwMDA0ODk1IDAwMDAwIG4NCjAwMDAwMDQ5MTYg MDAwMDAgbg0KMDAwMDAwNTc5NiAwMDAwMCBuDQowMDAwMDA1ODE3IDAwMDAwIG4NCjAwMDAwMDY3 NzQgMDAwMDAgbg0KMDAwMDAwNjc5NSAwMDAwMCBuDQowMDAwMDA2OTc1IDAwMDAwIG4NCjAwMDAw MDcxNjMgMDAwMDAgbg0KMDAwMDAwODEwNyAwMDAwMCBuDQowMDAwMDA4MTI4IDAwMDAwIG4NCjAw MDAwMDgyMzggMDAwMDAgbg0KMDAwMDAwODU0MiAwMDAwMCBuDQowMDAwMDA4NjI1IDAwMDAwIG4N CjAwMDAwMDk2ODggMDAwMDAgbg0KMDAwMDAwOTcwOSAwMDAwMCBuDQowMDAwMDA5ODkwIDAwMDAw IG4NCjAwMDAwMTAwNjUgMDAwMDAgbg0KMDAwMDAxMDk4OCAwMDAwMCBuDQowMDAwMDExMDA5IDAw MDAwIG4NCjAwMDAwMTE4ODYgMDAwMDAgbg0KMDAwMDAxMTkwNyAwMDAwMCBuDQowMDAwMDExOTg1 IDAwMDAwIG4NCjAwMDAwMDEyMDQgMDAwMDAgbg0KMDAwMDAwMTU0MSAwMDAwMCBuDQp0cmFpbGVy DTw8DS9TaXplIDk1DS9JbmZvIDU1IDAgUiANL1Jvb3QgNjAgMCBSIA0vUHJldiA4Njg0MiANL0lE WzxkNjE2ZWI0NWQ5MWJhNmJlMTQwZGZkNmYyOGNlYTc3ZT48ZDYxNmViNDVkOTFiYTZiZTE0MGRm ZDZmMjhjZWE3N2U+XQ0+Pg1zdGFydHhyZWYNMA0lJUVPRg0gICAgIA02MCAwIG9iag08PCANL1R5 cGUgL0NhdGFsb2cgDS9QYWdlcyA1NyAwIFIgDS9PdXRsaW5lcyA0MCAwIFIgDS9PcGVuQWN0aW9u IFsgNjEgMCBSIC9YWVogbnVsbCBudWxsIG51bGwgXSANL1BhZ2VNb2RlIC9Vc2VOb25lIA0+PiAN ZW5kb2JqDTkzIDAgb2JqDTw8IC9TIDIxOCAvTyAyOTQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xl bmd0aCA5NCAwIFIgPj4gDXN0cmVhbQ0KSIliYGBgZmBg2cLAysDAZ8LAy4AAvAwsDGxAzPHCiYGB kYOB4UXzvAvL1j3b5tjwzbCvkOvnBQcGhpsdy7ZndCA0hVwNcLyZ+OCISOJ5x4cuMob9jg+ZJQIf tDLfEwg4LvmwRcT4uKNrcKWpqEihnEDgYUlklRD2A6BzXEPTGoCGcYAxo5AG0JGmERA+G0iGg4EV yGWUYGAUlACpweYyIFBjYHRXBtIKQKwEFhFl4JY8NCegW+SfVp/DfAb+AgbTDQyvGBjmi7zLYWCd ZMa/YAKDCFPmYQa2DQyMpyo0mM7APKbBwOgJMpUf6IQCgAADANZ1Te8NZW5kc3RyZWFtDWVuZG9i ag05NCAwIG9iag0yNDUgDWVuZG9iag02MSAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQg NTYgMCBSIA0vUmVzb3VyY2VzIDYzIDAgUiANL0NvbnRlbnRzIFsgNzAgMCBSIDcyIDAgUiA3NCAw IFIgNzggMCBSIDgzIDAgUiA4NyAwIFIgODkgMCBSIDkyIDAgUiBdIA0vQW5ub3RzIFsgNjIgMCBS IF0gDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSAN L1JvdGF0ZSAwIA0+PiANZW5kb2JqDTYyIDAgb2JqDTw8IA0vQSA8PCAvVVJJIChodHRwOi8vd3d3 LmlldGYub3JnL2lldGYvMWlkLWFic3RyYWN0cy50eHQpL1MgL1VSSSA+PiANL1R5cGUgL0Fubm90 IA0vU3VidHlwZSAvTGluayANL1JlY3QgWyAzMDIgNTI2IDQ2NCA1MzggXSANL0MgWyAwIDAgMCBd IA0vQm9yZGVyIFsgMCAwIDEgXSANL0ggL0kgDT4+IA1lbmRvYmoNNjMgMCBvYmoNPDwgDS9Qcm9j U2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAvRjIgODAgMCBSIC9UVDIgNjUgMCBSIC9UVDQg NjcgMCBSIC9UVDYgNzYgMCBSIC9UVDggODYgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgOTEg MCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDY4IDAgUiA+PiANPj4gDWVuZG9iag02NCAwIG9i ag08PCANL1R5cGUgL0ZvbnREZXNjcmlwdG9yIA0vQXNjZW50IDg5MSANL0NhcEhlaWdodCAwIA0v RGVzY2VudCAtMjE2IA0vRmxhZ3MgMzQgDS9Gb250QkJveCBbIC0xODQgLTMwNyAxMDg5IDEwMjYg XSANL0ZvbnROYW1lIC9UaW1lc05ld1JvbWFuUFMtQm9sZE1UIA0vSXRhbGljQW5nbGUgMCANL1N0 ZW1WIDEzMyANPj4gDWVuZG9iag02NSAwIG9iag08PCANL1R5cGUgL0ZvbnQgDS9TdWJ0eXBlIC9U cnVlVHlwZSANL0ZpcnN0Q2hhciAzMiANL0xhc3RDaGFyIDE4MiANL1dpZHRocyBbIDI1MCAwIDQw OCAwIDAgMCAwIDE4MCAzMzMgMzMzIDAgMCAyNTAgMzMzIDI1MCAyNzggNTAwIDUwMCA1MDAgNTAw IA01MDAgNTAwIDUwMCA1MDAgNTAwIDUwMCAyNzggMjc4IDU2NCA1NjQgNTY0IDQ0NCAwIDcyMiA2 NjcgNjY3IDcyMiANNjExIDU1NiA3MjIgNzIyIDMzMyAzODkgNzIyIDYxMSA4ODkgNzIyIDcyMiA1 NTYgNzIyIDY2NyA1NTYgNjExIA03MjIgNzIyIDk0NCA3MjIgMCAwIDMzMyAwIDMzMyAwIDAgMCA0 NDQgNTAwIDQ0NCA1MDAgNDQ0IDMzMyA1MDAgDTUwMCAyNzggMjc4IDUwMCAyNzggNzc4IDUwMCA1 MDAgNTAwIDUwMCAzMzMgMzg5IDI3OCA1MDAgNTAwIDcyMiANNTAwIDUwMCA0NDQgMCAwIDAgMCAw IDAgMCAwIDAgMCAxMDAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDMzMyANNDQ0IDQ0NCAwIDUw MCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDUwMCAwIDAgMCAwIDAgMCAwIA0wIDAg MCAwIDAgMCAwIDQ1MyBdIA0vRW5jb2RpbmcgL1dpbkFuc2lFbmNvZGluZyANL0Jhc2VGb250IC9U aW1lc05ld1JvbWFuUFNNVCANL0ZvbnREZXNjcmlwdG9yIDY2IDAgUiANPj4gDWVuZG9iag02NiAw IG9iag08PCANL1R5cGUgL0ZvbnREZXNjcmlwdG9yIA0vQXNjZW50IDg5MSANL0NhcEhlaWdodCAw IA0vRGVzY2VudCAtMjE2IA0vRmxhZ3MgMzQgDS9Gb250QkJveCBbIC0xNjcgLTMwNyAxMDA5IDEw MDcgXSANL0ZvbnROYW1lIC9UaW1lc05ld1JvbWFuUFNNVCANL0l0YWxpY0FuZ2xlIDAgDS9TdGVt ViAwIA0+PiANZW5kb2JqDTY3IDAgb2JqDTw8IA0vVHlwZSAvRm9udCANL1N1YnR5cGUgL1RydWVU eXBlIA0vRmlyc3RDaGFyIDMyIA0vTGFzdENoYXIgMTIxIA0vV2lkdGhzIFsgMjUwIDAgMCAwIDAg MCAwIDAgMzMzIDMzMyAwIDAgMjUwIDAgMjUwIDI3OCA1MDAgNTAwIDUwMCA1MDAgNTAwIDUwMCAN NTAwIDUwMCA1MDAgNTAwIDMzMyAwIDAgMCAwIDAgMCA3MjIgMCA3MjIgNzIyIDAgMCA3NzggMCAz ODkgMCAwIA02NjcgOTQ0IDcyMiAwIDYxMSAwIDcyMiA1NTYgNjY3IDcyMiAwIDAgMCAwIDAgMCAw IDAgMCAwIDAgNTAwIDU1NiANNDQ0IDU1NiA0NDQgMzMzIDUwMCA1NTYgMjc4IDAgNTU2IDI3OCA4 MzMgNTU2IDUwMCA1NTYgNTU2IDQ0NCAzODkgDTMzMyA1NTYgNTAwIDcyMiAwIDUwMCBdIA0vRW5j b2RpbmcgL1dpbkFuc2lFbmNvZGluZyANL0Jhc2VGb250IC9UaW1lc05ld1JvbWFuUFMtQm9sZE1U IA0vRm9udERlc2NyaXB0b3IgNjQgMCBSIA0+PiANZW5kb2JqDTY4IDAgb2JqDVsgDS9DYWxSR0Ig PDwgL1doaXRlUG9pbnQgWyAwLjk1MDUgMSAxLjA4OSBdIC9HYW1tYSBbIDIuMjIyMjEgMi4yMjIy MSAyLjIyMjIxIF0gDS9NYXRyaXggWyAwLjQxMjQgMC4yMTI2IDAuMDE5MyAwLjM1NzYgMC43MTUx OSAwLjExOTIgMC4xODA1IDAuMDcyMiAwLjk1MDUgXSA+PiANDV0NZW5kb2JqDTY5IDAgb2JqDTkz MSANZW5kb2JqDTcwIDAgb2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNjkgMCBS ID4+IA1zdHJlYW0NCkiJhFVLj9s4DL77V/AoAWuNJFuSfW1abGeAAkXjW7sHw1EybtPM1Hba6b8v KSpO0N1iESB68fGRH0m/6oq7rrNgoNsXrWo9aPyljdNetVY7qBrlrEaJr8XdZnYwzElIwzwUd39v DRzmQiuta+iGoqQdyv4oxLMCI7vPRVkb1ThXg0eDxgXoXid5G0gBd8bWpPBR3J9ko4JY4sSbuMiy UV6U0mhVi9fSiqmXpTGqEnu+Q4nK1K1GqQ8yiF5q1QjW3qGkQ5Gf0tIlvO37A5vNJlhslv90D4WG 0qjg2hVcwsawdlO/ly0BKx9lI5KZZMSSCTRQenRTJpFRli2anqVDRAMv4/QNhb0y4ivHt2qQZ3PJ JcZjdd3gATGkPHoCIV5eKItvOpQcoTCtUb4x4FviqakBJVFpisW+eNX9xiHSRtx58E0gKaTwalcn dlYqDRHBzlMCbGZH64bToJYXhG8p5dbUtVWteMupbMQPaSrlxFHSbVwWyoJJKWnFe1kGXPrhS1Lv px1spEfpJ+QOdVVKQ4ovQ2HYGGODaeJIbqJcEbsKMdcGgrb0qFWwyGJA1T+kA8tE15SNgJs1G3W4 Fm7gUN+8UHUF8TxKqqApzgi9EqADPPQnfsPkat4ZTEJrNLKRag4FszxrH/juonbgdZJUvk/n7AhS kRmbSpq0gOoLc/idBSJmthETBln/BdlvYMPwkJxlS0eZa57kCaP+z/quzBo0t+tHsZFYI+JREnPU ZRpJOyHXuBzSZZQG+ZuRu7DCS22IwAiKJUaNwyq3BpXg+40WIa+gnyKwg1ES9Qc+HEfqc5vd8N2S BOJOAXTJl88PkQFEGH6HSqcLUr7LWKeYYR7Z6LDALrtkiSFhPPOB/xkgB8SmYczrwqoZDuziReMm iCyas8R3lySd0+kZlniMw9MpY4tTZKUhXhkzzvsLZf5ap1WuUw8PZyoaop3IA625m3Co1zzUU09r wMWioKXm90jPdRgYt446MW4323ugsXDtRRX8zVRoVgw2j4VPAiuuEfc4nE9L2kZMh8do6R8LyOQL foTtRjqxvf8k+Zhjzd68apqrt8rdTOEPKE99ELDWv53HKW0i1WfAijM0UPgRUdBh/vdYqZ2qaKzg 98B4aLDZakqy/cNc8enLd5kr2N7/N1Z0RaKY19UlN5i7jFIKiGLZYgnVGH1PHwIn8uk8wxNf7PkC 8sMjlmctxhneYbg6BVrSpMBGw4hEVkrM/xJgAAOfs4oNZW5kc3RyZWFtDWVuZG9iag03MSAwIG9i ag04MDIgDWVuZG9iag03MiAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcx IDAgUiA+PiANc3RyZWFtDQpIiZRVS2/bMAy++1fwKAG1Jsnv69YW6A47bL51O7i20xhLk8JJ2u3f jw8pj3YtNgRQJJr6+JEfTScf2taDg3aR1Ka2lQcL7WVijbVo7HnjoX1ObhVAq1OXm0otdaWmLQw6 LdSGll6n1jRqT/sH7RpTqxF9M7VGzx2gb3c4wg0bxxkthcnZNu5SXSPwpbYmU7M4L9i00w7Xk/sc lSAnNJE7gy72uK5W0DOfNS1yn88xVuB2BOtHwL03Tj1r/MvUxOF2AZoShQ5hHwNKhkZGfOISTLh3 Fd4mrK3OTalgI7F/tJ+TqzZpsIC5g9J6U+aQ+9KUgLUqYR6TRWKBfts+KTInfg45owa0EHLhxfEj IpmmZHfeMK4toWi8QX0eEhehSLGcpEuDiKTdN12gPmO/06TfxCsS9UjcWQXOwmahHVf6K5YtU9c6 LTGvT3zwFnkDCYfOAjFqKtrMllFMqQBc6ixqaFUAFYetrqhA3XxSdF+hLMxkZluufhIkaluxtLi/ Z4rDpt+jfJZjE/QJmQB8SGEnt5dMDmm/SRmu1vcvw43j/Do6dT5Bd4wh8ZgoXHNlJYN+fCF70Tjj POTYfAXrntdvCW+dOeju/kH3GpuoJnlPpL9V39XNFTVyoYRxiUKW6ru+ANK85nrVzL9iITrZXkBH yWJqw98cT9XiN95hyqJXxmXAh3JtLc/uxRYvhuNMU8KrzV6Oj4Ju4At2TB5hJTBp5UlIOizFv5MT bFheH83iOsO7QSIT7iAeAFhN9Vt7DtOtBFF8AxEYppMaSOxZTHcB/IzrcY4Ux2zmQ3U4QChyLM+h p3lcEqN1BD7hAyjSi7aqGpwgOapRv9tURY1+9f90VdkYX78aJ5k/zJP4McBZgK40xmWDL1RaI/OU EmhwCFAPyhDA7BZi3aHQDWZVoslokomHSgCiocw1egfvHIhHyXC0hoJ6LyOC+5HBJWSU6ImNHU76 TK2mIZoDCr3KJDV0wFAxh198axJaD7qR7x1+DxvFfjT4+RRwQIKGC3L72IZ4IfoHjuK35EPg251l MZzdjs+whykPuIuz548AAwBP3rkcDWVuZHN0cmVhbQ1lbmRvYmoNNzMgMCBvYmoNODc5IA1lbmRv YmoNNzQgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MyAwIFIgPj4gDXN0 cmVhbQ0KSImUVU1v2zAMvedXCDtJwKxYsvy149YdstOA5Tbs4DhqayypDcVDt3+/R1H21hbIVhSo FZF8pN4jpU2b69wZUVaNtsLlOf5nuXaNCH5zu8kF/V36TVkY9qvh14hc1/AzRpeWHd/vN61uq+ge FxE3r0RZ1rquxP68MQsWLLltxZ5XuRH7x81X+UM1upXTsZv98S1Ap1PXY6WQpJViDEIZK8fDRVW6 lOPJw00cfilrYJzvY7BXRgZxHPsfZ+zrGhtUo5EP0TxzrOhm0a2GrMCHYebhrFpdrVFagQns72Yx pNBBZY12Ca+bpjBOYUDFYh4FH4AdvdilnD7wws8cmimTI8eNstrK0N2uRV3Ut/2nzUfwmBQpK101 wtmaPlc1qSqi+BWauEpb90IT4xZNTMWaBH8LRgwxkqPI4B9UjTP0XoAiC8MZZzOyYzPOOnQnMarM 4YRBZZaUm9Nv0Q/E031E8DG+QrxFx8klBhyhULn6xKRLTJdiuAYQJt4ghUP+R4VPI58m/s5uA0Om oOmJSwq4i7bgLxf95rkIrtQgw4Lh5h8ilCVx+goRbEO+LwZj1YAl2OPUBVoZHY6uOqERSyJKjKRM DVbRdjKQMg3os7S1Q4jjLk39F7uv1QW6j4JulME6REIl43Aog4ueLRQoDl50fe/Zwv+9orE40hxR VfM8vdsyfPo8KpSMT/rqeAQjh5hjKYXzkk2O4Q4428HPvJlgDOSRwxEeFaK59C7mPiQWQtfPvIw4 889ZCyGuUPZMXZothxErrkvrClIqSVv/h7RweqmsMau0LWu7o640rBEtiBpD85BhrBrIlFNfct+3 IIw2Z2gHWb6AojgnmL4jtXHsZlPzLOA+uRmC73n2Wur3ArCDvyRt00TgcjuwRxw24EatLz5BssSx xnnirXfb7WO8H+uor0WFxD2EioWtR+BqybQM2lrGXUTcRv+LympIfeUkjJ6qSOBn/GrkST/XkxrF isIhtbsuqcUN+IphdW1Lt/BzScuCJM3i3BZ/5pVku6eHqwArNDX0ACxXJo6FwaczIr0M39lPHD29 DzV4Mxz1wIa0nTC6PxdvvPk6VSDXNMWAuwQ1RozF8wu4svJDfNl4vVP02CEi1SCWmtnp8xam3We8 vI7QUoGpnDGiNjTilV6LpPf0ryrVt98CDACSed0FDWVuZHN0cmVhbQ1lbmRvYmoNNzUgMCBvYmoN ODY2IA1lbmRvYmoNNzYgMCBvYmoNPDwgDS9UeXBlIC9Gb250IA0vU3VidHlwZSAvVHJ1ZVR5cGUg DS9GaXJzdENoYXIgMzIgDS9MYXN0Q2hhciAzMiANL1dpZHRocyBbIDI3OCBdIA0vRW5jb2Rpbmcg L1dpbkFuc2lFbmNvZGluZyANL0Jhc2VGb250IC9BcmlhbC1Cb2xkTVQgDS9Gb250RGVzY3JpcHRv ciA3NyAwIFIgDT4+IA1lbmRvYmoNNzcgMCBvYmoNPDwgDS9UeXBlIC9Gb250RGVzY3JpcHRvciAN L0FzY2VudCA5MDUgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQgLTIxMSANL0ZsYWdzIDMyIA0vRm9u dEJCb3ggWyAtMTY3IC0zNzYgMTA3NSAxMDQ4IF0gDS9Gb250TmFtZSAvQXJpYWwtQm9sZE1UIA0v SXRhbGljQW5nbGUgMCANL1N0ZW1WIDEzMyANPj4gDWVuZG9iag03OCAwIG9iag08PCAvRmlsdGVy IC9GbGF0ZURlY29kZSAvTGVuZ3RoIDc1IDAgUiA+PiANc3RyZWFtDQpIiXxVyW7bMBC9+yvmSBYR Q0qiLB3bNAhcoEAAC+ghyYGWlViNbQmymjR/3+EM5TUoDNhcZt5s79Hlj8ltOSm00qkBq2MVQ5wU KoVIqzSHvp48T76hgSoy0PihRRKjRaozSItCZTmUmwmaa20KKKuJhvJ98iCauYxMLG7mM/lUUhR2 O4RRef7/MDZRSXwSRvsAQsnyNwGSwR5Q2fgYTxPSrprYxHB5JvUwWk3Rzky9+Rj4uixTMFA+n6VA jfEZ5BiBM/B1+iwexLxqZWRFJ2NViFpRmQgUM1CsprZAnPI7+aTWO3GbLHcI4FcNXS8NYou2a3cy V0bUMEjrD7B9ygrAa7Gsn9mqoautNBqvagycCXDgG42eG4k9TYSj465rZJSpfDR+4R9oA9LoNJeZ SsWNjBI85M1Mam/Q9W3IpGrXtPgksxLXBs9uZKyxC/fXZDG7JyDgikZ79l5xIo53MMbPMdWT8LsQ rHch9RqqNhTDV31Ii7+5G/2OxhBx9zGusVk2zsBPsKKVjnkEj6Lrmy0Fr6QvoOncei2jHIN8cEmw bHY4ZsR+JTtwbL+EwXU1uL53H3Sw45r31+tm0bu+qfn8UUIVrmCBbsPgqhVta0RqAVsWTXF6wDY1 ZTNwJu8yzqgV2MhknwjjXgWHlic4uMVZ+mHOd80L2TkCXjQD3Ab01VHAfozOUx2omeUXZnByYHAy tu9O+oQWt1ieo+EkWLnRvAK+RdIy/ZibKUaXNEPhc7DMiIQGmPoMeEf3j8LogLLwLJoKjLRXWpCs 9nP24z6RWniGfrZD8+YCbRokkG/auVYt6/RCndK/GCON6/o1aIjpF/S1kdQqpt+az7D3LM1l243G R8rZnmvRfSa2CzmT9Ecdh7EGhADcw2IdUMZcOfdLLbVvYdUj9eDkRWHjd2ms51X/yhDcs8heiMru RRU691UiS1FQVdsvm62nnRVEypxJmYkrGR0X8c4OnoLZ/nBFfo6YaMUb7YJFxXetV4AJZPVdzWk4 7QE2tshpUgb2hsRTkHguQjeSxMNJrgIQ/9YwBvhL+4ajBlt2DEXiY4YazvA18ySYz/BxCKk2Q10F lIE8/tCxz8kXzo+GRen4vxTjH5uNLNCMv91JhUuuh9M4eLrQYj4PRe+LkE//BBgAppzJOA1lbmRz dHJlYW0NZW5kb2JqDTc5IDAgb2JqDTk4NSANZW5kb2JqDTgwIDAgb2JqDTw8IA0vVHlwZSAvRm9u dCANL1N1YnR5cGUgL1R5cGUxIA0vRW5jb2RpbmcgODIgMCBSIA0vQmFzZUZvbnQgL1N5bWJvbCAN L1RvVW5pY29kZSA4MSAwIFIgDT4+IA1lbmRvYmoNODEgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVE ZWNvZGUgL0xlbmd0aCAyMzAgPj4gDXN0cmVhbQ0KSIlUULtuxCAQ7PmKLS9KAcaK0lgUuWtc5KE4 SY9hbSEdD2Fc+O8DHDkpBSPtDDMMS8/jZXQmAf2IXk2YYDFOR9z8HhXCjKtx0HHQRqU2VVRWBqDZ PB1bQju6xcMwEPqZxS3FA07TYWd/fWQPQN+jxmjcCqev7vsnE9MewhUtugQMhACNC6HnVxnepMUs N2vlu/ag17gFqTBKtyIMrBMZegHo9H+N9DfHvNzGdrUAZ5yLTPBCVGCX50L0LSwTL0+C5Mw/d4kv f723U3uMuXhdSO1cWhqH950FH0qpcsivAAMAGblv7wplbmRzdHJlYW0NZW5kb2JqDTgyIDAgb2Jq DTw8IA0vVHlwZSAvRW5jb2RpbmcgDS9EaWZmZXJlbmNlcyBbIDEgL2J1bGxldCAvbXVsdGlwbHkg L211IF0gDT4+IA1lbmRvYmoNODMgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0 aCA3OSAwIFIgPj4gDXN0cmVhbQ0KSIlsVUtv4zYQvvtXDHIiC4sR9bSP2/SSAgUCWD1l96BIjKOu bLkSvdv8+86DlGG0MGByON8Mv3lw1Py+SSGxxhZFBc1vm9SkaVFA020S2lrc/ty8KvePtqkp1KBL Y9Wid8rrpDL7cHA+ihoanVhrcvWkd6ZWL4+sfX4Bz5tZp/jfngVMXi7TLCoYeWl1Zkr1qTNaHAuz Afh1EtRHuOZOcuTIwWUOoKmbxpUmtLMDJJWh/FNntdkFyr0bJYJP8ZJoxvSINZVyF3ExoZhjJHeM eogh9DfXtkSY6MU0uA9+r2Jw7t0s2WPvU4/B/anpBjoNyRQKR/2ticUpq7U4aUbFoV22k9r4D12Z DO8mqjt10navIq+TJg6kwivagDgzfoH2HjVw7WxQR0l0V52U7JsEmER617W5XXVeWeBVmA+LmYi6 e9c/7vjKmR8o1eRP5K1OMlRCDO6eaiCweJjbi+wH7K09lY8ECIcRtwxvshsZ5aDt7wABziTKlVPk 0IoYfa+UjrzOOimQmGu9Qz5dDIPLEPy2PsQfaQzjEI8+uczNL1LSOhY3zaW4mEh6dTn2DlrssC8+ +GnB89m7+cx757HLu4/Bu85f+WQOXbzlV0EPYkGuOUWU4kIREawH97cYyA1IF9u4QO6Z5W6Xxyxg b24NWZf7tR9XylmgTDOgVJQkpAbD4enwLPXIiVeOvCaRvLZ4WRA6opxFaWQVDNSklElk0wogeArL wLizCEe+EoKLd2SNLg462eP6pEuFPHxQgpCk0xcBcLHzUOxaio1iFwzkdNEF8R4wzSR66eGc39ID d+hOHXAGVIqC9nNwtgS2010WfLSAdoFwn5MghXsMjhMpikjrjbusoOG1W3uSUP87M8p1nmdSogO/ oie9R1+yf8aHgbk7fOEI/+CpVasM+qm7nugxo3McepYnFwJfI5KjzbDRDORbQDP10FDclucv6sKI R5uXWbxOIoc5TVnN6PIH6u9SfZM+e2yaAiw07xubQYo/XPYYQpFWkBe1sRjdCUOl5qOglNXNX2RV iVVqSjTCBOAQEkURFVnQcEIseXhVXy6XceiQzl61b8OImcGBKFP6M/LJxAGqKmbEm5VThl3ElOiz mcdHYevboyiwYFR7bGVLle1pcuDb47FDuzMOVNJiqSo14gfEplTh0J5UA/jBkIAXxFWEE30FxfTx LegDrvsua6L30XQenNzmXaABi4865nhktZOJVvFQZLzGkRfvH/FrBl+VMwI2W4gxIWr5zoc4m+aW O72OwbDPW8D/cbkFT2O9kL6niGAcQkxzO/OU4rr8K8AAvqL9zA1lbmRzdHJlYW0NZW5kb2JqDTg0 IDAgb2JqDTg0NSANZW5kb2JqDTg1IDAgb2JqDTw8IA0vVHlwZSAvRm9udERlc2NyaXB0b3IgDS9B c2NlbnQgOTA1IA0vQ2FwSGVpZ2h0IDAgDS9EZXNjZW50IC0yMTEgDS9GbGFncyAzMiANL0ZvbnRC Qm94IFsgLTIyMiAtMzI1IDEwNzIgMTAzNyBdIA0vRm9udE5hbWUgL0FyaWFsTVQgDS9JdGFsaWNB bmdsZSAwIA0vU3RlbVYgMCANPj4gDWVuZG9iag04NiAwIG9iag08PCANL1R5cGUgL0ZvbnQgDS9T dWJ0eXBlIC9UcnVlVHlwZSANL0ZpcnN0Q2hhciAzMiANL0xhc3RDaGFyIDMyIA0vV2lkdGhzIFsg Mjc4IF0gDS9FbmNvZGluZyAvV2luQW5zaUVuY29kaW5nIA0vQmFzZUZvbnQgL0FyaWFsTVQgDS9G b250RGVzY3JpcHRvciA4NSAwIFIgDT4+IA1lbmRvYmoNODcgMCBvYmoNPDwgL0ZpbHRlciAvRmxh dGVEZWNvZGUgL0xlbmd0aCA4NCAwIFIgPj4gDXN0cmVhbQ0KSImkVc1u2zAMxq55Ch2lYXYs/3vX rgWyw1Cgvm09OLaSeE3jQHFS7Dm2Bx4pUo3Tn112sUWRIj9SH6n66ywSgQ51muai/jKLwihKtajb WYBLDcun2XfZDjulo1DLUaVhKe2wdX/6GhWHsbQHVYSF/KHEhkwbt30igWyECnQMEpkeSbXfD3ac OOoESezGiDuVyysVJGEucblQAC2WgdIarBJwGVZS7O1Ax4aWwX0SDYPu/gVpaQybiYY80LdpPQAP aOBQLS5S+ajiGMT9kcFzAS7R24GzXG9epsW1eJXdfe3vJMvPd5Kd7ySjO9k3ttmqDHKgL+aT81os XdgMKl1iZoOF6KjY0DbDyuQac8r8trhReZjK3pkurRFXrGh2O1pMg4QKj4qLQx5KUIEKrghLPFAM SxaGPY0T0J04kDjYZs2BBHPOWyLnXiZrOT2PksKeplDFgTMlmk1jXlTEiG8qgeuhZO7clwmUyW7O 6Nzl1B/fa5MrgJN6goInd6NvUJOY8gnqp+V/stSpW0y8kmfCdj31qWlHbtSgALtfSNlYvuDz4lbs psFGMn5ScQG5D/aBiX1gKl9GEw2rr/ncZurLes8UMjyzu8gqT26dYCXdqqBC1thhUMoNuE7gbha3 Kijh/Px65C1w7P5Y61yOoid5pSppoZuQC0C+AkqdAtbRHp26ZXNQRpK2gOVQiBRMV3gOHbBTe6BW zyEwNLubVWRpttveudiRvHaeRNOReHJiQ17YZFSxbNYe8wSawIiMwAo6yuKWYTrhUZUe2HzJerZr H+gfTDz1hgoyGgYFLeZ1rj6MBeg0Ni0j2rwK9Yz+zamUxM8tEMX+pcC5iIWKkZ4Wp+cg2qOqkA0W UFWOYgmm3uyhN3DQ4p45KKz0ZxdqfhMLaKrVDOaihuEHgcsEeQKBHVkwmvyg6p+zeV2XZBuFaVaU IkIreGtIx350mKRVSrp3oC8IW2tNQ1D63drtiL2xK+RxhSMg0AnsuSyRZBEY8kGDvVy5JwzyPvJm Bywif5x1Z/sTZXxxXCwd6SvsUo10Yjyj8TFJNqMwOBsrHq14xKHpnfuWnVLIMyJ+bn4rcJXgjMLh A22V05NGZh4XxrfrN0C25o+6/yvAAD9h5EENZW5kc3RyZWFtDWVuZG9iag04OCAwIG9iag03OTkg DWVuZG9iag04OSAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDg4IDAgUiA+ PiANc3RyZWFtDQpIibxVS2+cMBBWr/sr5mhLgWAw4O2talMplfrScqt6cMFJ3BJYsWRX+R3tD+6M x2x2K7WnqBfAnm9e3zxo3q0u3+agoLlZJSo1WZ0DvnVR1NC8WWXQtPQ4rMQL2XxfXTaNYXCW6rI2 kBFKQJRFQyot9FqzLEuzLDfBDH5lOdn6IjbOykSpNBf3MldpJXoXzztZpTo+oR2HvTSpEW7iGy+L tBQjYsu0EEOQwY1UGd5OeHsijIahH1vbwzwC3uepEgepNL7YUufATs7CQ7C0+EgMigaZFBjPLfu4 /iSTGq3CNDJ2DvoU19fmLxzm6jk51EcOK+bwChMqlkRzDIVCjeGvxcAnsPvwtr6333zPMspdYQbh 8ChzYg+Yt1oENpWAnxLpKsU1sboWIf1aQGdnO49DsOl+ATImnAzs733rICpPrLSYjBFCa6fJh1KK C5lkVB3PpmA7uajrJm6FSrAIrT4V75XElGvRxNTfwzjB5uOHK1I1eE21yIh8pXW1UKcVUYf+MlUo 5g7VmuglkfgyQl38eVMEp9gUlLiTOWYw7TlGLzUCWveP4uvSPF/xdb1kkFFPUQafpnGWJfIzttTQ Rox9OO6QNGxTO1CktegW5rAG1LA+gLqOwcj0GtEHmRuE4CQs6Bsqvg4FRPkE90QJdhHjo22LnBgc kHCIpgKwXE4RyIFeAMfmglrLiAdGTBwXtmSFBcCWJAicu+vgszyGtPlPY5cdV1ceV9dViyGU2AwD bgsT4qk58TWG7t0ON4rf0bjgaWDobYA+kWt4XS277EQf5ruAPVaCvTjXsSWchfjhWRI9zZJot31P laf7BR/hW37RaNF82tnBWRq4GcISfZSKlgLsfLw4y6DHWTwHntuFJRtOEWLwLFw0DzIkFt1HGn6E euJNKB/ia10vNVDrpQYqji/PvxF3OJtrQWuHuEOLtNn7nqKo0Db2Nf0JPLWRxkwIfBueYLfb3rd2 9guWpTtsw/KkUCdmJ/Cb15KgG0m+r+nPQX+ooDhjU1ch0zhs1LHbaLsPQuS8e3netkVaKV1SuqZQ xbN07W8BBgAEdr6jDWVuZHN0cmVhbQ1lbmRvYmoNOTAgMCBvYmoNNTc1IA1lbmRvYmoNOTEgMCBv YmoNPDwgDS9UeXBlIC9FeHRHU3RhdGUgDS9TQSBmYWxzZSANL1NNIDAuMDIgDS9UUiAvSWRlbnRp dHkgDT4+IA1lbmRvYmoNOTIgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA5 MCAwIFIgPj4gDXN0cmVhbQ0KSInEVMFunDAQVa98xRxtqRBsswt7blOpVU8ttyYHFrwJCVmvDDTK 33fGY9hsld6q9mSbmXmeee+ZJM/yXG+hbhPa5QXUz8kP8VUqJZxMTVaIthlglNtMi0mazAjnmztZ ZVthoWlbO3LsvUxzTIbWHUOQv7qh75qpRyS1yTaCQ2vqMEdc6/sjY0LDOR2cHBYvX29Ew6nQcxwm mVZZKe5jJ3hLA63lIOLdSHlbf0muPmlQUB+SVGVVXmrAtTCmhPpjkoehaV7xTtYPyVVdV5yM3W3K CnLKEhBjEUhlptgVHFsYY+6IReLum8VpDbbxJNWOOJDpJjQFHe96qTKFBKWVeMShkInAI53HMCkc cAy1FHpoYJyWA2myE6gAFVqYI3g/9MiIyjH2EkL/jIDCEExKW2VW92jUhlTHBf0jN0FYbM+gsh4J 2mHgSWrDY1LahElWaqJjfOHUY3vPGy9z4WK9C+xWYubjKEucGRqGXO9oVgxeWo6/hTdHCMTV2E7s qsd2vHeeVnRhSIW9RZJL8SwVkW5thFhKGYgGCW8kxGx4EWFLAe+IjGGd1Y9/lkmrvyiTOcuUn2VS LJNmmUij3SoR9hw00qwR5aCH9037OIesE397XURvlAgokAD7//x3/TNKSzzPZ/JJy8U+v0kXJaWM IHE0CnwPf50P/Cfkw2c0D0wRighg2eM9p5PzFAR3wFcg4NLr1t9dGMrtH+g221JJSgURNPjOvjIO 26WDtz126V3X2SGwf10nvwQYAHiXWqMNZW5kc3RyZWFtDWVuZG9iag0xIDAgb2JqDTw8IA0vVHlw ZSAvUGFnZSANL1BhcmVudCA1NiAwIFIgDS9SZXNvdXJjZXMgMyAwIFIgDS9Db250ZW50cyA0IDAg UiANL0Fubm90cyBbIDIgMCBSIF0gDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94 IFsgMCAwIDYxMiA3OTIgXSANL1JvdGF0ZSAwIA0+PiANZW5kb2JqDTIgMCBvYmoNPDwgDS9EZXN0 IFsgMSAwIFIgL0ZpdEIgXSANL1R5cGUgL0Fubm90IA0vU3VidHlwZSAvTGluayANL1JlY3QgWyAx NjEgNDMyIDE2NiA0NDAgXSANL0MgWyAwIDAgMCBdIA0vQm9yZGVyIFsgMCAwIDEgXSANL0ggL0kg DT4+IA1lbmRvYmoNMyAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8 IC9GMiA4MCAwIFIgL1RUMiA2NSAwIFIgL1RUOCA4NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dT MSA5MSAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgNjggMCBSID4+IA0+PiANZW5kb2JqDTQg MCBvYmoNPDwgL0xlbmd0aCA2Nzk5IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJ vFdLb9zIEUau8ysaewm50NB8P3LTyruBFoZgQAP4sM6B4rQkxvRwQvbYVv5G8oPz1aM5M7J0CYyF YQ27u6q66qtn/7JZvdlsUpOYzf2qiZrSxPjHH0VcRk0aFyaroyKNQfF59eZqLkw3M1Fs5m715u+3 iXmYV3EUx7nZdKs1fYH26yrYRyYNN/9crfMkqosiN2UVFUmNw7dCnx0ZMuL4I7gMs6gMdmESR1mw NS4sgkdZWBOukzRKgvswSYMRi6gKBpzTf1pmUR18DZMC/D22dg/CZ7pxp/JsR/L6E3o94HvG/cii lI/IbJhGeTCHVdQEpp0syZJTx6frBjp8DtMMau2JueX97d/Cf2x+X735TWGFSUleGGhcZ2QorCfD YwbpL4QQXFALbRzlRVUDW1AFRs9UThJleZPLGeNXHfGrBL/3I/SGzqxmRWpisQZi+JEFI5f5vb2n l0OzleXEUNGGWlx5if5A6MbdfG4q2RhXKdmaZ1n142xN2Vb6ijVU3tpt37XObs0cluySjDSa2oew JkXNu8swyYObC7JBSPo5XBcES78TIjPek1mwk2AR4IR5nAwcm0QcKxl4pyUAVa5eiyhICFAR+Dls IMTv7YTV/VkgJaUHKckFpNtHBG8ZIHa34TonQ99dEiLtND3xSb+T/QdemdYbCeNjdjPH+Dc5HYX2 HkZS0kytCu1dT/iVQGsn94VrypLhFLIUSXRDTPfC3yO5sqAze5ExyE0H5p8RXWlgZqc3TuE6xWHL 4ryy9nthr+OcJj8IZ063svBAN4UA/Q6qxbD0MsxRV26waiTLYIejqKspkNIceq8/4DQHVpfhusLx jfwYRE0N474RdoVnwmZKS0QwMTXkERUGKLBvDvo1E0Z8nlRU2h7MNetUB+/ZN/DfpBue+XC8pBaU +Xg2nsDvmE4o2nNC/9u/KGa9fybFS/2i9FtdK59fmv98p/hbgUivV63PVwsgKu2/f1bOpUvO1VqD SYGGrKa+0H8JWXVnzY6/LPmXSrMEzNeQ1eZArz3TJ6acOZaMRH0dCPtWCI17FGmSPzUnThrcyWLo O/Y+ZZGwOTvp9Y6BAWacE9RGYZC3Jq69NWkq1mw07h5R7eBi099e3V4jM1tuoSS+FFcW4qAE6k+6 9On/NUTgp0E/DKRtISAwIetYotogXVCMnOe4vQrXmD4CXPWFLxiFbzjI72fYUngpb+70UIk6Ve0T c65RmAvPP/X2TG9nvarbM8Xk0j6kBtDZ2TjlN3dnZKZ1ru0UmmeShJk66cCuLoKnMKHAOAq7fk9F D6VUlFGhTskVNa+5QiVGSUeJyX1JUZZLMC7uS9R9M/dpCo6acUkh6pFXppUz86sjLQrd5vCEZ3ay cpExG4as8QQGzinJR9ccGihoIbFnFHopTQyctRl8RMDL31akitCtUrINiZ8kqX0UeXrMrCJbEvSP YD+NLiyDscOQVirrrxtw92aVFgjQNDFljOZbm6w0uAwfk13dr37ZPBtr0yamsbYEeUlrTLVc09OG rgNiVnLIzUZB9gpGFSau40SSLROJDl8fg62leYKaak2jwZbCJaHq9CTt1BDUwLE+HWjRG2novUG8 lBTwN+p/QrkMrgX9PLg1NPj6VRKjJnMz5PmEJw/+3ztn7cfwmHosvA8z7rD4syCnZgkehB4URQFE kahPAVzsL7IkinMQwiICNOJyiiBPX0EaguMcQBcN4iwnoBmtZUhJdUjhnJKpiZKBi0EutSQL9nK4 P6PZykozCR8bxjkBZlSr3iNopU71MxVKpsHnMttUkPGifErajGtDiSmD/l5QW82lCmc8INKuVwBC W9nx5wNLUL1EhC4O8kNVPT1R/aO3/8DEjvnN7pTTkREljzxqwuyvU7A8dn5bLkEYUCGtOFnXHHW/ hdRtrt5LyuPy9pOq/2JByevjE0Pb20/U3ioIKvH3CqmfoOPRtx58DEOK3guD0KMywe+3mh8MqKVh yoVEpoqeijxNLrXkRhEV/swenUUhBv2Icb8XMfK4q2V+55nhVjS5ZU2uWQMVyA8eAC4KC/8dpvkr vamV52HNrz1KECKIjnBUxdIdk3wpr6Uf+xppTlQHu3ZYsxatjHCTbfFIWFOvJ0uUapI5raUhFo7j qSCXsstTX5pK5a9PGJKCRwLyK+zikaGS2byRwZl+6A5zJzwqzi8PzNwPjqbFXAZsHf3LQBTRNiBV ftFnUu3c+SRfYZL3kUgGeo2ctJjyXMisPSRH/OnB/P/YvURELRjzTU+UAMV3hm/PlrrqGSN+NJ6E e54f+6dWcnl3VMsLpOLBTHyGyn1hTnfN/CgFRR9Z1fncQ45KpakHQri8nCp9OdHXzhelGIoO7I2F wcruJDRHZxx4PXPNs7xX6eoC3d3oHcqNCVCLJL/tEn4pxHrvovkdN+9aYM2lm9PB7C5k0lzkTXh5 BUccTiWrFSK6A1zeXLju7Fr5noHHMxwuKGkrP/v63e3LRSpd0jIu/dQD13cHritT72Qgf5Iy0407 /kDgIiK2lu/2W2WAWL3e+YLEfI+8smYYkeAGXm4vdCTKUP8w0vFDpYKWfw15TiIxMLndOwz/wjtZ 2LjjKz9zieiJSFQIaeIf/Z29XCpUlgvZxAJPG3qlXIMQD7wwmJHkPlhlaBCRokP9fzKtoEHxDFeR K9KEJ3ZOv75jaDc/C6DVAmgmgLakXhVszdTu+62x00TidfjIIH5rHSZdAZL8ttDbjpWq2BY8cHgS qhGgnWiSiyYFdeBMswjN0bQDP1mIsxUcIcLgkpBmDiokTC4UqsYwPvCSwKEiE9IY+nIpT4vFxEJM vGYnAuh3ZFojY1kdyDSWYcjFTxp8wJKeHv7UTOOBJ1wX0qvCkq/IF6fe4pinMMH7584aoRc6sdV+ ExGWbSvYKC5aGbexR15as2govHLjOLUPei5shDONEXyj3j9On/iMLtMTL/S5oiQT4SNhvrcTeweZ qmyu38l18KxqqxSjqAsjXn6afNc7z4ryLHXetB2eW9RSZppNZoRxc9I6Tesr8vd9VDub64deO9kT n2AeYcIL6cWyKc3nvFupQBUvFKPvn0vXpNCmE9WeXlE/QupklYtLaOpJnD3vegLWNHkMehEoGHDR LH3R9Jd5RV+/QVtn24kPZFjYnxWEZWaP85OCUHKCUzSmHI0IAgxW5DfekhqkkQEkjGv31nxhRqo7 VXDg70GKXhWIzAf+S26jFuDrcan1uOLIpVFYUjGXVGyCG1+VE345NVKUAa1VVdeijRv1w7IKeigq nmVDw9mQSZGtuMjShC36j6iCzmuiXCLKIqefGy547GVpu74d+n+/NoPEzfI+SgRr+y96BiT8QGj4 iVKSLHqpyfzPPZkt3zGJgwK1fz4gXMxeP0QB3XYsSBcUtP574AMu0MUi8wv/pXvQj/qzI34iJQw0 786iKEJhKxsUkIFrEStD397JnqQpNxGaNRYh/lZVmS88cUsKmwUBz/3Il5ndUT9/ouTjmfGfhJy7 DwEofA8LN6E4u5cNbGWpVuljy/vFjecu8afore0wLKa+WBzjJb9Sn1/6wmz3/gkoIwQ/1qjkoBbr 26/XpXrFP1GXAZ2mK/tw+r618/H5+YGYOYkSSiKj9+qDcvAPyie9mJ6Vr3RUHRU2PLY/8mvLciDK 07ai9oeugtD/H+PVshQ7ckT3/oraWR0BM+jReviuxth3zOYGY9j5btTqAmREq61WE2a+3pnnZOkB zA0HEbSkyspXVZ48SbYXF+ar8AV4k0aUdu2dAngWXd/dYNghT1Sk6Fx4PHDHK/bb9u7VrI4mtPYi SHXB+Eq3KXykQ77zRvFe2F/oaFCsiUijvwjWxdz2feOug9oXjiRHM+ftNqWLYCFn0bp/ssBTS1M7 vDNtb+aBNsZ1fHqoSl4/J8cTnFwZnOz75oyeLZk5jOI4sFyutbs7CjYBu0hLdaRFk9db3FPKcXvF 3Qps2+iLuJSIor/5V8j4jYBIfwS/S6JgyzRRzeh6M4RRp0Qhh70DZeZBh2ZGuGSER5bdt5trbV2l rc+BCNDokOf2Q2seDdPisq9lSM4lWEnF7AiJHjdbtdP0HR4EvZXw6KmJKw8bNFHxOccNSkqMbcKB SXzka+mu5A8PcS6tNpPyztJE3+9f5IT0PNSY3J37f/9JmEqODXiIc6m2UjckxU9Vgg3qXTKxp5J+ as5TRm+ZV+zBbCaV1nPliGmuQsvXK6wiI+6ffn3ggTb9UurInVKiOz51PnCukhgak3OJMZsOS60d /dGOIrhvH7tO2PAX8qWENKSMxOts8lqhIvtpC6ak6rr2mU+Ey5hTqDRd22zR/R+h4t6U0SJilXnA CQnepUW6/VAo6VQoBVMMvNXjFYbGKUvTItCk8eoQ9ohVxx9lNBk4fKJroBB5xPopMHwGRcrCbVZL MatxJOUwI+jc0IitK5mNMDUJVuw8Xk2596adGx3WnqhEysqe4NeAVMfBxOjoipRUKKh0TWTCxqU9 jhEpO99s15iQKvja7qTtX6NZPDFJa9ml6u5zArSdajKZuGYcR/ufNzq39sJm6qHu8ML/fpOCtejz pZZnEe3OuuVE3nh3vdFbdIc2fQPWJyhm7Vnuiw4+NXS0WtXq5LT5814Xx1M5GmzcappQHko4CtGF O1uBTYNe6dQr4L2kMtKPp46W4LKKC6OwQ/sSflEGwqd3VOmpjLdayxhWtZrLsDZ4M+/42UQH92v7 CDM15UxjOwKFYewJ60EPnZL7Uo9jzcmmMhHtTqWVWkUIqAABpcxa78IcucUN9dEsQi44Oc04cYU1 ekJPLankDBksQdVory7m75W9/mox/V0mh68h7j+YScvpGO2qycCrE6eg/YAxiQWc2OmUUeMDXKaR CSAggaORY+c2al+ZmNUwgbulq8iEHF7djZ5FmZtuYouIhP132hdTTDNxJFRI8UjLVoazfhj5dOHk 7NYW5ErtvKNO38h4dknqBjqpxxpHwfAbE66AZJtDUJ23uF/Q/PwqC6OzUJ/sCixaalpO5XtVLpIq GGBZTSyrvPMVs3qsxw1bQWVXO85ACLHPLRenxD6c8d5hjf8d2rc+oX3j0wksil91xoWBFq80H/RR fc0mtjcPpMNrD7GvXi7VPQCtCt5QvSa8snLQwORIMCOczBvFWDr/DUhzowexlbMtQW9UhRyjnWIx xROiWmQQzuaRJXLKBZfPOqkkq82XXFllwZxs7Pd5xYnimRPFEydqEGcR8b9mT7srmvQjf3hmdETv Nwslo6MV+jOaKBu5ejkunhWklQQKSDM3CQ9dz0PXT8+0cqF8Ak81VdGI+dCf+UsvTUFPy+FtX+vd 1ZYK0T/rQeVhh2VTvfjoj200+3v3dZNbJtj0SGLpXBCFadvBSOm2+8dGx6q/giMXOn5tweHVi+mO 5UGdDy4nCRNzWaAt4PVIHxptGwW7BleXh5rPh5pbV9WGmaLNJcxRFs2QlZRshuyxqf13pzNJ+1ER iJ/qlcAT100flZugMes0MOtg2/Gr7axN2gxRxFtCYpxKFd0K7G04UUGqoT7bs6dx899EHm3a0PKr oq86cG1BCbbRxUZT7aha+UEafb2bJgqqxHlVUXDj/nZJDLIidJRP0O8XQwJL7fMKJt50NpobIDGi ZxHXVvyEKsOIiaEtBFnUhpH2TUag/xAeuX0g3vi51QI++j2XDT740hgWt0EhgJAmzY21rTU6B1Sh +WHw3Gle3txqUvXDXjnHYiMR1mB0DLEDDA80ZF5aLPUS0gIs0mgDXVwxN/w7n0/8zov8oU/MeP0H zGEaFuJ5WBDmED20DXkZx5RUJ0+OOZncXUWaW+3qpRBLvTmRc9+8CWpUeTSAIEyvnT+dLhy4WWQL 7kj9YYOaNZMD4CRmFzeSR3qRapMllynsw5573Mk39v2M70M7vuFB7xBluNAO3gSFFER+pUyYrfmg 7oRbZoMjHEs5NqYrYqRHphy4GXw9rnhEUk3jmHWhXs4KJTqMZ/we8F8Is7eqPLFoHyYM0n466CVS aH7RWisim7xAo7HrRQ46n96odAyb+OqoWa4oeqbK0YVBSZ/KaSET4hTdThN6qNDe/dbfvfei+6F5 ZbsHarFuoOPsE/129X7fqgB96y1ec1JGOoxqGOqqNTlLsimp+eLaosfozRUe2PXSSC6tpSOATAJY Ullt/5rYvbcmdRl6nlalHKRruU0ILh+C4Ih94mbkvkdBt56SfPCvK1EuqkOYLTNee/TKM953nfh7 80ND3zcLtItB4rj8puWxnW5pcHfo99QtpYLTqswNvZ3qtQxVS6Y7E6VwRQ10YyuVK9HwQoT3h1GP p1gStqsA4TqYhYMow4omWZBsDgGIf2jHfmgPj7BCJFUq4u0L3CatFtbgZ8nVguZUf3u65F1w9kBv Xk2XOjD0B5qeIvi8811Ns1Ris9SNFNEJLHALyzmaQGzTRIrZIq44QW3BW3RNYGDPd6Ie+kOO8WhO RWEkFgUD4TfIODOGmXEr7RQbV4ItrPzOcZKO4ee0UYjWq0uxsa27zhww5UvYiglbFgg9NIX9UqHH nIpw7Nei9fRqkHjdaVz7XdNxnSlyIwsiGtIh5CybzRDyc+EVOdozothxzSI/TRldnYVrhXMLRaqE BimQ33ze6pJiwozUMEMrstADqaWtonoKQYOjtycNIrVaTCN+HA2mMi39Vi9SSoAW5qhH7mxVwzH+ uA1oudVKBs7mEU37dnDNGY/D8M4QeLXb12Pt+MVvtJxGObZs0mtmnumI8/+FZOOP46o+s+Dj4Cjp g1Xvhv68MNjSPVp8xP854tANoYLyhs/FjM//QolngNHSFNUWWhDlG+U+DVtRxKy2+NJY+0hD5iwO 1w+PZmI2xy2/B6vWYgpzfHH/p3ilBnb+sPIumH+fyc9PVqDIuyezDpWPfLOZF2cwoMyKsKdeRd74 n5t+mdo57ze3Sh/lQWF5DvC4duH9Ra3DNV42TsxPMjQ5VcRzzulxwQBTKJfM7peInSK/hcnT8QzH GVp+ZnJ+ITdCk1RXYyJPNDf6ZqRJejGQQ/oLoGysEw1iQogaqdTnXnto3VmL1PmqfaV5FwzTRg2+ kYT0if0drbTd3gI9UCvddxSnr3ad85ncHSzGK1xObAd7EK46PGON++V+2o5fTsuMx/HEsCvCzkmL r1DMbMdpfCqsDeovCK0Qot96Ptw5ctQKVDyO7LP2mdI+gUOXkYyT4wreM8D7jhs8sluwXVVhz7Ib lUG34oJwBXujW7Wp6dgL4KJ9YjySYPu8NmRCLdX4IHQIniO0kAp0he3kB65DKl2hAPBkC+YRArSN mDdiPc5LZed804uzPI2rfOruGU+DjTKTy18qPurNEDA5jcNZjjUHTUZFqhG9r/px8F/m65FbCYjU 6/JlCHuell9lDlL/2OESHpiwi6414eeVsH18w0dlO7lVX4oJT2XcqWcAR/5QpjUrY9sEIiLwceYv TfvhtMikcruZCQQ1prpf7TMmIPrMWW+mg4Pa1U5BKGT00QKbe3OWzb35w5DEGYDDye6wmnNsUpGL MjXfOCK0bqNniNrcYqMK33r054sPU9UYppGdXw4ecodthsHr0HfvB5QfTlGYaJZz05JtbqsQeLad uHe8ZeT3UKlcDCxJ5oVBL7oQpaO0h/J/hFdNb9s4EL37V/AoAU7Xoix/XNdId3XpBrWBXHpRZDpx 41qqLAObf9+Z90YO7Q26ycEkNRwOyeF7b/CCXG1fK8kyqjCtQDuE6Kk5i9hHeT+4/SyxK5VnPhFa QXuNn5X4XiRsl6m+KPeoEAcP+9QkZgY5ijH3F4U1bEVEYLAVWVQ+0MsjVSDNvqX0uQ3cRM2I6fjJ Bm/DfuqaSlhaDLfOVm0YQ3VIiwHksWEhjBw0pW7bdq+KUPUAJj1/LAmz/ILNcx7+erUuDd3m5F09 FMH2R/UnxAuUyYllOTJef0VCPnGOpnti8ysF2BnhXLzVZtsJh42jio41J0yPNH0z511lA6eWv020 SD+wIQMAgIANhApP7+5ZN3h4zZYmqqfUORPmOft0BNwQKLegQxesJWfwzxkmnbPdo7a4QuPZJcLX GHWn70nOY94p4GA7PqnPJ/E2QworxGvbNXpxiWPSzpGbefIw1hcl46IFwASaJ5KaOCHO+wpDuvp5 5ieWsEpGE2ybsC0jPZnCpm6b+iy34a1QmQw2uLv9lWkX2ODHjsvU7IUhOGwuDrQyX1tN4Snv3qOu 86x45xBvuu/GDsjFo+Yf1tcRRNojBV3aSv1AhTI7aKEhv1JnHPsrSVhcsNcQqN8z7aHBqOhmyTi6 6BxnNOGpoTopiKcCngBHwajwimFXxTLSD2aiL57xnW6OUbvb13LAELqCO1YVFTZ96HFCz7WU8gtD aikW+5spZitvVZ6psIaa6uXo+Z0PW/edu+WC9EhPdgw7RZyFPB+PHfccfWHI+qqx/X0djCA881Y9 rFCgQr3PtGCUCsCASJRLMV8AiHD8PHn327/0boly4Hcm6v5+MxLDyTRzWSZlw8IJ1Tp5ftLqwmg3 +nMzmmlnIv9owFqkUTaRi/L6SH+MkizdfB+JfJvBDo2l7Gkhdsv5p2JGM2TQ4pJB3vaxXyMDVxBh bJdx3YWE10Q46Gnqj3aEgsnZpv8FyA/BeMsGhu4RbNVXT6JGzyfQjc2EuPcQ9yJuUecukPWcwYUs lA+iJL24qm0Pqp1YOoqjQ2Urv6GvuSqE77h0wzV3PUOwHVa00bLm4yg2pn0ZwANf6x/4VlrPHaPl Lu69VhyNgazcbO6L/P/Jrf0vRfWpH5o1+cB6B/lwEhVX1a/khTGrk/ydzowNdE9zYI4k/pcSgj1L VqfxDTPsU08ZnbwYvUWd4Crzaz/CLVe85Ky3S5fJEIkj561TPrU7vQl2Sl5XxHrGdt0N+zEGizDY Ti7ZMfgsb/e+AYKrZipk3qASWMU69/eFCjN9/qhoLisaQ1s44xiJ8wgJXmSf+kLmyfPLHZmrDd2O raYzlY5HQYZXTlsmAkJR5p5YN0gR5MK/8rUNda/81BBglwqvTlcSkaWksrXLQ4JNh1FxWdfhEDq+ hmXyaUCZXwMA/L1RIwplbmRzdHJlYW0NZW5kb2JqDTUgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0v UGFyZW50IDU2IDAgUiANL1Jlc291cmNlcyA3IDAgUiANL0NvbnRlbnRzIDggMCBSIA0vQW5ub3Rz IFsgNiAwIFIgXSANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEy IDc5MiBdIA0vUm90YXRlIDAgDT4+IA1lbmRvYmoNNiAwIG9iag08PCANL0Rlc3QgWyA1IDAgUiAv Rml0QiBdIA0vVHlwZSAvQW5ub3QgDS9TdWJ0eXBlIC9MaW5rIA0vUmVjdCBbIDIwMSAyMjYgMjA4 IDIzNiBdIA0vQyBbIDAgMCAwIF0gDS9Cb3JkZXIgWyAwIDAgMSBdIA0vSCAvSSANPj4gDWVuZG9i ag03IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUMiA2NSAw IFIgL1RUNCA2NyAwIFIgL1RUNiA3NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA5MSAwIFIg Pj4gDS9Db2xvclNwYWNlIDw8IC9DczUgNjggMCBSID4+IA0+PiANZW5kb2JqDTggMCBvYmoNPDwg L0xlbmd0aCA1MjIyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJjFfbcttIDn33 V/QjOWUxvF/2LeNUbc1WMpUqOU+OHyiqLXOiiFqScuL9ofnNBXDQFEUlM1OuMtXdaDQaDRwc/Hp/ 8+b+PjaRuX+6qYIqNyH9yY8szIMqDjOTlEEWhyTx9ebN3ZCZZhCh0AzNzZt/ryOzG27CIAxTc9/c rPgXyX678Y6BSfz7P25WaRSUWZaavAiyqKTFdyIfxbxBfpW84cGrD34RxF7nr9Kg9EY/CirvWaas T4oLrzdHrPX+KgmiS0kdNBDV0V6WPvumHszY6wGDaukulI23Zmgh0Fgz6sHGX0UxrX6lYTXZ0Y6D od0Ruch78kv6b05Q7a/4Q3pE9U4mzT2JpkHm3fkZ/f/or2RH3VvT29qPoiDxttjQ7l/9qKKxsXJw 6b2IilbXrSoezajWX1n6zadPNl3uC/bTYkiLuJ+apRswunQpbnP0H+//cxOaVRREWZ67lwsTeTk8 GvkyI5fUpCzIvX5sm9ZNHfycbjIOgaihQEsRaPTyHD/0qUhdGuYUGhRzMYdYyKq9mAOHNuTYEAYZ ydPpZDMWUrcQ64rYJSH44L2zT/4qJ2v41nSZQzu23WFwVvww3CdL8jiIS7GENCblFNShBukdPXdE gcBx1Y8+P2vdyufgV/weWyu+jikw6GUzTxfxwQb8ZpOKIEUgdCqtH9Vvt+ZZtLrx7eyVM/o4rRR1 mQdRiqmFEhjWyMgZp0fDHnr0mAMCIz3RdD2072Tc4gzoqveyYrZdc/rqk0I67IDdCwNVGd2wPh73 s9s39QZDWHlrLpzY9VvbT9b9MAoXePPgNayPwk7MLsiTtJh4r37M5hjxUUL2RCHdbQsZ9kLOkEGT J14ZOPxTnYdMd+AFfaCC3fvfE1Z6cSZBQxzLE/FAhMeZ8sCnwwUBeLcc33dy1o68jgn2P87HzoWx mIT+72epW3PCEfZs3NbqrxYaxBzrswe2kx4e9fJMqSev52w1c6+Yi9sdBcIimFQ7/8gZ/6Mo3bzK I93/Ik8TFw7aGSoEJfpue6LMKbyGsJG/O/nPNy1J1bOMrHgp4TM5fskcWcQGs+n2W/PE9qZezRD9 KvNHGxhDKa96Wz+hdVWLQYeVQeBoFp+ppweMOJ4RPSN5GDoikfg064cMpfbQWFXCKau77IXGjmE0 YdgVm51KVXTeU+tV9ay5F7reHQK7t2TIy+xuzktynu2HuefT6pwVFVzPsRHze45UfXJ5QMq3lhGS A70853Gc4A0qKRQihj1LAJdcLLJqAt7cvXec4tD3HdeAlGOcP63PBjc1RnvziSDZfPbef/rs/2sB y1RmWbOAOt8jihJVHqUaTOatWVaUII7y6lwJkmKqUN5Y9zs7at3QQ6hAFmVylo+qiYsomKwIh3LO Ek4SenUu3O3WHoQGjFqdUU9p3I7stZTAhodGR88yqkeO40pyLeZcO+7tRCr41Re6cJapKZdf2mZG QGRzt7V7U0N0a+x30JJGhU5Q4ccztapvfcfOL731b6YBdYn1/K8SW04nwknQqiBnzEBXoiqbPFVq fA2WYYCq7MgHUBXZkNPYFZHUFA50qrB0O+YfPCwUThj23XePvS0+DYRUdrxY63THwajU5VbrjnPW PGjBW/Odvbc+B+AHspDTcRWrslulZH/qONFvgPnocpiFi21HXTCFLjzqoT9OnLT4OQd+3znmCD4o xZETR9gsEuf3E44j7E7ps/GZuFG5pITirM68T79f5xWTQmbfU8yn8RkqYpdYXLQpWPJ05ctFNlwR Kik4FcpDxVQ1Qr3jUi1AEauEVIQIJKbyIE6G6bjDxp5iey9LOrGDCog3NdbMFNKyeJjLqDkP67d0 9Yyek7WoxTF03po/ySvUudz6UcJxQO1IAR2PTAyiq7dZOQ8tsC2bsC2Dl9Z3lEPvCJJXAgp0OHcM S38TblILNYOkaypp3B3fgitxxivXEoZkzQWxq5VgYVUJXNMdQJlsg2W7XZC62kBU2eDiBIITJWVy 0sucbb6Ciw0n+W4GjKFND9M4FxaRI9nxWiy51ZOx/3jslC872qtcb+1zt3DHhbPyMPiNn4hBcMkZ L/gzLngAYKm/lyRxqodRUk1UBFWta7q9/KBAooJb8jMwKn7gwM45nhhGJZ4iD8Tbk6DiiKq8orzl oJLp6hHCf1co4ynfXQysbS8o/87u2xfbv5qP5CXK5PW7j599n5HsB4mchNJCOaVT9Q3Tcx6ziRJQ jgdR/QVzSPlqTI2ZA3LFZ7oRo7oJGwLvKjwdWpZLyD9P2Mbc64Q9YH5SmMDkCuEwTEn56Vp6sLaD GtPsW3e+Khajbq+MNFPhkiN81t5DxPa31BpohVI/LJvTcspXRbWxHr7I+55LKYd6Lbfi+MlRa/iZ dzKwGFAD0mkVsZAiX5jLfVsV6O3Yt3SR2XbuFLB6wvQwDj/dPRyn8pZNy4MdnM1PQtR6TpLMCeEe 7HjpwJ5xihnkwTiwYQ6H1/ZcHUkZhRpknSAV63KGuQODQa5mb7BV515hmM9d0mhhQTBnoPnk/1Lp 2lqogFQK/nSHV+IykoATB5aFQTnyNzaGW9sVelsgoIgseV8xr2kT5aMcJ9XE3sZl6gRlmqSzEpif S2CuqUOpJ3BcEQ4wvAgMJIABMk4AII8++wCNzPsLYa1BJcCxcpNHARDKiOrxqgZVQZJH6YIlJNns ave+UqOY44GY7RIfKLrS6J+gg+blGsh/R7epmB1epeNFYtfcFjW2fZmoayRtBKfqlaoG7U9GkVpJ W8f/6wsQ2HJEp9rr0NBsW9I/qkJ8TrLU6L5nhZN/rFt1jZ1RKLKEITNazRs6iFJXt+928+sTGxG0 gVeX1SU5x48G+wkYecCHqyr3UFLyzvnFHRnIDyEbk3gRto1u5qpPWzNuXGVCmI4k6gctsBxoCLNC Y7KCaIE5F2W0NwqxchVs8+K0Ol/HxRlqecKJT0YX3hfycHFVjag9SrNyFm4TXQqVLs1ITiIoIB1n t/mDw0p4i5P4xrBQieeZUMrys4i3GOjeEZPyVlRi9meqyNOH+Q6O1mMvFY4oMpbGC2071Tk/zp4t Eps7wG7/BbL1wJBYedCqi01bj1QOLze2aiqRMI3XXCI11v+IV2fzFpo4Omuzc4DPbLbD1Y6me0I4 qtuX8XjmmK6f2KNzlf8HVhbTQ9K7cpWKpYcUwxJJIbGFMoLRqWYWpLy75Law5DSVQORbC0kqAHgZ IjF2gFeV2AR+RIEahY+Q+9sQVLTSEOxRAmp+m5zNZyow6LBx9WHFaNN2KnTNwy/bHoDp6iJC33Ij VTKmdEfbC8lEqxELIeaAwOgA2IUAs2QqMnUzl+8gcZBtZmNHNKnfcIK1Wqt1fVqN+Q4dsqKY2Hws uEHk0w8d8Z9sgGF63H5mawsNOGF3cc6zXsZZ8V3GDabryTRRiZ24I4ddxA9sLpTrpNrQS+hX6AVi r/65Vyi+saj2OALvuq8oTfO/6FKBl7HDS4GKQgiD9JTaKarNlSdNiOSwVD0xr5LoT9waBrgAu3He jlqz0Z1EdGWPO8G654r0uarzGr38s/pBdKOndRZNpqnlz5NCqHFjmOXQP1P0T37c6qLPjcJq1vFq 8uGHeVA1SxaVUNEmKj3LkWKWhWNPtZmBi32VgHlQYRRif5lqK6doBkurSw6SKc0aLLKXIzEXUEyZ dwrhlvK+wVfFmDbLl4sRF21p1HRyz26oPBRqvkVohoaoYVFlsy4pqqY7Me/9P+dV09w2ckTv/hVz BFMSTYAACNTeok2lVJVkVZaqcpB9gEGYQhZLcAHQKefXp1+/HnzQlPZDBxGYacz09HS/fu/pP+82 3lRQJdvOKVM49jDTjO4MhJYUG3BluaYu3gsOC1xfhDRe77aiPccljRXr7qpHZP85L106oH04WH+C 1d+epjOFrnbvwlR4uBRKjGR1eHYSZIH+rnr3ZTpUsg3XG6EssSRtJA16vYNUEoCPaPjXp5m7Kic3 Tn5y2T/epG6bb/HF0y/k1cHWPE4NuNcJ3ZVk4oRH9HWUXILtc/BBEkiEpibQr+e604cKmbTjZSY2 eRyUVPcXmZWv81Q91IfRx12CV/hod1WO94dbuz8qEQVfwK9o0mEoSq3mMLAxFUaCsE5g/KxDwMGg 8eItDDgqrBRih5QX3x85K5KuHwobGxQtvYUcLNdT+w+wDsVarAoKeS2M84vyzUa/9Ssd1Jj/sUBg fs74RaQYhY/8mjySe/4gBe8+VA0IiX1mv7TvfQx4tJ6sva9bGlnU6v4Vz9TIfq77FsXkvuqVRB+X ubkkKyLExpJ4/lHYqZbkxAeui5en8frI5OWaKA/0gpTLe1FCsu+lBCLNx6Vw8Nb/pSh6oRAQmVN0 n2vBvq5uqH8E/kE/HOVBR57dcvOTML/6KAqjlJjToO3qivutwVuEO7ml77U55n8nbQCI94JEOAnP 4QUXfDjWFQelO7XHpX/jbv9qr8d9M6JblE9g7HtjZhVAgQimN9TlJJZS7ZbadpZW9ZGc+KBvSJ6h 5QigPLE0BsmqNeCR3+zE2aZmrs5FElqzUkDeWeY/AQbrEi+k9RZ1I+NYo9O2spWYyAdTbOlqMZ52 dpr+kppegUTJmiweIZGZK/e5RMZwBMDXofF2IuvPwd/lLn2vk97WaVfLrJ39FgRGuZzPI2AcT1zJ yuX5wwq58ck9rhAPJUxxcDq1nahSXNTnRn/b8mdOSY1kAefarjhwsHL3P4kok+Rvv9pI5+4f3NFe VtFExKX6lW51tiDWY2xFVYRxclHh4Vjho8s/fnI/nRVrOykqfaiHWmRecUJo4sB+OvaPlm+F0FiY ujPfezmDUKheTn4L7XS3QmI83guo5awvXYP9fBfwYyV7GG+UeplRiZLaim7S9fuhXewvQeKnV90a rizUzveQzsMz7jmqVA1IAUJEpx94gvdqL2G/PIH3/egbFDaxc2mjyIOlyz+/ggzbS9qHnOm1Qncs rZ0WZDDwRoWbeJoR7YQIpVdpxiuZm0hX0WoKJx4mPmQjOkU+hT+5f9ZHLdbaNApacaLdLAn+JxgI 1yJCiciB2t75Dcp9jY4+4gqCWRXa7hVgtnjXLpgGPxCuooCzw6LhArp1suLSrmgaIBOYixlU3G1P uxvEW/DnYxBy4OMKYhOG33259M+O0L19NnecMIykIfNfSsWosW7h3R1c7w8KAhIH03J2n3o9cp2J hJkMK4rf5JdRmuIe7eJ3v+Pi4806jNCFLq7+mtSjPqMqopL6TClFddazO7xI59dOnqs2NVGoguyj Sa/2wC9s9uMKVRIGP3irSG7moWsX2m/P1gCGpnqr5XtnqpFGB35x/F7t1RR3R7NAnfOh6rq2kz27 ahWCKV2ISPFEQTRFtoW83tnBpnUsYc7eAzYy9c3GCPTxqHf3/iiS/jpsdh2jUTkfVqxBbmVKVD2Y 34YtuL5MnG0uv6I6JLt+I3HifI0m+weFSRRpZl524ehPduEoZhd+kKpT6qjk7gvOG0sT4/sviIJS 4Eh55S0IW1m9R/alwV3bK6NAziEYGQIAr/VBMn0db9TxdA1084oqWmrAV4plE60z39/DkbmF+ahw kDnuoD+VJgRy3N42yNaiuXH14x2cToPHlXSQic4xVWDdDyOYEZCkE9U0OOkgVEA2bsEZMBfkwkCo YX3dMoNi0IXBRh3EFPNtRwYG1xpn04rBEanhhtQwJzWczZrtVx7OjftpALQHKhU0B7lR11U0GNR7 k2qR6jCWti0D0M1myxTdMMfjzO/euce7x/tZ+4zjqX1Gs/apDUsyoupKhdiMgjELOC61bozzUvJs 8xEFN/mMxEVC4v6Bi9HOHwL25J6A3W5fAXAo0VDGogRktPRmlr32cz7WalwqAqQB31qkNggy4v4G XftekIGu3aliNMmylCsLKVablilsuDFFNgi53JJvUtUVZYk+lSjmLFRSa/rIFf5+FkucbjldcKO9 ST0bNY3nt68HJzjceSdNPtoxBiF0XVtW+/Nc70FrNA1I40xynaq1Q1vH2qdTU5coh2P/CslKRoKT GNPSa8soVlJ2epIbARnZbTntKNbIgSJv3ag12306mlqEvkqEEpIEpYAhSKW3aWpbfxga5bDT5vZi Hzaa0kng7Y+cPeggaUqixRJRoMHmhTvXHVkzLLSaM7BmnfJXvQNoVL2yYUzsae361j9cLmBhWKzT u/3SZ2/1TS/j6S+j7rUkDnkF+7qTimxwqzFuNVQsynxSyBUX5cCBnuc9Vd0XFhSahF2ImharUPM9 lw4hefGkl4BQYKhCsTL/wusbtdyg2oOIan3TrLSPzrqOAlumOLZVUcJj880KCbvVZgUYOvK5Wp7k hq3fW5YWBG6zR6xDAbPDdcibCwaplqEt20YFjJYONkgD/q9VPwq+yIXlsunGSnrKUowLISIK2VOn 6Z9J6X6TMYZqq/OAahyw02COC/MzZGOYKHaHKFAXbjZOA5yZI2XXcvdK4rLKDShCc3YN9+zmIA2x 0XmV22xlICTrllCdyvTo1+1WvUAw82AOUKlezY5XAzcRztsJYrMkiTWoZCPuzT8hJxv057dMlqws 3CRgFeGSkgnTSvEC1qEPnnXIrzIscpQLZiK9OBObLAW7MWZCFqUVlTIb3ANqWgBzBb3GWgnl+HoL kUFwqoUSg0qhheyIvUiZWwYsZJbMtJEuo2AsUfz1bKwNe3Apze08aL7p2437TJOBuZOblYCFOcBp W7AYqoZmiuxWHGnge0TqHbBTdVKlQ+sKdfHiBHKA9z4G4/E5id2s4CN2qphY8W9UcWqrS/up/HYd 0p1H1cSRZhYl28vWEoeTgDJge9C9kfca+ojdL/dCKFQWG8v0nS6vlEFp/k5pvlAi/zkj1OlFbEkm TGaEqncyvzC/q/W/fWV73bA4ubxNOZq/0GKxhOMa3tGuKno/op8eOOHmZxzd+11HNamzmza8RU/l mLP123Z/9Leuy3Ga/0f58/8BACP0dFAKZW5kc3RyZWFtDWVuZG9iag05IDAgb2JqDTw8IA0vVHlw ZSAvUGFnZSANL1BhcmVudCA1NiAwIFIgDS9SZXNvdXJjZXMgMTAgMCBSIA0vQ29udGVudHMgMTEg MCBSIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0g DS9Sb3RhdGUgMCANPj4gDWVuZG9iag0xMCAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0 IF0gDS9Gb250IDw8IC9UVDIgNjUgMCBSIC9UVDQgNjcgMCBSIC9UVDYgNzYgMCBSID4+IA0vRXh0 R1N0YXRlIDw8IC9HUzEgOTEgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDY4IDAgUiA+PiAN Pj4gDWVuZG9iag0xMSAwIG9iag08PCAvTGVuZ3RoIDYwNDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUg Pj4gDXN0cmVhbQ0KSImMV8ty3MYV3fMrurwCUpwhgMFj4Kwi2Y6ZihWXOV5JWoAYiAMTIsYYkLL8 G8kH59x7bgMYUVJSrOKgu++77+P0i93F1W6XuNjt3l2U6zJ3Ef70I4vydZlEmdts11kSgeL9xdXL U+bqkxJF7lRfXP39JnZ3p4toHUWp29UXK/kC7YeL4Lh2abj77WIFAUkau7xYZ/EWZ98pebIRevmK CqF/HdT9Q7hdb4Ix3ARD33XNcAqzdRm4cBUn6zh4HyYxfpowWqdBhc14XQRkMcIxXJXBQXeqcWYr QS08Dx/1yJFa1PRDdad7javqujGZPOf/8z1X91hn66035hiuNjjtmhESOlFvBz0PTuEqx0p0uVaO l/bK7rkfo4qgmZchglkEruLR3sw2wW48mOH7pvNCPoaJ+Om8ke/CODqLShO+3f3jInIrmJjl+XQX +XQXKe+iFaV5MDYDboXfTT26uuK3j+1BV1SfBk+kC2m24yG83mJ9pwtuuQEmj3ZwxmR7GsM0OFa1 SUak5OJ6094b01B1nTs2g/i5ltsE+cZfTjrZRa66Wav3CF8ep5nEoEgLHwJsTOmbMQavfwHfNnjr fgxFaIvwJ8HdQXRlgbtFKPRrr0rz4EOoMSfZftQfI74M4/+fxVV3dyQbEJQkaGwFb0r8kIoscs/Y Mql0b+FdVnrvNuXkXVSad9/BqwTehXES7DRSW9hbSs6rpfIlhqag+hAmBQLchimshqGrAmElsXsD k2R7MB6kfAprcCexXmwSDG6oSCOrS/fTiytdnUK0BOTVm9Dx81EFHI/9MJPv3S3SOgmMhAdatzRV K1JspHazpe/0t5vEDGR37UIzzDqaSx1doqaWCwqsh6Yi+dk21V9+vp7Y2hjnPZyCmWMPB3I1RtrD k2zqF3LYaf4WuGHZ0JzXjC/gJ5m+1RVCHSNW17DAzFKBrpUNt2+5vg/nlskdaVKBEu27RvoJPpx0 29LUsNtOlVT4SkpZSVtWUqB6m7/CjARm/OrV98bkGvXKs640Nru/MCLxnIAxA9NVgwQRDX1AX6kP /BahBXJENEbrUqMdw1ztDAntQYoMaAFKv595jYQcFK09WTa5RKok4B7a8YCpAL+QqxnyWlzaWGTl Biy2hYjnR1113tFtYHujtgVLxVhTkfw88CMs5809PFLlLQK0DMzzyuytdL5aFDw7ikOx1aWVrOU8 CS7PCRs4NIlfcs2ih08Kiv/R/Jn9FPcmlOGUBmvnXmkb6SmfYlxPprOimqYxQpjDqRXzc65pd9ZW GjPvRgZoErwM5RZvQmmX186uv/QNyZxz5pkZM/j149Iahsc9LtxulycU+Wc1LlYm8GF5cdHWz8wk sXuTZErQ+LbSW6on3Dwqp+2qWx50OgUa17Wa29K3df9eCZ1RSeKrLtmU/iuNF4OC9xxzegjhqIuD 0l2KgdK5Rs14lX4ye+bIM5uE/jS6Rss0Cf7QjePSRGOEsOOZTTJumw5tBuHK5SZLieQbfGnHfzQZ Y+ulsTtnmkv2I8qQTO35/glhTpFVDrBtpEsANjxrqPzJjFIL6yWQSdMZyKQTkMkXQCYVIKPFGweV wTwZArmhqxQFZVFCpKUjtshqSWc0D2KnlO2FAC7VZJ2PtAWbGsNLqeIlE+pVh3Ivt0NbP0ci86yO p7zyMOSt+yeyNdVxX9DGRODQy58FEsTBr8jmQvNBDF8pFupapLBGGRYXmiBo0L/LWojlRkGEsSQ7 Tv8Pgiw3TESJurTfER7JjYBDxLhamIeheZgUurGpVZgQybbK6rytd7L6KKkSfw2gxNPFxbnHJ2/d D71kFsa55lGqSZcIJjpy1YytwEip5KdQSlithj/+nEtcvXV/FSbg2AuqSPGg3HVzqWBZ77Xgvd4Q kb68ufZCBw4RkzaGySy6pjRbdXrk3iMbYgm6dE+pT42NTAW9xyI41zk0jbv3zuA8R/Rk5czROQIq g0o8qR36Jd0aK1PQLm19+Dx0icop/WxO11KPJZsCW5Bg6RNTRxy85a8oFWhfIfWaPfe+FcjCTxR3 eyNhlD6Us1nEEpB5MJRrS0ws71UpZfpGFHsjTqfWlHZqTvPMmEdlbzuzw11f/ctVtqhMGFL7xG5T Lp6LthbFD/ppXIKczDbuu4a/Yzs0fjDoHE+yBfwTl2WEocEr0dVulzpE9t0FShiF4CIf+VwHCm7r dB/ufhPChIQolC3w00S42c6QYUstwEsnZkklU4wjNVOEWWrCJzbpMZB4pDMvExyqvVYmK7JcOkAZ cHFttUT6gavqwXCTKDNIkhli1qISRcfOtEsrSjjy5yUlkLFSA71FKWsH4GMTLPW4fQ+gQdQ5v1ZK RQpi/FF/zcKe0vq6pxXOjHrkiKREM/vYDHPhJDbuffDcN1I1RfBnM/TEsqgbHTaflEyST0MntZLp jx+lEovgG4Oxc5bry7Sxl6kte4l7Gvjnq/G6aqiN7qAv2HZUGCb96FE3hEFQqoBMeVh9ODRDIym9 1aJg0ldH6eGoyz3JXD84vOcmbki1g6kCtsQ8G8V7uWINnQyTEyVW/8sHEeKaP/S38nZ3/PEe9g96 XDd6FZ8d6emU7NaP9nJJhV6kPHbiiClaMAE2SFEeic5UJ6L8FwiHpAL1OowDw/tMpAIeY78RWImZ JrVq2VNI9ujb5lEoKJdSfSxUpAq3fS2FDRMKuKon0/ROjacT4aESrQwAMXGlRyCqzxtOSRIz+qx6 zPTDZMbnjOSeHiskltm9Mlwp759KB8PQfOE5GxXTTNjwDmRyQbU+nlod/2pKvtCJFEvZsWOpuMeQ 00pSZU9GR6ySaz8t+OoST3J6LSQYDzOR7QwURQypssnPGbEhWMV1mDldY+aR6o5E/nRoumr0Kp74 5iO/jZSNtFQUELmf0Z6+huSiCdQk+YzlXqrcXHyr6UHB4ZcQhU+wRhuH6GG/kJY0g1RivZggVSkP SklrBRyow+lk8iTKnUazYGC7x/uVB1NKL1R66abynPOeKg0BYp82UICRdvZ7xx/a7OM2T8M4wYCL HH5KBCwFfE8L3C3m23sZpTZPA1ykDcfcD0eMWx2NiA5PTGI0ncxDU+YxUD9Gsmm3EQt0lKt+/Zgs yND/E7VAJFDA619CqRzco/5K2ws6vPH6QfA2xtayAnL/btCMm4/+FuqQeyUTZot5myKY1xJwpNsr Cr7eKW1hp84vY4iJ0L+4TVoT8Cb8MsB+jusEYBORtSdXuQcPuJLMV+/c8RE1xTAg/vc4VKQ96Zg8 R2YDp8D4H/f6RpxLxFWZuz8xnRKSvSUZpvoLClag11LuHeGVIbPGOkBqsEtQm+Ubcr81vQbJbPWO w2cyHmafYzcPC3fak7cIo7wtf6bF8laSQwZngQdPX3h0ZtP8zxjaYz82MtlQPm3VOc2ALQflJjhi MHHwZTRgA51xJAguF7Vz9mR6IdJtum469pzX33veH3APGssfpZ1IkZ3xPik9vwcARUdBMtLx/hmb PZrg749KJEhWJmKmIx7viIcRmXbbdJ8IFixv5oz07aD8QNGtpIoE80atun4+UV4HZDUWuLbAHZlG OlZkOpvt6LNCDKLNZ/E0Az8NKzki7gJPUYcCKbqsIEhAZC+PTLrN/94QitybbtpOs0YuEHk0FUEN pFl2g+151xZk1Grgz+K2TAMLMak6UnW6cLd8Z8TaqU61PDi0MRWL94bepcBW3OWeT47vd2BpHVps jr4Wu1Rg+NZtsv9SXiXLjSNH9N5fUUfC0eJgBzg+2dMehyK8dFgKX9w+QCAkIgSRHBDq8Pg3PB/s l/mysJCSunUhUVvuy8t1WTpAJhTZvvlw/+GPt2d1MMqBsmMphCGaVO4LYbRhJZTpEkGGWRDZk634 WwWh2lRKg6A8G3tCLzMwQTKvy/HYI0PrkcPBVcS9uZonVVvA5l910TDkNuOZLuCJlY/jdIzjnDES r/xyr8fDerKLiUVVYZswX2MQSyNYam6bUf4sicyIuZgklAahcRK/ZkS1OpRL0FZE26eJ69hWwqkk f39bSYjjSoVe8w4RujrQYqdU5G0tAEatJNk0e9Q3gL3yUfGssSvMJ6mkCn9n3QcUQ+GGrGsxPSKX SO1XfP+oMqFkRKgQnwViRjriRcZ50HPky8+H3gj7ec4EN+VU8P5xdJNFblJu1hEsmYfr8hvuCTMx trmn+B73FKU1+4V7LjFcbYBmidm6BpWUn84Drr3Bo53hK7tasRWeYDdYRKyDqsy7nsIS5xkU6zqj e4Uojvxdk6bmneeFbI3BMhIfzgW7VScn4lR5FBrz07CUxuTtbXnyejzrpTvTv60r68aen/ZR0INP E2vxyaVyXib0gVPj7xiwrE4nf6xCUVMggv1FZAAs5ilyDB0hfjMykrIQP78ncfNifRkY8Qio/GCK 8RHDW4eqJ+BBSo2ilWMPPL9/0CVTL9XatVkdJCsyQeT6L9kBa/Phtuq33P69biOpYp9UMbtWymwF AePFxHK3Po9YRgSVPb7AXZo131u2xyvSoaj2tOWT+Qn8GioS6WG52jOH3uc+ydEdY/yVYuQ3fVLk 6/e5JEPzunBJNOZqNMvVlG0gZK6mU66mNOsdv54DGSWHIJ5hsL3uGY1hFrixBm7MLJTDRmySSb+G oKiFSL9MJ9KJtPHp9FnLi+iZ8EbBcSskzJBLQPLIHZj6jKfdOpdkzJ1Yc0fowYJfSYty+lN74kyp e2CFaFxR0P6MvB32RvhOhktseyVNFdpqO+f5VbdGO+krsqi6kzQQvmx5r5konQdSlkpoJKXiljcD KQeaf1cgpanQvGjKmtxXfnCSULpVCFHK/AUZQy3Z8rX9IRCNkQyDfux0F8W+6rnR6m8dxPZ15HhR zU8P3CNFd7gPFK0sKKJYFFos+MKYt1IPUlhaDakn22ddVB2P+GzxSupAoXWAonKUEOV+BoYrEbf9 QVBdoV2q0BpkyZKgBvESuVBCfveNFCA9NH36C0+mALL6+w0/Zom45T2OTGKDqMuSMHbvWfNmeyy1 dcMPrbQwJJF2mkQBjX5ogEfLZEusXyUrdPViPLtHWK8G+ekbGyoVK2XaVLUFpqwkmeYRErv17TiB jdW6sDg9hpJayOQozXDVOeVUTddvAikNGLMkn7i4dqqDlgfBZq5rHvCoHfybJ7ixkE4ealOS4C20 v0uhoFEe5Ee22/82PPmfhGHk+d1IdkY20c19Cp8BMiebVP7e9Cpg9Tsxs6D3JWbWpEzGSo+mz6mh F+OpigJp9oEUu9NRZFa7y59dGX7jnY9iQRS0vR3DaDw4ndzJ7w3mzJ0SXFA/O7PHqMSBIE4kpe0s JeuckR4kR/B4ecvdKATK4F9x1c21bV/opzReUfC0NtXczU8jhZovJTUFzj09VSbnfmu6AmPN9bPt aiGfbZ585D8928ZwERdouQLWi/CbkCwJZ2D9u+Ii3LwI1osx3ROGBZpgzhamtg8iLZa5B9XxIruR 2xEdg5yRupuP+JSYtqDPp8PG1QdvslzHsYi9HC5MVv/hnrMr9zp8ugrdenmjPXmqFGDP+w/keEMD /wSocHONOKjs+HQcHWr+EZwovj93RFgKNo6x/EZ+RuU7oXG8iQmNPeiagJifZ28GxQ8anBLvFQqT ZIurhqGqdzx7CmSE1PKlsZ0aFEM6+efu+rNAUZSlPQnwsmKIAu6Ls9WS0aNeA8Kil1GArYZvFDA1 xubBU8NJuiLtLY+aXuukpPozd7g6LvgAAvv7PB782tnFw1whf6mtOifoKxnFdp4MNfuq91p9VDfq 1BBuWUdpCkN/emUIIaI3hN+/gN7Xht61TcXogrzrR4GqO/B2M2Y4hOQQKM2A040dGXmga5nkjkB0 s4aoUwH/rCkKt9dmE0athHCUZqJmkW28lohDj8Zi6+P/+kcg4BPhBStFACCRJPPxeADoykT6QNJr 65TqD7e3qYN97j/gSOiGnnKYjAWjIGEpmO6TYOdMkz/ytgeR2BNJoiSZiKTJKF5kZUcRlthKYuqo UEpUjCSg0ceRyUK/RFIjNbHA+IMeftoF2qwB3WSUgDdOz9wxxcQ9/oiPFgSCcOW2EJ0wBpfqxgtC KmeC/F0nS2PRA/UJmhRvil+q3lxpjxGsqJAyKcIuNbPOnpKq09f2fSCIl5iJy02scZvlY9wm5YSv Sx+40osESZambgoHt/q7l0yLEcByKLAkAY9Oj0zM3A4bqIZ4+03tIBskADT7o364ne4eSPBkM5+e SCVHOxesFMfMcl1SJF5qej5xlRbsWAF1iTibUzr0lQkDmvvZ6/5AmflrohrBL4LlUwXcOkE9Uteq 7z0ryTwdGeX+R0eSlRI5Nq6bmeuO2vVVb28m3b5TnC8B8/G70/GT0nhvOqKLJHk0S8dsTMfsMh0z bY+xOPMiI/NNWU50zlG/O8PuKP9nIF9HASGP2Dbc3QUvjgEKxvUGCvSEuBUxN7/I73MwR9pjFY0Y n/KSaJzTADYyWNxzGXTXTbh9mlQ6FW9Ogd86LYwKiXzTDALUgL3PYzIWSbq5SMZiSkarghIg2uAE lOigy8xGDWFmSgpwewuM8MuzfrbzV83WcX0DEXIpVPZ17UipGaefYvzlCwZruiIDu06uFf+ajxP7 CtE9v7K4z+pQrChgrYNvbisvqOPVnW4aA1dXJp4Xgm+Ok/7/DJLVXwJ1rilGelIoI2oRa12959NO n3WU4Ip3FqrBvaVUVmX7qC67/d0H31YMWZmDFFmoCoYfBHvG/rOmJLbqcOCqum4w3yhXfYW51873 KsSald39udl79wiNvmIiMEBTGZr0WqU1ZXou3HvPsANymuFqgTFIuTueGmHX7k3upjdt7PkJ5kDJ 2HJ19kzDXOCe7pmsDH4JfRHctTeCS9BdUTu+2AsT3zi1NdT69RVMNc6WALhq7D8NO4XuDUYyZOye i+FLIPXdVrWaM8rUIKW6Ntccke+tCCgox+i4bd9+tc92DwPcSyNFz0XDVoHlLmq3sDsS14vE3O8W x+5YiYuaTuaEQjNepgZofujdz+0doBd3yJowLjcZ9zO1OtR85665dSbvzFCzupFGE+4x9CkxXhIk bEBdlXrgXxNoT9JkK/1Kg8Q67gZNrdVDXplFkBRIQRd4Qg78dXcNG5snQKpMQM+jEqjCtskKVUp/ jAxA+Fsf3fbZ5DQJTQlb7fzhVM5DbFdUo7ZjdzDWCy1rSmB3zCqU2+z1oEgxtq4+8T6ZTL7qlCuz EACfQr9Uv67nteIyfJv6QK/6AFW8LtOu4LxaE5wRpa5vubJA2MpgsHrkwpSXHR+WFuBdM4v16jiz k7JiiNUWfC+FIthc5TprRCvnBR5UQuCUToBX/uaQ8AIKx5Ag6PjfhpXPsfQrCPrmDxJw+eqvhruu BPfGQGL1bkLWQqDRdjLiZ7ydqy2SH7aNYPU3sFSqWEoBVKIA6nrQ6idDUsESrmBXmTD6NzDxXeN2 3Gl0PuS+IFYEx/3isiPBA88A8g9bO0exqdz2UNtSZJYqK3hjzy2TRZEpU6hkNJdeIEWrsR1pikb+ 8QP/nO2KhSO1dmbf11AyUVSnF8jM1AKoIB9NqI2p2Tf7UfSlWnRaZk5L2F83qxgl7W/K5PBysU+Z LdZU+4M6TlLjKleNkcPy81bgvTDLaOBF8Cb/Ca3nCJ2w/c6Qt6I1Rd/S4zICrtKQ/jgKrQOpOM5J OPoHO5szSFVgSKEmjuz7ehaQ0oZtXDjaWGUDFfSeD0tbjg+NTSaKmbKR1QtMbAJTm2erhM/n81Qk Wtnf2eRkl22YW446NpIsGkFkQi6k+j/fVbCCMAxD737Fjh4c6GFQz+LRmz/QYZDAkGGnsL+37yUd FcZO60LavGTp3kvu47ds1Nfnna3qHtY57nheRgmnOGp8zjjoECDeD6baSQD2hMJC7Z508jeJmekh e2ADYI29ueug02xmk/FQozz+tWaVZrR9jF7o/4KeDWD+vkaYKJWxejjG8Q+jI/7SRd0H2qwjdzO8 eng7p+Ufs2TqO3JmZcWSqB0oNt901F8Qj/iQTUwLlXxKoCqepEkKWM9lrikunKpLi5sPZXdjL7bo kZDvPf2v991PgAEAsIa4CAplbmRzdHJlYW0NZW5kb2JqDTEyIDAgb2JqDTw8IA0vVHlwZSAvUGFn ZSANL1BhcmVudCA1NiAwIFIgDS9SZXNvdXJjZXMgMTMgMCBSIA0vQ29udGVudHMgMTQgMCBSIA0v TWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDS9Sb3Rh dGUgMCANPj4gDWVuZG9iag0xMyAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9G b250IDw8IC9UVDIgNjUgMCBSIC9UVDQgNjcgMCBSIC9UVDYgNzYgMCBSIC9UVDEwIDM4IDAgUiA+ PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDkxIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA2OCAw IFIgPj4gDT4+IA1lbmRvYmoNMTQgMCBvYmoNPDwgL0xlbmd0aCA2NjcxIC9GaWx0ZXIgL0ZsYXRl RGVjb2RlID4+IA1zdHJlYW0NCkiJhFfbjtvIEX3XVzT2JWRg0ermHXmyZ3OZBXaRZATsg+EHiuyZ 4VojyhQ1znxAfiAfk+9LVZ1qirLHWQwwYndX171OVb/frt5ut85Ys71f1UldmA39yUe+KZLabXKT VknuNkTxtHp7c8pNexKijTm1q7d/vbPm4bTaJJtNZrbtas1fRPtlFR0Tk8fb31ZrYuAya4oyyW1F Zz8KecrkG6b8EH348aO5i7Okjrw3H/D1LrZZkkY/x2trkzxax3aTlJGL12n0xhBvWyaxddFHw/9/ 2BKZo/PHmHjENrGRuceNM9840HYr21M/8MIMdCGntVKZSW4aYWOjU1wkhTIan+mkb73p/L6P1xUp 9TxLGV+Y2elM/3cn/sbNybPa0VNcM2+9hSMzDWYam8NMexzGycja+HEcxnX8cfsTeWZN/POiYH99 CGqOxJY4kZfa4fgSOzH0O6aM/vNZ9GRNVC0zjLQf5F50UJft/ARVv8SuSDKSI5dFNz0B7+UtPu3Y yX6EGR1pwOJb3y8c9e8fiNeGIkkB/r3YOvdqWC9+ybIi5JErJJHoizOQs4m9nkZNzDwPh7hiTUcE gTZkHeL8JbYp5dqjbPat6JPpknzV+bHHhQdstQOWp3idJ1VEcbxiO8WpxtlFphm98fiemt1eCbEB CV5U7EyvN5QE+h/3Hh9etTqQe+hnajiFxWVB1BEnvlUG9zjsRZs2McHad7D28G2CiSODHy3cSMxY /FPsNmQqqcNcaMkSPNlLmsiOVFQ1uxQEOHrhu5RXrazI6GgYTp4rgGrJk08lSlSUtPfYnzg1j8KM /Mv518cc/F2/7ycoowyTJOEqwh6zpcJlmwjLMmAZpW/hipxQSsrnFOeSBc+hjp/9+GK48sI9xUDS Pa9xS0DqAmohu8idt5TlpB1VQRlx8WUSNOcoFEjSdZonaZXn37i4nHM1Bbe24RBSkPHzxvQIbcaM OXycg9AYjmcESOnjBWG5eN1xJTWSK+wnJqZ65oqsZgwqIqym8WoJ2ZwuWdSa4zgc/TgFAUCXDEmV CbqIIgxiKm3JRdAMn2fZ3kmtlKqSLoCPyC7Wm3GpAWIxVT9K7meRv5KAYG3/SH5Tf0lhKU/VK2at O7ML+k+wNGh9F69LYnxDobu7ZVunoR32qvEkLlCXdc3YUbD/S3ZHf6GWVESkJACgjJ4XwrAD910r 3Po338i/IW1ZdDO22Osn3yoRrBeQLy7NSH00UBYEB0gyZXMyFUgmziUpHWl6xEJuOlWKjwjCNMEs Eqy4TrAMCZYhvXJpVpxE7Cg5PuJSyA+O3D3YwVBgaYYWIf7JI3AE+7n+XhM2N981XJFJ8hSSOplo PwVTOD0IB2IqTbLJbKUqUhXei6Hf84aaoOpy2MgXB7nISLzwcFpdqr8KvUV6WEM9jRodFQElRs6u 5OJnQBDUohTLgQgFi+OzvRDq0T1NCshN3nyUBfsF91thTiXKh8dJL6ocaZwHXGfXS/XsPHg00mId zye00jvoylIU3aAiDHZUb4iQ3OZdkCzlTlD7LMc48KFeL5JGjkYu2cZKqKv8YXErUXhMCpsJOpZZ GcCRNuYBMte58J8Uliri4ZDbwRkcj4LbOTucHMAGsF9YAaW7kVv4vo1ZF0kQDn4TOymEioL+CezM PygFqsDcc1PJw6rnPsQVLqsH/MCIhQ3UMYIN7mKDC7Mtsc/VBunBOS1/FielPPpwsB0F/x6fkiIW Irl0nbT5MqRLLSbQ1icQmM/nQLmugxlOIsKdntSroLeL3sRW5hIQqoDOgPkAUl1B0gguLVZ7KPai ckF4xDXffApyVftXBF9mrlwczWZQSh7+wBulyjTBcN+pvUGzttmrHkKHw8crseqnQZ0wfWfQSWfg LBEjHg9qAEPJYxRDGiMDtec9mV0LAqXc+v+GdlhhcgTCAdtS/R4J8huZXzjDeN9IVwuDXBlhxeOV tIlUhstUWnStqhgcDNBouhpXC9GMm8GsGK4+4T2ASc1ikJyFguHEXh26RjcFeLPoT8QPg0LK0+hr 4kTYnul60LWjb6Cr560Fatq5L9liMUnWmCQFf6WZ4mdkCKqpkR+AfX4h3Elb5cUEDo8Ar1+JhNvl O5D8Ym7/jsHXgIkHEE3K4iLJ5hSSUnpJTrPWAw4g4jlctU4GI5F0UmUpG/Gxlr7DSVDIoM0r7O2h Yq9MHxVEmbXRy6rXvlEhL7MulFm/csrMNPJ6UOENtsapb89yYy9cm1FFUkUyZHEYRKbnF966VOFB 86+ilF16m61CFVQBZKsZZBkD+b9UtENFCxTl0gY5doCiKkARocH51TuDoDUpesTxkVJ4KQB3O2or bBc3EJciUawUc0ANKo9d04JGRQelQcq5bDEGNqcXfKou7SOW46Akg04OygE+MB10mi2bJzHptzgc x2G8Uv3hAjeLlpBeWsJmbgkfzU/cOZ0MzZXMnWyVb5vlto+tFPcdp0bNMysJuyORNro1J1BKnEAu uMMfn+TEfAYF+nEVYSWDvwC0E5WFUuYCNDrZ/g8bVAe1LmOoU5UaZXnA9emxmfhdINKMvzrsDBS+ 4SyctedgWEwhjgdbHrzEBB0pHD8BGKoWCJ5lFwQvZgTXV1k3yAjvAJkUwWZ/taYCo+SLMCdaeXbi xeIEpAsFaao//PQAlIMsHkChe7gFvKTd4Tj1A77BdR8cJoMxwX4zgafM9fwGGPQkEMqbgH9vv52P FrMF2tZiMipoqpALll6sG3NqV5SdNblE3q+obXWULSwu3nDC18hlbeBl1Ez9jn/33nyBaj1jDj+I WEOeq1nMn7ckqTcru6EOWFmTEwcajAuKXGaInjBl9Kv71fvtinCpEKXkw5aEw46eKTmNr3nBWj2J TaVd2HR3jiU4gJ5SqhHhV+tojiqrdGnd3GoyzQOK+57eWHecYcxhLe+6FI8+mnDFeJrNn5pDp5vX HqQ3gc1LyGDG+dLr/FykRwW/GGNMkbXeJ9e4mkaQYvZKWv+eV9K0xONt6RWxy9UslBDZc9UQPpxm KXAlS0mZc0r44/6/lDxLsq+lzC4VL87zkNV56MOdNNSKq5aHaOpwkWNVSuqad++xefPG+KlNPi5K iyRJ46OkHxeN3PHFngKzAw+vz7aDkA6X3ApxFsfMNpI/XbW0cdY+T22yyYjQUVDJZvKM4biTS153 Rk1WZuyLTZowdDx9b2oXHFpM0vK2sWHI5s5EDxudWHcCKtyHnYy0eGvkkd3MEyxGZrk2dOB4Hwbp CbOrTrKespfyKo/wXLiFXGRuKkOUQLP+4C0QRnkdlLuhPet50BGDMUXqF3mMhJH6guwLvcIzREZ/ fSyAS7+cx1Wojtw8rnzPZ50+VL8e2ucMgak3jNQXu18d3QOchU7KbqFqJh1K/c+tiT1Ssjto1thj Q2Yki8wjzRjLBedw6cik+9giNy1v21oacGA2NZPe6IVM5PIBFcG/6Kf1wmMy8iOnpxODKvPFRZr2 Iu5oopU/8O8Dc9ZjqhsKF9OI6W+328yQufcrqp6sqOsLJBXVDEmkF80o29+Y3IGctKtSe6F2+dwr c8XIqUd1P+l8wMU4nsTvpKBp2havnEIpnrRNN5O/ehpU/DTgxPNrRLQBK9/Qc3bfaBm8SDoDyteO eidB6ldh3cz93IV+zh2Rh1GZieG2MJDzhwzkuUCTPpR4cY/VnofwXLCIr08YoyQRaPMgm11gyNUn z7n/cV5tu3EbSfRX+nG4kLTTzesgT4YBZwXsBkYcYAPYfqA4lMR4PCOQlBJ/x+4Hp86papIzGstO AEHDvlXXreucYtaPje3So5ompZ2543/F6LMQva4mOyxBHbjeT1RPKGwLlkRmJ49lGPuuEe6ga3uj cDaMGJaz54ADU1tgbEpL3dz+10qoVMjWdp5QA/FMWqbfiWxWktO1sAxB+qzInRTl9BtQkxUazKwI 2DXBWVjCmXRzJ7jur7JN9lLGSuFS5hYkLnAG4IUd0mFs95wZu3rnxIqbXSvlDAG3hD6XsKPm//1z 5KEFsFd0coJnZfEi8GRlicVvA4+XxCfyZLkHOzDkWW+mhDF+hhzbMJ6SLsZW2ccGLW0y/MItQwI+ DfDJtbB5KyVZ3N/G83sekD6CcF5JiQXJvwaFxcJWRQhRZ0v1OkH2vLu+JMMMtvigv1SsF5LV7uLV j6qO286a4NInTqNaFlIdBwnh9bjoozaxVk8GOdXmoGLGI8vcrZJzfMdT6oDOtu1ikUAN682ywbS2 hV57gfE8tmQzCTA+Ko5A14WSot/X1B9FxwMExDfxQWZq1Zo4xB9AsydKwsEKlGgXbPMTFloAERiR XWW3iK9+TFL2WZAxJkaa5ISQKZBWNmAAtXWUSxaRElqFzvPEIvVzpfQlnwX2c5/TfQeaMQC7jXvz tFQC/G5FybOeAGgdOWLb0ii1SEKuis0hX1vI18qU5repSnNF3UKzJ+0QrV/+oTHK5yDlU+X6A7Jz lTrfeKuvgHQlF8gfmdf3uvcA8i7JBaU7LkBhTF24R/3o9LHs9cCd/ig7q8hSMuYDYONSr0r1qkjW AJ+yHOIPollEeVvdq1e91PY9LxHvf/7o3swpLcHNlWRLwvTxMRtMhPjW4vsd44Ppdl2sFyBHcFvI 5mLTEak2szf1mccjOiLpWz70oYvX2gN0pqe+32XDy0IkNahv9KJubJsT6fpc29M69XD0mmO94P02 AIjH7x0X5Em9UsNGIAZN6WyHlcepfDS7NtaY/flSkfspDWNQdpJEFZK40k4gJbPC/77j0v4OWSDi D0xMwXpiLpaIujn73UpC9YxgZOXX+C/Yxdx8lSSR3eDkr2Y9NrZ6e0vBHcmNeLdc7UbQdezAqOtb JcKR8JLKGletG0rVpaekjCTadWSrJLyMhaSf8JmmvXBDpNd7UuNFIUK7Es839T6y6MOCPJusVvVv RnevG3wZm8nMsMoTq1KkkYmO9+6m3/NtRJg63kodKVlcarHIJA7Imc1Kp6htiQT6F5RAqvCyPK4c besvxKy+ldeCSZXjauYcJTPpKGTPHXRyxmzQX1rvoyIjBdig0SuOdJT+hq1NOinQ2U13Kv9YbKe3 Nt3DUkg7HEVIs2ZjTvTxpvulnTsT4KJhg1osFWG0Y7pfUlGoLqAnrMwp418rdw717sd6wYa01J1j Q9MbPkNXFkUndlh+bT3TxETDVemFnk1ctMoWJPl9j7fWyoW5FLQNC7cHHH+MRDJIaAvhhWmeg+UJ Dy1OGPPcqB0TxJBWVwEEMc0lAkUkiFmq1Pn9uwQEfvUWdgl3S9YEndWvCXDm4zGfFrcUValGTFjZ TABKp9b7LYJiPisV2tByQPylmBhWbxLU4tdvj/sICeE6ix6aG1GHLhQdQyawVsweACV+5oHo99P2 IZMjz1ygIfge76uAv+P9tJBG6/nV2YITWmP6/o0okJILlRqNUijDmUlyAr8Kc24wwtDOQyXpIET8 S+1UJgacpkSMAh9IOfVJ6dQnCcmWB1KQvwAed0CkQn4wcDdomQYqq/91042y52fdEL0yaSxePdb4 WTuUFiUU/XY7tFlrN4RsAfG2bmgqy8HKcnPQhNwzIYcELdzYP3IEMBM/T03F/xKW/DEBuPRaHo5P 7uLkiCKZs0hi9f8qKQrSydrEt3bmd8B3Hqe/mFhxLrizqcna51n7KGKUngtABWp/7XTSttpR1XYg FhbEg6jj7pNNRNlREisdXqrH+qXU1I2UgUuy063ubVXJJ0roeAXoLW/8YaHtpObEy/6GXNN3OA+1 IZ2StLSYiqkoZWKq2KhfB9fWDX0HDsO5w6gfLatQ7w5POuyF2rCTlWwUGp6JJtduv9gqMSBEozAe +k8/OLtv6wa2LgdAYUnrMb203oReX6IL+ZXEce5RsNnOqqAxSe1LwmfKz6ZBi9m8ZxqjWar0ATI2 FPNhZd2hqX/f9m1sfy6jD9WbqXnTQgZuJCFDwDweeg/yJKFBx3GrnA2fnB6kDzG+W1wpgyu0NZu2 2LfiJvG8ksSDSoQIqXiwp6KzMMbVb+K+t/Zx4ewGiozwsoqzT+RurS6CCmLyS1yVhwHzuTiqTvvB NF2KHT8k+HmBM2fFXMSLSJvFls1q5s7eulAxsKNaYOLy+IQjPy6n+2StVGsjpn8Gy8/iaK/7Ru5z napoYmxkBx523KMS0euWkps3fEbWxEpQBrv34eHQm8xb5bgHDXHv3hEKX9GU/1zqYrhwN48nmizs mXq8UE1yRlXuPt58ME3sOnds7YOtmrdqFXxk7tVcCRbELt1MYQDHm8IQjsJQieANKWkCTiK9Hxll pompvSHXE7GmfhBXZgwO/jf1uBgdkMLo4LBdey1Oo9fCh7vjSp2gCGOiNd4qzimNX67JLzfkl1Dn AnmK4TaWBR3aXSrH/QY+FRXVg7Zy06tuW91/N5nyz6WFBz0ZtRJSrUIE3U+c5LYq7xbYNP3wSLRm b7LOF+diKs5GbwaWdqls8ACKmShb91uddvec0FqXrZ44QtFF1uhm0g3AIwaAw8oqumx0WgO9FVi5 IUU5lWoFmPe5mPg5QSuj/+M1poVtPSajqHthnc98PU8XfJ1Bt5AXdhREbFOQooYSdFFIy4ssLE1B voSrhBLMdu3BVpjGISgxHVq0QB5M5IiG+6tUytGsW5jozdroDZq3sDbkkuriur2+U3PIZvVZnh27 QY501fJAXyEmCB+CzpLSFZkLeVBruUIsyc2dFUMTVnqCr0uFEtkqS5QjEkj74S06SWR9gwSGar0g gf7rJNALTSULDMV6wQLTai4U5qfPSBfgGRpRUfmhbj5JjojJo/Z9t0rO+A4D3pEHfInlbAL7bcsn lMF1/4XFPNtunR63Y26no4PKdh2Huvl4o47uVYxu6509uYBKGWIBCnZSVUcZ+kNCzu+bVgXUduyJ I5V26M8XUXHvhGV+UUSlCXKvYG2GhwLxBlweBc60qAg2ZqfeVNlyh5IVWKwEXvd6MtY4er+SekbB ukfloZ9olxMq7WZ5wYDHkSpeeNNgKp+/q866U0XYRnHnbiFyqT+I3PJK0mLTWYsVVt3NYatyTcML 8b5O2N5GRw+1XrRTHxAGw2S8efUnOlkQd3WNIi3rWoqLlVe3rM9X2GxZjz5AgJx9zYqm39cfEg21 tKaZtqY+8I3IT2yTgmezyAZQxK1El+SX33CksCNXwYqgKK4rJmw9rcQHdfK2OvAy964dhu6ggRYq JrD/7/qL0C/TLKiw802c30jliK/3OaS8/1kMBdmS8o8Pue376/2F29ZjfeEiFklaKBY9GODoyEBo 7OvlbN2MXbxpOe8OTfPIcQ9qVhlvF4iorbL6ler6FjmRn6KO8IMs+HLh1s0iys09j7Kh26z0P0rD ajchUFrIFVJTvSBRGgRirr4FQoCscOrrTPGnOeyR0au2wYso+ZpIktlKcOkEmYQzFGGpfx6DJslC I9gaKmik5C+ePHbQTjDXTlB6LTB0BarEb0hqLy2HcLzdPocTWgLTAwAiowNehJN1JRYbmpRfR5Mp Gyt5vpOL/NR+emuYblCPwFA8WC0oivjnPgHlax3VF+Ux6sauRo+E3f0FoBmzrt7q1FMSWWJg/mDU tM4O3CZS4QdkD2ihpGI/igAu3amc44tPaE1xVa3LoAHSar/MMPCojfVizOeTAjL5UPh7mS6ef8Hn vxrr4ZNViRDrR1ml3jZyzxU2LAMXcsl6iYevJBWc9AL5y3HzG484/AUa4AtxjxEs2jwlZUS6uVPI SCsl1wf0aOgAHzH6k/Nq2W0bBoK/wqNcJIFE6kGf+gsB2mN6UFzFESDIhq34+7szu1RkOYiBHhKL FEktydmZWWpwUDDGTBtyNWLN2mFARjQqOp4lET258fyE1QwATzopAQDirr+uFZa0EXt0YJizqX9Z pEIdxG+3ykXSc9Zq4G0DmulO7ohOzGjPREtlr4R7VtcoBxlidXN9BMGdOyRmZia2cguQq6yARPWE wASl+57nErJLpz0jtlLAR7Jps1pG9xjAXLFZ61xRznlWG4Ng/9g5DgD5APU/uY5ZUDMLIPxaROG5 w9GxxppwDSrqHCT3YI6OfI7p+sZZS1fH/jDrXZdbH6cQ2TYP9f8c5yIlHkFzeeHXxiwkYS1CmHUP mvITQTXZn02ePENpBuAZBxvX4iKcXZb1jbYgDYq7AvOSzYQL9IBng7j7Uooc5uE3EiO3amkYKhlr aZii4vebpcpRf1U4G0rPI/bUS/dqO2Hb1MtjfNkYtcDVxPIzxHsi6EVofP5thDPb58b2yPtu39MW gJ+lAPgFWEcKfL0wIxE2hH9qRHRrQY1Iw4Jpmz0Ao5KrMCXOFoUdrZgnkiSAYPbBfu0llG2pN0jx Qdc7mTzzg1N/0BGJja5GCjPs+ZZLXfgIw1NmadI10D0uMvrvgS7nPzN1RZL+SmLvWj9fQr8TYc+M XRjlnDdG1ChEhuFze5GGG/UoWVk5ucwWdYEnPRSsC7RA0mV6a41skXzBLGmmflH5AtfZqA7Ak7iz 2aPlAnv+11A6p9JSCzhg+p9x3g5Ib4ylCiC915C1reY9PNVFWa04wc+uKq9uFSxwR1EYr4crLeCu 9Ld1+iCAQjF7GPccJuS57G+njjhVEy0WeJwckzDasoRnlV13iuSM9tC+Hi76aPa6EiinGIbubUpH ajMFcWKYZd4c5vEwdTpRGHqAEH/ohk43oT18XR7ls2x4kw1Kt1dUIO2mgde1FcNWAyBnSCZ0e7L+ d/ZDM6YFepC/vQDuVUfTrspZ7IauPblpnnO9iGo4WEzgIFDa8ZpqyedadUyq2d6+P2o4kzWPy4m7 8+q7g42ywNzx6u1panuLY7VqirRz+sK6DYGkqNpksKIMRlxMCtt2OS4DTgHxOn7/0GT1C14nXwqy PfeytUyKRLsHgSaB+SfAAAyC0IcKZW5kc3RyZWFtDWVuZG9iag0xNSAwIG9iag08PCANL1R5cGUg L1BhZ2UgDS9QYXJlbnQgNTYgMCBSIA0vUmVzb3VyY2VzIDE2IDAgUiANL0NvbnRlbnRzIDE3IDAg UiANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0v Um90YXRlIDAgDT4+IA1lbmRvYmoNMTYgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCBd IA0vRm9udCA8PCAvVFQyIDY1IDAgUiAvVFQ0IDY3IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1Mx IDkxIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA2OCAwIFIgPj4gDT4+IA1lbmRvYmoNMTcg MCBvYmoNPDwgL0xlbmd0aCA3NDUyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJ tFfdjtu6Eb7fp+AlVawVkfo/vcpJ0GKLHpyg66IXm73QyrRXjWK7Er056Wu079DX7PyRlr1N2gIt gqxFcjic329mflzfvFmvrTJqvb1p07ZSGfyjjzKr0tZmpcqbtLQZUHy+efNuLlU/E1Gm5v7mzW/v jdrNN1maZYVa9zcr/ALaLzf6mKoqWf/5ZgUMbGFUVaelaeDsPZFnNdLjl82R/kE/vH9U928TY/VP ycrkaaNXSZMW2iarMs21mp2j/Vx/TmrYn5U/yNHk/sJfp6RKaz1MTvln/oQrWVrpOSmQsOfdThjt +daeNpF7avWounEML+2EiYH3Oibrw9kP6m8J8G702yTTKnlc/w6NWbAxrU0b0+ZgJtG3akhfMo3v 5k9omrPtTVq3pT1TZ1W0TsHWUQMJW+gXkKLSB16ML/wbDnd0qOR0Tyun2Ai1rAc/dF4oJjgpxdI5 yDUxA+fVkQnQUnAyTKTfyhZpZYpSwaYpqyqKW0RxKxZ3RkPD9XFUwMGmhh4vWO5KezUf+cv1LF2r t18TA7GmVae8bDGTYdow7bWR27Sqi1qsFiSQaLp/d3+nNsTHsGtbMN0K1E8y9qLwEg+UrFD0gDWB YdaKB/xBHbvJD/1w7LxDhzTAW/mp468Z2FN0NmDRwYMSLcTTsN8xnVBtIFp7N7zEfaF7ppWDUxAW pf2cWLQBWCyHH88Cr0yRNrnJyQNFET1gbRS3YXGP06F3mxNwtehmS8yBp9GHjRuVG/mdRtNeWOyJ 3s8JRkVw3TZpgyCNnjC1kKjz1/4A2TJj/4uYB03O1KYMKuAXqpAqtU5QY7QOwAekN/5F4Yy+VR2J V4LMuKvuf/ojRnpLWuBOCPQCvif14NIdbadwUzE/OZy96uHTYOiT1XN9lCW/6BMLMrjpUc18sRv5 oVHtl4y8+KjkdL7KEtNGBRki+8lhIHVKDCNmBHAp2uJsmNJGMz6gHSm+Pl3FL1izNgtrFgs8DhBy gqsQ/yVlIwhXAV5AfhQAGHv+cNcuBdRsynKRFlEFyfOEknh6GXqnIK6GFzd9VcfD5K/ka9K6bBd8 8hivtUgnwfYlsSVhLuARGJnyApDb869THKm1/oXWR9f7zg+HvVCdb0B4LlYgXOe7kKs55ypGtslA enWYFGoCaZbgTTAUOxLivCrNtSMt2fZBT86fqERNe/o55wuhDiVMFRKmIIk6fBictyT/kkA9qQM5 cxyTnLDi0PfyhHoZOgiVzbAV9m5y/OzrPIQAyepFIpomus2e/Vac/UaSQO37nvsuw8ucw0tC4e/q 4T6poBoaZP0TQW8BlYVwxTIC36p/qCI1rEDGe2l9iwYAJDymOZ/kfPKYStWBFAOseFV1olZWtPp5 m1AF6anMWGoHGkS/HLPc3UolLKkYNYCp05EJwZMYRbzgQyUrYenl5jMdDrPqR+4iwiuzUxu+QdXG 6OPFipkKxYzMTS2tR6O5W2n0NYvtwOdB7O+LeN2C1Fwh/l3uou/1f5TAFxGQnwEmKyJe323R31jt rqRZ5ZBepqxf5dL/CFBilR6QQ41dxJr8U+l31EV8gKjgCsx/ATgG2bhNVlhpryqxlPdeyjHvdnu5 DV1E6BV3Qyc0fAaCByGOEFjC9SsWFgv8RtdNY9ijJ6B9PdEXNK4bMOKP3O7wntQUVve6osQmxYj6 w5aqA9aUK0xIy7Kw/wcgj61fVp09UEm7AyA33CerGmzxjrrne+rD7xSSMJJjb87/B8K8A7WktZbr gMyBU/eEVJ6opq5/DXs5t0bnEK2X/ce0c/6yA7EMK2eYjLqYoMsGCzuJivn3QoPHkFjy8i2mLW64 Pf3EJbchreZtRhnkElEGF3vBNrCKte11P/casQ/YhZXyCEFCK71QgUHVnQTSUVIjaQAnY8B2Xkd6 CEJG1wyfRpd+u3F4eJ9ggXpUf3LQOBGsd4kle3DXQAsFqY+GOZykmdhAFO5P0FAhSD5hlRUKdfDU hDwvr08QbNifnHuUp5F+XKiT8hofTsziFFiANSq6q0KDw51NzydB7P1++SY/oLCbg2lXlny0Wz7W MRO527sf+N5HbT4m6o4lUfi4iHOkvClpJog2vihdr+Y7nGlxfuWxzg8yf40uDn6z54IT5rrezWHQ 28buh+a8Z5n71pRKBlKvAO0+cMSEyY6qYh0GSZ8QZ1n1PPWFITPBhXL8LE+IA+3JgMnU6onJZXk5 dMpg2sfJNLwKlFN4Z3Qb/gqsaCIkLJ/P7BaDLlrnntXCcW92YqB5HuJD9O7YfY2iIsH0a3CeZZKP Cflo/atvzH2/6YaRoXhyirKw+RfVJVk1+tsVRn1h+w6M+2P4scRry0yBHhrx8bJcgOo8VrIIT1xY nKcOotIijg/To/vFd/vQZ8KtTzKZUjX6qC8qnEyWwpEKWmCEsx0n3WnBwF3Pqov5sIEkQQ4DOnFh TykvZBm0C1uFpiuyE5ofhtNco0dySKe3aKkcgkEszJUjJwtTPb7ktCfIwNqlusktpA4VbYMJBCbi PgemBg2lg4ycY/1llUEXMhgCG5LRFNiQUUzs0z8nGIxMEGUgCgqnAoQPIhPLSfoAEA1JEff5y4FQ MJgurFRUZ9CV0kOIVRHQ1gx9FUFTvZDHSJYaDGn0TgnreBFULBizqjDhISrcfVACT5CzAMh64wiX cwK7PNzgtzZKzsIrl49QqeO6XlES0s8d9D0Hpjv0h5E+biEFbAP3iA2/1NMTwipwwvHWanUh0uRY mS84W7Rik5Hpmf2KyeXWRJ1FI7fdhSU2LIHchjRDXjJoQD6boqSCWNQRq8uI1aXUwz88qnfcwVcE 3Fb+dsx7T1YgLCuxLkD/0xLWQZ/NwF4SgFpGs5LQrMT8HA+8IYwCi6eL3bHjB3qx5QUTTKecUKWf hNk8nyOGiwwShhpTIoo+01ZQafaqu9RBfh2/+EJnrEHvZKhqRG1od6jstFSVwCZQp3Aw8GrmV7Da Ev0Y9BK2SuSJYg9yPtJLLuTM62paRw/l7KETNZt+GIe/uqXyBVfYPGqfo/Zclbic1WTWXMxU62ly vI3+CzLm7AqrATtoObMqOatiNVfiCgMWBBcOQZBnWsmbs59VJy8y2YYP1OaC4wudDSQEVv4IvU0o 2zkmJ/N2QUzRQVbhSerBnZIls1CjLLvw/tOFphx0bB2EeV7tmIbTZ5E953Yyt+cx0YZ28lH9PqHW /4DYD4i8UU8dvoDFpaemYNjvaAXD0RYTtwVDULMAMwBVvANVAr7A8XSiFTmsJR9gdXs1JFSpNdVi gKmaOCQ8wJQwf2Lon6/GHjB21dSLWSGGnEwKC4fU3FzlklCE1NI1SB+TU+aD+erY4enBD51HJ+gJ 8gUrSUHoWEslgdLBUaMVU/Eb6C7u+ltAF6TZMZhB65Q3ZXmdLDZOflmzqM0yITTaT90AHUQnEAqV MMxkPN9a4E8ITb2AoV4AYpwvu2tzA/C2xfeM3RDkv7J1ky1sbeMwFKwNBoJZpBuwJyhJRqwgEBMb HD6ps8xxphiesH8Bpah1QdpnWjumeiUuqFwWYKH4dGkvBabbn5I6tvbflhr7xz72lBQjH3XPIcEF g4sHtxSU/C0nP/AWTLja5vyGVBgmyIux8y4cyO8/Sa+2HreVG/y+v2LQJxvY3WpGo9vjJmiBA7Rp 0V3gvORFa48d9WhtR5Jzkn9fkh9HF8e7SVAYkKW5cMgh+fGjSKI+Aa0lpWZV5D9Gy3oJPNRoRPwC fgAwiN7EmpJe1BSvC9u4bwauF2hnllgkQZyvhv7WtChkzFmnQ2/jMVJFCwBqM0SMhvDXZJr65xH1 db0U6sxyZimrM4NO0x1BqSbggLBgwyNsWO0u6r0EE54i24lQR27GIHmZlfMCgXeMTd1OwvCIQZTb FCBTgZ9KrvJmMix8xrozpEFEKwrzMIHSzJOlLkN95PlWCoXXel6OQKdI5vQg6MqVmlGgnSZlKzLm /zHjLYbm/cSefaRogizE0+T/qLxcKLONf8wjM/BCD17IRLnbho6Jo40ze/xNCC9/R6wHIY6yToEK 2eYPbDjj76SC63iQqftveNWRDSNpDiJMFFtaiqPOHc89NyldEJ1dPBaHBjl00Q/YFXTvuuNVK+5B Y4iPyeTFXlW518lnirxjL71UqXKfW5ysZqs65oTZU8D04uRwnbVdYQab48uanY9nPfJKLudbUIWp /NdmSynGU41U+k24NUohuL0i4KmUkrMQrASHYBitNM8SqXF8nImfXzG5AZ8YuINE7wnhiBDz/I2r ziSsHkifxWEztVTrUxfwtqHKRHJHGY/cjVSrhzXV/Gz1T6m90tRwVXdv0CvrR0DJJ3b1r27LSZUj 0zLwAokBfppNjU+meLNVUyy4BDeXCM2/qzQPqazWi8+gx7Agi3aKkpck1S3TSRlixBacdXKDsl71 CPVGjuTQh2Jx08u6FEXKiFiZnhgPBHrliy2i9QnjbSDUXJq3W1cIWB4TRkUlZdWfRe5o1ly/AWoF c1gI+oqvQSy6vx7cdiz+Vov/O64PgFdqBiG74qgZb72UIxKFPisWV8LKMmVbfoVdIISYwCoxixG2 oRRsMNbEvlem9jJ2z/lK68zfj4RcdcSrQutDhfujBSoj6kmQftzUram7UN/GQT1AFVnYEW2gAE8E R+pt+IwTajJxQFLMK+PISqyyEjmEexlSk5VKcf8OPvfCoQtpzByjpo61Lecm3s8yv2PHkxc7jfti FeejvHncZ4g3h0ib1vScoQW3j156Jha5Y0gisrNpwgHj0KUFnn9T1t/WnZL5fbRIlR0t+YIJnNWZ 0XTNIs9ZlE7z27l9w/I0pFRK3zy5ieU76hz3qpXKFRG/3uev4wr1BUhBSRskU+h6wdx5Zupl/uMB ef/B9JqSIaJFz1FkIKFWACBiGCAn5vNZEUu2KD6MWa+ZjTVdPyB770TkrmlboyMxgRfrd1F56iLe sVtTnRjMAt2m/FebFlC2RFhNjPoCQ3V2oY3asL0OHEk2svMMFy/EMhP25ZTqZkJbBez4Q0gohZ0w VUFsrZv8tZen0cYDLYfTJ5qMTFduIS32HhfDREF6PU81wnQfehNPZM5uGZJ4IgTdGf9Nsxs7R1n+ CetM30bl/lxL00m5+jvz/Wz18CH2BZn0BSxledig2rCPZ5dhcBbFQtjOcYZDXO7XxfvdgGqBU6VC XRhaRMoZM0LCSFoLECBQ4zgwpG0mMJhJ6svcJ+wwa8fabWsIZXZYzGvFrIS7MdWSWQnnPudp7HJK LlFEHchKruX8uZfPs7yf0JOI45iOYYWRHQxcKywj0BUmw1TeMslEVGRSPSuJWuQk6m2prDiTbiyX GDf8EmYLhnUq5QJy9eBts9MTqFLpOhOteS9t0b/Rzxm5EmsS+vWbG1fcJ7509BEvB3VUPcUXQQXk cJB/InPskVIwicX87YkkNeYmrbJ7761JMwqI3KTUXjhDiehLCuKb3c27pxtKx1wOlRefJvdpTtef ZikpYJ5eZrEiEaIhEjbzIDkeevWoqk8YX2VRe95LLn/6L/SKS+Qo1s3KyRT1fq7bKCtLLalCC/OS VUruC7KBkoTkXzfCupwWshHe8ioygsm15Xoq8WW9VtaHVlKII/7OgX5J70rB4CqJevncS1KmcVZh AbWwEBzE7sNyeRQe4nJyvCvI7XfidSfZuXQ7HVr5yenJhdNzcbr84cmiqYWNTneEtXlCV+U9X1VK UZa/6W9Hbq7yxVWJvxfuFmd7sdNrC+m1qVH9Z37Pc1dMBqTV1NpUMGNLnRHDfiPPL9JxUPlcF1I8 5MrQYFTaYGh7QCV6jTojF19INQGDjbteAEe3jKuFcMQqckRZqYeMrLICX+SP2O16cSNP7KdMijkp l8W3Szfqjad66d6O2CzhS9WItT8TsWkSfeDG+pfM8LnkQkOut0xDmsNAjHLgzC+pYjjG4BAE+oBK 2UrBxgnYlACb/Puoq5I0n7ymFHQMO45xxpoZNWv7KebozkoyNnVsIGFM+YOQoyAvl7YmZQw4TZ2D aIsnRV+zHH8LbDjoIIxIzeMxAjsgvdCnlAf2M8N6yuGUCwikNNc1p+awpylTb7ojZvCUu+QAe79m XeQu7eVdUmr4vKouc/j19OWqMd2mJzQUVLx+m7g8mnYlQ/lPgCZl4K/En6Wehb0y2jNxYARgIY2U sEH1CxexSIizS0DILsqYdyMgWIcAE/KQCR5UjJl8qa18MNMiwSXYh4uTAyax4YAppLwF6yC6Cb9T vpLHaWT8q5mAlnEXF3WJJz7EnI54wRPin/VMYTopN3ANBlTCXo8PkPtVj9+coSSBFe3amqNaoKof d0zvIsDdGp09RDNla69qYU807DtUgu/I0yiJBDk2fzskiFvkvxQUCZGRMuZqaidItxGYOK3KFZ5A 40pgVLoDRx+9cB0jWOUEmK1EP1xTya3nAi8WjuZlX7BM9zLpOpxHelauniGaK4Q0X3Kc9Lb8Rq4y JywnlKw3f2C0gWisGY7wAEPFewnhx9/M9hhPxKIjjhnMqTtCo2YbTH29a0nL6XZK3I5EHtHx+tCj mxLnZ5DLTaxoWUpsSIdUaDYwhsuS/RoNLSpvGiXqH+4bJpW4b7fSo9TuUuwuRKJ8adw9z0+mMPiM bxzeoWULKgqLPsHFumOH9RC6UT0wBxlq40EDw0qGY/qumKyl7LJTXmSazfG0YP7C2VWo5K4+aMCw Xqdjh2Fdc721sGMxtdnUWvzOnUqGkuZxv7h1qMaaB+lfU24seKqVwXoI3GSpgdOWLT6aiDGFdJ3C OV64MWEgg5ihiQKDVBUvFTpDhS6/r9DcE5b5jFhZNxpUzuq04/pSyI0XghZcsEquL57d/nEV4SNN yBDKX1cm3BkQo8nfLtupJ0Fctl1Rxs5gXiO0nRyZypt9gr1auqVP+LjWRiGtaL8fNfwhlU0rj87x FQVtvDAtO4baRwopciDzTn43zSN7zjJD5/x/XHPe/2b60INP9iCuskU2KJPcEdUSXkzBaQbywwTT EXRdlf5i8+Jyik12keKuzUePe1ggIcRNClO6Tr+O+KNSLGVQP3eUMVS99KvjrCwQrh7hnxKN5twb RzUbPKFMKZDDzxpLD3JiXNgPXaOHN8hBzO/lec8lllbFu3pYW75VgFkuKLaQT7P+4oAuQP1RUUxv wsTk+fs81/dZbeso5t//h/1pVx/WXG0+ruO2P6FK07ambtuoj45SMLzHtkeihRmFAYXKVcj/PhW7 sAnNl5iBCZcSNZeLXTE+a0wiV8VaQjVGau2BeBg+K9lnKcopf6nYznDoVXFSlzJkjbE5ABbChdhm aGoFhyGeQDWQyt9rmsrahaqmjcPf1nZ0EGkl1Xi+cKl1G++mY5wXpT4t4AoG/NI9RCNMM/QzN3k/ usn5sZ/KNX8CaQDuAkphyc5EWycvrQEjsVhp6e2bDAdZ0jF8lKiLd6XwM5kEbv/16ckbytzdDROc jOJiak6m3mo11N2e2jfCO9rgsIFof1Gms/ZhrFxOKxeRlQ+MNVyjAsMNR209xLemj65/Gb3CGbgR Psw1NVfSkIqp2MMAB2cTTGyWs1zrMtRJSeWgsyr/hNlWmBsatf9RXiU7bhtB9J6v6CMVaAxxJ09G losPQYzYt9gHjtSeIUBTAkUZ8W8k+eDUq1fdojiTgX2QuHVXVdfy6hV8hUki8Qd+dGKf1JibGdRG NWVB4nmrXlOpuSDeqsCyNkbOCOe933cXzfzzhh1cmN3BI7EaZYTK8goNZCtA1ETEASp+RqwrdEq8 P1CEu9lCguT4AOlf4t3kHql6mrlzixwU/T6sQbakQZCJ/cR2iCDpohAjgAzyVgrP8kxnjdRU2e7P 4Ed1cjqG43KhWU5NRxYGLe/DyeHX9z9GdNr/sJy8OM812iyKZJ66fpx571jZmZai2E1MAPLfFUpI OZk86tXHtY2YI+n53kYvfu5NZLjOaBKlffSOb4M+IL0xMpxPALhSJgZ6erzoFj76rTZwmBG7MLZ2 +1ngJ6icw0duPdF8qj7yYTgEu4JFITTwfY6GBb/+pm/r5E61ZM7/RYlCcLq5N0e64KdPS8dndczf nI7vxzvkQ8GgSUtDsoX8FfYwfYXbGzRPdDwHplpqKeMtdGMZdDOhZIvr7vuhJ3mceTEhgo3iZNlf JxfuA2d2nUP6BJm8538vDU9MG3WxlM1or3kx4Vw0PnDR8RPPc4WeVpmJng5sV3junpYyyzWVcr2o +WrEQA0w+N4HD3KWqhfs8kh6duJl7u15VIrZgdtUyfDqlmgKshRZtuDPazbo3sxXXpApmRG8AvIx G6WUAXRkeedzr1+AefD9DQHsHHAdJHB2Ri+FwUgCJmcfXnTAxeTAe3imVuKurFLhNRniUvDPc09g 1W5oM46wy7QST2bCLnc7sEsw3/qGJUcGHHnoDqwy8ND0/3louyMNTYV+pwsa+rSN/vnHBu7/6Hj1 ++OXZaPEdMNUZBtY9UsJueFZhfTTH3IrZQIW6L+h8XLVFBEmX0mBipUMhfw6iaJiK1d7Qi/HhhPb vd/3NFAzOjU5+07TbCkL4GT60dm0JmUcHfogYNadAx+iAw6HyQcbxfXf5azn+WcWB4OdDQYoo0xF 1nbPf5wlN5+Q5VzpjamtbRt4s/q4oSMEo8I+bUZoEc6+dLqjH/zB4oIJCgzyLVhFqRCiuvgvqNXb C22dxQLBVZ6Cokwp9GmVBANNsWDJhw1LQAKWFiUcUpdtLO2YoWnI0F8/OmH1CawrRMEvG8TxLXwr 2KWEv9ZyRzzQFaoEDNVGo1JpNtihzk2tNcOavTEzdluRzQsn69xgQsCq0FzjkgulGlutlFnpSDk7 P032cjq7cKdaXbBp1N0PS8vOmpbXjzptiVz3MzAxN4XzzcRTSzrR4NnpwNYmPJ5E9HkP3RrAf41i eP18XuYxL2vGAfHVbAG4ousKqGqaFToC6AQ4PvDDh81rDXuN0dFih6zgMtKGlkULzOlVtiQ4S3XU Vdf5UPf4g+mU/mJ1WCqF5EtK7u0RnknCAp0DL5QSwTdAaloI7am+AVLTrDJMzTLAtSBq7E/qpshz s/SauD9tlBiNWlHK+9OEY1OhXJ6MU3J036G/3HzpbiNPwq3QVdskEEVe9GE48ElDXqrrKnoAG3r7 KmRFpozBlpw1X0utX0WUszcUiG0rmO/OQpY8gM33qzNMK9M9u6jezkHBRGTgqNPGobYw8Rhq57B0 G5ukul3iJNUqUchbAcGXm2Sa14iPRbT+loiKFaV1yZuYCucIRdCEImgDW28CxmWGcY1iHBBuuwgc pzZd70ddTY6hhZPpeJPBSZq6wvdOROo27AWi50gQ06FAL4UigC2t9N2j7g97Lvo0HMCd773DyFHa iCBUfgP8ux+AYI71Qz4uB5o8Syiz4mxM79P9QZWdpbfr/CgHWMdsJzyycHktYJa9HDOZRb+J2MSY tW0kNjchS2PIUgsZKmtLUG6Yg2lyQxYr7R7LUjPWKOj1qJMu55tG+0oWv80mU0w98e5WySy1NHFL v/fsIk3YjWlBhU02137diA/yRTcFdei4uB+0UjgIZUG+v1ow007v3pFyoAOUcSzNNG1UvGmftrd6 guGxh67OOgXryGrUFnPN7Ab/nHuG1+t0aBvUZd5K2r+cDLtGYvs9ydDUr6onmZAvho4344ZNArAj CbHBZOodc9moS43CJKcDKqeAISFFtlafec9/0LEaNLdNluVeqJOx+G+pqb1+3/tpjDIPMusdL/IY 0f0fzqgP8u7xSOGze/M77/7VTaSSYMzLtoA5zUYywEK6SOweCrvTaTqeJoFVP3D/13VMGh0o8gYA +GJQWjHme0IiNQ/MXMdEm+Rd6JcIzbDRCtyQi9xl9IQ2xky5G8cGcs2joaYAbcaAnSb/RZdTwvFy 3ihr5cVbrvKJ/7bQJI26+dXaL3Wpfb5pXnRKU+KQT7zy3wDhwl2rCmVuZHN0cmVhbQ1lbmRvYmoN MTggMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDU2IDAgUiANL1Jlc291cmNlcyAxOSAw IFIgDS9Db250ZW50cyAyMCAwIFIgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94 IFsgMCAwIDYxMiA3OTIgXSANL1JvdGF0ZSAwIA0+PiANZW5kb2JqDTE5IDAgb2JqDTw8IA0vUHJv Y1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL0YyIDgwIDAgUiAvVFQyIDY1IDAgUiAvVFQ0 IDY3IDAgUiAvVFQ2IDc2IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDkxIDAgUiA+PiANL0Nv bG9yU3BhY2UgPDwgL0NzNSA2OCAwIFIgPj4gDT4+IA1lbmRvYmoNMjAgMCBvYmoNPDwgL0xlbmd0 aCA1MjA2IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJjFdLc+PGEb7rV8wtQErk Em/iuFnbybo2m9hiyof1HiBwRCKiCHoISiv/+nT31wMOuFzHpSoRM9Pvmf66+2+rmzerVWoSs3q4 qed1aRb0Jx/FopzX6aIw2XJepAuieLp58+5YmPYoRAtzbG/e/P0uMZvjzWK+WORm1d7M+ItoX26i w9xU8eq/rCCHgiQVPvqpiSxflKZaLFkXiV4w90IYs3mhfKXyzdOC+Fbf3URmKnExnrDeRTqawJ8v N5+ilWv2x0Pvhlvz0Q4v8ayal1HvHk2zX8ezJM1oaeJZuYg+xLMsnS+jjrej/WP8efXjN4Pj7S9r sk3MZ7clAvJVQvunnz+bX0g664xnOf24x7ia59HRsJZ5Er3EaS5KeTHEM5IebYXEWCEpoy+y7I56 CtI9xG1A2uFoD9lDs9vZNQjM9xOhJHMxLyIHSjuYRpVgwzO9/3c8W86ryBv/NiaiKvpoOtX7ENdk hItnKYlujoM7CX8L8aoSe87Ojenu3rGkLLqLM/r/3hzpJtQmjQyod96EfXg2GGd/C8k6Z+kGsfMq N7Uws2SeFGXpH0P29WN4itM06tcdO5eQD8lC4pnR/7YZsNv1e/ItjczQU+y28m3jhIx2e13MiG9J TjKf2Z5k9/4Yl7R5y44VtH6Jk5wioapA224nAsBhesdR5kfwFjwfjeshFGzWKSVZ1KiMzgt5Dgz0 b+oprke/9moB9k58YUtdkLDWHobmfmfNwToNR0+XmmQkU4LViGA43tpb81N/F0Q7z8dop0v/9n2w 6XqWpGtt2Hp6a+1J1q6TQBd0bbycizjKwiQvWGZV1GM2jyJTFfnpu8/mP0d+gxziDaJiv8hvJwEp KKo5aRv8ci+HStqoa9hcQ4zp8ftA+U8MDzEbN5P/gwrZCj2HnCKJoyOQZAtePdopPWSYwbaqEAKg VtXteu+F/LzCwhEV5Kfb7UzjhTJDUtFtCGSkkvRycBdzbkp+1bp4T29FvYMtJ1Xqhb3GyZKfwM6q hc+hi67ZhEtzEQfTqvDjcHs99/KgFuS4O7ww9qp39LRYZ8MwwdBDwgk7rJlurI1rDl1cEMva4FcZ D65/Bo0Fx1OcsKFWOQch5osl36pI9QbBLWhXJdsdAvcK1hlouTaQXHvYCZWX9Eo5QXGwMf9fE0ya D28h7WPM54xJajvlNB0IdAbnsMy2W6XroeesRTBdjmCfOniMfaqs/oqUKMd80wCvxMaELmnJljsr CZ1RuClvGX/xPBhh3ptWspNDvURtyjhsjLIn2WF8HVADVB5BPycyQCmLqMQ0StDJRg9Rple1BtQH iLZt1+xu/Q2o1v0MpBO5RJIQA0jWjVsbFWi9PljIMEULC2YuTKL+wmhVeAqcXatFbgfSVzi830zC m43hrRBeSmYWmomUDCGrkJQFIww9Db41qnX/8Hv00Li9sOJSRSnGjDZmMkdXMpzxOuXX2yidSjL3 1qzJvQJYTknJQM014HDYeZrWuqHh6q9mmf4wyFJldr9z1GqpaksJNYszsgd3miHICzZX2Al47F4J 1CwIoCyFDLoueloEFHAqiF2enJM/0eR3/RBzievbficfhvJMMSHhfcYE6SV8J0N3nkjJofoI3i32 fLECm9ieRSN3KvCuzIIAeaTEpp/KEVpnvrbtSC1GxfiaC2gL9Vc2OAsyG2jzigYuDumZ+KgKOFyq +2oRLbKxCdaYlbCq1OSfXS2UVwL+6We6lzz6bH7sJ27AEHol1jX4xrFAzxh282vU9lRXZdEFMQHL 2eVf4wmo5ko89f8bcSfwFBcTxqWL2M9jqTt6BXfk/zvxh7+4d6wkxUUc10dWps9KTVN/1F26rBbn JxyvccyVzPsiVumxc3b/fyN1tfB9DcuSVbWC5ASdzBnrBJEkpyuUrUK6CADp4MGMw8WB+mPI4969 4vbj4LQACJPiscL7QImlJWCtJ+BvFBV3DD+N2QCpe6p1oB6swwfNDm03dL9bLP+gj0vGNu7cxf2T m3dpQ2qt1AUjOseIRhrBHEZaLi6MPYRindAIF7d45bRXyqRXIoSupL+K0Csl0dGiE1IVB/4nM1BN eDpYrot39KCG0ICGLbA8gHhSijDBS47OK488sbx4mue8Qc041R2taGoCTVNxcjFqeag7kKaiuH+r /LF36+n660uqcQDNtGptAVIdgahrZLaiUEkwc6llfCpmUgYZPdf10BicW11TFPgyKArarTAxxyGn 3Uch5dqQ0RR0T7ulvGm6YO68aaYB/UmBJJUabCaG7NDsvAJ/ecbMRzVBseRWtFHgHz2Q1cbbq0Vh 6UfeVAmhl2OhNA88EWGeZfTi05Zq+EoKB4LXHankworD6HOKtknC1YLOh08V7rpnWEPjW9hZjHe0 ON8RX7GdtAN4wvzAODsz6Y7y6IIEAIKWpBpXG4iTdsHun2Pfo1HX0YPySby1I7e8rQGNVo7Oqhz1 qy66Qj8k5n5IrK4bgk4+2ADEZDoDVn4GLKX3osAdMOJSfG07dLCYAepCSAciABf5u/MmwuMewo/X a2u2HIskD5YSeGmGe77MFAWeVuvmNeYn+xcuaJnUo3PcsbIxV4BDg8JQ6Nr8EGOs4K176p+5ZHE7 ztIaSXl+hbUU41qmK2bbCf1UgZrC7YIQGs8AIQMAnvKQC5/2QCVlYK0FNKcM9MRfZLe1Bwg10qOM Xjt1e6uK+vtn+YIX/Undv7dTC1QIqCBBKdU29Vp1rq0D6bNKuY5eY8+daM9NWSnIJ2+RR09J/vMk x1iSCV5ygSXTJMsBA9lI9eHtRyPd+yyXNrYCkObavvMvGCj9FTsqFUpI8C8wGAFtQYQsskb5CE78 Fic9TSUGy0bFgHkN6ltzvHBEubw48XXjBSxkZJEDtkxrIHyQH9g4qYOl1MGxdNSjIc+h2878ucix AB8QFe/9DCEtGSt8UuPiWJh6t9Ryl0KCAjrulTYfsWJLOEANONfYPcNdUntD5UlX0pf6lBLajWyi lAfyG+xbo9q7cw7J7xZsykTdqDSqgfXOeFKpZSlqgI8LWe55uVAvz6XFb+B/J4LHykIRhBKqLYep vQSArZ3caS53ej+qXcj4NImM99leSLPPoqXZWY3SwF1jWI3ysRqVQTXKIzuZClFA6F7sTLvumIuw s83tBd2WFdMzcTy8rruWWh9BfylgiZ8YWf5O+ldqoPjU0LTBi6YVzCdyqjewpONSUUdgwxa1thQk CuYPPByqwSzGGjKJP/o9/7814HLo2ZvWqvQHeHGCsdQ49rMwKldqhf0S4GKI0jYOkZ2uLiQhc347 YcPZNaOJREFh0IMvCJ7YpoLrXgcMlZm25DlQ6SmQJ/k6HHr3p+D8JwQdh3dSnzjF+HBtwmLWqheQ 7zywix2vXOPrABREo230WFn4XRZ8C+K3ugBV26Dc6WRVBCRXy/XXxYDnGsmwZgKq5vthisA4pfHk EIBqgNf62YLcFwI6OJoAQql3HLHzMZRvuuGcfMsRUe2lWWG7mgRk/ED7sBQ192oCRhVhbi/qhJJ0 6tvYLQxh4Wg201PvTaxop5Xq27XrelnOszEZxqmCro6fBWPTQa6cnkdPDyMd/d5j9rTSUQv9CzdU FHD36OfamMeeBYe5JnuEXf51dNDOz9YEs2RajniVn6fJX2iCUnAgDMBAqlMtw07mYYcBosO+4nRC bKDHNicW5Vn/DGpFl/ICDmuBQ7prIngP9sE6fNjhdvJE6v9xXjXNjRtHtCpH/4q5GayS1sSXSFRO ydpJtsp2bVly7cG7BwiEloxoggEhOZsfkt+bfu/1AKAkK6lcSMxMT0/3THe/1+x5LnNn8tEio1Rh 0x2iUW5z/eSYaOIXLbsx9WBR5Ctutg4w9nz6nUec3lDXdsP9qwQkNU/ek+2qatk5tzWeD6FsUJWK aBYsU0bDmfa5XbZJzGBoyQ4kPr0IrxVt+2GFjCKtRAbqCn6OxftBojJAqzqvcyMoGepH/HVuiraj 3djLKCpBuVMJW7mmLWbcAJ3gQJxy0fZI2t3fSZmPOh7SMr4hH+YX4IADQUDO6xjytFbPC7DaAHtA R4RZ+VQT4PZWidqK96EeGfe84mvk1bg+SGaEizcLkofwoY3A0bbegMzVRN7+9MjnfN93u7Jz7KuD LO+6DbKkIhWc+qXeWwo/q3Gf55D4L2871ApOKGsJ6qhjaRTBeE5IM68Sae6EtEGlM3Z14F8D99eR N0HNgmHsZbdi2RXRw2XsbvW12+/EBVNxwTxRkQKJSIuSpapYxZQr0zEAMk+6X34yq4vkU/jZ3EPu stHVHy8bZfgtZd5/Y2i9NtB7E1hWdxxeL1a+jq93PusKaqNF3cCZrun2/DAcy1RIQAePR21AsqWg vSi3BG+EBZzPkSagquF6do6ixlUjtKpn5uomZhcx1exiuog0XsS3i5wX8WErJob3XjMJ1iwR2ayt yVaMGU5ayfxaODJQ1lxuG217kKaNuF6LmoLQWLnuON/X2qnZTrMH7fUcB8eQzmFmUE+D8ih7puZR c0o/J0OZgG1ULmM8Fdc+nNPnksWAbm4jy6aCqP3sXtxSd6B5mB8iSWYJZ/vg1kZ/5na9eB0vg0g5 PWSph2RxnEpFxlKxtnAIf7UXhtrSJh+d2i4ytrTaMXc8CrK/yHh1ELXaGu4efEPDqYFyksYDozAF DaPi2ppArQP1dTKiQZ9a6pkfWaJfGdZfCI9MaBMkto3b+/b0oG/tiGZcXnk68WTf3RqDuJMfvFCr x0/vJ8hy1yk+D40Xrzdn3jJZxLMneuAInc4OB5XklGyQOu9w4BEn6r0xFECpS1hVk47H2I51Tb1/ 2t2BJbGTgoEzSpsqutlLfQFFFZ9fs3RWJnN0C7/46W6Fy7gXfWzHtKfV9BCiL9GHppsayIsgqU6t G9rE+TJbTZn/uDt8xuiPZ1eajVfqoEzTkCxG2pq2H+qdJvZIkpQOZOjjrNCcWABbrQ9a11xwLUbk olfk7mkSB74JD4PcE8MvBTWp+JvVvyj9T0obczeBYW7Ic8CZ6myaji1TFfHmU/jeziOhcPxcma57 /oZ3NCH3STYulZqBtcBvHJ1LaDQYh9pdm3HJ22tDolM4V3C3k5gPfXGqXus33jxVKE/WcaG/q5BD s12Dm20gUIg5UR63lbkXJzI3MYSMsTUzFWbtn/rOlUsasFvobTj1/1yAu/MnZHuRjF5Ie6O9bvWv tmlUW7t/ohYVXn4quUUxltx8NbHIlR71ezusTH5cWA8hVpSTFRkdDh+mpT0ZBjmVldx7CpzU/9XW hZ4eOHN0OBILtVcyl64XgNcjSKr1b82Oy0QbtQCWuLW076X9C+oBSJo0bam5tUNwkEzrpC1s2mPr xrp+jfyU1v1xQzsthu/OFNNQgfHKPW8l8AoHWVYxN0Yupr7Rgy0FU0or9jDGgpzhzbrZK8UClx7H UdJbSzYO9lHgdJqhW+a1Mh/1DIuU3VyHttVOvtU/dZyfdMd+b2onNe7V7aGCgM264WP1goIY9Sj3 VH6v6aY+alz7oeC0u/Y0yd9P/uDiNTlzsncLh+Ca3As3aniZOizz8f49jA2ar9ix4dc2F+70yqcM oKh5xSvxYM/H4UZrAekGQCXj8YE95cqwEWJ7wFyaIiEtXNNEI6m45+8bBKIpCn9GMBS2j4qHwJe1 CA2no6baxtDlErCtFTfPN0B7NOowXx+CT8M/ZdZsdO7NucfRglMb323l77byd/MrOaGdief1hmRn nehSmKdrR+wCWcDhpWvsa1JG/dV/Q5kxk0YyD5SxyoOa/ONF+IGxepXEGa/MKWuoV1mLrQ/gTukk 9W/N09HCAxT/+xZlFvZ8c3NTBDvz7itQ5SxdheXvh9c1QOnb9tFvyx6naaOWTFoM+tI8n5RkI34u HT8B7is0PhZWIINr8n6wmIp5hyJ/dNCufEx6YOBAbFwb37tEOb1eFNau/QZnWZWx39Dili9b3+5b EANeupGpVV5UTxMon71h0/3Ksqffh8OucTZHJLD8qebsurBpX7WAmpaMG6G2aEWqdvu632viywIo ZslvkfoP8gSw1crkc+KGUdyN3W6caFqpAxGyNw44CBTNTxkoFKLhx5lCLbW97+98CapbIwYHogvI ynsVayAKQQCdC011H7veUM4I9DIJv21bX9uOwr0Y7sXLOPucOwmuIw/pHcQdvn1keVOLC1yocQrn DGXWwxYkYenEhFrnK6IKzgdAuZ03hZNZ274CaVk5hms5Qdru+u0CPck1ERG8bIE0i5agYygTzdkT Hjgkly5BTjVmCKzJOMt5v7lUQ5OzoeFtAuLrsPFD7rXh1pb8s24UGKkfKZ7M/qo+Wswf2x51/jCz yrL9nbH6vUZ+3OjGE/tftu1lCErH+4rNa683KMHBlyrYeKahFZUtScCz5ASiOTpzCfQ7hY9Jjapb 6diCV4LGMw6AKmuA3Ir943j9GA4quBXvGBMfFxK/CP+rSTLDDZv6siVzHRJ9Hy3ZHfQhU9jj4sP1 vXLirb7q5l4fMv7o+56peep7O8egIh15bIQN3KCpsLJAckn8StkNgM0hJeCHRcljbEoxftBgTxEJ kuQavf+spY8LC6GfFyi00r+j1EHaXSjceOv5lt3k+xCFtCrFW41aX7R6HuWvra98F1q8YRU3IZv4 0W1ad2x6mIwZ71RwtskFB7ngB9Y6/slsG459p5Wu6XQBbplr0Z/Yud3frf51bLzkHoWGMZsWZtjs iZ4Du8EFELv1XB+6MM5M8JJ7o3qVbMj462fkYV3kBZKQr1+OWbiOVUuZ8BPeJ0s+OYX7GyKctwVS dtjwuVia5BOnR5q93Wn+s4ttjd+UJNYczrRkZpeZXSpuCtVjCoECoXJunW38xWkCwqRcXjlPGGE4 +cPi5u8zNmEHlkU2sYkimyI+k6eGmHiSSjGA5M/YFfFpKgZ4ntz2O8o0HJ2IQrrS726++s8AP1LL JQplbmRzdHJlYW0NZW5kb2JqDTIxIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1NiAw IFIgDS9SZXNvdXJjZXMgMjMgMCBSIA0vQ29udGVudHMgMjQgMCBSIA0vQW5ub3RzIFsgMjIgMCBS IF0gDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSAN L1JvdGF0ZSAwIA0+PiANZW5kb2JqDTIyIDAgb2JqDTw8IA0vRGVzdCBbIDIxIDAgUiAvRml0QiBd IA0vVHlwZSAvQW5ub3QgDS9TdWJ0eXBlIC9MaW5rIA0vUmVjdCBbIDM4OSA3MDQgMzk1IDcxMiBd IA0vQyBbIDAgMCAwIF0gDS9Cb3JkZXIgWyAwIDAgMSBdIA0vSCAvSSANPj4gDWVuZG9iag0yMyAw IG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9GMiA4MCAwIFIgL1RU MiA2NSAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA5MSAwIFIgPj4gDS9Db2xvclNwYWNlIDw8 IC9DczUgNjggMCBSID4+IA0+PiANZW5kb2JqDTI0IDAgb2JqDTw8IC9MZW5ndGggNzU5MCAvRmls dGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZRX227bSBIF9lFf0Y8kYDNkX3jZeRhMks0g C8wiSAzkIckDJdE2J7Kkoehk8vd76tKiLh47hmGR3azursupqtMvr2Yvrq6sKczV9azJmtLk+OOX kJdZY/NgXJ0Fm0Pibvbi1S6YxY6FcrNbzF78/qEwN7tZnuW5N1eL2SW9Qfb7LNlmpk6v/pxdYgPr C1NWWShqfHvN4gXL81tJ8p+ST6+/mLSwyVV6WfjMJbdplfmk35mh+yu99FmZ3MvM0EHCYnyX2jyz SRyu+fNoeFjga43fe1m6G81c3iCdZyExi1U79NdpA5m+W8q3uFSm5bhhTC/hEVVHVw8Xpt2ZVk40 UYG/RYFurWemIek38q6SGBVlVukBZtRNzUfM1xD7DQ7A5/89ZjNZFY8Ys/TL1X9nubks2MnqXO+m YDjx7quU9qfzsf86LXJ4eEebwHzouZQP3WBGjG7le0vvhxOdoeAUUEi2e2e2tHjbGREtcpEzv6cO n+fbHbRv4FZS8sUbBVoRsqoJFhAidRkGjJh/EVwmPMLPwUcpMipMRgUxivFi89wACY4NYucsDOsb xNyQrNIAmCxlIDLGhtz88ZIVKzNfM6L5xdVNltd5aarcZZZgf6CiIxWPE8U1NnMW8mUDl3heABUx ofi2XpTNDIWZo7la7ZEeoMyr1MNL7whncOKOQh8gdp+WhNc1PxadWWwO52Ua4ISlVbIjeAcGKJl0 aYEy5xvCReF9GX2Ysw8/8RaIGy2hCG/iyy3P97sLo1NDeumyep8Y31NEv2YDVJZw6fD8waMTOV3d nh4Uz7tGLGyUErOD6kb7wpw1D8TKGoGU54+Us+VXtvU/V7PC9GYWXJHlVGiaErlg4BVrCKxA2tDN rmcvr04i18AjngJXNiSLwBWxsP1DEn16z7D/YuTZLTbfNDFSS6VCrf+RWhuznSy7pmCbJedIy5JG EoMmhq6lwFcJYxjI3A6bRSff5bcHamLK3miC8dwd9sdPUej5S0pGPqVb0fcfKtxe85zoeJbCcoh8 vVElujXn83LYbLPoZHYX3FsyzB0FuDZwD3KG/bvvCvtA1J78qoGo/jkQBdAqkQicS8VZLGwVsyl3 U7d4u4YJFexDasjrN/7tUgKJzAA6ZGavAi1/WrETHFxQUdpxae5FEA6gKrFddXeo0VzwYamLu7Uj VfQiMDRp5oIaAny82uwAVSpB9Gs21+R7T7qdHyUyouUNHRMoy3n/qXnJ4STSHqmgB6hdt2LVqHuO qLteIMVLZJsIqUm/nuXWN/sE4gggZIH6CGpgSTF+NLjo5+55wfUw5iyy53XyDnh01LfJyTuyDF6r YRoZpCVwKw9zVAr7lYnl8JZn2r07d92NrJDNpWj5uNAwyYjl9ZuUXao/Dq5cyizizIl7WOAaPadf 6H4yNGOr46+HW4EvmPXhBJRFnajjXno8V8NKl5rhgZqvGplx6KMfRhUz7YLMoHhQYWgmV6169aPo mJ1F3nERdCVKwxOR9w3i+HR5nQJvm8yeR57oSqyvhYT+TaoUpUGp91zw6ZemPNxkySyqfGgaa66E xKFY5FZGHXExYmt/y7iV0R0V5AZJzbJgK3Mp3FhwcRBR6Z8Wu59Mbu5l5dLMO7OZy6DnX1WjW6oi qr5ZtDJeqGZCONKCYvuSQlxKkaA2ya3Bc6bz4bLzWhYeb9vq7P5ls/4hL3f0aNRX9zK3O4uyRcNF ensK0KNBJtb+vCgDdFV5FOVPyVwVuVZLwS/nnb6rSUM3OZsDrBFYTE4fuB1WVBMKjiaTAJWTT+Il 7aV7kamXVon8qms1ar/wyKwOnC8byd5g+aOgTYMoq2Wn06AdG3EUhmiHhqw7A8u8m+o+I1U/RJDy CaeLRJlRVToNNK4ZJZHZInsi0MQUy2cFOmd+dRxoOHrbDQrn1aGeMRH6w0n1RIaaDiybj52Gseu+ Kr5lLG5Upxyn5tGkhn0dYc8BksUxx7b3h/kUdTquJSchXfaKqsV4UI2Y3bEe1k8lIhYgwPkwMAit eiMeIspdiOXHei9NqxHfyB7Lo0oQtTwucYp8oiHkh1Mc5Lhg0CUlUJQfBQLM+CnavAdCaPxDvNk1 +7qeN1LXxa8l+TUxHDgNLVT+ljYcxsupilGHRFT4gyxkvssNm+YkK8nnvBS1lpyKyqEVtRT4hP2u N/ttZXy09RHJjX4LjSMGVFQ0fNxvuCZOLPen/FYxdTpjQvWeCdmJ4+o9EWsbqi3kMe71cORWnjrc 7fq5vPWrfuw70A1g8atMEXGoo+T3VHxAU0aoimV6FW+AnBhwVABYPdOLOL7buxU7/Nt8TgrZ8nNq fks5Lc37TncUWT1z7MzrP/hDlURJTWS6BGLPdtQz7lkDpkEU0nYpO+hDz9aREh/KRvETINWAawP0 uNmILV3coRvOymSoSuG7jrLksTCHxj6T74YyPMB3bbO/ySjp+Zz0xOwbdn7NgN5sR7mwNMzXw4Gv GMdE34Hm1RJ+n/pOKfyY8hhOrJlEkRzKymoVP6BceXYc3R98Ek9EhrZmuxn25+oNppTbStAN5zLo /pKhXI6KRJaMfBrfd4jW4saDg95JidVdurSQG09JV0y1bdxQA9yu2gXI9mmASi+0tCaPPxqgyj+P lobgT2kpKlVP1X4lzkGlL5IL0317wNgYrGOH39zqx2Pv9h9epZdUpj9wsX5LvGTcLDYroxLtksDJ 3pCZFo6KxyJV1vEsRxUf7upUi22r67rlL0hHK4noj/tkQZ01LfQiGS3Q86IBMdx0EZpOHvrzS2II jkhkg0b+REhK90wSGbw9JZF8VXDTVcFJ1vRpgDeRvcBZDb3l/e1BooC/pbkwg4r6SsFr1lQ/Kuml jvpK4I6sTUXaOC6IjGJchXT3d0aERogryaQ9rTuoY8z8pK85BPledlphBSUgHvQfvzJrKOX8D1Se 6hM7CCEkD5DggSEZpGS2UZXZnmjJqLsT81ls1jJLkwdWq07EmXCj1TuEY2Z5aDu163DOJYIvmFOi Ej7RE0MonkkqaaX1VBJPu2LYd8UgkZeSD5/M5ckdwWs7lPYRkCP9WvDdj7v4iXuKtsGQbI+WD2al 8yCsvb7KFufkwOXULgrrHveCz3+OUTW5OoGgwcyaQV9PoK+VELxH6SySXwl/NvmSUrs0b6BkRVjH oGVMUoO4LFl/vgbdyIPYNjq++cALBImOM0NvXc1h0XBCwxj0vNmt7LJZcmuwdLQ+9WzRSvd7UAOj s6LBK3lErGcR7bTzvSzQ5T1Piswupco6JR0uOsVe4FDRYw2MrNZvkt85c4WGuYJsK0Ka4Ta6wTAA 9rB0KK9loOHVawRFhe7VnzvZg1raslMHHasiXt23Og+4OAu4wBKH65UjSBzg6gwuHmTe1kiMCS+E kqIWAilQuYMJFtBDE4vn8DI6h2ssTqsfh6+tafenGc8evrl9iNe6aroPVKIcfDMQh6Aq2jPF5PsL ACde2eviM3RpK57O99YByVd/ckJGOVYBKudURYznm8yjthXFAZv7mQLlay9sLpaiqTxFqv4xFhTu o8S+75narvl3oDqD4FPbxQVv9U1oLwujQcg0caGpnQTA5FZ5MDOKmhlFXDUIB2F40TfdotdnKz5m gUHICxF4wnUnKo3ZMbCBY+8QLnE3hS5MoVNDjfI7y7cvIuhTybBCQLkkdGbOt7KCA2yTRYx40AmO eK1XshCXx+GaJUeWvJA6t+xkv0VKN7xBdplPlNGFJvOIra/5QlaU+RMA983/Ka+a5riNHHr3r+jb crYk1fCbTA5bib2p8iG1rrV9Wu+BM6RGrIyGWpKjWPkZ+cULvIfmx0h27HJZw+4G0GigG++hnBnj tzD6JI8vGSNuw9S4ba1xq3agUHK6K6Yj8Sy32h0bJFnpiTGtAvksfT61OklF3DdU5F9l/LxJWkz1 rxG3mMQtDWZlcHmqjVCTenaPePpctfccc9+6rUbejygA802V+eqKq27HpnfeGsUPXpca9Gl0dd89 PEPKJI9AFaUf+ItcFNF3UsUkCy+pIi7q9DQNM3/ptZbHIL+lolNs/a22S6eD85+D0QgNq1LkUE+Y LTDx9002mXlSxpRYvyVvzfYwjT1/uPZokyaiXERmbUQRkzfBYWhp+3R4Hk+51ELBwkyv+tcDmm+/ k4IlSaJl09BEJiyOpVU4vMw4eKNe5kI+OMRNlpKAJuNOCHejDZ9Ebc9fPaFcEo1CtA1Oo9bIWOOf 2vFlsaLokZNzvFPGW/QQ71njApDleEUSJ3PdyqayhW4hqDvRJ2REckohVHLaDCiYKuZ+DW0jKStx 9qX4MDZ3dp7B54tK8y7hRff6fBdJV1GsdnkZaI6d8NOp6Pr6L3WJvae8yx5EpAyutS7oi9bFxsMN YYCT6zY0ndtQRRFChIcYEqYY7ZYQh+CdpkxQjtIjzKHMrOBbLkUZl8WcmEs2D7S6wHKGQmMXK8uO QiFmX7/qEuzoW4jKfNXjaEGctulcMyzKA6tCwXoRBw9T3Dj9REiSDgTc70lpXmZvm1Q4J6kpgctS WcrAm5Sa45tUE+jMjG2mdx2Sd9iloTHdzJg6xofJ3skWUDaKwA8PLGXydbtRvP6C2WGkmvli0yhz gOUQCAxUcHauEwzUZt2dGhN+bFbnnpkBjuVN/jY57h1lMM34hZYX9m6a1G/LM9yYe+7tKuAXB7bw DHZFRSRMs8zfTGGZU7dlKK50owCvl5/bW+1rSpAXrfARfEtYtkvfqkSgLCU7NO21/kUAVq4TBleO Ju8o2p39C03QD2CfBf3LrQObS6Ip0Epj1jp77/cbNDSm0dER24qT/SK2EK1ONOEGs7w3eVf5SGJ5 Z1s8kaSNdm5sRJFRMyEOrsN0Opi5rncXvncm6EzTO+oezqsguR1N+bhq8j78HSkLp1rCfP27OUpp lDimemFF9W7DNjdnowh3sdzpn0WkE5Cba1Ae4cbX2mXiZp3FglDZUS1VI5UrCgrHqvajCtjE0RDR /0pRKFFJbTzICaRGjH213OFeDitN0LhyXCTyAOOFv+bgAO2/sbEcNwq77qhvSwiDaB10Vb2atlXf JydURPicFvNENF8DY99pskOkmbq3UvP9X9u2X0beYMk/EzQ0YKNaw6ToxxPQ/E6+5buYvvnH/Pzy tJxeXzK/vsR3VPJ0hGe4nzZiOJdtCruWGd+alIrmMz9s+l6vVBg8HOVUod2xCF0VmdUdpUGvdyYk /ip2/YCRvxK615nCDw+djhoLmp7tNeLyzmF+o2k4mJVm5d5Imy1+4D731FCJ1GDr9dm7NS6X3b4z Owdbp3E1RMF2KW6yVxuQ2HWcavNhfF76iH/JhH8W/DuUWY97yfTbcPo0kP/yeWHHPVf8+GTrZodV PA9OziZO6GCdn3/P+/EadSb0Q5Rzua93tq3p1mtnekoBBMQRuS8p6gX2NwwANZUNzYJf9MfyXrec 985fmcnBxrbT7I088AQQiTp4jav+hI1OZtHCcmeqU3ggZNv9sbZnm7XrAJ+WtLj0q0c73dEOtFuH xh3XRoaLg1k4hkvoCfViz3dFWKx/qHE8PVRy6/8Ee6lWJVJqHU0kO0i5nuxppVeHCxOpKeJU5g4z jeNKgyqVB58xgg3ovZczK/NUROT325kBQ1GZ641zv2DRfBBcxf8Oxa7QQp8xsN6TA/7qHSkmTxpu XpkrCpVaUfR9VrvukVJXi1glgGlK/7lhb4fqs1MtQS/bpqlxYnPOCY2X0Z7bSpexdb/+/DI1ics5 5taMET3lAdVqBRdbobKxaewRKnIt2YTGN9KN9fCCdH0znMEEMi60sPRea2WEIhch2IUEu3toetuJ UrYDLd1sgJzurffqpV278xGatTOp1pPJkuzHKrXK3GFu8p4iauYzvih04a931XT1Qlx9wRFImDcr QhH5OhgWjDNhUwmTNKEPJJDHTUjWJ6YL4LeMBFeV/OmLwjTHysBVxTTHje8DJpvtH15jbDvP36H6 yT46PPHSRteSltIM9VVNjRa6CykwEnkA7hbiZrg32E8B+6nCvq669htOR/+rhZNGpvXvpw0Ykm48 Lh6GIqP364xlGjev5f79j19cbOW68otW3fpMRwvb0zJjSTjzhpApG6HcWQuAVyZjEttQuc9AxoVp EtQQl4FgjndwnePuZ3gFCb7errizlx1orDIzgOhSqsC1p7q5ZCwMUXyExVgtCvqRW3in+JwL/5y1 y1HrUtB06zOldrSnQQsTP0k3ThxQ1/Fo+i70Vajrk89wj3bMc04dlta4goeyCE4t5cXVzb6rm/rK 54D0IZvoQ8YcAFdSoFxicJ0FpAwlkNgQivwWq6ZiMjYygNYQbD0ylsESCRXYKWIGex7Im2+p4jV3 a+EjpzubldvAj0Fe38rOI1tcEVmfQsugslT659W5RkqUTjL395U3MoXE72xe1y+exc5wOiws0t/B nDFBi1rr342tDuuATT7b6SdnHtbOmBP7VajOs4sv8oNtOTVl9h6lSWq1ZZH9Dxt1Sx65/jxwruPP iUsjHrnSggwAUHNVA6cN13kpPLbHZ/L3oAitzQ6DBWS9/3s9X4QiKK/8rVx09IhRQG3+rYyKci/z w7WDVK190z5yP3XLL1UXkuYDZwc7bbNnn6cSWpszf/7e+Q+lvok3gx36lkYOL9ODKF40Y9o0FKB0 IMF91Z5Gfn/aaGt5Axv//PAqjcMbLZ9Rmd8I6xe4iBxatkhO+Or21c8fXsklyNxW/uEjFPK0TbaZ i3Lhu9FW0nv/KsT6wIwnxVyMi6mJS+QY/9U6BuYciSePcwVVEDsCjObyKpczs0X0bSnLpNT0ekA5 djaumMeVOSlSzYkTrm5hmUp7Cp+5NmiUM1uiJniJhyyR7LVLNNs0VHOBA9vkYHvd9uaN+d/S85Nf 3z3Zxwfci1JuXymnffcDD/8pCAVEpdPNeDPgCANTeRNgUPBDDunTyKxIGnNBnMTF8pMVTnIgyUAe tz5DU8KLTNLnE55/S8LT8qZ4nu052fbQa0EGgE+O6JDM5ciePM0PKBw5IDUO3jk5Y9A9UIRKWpio OeAJLvUrfhObY0rVVNvbWjNpseVaqi3nqGXC/AF9Dw6AydxG2MLUd82BmvPmpl85jKjymSoQAuyj B5ksPkKSM+5d39F4t+9s6s0mRuMRGZTn9u0+YuE0Oz8+S39aaD7jQmrH17OfFZLL73ntifQBz7Mf TXXep/+T8kmtom8+ftr8qDe62oQxyo62nu7XRjgi2GjtQkGyEL2iltgzpo+bGN2BsIx9J69Io7ej QHuChNyuZdelL1FVRjazNC4J6RvcNJSSkNdKjYyNe/PrT3OZKZH3UAsDpKlfcauWNgetwqkJ67Pc smLHqNgfjCS/Bj+10/8In+T4kZwc8rjpXpa7NM90L/OZyLSkUx7eX7zmNNf0fE9CY6nE2Qvle1tM 0B0Zle4BaiiXHhkFTcex6X0cFcTSALgcBRTWgGq8W8JqbNOA2RQlOLM+IyXMReBl+ttQVflq6bEx 8kJE1NS2bKHfeAODyWojkhEbBL4jdEpKA0fX2IzZ9tYoMMi9YXNUgheEms5SfHeDPH1YNj1vhoca 13EwJ3rXPGK5ETQX2p9J0xIWC+ZKZTvtZ1oyf65hiNHxe5mxozOn3ny8enZh4lQLf1ykWty/emMS IYHZd92YSIxmzwEgnREg5YVpgFxRYD8KrBmRVwOdGngaZKfsR0At9XxPmyiy+p2hKYRMf6LK/ymv mubGbRg60976K3i0Z3YTUaJkaXZ6aKbTzh463UNu3Ysi07EaxfbKcrvpry8eAOrLXm+SQyySIAgS wMPDDo+VSr1NF1pNYq4mOVWTIKQ+xWAvOxgRmtHhJVv1IFP+nQEJsGrrFz3Vq6aKeh1uyoaNakG7 jHiKb8BFn4LlAFDnswHqqbRwSP9PwwKgi0JbluVsNUHPllYVhotylRHjwaMct3X08/EsCmIHNEjy CD9XoyBx8OlbooBCMXeXcMP1uKHt379S+yT/V5zwjkNa2XKmaeo0wjNOCKoR8uN9L7XXj6ZW2gQI ooLQzRXUrL2VDFlRhhxHm/3xGOTL6YGdqlVlQ3WwhXiXhb/y6tSEkyiXAYcjJQmFI8qzGhGwI0hN L9uZY/2f50bTZuHJtiIvt6nEKhMOPp65m4ylVE4y8vp1b1NmvZHkR/E3SP6c9gUWpoxFudgajDdm 9tIoxRu40xOzQ267HKNhz6dE1xpE0LflhBzuROLz8p1yzI08mfn4yegnoW0EBjUQOQbbiWVzqhkY 5YRy8jKYSMmcba9m+fXNmQsiC94dr6hyfMcH1o5492t8YHN3iXefw+5fYI45dVl/8q8gzTZgLECr nbRXic4q0qqktFeWGw2ZEYwasa54Bf4muNo+BZh62Ac8n8MyoZ90NsDAOwV4MU8sUOQLkL/RwqFo HsC+lmU1r1RsNvtgn2qstroi23ssH2o0Kw/2Gx3PkNzawQBR8Dh3uiWaDbZNzVaaXXW6LYqBbr+m 17Kr5BLdTore6VHRg6zUNCeeXDEHLoS8YrDjAagVk2W8PrcV4DBxT0WOzOlkvvUlQw0DIKirLH7g RRNOcv3CCC4hTfVxdva6Hkljef0PL4RjRExUhrlHnmMTx7vN5Iql7GFSh02ilulWTMJ812/e/Pk0 vjTSnOp25WVS/kNOIWx28jiaLN4MmOMWw8NRjK/UK8RZYr7RPH5WMdj9Kgd0XA2fPH4ju7eZvczu wekHno8AepZWpEJLIrXOUpLteFQjGxAclHgxN1KLd7hovhi1YAk91BbNUL6oK90uymZCexE68VpD iV8MoCJWlEsA9xMLeP4e4jOlrapho/HCxqly2dp6ozaLwkPj1SK9ptyrC9ek7kRmZtvasurqqmw+ SAdnfpFDjJhxEDN8VZfNpQu0eoPWyDv4ck0xOfd+FjFVL5Lrvid0eRtPt2l0iacn8YAd2tgB4Qv2 RcK+QAo0JZFdBuqDb+v9Gq/QUAJYOKPesTxSARi85VHoYlPuYlPQngxZf9SGBroDpOScGaDDYY/q EGFvdIgHk6lR1cmkF+Tu7MzwzZLJ0f4kJq417YtFsH29V+vP89AVTJfT9Dv8yabF69hyEakr4vwG VOmZdCHt5NnN1T+KQyDJVRG9AB9DZiVcgKyboAgZlWEAo/ijN4oKRsS1ZZEs7/+em47Cw1KkVJkf A0faA4fSDUkK1GMiq0X4OSE3uBWyTDAipt6SbynDSgpil3CPa1FyH2QUROB2C+bIxFBoc8Hog/+q XvTU5P54UZWaxekUcHLAF/hHt5XFcgATHHHYd0ZWSray2Yg9MyXBrhdV0iLE+BZGFfvZjr3Y2DZr U3aDkP9yGp3W7VvKgo+PPCUKwz7P/gWHjVMCB+oB0ozy+Vd+fWuDH6wygJKeE4GuPzW9hnztuN0o BbAbE9aXAL5jV8py5cPCSboTbpgwNlX5IF+NN+1+LNSxDmlIYj3nkf/fgFXRXtxsZITuJnhu/UG+ fdkRQKoVagJhBrNwSnYH/3YVvywaM+73LhwYrPd6zVJ3vPDqkRn7B44lsYZ/ZLoOW1szebFG9vbR Egz2Kv+eWa1u2fBAppr6Eb0uqqHjAgU/RvBgTGmqDnTZQNwz7Z08KGwCakRmvQh3gE1cdJifEoCl kve3v8WGlG1+Ik6zWgHiobiHl8WPyOnb+3uVAjRkiUohj4s+j7VzQzZQfFbm1jwhybIBnqWQg+1U dVudZHUjNM8vLflzx+sURJKscViVsB/xpEceArAJxNFSdYOIZAaqp0VemFpMMuWBGNlXXroxjiCK P6PIPPEHYww91GcknmPKgFQdPCfGkd8Q05xT1Em5xLFH4rzoc8r1OZWFyggXZhwQDuRDgyVDsLS6 eFoKx8V/XX3ftfVBPk3YgMCkJqp8WdqCGaxuD1Y+sQLvw75uKxNGJ2puXw8TbQYUyAINMdk0n5fk MyujSPX8/iAft0fy7TR0qPnLbZGcxc4PZ7GTuniIHUKjELpxokXAgf8mODSCL6JIx4glSmI4Yxa1 qzw5j9rvnTwczOJIBmwIWqk+2/gVqdArfT8lw6kyoiNCCFy3wtuR8ZQTKCIoP44LzOQyFMwRcfc3 XsYNXVzAcHNnbpeodSZ/MD+bWMhMGpk/7rjs3YSS//8AKLjaJwplbmRzdHJlYW0NZW5kb2JqDTI1 IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1NiAwIFIgDS9SZXNvdXJjZXMgMjcgMCBS IA0vQ29udGVudHMgMjggMCBSIA0vQW5ub3RzIFsgMjYgMCBSIF0gDS9NZWRpYUJveCBbIDAgMCA2 MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANL1JvdGF0ZSAwIA0+PiANZW5kb2Jq DTI2IDAgb2JqDTw8IA0vRGVzdCBbIDI1IDAgUiAvRml0QiBdIA0vVHlwZSAvQW5ub3QgDS9TdWJ0 eXBlIC9MaW5rIA0vUmVjdCBbIDI0MiA1OTMgMjQ3IDYwMSBdIA0vQyBbIDAgMCAwIF0gDS9Cb3Jk ZXIgWyAwIDAgMSBdIA0vSCAvSSANPj4gDWVuZG9iag0yNyAwIG9iag08PCANL1Byb2NTZXQgWyAv UERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNjUgMCBSIC9UVDQgNjcgMCBSID4+IA0vRXh0R1N0 YXRlIDw8IC9HUzEgOTEgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M1IDY4IDAgUiA+PiANPj4g DWVuZG9iag0yOCAwIG9iag08PCAvTGVuZ3RoIDY3MTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4g DXN0cmVhbQ0KSImUV9uO28gRfddX9FvIYESTzXvyYOzKRjDBxjB2tLsPYz9QFDXmWkNqScrO/Ea+ OHWqqklpbEyyECCS3dV1PXXpH7erV9utNZHZHlZlUGYmpB+/pGEWlDZMTVwEqQ2J4nH1ajOmph6Z KDRjvXr1j7vIPIyrMAjDxGzr1RpvRPt15Z0CU/rb31eRI14TJ5tEJsuDNCqI6A2fs3wQb2GGg/de NbTTp0e/DCKv8cMg9aa2Nn3nF0Hima2/juIg9jZ+FuTee3+d08OMfqa0f5yZrhHyujHycqZjKVEK 252/jgM6MeBc4JPWmWdm1p+EA31FtP7kW/A1tWow+TFxqFp/XQSlp1q13cGPQlrvhfNAhxMSZqPA etUktK2zodr1ouVkdEtFmubf/FLVkzn2dTW1veqtJ3sVY9q7jb9OSdk7P6H/W//j9p+rt1vydmtW kc2CkDydhxRKa+IyCZLEkJVJYYZmdVjNAUzjSCgtLQS5NbA5tUL14/YZJoQvhSkrS/AiSCzR5Qjm HEsJI8WGdC28N7+MwayeapaVZI6hD1u8rFdYYPPPqFbYILxWjVGWzihLRT1z/zNBqPA+mtvJwP+p V/kAAmECHt41jCGKvDw4soTFZm8mR84qMSIsRZVU44jHnkQtI2Lz5l/0br0ffIJW4ZnFEcQhI0cU SUDaZ6TJpSO+sc+WFOzsyr5vPU5ikRQneTB4MsYOgDKLFlZONOkRviw6tjHB50o053u5JHwp8gmc lqyG9eTBgRMKCNhQ0lnvvTn1IztO/lsfLtod+UFZFiPHzsB36cni3pyG/ousCPVeWDeUN/j8JHuN 2TeXPIdm74wV5Wc/5zawl8bOeKR6ZIOXcyQrc5ivWMz/HyzmVDmyb9MkvwzaYWCAVN+kSB5DHCEs yl6MTxTngQ2vxEFKVBSQcu89UqmKUTQ6f20TCscDBNJzyUrmMIu0FlXjRVfkJST977QsQ/VESpwT B504m6Hjaj4lY06RfM2ISbyPqMlU198OfoSEGvrBELi8fTM19YRCWgao3FK47yjqSMOEKWXvgf8b PyI2g9kI+w3/i91UpqMkJfWDPC1dM4oXUIcK6vs3nN4fWf6WU74g4JWMO/e9IfQB4bVu1J/5OZK8 FL3HkvsfAVZStyU9M90yA5pDThbrQTITwYm8r77NyZzGt1xnED/haSo92gzwCnmE+ZFb+CncexHZ yYkHfuhaI8cDQ3WPaa8VGs9MfDr1gzImDfZmRwlKNU6WnPVHlcgGdKqeLFZ06knI6G1gj4dwdpRm mfN2YpcSYsXbNae0ZQ/yC9WEPatMxfODR3FEmm+kFq8BjtKLrTnIG1eeguL9FoqWnmPmWyCj0y/Z ++DDBcqZgxA5+koo4Av+Pij7lpdr5ibvNCSUdF45y7HXsmbMb1SXet36i89tXBk7OUL55VJLIxoJ IYH5Qd1AlVDWJNBOGbVYpTg/DEgfGkYepT5ScoqkNcdh+1dpieXcEiMdvND6Uk/+Zf6JZIjJZWSK qDCvMYg8+ZhaBIJCTs5Ejlnpk5F3Zgq0Rp2zAC7FtrLs961wOyg77qsAltIdJGNQIDEUXSvUGLfQ 8ak9WVlL47WO5vQdXWTkSjwni7CEoijKb/g/EPcZ81Yr5RehFUEUIZT/jFI0Soi6nWpVRFn6kbrw exqdp+YiGZKEk4FTgOrq0hUI6FpZRmnccwehDhsXKLGaRGk0J5HVOH7w+HAsDbnwqEz8SmhJvJ98 zCs0i2T08Y6XnK/RP3MkAAxpMThYdxy9tnBrACRIPvhz90gJtyQ6C8m/piClk5dHmTihloamEEaX 80S6FAMd0b4iaQrJylRyLRWok2+r/R7dXjTilV8R2JiMBOZ/kLPv/FjyO3X5lnpKP2oc1K8RWlVh Lxy7+FXUuTEV5hJkMbkiZmd5/RkOkXfnS17p+71phZBiDy1Ochp2uCV+70A+8V7Flwt5b/m/50tJ JEQIEhXsfW+EhKUShH/eLPO/gwl7GVEht7PfX2zosP5PzdlpgRFA7oUcs9xVkygWdwH8j3J5ojxk yJ39nN1nkc7cxvn21iIumSdfNYpVZNltepPjQ5WuHo9C/cQmZ3w9oB+/0PWSrpjQjq48aCmPyCdN Ki/BdfQZFhPKgeI75nxbHAO5IFoGpJYpnQgsTwCoHHwfRdNSEupiBc8BmcwBMdevUroulyGtFL3U qEHuuTQAuwp7XePWUhEnV1X0GyVHKUQGtwudlAoeebSi3WpNfV4Fn5sB33IBKjF9Pmvb8ZKp8WXb jmHwyDPWGTU7x8wTsam5J+umE8Je9gnGaLMoNFRtQmCe+2vCbw2/dSC6Aei5HmS6bECh3EAx0PF+ f5aFujEiT9KnEEX04OeLk3vRw/SsgqmBrdxbrJEX0ys5DwrztrnbcELe+aiMt6SA2VfQpRJBO2dY ZFX2oF7YqVpPM9t05nrZozEz6wVWB+X2bnN3a44VM0AWIK+AEkLR8FpXjXlPhsNX49ju5K09tlPb jH9j2cgoOUpX1Fppz7zQycd0g6F5jRl0Q9D/Da+Y/DJO3UYFfRHxl1OrljXoY/bCS/nfYAAF8qdl EADaRC51VbQagN2oyo7N6YrNYtLR18ug7syWULRx4zxd2iWsr7xrsznJE/EuTyAZZmREtDHS2SVN Mk3nzM2kIJF8dfmUO0JzOMr3EwpgDtNxrTGVniKYVKbVj2moVIzwGqF2IdMP4WGg66zRj0HgKumf ylBBbN/dbogVJqh+cW3JYBLGSsizGaHwieXcLFGjYGKiZJXVB07Gmb+ma76VD6vEG3qM2jA7ojLq vFpds38Z0BqkSmI9jkZjOENrEtA1Dk1KSQ5Uyo73KeUJWO2IasNTrywbYAxxX9sLZOhJuNPlULCM ZXmSzxfCi5ns/s1H5EHKt0pCaTUAINyd4Vjctrick0v2zdDik25e81RwXrZlUOlwiCnIRBTh2/dy DRGqpjZwdi5B1EXGRqIfFesBDqrRXjkvS0KI0TefFTlAE2hXM8kkZ1qm6bEuaMvFqsEoAXRshZ85 gcVw6lVTvkz+Rw4pRXUcmmrPGIq0V1/d/7h9pHOXLUrx8Z4ByxcrXIMOLeylGU+/3fYNVCJFK1nu fHQ3t2n00CQPJDGIet0+H2V9Jt9dyzTn8Zk0J2ZEAYvnT64OJV8zQ4EYVlXaZ157LvuZeYYn9VAA yRvdkauzRWew9KAxWwhmFI2tUnQPS/6m0gTA0zFwTxoRoE6nAkiQTfRE5xh8btSSp/Hv6jTd+aRc OzVEP53/rxRY3M6CxSuz1/U5uWogAJinxiLX/oYEQzxjjh+naKhccvVfTty4y0tTYofAHSyZ5cai KOFQTaNKbNxhFjFecOcFQgH3rvMim9ed6A5tiN+uVdrp8yiVQYQ9SlkIpDQa8wsuAokAKFoUvmVO 7zlLCF6qs5plLlxhdK1XJbvfmcW50/UrdYXiMjaZ7F36BxMf94QNymkyK4EhK5fcF3ewXc51pnNm s0QjrlC9/7gKTrv4t7mOOXfde4+zKPWWxkIuc4mQqURWPCHYHPRZyyE9O8mi7onKiNQ0LzAPtTUW W2O1NZfwYFr6gW9TRpm7Y51+N/IYMBtShhoVvlMyRw6zE6eSkTzNuA/J/oNT6xMTKldgAGydTqOu 19/1juALM1Clbw+OzzOHPcykTqPDfzmvtuW2kSP67q+YRzAlOcSVYOUh5VUupUrVRiVpax9iP4DE UMIaJrgg6F3nM/LF6dOnBwAp2cluuSxwZnq6e6Z7us+hodPF6V7cIVddcB85G527HmwHzbY3DD+F 9S8u3NE57BoxwNIwgMKThPBEurK3RrYmBpKOhbiJj9uTUsKWiwPgzrazizMdB+qgIPoeAGI1uOfm 6VnnAE+ClBdkNXC6CgenkKKYyLxAG47DnlYgyQ0qXRrdS6tbRTf6dw4f0NYIH+I8HBW/Aoj4DtUA uFfvH5gF9CuTqZsF3uCd5eXQU6RdJOwIShwrTQlUBo2757fWcBVS7oFvgIgzmeXIn+2ouYRyljGt E3aoTLMhYSAy5uCSqDFTL8TFNPqzKXMOtCY4/qyqvaOJoKyxH+aqjXbqlKt+0wGWpCbZ7ACCjE2q d1tzQBC6KpU7HZDvpXkmGdAMMxSSZROdjSc6GzNI1Z55J9mxyFSHDr3SO0xUek2S+Bhwyh36jr/8 ll9wPakqh4o/bHimEQeIs+jE0RPT3NaUa+rVmnFzyqttW5Rqjdbb9R9tU4AJJry13ZUVL1p4mmuq r4SociKcpJt7zENuWhoUQqNHP9qWzU+cD4e283U0xBjxqHZ+8/jZnrRKt7y3lyjxdZYWOFfl2mZv lKzrXTNOj0VTO9xEgkBdAgdSBlsGgiYI1ja9IxlC6UgjFwih6/3WG9P5fEZ7dGDcJ1gy4uP4FbJ4 hQSdU8gGcCXoNsvkcPeoLWEQlmjSK+vqGyODjboYbAYyGZRuOyGO24GOH4Z5AU6TMeOXiV2p6lyT mQLO7/lIoZXRzLQQJokSCyCSimuen5FMcnjS7VsFr8h2vniukVDENuulVVKRG52o1NvYLeXfcftG AElaxJkMAnFYjsQhNeLgHv6pEZV6/I8HSyTbLuEu4zV3f+XB4/H+GlJ7PMhKyRfm+s/2Ynok2sH3 4VkdLKsr9/0CafWO2f1o3eSKvOy8iqh3f32czpespcsUscsSiXnhUilphUPFKyV6b3bTSfI0fgv3 s3QlXyfZkDgYyhMKfvf4Zg0NENcfsWhcZtJmM6mtIvX4iQ+pDNeXWPzRRpVoaSj4V7IHBRy3bzMo q3Y7+5AYCvlRxAsyplw5R4nyHzZz6DU+wYRJXgEi5NoMMavNUFKY+bA9t76tBjNTmSr1b3Lb2Win 9pQOLOXHLwv12Bbdq3581ePxqHVgk6PJmWW7qtn8UUKeRXxJTHo7lVnn+isl73p6ofhVhACxLFT8 HAOEnAZXBBVix6YVwaE7PhD13uh1rMPw1nDjnTqURn/5Qdko5j4hiQt1FaWdYBO/nkw1WOkFOsWG IKa4z/q2emJeX0wPg/3oTdv5GfeLhNeOLQ1QSh5GJhH2u13H+aDo5v4G8GY2ZQID1dg+s6ehw+r+ bXia40OLf9MrS0vhQXhk49MORcZ+rScQ+CPitVZ4IjVhrx9Nrkz79EoDm6H/1pzzfaMRTGz1iVsq oFEAp3t+ACEBYAdknOp2Yd/DQrMAUvh561piyTWxpeGsMuqvJvKYZLwvzMusDGM+qCxq2tY9BfYQ y/pn2jPh6Sz6uTxMZegwHNYNNnRHf+CvytyRd4+Q0vMjApqCzpQBtq4lyqGmaihiiYTWu7SUypp8 u5YuE4mZBXn1/wR5JT/KF1FepmNNXTHKzcPNw224Sbx+adaVlhFpK/0RABhzNuPsi5CAW1VXlwJD F36YMsd1YTGXEg1XBjFzMINU2x898mUl4bvn50biIrcKXGb0MQYQxF+a4QwVutYs7Pl54uyl67O6 sGZGaGdmIUnHfebT8e2L6K1yl+ZSpb7dBtMyRyT+9wNdLy10eYnW+tUu+K/7D+7Bt6jmn60HPRr0 wbOKwcyAIcXM0Fcs7JqPhRwVZaY5Hi9aJs8mxDHO8gtymK0mHLIKhWGRRR/c3+Vvo+9YcQdRCNqy DtDY8BKebc21Ktrtn2y99pyp6O0XTh/lwaToCrki1KBqBONIKClKMZitwJgi+n6RKYARsRM3h70q Hawp1pH7WUd3jmL0lj6AF+B7fhxcoE702ruq/fHTQlEsxajm2Ni5zNCUVJA80QuKUpnbePPgxC2H Q9dziS7VbvPldZKRFhMkLuz5LkB3H2R7LqdbaefE79srFNSVPoI1qUctuHBQ8Q4NRjwBUtbjzARt 8Cmgd8j/29vGZxXxblNRuLbmLc8yM54klWCgCgq7UaHaOlSE21lEjQqnY8tWnelRT1eRWQByTSZv rsn4rgNzTcxmZ45wE9P5+vV8zqZ8zsKDWoAkfXB/s1Ki7d4jLws0lxXBHEatrvGvFHcNNbIjtl22 SZMh0+vRTdpEFbnnQFsE3infhoQdwH7Q/d22o/L/LAzeS486UQy/ST+w4m7vXFXXvcc0/tMBPi+0 1uDDaO1GT3nnDh1Xepp0JniisxInIL8NR1KaF+WMbJTB585qpPlGp+28yHDz8vIuY1Zg3EO4yinT s2zClgW5EwO06/UICrAKIgC01X1twEAuOC4B6ZA6629Vs6QI1XSZTShHfFUylGoLBDNtjKhKYDbV ppXQnT/uJNbnurR+ErPgpC8YrgCcSgCIcoRc87PU9MBfFJ1rhKTpOCFP4OcTl6jF1k3lEf2rNAc7 CrY1LtvyoFDEzguCZ7XALo4P9M1vm114SUDT2xmEksOvbV/b0u41ZT/OfEblWPHBpwqKXq1Tr3DX wDWRhJkkIa5c616m7UcJKObGWMShdwjUptQM7sEFfZFr9o04cFgtJqUWhdLSbs20yzTtEuTKSj+h hEhYt6ZrT13nWmwDfZASuqm4y6u62nVhF6cRaVF14qQWsyTaenuP8kak8VGSGhuq358dlZZNVdA/ PXgWwckJ/rYrGA1pbB7/YP1iFRI/tS5e4y0p5EpQZwdBY2iqRLyxvi+dAOGCpj8+PmZOorl7I6W+ FEAzdqXV+FQFVfVPUscff4J4QnF57KsyneSX6xHRWHK8XzAfyHS6kW8hZwfFj7hQnVWwtkJ50nLb WzXN9GFATpqrKTjp2NRIBN5HDVU1Q1MNwYrY/j125VkHO+5Iy50RS9pt64lOWmeIozPGQmipr01r 6LW+sI+6ex80JmA8SmUSe21gikWRzt+cVMjwOvUExN64i2bPKRy5HW8KMDEhTExIgUhlJ8sAl2fT b5EmcoBwgnd0yQ5Zj1bONlV6vt9rVj8gZHnk7AYurngRE4PoPUvxnMe96b0Lbqncjnfke2+qh69m yusJEd6T5vx6QmJrJjGgh6aLwb+lGNtwhC6KkiUpuCOB6Xp34oZGmc3PFDzplNeEiE1jGFEF/75f oK4I8HUbbhlUi9wph60OtTd0Bx/0oFpV5gA3YK7XLgGtfsvdPX2iyBedcjXnKnOGBroncxhFUMW4 59lmt4RliZ2Ei33XmrMUq7i495x+id3AQKyv5FNfyWfYrRTsdtcBVCtGMdwhIKbVj1cCwexBFBTi pGGoaD/XSipO1RVlKxtSUU+CkBDKlYRyxficd8xOAMHEkJEBKSmf2oRyVvE8qq8pO3kEWZukD0GS 7s1bdP62tJMN9OOZot2JXyp1zxVN2sm4g8Awj4zUBG8DqUlM7IhkX5/dYmcOSa7tdnNHe0o5M2BS 5i6NmIP+jBYFW3Vt8fkWYMtHwJa/BtgSxUM5ARt8sS9RGQFXriQEaaLALVXgUXFlWEDHtI7SdGg9 f/hRiVyKyKI3msLaD1XTSve5tYlLgw/o68A7iNODpHYW3RLCAbOU+pQLvad0Zol2r5yvtjyXI3Is IgVuAVjiRv2Zwdrx29XebuB92NL1r6O0ZIQEy5RX639VilAZYdDbynEdIXzPut7hOlL1P1X/sygg 3GzcZMOT7qAQuuZex7Wbb+766kmnUTH2gxSI1qMvP5s3S0VqkHdfdw03R0he2m9zszem8kW1/QkV U11yiq4LO5O2Srqqs1TUf/4v41Ww2zYMQ+/5Ch0dIAks27Hj4xIMww7toSiww3ZxHbc16jaB4w7Y 3++Rj3KcLii2wxpLlEiR0nuPnDSvtGhpwQ8GHoLgmNZgQhbJKHjiIHii2+87bV8XvBOqqAUp26FW Z8y1oLbXsxPzpc9w+4PWuuQBcYNwC38EkeGlFJyW5OYWu/UsBTJWDRCjYcEXer2l/l2SteivrpBc gEUPiGCIklZtjfSLVq06MY+hgVH2Hf2WkygsssMjPTk875vtfKny/lfkp1m70kR80+vycKTexQ0B jQoUJOvY3cy1ylvJSc7NlKlid7lqYTq9wQuq+m4it6ml+6D+DwdqcZPTJsaHiemvOTJ/pV+BxAh7 9M3CcXfrYV6FOVQmeIXZcyPgrjkSBrfxprZ2Zqr/2dP0ev1S0lIypSV/biOQKsmuhxCP3amepUCJ FPk/6/J81PE/I1WQqFuji77ez7INkAuC3RewRSOA4udORMgGVZ89zuKw7zr1KymeL2CSODSUiVOk Smi4vZ/hFLma6w+f5FgA7wCnlU/c/auGI7/4avyGMeGFaifW4FkWBpqJVqhQTgmqSa/Tkw46zh04 tuefRsQTMugGGxeNLn/f1ViFYiloZfYV7YfqgzZBIQNTTTLnPv2HEopw+9TEMl7GzCPgw2fZNNdI YS4fkkL9oaaSQbyBdSIo8zqLMmnDLlNdgms2YhfjhDntrlMBea2I6r65hNzmbUAnsQMGRHfzAvxW yLP/H/i3zlKGubchVWHg3u5FJyvM23hHpCVyf8Tr+vmCrN6I9kYkit7Qv1w6NLbvBXSTSouzOFFQ yvQ1mdTluZVYFmfuGtchfm5M2PtIFOwYUDc8M/ZrWXZm33RMecGU64FSKLIn/StnyEZITQVS9ySp VHNRRrRvtBgwvdMXu9P/V7yyzlElZRNbmb7jCn6IZhJVMiWxlOQiNDkW1+STl+LKFMNCbSMJTAKy 4NrBNmQ6ChsfWpGeS5E/iR7twbb9w2UjkauTyta+8PC6PRMD9jpW9GB+Dr95OuiLrrvMDfS8Pdt/ 9Y8fK2At3FbSAtCrq3frJdZ2mUQivulYfeilG+jYxepRtJk9tsiCeDvr9WStBCkQZeadfjjudKDU Htxj+0R/fRO86CFM3HoRJ/y0EOxNlpKr2FpE+eS0vE+LQQqcsdq5fYR65mHxkWEwgkEDrMw71a6P FnKbSnswEk6PB7WUi89VXbDXCiH6HT2Ojie0PiY9Yc71LPLKfBmgthSkB76cUGkgfRjs2r2MuloX PGOiqV/mxYTuTJyuwXWDzp/Q+vWV2oMIolPdt0eZVyRa65hr+p5rTquAvH8FGABqlMGYCmVuZHN0 cmVhbQ1lbmRvYmoNMjkgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDU2IDAgUiANL1Jl c291cmNlcyAzMCAwIFIgDS9Db250ZW50cyAzMSAwIFIgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzky IF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANL1JvdGF0ZSAwIA0+PiANZW5kb2JqDTMwIDAg b2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUMiA2NSAwIFIgL1RU NCA2NyAwIFIgL1RUNiA3NiAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA5MSAwIFIgPj4gDS9D b2xvclNwYWNlIDw8IC9DczUgNjggMCBSID4+IA0+PiANZW5kb2JqDTMxIDAgb2JqDTw8IC9MZW5n dGggNjI5NSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZxXzZLbuBG+6ylwC5Ay aRIk+HNK7dpJalIblyujrA+2D5SImeGaI2lJanb9HMkDp/9AUhrbSVxTIxJEo9Hd+Lr7w4/bzcvt 1qpUbe82dVwXKoE/enFJGldVUqisip1NQOJx8/LV6NR+JKFEjfvNy7/epup+3CRxkuSV2u43Eb6m 8PrbRp9ilSZm+8smytM4y+tcFWXs0gp0vaYltsQl+JZkuOK9fv8PU+qP6hX8Hk2UxZk+mCou9WgK ePeDidIMhk/0sTFJXOmpM1EVF/LN0zfV8TIF8jZO9XmtRIHm1MW5vjNpAg+1Ra0a93xrohK/NLy8 VbKElPI7/3YmA7njIWIVfQMqwEf9eWXEoPZH1sO/fj91Yeu1V7ESCx5YTPHkmf1/NDXI73jgBzX4 X9mbbvDBPl54PPetrJXYTTK/b3pvPm7/tklUlMapK4pwArlbDs3xEbSdcbHVA5hLL72JCvDms7EW ohwi+psBPyrNshOLPGAwnFa8TkY+rDjwGNxK4fFoIF4W/cIR+HWE07DwRY1nFjydjgNr8gZ3bhW5 AHjNGa8YSFdWgERwhlzIM8ITge+nfyLyFnTD6VTOrYSzAoX1GBNCAVbWVjXGp8zLbwD09UelwFL9 HDRfOG1B4Qyn7pZlcdmtycGrmxliDKAvgewCyjw10VQzedXxhzAtmObU6Iaec4Nhqfzv9DyFzdYo 7J5AF0vLhtOcbo/G4inJTtfpIzuLUNDNuBUgBkUAIkpPCNVoIkcor2gDFPbtV0BqF5BaPobdcYLw 1QwyAM3xwC8RwqqGiJuoho3ka0vowok7EdjzvCzvSNcJsQjGfdDkkQWM2xLC4dFc3s7zMnA106ee PjUyo37m4P0EkS/gcAsIyY0Bo1ONcS05QvOyWRMYMI/EXNbYAOozeWcDj2ygSH0wrB3Kh5zAD5iT TpaoZjkY2oN1HILHYhFNnel46jB8YTA1LzLdURLC4Y+mBBO8/ySBn9aWyeyZUxfPcvtHOsElL98j nmtNuMRyCZhJUQsNHwwG8nAmKFV6J2UhQTTuGXZYR2r6x/pEqzrSiOCE8kTlsebiOB1Vs0edHYES OgRGmjZci+0aXN+Kejj0ArCIuQcZuzJMsZhqPVf72fg9jVCkGZo9ZE3Htkwdz4wXkchmLIeSsvO4 C28MdQNtTaiM1BjwB3qA81ZStdYkzq9kFMxzJ8ApyLmhERX3NEf4pTecHY597weWjskyyNAihUaA pc/VmHTch8EF6sTwxNjnDHyExImtaaZu1/sVTrCgAczYQDZcQabK281bbhNPNMxZx+LA5XTB02zh MwOpFOShNqfFUpvfoZICE4pa0cFgufVQWSI0DiAhLw3WAafle5huVN/BATa9ugMrUi3SwyPWLQsO L2V2PU22V8RLcLsiaJu149cBPQIAjqMfkZ9QzyD30kCnbBYnpc24SaFnzq4T58w9c+BVf97Cwk5t MgeNOE9VAQ0ce3KuwHUgY4Pf3G1+3F7RuqxM4wJpXQF+IGV75PZGzfC9npqB8qAC7FAcJyU1OZgY u9zalYUznKUyt0cqJlztQE9jcNDhsWDDgXxQ9IsVmlkA8qWLMEBzybJS9lg6+oiNGrxmF9DhNM6V BYbwTY9zBxWqeOaxrebWLparu84Dg6Im6KgJVnoONWvhKGdQGv7LnlV1HWWKUbE6zvNAWZphLXTU 88NWtFgOFLKu+uZWLsliV8A5rNwLgaTg6VjiNsc3d1C3bAjgtTPzapelcYLAgi4IkwnJpcDg7WzK ioxZWgaPGhzNwVNXA9wt2qMhf4WMFYGMWaFiEGGeET3JPMOgsozLN430ZGpRyDshs5BGRwUgVXjh F+8xszklAGU+/Xwpw+ly9aio5L0BxBaET6Y+gt+Se2cG+9EjNkhzlHr3wGPPCw4XoycZDep0xOaI faqkZuz0rocf/0LhO/IFC7SwEu5gkRiK4gbl8H+hgCU1bZoVmkysWwdTlFiBHCKnLBMP7lUz7EUK 98VF1EjLoG8wCV6RkI1rdfsDxeLvEY4sMI3XNPak9gkVNKgBX5DRldKRSNFXyJxwASHXlAiGSwPd VPwwNR21Tf4wUe/GXjyuGk2Kjabv1Y5JJIqs7ibgL1bnhr+2lzJB3e+k/sSTfbfvphdoekGW00em HCW1vZL+w1ox7eXgx6MoIJPPYrJ8ZDGs9TR84gc0GNngG83N1nN1SpfmdjMpokal5vtmSsQeYr7j SyOSIk/Emsk6HI9JhXw74ivYAdVpOD6RdNd6aHlneuf1HZ3vr9QDrOaZoDvsiWS91FQmc1l2z2as rigOKHZKF9NargS8EMEDVhCrK/h6ABYquhdhAuDv7c31Vc9WcVol5Xdf9dIQzUKiCbe+CBl6MFn9 pdthxlK0iDJAFDWwn4jKzYOxHADsCyR2IIFetTw88oM+jgpQQcTK1lCai+xZCsy3SpstKYDsLqe0 jCg2B24L9zSFvKaEkTzeQJKOkI54CcWsJ+I4+D8YqpFkACI351TgSxmy9U6UtvzooITlet8IYjkJ S6LjZAIrBtZz+cHLvikvSvihbv2e/Ri6IP/ZpNxFQ+banIPl5FKIafCJN12y21Ju7698OPEqSNWG AI10Fs33amKzQuA4jOo075Ix0nC0EyW88jEw82fNeWK5B4hzaNNojqWrktX3D9cIBXvKbI25lPgF YPPNJTjBxwQ42kygsnq5ENRCRAZPdyAkwHAH6hiRB2TztZZTT2Nm2JieIsqsH6B54O+D3Hqh6LUe 72lUuMkfYHFaVgXttOZiwPst98gzjYNgLnUipzpBG75YnXMZLi2QUb0Rjk3CdJ+0oqFlEVh6ZsV7 cYJ9a8TpL9l1fQBAVorE/o8lIsoKJrLXqVnMqZnzYRyIBzPkBIs551O2ajjo7hkxk+tx4ra0SLUX Q7hfCMUWZd184SPGLdJwzu/mUrTeDJOIEqSZJj+bVYZmBrcvuuU4ul6sGmI+O/HJyBWJ1AazDR4O dH4e3pmKcvotFjYHhRlMSXVEX61ol/0ow0otC59IeTd2Mj6s1StWK1PDszO0cVHm31Pmv8Lob1p/ mDpqNk4LsfcmAfgMTLejLGfYIA7yfMZBXi78UGjKB/2zwcOiENMvsqsKmRkFYzi2RKFKvedp1TZr OXVq7nne87jKHnis3t9S0N6+MrVGqgVrLJ/OC3WKFZGvJPvIM+qDiRdqteYMbvbfLZzhHdMB9LrQ 1Mlh4FvkAUBfTMH1GPlB/8STK6A5iNQDf50welUYkbYBK+/FElYv+qCr8JjqUILHL2L/ujp3aBJ1 6spVUSyXY4cr6T3sflVEXVlli3wy0yUrDf7fqhvBwd4jHmuBPpPKmhpHoSVNoPl3t6+QDlRAvfFk bpDoypwfmRXWKIW5ACTl9kbaOzSROr8uIc+pG/Mh5D38xr9MtY5CtZhHQdM7BCKHaYzF9zJUFdyP sv8rUEhAloRKZpBkApI/UV2GU1KvwFPEiuomNZ7pdYdxQYsxeP5XHvGUl3W8ZjJ4wL10ps+sp2l5 gTwGvLRVENJxKUwFtQ+U7idZ3J1YvPfXzkOkagT87H6eXrs/XvtfF9lqQVYt3baSItFgW6l0+2y3 uHbQxb+Tc+Y2BLqUajRibaU8GLgSqV/o8DGg6Dc0jkZY6X9Ir7bltpEj+r5fMY9AymKIC3GpSmXL sZIqVdku10qufVj7AQIhCRWaZEBQtj5gfyDJB6fP6R4ABLXeRHkQBWBmenqmu885fWsTj9tavzwY WiWLZZmdpdxUuaDWEuqp+WlEdiRnp/kvxDOtB4cfMU/YOs3KcoY8STLeanICPSnhDmLL4ZYTDz2A NLYh+MY2RIfdkf/0heIAwvZgVZuaxsn9hJb6IFGEiXihRaAmxjUTZZlSw9KqqNqz4hJBlIH4R0oZ gqjZElw33WNbN+6y2bSPTffkPuy6WcmVGhq1MSaMs1m21XJWx9wIJHMdAnYuP3wKvXPPlLInuXog PmayQF67ZUmpJhY+H1iOgDlnuXhkOUtSyLIyuCYvvCF66/MVoKnnYPNN0VzfFiE7C3d1EA0irKYG dAz5jNLaGvxffXCNPa/3O525nRr7Uec79zpkYTg1CsJERPGsv7p2R1QfbPyvxh91UqfDR51bca+N +r9ubI6E/FMwVysiKspl+jLW+o2+VKJOzSfxY7fZI1lTXiRuQ/tLais2oQMqRFmRnxFROiRvptbr CuQi5m4bhJPP+it0w8L8kRgs3OiuXyuOvLsIKchjiUVG4lfox6wvIbzAd19jUtrNP440WXFqz9Lm pteXH3C4k1YvYQcCcqG+ULWRBs+UpiBJIff1Yolwro5AT0jwtddB+vsqvIDj7l+QW1nwaxiT3qbq x1X7PfpQKhyVAqK/s2h11k4Mm8a2ab1pmy35vyeUSUJrzcqOTVUzoVfBg36wgdZm9m2lkkWo1OmU yhY8msS4iDR59NVVtzq93bR+uyc17Q1pc4I9UMaH2j6bvc6b8vv0Wu0bm3Zv/uFjXW3GTjGjMFDL vZzQNreDDv7Z/Ot3H59pBdI4+j8kodbWxajDJN5dQw5LzuOUDHEytV/VteiUlveXCxyI+nG4lASn DapJ56d9n8zp2709nQmXct6qlBPGvhFEA+5coAx4sDnwlyDhYXE09DnR2OeAagXut0RHaXlaYc1z yEpncigeAAgKWrXqipRLllt7ltsLyx1mbpVKJaMWy0YyyTya/RqCcv8tHQ5BtwzeAbmsy0m0y0nR 5UibEwFoMv/NZkTS9mDfv95IlyBoIUCZlMKQmYuXghQihUQOxKL0f7j74S834lQJl5aOD6U4kwr4 JXmBSTdf1NF0dDQ1tfKTFHYefHYfQyDST4hFEryFC2CcIpCO4KonMzu8snGjGinF4w0H9BfQut9h Dv5afrvVocapBbCW9mJTI1/COCFhA27c9Rs6dM2MuwoBwCAjXaFW60YXYh9gLIfWz21+ASX0FMa0 PMdVaXuyfPVSlZsNCVT6Zkck3EWMZDxYg9CiKkA0WjL3IDNx+aO8A/F/wnUUwVtfUn7Zk8gNLpTp MS7gG4A5EsFB7nPuJkQHhmDwewOrB/DlSp/d1q/8YmMK1dpCzRDgmaSYBnoe4nt93N3aiJTaUadu fYx463chz2Qh73BEH+lP4SRqh+M0tJonFVNhSB67nplFt69O4y2KmJYcv/tVQ8LGBGbdVU2Y/Uky 3QoE0fuvYUz5ZYO27tnkBCPSkCVn1U/ezFm/vp7O1ITf6nSWGsO0ZIuR5gPoDYkWlb5mP7tLWM7l RiOASoVEkHsACEbD0fWt4lsJwKag5MzDgllkbUHOa37epPVjZvn9aI2zzNxAZwKfnNhSNl6+v5ZY H6B9Etum2mzQ46WTtUd9PYRsmdb6Rprmg2gUe9w8mqe9d/lPJOfYbO+mhvoTx6vh9PT+zwiLsPle V9jCDuAnmrm1dzuzvd2N0fm+0Bk4XsUArRbBZhQe9l8lQfPKv9vEkWAjdHNHHfQcuwLHXn3QtISO 47f1iQWh7wYB/hkX7yWVaS/Rl/pumx68OtqeqB3b3MsqjQF3t+kmkw7jqUZdIHf4nNffcfbVIIxs f8QQAf7OGobj5g8vwTAI9SlWPU4x4gQVrHCNAOcIdHe08dqk8m8V/gldSmH8LlX5AuZqs2VO7JvO +3GCqicIVzdSeHs7ZafbNSPGJIssSqkF81U56MZRxvi26ZfLEOXy2cmGwc2AqDgLukPtTONJ7GP0 bI2o4pyppLSg8MqJvfZBaqLqaeP0I2LOviQnkeZUBrBnFupTuwjncbKD9HLqmQ7rDmqgq22l7X5i 5143rxAJOPOk1ivbdUM7u+396eexnVNAwzZuvdv+M5QuMbHdxeUODcnuaC49iyPnPasGeslAy81G KpUjTauCQZYi/9uuc8036zt9c8rJ+41gy0lz2bCTHaeVTHT2fGOL2wOPImhh2x2sJ9WXBNdXcw0F DiiKl7cqv1G3ozz4oxZTd9YkLVKx9ALt9myDhF0zhO5T0N+upXJwNUVw5AXpUB8SmyXb2B/xm/62 HLndNCay8kW6FAX+u+G120UNKwXxzi12rqlqjdlZNHqLpUWRgfM1Y6+MI5602i8m5T7qiqQYb6GY NAORlPtliI6sqZD2q0max8j6Vmml59gDb0pL2L81rt5RWySKCFmwhziNAzXX8reu+snbbouKstNi 4R0qaAXD6cTugIulkTkY/iv6q8jEmWgQ9nFXEBfM69wQqQxqW3Dk8k53Ns+fcO+iNqqufvAD+G1q m2CL1GzjfinLC3UxTlddwcHPOvgdiJ2k+wCw6Lc+uw8VsR96sjf9QjkUeh5QRxPeV450ifWz68cP 9gzaExf33e6bN7ol2oHWl0xZmuL6exqpFBx1aTt1oLZ9wSV6fMFT6cgYoFNT6uza3fpde6/86M+E 1DR2GrlEg5TZcvVDd/OO0Io3OsojHGW3bjactQjRGV1N78QOw9W6hTvZ/FkczuIhRkmsQZIkXulG 4OhXIdgW2RYh2tdMuism2Byl5HB5MgGeZTxoRau4t1RPhUSB/1qpXMnTSt827uO27d3745db6Ach mI7yyDd0I8KVi1UaxxOEG8FQMM3Q0JwSp0tJxBE6IwKBoOb7OVhHyyIdjcbFAGHxiJqptHOAsJVR c0TNXRhYjsIgIuXiePs9SNrmVGdTpib0t+GyTrlsRS4rRDy4+/bRxnWNuuOhc9oC4m2n6l4nKQlL uzNU6QlYZ8NJjZXkEvXm6aZIsg4g13VSHhtJTGTiK79xNmz8OpQ1Es9aPUy5eWy/V7i1PLgUuqnc ccuFrQjTYqEInp+lUyTplyXlGJE8/y7plXk2CTNQSA9VWB85XH2FGimA0hHSeKGagR0ig/JwPLyy bzZV7h45GgePOBpyczv3VkgqjvKXUXQUJUMVil5Qb7+i4OSOhAAjRcKIGIhn8atE7M9kQp5H0cv0 CTfPcn9pZW5u4LYY8KeQMUU7K9cIrFayFdqIimieU+l4oshONAjdMtBfIGA6AKiYXkt7ENxB7kvt NxRFUJWrs4OKUMmK/Kyo/WWfV3a8XE1mq/QPDgvMAwGX+DTjrnw8QD7oBTQH15LPGSFbbmG/33Xo VMR7/f5G5ujTVQhN5xL6LlUI/bZ0fBDvF1kq0iheSswy0OMXLcBJ0nRruCfJlnEhH6I8XSyxMCrL RexXJaOvS/P1QqNFJZIEVUdWyZT3JZ93orbJvCnpj01XbG2E9h6JV1PpoojKZKamBmQfiH1smhJy EfTL30NFqzggaAhjtUB20NOKWu4C4HTkXK4IKREb1zWbqm90lut3gMKSzIqZgh3ETbSChXJySdfB K4Un95iz256MXpLRqWdiEvodzymHlnbRBH/+jOCPMz38y4R3NN6S8d9BOpifcWreC0Fvq31csw5x 0zitPdgHm9eTDAFBAslC87ZMgEBleLGIl5JY8yocpX8UeXaXvC1Ye7FqjZjKAX3H2u2oc2T3PXs+ TYWcOSL0dG+avKKwkKdKifrEppP25eFk2CavVfpvwnSce9uYCxTbojSbkyXiRtdzfjtZdZgTRcwC kX/l0qpjJXem5TGGS/LS4pX5eMUWLQGYmWiIz/HiF6nrC8rr+qgPXavKpX86EyjToh28SmJ5nRT7 atBHK48wn93ro6AHhTov9YFvAobTj8IFcg//YbtadhMGYuC9X5FjqGhFkhLRa6X+QMutJxRSEimE lgSk/n09nlmSAKd9+LH2rtceW09RO6BaxiQ/I7wtW44KMh6zd+4x9mBfQ+Dh8lyQYM4BKGg1l876 Ey9j+c1+yQ+F8AlxC31ddgZqn5DTae6maW6Zijps9SUAERjNqK6WUzxPPE0ZFBwBZFaxBAYl1Nt0 qNGWc9Cy+BVVpJfyzM4dX9NdEJy9DvlTYMG6CQReHh+KQ+MTQ9ypw4SQ5xJ/HURr7Qz4KubRrWRE Cnk33BOFki3uOInPznDgggzcKsdvR1QgOZ7uwSjKyTvKNCz1o23C/jGRUHF0hd4JpiwTSUy7qVtm kNvveBDeRu3YVPknJWVBBXI0ktuzFAWNtnkjlHIPudgNlLKjP9H6cfRzv+Ci8WRwakXkhhtcOAox lbafY24hYjjHd/zJcUO56thiVMEIEC+od5Xr9zH3fCCyMrS37vqbh1cG+4KdPvbIzwZlNB7J7oBx KWyZX8aaTO3OwWaO1o38ko66SpINNzQ42kocIj8xuq6s6DqpDowSDPzSH9ijVoTdlRvSIiP6qZLQ WOAZ0BPuKlHkdDQ99Dy1WURLECgOon3Pxt1tJ9aCw8mRk5TLsl5n/xFVhWXlS5nd3qNdObEPTrRh sj/Jpv1cjzN1v3DIb21BK1N+zROAnJP0380pL+lQetU/4nvxKdQubkuPfSYQFb3Ui1wWbxm27+uH fwEGAKEzMP8KZW5kc3RyZWFtDWVuZG9iag0zMiAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJl bnQgNTggMCBSIA0vUmVzb3VyY2VzIDMzIDAgUiANL0NvbnRlbnRzIDM0IDAgUiANL01lZGlhQm94 IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0vUm90YXRlIDAgDT4+ IA1lbmRvYmoNMzMgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAv VFQyIDY1IDAgUiAvVFQ0IDY3IDAgUiAvVFQ2IDc2IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1Mx IDkxIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA2OCAwIFIgPj4gDT4+IA1lbmRvYmoNMzQg MCBvYmoNPDwgL0xlbmd0aCA1MzI4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJ jFdLb9vKFd77V8ySLCyGQ5EUuSoaG7hwex9BJLSLxAuaHFu8kUWFpBS4P6O/uOec7wwpJXZaGBbn cd7veb+5erfZJMaazeNVGZW5ielPFllso6KIc7MsoiyJCeL56t3NkJl6EKDYDPXVu1/W1jwNV3EU x2lhNvXVgpeWlt+ugkNkrA03f14tUhst0zI1+SrKbEG0bgUlXjEKr5IlY3wKPn28N7fVWBm3D/No FdR9uLAZLV5CW0RFcAgXWbQMxrbDAlBRuEhoYxSVUJbT3c8o2FWUKpgZtvKtFHm3Mw+AZHJREhjl +LoEFYB21x7rKMej+RbaJNjOlGbBTEvbOMqD59CWUeYJ70JLQnnQ53B1tgPi6BqAXqva/1N0hb+Q 2JlW5agMgSWR9YJUE7dz8D683/z9KjZE0WZ57l2Y2tnrFj489G4gsdn0bZiSyR9YjDJwjXl4CZOE JcLFOsyDG/65u56vepKaIjB4DC1FIOnApz3FAxnLBrUzo+B2IGo6bLe4VmgDAXBVgZ4SaKq+wW00 a7RKV6wQ1Eh8KIYU+cG9uZFvJ1YpyUiJBAHLVjGDMmjl9wFnO9mwHOQZNSwFQUYkADeGixWdicCk 7t0Hc+hJCbsKuroDtohfUAwJiEc7o4t7VysE2VloKmmxJEEJXcaaCAPPiGlLmLAgawFKJXJGHUgM 9jjCvRFDFoEeNubx6K+FrRfGwbDLKLdpJtbNSh8udkp4NfLtvfkXoXOoxsJjH5Lf2rrldCADS5hS KHHsBr0cYhkuOEarUa6f6J8syfLZYHC1+UzarAA5kPB5MC7Ii0QZWW8DbBVIThTucwgZOGcCs17/ +m5DACkBr0H+17CklCQO/vxG+ZqazWrBlUnwv6tRKWTzOYTUhkTkcCKTrch5oxSCJfmNqwKnHcv5 arotyynd4hImHEXubcgx7iSxchY1J1dw5Ca0ychrdz4cch8OeYCk45STiMg5vCTVmFbtrtnjkjQ4 aUzFcUFm6Q6gBG4d1yEPBIjI+Nj/G8d+wuZMJOpLRPEZGzOeUzFPctx1jYGQXgYRVTAl2lPlB/CL TOOUEQgQUF3dHqIdBaF35EGvzlz/uB4DoFa2W3HE5i9Tm6qn1sXWJxIZ8z90WOC3DZfBw85RLI1b 8gHHqWdClo6JBR+S30tdkjjAdGFMgjNzyht/2Y4vsjCN04uT7AG86w7S4gquTVaKIID24KISkgjC EAI5SRepBhXgGrOREpdS2ORkhg9mV5HNY+k//Nsrobq6oG8enFF92h0zaP+tlZ6RTLu+4T6SUhim 5Jk7FIfFq9UhKSfz2qkIrwjr3oiw7H8b9KLmKuC8KYjwQOKSE4/4OtM9wo/mse1hCo6LJGc36JZa pAJ1nJFWOoZgE/1agVipEunJbEghO4UKoGuWxrm9ALYA24Pgk5z9rBImXldMS6iFd+N/BHGQzqUj AkdmEnPT0cbHnx58Ri5WUoOlG456a6rdjgtbToc8grCBqjkEIa+VPJDDwkNVXIdXbKCJ1AO+Djfk a+YHAXkSEZgJWL9g8bjTrVL10BVoUG7Ap5BAFWnrCsNZLk2bzwzIqcr+TozidfK0zCU1zxE7laa5 UIhKFSp5FoBGO5zV3jSdau+Pyd/61Ho9/ee6omnJCXGZPEpAQvYsvWULeE1OVp1Cfv+EnPNUsK1a zrtziqO5vawCLFK1Y/dIpQC/tetRSNqay+Ftt+buN3DzsEqoEiGQ35LdWk6uJYHYJWJWLT9ii53z thA6C4YbiX0vVrZKTOWkc6oWVe+q1xvejyVhSm8prdzz1MxIQ85gzVgKaS45Xf9Fk3HBuWR44MjE H7SpkMkDH3FvkAjTAiMASgu8UEPnvftpjfmsZYLbPg/1RokowGmSdw6UUvpPof3HSv/hnXnyGSCv A5qLeu7ntfPi5zJfagUqpC8W6EvinMQbQwro98XvvMWl84RhdcJoWvJVHgw8+nIQsl+57BbCuQBn PqLotDLFPXS6OuLe1N0eB/px5PcaY9MoxMFCJlLrgWgu+XpU+AFgSm4Evy0uKw/fdAZgeiHVoJwJ NkYXXeNACQX63WaTGgqvxyuqyvy0pU9JFkjj3GRZEvHT9plikyNRXrbLqOB3LeHlihclGeFR0JIz caMU4+kGU9tcROTJG2y6Q7frnl7MbTvU3cn1L4qfAJ9mkVxkksUk1ZImz0Sk0lFQx2o795I//nE9 j0VJqu9BGj6rtkEGnE1NiRZijgduQXngYXauOoW5WBODKtud9+1w3lByPC85unYcnlRW37OX+BmT S0W6ZLYF9Au9OYM5g3M8S/hmNLo4+HRdeAhOfUikQnIvo+CoVQFwHHhkoAhtlQ5w/3r26kM3Fr9k qTdhkaoJOXRWwUeE5D123JH1LbAk2jfrOz0fhDW/Q2PMrCsUTAbWj9dyywWA3xAnBzCz7yTZUzT8 nGBpFOVJ7KDnSq/GZ1R6Hmsfoh4qLkTwtJXpcfAHHusRT0mlOdPC9+SJiKTgoHzVBBOGV8erqSb5 IIWHknXvJrFFPgqAYsLuAfyFPSkH3gLDd4KdzrXitsNP1xeOpZWonkrB9mYSgbZeQpX4q+6PF7qx 54jUG6OaBEfuY6OcQkMebLcc4CuODdmy5rL4wPwodVgneYxk3PBEIQlvOiR1IC6KOp5/4ilWJkOy rtAHaMzZCtJeSX2V3REk6Hbgp1MqU8E5QaVwlngrGebldCfPXolRJrkApROk08MzIaFKxBOAMf+E 3t+xU+E6FY5/vWB7pTIq02/hDOnRv0xdsBJR9kr+CZI947NX65jDuaDVJfnHiXLC7J9DmQiH8wRk 8NMPBlJaxOflrSHw7XmkaedXQafNvcfERjzu9AVjqqbpL9o2t2SsrnVCwe7VOaC6GCpXlz3/tQGg 7448LNRVrSMLiEf/1xhUSp7Yadb6Yf7xb67RzXIKrheIEir4etRJBxD6XtOBRGWmB8eDe30EjIup uyUwOdf3Qp4gqTxBeL655ndsjume9xfthjoEMTjrVjKykNuy4BKJnk1oO4mCodXQY0EcWHLkuUuU A76KxZ3IEx1lPWjbzAJQIhvWCnwOVl0KZNo12FGLMThSJOXr0JcG1K3Fq2/MZLJcrJb79JGeONTN iPyNrNYhV8Y7I57Pxa0kIlmBpsuTbNrGSUCmOucvJS4yeZ5QfJ1vMNkm4vwiAJLxWJImGT9yKB6l W+YI4hwxkCkaMqbQyTvzIo0U+siwlHJChjjI53p6QCC74Bw+bGjwbbGkJwaz1Z1RMap6NMpW5awU gC1D5bucTINzGhnfaBFi6WyydDZPYBR6wUZUTUSNLBAFrE62cqDPhwy2WAZ+u5drFRLGJLw/sTuG XnAr+iXyBt1VtQObbtfIBPaLAxknHa8nvwL/JUykZClxAKkRp0pmxc457Ew04LTSI0uxyrQ08LUz XjtNNOzg/3zWTzjP2clTU6VKv0AQKqk/VAJ6hnDgcLhRlu19DkN8GcaepoQtUJO8rrw9OINCnQT+ wsO9oS1FVeNOyodNXDuyqJqoA0sg4hdAD99ZeNx6o/hvo/Y4ejG7WeQDOgMVZShYvy31+nftm799 eL1TpfM7DhFJGT1KSHd1t5OFkZcTJcsQslJUyaiu1K3cEVv0F97U5re79+bEwFXIfutxXj2AkAvh b6YScb8wd0LZgCE/x5QSYKgdYeGuzZq6bhn8Du1++y/nVdPcuHFE7/4VOIIuSSG+waNLdpKtSqVc XiW+yAcQgCjEtEBDkGz9e3e/1zMYiNzd1B4EEYOZnv5+r3+MVvuC871MlWYAP/L31JyiwDIFFck5 qiXx+m6D5sP3OVB2goINlwZ2DxjYH/nypi9XlyHpAgsg4PXS24l2R0Bfp7rt+4h0oNmo52yDweAd KqGOtR2nsSMJJ2wZiaXTHFG6x2VA/55v/WSSJJGW3lp4guAUM6B1bGWN41K8/Z8hh6DM2UR71vEY wPrBnbULrymZhvdGgAAks2ZcJL5WZ959+wls6klRpAWfRjH5Pp75akCDRfZi+CF16LHnm5CcZdot FZFfjp0QkwX4UZPi1Zeg++SOskhT8debxvcbSaC7EW1cYmj9h4hFRfqOY26O4ABtohWYSYW98Ufb EGf6q5UXMu+Fil54RjmwadY0MYEL8nihfuLY5ok7ta8UAYnVe6XL9p30L5cRJmVQ9m7f2XMy4Cqa b2VfQDwqJGQphioIpgKC/dM8zHoeTlx0cbejc+mvnmJE7Jlk4kttIJ0tIK2zYRZ/JZP5mXzIcsW5 3Xp3jVxFIzWGwzCdNJsM+ENCXbrcOs7D6dh/mVdQBrAwjc94EiRa/i7B+wJfukyU9JTd7DMMmjq2 ZA5YVG5HbrdUbefBKWZOmsfTeBwPb5/ocbl3eUmXK4LFKMGcCXZGiPLKny796XwhRN9twCAsOCWC k2lwAudk5pacqVat6ESK+JWM3/oUP1jcEK5yCZeihTitMiGI1xJ6u0iDUmkFz/+XPpSLHp3LKekQ U99Q/shvAo/A973p0HEvmsIFgtOOyJicHS1nnCu7p3HhK2zBEiWzVl6hAiTbLtqubqZqoo+JOXA/ jbbDU7S6VHRv7Da2hQJtgdy4wNhQ2iySGGik9mRo1ypLs/z7OBFpSu9WM1rCdqUsVgPbjqHi03g8 imb3cfOrHfwsjlgbb56eAvjwIMBy0bpSnmzd4H4j/XnqIqtYV5fWuk2eLb6GoGS16grrJRQeBT1m 9lNM6+aMvRKal6BCO9azBOhgV/BCV9lK6MkWdq7bWcNhKxqnxg760ucH89+6zy02Bs7M04U0pg6P KiAuSCjKbpOb5YkB1fJ90rpKmJr6VVOzAsLic09hfbSIUU3xje8TmNsu5lOZmoSfp66Cak0xSKQA DogxIaaNLTahrNGmlz82SMj3mkmNYpsJ6ijIreHl4Il4KM9WQ22cTBRThU4RmGg6SqqbN3DymS9D uMWOLd68szHpFjPAj6syqHwZZFYGyAFWchW0/9xWNHWyFSzlkhRbTUSZOrhHZqMHpHiuFJFrc1ii FVIstxSjyJultwUInidLaiUOEXD2F5J0qRLpnxqsOj6SBiPSaQAAlePqIH9aCCd1FWNMV77fPoYb kKYJAkFf1/ie09UJY1u7M/b2yG92L6WC1tulpklAKgsjAbm7kDs7rfn1DV2P2DplTA2zowlKNrVy SLVwdy5RnZGWIrUK5EqLvRPF7k13O3MwY1GFl2fHbAmWpdO8kQ4jfEH/HQg8CI+Sn0QTCXpVvCPj 4PPK34OcQUFJGmrfsDqV2U0j3ulnO3XwiFbQ7UpQN4UzawezcjWriE/6QX+YMBMR9WpsHEhYWh/u njbbWIA2SdTBOtYKIkGHSWpE/qmFZlm0HEC1g49nzJPrysuP9PRHoN4t/334BKU6Y/tHR5GNVpPo K9Zp+nCWKZj0O5BksnbRvF1GhB3+1mQbYFLFMsJ8hlVfqsmfQO9+icQQ9JlcLRODSHB1NGICtyOT +pqJ1zGh+1NvSda5H9xved+3j0+unPT997BKLHWjZ6s43wxyw8sXbugjy3aroil6MAnTqmJZL4hu Dn69FAar89X2TciPzPebfc8as/dVi+gu9QZNAX0VjrTyWkTdGycuVOuMPH922hHy/P1AtG9BPUbj IW5CsOE2sk1Lhvg55GSTjeMVf/ppt+15iM8bi/X7wfYj54pbzF0fP7DAMY90dt7oiF3Ut4PRI5vB XBmSvIzjr+HYs5SoEaqMsXPjlhtzOPs4/uN7bloyUBlZCFxC+VdR42a35nj8RGGm3uE1Hf6HTiqc 07IV40919FFS343q6wJKTQ0/8R34CNaWgoshDFqkNZ5pYrVam8DOxMjwcF16KTCmjtuhmeXCBVwy OFTxYNa0ojvT2F04cNFEWnAL+FmfCK58iP7RmzWwcFLX4CDyaDHZm4Ho70D/pdl9WLHGpYvQeSi4 CgVXouCSGFjgKi1nZSVeP93HZ8SjQrlSlKe6ISUfyLhg252CWvl2NjiqQGIcL0ORLrnyE/pjD62M h+pMp/b87e4uj8SGh28SsbXMdtFWM4SmsSbVwPhf//n35u5/uj3ldpG1rXPudlnU+swCjWYeVChH naP4bJCqiswlm2jhRjT8sAWtWI1FPwlu7WDpNcqEx4IysPrCqaE14Y/YdWZielMXRR6YSNR3Jq4t FIiX3cHmrHQmahnRxEYvLrT+KtAVIk9+kyVZ9p5inBPWpuumnhSST7LR/bGPZG5qogP45gDO+EpS izLKjNC+N3B3IzHJlqD4Gk/cZFNg5H0d2l4IxnF47ae3hRQtttfozYvphVe9Nhi/0a4kGkQfX/YG w1LLKZr9tabm77b6oiqDJkluKteQlFBcII+ikBmLj6hLOx5htxbPFiKsUq7JqxJzBenXFnPItTYJ eVDYWzTavoeNpJVJaPivtaOPZxkijTkv6q/IELg78QHe4YCQmqG/0kPXmXi+yqv3bfgCHzHUNQRd EeH7TbTm5Jdw3kO89NA9cNyBPKlnwLodLf4v48BvswG4HjEiIj2rM77RGrnpmnBfdGoOKyJfZ8YT bgRU3f0tacV6jmkmpxJenUrvApNtb8pSCuhrS7fwpVvQy9EHdWQdfy/Xov6uFIkKBZVeazpHK9oB eSStxcctl7U7K7d4FUMVLbE4IaGU77ABpDfbvE6/TIY9WO1iPqO2IZtVwrclCV7RW4KeDgnLsHTA irQJNInMmoTjxWc5Ltrv6npJ26r6rCd3VbnsTXwjcFMSKYES9FZNESdWMVcGTC7azERnTdBphLOV /errQcmWYhVS2Q9+td8AfC7jeTiJjFNjUijcSL50++2uzN+xyST3WpYLm/xZlSvVOwok0ZNIy5Xa qBrSXEb+5/LxTWGn8Gq9cjOPTsPDG96jmUX6iLcGFAhChmdlD1pTtxsdZoQ7BvYlOhTwCFbdpQOl Peuk9ZuSkhQtj2uFck358MIbHjY7cMcdBguV3JosvvUUOUv9/dMsO/GfGfHwQkOPjqlXbMhJwDSA rpX32Hp1OB4XllixMlT+fOZZAffLHPQdPDXCu+PDbx7olCQYDVen3QIixZV0dI0ZRJSXNUFIJWrX 8DVeZBBp3nTPBlMS0+WHu2/+GgBd0EIfCmVuZHN0cmVhbQ1lbmRvYmoNMzUgMCBvYmoNPDwgDS9U eXBlIC9QYWdlIA0vUGFyZW50IDU4IDAgUiANL1Jlc291cmNlcyAzNiAwIFIgDS9Db250ZW50cyAz NyAwIFIgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIg XSANL1JvdGF0ZSAwIA0+PiANZW5kb2JqDTM2IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1Rl eHQgXSANL0ZvbnQgPDwgL1RUMiA2NSAwIFIgL1RUNCA2NyAwIFIgL1RUNiA3NiAwIFIgPj4gDS9F eHRHU3RhdGUgPDwgL0dTMSA5MSAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgNjggMCBSID4+ IA0+PiANZW5kb2JqDTM3IDAgb2JqDTw8IC9MZW5ndGggMzI0OCAvRmlsdGVyIC9GbGF0ZURlY29k ZSA+PiANc3RyZWFtDQpIiZxX227bSBJ911f0Y3NhMWzeiX3KONkdDeCsEQmYBeI80FTbZkaRtBQd b35kvner6lRTUiZOjIVhsS/V1XU9Vf3LavZqtUqNM6u7WRM3pUnoTwZF4uK6TkqT1XGRJkTxefbq 8lCY7iBEiTl0s1f/XDpzf5glcZLktVl1szkPHQ2fZnYfG5dGq098R447XCpH6dMQYU7sq6Tm64h7 wucTOZnFjZ4r9VycFnRu9WZmzTnHZNqRm5nHB3vVbtt7H82LOLWfI5dbH+Vxbbdj9HH127MqB5HK hq4TiVit9KhWysJ9sB/eR2QT+9EsrueRc3Fjb6M5Wcu2h6iij1+bz1Ga2TaaN3xr5EjIMLvHzGMm dP6MbiRRG2v2ww6jXbfbyADM44iEs8YslA6rD3oHKPFrbr3Z77CP317Wb7HtzY29fTy7UyXYQRtd HPx/lArHh4iM4UjJm8iohKA3SqZG+KF1iLAhi6Rx9hPzCKEHIbyXGOLmirIMPs+ayUNJAw+p8cpg vFKEKmCtwj5G5G/bPcjHtLq3jAq68l2U0f1X16bdyvbavL9amF6YYMV0Oww+RfMqzu0jZh1uBCWZ JCUj6QHSn2dPkSvEjHIM5CqDrkGES3wWsTG/ijRs4Jy+T1FaxRwvqf0i53g0XJAb5DR4kdMHD5U8 7sdNhmzq2HN37MCcPuScSdJBiSYZebYBX1jQgJnSK/naKxtVYKtiHR2V5+IocU+ZSYrTqMngqJFz lK1A+5IINQe32ou8n5IWFBkp727DgDNbBp/1wFbZmBbzQTQtbad0yr4HlRLr2Q4fWnQVrT6Gs0GW u51Ik01MjbJZRvOcNL5k34TJItBs/Tnbp6iw3zL6Q+cHtmhFiT0ZrSqaENx5doSf7AR+SoKfZVTS FZcyw3gRJafWc5JiOVJM8ahCiinqVCDzIBtJSgBPgcwpJG8aAZ0MoKNJ1AByMoacwvL/EW4aPdXT 6i1veTOq/nQhYCJTGv+cFh2TZ8CASRmRWLVZgx3YBKUpGtNw0XC0RJrF/5ctbqxXigtYYHku53fx KA1B7uCvt9uott2GTMNh80iTgT2ese5+kNTOKZ8JnjsPmguzfCvaV3YJQ5RQI0HJ4E/LytntWjdx 8CYCZzPuTpQHeSJ4FA7e08jLmumXHKuOXFDS0SUhTmMXlNkiUsSh2Xk5gzviUEOfLemla+KyRstw VtRd8m1Vr15U1dMSZX3BXpHcLMlkg/zuSMLS7k+XWlm67Tc9yL++rOpTw9NMZT+tgheTY9pVXPNH Kl8Nw340zySM2WojWS2fUqCc1ntZv914c8dxxOLKqcE8UCBVYXrKBGOGMuekPDLdGv6gEThypAjZ yL5jAOW9dmRw4jlE3G+8csHcg9loOKQKSQaeb3DmKyLlnKVq408SjNXwrVI9ChVHMftFhcJnD918 pzLpUShAXUCP28/v84bN/A9Uk8uIt3ffTzOXTaH1wRIvroyjH3aMxBVfzh/PyF/aoR39M/h6bB5c Mzm6oewGvtZAEyALigeHGJXJUpoWcqVno/Oi4AxdvdEgJUwBckGQRhJQGhrEJZ/GqJdfIF4aiAxW D9JsSlzxAL/YutWbgnMecTCQiFBbiH+vPLl3kdGX0Fr9Va7vSsPC4jqzkgJGfYrY6fqVrC6uDWMv qod0XtRJ2J0egUiqdvcHWMIjWVy6vBC35NXklqNX1ClvIg6mj+Y1NyiZ9CSVlcYnk1Jkd4+sI8bB IFwcSm2fBApTbkAj3jLgEPaewHYOb+0Fl9PQJRFZmCsPzs5ETEOpyCVz/SjjUZji2hMEPh75KuLB CxbU0peh1ZXgySWTsWVOLjIt9vCrkp5rqNJy19EgSgupddLheXSZWRBIrzq73m9xMY5g3AdDIRao FH6/7BUTYhbwGV2fxSgz2n46CoL/SijsfTea8UHG/mipRpKK6T8xKtQAyMIO/SjwRHyAXQonKcoX 2+BSsnUZSR+mLmHu3chw5KQmBhnIAoyK+01gFphDIEgwrHVT9dBVcffI7ZFpu85vPIFLv9vKwZjh hpJVUmGlCQSe/UHMtvobwMtNTYKiDj2zGLIaSuKKpO0Hqbk0J89IT73F9kgPsnazkRRMOZwaMYLO D+IrUoN5CPjVLPG4w2Fz0MHmCygYfOXgUyThHej2+A5s/TSs3urZyIlXz8Q7UK/ksE/vwmtCleNh vklZ3EXsVHM+E/B2E3g3Vm8P8rd6sRT0M70vGNz4pctt9mQM6ZiqIJkyIycd2SiXI7cT1xRuwp7U TYFcSuXiZ5lUAPaqY1w10ug3FNUyHYWklVfbFhOFy1LgMrfXBmykWDQoFrVVLlIsasnCIx+McWq3 PUiZ/DtY39iUrT2w82gTlPr5yhekrH4ivbhIvYWY9/hcAAmPz9NSXpFywA/StRMj1dSr4JRuheqy pPq80AcDXyxPBvGofCCquYN0WBuwOAY6kUrV/qEtvNhifeqpvyJOjzQ0yPhvkGIEonybzY8g7rf3 OKtkCk3Y9IzYX7CynbIdLDbHYlNKBLCcVOMwnRDhHA0mvoIlAdv2ADvdPPFLJvbj4xtIt5GJ2fR/ gJ0uf4UKt74LuivSKQaf62Z8O0wHT8Epn8CpDOB0FzUBUQiNPWcsd6HswUaclSYINoohLlEpqKVn 0s0wxdmRUVPJsULPixc8LvKqjrP0m6dFfv6uIBl/9qyQ/M70XfGe1EOOelZtXmmH/ZJnQ17SuTo8 GzIw/MBlqLHUq+Rk/CutOnPU2JS8TI2Meldp3kWpPL742IJRjdrod5fS1C5WQlrpbhxRK2B+97d+ uOBmWQo8J+hm/xARFv4rvjB+3Y+7IeYqZAyOgdf5BXr3gGn30I9UlTkgMqpCTVj35mq39htzKj4h z09VvIlihD3L7ohfQnVBqwrz3dHSJ9zG17Kx365mzvRmljZF7Ah984JeksYV8jIjl+U1Vb/Z3eyX 1XPeyClMy+CN4ojkiGPnimoeiUBvIpY7jthCA72tHCWcyy5IHWvS1Fy13DtR12TShPeS5Pvvhyyb 7pgeiksCLOqLGdtqsXsp3RpbhW/+CLuX5HPKncKCnH2iLi+Dy3PdI5ezu9TpLD/N3gsluz6H63nx xPnseD4MjqeXqGTsBttLM+hsOwg+OS4bzDGE5+VJf1lLl4gf6UsldfneNTRj3KII/NOkEh8/tAKi wyWQ+1qIRaKdrHBgnIZF2rg4rSkiMkpuDomM3f6ikCDwcenzIZFm5Y9CotaQcBQSUlJpMU1COATE zCbErDQILvH8eo1n4xVeYXMK+Qwv01MASCUYGH+WC/NOU+kSz6qVmJ4igcz1hhq/9nBhfmfe1Jhv Nn2r+SUYS6K9kQDAFT3avspyMCy2AgCE4zasSolw4SjwOFc4ztC38M2+e5AFLCvNBthdUyshc+m9 yE5/Ymouw/5nZR/myvx1xwA7577vYK78+HDKXOUnZur9LMnlKZgnZJmUnFG+1PsZx01wPr8vw7NS /ZRRpFIzw14QLzUk4hy9cAaYuokkVP8d8YrEPBvGWZdIl4/miztn2fvEriklcgu2R9NgPaHgsrHG FS+Y7ELOE1pdtQOGgbiOg+KihiMtJIbzmhV+kdpVdlKV8qPaGp1No+/HNC8GjOqP5tdWH5DyzuP6 8lv/P8qrpqVhIIj+lRwb0DSb7X55rQi9FMFLoXiIWFEIban+Fn+v82Zm06QRpZe0SWY3u+/Nzpun ZpG0BLamT1n4yuJ+99V+dLtXqSgGGYVCdTgexJa2XaETsns8FTqQuUeAXM+1JkoOqNEU+3k6dBKm aoLpTD0f7ACxTpFVtyoj1LnyZEW8KRqfV3CU9dBBxru6x9sQgJ5QsqGB8Hg6WiPhcdYIITEC35pf m4Ck/IcQ51WYDL/8lHLR9HUja8cjFNGgUqPKU8GgHWxKJCLrhuVeAbdaKmqurj73CRZ9Amq/PKMg nRDNKjFEFYSknIpBhN964+lb6uO+i80k6Zyr6kgYxWqEQZ030IMRDPZ2BRiW0FtMwKAP9HkaFY+H EpVuyd0n7cRlHJBua6LXkM4gYlVym72GbEj0SnxX0oCKXStE8TzhcCgfeieHngRAvgvP4mYv5BKW 7yVD3e738mdHDVuadRxADowtkMsWiB5tf1skqgjd6Ueqxqc7jjYyZ0pe/jxfUmEDVM/Wl43QlAsS RoL2Ci5I7kUeR1xMdJKZSAycFSbMTHWzgWwrK8hOquqRd5wY2wBaoKwyVnkJGiFy+zSYfDhSv2C5 b5EVgJPQc4LEvk3sa+UCn2iYl9DzEjIvQZqTbV6C4dYOl3lpsGpjFrhbjDqCCRsNfosmkXeNf7Nh PcCdsPEjwADG71+KCmVuZHN0cmVhbQ1lbmRvYmoNMzggMCBvYmoNPDwgDS9UeXBlIC9Gb250IA0v U3VidHlwZSAvVHJ1ZVR5cGUgDS9GaXJzdENoYXIgNDYgDS9MYXN0Q2hhciAxMTQgDS9XaWR0aHMg WyAyNTAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAw IDAgMCAwIDAgMCANMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAzODkgMCAzODkgMCAwIDAgMCAw IDAgMCA0NDQgMjc4IDAgMCAwIA0wIDAgMCAwIDAgMCAwIDAgMzg5IF0gDS9FbmNvZGluZyAvV2lu QW5zaUVuY29kaW5nIA0vQmFzZUZvbnQgL1RpbWVzTmV3Um9tYW5QUy1JdGFsaWNNVCANL0ZvbnRE ZXNjcmlwdG9yIDM5IDAgUiANPj4gDWVuZG9iag0zOSAwIG9iag08PCANL1R5cGUgL0ZvbnREZXNj cmlwdG9yIA0vQXNjZW50IDg5MSANL0NhcEhlaWdodCAwIA0vRGVzY2VudCAtMjE2IA0vRmxhZ3Mg OTggDS9Gb250QkJveCBbIC0xODkgLTMwNyAxMTIwIDEwMjMgXSANL0ZvbnROYW1lIC9UaW1lc05l d1JvbWFuUFMtSXRhbGljTVQgDS9JdGFsaWNBbmdsZSAtMTUgDS9TdGVtViAwIA0+PiANZW5kb2Jq DTQwIDAgb2JqDTw8IA0vQ291bnQgMTQgDS9GaXJzdCA0MSAwIFIgDS9MYXN0IDQyIDAgUiANPj4g DWVuZG9iag00MSAwIG9iag08PCANL1RpdGxlIChBcHBsaWNhYmlsaXR5KQ0vRGVzdCBbIDYxIDAg UiAvRml0QiBdIA0vUGFyZW50IDQwIDAgUiANL05leHQgNDQgMCBSIA0+PiANZW5kb2JqDTQyIDAg b2JqDTw8IA0vVGl0bGUgKFJlZmVyZW5jZXMpDS9EZXN0IFsgMzUgMCBSIC9GaXRCIF0gDS9QYXJl bnQgNDAgMCBSIA0vUHJldiA0MyAwIFIgDT4+IA1lbmRvYmoNNDMgMCBvYmoNPDwgDS9UaXRsZSAo UmVxdWlyZW1lbnRzKQ0vRGVzdCBbIDUgMCBSIC9GaXRCIF0gDS9QYXJlbnQgNDAgMCBSIA0vUHJl diA0NCAwIFIgDS9OZXh0IDQyIDAgUiANL0ZpcnN0IDQ1IDAgUiANL0xhc3QgNDYgMCBSIA0vQ291 bnQgMTAgDT4+IA1lbmRvYmoNNDQgMCBvYmoNPDwgDS9UaXRsZSAoRGVmaW5pdGlvbnMpDS9EZXN0 IFsgNSAwIFIgL0ZpdEIgXSANL1BhcmVudCA0MCAwIFIgDS9QcmV2IDQxIDAgUiANL05leHQgNDMg MCBSIA0+PiANZW5kb2JqDTQ1IDAgb2JqDTw8IA0vVGl0bGUgKEdlbmVyYWwpDS9EZXN0IFsgNSAw IFIgL0ZpdEIgXSANL1BhcmVudCA0MyAwIFIgDS9OZXh0IDU0IDAgUiANPj4gDWVuZG9iag00NiAw IG9iag08PCANL1RpdGxlIChJbnRlcm9wZXJhYmlsaXR5KQ0vRGVzdCBbIDM1IDAgUiAvRml0QiBd IA0vUGFyZW50IDQzIDAgUiANL1ByZXYgNDcgMCBSIA0+PiANZW5kb2JqDTQ3IDAgb2JqDTw8IA0v VGl0bGUgKE1hbmFnZW1lbnQpDS9EZXN0IFsgMzUgMCBSIC9GaXRCIF0gDS9QYXJlbnQgNDMgMCBS IA0vUHJldiA0OCAwIFIgDS9OZXh0IDQ2IDAgUiANPj4gDWVuZG9iag00OCAwIG9iag08PCANL1Rp dGxlIChUb3BvbG9neSBEaXNjb3ZlcnkpDS9EZXN0IFsgMzIgMCBSIC9GaXRCIF0gDS9QYXJlbnQg NDMgMCBSIA0vUHJldiA0OSAwIFIgDS9OZXh0IDQ3IDAgUiANPj4gDWVuZG9iag00OSAwIG9iag08 PCANL1RpdGxlIChTZWN1cml0eSkNL0Rlc3QgWyAyOSAwIFIgL0ZpdEIgXSANL1BhcmVudCA0MyAw IFIgDS9QcmV2IDUwIDAgUiANL05leHQgNDggMCBSIA0+PiANZW5kb2JqDTUwIDAgb2JqDTw8IA0v VGl0bGUgKE5hbWluZykNL0Rlc3QgWyAyOSAwIFIgL0ZpdEIgXSANL1BhcmVudCA0MyAwIFIgDS9Q cmV2IDUxIDAgUiANL05leHQgNDkgMCBSIA0+PiANZW5kb2JqDTUxIDAgb2JqDTw8IA0vVGl0bGUg KFRyYW5zcG9ydCwgTmV0d29yayBhbmQgTGluaykNL0Rlc3QgWyAxOCAwIFIgL0ZpdEIgXSANL1Bh cmVudCA0MyAwIFIgDS9QcmV2IDUyIDAgUiANL05leHQgNTAgMCBSIA0+PiANZW5kb2JqDTUyIDAg b2JqDTw8IA0vVGl0bGUgKGlTQ1NJIFNlc3Npb24gTGF5ZXIpDS9EZXN0IFsgMTIgMCBSIC9GaXRC IF0gDS9QYXJlbnQgNDMgMCBSIA0vUHJldiA1MyAwIFIgDS9OZXh0IDUxIDAgUiANPj4gDWVuZG9i ag01MyAwIG9iag08PCANL1RpdGxlIChTQ1NJKQ0vRGVzdCBbIDkgMCBSIC9GaXRCIF0gDS9QYXJl bnQgNDMgMCBSIA0vUHJldiA1NCAwIFIgDS9OZXh0IDUyIDAgUiANPj4gDWVuZG9iag01NCAwIG9i ag08PCANL1RpdGxlIChQZXJmb3JtYW5jZS9Db3N0KQ0vRGVzdCBbIDUgMCBSIC9GaXRCIF0gDS9Q YXJlbnQgNDMgMCBSIA0vUHJldiA0NSAwIFIgDS9OZXh0IDUzIDAgUiANPj4gDWVuZG9iag01NSAw IG9iag08PCANL1Byb2R1Y2VyIChBY3JvYmF0IERpc3RpbGxlciA0LjAgZm9yIFdpbmRvd3MpDS9D cmVhdG9yIChNaWNyb3NvZnQgV29yZCA4LjApDS9Nb2REYXRlIChEOjIwMDAwNzA3MTIwNzI5LTA3 JzAwJykNL0F1dGhvciAoUmFuZG9scGggQi4gSGFhZ2VucykNL1RpdGxlIChTQ1NJIG9uIFRDUCBS ZXF1aXJlbWVudHMgRG9jdW1lbnQpDS9DcmVhdGlvbkRhdGUgKEQ6MjAwMDA3MDcxMjA1MjYpDT4+ IA1lbmRvYmoNNTYgMCBvYmoNPDwgDS9UeXBlIC9QYWdlcyANL0tpZHMgWyA2MSAwIFIgMSAwIFIg NSAwIFIgOSAwIFIgMTIgMCBSIDE1IDAgUiAxOCAwIFIgMjEgMCBSIDI1IDAgUiAyOSAwIFIgDV0g DS9Db3VudCAxMCANL1BhcmVudCA1NyAwIFIgDT4+IA1lbmRvYmoNNTcgMCBvYmoNPDwgDS9UeXBl IC9QYWdlcyANL0tpZHMgWyA1NiAwIFIgNTggMCBSIF0gDS9Db3VudCAxMiANPj4gDWVuZG9iag01 OCAwIG9iag08PCANL1R5cGUgL1BhZ2VzIA0vS2lkcyBbIDMyIDAgUiAzNSAwIFIgXSANL0NvdW50 IDIgDS9QYXJlbnQgNTcgMCBSIA0+PiANZW5kb2JqDXhyZWYNMCA1OSANMDAwMDAwMDAwMCA2NTUz NSBmDQowMDAwMDEyNjM4IDAwMDAwIG4NCjAwMDAwMTI4MDggMDAwMDAgbg0KMDAwMDAxMjk1MSAw MDAwMCBuDQowMDAwMDEzMTA4IDAwMDAwIG4NCjAwMDAwMTk5ODEgMDAwMDAgbg0KMDAwMDAyMDE1 MSAwMDAwMCBuDQowMDAwMDIwMjk0IDAwMDAwIG4NCjAwMDAwMjA0NTIgMDAwMDAgbg0KMDAwMDAy NTc0OCAwMDAwMCBuDQowMDAwMDI1OTAxIDAwMDAwIG4NCjAwMDAwMjYwNjAgMDAwMDAgbg0KMDAw MDAzMjE4MiAwMDAwMCBuDQowMDAwMDMyMzM2IDAwMDAwIG4NCjAwMDAwMzI1MDggMDAwMDAgbg0K MDAwMDAzOTI1NCAwMDAwMCBuDQowMDAwMDM5NDA4IDAwMDAwIG4NCjAwMDAwMzk1NTUgMDAwMDAg bg0KMDAwMDA0NzA4MiAwMDAwMCBuDQowMDAwMDQ3MjM2IDAwMDAwIG4NCjAwMDAwNDc0MDYgMDAw MDAgbg0KMDAwMDA1MjY4NyAwMDAwMCBuDQowMDAwMDUyODYxIDAwMDAwIG4NCjAwMDAwNTMwMDYg MDAwMDAgbg0KMDAwMDA1MzE1MiAwMDAwMCBuDQowMDAwMDYwODE3IDAwMDAwIG4NCjAwMDAwNjA5 OTEgMDAwMDAgbg0KMDAwMDA2MTEzNiAwMDAwMCBuDQowMDAwMDYxMjgzIDAwMDAwIG4NCjAwMDAw NjgwNzAgMDAwMDAgbg0KMDAwMDA2ODIyNCAwMDAwMCBuDQowMDAwMDY4MzgzIDAwMDAwIG4NCjAw MDAwNzQ3NTMgMDAwMDAgbg0KMDAwMDA3NDkwNyAwMDAwMCBuDQowMDAwMDc1MDY2IDAwMDAwIG4N CjAwMDAwODA0NjkgMDAwMDAgbg0KMDAwMDA4MDYyMyAwMDAwMCBuDQowMDAwMDgwNzgyIDAwMDAw IG4NCjAwMDAwODQxMDUgMDAwMDAgbg0KMDAwMDA4NDQ0NiAwMDAwMCBuDQowMDAwMDg0NjQ2IDAw MDAwIG4NCjAwMDAwODQ3MTAgMDAwMDAgbg0KMDAwMDA4NDgxMSAwMDAwMCBuDQowMDAwMDg0OTA5 IDAwMDAwIG4NCjAwMDAwODUwNjIgMDAwMDAgbg0KMDAwMDA4NTE3NCAwMDAwMCBuDQowMDAwMDg1 MjY4IDAwMDAwIG4NCjAwMDAwODUzNzIgMDAwMDAgbg0KMDAwMDA4NTQ4NCAwMDAwMCBuDQowMDAw MDg1NjA0IDAwMDAwIG4NCjAwMDAwODU3MTQgMDAwMDAgbg0KMDAwMDA4NTgyMiAwMDAwMCBuDQow MDAwMDg1OTUxIDAwMDAwIG4NCjAwMDAwODYwNzIgMDAwMDAgbg0KMDAwMDA4NjE3NyAwMDAwMCBu DQowMDAwMDg2Mjk0IDAwMDAwIG4NCjAwMDAwODY1MzUgMDAwMDAgbg0KMDAwMDA4NjY3OSAwMDAw MCBuDQowMDAwMDg2NzUzIDAwMDAwIG4NCnRyYWlsZXINPDwNL1NpemUgNTkNL0lEWzxkNjE2ZWI0 NWQ5MWJhNmJlMTQwZGZkNmYyOGNlYTc3ZT48ZDYxNmViNDVkOTFiYTZiZTE0MGRmZDZmMjhjZWE3 N2U+XQ0+Pg1zdGFydHhyZWYNMTczDSUlRU9GDQ== ------_=_NextPart_000_01BFE848.21FFBA00-- From owner-ips@ECE.cmu.edu Sat Jul 08 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15747; 7 Jul 2000 23:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA08815 for ips-outgoing; Fri, 7 Jul 2000 21:50:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-ID: <5B0C98C89640D311A319002048406A1C05B31C3C@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: "'David Robinson'" , Jim McGrath Cc: ips@ece.cmu.edu Subject: RE: Security Considerations Date: Fri, 7 Jul 2000 17:15:37 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim McGrath wrote: > > My impression (which I admit could be quite mistaken) is that we may be on a > path of requiring a higher level of security for iSCSI than is warranted by > alternative protocols. Of course, enhancing security has its value, but > since SCSI starts off with essentially no security, iSCSI is a poor protocol > upon which to require lots of security. > I've quoted below portions of section 5 of "Guidelines for Writing RFC Text on Security Considerations". My email is was intended as a start to the "due diligence" process. While it is not a requirement that any given protocol or system be immune to all forms of attack, it is still necessary for authors to consider them. Part of the purpose of the Security Considerations section is to explain what attacks are out of scope and what counter- measures can be applied to defend against them. There should be a clear description of the kinds of threats on the described protocol or technology. This should be approached as an effort to perform "due diligence" in describing all known or foresee- able risks and threats to potential implementers and users. At least the following forms of attack MUST be considered: eavesdrop- ping, replay, message insertion, deletion, modification, and man-in- the-middle. Potential denial of service attacks MUST be identified as well. David Robinson wrote: > The benchmark that iSCSI will have to meet is what is being done > in the NFSv4 WG. Using the ONCRPC WG mechanisms based on GSSAPI > and RPCSEC_GSS they provide authentication, integrity, and privacy > using one of two manditory to implement mechanisms, Kerberos V5 > and LIPKEY. > > I suspect that if iSCSI doesn't address security to provide similar > capabilities it will not pass muster. Of course the security should > always be able to be negotiated to the desired levels. I agree on the pass muster comment. I also agree on the negotiating of security level. Hence, the 3 layer model of no security, connection-oriented authentication on logins (thanks to Brent on the distinction), and encryption. Luciano Dalle Oro pointed out that IPsec provides excellent data integrity in addition to security. Paul From owner-ips@ECE.cmu.edu Sat Jul 08 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15751; 7 Jul 2000 23:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA08787 for ips-outgoing; Fri, 7 Jul 2000 21:49:03 -0400 (EDT) From: owner-ips@ece.cmu.edu X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-ID: <0F31E5C394DAD311B60C00E029101A070148FAE1@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Cc: Black_David@emc.com Subject: Administrivia Date: Fri, 7 Jul 2000 19:40:17 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Time and space has been requested in Pittsburgh. Date and time will be forthcoming, one two hour slot is expected. Your future co-chairs and ADs are editing the charter as I write this. Expect an initial version to appear for further comment on the list sometime early next week. Another thing that will happen next week will be initial assembly of the agenda for the meeting. For those who want time on the agenda at the meeting in Pittsburgh, submission of an Internet-Draft is highly recommended. While there's no guarantee that everyone who submits a draft will be on the agenda, no draft almost certainly implies no agenda slot. So, I'd ask those who desire time on the agenda to please send me an email indicating the topic, the title and filename of the draft, and the name and email of the presenter. A few matters for those who may not have written Internet-Drafts before: - Please read the Guidelines to Authors of Internet-Drafts at http://www.ietf.org/ietf/1id-guidelines.txt . - In particular, note the requirements there for boilerplate text on the first page, including the required reference to RFC 2026. Drafts without the required boilerplate will be returned. - Since ips is not yet a working group, there are no such things as ips working group drafts, and hence names of the form draft-ietf-ips- ... are not appropriate. Drafts should be sent as individual submissions; a useful convention is to use the WG acronym as the third word of the draft name. For example, if this email were turned into an Internet-Draft, one possible name would be draft-black-ips-administrivia-00.txt . - The draft deadline is next Friday (7/14) at 5pm Eastern Time. The deadline is absolute, and what counts is whether the draft is received by the deadline, not whether it was sent and got hung up in email. Submitting drafts hours/days earlier is a good idea, and authors in California should remember that the deadline is Eastern time, *not* Pacific time. I'd ask those who submit drafts close to the deadline, to please consider posting a copy on a web site somewhere and sending the URL to the list; this helps with the fact that there can be a multi-day processing backlog for last-minute draft submissions. My apologies for using email bandwidth for administrivia, but this is occasionally necessary. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sat Jul 08 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15739; 7 Jul 2000 23:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA08956 for ips-outgoing; Fri, 7 Jul 2000 21:56:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA08064 for ; Fri, 7 Jul 2000 21:10:38 -0400 (EDT) From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id VAA16602 for ; Fri, 7 Jul 2000 21:08:08 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id TAA91918 for ; Fri, 7 Jul 2000 19:10:36 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256916.000672FC ; Fri, 7 Jul 2000 19:10:26 -0600 X-Lotus-FromDomain: IBMUS To: ips@ece.cmu.edu Message-ID: <87256916.0005A6A8.00@d53mta03h.boulder.ibm.com> Date: Fri, 7 Jul 2000 18:01:18 -0700 Subject: noop and ping Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Can't the functionality of the NOOP command be subsumed by the PING command with a bit advising the target to ignore the task tag? Also, aren't we doing register synchronization on almost all commands to need a noop command of its own. if the argument is the need to do synchronization when there are no outstanding commands, i have a suspicion that this could be done without adding a new command. Alternatively, if there is a need for both, it should be spelled out in the document. Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose From owner-ips@ECE.cmu.edu Sat Jul 08 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15764; 7 Jul 2000 23:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA08870 for ips-outgoing; Fri, 7 Jul 2000 21:51:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Message-ID: From: Jim McGrath To: "'Black_David@emc.com'" , wmain@gis.net, ips@ece.cmu.edu Subject: RE: iSCSI has already been done before, sort of... Date: Fri, 7 Jul 2000 18:12:32 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I think the question is whether there is a document similar to iSCSI for IPI over a network that we could learn from. My problem with the actual IPI-3 document is that I believe it is similar in scope to the SCSI documents themselves - good at defining the protocol for the traditional storage model, but not good at highlighting the changes/issues associated with putting it over a network. If there was a iIPI (IPI-3 over some network) document(s) out there, then that could provide some useful guidance. If I am mistaken, and the actual IPI-3 documents provide this level of guidance, then it would probably help others to identify the appropriate sections, since the document itself is rather large. Jim -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Friday, July 07, 2000 11:49 AM To: wmain@gis.net; ips@ece.cmu.edu Subject: RE: iSCSI has already been done before, sort of... It's not clear to me what the point of this message is, since the author says: > Right up front, this is **NOT** proposed as a protocol for adoption but it > does offer to us some valuable insights in how to handle some of the > questions at hand. but then has very little to say in terms of exactly what the insights are. The bullet list of IPI-3 features isn't particularly helpful; is the intent to put them forward as requirements? Could I suggest writing an Internet-Draft on the topic of what the author thinks is wrong with iSCSI, why, and how to fix it as a productive alternative to making disparaging comments on the efforts of others? --David (future WG co-chair) --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Bill Main [SMTP:wmain@gis.net] > Sent: Friday, July 07, 2000 10:13 AM > To: ipSCSI list > Subject: iSCSI has already been done before, sort of... > > Folks- > If this seems late in the cycle, I apologize but I just recently > joined this project and I realize there is an IETF deadline looming > but.... > > How does the adage go? Those who fail to learn from history are > doomed to repeat it. > > That I believe applies here. Many protocols have been put forward or > mentioned but one seems distinctly absent from this discussion. Right up > front, this is **NOT** proposed as a protocol for adoption but it does > offer to us some valuable insights in how to handle some of the > questions at hand. > > The important thing is that it does everything iSCSI is trying to do and > it has been running for years at performance levels that iSCSI will > struggle to match. > > Here are some of its features: > > - Been around a long time (not well known even though has an ANSI spec) > - Specs revised and improved 1986, 92, 95 and latest twik to spec in 97 > - Up and running and supported by IBM, Compaq(Digital), NEC, Cray and > SGI & probably others. > - Connects large storage farms to hosts over switched network > - Deals well with congestion delays > - Handles tapes > - works with all common unix host based file systems. > - priority classifications for what now is called QoS. > - adaptable block size on transmit > - allows coalescing of data for transfer (sorts transmit queue to > optimize transfers) > - Designed and built to function on full duplex 100 MB/s networks albeit > LAN only. > - Optional multiple path capability for load balancing or striping for > both hosts and targets. > - Optional out of band command and response channel (usually Ethernet). > - Header layout done so receiver has a priori knowledge of pay load and > can segregate address data such that payload goes to page flip memory > (Does exactly what RDMA wants to do). > - Has a transmit credit system. > - Sports a well thought out error and recovery system (this could be > very useful) > - Handles partition, spanning and volume work. > - Performance numbers are commonly in the range of 85MB/s raw and 50 > MB/s with a file system (WRITE) on single path with in line command and > response using 8KB file blocks. The 50 is a file system limitation > regardless of storage system class. > - relatively easy to implement, only about 1.5 man-years to get it > designed, coded and working into production envirnoment > - has stages of operation that are very SMP (or ASIC) friendly > > > It is called IPI-3 and runs currently only over Hippi networks which are > as close to being a SAN as any FC layout. Specs are available from the > T11 group (http://www.t11.org/index.html) > Grab the IPI-3DR and IPI-3TR versions (disk and tape respectively). > > Having been the original architect on one implementation of IPI-3, > there is a lot of knowledge in there that has been stepwise improved on > over the 14 years of its existence, but it does exactly what iSCSI is > trying to do. > > Now the bad news: Does not use IP or TCP but IPI-3 is very efficient. > Note though, the Hippi-PH can be replaced by IP if one really wanted to > without much effort. > > First, from my perspective it appears that iSCSI is a re-invention of > the wheel - Not that a redesigned wheel can't be made better. But it > always seemed better to study the original while doing it. > > Second, when you pull the specs, you will notice that they are weighty. > Having designed an implementation of a network disk transfer system (ala > iSCSI) for IPI-3, I assure you there is very little excess in there. > Eventually every issue that generated paragraphs in the IPI-3 spec will > show up in the iSCSI implementations as well. > > Regards, > Bill Main > From owner-ips@ECE.cmu.edu Sat Jul 08 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18021; 8 Jul 2000 11:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA27657 for ips-outgoing; Sat, 8 Jul 2000 10:12:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA11088 for ; Fri, 7 Jul 2000 23:55:41 -0400 (EDT) Received: from beppo.feral.com (beppo [192.67.166.79]) by feral.com (8.9.3/8.9.3) with ESMTP id UAA19970; Fri, 7 Jul 2000 20:55:30 -0700 Date: Fri, 7 Jul 2000 20:55:31 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: Jim McGrath cc: "'Black_David@emc.com'" , wmain@gis.net, ips@ece.cmu.edu Subject: RE: iSCSI has already been done before, sort of... In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > > > I think the question is whether there is a document similar to iSCSI for IPI > over a network that we could learn from. My problem with the actual IPI-3 > document is that I believe it is similar in scope to the SCSI documents > themselves - good at defining the protocol for the traditional storage > model, but not good at highlighting the changes/issues associated with > putting it over a network. > > If there was a iIPI (IPI-3 over some network) document(s) out there, then > that could provide some useful guidance. > > If I am mistaken, and the actual IPI-3 documents provide this level of > guidance, then it would probably help others to identify the appropriate > sections, since the document itself is rather large. I don't believe that there have ever been any public specs in this area. I know of a couple of implementations of IPI-3 over both TCP/IP and Hippi raw framing protocol- but I don't recall that they got to any kind of standardization level (possibly in the Hippi specs somewhere- I don't have them handy). Anybody on the list know anyone from MaxStrat (mass storage for SuperComputer's company- maybe bought by Sun???? I think they may have also done this as well. -matt From owner-ips@ECE.cmu.edu Sat Jul 08 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18039; 8 Jul 2000 11:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA27798 for ips-outgoing; Sat, 8 Jul 2000 10:18:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA27768 for ; Sat, 8 Jul 2000 10:17:41 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id KAA19178; Sat, 8 Jul 2000 10:17:41 -0400 Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA12562 for ; Sat, 8 Jul 2000 01:24:43 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA113362 for ; Sat, 8 Jul 2000 07:24:00 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA142164 for ; Sat, 8 Jul 2000 07:24:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256916.001DACDB ; Sat, 8 Jul 2000 07:24:07 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 8 Jul 2000 08:23:05 +0300 Subject: Re: Third party naming vis-a-vis iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, Your first solution is what we already did in the working draft (0706). The only element on which we in fact differ is that we decomposed the action in two elements - the address - that is given out by the initiator and what you call an alias (in the draft it is a SAR) that is set by the target. This way the target can manage by himself the small name server in the best way he sees fit. This can be extended to several other protocol elements. The disassociation between mapping and command is not a major impediment as we are talking about long and complex commands anyhow and can be even an advantage if the commands are repeated as the decoding logic in the target gets simpler and more regular. The one point in which I would like us to interact sometime with T10 is the logical unit name and mapping - this messier than the target address mapping - your access control merging with LU identification does not make things simpler. In the current draft we allude to the fact that the proxy mapping has to be discussed with T10 but I think the issue is larger than that. Regards, Julo hafner@almaden.ibm.com on 06/07/2000 19:56:45 Please respond to hafner@almaden.ibm.com To: "IPS (E-mail)" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Third party naming vis-a-vis iSCSI Here's my promised suggestion for third-party naming. There are two approaches, one fully integrated into SCSI (and so independent of iSCSI, though iSCSI can best take advantage of it). The other approach is more attuned to iSCSI, though could be used in other (probably future) transports. The real problem in this space is not in the higher application layers which can do naming in anyway they want, so long as there is an application that can translate from one form to another. (So a URL can get "resolved" by different application layers into an ethernet MAC and IP port, etc.) The problem is that an arbitrary naming scheme doesn't map well into existing SCSI paramter data where third party addresses are used (e.g., third party reservations, RAID commands like XDWRITE, extended copy, etc.). The biggest problem in that space is the fact that SCSI has only (at the moment) set aside fixed sized fields for this purpose (and we know iSCSI is going to need much longer names). In some cases, this field is 8 bytes (e.g., third party reservations) and in others 16 bytes (bytes 12-27 in extended copy target descriptors; I'm referring here to the identifier of the "target device" and not the logical unit within that device). The first suggestion, then is to use the existing fixed field (and only 8 bytes of that) of third party addressing as an "alias" field. The requesting initiator will (through a mechanism described later) define for the target a translation mapping of this alias to the transport specific identifier of the target (e.g., the portion of the url, the FC WWN, the parallel SCSI address). A flag bit or field (TBD for each usage) will indicate that this is an alias field and not a specific address identifier. Using aliases in this way, we don't need to change the size of the fields currently dedicated to third party addresses and we don't need to change their existing usage. Here are the two approaches for an initiator to define the alias<->thirdparty mapping. Note, in all cases, (without a completely different protocol) the mapping would be initiator specific, not a global aliasing. 1) Within iSCSI, a Text command could define the mapping. Advantages: a) you'd only need to change SPC-2 to allow for alias-type usage, and leave the mapping to "the transport". b) it would apply to all third-party addressing contexts. Disadvantages: a) there's a disassociation between the actual mapping and the use of the mapping (that is, state must be maintained) and so all the issues about state (persistence, etc) have to be addressed. b) it's not clear how other transports could take advantage of this. 2)an additional "chunk" of the parameter data could be used to define the alias. For example, in extended copy, an additional "page" of parameter data can define the mapping for that instance of the command. Advantages: a) can be used in all transports and future transports b) is a general extensible mechanism c) has state only for the context of the command itself d) can handle whatever size of device identifier you want (e.g., arbitrarily long ) - that's because you can define the parameter data in this way. Disadvantages a) would require more changes to SPC-2 (both alias usage and the definition of mapping), and would (might?) affect any command where third-party addressing is used. As for how logical units are referenced within a target, the existing mechanisms of LUN and Proxy Token (from access controls) are sufficient. In all cases, the harder part is finding the device. SCSI already knows how to find the logical unit within the device. But I don't see a fundamental problem with adding this logical unit identifier (in whatever form you like, LUN, Proxy Token, EVPD data) as part of the alias mapping. The subtle issue is that in some cases there is no logical unit to reference (e.g., third party reservations reference an initiator, not a target). Political comment: After having worked with the T10 guys over the last year or so, I find them very amenable to additions/changes to SCSI so long as it doesn't break anything and that it meets someones perceived need. I think the issues raised here meet both those requirements. So, I wouldn't be reluctant (if I were more active in this iSCSI thing) to take specific proposals to T10 to enhance what iSCSI can do. Jim Hafner From owner-ips@ECE.cmu.edu Sat Jul 08 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18041; 8 Jul 2000 11:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA27636 for ips-outgoing; Sat, 8 Jul 2000 10:10:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA27724 for ; Sat, 8 Jul 2000 08:19:36 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id IAA208120 for ; Sat, 8 Jul 2000 08:20:12 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256916.0022CDDA ; Sat, 8 Jul 2000 08:20:09 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 8 Jul 2000 09:18:48 +0300 Subject: Re: multiple tcp connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Parallelism on end-points could be another reasons. A long time ago when running some benchmarks we achieved considerably better numbers with 4 sockets than with one. This type of result repeated itself in different contexts and with different stacks. That is probably due to some serialization that is inherent in the way stacks are built and API's are used (this is a long discussion subject). With SMPs the differences are even more striking. Julo Matt Wakeley on 08/07/2000 02:11:57 Please respond to Matt Wakeley To: Randy Haagens , Julian Satran/Haifa/IBM@IBMIL cc: Subject: multiple tcp connections Hi, I was asked the question: If there is only 1 MAC (available to the system), would there ever be more than 1 tcp connection per iSCSI session. I couldn't think of a good reason why there would be, but thought I'd ask you. The only reason I could think of using multiple tcp connections per iSCSI session was for port aggregation when there are multiple MACs available. Any comments? Thanks, -Matt From owner-ips@ECE.cmu.edu Sat Jul 08 21:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19018; 8 Jul 2000 17:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA03577 for ips-outgoing; Sat, 8 Jul 2000 16:21:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA02011 for ; Sat, 8 Jul 2000 14:40:26 -0400 (EDT) From: hufferd@us.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA17718 for ; Sat, 8 Jul 2000 14:37:55 -0400 Received: from d53mta08h.boulder.ibm.com (d53mta08h.boulder.ibm.com [9.99.142.8]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id MAA80450 for ; Sat, 8 Jul 2000 12:40:24 -0600 Received: by d53mta08h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256916.00668E36 ; Sat, 8 Jul 2000 12:40:10 -0600 X-Lotus-FromDomain: IBMUS To: julian_satran@il.ibm.com cc: ips@ece.cmu.edu Message-ID: <87256916.00660A7D.00@d53mta08h.boulder.ibm.com> Date: Sat, 8 Jul 2000 11:34:22 -0700 Subject: Re: multiple tcp connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Since a session is only an iSCSI thing, your statements would be correct even if there were only one TCP connection per session, but several sessions on the same NIC. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 07/07/2000 11:18:48 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: multiple tcp connections Parallelism on end-points could be another reasons. A long time ago when running some benchmarks we achieved considerably better numbers with 4 sockets than with one. This type of result repeated itself in different contexts and with different stacks. That is probably due to some serialization that is inherent in the way stacks are built and API's are used (this is a long discussion subject). With SMPs the differences are even more striking. Julo Matt Wakeley on 08/07/2000 02:11:57 Please respond to Matt Wakeley To: Randy Haagens , Julian Satran/Haifa/IBM@IBMIL cc: Subject: multiple tcp connections Hi, I was asked the question: If there is only 1 MAC (available to the system), would there ever be more than 1 tcp connection per iSCSI session. I couldn't think of a good reason why there would be, but thought I'd ask you. The only reason I could think of using multiple tcp connections per iSCSI session was for port aggregation when there are multiple MACs available. Any comments? Thanks, -Matt From owner-ips@ECE.cmu.edu Sun Jul 09 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19423; 8 Jul 2000 20:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA05830 for ips-outgoing; Sat, 8 Jul 2000 18:55:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ophidian.com (ophidian.com [209.136.122.245] (may be forged)) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA05397 for ; Sat, 8 Jul 2000 18:25:26 -0400 (EDT) Received: from boa [216.84.97.93] by ophidian.com (SMTPD32-5.05) id AD5938760150; Sat, 08 Jul 2000 16:38:17 -0600 Message-ID: <006901bfe92b$ea624ec0$0302010a@boa> From: "Edward A. Gardner" To: Subject: Fw: 00-279r0 Large SCSI Device Identifiers Date: Sat, 8 Jul 2000 16:29:00 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id SAB05830 FYI, the following will be discussed at the T10 working group next Wednes= day (July 12). I'm leaving the next day on an overseas trip, so DO NOT try t= o ask me how it went. However, I do plan to be in Pittsburg. Edward A. Gardner eag@ophidian.com Ophidian Designs 719 593-8866 voice 1262 Hofstead Terrace 719 593-8989 fax Colorado Springs, CO 80907 719 210-7200 cell -----Original Message----- From: Edward A. Gardner To: T10 Reflector Date: Saturday, July 08, 2000 4:31 PM Subject: 00-279r0 Large SCSI Device Identifiers * From the T10 Reflector (t10@t10.org), posted by: * "Edward A. Gardner" * Document: T10/00-279r0 Date: July 8, 2000 To: T10 Committee Membership From: Edward A. Gardner, Ophidian Designs Subject: Large SCSI Device Identifiers The current SCSI standards specify that every SCSI device shall have a 64-bit device identifier. Quoting from SAM-2 revision 13, page 24: An Initiator Identifier is a field containing up to 64 bits that is a SCSI device identifier for the initiator device. A Target Identifier is a field containing up to 64 bits that is a SCSI device identifier for the target device. Unfortunately, 64-bits is no longer sufficient to identify a device with forthcoming protocols. For example, SVP will need to identify an initiato= r or target by a VI address. Many VI implementations construct a VI address from an IPv6 address plus a discriminator. Since an IPv6 address is 128 bits, this cannot fit within a 64-bit field. For another example consider the various proposals for SCSI over IP (Internet Protocol). At least one, iSCSI, is proposing to identify devices by URL strings. There are two approaches to dealing with this problem. One is to have eac= h protocol that encounters this problem define a protocol specific device identifier mapping. For example, SVP would define an SVP specific 64-bit = to VI address identifier mapping, with a protocol specific way for initiator= s to load and maintain the mapping table in targets. While that could be do= ne, I believe it is fundamentally a kludge and will lead to long-term support and compatibility problems. The other approach is to alter the relevant SCSI standards to allow much larger device identifiers. This document sketches how that might be done.= If the committee agrees that this is the proper approach, I will prepare detailed proposals for the affected standards for discussion at this fall= =92s T10 meetings. So far as I am aware, device identifier size only affects third-party operations, those where one target is asked to become a temporary initiat= or to access another target. The two important third-party operations are EXTENDED COPY and the third-party disk XOR operations. My impression is t= hat other third-party operations (e.g. third-party RESERVATION) are obsolete = and need not be supported. If anyone disagrees with these assumptions, I trus= t they will let us know. With these assumptions, the affected documents are SAM-2, SPC-3 and SBC-2. The changes I plan to propose to each are summarized below. SAM-2 The existing statements about the size of device identifiers would be removed. If SAM-2 says anything at all about device identifier size, it should say that it is protocol specific. SPC-3 The EXTENDED COPY command references a lengthy parameter list that contro= ls its operation. The parameter list contains a header, a list of target descriptors, a list of segment descriptors and an inline data area. Target descriptors are fixed length, 32-byte constructs. They include a 24-byte field available to identify the target and LUN. Various formats o= f target descriptors are defined that include Fibre Channel and parallel SC= SI device identifiers. Segment descriptors define a data transfer to be performed by the EXTENDE= D COPY command. They identify source and destination target devices with offsets into the list of target descriptors. Various formats define vario= us kinds of data transfer operations. One format copies data from the inline data area to the destination device, presumably for writing labels and ot= her control information. The inline data to be copied is identified by a 4-by= te offset and a 4-byte length. I plan to propose one or more new target descriptor formats. The actual target identifier would be stored in the inline data area. The new target descriptor format would contain a LUN (8 bytes), offset to the target identifier (4 bytes), length of the target identifier (2 or 4 bytes) and = a target identifier format code (2 bytes). Target identifier format codes would include values for unspecified, FC-VI, Infiniband, etc. The format codes serve much the same purpose as the current multiple formats of Fibr= e Channel target descriptors, one for port identifier, one for port WWN, on= e for both, etc. I am also tempted to allow the LUN=92s device identification VPD data to = be specified (e.g. the LUN=92s WWN) within the inline data area, analogous t= o the current Identification descriptor target descriptor format. I welcome comments on whether this would be useful. SBC-2 The third-party XOR commands identify the secondary target using a one by= te SECONDARY ADDRESS field. For parallel SCSI this is the target identifier. For Fibre Channel it specifies the low byte of the second target=92s port identifier; the other bytes of the port identifier are the same as the primary target=92s. The third-party XOR commands also contain a TABLE ADDRESS flag. When that flag is zero, the commands operate as described above. When that flag is one, SBC / SBC-2 specify: A TABLE ADDRESS bit of one indicates that the SECONDARY ADDRESS field contains a pointer to a look up table of ANSI X3.270 SAM compliant target identifiers. The look up table is reserved for future definition. I plan to propose that we define the format of the lookup table for SBC-2= . I plan to propose a table format that is a strict subset of that used by EXTENDED COPY. The header, target descriptors and inline data area would = be identical. Segment descriptors would be prohibited. I believe that all th= e target descriptor formats defined by EXTENDED COPY should be supported. I would like to have SBC-2 cross-reference SPC-3 for the table format, rather than redefining it within SBC-2. Is that legitimate? Does it requi= re any re-structuring of either standard? I believe the table should be volatile, reset on power-up or anything tha= t resets the LUN. There should be exactly one copy per LUN, shared by all initiators. The maximum table length is close to 256*32 or 8192 bytes plu= s the inline data, perhaps about 64Kbytes total. That=92s too large to fit easily into our mode page structure. We could segment the table, but I do= n=92t like that, I think it should be loaded by a single command, The other choices are a new command or to define one of the reserved bytes in MODE SELECT to indicate the type of mode information provided. I have no preference between these last two choices. I would also welcome other suggestions. Edward A. Gardner eag@ophidian.com Ophidian Designs 719 593-8866 voice 1262 Hofstead Terrace 719 593-8989 fax Colorado Springs, CO 80907 719 210-7200 cell * * For T10 Reflector information, send a message with * 'info t10' (no quotes) in the message body to majordomo@t10.org From owner-ips@ECE.cmu.edu Sun Jul 09 14:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22081; 9 Jul 2000 9:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA23692 for ips-outgoing; Sun, 9 Jul 2000 08:20:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA10969 for ; Sun, 9 Jul 2000 01:39:34 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA116558 for ; Sun, 9 Jul 2000 07:38:35 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA06676 for ; Sun, 9 Jul 2000 07:39:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256917.001F0643 ; Sun, 9 Jul 2000 07:38:52 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 9 Jul 2000 08:37:49 +0300 Subject: Re: multiple tcp connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk John, Correct - the only difference would be that the higher levels of software would have to be aware and handle it (error recovery, exceptions, balancing, increasing the number, decreasing the number). Those where the considerations that lead me to invent the session (channel group in our own slang) - as we think we can handle those well under the hood. Julo hufferd@us.ibm.com on 08/07/2000 21:34:22 Please respond to hufferd@us.ibm.com To: julian_satran%ibmil.RSCS@DEVM.DE.IBM.COM cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple tcp connections Julian, Since a session is only an iSCSI thing, your statements would be correct even if there were only one TCP connection per session, but several sessions on the same NIC. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 07/07/2000 11:18:48 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: multiple tcp connections Parallelism on end-points could be another reasons. A long time ago when running some benchmarks we achieved considerably better numbers with 4 sockets than with one. This type of result repeated itself in different contexts and with different stacks. That is probably due to some serialization that is inherent in the way stacks are built and API's are used (this is a long discussion subject). With SMPs the differences are even more striking. Julo Matt Wakeley on 08/07/2000 02:11:57 Please respond to Matt Wakeley To: Randy Haagens , Julian Satran/Haifa/IBM@IBMIL cc: Subject: multiple tcp connections Hi, I was asked the question: If there is only 1 MAC (available to the system), would there ever be more than 1 tcp connection per iSCSI session. I couldn't think of a good reason why there would be, but thought I'd ask you. The only reason I could think of using multiple tcp connections per iSCSI session was for port aggregation when there are multiple MACs available. Any comments? Thanks, -Matt From owner-ips@ECE.cmu.edu Sun Jul 09 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22759; 9 Jul 2000 14:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA27560 for ips-outgoing; Sun, 9 Jul 2000 12:51:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA27269 for ; Sun, 9 Jul 2000 12:32:14 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA75020 for ; Sun, 9 Jul 2000 18:30:58 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA145888 for ; Sun, 9 Jul 2000 18:31:42 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256917.005AC626 ; Sun, 9 Jul 2000 18:31:29 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 9 Jul 2000 19:30:27 +0300 Subject: working draft last comments before it goes out Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk A new version is on http://www.haifa.il.ibm.com/satran/iSCSI-Working-Draft0709.txt date July 9. It will be out there for a very short time (24 Hours?) before I send it (+ some editorial corrections some of which are already apparent) to the IETF. It includes the things we agreed already. Security is in (first pass). Length is changed to accommodate even the case of immediate and non immediate data (ugly). RDMA is out (not out but not reference as another doc). Audio/Video is out. Please have again a look at mapping and suggest wording. Julo From owner-ips@ECE.cmu.edu Sun Jul 09 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22885; 9 Jul 2000 15:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA28479 for ips-outgoing; Sun, 9 Jul 2000 13:54:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA28234 for ; Sun, 9 Jul 2000 13:40:26 -0400 (EDT) From: hufferd@us.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA22356 for ; Sun, 9 Jul 2000 13:37:56 -0400 Received: from d53mta08h.boulder.ibm.com (d53mta08h.boulder.ibm.com [9.99.142.8]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id LAA43990 for ; Sun, 9 Jul 2000 11:40:24 -0600 Received: by d53mta08h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256917.00611009 ; Sun, 9 Jul 2000 11:40:10 -0600 X-Lotus-FromDomain: IBMUS To: julian_satran@il.ibm.com cc: ips@ece.cmu.edu Message-ID: <87256917.0060B7D3.00@d53mta08h.boulder.ibm.com> Date: Sun, 9 Jul 2000 10:36:10 -0700 Subject: Re: working draft last comments before it goes out Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Since the RDMA stuff solves some basic and key problems, I would think that we need to refer to it some how. We are going all around the barn to solve a problem that is solved in an efficient manor with the RDMA option. I think this should be stated in some way. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 07/09/2000 09:30:27 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: working draft last comments before it goes out A new version is on http://www.haifa.il.ibm.com/satran/iSCSI-Working-Draft0709.txt date July 9. It will be out there for a very short time (24 Hours?) before I send it (+ some editorial corrections some of which are already apparent) to the IETF. It includes the things we agreed already. Security is in (first pass). Length is changed to accommodate even the case of immediate and non immediate data (ugly). RDMA is out (not out but not reference as another doc). Audio/Video is out. Please have again a look at mapping and suggest wording. Julo From owner-ips@ECE.cmu.edu Mon Jul 10 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24262; 9 Jul 2000 23:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA06498 for ips-outgoing; Sun, 9 Jul 2000 22:26:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA02549 for ; Sun, 9 Jul 2000 17:57:45 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id OAA18833; Sun, 9 Jul 2000 14:57:41 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id <3S8N65TL>; Sun, 9 Jul 2000 14:57:39 -0700 Message-ID: From: Jim McGrath To: "'VonStamwitz, Paul'" , "'David Robinson'" , Jim McGrath Cc: ips@ece.cmu.edu Subject: RE: Security Considerations Date: Sun, 9 Jul 2000 14:57:39 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I have no concerns over examining the protocol for security implications (the SCSI standard itself has a similar requirement with respect to testability). My concern was over jumping to the conclusion that a certain level of security is required. Once again, what concerns me is the application space. In the one I am most familiar with, where iSCSI can be used to complement or replace Fibre Channel/Infiniband in a storage area network environment, it is not at all clear that high levels of protocol security are required (security is typically provided by means outside of this layer of the protocol). What especially triggered my concern was mention of encrypted traffic, since that would appear to imply either dedicated hardware or software running on fast general purpose hardware. Levels of security is one possible solution to the issue of varying application environments, although the marketplace has a tendency to focus on a certain level of functionality for optimum price/performance products. This is an area that I would suggest people get T10 feedback on, since I think it is an area where the storage and network communities have differing histories and expectations. Jim -----Original Message----- From: VonStamwitz, Paul [mailto:paulv@corp.adaptec.com] Sent: Friday, July 07, 2000 5:16 PM To: 'David Robinson'; Jim McGrath Cc: ips@ece.cmu.edu Subject: RE: Security Considerations Jim McGrath wrote: > > My impression (which I admit could be quite mistaken) is that we may be on a > path of requiring a higher level of security for iSCSI than is warranted by > alternative protocols. Of course, enhancing security has its value, but > since SCSI starts off with essentially no security, iSCSI is a poor protocol > upon which to require lots of security. > I've quoted below portions of section 5 of "Guidelines for Writing RFC Text on Security Considerations". My email is was intended as a start to the "due diligence" process. While it is not a requirement that any given protocol or system be immune to all forms of attack, it is still necessary for authors to consider them. Part of the purpose of the Security Considerations section is to explain what attacks are out of scope and what counter- measures can be applied to defend against them. There should be a clear description of the kinds of threats on the described protocol or technology. This should be approached as an effort to perform "due diligence" in describing all known or foresee- able risks and threats to potential implementers and users. At least the following forms of attack MUST be considered: eavesdrop- ping, replay, message insertion, deletion, modification, and man-in- the-middle. Potential denial of service attacks MUST be identified as well. David Robinson wrote: > The benchmark that iSCSI will have to meet is what is being done > in the NFSv4 WG. Using the ONCRPC WG mechanisms based on GSSAPI > and RPCSEC_GSS they provide authentication, integrity, and privacy > using one of two manditory to implement mechanisms, Kerberos V5 > and LIPKEY. > > I suspect that if iSCSI doesn't address security to provide similar > capabilities it will not pass muster. Of course the security should > always be able to be negotiated to the desired levels. I agree on the pass muster comment. I also agree on the negotiating of security level. Hence, the 3 layer model of no security, connection-oriented authentication on logins (thanks to Brent on the distinction), and encryption. Luciano Dalle Oro pointed out that IPsec provides excellent data integrity in addition to security. Paul From owner-ips@ECE.cmu.edu Mon Jul 10 14:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26285; 10 Jul 2000 10:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA22806 for ips-outgoing; Mon, 10 Jul 2000 08:36:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA12095 for ; Mon, 10 Jul 2000 05:23:00 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id A2E223163 for ; Mon, 10 Jul 2000 05:22:59 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id CAA11664 for ; Mon, 10 Jul 2000 02:22:58 -0700 (PDT) Received: from agilent.com ([15.3.254.111]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA14C7; Mon, 10 Jul 2000 02:22:52 -0700 Message-ID: <3969957B.F5AB40AC@agilent.com> Date: Mon, 10 Jul 2000 02:21:00 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: working draft last comments before it goes out References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk comments on the July 9 draft: 2.2.2. Ordering and iSCSI numbering: "It is assumed that the SCSI layer frees all the resources allocated for the task execution; the result summary, just enough to rebuild the status PDU, will be kept by those iSCSI target implementations that support status recovery after connection failure." This statement in 2.2.2 conflicts with 2.2.4 and 4.1 The target cannot "deallocate" all resources and only keep the result summary around. It must be able to send all the data if necessary (as it says in 4.1). 3.3. SCSI Response: I'm not sure if it's required, but the new "Res_len" & "Sense_len" fields are not described. 3.10.4. Flags: New text about numbering iscsi data messages appears here. What's that got to do with flags? Also, I'm assuming that I->T transfers use CmdRN and I<-T transfers use StatRN? 3.13 Login: "RSA" is not defined. Does there need to be some mention of IPSec? There are so many "ways" to login, I think there might need to be a two step login process. The first step is more of a "query" that would indicate & request what options are supported, and the second step would actually perform the login. It's lookin good! Matt Wakeley From owner-ips@ECE.cmu.edu Mon Jul 10 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27273; 10 Jul 2000 14:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA03024 for ips-outgoing; Mon, 10 Jul 2000 13:39:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA00054 for ; Mon, 10 Jul 2000 12:25:16 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA56816; Mon, 10 Jul 2000 18:24:31 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA94748; Mon, 10 Jul 2000 18:24:43 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256918.005A2640 ; Mon, 10 Jul 2000 18:24:40 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: internet-drafts@ietf.org cc: ips@ece.cmu.edu Message-ID: Date: Mon, 10 Jul 2000 19:23:33 +0300 Subject: new version of iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk On behalf of a group of authors from several organizations I submit a revision of an IETF-draft for immediate publication. It specifies iSCSI - a SCSI Over TCP protocol and it completely replaces It contains many changes. The draft is http accessible at: http://www.haifa.il.ibm.com/satran/draft-satran-iscsi-01.txt Julian Satran - IBM Research Laboratory in Haifa From owner-ips@ECE.cmu.edu Mon Jul 10 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27271; 10 Jul 2000 14:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA03008 for ips-outgoing; Mon, 10 Jul 2000 13:38:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA29527 for ; Mon, 10 Jul 2000 12:11:43 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA150726; Mon, 10 Jul 2000 18:10:41 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA257624; Mon, 10 Jul 2000 18:11:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256918.0058E89D ; Mon, 10 Jul 2000 18:11:07 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ietf-drafts@ietf.org cc: ips@ece.cmu.edu Message-ID: Date: Mon, 10 Jul 2000 19:04:00 +0300 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk On behalf of a group of authors from several organizations I submit a revision of an IETF-draft for immediate publication. It specifies iSCSI - a SCSI Over TCP protocol and it completely replaces It contains many changes. The draft is http accessible at: http://www.haifa.il.ibm.com/satran/draft-satran-iscsi-01.txt Julian Satran - IBM Research Laboratory in Haifa From owner-ips@ECE.cmu.edu Mon Jul 10 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27278; 10 Jul 2000 14:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA03075 for ips-outgoing; Mon, 10 Jul 2000 13:40:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from c014.sfo.cp.net (c014-h023.c014.sfo.cp.net [209.228.12.87]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id NAA02893 for ; Mon, 10 Jul 2000 13:36:02 -0400 (EDT) Received: (cpmta 8994 invoked from network); 10 Jul 2000 10:35:30 -0700 Received: from 3ff8c878.dsl.flashcom.net (HELO gis.net) (63.248.200.120) by smtp.flashcom.net with SMTP; 10 Jul 2000 10:35:30 -0700 X-Sent: 10 Jul 2000 17:35:30 GMT Message-ID: <396A08A0.8E0FA97C@gis.net> Date: Mon, 10 Jul 2000 13:32:16 -0400 From: Bill Main Reply-To: bill.main@bigfoot.com X-Mailer: Mozilla 4.5 [en]C-CCK-MCD compaq (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ipSCSI list Subject: Question on TCP hardware up lift Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Is there a list any where of companies currently in the TCP hardware busines? Regards, Bill Main From owner-ips@ECE.cmu.edu Mon Jul 10 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27267; 10 Jul 2000 14:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA03054 for ips-outgoing; Mon, 10 Jul 2000 13:39:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA02697 for ; Mon, 10 Jul 2000 13:31:51 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 400D3837F3; Mon, 10 Jul 2000 12:29:25 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 34Y8PWGT; Mon, 10 Jul 2000 12:37:53 -0500 Message-ID: <396A08C5.B8E1B624@nuspeed.com> Date: Mon, 10 Jul 2000 12:32:53 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: working draft last comments before it goes out References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian- Here are some comments on logins and on connection recovery. The draft is looking pretty good so far. -- Mark Login Security- Looking at the latest draft, I noticed that section 3.13.5 (pg 37) provides for a clear text login, while section 6.2.1.1 (pg 52) assumes that long-lived passwords are never sent in the clear. These appear contradictory. I also do not see any method to do digest authentication; all of the strong authentication schemes are based on public key encryption, which I think will require some sort of key distribution or certificate system to be in place. Anyway, I would like to see the clear text login replaced with a digest login, based on MD5 and a nonce, like HTTP/1.1 uses. For those who wish to base authentication on knowing a user name and password, this will be an easier scheme to configure and deploy. This method of hashing passwords is so easy to do, and requires no additional configuration to be done, that it could simply replace the clear text logins. I had proposed the text fields to accomplish this in a previous posting; I will be happy to re-send it if anyone is interested. Login Version- The Version field (either a text field or part of the login command header) still needs to be added. Asynchronous Events- (this one might not make it in this draft) In conjunction with the connection recovery discussion, I would still like to add an asynchronous event type indicating that the target is planning to close the connection: 3.6.2. SCSI Event Indicator 5 Connection will be closed by the target Two reserved fields would be used to communicate: - MaxUpTime - the number of seconds the target intends to keep the connection alive. - MinHoldTime - the number of seconds the initiator should wait before establishing a new connection. This feature would allow a target to indicate that it wishes a connection to be re-established, in the case of a controlled shutdown, failover, or other operation on the target itself. The initiator would perform normal connection recovery, and would not necessarily have to lose commands and data. An open question- I've documented this as an advisory event, since the initiator has to deal with "hard" shutdown or failover of a target anyway. Would it be cleaner to allow the client to acknowledge that it is finished with the connection, or have the client close the connection within a certain time period? julian_satran@il.ibm.com wrote: > > A new version is on > > http://www.haifa.il.ibm.com/satran/iSCSI-Working-Draft0709.txt > > date July 9. > > It will be out there for a very short time (24 Hours?) before I send it (+ > some editorial > corrections some of which are already apparent) to the IETF. > > It includes the things we agreed already. > > Security is in (first pass). > > Length is changed to accommodate even the case of immediate and > non immediate data (ugly). > > RDMA is out (not out but not reference as another doc). > > Audio/Video is out. > > Please have again a look at mapping and suggest wording. > > Julo -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Mon Jul 10 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27406; 10 Jul 2000 15:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA04440 for ips-outgoing; Mon, 10 Jul 2000 14:19:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA04066 for ; Mon, 10 Jul 2000 14:09:18 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel3.hp.com (Postfix) with ESMTP id B1E5FB9F; Mon, 10 Jul 2000 11:05:11 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id LAA23846; Mon, 10 Jul 2000 11:05:05 -0700 (PDT) X-OpenMail-Hops: 1 Date: Mon, 10 Jul 2000 11:04:54 -0700 Message-Id: Subject: RE: Re: multiple tcp connections MIME-Version: 1.0 To: ips@ece.cmu.edu, julian_satran@il.ibm.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't think we want to invent sessions because some implementation could not achieve link speed using a single connection. First of all, we have demonstrated link speed on high speed links with a single TCP connection. There are two reasons I can come up with on short notice that may prevent one from acheiving this number. One is that the TCP window size may not be large enough. With the large window size option, this should not be an issue. The other is poor design - and that is something that should be solved through better design. There may be other reasons why this is not achievable that I don't know. -- When we get to adding all the other objectives, that is another can of worms. We should at least spell out the requirements we are placing on a session in the spec. Somesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Saturday, July 08, 2000 10:38 PM To: ips@ece.cmu.edu Subject: FW: Re: multiple tcp connections John, Correct - the only difference would be that the higher levels of software would have to be aware and handle it (error recovery, exceptions, balancing, increasing the number, decreasing the number). Those where the considerations that lead me to invent the session (channel group in our own slang) - as we think we can handle those well under the hood. Julo hufferd@us.ibm.com on 08/07/2000 21:34:22 Please respond to hufferd@us.ibm.com To: julian_satran%ibmil.RSCS@DEVM.DE.IBM.COM cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple tcp connections Julian, Since a session is only an iSCSI thing, your statements would be correct even if there were only one TCP connection per session, but several sessions on the same NIC. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 07/07/2000 11:18:48 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: multiple tcp connections Parallelism on end-points could be another reasons. A long time ago when running some benchmarks we achieved considerably better numbers with 4 sockets than with one. This type of result repeated itself in different contexts and with different stacks. That is probably due to some serialization that is inherent in the way stacks are built and API's are used (this is a long discussion subject). With SMPs the differences are even more striking. Julo Matt Wakeley on 08/07/2000 02:11:57 Please respond to Matt Wakeley To: Randy Haagens , Julian Satran/Haifa/IBM@IBMIL cc: Subject: multiple tcp connections Hi, I was asked the question: If there is only 1 MAC (available to the system), would there ever be more than 1 tcp connection per iSCSI session. I couldn't think of a good reason why there would be, but thought I'd ask you. The only reason I could think of using multiple tcp connections per iSCSI session was for port aggregation when there are multiple MACs available. Any comments? Thanks, -Matt From owner-ips@ECE.cmu.edu Mon Jul 10 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27413; 10 Jul 2000 15:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA04423 for ips-outgoing; Mon, 10 Jul 2000 14:18:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA03752 for ; Mon, 10 Jul 2000 13:59:52 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 702F6837F3 for ; Mon, 10 Jul 2000 12:57:26 -0500 (CDT) Received: from nuspeed.com (marvin.warner.nuspeed.com [10.1.0.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 34Y8PWHQ; Mon, 10 Jul 2000 13:05:55 -0500 Message-ID: <396A0F57.2F6DF0A1@nuspeed.com> Date: Mon, 10 Jul 2000 13:00:55 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-32smp i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI MIB draft Content-Type: multipart/mixed; boundary="------------6139FF0C7285D1D87E83A475" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------6139FF0C7285D1D87E83A475 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Here's a MIB draft that we intend to send to the IETF by Wednesday. I apologize for sending this out late, but I suspect that the MIB can't really solidify until the protocol is done. At any rate, I will try to incorporate any comments received before I send this on Wednesday. -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 --------------6139FF0C7285D1D87E83A475 Content-Type: text/plain; charset=us-ascii; name="iSCSI.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="iSCSI.txt" Internet Draft Mark Bakke NuSpeed, Inc Expires January 2001 Jim Muchow NuSpeed, Inc July 2000 Definitions of Managed Objects for SCSI over TCP 1. Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference mate- rial or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 1.1. Copyright Notice Copyright (C) The Internet Society (2000). All Rights Reserved. 2. Abstract This memo defines a portion of the Management Information Base (MIB) for use with network management protocols in TCP/IP based internets. In particular it defines objects for managing a client using the SCSI over TCP (aka iSCSI) protocol. Bakke, Muchow [Page 1] Internet Draft iSCSI MIB July 2000 3. The SNMP Management Framework The SNMP Management Framework presently consists of five major compo- nents: o An overall architecture, described in RFC 2571 [RFC2571]. o Mechanisms for describing and naming objects and events for the purpose of management. The first version of this Structure of Management Information (SMI) is called SMIv1 and described in STD 16, RFC 1155 [RFC1155], STD 16, RFC 1212 [RFC1212] and RFC 1215 [RFC1215]. The second version, called SMIv2, is described in STD 58, RFC 2578 [RFC2578], STD 58, RFC 2579 [RFC2579] and STD 58, RFC 2580 [RFC2580]. o Message protocols for transferring management information. The first version of the SNMP message protocol is called SNMPv1 and described in STD 15, RFC 1157 [RFC1157]. A second version of the SNMP message protocol, which is not an Internet standards track protocol, is called SNMPv2c and described in RFC 1901 [RFC1901] and RFC 1906 [RFC1906]. The third version of the mes- sage protocol is called SNMPv3 and described in RFC 1906 [RFC1906], RFC 2572 [RFC2572] and RFC 2574 [RFC2574]. o Protocol operations for accessing management information. The first set of protocol operations and associated PDU formats is described in STD 15, RFC 1157 [RFC1157]. A second set of proto- col operations and associated PDU formats is described in RFC 1905 [RFC1905]. o A set of fundamental applications described in RFC 2573 [RFC2573] and the view-based access control mechanism described in RFC 2575 [RFC2575]. A more detailed introduction to the current SNMP Management Framework can be found in RFC 2570 [RFC2570]. Managed objects are accessed via a virtual information store, termed the Management Information Base or MIB. Objects in the MIB are defined using the mechanisms defined in the SMI. This memo specifies a MIB module that is compliant to the SMIv2. A MIB conforming to the SMIv1 can be produced through the appropriate translations. The resulting translated MIB must be semantically equivalent, except where objects or events are omitted because no translation is possible (use of Counter64). Some machine readable information in SMIv2 will be converted into textual descriptions in SMIv1 during the translation process. However, this loss of machine Bakke, Muchow [Page 2] Internet Draft iSCSI MIB July 2000 readable information is not considered to change the semantics of the MIB. 4. Discussion This structure attempts to put together a starting point for an iSCSI (SCSI/TCP) MIB. Although derived from NuSpeed's enterprise MIB, it is not intended to reflect or depend on NuSpeed propietary MIB objects. This structure is intended to satisfy fault, configuration, performance, accounting, and security management for an iSCSI implementation. It is currently structured around objects such as targets, luns, ports, and connections, but we need to keep the other categories in mind. We have looked at using a fault/config/perf/acct/security structure for the MIB, but this would result in a larger number of tables. Any opinions on this? This structure basically covers the server side of iSCSI. We will prob- ably need to do something with the client side as well. Perhaps adding an initiator table would help, but I'm not sure that's enough. Any thoughts on whether managing the client side via SNMP is useful, and what the requirements might be? Below the iscsimgmt object, there are basically six tables: iscsiTable, main table, list iSCSI instances known. Normally, there might be just one instance in this table; however, it is added as a table to correct a few shortcomings of other routing MIBs, in that most of these do not provide for multiple virtual instances. For example, many IP routing products now allow multiple virtual routers. iSCSI should allow for the same idea. iscsiPortTable, sub-table of iscsiTable: a list of server ports on which the iSCSI server can listen for client connections. targetTable, sub-table of iscsiTable: a list of iSCSI targets. It includes per-target configuration parameters, as well as statistics and login information. iscsiConnectionTable, sub-table of iscsiTargetTable: a list of current connections to the server. Entries in this table are fairly transient; it might be a good debugging tool, but should not be used for accounting or anything serious. This should probably be optional. iscsiLunTable, sub-table of iscsiTargetTable: a list of iSCSI LUNs, indexed by target name and LUN. It is used to gain finer-granularity statistics. Bakke, Muchow [Page 3] Internet Draft iSCSI MIB July 2000 iscsiInitiatorTable is a list of iSCSI initiators. It includes mainly the initiator's name and IP address, along with statistics. This should primarily be useful for troubleshooting errant client implementations, or tracking usage by initiator. A fairly standard list of statistics have been added to each of the above objects. At first glance, these appear to be redundant, and could be computed from lower-level objects. However, some of the objects are transient (connections can come and go, and initiators that have discon- nected may not be seen again, LUNs could be added and removed on a disk array), so we felt that duplicating them was not too high of a price for the availability of good information. There are no doubt still some important statistics and other information still missing; the authors would welcome any comments. Note: We still need to add counters for the new messages, NOOP and MAP. 5. MIB Definitions ISCSI-MIB DEFINITIONS ::= BEGIN IMPORTS MODULE-IDENTITY, OBJECT-TYPE, Unsigned32, IpAddress, Counter32, Counter64, experimental -- mib-2 FROM SNMPv2-SMI DisplayString -- DisplayString, RowStatus FROM SNMPv2-TC InterfaceIndex FROM IF-MIB MODULE-COMPLIANCE, OBJECT-GROUP -- MODULE-COMPLIANCE, OBJECT-GROUP, NOTIFICATION-GROUP FROM SNMPv2-CONF ; iscsiModule MODULE-IDENTITY LAST-UPDATED "0007010000Z" ORGANIZATION "NuSpeed, Inc" CONTACT-INFO " Mark Bakke Bakke, Muchow [Page 4] Internet Draft iSCSI MIB July 2000 Postal: NuSpeed, Inc 6450 Wedgwood Road Maple Grove, MN USA 55311 Tel: +1 763-398-1000 Fax: +1 763-398-1001 E-mail: mark.bakke@nuspeed.com Jim Muchow Postal: NuSpeed, Inc 6450 Wedgwood Road Maple Grove, MN USA 55311 Tel: +1 763-398-1000 Fax: +1 763-398-1001 E-mail: jim.muchow@nuspeed.com" DESCRIPTION "." ::= { experimental 1 } iscsiMgmt OBJECT IDENTIFIER ::= { iscsiModule 1 } iscsiObjects OBJECT IDENTIFIER ::= { iscsiMgmt 1 } iscsiNotifications OBJECT IDENTIFIER ::= { iscsiMgmt 2 } iscsiConformance OBJECT IDENTIFIER ::= { iscsiMgmt 3 } -- Textual Conventions -- IScsiLunId ::= OCTET STRING (SIZE(8)) IScsiConnectState ::= INTEGER { unknown (1), notLoggedIn (2), fullPhase (3), loginDenied (4), terminating (5) } IScsiAuthType ::= INTEGER { unknown (1), none (2), digest (3) -- Work to do here. Should this be a string, instead of an enum, -- for better flexibility? Bakke, Muchow [Page 5] Internet Draft iSCSI MIB July 2000 } IScsiCheckType ::= INTEGER { unknown (1), none (2), -- No extra checksums checksum16 (3), -- 1s complement crc32 (4), -- 32-bit CRC md5 (5) -- MD5 hash } -- We need to get an experimental number for this MIB. iscsiNumber OBJECT-TYPE SYNTAX Unsigned32 MAX-ACCESS read-only STATUS current DESCRIPTION "The number of iSCSI instances present on the system." ::= { iscsiObjects 1 } iscsiTable OBJECT-TYPE SYNTAX SEQUENCE OF IscsiEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." ::= { iscsiObjects 2 } iscsiEntry OBJECT-TYPE SYNTAX IscsiEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." INDEX { iscsiIndex } ::= { iscsiTable 1 } IscsiEntry ::= SEQUENCE { iscsiIndex Unsigned32, iscsiName DisplayString, iscsiDescription DisplayString, rxMsgs Counter32, rxPingCmdMsgs Counter32, rxScsiCmdMsgs Counter32, rxScsiCmdMsgsRejected Counter32, rxTaskMgtMsgs Counter32, rxTaskMgtMsgsRejected Counter32, Bakke, Muchow [Page 6] Internet Draft iSCSI MIB July 2000 rxLoginCmdMsgs Counter32, rxLoginCmdMsgsRejected Counter32, rxTextCmdMsgs Counter32, rxTextCmdMsgsRejected Counter32, rxInvalidMsgs Counter32, rxDataMsgs Counter32, rxDataBytes Counter64, rxDataMsgsDiscarded Counter32, rxDataBytesDiscarded Counter32, txMsgs Counter32, txPingRspMsgs Counter32, txScsiRspMsgs Counter32, txScsiTaskMgtRspMsgs Counter32, txLoginRspMsgs Counter32, txTextRspMsgs Counter32, txRttMsgs Counter32, txAsyncEvtMsgs Counter32, txInvalidOpcodeMsgs Counter32, txDataMsgs Counter32, txDataBytes Counter64, connectionsCreated Counter32, connectionsRefused Counter32, connectionsTerminated Counter32, connectionsErrored Counter32 } iscsiIndex OBJECT-TYPE SYNTAX Unsigned32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 1 } iscsiName OBJECT-TYPE SYNTAX DisplayString MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 2 } iscsiDescription OBJECT-TYPE SYNTAX DisplayString MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 3 } rxMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current Bakke, Muchow [Page 7] Internet Draft iSCSI MIB July 2000 DESCRIPTION "." ::= { iscsiEntry 4 } rxPingCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 5 } rxScsiCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 6 } rxScsiCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 7 } rxTaskMgtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 8 } rxTaskMgtMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 9 } rxLoginCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 10 } rxLoginCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 11 } rxTextCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current Bakke, Muchow [Page 8] Internet Draft iSCSI MIB July 2000 DESCRIPTION "." ::= { iscsiEntry 12 } rxTextCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 13 } rxInvalidMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 14 } rxDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 15 } rxDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 16 } rxDataMsgsDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 17 } rxDataBytesDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 18 } txMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 19 } txPingRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current Bakke, Muchow [Page 9] Internet Draft iSCSI MIB July 2000 DESCRIPTION "." ::= { iscsiEntry 20 } txScsiRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 21 } txScsiTaskMgtRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 22 } txLoginRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 23 } txTextRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 24 } txRttMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 25 } txAsyncEvtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 26 } txInvalidOpcodeMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 27 } txDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current Bakke, Muchow [Page 10] Internet Draft iSCSI MIB July 2000 DESCRIPTION "." ::= { iscsiEntry 28 } txDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 29 } connectionsCreated OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 30 } connectionsRefused OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 31 } connectionsTerminated OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 32 } connectionsErrored OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiEntry 33 } ---------------------------------------------------------------------- iscsiPortTable OBJECT-TYPE SYNTAX SEQUENCE OF IscsiPortEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." ::= { iscsiObjects 3 } iscsiPortEntry OBJECT-TYPE SYNTAX IscsiPortEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." Bakke, Muchow [Page 11] Internet Draft iSCSI MIB July 2000 INDEX { iscsiIndex, portIpAddress, portTcpPort } ::= { iscsiPortTable 1 } IscsiPortEntry ::= SEQUENCE { portIpAddress IpAddress, portTcpPort Unsigned32, portIfIndex InterfaceIndex, hostName DisplayString, rxPortMsgs Counter32, rxPortPingCmdMsgs Counter32, rxPortScsiCmdMsgs Counter32, rxPortScsiCmdMsgsRejected Counter32, rxPortTaskMgtMsgs Counter32, rxPortTaskMgtMsgsRejected Counter32, rxPortLoginCmdMsgs Counter32, rxPortLoginCmdMsgsRejected Counter32, rxPortTextCmdMsgs Counter32, rxPortTextCmdMsgsRejected Counter32, rxPortInvalidMsgs Counter32, rxPortDataMsgs Counter32, rxPortDataBytes Counter64, rxPortDataMsgsDiscarded Counter32, rxPortDataBytesDiscarded Counter32, txPortMsgs Counter32, txPortPingRspMsgs Counter32, txPortScsiRspMsgs Counter32, txPortScsiTaskMgtRspMsgs Counter32, txPortLoginRspMsgs Counter32, txPortTextRspMsgs Counter32, txPortRttMsgs Counter32, txPortAsyncEvtMsgs Counter32, txPortInvalidOpcodeMsgs Counter32, txPortDataMsgs Counter32, txPortDataBytes Counter64 } portIpAddress OBJECT-TYPE SYNTAX IpAddress MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 1 } portTcpPort OBJECT-TYPE SYNTAX Unsigned32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 12] Internet Draft iSCSI MIB July 2000 ::= { iscsiPortEntry 2 } portIfIndex OBJECT-TYPE SYNTAX InterfaceIndex MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 3 } hostName OBJECT-TYPE SYNTAX DisplayString MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 4 } rxPortMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 5 } rxPortPingCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 6 } rxPortScsiCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 7 } rxPortScsiCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 8 } rxPortTaskMgtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 9 } rxPortTaskMgtMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 13] Internet Draft iSCSI MIB July 2000 ::= { iscsiPortEntry 10 } rxPortLoginCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 11 } rxPortLoginCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 12 } rxPortTextCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 13 } rxPortTextCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 14 } rxPortInvalidMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 15 } rxPortDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 16 } rxPortDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 17 } rxPortDataMsgsDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 14] Internet Draft iSCSI MIB July 2000 ::= { iscsiPortEntry 18 } rxPortDataBytesDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 19 } txPortMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 20 } txPortPingRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 21 } txPortScsiRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 22 } txPortScsiTaskMgtRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 23 } txPortLoginRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 24 } txPortTextRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 25 } txPortRttMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 15] Internet Draft iSCSI MIB July 2000 ::= { iscsiPortEntry 26 } txPortAsyncEvtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 27 } txPortInvalidOpcodeMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 28 } txPortDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 29 } txPortDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiPortEntry 30 } ---------------------------------------------------------------------- iscsiTargetTable OBJECT-TYPE SYNTAX SEQUENCE OF IscsiTargetEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." ::= { iscsiObjects 4 } iscsiTargetEntry OBJECT-TYPE SYNTAX IscsiTargetEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." INDEX { iscsiIndex, targetIndex } ::= { iscsiTargetTable 1 } IscsiTargetEntry ::= SEQUENCE { targetIndex Unsigned32, targetName DisplayString, Bakke, Muchow [Page 16] Internet Draft iSCSI MIB July 2000 rxTargetMsgs Counter32, rxTargetScsiCmdMsgs Counter32, rxTargetScsiCmdMsgsRejected Counter32, rxTargetTaskMgtMsgs Counter32, rxTargetTaskMgtMsgsRejected Counter32, rxTargetLoginCmdMsgs Counter32, rxTargetLoginCmdMsgsRejected Counter32, rxTargetDataMsgs Counter32, rxTargetDataBytes Counter64, rxTargetDataMsgsDiscarded Counter32, rxTargetDataBytesDiscarded Counter32, txTargetMsgs Counter32, txTargetScsiRspMsgs Counter32, txTargetScsiTaskMgtRspMsgs Counter32, txTargetLoginRspMsgs Counter32, txTargetRttMsgs Counter32, txTargetAsyncEvtMsgs Counter32, txTargetDataMsgs Counter32, txTargetDataBytes Counter64 } targetIndex OBJECT-TYPE SYNTAX Unsigned32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 1 } targetName OBJECT-TYPE SYNTAX DisplayString MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 2 } rxTargetMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 3 } rxTargetScsiCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 4 } rxTargetScsiCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only Bakke, Muchow [Page 17] Internet Draft iSCSI MIB July 2000 STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 5 } rxTargetTaskMgtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 6 } rxTargetTaskMgtMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 7 } rxTargetLoginCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 8 } rxTargetLoginCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 9 } rxTargetDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 10 } rxTargetDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 11 } rxTargetDataMsgsDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 12 } rxTargetDataBytesDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only Bakke, Muchow [Page 18] Internet Draft iSCSI MIB July 2000 STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 13 } txTargetMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 14 } txTargetScsiRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 15 } txTargetScsiTaskMgtRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 16 } txTargetLoginRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 17 } txTargetRttMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 18 } txTargetAsyncEvtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 19 } txTargetDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 20 } txTargetDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only Bakke, Muchow [Page 19] Internet Draft iSCSI MIB July 2000 STATUS current DESCRIPTION "." ::= { iscsiTargetEntry 21 } ---------------------------------------------------------------------- iscsiLunTable OBJECT-TYPE SYNTAX SEQUENCE OF IscsiLunEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." ::= { iscsiObjects 5 } iscsiLunEntry OBJECT-TYPE SYNTAX IscsiLunEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." INDEX { iscsiIndex, targetIndex, lunId } ::= { iscsiLunTable 1 } IscsiLunEntry ::= SEQUENCE { lunId Unsigned32, rxLunMsgs Counter32, rxLunScsiCmdMsgs Counter32, rxLunScsiCmdMsgsRejected Counter32, rxLunTaskMgtMsgs Counter32, rxLunTaskMgtMsgsRejected Counter32, rxLunDataMsgs Counter32, rxLunDataBytes Counter64, rxLunDataMsgsDiscarded Counter32, rxLunDataBytesDiscarded Counter32, txLunMsgs Counter32, txLunScsiRspMsgs Counter32, txLunScsiTaskMgtRspMsgs Counter32, txLunAsyncEvtMsgs Counter32, txLunDataMsgs Counter32, txLunDataBytes Counter64, lunInquiryString DisplayString } lunId OBJECT-TYPE SYNTAX Unsigned32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 20] Internet Draft iSCSI MIB July 2000 ::= { iscsiLunEntry 1 } rxLunMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 2 } rxLunScsiCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 3 } rxLunScsiCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 4 } rxLunTaskMgtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 5 } rxLunTaskMgtMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 6 } rxLunDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 7 } rxLunDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 8 } rxLunDataMsgsDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 21] Internet Draft iSCSI MIB July 2000 ::= { iscsiLunEntry 9 } rxLunDataBytesDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 10 } txLunMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 11 } txLunScsiRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 12 } txLunScsiTaskMgtRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 13 } txLunAsyncEvtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 14 } txLunDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 15 } txLunDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiLunEntry 16 } lunInquiryString OBJECT-TYPE SYNTAX DisplayString MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 22] Internet Draft iSCSI MIB July 2000 ::= { iscsiLunEntry 17 } ---------------------------------------------------------------------- iscsiInitiatorTable OBJECT-TYPE SYNTAX SEQUENCE OF IscsiInitiatorEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." ::= { iscsiObjects 6 } iscsiInitiatorEntry OBJECT-TYPE SYNTAX IscsiInitiatorEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." INDEX { initiatorIndex } ::= { iscsiInitiatorTable 1 } IscsiInitiatorEntry ::= SEQUENCE { initiatorIndex Unsigned32, initiatorName DisplayString, status INTEGER, connections Unsigned32, rxInitiatorMsgs Counter32, rxInitiatorPingCmdMsgs Counter32, rxInitiatorScsiCmdMsgs Counter32, rxInitiatorScsiCmdMsgsRejected Counter32, rxInitiatorTaskMgtMsgs Counter32, rxInitiatorTaskMgtMsgsRejected Counter32, rxInitiatorLoginCmdMsgs Counter32, rxInitiatorLoginCmdMsgsRejected Counter32, rxInitiatorTextCmdMsgs Counter32, rxInitiatorTextCmdMsgsRejected Counter32, rxInitiatorInvalidMsgs Counter32, rxInitiatorDataMsgs Counter32, rxInitiatorDataBytes Counter64, rxInitiatorDataMsgsDiscarded Counter32, rxInitiatorDataBytesDiscarded Counter32, txInitiatorMsgs Counter32, txInitiatorPingRspMsgs Counter32, txInitiatorScsiRspMsgs Counter32, txInitiatorScsiTaskMgtRspMsgs Counter32, txInitiatorLoginRspMsgs Counter32, txInitiatorTextRspMsgs Counter32, txInitiatorRttMsgs Counter32, Bakke, Muchow [Page 23] Internet Draft iSCSI MIB July 2000 txInitiatorAsyncEvtMsgs Counter32, txInitiatorInvalidOpcodeMsgs Counter32, txInitiatorDataMsgs Counter32, txInitiatorDataBytes Counter64 } initiatorIndex OBJECT-TYPE SYNTAX Unsigned32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 1 } initiatorName OBJECT-TYPE SYNTAX DisplayString MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 2 } status OBJECT-TYPE SYNTAX INTEGER { someStatus (1), someOtherStatus (2) } MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 3 } connections OBJECT-TYPE SYNTAX Unsigned32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 4 } rxInitiatorMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 5 } rxInitiatorPingCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 6 } rxInitiatorScsiCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 7 } Bakke, Muchow [Page 24] Internet Draft iSCSI MIB July 2000 rxInitiatorScsiCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 8} rxInitiatorTaskMgtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 9 } rxInitiatorTaskMgtMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 10 } rxInitiatorLoginCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 11 } rxInitiatorLoginCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 12 } rxInitiatorTextCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 13 } rxInitiatorTextCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 14 } rxInitiatorInvalidMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 15 } Bakke, Muchow [Page 25] Internet Draft iSCSI MIB July 2000 rxInitiatorDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 16 } rxInitiatorDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 17 } rxInitiatorDataMsgsDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 18 } rxInitiatorDataBytesDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 19 } txInitiatorMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 20 } txInitiatorPingRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 21 } txInitiatorScsiRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 22 } txInitiatorScsiTaskMgtRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 23 } Bakke, Muchow [Page 26] Internet Draft iSCSI MIB July 2000 txInitiatorLoginRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 24 } txInitiatorTextRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 25 } txInitiatorRttMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 26 } txInitiatorAsyncEvtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 27 } txInitiatorInvalidOpcodeMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 28 } txInitiatorDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 29 } txInitiatorDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiInitiatorEntry 30 } ---------------------------------------------------------------------- iscsiCxnTable OBJECT-TYPE SYNTAX SEQUENCE OF IscsiCxnEntry MAX-ACCESS not-accessible Bakke, Muchow [Page 27] Internet Draft iSCSI MIB July 2000 STATUS current DESCRIPTION "." ::= { iscsiObjects 7 } iscsiCxnEntry OBJECT-TYPE SYNTAX IscsiCxnEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "." INDEX { initiatorIndex, cxnIpAddress, cxnTcpPort } ::= { iscsiCxnTable 1 } IscsiCxnEntry ::= SEQUENCE { cxnIpAddress IpAddress, cxnTcpPort Unsigned32, isid Unsigned32, -- Session ID (0..65535) tsid Unsigned32, -- Session ID (0..65535) state IScsiConnectState, allowNoRtt INTEGER, -- (0 == No; 1 == Yes) initiator DisplayString, authenticator DisplayString, authType IScsiAuthType, msgCheckType IScsiCheckType, dataCheckType IScsiCheckType, rxCxnMsgs Counter32, rxCxnPingCmdMsgs Counter32, rxCxnScsiCmdMsgs Counter32, rxCxnScsiCmdMsgsRejected Counter32, rxCxnTaskMgtMsgs Counter32, rxCxnTaskMgtMsgsRejected Counter32, rxCxnLoginCmdMsgs Counter32, rxCxnLoginCmdMsgsRejected Counter32, rxCxnTextCmdMsgs Counter32, rxCxnTextCmdMsgsRejected Counter32, rxCxnInvalidMsgs Counter32, rxCxnDataMsgs Counter32, rxCxnDataBytes Counter64, rxCxnDataMsgsDiscarded Counter32, rxCxnDataBytesDiscarded Counter32, txCxnMsgs Counter32, txCxnPingRspMsgs Counter32, txCxnScsiRspMsgs Counter32, txCxnScsiTaskMgtRspMsgs Counter32, txCxnLoginRspMsgs Counter32, txCxnTextRspMsgs Counter32, Bakke, Muchow [Page 28] Internet Draft iSCSI MIB July 2000 txCxnRttMsgs Counter32, txCxnAsyncEvtMsgs Counter32, txCxnInvalidOpcodeMsgs Counter32, txCxnDataMsgs Counter32, txCxnDataBytes Counter64 } cxnIpAddress OBJECT-TYPE SYNTAX IpAddress MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 1 } cxnTcpPort OBJECT-TYPE SYNTAX Unsigned32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 2 } isid OBJECT-TYPE SYNTAX Unsigned32 (0..65535) MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 3 } tsid OBJECT-TYPE SYNTAX Unsigned32 (0..65535) MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 4 } state OBJECT-TYPE SYNTAX IScsiConnectState MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 5 } allowNoRtt OBJECT-TYPE SYNTAX INTEGER { no (1), yes (2) } MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 6 } initiator OBJECT-TYPE SYNTAX DisplayString MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 29] Internet Draft iSCSI MIB July 2000 ::= { iscsiCxnEntry 7 } authenticator OBJECT-TYPE SYNTAX DisplayString MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 8 } authType OBJECT-TYPE SYNTAX IScsiAuthType MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 9 } msgCheckType OBJECT-TYPE SYNTAX IScsiCheckType MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 10 } dataCheckType OBJECT-TYPE SYNTAX IScsiCheckType MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 11 } rxCxnMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 12 } rxCxnPingCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 13 } rxCxnScsiCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 14 } rxCxnScsiCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 30] Internet Draft iSCSI MIB July 2000 ::= { iscsiCxnEntry 15 } rxCxnTaskMgtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 16 } rxCxnTaskMgtMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 17 } rxCxnLoginCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 18 } rxCxnLoginCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 19 } rxCxnTextCmdMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 20 } rxCxnTextCmdMsgsRejected OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 21 } rxCxnInvalidMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 22 } rxCxnDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 31] Internet Draft iSCSI MIB July 2000 ::= { iscsiCxnEntry 23 } rxCxnDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 24 } rxCxnDataMsgsDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 25 } rxCxnDataBytesDiscarded OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 26 } txCxnMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 27 } txCxnPingRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 28 } txCxnScsiRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 29 } txCxnScsiTaskMgtRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 30 } txCxnLoginRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." Bakke, Muchow [Page 32] Internet Draft iSCSI MIB July 2000 ::= { iscsiCxnEntry 31 } txCxnTextRspMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 32 } txCxnRttMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 33 } txCxnAsyncEvtMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 34 } txCxnInvalidOpcodeMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 35 } txCxnDataMsgs OBJECT-TYPE SYNTAX Counter32 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 36 } txCxnDataBytes OBJECT-TYPE SYNTAX Counter64 MAX-ACCESS read-only STATUS current DESCRIPTION "." ::= { iscsiCxnEntry 37 } ----------------------------------------------------------------------------- iscsiGeneralInfoGroup OBJECT-GROUP OBJECTS { iscsiNumber, iscsiIndex, iscsiName, iscsiDescription, rxMsgs, rxPingCmdMsgs, Bakke, Muchow [Page 33] Internet Draft iSCSI MIB July 2000 rxScsiCmdMsgs, rxScsiCmdMsgsRejected, rxTaskMgtMsgs, rxTaskMgtMsgsRejected, rxLoginCmdMsgs, rxLoginCmdMsgsRejected, rxTextCmdMsgs, rxTextCmdMsgsRejected, rxInvalidMsgs, rxDataMsgs, rxDataBytes, rxDataMsgsDiscarded, rxDataBytesDiscarded, txMsgs, txPingRspMsgs, txScsiRspMsgs, txScsiTaskMgtRspMsgs, txLoginRspMsgs, txTextRspMsgs, txRttMsgs, txAsyncEvtMsgs, txInvalidOpcodeMsgs, txDataMsgs, txDataBytes, connectionsCreated, connectionsRefused, connectionsTerminated, connectionsErrored, portIpAddress, portTcpPort, portIfIndex, hostName, rxPortMsgs, rxPortPingCmdMsgs, rxPortScsiCmdMsgs, rxPortScsiCmdMsgsRejected, rxPortTaskMgtMsgs, rxPortTaskMgtMsgsRejected, rxPortLoginCmdMsgs, rxPortLoginCmdMsgsRejected, rxPortTextCmdMsgs, rxPortTextCmdMsgsRejected, rxPortInvalidMsgs, rxPortDataMsgs, rxPortDataBytes, rxPortDataMsgsDiscarded, rxPortDataBytesDiscarded, Bakke, Muchow [Page 34] Internet Draft iSCSI MIB July 2000 txPortMsgs, txPortPingRspMsgs, txPortScsiRspMsgs, txPortScsiTaskMgtRspMsgs, txPortLoginRspMsgs, txPortTextRspMsgs, txPortRttMsgs, txPortAsyncEvtMsgs, txPortInvalidOpcodeMsgs, txPortDataMsgs, txPortDataBytes, targetIndex, targetName, rxTargetMsgs, rxTargetScsiCmdMsgs, rxTargetScsiCmdMsgsRejected, rxTargetTaskMgtMsgs, rxTargetTaskMgtMsgsRejected, rxTargetLoginCmdMsgs, rxTargetLoginCmdMsgsRejected, rxTargetDataMsgs, rxTargetDataBytes, rxTargetDataMsgsDiscarded, rxTargetDataBytesDiscarded, txTargetMsgs, txTargetScsiRspMsgs, txTargetScsiTaskMgtRspMsgs, txTargetLoginRspMsgs, txTargetRttMsgs, txTargetAsyncEvtMsgs, txTargetDataMsgs, txTargetDataBytes, lunId, rxLunMsgs, rxLunScsiCmdMsgs, rxLunScsiCmdMsgsRejected, rxLunTaskMgtMsgs, rxLunTaskMgtMsgsRejected, rxLunDataMsgs, rxLunDataBytes, rxLunDataMsgsDiscarded, rxLunDataBytesDiscarded, txLunMsgs, txLunScsiRspMsgs, txLunScsiTaskMgtRspMsgs, txLunAsyncEvtMsgs, Bakke, Muchow [Page 35] Internet Draft iSCSI MIB July 2000 txLunDataMsgs, txLunDataBytes, lunInquiryString, initiatorIndex, initiatorName, status, connections, rxInitiatorMsgs, rxInitiatorPingCmdMsgs, rxInitiatorScsiCmdMsgs, rxInitiatorScsiCmdMsgsRejected, rxInitiatorTaskMgtMsgs, rxInitiatorTaskMgtMsgsRejected, rxInitiatorLoginCmdMsgs, rxInitiatorLoginCmdMsgsRejected, rxInitiatorTextCmdMsgs, rxInitiatorTextCmdMsgsRejected, rxInitiatorInvalidMsgs, rxInitiatorDataMsgs, rxInitiatorDataBytes, rxInitiatorDataMsgsDiscarded, rxInitiatorDataBytesDiscarded, txInitiatorMsgs, txInitiatorPingRspMsgs, txInitiatorScsiRspMsgs, txInitiatorScsiTaskMgtRspMsgs, txInitiatorLoginRspMsgs, txInitiatorTextRspMsgs, txInitiatorRttMsgs, txInitiatorAsyncEvtMsgs, txInitiatorInvalidOpcodeMsgs, txInitiatorDataMsgs, txInitiatorDataBytes, cxnIpAddress, cxnTcpPort, isid, tsid, state, allowNoRtt, initiator, authenticator, authType, msgCheckType, dataCheckType, rxCxnMsgs, rxCxnPingCmdMsgs, Bakke, Muchow [Page 36] Internet Draft iSCSI MIB July 2000 rxCxnScsiCmdMsgs, rxCxnScsiCmdMsgsRejected, rxCxnTaskMgtMsgs, rxCxnTaskMgtMsgsRejected, rxCxnLoginCmdMsgs, rxCxnLoginCmdMsgsRejected, rxCxnTextCmdMsgs, rxCxnTextCmdMsgsRejected, rxCxnInvalidMsgs, rxCxnDataMsgs, rxCxnDataBytes, rxCxnDataMsgsDiscarded, rxCxnDataBytesDiscarded, txCxnMsgs, txCxnPingRspMsgs, txCxnScsiRspMsgs, txCxnScsiTaskMgtRspMsgs, txCxnLoginRspMsgs, txCxnTextRspMsgs, txCxnRttMsgs, txCxnAsyncEvtMsgs, txCxnInvalidOpcodeMsgs, txCxnDataMsgs, txCxnDataBytes } STATUS current DESCRIPTION "." ::= { iscsiConformance 1 } iscsiCompliance MODULE-COMPLIANCE STATUS current DESCRIPTION "." MODULE -- this module MANDATORY-GROUPS { iscsiGeneralInfoGroup } ::= { iscsiConformance 2 } END 6. Acknowledgments Some sort of acknowledgment... Bakke, Muchow [Page 37] Internet Draft iSCSI MIB July 2000 7. Security Considerations I think there's a standard blurb to insert here if the MIB has any read-write or read-create variables... 8. References [RFC2571] Harrington, D., Presuhn, R., and B. Wijnen, "An Architecture for Describing SNMP Management Frameworks", RFC 2571, April 1999. [RFC1155] Rose, M., and K. McCloghrie, "Structure and Identification of Management Information for TCP/IP-based Internets", STD 16, RFC 1155, May 1990. [RFC1212] Rose, M., and K. McCloghrie, "Concise MIB Definitions", STD 16, RFC 1212, March 1991. [RFC1215] M. Rose, "A Convention for Defining Traps for use with the SNMP", RFC 1215, March 1991. [RFC2578] McCloghrie, K., Perkins, D., Schoenwaelder, J., Case, J., Rose, M., and S. Waldbusser, "Structure of Management Infor- mation Version 2 (SMIv2)", STD 58, RFC 2578, April 1999. [RFC2579] McCloghrie, K., Perkins, D., Schoenwaelder, J., Case, J., Rose, M., and S. Waldbusser, "Textual Conventions for SMIv2", STD 58, RFC 2579, April 1999. [RFC2580] McCloghrie, K., Perkins, D., Schoenwaelder, J., Case, J., Rose, M., and S. Waldbusser, "Conformance Statements for SMIv2", STD 58, RFC 2580, April 1999. [RFC1157] Case, J., Fedor, M., Schoffstall, M., and J. Davin, "Simple Network Management Protocol", STD 15, RFC 1157, May 1990. [RFC1901] Case, J., McCloghrie, K., Rose, M., and S. Waldbusser, "Introduction to Community-based SNMPv2", RFC 1901, January 1996. [RFC1906] Case, J., McCloghrie, K., Rose, M., and S. Waldbusser, "Transport Mappings for Version 2 of the Simple Network Man- agement Protocol (SNMPv2)", RFC 1906, January 1996. [RFC2572] Case, J., Harrington D., Presuhn R., and B. Wijnen, "Message Processing and Dispatching for the Simple Network Management Protocol (SNMP)", RFC 2572, April 1999. Bakke, Muchow [Page 38] Internet Draft iSCSI MIB July 2000 [RFC2574] Blumenthal, U., and B. Wijnen, "User-based Security Model (USM) for version 3 of the Simple Network Management Proto- col (SNMPv3)", RFC 2574, April 1999. [RFC1905] Case, J., McCloghrie, K., Rose, M., and S. Waldbusser, "Pro- tocol Operations for Version 2 of the Simple Network Manage- ment Protocol (SNMPv2)", RFC 1905, January 1996. [RFC2573] Levi, D., Meyer, P., and B. Stewart, "SNMPv3 Applications", RFC 2573, April 1999. [RFC2575] Wijnen, B., Presuhn, R., and K. McCloghrie, "View-based Access Control Model (VACM) for the Simple Network Manage- ment Protocol (SNMP)", RFC 2575, April 1999. [RFC2570] Case, J., Mundy, R., Partain, D., and B. Stewart, "Introduc- tion to Version 3 of the Internet-standard Network Manage- ment Framework", RFC 2570, April 1999. 9. Authors' Addresses Mark Bakke Postal: NuSpeed, Inc 6450 Wedgwood Road Maple Grove, MN USA 55311 Tel: +1 763-398-1000 Fax: +1 763-398-1001 E-mail: mark.bakke@nuspeed.com Jim Muchow Postal: NuSpeed, Inc 6450 Wedgwood Road Maple Grove, MN USA 55311 Tel: +1 763-398-1000 Fax: +1 763-398-1001 E-mail: jim.muchow@nuspeed.com Bakke, Muchow [Page 39] --------------6139FF0C7285D1D87E83A475-- From owner-ips@ECE.cmu.edu Mon Jul 10 22:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27847; 10 Jul 2000 18:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA10969 for ips-outgoing; Mon, 10 Jul 2000 16:48:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA10842 for ; Mon, 10 Jul 2000 16:47:07 -0400 (EDT) From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id PAA11590; Mon, 10 Jul 2000 15:40:26 -0500 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.9) with SMTP id OAA160056; Mon, 10 Jul 2000 14:47:05 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256918.00722BA3 ; Mon, 10 Jul 2000 14:47:02 -0600 X-Lotus-FromDomain: IBMUS To: somesh_gupta@hp.com cc: ips@ece.cmu.edu, julian_satran@il.ibm.com Message-ID: <87256918.007201BA.00@d53mta03h.boulder.ibm.com> Date: Mon, 10 Jul 2000 13:44:43 -0700 Subject: RE: Re: multiple tcp connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I would second that, we have also managed to obtain link speed on a single connection, Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose somesh_gupta@hp.com@ece.cmu.edu on 07/10/2000 11:04:54 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Re: multiple tcp connections I don't think we want to invent sessions because some implementation could not achieve link speed using a single connection. First of all, we have demonstrated link speed on high speed links with a single TCP connection. There are two reasons I can come up with on short notice that may prevent one from acheiving this number. One is that the TCP window size may not be large enough. With the large window size option, this should not be an issue. The other is poor design - and that is something that should be solved through better design. There may be other reasons why this is not achievable that I don't know. -- When we get to adding all the other objectives, that is another can of worms. We should at least spell out the requirements we are placing on a session in the spec. Somesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Saturday, July 08, 2000 10:38 PM To: ips@ece.cmu.edu Subject: FW: Re: multiple tcp connections John, Correct - the only difference would be that the higher levels of software would have to be aware and handle it (error recovery, exceptions, balancing, increasing the number, decreasing the number). Those where the considerations that lead me to invent the session (channel group in our own slang) - as we think we can handle those well under the hood. Julo hufferd@us.ibm.com on 08/07/2000 21:34:22 Please respond to hufferd@us.ibm.com To: julian_satran%ibmil.RSCS@DEVM.DE.IBM.COM cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple tcp connections Julian, Since a session is only an iSCSI thing, your statements would be correct even if there were only one TCP connection per session, but several sessions on the same NIC. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 Julian Satran/Haifa/IBM@IBMIL@ece.cmu.edu on 07/07/2000 11:18:48 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: multiple tcp connections Parallelism on end-points could be another reasons. A long time ago when running some benchmarks we achieved considerably better numbers with 4 sockets than with one. This type of result repeated itself in different contexts and with different stacks. That is probably due to some serialization that is inherent in the way stacks are built and API's are used (this is a long discussion subject). With SMPs the differences are even more striking. Julo Matt Wakeley on 08/07/2000 02:11:57 Please respond to Matt Wakeley To: Randy Haagens , Julian Satran/Haifa/IBM@IBMIL cc: Subject: multiple tcp connections Hi, I was asked the question: If there is only 1 MAC (available to the system), would there ever be more than 1 tcp connection per iSCSI session. I couldn't think of a good reason why there would be, but thought I'd ask you. The only reason I could think of using multiple tcp connections per iSCSI session was for port aggregation when there are multiple MACs available. Any comments? Thanks, -Matt From owner-ips@ECE.cmu.edu Tue Jul 11 03:29:59 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29020; 10 Jul 2000 23:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA21305 for ips-outgoing; Mon, 10 Jul 2000 22:12:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imo-d05.mx.aol.com (imo-d05.mx.aol.com [205.188.157.37]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA20130 for ; Mon, 10 Jul 2000 21:12:18 -0400 (EDT) From: AndInc@aol.com Received: from AndInc@aol.com by imo-d05.mx.aol.com (mail_out_v27.10.) id 8.3f.74ea129 (4401); Mon, 10 Jul 2000 21:11:41 -0400 (EDT) Message-ID: <3f.74ea129.269bce4c@aol.com> Date: Mon, 10 Jul 2000 21:11:40 EDT Subject: Re: Question on TCP hardware up lift To: bill.main@bigfoot.com CC: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: AOL 5.0 for Windows sub 114 Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Bill, There used to be a company called Jato that made a TCP acceleration NIC. They had some nice white papers. Larry Bouchet has a little company called Alachritech, they're in a similar business. Level1 communications, bought by Intel, had some nice white papers in this area. Also Xaqti has done some development in this area. They focus on ICs for the IP switching business. Of course TCP stacks in hardware are nothing new. They're used in practically all the existing high speed (gigabit class) switches. I've seen some nice Verilog implementations targeted at FPGAs. They newer CAM based Altera devices seem (to me) to have the most promise. I have a small project developing in this area, targeting Gigabit Ethernet RAID systems. Give me a shout if you'd like to chat about ipSCSI implementations. Mike Anderson VP Engineering Medea Corp. 818 597 7645 From owner-ips@ECE.cmu.edu Tue Jul 11 13:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30909; 11 Jul 2000 9:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA14626 for ips-outgoing; Tue, 11 Jul 2000 08:10:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imo-d05.mx.aol.com (imo-d05.mx.aol.com [205.188.157.37]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25325 for ; Tue, 11 Jul 2000 01:45:03 -0400 (EDT) From: AndInc@aol.com Received: from AndInc@aol.com by imo-d05.mx.aol.com (mail_out_v27.10.) id p.5b.85d08ab (4560); Tue, 11 Jul 2000 01:44:26 -0400 (EDT) Message-ID: <5b.85d08ab.269c0e3a@aol.com> Date: Tue, 11 Jul 2000 01:44:26 EDT Subject: Re: iSCSI MIB draft To: mark.bakke@nuspeed.com CC: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: AOL 5.0 for Windows sub 114 Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Mark, I'm trying to keep up on what's going on with iSCSI, and you seem to be in the midst of the action. FYI, I'm in the middle of adapting our Fibre Channel RAID system to GBE, and iSCSI seems to be the protocol of choice. To be honest, I'm not sure whether to focus on the Adaptec version or the Cisco/IBM/(Nuspeed?) version. You can see my products at Medeacorp.com. Our target is high bandwidth audip/video applications. You won't see our fault tolerant products yet (late Q3), but that's our plan for iSCSI in Q4/Q1. Can you shed any light on how to choose between Adaptec and Cisco/IBM/etc? Mike A From owner-ips@ECE.cmu.edu Tue Jul 11 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30926; 11 Jul 2000 9:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA14612 for ips-outgoing; Tue, 11 Jul 2000 08:10:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imo-r08.mx.aol.com (imo-r08.mx.aol.com [152.163.225.8]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25009 for ; Tue, 11 Jul 2000 01:27:27 -0400 (EDT) From: AndInc@aol.com Received: from AndInc@aol.com by imo-r08.mx.aol.com (mail_out_v27.12.) id w.c8.75a95b4 (4560); Tue, 11 Jul 2000 01:26:13 -0400 (EDT) Message-ID: Date: Tue, 11 Jul 2000 01:26:12 EDT Subject: Re: Question on TCP hardware up lift To: echen1@ms21.hinet.net CC: ips@ece.cmu.edu, samba@vitesse.com MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="part1_c8.75a95b4.269c09f4_boundary" X-Mailer: AOL 5.0 for Windows sub 114 Sender: owner-ips@ece.cmu.edu Precedence: bulk --part1_c8.75a95b4.269c09f4_boundary Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Hi Eric, I've had a few questions about the Jato reference, so I've included an old PDF from their site. Level1 bought Jato a while back, however I think their white papers are gone. I'm in the midst of adapting our Fibre Channel RAID system to Gigabit Ethernet. Our design is based on the Xaqti MAC and Vitesse phy with an Altera Apex FPGA as the protocol chip. I am (of course) very interested in the Cisco/IBM/Adaptec versions of SCSI over IP. My partner (Stu Mabon) and I believe this is the future of SANs. Besides that, what great fun to see a new protocol born (amidst the blood and the screaming). Mike Anderson --part1_c8.75a95b4.269c09f4_boundary Content-Type: application/pdf; name="Jt1001.pdf" Content-Transfer-Encoding: base64 JVBERi0xLjIgDQol4uPP0w0KIA0KOSAwIG9iag0KPDwNCi9MZW5ndGggMTAgMCBSDQovRmls dGVyIC9MWldEZWNvZGUgDQo+Pg0Kc3RyZWFtDQqADEQQKBHIziAGjEcQOFjUbjYQDcaxA5GU QGYGkIqA0ajQQDYbjgXDcZCAQFQiA0YCCVyuDQgXkaVjEYC4YQuTxgWzUYDYbTgqGOWC4YjQ aDWTFQ7iAUFApE8iFUhlQQEIpEkikYUyc1A0ZDOajiFzUZR6TkSWUMajCSzW1yWXg2YwQZi4 cyWcyqbDUZRCT0KVyeliglFSaDAY1sqV0ixuFQwQWWPyQQRWLg2wZCJx4Y3CLRiW2m4jaVjM bjQXDaPWWZ6mF5aMDPJabUarI3yB67K5+9ZSlwkQEqEQI1QjQyspEeEDPOiAZjgY6kQG3lz3 nRLpGzl80ZjYc9ntwvTDXwZjudDymIVRmNjWRbLnZmzyq1Wz63CD3IjQSeUCdJ47ocqSwCbQ CpLBpoF7DwUGEGiaMQ4DmEAijoNAyjkNwyjoEAkjaOA2DKNoyjcOgwjoNI3jci43jkEAwsUr qVha5qzhAnabBkHKgQInsdQOpghjeOcNiKMwzDKMcTjsiwmjSM45RNFEVChEw0Rg+kZuiGSg LQ47KvymKSpEsSktAm0aKCtLBKYLgZBkGsrzGsaky6mwaIEv6hhgGsBTWFAjDqNjtDCNwyBA NAwjYMwWjIOsPjKPAQDeOELyjFMru+5sshdLc6KHNDAR+FE2r7K7HsClE9I6pE8xvPaj1FKg xjXDSqjqOUhhANMVURRVGUdEA8B2EApjyNsRDoOQ0jGF4wjnY1kWUMcrxlTK8VSngaufAcbJ qGIY23PwzUCNlf0fSIzDYN6ljHFNkjeNkrha7sbW/TkuT1UE1KVNk3TgrgGzlfc6p6HC/TTG 7O4NWI0DyOdl0SEAmDCPMLhAIgyjtZaLC4pokCyLitiTEkLjMMIxjLTCiJLTdOxrb1r1DP1S BtU055fOwbhnbkAJBUQjycMIxDTDYmjKMg0jDDlCjLSmmRJpY6ZLk+OBQI4miSJOQxWOVqRt a1PJ5g+ZX4FDDhgIQgimIoWipeTH5bfGYW5Uc3ZrgFT7CvYZBvblXBjn0/aNpGlZGMmmxHw+ oZHqQ5ZNlAQY7rGtK2M0WIGGEGbRtQi7ehe471fVUMHmmbYHPQaJ9nkCtPUQqRGFohaJqOp8 hjoqCFymucxBu07WFosIuNIxQveV6dBnDEZjffSX9OIXTJVGCBwGCzYRbwaerUQchsJYhBBE PiDJw9DCMJIjCflVNXt7WcdFUXS7xm9sJsn1WeumyQ/uwbZe8ECyVCBzDa7N24WGQgsI+DR/ xFWUBpSW5EFAUoDApc8vVLTcmVs8fg3Z0z0k9GkT6/gnoMIQmDCKGwNMAw3ImDKhJDLR3dqQ cbCs7T5n0PGIgC0r5I2DpdPvD8uJc3ME2P8A1VwNDoOrJpElPwSEhIbCekZdQYWkBuIOxppQ Qw0BpDgC0JjGQynacYhdDIdGvIzBkSNa5aFXHMesUJVyW2DmDCgHIN4dA3rtO0x0IYTAkhQT acxrahFDBQXgssPLsVnQwCEGVRDGkWRnXsy5bBaSXJgCMmJ6ERSePvZm85gDAoPLZBgztPJP ISN9T9H8OxHgxoWVmHMOp1AxqJDGoFSyKkUgglfFx9TLJJwYk82V+JiyEvzYIT6U0IiQzLMH H+BAVAhhQRcoUEAVQiTUlekgNcsg2oSlkHAOCLGpBkgq8h+kw3mqlfk6dbKeIRAxf2UxWStE NhmDSGxxquyDhikWoaXc25YyzfAHKOyuIcQWXu3p5UGpPpvee9GhgNn7xwbFPMFD1I0gzZPN VQwVgmBBCc/8MJB5wzjDlGZgDYJ0J1nUv2dkxm8vJBoDeEKPDOzwjpHZJ4YVjtCRAQMiCI1k h5pIGJECwwjT5n3FYEE/g5wwUuwBasGXkmqmXRZlYNJnFMgYG8M4bg0h6hgldO8PJ3RELvQ5 stIKRBzSulsFxE60k9Ry34nhp4SlMWHW6kaJaTLJi4HCfikkVQMDLA4MoO15A3JqfAFrYHkl 8nhVpBoOautWXVP5QahQXuXUoi0ODJ57Agr9SQg6u6o0pSlYaM9kn6J7KLWwwcAA3QCaIqYG kaSbmROwQ9vREq7k1uEgK4hfEBRBP2QNa5/0zrarwUR6sqmyhGCKEEKgVQpBFYuEUKYQyrhQ CoEkJ4Tl5E1iQUO9LL4fluPsXFP917s3bu7d+8N47ypxO+p2HRqDJI1iETSIiZYjIABxdSyx iLMhPDgieAdZFDBCDqhIKiULbpHRbgxSqJ6pzGRkvY3F7DQ3Kk0mQvMnXlujpgv+Y0oqGETp unqUtOimBIScGgFoU1KQwCgEMJLtHHNUl/QqSlLsUwbpiV2mdsS1o7xlWtP2PcfxpIEu1D0K FCUqmMjNz77HrZGtpTBu9MpkOoBhghVVmFRJFnwGMNNRAQZSOms5xqkrQy5nPMF0OR6H5jyV mVsWMWE13T8DMGYTQkB6gQbIF5tmhobUnhtKWeYL57zC3XJMx66l9xit5bWNAUaOdnFQMhFQ 5sPitpShb7s+TEg5O2UaBXARKiQ9YwYU1HUoDohJRrEdSamQkGMPIY0QVwYAvOHOenk0v0xi wruLqr3Rfa2XXE4pyISQqRUiwSQXhPOmhoNAbwybGmNSzZU6dWzrz9prWJPboKtZhu9ssdaw JQWPDDbj6djtw3PmCU+SN15LYJVyyqeiH17BRhWKjREpMRDnLUOjjQWhnSqxZUgMwxYTzmkN C6pCPBECaEHVWRVP7pzFB3F4Mc0KuJvwjegcFAsPRTIEgTUpXhuXhWCowLQQWjVmrXjKuET1 OZPHbU+ccfVPwna+q1sYkcrxRdQwauwxkVkXthCz/w0R2DqGcNHMJ7uXgEok7XPp7bkK7sih W02CbMVJs5gMm6601xSUQ2RQKdqT5ja1I6JlbwtyHS3kul5i5/rrZR1aDabKxtJ0BW6Q7C9F SEhLOXGe0RG35pXVnhNX5k7mbaU6Z16OCV3CmscLelISSuDU8hRu55q3+n4MIctHpQw5bexp dfXeCbmnnTHAdAJ2optImlGIT+mhW1JCS6gzsQO1HjnrDWHy1UG+PYD/8LBzDZnjfeXfNbo8 5pngSql9RxTRrf6nqPoB3DS4dFvlgQfsDIhVCUhCL50YtsSPIazKws8uy4yImE5M9+5QsmIc 2k7oVESarEwe9QV2caDkUc12qeqMXaUShazcqctsDmww/q6q8m60Isyk6YTQ4GYW9ibK8soS h0LquAxErmPsvcPwJguWwEt6LyjaK+8SNksyCCfG4YRSYiCgZKRYDaUIcgCMDK781Mkk++h8 xGkwxKk4TPAG7eoi+CT2O88SxmVEcMcQaeQ2CoeC/sgkfAisV28A2OYE95Co/C7g2gyYdEJs Ju7wKYjGUESczhAeQvAiwazmhWDORCqI8C368HBQ3VAK6dBOTSne1sKYCcDeV0ZIoNAksMJO Cw5HAE/C+ArqO6oqU+q4owCmm2UaRAUND3EowaQkcsRaDIRMaUSG6shUDPEy0tEO5O1gom2k BqnkVip43tCMqQIsi268BeXUKWDuhYDlCMDk/5FRD7AnFYBBDK/tDFBIjWg+BxE+xUT+fOCe /+LABuU4W6BcvWVSTCIiLsuaL0JuPs3gKIeyWuMGCkYyIG6Yh6dQe40uW+BeLKBeR0rMO9HG LrHwneuiW+1ASCRIV2DqqkRUn8VmXkdSNQTIW0Bs7lBeLeiAkwIJHW5YcCfwW+0MZ+CKCcCK CkCCCYvqvAj+vwvMqoRsBmNQroxENFCiuYwIxQiVBubKCYDSVoRcUEBArMseLaoZF69C+KVE CcQ0CGjCw2cuSuoooWlGrwVEl6QigQMKMOBiVIBkQlGIDQDYDybeT2OkRmNQy+T0bs8S5VEc BQUmwc9OUM9ml65sDo7+V0yvEERJLAiekkNQ1meSeqjeU+nkjmKYDeikDe4WisgQ/gZOZQRA 9sSkgQ/tLCBaDmx2UNFcRLMBHLKO8THkKZA2wxDM+aQyQkV2RcBA1QDOqDKwBcKSQsqiIvCV LxCYYACoBUXyJDC2BtNGBQ6o6tNqXGDcSS4ahQDoDygQIqUacgCeCKCaDnK8f+hSIsjwSvN3 LWBsqzHIJvO6VFGYntMsmtGOYsQqIsDeSWShKEXaVzMVBCBAjKDuRYDXO1N4k6XCnjP26nEm ceDLNkBBCTCXAdOODYDqcOcwc0c0BgQeQjPxLXB1HeerPCT8QoQsQwVq0jMoRSgQ+2IMIs4y SMQuDnPKoAwa9MwhQixQbHO+M7MQBRM6aVNKYsWQ3C3GBcrMb4N00MjS9BHOkyiHJ4ucLYTf NFSOT8CeDk/cJOi2QkCIjylnEHJgjQU80G5UdWLLSyT8CEqMt0KIIgg8wCP6wIVcNVRdG5K2 QaBiCgCEBeCJIkKMLshyByS1SuJ4pqzRTyBu6kKYi+SWO0CeQzK89UYAW1KooYhJO9G5KYDp KdMmRMkjUPIuNsg8jiBzG2RxUyg2OZPmQ1PrGaRcDHMlKgRbKxNUZIp6alU+DpVDGdP+aoIu DrOO9uQk/YQq56jsjwj0BARBUCJMDgjsZQ1On5R032hJHUXrGwjbB3QsbKCo6zTWMQV02w9n ECnKd2tXPYo8/jPsSGlyRcnEhQ4gSkDnQECSQ2DS6wM+DS6EJNQwjIVrPpPtVHVK9scu+hMU wwklWYwM+IBmowQSQWbPQeQkKwCKu4o0BcZ3Q5XC/tD3VWhgmkCgBej/W6tPWEjyhbNdVnVr XNElNZY1WIDmRYBa2K14haScDdWOmMRyNRH1BZHFHxHQTQTMMRG3KXKbKfXwa6YBOBTCrrS5 EYRwsyBACCDlLsSRNwIsCEXUVmYuaSp6DaSuMaL0MkN+IIIGS+OWq5HGTvZgIgTcjSMkBiLv HKNfREMxa8JKpq9aI89ZHGXAM8qfauBwMkJqO8TuKGwOVgLjYCW8IgPcOiNwrOU65UoWMsDE PYISZWKQNCIEO7cEswrmWuDaL0JoLaulc1cza4P0Z2LqBwetZusdZyyiY/IkU4J8hzJnIK+E ycBQjGyCZQriLsYMb6BbdbTwbpac/3aqMda2IKPyBmR1NAiQJE1mPcrmIgPRdEN3bqKPIvH1 ePaDKmU7eVbTehcQXoIcjSTINkNSNWTHeeCmIQI1caKKuGLSIEt3ZmIaBzbyetcuS8uUNKeh dGwKRwPcboCmQuaSXi2OyoRtd0RqVcJIzQBRYS3pao2PbyZ3dzeXTwL3HcTTdjVjdoYBata1 eEIRbdWU9YjSMoRyoWrlezcXfOOjfSuNfWBAiRe8IaJ/YZcsPpCgIQjjdhIUXfgAy2OiBwQF ghJoVThvE+giDK+a44oRgzeBa0LipqI8dTIurpfATJcQU7cVcZhSt3hWJnhaPQrWprfGJPfm ktc9gM2mR4KM1Aawe+kIvONTgfgJiEJ4LLMLG4CmRKRO8gDG/+vQZZjijZjnG0bpHpiOca8v g1eDc9YDIuTIiRdCI9JlgiBwLritbVhQU5bHjJcjaANsW1koKBjHhqP0JWPI2nSKXBgQCg+k +cSuLq0NbuRtIuv/jkiIBs4KBQYmYrZ9h4rmsdh/HxgMBzgQYwY1gwy2yohzllLVTOL6bpdl QBd+OBiYPyuRa/biU6LBTouZlcIpksMdkwTdk0BBk8oXF4JFflhpJthsouydjmMoyi6Swlj4 rnh8IEBbmAbFmbFvgscbmhiVmla3b+scehhaBkuIJLbLHFkrbrkvhVnENNnOIWq4LcYPlDnV lGOdfvTMQBH098CYCLTi2Ov9gHgjgKlIeXn4dqZTn/g3c8TuvRoLoOMiM5oVhNixHhfVi4Tv ZnhbfBo6CposkvnWQLnC98CEDejvjct3pIXwiPOBkGCeCaeMKIKRiBnxgnpRmeyFpZkTiba8 e0t3plR4Ihirptobi1nEKNIvifmziDqDjLndiIYw4eWUwbUmmNIpjhpLloW/b299JRqlq5mn g7T6OlrCjUOcNQPhrLefhPm/odchhbfhoIq4jTp/reLijaI7d46XgCJHr1qaQBJ4KEBQCa/x l27ThDr1iDkARxqe99hyjsUEeLsFoCPy7phlnHhGrXmzH0JENtivrPpza2pqv2Ia0MTsKTsw PyYST28SNPLeCkUihs30mNkRsGAbtwPgBpTtuTt7eZfDsbpvshhZuKoXu5MDornTqFfyYK06 TskoMGCpunG9miLJbuI9byBzb2LCNOKRiaJBnKM7vDTnk6rRuDsfSPsiprcDnGKLtzjFvXc8 JjdBo0xOTPr9P4syCJFeVtRIRbd7Ijn+DiIRoM7kNDm5hbURe4Ifm0DGOpwkLidyAaAUBeCu BAwPxoBeCQIGBpx0CEx6IHx0CSCa+xIbx0LQC3a2JWC6IQAUCSLQAAAAHhylyqAfyqAHypyq AAAeGBykA/y1yty0D/zDykAeH8AAAGH/zLy4H+EAAfzXy3zaHAB/zjy3zgHwH/ztysH+H9zh zZz/zV0BzcABzJzlzPzN0By1yz0OEByqDhzkAdylxoCKCTxoCjg6Idg+epgjxgKPHErWIdek IgO1uzxXcEJDvD1L0/03opV8IRfLuzIEJ/nH04rp091mIbxbH11WdaIdt+I91X1z1r1d1L1j 1Z1D1SNt1wO/1p1/1V1gIRxIITHENxxRYZcEJvgi9ZkYl5xhfoPzxnxrxuZ1x1x4KLx/yCBj yHyKWTyPxryTyWBBybxpygBAAAACHByxy9ykAP31yl0Zy/3/zSH5yqH/4HzV0dzV4R4Pznyw H+HwABzr4ZzRz14ZzcA/4b4AH/4L4X4f3/41zT0IAAA/4R4L4B4R4GAB4R0cAB0p0sAV0wI5 wHIqNJNBxhF5hfnGInoJ1Kyb23eUvT595phhph2MIQspcrnH5sI95wBlcIJL25576QTfo09Z oj1eI56f6UIn6N2j5n516vbQOn6p6h536x6OAb2mTdgj2ve4J67kpqLqNxxeONjJxl5hxsBB v33Nx73SCHyFxryJyMDLyQBByUIFyZyd3t3x5VykA9yt8d5J34B/8lzB3x5DyqA/zRzh8sH+ GB4nzl5JzX4t9F4yH54z88HAAP8zy/zcAD9b5JzR4l8ty0Af8t34AH8ty38kA50mAV0r0v6Q knhbsKOb5x+J7l6V597hkaOw9d6HTvT0gz7SL5Ipol+MJL6d+v+L7mJL59hHbb+f2D6p+5+m Ob+r+T/H7J61/N/X+r6qIgKMO+Nx+QLJifu5+Xg7v3wJ/yNwDYIAIAaNByNBcNhoIBoNBiLh iMhAbIEU4ENRlBhwOIUNYbEDbFRkORdEIXIotEZBGI1CxvB5QDYtKo3HZeYhVAxyNhdK4ZLo /OINCIUM41EpwMhcN5JPRtL4JOp5DabRptMBkNZcNBrWBlGp/BI7K63I6LAq1OpzG65ZYGNb RTYXDYfL4pVplZxdaZ/FpFJLHXZfMZ3YhmLhnU5Tg43b5rN61IrncZ9ZrdebhRKcNYxCqZmc hS6ljYGMqRCRoNsKOY9ZtILtMNRhlqdrdNPbnRrqNIuLrdnNDX9pQ7ZutjF99LqNuoNvcliA bVd1pYVqLzqwbRKRK+pqpf2J3x9vE4FRINaeaIJ/3pXmKNRNjQvP7Rx5bhnapjhhSKVChzLc Ar78qSkgcsKGy2BmHIcNc/isMOp0Av2gj/LY6EIQHBqmuA6SCQwuiBOKHDjPegoYhA2IcBur QQDkM6zRQ6oQBqHLYhipobIa7kFKEOQyhAMQGiEKgGrkq8TSNEqCQKjUZMKwAQCoNoGxo/MT IdKkpohFkPhcGCLBuEEnjGEAWtiGAaBvL8njuEAUCmPI5joMqPiSN04jkMwwjHHohDYN4xjW FMnjUBoiyFEERIPEkqxRFUtIGh8uSWyoZogGaDUoEAYzKjUeR9IEhSIGsjBhTKFBilq5rc2K EyfKMsSrTSIVfR1STDMcyhgGCNTVNgmiCIYQT5P1AUFQlQVLEtHL4kbOMKG6mhm2LzU3Fcex /IMhpG0lZSPGIZpauAbtDVspVHFcWgaF4jVIrEzTAKgzAbMkuBuGYZ3fMVaipNYUCSIt/0CK lBhahtn3rMadONJ4iVvLgaRRfE2BiGKCIdgNBrkG8Q4Rd+GXmh+NYiFAlCoIIj4vbODXuFuE 1YKmPTKiym1sFAoDeOQ6ZRQqrKwuDtutM7TZ+2cN6G5Lgsk8IGrqwSMunnr0JAvrptSiCjaa 7WrMTpzT6g+7RuW+rftZDb2NY4ratDo+wuO5zoNhGKQxg4CsWfuKW3c3CKhitb+QU4y94nGC CNKtjc0hXW/JGhMAQVCLVNdwyBRLylzrMGG8NNBPF0y2NMR0hNO3jbC5N0HNzSRTTFNghsHX JUnYctdN1hAwr53feN5hgGdRZoHGdWPytlS7ZiCb+hIYyUEHQWqEHR1Ah3TdRGIYJMiHWMNm coXL2NHBRSngSHZHZOjBYa94g6NBlcFM8z5tr+hbduVJJG5QczW6+1V1zUddV2MOdwvJMoNg bu9ComIFCvghsoBaUgrZgGWMdYamYHC92aBDDenUOQbw2PhcqsldD5TTEZIcU19ZWSWknU6/ BbL8npumeyjF1TD13v7e6uh3UBIDQIgU+ErSl0BoTaiW2IB/HlHJbghyGMSIioSLIaKH5hkL wxK+UqKSDIlniJg5grJ/Ynl7i4XBAj6TAu7ifEpBzV4wn8iE182D7I0IZak8UHKHW9KPQU4l 85BoaFfBjHlJa0TFNHOk+c7JTo/qRW9HxChN0EIKITIY78fpALekPHcjjfXzs9K83tHD125H cjuQwpEen0IGiGDRXSAm4kilFFqD75IuLuk3GR1TnyDuhR689bLFCQwvdU0586CldvbdiqR/ rtHbMudymV3jInfrFZ3LF4ZXFvScUy8p5jolPy9ek7FEpsHrLemJDV7iRnvPgmkkI08gZGSp lQ+dE5w4rKYkkcOeKlnIxQbhPeeEgZLxaaxIufZezdzCkE05q7cnrqUO/QsmUe59xujMYCf1 BqI0OckQMG1DZsFfBq8xZcr2luTfGo4rUkJrxkmIRqbcu5uull/OA6ZDSTzDO+69/i6AUTRY EsajiA3jyppDLk/jhUPPieFCGop8HNnGeZS951MXo0zW6adgpTXsOumNTsgVPXwhxcnKdczL DNm6BAs8iAYyfgvDSG1EoRA3gNICDQplbmRzdHJlYW0NCmVuZG9iag0KMTAgMCBvYmoNCjY4 NTANCmVuZG9iag0KMTMgMCBvYmoNCjw8DQovVHlwZSAvWE9iamVjdA0KL1N1YnR5cGUgL0lt YWdlDQovTmFtZSAvaW0xDQovRmlsdGVyIC9MWldEZWNvZGUgDQovV2lkdGggMzI4DQovSGVp Z2h0IDE1Nw0KL0JpdHNQZXJDb21wb25lbnQgOA0KL0NvbG9yU3BhY2UgWyAvSW5kZXhlZCAv RGV2aWNlUkdCIDI1NSAxMiAwIFIgXQ0KL0xlbmd0aCAxNCAwIFINCj4+DQpzdHJlYW0NCoA7 4FA4JBYNB4RCYVC4ZDYdD4hEYlE4pFYtF4xGY1G45HY9H5BIZFI5JJZNJ5RKZVK5ZLZdL5hM ZlM5pNZtN5xOZ1O55PZ9P6BQaFQ6JRaNR6RSaVS6ZTadT6hUalU6pVatV6xWa1W65Xa9X7BY bFY7JZbNZ7RabVa7Zbbdb7hcblc7pdbtd7xeb1e75fb9f8BgcFg8JhcNh8RicVi8AUT5jshj 8lkcpk8tlcxl81mc5m89nSjjKvn9JoNLp9NqdRl9FVdVr9XsNlsc/rantNnudxu9TtqjuuBv OFwdDvqdw+RxOTqONTeVz+X0MxzaX0et0uv1KT2O51+T2qP3u74954KL5PR4tj5qH6vT79L7 KD8Pp7s38p/9vr+8nZwAAC2v5AT7JG/8DQPAybQOgsALM/UHwGyCQQRCkKohCsFotBCDQxDs OobCidQjCEHo9D0TwbEELIvEMORRF6Fv/BicxJEb0I5F8YRVDcNRbF0cw+hMUoJGSbxtGr1I 3HJ3yXGMeIrDCESBD0hQbFMrJvJEju4jMToZIKFSeicwQZHSBxXDkzoFK8jS1NzyIxKiHyjM MMzHMkiR9L8EoPNkmTUm030FLkezRC9DR/IqJS9Ps9RVKUzztIaZS3QbkULR1D0zSM+UXOVE 07C8+0BJkAUmmVLUq4FMTsik6VBO9PzLMUWVJP9TpjVVUt3KFEVdTdS1Ch0zVnVs4zXBdFJp XddNlX9fWfY1kWFPcQ1pTlpR7Uc2uU6rTWjbNwWVbFcSrC1r2ncNtWRGjnqY+NPWBXtr3Rc0 eXraEu3GmzoKa0lY3redpYDUFO3xeSqui47OYBal9Vpgk8z1g+IqnfuFs1ht94fY2K2DiFw3 yq2FINZraIK+6I1fHFHYrPGKXU1zloPZjhZQyt449nNhZdX2YYc0d3ZLmrdZu/uVYRVlx4jl dySlpOLZmg2iZsgjLZ3jeOVbgmm6dRuA3Ld+hILk2i6tCWkZ1rFlbBkWfxQqmL6NsuT7OPm1 wnQ223lt9GKlkmyao4Ob7wj+V77sOP3LKfEqVuW7bo2eUcLE2W2zrti8XKaqcAgnI6rgWgTj y1qcxzOnyBuOpbnwXQY1rMWdJpeRYlrPac5sfPdbbvQ9h3vZ5529081tSn8egfd9XyklURMX TVhr+Y9w7+h8/wfX8relJeFr2C8bqPedZ61eex5lM3v7nu8z33puIg/x/DTXpbT7UEzxJ2gf SqHjoF5LpLie+sNaCG3nuods1Bv7uSBvwUu6+AL+HEPFeHAZ0TMn4vIgYquAED4KMhb8/J4j 84EvUcDBl8kG4Nthg/CCDsHHjPKgw/6EkKH6NcVlCx6MFVbNihnAuGUF4HQcfu+uEUFIWv4f 3DB/sJjctahtEOIj7IIQhcYVBzsPomN1ic4xnr+ocsbi5Do88Sh3wNJYb180YYxPDjWnV+sY YrQKIE+4ly3yOxqiFAVarsouRJiBGV68dTauGc2nOPUU0Wx4jaUKK8czhq5M8SSKB+H3xkjM S9hklCdSNHfHSSDOJNE5f5IB10jj6SVMpKFdsPZTHlfEfqVBkixQuLFKOS8rZTvVMeWGKSDo 5SdkfK898sW1xqijH1gENZFTHiqSWW0gYsJalREGZrtZlwglpGyOEzJCkjk5J6GKApKwpm6u SPE2JsuKnPNxJpI5nylmAhF985JyzamRAKBD3p7zmm3N6X84IlqDZZIpU9BI2yTjdQSdk7SQ zvbM7pG0aaDulkOx+fEXp7I7X3PmCRE5vzBchQKLamoiNIo1RidUtKNwSkWRSh0J5o0RdHR1 SdKIpQrpPL1Mr61PT+lZPGV1IaZNKnSkOlCf6Ev2iLPxMdPKm0NktNCXClqZ1LXtS2Q0BKrU piclmf9IJwq7djR170hH0U6qZVWtBKKXuShKzWolaq10XYNVao7BavU/oBKSNBFTVO/rlOmR CV67T5kQtyW9AaH0xX+Revry4APZeDIuu9Tia0fg1UKQdjrGvlrjQNjtS7KzWsFO6qM8DYEa NBYCsdW492TrRaOc1ea91AibMJlJGTO2ss+RqAcRbZRssRVKxVMKp27I3Jmz1vKRvQtfXODt tKwWMWdbg6ZHDM3MsjRKA8a7gzquHPCvlxrbWds1cqiDR7l3bUhVldTHnaQPsMSizFQaw2rl 1Y+xl2r1rTmVCqEVv482umdaexdx5ItTurW84t/Zi0Ii+4mLtDLDqBq/Zm+9m70mvmJex38N 7BxTufTiq9pSRVti1dTBN1rzSmv5g+itaar4jxBiW6WGMVYcxZKm/OKZHYvv/jXG1r7B5Ci/ eHA9xa3YbxbkrH0ncgToTDXSXrTG+XuugSW+tt7z25yZkuwOYcxZExLia2a/MDXktTgqxOHs 3YOiNmWruFq9XTvHfjBl4rIZwylXSpOWa8Zoj/YnNeOzV5jyjlitbe8CaBsvhe+2Tr0Y5vJf /RGfM44RzmTXFGhdKSg0Np7Peo8+6KiPXLG+kcEY6y7dfPMf8qZmq5b3LGp7W6AtNoO4motJ Z20VrK0dR2+6bWXpDLmGdJ6rz1TnYF86ytY01qjQWbdCaHzY//N+pEZ7PYbtFGOFdH51xxsr UGn9K6x0uo3bzO91tP3BuG4m5F4avxXubBe2tfxgcvs60k+qCxPrJlrWFtdeHYhpUTCms1xb +bhmXWRIYyXl3vr27O9Njzo2bP3GVy5rwt4fxDgedj149yTrHXFKXU2k5O3qdeEasEp4jXvk fFuZ750bQqfmgEwc44ZycknMbxc14ptYoyfpJVIVcU/NOYNkZe6HfqVRO+l5Pzvhrp+Teoyb 110HVnTeodZ61tTePQuq1/7AfnrfJeyaq7P2Hsfbzmahav23tHYu2dlYztfuPdCfdT153jV3 Xu8988J4Xw3h/EeJ8V4vxnjfHeP8h5HyXk/KeV8t5fzHmfNebJyQEA0KZW5kc3RyZWFtDQpl bmRvYmoNCjE0IDAgb2JqDQoyMjAyDQplbmRvYmoNCjEyIDAgb2JqDQo8PA0KL0xlbmd0aCAx NSAwIFINCi9GaWx0ZXIgL0FTQ0lJODVEZWNvZGUgDQo+Pg0Kc3RyZWFtDQp6ISNiaDs5YFAw RCEhJz4qITlPN11ydShrOzFCN0QyISZIRDJBaHNGVDFCPldUITw1YW4hK2wzMkFjT1chIStv Z0MNClI6WkhDQWNWal0hISVOTEo9LUI7Ui9mJTIhK3FXIVJAKychUi9tOG4hISg0QyE2YkYh YlEpOCEhMUQlMmJmZyhlDQpqbz5EXSEhKiQhMV1JR2VyckFKZSE2a0hDcnQsaFQhI2JoXStz SmchISZIRGUxR2BXITFSVSRUYlZNJ1RBaHNGIQ0KMUdgVyFBbkVpZUFpJVlDMV1OayEhMT5X IVI1OEhDMVJYWGViYUM5ZWJWSzdDMUdkNjJCJDwnIWJWUkplMV1SSjINCiE8NGkycyJlOWUx Ulw4IWJsOUwhNnBzRUM5YFMwbjw8LVYyIStuImVBbkQkVFIvZ2pDISt1NmVBY09XITFHYFdU DQoxTTMlMkIkPCZDcyg1akNBbkVpMkFuR1hlUjpeJ1RBbk0oMkFjUzYyMVJWaWVSOl9rVEIk P1pUczMsJ1RiW3JaVA0KQW5LOCFSRVQ5ZWJcJG5UQWNWakMxXU0nIXMoN10hQiRDOWVzKiZb MiEuWSVMSjpOMCMhLl1UTUxrcGllUi9kNVQNCjFCPGhUITFCNVRSRVBaIXMiZyZDMVJVJFRS Olw3Q1I1OjhUMVJcOFRSL2dpZTFNMyVlQXNvJyFSRVQ5MnMtXThUDQpSQCxrIVI6X2tUUkAw SmVSQDQqIVIva0khMVgpOCFiYUZtQ1JFV21DczhTSmVzLVlcQ1I6Y0plUksmXSFzLWBuWw0K VFZTVGZXVzlAKl09WVxqXnFjaGluLFVaNCE2ZDUhYltwayFSL2tJVCE2a0khYlEnSEMxR2Q2 ZTFYKTdDYmZobGUNCnMoOUlUQiQ8JkNiW3RKMlI6YVtlQiRDOkNiUSsnVDFSWkkhUkVWKGVi ZmxMIXMzL1tlYmZobGViXCMpQ1JFV24hDQpiZnArZWJRLltlMV1QWzJzMy1vMmJmcCsyczFt RipoOy1sZWpvRmk7bWYzMVIhPDMkIXMiYVpUQWNWayEhPDo3Qw0KczhPblQhJk9YMjFdTSZD cy1ZXENiVlQ7IUIpX2pDcyJlOWVBbk0oMkIpZyhlczhTTWUhMUVqQ1JLJGxlcy1dO1QNCmJh Sk0yYmw3W2VzImhuIUIkQzpDYmw+bzJzOFctISE8PCdUczhRXjJzLWBvZWJsQF9DczhXLSFz OFctIXM4Vy0hDQpzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhX LSF+Pg0KZW5kc3RyZWFtDQplbmRvYmoNCjE1IDAgb2JqDQo5ODYNCmVuZG9iag0KMTYgMCBv YmoNCjw8DQovTGVuZ3RoIDE3IDAgUg0KL0ZpbHRlciAvTFpXRGVjb2RlIA0KPj4NCnN0cmVh bQ0KgBREANgIDQplbmRzdHJlYW0NCmVuZG9iag0KMTcgMCBvYmoNCjYNCmVuZG9iag0KNCAw IG9iag0KPDwNCi9UeXBlIC9QYWdlDQovUGFyZW50IDUgMCBSDQovUmVzb3VyY2VzIDw8DQov Rm9udCA8PA0KL0YwIDYgMCBSIA0KL0YxIDcgMCBSIA0KL0YyIDggMCBSIA0KL0YzIDExIDAg UiANCj4+DQovWE9iamVjdCA8PA0KL2ltMSAxMyAwIFIgDQo+Pg0KL1Byb2NTZXQgMiAwIFIN Cj4+DQovQ29udGVudHMgWyA5IDAgUiAxNiAwIFIgIF0NCj4+DQplbmRvYmoNCjE5IDAgb2Jq DQo8PA0KL0xlbmd0aCAyMCAwIFINCi9GaWx0ZXIgL0xaV0RlY29kZSANCj4+DQpzdHJlYW0N CoAMRBAoEcjOIAaMRxA4WNRuNhANxrEDkZRAZgaMBAMBcNRgMo3HY+IINCIdGoUIBkNBANhv IIrF4TIxpIBoOZBFItGCEVAaMxiLpBL45KxAICoRAaLyNBI4MIXSIwLaeNBrUSoY5DUKvRyo dxAKCkbzEdTmdBARDkaTsZTkcxAYTcZBATbkYTOZTaZTcdBTSDVGRALRjQRrIKQRI3i41JaW RpAOBcOKxGKeMcRWcXSLAKC4MhkNb+VMDkspm6TW48Na9WqoLhgMIfXs6UzqcDgbzkdDSboO czeZjodzDMd6YzYdTIZbgY7dvDMaTLdOWdjTzhAZLXbbfcbmIL4ZNHgY0LRyLsxXsVVYdrZC MRmNBntLDZrd2TTeDcb7P1zm8YGvO9LCKCGSsPW9DMq0jTOLCz4ZBtACUwY1KnhqHL3KeGCB QaFArt6Mg3juObPomEAnCoEAZo6GIWBAGjYO8ugatgHYQQ+ucRRI0KIBy1gnimKQZB3ADywE zMENkqD3NeGAbNA+kPRBHQQRLHocRsJwyjoK7ijKz6gKPGDCxkEEaBhADCIXAihQO98FNQzs HwiwCEoXCkkhslikNcjjMT2r6wts3DdLQMzdBAOg0ItACJhdPTUSSHCcQy2AcyXDonjcJ0RB BAAZo4HKITwrYaBpDjNSaldUM6JI3DS3gwt4tsyigIYkhAKERLdNIcBvR7ByO9StpcrEFyiu w3LwvS+L9OthMIp83KYoYXUoqQGte+E4M80DRTq0072Gy8fUqGAaBg1kOy6NYyhaN43BaJgg idMCBRcuwzuvXIwjHdq0XtXrJzWGNpWHbQZzhCk5W9AFw0jYgc1RPsE4lKNbVwKAmiKEApjS M9lDZNIZohNlz2HagQYfbE/YVKM5wlcTEtVdVU3JKIpDKOI6jSiq6DKNlmL6uDg0TReVI+Fz 5iaJIhZHkoZRW2eZo0jiPJBqyRscpinNgytswsiUmKfJ+as6IIyDJWA03gMI2U9OoaV+2dSQ 0HL5z5OKwy0Ogh6At1ZN1RobUellSSarmxvQwuzLCEAjDLWQ6oq/862jGCWTZA1xsYkiDsey OBq8yzYPTvOFwdhtwdFurYBuGOa4o+HYSiIoiigKQniaEFXDotwzX6MsAWgws23HBMM5e0E6 NJO2IKrMPTtgicMQ7jzfaCEA7t0uiKjCMgXjutffaIOSSciMi4WbnqLDHeHfaHNKU81N2W+T DuYTrCfj3PYzVGZM6ENRa/g5h1DaXE4R93vBkRcGEOodA3gtOW74MZvF4NPMG8VzbVHSsufw 6p5rD3WrpamZpDRRkOhFDwHAuQc22BuBA7l3bvXfvBeG8h+jx3TQmeUhBmLzzYA0V8uYHAM2 8KBBQoM3Juy4QyRc7eJyZQzBsDCHMNAIC9htN0Hl+TBECv1eQ6eHrzDAv7g4udcr0kNrpSjD QOTIQ2B5BBAVQkTAQG3USG9F4SzwQrhbC+GLuouwZKCyeDkO1jwfh8/pmSFXptRcUR+SCHQq hwjyCBeBFi9hnDCGIPLvgQBjLkCAMRFo6RLd8XQsxvSDhlDw76N7bkirBjDI5/r94kBsDevo McGIcxnkQ6k0LDnWP8VMxMrYMwcvViREpQpcI8QQIuG9yYIEQFsDSGQOrbo4ggDgHKXYcgwh tDbJ1oKEgbp+VHMY2SlT4N3SiCAJgRQiOVeaYQG552Ty/ltMF1C3ZhurNPCMq7sTBlPBmDhx oKAku2Y4YUm56AwmEmWfMLgKAlBUCCEcLhfyzlyDIcUuj7g2hwDYfkvspJqFzOLHIOcog3Mx BkegiBoVfyGQqSFq9OmtOfa4QNgy2Emg1qIuaolCwjBFCCFQKoUmOBECKFMIYUgkhQCoEkJ4 TkqgofcX03odTp0dQk0lURg1QFCUAYpVQNojsUBg3cG6UQqBlDGGg/cugzxycg5Jyks02GHc 41VzxCGUsrCo6RDcHokIPW/CGYsZwa0JXMk6ACgg6Kyk0v0NBvSLHLhcyBgRp5+JJn9D2xpp bHy2VNQYy9RUOtKNYG0684IYTfDec4Oc9jAgtYfaMrdpZFWnAbCJ/lRJmVuBjMti8VD+laDF NwEAZy1l0OLOKOVFwoBCCOEGjs3l+hrWXaGL0QX+XAsXCC1FA3+TKmQU8GVb0oorPmHYN4bA 6AvNZfS+0eWgzipSEkF4T7dLZt7BqMFiZcMMoDY69UZ25KAQWbAGQOa4odCocUvMqZvK7fNV 2AtJYLQwaK0G3LRpRmGCvVsOgeQ4HXbdWJy1vTQlCsCSJrGNrBmPa6piw7YDYHxkiDgiCHQo Tgm1BUEAQQ7BhDTFQMWTVYRcTqeXA1gTFtbMgyp0TLIO4JmFcK4kZyXJwW0DIl6UQpzjpMcw uOS8mzmIsfsO94pCPGmBYrBWYLUpJiEwp6ZKciZGDrBWQF+oDEWDDm7Jz2QaBRByDhhyv0Bo XKCyRlGWVfrXx6VWSbNnpnyeUDUgQTAyltbepoiwQw3zkDqq+UWIQ5ouVcGMFxA5l6QctMol F73+Ggf9QjM6HVXKHDlOXEM1oYKKDSXBEIY9D0p2XN6cB1jll0N7KFeAbq6bHDurCK+pNTAg 1QCDVWrNXKyhfrF3gbtabSDfkcOgc9aonj1Sc5wbg5hlRdK6b5zN8l0UQGmktJzpyXLlHI3r vg2NBgrNtkWU5aZDlsSt/ziAY4ViRbYOBz45FrDOGjeM1i4XSLlhqTzRto7N2eHTeYRTuRyV k77EAIA0RVlIGUvkWHvkWmlFWApMVFKyaMRY3uxNjSA37vkvvBdr7K2ZbflfIi4hj2dOJ328 9wNA3FtpAAVAVW/jTCaIKT0o7lDbq3F2sNZbs1rz7Q5cD9ggpPJ3KGK3tc13iG/UsCmgboN9 Jc6IdNtYl2IXG7xuzrh1io+YODkzc75RcXsOVdeS90pPitF3AQ3BmLWb5oRaGi8Hu8/AOiLn 3BwDzx7kEmHzBvUUrziB5jJ1pK3mattB2vA3iOZ3hLQOGB04du7jZu+Osf5AC5AEQjB3vppr 3jFyIdmd6zqfbW5NV9n3P2rde7XKFuLaXD15JPjFomlOVdsoea+e/BHoOfGwxhpOjq/dKZbb bwOycwMZaw4awLiWj0Q3gvZFzbpRSagtA/Y3g5z5BOooxSbOw1JlJgprx0bHwj4HDCD3AGAo DiQzoIxRB75tTEJtzZDozdBeCUK6BCTGawBuqnjG6nbLDHZr5Jri7573BgrTpOQFDUQGS7pd AG4FoGRgpNBOpJ5wp55zrLAkECLHgqahBKCNQiayoFD1gEA6yFwtD6brZL40AGQuBhxGCspw 4y5JxxRgoHJQAzoO4MoMUK5RhOqIx5EMTH5Sadwq6ZgzroSG8C0I5rMFynqwgpqoECSoS9x2 iMRDoO8RIFxoLUwoKkhkZAwyaDJJBzoxqnzLMJcGS9w0KyZixDoIwMIPCrcHcHow4Fq4wxDc qFgNyORRZ75nY4qWAuBLzuS26zC6gtDCgjxABHgjqdZmcGbQDTxgoHEKQEAI7vT15T4oJUxi BVSyRvJJpkkYwsZ75FzNL/KcZZoN5FwIYIIEBH0YoG6LpGhzbCadTS4ghFbTSxBA0C4p5iUD YsKFTNR7MIQGAJoMQOAuAIr16N5LYEAIQN6kSXy96mbHgxSwrLbTbLsQ687BaMoyQoyEYmq4 494GZ15KKjMIQpwF8IUjw2IjTVQvqcDhb2Ce4hStESYFz5TO7LygC4UlMia4rHjCIzDTQzse 8fMfZjoIoKSqAKaLslQwimclqfrPDL5mMlSgkOiNRDZVgsIIwNKUp8wJ7/ZfYKgLAF4KQLEo Qo0oklUl0hzPMpUmbBywB6Q+Eo0nI0IJoJAPTciXRfxCSiI2cjDGksSHi4Mspwy4qdz3ceaP zfCQCGUrxzIzEljA680siRkpZ/giSgwFEgUghy0mUw8osxRbixkviIBJzshvMyScRfxLcw0l aFEo490mEzigkIKdzM0O5QQtw/Jt52x3CQROoIonwookJUcljHI0CL5FQqB0RVclglglYFzM 4mIjAGI+QjokAoBGkPYoA88JQoC8k5YjJawiAsAhIEAJQhAgQNQhCwQjQKQI4hAlZX8ZpUBY SA4Bs9U4wlRUUIxt8+EIJ4zCZwhSE+wKc9KyJa06Ct55E940JMams+k/ghE/0+FABSk9tAk/ 8dc6ETLuVCU+QoCQolk+w0KmbM9DM5IiFDjUVEJFUxAo1EYjghYoEzFC0+CiM6dE9Dc9KIR0 VEA2dDlGoGIiFEBk9HJX4G1FdCtH9DFAY9M/ohCZQyQllCA9M99JU30/U+tBdJJiVAM4U91K ooJSlKVBQBtBg+U4NJokE99MJ4woBg0+w+VFVE1DVF0ir2dNtEtNTCZ5FFklVOjTNClFtOhw istO9FAhAmtHh4tHFQTGZzdHtGYBompGFFdIwkFJFRk5AhdLollMtSlLB5FSVKFJlSCLFKtJ dE1NNKtPVOVH1KtP1QgoNQwn8+go1QFRaZRM9JkxD5U+yZRGBJdWNFyZRFY9NG9EVKsotU9W VK1B9T9SU5qmZExU1IFUAhImsX0+c/dRdBlTtakI1TFR1bNa1LVK9S1aA0NCdbtF1BlcdK9M daC9ldNZNKgBoOM8IwxrAxZGg2ddFZC+QiAMc94IQJIBoBQF7FNHdgAF4JBlVgoIRWwgdgox QLYgYxYLohFgIIc/wF4IjUo64MoI661gAJIxQB9kNkVkdkgB9gAItf4BQKIhFeIhNeZzteym oGxGk9kDJpVfdftlNgQgYG1gtg4HFhNhYGNhoEFh4gQjViVgtioEFi9jI51jgMIPNj1kFktq tkNk9lNldeFeU59mFErM1VU4VfSUNnNgtgdntgNn9oIIdhlgNh1iFpFiYF9pdpo61p9joBVj 4EFq1q1rFgFrVlp4sFdetr4ykxNsVm9sghFf1s1nln1hFgNhVtlodt1otuAEFpNili1jFu1j dvFvVvlqtv1lVllrlwYjVmM4QjRUFsdflxdnVs9x9oFyNoVolo1iNuVulzljVqFqVvNql0Nk d0dwF01el1FEtECwF1lxN1wBtxlgN2NtNyAF9yVtti9y1o9zF3Nzdp1z1qNqdvd4N4QBVlFv 90tl1rqwV1JhJFZk95bS15t59ndgl6V2d6l2tyt29uNpV7lzt3t8F8V8d8t0lrd9F05M15A+ Nad99nF19xt+lg16d6tyl69/V7V/lpl3du979398OANk18lrN89wV42BEuwGhwlXdm1+Fst6 Fx1+ttd61t97NzNud/t3lz94GAN4eEdl99V5AhxYGBlxV512GF+CN+2Cd21y+Gt3V7t/+DuD 9q+EN82AuElr0uwGxYWId+OI2CFtV2lyeJeGl7eDOJ+HOD2HeKmAlwM4OH8uwHAjhSGLmFt+ dtGJGGOCmGd3GDFuuHGDl0GKWHmK2H1wle9BwkDCdZ+LuB+O+MF++MV/OJmMuP2Dd32QOD+Q eNt9OQ1mVmk47ChYGRmF2L+CV/GCuSePuDV72S+HV8WTV4uLCmoh87c+eReOt6OPGMOGV7GP lzWM1/2NGKWEGAd4mA2Et1JA1JdSuUOBuIuRt2WPOMeX2G2YGP+VuNOV+NeY2K+N9Hg2OW2U WXGI+R+JWSWMmVWM+QGV14OWGY+WVOSwGRWcWB2UmR2U2SOVGdGX+SuVmAGQWbeHuTl48uxq K8meeZ1+WXOcuU+Pd/efmVeKGTGNWYugWA99eBRE2hGImhWcmfGXmC2JuG+S2f+TOgOQmgeE 1HmFECWjeUeO2aOXePWXuh+aufuiWdl0Od2buTpFWINBGW+eumGGGmWaemuJ2YOdebOduk+T ei95GLR5Gl2ceUuJOhummC+iGdWbGYeneN2nqhOOWUGoOZ+e2mOSGkGVOrWpOrmgGiulGA5U KmuRFbJSGl+hd62c2fWampGa+f+r2lOuQlVmc+VcOu+j2CmvWh2rOm2iOYWYmEWuGEuwVsGW uw2qme+xOq+kOSmx2pWKet+p2yey1wz5Wy+oWvGzWfOxekWa2kmKOyGKu0USuWpUAjW0+suo eCO1WtOfexurev+puWM8u2uK+3GjuUu3mmezmdOtm4O0O4Yxm2ugxk+4+L2zOaWc+vmkef22 GwGuO2ujNBFaujm69x+5Wo2xmvu1+QO7+0dP+lhJe62aF6W9G7Wo+7mnF8O922lP+n+uolmw +5O7Ove/G127u9u4Wd+4lP+qI9O+es2+vAm1mzu4G73BTKvBlHmOMI3CG3Vg++3Au9W/Ox+Q dfGRM+glg+EiXFVDwmAi1Bm2ebxFTR832Om1Gj2q21erG1um+x+k26HBenpdA89fN5mzGs+x XHnCu522HIGyNltsfGYmsZlT11ucYGvJOzetW3/JuieAXKG6OglHnGuseem3OvHHW3u7fA+/ WrvDGQvMZF9AdLmZu8u+mXWtG5fLm9fBGnNvmd3KXIc/EZuqfHHLOonPW9PHuz2ttku/nDXG lUXD21PCfJe5uv3C/IOnhquWvIlcFBPAPJHCWou+/EfNvEvBXQXTqsvKmwvUPO+F3RG3fS25 mtfTN0GQdbFKU4+8kxE5XGGi2ZFr8/FROFehO83RPJXW3LvXHP9vvOGlOZJhMSXG/NHHPWvP nEmz+t3MIn8OOnolZGBhNxGFnHAGnLXHfZnPvN10XaOp9e/YuZmsm5Ge/NXPe33dnH+inb3T mlQlXanefM/evdPNfA3H3bnJ+2VJPcHOXcRpXFHemI3dHZXLffPbfR1knSG6Sss4E/PWHAW7 HUvEXRnC3BPTfOIkPjvgPAHWPD+vPi3NnhGtvQPhvlSmpU3iHluw/inWnkfCnTG9lkHE3eVc pCE33X/F5js9PotdVAvpp2BYFZVaWjXWE99HYyTM/Xlc3pimdWDi68nq+qNCim9RdHc6s+Yy RJfqdZgiFZxYFMsIrM/t9YVL9QXuVClA9aGFCmfVullb07UjETgjnwQ1nwgl41gxxhJGFD+6 glgw9aYj03wmIMU7wgjHJhJX86c/EPe+IhfKp84mXGQxf0HzNOJhIoMPfkNx4GXAnfWpQH/2 IH/jYlFDCZVxP1FOP1d6X1vkf1+bH2X2fd+Ev0vyQ0/3P1XUeCP3vRXW/oQEH4P2ggdDEItJ nznUXHGUv5mvX398H6Ok/03zYoPtxwncn8YhYmPGM8JR5cXOUjHspFRPUSWw/5mhmfP7uDoA AAAf//n/unYFwgA2HAgGEEEA1Fw3GwgGY3G4uGA0hg1HMQiRjNogBpCJINBQvK4gGUeF5IEA 4khCKBDEAxkhDKYgF5EMp2NJjMpHORhPMeJJEEAAAD/olFjxFjoKKMaOMaGIugUGgsIhUMh0 WiYzFwygcYjUckkhkcfk0oj8qlkuj8wmU0m04nU8n1AoVFowKpEepYNpoNp9RgtThMLhsPiM MGkPGoyEFejdJkEiklllMrlsvmMzms3nM7noKn9Bod2f9HpN7vt/geBg+Dq2GiQzGlPquOsE fsWTk+VtOYtubuGeuei0mlvGnplOqGrg1UwlXw4zGcL22Q3Nk3dny1qF9szVvzty0F00d201 65N+5dS1tVwtY6QwrcXjOPsOS7FmF9oy9rzK3M4uLPtCurSPOpT0tU9itBw2LoNiGQbBc+iv us/CSoO3j+u6/7gPDAbyLuvMEAa97owiFwYoWp4csYGgaQmEA5DKECYr4pynsW9kWMYGYZIf BqrK1F7Gvq27IxU3QZw07jvQA4LxNCH8pypA7URwF0dNZHiGBmirpyFCcKPs3CWhtJUmN878 AuE8YQSpKrjvRG6/RyxkthdFqJvkHCBobIcxyOkMkuxJbtN6/zfvBAThzhKcrQTO0dzzHobB lGM/zFIsKvvQkMUM/btzVJ8P0bR1IRLH4XSDTMXQlHyRRShcZxrBNKPY5yGBlS8/BtGCuyNC 1QMpQ8NydD1GTdArzTlEjU1u1lcx9XiJoqw7qvvYbs1DREOUVNkoxDA1myu9U9Wi11p1WrKt 2BTky20/T+SbDtFzbAjyxE5E6S5dD3V3dbZMUxlsXg3V5VFRM1yhEDiXHEdy365t04BPzZtd grI3jNOFVJZN8OLVFn3Pid/2o6TqWDbOD45b2F1LZV8qJkTlZIwWTYCGb5BlQNhZZYt6W/hj h2XfU55HO6DQZCGTxRnuVuwGuW2Pe1w4c42II1dU/RRFVY69F8Y1pGz5BoGwbBugkJ7PtOy7 ZGQz61rrCBmp7GPrH0JNjH1ZBANm5QlPsuq0GLGb/VKH7orSF8PHwcSzukYBggfGhkisdOlG FYcbnUJz82SIcprW6006XF790dL6900U8N1Kob31XGdfX0u52iXOdlXSKq51ES9JInHch3z4 89XXH8n4iueH4Xk8rxPj+HyvA653nRVTvXbPn5UJdX0nXAaMQVAa+SGhkGu1fN9Hyhv8+4a0 GKKqiGYcKfMAYq0xn5RlGgxAazxxKfkGp5MI/t+jfVaP+WeYBkpCyuGwBAbMhBUWMnXQwwhb rVFwMNaKzNcitoGM3gcDiCANAYEVT1BVC6xFuLGXrBtojMmsL7gWcyERIoSFYhMjBr0KixwX anC9obMWQwfX4etf0I4IJece09MsP4WLzVGshe64lmNZiPCE9sSj4ItdgptMhkYoLbikx2Kj VoOwzaOcqLSuYHnwBw9SMCgoVxkYSy5j0VWrtGRJG86McYvpcdAQNsakUstJIKgpniXzCA4K 0bVlSZVPEmh/GWPEZ2GpwZonWQ6kyooRT5AKR7KV3pITOdiSsd4NRDSknGLBqVJJ4k++1TEj mMSRlMbqVMGYhMwlao+I0NZPEDK4VpPT9JRxzQtGOXcLmhS+KBMCLEiwXJgmRJ06bxkuSFiz DaLZImzQRBjBNd0YYLRRlVL1j8Vo+MRiRA2cBC4TQoYJLic8dpeTPnXHuD0r42TejdOGHbfY fM/ha0Fl8+40ybQVEmeJDAcxNmUfeMcGJnUJj1QuYM/1cGuZ5AVS7taCn5iDPqjMMqGTvhvR 8hkgHBUjiA0CKbVYOUoiNSx+iEnayCV+/xWq/JYkGKekSaiYAaAycJPWUqg5ToYmbQiPLVpN Ubk4lqoSmpQKrIHUepNE5JVNkpSWjFUpXQ0SxVaRNWJaESq462r0uZUViqjJmssaz1RaqGRK Yqt62uFrfPep9M4YJumkvuos8mKq9m0eubgQgqP/MXASCIMCEV+BAFQIhfiW2bDk3EBoLwjA xBAQuy4ZgGgtPkDAGBDQQWXDGQay4dwQAoBjaoGATQxBwDmCmy4anyAgBaQ9IllygWpBna0K lrwUBTCKFIIgRQp28Cpb5tCmgWgxV+QO4lwLU1HuRcoIoRbpW+CLY99j7rzvrmq+19FnWtJ9 dCVZGFbLKmMBjE2Qj/SN2PhMpJ+hf7kWZtFgN99nwjGMtIFS01xrj2utmEULAUApBPCbeMBt 5XyJ5BiVU+QOcNkLw7h/At2CtI6hMnwg6lwanPVvAm/ZflLWSqO7fANvzWXuwyDC2t2rk3cI gDd5NsbZhGDSGIMocgQYWBahIHJC7gqau3agiAMr73ftgFS2QKAqE7DcHMnAaQ7ZHwthi9L6 b2ZmfdjiN57kvKrIk/hTThSEmMxcfIG5iwctqzu5bPWeMC5rMJZFWGcEiISfbT5/2BLRZqw9 W4GYNZyAgPq5Z+2gY5OH0pm4ib0NMRexWRNx7myNOWxKj0Grj3ruWvmn4ihWNOkIeDqddenX s6P1C+ByxD6/aP05qMHL0NH6X18kA2OkD16d2ADVGGh9kaO2U75y2tQav51SDnWGxVqad1WR PFTs3/g51LtzWevtursd7p0p6rNn7omroHZeuNG672MVFw74tv7EMRakgek9f6aZRZJw6NtV PGNlvrSWvtt7/ya77gW8UewmvjvzSqXcmbeRtY5/4OJ6GIMZcTAxjCHp6tLjnBNyitAzzHeZ deN7PYkmwDSY2CM52jeNi7lzmNHyd0NzIGl+QQaJ5hNhVVfud8057ogv3QXMX35n0VvXPn/Z 2MZbKzWBMcGycfodLz++ezabt1DF+j9twmfzjU1hBccWgwRci02UgYVIwbj1ndq+45ZcK+gJ oSA9AgCGEwJfKcMp3Kg2oiWOObmMctpF856wZHydqrTBZUKItqBt5Px3k/DdK8QDeRnB+k7h bM9dws8t6FOQjBEG3MXfP4422b1Th/WK3R/532HmiRec3a77e2HSF9UzL7+9eabPX9Pg53o+ cCo5y0PnUhOfs7fO+bnzHHxDo1Hsl8ggfT+kaKwLQM2LZpqt3I1QN2rAl2u+hNfUxDj2efos pv7rvC3DwmcCYgiqRP5uT5mbL+7uPxoTlbusOAPxraovwBP2v5gYintdvwNRAGjZvHPvr5v3 ONjZPXv/usuHlAP3CHj3OgnwP6GAugv8P/iEFWOgk9P8v1DZQJv8kYPygaCEEwQXFMQGnwN7 DZwFkenOtDj6wctHFVOLP/wOm6OpPPITQiFdNegGkbPvHtQev/wJkfQluLr+O3rJDpFLiJOP O1AQOQuOsFOSMrCCshAUOTslCrkdKswtrMCpEsu3m1Aaw4O0rQiWuDORu3GdM8sHO3ONLRQy gggxA3g6g6AQAmCagym/gng3AysLCCrrwJrtuzsCwuwCsgwww8gbseLXrUiHMeMsxDswxFRG O+A3g2g2g6g3Cbgwg6A0g3sugQAuAUMLQFNUMrigQ8pHQxrkMsgnAwg5gyAwg4sLGdRbsfMd AYOSsfLagcw2MsgdRDAigrAmAuLeA4A5A3ibAyAyrdrerTrsFrCJLrwwRcLUkVH0Q+LUvUx0 MsLZg5g0g2CbxXjGxXxGAxxWibA6A8gWRDAggnAQA5g7g0g6Axg0A0g3G4gwg3AyAQArx/AQ AwgxicA5xurprfxxsAxjnesHLUjDwyg5g3g2RCRXRYAzA3skSDAzg0AWg5g4AygyyGA6REgy gxxTSISFgQRGA6A7yTg1yDgzsLAqAVSNAaQ9seu3EIsEssgwg4A4R4gxxWSSA5gXRDRERRka Sag2gxSDxuAQSBg5yAA6EZyEA6A0SwSDybgwg2A3ygRvShRjmBQxiIMNx2SlycAyA0gzyBy1 DGg0g5Axg6yBgQRtx3gzg3R+SBSyyvA6SwSsxURVSoRWxXywA5g8g5yZA2yKrfS3xcyjROMf uNReLZgyg8SXA5RWg5gyx+A6A3gQRrg3gyA6icTBy8y9m/yFSGA2g0g8SYSWS8g3S+SQSRTJ Muygyhu3JxsrO3Aam0wyzEg0SvA3AxkZxfyYAQAzRUx7ySS1SBg8ybyGAzzqSZMkEZx4gwyt x4x9AWTjSNO4rXu3SizRAUEZg5g6g2RWyETom/A3g7sjgQTWA6TgzLTMTGg3zLgXMlMNxxRw DXLtsCO0LPQusEu2j5MqyOT5AuFdgasLHdQyQ2xJscSkHBMHAUAmAigiLVMlHdMnrhw20HsC spULMegUUMnz0OG+0PLij2UQrUiuRNrZ0TUUAY0VG+snxyLNrRUICNUY0f0aUNUbmvUcj2Ul LT0e0R0Z0gu30iGvUjSM0X0eEU0m0a0NxvUOyM0QLPURUm0suURvMNiKiqn4it0vUkUYUen3 R0iICGy6rZgnAyg6AhyZidzWMkRiUzRJK3ULwy0Mm6slG+UuHNUjwuleMePIiIpw0SAlA6La sdAWgsAsRHLgHT1ERkoSMrUSnCrVga1QS3Shw70ZgqApAigcsPAYT2UPLlAqgpggvAC+gbRJ gWnlk/CCmzowAXg0g2jGAiA3gGiAgA0KZW5kc3RyZWFtDQplbmRvYmoNCjIwIDAgb2JqDQo4 ODIzDQplbmRvYmoNCjIyIDAgb2JqDQo8PA0KL1R5cGUgL1hPYmplY3QNCi9TdWJ0eXBlIC9J bWFnZQ0KL05hbWUgL2ltMg0KL0ZpbHRlciAvTFpXRGVjb2RlIA0KL1dpZHRoIDMyOA0KL0hl aWdodCAxNTcNCi9CaXRzUGVyQ29tcG9uZW50IDgNCi9Db2xvclNwYWNlIFsgL0luZGV4ZWQg L0RldmljZVJHQiAyNTUgMjEgMCBSIF0NCi9MZW5ndGggMjMgMCBSDQo+Pg0Kc3RyZWFtDQqA O+BQOCQWDQeEQmFQuGQ2HQ+IRGJROKRWLReMRmNRuOR2PR+QSGRSOSSWTSeUSmVSuWS2XS+Y TGZTOaTWbTecTmdTueT2fT+gUGhUOiUWjUekUmlUumU2nU+oVGpVOqVWrVesVmtVuuV2vV+w WGxWOyWWzWe0Wm1Wu2W23W+4XG5XO6XW7Xe8Xm9Xu+X2/X/AYHBYPCYXDYfEYnFYvAFE+Y7I Y/JZHKZPLZXMZfNZnOZvPZ0o4yr5/SaDS6fTanUZfRVXVa/V7DZbHP62p7TZ7ncbvU7ao7rg bzhcHQ76ncPkcTk6jjU3lc/l9DMc2l9HrdLr9Sk9judfk9qj97u+PeeCi+T0eLY+ah+r0+/S +yg/D6e7N/Kf/b6/vJ2cAAAtr+QE+yRv/A0DwMm0DoLACzP1B8BsgkEEQpCqIQrBaLQQg0MQ 7DqGwonUIwhB6PQ9E8GxBCyLxDDkURehb/wYnMSRG9CORfGEVQ3DUWxdHMPoTFKCRkm8bRq9 SNxyd8lxjHiKwwhEgQ9IUGxTKybyRI7uIzE6GSChUnonMEGR0gcVw5M6BSvI0tTc8iMSoh8o zDDMxzJIkfS/BKDzZJk1JtN9BS5Hs0QvQ0fyKiUvT7PUVSlM87SGmUt0G5FC0dQ9M0jPlFzl RNOwvPtASZAFJplS1KuBTE7IpOlQTvT8yzFFlST/U6Y1VVLdyhRFXU3UtQodM1Z1bOM1wXRS aV3XTZV/X1n2NZFhT3ENaU5aUe1HNrlOq01o2zcFlWxXEqwta9p3DbVkRo56mPjT1gV7a90X NHl62hLtxps6CmtJWN63naWA1BTt8XkqrouOzmAWpfVaYJPM9YPiKp37hbNYbfeH2Nitg4hc N8qthSDWa2iCvuiNXxxR2Kzxil1Nc5aD2Y4WUMreOPZzYWXV9mGHNHd2S5q3Wbv7lWEVZceI 5XckpaTi2ZoNombIIy2d43jlW4JpunUbgNy3foSC5NourQlpGdaxZWwZFn8UKpi+jbLk+zj5 tcJ0Ntt5bfRipZJsmqODm+8I/le+7Dj9yynxKlblu26NnlHCxNlts67YvFymqnAIJyOq4FoE 48tanMczp8gbjqW58F0GNazFnSaXkWJaz2nObHz3W270PYd72eedvdPNbUp/HoH3fV8pJVET F01Ya/mPcO/ofP8H1/K3pSXha9gvG6j3nWetXnseZTN7+57vM996biIP8fw016W0+1BM8Sdo H0qh46BeS6S4nvrDWght57qHbNQb+7kgb8FLuvgC/hxDxXhwGdEzJ+LyIGKrgBA+CjIW/Pye I/OBL1HAwZfJBuDbYYPwgg7Bx4zyoMP+hJCh+jXFZQsejBVWzYoZwLhlBeB0HH7vrhFBSFr+ H9wwf7CY3LWobRDiI+yCEIXGFQc7D6JjdYnOMZ6/qHLG4uQ6PPEod8DSWG9fNGGMTw41p1fr GGK0CiBPuJct8jsaohQFWq7KLkSYgRlevHU2rhnNpzj1FNFseI2lCivHM4auTPEkigfh98ZI zEvYZJQnUjR3x0kgziTROX+SAddI4+klTKShXbD2Ux5XxH6lQZIsULixSjkvK2U71THlhikg 6OUnZHyvPfLFtcaoox9YBDWRUx4qklltIGLCWpURBma7WZcIJaRsjhMyQpI5OSehigKSsKZu rkjxNibLipzzcSaSOZ8pZgIRffOScs2pkQCgQ96e85ptzel/OCJag2WSKVPQSNsk43UEnZO0 kM72zO6RtGmg7pZDsfnxF6eyO19z5gkROb8wXIUCi2pqIjSKNUYnVLSjcEpFkUodCeaNEXR0 dUnSiKUK6Ty9TK+tT0/pWTxldSGmTSp0pDpQn+hL9oiz8THTyptDZLTQlwpamdS17UtkNASq 1KYnJZn/SCcKu3Y0de9IR9FOqmVVrQSil7koSs1qJWqtdF2DVWqOwWr1P6ASkjQRU1Tv65Tp kQleu0+ZELclvQGh9MV/kXr68uAD2XgyLrvU4mtH4NVCkHY6xr5a40DY7Uuys1rBTuqjPA2B GjQWArHVuPdk60WjnNXmvdQImzCZSRkztrLPkagHEW2UbLEVSsVTCqduyNyZs9bykb0LX1zg 7bSsFjFnW4OmRwzNzLI0SgPGu4M6rhzwr5ca21nbNXKog0e5d21IVZXUx52kD7DEosxUGsNq 5dWPsZdq9a05lQqhFb+PNrpnWnsXceSLU7q1vOLf2YtCIvuJi7Qyw6gav2ZvvZu9Jr5iXsd/ DewcU7n04qvaUkVbYtXUwTda80pr+YPorWmq+I8QYlulhjFWHMWSpvzimR2L7/41xta+weQo v3hwPcWt2G8W5Kx9J3IE6Ew10l60xvl7roElvrbe89ucmZLsDmHMWRMS4mtmvzA15LU4KsTh 7N2DojZlq7havV07x34wZeKyGcMpV0qTlmvGaI/2JzXjs1eY8o5YrW3vAmgbL4Xvtk69GOby X/0RnzOOEc5k1xRoXSkoNDaez3qPPuioj1yxvpHBGOsu3XzzH/KmZquW9yxqe1ugLTaDuJqL SWdtFaytHUdvum1l6Qy5hnSeq89U52BfOsrWNNao0Fm3Qmh82P/zfqRGez2G7RRjhXR+dccb K1Bp/SusdLqN28zvdbT9wbhuJuReGr8V7mwXtrX8YHL7OtJPqgsT6yZa1hbXXh2IaVEwprNc W/m4Zl1kSGMl5d769uzvTY86Nmz9xlcua8LeH8Q4HnY9ePck6x1xSl1NpOTt6nXhGrBKeI17 5Hxbme+dG0Kn5oBMHOOGcnJJzG8XNeKbWKMn6SVSFXFPzTmDZGXuh36lUTvpeT874a6fk3qM m9ddB1Z03qHWetbU3j0Lqtf+wH563yXsmquz9h7H285moWr9t7R2LtnZWM7X7j3Qn3U9ed41 d17vPfPCeF8N4fxHifFeL8Z43x3j/IeR8l5PynlfLeX8x5nzXmyckBANCmVuZHN0cmVhbQ0K ZW5kb2JqDQoyMyAwIG9iag0KMjIwMg0KZW5kb2JqDQoyMSAwIG9iag0KPDwNCi9MZW5ndGgg MjQgMCBSDQovRmlsdGVyIC9BU0NJSTg1RGVjb2RlIA0KPj4NCnN0cmVhbQ0KeiEjYmg7OWBQ MEQhISc+KiE5TzddcnUoazsxQjdEMiEmSEQyQWhzRlQxQj5XVCE8NWFuIStsMzJBY09XISEr b2dDDQpSOlpIQ0FjVmpdISElTkxKPS1CO1IvZiUyIStxVyFSQCsnIVIvbThuISEoNEMhNmJG IWJRKTghITFEJTJiZmcoZQ0Kam8+RF0hISokITFdSUdlcnJBSmUhNmtIQ3J0LGhUISNiaF0r c0pnISEmSERlMUdgVyExUlUkVGJWTSdUQWhzRiENCjFHYFchQW5FaWVBaSVZQzFdTmshITE+ VyFSNThIQzFSWFhlYmFDOWViVks3QzFHZDYyQiQ8JyFiVlJKZTFdUkoyDQohPDRpMnMiZTll MVJcOCFibDlMITZwc0VDOWBTMG48PC1WMiErbiJlQW5EJFRSL2dqQyErdTZlQWNPVyExR2BX VA0KMU0zJTJCJDwmQ3MoNWpDQW5FaTJBbkdYZVI6XidUQW5NKDJBY1M2MjFSVmllUjpfa1RC JD9aVHMzLCdUYltyWlQNCkFuSzghUkVUOWViXCRuVEFjVmpDMV1NJyFzKDddIUIkQzllcyom WzIhLlklTEo6TjAjIS5dVE1Ma3BpZVIvZDVUDQoxQjxoVCExQjVUUkVQWiFzImcmQzFSVSRU UjpcN0NSNTo4VDFSXDhUUi9naWUxTTMlZUFzbychUkVUOTJzLV04VA0KUkAsayFSOl9rVFJA MEplUkA0KiFSL2tJITFYKTghYmFGbUNSRVdtQ3M4U0plcy1ZXENSOmNKZVJLJl0hcy1gblsN ClRWU1RmV1c5QCpdPVlcal5xY2hpbixVWjQhNmQ1IWJbcGshUi9rSVQhNmtJIWJRJ0hDMUdk NmUxWCk3Q2JmaGxlDQpzKDlJVEIkPCZDYlt0SjJSOmFbZUIkQzpDYlErJ1QxUlpJIVJFVihl YmZsTCFzMy9bZWJmaGxlYlwjKUNSRVduIQ0KYmZwK2ViUS5bZTFdUFsyczMtbzJiZnArMnMx bUYqaDstbGVqb0ZpO21mMzFSITwzJCFzImFaVEFjVmshITw6N0MNCnM4T25UISZPWDIxXU0m Q3MtWVxDYlZUOyFCKV9qQ3MiZTllQW5NKDJCKWcoZXM4U01lITFFakNSSyRsZXMtXTtUDQpi YUpNMmJsN1tlcyJobiFCJEM6Q2JsPm8yczhXLSEhPDwnVHM4UV4ycy1gb2VibEBfQ3M4Vy0h czhXLSFzOFctIQ0KczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4 Vy0hfj4NCmVuZHN0cmVhbQ0KZW5kb2JqDQoyNCAwIG9iag0KOTg2DQplbmRvYmoNCjI4IDAg b2JqDQo8PA0KL0xlbmd0aCAyOSAwIFINCi9GaWx0ZXIgL0xaV0RlY29kZSANCj4+DQpzdHJl YW0NCoAURADSEVAaOBoLhoOBANxyLhtDBAVCIDRgIIvFzkZ4GLyMMhANhcORsIImZosLhgMx tJYmYxALRhKpZLiodxAKCcZToQzKbDKcjCdDechYICGTCSUBAYTcZBAUDkbzgdTYczSbzdTT kZRAdKEZDKbTCcjWcxAbzMICYZTtPxATzdXiGbzabTqbjSY6HWTcc6OSTcYxcICuYTWZRbWh aTCCThAabQYa/YbHZTXabWSSETcLJiaYTPe6iYTHiTpkcnlTDYrJZs0KYmagaNRpMRxCpNFI xNBmM93MJlKpXwInOBQQTIdqcYzLUCbe6mICJbr2ZcAIMFhBAQTcebSdDRQdZrsxaKKIK7oz mdKDz/Ll7Nk67X/GIDhUzhQTp4LU+wyjSOT1Ow/YxjoNK3rSO65jkObZCo2gioMOKBhokCMp iGIbhcGSGN+i6ahAMY2o6NI2uAIg3gagIA0KZW5kc3RyZWFtDQplbmRvYmoNCjI5IDAgb2Jq DQozNDUNCmVuZG9iag0KMjYgMCBvYmoNCjw8DQovVHlwZSAvWE9iamVjdA0KL1N1YnR5cGUg L0ltYWdlDQovTmFtZSAvaW0zDQovRmlsdGVyIC9MWldEZWNvZGUgDQovV2lkdGggMTA2DQov SGVpZ2h0IDQ0DQovQml0c1BlckNvbXBvbmVudCA4DQovQ29sb3JTcGFjZSBbIC9JbmRleGVk IC9EZXZpY2VSR0IgMjU1IDI1IDAgUiBdDQovTGVuZ3RoIDI3IDAgUg0KPj4NCnN0cmVhbQ0K gBXgUDgkFg0HhEJhULhkNh0PiERiUTikVi0XjEUAEbjkdj0fkEfjMjishk0nkMklUQlEtl0r mELl0zk8xm0Gmk5kEIEQAnotAEGHs+AA9hFAgVEn4AoFGK9FiM6qUcnEmglDjcIqlPqwintO h1TqcFjsEDlbpNQqtZK9YjYcFtnotYsENsVSgkcusDvUCr8GuVsulCK9xqN3nN8jd7vNsxcF t2Cx8jq0YstttEHntQv8Dt1YgWDykgEWjoOYtkJqmTtNQyVqjNWyNjrmtxlkrOdK+b0+v28a kAt1GIpMCuUM1dqt2K0Mii9WoGI1Nc2FarNN1uN5sey0gDnD6WejelhPLx+Z2uo38T2XSy/U hV9s+f3Hb9cSkOl2fE20I5bqPGqr7JI9r3OmvqCuitUFNO+r1QI0jwI48jkPe1qmO21kLO08 DeItArnLs9ArsCj0KOG28JwAjrhJLCL9wwoagLOoClrg+CtRo77IIc+8PpS3juJo9Kbpu/MJ I24USpQp6+KBGqmKhGUoxnBqCKQyCep7GgAA4owey0oknoPEEhMS8SUxW7jGy6tadzVDc4J9 JE2wYl7FSSGKsOEjj5t26bIoM3ikK807eJ7EksTYkUYKC4MuyjNa0x2s0cKugkgr3EbC0tSt FxZOjhTs60DurNiZMXUzooSzcRJ3Rs1KTKqjMy1jcSsoUAunTtb1cjzvxhJU1o/WkrVtU7VM c6rjqPXdSJFOzksukFaMa39Nzw7dkLbXsKo+4UYO/Uc5U1U08Q2zawOnaKC2ZVFXpS5yTLrY 9zvRWsMQve9nTI71QsLYbKrTa9+OzDk/1xK+C1vRiUyXYs0rTFq12bhLU2xXlk3gndiTlY1z YNil2rZVeKofMqy47iKuYJK11TbAVeZfhL4pTWGVRCrFKVvLje4Kx7Ay4sF3LDh14o9iC0Tj OigsDUk5UpPqMx9Hjys8kiwR9rMi65ruva/sCBoCDQplbmRzdHJlYW0NCmVuZG9iag0KMjcg MCBvYmoNCjczMg0KZW5kb2JqDQoyNSAwIG9iag0KPDwNCi9MZW5ndGggMzAgMCBSDQovRmls dGVyIC9BU0NJSTg1RGVjb2RlIA0KPj4NCnN0cmVhbQ0KeiEjLEQ1L2NZbEIhISRGLSEuK1xH UFM9KXQnRUFoIiEhIlZPL2hSLTQvY18sLCEhJEYtISs+altAL3NfRSEwaGcxDQohMGRIYFBR Mz4jIStEKWxQUkBqLCEjLEQ1J0dNNXEhIy0rXSo/Py4jL2NZbCgvY1tNNCElc0tnL2NbTFYv aFNiYg0KL2hXQXMvY108NC9uJEFAQDRtZS8vY18rZy9zSXRzUFYvMVgyKiFsYzU2KzRTOmYn dGc8RTIrZktgR05dIStASlsNCkA6OS5FUFYtQUUvbiRAYkA6OmNzUFtSdSNAOjpjc0A6PFNR UGEjU1ZQW1EyL0A6PkMvUFo7XidFY1ExI0hOOFxODQpLOVZJTyEwZEhgUFYpcyNAL3VPVi9z SD5nUFYrU1FANG1lL0A/XmIjUFYtQy9AOj5DYlBgdTA0UFYvMmJAP2QiQA0KczhXLSFzOFct IXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhX LSENCnM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0h czhXLSFzOFctIXM4Vy0hDQpzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4 Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIQ0KczhXLSFzOFctIXM4Vy0hczhXLSFzOFct IXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSENCnM4Vy0hczhXLSFz OFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0h DQpzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4 Vy0hczhXLSFzOFctIQ0KczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFct IXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSENCnM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFz OFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hDQpzOFctIXM4Vy0hczhX LSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIQ0K czhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hczhXLSFzOFctIXM4Vy0hfj4NCmVuZHN0 cmVhbQ0KZW5kb2JqDQozMCAwIG9iag0KOTg2DQplbmRvYmoNCjMxIDAgb2JqDQo8PA0KL0xl bmd0aCAzMiAwIFINCi9GaWx0ZXIgL0xaV0RlY29kZSANCj4+DQpzdHJlYW0NCoAURADYCA0K ZW5kc3RyZWFtDQplbmRvYmoNCjMyIDAgb2JqDQo2DQplbmRvYmoNCjE4IDAgb2JqDQo8PA0K L1R5cGUgL1BhZ2UNCi9QYXJlbnQgNSAwIFINCi9SZXNvdXJjZXMgPDwNCi9Gb250IDw8DQov RjEgNyAwIFIgDQovRjIgOCAwIFIgDQo+Pg0KL1hPYmplY3QgPDwNCi9pbTIgMjIgMCBSIA0K L2ltMyAyNiAwIFIgDQo+Pg0KL1Byb2NTZXQgMiAwIFINCj4+DQovQ29udGVudHMgWyAxOSAw IFIgMjggMCBSIDMxIDAgUiAgXQ0KPj4NCmVuZG9iag0KNiAwIG9iag0KPDwNCi9UeXBlIC9G b250DQovU3VidHlwZSAvVHJ1ZVR5cGUNCi9OYW1lIC9GMA0KL0Jhc2VGb250IC9BcmlhbA0K L0VuY29kaW5nIC9XaW5BbnNpRW5jb2RpbmcNCj4+DQplbmRvYmoNCjcgMCBvYmoNCjw8DQov VHlwZSAvRm9udA0KL1N1YnR5cGUgL1RydWVUeXBlDQovTmFtZSAvRjENCi9CYXNlRm9udCAv VGltZXNOZXdSb21hbixCb2xkDQovRW5jb2RpbmcgL1dpbkFuc2lFbmNvZGluZw0KPj4NCmVu ZG9iag0KOCAwIG9iag0KPDwNCi9UeXBlIC9Gb250DQovU3VidHlwZSAvVHJ1ZVR5cGUNCi9O YW1lIC9GMg0KL0Jhc2VGb250IC9UaW1lc05ld1JvbWFuDQovRW5jb2RpbmcgL1dpbkFuc2lF bmNvZGluZw0KPj4NCmVuZG9iag0KMTEgMCBvYmoNCjw8DQovVHlwZSAvRm9udA0KL1N1YnR5 cGUgL1RydWVUeXBlDQovTmFtZSAvRjMNCi9CYXNlRm9udCAvQXJpYWwsQm9sZA0KL0VuY29k aW5nIC9XaW5BbnNpRW5jb2RpbmcNCj4+DQplbmRvYmoNCjIgMCBvYmoNClsgL1BERiAvVGV4 dCAvSW1hZ2VCIC9JbWFnZUMgL0ltYWdlSSAgXQ0KZW5kb2JqDQo1IDAgb2JqDQo8PA0KL0tp ZHMgWzQgMCBSIDE4IDAgUiBdDQovQ291bnQgMg0KL1R5cGUgL1BhZ2VzDQovTWVkaWFCb3gg WyAwIDAgNjEyIDc5MiBdDQo+Pg0KZW5kb2JqDQoxIDAgb2JqDQo8PA0KL0NyZWF0b3IgKE1p Y3Jvc29mdCBQb3dlclBvaW50ICkNCi9DcmVhdGlvbkRhdGUgKE1vbmRheSwgTm92ZW1iZXIg MzAsIDE5OTggMTA6MTk6NDQgQU0pDQovVGl0bGUgKFByb2R1Y3QgQnJpZWYpDQovQXV0aG9y IChJUykNCi9Qcm9kdWNlciAoQWNyb2JhdCBQREZXcml0ZXIgMy4wMyBmb3IgV2luZG93cykN Ci9LZXl3b3JkcyAoKQ0KL1N1YmplY3QgKCkNCj4+DQplbmRvYmoNCjMgMCBvYmoNCjw8DQov UGFnZXMgNSAwIFINCi9UeXBlIC9DYXRhbG9nDQovRGVmYXVsdEdyYXkgMzMgMCBSDQovRGVm YXVsdFJHQiAgMzQgMCBSDQo+Pg0KZW5kb2JqDQozMyAwIG9iag0KWy9DYWxHcmF5DQo8PA0K L1doaXRlUG9pbnQgWzAuOTY0MyAxIDAuODI1MSBdDQovR2FtbWEgMS41OTk5IA0KPj4NCl0N CmVuZG9iag0KMzQgMCBvYmoNClsvQ2FsUkdCDQo8PA0KL1doaXRlUG9pbnQgWzAuOTY0MyAx IDAuODI1MSBdDQovR2FtbWEgWzEuNTk5OSAxLjU5OTkgMS41OTk5IF0NCi9NYXRyaXggWzAu NTM0NyAwLjI5MTkgMC4wMjUyIDAuMyAwLjY1MyAwLjExNzcgMC4xMjkgMC4wNTUgMC42ODEy IF0NCj4+DQpdDQplbmRvYmoNCnhyZWYNCjAgMzUNCjAwMDAwMDAwMDAgNjU1MzUgZg0KMDAw MDAyNjgxMyAwMDAwMCBuDQowMDAwMDI2NjU2IDAwMDAwIG4NCjAwMDAwMjcwNDAgMDAwMDAg bg0KMDAwMDAxMDYzNSAwMDAwMCBuDQowMDAwMDI2NzE0IDAwMDAwIG4NCjAwMDAwMjYxNjkg MDAwMDAgbg0KMDAwMDAyNjI4NCAwMDAwMCBuDQowMDAwMDI2NDEyIDAwMDAwIG4NCjAwMDAw MDAwMjEgMDAwMDAgbg0KMDAwMDAwNjk1NCAwMDAwMCBuDQowMDAwMDI2NTM1IDAwMDAwIG4N CjAwMDAwMDk0MjcgMDAwMDAgbg0KMDAwMDAwNjk3OCAwMDAwMCBuDQowMDAwMDA5NDAzIDAw MDAwIG4NCjAwMDAwMTA1MDEgMDAwMDAgbg0KMDAwMDAxMDUyNCAwMDAwMCBuDQowMDAwMDEw NjE0IDAwMDAwIG4NCjAwMDAwMjU5NjAgMDAwMDAgbg0KMDAwMDAxMDg0NiAwMDAwMCBuDQow MDAwMDE5NzUzIDAwMDAwIG4NCjAwMDAwMjIyMjYgMDAwMDAgbg0KMDAwMDAxOTc3NyAwMDAw MCBuDQowMDAwMDIyMjAyIDAwMDAwIG4NCjAwMDAwMjMzMDAgMDAwMDAgbg0KMDAwMDAyNDc1 MiAwMDAwMCBuDQowMDAwMDIzNzc1IDAwMDAwIG4NCjAwMDAwMjQ3MjkgMDAwMDAgbg0KMDAw MDAyMzMyMyAwMDAwMCBuDQowMDAwMDIzNzUyIDAwMDAwIG4NCjAwMDAwMjU4MjYgMDAwMDAg bg0KMDAwMDAyNTg0OSAwMDAwMCBuDQowMDAwMDI1OTM5IDAwMDAwIG4NCjAwMDAwMjcxMzcg MDAwMDAgbg0KMDAwMDAyNzIyNSAwMDAwMCBuDQp0cmFpbGVyDQo8PA0KL1NpemUgMzUNCi9S b290IDMgMCBSDQovSW5mbyAxIDAgUg0KL0lEIFs8YjU4MmU5OGJiMzkzNmI3Y2ZiYjM2OTFi MWMzYjA0YTU+PGI1ODJlOThiYjM5MzZiN2NmYmIzNjkxYjFjM2IwNGE1Pl0NCj4+DQpzdGFy dHhyZWYNCjI3Mzk3DQolJUVPRg0K --part1_c8.75a95b4.269c09f4_boundary-- From owner-ips@ECE.cmu.edu Tue Jul 11 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31663; 11 Jul 2000 13:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA21404 for ips-outgoing; Tue, 11 Jul 2000 12:02:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA21314 for ; Tue, 11 Jul 2000 12:00:32 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id MAA26597; Tue, 11 Jul 2000 12:00:31 -0400 Message-Id: <200007111600.MAA26597@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: iSCSI (Internet SCSI) Requirements Date: Tue, 11 Jul 2000 12:00:31 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : Author(s) : R. Haagens Filename : draft-haagens-ips-iscsireqs-00.txt Pages : 21 Date : 10-Jul-00 This document explains the motivation behind an efficient transport of SCSI commands on top of TCP/IP and describes scenarios where such a transport will be used. The document also enumerates and discusses requirements for supporting SCSI on top of IP. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-haagens-ips-iscsireqs-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-haagens-ips-iscsireqs-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-haagens-ips-iscsireqs-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. - --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" - --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20000710151524.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-haagens-ips-iscsireqs-00.txt - --OtherAccess Content-Type: Message/External-body; name="draft-haagens-ips-iscsireqs-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20000710151524.I-D@ietf.org> - --OtherAccess-- - --NextPart-- ------- End of Forwarded Message From owner-ips@ECE.cmu.edu Tue Jul 11 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32693; 11 Jul 2000 18:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA02188 for ips-outgoing; Tue, 11 Jul 2000 17:11:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch11.nsmg.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA02149 for ; Tue, 11 Jul 2000 17:10:45 -0400 (EDT) Received: by lmoxch11.nsmg.veritas.com with Internet Mail Service (5.5.2448.0) id ; Tue, 11 Jul 2000 17:11:22 -0400 Message-ID: From: Roger Cummings To: "'mjacob@feral.com'" , "'Bill Main'" Cc: "'ips@ece.cmu.edu'" Subject: RE: iSCSI has already been done before, sort of... Date: Tue, 11 Jul 2000 17:11:15 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, Bill, There is no iIPI document that I've ever seen. However the definition of IPI-3 on "HIPPI Raw framing protocol" is defined in section A.2 of the latest IPI-3 Disk standard - ANSI X3.291:1997, the Revised Device Generic Command Set for Magnetic and Optical Disk Drives. The standard can be purchased in paper for from the usual sources, and online as a PDF file from http://www.ncits.org. Details of all IPI projects are still available from the link that Bill gave, http://www.t11.org, which is the T11 committee web site that I happen to maintain - follow the "IPI" links. I was the editor of the IPI-3 Tape standard, and I worked on IPI very actively from 1984 thru the early 1990s. IPI was mostly used by large system companies such as IBM and what used to be known as the BUNCH (Univac, Control Data, Honeywell etc.), because its architecture used some of the same principles as their previous proprietary channels. With the demise of the BUNCH in the early 90s, IPI basically faded away and was replaced by SCSI for server storage. Maximum Strategy certainly did have an IPI-3 over HIPPI and over FC product, and I think that they had plans for a IPI-3 over TCP/IP product as well. I still have contact with some of the folks that worked there, so I'll see if I can find anything that will be useful for iSCSI. The reason why it was fairly straightforward to create definitions of how IPI-3 was transported on the multiple interfaces was that IPI-3 was conceived from the start as a "packetized" protocol. This was most certainly NOT originally true of SCSI, which has the message system and other physical layer signaling aspects. However in the last 10 years T10 has put a LOT of effort into the SCSI-3 architecture to enable SCSI to be "packetized" and work across multiple transports. A number of people who have been involved in this process have been up front about using the functionality of IPI-3 as guidance for that effort. It can certainly be argued that things like status reporting and error recovery are unnecessarily complex in SCSI because of its heritage. However the fact is that SCSI, complexity and all, is implemented in every major server OS, and that it made eminent sense in the mid 1990s to take advantage of that support to speed the adoption of Fibre Channel. Supporting essentially the same protocol over parallel SCSI & FC enabled SAN configurations that take the best advantage of both technologies, through the equivalent of bridges and routers. I submit that it makes equal sense to take the same approach for IP Storage. Bill's list of the attributes of IPI-3 is essentially correct, but I'd like to add a few comments: 1) IPI-3 certainly had the concept of a "Transfer Notification" that preceded a data transfer, and even allowed information such as address and length to be included, but I don't think we ever did standardize the format of that information. 2) The IPI-3 packet format only allocated a byte each for the "Target" and LUN address, which would clearly not be sufficient for today's SANs, and changing those size would have a major impact on the protocol because a number of the protocol aspects assume a fixed length header. 3) IPI-3 did have multiple port support, but this was achieved with simple bit masks that clearly would not work for multiple different transport technologies in today's SANs. 4) IPI-3 predated the use of worldwide unique identifiers for configuration discovery in storage. 5) There has never been any IPI-3 "controller chips" at any level approaching the level of integration and complexity of today's intelligent SCSI silicon, as far as I know. The highest level of integration achieved was a single chip physical interface state machine. Therefore, despite its technical excellence (and there are still a number of people around who will confirm that it was a good technical solution), I cannot believe that IPI-3 is a viable candidate for a 21st century storage protocol over any transport mechanism. Regards, Roger Cummings Veritas Software roger.cummings@veritas.com -----Original Message----- From: Matthew Jacob [mailto:mjacob@feral.com] Sent: Friday, July 07, 2000 11:56 PM To: Jim McGrath Cc: 'Black_David@emc.com'; wmain@gis.net; ips@ece.cmu.edu Subject: RE: iSCSI has already been done before, sort of... > > > I think the question is whether there is a document similar to iSCSI for IPI > over a network that we could learn from. My problem with the actual IPI-3 > document is that I believe it is similar in scope to the SCSI documents > themselves - good at defining the protocol for the traditional storage > model, but not good at highlighting the changes/issues associated with > putting it over a network. > > If there was a iIPI (IPI-3 over some network) document(s) out there, then > that could provide some useful guidance. > > If I am mistaken, and the actual IPI-3 documents provide this level of > guidance, then it would probably help others to identify the appropriate > sections, since the document itself is rather large. I don't believe that there have ever been any public specs in this area. I know of a couple of implementations of IPI-3 over both TCP/IP and Hippi raw framing protocol- but I don't recall that they got to any kind of standardization level (possibly in the Hippi specs somewhere- I don't have them handy). Anybody on the list know anyone from MaxStrat (mass storage for SuperComputer's company- maybe bought by Sun???? I think they may have also done this as well. -matt From owner-ips@ECE.cmu.edu Wed Jul 12 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00858; 11 Jul 2000 23:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA09048 for ips-outgoing; Tue, 11 Jul 2000 22:27:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ophidian.com (ophidian.com [209.136.122.245] (may be forged)) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA06699 for ; Tue, 11 Jul 2000 20:13:31 -0400 (EDT) Received: from boa [216.84.97.97] by ophidian.com (SMTPD32-5.05) id AB1A1D4017A; Tue, 11 Jul 2000 18:26:02 -0600 Message-ID: <005b01bfeb96$870de600$0302010a@boa> From: "Edward A. Gardner" To: "IPS Reflector" Subject: Pittsburgh schedule? Date: Tue, 11 Jul 2000 18:17:04 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Does anyone know when we can expect to see the schedule for Pittsburgh? The ietf web site says that scheduling closed July 7, but gives no hint when they might be posted. Simply knowing when they will be posted and where we should be looking would be useful. Edward A. Gardner eag@ophidian.com Ophidian Designs 719 593-8866 voice 1262 Hofstead Terrace 719 593-8989 fax Colorado Springs, CO 80907 719 210-7200 cell From owner-ips@ECE.cmu.edu Wed Jul 12 17:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03791; 12 Jul 2000 13:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA03237 for ips-outgoing; Wed, 12 Jul 2000 12:00:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03062 for ; Wed, 12 Jul 2000 11:54:59 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id <333486D5>; Wed, 12 Jul 2000 11:54:27 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB06@corpmx9.isus.emc.com> To: eag@ophidian.com, ips@ece.cmu.edu Subject: RE: Pittsburgh schedule? Date: Wed, 12 Jul 2000 11:54:25 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Does anyone know when we can expect to see the schedule for Pittsburgh? The > ietf web site says that scheduling closed July 7, but gives no hint when > they might be posted. Simply knowing when they will be posted and where we > should be looking would be useful. Soon, I hope. What closed on July 7th was requests for time/space; the actual scheduling is in progress. Everyone should be aware that IETF agendas are living documents and can change even after they are initially posted - I know from personal experience that in the past the agenda has changed in the two weeks prior to the meeting. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Jul 12 18:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03886; 12 Jul 2000 13:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA05268 for ips-outgoing; Wed, 12 Jul 2000 12:56:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemail2.firewall.lucent.com (hoemail2.lucent.com [192.11.226.163]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA04344 for ; Wed, 12 Jul 2000 12:31:20 -0400 (EDT) Received: from hoemail2.firewall.lucent.com (localhost [127.0.0.1]) by hoemail2.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id MAA27895 for ; Wed, 12 Jul 2000 12:31:19 -0400 (EDT) Received: from il0015exch001h.wins.lucent.com (h135-1-23-83.lucent.com [135.1.23.83]) by hoemail2.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id MAA27887 for ; Wed, 12 Jul 2000 12:31:19 -0400 (EDT) Received: by il0015exch001h.ih.lucent.com with Internet Mail Service (5.5.2650.21) id ; Wed, 12 Jul 2000 11:31:19 -0500 Message-ID: <80B684C5E29FD211AA8000A0C9CDD91904D09736@IL0015EXCH005U> From: "Rodriguez, Elizabeth G (Elizabeth)" To: "'bill.main@bigfoot.com'" , ipSCSI list Subject: RE: Question on TCP hardware up lift Date: Wed, 12 Jul 2000 11:31:17 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Lucent Technologies Optical Area Networking Group is developing products and silicon that implement TCP, UDP, and IP in hardware. Product availability is late 2000. For information, please contact: Rob Preece Product Manager Lucent Technologies Optical Area Networks email: robpreece@lucent.com For more information about OAN products: OAN Info Line: (972) 231-2680 EMail: oaninfo@lucent-optical.com Web Site: www.lucent-optical.com/oan -----Original Message----- From: Bill Main [mailto:wmain@gis.net] Sent: Monday, July 10, 2000 12:32 PM To: ipSCSI list Subject: Question on TCP hardware up lift Is there a list any where of companies currently in the TCP hardware busines? Regards, Bill Main From owner-ips@ECE.cmu.edu Wed Jul 12 18:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03888; 12 Jul 2000 13:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA05344 for ips-outgoing; Wed, 12 Jul 2000 12:58:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA05291 for ; Wed, 12 Jul 2000 12:57:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA246032; Wed, 12 Jul 2000 18:56:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id SAA42700; Wed, 12 Jul 2000 18:56:46 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125691A.005D14F4 ; Wed, 12 Jul 2000 18:56:42 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: ips@ece.cmu.edu Message-ID: Date: Wed, 12 Jul 2000 18:58:51 +0300 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Can you shed some light on what the IETF schedule for our group will be? Also I sent the iSCSI draft two days ago and I don't see it appearing. Is it considered bad form to send it twice? Regards, Julo From owner-ips@ECE.cmu.edu Wed Jul 12 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04480; 12 Jul 2000 16:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA13118 for ips-outgoing; Wed, 12 Jul 2000 15:35:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA08868 for ; Wed, 12 Jul 2000 14:28:45 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id OAA01397; Wed, 12 Jul 2000 14:28:37 -0400 (EDT) Date: Wed, 12 Jul 2000 14:28:37 -0400 (EDT) From: Scott Bradner Message-Id: <200007121828.OAA01397@newdev.harvard.edu> To: Black_David@emc.com, julian_satran@il.ibm.com Subject: > Also I sent the iSCSI draft two days ago and I don't see it appearing. Cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Precedence: bulk the closer to the deadline that we get the bigger the lag due to the larger number of IDs that are being sent in Scott From owner-ips@ECE.cmu.edu Thu Jul 13 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06508; 13 Jul 2000 3:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA05990 for ips-outgoing; Thu, 13 Jul 2000 01:45:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA05876 for ; Thu, 13 Jul 2000 01:39:52 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA133420 for ; Thu, 13 Jul 2000 07:39:18 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id HAA216790 for ; Thu, 13 Jul 2000 07:39:17 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125691B.001F0F67 ; Thu, 13 Jul 2000 07:39:15 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 13 Jul 2000 08:37:56 +0300 Subject: FYI - draft-satran-iscsi-01.txt published Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk FYI ---------------------- Forwarded by Julian Satran/Haifa/IBM on 13/07/2000 08:36 --------------------------- Internet-Drafts@ietf.org on 12/07/2000 13:31:19 Please respond to Internet-Drafts@ietf.org To: IETF-Announce: ; cc: (bcc: Julian Satran/Haifa/IBM) Subject: I-D ACTION:draft-satran-iscsi-01.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : iSCSI (Internet SCSI) Author(s) : J. Satran, D. Smith, K. Meth, C. Sapuntzakis, R. Haagens, E. Zeidner, P. Stamwitz, L. Ore Filename : draft-satran-iscsi-01.txt Pages : 61 Date : 11-Jul-00 The Small Computer Systems Interface (SCSI) is a popular family of protocols for communicating with I/O devices, especially storage devices. This memo describes a transport protocol for SCSI that operates on top of TCP. The iSCSI protocol aims to be fully com- pliant with the requirements laid out in the SCSI Architecture Model - 2 [SAM2] document. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-satran-iscsi-01.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-satran-iscsi-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-satran-iscsi-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. mailto:mailserv@ietf.org ftp://ftp.ietf.org/internet-drafts/draft-satran-iscsi-01.txt From owner-ips@ECE.cmu.edu Thu Jul 13 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08576; 13 Jul 2000 12:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA26594 for ips-outgoing; Thu, 13 Jul 2000 10:47:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA26025 for ; Thu, 13 Jul 2000 10:29:56 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id <3SR1L7HD>; Thu, 13 Jul 2000 10:29:22 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB18@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: Pittsburgh schedule? Date: Thu, 13 Jul 2000 10:29:18 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk The preliminary agenda for Pittsburgh is now available. It shows IP Storage (still as a BOF, we're not a WG yet) meeting Wednesday afternoon (August 2) 1530-1730. Please note that this is subject to change. An initial draft of the charter should appear this week, and there should be a preliminary agenda and the associated electronic agenda bashing on this list next week. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Jul 14 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11669; 14 Jul 2000 3:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA09421 for ips-outgoing; Fri, 14 Jul 2000 02:23:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA05849 for ; Thu, 13 Jul 2000 14:33:11 -0400 (EDT) Received: from mail.nuspeed.com (mail [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 4237D837F3 for ; Thu, 13 Jul 2000 13:30:34 -0500 (CDT) Received: from nuspeed.com (marvin.eng.nuspeed.com [172.18.48.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 34Y8PX74; Thu, 13 Jul 2000 13:39:18 -0500 Message-ID: <396E0BA9.7AD1077E@nuspeed.com> Date: Thu, 13 Jul 2000 13:34:17 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: IPS Subject: iSCSI MIB draft is available Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk The iSCSI MIB draft has been posted, and is available at: http://www.ietf.org/internet-drafts/draft-bakke-iscsimib-00.txt The draft is essentially the same as the one sent to the mailing list a few days ago, with a few minor fixes. -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Sat Jul 15 02:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15476; 14 Jul 2000 22:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA03884 for ips-outgoing; Fri, 14 Jul 2000 21:27:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA00044 for ; Fri, 14 Jul 2000 19:50:09 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id 08800179 for ; Fri, 14 Jul 2000 19:50:09 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id QAA18051 for ; Fri, 14 Jul 2000 16:50:07 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA892 for ; Fri, 14 Jul 2000 16:50:04 -0700 Message-ID: <396FA72A.F8593E8C@agilent.com> Date: Fri, 14 Jul 2000 16:50:03 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: iSCSI Subject: iSCSI Liason Report presented at T10 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk The iSCSI Liason Report presented by Ed Gardner at the T10 meeting is available at ftp://ftp.t10.org/t10/document.00/00-280r0.pdf From owner-ips@ECE.cmu.edu Sat Jul 15 02:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15481; 14 Jul 2000 22:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA03472 for ips-outgoing; Fri, 14 Jul 2000 21:19:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA09314 for ; Fri, 14 Jul 2000 12:18:09 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id <3SR1MPB8>; Fri, 14 Jul 2000 11:58:42 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB2A@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: IPS Draft Charter Date: Fri, 14 Jul 2000 11:58:36 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk As promised, here's the current charter draft that the co-chairs and ADs have been working on is attached. A couple of important notes: - The scope is considerably broader than iSCSI; this is deliberate. - The schedule/milestones is/are particularly drafty. Comments and suggestions are encouraged, promptly so there's a reasonable chance of having a close- to-final revision prior to Pittsburgh. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- IP Storage (ips) Working Group Proposed Charter - DRAFT There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for the shared traffic environment of the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks that support other services and protocols. - Management, including appropriate MIB definition. The WG will not require either modifications to common transport protocols such as TCP and IP or presence of options that are not widely supported, although the WG may recommend use of such options for block storage traffic. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP-transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Launch framework discussion to select encapsulation approach or approaches. Sep 00 Publish version of framework draft reflecting WG consensus on encapsulation approach or approaches. Dec 00 Discuss framework, and first set of requirements, applicability and protocol specification drafts at IETF meeting in San Diego. Jan 01 Submit framework and first set of requirements, applicability, and protocol specification drafts to IESG. Mar 01 Discuss MIB and any ancillary drafts required to use the specified protocols at IETF meeting in Minneapolis. May 01 Submit MIB draft to IESG. Aug 01 Meet at IETF to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Sun Jul 16 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23063; 16 Jul 2000 10:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA17313 for ips-outgoing; Sun, 16 Jul 2000 09:38:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA05630 for ; Sun, 16 Jul 2000 04:50:13 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA215910; Sun, 16 Jul 2000 10:49:41 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id KAA17484; Sun, 16 Jul 2000 10:49:41 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125691E.00307CD9 ; Sun, 16 Jul 2000 10:49:36 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: ips@ece.cmu.edu Message-ID: Date: Sun, 16 Jul 2000 11:48:19 +0300 Subject: FCP over IP item on the agenda Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I've seen the proposed agenda of the BOF on the IETF site. I think that we have far less time than we asked for but we may try to make up for this. I was also under the impression that we are going to discuss thing that where already published on the IETF and /or IPS board. Considering the limited time we have - and the fact that we went through the FCP over IP thing several times in public forums and designer group I wonder why it appears as a tentative item on your agenda. If we have to consider new facts we have to have time to digest them and we did not see the yet on any forum (unlike all other items). I feel very uneasy about having to give up precious time during the only face-to-face session we have to an item that has yet to be brought to us. I think this is also contrary to the IETF practice - or the little I've got to know about it. Again I have nothing against discussing again FCP over IP - but in order to see if there are arguments for it that we did not consider before I think we need time off line and getting it a first hearing during the only session we have will do a disservice to the community. Regards, Julo From owner-ips@ECE.cmu.edu Mon Jul 17 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29608; 17 Jul 2000 17:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA15721 for ips-outgoing; Mon, 17 Jul 2000 16:11:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA08786 for ; Mon, 17 Jul 2000 13:22:45 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id 30402DC for ; Mon, 17 Jul 2000 13:22:44 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id KAA00906 for ; Mon, 17 Jul 2000 10:22:42 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA28A2 for ; Mon, 17 Jul 2000 10:22:40 -0700 Message-ID: <397340CD.609EAE66@agilent.com> Date: Mon, 17 Jul 2000 10:22:21 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: iSCSI Subject: The posted IPS agenda Content-Type: multipart/mixed; boundary="------------E532189CCBDB8AF36BB53D3D" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------E532189CCBDB8AF36BB53D3D Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit http://www.ietf.org/ietf/00aug/ips-agenda.txt --------------E532189CCBDB8AF36BB53D3D Content-Type: text/plain; charset=us-ascii; name="ips-agenda.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="ips-agenda.txt" IP Storage BOF (ips) Wednesday, August 2 at 1530-1730 ================================ CHAIRS: David Black Steven Bellovin DESCRIPTION: An IP Storage working group is being formed based on the IP Storage BOF held in Adelaide because there is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols (SCSI and/or Fibre Channel) in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, and discovery), as opposed to modifying the existing block storage protocols. The protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Use of an IP-based transport raises issues that do not occur in existing storage transports. At least the following must be addressed by this effort: - Congestion control suitable for the shared traffic environment of the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks that support other services and protocols. - Management, including appropriate MIB definition. The WG will not require either modifications to common transport protocols such as TCP and IP or the presence of options that are not widely supported unless there is no other reasonable alternative, although the WG may recommend use of such options for block storage traffic. AGENDA: Agenda bashing - co-chairs A Few Words From The Area Director - Allison Mankin Charter presentation and discussion - co-chairs This will be the proposed charter if the IESG has not approved it prior to the meeting. Framework remarks and discussion - David Black Between SCSI, Fibre Channel, end systems and gateways, there are multiple problems to be solved. Internet-Draft presentations iSCSI requirements iSCSI Fibre Channel over IP (? tentative ?) MIB Security remarks and discussion - Steve Bellovin Further discussion and organization of items to be worked on - co-chairs. MAILING LIST: The mailing list is at ips@ece.cmu.edu, to subscribe send "subscribe ips" as the text to majordomo@ece.cmu.edu A web site has been setup at http://www.ece.cmu.edu/~ips/ --------------E532189CCBDB8AF36BB53D3D-- From owner-ips@ECE.cmu.edu Mon Jul 17 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29606; 17 Jul 2000 17:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA15474 for ips-outgoing; Mon, 17 Jul 2000 16:06:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03750 for ; Mon, 17 Jul 2000 11:05:18 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id <3SR1NAYW>; Mon, 17 Jul 2000 11:04:42 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB41@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: FCP over IP item on the agenda Date: Mon, 17 Jul 2000 11:04:31 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: Thanks for your comments. Here's an attempt to provide the missing background/explanations: - The ips agenda for Pittsburgh posted on the IETF web site is very drafty -- something had to be invented on short notice to get the space scheduled. It is definitely subject to change. - One of the things that needs to be done this week is to come up with a real agenda, and the first step is agenda bashing on the mailing list -- thanks for starting this. It would be good if the agenda were mostly finalized this week. - The draft charter has been expanded in scope from what was previously discussed based on direction from the ADs. Both SCSI and Fibre Channel protocols are noted, and a framework document has been added. The FC over IP work is at least relevant to the framework discussion. The WG is free to decide whether or not to undertake FC over IP, and for what purposes. - There is a draft on the FC over IP work, draft-ietf-ipfc-fcoverip-01.txt Despite the "ietf-ipfc" draft name, it may be the case that this work is to be pursued in the ips WG if at all; further clarification should be forthcoming from the ADs. I appreciate the concern about needing the opportunity to review drafts prior to the meeting -- hence the goal of getting the agenda done this week, complete with draft names. - Please be careful with terminology - FC over IP encapsulates FC-2; this is the basic Fibre Channel frame protocol, and is at a lower layer in the FC protocol stack than FCP (the Fibre Channel encapsulation of SCSI). At the moment, this is not a source of confusion, but a stitch in time ... Finally, when something is discussed among the design team, the result is a design team recommendation, which is not to be confused with WG rough consensus -- IETF operates by rough consensus of the WG, usually on the WG mailing list. One of the things that would be a good idea in Pittsburgh is to put the design team's recommendations in front of the audience to check what there is and is not consensus for. Thanks, --David p.s. From a procedural viewpoint, I want to encourage people to do what Julian has done -- if you think there's something wrong with the way the WG is being run, please bring it up early and by whatever means you're comfortable with. Nobody can promise to make everyone happy all the time, but I'd at least like to know what people are concerned about so that I can do something about it. --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Sunday, July 16, 2000 4:48 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: FCP over IP item on the agenda > > > > David, > > I've seen the proposed agenda of the BOF on the IETF site. > I think that we have far less time than we asked for but we may try to > make > up for this. > > I was also under the impression that we are going to discuss thing that > where > already published on the IETF and /or IPS board. > > Considering the limited time we have - and the fact that we went through > the FCP > over IP thing several times in public forums and designer group I wonder > why > it appears as a tentative item on your agenda. > > If we have to consider new facts we have to have time to digest them and > we > did not see the yet on any forum (unlike all other items). > > I feel very uneasy about having to give up precious time during the only > face-to-face > session we have to an item that has yet to be brought to us. > I think this is also contrary to the IETF practice - or the little I've > got > to know about it. > > Again I have nothing against discussing again FCP over IP - but in order > to see if there > are arguments for it that we did not consider before I think we need time > off line > and getting it a first hearing during the only session we have will do a > disservice > to the community. > > Regards, > Julo > From owner-ips@ECE.cmu.edu Mon Jul 17 22:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29615; 17 Jul 2000 17:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA15996 for ips-outgoing; Mon, 17 Jul 2000 16:17:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA15355 for ; Mon, 17 Jul 2000 16:03:58 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id D5117257 for ; Mon, 17 Jul 2000 16:03:57 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id NAA13175 for ; Mon, 17 Jul 2000 13:03:56 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA2CF0; Mon, 17 Jul 2000 13:03:49 -0700 Message-ID: <39736693.3C376D5@agilent.com> Date: Mon, 17 Jul 2000 13:03:31 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: FCP over IP item on the agenda References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Notice that FC over IP is also being discussed in the IP over FC WG (for some reason): http://search.ietf.org/ietf/00aug/ipfc-agenda.txt -Matt julian_satran@il.ibm.com wrote: > David, > > I've seen the proposed agenda of the BOF on the IETF site. > I think that we have far less time than we asked for but we may try to make > up for this. > > I was also under the impression that we are going to discuss thing that > where > already published on the IETF and /or IPS board. > > Considering the limited time we have - and the fact that we went through > the FCP > over IP thing several times in public forums and designer group I wonder > why > it appears as a tentative item on your agenda. > > If we have to consider new facts we have to have time to digest them and we > did not see the yet on any forum (unlike all other items). > > I feel very uneasy about having to give up precious time during the only > face-to-face > session we have to an item that has yet to be brought to us. > I think this is also contrary to the IETF practice - or the little I've got > to know about it. > > Again I have nothing against discussing again FCP over IP - but in order to > see if there > are arguments for it that we did not consider before I think we need time > off line > and getting it a first hearing during the only session we have will do a > disservice > to the community. > > Regards, > Julo From owner-ips@ECE.cmu.edu Tue Jul 18 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30508; 17 Jul 2000 22:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA29427 for ips-outgoing; Mon, 17 Jul 2000 21:15:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA28985 for ; Mon, 17 Jul 2000 20:59:21 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id RAA00677; Mon, 17 Jul 2000 17:57:28 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id RAA13623; Mon, 17 Jul 2000 17:57:04 -0700 From: "Douglas Otis" To: "Matt Wakeley" , Cc: , Subject: RE: FCP over IP item on the agenda Date: Mon, 17 Jul 2000 18:01:16 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <39736693.3C376D5@agilent.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk You will notice that this work group is developing standards for IP over Fibre-Channel whereas the iSCSI is Fibre-Channel over IP. Similar but different. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Matt Wakeley Sent: Monday, July 17, 2000 1:04 PM To: julian_satran@il.ibm.com Cc: Black_David@emc.com; ips@ece.cmu.edu Subject: Re: FCP over IP item on the agenda Notice that FC over IP is also being discussed in the IP over FC WG (for some reason): http://search.ietf.org/ietf/00aug/ipfc-agenda.txt -Matt julian_satran@il.ibm.com wrote: > David, > > I've seen the proposed agenda of the BOF on the IETF site. > I think that we have far less time than we asked for but we may try to make > up for this. > > I was also under the impression that we are going to discuss thing that > where > already published on the IETF and /or IPS board. > > Considering the limited time we have - and the fact that we went through > the FCP > over IP thing several times in public forums and designer group I wonder > why > it appears as a tentative item on your agenda. > > If we have to consider new facts we have to have time to digest them and we > did not see the yet on any forum (unlike all other items). > > I feel very uneasy about having to give up precious time during the only > face-to-face > session we have to an item that has yet to be brought to us. > I think this is also contrary to the IETF practice - or the little I've got > to know about it. > > Again I have nothing against discussing again FCP over IP - but in order to > see if there > are arguments for it that we did not consider before I think we need time > off line > and getting it a first hearing during the only session we have will do a > disservice > to the community. > > Regards, > Julo From owner-ips@ECE.cmu.edu Tue Jul 18 05:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31593; 18 Jul 2000 1:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA02662 for ips-outgoing; Mon, 17 Jul 2000 23:52:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA01566 for ; Mon, 17 Jul 2000 22:54:20 -0400 (EDT) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id TAA09300; Mon, 17 Jul 2000 19:53:49 -0700 (PDT) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id TAA04512; Mon, 17 Jul 2000 19:53:48 -0700 (PDT) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Mon, 17 Jul 2000 19:53:47 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C05B31C58@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: "'Black_David@emc.com'" Cc: ips@ece.cmu.edu Subject: Some more agenda bashing Date: Mon, 17 Jul 2000 19:53:40 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I propose to add a couple more items on the agenda. 1. SEP - current status We are evolving the draft to essentially map "packetized SCSI" (Information Unit from T10 standard) over IP. 2. Security Considerations - current status Luciano Dalle Ore and I have done some work in this area, most of which has been incorporated already in the iSCSI draft. Paul von Stamwitz From owner-ips@ECE.cmu.edu Wed Jul 19 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02879; 18 Jul 2000 19:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA13928 for ips-outgoing; Tue, 18 Jul 2000 18:55:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA06477 for ; Tue, 18 Jul 2000 16:24:15 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id WAA298750; Tue, 18 Jul 2000 22:23:42 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id WAA20054; Tue, 18 Jul 2000 22:23:37 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256920.00700576 ; Tue, 18 Jul 2000 22:23:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: "Jon William Toigo" cc: ips@ece.cmu.edu Message-ID: Date: Tue, 18 Jul 2000 20:22:32 +0300 Subject: Re: FCP over IP item on the agenda Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jon, I am taking the liberty of forwarding this to the mailing list. In fact I DO NOT WANT FCP to e discussed AT ALL during the limited time we have. But the new WG chairmenn David Black (EMC) and Steve Belovin (AT&T) suggested we do on their draft agenda and that generated my answer. I still think it is a very bad idea. Julo Julian Satran IBM Research Laboratory in Haifa "Jon William Toigo" on 16/07/2000 22:28:03 Please respond to "Jon William Toigo" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: FCP over IP item on the agenda Mr. Satran, I have been reading in earnest the traffic regarding iSCSI and storage over IP. I cannot understand why you would want to obfuscate the important work you are doing to include a debate or discussion of FCP over IP, as is proposed by EMC (see below). Clearly, a few companies have already deployed FC-based storage networks, but have discovered the great expense, learning curve, and difficulty in building, scaling and managing the architecture. That, in a nutshell, is why IP SANs are so interesting. FC-based SANs confront numerous problems that I have addressed in articles for the trade press. (See SANs and Sensibility, InfoStor Magazine, May or June issue and my current column on storage at PlanetIT.com.) To bridge "island FC SANs" across the corporate Ethernet backbone, an assortment of gateways and bridges will shortly come to market -- including a joint Brocade/Cisco solution that will likely involve a blade in a Cisco router, and standalone products from NuSpeed, NetConvergence and others. IPS should focus on IP SANs initially, then embrace the needs of those who want to bridge their storage solutions to IPS-defined SANs. In my view, for what it is worth, the handwriting about FC is already on the wall. It is not a true SAN technology, if only because FC is a fabric and not a network per se. Efforts to do SANs using FC have run afoul of the technology's lack of in-band management, lack of interoperability between vended products, and large staff retraining requirements. In the end, FC may provide a nice technology for use within array cabinets, but it does not provide a sound basis (in its current manifestation, at least) for an enterprise SAN. For what its worth. Jon William Toigo, consultant and author The Holy Grail of Data Storage Management jtoigo@intnet.net ----- Original Message ----- From: To: Cc: Sent: Sunday, July 16, 2000 4:48 AM Subject: FCP over IP item on the agenda > > > David, > > I've seen the proposed agenda of the BOF on the IETF site. > I think that we have far less time than we asked for but we may try to make > up for this. > > I was also under the impression that we are going to discuss thing that > where > already published on the IETF and /or IPS board. > > Considering the limited time we have - and the fact that we went through > the FCP > over IP thing several times in public forums and designer group I wonder > why > it appears as a tentative item on your agenda. > > If we have to consider new facts we have to have time to digest them and we > did not see the yet on any forum (unlike all other items). > > I feel very uneasy about having to give up precious time during the only > face-to-face > session we have to an item that has yet to be brought to us. > I think this is also contrary to the IETF practice - or the little I've got > to know about it. > > Again I have nothing against discussing again FCP over IP - but in order to > see if there > are arguments for it that we did not consider before I think we need time > off line > and getting it a first hearing during the only session we have will do a > disservice > to the community. > > Regards, > Julo > > From owner-ips@ECE.cmu.edu Wed Jul 19 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02878; 18 Jul 2000 19:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA13591 for ips-outgoing; Tue, 18 Jul 2000 18:43:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA16603 for ; Tue, 18 Jul 2000 05:43:34 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA288950 for ; Tue, 18 Jul 2000 11:42:58 +0200 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id LAA162302 for ; Tue, 18 Jul 2000 11:42:57 +0200 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256920.00355C5A ; Tue, 18 Jul 2000 11:42:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: ips@ece.cmu.edu Message-ID: Date: Tue, 18 Jul 2000 12:40:46 +0300 Subject: RE: FCP over IP item on the agenda and thje charter Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, We will certainly bring to the table our considerations and why we choose what we choose. I think this should be a 15 min point on the agenda. Many of us would also like to see a 30 minutes discussion on each of the following The separate data/command channels approach with the Kalman Meth proposed twist, security and authentication - details about how it works and how it should be worded, recovery how it works and what is the minimum required. The draft you suggested we look at is not entirely relevant to our group . It is a networking layer on another networking layer. The IP area is there territory this should be discussed (if somewhere). There is also a growing concern (expressed already in a previous message) to the community versus commercial value to some industry players of the FCP inclusion. I strongly favor taking it out. Also about the charter - due to mail failures I got it late. It is good and in covers the whole area. The time table should however focus on getting out the things that require the longest "gestation" first - the transport mapping. The others are at least as important and if IESG decides that 1 group is enough to cover them all (I am not sure it has to be) then we should schedule the to activities in areas late discovery, management etc. to follow. I am also not completely confident that I understand the "framework" stuff? Is it a requirements document? Is it a recipe to derive a myriad of incompatible protocols? In those areas that cover basic transport issues is framework is not a unifying thing but rather a way of explaining the differences when they arise. Do we want this? Do we really need this. I wonder why you used so little of the initial, modest and carefully worded charter we suggested for the first phase 3 months and added to it for later stages discovery and management. Did you get to have a look at it? Do you what it reposted? Regards, Julo Black_David@emc.com on 17/07/2000 18:04:31 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, Black_David@emc.com cc: ips@ece.cmu.edu Subject: RE: FCP over IP item on the agenda Julian: Thanks for your comments. Here's an attempt to provide the missing background/explanations: - The ips agenda for Pittsburgh posted on the IETF web site is very drafty -- something had to be invented on short notice to get the space scheduled. It is definitely subject to change. - One of the things that needs to be done this week is to come up with a real agenda, and the first step is agenda bashing on the mailing list -- thanks for starting this. It would be good if the agenda were mostly finalized this week. - The draft charter has been expanded in scope from what was previously discussed based on direction from the ADs. Both SCSI and Fibre Channel protocols are noted, and a framework document has been added. The FC over IP work is at least relevant to the framework discussion. The WG is free to decide whether or not to undertake FC over IP, and for what purposes. - There is a draft on the FC over IP work, draft-ietf-ipfc-fcoverip-01.txt Despite the "ietf-ipfc" draft name, it may be the case that this work is to be pursued in the ips WG if at all; further clarification should be forthcoming from the ADs. I appreciate the concern about needing the opportunity to review drafts prior to the meeting -- hence the goal of getting the agenda done this week, complete with draft names. - Please be careful with terminology - FC over IP encapsulates FC-2; this is the basic Fibre Channel frame protocol, and is at a lower layer in the FC protocol stack than FCP (the Fibre Channel encapsulation of SCSI). At the moment, this is not a source of confusion, but a stitch in time ... Finally, when something is discussed among the design team, the result is a design team recommendation, which is not to be confused with WG rough consensus -- IETF operates by rough consensus of the WG, usually on the WG mailing list. One of the things that would be a good idea in Pittsburgh is to put the design team's recommendations in front of the audience to check what there is and is not consensus for. Thanks, --David p.s. From a procedural viewpoint, I want to encourage people to do what Julian has done -- if you think there's something wrong with the way the WG is being run, please bring it up early and by whatever means you're comfortable with. Nobody can promise to make everyone happy all the time, but I'd at least like to know what people are concerned about so that I can do something about it. --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Sunday, July 16, 2000 4:48 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: FCP over IP item on the agenda > > > > David, > > I've seen the proposed agenda of the BOF on the IETF site. > I think that we have far less time than we asked for but we may try to > make > up for this. > > I was also under the impression that we are going to discuss thing that > where > already published on the IETF and /or IPS board. > > Considering the limited time we have - and the fact that we went through > the FCP > over IP thing several times in public forums and designer group I wonder > why > it appears as a tentative item on your agenda. > > If we have to consider new facts we have to have time to digest them and > we > did not see the yet on any forum (unlike all other items). > > I feel very uneasy about having to give up precious time during the only > face-to-face > session we have to an item that has yet to be brought to us. > I think this is also contrary to the IETF practice - or the little I've > got > to know about it. > > Again I have nothing against discussing again FCP over IP - but in order > to see if there > are arguments for it that we did not consider before I think we need time > off line > and getting it a first hearing during the only session we have will do a > disservice > to the community. > > Regards, > Julo > From owner-ips@ECE.cmu.edu Wed Jul 19 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02886; 18 Jul 2000 19:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA13706 for ips-outgoing; Tue, 18 Jul 2000 18:47:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from auemlsrv.firewall.lucent.com (auemail1.lucent.com [192.11.223.161]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA21834 for ; Tue, 18 Jul 2000 09:59:06 -0400 (EDT) Received: from auemlsrv.firewall.lucent.com (localhost [127.0.0.1]) by auemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id JAA19990 for ; Tue, 18 Jul 2000 09:59:05 -0400 (EDT) Received: from il0015exch001h.wins.lucent.com (h135-1-23-83.lucent.com [135.1.23.83]) by auemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id JAA19969 for ; Tue, 18 Jul 2000 09:59:05 -0400 (EDT) Received: by il0015exch001h.ih.lucent.com with Internet Mail Service (5.5.2650.21) id <3ZJZ7THL>; Tue, 18 Jul 2000 08:59:04 -0500 Message-ID: <80B684C5E29FD211AA8000A0C9CDD91904D09764@IL0015EXCH005U> From: "Rodriguez, Elizabeth G (Elizabeth)" To: "'ips@ece.cmu.edu'" Cc: "Marjorie Krueger (E-mail)" , "'rajb@lightsand.com'" , "'wayne@gadzoox.com'" Subject: FCP over IP/ FC over IP in ipfc Date: Tue, 18 Jul 2000 08:58:59 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi all, I see some discussion on FC over IP with respect to the agenda item on iSCSI: FCP over IP. I am not certain who proposed this agenda item, so I can not speak to it directly, but as one of the authors of FC over IP draft that is being done in the ipfc working group, I can speak to that. The FC over IP draft is not directly dealing with encapsulation of FCP commands over IP. It is standardizing a way of bridging FC traffic. It focuses on the encapsulation of FC frames over an IP network. No knowledge of the type of traffic being bridged (e.g. traffic can be link layer, FCP, IP encapsulated in FC (e.g. following the ipfc draft), etc.) is required. This specification focuses more on the bridging of two disjoint FC regions to form a virtual FC only network. In other words, with the exception of the FC over IP bridging device, none of the FC devices are aware that any of their traffic is going over an IP network. Instead of developing large FC only networks, we are taking advantage of the existing IP network infrastructure and using it as a backbone to bridge FC networks. We feel that our efforts address a different concern than iSCSI and that FC over IP and iSCSI are in many ways complementary. This work is being done in the ipfc working group, since that is/was the only group that was dealing with Fibre Channel. Draft 2 of the FC over IP document has been submitted to the IETF, but I do not believe it has yet been posted. (draft-ietf-ipfc-fcoverip-02-txt) If anyone needs or wants more information, feel free to contact me. Thanks, Elizabeth G. Rodriguez Optical Area Networking Group Lucent Technologies From owner-ips@ECE.cmu.edu Wed Jul 19 14:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05892; 19 Jul 2000 10:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA07675 for ips-outgoing; Wed, 19 Jul 2000 09:02:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA07240 for ; Wed, 19 Jul 2000 08:45:48 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id IAA08321; Wed, 19 Jul 2000 08:45:31 -0400 (EDT) Date: Wed, 19 Jul 2000 08:45:31 -0400 (EDT) From: Scott Bradner Message-Id: <200007191245.IAA08321@newdev.harvard.edu> To: Black_David@emc.com, julian_satran@il.ibm.com Subject: RE: FCP over IP item on the agenda and thje charter Cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Precedence: bulk > I am also not completely confident that I understand the "framework" a freamework document is for people to read to get an overview of what the system level view is of the technology - they are informational documents note that WG charters are finalized by the ADs and the IESG with the advice of the IAB Scott From owner-ips@ECE.cmu.edu Thu Jul 20 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07745; 19 Jul 2000 19:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA05663 for ips-outgoing; Wed, 19 Jul 2000 18:51:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA05210 for ; Wed, 19 Jul 2000 18:38:12 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id <3SR1333M>; Wed, 19 Jul 2000 18:37:42 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB5C@corpmx9.isus.emc.com> To: jtoigo@IntNet.net, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: FCP over IP item on the agenda Date: Wed, 19 Jul 2000 18:37:40 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Apologies to David Black if I misunderstood who had proposed the inclusion > of FCP over IP in the agenda. Nobody has made that proposal, AFAIK. At the risk of splitting hairs, the distinction comes down to one letter in an acronym -- FC*P* over IP has not been proposed. The initial draft agenda contains an FC over IP item; this involves encapsulation of FC-2 frames rather than FCP information units; FCP is the mapping of SCSI to Fibre Channel including definition of FCP information units that are carried in FC-2 frames, but FC-2 frames can carry payloads other than FCP. No more FCP discussion, at least for now, please ... > Whatever decision is made regarding the agenda, I would hope that it advances, > rather than slows, the realization of iSCSI SAN standards and technology. I agree ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Jul 20 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07740; 19 Jul 2000 19:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA04535 for ips-outgoing; Wed, 19 Jul 2000 18:19:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhost.IntNet.net (mailhost.IntNet.net [198.252.32.150]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA03347 for ; Wed, 19 Jul 2000 17:55:45 -0400 (EDT) Received: from jtoigo by mailhost.IntNet.net (8.8.5/INTNET/SMI-SVR4) id RAA02002; Wed, 19 Jul 2000 17:59:51 -0400 (EDT) X-Auth: cs-clw-5-105.intnet.net [207.90.1.105] Message-ID: <0b9f01bff1cc$70536840$dd9cfea9@jtoigo> From: "Jon William Toigo" To: Cc: References: <0F31E5C394DAD311B60C00E029101A070148FB5A@corpmx9.isus.emc.com> Subject: Re: FCP over IP item on the agenda Date: Wed, 19 Jul 2000 17:58:12 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Apologies to David Black if I misunderstood who had proposed the inclusion of FCP over IP in the agenda. I admit that I assumed this based on the message traffic in my email. I have been covering the IP SAN effort for some time and frequently encounter FC advocates who turn Fibre Channel vs IP-based SAN discussions into holy wars. (I have an interesting email from Tom Lassen that illustrates the point, if you are interested.) Whatever decision is made regarding the agenda, I would hope that it advances, rather than slows, the realization of iSCSI SAN standards and technology. For what it is worth, Jon Toigo ----- Original Message ----- From: To: ; Cc: <> Sent: Wednesday, July 19, 2000 5:37 PM Subject: RE: FCP over IP item on the agenda > > In fact I DO NOT WANT FCP to e discussed AT ALL during the limited time > > we have. > > For the record: > - I have not seen any proposal to discuss FCP or FCP over IP at the > ips meeting in Pittsburgh. > - Jon William Toigo's statement that "EMC" "proposed" "a debate or > discussion of FCP over IP" is simply not true. I believe that this > confusion stems from Julian's original misuse of "FCP", which > has now been repeated in a rather unfortunate fashion. > Misstatements like this have the potential to cause serious > problems. Could everyone please be careful about this sort > of thing? > > > But the new WG chairmenn David Black (EMC) and Steve Belovin (AT&T) > > suggested we do on their draft agenda and that generated my answer. > > I still think it is a very bad idea. > > I believe that this is about the following item on the drafty draft agenda: > > Fibre Channel over IP (? tentative ?) > > Julian's request to remove it is noted. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Thu Jul 20 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07835; 19 Jul 2000 20:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA04529 for ips-outgoing; Wed, 19 Jul 2000 18:19:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA02624 for ; Wed, 19 Jul 2000 17:38:02 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id <3SR13N7Y>; Wed, 19 Jul 2000 17:37:32 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB5A@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, jtoigo@IntNet.net Cc: ips@ece.cmu.edu Subject: RE: FCP over IP item on the agenda Date: Wed, 19 Jul 2000 17:37:28 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > In fact I DO NOT WANT FCP to e discussed AT ALL during the limited time > we have. For the record: - I have not seen any proposal to discuss FCP or FCP over IP at the ips meeting in Pittsburgh. - Jon William Toigo's statement that "EMC" "proposed" "a debate or discussion of FCP over IP" is simply not true. I believe that this confusion stems from Julian's original misuse of "FCP", which has now been repeated in a rather unfortunate fashion. Misstatements like this have the potential to cause serious problems. Could everyone please be careful about this sort of thing? > But the new WG chairmenn David Black (EMC) and Steve Belovin (AT&T) > suggested we do on their draft agenda and that generated my answer. > I still think it is a very bad idea. I believe that this is about the following item on the drafty draft agenda: Fibre Channel over IP (? tentative ?) Julian's request to remove it is noted. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Jul 20 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07905; 19 Jul 2000 20:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA06789 for ips-outgoing; Wed, 19 Jul 2000 19:22:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from prserv.net (out4.prserv.net [32.97.166.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA06510 for ; Wed, 19 Jul 2000 19:15:10 -0400 (EDT) Received: from sanrad.com ([139.92.185.15]) by prserv.net (out4) with SMTP id <2000071923150823902haan4e>; Wed, 19 Jul 2000 23:15:08 +0000 Message-ID: <39764464.80A132F8@sanrad.com> Date: Thu, 20 Jul 2000 02:14:28 +0200 From: Yaron Klein X-Mailer: Mozilla 4.73 [en] (Win98; I) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: iSCSI Security draft Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear all, We submitted an internet draft (ietf-klein-iscsi-security-00.txt) to the IETF. The draft adresses the iSCSI security issues and introduces the concept of VPSAN (Virtual Private San). A postscript version of the draft (much more readable than the text version thereof) can be downloaded from: http://www.eng.tau.ac.il/~klein/ietf/iscsi.html Comments, suggestions and reviews are encouraged. Hope we will have chances to discuss the draft in the next meeting. Regards, Yaron Klein SANRAD From owner-ips@ECE.cmu.edu Thu Jul 20 03:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08360; 19 Jul 2000 22:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA10492 for ips-outgoing; Wed, 19 Jul 2000 21:28:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA10258 for ; Wed, 19 Jul 2000 21:20:56 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id DAA262650; Thu, 20 Jul 2000 03:20:20 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id DAA118080; Thu, 20 Jul 2000 03:20:21 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256922.00075A37 ; Thu, 20 Jul 2000 03:20:18 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: ips@ece.cmu.edu Message-ID: Date: Thu, 20 Jul 2000 04:18:50 +0300 Subject: RE: FCP over IP item on the agenda Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I am even more confused. I assumed that you want to talk about FCP over IP as FCP has something to do with storage - but FC over IP has not and is handled elewhere. And if this created confusion I appologize. FC over IP - what does have to do with our charter? Julo Black_David@emc.com on 20/07/2000 00:37:28 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, jtoigo@intnet.net cc: ips@ece.cmu.edu Subject: RE: FCP over IP item on the agenda > In fact I DO NOT WANT FCP to e discussed AT ALL during the limited time > we have. For the record: - I have not seen any proposal to discuss FCP or FCP over IP at the ips meeting in Pittsburgh. - Jon William Toigo's statement that "EMC" "proposed" "a debate or discussion of FCP over IP" is simply not true. I believe that this confusion stems from Julian's original misuse of "FCP", which has now been repeated in a rather unfortunate fashion. Misstatements like this have the potential to cause serious problems. Could everyone please be careful about this sort of thing? > But the new WG chairmenn David Black (EMC) and Steve Belovin (AT&T) > suggested we do on their draft agenda and that generated my answer. > I still think it is a very bad idea. I believe that this is about the following item on the drafty draft agenda: Fibre Channel over IP (? tentative ?) Julian's request to remove it is noted. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Jul 20 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08548; 19 Jul 2000 23:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA12591 for ips-outgoing; Wed, 19 Jul 2000 22:44:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA12025 for ; Wed, 19 Jul 2000 22:22:21 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id C81B741B1 for ; Wed, 19 Jul 2000 19:22:20 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id TAA27387 for ; Wed, 19 Jul 2000 19:22:18 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA67F4; Wed, 19 Jul 2000 19:22:05 -0700 Message-ID: <3976624D.6A514A3C@agilent.com> Date: Wed, 19 Jul 2000 19:22:05 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.73 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: FCP over IP item on the agenda References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I think FC over IP means "tunnel FC over IP from one FC fabric to another". FCP is a subset of FC, so if FC over IP can mean tunnel FCP over IP. -Matt julian_satran@il.ibm.com wrote: > David, > > I am even more confused. I assumed that you want to talk about FCP over IP > as FCP > has something to do with storage - but FC over IP has not and is handled > elewhere. > > And if this created confusion I appologize. > FC over IP - what does have to do with our charter? > > Julo > > Black_David@emc.com on 20/07/2000 00:37:28 > > Please respond to Black_David@emc.com > > To: Julian Satran/Haifa/IBM@IBMIL, jtoigo@intnet.net > cc: ips@ece.cmu.edu > Subject: RE: FCP over IP item on the agenda > > > In fact I DO NOT WANT FCP to e discussed AT ALL during the limited time > > we have. > > For the record: > - I have not seen any proposal to discuss FCP or FCP over IP at the > ips meeting in Pittsburgh. > - Jon William Toigo's statement that "EMC" "proposed" "a debate or > discussion of FCP over IP" is simply not true. I believe that this > confusion stems from Julian's original misuse of "FCP", which > has now been repeated in a rather unfortunate fashion. > Misstatements like this have the potential to cause serious > problems. Could everyone please be careful about this sort > of thing? > > > But the new WG chairmenn David Black (EMC) and Steve Belovin (AT&T) > > suggested we do on their draft agenda and that generated my answer. > > I still think it is a very bad idea. > > I believe that this is about the following item on the drafty draft agenda: > > Fibre Channel over IP (? tentative ?) > > Julian's request to remove it is noted. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Jul 20 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08783; 20 Jul 2000 0:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA13660 for ips-outgoing; Wed, 19 Jul 2000 23:22:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA13297 for ; Wed, 19 Jul 2000 23:08:05 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id <3SR13PY4>; Wed, 19 Jul 2000 23:07:35 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB64@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: IPS Draft Charter update Date: Wed, 19 Jul 2000 23:07:32 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Slightly updated version of the draft charter. Comments are welcome/encouraged, especially on the proposed schedule. There will probably be an opportunity to bash the charter in Pittsburgh, but bashing done on the list doesn't consume time in the meeting. --David IP Storage (ips) Working Group Proposed Charter - DRAFT 4 There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for shared traffic network environments, such as the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks, including both discovery of storage for access by the discovering entity, and discovery for management. - Management, including appropriate MIB definition. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). The WG will strive to support the approaches to discovery, multi-pathing, and booting taken by the existing block storage protocols it encapsulates at the levels of those protocols. It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Launch framework discussion to select encapsulation approach or approaches. Sep 00 Publish initial version of framework draft reflecting WG consensus on encapsulation approach or approaches. Dec 00 Discuss framework, and requirements, applicability and protocol specification drafts at IETF meeting in San Diego. Jan 01 Submit framework and at least one set of requirements, applicability, and protocol specification drafts to IESG. Mar 01 Discuss MIB(s) and any ancillary drafts required to use the specified protocols at IETF meeting in Minneapolis. May 01 Submit primary MIB draft to IESG. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Thu Jul 20 14:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10894; 20 Jul 2000 10:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA07830 for ips-outgoing; Thu, 20 Jul 2000 09:03:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA07460 for ; Thu, 20 Jul 2000 08:55:13 -0400 (EDT) Received: from centralmail2.Central.Sun.COM ([129.147.62.11]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id GAA19329; Thu, 20 Jul 2000 06:55:06 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail2.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id GAA23205; Thu, 20 Jul 2000 06:55:05 -0600 (MDT) Received: from sun.com (hobo24.Central.Sun.COM [129.147.8.24]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id GAA04807; Thu, 20 Jul 2000 06:41:40 -0600 (MDT) Message-ID: <3976F81F.5018CFE7@sun.com> Date: Thu, 20 Jul 2000 07:01:19 -0600 From: "Mark A. Carlson" X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: ips@ece.cmu.edu Subject: Re: IPS Draft Charter update References: <0F31E5C394DAD311B60C00E029101A070148FB64@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk In addition to the network quality of service work, I would hope that it is in our scope to work on how storage quality of service would be specified for IP based storage. I have suggested some words for the charter below: Black_David@emc.com wrote: > > Slightly updated version of the draft charter. Comments > are welcome/encouraged, especially on the proposed schedule. > There will probably be an opportunity to bash the charter > in Pittsburgh, but bashing done on the list doesn't consume > time in the meeting. > > --David > > IP Storage (ips) Working Group Proposed Charter - DRAFT 4 > > There is significant interest in using IP-based networks to transport > block storage traffic. This group will pursue the pragmatic approach of > encapsulating existing block storage protocols, such as SCSI and certain > Fibre Channel protocols, in an IP-based transport or transports. The group > will focus on the transport or transports and related issues (e.g., > security, > naming, discovery, and configuration), as opposed to modifying existing > block > storage protocols. Standards for those protocols are controlled by other > standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG > cannot assume that any changes it desires will be made in these standards, > and hence will pursue approaches that do not depend on such changes unless > they are unavoidable and in that case will create a document to be forwarded > to the standards group responsible for the technology explaining the > issue and requesting the desired changes be considered. The WG will > endeavor to ensure high quality communications with these standards > organizations. > > The storage protocols to be encapsulated expect a reliable transport, > in that failure to deliver data is considered to be a rare event for > which time-consuming recovery is acceptable. This has implications > for both the choice of transport protocols and design of the > encapsulation(s). Encapsulations of the storage protocols may require > quality of service assurances (e.g., predictable latency) to operate > successfully; the WG will consider what assurances are appropriate and > how to provide such assurances in shared traffic environments > based on existing IETF QoS mechanisms such as Differentiated Services. Add: In addition the WG may consider emerging standards for storage quality of service and how they may be used in IP based storage. This would include the specification of performance and availability service levels for block storage devices and the management of policies related to those service levels. > > Use of an IP-based transport raises issues that do not occur in existing > storage transports. The WG will address at least the following issues: ... -- mark From owner-ips@ECE.cmu.edu Thu Jul 20 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12192; 20 Jul 2000 17:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA25694 for ips-outgoing; Thu, 20 Jul 2000 15:30:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA25629 for ; Thu, 20 Jul 2000 15:29:24 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Thu, 20 Jul 2000 15:28:37 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB72@corpmx9.isus.emc.com> To: mark.carlson@sun.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: IPS Draft Charter update Date: Thu, 20 Jul 2000 15:28:36 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark Thanks for the suggestion: > In addition the WG may consider emerging standards for storage quality > of service and how they may be used in IP based storage. This would include > the specification of performance and availability service levels for block > storage devices and the management of policies related to those service > levels. Would those be IETF standards, or some other standards body/bodies? If the latter, which ones? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Mark A. Carlson [SMTP:mark.carlson@sun.com] > Sent: Thursday, July 20, 2000 9:01 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: Re: IPS Draft Charter update > > In addition to the network quality of service work, I would > hope that it is in our scope to work on how storage quality > of service would be specified for IP based storage. > > I have suggested some words for the charter below: > > Black_David@emc.com wrote: > > > > Slightly updated version of the draft charter. Comments > > are welcome/encouraged, especially on the proposed schedule. > > There will probably be an opportunity to bash the charter > > in Pittsburgh, but bashing done on the list doesn't consume > > time in the meeting. > > > > --David > > > > IP Storage (ips) Working Group Proposed Charter - DRAFT 4 > > > > There is significant interest in using IP-based networks to transport > > block storage traffic. This group will pursue the pragmatic approach of > > encapsulating existing block storage protocols, such as SCSI and certain > > Fibre Channel protocols, in an IP-based transport or transports. The > group > > will focus on the transport or transports and related issues (e.g., > > security, > > naming, discovery, and configuration), as opposed to modifying existing > > block > > storage protocols. Standards for those protocols are controlled by > other > > standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The > WG > > cannot assume that any changes it desires will be made in these > standards, > > and hence will pursue approaches that do not depend on such changes > unless > > they are unavoidable and in that case will create a document to be > forwarded > > to the standards group responsible for the technology explaining the > > issue and requesting the desired changes be considered. The WG will > > endeavor to ensure high quality communications with these standards > > organizations. > > > > The storage protocols to be encapsulated expect a reliable transport, > > in that failure to deliver data is considered to be a rare event for > > which time-consuming recovery is acceptable. This has implications > > for both the choice of transport protocols and design of the > > encapsulation(s). Encapsulations of the storage protocols may require > > quality of service assurances (e.g., predictable latency) to operate > > successfully; the WG will consider what assurances are appropriate and > > how to provide such assurances in shared traffic environments > > based on existing IETF QoS mechanisms such as Differentiated Services. > > Add: > > In addition the WG may consider emerging standards for storage quality > of service and how they may be used in IP based storage. This would > include > the specification of performance and availability service levels for block > storage devices and the management of policies related to those service > levels. > > > > > Use of an IP-based transport raises issues that do not occur in existing > > storage transports. The WG will address at least the following issues: > > ... > > -- mark From owner-ips@ECE.cmu.edu Thu Jul 20 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12206; 20 Jul 2000 17:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA28587 for ips-outgoing; Thu, 20 Jul 2000 16:14:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA27702 for ; Thu, 20 Jul 2000 16:03:13 -0400 (EDT) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id OAA24032; Thu, 20 Jul 2000 14:03:09 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id OAA13434; Thu, 20 Jul 2000 14:03:06 -0600 (MDT) Received: from sun.com (hobo30.Central.Sun.COM [129.147.8.30]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id NAA11339; Thu, 20 Jul 2000 13:49:43 -0600 (MDT) Message-ID: <39775C74.AAE6BFF1@sun.com> Date: Thu, 20 Jul 2000 14:09:24 -0600 From: "Mark A. Carlson" X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: ips@ece.cmu.edu, Lynne VanArsdale Subject: Re: IPS Draft Charter update References: <0F31E5C394DAD311B60C00E029101A070148FB72@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com wrote: > > Mark > > Thanks for the suggestion: > > > In addition the WG may consider emerging standards for storage quality > > of service and how they may be used in IP based storage. This would > include > > the specification of performance and availability service levels for block > > storage devices and the management of policies related to those service > > levels. > > Would those be IETF standards, or some other standards > body/bodies? If the latter, which ones? As you know, I am a co-chair for the Storage Network Industry Association (SNIA) Policy working group. For the benefit of others on this list, we are working on storage policies and storage service level standards. The consensus to date has been that these policies would leverage the work of the IETF Policy Framework working group as well as the Distributed Management Task Force's (DMTF's) Common Information Model (CIM) Policy Model. The standards we come up with would be incorporated into DMTF's CIM Model, but could also be standardized as a LDAP schema or model in an appropriate IETF WG. There is already a precedent for this in the current DMTF/IETF service level/policy working groups. The main IETF WG to date that has done anything relating to storage is the IP over Fibre Channel WG and it might be appropriate to standardize it there by extending its charter. However, this working group might be a more appropriate place to do this standardization as it appeals more to storage and system vendors rather than Fibre Channel switch vendors. The Policy Framework working group has already pushed policy models specific to a given technology (DiffServ, MPLS, Security) out to the respective working groups to standardize, and this could be an example of that. As far as this working group is concerned, the standards might take the form of signaled service level objectives (requesting a certain performance goal, for example) to the storage device or perhaps a protocol for negotiation of the service level. Service level metrics, such as performance, availability and perhaps even cost might be exposed through a storage MIB. Discovery could find storage devices that would provide certain service levels and even sort them by cost or distance (latency performance metric). These all involve some form of IP protocol, however, they go beyond the initial work of simple connectivity and data transfer. My intent is to allow this sort of work in our charter so that teams that are interested in this work would have a framework within which to collaborate in the IETF. -- mark From owner-ips@ECE.cmu.edu Thu Jul 20 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12211; 20 Jul 2000 17:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA28596 for ips-outgoing; Thu, 20 Jul 2000 16:14:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA27702 for ; Thu, 20 Jul 2000 16:03:13 -0400 (EDT) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id OAA24032; Thu, 20 Jul 2000 14:03:09 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id OAA13434; Thu, 20 Jul 2000 14:03:06 -0600 (MDT) Received: from sun.com (hobo30.Central.Sun.COM [129.147.8.30]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id NAA11339; Thu, 20 Jul 2000 13:49:43 -0600 (MDT) Message-ID: <39775C74.AAE6BFF1@sun.com> Date: Thu, 20 Jul 2000 14:09:24 -0600 From: "Mark A. Carlson" X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: ips@ece.cmu.edu, Lynne VanArsdale Subject: Re: IPS Draft Charter update References: <0F31E5C394DAD311B60C00E029101A070148FB72@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com wrote: > > Mark > > Thanks for the suggestion: > > > In addition the WG may consider emerging standards for storage quality > > of service and how they may be used in IP based storage. This would > include > > the specification of performance and availability service levels for block > > storage devices and the management of policies related to those service > > levels. > > Would those be IETF standards, or some other standards > body/bodies? If the latter, which ones? As you know, I am a co-chair for the Storage Network Industry Association (SNIA) Policy working group. For the benefit of others on this list, we are working on storage policies and storage service level standards. The consensus to date has been that these policies would leverage the work of the IETF Policy Framework working group as well as the Distributed Management Task Force's (DMTF's) Common Information Model (CIM) Policy Model. The standards we come up with would be incorporated into DMTF's CIM Model, but could also be standardized as a LDAP schema or model in an appropriate IETF WG. There is already a precedent for this in the current DMTF/IETF service level/policy working groups. The main IETF WG to date that has done anything relating to storage is the IP over Fibre Channel WG and it might be appropriate to standardize it there by extending its charter. However, this working group might be a more appropriate place to do this standardization as it appeals more to storage and system vendors rather than Fibre Channel switch vendors. The Policy Framework working group has already pushed policy models specific to a given technology (DiffServ, MPLS, Security) out to the respective working groups to standardize, and this could be an example of that. As far as this working group is concerned, the standards might take the form of signaled service level objectives (requesting a certain performance goal, for example) to the storage device or perhaps a protocol for negotiation of the service level. Service level metrics, such as performance, availability and perhaps even cost might be exposed through a storage MIB. Discovery could find storage devices that would provide certain service levels and even sort them by cost or distance (latency performance metric). These all involve some form of IP protocol, however, they go beyond the initial work of simple connectivity and data transfer. My intent is to allow this sort of work in our charter so that teams that are interested in this work would have a framework within which to collaborate in the IETF. -- mark From owner-ips@ECE.cmu.edu Thu Jul 20 23:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12616; 20 Jul 2000 19:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA05857 for ips-outgoing; Thu, 20 Jul 2000 18:02:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from c014.sfo.cp.net (c014-h003.c014.sfo.cp.net [209.228.12.67]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id RAA03851 for ; Thu, 20 Jul 2000 17:42:50 -0400 (EDT) Received: (cpmta 20152 invoked from network); 20 Jul 2000 14:36:07 -0700 Received: from 3ff8c878.dsl.flashcom.net (HELO gis.net) (63.248.200.120) by smtp.flashcom.net with SMTP; 20 Jul 2000 14:36:07 -0700 X-Sent: 20 Jul 2000 21:36:07 GMT Message-ID: <39777066.D76A07D0@gis.net> Date: Thu, 20 Jul 2000 17:34:30 -0400 From: Bill Main Reply-To: bill.main@bigfoot.com X-Mailer: Mozilla 4.5 [en]C-CCK-MCD compaq (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ipSCSI list Subject: questions on the mapping in iSCSI Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks- Been working on trial implementation of iSCSI as it seems to me the easiest way to find loose threads in a new spec. couple of items have showed up so far: Question with regard to the mapping. I see latest spec descriptors that move from initiator to host but the response to mapping has little data in it other than status. Who generates the SRA's for the host to use. Are the SRA's part of the descriptor and the host is therefore informing the target what LUN id it will use for this "view"? Or should the SRA be returned from the target to inform the host what SRA to use to reach the desired view? If so the data layout needs updating to accommodate this. Next, multiple descriptors are allowed in the map command. What action should be taken if one is bad? Do the others and respond with failure or disallow others from succeeding? or should we limit the map command to single descriptor? I am thinking also of the poor sys admin trying to fix a broken mount command. Next, on the unmapping. What happens when an unmapping is done and outstanding IO's are pending on the device? in flight? Does the target can them or let them complete? It is not clear, but I presume the response to the unmap occurs after the IO's are either completed or aborted and abort is confirmed by the hardware at the target end. Is this correct? -Bill From owner-ips@ECE.cmu.edu Fri Jul 21 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12795; 20 Jul 2000 20:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA09153 for ips-outgoing; Thu, 20 Jul 2000 19:02:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail-gw.hursley.ibm.com (mail-gw.hursley.ibm.com [194.196.110.15]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA08734 for ; Thu, 20 Jul 2000 18:51:25 -0400 (EDT) Received: from sp3at21.hursley.ibm.com (sp3at21.hursley.ibm.com [9.20.45.21]) by mail-gw.hursley.ibm.com (AIX4.3/UCB 8.8.8/8.8.8) with ESMTP id XAA113968; Thu, 20 Jul 2000 23:50:08 +0100 Received: from hursley.ibm.com (gsine04.us.sine.ibm.com [9.14.6.44]) by sp3at21.hursley.ibm.com (AIX4.2/UCB 8.7/8.7.3) with ESMTP id XAA24134; Thu, 20 Jul 2000 23:50:37 +0100 (BST) Message-ID: <39777948.E9492486@hursley.ibm.com> Date: Thu, 20 Jul 2000 17:12:24 -0500 From: Brian E Carpenter Organization: IBM X-Mailer: Mozilla 4.61 [en] (Win98; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: "Mark A. Carlson" CC: Black_David@emc.com, ips@ece.cmu.edu, Lynne VanArsdale Subject: Re: IPS Draft Charter update References: <0F31E5C394DAD311B60C00E029101A070148FB72@corpmx9.isus.emc.com> <39775C74.AAE6BFF1@sun.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "Mark A. Carlson" wrote: ... > As far as this working group is concerned, the standards might take the form of > signaled service level objectives (requesting a certain performance goal, for example) > to the storage device or perhaps a protocol for negotiation of the service level. > Service level metrics, such as performance, availability and perhaps even cost > might be exposed through a storage MIB. This seems very strange to me - why would any of this be specific to storage? These sound like quite generic QOS issues. They are also very tough ones, that the IETF has not yet got a handle on. See draft-iab-qos-01.txt for more on what is missing in this area. Brian From owner-ips@ECE.cmu.edu Fri Jul 21 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13910; 21 Jul 2000 1:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA21188 for ips-outgoing; Fri, 21 Jul 2000 00:32:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail-gw.hursley.ibm.com (mail-gw.hursley.ibm.com [194.196.110.15]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA12587 for ; Thu, 20 Jul 2000 20:49:03 -0400 (EDT) Received: from sp3at21.hursley.ibm.com (sp3at21.hursley.ibm.com [9.20.45.21]) by mail-gw.hursley.ibm.com (AIX4.3/UCB 8.8.8/8.8.8) with ESMTP id BAA65928; Fri, 21 Jul 2000 01:47:54 +0100 Received: from hursley.ibm.com (gsine01.us.sine.ibm.com [9.14.6.41]) by sp3at21.hursley.ibm.com (AIX4.2/UCB 8.7/8.7.3) with ESMTP id BAA15178; Fri, 21 Jul 2000 01:48:22 +0100 (BST) Message-ID: <39779D52.BB5B63D4@hursley.ibm.com> Date: Thu, 20 Jul 2000 19:46:10 -0500 From: Brian E Carpenter Organization: IBM X-Mailer: Mozilla 4.61 [en] (Win98; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: Scott Bradner CC: mark.carlson@sun.com, Black_David@emc.com, ips@ece.cmu.edu, Lynne_VanArsdale@exchange.bmc.com Subject: Re: IPS Draft Charter update References: <200007202357.TAA14331@newdev.harvard.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes, it's still a draft, but I don't quite see your point, Scott. Brian Scott Bradner wrote: > > Brian sez: > > See draft-iab-qos-01.txt for more on what is missing in this area. > > See draft-iab-qos-01.txt for a view on what is missing in this area. > > Scott From owner-ips@ECE.cmu.edu Fri Jul 21 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13908; 21 Jul 2000 1:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA21222 for ips-outgoing; Fri, 21 Jul 2000 00:33:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA15495 for ; Thu, 20 Jul 2000 22:01:23 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id WAA14694; Thu, 20 Jul 2000 22:01:19 -0400 (EDT) Date: Thu, 20 Jul 2000 22:01:19 -0400 (EDT) From: Scott Bradner Message-Id: <200007210201.WAA14694@newdev.harvard.edu> To: brian@hursley.ibm.com, sob@harvard.edu Subject: Re: IPS Draft Charter update Cc: Black_David@emc.com, ips@ece.cmu.edu, Lynne_VanArsdale@exchange.bmc.com, mark.carlson@sun.com Sender: owner-ips@ece.cmu.edu Precedence: bulk > I don't quite see your point, Scott. my point is that it is not a universally shared view and in fact is quite disputed by some people Scott From owner-ips@ECE.cmu.edu Fri Jul 21 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13895; 21 Jul 2000 1:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA21202 for ips-outgoing; Fri, 21 Jul 2000 00:32:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA13340 for ; Thu, 20 Jul 2000 21:12:05 -0400 (EDT) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id TAA09205; Thu, 20 Jul 2000 19:11:59 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id TAA25830; Thu, 20 Jul 2000 19:11:58 -0600 (MDT) Received: from sun.com (hobo21.Central.Sun.COM [129.147.8.21]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id SAA25056; Thu, 20 Jul 2000 18:58:35 -0600 (MDT) Message-ID: <3977A4D9.B1484045@sun.com> Date: Thu, 20 Jul 2000 19:18:17 -0600 From: "Mark A. Carlson" X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Scott Bradner CC: brian@hursley.ibm.com, Black_David@emc.com, ips@ece.cmu.edu, Lynne_VanArsdale@exchange.bmc.com Subject: Re: IPS Draft Charter update References: <200007202357.TAA14331@newdev.harvard.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a very comprehensive overview of what is needed to ensure QoS - Quality of Service for IP networks. Which is fine up to the access point of the storage device. You obviously wouldn't want the network QoS any less than what the storage can provide or you are wasting money on your storage capabilities. Conversely, you may also want to specify the Storage quality of service as well to better match what you can get out of the network or to match what the application needs. The referenced draft does not cover that area. I know everyone thinks of disk drives when they think storage device, and this is equivalent to a "best effort" storage QoS. If the device dies, your data is lost. You get the performance that it gives you and it typically can't be tuned other than laying out the blocks differently. However, we now have storage devices that can be configured to provide different levels beyond "best effort" storage with integrated RAID, Backup, Snapshot Copy, Remote Mirroring and Caching. These are equivalent to different forwarding treatments and so forth in the networking QoS world. Of course, we can have separate MIBs for the parameters of each of these different data "treatments", but the administrator really doesn't want to care about RAID levels or how to tune his storage with all of these parameters to get the service levels that he wants out of his storage. He wants differentiated storage services that he can match to his application needs, just like in the networking world. In addition, there are now "Storage Service Providers" (SSPs) that will statically configure (provision) these parameters for their customers and offer different service level agreements for storage services. These businesses would like to offer their services over IP for all the obvious reasons. With the integration of both Network QoS and Storage QoS, IP based storage services can guarantee service levels all the way to the storage client with matched performance and availability from both the network and the storage. This is why I am interested in having this in our charter. Perhaps I should do a draft on this to help explain some of these issues. I am feeling a little alone out on this limb here. Does anyone else see this as important? Are there any SSPs out there lurking? -- mark Scott Bradner wrote: > > Brian sez: > > See draft-iab-qos-01.txt for more on what is missing in this area. > > See draft-iab-qos-01.txt for a view on what is missing in this area. > > Scott From owner-ips@ECE.cmu.edu Fri Jul 21 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13893; 21 Jul 2000 1:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA21061 for ips-outgoing; Fri, 21 Jul 2000 00:28:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA10999 for ; Thu, 20 Jul 2000 19:57:47 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id TAA14331; Thu, 20 Jul 2000 19:57:34 -0400 (EDT) Date: Thu, 20 Jul 2000 19:57:34 -0400 (EDT) From: Scott Bradner Message-Id: <200007202357.TAA14331@newdev.harvard.edu> To: brian@hursley.ibm.com, mark.carlson@sun.com Subject: Re: IPS Draft Charter update Cc: Black_David@emc.com, ips@ece.cmu.edu, Lynne_VanArsdale@exchange.bmc.com Sender: owner-ips@ece.cmu.edu Precedence: bulk Brian sez: > See draft-iab-qos-01.txt for more on what is missing in this area. See draft-iab-qos-01.txt for a view on what is missing in this area. Scott From owner-ips@ECE.cmu.edu Fri Jul 21 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13912; 21 Jul 2000 1:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA21245 for ips-outgoing; Fri, 21 Jul 2000 00:34:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from creeper.bmc.com (fw-us-hou-2.bmc.com [198.207.223.251]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA20534 for ; Fri, 21 Jul 2000 00:13:31 -0400 (EDT) Received: from ec02-hou.bmc.com (EC02-HOU.bmc.com [172.17.0.151]) by creeper.bmc.com (8.10.2/8.8.6) with ESMTP id e6L4D2h24533; Thu, 20 Jul 2000 23:13:03 -0500 (CDT) Received: by EC02-HOU.bmc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 20 Jul 2000 23:13:05 -0500 Message-ID: From: "VanArsdale, Lynne" To: "'Brian E Carpenter'" , "Mark A. Carlson" Cc: Black_David@emc.com, ips@ece.cmu.edu, "VanArsdale, Lynne" Subject: RE: IPS Draft Charter update Date: Thu, 20 Jul 2000 23:14:50 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Storage involves issues related to data in place. For instance, the data stored needs to have longevity (reliability in place) and different kinds of security. Also, sharing has different meanings for data in place and data on the move. The words may be the same. The application is, in part, different. lynne_vanarsdale@bmc.com 2101 City West Blvd Houston, Texas 77042 ( 713 ) 918 - 3167 (voice) ( 713 ) 918 - 1750 (FAX) ( 800 ) 791 - 4526 (pager) -----Original Message----- From: Brian E Carpenter [mailto:brian@hursley.ibm.com] Sent: Thursday, July 20, 2000 5:12 PM To: Mark A. Carlson Cc: Black_David@emc.com; ips@ece.cmu.edu; Lynne VanArsdale Subject: Re: IPS Draft Charter update "Mark A. Carlson" wrote: ... > As far as this working group is concerned, the standards might take the form of > signaled service level objectives (requesting a certain performance goal, for example) > to the storage device or perhaps a protocol for negotiation of the service level. > Service level metrics, such as performance, availability and perhaps even cost > might be exposed through a storage MIB. This seems very strange to me - why would any of this be specific to storage? These sound like quite generic QOS issues. They are also very tough ones, that the IETF has not yet got a handle on. See draft-iab-qos-01.txt for more on what is missing in this area. Brian From owner-ips@ECE.cmu.edu Fri Jul 21 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16595; 21 Jul 2000 14:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA23733 for ips-outgoing; Fri, 21 Jul 2000 12:29:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA24611 for ; Fri, 21 Jul 2000 02:33:16 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA222240 for ; Fri, 21 Jul 2000 08:32:44 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id IAA278834 for ; Fri, 21 Jul 2000 08:32:43 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256923.0023EF27 ; Fri, 21 Jul 2000 08:32:29 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 21 Jul 2000 08:19:57 +0300 Subject: Re: IPS Draft Charter update Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, The charter looks good and is well balanced. I am however concerned that if we include all the work items - including discovery - in the first version we might either miss our schedule or have a half baked solution. Can we exclude those from the first version and add them (either as a separate/complementary spec) in the MIB timeframe? A separate spec. could have the added advantage of being a good fit for several infrastructures. Julo Black_David@emc.com on 20/07/2000 06:07:32 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: IPS Draft Charter update Slightly updated version of the draft charter. Comments are welcome/encouraged, especially on the proposed schedule. There will probably be an opportunity to bash the charter in Pittsburgh, but bashing done on the list doesn't consume time in the meeting. --David IP Storage (ips) Working Group Proposed Charter - DRAFT 4 There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for shared traffic network environments, such as the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks, including both discovery of storage for access by the discovering entity, and discovery for management. - Management, including appropriate MIB definition. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). The WG will strive to support the approaches to discovery, multi-pathing, and booting taken by the existing block storage protocols it encapsulates at the levels of those protocols. It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Launch framework discussion to select encapsulation approach or approaches. Sep 00 Publish initial version of framework draft reflecting WG consensus on encapsulation approach or approaches. Dec 00 Discuss framework, and requirements, applicability and protocol specification drafts at IETF meeting in San Diego. Jan 01 Submit framework and at least one set of requirements, applicability, and protocol specification drafts to IESG. Mar 01 Discuss MIB(s) and any ancillary drafts required to use the specified protocols at IETF meeting in Minneapolis. May 01 Submit primary MIB draft to IESG. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Fri Jul 21 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16608; 21 Jul 2000 14:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA23751 for ips-outgoing; Fri, 21 Jul 2000 12:30:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA24862 for ; Fri, 21 Jul 2000 02:44:39 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA258454 for ; Fri, 21 Jul 2000 08:44:08 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id IAA195042 for ; Fri, 21 Jul 2000 08:44:07 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256923.0024FE80 ; Fri, 21 Jul 2000 08:44:04 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 21 Jul 2000 09:42:46 +0300 Subject: Re: questions on the mapping in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Bill, SRA's are returned by the target - the initiator only provides the descriptors; the idea is to let the target organize them as it sees fit (e.g., store the strings only once even if they originate from different initiators). I expect you would want to do as much work as possible and explore all descriptors. However finding out that a descriptor is bad can happen even after mapping. I refrained in fact specifying when will the name-to-address mapping be done (lazy or eager). I assume that unmapping should not have any effect on outstanding I/O - but I will make a not to clarify this point. Good luck with the implementation and thanks, Julo Bill Main on 21/07/2000 00:34:30 Please respond to bill.main@bigfoot.com To: ipSCSI list cc: (bcc: Julian Satran/Haifa/IBM) Subject: questions on the mapping in iSCSI Folks- Been working on trial implementation of iSCSI as it seems to me the easiest way to find loose threads in a new spec. couple of items have showed up so far: Question with regard to the mapping. I see latest spec descriptors that move from initiator to host but the response to mapping has little data in it other than status. Who generates the SRA's for the host to use. Are the SRA's part of the descriptor and the host is therefore informing the target what LUN id it will use for this "view"? Or should the SRA be returned from the target to inform the host what SRA to use to reach the desired view? If so the data layout needs updating to accommodate this. Next, multiple descriptors are allowed in the map command. What action should be taken if one is bad? Do the others and respond with failure or disallow others from succeeding? or should we limit the map command to single descriptor? I am thinking also of the poor sys admin trying to fix a broken mount command. Next, on the unmapping. What happens when an unmapping is done and outstanding IO's are pending on the device? in flight? Does the target can them or let them complete? It is not clear, but I presume the response to the unmap occurs after the IO's are either completed or aborted and abort is confirmed by the hardware at the target end. Is this correct? -Bill From owner-ips@ECE.cmu.edu Fri Jul 21 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16597; 21 Jul 2000 14:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA23713 for ips-outgoing; Fri, 21 Jul 2000 12:29:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail-gw.hursley.ibm.com (mail-gw.hursley.ibm.com [194.196.110.15]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA21361 for ; Fri, 21 Jul 2000 00:38:34 -0400 (EDT) Received: from sp3at21.hursley.ibm.com (sp3at21.hursley.ibm.com [9.20.45.21]) by mail-gw.hursley.ibm.com (AIX4.3/UCB 8.8.8/8.8.8) with ESMTP id FAA40602; Fri, 21 Jul 2000 05:37:30 +0100 Received: from hursley.ibm.com (gsine04.us.sine.ibm.com [9.14.6.44]) by sp3at21.hursley.ibm.com (AIX4.2/UCB 8.7/8.7.3) with ESMTP id FAA18152; Fri, 21 Jul 2000 05:37:59 +0100 (BST) Message-ID: <3977D320.4EF63D97@hursley.ibm.com> Date: Thu, 20 Jul 2000 23:35:44 -0500 From: Brian E Carpenter Organization: IBM X-Mailer: Mozilla 4.61 [en] (Win98; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: Scott Bradner CC: ips@ece.cmu.edu Subject: Re: IPS Draft Charter update References: <200007210201.WAA14694@newdev.harvard.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Scott Bradner wrote: > > > I don't quite see your point, Scott. > > my point is that it is not a universally shared view and in fact is > quite disputed by some people I think we will succeed in fixing that in the next version or two of the draft; anyway let's not distract ips with that... Brian From owner-ips@ECE.cmu.edu Fri Jul 21 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16599; 21 Jul 2000 14:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA23785 for ips-outgoing; Fri, 21 Jul 2000 12:31:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA18911 for ; Fri, 21 Jul 2000 10:37:57 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA71346; Fri, 21 Jul 2000 16:37:24 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v2.07) with SMTP id QAA98258; Fri, 21 Jul 2000 16:37:25 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256923.00505363 ; Fri, 21 Jul 2000 16:37:22 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: bill.main@bigfoot.com cc: ips@ece.cmu.edu Message-ID: Date: Fri, 21 Jul 2000 17:36:03 +0300 Subject: Re: questions on the mapping in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I sent the response to the list a while ago. But here I attach it again and again Good Luck, Julo ---------------------------------- Julian Satran 21/07/2000 09:42 To: ips@ece.cmu.edu cc: From: Julian Satran/Haifa/IBM@IBMIL Subject: Re: questions on the mapping in iSCSI (Document link: Julian Satran - Mail) Bill, SRA's are returned by the target - the initiator only provides the descriptors; the idea is to let the target organize them as it sees fit (e.g., store the strings only once even if they originate from different initiators). I expect you would want to do as much work as possible and explore all descriptors. However finding out that a descriptor is bad can happen even after mapping. I refrained in fact specifying when will the name-to-address mapping be done (lazy or eager). I assume that unmapping should not have any effect on outstanding I/O - but I will make a not to clarify this point. Good luck with the implementation and thanks, Julo Bill Main on 21/07/2000 00:34:30 Please respond to bill.main@bigfoot.com To: ipSCSI list cc: (bcc: Julian Satran/Haifa/IBM) Subject: questions on the mapping in iSCSI Folks- Been working on trial implementation of iSCSI as it seems to me the easiest way to find loose threads in a new spec. couple of items have showed up so far: Question with regard to the mapping. I see latest spec descriptors that move from initiator to host but the response to mapping has little data in it other than status. Who generates the SRA's for the host to use. Are the SRA's part of the descriptor and the host is therefore informing the target what LUN id it will use for this "view"? Or should the SRA be returned from the target to inform the host what SRA to use to reach the desired view? If so the data layout needs updating to accommodate this. Next, multiple descriptors are allowed in the map command. What action should be taken if one is bad? Do the others and respond with failure or disallow others from succeeding? or should we limit the map command to single descriptor? I am thinking also of the poor sys admin trying to fix a broken mount command. Next, on the unmapping. What happens when an unmapping is done and outstanding IO's are pending on the device? in flight? Does the target can them or let them complete? It is not clear, but I presume the response to the unmap occurs after the IO's are either completed or aborted and abort is confirmed by the hardware at the target end. Is this correct? -Bill ___________________________ Bill Main on 21/07/2000 17:07:52 Please respond to bill.main@bigfoot.com To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: questions on the mapping in iSCSI Julian- I have not had any responses to the spec question so I thought I would ask the author directly. I do have to say however that the state table on this has come together very nicely in general. Although I am not done yet, it seems be a well laid out spec. My compliments! -Bill Folks- Been working on trial implementation of iSCSI as it seems to me the easiest way to find loose threads in a new spec. couple of items have showed up so far: Question with regard to the mapping. I see latest spec descriptors that move from initiator to host but the response to mapping has little data in it other than status. Who generates the SRA's for the host to use. Are the SRA's part of the descriptor and the host is therefore informing the target what LUN id it will use for this "view"? Or should the SRA be returned from the target to inform the host what SRA to use to reach the desired view? If so the data layout needs updating to accommodate this. Next, multiple descriptors are allowed in the map command. What action should be taken if one is bad? Do the others and respond with failure or disallow others from succeeding? or should we limit the map command to single descriptor? I am thinking also of the poor sys admin trying to fix a broken mount command. Next, on the unmapping. What happens when an unmapping is done and outstanding IO's are pending on the device? in flight? Does the target can them or let them complete? It is not clear, but I presume the response to the unmap occurs after the IO's are either completed or aborted and abort is confirmed by the hardware at the target end. Is this correct? -Bill From owner-ips@ECE.cmu.edu Fri Jul 21 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16761; 21 Jul 2000 14:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA27622 for ips-outgoing; Fri, 21 Jul 2000 13:38:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA26884 for ; Fri, 21 Jul 2000 13:23:05 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id NAA17177 for ips@ece.cmu.edu; Fri, 21 Jul 2000 13:23:04 -0400 Date: Fri, 21 Jul 2000 13:23:04 -0400 From: Dave Nagle Message-Id: <200007211723.NAA17177@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: ignore Sender: owner-ips@ece.cmu.edu Precedence: bulk test From owner-ips@ECE.cmu.edu Fri Jul 21 20:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17194; 21 Jul 2000 16:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA01573 for ips-outgoing; Fri, 21 Jul 2000 15:00:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA01557 for ; Fri, 21 Jul 2000 15:00:34 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 21 Jul 2000 15:00:02 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB8B@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: IPS Draft Charter update Date: Fri, 21 Jul 2000 15:00:01 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I am however concerned that if we include all the work items - including > discovery - in the first version we might either miss our schedule or have > a half baked solution. > > Can we exclude those from the first version and add them (either as a > separate/complementary spec) in the MIB timeframe? Sure - discovery sounds "ancillary" to me. In any case I've been concerned that the milestones were aggressive to the point of unrealistic, and given this feedback, I'll stretch them out and try to make the next charter version more explicit about what has to be done to get the first protocol version out - that does include security. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sat Jul 22 00:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18016; 21 Jul 2000 20:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA10665 for ips-outgoing; Fri, 21 Jul 2000 19:08:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA10651 for ; Fri, 21 Jul 2000 19:08:37 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id TAA24668; Fri, 21 Jul 2000 19:08:36 -0400 Message-Id: <200007212308.TAA24668@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: Internet-Drafts@ietf.org: I-D ACTION:draft-klein-iscsi-security-00.txt Date: Fri, 21 Jul 2000 19:08:36 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk fyi ------- Forwarded Message Date: Fri, 21 Jul 2000 09:40:04 -0400 From: Internet-Drafts@ietf.org To: IETF-Announce: ; Subject: I-D ACTION:draft-klein-iscsi-security-00.txt - --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : iSCSI Security Protocol Author(s) : Y. Klein, E. Felstaine Filename : draft-klein-iscsi-security-00.txt Pages : 27 Date : 20-Jul-00 The iSCSI Security Layer is an SSH based protocol for secure, iSCSI aware, network services over an insecure storage network. The protocol supports login, server authentication, integrity protection and encryption, thus enables application such as Virtual Private Storage Area Networks (VPSAN). Key exchange method, public key algorithm, symmetric encryption algorithm, message authentication algorithm, and hash algorithm are all negotiated. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-klein-iscsi-security-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-klein-iscsi-security-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-klein-iscsi-security-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. - --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" - --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20000720141821.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-klein-iscsi-security-00.txt - --OtherAccess Content-Type: Message/External-body; name="draft-klein-iscsi-security-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20000720141821.I-D@ietf.org> - --OtherAccess-- - --NextPart-- ------- End of Forwarded Message From owner-ips@ECE.cmu.edu Sat Jul 22 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18403; 21 Jul 2000 22:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA14287 for ips-outgoing; Fri, 21 Jul 2000 20:46:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hplms26.hpl.hp.com (hplms26.hpl.hp.com [15.255.168.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA14263 for ; Fri, 21 Jul 2000 20:46:46 -0400 (EDT) Received: from hplajw.hpl.hp.com (hplajw.hpl.hp.com [15.4.88.193]) by hplms26.hpl.hp.com (8.9.3 (PHNE_18979)/HPL-PA Relay) with ESMTP id RAA23747 for ; Fri, 21 Jul 2000 17:46:45 -0700 (PDT) Received: (from wilkes@localhost) by hplajw.hpl.hp.com (8.9.3 (PHNE_18979)/8.9.3 HPLabs Workstation) id RAA15157; Fri, 21 Jul 2000 17:46:44 -0700 (PDT) Date: Fri, 21 Jul 2000 17:46:44 -0700 (PDT) Message-Id: <200007220046.RAA15157@hplajw.hpl.hp.com> From: john wilkes To: ips@ece.cmu.edu Subject: Re: storage-device QoS [was: IPS Draft Charter update] Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, Like you, I believe that defining - and controlling - the QoS specification for a storage device is indeed an interestingly different problem than doing the same thing for network traffic. Indeed, we've been researching this topic for some time now :-). (See for a little more data.) However, I'd suggest that storage device QoS specification seems to be outside the immediate scope of the IPS work. This seems clearly focussed on achieving transport of existing block storage protocols (such as SCSI) across an IP-based network. Expressing - and controlling - storage device QoS seems to be something that might better be pursued in the T10 forum, where it would probably show up as extensions to the SCSI device mode pages. Given that changing SCSI itself is expressly outside the scope of the IPS WG charter, this seems to place this issue outside it's scope too. john From owner-ips@ECE.cmu.edu Sat Jul 22 04:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18804; 22 Jul 2000 0:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA18436 for ips-outgoing; Fri, 21 Jul 2000 23:06:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.5]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA18421 for ; Fri, 21 Jul 2000 23:06:44 -0400 (EDT) Received: from pirus.com (d7.isdn-1.mbo.ma.ultra.net [146.115.101.199]) by apollo.pirus.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id N5N6J4SQ; Fri, 21 Jul 2000 23:15:46 -0400 Message-ID: <39790F36.81BC1A2B@pirus.com> Date: Fri, 21 Jul 2000 23:04:22 -0400 From: Howard Hall Reply-To: howard@pirus.com X-Mailer: Mozilla 4.5 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: john wilkes CC: ips@ece.cmu.edu Subject: Re: storage-device QoS [was: IPS Draft Charter update] References: <200007220046.RAA15157@hplajw.hpl.hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk We too have been looking into this issue, but have come to a different conclusion. The propagation of QoS information through SCSI ignores the transport protocol, the network layer, all the IP aware devices and QoS magic that may allow for the reduction of latency that certain applications demand. I can envision lots of iSCSI aware devices out there that don't care about the encapsulated protocol, but sure may want to get involved at the network/transport layer and do their value add. These devices may know far better than the SCSI layer queue depths, windows, session loading, lan/wan bandwidth utilization, etc. I can also envision initiator application software that is iSCSI aware. These applications may know far better than anyone the block content and its significance. The scope of QoS inclusions could be as simple as a guideline for the setting of TOS bits or a high priority channel, or the ability to push real flow control and apply end-to-end priority queuing between the target and the initiator through the iSCSI protocol. We won't know unless we explore it. Howard Hall Pirus Networks www.pirus.com john wilkes wrote: > Mark, > > Like you, I believe that defining - and controlling - the QoS > specification for a storage device is indeed an interestingly > different problem than doing the same thing for network traffic. > Indeed, we've been researching this topic for some time now :-). (See > for a little more data.) > > However, I'd suggest that storage device QoS specification seems to be > outside the immediate scope of the IPS work. This seems clearly > focussed on achieving transport of existing block storage protocols > (such as SCSI) across an IP-based network. > > Expressing - and controlling - storage device QoS seems to be > something that might better be pursued in the T10 forum, where it > would probably show up as extensions to the SCSI device mode pages. > Given that changing SCSI itself is expressly outside the scope of the > IPS WG charter, this seems to place this issue outside it's scope too. > > john From owner-ips@ECE.cmu.edu Sat Jul 22 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21494; 22 Jul 2000 12:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA12813 for ips-outgoing; Sat, 22 Jul 2000 11:37:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA12791 for ; Sat, 22 Jul 2000 11:37:38 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id BDCAF482; Sat, 22 Jul 2000 08:37:37 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id IAA22685; Sat, 22 Jul 2000 08:36:29 -0700 (PDT) Message-Id: <4.2.2.20000722062021.00b07b40@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Sat, 22 Jul 2000 06:33:05 -0700 To: howard@pirus.com From: Michael Krause Subject: Re: storage-device QoS [was: IPS Draft Charter update] Cc: ips@ece.cmu.edu In-Reply-To: <39790F36.81BC1A2B@pirus.com> References: <200007220046.RAA15157@hplajw.hpl.hp.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:04 PM 7/21/00 -0400, Howard Hall wrote: >We too have been looking into this issue, but have come to a different >conclusion. The propagation of QoS information through SCSI ignores the >transport protocol, the network layer, all the IP aware devices and QoS >magic that may allow for the reduction of latency that certain >applications demand. I can envision lots of iSCSI aware devices out >there that don't care about the encapsulated protocol, but sure may want >to get involved at the network/transport layer and do their value add. >These devices may know far better than the SCSI layer queue depths, >windows, session loading, lan/wan bandwidth utilization, etc. I can also >envision initiator application software that is iSCSI aware. These >applications may know far better than anyone the block content and its >significance. > >The scope of QoS inclusions could be as simple as a guideline for the >setting of TOS bits or a high priority channel, or the ability to push >real flow control and apply end-to-end priority queuing between the target >and the initiator through the iSCSI protocol. We won't know unless we >explore it. For many of the items mentioned, it will take some time to develop the modeling and prototypes to validate the concepts proposed by a given QoS policy. If QoS needs to be addressed within the first version, then perhaps the focus should be on the interfaces to communicate QoS to the iSCSI layer. This would allow designs to include QoS information in a standard way while not being locked into a model that may not be suitable for all of the implementations envisioned - perhaps being too simplistic or overly complex. As such the actual QoS policies would be better implemented within a separate specification while the QoS interface to iSCSI is defined within this spec. Note: You might want to expand the concept of QoS to also include attributes of the fabric that can make a difference in these policies beyond what you had listed, e.g. there may be N paths between endnodes which support different bandwidths, have different congestion (response times), have different costs, etc. QoS might want to address how one arbitrates commands across a set of paths, how it changes should a hot-plug/removal occur, etc. The workgroup will need to define what QoS means which I would contend is much more than a mapping of priority reqs to the TOS or TClass fields. Mike From owner-ips@ECE.cmu.edu Sat Jul 22 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21875; 22 Jul 2000 14:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA16355 for ips-outgoing; Sat, 22 Jul 2000 13:37:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ligarius-fe0.ultra.net (ligarius-fe0.ultra.net [146.115.8.189]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA16337 for ; Sat, 22 Jul 2000 13:36:52 -0400 (EDT) Received: from howard ([63.91.118.41]) by ligarius-fe0.ultra.net (8.8.8/ult/n26500/mtc.v2) with SMTP id NAA09448; Sat, 22 Jul 2000 13:36:24 -0400 (EDT) Reply-To: From: "Howard Hall" To: , Subject: RE: IPS Draft Charter update Date: Sat, 22 Jul 2000 13:45:58 -0400 Message-ID: <618471D08ABDD31188B6009027E500930F128E@apollo.pirus.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal In-Reply-To: <618471D08ABDD31188B6009027E500932B61D4@apollo.pirus.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, Discovery may be ancillary, but it I believe it will be critical to the acceptance of the protocol. Is there a way to keep discovery in the charter without it becoming a bottle neck for a first protocol version? Is this what you are planning on doing? I am hesitant to support the removal of it all together given its importance. -Howard -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Black_David@emc.com Sent: Friday, July 21, 2000 3:00 PM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: RE: IPS Draft Charter update > I am however concerned that if we include all the work items - including > discovery - in the first version we might either miss our schedule or have > a half baked solution. > > Can we exclude those from the first version and add them (either as a > separate/complementary spec) in the MIB timeframe? Sure - discovery sounds "ancillary" to me. In any case I've been concerned that the milestones were aggressive to the point of unrealistic, and given this feedback, I'll stretch them out and try to make the next charter version more explicit about what has to be done to get the first protocol version out - that does include security. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sat Jul 22 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22210; 22 Jul 2000 16:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA17194 for ips-outgoing; Sat, 22 Jul 2000 14:15:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ligarius-fe0.ultra.net (ligarius-fe0.ultra.net [146.115.8.189]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA17178 for ; Sat, 22 Jul 2000 14:15:23 -0400 (EDT) Received: from howard ([63.91.118.41]) by ligarius-fe0.ultra.net (8.8.8/ult/n26500/mtc.v2) with SMTP id OAA32139; Sat, 22 Jul 2000 14:11:05 -0400 (EDT) Reply-To: From: "Howard Hall" To: "'Michael Krause'" Cc: Subject: RE: storage-device QoS [was: IPS Draft Charter update] Date: Sat, 22 Jul 2000 14:20:38 -0400 Message-ID: <618471D08ABDD31188B6009027E500930F128F@apollo.pirus.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal In-Reply-To: <618471D08ABDD31188B6009027E500932B61E4@apollo.pirus.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, Focusing on the interfaces to communicate QoS to the iSCSI layer sounds like a good first step to me. I agree that attributes of the fabric extended through the network is a desirable addition to those I listed. It is also clear the TOS or TClass fields will not cut it in the more complicated (and desirable) QoS concepts. -Howard Howard Hall Pirus Networks www.pirus.com -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Michael Krause Sent: Saturday, July 22, 2000 9:33 AM To: Hall, Howard Cc: ips@ece.cmu.edu Subject: Re: storage-device QoS [was: IPS Draft Charter update] At 11:04 PM 7/21/00 -0400, Howard Hall wrote: >We too have been looking into this issue, but have come to a different >conclusion. The propagation of QoS information through SCSI ignores the >transport protocol, the network layer, all the IP aware devices and QoS >magic that may allow for the reduction of latency that certain >applications demand. I can envision lots of iSCSI aware devices out >there that don't care about the encapsulated protocol, but sure may want >to get involved at the network/transport layer and do their value add. >These devices may know far better than the SCSI layer queue depths, >windows, session loading, lan/wan bandwidth utilization, etc. I can also >envision initiator application software that is iSCSI aware. These >applications may know far better than anyone the block content and its >significance. > >The scope of QoS inclusions could be as simple as a guideline for the >setting of TOS bits or a high priority channel, or the ability to push >real flow control and apply end-to-end priority queuing between the target >and the initiator through the iSCSI protocol. We won't know unless we >explore it. For many of the items mentioned, it will take some time to develop the modeling and prototypes to validate the concepts proposed by a given QoS policy. If QoS needs to be addressed within the first version, then perhaps the focus should be on the interfaces to communicate QoS to the iSCSI layer. This would allow designs to include QoS information in a standard way while not being locked into a model that may not be suitable for all of the implementations envisioned - perhaps being too simplistic or overly complex. As such the actual QoS policies would be better implemented within a separate specification while the QoS interface to iSCSI is defined within this spec. Note: You might want to expand the concept of QoS to also include attributes of the fabric that can make a difference in these policies beyond what you had listed, e.g. there may be N paths between endnodes which support different bandwidths, have different congestion (response times), have different costs, etc. QoS might want to address how one arbitrates commands across a set of paths, how it changes should a hot-plug/removal occur, etc. The workgroup will need to define what QoS means which I would contend is much more than a mapping of priority reqs to the TOS or TClass fields. Mike From owner-ips@ECE.cmu.edu Sun Jul 23 17:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26643; 23 Jul 2000 13:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA14732 for ips-outgoing; Sun, 23 Jul 2000 11:38:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA14711 for ; Sun, 23 Jul 2000 11:37:56 -0400 (EDT) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id JAA25074; Sun, 23 Jul 2000 09:37:40 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id JAA28548; Sun, 23 Jul 2000 09:37:39 -0600 (MDT) Received: from central.sun.com (hobo21.Central.Sun.COM [129.147.8.21]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id JAA16197; Sun, 23 Jul 2000 09:24:14 -0600 (MDT) Message-ID: <397B12CF.D10DD86E@central.sun.com> Date: Sun, 23 Jul 2000 09:44:15 -0600 From: "Mark A. Carlson" Reply-To: mark.carlson@central.sun.com Organization: Sun Microsystems, inc. X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: hhall@ultranet.com CC: "'Michael Krause'" , ips@ece.cmu.edu Subject: Re: storage-device QoS [was: IPS Draft Charter update] References: <618471D08ABDD31188B6009027E500930F128F@apollo.pirus.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk While adding storage QoS to the SCSI mode pages would have the advantage of working both with IP and FC transports, I would hope that we could leverage some of the network QoS that has already been done in the Policy Framework group with models. This is what I hope is in scope for our charter, even though we obviously wouldn't do it in the first deliverable. -- mark Howard Hall wrote: > > Mike, > > Focusing on the interfaces to communicate QoS to the iSCSI layer sounds like > a good first step to me. I agree that attributes of the fabric extended > through the network is a desirable addition to those I listed. It is also > clear the TOS or TClass fields will not cut it in the more complicated (and > desirable) QoS concepts. > > -Howard > Howard Hall > Pirus Networks > www.pirus.com > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Michael Krause > Sent: Saturday, July 22, 2000 9:33 AM > To: Hall, Howard > Cc: ips@ece.cmu.edu > Subject: Re: storage-device QoS [was: IPS Draft Charter update] > > At 11:04 PM 7/21/00 -0400, Howard Hall wrote: > >We too have been looking into this issue, but have come to a different > >conclusion. The propagation of QoS information through SCSI ignores the > >transport protocol, the network layer, all the IP aware devices and QoS > >magic that may allow for the reduction of latency that certain > >applications demand. I can envision lots of iSCSI aware devices out > >there that don't care about the encapsulated protocol, but sure may want > >to get involved at the network/transport layer and do their value add. > >These devices may know far better than the SCSI layer queue depths, > >windows, session loading, lan/wan bandwidth utilization, etc. I can also > >envision initiator application software that is iSCSI aware. These > >applications may know far better than anyone the block content and its > >significance. > > > >The scope of QoS inclusions could be as simple as a guideline for the > >setting of TOS bits or a high priority channel, or the ability to push > >real flow control and apply end-to-end priority queuing between the target > >and the initiator through the iSCSI protocol. We won't know unless we > >explore it. > > For many of the items mentioned, it will take some time to develop the > modeling and prototypes to validate the concepts proposed by a given QoS > policy. If QoS needs to be addressed within the first version, then > perhaps the focus should be on the interfaces to communicate QoS to the > iSCSI layer. This would allow designs to include QoS information in a > standard way while not being locked into a model that may not be suitable > for all of the implementations envisioned - perhaps being too simplistic or > overly complex. As such the actual QoS policies would be better > implemented within a separate specification while the QoS interface to > iSCSI is defined within this spec. > > Note: You might want to expand the concept of QoS to also include > attributes of the fabric that can make a difference in these policies > beyond what you had listed, e.g. there may be N paths between endnodes > which support different bandwidths, have different congestion (response > times), have different costs, etc. QoS might want to address how one > arbitrates commands across a set of paths, how it changes should a > hot-plug/removal occur, etc. The workgroup will need to define what QoS > means which I would contend is much more than a mapping of priority reqs to > the TOS or TClass fields. > > Mike From owner-ips@ECE.cmu.edu Sun Jul 23 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26759; 23 Jul 2000 13:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA15971 for ips-outgoing; Sun, 23 Jul 2000 12:59:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA15954 for ; Sun, 23 Jul 2000 12:58:52 -0400 (EDT) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id KAA07504; Sun, 23 Jul 2000 10:58:49 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id KAA01170; Sun, 23 Jul 2000 10:58:48 -0600 (MDT) Received: from central.sun.com (hobo21.Central.Sun.COM [129.147.8.21]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id KAA16800; Sun, 23 Jul 2000 10:45:24 -0600 (MDT) Message-ID: <397B25D5.3CF7F7AA@central.sun.com> Date: Sun, 23 Jul 2000 11:05:25 -0600 From: "Mark A. Carlson" Reply-To: mark.carlson@central.sun.com Organization: Sun Microsystems, inc. X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: storage-device QoS [was: IPS Draft Charter update] References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > > I agree that QoS (and the way storage will want to define SLAs) will be > important to this group and there is no other forum where we can leverage > QoS better than IETF. > > However I think that for practical reasons we should postpone this until > after we are able to outline our discovery, basic management interaction > with "control points" etc. Agreed. > > The SLA issue is getting more mature and IETF is better positioned to > handle it > than any other standards body. > > I think that our chairmen would like to add a "Phase 3" to our schedule to > do this. Sounds good to me. -- mark From owner-ips@ECE.cmu.edu Sun Jul 23 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26818; 23 Jul 2000 13:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA15720 for ips-outgoing; Sun, 23 Jul 2000 12:42:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA15704 for ; Sun, 23 Jul 2000 12:42:39 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA55256 for ; Sun, 23 Jul 2000 18:42:07 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id SAA38428 for ; Sun, 23 Jul 2000 18:42:07 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256925.005BB738 ; Sun, 23 Jul 2000 18:41:46 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 23 Jul 2000 19:37:55 +0300 Subject: Re: storage-device QoS [was: IPS Draft Charter update] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree that QoS (and the way storage will want to define SLAs) will be important to this group and there is no other forum where we can leverage QoS better than IETF. However I think that for practical reasons we should postpone this until after we are able to outline our discovery, basic management interaction with "control points" etc. The SLA issue is getting more mature and IETF is better positioned to handle it than any other standards body. I think that our chairmen would like to add a "Phase 3" to our schedule to do this. Regards, Julo "Mark A. Carlson" on 23/07/2000 18:44:15 Please respond to mark.carlson@central.sun.com To: hhall@ultranet.com cc: "'Michael Krause'" , ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: storage-device QoS [was: IPS Draft Charter update] While adding storage QoS to the SCSI mode pages would have the advantage of working both with IP and FC transports, I would hope that we could leverage some of the network QoS that has already been done in the Policy Framework group with models. This is what I hope is in scope for our charter, even though we obviously wouldn't do it in the first deliverable. -- mark Howard Hall wrote: > > Mike, > > Focusing on the interfaces to communicate QoS to the iSCSI layer sounds like > a good first step to me. I agree that attributes of the fabric extended > through the network is a desirable addition to those I listed. It is also > clear the TOS or TClass fields will not cut it in the more complicated (and > desirable) QoS concepts. > > -Howard > Howard Hall > Pirus Networks > www.pirus.com > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Michael Krause > Sent: Saturday, July 22, 2000 9:33 AM > To: Hall, Howard > Cc: ips@ece.cmu.edu > Subject: Re: storage-device QoS [was: IPS Draft Charter update] > > At 11:04 PM 7/21/00 -0400, Howard Hall wrote: > >We too have been looking into this issue, but have come to a different > >conclusion. The propagation of QoS information through SCSI ignores the > >transport protocol, the network layer, all the IP aware devices and QoS > >magic that may allow for the reduction of latency that certain > >applications demand. I can envision lots of iSCSI aware devices out > >there that don't care about the encapsulated protocol, but sure may want > >to get involved at the network/transport layer and do their value add. > >These devices may know far better than the SCSI layer queue depths, > >windows, session loading, lan/wan bandwidth utilization, etc. I can also > >envision initiator application software that is iSCSI aware. These > >applications may know far better than anyone the block content and its > >significance. > > > >The scope of QoS inclusions could be as simple as a guideline for the > >setting of TOS bits or a high priority channel, or the ability to push > >real flow control and apply end-to-end priority queuing between the target > >and the initiator through the iSCSI protocol. We won't know unless we > >explore it. > > For many of the items mentioned, it will take some time to develop the > modeling and prototypes to validate the concepts proposed by a given QoS > policy. If QoS needs to be addressed within the first version, then > perhaps the focus should be on the interfaces to communicate QoS to the > iSCSI layer. This would allow designs to include QoS information in a > standard way while not being locked into a model that may not be suitable > for all of the implementations envisioned - perhaps being too simplistic or > overly complex. As such the actual QoS policies would be better > implemented within a separate specification while the QoS interface to > iSCSI is defined within this spec. > > Note: You might want to expand the concept of QoS to also include > attributes of the fabric that can make a difference in these policies > beyond what you had listed, e.g. there may be N paths between endnodes > which support different bandwidths, have different congestion (response > times), have different costs, etc. QoS might want to address how one > arbitrates commands across a set of paths, how it changes should a > hot-plug/removal occur, etc. The workgroup will need to define what QoS > means which I would contend is much more than a mapping of priority reqs to > the TOS or TClass fields. > > Mike From owner-ips@ECE.cmu.edu Mon Jul 24 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30104; 24 Jul 2000 0:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA21607 for ips-outgoing; Sun, 23 Jul 2000 23:27:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA21584 for ; Sun, 23 Jul 2000 23:27:43 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Sun, 23 Jul 2000 23:27:11 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FB8F@corpmx9.isus.emc.com> To: hhall@ultranet.com, Black_David@emc.com, ips@ece.cmu.edu Subject: RE: IPS Draft Charter update Date: Sun, 23 Jul 2000 23:27:10 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Discovery may be ancillary, but it I believe it will be critical to the > acceptance of the protocol. Is there a way to keep discovery in the charter > without it becoming a bottle neck for a first protocol version? Is this > what you are planning on doing? I am hesitant to support the removal of it > all together given its importance. That is exactly what I'm planning on doing. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Jul 24 15:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32446; 24 Jul 2000 11:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA13893 for ips-outgoing; Mon, 24 Jul 2000 10:24:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail-gw.hursley.ibm.com (mail-gw.hursley.ibm.com [194.196.110.15]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA13875 for ; Mon, 24 Jul 2000 10:24:42 -0400 (EDT) Received: from sp3at21.hursley.ibm.com (sp3at21.hursley.ibm.com [9.20.45.21]) by mail-gw.hursley.ibm.com (AIX4.3/UCB 8.8.8/8.8.8) with ESMTP id PAA30890; Mon, 24 Jul 2000 15:23:20 +0100 Received: from hursley.ibm.com (gsine01.us.sine.ibm.com [9.14.6.41]) by sp3at21.hursley.ibm.com (AIX4.2/UCB 8.7/8.7.3) with ESMTP id PAA22004; Mon, 24 Jul 2000 15:23:51 +0100 (BST) Message-ID: <397C50AF.5E232309@hursley.ibm.com> Date: Mon, 24 Jul 2000 09:20:31 -0500 From: Brian E Carpenter Organization: IBM X-Mailer: Mozilla 4.61 [en] (Win98; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: hhall@ultranet.com CC: "'Michael Krause'" , ips@ece.cmu.edu Subject: Re: storage-device QoS [was: IPS Draft Charter update] References: <618471D08ABDD31188B6009027E500930F128F@apollo.pirus.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Howard Hall wrote: ... > It is also > clear the TOS or TClass fields will not cut it in the more complicated (and > desirable) QoS concepts. This is a pretty unclear comment. Apart from anything else, the IETF obsoleted the old concept of the TOS field in RFC 2474. The replacement concept of the Differentiated Services field (which applies identically to IPv4 and IPv6) is considerably more complex and subtle, and goes way beyond the notion of simple priority that has been mentioned on this thread. I agree that this question should be deferred to a later stage. Brian From owner-ips@ECE.cmu.edu Mon Jul 24 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32715; 24 Jul 2000 13:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA16923 for ips-outgoing; Mon, 24 Jul 2000 11:34:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antiochus-fe0.ultra.net (antiochus-fe0.ultra.net [146.115.8.188]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA16902 for ; Mon, 24 Jul 2000 11:33:54 -0400 (EDT) Received: from howard ([63.91.118.41]) by antiochus-fe0.ultra.net (8.8.8/ult/n20340/mtc.v2) with SMTP id LAA09705; Mon, 24 Jul 2000 11:29:37 -0400 (EDT) Reply-To: From: "Howard Hall" To: "'Brian E Carpenter'" Cc: , , Subject: RE: storage-device QoS [was: IPS Draft Charter update] Date: Mon, 24 Jul 2000 11:39:30 -0400 Message-ID: <618471D08ABDD31188B6009027E500930F1294@apollo.pirus.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300 Importance: Normal In-Reply-To: <618471D08ABDD31188B6009027E500932B6203@apollo.pirus.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk The comment should have read "...will not cut it ALONE in the more complicated.." and was in reference to Mike's comments about just mapping of priority reqs to these fields. I've read 2474 in the past, and I admit I am rusty on it, I will read it again. From what I remember, some of the more complicated concepts of QoS we have thought of here at Pirus will need to extend into the protocol, in some cases complementing the DS fields. Dave: How widespread is the adoption and use of 2474? -Howard Howard Hall Pirus Networks www.pirus.com -----Original Message----- From: Brian E Carpenter [mailto:brian@hursley.ibm.com] Sent: Monday, July 24, 2000 10:21 AM To: hhall@ultranet.com Cc: 'Michael Krause'; ips@ece.cmu.edu Subject: Re: storage-device QoS [was: IPS Draft Charter update] Howard Hall wrote: ... > It is also > clear the TOS or TClass fields will not cut it in the more complicated (and > desirable) QoS concepts. This is a pretty unclear comment. Apart from anything else, the IETF obsoleted the old concept of the TOS field in RFC 2474. The replacement concept of the Differentiated Services field (which applies identically to IPv4 and IPv6) is considerably more complex and subtle, and goes way beyond the notion of simple priority that has been mentioned on this thread. I agree that this question should be deferred to a later stage. Brian From owner-ips@ECE.cmu.edu Mon Jul 24 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00714; 24 Jul 2000 15:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA23350 for ips-outgoing; Mon, 24 Jul 2000 14:15:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA23320 for ; Mon, 24 Jul 2000 14:14:58 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id <3SR1Q34B>; Mon, 24 Jul 2000 14:14:19 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0703C4EECA@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: Draft Pgh. Agenda and charter Date: Mon, 24 Jul 2000 14:14:17 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Draft agenda for Pittsburgh along with latest rev. to proposed charter follow (both should appear on the agenda area of the IETF web site). Important Agenda notes: The purpose of the Overview and Rationale talks is review of major design decisions. The 15 minutes available for FC-over-IP limits discussion to major issues; detailed review of FC-over-IP will not be done during the ips meeting. The 15 minute slots should be no more than 10 min of presentation; iSCSI Multiple Channels and Error Recovery should be no more than 5 min of presentation. Presenters should assume that the audience has read the drafts, and hence only cover material necessary to get to design decisions, questions, issues, etc. The 5 min time slots are for updates only; clarification questions will be entertained if time permits, but discussion will need to take place on the mailing list. Expect time slots to be enforced in some fashion. Could those who are going to be presenting each of the iSCSI items on the agenda please contact me and let me know who's doing what. I believe I know who the presenters are for the other segments. Thanks, --David IP Storage (ips) BOF DRAFT Agenda - subject to change -- Organizational Matters (20 min) 5 min Agenda Bashing 5 min A few words from the AD 10 min Charter Bashing Draft charter for bashing is appended to this agenda. Please bash the charter on the mailing list in preference to in the meeting, as the charter bashing time can be productively used for other purposes. -- Internet SCSI (iSCSI) (80 min) draft-haagens-ips-iscsireqs-00.txt draft-satran-iscsi-01.txt draft-bakke-iscsimib-00.txt 15 min iSCSI Overview and Rationale 30 min iSCSI Security, may include draft-klein-iscsi-security-01.txt 15 min iSCSI Multiple Channels 15 min iSCSI Error Recovery 5 min iSCSI MIB -- Related Matters (20 min) 15 min FC-over-IP Overview and Rationale draft-ietf-ipfc-fcoverip-02.txt 5 min SEP and Parallel SCSI update draft-wilson-sep-00.txt IP Storage (ips) Working Group Proposed Charter - DRAFT 5 There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for shared traffic network environments, such as the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks, including both discovery of storage for access by the discovering entity, and discovery for management. - Management, including appropriate MIB definition. The WG will address security and congestion control as an integral part of its protocol(s); naming, discovery, and management are important related issues, but may be addressed in companion documents. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). The WG will strive to support the approaches to discovery, multi-pathing, and booting taken by the existing block storage protocols it encapsulates at the levels of those protocols. It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Discuss selection of encapsulation approach or approaches. Sep 00 Submit initial version of framework draft on encapsulation approaches. Dec 00 Discuss framework draft, as well as requirements, applicability and protocol specification drafts for at least one protocol encapsulation at IETF meeting in San Diego. Feb 01 Submit requirements, applicability, and protocol specification drafts for at least one protocol encapsulation to the IESG for consideration as a Proposed Standard. Mar 01 Discuss related drafts (e.g., MIBs, discovery) for at least the first protocol encapsulation at IETF meeting in Minneapolis. Jun 01 Submit MIB and discovery draft to the IESG. Begin revision of WG charter as appropriate in consultation with ADs. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Mon Jul 24 23:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01328; 24 Jul 2000 18:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA29593 for ips-outgoing; Mon, 24 Jul 2000 16:39:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA29576 for ; Mon, 24 Jul 2000 16:39:19 -0400 (EDT) From: mako@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id QAA76794 for ; Mon, 24 Jul 2000 16:28:01 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id OAA82628 for ; Mon, 24 Jul 2000 14:39:18 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256926.0071754F ; Mon, 24 Jul 2000 14:39:15 -0600 X-Lotus-FromDomain: IBMUS To: ips@ece.cmu.edu Message-ID: <87256926.00716DAB.00@d53mta03h.boulder.ibm.com> Date: Mon, 24 Jul 2000 13:17:00 -0700 Subject: TSID in Login Response Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk There appears to be a minor inconsistency relating to TSID in the Login Response (sec. 3.14) in the latest iSCSI draft. In sec. 3.14.2 on pg. 40, it stated that "The TSID is an initiator identifying tag set by the target. A 0 in the returned TSID indicates that either ... In both cases the target is rejecting the login." But in sec. 3.14.1 on pg. 39, it stated that "[InitStatRN] issignificant only if TSID is 0". Maybe the wording should have been "if TSID is non-zero"? Mike From owner-ips@ECE.cmu.edu Mon Jul 24 23:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01511; 24 Jul 2000 19:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA03617 for ips-outgoing; Mon, 24 Jul 2000 18:27:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA03597 for ; Mon, 24 Jul 2000 18:27:12 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id AAA305480; Tue, 25 Jul 2000 00:26:40 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id AAA196452; Tue, 25 Jul 2000 00:26:37 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256926.007B448B ; Tue, 25 Jul 2000 00:26:24 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: ips@ece.cmu.edu Message-ID: Date: Tue, 25 Jul 2000 00:37:50 +0300 Subject: Re: Draft Pgh. Agenda and charter Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, It looks good. I still don't understand the FC stuff and I assume that for the world at large a good encapsulation of serial ATA would be more important, and SSA - a very widely deployed ANSI standard protocol - has in this time frame the at least same merit as FC but I'll let it be! As for the agenda: I assume that Randy will need at least 15 minutes for the requirements (Randy ?) and that storage "consumers" (SSPs, large users like National Labs) will join the discussion (or you will invite them). I have a already prepared some of the summaries for overview/design considerations/ multiple channels. Can we have those in one block (i was thinking about 25-30 min)? Security and some of the open things will certainly fill the remaining time. Thanks, Julo Black_David@emc.com on 24/07/2000 21:14:17 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Draft Pgh. Agenda and charter Draft agenda for Pittsburgh along with latest rev. to proposed charter follow (both should appear on the agenda area of the IETF web site). Important Agenda notes: The purpose of the Overview and Rationale talks is review of major design decisions. The 15 minutes available for FC-over-IP limits discussion to major issues; detailed review of FC-over-IP will not be done during the ips meeting. The 15 minute slots should be no more than 10 min of presentation; iSCSI Multiple Channels and Error Recovery should be no more than 5 min of presentation. Presenters should assume that the audience has read the drafts, and hence only cover material necessary to get to design decisions, questions, issues, etc. The 5 min time slots are for updates only; clarification questions will be entertained if time permits, but discussion will need to take place on the mailing list. Expect time slots to be enforced in some fashion. Could those who are going to be presenting each of the iSCSI items on the agenda please contact me and let me know who's doing what. I believe I know who the presenters are for the other segments. Thanks, --David IP Storage (ips) BOF DRAFT Agenda - subject to change -- Organizational Matters (20 min) 5 min Agenda Bashing 5 min A few words from the AD 10 min Charter Bashing Draft charter for bashing is appended to this agenda. Please bash the charter on the mailing list in preference to in the meeting, as the charter bashing time can be productively used for other purposes. -- Internet SCSI (iSCSI) (80 min) draft-haagens-ips-iscsireqs-00.txt draft-satran-iscsi-01.txt draft-bakke-iscsimib-00.txt 15 min iSCSI Overview and Rationale 30 min iSCSI Security, may include draft-klein-iscsi-security-01.txt 15 min iSCSI Multiple Channels 15 min iSCSI Error Recovery 5 min iSCSI MIB -- Related Matters (20 min) 15 min FC-over-IP Overview and Rationale draft-ietf-ipfc-fcoverip-02.txt 5 min SEP and Parallel SCSI update draft-wilson-sep-00.txt IP Storage (ips) Working Group Proposed Charter - DRAFT 5 There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for shared traffic network environments, such as the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks, including both discovery of storage for access by the discovering entity, and discovery for management. - Management, including appropriate MIB definition. The WG will address security and congestion control as an integral part of its protocol(s); naming, discovery, and management are important related issues, but may be addressed in companion documents. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). The WG will strive to support the approaches to discovery, multi-pathing, and booting taken by the existing block storage protocols it encapsulates at the levels of those protocols. It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Discuss selection of encapsulation approach or approaches. Sep 00 Submit initial version of framework draft on encapsulation approaches. Dec 00 Discuss framework draft, as well as requirements, applicability and protocol specification drafts for at least one protocol encapsulation at IETF meeting in San Diego. Feb 01 Submit requirements, applicability, and protocol specification drafts for at least one protocol encapsulation to the IESG for consideration as a Proposed Standard. Mar 01 Discuss related drafts (e.g., MIBs, discovery) for at least the first protocol encapsulation at IETF meeting in Minneapolis. Jun 01 Submit MIB and discovery draft to the IESG. Begin revision of WG charter as appropriate in consultation with ADs. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Mon Jul 24 23:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01455; 24 Jul 2000 19:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA03635 for ips-outgoing; Mon, 24 Jul 2000 18:27:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA03601 for ; Mon, 24 Jul 2000 18:27:14 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id AAA130816 for ; Tue, 25 Jul 2000 00:26:41 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id AAA227200 for ; Tue, 25 Jul 2000 00:26:41 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256926.007B4905 ; Tue, 25 Jul 2000 00:26:36 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 25 Jul 2000 00:53:29 +0300 Subject: Re: TSID in Login Response Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, Unfortunate wording. It should say that it is significant only as a response to a login with TSID 0 (the first in a session). Will correct. Thanks, Julo mako@almaden.ibm.com on 24/07/2000 23:17:00 Please respond to mako@almaden.ibm.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: TSID in Login Response There appears to be a minor inconsistency relating to TSID in the Login Response (sec. 3.14) in the latest iSCSI draft. In sec. 3.14.2 on pg. 40, it stated that "The TSID is an initiator identifying tag set by the target. A 0 in the returned TSID indicates that either ... In both cases the target is rejecting the login." But in sec. 3.14.1 on pg. 39, it stated that "[InitStatRN] issignificant only if TSID is 0". Maybe the wording should have been "if TSID is non-zero"? Mike From owner-ips@ECE.cmu.edu Tue Jul 25 04:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02435; 25 Jul 2000 0:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA13947 for ips-outgoing; Mon, 24 Jul 2000 23:17:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA13931 for ; Mon, 24 Jul 2000 23:17:16 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id UAA02285; Mon, 24 Jul 2000 20:14:04 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id UAA05522; Mon, 24 Jul 2000 20:13:43 -0700 From: "Douglas Otis" To: , Cc: Subject: RE: Draft Pgh. Agenda and charter Date: Mon, 24 Jul 2000 20:19:08 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Are you suggesting iSCSI should distribute the DMA context from the initiator adapter to the target device? Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, July 24, 2000 2:38 PM To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: Draft Pgh. Agenda and charter David, It looks good. I still don't understand the FC stuff and I assume that for the world at large a good encapsulation of serial ATA would be more important, and SSA - a very widely deployed ANSI standard protocol - has in this time frame the at least same merit as FC but I'll let it be! As for the agenda: I assume that Randy will need at least 15 minutes for the requirements (Randy ?) and that storage "consumers" (SSPs, large users like National Labs) will join the discussion (or you will invite them). I have a already prepared some of the summaries for overview/design considerations/ multiple channels. Can we have those in one block (i was thinking about 25-30 min)? Security and some of the open things will certainly fill the remaining time. Thanks, Julo Black_David@emc.com on 24/07/2000 21:14:17 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Draft Pgh. Agenda and charter Draft agenda for Pittsburgh along with latest rev. to proposed charter follow (both should appear on the agenda area of the IETF web site). Important Agenda notes: The purpose of the Overview and Rationale talks is review of major design decisions. The 15 minutes available for FC-over-IP limits discussion to major issues; detailed review of FC-over-IP will not be done during the ips meeting. The 15 minute slots should be no more than 10 min of presentation; iSCSI Multiple Channels and Error Recovery should be no more than 5 min of presentation. Presenters should assume that the audience has read the drafts, and hence only cover material necessary to get to design decisions, questions, issues, etc. The 5 min time slots are for updates only; clarification questions will be entertained if time permits, but discussion will need to take place on the mailing list. Expect time slots to be enforced in some fashion. Could those who are going to be presenting each of the iSCSI items on the agenda please contact me and let me know who's doing what. I believe I know who the presenters are for the other segments. Thanks, --David IP Storage (ips) BOF DRAFT Agenda - subject to change -- Organizational Matters (20 min) 5 min Agenda Bashing 5 min A few words from the AD 10 min Charter Bashing Draft charter for bashing is appended to this agenda. Please bash the charter on the mailing list in preference to in the meeting, as the charter bashing time can be productively used for other purposes. -- Internet SCSI (iSCSI) (80 min) draft-haagens-ips-iscsireqs-00.txt draft-satran-iscsi-01.txt draft-bakke-iscsimib-00.txt 15 min iSCSI Overview and Rationale 30 min iSCSI Security, may include draft-klein-iscsi-security-01.txt 15 min iSCSI Multiple Channels 15 min iSCSI Error Recovery 5 min iSCSI MIB -- Related Matters (20 min) 15 min FC-over-IP Overview and Rationale draft-ietf-ipfc-fcoverip-02.txt 5 min SEP and Parallel SCSI update draft-wilson-sep-00.txt IP Storage (ips) Working Group Proposed Charter - DRAFT 5 There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for shared traffic network environments, such as the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks, including both discovery of storage for access by the discovering entity, and discovery for management. - Management, including appropriate MIB definition. The WG will address security and congestion control as an integral part of its protocol(s); naming, discovery, and management are important related issues, but may be addressed in companion documents. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). The WG will strive to support the approaches to discovery, multi-pathing, and booting taken by the existing block storage protocols it encapsulates at the levels of those protocols. It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Discuss selection of encapsulation approach or approaches. Sep 00 Submit initial version of framework draft on encapsulation approaches. Dec 00 Discuss framework draft, as well as requirements, applicability and protocol specification drafts for at least one protocol encapsulation at IETF meeting in San Diego. Feb 01 Submit requirements, applicability, and protocol specification drafts for at least one protocol encapsulation to the IESG for consideration as a Proposed Standard. Mar 01 Discuss related drafts (e.g., MIBs, discovery) for at least the first protocol encapsulation at IETF meeting in Minneapolis. Jun 01 Submit MIB and discovery draft to the IESG. Begin revision of WG charter as appropriate in consultation with ADs. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Tue Jul 25 06:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02953; 25 Jul 2000 2:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA16973 for ips-outgoing; Tue, 25 Jul 2000 01:25:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA16958 for ; Tue, 25 Jul 2000 01:25:35 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA305640 for ; Tue, 25 Jul 2000 07:25:04 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id HAA135834 for ; Tue, 25 Jul 2000 07:25:04 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256927.001DC1A8 ; Tue, 25 Jul 2000 07:25:01 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 25 Jul 2000 08:01:48 +0300 Subject: RE: Draft Pgh. Agenda and charter Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, It is true that RDMA - in a form or other - would be valuable in any fast context. As it stands the current proposal relies on information already in the headers to perform the association between packets and buffers. In this sense it is different from more direct approaches - like in VIA - in which the context is specifically handed over to the remote site. However the team that put the draft together feels that we probably have some more things to do until we can call it good enough and I hope we have a chance to discuss it in Pittsburgh. Regards, Julo "Douglas Otis" on 25/07/2000 06:19:08 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, Black_David@emc.com cc: ips@ece.cmu.edu Subject: RE: Draft Pgh. Agenda and charter Julian, Are you suggesting iSCSI should distribute the DMA context from the initiator adapter to the target device? Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, July 24, 2000 2:38 PM To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: Draft Pgh. Agenda and charter David, It looks good. I still don't understand the FC stuff and I assume that for the world at large a good encapsulation of serial ATA would be more important, and SSA - a very widely deployed ANSI standard protocol - has in this time frame the at least same merit as FC but I'll let it be! As for the agenda: I assume that Randy will need at least 15 minutes for the requirements (Randy ?) and that storage "consumers" (SSPs, large users like National Labs) will join the discussion (or you will invite them). I have a already prepared some of the summaries for overview/design considerations/ multiple channels. Can we have those in one block (i was thinking about 25-30 min)? Security and some of the open things will certainly fill the remaining time. Thanks, Julo Black_David@emc.com on 24/07/2000 21:14:17 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Draft Pgh. Agenda and charter Draft agenda for Pittsburgh along with latest rev. to proposed charter follow (both should appear on the agenda area of the IETF web site). Important Agenda notes: The purpose of the Overview and Rationale talks is review of major design decisions. The 15 minutes available for FC-over-IP limits discussion to major issues; detailed review of FC-over-IP will not be done during the ips meeting. The 15 minute slots should be no more than 10 min of presentation; iSCSI Multiple Channels and Error Recovery should be no more than 5 min of presentation. Presenters should assume that the audience has read the drafts, and hence only cover material necessary to get to design decisions, questions, issues, etc. The 5 min time slots are for updates only; clarification questions will be entertained if time permits, but discussion will need to take place on the mailing list. Expect time slots to be enforced in some fashion. Could those who are going to be presenting each of the iSCSI items on the agenda please contact me and let me know who's doing what. I believe I know who the presenters are for the other segments. Thanks, --David IP Storage (ips) BOF DRAFT Agenda - subject to change -- Organizational Matters (20 min) 5 min Agenda Bashing 5 min A few words from the AD 10 min Charter Bashing Draft charter for bashing is appended to this agenda. Please bash the charter on the mailing list in preference to in the meeting, as the charter bashing time can be productively used for other purposes. -- Internet SCSI (iSCSI) (80 min) draft-haagens-ips-iscsireqs-00.txt draft-satran-iscsi-01.txt draft-bakke-iscsimib-00.txt 15 min iSCSI Overview and Rationale 30 min iSCSI Security, may include draft-klein-iscsi-security-01.txt 15 min iSCSI Multiple Channels 15 min iSCSI Error Recovery 5 min iSCSI MIB -- Related Matters (20 min) 15 min FC-over-IP Overview and Rationale draft-ietf-ipfc-fcoverip-02.txt 5 min SEP and Parallel SCSI update draft-wilson-sep-00.txt IP Storage (ips) Working Group Proposed Charter - DRAFT 5 There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for shared traffic network environments, such as the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks, including both discovery of storage for access by the discovering entity, and discovery for management. - Management, including appropriate MIB definition. The WG will address security and congestion control as an integral part of its protocol(s); naming, discovery, and management are important related issues, but may be addressed in companion documents. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). The WG will strive to support the approaches to discovery, multi-pathing, and booting taken by the existing block storage protocols it encapsulates at the levels of those protocols. It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Discuss selection of encapsulation approach or approaches. Sep 00 Submit initial version of framework draft on encapsulation approaches. Dec 00 Discuss framework draft, as well as requirements, applicability and protocol specification drafts for at least one protocol encapsulation at IETF meeting in San Diego. Feb 01 Submit requirements, applicability, and protocol specification drafts for at least one protocol encapsulation to the IESG for consideration as a Proposed Standard. Mar 01 Discuss related drafts (e.g., MIBs, discovery) for at least the first protocol encapsulation at IETF meeting in Minneapolis. Jun 01 Submit MIB and discovery draft to the IESG. Begin revision of WG charter as appropriate in consultation with ADs. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Tue Jul 25 18:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05389; 25 Jul 2000 14:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA14191 for ips-outgoing; Tue, 25 Jul 2000 13:16:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA14174 for ; Tue, 25 Jul 2000 13:16:23 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA02429; Tue, 25 Jul 2000 10:13:04 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA07538; Tue, 25 Jul 2000 10:12:44 -0700 From: "Douglas Otis" To: , Subject: RE: Draft Pgh. Agenda and charter Date: Tue, 25 Jul 2000 10:18:17 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, Architecturally, there is a significant difference between serial ATA, FireWire with ORBs as a container for direct DMA which could be your meld of VIA, or perhaps that of Fibre-Channel or SSA. I do not have a clear sense as to which architecture you advocate nor have your statements shed much light in this area. What happens if there is a 10 milli-second response delay within a network? What delay is tolerable? From looking at your spec, it would appear architecturally to be that of Fibre-channel on a different transport, but to even suggest this seems to create heated comments. Perhaps you could indicate your short and long term goals with respect to architecture in neutral terms. 1) Target manages exchanges; Initiator manages DMA. 2) Target manages exchanges and DMA. 3) Initiator manages exchanges and DMA. Where intelligence is placed has great effect on security and scalability. Which end do you want to see secure and which end do you want to see scale? Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, July 24, 2000 10:02 PM To: ips@ece.cmu.edu Subject: RE: Draft Pgh. Agenda and charter Doug, It is true that RDMA - in a form or other - would be valuable in any fast context. As it stands the current proposal relies on information already in the headers to perform the association between packets and buffers. In this sense it is different from more direct approaches - like in VIA - in which the context is specifically handed over to the remote site. However the team that put the draft together feels that we probably have some more things to do until we can call it good enough and I hope we have a chance to discuss it in Pittsburgh. Regards, Julo "Douglas Otis" on 25/07/2000 06:19:08 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, Black_David@emc.com cc: ips@ece.cmu.edu Subject: RE: Draft Pgh. Agenda and charter Julian, Are you suggesting iSCSI should distribute the DMA context from the initiator adapter to the target device? Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, July 24, 2000 2:38 PM To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: Draft Pgh. Agenda and charter David, It looks good. I still don't understand the FC stuff and I assume that for the world at large a good encapsulation of serial ATA would be more important, and SSA - a very widely deployed ANSI standard protocol - has in this time frame the at least same merit as FC but I'll let it be! As for the agenda: I assume that Randy will need at least 15 minutes for the requirements (Randy ?) and that storage "consumers" (SSPs, large users like National Labs) will join the discussion (or you will invite them). I have a already prepared some of the summaries for overview/design considerations/ multiple channels. Can we have those in one block (i was thinking about 25-30 min)? Security and some of the open things will certainly fill the remaining time. Thanks, Julo Black_David@emc.com on 24/07/2000 21:14:17 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Draft Pgh. Agenda and charter Draft agenda for Pittsburgh along with latest rev. to proposed charter follow (both should appear on the agenda area of the IETF web site). Important Agenda notes: The purpose of the Overview and Rationale talks is review of major design decisions. The 15 minutes available for FC-over-IP limits discussion to major issues; detailed review of FC-over-IP will not be done during the ips meeting. The 15 minute slots should be no more than 10 min of presentation; iSCSI Multiple Channels and Error Recovery should be no more than 5 min of presentation. Presenters should assume that the audience has read the drafts, and hence only cover material necessary to get to design decisions, questions, issues, etc. The 5 min time slots are for updates only; clarification questions will be entertained if time permits, but discussion will need to take place on the mailing list. Expect time slots to be enforced in some fashion. Could those who are going to be presenting each of the iSCSI items on the agenda please contact me and let me know who's doing what. I believe I know who the presenters are for the other segments. Thanks, --David IP Storage (ips) BOF DRAFT Agenda - subject to change -- Organizational Matters (20 min) 5 min Agenda Bashing 5 min A few words from the AD 10 min Charter Bashing Draft charter for bashing is appended to this agenda. Please bash the charter on the mailing list in preference to in the meeting, as the charter bashing time can be productively used for other purposes. -- Internet SCSI (iSCSI) (80 min) draft-haagens-ips-iscsireqs-00.txt draft-satran-iscsi-01.txt draft-bakke-iscsimib-00.txt 15 min iSCSI Overview and Rationale 30 min iSCSI Security, may include draft-klein-iscsi-security-01.txt 15 min iSCSI Multiple Channels 15 min iSCSI Error Recovery 5 min iSCSI MIB -- Related Matters (20 min) 15 min FC-over-IP Overview and Rationale draft-ietf-ipfc-fcoverip-02.txt 5 min SEP and Parallel SCSI update draft-wilson-sep-00.txt IP Storage (ips) Working Group Proposed Charter - DRAFT 5 There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for shared traffic network environments, such as the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks, including both discovery of storage for access by the discovering entity, and discovery for management. - Management, including appropriate MIB definition. The WG will address security and congestion control as an integral part of its protocol(s); naming, discovery, and management are important related issues, but may be addressed in companion documents. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). The WG will strive to support the approaches to discovery, multi-pathing, and booting taken by the existing block storage protocols it encapsulates at the levels of those protocols. It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Discuss selection of encapsulation approach or approaches. Sep 00 Submit initial version of framework draft on encapsulation approaches. Dec 00 Discuss framework draft, as well as requirements, applicability and protocol specification drafts for at least one protocol encapsulation at IETF meeting in San Diego. Feb 01 Submit requirements, applicability, and protocol specification drafts for at least one protocol encapsulation to the IESG for consideration as a Proposed Standard. Mar 01 Discuss related drafts (e.g., MIBs, discovery) for at least the first protocol encapsulation at IETF meeting in Minneapolis. Jun 01 Submit MIB and discovery draft to the IESG. Begin revision of WG charter as appropriate in consultation with ADs. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Tue Jul 25 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05579; 25 Jul 2000 15:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA15628 for ips-outgoing; Tue, 25 Jul 2000 14:00:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA15613 for ; Tue, 25 Jul 2000 14:00:30 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id TAA14916 for ; Tue, 25 Jul 2000 19:59:53 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id TAA100198 for ; Tue, 25 Jul 2000 19:59:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256927.0062D8B9 ; Tue, 25 Jul 2000 19:59:40 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 25 Jul 2000 20:58:16 +0300 Subject: RE: Draft Pgh. Agenda and charter Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, We had different assumptions for Reads and Writes. For Reads we assumed that the initiator manages the DMA the target the exchange. For writes we have again two options: - low latency pairs with enough resources would like to expedite writes by giving-up the RTT trip (large penalty for small writes) and use either immediate data or unsolicited writes. The host DMA is managed by the initiator again and the target should have the buffers ready. -higher latency pairs (bandwidth can be the same) and more cost conscious targets will operate only in solicited mode and the they could also use the Target Tag as a fast "buffer binder" (or cache DMA) while the initiator manages the host DMA. In all case we are not mandating any descriptors that can be fetched from memory although some adaptor vendors may choose to do so; we are limiting ourselves to the wire protocol and make only minimal assumptions about the APIs (even the age-old sockets can do!). And we would like to remain so unless there are compelling reasons not to. But as I said we still have some issues open. Julo "Douglas Otis" on 25/07/2000 20:18:17 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Draft Pgh. Agenda and charter Julo, Architecturally, there is a significant difference between serial ATA, FireWire with ORBs as a container for direct DMA which could be your meld of VIA, or perhaps that of Fibre-Channel or SSA. I do not have a clear sense as to which architecture you advocate nor have your statements shed much light in this area. What happens if there is a 10 milli-second response delay within a network? What delay is tolerable? From looking at your spec, it would appear architecturally to be that of Fibre-channel on a different transport, but to even suggest this seems to create heated comments. Perhaps you could indicate your short and long term goals with respect to architecture in neutral terms. 1) Target manages exchanges; Initiator manages DMA. 2) Target manages exchanges and DMA. 3) Initiator manages exchanges and DMA. Where intelligence is placed has great effect on security and scalability. Which end do you want to see secure and which end do you want to see scale? Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, July 24, 2000 10:02 PM To: ips@ece.cmu.edu Subject: RE: Draft Pgh. Agenda and charter Doug, It is true that RDMA - in a form or other - would be valuable in any fast context. As it stands the current proposal relies on information already in the headers to perform the association between packets and buffers. In this sense it is different from more direct approaches - like in VIA - in which the context is specifically handed over to the remote site. However the team that put the draft together feels that we probably have some more things to do until we can call it good enough and I hope we have a chance to discuss it in Pittsburgh. Regards, Julo "Douglas Otis" on 25/07/2000 06:19:08 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, Black_David@emc.com cc: ips@ece.cmu.edu Subject: RE: Draft Pgh. Agenda and charter Julian, Are you suggesting iSCSI should distribute the DMA context from the initiator adapter to the target device? Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, July 24, 2000 2:38 PM To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: Draft Pgh. Agenda and charter David, It looks good. I still don't understand the FC stuff and I assume that for the world at large a good encapsulation of serial ATA would be more important, and SSA - a very widely deployed ANSI standard protocol - has in this time frame the at least same merit as FC but I'll let it be! As for the agenda: I assume that Randy will need at least 15 minutes for the requirements (Randy ?) and that storage "consumers" (SSPs, large users like National Labs) will join the discussion (or you will invite them). I have a already prepared some of the summaries for overview/design considerations/ multiple channels. Can we have those in one block (i was thinking about 25-30 min)? Security and some of the open things will certainly fill the remaining time. Thanks, Julo Black_David@emc.com on 24/07/2000 21:14:17 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Draft Pgh. Agenda and charter Draft agenda for Pittsburgh along with latest rev. to proposed charter follow (both should appear on the agenda area of the IETF web site). Important Agenda notes: The purpose of the Overview and Rationale talks is review of major design decisions. The 15 minutes available for FC-over-IP limits discussion to major issues; detailed review of FC-over-IP will not be done during the ips meeting. The 15 minute slots should be no more than 10 min of presentation; iSCSI Multiple Channels and Error Recovery should be no more than 5 min of presentation. Presenters should assume that the audience has read the drafts, and hence only cover material necessary to get to design decisions, questions, issues, etc. The 5 min time slots are for updates only; clarification questions will be entertained if time permits, but discussion will need to take place on the mailing list. Expect time slots to be enforced in some fashion. Could those who are going to be presenting each of the iSCSI items on the agenda please contact me and let me know who's doing what. I believe I know who the presenters are for the other segments. Thanks, --David IP Storage (ips) BOF DRAFT Agenda - subject to change -- Organizational Matters (20 min) 5 min Agenda Bashing 5 min A few words from the AD 10 min Charter Bashing Draft charter for bashing is appended to this agenda. Please bash the charter on the mailing list in preference to in the meeting, as the charter bashing time can be productively used for other purposes. -- Internet SCSI (iSCSI) (80 min) draft-haagens-ips-iscsireqs-00.txt draft-satran-iscsi-01.txt draft-bakke-iscsimib-00.txt 15 min iSCSI Overview and Rationale 30 min iSCSI Security, may include draft-klein-iscsi-security-01.txt 15 min iSCSI Multiple Channels 15 min iSCSI Error Recovery 5 min iSCSI MIB -- Related Matters (20 min) 15 min FC-over-IP Overview and Rationale draft-ietf-ipfc-fcoverip-02.txt 5 min SEP and Parallel SCSI update draft-wilson-sep-00.txt IP Storage (ips) Working Group Proposed Charter - DRAFT 5 There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing block storage protocols, such as SCSI and certain Fibre Channel protocols, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing block storage protocols. Standards for those protocols are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The storage protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). Encapsulations of the storage protocols may require quality of service assurances (e.g., predictable latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide such assurances in shared traffic environments based on existing IETF QoS mechanisms such as Differentiated Services. Use of an IP-based transport raises issues that do not occur in existing storage transports. The WG will address at least the following issues: - Congestion control suitable for shared traffic network environments, such as the Internet. - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - Storage naming and discovery mechanisms for block storage services on IP-based networks, including both discovery of storage for access by the discovering entity, and discovery for management. - Management, including appropriate MIB definition. The WG will address security and congestion control as an integral part of its protocol(s); naming, discovery, and management are important related issues, but may be addressed in companion documents. The WG will consider issues raised by bridges and gateways to existing implementations of block storage protocols in order to support effective interoperability of the protocols developed in the working group with other implementations and/or encapsulations of the same block storage protocol(s). The WG will strive to support the approaches to discovery, multi-pathing, and booting taken by the existing block storage protocols it encapsulates at the levels of those protocols. It may be necessary for block storage traffic to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for block storage traffic are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by existing block storage implementations. The WG will consider what levels of data integrity assurance are required for block storage traffic over IP networks and how they should be achieved. The WG will produce a framework document describing the encapsulation or encapsulations it intends to pursue, and requirements, applicability and protocol specification documents for each encapsulation. The framework document will consider whether both end-system and gateway node (including gateways to Fibre Channel) requirements can be accommodated in a single protocol family (e.g., as has been done by the IP Security Protocol). The applicability and requirements documents will consider both disk and tape devices and take note of the variation in scale from single drives to large disk arrays and tape libraries; the protocols need not be applicable to all such devices. The WG will not work on: - Extensions to existing block storage protocols beyond those strictly necessary for the use of IP-based transports. - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - Support for environments in which significant data loss or data corruption is acceptable. - File system protocols. Milestones Aug 00 Initial meeting in Pittsburgh. Discuss selection of encapsulation approach or approaches. Sep 00 Submit initial version of framework draft on encapsulation approaches. Dec 00 Discuss framework draft, as well as requirements, applicability and protocol specification drafts for at least one protocol encapsulation at IETF meeting in San Diego. Feb 01 Submit requirements, applicability, and protocol specification drafts for at least one protocol encapsulation to the IESG for consideration as a Proposed Standard. Mar 01 Discuss related drafts (e.g., MIBs, discovery) for at least the first protocol encapsulation at IETF meeting in Minneapolis. Jun 01 Submit MIB and discovery draft to the IESG. Begin revision of WG charter as appropriate in consultation with ADs. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items. From owner-ips@ECE.cmu.edu Wed Jul 26 23:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11895; 26 Jul 2000 19:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA08930 for ips-outgoing; Wed, 26 Jul 2000 17:54:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA08905 for ; Wed, 26 Jul 2000 17:54:38 -0400 (EDT) Received: from 7h16a (gn77.giganet.com [208.239.8.77]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id RAA16441 for ; Wed, 26 Jul 2000 17:54:16 -0400 Reply-To: From: "Alex Nicolson" To: Subject: IETF Draft for VI/TCP Date: Wed, 26 Jul 2000 17:49:23 -0400 Message-ID: <002601bff74b$5c472ec0$4d08efd0@7h16a.giganet.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0027_01BFF729.D5358EC0" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3612.1700 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0027_01BFF729.D5358EC0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Attached is the IETF draft submission for the VI over TCP by S. DiCecco and J. Williams. This document was sent to the IETF today but may not be posted in time for the meeting next week. This document is referenced in the draft titled "iSCSI/VI/TCP proposal" (draft-csapuntz-ips-iscsivi-00.txt) available as an IETF Draft. Could someone put this on the Web site? Alex Nicolson Giganet, Inc. Concord Office Center 2352 Main Street Concord, MA 01742 978.461.0402 (tel) 978.461.0430 (fax) anicolson@giganet.com ------=_NextPart_000_0027_01BFF729.D5358EC0 Content-Type: text/plain; name="draft-dicecco-vitcp-00.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="draft-dicecco-vitcp-00.txt" Network Working Group S. DiCecco Internet-Draft J. Williams GigaNet, Inc. Expires January 2001 July 14, 2000 VI / TCP (Internet VI) Status of this memo This document is an Internet-Draft and is offered in full accordance with all provisions of Section 10 of RFC 2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also=20 distribute working documents as Internet-Drafts. Internet-Draft documents are valid for a maximum of six months and may be updated, replaced, or rendered obsolete by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to site them other than as "work in progress". The list of current Internet-Drafts can be accessed at http://www.ietf.org/lid-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.NHtml The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this memo are to be interpreted as described in RFC2119. Table of Contents 1 Abstract 2 2 Overview 3 2.1 VI Architectural Components 3 2.2 VI/TCP 4 2.2.1 Extensions to VI 4 2.2.2 VI/TCP Overview 4 2.2.2.1 Basic VI Components 4 2.2.2.2 Introduction to VI/TCP 5 2.2.2.2.1 VI/TCP Addressing 5 2.2.2.2.2 VI/TCP Connection Management 5 DiCecco, Williams [Page 1] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 2.2.2.2.3 VI/TCP Protocol Messaging 6 2.2.2.2.4 TCP/IP Options and VI/TCP 6 2.2.2.2.5 VI/TCP Retransmissions 6 2.2.2.2.6 Note on Outstanding RDMA Reads 7 3 The VI/TCP Protocol 7 3.1 VI/TCP Segment Header 8 3.2 VI/TCP Connection Establishment (CE) Header 12 3.3 VI/TCP RDMA Header 14 4 VI/TCP Connection Establishment 15 4.1 Basic Connection Establishment Timeline 15 4.2 Connection Establishment - Active 16 4.3 Connection Establishment - Passive 17 5 Security Considerations 18 6 References 18 7 Author's Addresses 19 1. Abstract The Virtual Interface (VI) architecture [VIAR] describes a high performance design for interfacing distributed applications to=20 accelerated protocol processing. VI seeks to improve the performance=20 of such applications by reducing the latency and overheads associated=20 with standard communications protocol stack processing. VI greatly=20 reduces the processing overhead associated with traditional network=20 architectures by providing applications a protected, directly=20 accessible interface to network hardware - a Virtual Interface. This memo describes extensions to the VI Architecture designed to=20 facilitate operation over TCP/IP. These extensions take the form of=20 enhancements to the VI Provider Library API defined in the VI=20 Architecture Developer's Guide [VIDG], and a "VI Protocol" which=20 supports VI functionality during operation over TCP/IP. The extensions to the VI Architecture which support operation over TCP/IP are intended to be fully compliant with the VI Architecture [VIAR] and its associated Developer's Guide [VIDG]. DiCecco, Williams [Page 2] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 2. Overview This section contains a brief overview of VI components and a=20 functional overview of VI operation over TCP/IP 2.1. VI Architectural Components VI is comprised of four architectural components - Virtual=20 Interfaces, Completion Queues, VI Providers, and VI Consumers. Virtual Interfaces (VIs) are the mechanisms that allow VI Consumers direct access to the data transfer services of VI Providers. VI=20 Consumers post data transfer requests, in the form of Descriptors,=20 directly to the VI Provider. Descriptors are structures that contain=20 the information necessary for the VI Provider to process the data=20 transfer (e.g.,data location). Descriptors are posted to Work Queues=20 (send and receive) associated with the VI. Facilities are provided to=20 signal VI Descriptor postings to the network adapter. Processing of=20 posted Descriptors is asynchronous and descriptors are marked when=20 processing completes. VI Consumers remove completed descriptors from=20 Work Queues for reuse in subsequent requests. Completion Queues provide a facility whereby VI Consumers can create a single point of notification for processing completed Descriptors. =20 Once a Work Queue is associated with a Completion Queue, handling of=20 all completions are handled via that Completions Queue. The VI Provider consists of a physical network interface (NIC) and driver functionality. The VI NIC implements the Virtual Interfaces and Completion Queues, and directly performs data transfers. VI NIC=20 drivers provide the control and resource management functions to=20 maintain the VI between consumers and VI NICs. VI Consumers are typically applications programs and their supporting operating system functions. VI Consumers represent the users of a=20 Virtual Interface. Access to the Virtual Interface is through a=20 library referred to as the VI Provider Library [VIDG]. The VI Provider=20 Library provides an application programming interface for hardware=20 connection, endpoint creation and destruction, connection management,=20 memory handling, data transfer, queue management, informational DiCecco, Williams [Page 3] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 queries, name services, and error handling. 2.2. VI/TCP This section introduces the fundamentals of VI operation over TCP. 2.2.1. Extensions to VI The proposed protocol supports the VI Architecture as currently defined. In addition, the protocol supports certain enhancements to=20 VI. Extensions to the API defined in [VIDG] would be required to=20 exploit such enhancements. Proposed enhancements are as follows: - Descriptor Flow Control: Transmit descriptors may be posted in advance of the corresponding receive descriptors. The VI Provider will supply flow control. - Security Field: A security field is contained in the=20 Connect Request and Connect Accept PDUs. Use of this field is to be=20 defined. - Attribute Negotiation: VI Architecture requires that incoming=20 connection establishment attempts be rejected unless the calling and=20 called VI Attributes match (e.g., Maximum Transfer Unit Size). The=20 protocol permits downward negotiation of MTU sizes. 2.2.2. VI/TCP Overview This Section provides an overview of how the components of a=20 Virtual Interface are created, managed, and destroyed, and also=20 introduces the data transfer models. 2.2.2.1. Basic VI Components Operations on basic VI architectural components remain largely unchanged with VI/TCP. VI functionality is invoked by a VI Consumer through the API defined in [VIDG]. Access to a VI NIC is achieved by opening a handle to the driver representing the NIC. This handle is used in subsequent operations. All memory used in data transfer is DiCecco, Williams [Page 4] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 "registered" with the VI Provider. Memory handles are used to identify the region and to qualify virtual memory addresses. VIs are created by the VI Provider upon request by the VI Consumer. Connections are not established by creation of a VI and no data transfer can occur until=20 the VI is connected to another. VI Work Queues may be associated with=20 Completion Queues to provide a single handling point for completed VI Descriptors. VI provides a connection-oriented data transfer service. Newly created VIs are not pre-associated with other VIs; a VI must be explicitly connected to another to enter its data transfer phase. VI provides two types of data transfers - traditional Send/Receive, and Remote Direct Memory Access (RDMA). 2.2.2.2. Introduction to VI/TCP This Section serves as an introduction to VI operation over=20 TCP/IP. 2.2.2.2.1. VI/TCP Addressing The VI Architecture defines a generic "VI Network Address" format consisting of an "address" portion and a "discriminator" portion. When operating VI/TCP, the address portion contains an IP address and the discriminator is per the VI Architecture [VIAR]. One transport layer port is reserved for passive connection establishment. All incoming VI connections are through this port and VI applications distinguish=20 themselves by the VI Network Address discriminator. For active=20 connection establishment, multiple transport layer ports are used. 2.2.2.2.2. VI/TCP Connection Management With VI/TCP, all VI connections are implemented over an underlying TCP connection. The VI/TCP connection establishment process requires=20 an underlying TCP connection over which VI/TCP protocol may be=20 exchanged. VI connections have a one-to-one correspondence with TCP=20 connections. This is referred to as the VI/TCP connection. When a VI=20 connection is closed, the underlying TCP connection must be closed. =20 Similarly, when a TCP connection is closed, the associated VI=20 connection must be closed. VI Provider's handling VIP_Connect Request=20 primitives [VIDG], first request TCP establish its connection and then=20 perform VI/TCP protocol messaging over this underlying connection. VI=20 Providers must have DiCecco, Williams [Page 5] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 accepted an underlying TCP connection before the associated VI=20 connection is accepted. VI/TCP Provider's MUST check that=20 address/handles are valid for the underlying connection. From the=20 perspective of a VI/TCP Provider, TCP connection setup is an atomic=20 operation that either succeeds or fails. If the operation succeeds, VI=20 connection establishment is initiated; otherwise, the VI connection is=20 rejected. 2.2.2.2.3. VI/TCP Protocol Messaging VI/TCP functionality is invoked by a VI Consumer through the API defined in [VIDG]. The VI Provider supplies this functionality. The=20 VI Provider, through use of the VI Protocol, supports this VI/TCP=20 functionality. The VI Protocol defines "messages" to implement VI=20 these functions (e.g., connections establishment). Typically, there is=20 one message per Transmit Descriptor. Each message has a type (e.g.,=20 RDMA Write). VI messages are divided into "segments". These segments are sent, in order, over the associated TCP connection. It is recommended, but not required, that there be exactly zero or one VI segment for each TCP=20 segment and that VI segments not be fragmented to span multiple TCP=20 segments. All segments for one VI message will be transmitted before=20 the next message is started. An exception is provided in that RDMA=20 Read Response segments may be interleaved with segments of any message=20 type other than another RDMA Read Response. 2.2.2.2.4. TCP/IP Options and VI/TCP It is strongly recommended that TCP connections supporting VI/TCP implement the timestamp option for PAWS (protection against wrapped sequence numbers) as defined in RFC1323, TCP Extensions for High=20 Performance [PAWS]. 2.2.2.2.5. VI/TCP Retransmissions VI/TCP will retransmit dropped segments, as required. It is=20 recommended that retransmitted segments contain the same data as the=20 original dropped segment. In certain circumstances, this will not be=20 possible without undue burden on an implementation. The following=20 exceptions are=20 DiCecco, Williams [Page 6] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 noted: - The "RX Descriptors Posted" field of the protocol's "VI Segment Header" may contain the current value instead of the value at the time of its original posting. Senders must not assume that this value has actually been received since there are no facilities for determining which of the two values has been received from the acknowledgement. - Retransmission is required, but data access results in an access=20 violation and retransmission cannot occur. - Retransmission is required, but cannot occur because the VI/TCP=20 connection has been closed. If a VI NIC is unable to retransmit original data, it may pad and=20 should set the "Transmit Error" bit in the "Type" field of the "VI=20 Segment Header". 2.2.2.2.6. Note on Outstanding RDMA Reads For each RDMA Read Request received, memory allocated for the request must be held until the response is acknowledged. The number of outstanding RDMA Reads must be limited to control resource exhaustion. Discarding excessive RDMA Reads pending completions of outstanding requests does not seem viable in the absence of a deadlock avoidance mechanism. It is proposed that the VI Protocol be extended to provide negotiation of the number of outstanding RDMA Reads during connection establishment. This number would represent a per VI limit and the=20 negotiated value would remain for the lifetime of the VI/TCP=20 connection. 3. The VI/TCP Protocol This section provides the VI/TCP protocol data unit formats. All multibyte formats are to be represented in network byte order (i.e., big-endian). Each VI/TCP PDU contains a VI Segment Header. Optionally,=20 an RDMA Header or CE (connection establishment) Header may be present.=20 The VI/TCP Segment Header provides sufficient features to support non- RDMA send/receives. The RDMA Header must be included for RDMA=20 transfers. The CE Header must be included for connection=20 establishment. Encapsulations are as follows: DiCecco, Williams [Page 7] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 | Lower Layer Headers | +----------------------------------------+ | IP Header | +----------------------------------------+ | TCP Header | +----------------------------------------+ | VI/TCP Segment Header | +----------------------------------------+ | (optional) VI/TCP RDMA or CE Header | +----------------------------------------+ | VI/TCP Consumer's Data | 3.1. VI/TCP Segment Header The VI/TCP Segment Header is defined as follows. | Byte 0 | Byte 1 | Byte 2 | Byte 3 | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ | Version | Type/Flags | Segment Length | +---------------+---------------+---------------+---------------+ | Data Offset | +---------------+---------------+---------------+---------------+ | Immediate Data | +---------------+---------------+---------------+---------------+ | Message Number | +---------------+---------------+---------------+---------------+ | Message ACK | +---------------+---------------+---------------+---------------+ | Rx Descriptors Posted | Remote Error Code | +---------------+---------------+---------------+---------------+ | VI/TCP User Data | | | Version This is an 8-bit field indicating the VI/TCP version. DiCecco, Williams [Page 8] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 Type/Flags Type is a 5-bit field indicating the packet type. Valid message types are: - Send - RDMA Read Request - RDMA Read Response - RDMA Write - NOP - Connect Request - Connect Accept - Connect Reject - Connect No Match Flags is a 3-bit field defined as follows: - BIT 1 : Immediate Data Valid As defined by the VI Architecture [VIAR] - BIT 2 : End of Message Indicates the current segment is the last of a message - BIT 3 : Transmit Error Indicates either transmit length or protection error Segment Length Segment Length is a 16-bit field containing the length of the VI/TCP segment including the VI/TCP Segment Header Data Offset For the initial segment of any message, this 32-bit field will contain zero. For subsequent segments, it will contain the number of bytes already transferred for this message in prior segments. Only segment payload is included in this count; headers are specifically not included. Immediate Data May hold 32-bits of optional user data as described by the VI DiCecco, Williams [Page 9] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 Architecture [VIAR]. Message Number Messages are sequentially number by the VI/TCP Provider. The initial Message Number may be varied by an implementation. For RDMA Read Responses, Message Number carries the message number of the=20 corresponding RDMA Read Request. Any two segments with the same type=20 are part of the same message if and only if their message numbers are=20 equal. Segments within a message may be processed out of order. =20 Reordering messages is not supported. Rx Descriptors Posted Indicates the number of receive Descriptors, modulo 2^16, that have=20 been posted during the lifetime of the VI/TCP connection. If=20 Descriptor flow control is in effect, the VI/TCP provider must delay=20 any transmission which would consume receive Descriptors until receive=20 descriptors complete and become available. Notes on NOP NOPs are used to send a Message ACK when a VI has no data to transmit. If a VI has data to transmit, the Rx Descriptors Posted number is included in the transferred segments. However, if a transmitter is idle, a NOP is utilized to permit conveyance of Rx Descriptors Posted=20 in the absence of use data message segments. Idle VI/TCP connections=20 that have an updated Rx Descriptors Posted value, must use NOP to=20 convey this information. NOPs should be sent whenever there have been=20 Rx Descriptors Posted since the last segment sent and the number of=20 unused credits falls below some threshold. Unused credits are the=20 number of Rx Descriptors of which the remote VI has been notified but=20 has not yet used. The threshold value should be set at the time of=20 connection establishment. Message ACK Message ACK is valid only for VI/TCP connections at the Reliable=20 Delivery Level [VIAR]. Message ACK is used in conjunction with Remote=20 Error Code to provide information relating to memory protection or VI=20 Descriptor errors and also to provide facilities for implementation=20 specific error handling. If the VI Error subfield of the Remote Error=20 Code DiCecco, Williams [Page 10] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 (Remote Error Codes, next section) indicates "No Error", then Message ACK contains the Message Number from the last VI Message received=20 without error. If VI Error is OTHER THAN "No Error", then Message ACK=20 contains the Message Number of the message segment in error. Message=20 ACK should not be indicated for messages until both message data has=20 been written to host memory and the doorbell has been rung. Message=20 ACKs may be included in any VI Message Segment including that of a NOP=20 message. When a VI/TCP connection is supporting level Reliable Reception, the Message ACK field must be valid and will be used determine when=20 transmit Descriptors will be completed. RDMA Reads are completed upon=20 receipt of a valid response. Message ACK are indicated for messages=20 received in error. In this case, the VI Error Type field of Remote=20 Error Code is set to reflect the appropriate VI error. Remote Error=20 Code is defined in the following paragraph. Message ACK is invalid on=20 subsequent messages. When a VI/TCP connection is supporting level Reliable Delivery or=20 Unreliable Delivery, the contents of Message ACK are undefined and must=20 be ignored by a receiver. Remote Error Code Remote Error Code is comprised of two subfields - the VI Error Type,=20 and the IS Error Code. Both VI Error Type and IS Error Code apply to=20 VI message identified by the Message ACK field. These are defined in=20 the following paragraphs. IS Error Code IS Error Code is an Implementation Specific error code, its semantics are implementation dependent and considered outside the scope of this document. If the IS Error Code is set, Message ACK must be set to=20 indicate the VI Message on which the error occurred. Note that VI=20 errors (see next paragraph) and local errors need not be mutually=20 exclusive and this field may be used to provide supplemental status=20 information. VI Error Type VI Error Type indicates one of the following - No Error DiCecco, Williams [Page 11] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 - RDMA Memory Protection Error - VI Descriptor Error When a VI/TCP connection is supporting level Reliable Reception, the VI Error Type field must be valid and is used to update the Status of the VI Descriptor's Control Segment. When a VI/TCP connection is supporting level Reliable Delivery or=20 Unreliable Delivery, VI Error Type shall indicate No Error and be=20 ignored by a receiver. 3.2. VI/TCP Connection Establishment (CE) Header The VI/TCP CE Header is defined as follows. DiCecco, Williams [Page 12] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 | Byte 0 | Byte 1 | Byte 2 | Byte 3 | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ | Calling Attributes | Calling Discriminator Length | +---------------+---------------+---------------+---------------+ | MTU Size | +---------------+---------------+---------------+---------------+ | | +- -+ | | +- Calling Discriminator -+ | | +- -+ | | +---------------+---------------+---------------+---------------+ | Calling RDMA Read Window | Called Discriminator Length | +---------------+---------------+---------------+---------------+ | | +- -+ | | +- Called Discriminator -+ | | +- -+ | | +---------------+---------------+---------------+---------------+ | Security Information | Calling Attributes The Calling Attributes field contains the following flag bits - Reliable : Reliable reception and delivery are merged - RDMA Write Enable - RDMA Read Enable - Descriptor Flow Control Enabled - Peer-to-peer Connection Establishment Calling/Called Discriminator and Discriminator Lengths These fields are as defined by the VI Architecture [VIAR] DiCecco, Williams [Page 13] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 MTU Size MTU Size is "proposed" in Connect Request PDUs and is considered an "agreed" value in a Connect Accept. The agreed value must be the=20 lesser of the called/calling VI/TCP Provider's MTU capability. Security Information To be supplied. A VI/TCP Consumer may use this feature as an=20 additional basis for accepting or rejecting calls. This feature is=20 currently unsupported by the VI Architecture [VIAR] and the programming=20 API [VIDG]. Extensions to the API to expose this feature are to be=20 supplied. 3.2.1. VI/TCP RDMA Header The VI/TCP RDMA Header is defined as follows. | Byte 0 | Byte 1 | Byte 2 | Byte 3 | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ | | +- RDMA Address -+ | | +---------------+---------------+---------------+---------------+ | Registered Memory Handle | +---------------+---------------+---------------+---------------+ | RDMA Length | +---------------+---------------+---------------+---------------+ | | | VI/TCP Consumer's Data | | | RDMA Address The RDMA Address field contains the 64-bit data address of the first data segment from the VI Descriptor DiCecco, Williams [Page 14] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 Registered Memory Handle The Registered Memory Handle field contains the Memory Handle returned when the region of memory containing the data segment was registered with the VI Provider. This is the same memory handle required by the=20 VI Descriptor. RDMA Length The RDMA Length field contains the length field from the VI Descriptor that indicates the total number of bytes to be transferred across all segments of a message. 4. VI/TCP Connection Establishment This section contains the state machines governing VI/TCP=20 connection establishment. Both active and passive (e.g., listens)=20 scenarios are presented. For peer-to-peer connection establishment,=20 conflicts are resolved lexicographically on IP address. "Lower" (in=20 the lexicographic sense) IP addressed hosts concede to "higher"=20 addressed host during peer connection establishment and the process=20 reverts that of the active/passive case. Receiving a Connect No Match=20 VI/TCP packet type during peer connection establishment results it=20 repeated attempts for a period specified by the VI Consumer's=20 connection timeout value. 4.1. Basic Connection Establishment Timeline DiCecco, Williams [Page 15] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 VIPL API [VIDG] | VI/TCP Protocol | VIPL API ------------------------------------------------------------------ | | | | VipConnectWait VipConnectRequest | | <----------------- -----------------> | | | setup TCP connection | | | | Connect Request | | -------------------> | VipConnectWait(ret) | | -----------------> | | VipPostRecv | | <----------------- | | VipConnectAccept | Connect Accept | <---------------- VipConnectReq (ret) | <------------------- | <----------------- | or Connect Reject | | or Connect No Match | 4.2. Connection Establishment - Active The state machine governing active VI/TCP connection establishment is as follows: DiCecco, Williams [Page 16] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 +----------------+ (Legend: event - action) | Disconnected | <--------------------------------<+ +----------------+ ^ | VipConnectRequest | | - Setup TCP connection | | | \|/ | +----------------+ TCP setup fail ^ +------>| Connecting +>--------------------------------->+ | +----------------+ ^ | TCP Closes | TCP connection established | | - | - ConnectRequest | | Reestablish \|/ | | +----------------+ ConnectReject or Timeout ^ +------<| Pending Accept |>--------------------------------->+ +----------------+ - close TCP connect. ^ | | | ConnectAccept | \|/ | +----------------+ Vip or TCP disconnect ^ | Connected |>--------------------------------->+ +----------------+ - close TCP connection 4.3. "Connection Establishment - Passive" The state machine governing passive VI/TCP connection=20 establishment is as follows: DiCecco, Williams [Page 17] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 +----------------+ +--------------+ | Listening on | ( Listen is implied by + Disconnected + | VI/TCP Port | 1'st VipCreateVI ) +--------------+ +----------------+ ^ | | | incoming TCP connection - Accept TCP connection | | (Legend: event - action) | \|/ | +----------------+ Timeout - close TCP connection ^ | Incoming |---------------------------------------------->+ +----------------+ TCP connection closes ^ | | | incoming Connect Request | \|/ | +----------------+ No Matching Discriminator ^ | Matching +---------------------------------------------->+ +----------------+ - Send Connect NoMatch; close TCP connection ^ | | | Discriminator match | \|/ | +----------------+ ConnectReject - VipConnectReject, close TCP ^ | Pending Accept |---------------------------------------------->+ +----------------+ ^ | | | VipConnectAccept - ConnectAccept | \|/ =20 +----------------+ Vip or TCP disconnect - close TCP connection ^ | Connected |---------------------------------------------->+ +----------------+ 5. Security Considerations No special security considerations exist at this time. 6. References DiCecco, Williams [Page 18] =0C Internet-Draft VI / TCP (Internet VI) July 14, 2000 [VIAR] "Virtual Interface Architecture Specification", Compaq Computer Corp., Intel Corporation, Microsoft Corporation, 1997. [VIDG] "Intel Virtual Interface (VI) Architecture Developer's Guide", Intel Corporation, September 1998. [PAWS] Jacobsen, Braden, Borman, "TCP Extensions for High Performance", RFC 1323, May 1992. 7. Author's Addresses Stephen DiCecco James Williams GigaNet, Inc. Concord Office Center 2352 Main Street Concord, Massachusetts 01742 978.461.0402 (tel) 978.461.0430 (fax) www.giganet.com Email: sdicecco@giganet.com jimw@giganet.com DiCecco, Williams [Page 19] =0C ------=_NextPart_000_0027_01BFF729.D5358EC0-- From owner-ips@ECE.cmu.edu Wed Jul 26 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11914; 26 Jul 2000 19:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA10382 for ips-outgoing; Wed, 26 Jul 2000 18:40:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA10337 for ; Wed, 26 Jul 2000 18:40:34 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Wed, 26 Jul 2000 18:39:59 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FBAB@corpmx9.isus.emc.com> To: anicolson@giganet.com, ips@ece.cmu.edu Subject: RE: IETF Draft for VI/TCP Date: Wed, 26 Jul 2000 18:39:59 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk FYI - this draft was submitted well after the cutoff two weeks prior to the Pittsburgh meeting, and hence will not be on the ips agenda for Pittsburgh. I'd like to discourage the announcement of drafts that have not made the cutoff during the two weeks prior to the meeting - there are very good reasons for that two week prior cutoff. Costa's draft did make it in time, and will have a brief agenda slot. Also, submitting drafts to the Internet-Drafts servers is usually a better approach than broadcasting them indiscriminately to the mailing list, as not everyone enjoys high-bandwidth email connections all the time. If for some reason, the Internet-Drafts servers are inappropriate (e.g., a near-final version is being announced for review before submission), please put the draft on a web site and send a URL to the list. As for putting the draft on the IETF web site - that's the responsibility of the draft authors once the site reopens for submissions after the Pittsburgh meeting. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Alex Nicolson [SMTP:Anicolson@giganet.com] > Sent: Wednesday, July 26, 2000 5:49 PM > To: ips@ece.cmu.edu > Subject: IETF Draft for VI/TCP > > Attached is the IETF draft submission for the VI over TCP by S. DiCecco > and > J. Williams. This document was sent to the IETF today but may not be > posted > in time for the meeting next week. This document is referenced in the > draft > titled "iSCSI/VI/TCP proposal" (draft-csapuntz-ips-iscsivi-00.txt) > available > as an IETF Draft. > > Could someone put this on the Web site? > > > Alex Nicolson > > Giganet, Inc. > Concord Office Center > 2352 Main Street > Concord, MA 01742 > 978.461.0402 (tel) > 978.461.0430 (fax) > anicolson@giganet.com << File: draft-dicecco-vitcp-00.txt >> From owner-ips@ECE.cmu.edu Thu Jul 27 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12297; 26 Jul 2000 21:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA15513 for ips-outgoing; Wed, 26 Jul 2000 20:19:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from siamese.i3ware.com (mail.3ware.com [209.150.174.78]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA15496 for ; Wed, 26 Jul 2000 20:19:46 -0400 (EDT) Received: by SIAMESE with Internet Mail Service (5.5.2650.21) id ; Wed, 26 Jul 2000 17:19:15 -0700 Message-ID: <53B208BD9A7FD311881A009027B6BBFB216FFB@SIAMESE> From: Bob Horst To: ips@ece.cmu.edu Subject: IP-attached ATA storage Date: Wed, 26 Jul 2000 17:19:14 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01BFF760.4BCC70D6" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01BFF760.4BCC70D6 Content-Type: text/plain; charset="iso-8859-1" There was some recent discussion about supporting ATA protocols over iSCSI. At 3ware, we have been working on IP-attached ATA storage for quite a while. We are already shipping a line of ATA PCI controllers with internal packet switching hardware that exports virtual SCSI devices, and have device drivers for several operating systems. We are just about to come out of stealth mode on the way we will extend these products to network-attached block storage subsystems with our Storage Control Protocol over TCP/IP. We will be showing our product at Networld+Interop in Atlanta and plan to ship production units by the end of the year. Although our protocol has some differences from iSCSI, we hope in the long run to work with the committee to ensure that the final iSCSI protocol is suitable for ATA storage as well as SCSI. Our initial protocol, which will ship before there is an agreed standard, will naturally be a subset of what is eventually standardized. We will upgrade the customers to the standard appropriately as it emerges. We are not quite ready to disclose our protocol, but I will be attending the IETF meeting in Pittsburgh and hope to begin interacting with the committee then. -- Bob Bob Horst 3ware, Inc. 490 California Ave. Palo Alto, CA 94306 http://www.3ware.com ------_=_NextPart_001_01BFF760.4BCC70D6 Content-Type: text/html; charset="iso-8859-1"
There was some recent discussion about supporting ATA protocols over iSCSI.  At 3ware, we have been working on IP-attached ATA storage for quite a while.   We are already shipping a line of ATA PCI controllers with internal packet switching  hardware that exports virtual SCSI devices, and have device drivers for several operating systems.  We are just about to come out of stealth mode on the way we will extend these products to network-attached block storage subsystems with our Storage Control Protocol over TCP/IP.  We will be showing our product at Networld+Interop in Atlanta and plan to ship production units by the end of the year.
 
Although our protocol has some differences from iSCSI, we hope in the long run to work with the committee to ensure that the final iSCSI protocol is suitable for ATA storage as well as SCSI.  Our initial protocol, which will ship before there is an agreed standard, will naturally be a subset of what is eventually standardized.  We will upgrade the customers to the standard appropriately as it emerges.
We are not quite ready to disclose our protocol, but I will be attending the IETF meeting in Pittsburgh and hope to begin interacting with the committee then.
 
-- Bob
 
Bob Horst
3ware, Inc.
490 California Ave.
Palo Alto, CA 94306
http://www.3ware.com
 
------_=_NextPart_001_01BFF760.4BCC70D6-- From owner-ips@ECE.cmu.edu Thu Jul 27 03:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12607; 26 Jul 2000 23:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA17841 for ips-outgoing; Wed, 26 Jul 2000 22:00:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA17821 for ; Wed, 26 Jul 2000 22:00:00 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Wed, 26 Jul 2000 21:59:26 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FBAF@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: All-but-final ips agenda for Pgh. Date: Wed, 26 Jul 2000 21:59:21 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk This should be the all-but-final agenda. There's time to bash (modify) it at the start of the meeting. --David IP Storage (ips) BOF DRAFT Agenda Drafts can be retrieved from http://www.ietf.org/ID.html , among other places. -- Organizational Matters (20 min) 5 min Agenda Bashing (David Black/Steve Bellovin, ips co-chairs) 5 min A few words from the Area Director(s) (Allison Mankin/Scott Bradner, IETF Transport Area Directors) 10 min Charter Status and Bashing (David Black, ips co-chair) Please bash the charter on the mailing list in preference to in the meeting, as the charter bashing time can be productively used for other purposes. -- Internet SCSI (iSCSI) (80 min) 15 min iSCSI Requirements (Randy Haagens, HP) draft-haagens-ips-iscsireqs-00.txt 15 min iSCSI Overview and Rationale (Julian Satran, IBM) 15 min iSCSI Multiple Channels (Julian Satran, IBM) draft-satran-iscsi-01.txt 30 min iSCSI Security Requirements (Steve Bellovin, co-chair) 5 min iSCSI MIB (Steve De Groote, NuSpeed) draft-bakke-iscsimib-00.txt -- Related Matters (20 min) 15 min FC-over-IP Overview and Rationale (Elizabeth Rodriguez, Lucent) draft-ietf-ipfc-fcoverip-02.txt 2.5 min SEP and Parallel SCSI update (Paul von Stamwitz, Adaptec) draft-wilson-sep-00.txt 2.5 min VI/TCP for iSCSI (Constantine Sapuntzakis, Cisco) draft-csapuntz-ips-iscsivi-00.txt Some notes to presenters: - Assume the audience has read the draft(s). The agenda time is for discussion of significant issues, rather than presentation of drafts. - Please use hardcopy slides. Given the number of presenters, we could easily lose 15 minutes fiddling with laptop connections to a projector. Black text on clear slides is just fine; if you want to do something fancier, make sure that it can be read from the back of a reasonably large room. - Please let one of the co-chairs know that you're here before the session. Sitting near the front will reduce the time spent in transition between speakers. - The purpose of the Overview and Rationale talks is review of major design decisions. The 15 minutes available for FC-over-IP limits discussion to major issues; detailed review of FC-over-IP will not be done during this ips meeting. - The 15 minute slots should be no more than 10 min of presentation. The 5 min time slot for the MIB should allow for at least 1 min of questions - substantial discussion will probably have to be taken to the mailing list. The 2.5 min time slots are limited to 2 slides; clarification questions will be entertained if time permits, but discussion will have to take place on the mailing list. From owner-ips@ECE.cmu.edu Thu Jul 27 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13327; 27 Jul 2000 2:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA21928 for ips-outgoing; Thu, 27 Jul 2000 01:13:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA21915 for ; Thu, 27 Jul 2000 01:13:23 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA234790 for ; Thu, 27 Jul 2000 07:12:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id HAA176640 for ; Thu, 27 Jul 2000 07:12:52 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256929.001CA396 ; Thu, 27 Jul 2000 07:12:48 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 27 Jul 2000 08:11:22 +0300 Subject: Re: IP-attached ATA storage Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Welcome Bob. That is a serious development and we are looking forward at working with you. And there is no "comitee" - just a working group and 2 chairmen (the working group is an open forum). Julo Bob Horst on 27/07/2000 03:19:14 Please respond to Bob Horst To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: IP-attached ATA storage There was some recent discussion about supporting ATA protocols over iSCSI. At 3ware, we have been working on IP-attached ATA storage for quite a while. We are already shipping a line of ATA PCI controllers with internal packet switching hardware that exports virtual SCSI devices, and have device drivers for several operating systems. We are just about to come out of stealth mode on the way we will extend these products to network-attached block storage subsystems with our Storage Control Protocol over TCP/IP. We will be showing our product at Networld+Interop in Atlanta and plan to ship production units by the end of the year. Although our protocol has some differences from iSCSI, we hope in the long run to work with the committee to ensure that the final iSCSI protocol is suitable for ATA storage as well as SCSI. Our initial protocol, which will ship before there is an agreed standard, will naturally be a subset of what is eventually standardized. We will upgrade the customers to the standard appropriately as it emerges. We are not quite ready to disclose our protocol, but I will be attending the IETF meeting in Pittsburgh and hope to begin interacting with the committee then. -- Bob Bob Horst 3ware, Inc. 490 California Ave. Palo Alto, CA 94306 http://www.3ware.com - att1.htm From owner-ips@ECE.cmu.edu Thu Jul 27 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13325; 27 Jul 2000 2:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA21838 for ips-outgoing; Thu, 27 Jul 2000 01:08:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA21823 for ; Thu, 27 Jul 2000 01:07:50 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA163308; Thu, 27 Jul 2000 07:07:20 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id HAA225414; Thu, 27 Jul 2000 07:07:19 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256929.001C20C6 ; Thu, 27 Jul 2000 07:07:13 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: ips@ece.cmu.edu Message-ID: Date: Thu, 27 Jul 2000 08:05:49 +0300 Subject: RE: IETF Draft for VI/TCP Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Can we get a room for some face-to-face design team meetings in Pittsburgh. If would be a vaste not to use the week to make some progress on some open issues. We can keep them widely open although I assume that they will not be as widely attended as the formal session. On site is convenient as people can slip in-and-out and attend some other sessions too. If we do make our private arrangements that would be more difficult. Thanks, Julo Black_David@emc.com on 27/07/2000 01:39:59 Please respond to Black_David@emc.com To: anicolson@giganet.com, ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: IETF Draft for VI/TCP FYI - this draft was submitted well after the cutoff two weeks prior to the Pittsburgh meeting, and hence will not be on the ips agenda for Pittsburgh. I'd like to discourage the announcement of drafts that have not made the cutoff during the two weeks prior to the meeting - there are very good reasons for that two week prior cutoff. Costa's draft did make it in time, and will have a brief agenda slot. Also, submitting drafts to the Internet-Drafts servers is usually a better approach than broadcasting them indiscriminately to the mailing list, as not everyone enjoys high-bandwidth email connections all the time. If for some reason, the Internet-Drafts servers are inappropriate (e.g., a near-final version is being announced for review before submission), please put the draft on a web site and send a URL to the list. As for putting the draft on the IETF web site - that's the responsibility of the draft authors once the site reopens for submissions after the Pittsburgh meeting. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Alex Nicolson [SMTP:Anicolson@giganet.com] > Sent: Wednesday, July 26, 2000 5:49 PM > To: ips@ece.cmu.edu > Subject: IETF Draft for VI/TCP > > Attached is the IETF draft submission for the VI over TCP by S. DiCecco > and > J. Williams. This document was sent to the IETF today but may not be > posted > in time for the meeting next week. This document is referenced in the > draft > titled "iSCSI/VI/TCP proposal" (draft-csapuntz-ips-iscsivi-00.txt) > available > as an IETF Draft. > > Could someone put this on the Web site? > > > Alex Nicolson > > Giganet, Inc. > Concord Office Center > 2352 Main Street > Concord, MA 01742 > 978.461.0402 (tel) > 978.461.0430 (fax) > anicolson@giganet.com << File: draft-dicecco-vitcp-00.txt >> From owner-ips@ECE.cmu.edu Thu Jul 27 06:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13348; 27 Jul 2000 2:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA22567 for ips-outgoing; Thu, 27 Jul 2000 01:53:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA22549 for ; Thu, 27 Jul 2000 01:53:18 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id WAA02953; Wed, 26 Jul 2000 22:53:11 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.153.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id WAA20932; Wed, 26 Jul 2000 22:53:09 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id WAA13741; Wed, 26 Jul 2000 22:53:09 -0700 (PDT) Message-ID: <397FCD4D.945EDAE7@ebay.sun.com> Date: Wed, 26 Jul 2000 22:49:01 -0700 From: David Robinson X-Mailer: Mozilla 4.72 [en] (Win98; I) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: IETF Draft for VI/TCP References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I would like to oppose this as it is not the IETF way of doing business. I would like to participate but there are also other WGs I need to attend and have scheduled travel around them. Last minute changes to the agenda really screws things up. Why can't we just bash things out on the WG alias so all can participate? -David Robinson julian_satran@il.ibm.com wrote: > > David, > > Can we get a room for some face-to-face design team meetings in Pittsburgh. > If would be a vaste not to use the week to make some progress on some open > issues. > We can keep them widely open although I assume that they will not be as > widely attended as the formal session. > > On site is convenient as people can slip in-and-out and attend some other > sessions too. If we do make our private arrangements that would be more > difficult. From owner-ips@ECE.cmu.edu Thu Jul 27 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13525; 27 Jul 2000 3:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA23022 for ips-outgoing; Thu, 27 Jul 2000 02:19:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA22997 for ; Thu, 27 Jul 2000 02:18:59 -0400 (EDT) Received: from scorpion.cisco.com (scorpion.cisco.com [171.69.18.74]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id XAA24853; Wed, 26 Jul 2000 23:18:43 -0700 (PDT) Received: (csapuntz@localhost) by scorpion.cisco.com (8.8.5-Cisco.2-SunOS.5.5.1.sun4/CISCO.WS.1.2) id XAA27069; Wed, 26 Jul 2000 23:18:28 -0700 (PDT) To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: IP-attached ATA storage References: cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 7.108) Content-Type: text/plain; charset=US-ASCII From: Costa Sapuntzakis Date: 26 Jul 2000 23:18:28 -0700 In-Reply-To: julian_satran@il.ibm.com's message of "Thu, 27 Jul 2000 08:11:22 +0300" Message-ID: Lines: 10 X-Mailer: Gnus v5.5/XEmacs 20.3 - "Vatican City" Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com writes: > Welcome Bob. That is a serious development and we are looking forward at > working with you. And there is no "comitee" - just a working group and 2 > chairmen (the working group is an open forum). And there isn't even a working group yet! -Costa From owner-ips@ECE.cmu.edu Thu Jul 27 11:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14561; 27 Jul 2000 7:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA21634 for ips-outgoing; Thu, 27 Jul 2000 06:05:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from radmail.rad.co.il (radmail.rad.co.il [207.232.32.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA21618 for ; Thu, 27 Jul 2000 06:04:50 -0400 (EDT) Received: from sanrad.com ([192.114.26.219]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-54115U1200L1200S0V35) with ESMTP id il; Thu, 27 Jul 2000 13:07:14 +0300 Message-ID: <39802580.46BF6169@sanrad.com> Date: Thu, 27 Jul 2000 13:05:20 +0100 From: Yaron Klein Organization: SanRad X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: David Robinson CC: julian_satran@il.ibm.com, Black_David@emc.com, ips@ece.cmu.edu Subject: Re: IETF Draft for VI/TCP References: <397FCD4D.945EDAE7@ebay.sun.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I'll oppose to that. I agree that WG *decisions* should be carried in consensus within the formal WG meetings. However, there are many issues that needs to b discussed and so many views that needs to be explained. I think that such face-to-face meeting is a great opportunity to hear the others. A meeting that is *open* to *all* of the community can only assist. Frankly, going all the way to Pittsburgh for 2 hours only, is a little bit of a waste - for someone whose main interest is Storage... Regards, Y. Klein David Robinson wrote: > I would like to oppose this as it is not the IETF way of doing > business. I would like to participate but there are also other > WGs I need to attend and have scheduled travel around them. > Last minute changes to the agenda really screws things up. > > Why can't we just bash things out on the WG alias so all > can participate? > > -David Robinson > > julian_satran@il.ibm.com wrote: > > > > David, > > > > Can we get a room for some face-to-face design team meetings in Pittsburgh. > > If would be a vaste not to use the week to make some progress on some open > > issues. > > We can keep them widely open although I assume that they will not be as > > widely attended as the formal session. > > > > On site is convenient as people can slip in-and-out and attend some other > > sessions too. If we do make our private arrangements that would be more > > difficult. From owner-ips@ECE.cmu.edu Thu Jul 27 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15405; 27 Jul 2000 11:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA28466 for ips-outgoing; Thu, 27 Jul 2000 10:29:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA28445 for ; Thu, 27 Jul 2000 10:29:30 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Thu, 27 Jul 2000 10:28:58 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FBB6@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Design Teams Date: Thu, 27 Jul 2000 10:28:57 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, > Can we get a room for some face-to-face design > team meetings in Pittsburgh. The short answer is "no", if for no other reason than there is no official IETF space available. The long answer is that the core iSCSI proposal has more or less reached the point where a small design team ceases to be an effective means of making progress. A focused design team is effective in producing an initial version of something, and the iSCSI design team has done well has done well in that regard. Once the initial version is done, wider review and comment is required to move from design team consensus to working group (rough) consensus. Working through the issues on the mailing list where all of the technical discussion is visible tends to be more effective than queuing up issues for design team meetings and interpreting the minutes to try to figure out what was considered by the design team. Design teams may still be appropriate for focused areas where it's necessary to make rapid progress on an initial version -- security comes to mind as a possible example. Also, the WG may need interim meetings between IETF meetings - if so, these will be announced well in advance and open to all. Let me also make it clear that this note should not discourage interested people from meeting in whatever informal circumstances are convenient, as long as everyone keeps in mind that the list should be the primary forum for technical discussions and decision-making rather than merely being used to review decisions made offline. I've been personally involved in an IETF WG in which design teams were used effectively (disbanded after production of the initial version) and one in which they were used ineffectively (persisted long after the initial version). IMHO, it's more than coincidence that the latter WG has encountered greater difficulty and consumed much more time than expected in making progress towards its goals. I'd prefer not to identify the WGs in order to avoid an off-topic discussion. I believe the AD(s) will have more to say on this subject in Pittsburgh. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Jul 27 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16088; 27 Jul 2000 15:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA06303 for ips-outgoing; Thu, 27 Jul 2000 14:16:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f227.law10.hotmail.com [64.4.15.227]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id OAA06287 for ; Thu, 27 Jul 2000 14:16:35 -0400 (EDT) Received: (qmail 75667 invoked by uid 0); 27 Jul 2000 18:16:01 -0000 Message-ID: <20000727181601.75666.qmail@hotmail.com> Received: from 192.115.216.67 by www.hotmail.com with HTTP; Thu, 27 Jul 2000 11:16:01 PDT X-Originating-IP: [192.115.216.67] From: "Julian Satran" To: ips@ece.cmu.edu Date: Thu, 27 Jul 2000 21:16:01 IDT Mime-Version: 1.0 Content-Type: text/html Sender: owner-ips@ece.cmu.edu Precedence: bulk

I've placed a memo about the framing and RDMA at:

http://www.haifa.il.ibm.com/satran/iSCSI-RDMA-memo.txt

As it is meant to get obsolete sooner (I hope) than the draft it refers too I was hesitant about cluttering the ID area with it and I will post it on my home site at IBM and on the document area on the IPS Mailing list archive (if Dave Nagle will help me on that).

C U in Pittsburgh

Julo

Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com
From owner-ips@ECE.cmu.edu Thu Jul 27 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16090; 27 Jul 2000 15:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA05951 for ips-outgoing; Thu, 27 Jul 2000 14:08:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA05928 for ; Thu, 27 Jul 2000 14:07:55 -0400 (EDT) Received: from scorpion.cisco.com (scorpion.cisco.com [171.69.18.74]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id LAA15169; Thu, 27 Jul 2000 11:07:38 -0700 (PDT) Received: (csapuntz@localhost) by scorpion.cisco.com (8.8.5-Cisco.2-SunOS.5.5.1.sun4/CISCO.WS.1.2) id LAA15321; Thu, 27 Jul 2000 11:07:23 -0700 (PDT) Date: Thu, 27 Jul 2000 11:07:23 -0700 (PDT) Message-Id: <200007271807.LAA15321@scorpion.cisco.com> From: Costa Sapuntzakis To: ips@ece.cmu.edu Subject: iSCSI/VI/TCP cc: csapuntz@cisco.com X-csapuntz: outbox Sender: owner-ips@ece.cmu.edu Precedence: bulk I have sent out a proposal (draft-csapuntz-iscsivi-00.txt) that argues for running iSCSI on top of the VI/TCP protocol. I argue that there are a couple principal reasons for doing this. 1) Generic RDMA infrastructure. A single NIC that support VI/TCP RDMA acceleration could accelerate SCSI, DAFS (http://www.dafscollaborative.org/), Oracle, and other message+RDMA protocols w/o the need for specific protocol processing in the NIC. 2) Reduce the need for TCP reassembly memory Why? For performance, TCP maintains a window at least as large as the application buffers awaiting data. The window buffer must be separate memory than the application buffers if the receiver cannot figure out which application buffers TCP payloads belongs to. Thus, buffering requirements are potentially up to 2x that of the base application. - VI/TCP could align its headers at the start of each TCP segment payload [D] This is not currently well discussed in the spec. However, there are a couple things techniques that could be used to indicate alignment. For one, a TCP option could be used to indicate that the sender will send aligned packets. As added check, each VI frame could end in a CRC. Failure of the CRC indicates loss of synchronization. Also, there are some fields in the VI/TCP header that should make sense (like the Segment Length < MTU). - The VI/TCP header, which appears in each segment, could direct the payload of the segment to the correct application-layer buffer (for both messages and data) 3) More exact flow control VI/TCP provides flow control on messages separate from data buffers. It is worthwhile to note that iSCSI could be modified to incorporate similar features to VI/TCP and do #2 and #3. Others have stated some valid concerns with iSCSI/VI: >- It adds another protocol layer (VI/TCP) >- it adds another dependency (VI/TCP is not a standard yet) > also heard, don't base new technology on new technology >- all SCSI class drivers and file systems that I'm aware of operate in kernel >space. So what advantage does VI bring when the app must enter kernel space >to perform the I/O anyway? RDMA can improve kernel I/O performance too. >- it adds more complexity to developing the iSCSI accelerator to add yet >another protocol layer. To this, I would answer that it allows the construction of a potentially simpler accelerator that handles more protocols. > There already is a SCSI/VI. Our group is looking for the best way to do SCSI over IP. For a long time, members of this group have though that some kind of RDMA layer would be valuable for building simpler NICs. VI/TCP provides just that sort of RDMA layer. SCSI/VI was born out of a different set of requirements than iSCSI. However, it may turn out that analysis of the problem brings us to a very similar solution. At the end of the day, this argument comes down to whether folks think that an RDMA mechanism is the right way to go and what RDMA scheme over IP is best. A suitably generic RDMA scheme allows innovation in protocol design, without worrying that the new protocol will be hopelessly slow in legacy hardware. It decreases the pressure to load features into old, hardware-supported protocols instead of introducing new ones. It destroy an artificial barrier (host overhead) that prevents IP from scaling to fit all our needs. -Costa From owner-ips@ECE.cmu.edu Thu Jul 27 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16097; 27 Jul 2000 15:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA06544 for ips-outgoing; Thu, 27 Jul 2000 14:23:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA06529 for ; Thu, 27 Jul 2000 14:23:24 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id EC8BC4D1 for ; Thu, 27 Jul 2000 14:23:23 -0400 (EDT) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id LAA05447; Thu, 27 Jul 2000 11:23:51 -0700 (PDT) Message-Id: <200007271823.LAA05447@core.rose.hp.com> Subject: Comments on draft-satran-iscsi-01.txt To: ips@ece.cmu.edu Date: Thu, 27 Jul 2000 11:23:51 PDT Cc: randy_haagens@hp.com Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk All: Please find enclosed my comments on the latest draft of iSCSI (draft-satran-iscsi-01.txt). The first section has comments and the second section has additional enhancements I am requesting. Thanks. -- Mallikarjun cbm@rose.hp.com Networked Storage Architecture HP Storage Organization, Roseville. Comments on draft-satran-iscsi-01.txt ------------------------------------- o Section 2.1, page 4. Paraphrases SAM-2 by defining a task as a "a linked set of SCSI commands". Suggest rewording as "a SCSI command, or possibly a linked set of SCSI commands". Wherever applicable, the iSCSI draft should also cite the corresponding SAM-2 clause numbers. o Section 2.2.2 on page 5. First sentence states "iSCSI supports ordered command delivery within a session.". While I realize that ordered completion is not required by the iSCSI spec, I suggest rewording as "iSCSI supports ordered task initiation and completion within a session". o I have a general question about all iSCSI PDUs from the initiator to the target carrying the CmdRN. Section 3.10.5 seems to leave it as an implementation choice, but why is it necessary? Once we stipulated task allegiance to a TCP connection and given that the SCSI Data PDU has the `Buffer Offset' field in it, why would the CmdRN be necessary for iSCSI Data PDUs from the initiator to the target? o Section 2.2.2 on page 5. Towards the end of the second para, "The target will reject any command outside this range or...." should be reworded to make it obvious as "The target will ignore any command outside this range or...". We do not want network bandwidth to be consumed for something that cannot be valid. o Section 2.2.2 on page 5. I suggest that the spec should spell out all the conditions under which the CmdRN is reset back to the initial value - target reset (currenly forces session termination, but even if doesn't in future), lun reset, and re-establishment of session. o Section 2.2.2 on page 6, second para. There's a sentence - "iSCSI initiators are required to implement the numbering scheme if they support more than one connection." Suggest adding "per session" at the end of the sentence, as per current status of the spec. If and when session recovery mechanisms are defined, this statement may have to be modified to perhaps even mandate numbering with one TCP connection, and across sessions. o Section 2.2.2 on page 6, third para - "iSCSI targets are not required to use the numbering scheme for ordered delivery". I take it that the iSCSI protocol layer which provides the service delivery port abstraction to the device server is not required to deliver commands in the CmdRN order. It appears though as if the target iSCSI layer shall always keep track of the ExpCmdRN. This should be stated explicitly. Also, it appears that the StatRN may not be valid coming from the targets which do not re-order received commands by CmdRN. It would help to state this as well. o This capability of a target to enforce the numbering scheme in full is something the initiators would be better off knowing. I suggest adding a Login/Text key for this. EnforceOrdering: --> EnforceOrdering: o Section 2.2.4 on page 7, first para. Contains a discussion about connection allegiance, which limits to commands. I suggest tasks be used instead. I would also propose that the abort task management request should also have the same connection allegiance as the original command. o Section 3.2.1 on page 14. Autosense is made optional through a bit setting in the SCSI Command PDU. Why can't it be made mandatory? That would make the life of the FC-iSCSI bridges a lot more easier, since FC mandates it. Also, section 5.3 discusses Autosense from the device perspective, but stops short of saying that Autosense is mandatory for all iSCSI targets. Are there any device issues complicating the situation here? It appears that compliance with FC has been fairly in place. o Section 3.3.4 on page 17. Suggest adding an additional iSCSI Status value in the SCSI Response PDU - "2 Non-existent iSCSI session". This shall be returned if a SCSI command were attempted without an established iSCSI login session. Returning a Logout PDU is another option (see the enhancement proposal below). o Section 3.3.7 on page 17. It appears to be in need of more definition about Response data. I propose the following response data values - - RTT-related: the data sent did not match the burst size, or the offset allowed in the RTT message. - SCSI Command format: invalid command format. o Section 3.3.7 on page 17. It appears that there's a violation of SAM-2 transport and application protocol layering here. The discussion allows certain iSCSI (transport) errors to be indicated when the Command Status (SCSI application protocol) is CHECK CONDITION. It would seem that all such transport errors should be able to be flagged with a non-zero iSCSI Status alone. The initiators would in that case are expected to look at the iSCSI status first, and proceed further only if it is zero. o Sections 3.4 and 3.5 on pages 19 and 20. Suggest adding one sentence each to NOP-OUT and NOP-IN sections to explicitly state the direction of flow of the PDU. o Section 3.7.1 on page 23. For the Target Reset task management function, the target is not expected to provide a response - and this is concerning to me. - how would the initiator confirm the successful completion of the target reset? - not having a response effectively makes the target reset an operation iSCSI hardware cannot assist. Having a response would make it no different from the rest of the iSCSI transactions and the hardware can gracefully deal with it. - SAM-2 (section 6.6, page 63) specifies what a target should do "Before returning a FUNCTION COMPLETE response". This seems to me as an implicit requirement that a response be returned on a Target Reset task management function. - I am also unclear as to why the sessions are allowed to be terminated. In the FC world, as far as I can recall, the process login sessions remain intact after a target reset. If it is required that the sessions and the associated TCP connections be cleared in iSCSI, it is helpful to mandate (as opposed to leaving it up to the implementation) that an Async event shall be reported to all the initiators currently logged in. o Section 3.8 on page 25. Suggest additional SCSI Task Management Response indications in addition to the two defined. 2 Function Invalid (the function is invalid as per current rev) 3 Function Unsupported (valid, but implementation doesn't support) Also suggest adding iSCSI Status and Response data fields to the PDU. - iSCSI Status can take three values: success, non-existent LUN, and non-existent iSCSI session. - Response data can take one value: Invalid message format. o Section 3.10.2 on page 29. The Transfer Tag is the Initiator Task Tag in SCSI Data PDU from target to initiator. Why then are both shown in the payload diagram? If this is done to retain some resemblance between the two types of SCSI Data PDUs, I would suggest that ideally there be one type of SCSI Data PDU for both READ and WRITE - with certain fields in the PDU to be ignored in each case. This makes it easier on hardware and software implementations. o Section 3.11 on page 31. Suggest adding statement "The capabilities exchanged and operations performed are valid for the entire login session including all TCP connections for that session." This makes it clear, for ex., as in the case of: should authentication be performed on every new connection, or only on the leading connection. o Since Text key pairs are used as part of Login process and outside, I advocate that they be named thus - "Text key pairs". This terminology can consistently be used across. The current usage has "Text Command format" (in section 3.13), and "Login/Text keys" elsewhere (section 10). o I suggest that Login dialogue should also be able to identify the alternate names the same target (task manager and the device servers, albeit possibly with different target "id"s) is available from. This would mean including more Text key pairs in Appendix B. A possible format is - OtherNames: where the Descriptor type is as defined in section 3.17.1 and the Descriptor value is based on the given type. o Section 3.14 on page 39. The first paragraph allows a target to respond to a Login Command with a Login Response and an unsolicited Text Response PDU. I suggest that there be a "Login reply" bit in the Text Response PDU to indicate to the initiator that the PDU is not in response to a Text Command, but as a reply to a login proposal. o Section 3.14.1 on page 39. States that the "InitStatRN is significant only if TSID is 0". I am somewhat confused by this. It appears that it should be non-zero. I am assuming that the TSID is non-zero in the first Login Response on a connection for a given session (leading connection), and is zero in all subsequent Login Responses on other connections of the same session. If this were true, only the leading connection's login dialogue should specify InitStatRN. [ I see now that Mike from IBM already pointed this out, but am keeping this to confirm the expected target behavior that I described. ] o Section 3.18 on page 46. Suggest adding comments to explicitly state that target should reject (with a non-zero Response) the Map Command when a map (or unmap) of a particular TAN (or SRA) fails out of a set of descriptors. Essentially, either all descriptors succeed, or all end up in a failure. [ I see that Bill Main also had a comment on the same issue. ] o Section 4.1 on page 49. Second paragraph from the bottom. "if they are not acknowledged yet or a new CmdRN if they where acknowledged;" should be "if they are not acknowledged yet or a new CmdRN if they were acknowledged". o Section 9.2 on page 59. The Write operation example depicts multiple SCSI Data PDUs being shipped to the target in response to one RTT PDU. This effectively puts a requirement on the target implementations to keep the target transfer tag valid until the expected data size is received. It would be helpful to explicitly state this in section 3.9. New proposals for enhancements ------------------------------ o I propose a new opcode to do Third party Logout - to provide a service with the (almost) same name in Fibre Channel (Third Party Process Logout TPRLO). This is issued by an initiator, and it requests the target to logout with all third party initiators who are iSCSI logged in with the given target. This should be effective on all the initiators having iSCSI access to the same set of task manager and device servers. This feature allows one host (initiator) to ensure that there there are no other hosts talking to a target device, in failover configurations. In order to ensure that this is not maliciously used by rogue hosts, iSCSI target may selectively allow initiators to do TPRLO with a TPRLO-password to be specified in the TPRLO command PDU, this password communicated in the login dialogue. o I propose adding a set of two new iSCSI opcodes for "Logout" and "Logout Response". - Logout and Logout Response enable the dual roles of initiator and target to be independently played by SCSI devices. - these can also be used as an error recovery mechanism by a target, forcing the initiator to re-login. - these also enable multiple sessions to be operational across a pair of SCSI devices (if and when we want it). This Logout is _across_ all channels associated with the iSCSI Session, and a graceful connection termination using TCP FINs for all the individual TCP connections (other than the one Logout is delivered on) is recommended before this Logout command. o Irrespective of the exact process to handle an error, I propose that an upper bound be specified on the time that a SCSI device (initiator or target) should wait before freeing up resources allocated for a SCSI task (and thus assume the implicit termination of the SCSI task). This timeout shall only be used in the case of a continued failure to re-establish the Login session for the said period. I know that this is a passionate topic, but even a ridiculously high upper bound (say 4 hours for task), varying on device class (disk, tape ..) is better than no upper bound. This is the only architected way for certain hosts (say rebooted on an average, once an year) to recover task resources like Tags. Note that this is not an attempt to impose new timers on an iSCSI implementation, this only requires that the resources must be set aside for this much time - one could envision an implementation which would reallocate resources only as needed with no timers, so the resources can be set aside far longer than the iSCSI spec requires. o I propose that a new iSCSI PDU "SCSI Conf" (analogous to FCP_CONF of FC) be defined as a payload sent from an initiator to a target. This informs the target that the initiator received the SCSI Response message on that Initiator Task Tag. A target should wait to execute the subsequent commands on an error until this SCSI Confirmation PDU is received. This handshake "allows subsequent queued stateful operations to be performed" (taken out of FCP-2 spec). In the context of tapes/asynch mirroring, this preserves ordering/coherency since the target device stalls for the SCSI Conf from the initiator. To avoid unnecessary overhead, a target would request the SCSI Conf message in the SCSI Response message only on a SCSI task ending in an error. Also, SCSI Conf should also play by the rules of the connection allegiance. o I suggest the following new Login/Text keys in section 10 - - Following are for capabilities. InitiatorCapability: TargetCapability: - Following for protocol revision. iSCSIRevision: The responder to a Login command may choose to propose an equal or a lower rev than proposed in the Login command payload. If the counter-proposal in the response is not acceptable, the sender of Login command should immediately log out. If the responder can only support higher revs, login is rejected. - Following, subsequent to the SCSI Conf enhancement request. SCSIConfSupport: --> SCSIConfSupport: From owner-ips@ECE.cmu.edu Thu Jul 27 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16559; 27 Jul 2000 17:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA11067 for ips-outgoing; Thu, 27 Jul 2000 16:17:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.5]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA11053 for ; Thu, 27 Jul 2000 16:17:51 -0400 (EDT) Received: by apollo.pirus.com with Internet Mail Service (5.5.2650.21) id ; Thu, 27 Jul 2000 16:27:49 -0400 Message-ID: <618471D08ABDD31188B6009027E500930B9BD5@apollo.pirus.com> From: "Merhar, Milan" To: "'Costa Sapuntzakis'" , ips@ece.cmu.edu Subject: RE: iSCSI/VI/TCP Date: Thu, 27 Jul 2000 16:27:48 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, Thanks for your comments on the value of using VI as part of the iSCSI solution. I was pretty much in agreement with you, until I got to your last paragraph. > > At the end of the day, this argument comes down to whether folks think > that an RDMA mechanism is the right way to go and what RDMA scheme > over IP is best. I agree. If you are transporting data between application memory and a remote periperal device, a scheme that permits direct DMA transfer makes sense. And, if we agree to that, it also makes sense to investigate a range of DMA transfer solutions, to see which provides the most appropriate capabilities, allows broad implementation, etc. > A suitably generic RDMA scheme allows innovation in > protocol design, without worrying that the new protocol will be > hopelessly slow in legacy hardware. It decreases the pressure to load > features into old, hardware-supported protocols instead of introducing > new ones. Here's where I must ask for a clarification. Are you saying that performance, or for that matter the ability to implement these solutions on existing hardware, is a "don't care"? Or, are you saying that a RDMA scheme is inherently efficient enough that even legacy hardware will work just fine? I hope you meant the latter, but admit that my first impression was that you'd give up "legacy hardware" support for a fast protocol that could only run in a new hardware environment. > It destroy an artificial barrier (host overhead) that > prevents IP from scaling to fit all our needs. > Allowing IP to be used across a wider range of solutions is a good thing. Removing the perception that IP stacks are "cycle hogs" is a good thing, too. But, I suggest that its history of maintaining compatibility with older hardware and software implementations is also a pretty good selling point for TCP/IP, and one that shouldn't be lightly dismissed. - milan Milan J. Merhar, Chief Engineer, Pirus Networks, 43 Nagog Park, Acton MA 01720-3425 voice: 978-206-9124 email: milan@pirus.com From owner-ips@ECE.cmu.edu Thu Jul 27 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16552; 27 Jul 2000 17:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA11089 for ips-outgoing; Thu, 27 Jul 2000 16:18:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA11070 for ; Thu, 27 Jul 2000 16:18:05 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA00430 for ; Thu, 27 Jul 2000 13:16:20 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id NAA14773 for ; Thu, 27 Jul 2000 13:13:55 -0700 From: "Douglas Otis" To: Subject: Agenda suggestion Date: Thu, 27 Jul 2000 13:19:52 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk As the impact of the VI/TCP proposal will have a significant effect on security, perhaps a bit more time should be spent with Constantine Sapuntzakis from Cisco rather than 30 minutes dedicated to Steve Bellovin. This is provided the co-chair is willing... 30 min iSCSI Security Requirements (Steve Bellovin, co-chair) 2.5 min VI/TCP for iSCSI (Constantine Sapuntzakis, Cisco) draft-csapuntz-ips-iscsivi-00.txt Doug From owner-ips@ECE.cmu.edu Thu Jul 27 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16509; 27 Jul 2000 17:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA10096 for ips-outgoing; Thu, 27 Jul 2000 15:50:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gifw.genroco.com (genroco.com [205.254.195.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA10081 for ; Thu, 27 Jul 2000 15:50:07 -0400 (EDT) Received: from gi2.genroco.com (gi2.genroco.com [192.133.120.3]) by gifw.genroco.com (8.9.3/8.9.3) with ESMTP id OAA20191 for ; Thu, 27 Jul 2000 14:50:06 -0500 Received: from don (don.genroco.com [192.133.120.101]) by gi2.genroco.com (8.9.3/8.9.3) with SMTP id OAA21964 for ; Thu, 27 Jul 2000 14:50:06 -0500 From: "Don Woelz" To: Subject: RE: iSCSI/VI/TCP Date: Thu, 27 Jul 2000 14:50:06 -0500 Message-ID: <002f01bff803$dcfcab40$657885c0@genroco.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <200007271807.LAA15321@scorpion.cisco.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > At the end of the day, this argument comes down to whether folks think > that an RDMA mechanism is the right way to go and what RDMA scheme > over IP is best. A suitably generic RDMA scheme allows innovation in > protocol design, without worrying that the new protocol will be > hopelessly slow in legacy hardware. It decreases the pressure to load > features into old, hardware-supported protocols instead of introducing > new ones. It destroy an artificial barrier (host overhead) that > prevents IP from scaling to fit all our needs. Costa, Your thoughts reflect ours as well, with one exception. We chose the Scheduled Transfer Protocol (STP), which is essentially ANSI RDMA, as the transport for SCSI (SCSI on ST, or SST, an ANSI T10 draft standard, Project 1380-D). STP is defined over IP (and implemented that way in Linux), so SST is an ANSI SCSI on IP mapping. Regards, Don Donald D. Woelz Tel: 262-644-2505 V. P. of Marketing Mobile: 414-732-7705 GENROCO, Inc. Toll Free: 800-243-6762 255 Info Hwy Fax: 262-644-6667 Slinger, WI 53086 From owner-ips@ECE.cmu.edu Thu Jul 27 21:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16591; 27 Jul 2000 17:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA12130 for ips-outgoing; Thu, 27 Jul 2000 16:47:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA12115 for ; Thu, 27 Jul 2000 16:47:46 -0400 (EDT) Received: from csapuntz-u1.cisco.com (csapuntz-u1.cisco.com [10.34.2.69]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id NAA20920; Thu, 27 Jul 2000 13:47:29 -0700 (PDT) Received: (csapuntz@localhost) by csapuntz-u1.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) id NAA04906; Thu, 27 Jul 2000 13:47:17 -0700 (PDT) To: "Merhar, Milan" cc: ips@ece.cmu.edu Subject: Re: iSCSI/VI/TCP References: <618471D08ABDD31188B6009027E500930B9BD5@apollo.pirus.com> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 7.108) Content-Type: text/plain; charset=US-ASCII From: Costa Sapuntzakis Date: 27 Jul 2000 13:47:17 -0700 In-Reply-To: "Merhar, Milan"'s message of "Thu, 27 Jul 2000 16:27:48 -0400" Message-ID: Lines: 33 X-Mailer: Gnus v5.5/XEmacs 20.3 - "Vatican City" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Constantine Sapuntzakis writes: > > A suitably generic RDMA scheme allows innovation in ^^^^^^ = I should have used the word protocol here > > protocol design, without worrying that the new protocol will be ^^^^ higher-layer protocol > > hopelessly slow in legacy hardware. It decreases the pressure to load > > features into old, hardware-supported protocols instead of introducing > > new ones. > > Here's where I must ask for a clarification. Are you saying that Thanks for pointing out the lack of clarity. I should have made the layering in my argument more explicit. I am talking about innovation in protocols at the iSCSI/NFS/CIFS layer. Hypothetical world: The world starts building protocol-specific NICs, for example, iSCSI NICs. There is a large installed base of iSCSI NICs. The NICs make data transfers on iSCSI run quickly while other protocols go through the software stack. In this world, let's say you want to create a new protocol/API that competes with iSCSI/SCSI. Well, to make it run quickly on the installed base of iSCSI NICs, you would probably end up figuring out how to run it on top of SCSI or iSCSI instead of creating a new protocol. This puts pressure to dump features into iSCSI. Admittedly, calling iSCSI "an old, hardware-support protocol" is a bit premature. -Costa From owner-ips@ECE.cmu.edu Thu Jul 27 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16853; 27 Jul 2000 18:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA13065 for ips-outgoing; Thu, 27 Jul 2000 17:17:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA13048 for ; Thu, 27 Jul 2000 17:17:28 -0400 (EDT) From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA60972; Thu, 27 Jul 2000 17:06:38 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id PAA139796; Thu, 27 Jul 2000 15:11:54 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256929.00747122 ; Thu, 27 Jul 2000 15:11:51 -0600 X-Lotus-FromDomain: IBMUS To: cbm@rose.hp.com cc: ips@ece.cmu.edu, randy_haagens@hp.com Message-ID: <87256929.0073EEA8.00@d53mta03h.boulder.ibm.com> Date: Thu, 27 Jul 2000 14:07:09 -0700 Subject: Re: Comments on draft-satran-iscsi-01.txt Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun and others, I would like to voice my support for a logout mechanism. The draft advises us to use TCP FINs but a standard mechanism to close channels/sessions would be preferable (generating TCP FINs is non-trivial in some programming environs) Personally, our experience has been that the logout mechanism has been very useful in Fibre Channel and would be a good resource deallocation/error recovery mechanism in iSCSI. I had suggested a logout mechanism in iSCSI at the early stages of the draft (Feb timeframe), for some reason it was not accepted. Perhaps this is a good time for a discussion on the issue. Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose "Mallikarjun C." @ece.cmu.edu on 07/27/2000 11:23:51 AM Please respond to cbm@rose.hp.com Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: randy_haagens@hp.com Subject: Comments on draft-satran-iscsi-01.txt All: Please find enclosed my comments on the latest draft of iSCSI (draft-satran-iscsi-01.txt). The first section has comments and the second section has additional enhancements I am requesting. Thanks. -- Mallikarjun cbm@rose.hp.com Networked Storage Architecture HP Storage Organization, Roseville. Comments on draft-satran-iscsi-01.txt ------------------------------------- o Section 2.1, page 4. Paraphrases SAM-2 by defining a task as a "a linked set of SCSI commands". Suggest rewording as "a SCSI command, or possibly a linked set of SCSI commands". Wherever applicable, the iSCSI draft should also cite the corresponding SAM-2 clause numbers. o Section 2.2.2 on page 5. First sentence states "iSCSI supports ordered command delivery within a session.". While I realize that ordered completion is not required by the iSCSI spec, I suggest rewording as "iSCSI supports ordered task initiation and completion within a session". o I have a general question about all iSCSI PDUs from the initiator to the target carrying the CmdRN. Section 3.10.5 seems to leave it as an implementation choice, but why is it necessary? Once we stipulated task allegiance to a TCP connection and given that the SCSI Data PDU has the `Buffer Offset' field in it, why would the CmdRN be necessary for iSCSI Data PDUs from the initiator to the target? o Section 2.2.2 on page 5. Towards the end of the second para, "The target will reject any command outside this range or...." should be reworded to make it obvious as "The target will ignore any command outside this range or...". We do not want network bandwidth to be consumed for something that cannot be valid. o Section 2.2.2 on page 5. I suggest that the spec should spell out all the conditions under which the CmdRN is reset back to the initial value - target reset (currenly forces session termination, but even if doesn't in future), lun reset, and re-establishment of session. o Section 2.2.2 on page 6, second para. There's a sentence - "iSCSI initiators are required to implement the numbering scheme if they support more than one connection." Suggest adding "per session" at the end of the sentence, as per current status of the spec. If and when session recovery mechanisms are defined, this statement may have to be modified to perhaps even mandate numbering with one TCP connection, and across sessions. o Section 2.2.2 on page 6, third para - "iSCSI targets are not required to use the numbering scheme for ordered delivery". I take it that the iSCSI protocol layer which provides the service delivery port abstraction to the device server is not required to deliver commands in the CmdRN order. It appears though as if the target iSCSI layer shall always keep track of the ExpCmdRN. This should be stated explicitly. Also, it appears that the StatRN may not be valid coming from the targets which do not re-order received commands by CmdRN. It would help to state this as well. o This capability of a target to enforce the numbering scheme in full is something the initiators would be better off knowing. I suggest adding a Login/Text key for this. EnforceOrdering: --> EnforceOrdering: o Section 2.2.4 on page 7, first para. Contains a discussion about connection allegiance, which limits to commands. I suggest tasks be used instead. I would also propose that the abort task management request should also have the same connection allegiance as the original command. o Section 3.2.1 on page 14. Autosense is made optional through a bit setting in the SCSI Command PDU. Why can't it be made mandatory? That would make the life of the FC-iSCSI bridges a lot more easier, since FC mandates it. Also, section 5.3 discusses Autosense from the device perspective, but stops short of saying that Autosense is mandatory for all iSCSI targets. Are there any device issues complicating the situation here? It appears that compliance with FC has been fairly in place. o Section 3.3.4 on page 17. Suggest adding an additional iSCSI Status value in the SCSI Response PDU - "2 Non-existent iSCSI session". This shall be returned if a SCSI command were attempted without an established iSCSI login session. Returning a Logout PDU is another option (see the enhancement proposal below). o Section 3.3.7 on page 17. It appears to be in need of more definition about Response data. I propose the following response data values - - RTT-related: the data sent did not match the burst size, or the offset allowed in the RTT message. - SCSI Command format: invalid command format. o Section 3.3.7 on page 17. It appears that there's a violation of SAM-2 transport and application protocol layering here. The discussion allows certain iSCSI (transport) errors to be indicated when the Command Status (SCSI application protocol) is CHECK CONDITION. It would seem that all such transport errors should be able to be flagged with a non-zero iSCSI Status alone. The initiators would in that case are expected to look at the iSCSI status first, and proceed further only if it is zero. o Sections 3.4 and 3.5 on pages 19 and 20. Suggest adding one sentence each to NOP-OUT and NOP-IN sections to explicitly state the direction of flow of the PDU. o Section 3.7.1 on page 23. For the Target Reset task management function, the target is not expected to provide a response - and this is concerning to me. - how would the initiator confirm the successful completion of the target reset? - not having a response effectively makes the target reset an operation iSCSI hardware cannot assist. Having a response would make it no different from the rest of the iSCSI transactions and the hardware can gracefully deal with it. - SAM-2 (section 6.6, page 63) specifies what a target should do "Before returning a FUNCTION COMPLETE response". This seems to me as an implicit requirement that a response be returned on a Target Reset task management function. - I am also unclear as to why the sessions are allowed to be terminated. In the FC world, as far as I can recall, the process login sessions remain intact after a target reset. If it is required that the sessions and the associated TCP connections be cleared in iSCSI, it is helpful to mandate (as opposed to leaving it up to the implementation) that an Async event shall be reported to all the initiators currently logged in. o Section 3.8 on page 25. Suggest additional SCSI Task Management Response indications in addition to the two defined. 2 Function Invalid (the function is invalid as per current rev) 3 Function Unsupported (valid, but implementation doesn't support) Also suggest adding iSCSI Status and Response data fields to the PDU. - iSCSI Status can take three values: success, non-existent LUN, and non-existent iSCSI session. - Response data can take one value: Invalid message format. o Section 3.10.2 on page 29. The Transfer Tag is the Initiator Task Tag in SCSI Data PDU from target to initiator. Why then are both shown in the payload diagram? If this is done to retain some resemblance between the two types of SCSI Data PDUs, I would suggest that ideally there be one type of SCSI Data PDU for both READ and WRITE - with certain fields in the PDU to be ignored in each case. This makes it easier on hardware and software implementations. o Section 3.11 on page 31. Suggest adding statement "The capabilities exchanged and operations performed are valid for the entire login session including all TCP connections for that session." This makes it clear, for ex., as in the case of: should authentication be performed on every new connection, or only on the leading connection. o Since Text key pairs are used as part of Login process and outside, I advocate that they be named thus - "Text key pairs". This terminology can consistently be used across. The current usage has "Text Command format" (in section 3.13), and "Login/Text keys" elsewhere (section 10). o I suggest that Login dialogue should also be able to identify the alternate names the same target (task manager and the device servers, albeit possibly with different target "id"s) is available from. This would mean including more Text key pairs in Appendix B. A possible format is - OtherNames: where the Descriptor type is as defined in section 3.17.1 and the Descriptor value is based on the given type. o Section 3.14 on page 39. The first paragraph allows a target to respond to a Login Command with a Login Response and an unsolicited Text Response PDU. I suggest that there be a "Login reply" bit in the Text Response PDU to indicate to the initiator that the PDU is not in response to a Text Command, but as a reply to a login proposal. o Section 3.14.1 on page 39. States that the "InitStatRN is significant only if TSID is 0". I am somewhat confused by this. It appears that it should be non-zero. I am assuming that the TSID is non-zero in the first Login Response on a connection for a given session (leading connection), and is zero in all subsequent Login Responses on other connections of the same session. If this were true, only the leading connection's login dialogue should specify InitStatRN. [ I see now that Mike from IBM already pointed this out, but am keeping this to confirm the expected target behavior that I described. ] o Section 3.18 on page 46. Suggest adding comments to explicitly state that target should reject (with a non-zero Response) the Map Command when a map (or unmap) of a particular TAN (or SRA) fails out of a set of descriptors. Essentially, either all descriptors succeed, or all end up in a failure. [ I see that Bill Main also had a comment on the same issue. ] o Section 4.1 on page 49. Second paragraph from the bottom. "if they are not acknowledged yet or a new CmdRN if they where acknowledged;" should be "if they are not acknowledged yet or a new CmdRN if they were acknowledged". o Section 9.2 on page 59. The Write operation example depicts multiple SCSI Data PDUs being shipped to the target in response to one RTT PDU. This effectively puts a requirement on the target implementations to keep the target transfer tag valid until the expected data size is received. It would be helpful to explicitly state this in section 3.9. New proposals for enhancements ------------------------------ o I propose a new opcode to do Third party Logout - to provide a service with the (almost) same name in Fibre Channel (Third Party Process Logout TPRLO). This is issued by an initiator, and it requests the target to logout with all third party initiators who are iSCSI logged in with the given target. This should be effective on all the initiators having iSCSI access to the same set of task manager and device servers. This feature allows one host (initiator) to ensure that there there are no other hosts talking to a target device, in failover configurations. In order to ensure that this is not maliciously used by rogue hosts, iSCSI target may selectively allow initiators to do TPRLO with a TPRLO-password to be specified in the TPRLO command PDU, this password communicated in the login dialogue. o I propose adding a set of two new iSCSI opcodes for "Logout" and "Logout Response". - Logout and Logout Response enable the dual roles of initiator and target to be independently played by SCSI devices. - these can also be used as an error recovery mechanism by a target, forcing the initiator to re-login. - these also enable multiple sessions to be operational across a pair of SCSI devices (if and when we want it). This Logout is _across_ all channels associated with the iSCSI Session, and a graceful connection termination using TCP FINs for all the individual TCP connections (other than the one Logout is delivered on) is recommended before this Logout command. o Irrespective of the exact process to handle an error, I propose that an upper bound be specified on the time that a SCSI device (initiator or target) should wait before freeing up resources allocated for a SCSI task (and thus assume the implicit termination of the SCSI task). This timeout shall only be used in the case of a continued failure to re-establish the Login session for the said period. I know that this is a passionate topic, but even a ridiculously high upper bound (say 4 hours for task), varying on device class (disk, tape ..) is better than no upper bound. This is the only architected way for certain hosts (say rebooted on an average, once an year) to recover task resources like Tags. Note that this is not an attempt to impose new timers on an iSCSI implementation, this only requires that the resources must be set aside for this much time - one could envision an implementation which would reallocate resources only as needed with no timers, so the resources can be set aside far longer than the iSCSI spec requires. o I propose that a new iSCSI PDU "SCSI Conf" (analogous to FCP_CONF of FC) be defined as a payload sent from an initiator to a target. This informs the target that the initiator received the SCSI Response message on that Initiator Task Tag. A target should wait to execute the subsequent commands on an error until this SCSI Confirmation PDU is received. This handshake "allows subsequent queued stateful operations to be performed" (taken out of FCP-2 spec). In the context of tapes/asynch mirroring, this preserves ordering/coherency since the target device stalls for the SCSI Conf from the initiator. To avoid unnecessary overhead, a target would request the SCSI Conf message in the SCSI Response message only on a SCSI task ending in an error. Also, SCSI Conf should also play by the rules of the connection allegiance. o I suggest the following new Login/Text keys in section 10 - - Following are for capabilities. InitiatorCapability: TargetCapability: - Following for protocol revision. iSCSIRevision: The responder to a Login command may choose to propose an equal or a lower rev than proposed in the Login command payload. If the counter-proposal in the response is not acceptable, the sender of Login command should immediately log out. If the responder can only support higher revs, login is rejected. - Following, subsequent to the SCSI Conf enhancement request. SCSIConfSupport: --> SCSIConfSupport: From owner-ips@ECE.cmu.edu Thu Jul 27 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16848; 27 Jul 2000 18:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA13546 for ips-outgoing; Thu, 27 Jul 2000 17:30:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA13529 for ; Thu, 27 Jul 2000 17:30:20 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id RAA00448; Thu, 27 Jul 2000 17:30:00 -0400 (EDT) Date: Thu, 27 Jul 2000 17:30:00 -0400 (EDT) From: Scott Bradner Message-Id: <200007272130.RAA00448@newdev.harvard.edu> To: dotis@sanlight.net, ips@ece.cmu.edu Subject: Re: Agenda suggestion Sender: owner-ips@ece.cmu.edu Precedence: bulk I'd rather see the security discussion the VI/TCP proposals is not a working group item at this point and security is one of the key issues for the protocol Scott --- >From owner-ips@ece.cmu.edu Thu Jul 27 16:19:05 2000 X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f From: "Douglas Otis" To: Subject: Agenda suggestion Date: Thu, 27 Jul 2000 13:19:52 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk As the impact of the VI/TCP proposal will have a significant effect on security, perhaps a bit more time should be spent with Constantine Sapuntzakis from Cisco rather than 30 minutes dedicated to Steve Bellovin. This is provided the co-chair is willing... 30 min iSCSI Security Requirements (Steve Bellovin, co-chair) 2.5 min VI/TCP for iSCSI (Constantine Sapuntzakis, Cisco) draft-csapuntz-ips-iscsivi-00.txt Doug From owner-ips@ECE.cmu.edu Thu Jul 27 23:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17017; 27 Jul 2000 19:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA15245 for ips-outgoing; Thu, 27 Jul 2000 18:28:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA15232 for ; Thu, 27 Jul 2000 18:28:09 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Thu, 27 Jul 2000 18:27:37 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FBC1@corpmx9.isus.emc.com> To: dotis@sanlight.net, ips@ece.cmu.edu Subject: RE: Agenda suggestion Date: Thu, 27 Jul 2000 18:27:37 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > As the impact of the VI/TCP proposal will have a significant effect on > security, perhaps a bit more time should be spent with Constantine > Sapuntzakis from Cisco rather than 30 minutes dedicated to Steve Bellovin. > This is provided the co-chair is willing... This co-chair is always willing to listen ... ... but has a few concerns: - Of the things that MUST (cf. RFC 2119 ;-) be done to complete iSCSI, Security is the biggest open area. Starting to do something about it is arguably the most important thing that needs to be achieved for iSCSI in Pittsburgh. - Adoption of VI probably affects the choice/design of security solutions, but the first priority is to get to consensus on security requirements. Does whether to adopt an RDMA framework and/or which one really have a dramatic effect on security requirements? - It is unfortunately the case that the underlying VI/TCP draft that Costa referenced is off-limits for discussion in the meeting due to missing the submission deadline two weeks prior to the meeting. Here's a possibility for creative agenda-bashing, based on the fact that Costa is last on the agenda before a 2-hour break. If the meeting is close to on-time and the consensus in the room is to spend additional time on his draft instead of the break, then the meeting continues into the break for discussion. Think about this, and let's take it up during agenda bashing in Pittsburgh. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Douglas Otis [SMTP:dotis@sanlight.net] > Sent: Thursday, July 27, 2000 4:20 PM > To: ips@ece.cmu.edu > Subject: Agenda suggestion > > As the impact of the VI/TCP proposal will have a significant effect on > security, perhaps a bit more time should be spent with Constantine > Sapuntzakis from Cisco rather than 30 minutes dedicated to Steve Bellovin. > This is provided the co-chair is willing... > > 30 min iSCSI Security Requirements (Steve Bellovin, co-chair) > 2.5 min VI/TCP for iSCSI (Constantine Sapuntzakis, Cisco) > draft-csapuntz-ips-iscsivi-00.txt > > Doug From owner-ips@ECE.cmu.edu Fri Jul 28 02:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17486; 27 Jul 2000 22:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA17804 for ips-outgoing; Thu, 27 Jul 2000 20:33:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA17789 for ; Thu, 27 Jul 2000 20:33:14 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id RAA00516; Thu, 27 Jul 2000 17:31:33 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id RAA15408; Thu, 27 Jul 2000 17:29:09 -0700 From: "Douglas Otis" To: , Subject: RE: Agenda suggestion Date: Thu, 27 Jul 2000 17:35:08 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 In-Reply-To: <0F31E5C394DAD311B60C00E029101A070148FBC1@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Thursday, July 27, 2000 3:28 PM To: dotis@sanlight.net; ips@ece.cmu.edu Subject: RE: Agenda suggestion >- Adoption of VI probably affects the choice/design >of security solutions, but the first priority is to >get to consensus on security requirements. Does >whether to adopt an RDMA framework and/or which one >really have a dramatic effect on security requirements? > >- It is unfortunately the case that the underlying >VI/TCP draft that Costa referenced is off-limits >for discussion in the meeting due to missing the >submission deadline two weeks prior to the meeting. Some type of VI approach such as the RDMA (or VI modeled after Giganet CLAN product) has always been included within iSCSI discussions. I have never heard it excluded, from the I/O Server convention to even most recently by the author, although a contentious point within the specification. I did not partake in the telephone conferences, so I may have been excluded from those decisions to exclude it. Declare fait accompli without knowing the architecture and related security issues? All the more reason to let those with experience speak. Doug From owner-ips@ECE.cmu.edu Fri Jul 28 02:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17481; 27 Jul 2000 22:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA18510 for ips-outgoing; Thu, 27 Jul 2000 21:13:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA18496 for ; Thu, 27 Jul 2000 21:13:09 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Thu, 27 Jul 2000 21:12:38 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FBC4@corpmx9.isus.emc.com> To: dotis@sanlight.net, Black_David@emc.com, ips@ece.cmu.edu Subject: RE: Agenda suggestion Date: Thu, 27 Jul 2000 21:12:37 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > >- It is unfortunately the case that the underlying > >VI/TCP draft that Costa referenced is off-limits > >for discussion in the meeting due to missing the > >submission deadline two weeks prior to the meeting. > > Some type of VI approach such as the RDMA (or VI modeled after Giganet CLAN > product) has always been included within iSCSI discussions. I have never > heard it excluded, from the I/O Server convention to even most recently by > the author, although a contentious point within the specification. I did > not partake in the telephone conferences, so I may have been excluded from > those decisions to exclude it. Declare fait accompli without knowing the > architecture and related security issues? All the more reason to let those > with experience speak. Oh, good grief ... that's simply not true -- no decision has been made to exclude the VI approach, and in fact it is not being excluded. Costa's draft is on the agenda, providing an opportunity to discuss the VI approach in general, and I've even suggested a way to acquire more time to do so than the 2.5 min currently on the agenda. The only thing that is being excluded is discussion of the GigaNet VI/TCP draft *at the Pittsburgh meeting* because it was not submitted by the cutoff date for submissions. This rule is necessary to avoid last minute submissions of important documents in a fashion that makes it impossible for attendees to review them sufficiently in advance. The GigaNet VI/TCP draft can be discussed on the mailing list at any time (although waiting until after the meeting might be a good idea), and at any meeting after Pittsburgh, assuming that it does get submitted in time for such a meeting. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Jul 28 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18587; 28 Jul 2000 3:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA24004 for ips-outgoing; Fri, 28 Jul 2000 01:36:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA23984 for ; Fri, 28 Jul 2000 01:36:04 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA322986 for ; Fri, 28 Jul 2000 07:35:33 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id HAA194838 for ; Fri, 28 Jul 2000 07:35:33 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125692A.001EB498 ; Fri, 28 Jul 2000 07:35:23 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 28 Jul 2000 08:33:54 +0300 Subject: framing & RDMA Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I've placed a memo about the framing and RDMA at: http://www.haifa.il.ibm.com/satran/iSCSI-RDMA-memo.txt As it is meant to get obsolete sooner (I hope) than the draft it refers too I was hesitant about cluttering the ID area with it and I will post it on my home site at IBM and on the document area on the IPS Mailing list archive (if Dave Nagle will help me on that). C U in Pittsburgh Julo From owner-ips@ECE.cmu.edu Fri Jul 28 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21098; 28 Jul 2000 14:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA20934 for ips-outgoing; Fri, 28 Jul 2000 13:27:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraab.compuserve.com (ds-img-rel-2.compuserve.com [149.174.206.155]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA20920 for ; Fri, 28 Jul 2000 13:27:14 -0400 (EDT) Received: (from mailgate@localhost) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id NAA10355 for ips@ece.cmu.edu; Fri, 28 Jul 2000 13:26:44 -0400 (EDT) Received: from compuserve.com (dal-qbu-zol-vty6.as.wcom.net [216.192.244.6]) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id NAA10340 for ; Fri, 28 Jul 2000 13:26:42 -0400 (EDT) Message-ID: <3981C306.E091C316@compuserve.com> Date: Fri, 28 Jul 2000 12:29:43 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: About the idea of layering iSCSI on VI on TCP Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Forgive me if I've read the recent reflector traffic incorrectly, but it appears that the idea is being floated of layering iSCSI on VI on TCP. Based on what I know about the activities of the SCSI standards community as the secretary for T10 and the project editor for SAM-2 and SPC-2 (SCSI Primary Commands) I'm wondering if iSCSI would be needed in such a protocol stack. T10 already has a project to layer SCSI on VI, it's call SVP or SCSI VI Protocol. The project editor for SVP is Ed Gardner (Gardner@acm.org) and the project has a draft standard document available at: ftp://ftp.t10.org/t10/drafts/svp/svp-r01.pdf Recent SVP status report documents are available at: ftp://ftp.t10.org/t10/document.99/99-316r1.pdf ftp://ftp.t10.org/t10/document.00/00-172r0.pdf ftp://ftp.t10.org/t10/document.00/00-240r0.pdf ftp://ftp.t10.org/t10/document.00/00-278r0.pdf This list is ordered by date from older to newer documents. As far as I can tell, should the IPS WG decide to pursue enabling SCSI over IP via definition of a VI protocol for TCP, then that one document would be the only document needed as the protocol stack would be SVP on VI on TCP. Effectively, the proposed use of iSCSI would be replaced by SVP. Thanks. Ralph... P.S. I am aware of the IETF policy to use plain text files and therefore apologize for the pointers to PDF files. However, the T10 policy is to use PDF files so there's no other choice. From owner-ips@ECE.cmu.edu Fri Jul 28 23:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22255; 28 Jul 2000 19:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA02696 for ips-outgoing; Fri, 28 Jul 2000 17:40:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA02672 for ; Fri, 28 Jul 2000 17:40:27 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA00751; Fri, 28 Jul 2000 14:38:32 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA18444; Fri, 28 Jul 2000 14:36:08 -0700 From: "Douglas Otis" To: , "IPS Reflector" Subject: RE: About the idea of layering iSCSI on VI on TCP Date: Fri, 28 Jul 2000 14:42:16 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <3981C306.E091C316@compuserve.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, Thanks for the information. This T10 standard would represent a substantial overhead for Ethernet as it was intended for Infiniband. A VI approach coupled with TCP is the only means iSCSI can exist. It would be futile to grope for headers and double handle memory without some other mechanism in addition to a vanilla TCP, regardless of the desire. The simple proposal presented by Alex Nicolson from Giganet, albeit a bit late, looks far more appropriate than the T10 spec. The iSCSI proposal can not stand on its own. iSCSI requires an extension to the present suite of IP protocols. It would not be appropriate to suggest security belongs to iSCSI because of the need for this IP protocol extension. It should belong to the protocol extension, otherwise you are not creating a standard mechanism. An extension to the protocol that will be placed in hardware as well. For David to discuss getting rid of the design team to make progress underscores his appreciation. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Ralph Weber Sent: Friday, July 28, 2000 10:30 AM To: IPS Reflector Subject: About the idea of layering iSCSI on VI on TCP Forgive me if I've read the recent reflector traffic incorrectly, but it appears that the idea is being floated of layering iSCSI on VI on TCP. Based on what I know about the activities of the SCSI standards community as the secretary for T10 and the project editor for SAM-2 and SPC-2 (SCSI Primary Commands) I'm wondering if iSCSI would be needed in such a protocol stack. T10 already has a project to layer SCSI on VI, it's call SVP or SCSI VI Protocol. The project editor for SVP is Ed Gardner (Gardner@acm.org) and the project has a draft standard document available at: ftp://ftp.t10.org/t10/drafts/svp/svp-r01.pdf Recent SVP status report documents are available at: ftp://ftp.t10.org/t10/document.99/99-316r1.pdf ftp://ftp.t10.org/t10/document.00/00-172r0.pdf ftp://ftp.t10.org/t10/document.00/00-240r0.pdf ftp://ftp.t10.org/t10/document.00/00-278r0.pdf This list is ordered by date from older to newer documents. As far as I can tell, should the IPS WG decide to pursue enabling SCSI over IP via definition of a VI protocol for TCP, then that one document would be the only document needed as the protocol stack would be SVP on VI on TCP. Effectively, the proposed use of iSCSI would be replaced by SVP. Thanks. Ralph... P.S. I am aware of the IETF policy to use plain text files and therefore apologize for the pointers to PDF files. However, the T10 policy is to use PDF files so there's no other choice. From owner-ips@ECE.cmu.edu Mon Jul 31 23:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04795; 31 Jul 2000 19:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA09577 for ips-outgoing; Mon, 31 Jul 2000 17:49:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f219.law10.hotmail.com [64.4.15.219]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id RAA09561 for ; Mon, 31 Jul 2000 17:49:31 -0400 (EDT) Received: (qmail 38902 invoked by uid 0); 31 Jul 2000 21:48:44 -0000 Message-ID: <20000731214844.38901.qmail@hotmail.com> Received: from 147.73.132.189 by www.hotmail.com with HTTP; Mon, 31 Jul 2000 14:48:44 PDT X-Originating-IP: [147.73.132.189] From: "Steve Byan" To: ips@ece.cmu.edu Cc: stephen.byan@quantum.com Subject: Re: multiple TCP connections Date: Mon, 31 Jul 2000 17:48:44 EDT Mime-Version: 1.0 Content-Type: text/plain; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't understand the motivation for creating the concept of a session consisting of multiple TCP connections in the iSCSI draft. If the motiviation is a desire to share the load over multiple physical paths, IIRC IP already provides this in the form of multihoming. I suspect there is some current activity in this area, although I'm not network-literate enough to point to any such work. If the motiviation is to work around performance flaws in the implementation of multihoming in some TCP/IP stacks, then I suggest that it would be better to fix the implementations. If the motiviation is to work around a fundamental performance flaw in IP multihoming, then I suggest that IP should be fixed, or that some other generic solution be engineered. If the motiviation is to make it easier to implement TCP/IP hardware acceleration by pushing the aggregation of multiple paths up into the application protocol, then I suggest that a general solution should be engineered (one that can be applied to multiple protocols), rather than creating a special hack for iSCSI. Is there a rationale for multiple TCP connections that I have missed? If not, which is the claimed rationale? Thanks. Regards, -Steve Steve Byan Design Engineer Quantum Corp. MS 1-3/E23 333 South Street Shrewsbury MA 01545 (508) 770-3414 ________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com From owner-ips@ECE.cmu.edu Tue Aug 01 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05561; 31 Jul 2000 23:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA15665 for ips-outgoing; Mon, 31 Jul 2000 21:56:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraaa.compuserve.com (ds-img-rel-1.compuserve.com [149.174.206.140]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA15649 for ; Mon, 31 Jul 2000 21:56:12 -0400 (EDT) Received: (from mailgate@localhost) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id VAA24240 for ips@ece.cmu.edu; Mon, 31 Jul 2000 21:55:42 -0400 (EDT) Received: from compuserve.com (mid-tgn-nos-vty167.as.wcom.net [216.192.78.167]) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id VAA24213 for ; Mon, 31 Jul 2000 21:55:35 -0400 (EDT) Message-ID: <39863BAE.A5589B41@compuserve.com> Date: Mon, 31 Jul 2000 21:53:35 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: Typos in draft-haagens-ips-iscsireqs-00.txt Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I have noted the following seeming typos in draft-haagens-ips-iscsireqs-00.txt 3.2 Performance/Cost2 ^ ??? ===== [D] See [SAM-2, p. 17.] "The function of the service delivery subsystem is to transport an error-free copy of the request or response between the sender and the receiver..." [SAM-2, p. 22] "The manner in which ordering constraints are established is implementation-specific. An implementation may choose to delegate this responsibility...to the service delivery port. In some cases, in-order delivery may be an intrinsic property of the transport subsystem or a requirement established by the SCSI protocol standard. For convenience, the SCSI architec- ture model assumes in-order delivery to be a property of the service delivery subsystem. This assumption is made to sim- plify the description of behavior and does not constitute a requirement. ^ " From owner-ips@ECE.cmu.edu Tue Aug 01 13:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07983; 1 Aug 2000 9:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA06008 for ips-outgoing; Tue, 1 Aug 2000 08:01:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sphmraaa.compuserve.com (hs-img-rel-1.compuserve.com [149.174.177.156]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA05990 for ; Tue, 1 Aug 2000 08:01:34 -0400 (EDT) Received: (from mailgate@localhost) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id IAA02594 for ips@ece.cmu.edu; Tue, 1 Aug 2000 08:01:03 -0400 (EDT) Received: from compuserve.com (mid-tgn-ngn-vty50.as.wcom.net [216.192.86.50]) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id IAA02543 for ; Tue, 1 Aug 2000 08:01:00 -0400 (EDT) Message-ID: <3986C964.258DD748@compuserve.com> Date: Tue, 01 Aug 2000 07:58:13 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: IPS Requirements discussion points Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Okay, I admit it, I also noticed a few things in the iSCSI (Internet SCSI) Requirements that go beyond the realm of typos. Regarding the last sentence on page 4: At the current time, we do not seek a more generic requirements statement that would justify the choice of TCP (or another protocol) as transport, since the merits of using TCP are readily evident to the working group participants. To me this looks like a boilerplate statement to which the parenthetical phrase "(or another protocol)" was added with the result that it's just plain confusing. If "the merits of using TCP are readily evident" then why would "another protocol" need any mention at all? Regarding the following in 3.3 near the top of page 8: [R] Support SCSI SAM-2 architecture model. [D] It would be helpful to produce a document discussing iSCSI with reference to SAM-2. No promises. Both SPI-3 and FCP-2 contain an annex that relates the protocol to SAM-2. The FCP-2 version of this is undergoing reconstructive surgery as part of the preparations for first public review. However, the finished FCP-2 effort should give iSCSI a substantial leg-up toward meeting the described requirement. Regarding the following in 3.3 near the bottom of page 8: [R] Support for SCSI Task Queuing. [D] SAM-2 defines task queuing, and so strictly speaking, we don't need to call this out specifically. However, task queu- ing is not widely implemented today; and it will increase in importance with WAN IP networks, given speed-of-light delays. We are particularly interested in supporting task queuing of pipelined remote backup and asynchronous disk mirroring [D] Just because iSCSI supports task queuing doesn't mean that the end SCSI node is required to do so also. Task queuing is an optional feature of SCSI. Tapes do not widely implement task queuing (well okay, they practically don't implement it at all). Disks on the other hand are almost universal in their support for task queuing. It is also worth noting that a bridging product that provides task queuing for underlying devices that do not support task queuing has cut itself a chore that is larger than just passing commands and data back and forth. Regarding the following in 3.3 at the bottom of page 8 and top of page 9: [R] Supports all SCSI-3 command sets [SPC-2, SBC, etc.]. There will be no requirement by T10 to modify the SCSI command documents. No modifications are required of the SCSI command layer implementa- tion, except possibly to lengthen task timers to accommodate wide- area delays due to speed-of-light and switching. T10 believes that commands such as EXTENDED COPY and the disk XOR commands require modifications to support target identifiers in the URL form and is working on these modifications. Regarding the following in 3.3 near the top of page 9: [R] Forward compatibility with future revisions of SCSI architec- ture and protocol. Attention to clean layering of protocols. [D] This is a difficult requirement to achieve in practice, since we cannot predict how SCSI will evolve. However, care- ful attention to protocol layering principles will help ensure this result. Actually the problem here is SCSI history. SCSI was developed with an eye toward minimizing implementation costs with much less emphasis on good layering practices. That baggage can be very difficult to shed. For example, the use of mode pages (command layer constructs) to manage link layer features totally ignores layering, but the Disconnect-Reconnect mode page performs just such a function and recently two protocol-specific mode pages were added not so much as an affront to layering common sense but because no other practical mechanism could be found. So there's one of my favorite layering rat holes, let's talk about what's giving you headaches. Regarding the following in 3.4 at the top of page 10: [D] SAM-2 seems to require this channel allegiance: "A task involving one initiator-target pair shall not specify a third SCSI device to participate in transmitting and receiving the remote procedure model elements for that task. Thus, an SMU initiator [e.g., a host computer] shall not create a task using one service delivery port with the expectation that the data transfer or status return for that task would occur via a different service delivery port" [SAM-2, section 4.10.7, p.33]. Of course, interpretation of this clause depends on the definition of service delivery port. If a service delivery port is a TCP connection, then channel allegiance is pretty clearly required. But if a service delivery port is an iSCSI session or an abstract target device, then the interpre- tation of this clause is less clear. Please be aware that the SMU concept will be removed from SAM-2 within the next 6 to 9 months. SMU was a proposal and it has fallen into disfavor with T10. The following diagrams illustrate the conceptual transition that T10 is in the process of building/modifying definitions to support: Past: +------------+ | Initiators | +--------+ +----------+ | A1, A2,... |----| Port A |----| | +------------| +--------+ | |+-----------+ | || | | Target 0 || LUN 0 | | || | +------------+ +--------+ | |+-----------+ | Initiators |----| Port B |----| | | B1, B2,... | +--------+ +----------+ +------------+ Probable future: +------------+ | Initiators | +--------+----------+ | A1, A2,... |----| Port A | Target A |--+ +------------+ +--------+----------+ | +-----------+ | | | +--| LUN 0 | | | | +------------+ +--------+----------+ | +-----------+ | Initiators |----| Port B | Target B |--+ | B1, B2,... | +--------+----------+ +------------+ Also, it looks to me like the multiple connections issue may be one that IPS can solve on its own using an internal T10 agreement as a guideline. >>From the point of view of the SCSI Architecture Model, more than one connection may be used to implement a service delivery port provided the modeled capabilities of the port are maintained. In Fibre Channel talk, the use of 'hunt groups' is permissible at or below FCP-2 and outside the scope of SAM. Should IPS wish to revisit this perspective, please be aware that the internal T10 agreement is not cast in stone. Regarding the following in 3.4 at the bottom of page 10: [R] Command striping (load balancing) across multiple host and dev- ice interfaces. It shall be possible to utilize multiple con- current paths between hosts and devices for the purpose of load balancing. [D] Load balancing refers to concurrent tasks from a single initiator. There is no ordering constraint among these tasks. We aim to distribute these tasks (commands and their related data and status) across multiple host ports, links, switch ports and device ports, in order to achieve aggregate perfor- mance equal to a multiple of single link performance. The statement 'There is no ordering constraint among these tasks.' assumes that all tasks carry the SIMPLE queuing attribute. While this is consistent with the normal case queuing mindset, it is a very restrictive assumption. In regards to this and a couple other statements related to multiple connections, it may be productive for IPS and T10 to discuss the architectural meaning of terms like initiator, target, and port realizing that the T10 definitions of these terms are in a state of flux at the moment because T10 is trying to develop an acceptable model for devices with multiple ports. Regarding the following in section 4 at the top of page 21: [CAM-3] ANSI NCITS. Dallas, William D., editor. Information Tech- nology - Common Access Method - 3 (CAM-3)). X3T10 Project 990D. rev 3, 16 Mar 1998. T10 has withdrawn the CAM-3 project. No further work will be done and no standard will be produced. I think the closest equivalent IETF classification of the last CAM-3 draft would have to be 'Historic'. From owner-ips@ECE.cmu.edu Tue Aug 01 15:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08313; 1 Aug 2000 10:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA09268 for ips-outgoing; Tue, 1 Aug 2000 09:55:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ertpg14e1.nortelnetworks.com (ertpg14e1.nortelnetworks.com [47.234.0.35]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA09230 for ; Tue, 1 Aug 2000 09:54:45 -0400 (EDT) Received: from zsc4c002.corpwest.baynetworks.com by ertpg14e1.nortelnetworks.com; Tue, 1 Aug 2000 09:51:23 -0400 Received: from zsc4c006.corpwest.baynetworks.com ([134.177.2.153]) by zsc4c002.corpwest.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id QBFJ6XD1; Tue, 1 Aug 2000 06:51:19 -0700 Received: from long-pc.ca.nortel.com (LONG-PC [47.129.50.26]) by zsc4c006.corpwest.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id NW28N59D; Tue, 1 Aug 2000 06:50:29 -0700 Message-Id: <4.2.2.20000801064615.00bdd300@zsc4c006.corpwest.baynetworks.com> X-Sender: long@zsc4c006.corpwest.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 01 Aug 2000 06:51:04 -0700 To: Steve Byan , ips@ece.cmu.edu From: "Lyndon Ong" Subject: Re: multiple TCP connections Cc: stephen.byan@quantum.com In-Reply-To: <20000731214844.38901.qmail@hotmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk We recently completed work on a transport protocol with multi-homing support, plus support for multiple information streams within one session. If this might be of interest (it was built for transport of signaling and control messaging) folks can find it at http://www.ietf.org/internet-drafts/draft-ietf-sigtran-sctp-13.txt . It was recently approved as a Proposed Standard and is on the RFC editor's queue. Cheers, L. Ong Nortel Networks At 05:48 PM 7/31/2000 -0400, Steve Byan wrote: >I don't understand the motivation for creating the concept of a session >consisting of multiple TCP connections in the iSCSI draft. > >If the motiviation is a desire to share the load over multiple physical >paths, IIRC IP already provides this in the form of multihoming. I >suspect there is some current activity in this area, although I'm not >network-literate enough to point to any such work. > >If the motiviation is to work around performance flaws in the >implementation of multihoming in some TCP/IP stacks, then I suggest that >it would be better to fix the implementations. > >If the motiviation is to work around a fundamental performance flaw in >IP multihoming, then I suggest that IP should be fixed, or that some >other generic solution be engineered. > >If the motiviation is to make it easier to implement TCP/IP hardware >acceleration by pushing the aggregation of multiple paths up into the >application protocol, then I suggest that a general solution should be >engineered (one that can be applied to multiple protocols), rather than >creating a special hack for iSCSI. > >Is there a rationale for multiple TCP connections that I have missed? > >If not, which is the claimed rationale? > >Thanks. > >Regards, >-Steve > >Steve Byan >Design Engineer >Quantum Corp. >MS 1-3/E23 >333 South Street >Shrewsbury MA 01545 >(508) 770-3414 >________________________________________________________________________ >Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com Nortel Networks 4401 Great America Parkway Santa Clara, CA 95052-8185 Tel. +1 408 495 5072 Fax. +1 408 495 1699 From owner-ips@ECE.cmu.edu Tue Aug 01 15:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08318; 1 Aug 2000 10:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA09266 for ips-outgoing; Tue, 1 Aug 2000 09:55:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f310.law10.hotmail.com [64.4.14.185]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id JAA09246 for ; Tue, 1 Aug 2000 09:54:54 -0400 (EDT) Received: (qmail 62282 invoked by uid 0); 1 Aug 2000 13:54:08 -0000 Message-ID: <20000801135408.62281.qmail@hotmail.com> Received: from 147.73.128.184 by www.hotmail.com with HTTP; Tue, 01 Aug 2000 06:54:08 PDT X-Originating-IP: [147.73.128.184] From: "Steve Byan" To: ips@ece.cmu.edu Subject: Re: multiple TCP connections Date: Tue, 01 Aug 2000 09:54:08 EDT Mime-Version: 1.0 Content-Type: text/plain; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo wrote: > >What standard are you refering to? I wrote the post in reference to draft-satran-iscsi-01.txt. I'm no network expert, I'm just trying to understand why what I read about IP multihoming isn't sufficient to provide bandwidth aggregation (if that in fact is the motivation for multiple sessions). Regards, -Steve Steve Byan ________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com From owner-ips@ECE.cmu.edu Tue Aug 01 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08514; 1 Aug 2000 11:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA11079 for ips-outgoing; Tue, 1 Aug 2000 10:39:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA11063 for ; Tue, 1 Aug 2000 10:39:14 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Tue, 1 Aug 2000 10:38:42 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070148FBE6@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: Congestion Control Date: Tue, 1 Aug 2000 10:38:32 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk I've occasionally had people ask me whether link and session/connection flow control is/are adequate for IP networks. The answer is *no* - congestion control is necessary over and above basic flow control measures. This is one of the reasons why iSCSI employs TCP. Further explanation can be found in the following document which will appear shortly as a Best Current Practice (BCP) RFC. http://www.ietf.org/internet-drafts/draft-floyd-cong-04.txt Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Aug 01 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08954; 1 Aug 2000 14:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16158 for ips-outgoing; Tue, 1 Aug 2000 12:46:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lsi.lsil.com (lsi.lsil.com [147.145.40.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16142 for ; Tue, 1 Aug 2000 12:46:28 -0400 (EDT) Received: from mhbs.lsil.com ([147.145.31.100]) by lsi.lsil.com (8.9.3+Sun/8.9.1) with ESMTP id JAA09859 for ; Tue, 1 Aug 2000 09:46:26 -0700 (PDT) Received: from inca.co.lsil.com by mhbs.lsil.com with ESMTP for ips@ece.cmu.edu; Tue, 1 Aug 2000 09:46:12 -0700 Received: from exc-colorado.co.lsil.com (exccolorado.co.lsil.com [153.72.104.182]) by inca.co.lsil.com (8.9.3/8.9.3) with ESMTP id KAA02866 for ; Tue, 1 Aug 2000 10:46:11 -0600 (MDT) Received: by exccolorado.co.lsil.com with Internet Mail Service (5.5.2650.21) id <307ARQGZ>; Tue, 1 Aug 2000 10:48:40 -0600 Message-Id: <2ADA45B8C19AD111BBEA00805FBE134102E6CD6D@exccolorado.co.lsil.com> From: "Bryant, Kelly" To: ips@ece.cmu.edu Subject: unsubscribe Date: Tue, 1 Aug 2000 10:48:39 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Unsubscribe From owner-ips@ECE.cmu.edu Tue Aug 01 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08959; 1 Aug 2000 14:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16010 for ips-outgoing; Tue, 1 Aug 2000 12:43:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA15992 for ; Tue, 1 Aug 2000 12:42:54 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0FYM003ARGYNNC@mta6.snfc21.pbi.net> for IPS@ECE.CMU.edu; Tue, 1 Aug 2000 09:39:59 -0700 (PDT) Date: Tue, 01 Aug 2000 09:31:30 -0700 From: Peter Johansson Subject: Re: IPS Requirements discussion points In-reply-to: <3986C964.258DD748@compuserve.com> X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000801092914.00bd1310@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:58 AM 8/1/00, Ralph Weber wrote: >Both SPI-3 and FCP-2 contain an annex that relates the protocol to SAM-2. >The FCP-2 version of this is undergoing reconstructive surgery as part of >the preparations for first public review. However, the finished FCP-2 >effort should give iSCSI a substantial leg-up toward meeting the described >requirement. Annex B of ANSI NCITS 325-1998, Serial Bus Protocol 2, is yet another example of language and organization that works to map a transport layer's services to the requirements of SAM. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Wed Aug 02 13:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12931; 2 Aug 2000 9:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA07603 for ips-outgoing; Wed, 2 Aug 2000 07:39:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA07590 for ; Wed, 2 Aug 2000 07:39:33 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA222246; Wed, 2 Aug 2000 13:38:55 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id NAA63870; Wed, 2 Aug 2000 13:38:56 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125692F.003FF8F8 ; Wed, 2 Aug 2000 13:38:45 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: "Lyndon Ong" cc: ips@ece.cmu.edu Message-ID: Date: Wed, 2 Aug 2000 14:36:56 +0300 Subject: Re: multiple TCP connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Lyndon, Several of us looked through it. Very good job. And in many respects close to what we need. But not close enough. We recognize that your objectives where quite different from ours - ours are simpler as far as functions goes (e.g., we don't demand out of order processing only out of order DMA not to have data pile up in adapters) but far more stringent performance requirements. Nevertheless we will continue watching what you do both for ideas and guidance, to influence it where we can and perhaps use sctp or some light weight version as it gets widespread. Regards, Julo Please respond to "Lyndon Ong" To: Steve Byan , ips@ece.cmu.edu cc: stephen.byan@quantum.com (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple TCP connections We recently completed work on a transport protocol with multi-homing support, plus support for multiple information streams within one session. If this might be of interest (it was built for transport of signaling and control messaging) folks can find it at http://www.ietf.org/internet-drafts/draft-ietf-sigtran-sctp-13.txt . It was recently approved as a Proposed Standard and is on the RFC editor's queue. Cheers, L. Ong Nortel Networks At 05:48 PM 7/31/2000 -0400, Steve Byan wrote: >I don't understand the motivation for creating the concept of a session >consisting of multiple TCP connections in the iSCSI draft. > >If the motiviation is a desire to share the load over multiple physical >paths, IIRC IP already provides this in the form of multihoming. I >suspect there is some current activity in this area, although I'm not >network-literate enough to point to any such work. > >If the motiviation is to work around performance flaws in the >implementation of multihoming in some TCP/IP stacks, then I suggest that >it would be better to fix the implementations. > >If the motiviation is to work around a fundamental performance flaw in >IP multihoming, then I suggest that IP should be fixed, or that some >other generic solution be engineered. > >If the motiviation is to make it easier to implement TCP/IP hardware >acceleration by pushing the aggregation of multiple paths up into the >application protocol, then I suggest that a general solution should be >engineered (one that can be applied to multiple protocols), rather than >creating a special hack for iSCSI. > >Is there a rationale for multiple TCP connections that I have missed? > >If not, which is the claimed rationale? > >Thanks. > >Regards, >-Steve > >Steve Byan >Design Engineer >Quantum Corp. >MS 1-3/E23 >333 South Street >Shrewsbury MA 01545 >(508) 770-3414 >________________________________________________________________________ >Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com Nortel Networks 4401 Great America Parkway Santa Clara, CA 95052-8185 Tel. +1 408 495 5072 Fax. +1 408 495 1699 From owner-ips@ECE.cmu.edu Wed Aug 02 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13351; 2 Aug 2000 11:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA10281 for ips-outgoing; Wed, 2 Aug 2000 09:54:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f320.law10.hotmail.com [64.4.14.195]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id JAA10268 for ; Wed, 2 Aug 2000 09:54:40 -0400 (EDT) Received: (qmail 36097 invoked by uid 0); 2 Aug 2000 13:54:09 -0000 Message-ID: <20000802135409.36096.qmail@hotmail.com> Received: from 147.73.132.189 by www.hotmail.com with HTTP; Wed, 02 Aug 2000 06:54:09 PDT X-Originating-IP: [147.73.132.189] From: "Steve Byan" To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: multiple TCP connections Date: Wed, 02 Aug 2000 09:54:09 EDT Mime-Version: 1.0 Content-Type: text/plain; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo wrote: >There is no standard way of doing link agregation. That's the main reason >for having >the session (+reliability). Being ignorant of networking practices, it seems to me that link aggregation is more appropriately solved as a routing problem than as part of the application protocol. IP supports multiple links per end-point. If an end-point has two links, then it must have at least two routes to it. From there it's just a question of getting the routing to load-balance between the routes. What am I missing? Is it that the currently-available TCP/IP implementations and routers don't do load-balancing on links, due to problems introduced by out-of-order arrival? I also don't understand the point about reliability. A multi-homed host already has hardware redundancy, and even if today's commercially-available routers and TCP/IP stacks don't do load balancing they surely understand how to use a different route in the event of link failure. Is the claim of reliability based on just having multiple links, or is it based on using the command reference numbers to recover a failed TCP session? Is there work going on elsewhere in the IETF with respect to link aggregation? Thanks. Regards, -Steve Steve Byan ________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com From owner-ips@ECE.cmu.edu Wed Aug 02 19:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14120; 2 Aug 2000 15:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA16535 for ips-outgoing; Wed, 2 Aug 2000 13:41:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (root@wireless-132-188.ietf.marconi.com [147.73.132.188]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA16521 for ; Wed, 2 Aug 2000 13:41:48 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id JAA00675; Thu, 3 Aug 2000 09:42:10 -0700 To: "Lyndon Ong" Cc: Steve Byan , ips@ece.cmu.edu, stephen.byan@quantum.com Subject: Re: multiple TCP connections References: <4.2.2.20000801064615.00bdd300@zsc4c006.corpwest.baynetworks.com> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 03 Aug 2000 09:42:10 -0700 In-Reply-To: "Lyndon Ong"'s message of "Tue, 01 Aug 2000 06:51:04 -0700" Message-ID: Lines: 18 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk "Lyndon Ong" writes: > We recently completed work on a transport protocol with multi-homing > support, plus support for multiple information streams within one > session. If this might be of interest (it was built for transport of > signaling and control messaging) folks can find it at > http://www.ietf.org/internet-drafts/draft-ietf-sigtran-sctp-13.txt . It > was recently approved as a Proposed Standard and is on the RFC editor's queue. One of the features of the iSCSI is that a session that involves multiple connections can be spread across multiple iSCSI accelerators cards with very little need for co-ordination between the cards. My impression was that this would be more difficult with SCTP, but I'm open to correction here. -Costa From owner-ips@ECE.cmu.edu Wed Aug 02 19:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14155; 2 Aug 2000 15:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA17572 for ips-outgoing; Wed, 2 Aug 2000 14:13:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtpmast01.marconi.com ([169.144.68.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA17548 for ; Wed, 2 Aug 2000 14:13:35 -0400 (EDT) Received: from littlejoy ([147.73.132.110]) by smtpmast01.marconi.com (Lotus Domino Release 5.0.3 (Intl)) with SMTP id 2000080214130340:22209 ; Wed, 2 Aug 2000 14:13:03 -0400 From: "Douglas Otis" To: Subject: Re: multiple TCP connections Date: Wed, 2 Aug 2000 11:15:05 -0700 Message-ID: MIME-Version: 1.0 X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 X-MIMETrack: Itemize by SMTP Server on SMTPMAST01/MARCONI(Release 5.0.3 (Intl)|21 March 2000) at 02/08/2000 02:13:03 PM, Serialize by Router on SMTPMAST01/MARCONI(Release 5.0.3 (Intl)|21 March 2000) at 02/08/2000 02:13:34 PM, Serialize complete at 02/08/2000 02:13:34 PM Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, Between the router and an intelligent switch, the packets are encapsulated to allow load balancing and multi-paths. The internal address of the packet does not change for this encapsulation that responds to failure at 10 second intervals typically. This would be true for any protocol other than iSCSI. The major problem with the charter of IPS are assumptions aimed at providing an OEM specification to a specific product providing a bridge between Fibre-Channel and 1 Gbit Ethernet to exclude these reliability practices. As most high-end disk drives can only handle about 150 operations per second, the throughput on the drive interface is largely dependent on the size of these transactions. In most SAN environments, the half-duplex data rate at the drive is less that 10 MB/sec. (Still possible within Fast Ethernet rates which provide up to 25 MB/sec. full duplex) Improvements to the operations per second figure require increases in power to the square of the improvement. With co-location sites running into the tens of mega-watts, a reduction in operations per second could have a significant power benefit together with the use of Ethernet switches at providing the actual reliable switching fabric rather than Fibre-Channel or Parallel SCSI. Second, because of the IPS direction being aimed at the Fibre-Channel bridge, security issues that result in this approach have been over-looked together with the reliability issue mentioned. Rather than mapping a process to a port nexus, process IDs are buried somewhere within the TCP stream. The overhead of the Fibre-Channel bridge (iSCSI) approach will overwhelm legacy equipment with this buried critical data and preclude third-party verification. The iSCSI headers are free-form and non-aligned. The redundant data-handling resulting from this simplistic approach will ensure protocols using an aligned structure will well out perform the iSCSI standard and ensure a short life for this iSCSI product. The iSCSI architecture is based on Fibre-Channel and not that offered by an IP networked device. Doug Julo wrote: >There is no standard way of doing link agregation. That's the main reason >for having >the session (+reliability). Being ignorant of networking practices, it seems to me that link aggregation is more appropriately solved as a routing problem than as part of the application protocol. IP supports multiple links per end-point. If an end-point has two links, then it must have at least two routes to it. From there it's just a question of getting the routing to load-balance between the routes. What am I missing? Is it that the currently-available TCP/IP implementations and routers don't do load-balancing on links, due to problems introduced by out-of-order arrival? I also don't understand the point about reliability. A multi-homed host already has hardware redundancy, and even if today's commercially-available routers and TCP/IP stacks don't do load balancing they surely understand how to use a different route in the event of link failure. Is the claim of reliability based on just having multiple links, or is it based on using the command reference numbers to recover a failed TCP session? Is there work going on elsewhere in the IETF with respect to link aggregation? Thanks. Regards, -Steve Steve Byan From owner-ips@ECE.cmu.edu Wed Aug 02 19:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14111; 2 Aug 2000 15:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA16438 for ips-outgoing; Wed, 2 Aug 2000 13:39:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (root@wireless-132-188.ietf.marconi.com [147.73.132.188]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA16425 for ; Wed, 2 Aug 2000 13:38:57 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id JAA00670; Thu, 3 Aug 2000 09:38:24 -0700 To: "Steve Byan" Cc: ips@ece.cmu.edu, stephen.byan@quantum.com Subject: Re: multiple TCP connections References: <20000731214844.38901.qmail@hotmail.com> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 03 Aug 2000 09:38:24 -0700 In-Reply-To: "Steve Byan"'s message of "Mon, 31 Jul 2000 17:48:44 EDT" Message-ID: Lines: 62 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Steve, First off, I'm not sure what you mean by IP mulithoming. To an extent, the draft supports IP multihoming, in that: - a multihomed host has multiple IP addresses - each TCP connection binds to only one IP address - howver, iSCSI supports using multiple TCP connections for a single iSCSI sessions. - thus, iSCSI supports multihoming. One advantage of using multiple TCP connections is that you can have multiple independent TCP and DMA engines operating in parallel, each at a slower speed (8 TCP engines at 1 gigabit instead of one TCP enginea t 8 gigabits). -Costa "Steve Byan" writes: > I don't understand the motivation for creating the concept of a session > consisting of multiple TCP connections in the iSCSI draft. > > If the motiviation is a desire to share the load over multiple physical > paths, IIRC IP already provides this in the form of multihoming. I > suspect there is some current activity in this area, although I'm not > network-literate enough to point to any such work. > > If the motiviation is to work around performance flaws in the > implementation of multihoming in some TCP/IP stacks, then I suggest that > it would be better to fix the implementations. > > If the motiviation is to work around a fundamental performance flaw in > IP multihoming, then I suggest that IP should be fixed, or that some > other generic solution be engineered. > > If the motiviation is to make it easier to implement TCP/IP hardware > acceleration by pushing the aggregation of multiple paths up into the > application protocol, then I suggest that a general solution should be > engineered (one that can be applied to multiple protocols), rather than > creating a special hack for iSCSI. > > Is there a rationale for multiple TCP connections that I have missed? > > If not, which is the claimed rationale? > > Thanks. > > Regards, > -Steve > > Steve Byan > Design Engineer > Quantum Corp. > MS 1-3/E23 > 333 South Street > Shrewsbury MA 01545 > (508) 770-3414 > ________________________________________________________________________ > Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com From owner-ips@ECE.cmu.edu Wed Aug 02 20:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14424; 2 Aug 2000 16:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA19580 for ips-outgoing; Wed, 2 Aug 2000 15:15:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hotmail.com (f96.law10.hotmail.com [64.4.15.96]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id PAA19563 for ; Wed, 2 Aug 2000 15:15:10 -0400 (EDT) Received: (qmail 31751 invoked by uid 0); 2 Aug 2000 19:14:22 -0000 Message-ID: <20000802191422.31750.qmail@hotmail.com> Received: from 147.73.132.189 by www.hotmail.com with HTTP; Wed, 02 Aug 2000 12:14:22 PDT X-Originating-IP: [147.73.132.189] From: "Steve Byan" To: ips@ece.cmu.edu Subject: Re: multiple TCP connections Date: Wed, 02 Aug 2000 15:14:22 EDT Mime-Version: 1.0 Content-Type: text/plain; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry, I didn't understand TCP multihoming. I've been reading RFC 1122 and RFC 1123 and discovered that TCP binds to exactly one of the local multihomed addresses, and that TCP applications must choose exactly one of the multihomed addresses for the remote host. Multiple sessions starts to make much more sense. I noticed that there is a standard for ethernet link aggregation (IEEE 802.3ad), and that routers are available that will load-balance over a set of links. Given these two, it seems to me that multiple TCP connections aren't needed for speed or reliability, assuming ethernet links at the end-points. Am I still missing something? The argument that makes sense to me is that of parallelizing the TCP load across multiple hardware accelerators. Thanks. Regards, -Steve Steve Byan ________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com From owner-ips@ECE.cmu.edu Thu Aug 03 03:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15773; 2 Aug 2000 23:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA29490 for ips-outgoing; Wed, 2 Aug 2000 22:11:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA29477 for ; Wed, 2 Aug 2000 22:11:12 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id EAA156892 for ; Thu, 3 Aug 2000 04:10:40 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id EAA95624 for ; Thu, 3 Aug 2000 04:10:37 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256930.000BF340 ; Thu, 3 Aug 2000 04:10:31 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 3 Aug 2000 05:08:51 +0300 Subject: Re: multiple TCP connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, Ethernet links are fine between switches and end-points that do this multilink. On other media (WAN) this is not done. Also on Ethernet when you have several NICS host Ethernet drivers have to do the "striping" (and most don't do it). Regards, Julo "Steve Byan" on 02/08/2000 22:14:22 Please respond to "Steve Byan" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple TCP connections Sorry, I didn't understand TCP multihoming. I've been reading RFC 1122 and RFC 1123 and discovered that TCP binds to exactly one of the local multihomed addresses, and that TCP applications must choose exactly one of the multihomed addresses for the remote host. Multiple sessions starts to make much more sense. I noticed that there is a standard for ethernet link aggregation (IEEE 802.3ad), and that routers are available that will load-balance over a set of links. Given these two, it seems to me that multiple TCP connections aren't needed for speed or reliability, assuming ethernet links at the end-points. Am I still missing something? The argument that makes sense to me is that of parallelizing the TCP load across multiple hardware accelerators. Thanks. Regards, -Steve Steve Byan ________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com From owner-ips@ECE.cmu.edu Thu Aug 03 12:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17771; 3 Aug 2000 8:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA13939 for ips-outgoing; Thu, 3 Aug 2000 06:18:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA13926 for ; Thu, 3 Aug 2000 06:18:41 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id MAA102760 for ; Thu, 3 Aug 2000 12:18:09 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id MAA199298 for ; Thu, 3 Aug 2000 12:18:09 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256930.00389711 ; Thu, 3 Aug 2000 12:18:06 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 3 Aug 2000 13:16:23 +0300 Subject: IETF48 - foils available Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The slides I presented at IETF-48 are available at: htpp://www.haifa.il.ibm.com/satran/IETF48-iSCSI.ps Hopefully they will appear on IPS site (with Dave Nagle's kind help). Regards, Julo From owner-ips@ECE.cmu.edu Thu Aug 03 16:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18672; 3 Aug 2000 12:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA20079 for ips-outgoing; Thu, 3 Aug 2000 11:20:50 -0400 (EDT) Received: from pilots.postech.ac.kr ([141.223.121.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA20063 for ; Thu, 3 Aug 2000 11:20:41 -0400 (EDT) Received: (from kotaeji@localhost) by pilots.postech.ac.kr (8.9.3/8.9.3) id AAA73981; Fri, 4 Aug 2000 00:18:12 +0900 (KST) (envelope-from kotaeji@pilots.postech.ac.kr) Date: Fri, 4 Aug 2000 00:18:12 +0900 (KST) Message-Id: <200008031518.AAA73981@pilots.postech.ac.kr> X-Authentication-Warning: pilots.postech.ac.kr: kotaeji set sender to kotaeji@pilots.postech.ac.kr using -f From: Kim Tae Hyung To: ips@ece.cmu.edu Subject: Several Questions. Reply-to: kotaeji@postech.ac.kr Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, all. I am a graduate student who is very interested in iSCSI. Although I feel very difficult, I am trying to follow up the articles in this mailing list. Reading the articles in this mailing list nowadays, I cannot fully understand every new issue. By the way, with my amateur's eyes, some very basic questions have made me write this article. The first question is about the SCSI protocol itself. iSCSI is just a mapping the SCSI protocol to TCP. But looking in details, I think that it is just the mapping of CDB. After removing the unnecessary parts of SCSI protocol due to the change of underlying physical media from bus to network, I think just CDBs remain, eventually. Then how can we map the operation sequence of SCSI protocol to that of iSCSI ? I wonder whether we can call this just mapping of CDB to SCSI protocol. I think CDB is not the everything of SCSI. The second question is the distinct differences between NFS and iSCSI. According to "draft-von-ipsissues-01.txt" file, there are the following paragraph. 4.2.4 NFS "NFS is the current solution for accessing storage over the network. Its strengths are ease-of-use, file sharing, and security. The general assumption is that a block storage protocol should outperform NFS in some applications, but that assumption needs to be tested. Also, some applications (e.g. tape library) could benefit from a block storage protocol. In order for a block-based solution to be accepted, clear benefits over NFS (e.g. performance, direct access) must be proven, Even then, NFS would still be the solution of choice for those requiring ease-of-use and file sharing." 6. Conclusion - Any block storage protocol must show clear differentiation from NFS. I want to hear many opinions about the distinct differences between the file protocol such as NFS and iSCSI. Thanks in advance. From owner-ips@ECE.cmu.edu Thu Aug 03 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19484; 3 Aug 2000 16:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA29382 for ips-outgoing; Thu, 3 Aug 2000 15:25:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.5]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA29360 for ; Thu, 3 Aug 2000 15:25:38 -0400 (EDT) Received: by apollo.pirus.com with Internet Mail Service (5.5.2650.21) id ; Thu, 3 Aug 2000 15:35:20 -0400 Message-ID: <618471D08ABDD31188B6009027E500930B9BF3@apollo.pirus.com> From: "Merhar, Milan" To: "'Steve Byan'" , ips@ece.cmu.edu Subject: RE: multiple TCP connections Date: Thu, 3 Aug 2000 15:35:19 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, The problem with relying on Layer 2 link aggregation such as 802.1ad, is that it protects against packet reordering by keeping all traffic between a given source/destination pair on a single pipe of the multi-link. So, unless you have multiple attachments at each end (with multiple MAC address and multiple IP addresses,) you're limited to a maximum of (in the case of TCP/IP and Gigabit Ethernet) around 920 Mbps of aggregate traffic between two end points. And, although 900+ Mbps seems to be plenty of bandwidth for any single storage device, concerns have been raised that it may not be sufficient for the case, for example, of a storage array controller talking to a mainframe. This, in fact, had been presented as one rationale for spawning multiple TCP sessions within a single iSCSI session. Each TCP could potentially have separate IP and MAC end point addresses, allowing them to be carried in parallel through 802.1ad inter-switch links. (Although, I agree with you that being able to take advantage of multiple protocol accelerators is in itself a pretty good justification.) On the other hand, It may be a close race between introduction of Storage-over-IP devices that can take advantage of multiple parallel Gigabit Ethernet links, and initial availablity of 10 Gigabit Ethernet LANs. We might find ourselves having designed a clever, but more complex, way of accessing greater bandwidth, which is superceded by the brute-force solution of faster transport technology. - milan > -----Original Message----- > From: Steve Byan [mailto:steve_byan@hotmail.com] > Sent: Wednesday, August 02, 2000 3:14 PM > To: ips@ece.cmu.edu > Subject: Re: multiple TCP connections > > > Sorry, I didn't understand TCP multihoming. I've been reading > RFC 1122 and > RFC 1123 and discovered that TCP binds to exactly one of the local > multihomed addresses, and that TCP applications must choose > exactly one of > the multihomed addresses for the remote host. Multiple > sessions starts to > make much more sense. > > I noticed that there is a standard for ethernet link > aggregation (IEEE > 802.3ad), and that routers are available that will > load-balance over a set > of links. Given these two, it seems to me that multiple TCP > connections > aren't needed for speed or reliability, assuming ethernet > links at the > end-points. Am I still missing something? > > The argument that makes sense to me is that of parallelizing > the TCP load > across multiple hardware accelerators. > > Thanks. > > Regards, > -Steve > > Steve Byan > > ______________________________________________________________ > __________ > Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com From owner-ips@ECE.cmu.edu Thu Aug 03 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19497; 3 Aug 2000 17:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA29404 for ips-outgoing; Thu, 3 Aug 2000 15:26:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA29377 for ; Thu, 3 Aug 2000 15:25:49 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA01130; Thu, 3 Aug 2000 12:23:05 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA05658; Thu, 3 Aug 2000 12:20:28 -0700 From: "Douglas Otis" To: , Subject: RE: Several Questions. Date: Thu, 3 Aug 2000 12:27:46 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 In-Reply-To: <200008031518.AAA73981@pilots.postech.ac.kr> Sender: owner-ips@ece.cmu.edu Precedence: bulk The simple distinction is direct block retrieval versus a meta-reference. With memory costing 100 times more than hard storage, remote referencing in a file system doubles cost of storage together with an overhead not desired in many applications using their own abstractions other than name/extent/permission. Memory should be kept close to the cpu for best utilization and not at the storage device. In the end, for reliability, even a NFS server would have a block interface exposed to a redundant server. With this interface running over IP, the redundant server need not be in the same facility. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Kim Tae Hyung Sent: Thursday, August 03, 2000 8:18 AM To: ips@ece.cmu.edu Subject: Several Questions. Hi, all. I am a graduate student who is very interested in iSCSI. Although I feel very difficult, I am trying to follow up the articles in this mailing list. Reading the articles in this mailing list nowadays, I cannot fully understand every new issue. By the way, with my amateur's eyes, some very basic questions have made me write this article. The first question is about the SCSI protocol itself. iSCSI is just a mapping the SCSI protocol to TCP. But looking in details, I think that it is just the mapping of CDB. After removing the unnecessary parts of SCSI protocol due to the change of underlying physical media from bus to network, I think just CDBs remain, eventually. Then how can we map the operation sequence of SCSI protocol to that of iSCSI ? I wonder whether we can call this just mapping of CDB to SCSI protocol. I think CDB is not the everything of SCSI. The second question is the distinct differences between NFS and iSCSI. According to "draft-von-ipsissues-01.txt" file, there are the following paragraph. 4.2.4 NFS "NFS is the current solution for accessing storage over the network. Its strengths are ease-of-use, file sharing, and security. The general assumption is that a block storage protocol should outperform NFS in some applications, but that assumption needs to be tested. Also, some applications (e.g. tape library) could benefit from a block storage protocol. In order for a block-based solution to be accepted, clear benefits over NFS (e.g. performance, direct access) must be proven, Even then, NFS would still be the solution of choice for those requiring ease-of-use and file sharing." 6. Conclusion - Any block storage protocol must show clear differentiation from NFS. I want to hear many opinions about the distinct differences between the file protocol such as NFS and iSCSI. Thanks in advance. From owner-ips@ECE.cmu.edu Fri Aug 04 00:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20103; 3 Aug 2000 20:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA09041 for ips-outgoing; Thu, 3 Aug 2000 18:17:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA09025 for ; Thu, 3 Aug 2000 18:17:29 -0400 (EDT) From: pat_thaler@agilent.com Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id E370BDF; Thu, 3 Aug 2000 16:17:28 -0600 (MDT) Received: from axatlbh3.atl.hp.com (axatlbh3.atl.hp.com [15.47.113.38]) by andom1.an.hp.com (Postfix) with SMTP id 62E92104; Thu, 3 Aug 2000 18:16:49 -0400 (EDT) Received: from 15.47.113.38 by axatlbh3.atl.hp.com (InterScan E-Mail VirusWall NT); Thu, 03 Aug 2000 18:16:23 -0400 (Eastern Daylight Time) Received: by axatlbh3.atl.hp.com with Internet Mail Service (5.5.2650.21) id ; Thu, 3 Aug 2000 18:16:23 -0400 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0025156AC@axcs04.cs.itc.hp.com> To: mmerhar@pirus.com, steve_byan@hotmail.com, ips@ece.cmu.edu Subject: RE: multiple TCP connections Date: Thu, 3 Aug 2000 18:16:22 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Milan, Perhaps you have not read 802.3ad (the link aggregation standard is part of the 802.3 standard rather than .1). It does not specify that "all traffic between a given source/ destination" stays on one path. What it says is: Frame ordering must be maintained for certain sequences of frame exchanges between MAC Clients (known as conversations, see 1.4). The Distributor ensures that all frames of a given conversation are passed to a single port. For any given port, the Collector is required to pass frames to the MAC Client in the order that they are received from that port. The Collector is otherwise free to select frames received from the aggregated ports in any order. Since there is no means for frames to be misordered on a single link, this guarantees that frame ordering is maintained for any conversation. Implementations are free to use any method they desire for determining what a conversation is. Many implementations use the simplest method and distribute based on all or part of the source and destination addresses. A more sophisticated implementation could look deeper into the packet and use session to identify a converstation. Pat -----Original Message----- From: Merhar, Milan [mailto:mmerhar@pirus.com] Sent: Thursday, August 03, 2000 12:35 PM To: 'Steve Byan'; ips@ece.cmu.edu Subject: RE: multiple TCP connections Steve, The problem with relying on Layer 2 link aggregation such as 802.1ad, is that it protects against packet reordering by keeping all traffic between a given source/destination pair on a single pipe of the multi-link. So, unless you have multiple attachments at each end (with multiple MAC address and multiple IP addresses,) you're limited to a maximum of (in the case of TCP/IP and Gigabit Ethernet) around 920 Mbps of aggregate traffic between two end points. And, although 900+ Mbps seems to be plenty of bandwidth for any single storage device, concerns have been raised that it may not be sufficient for the case, for example, of a storage array controller talking to a mainframe. This, in fact, had been presented as one rationale for spawning multiple TCP sessions within a single iSCSI session. Each TCP could potentially have separate IP and MAC end point addresses, allowing them to be carried in parallel through 802.1ad inter-switch links. (Although, I agree with you that being able to take advantage of multiple protocol accelerators is in itself a pretty good justification.) On the other hand, It may be a close race between introduction of Storage-over-IP devices that can take advantage of multiple parallel Gigabit Ethernet links, and initial availablity of 10 Gigabit Ethernet LANs. We might find ourselves having designed a clever, but more complex, way of accessing greater bandwidth, which is superceded by the brute-force solution of faster transport technology. - milan > -----Original Message----- > From: Steve Byan [mailto:steve_byan@hotmail.com] > Sent: Wednesday, August 02, 2000 3:14 PM > To: ips@ece.cmu.edu > Subject: Re: multiple TCP connections > > > Sorry, I didn't understand TCP multihoming. I've been reading > RFC 1122 and > RFC 1123 and discovered that TCP binds to exactly one of the local > multihomed addresses, and that TCP applications must choose > exactly one of > the multihomed addresses for the remote host. Multiple > sessions starts to > make much more sense. > > I noticed that there is a standard for ethernet link > aggregation (IEEE > 802.3ad), and that routers are available that will > load-balance over a set > of links. Given these two, it seems to me that multiple TCP > connections > aren't needed for speed or reliability, assuming ethernet > links at the > end-points. Am I still missing something? > > The argument that makes sense to me is that of parallelizing > the TCP load > across multiple hardware accelerators. > > Thanks. > > Regards, > -Steve > > Steve Byan > > ______________________________________________________________ > __________ > Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com From owner-ips@ECE.cmu.edu Fri Aug 04 01:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20293; 3 Aug 2000 21:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA11467 for ips-outgoing; Thu, 3 Aug 2000 19:43:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA11452 for ; Thu, 3 Aug 2000 19:43:14 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id QAA25026 for ; Thu, 3 Aug 2000 16:43:13 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.153.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id QAA28520 for ; Thu, 3 Aug 2000 16:43:11 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id QAA13951; Thu, 3 Aug 2000 16:43:11 -0700 (PDT) Message-Id: <200008032343.QAA13951@ha10nwk.EBay.Sun.COM> Date: Thu, 3 Aug 2000 16:43:11 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Requirements specification To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: cN1PBgbXWCmCH+NqDIGUug== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Fri Aug 04 03:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20782; 3 Aug 2000 23:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA15140 for ips-outgoing; Thu, 3 Aug 2000 22:08:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA15098 for ; Thu, 3 Aug 2000 22:08:16 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id EAA23724 for ; Fri, 4 Aug 2000 04:07:44 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id EAA256026 for ; Fri, 4 Aug 2000 04:07:45 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256931.000BADA2 ; Fri, 4 Aug 2000 04:07:33 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 4 Aug 2000 03:57:45 +0300 Subject: RE: IETF48 - foils available Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I will try. Dave Nagle will also try to post them on the archive site. Sorry for the inconvenience. Julo "Hughes, James P. (MRDC)" on 03/08/2000 15:35:00 Please respond to "Hughes, James P. (MRDC)" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: IETF48 - foils available I usually can distill .ps files to get the .pdf, but in this case, the fonts were not correct. Can you distill the file or (if this is a PowerPoint) print it with the fonts? Thanks Jim -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Thursday, August 03, 2000 5:16 AM To: ips@ece.cmu.edu Subject: IETF48 - foils available The slides I presented at IETF-48 are available at: htpp://www.haifa.il.ibm.com/satran/IETF48-iSCSI.ps Hopefully they will appear on IPS site (with Dave Nagle's kind help). Regards, Julo From owner-ips@ECE.cmu.edu Fri Aug 04 03:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20773; 3 Aug 2000 23:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA15147 for ips-outgoing; Thu, 3 Aug 2000 22:08:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA15107 for ; Thu, 3 Aug 2000 22:08:19 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id EAA30392 for ; Fri, 4 Aug 2000 04:07:44 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id EAA256020 for ; Fri, 4 Aug 2000 04:07:45 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256931.000BADE0 ; Fri, 4 Aug 2000 04:07:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 4 Aug 2000 04:59:11 +0300 Subject: Re: Several Questions. Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Kim, SCSI has a bit more than the CDB. It has a "task-tag" that enables several commands to be queued by a device, a selector for a unit, specification of the transfer length, an address where to place data, a response, a target address. All of them are specified in SAM-2. Their mapping is infrastructure specific. As for NFS versus SCSI - NFS is a file oriented protocol while SCSI is (for a while) a block transfer protocol. They are used by different application classes. It is possible that they will "converge" into a common successor but it will take a while until this will happen if it will happen. Julo Kim Tae Hyung on 03/08/2000 18:18:12 Please respond to kotaeji@postech.ac.kr To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Several Questions. Hi, all. I am a graduate student who is very interested in iSCSI. Although I feel very difficult, I am trying to follow up the articles in this mailing list. Reading the articles in this mailing list nowadays, I cannot fully understand every new issue. By the way, with my amateur's eyes, some very basic questions have made me write this article. The first question is about the SCSI protocol itself. iSCSI is just a mapping the SCSI protocol to TCP. But looking in details, I think that it is just the mapping of CDB. After removing the unnecessary parts of SCSI protocol due to the change of underlying physical media from bus to network, I think just CDBs remain, eventually. Then how can we map the operation sequence of SCSI protocol to that of iSCSI ? I wonder whether we can call this just mapping of CDB to SCSI protocol. I think CDB is not the everything of SCSI. The second question is the distinct differences between NFS and iSCSI. According to "draft-von-ipsissues-01.txt" file, there are the following paragraph. 4.2.4 NFS "NFS is the current solution for accessing storage over the network. Its strengths are ease-of-use, file sharing, and security. The general assumption is that a block storage protocol should outperform NFS in some applications, but that assumption needs to be tested. Also, some applications (e.g. tape library) could benefit from a block storage protocol. In order for a block-based solution to be accepted, clear benefits over NFS (e.g. performance, direct access) must be proven, Even then, NFS would still be the solution of choice for those requiring ease-of-use and file sharing." 6. Conclusion - Any block storage protocol must show clear differentiation from NFS. I want to hear many opinions about the distinct differences between the file protocol such as NFS and iSCSI. Thanks in advance. From owner-ips@ECE.cmu.edu Fri Aug 04 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20789; 3 Aug 2000 23:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA15803 for ips-outgoing; Thu, 3 Aug 2000 22:39:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA15789 for ; Thu, 3 Aug 2000 22:39:07 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id EAA21428 for ; Fri, 4 Aug 2000 04:38:36 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id EAA37268 for ; Fri, 4 Aug 2000 04:38:36 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256931.000E8433 ; Fri, 4 Aug 2000 04:38:33 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 4 Aug 2000 05:36:50 +0300 Subject: Re: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Fri Aug 04 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20802; 3 Aug 2000 23:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA15145 for ips-outgoing; Thu, 3 Aug 2000 22:08:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA15099 for ; Thu, 3 Aug 2000 22:08:16 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id EAA18932; Fri, 4 Aug 2000 04:07:44 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id EAA256014; Fri, 4 Aug 2000 04:07:44 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256931.000BAE1F ; Fri, 4 Aug 2000 04:07:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Peter Johansson cc: ips@ece.cmu.edu Message-ID: Date: Fri, 4 Aug 2000 05:01:33 +0300 Subject: Re: IETF48 - foils available Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Ready To Transfer - it is a "prompt" for data from target to initiator. Julo Peter Johansson on 03/08/2000 20:43:51 Please respond to Peter Johansson To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: IETF48 - foils available At 01:16 PM 8/3/00, you wrote: >The slides I presented at IETF-48 ... Julian, what does RTT stand for? Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Fri Aug 04 06:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21505; 4 Aug 2000 2:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA18480 for ips-outgoing; Fri, 4 Aug 2000 00:28:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA18452 for ; Fri, 4 Aug 2000 00:28:07 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0FYR002O42V02K@mta5.snfc21.pbi.net> for IPS@ECE.CMU.edu; Thu, 3 Aug 2000 21:23:58 -0700 (PDT) Date: Thu, 03 Aug 2000 21:23:46 -0700 From: Peter Johansson Subject: RE: IETF48 - foils available In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000803212134.00be5bc0@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: multipart/mixed; boundary="=====================_45058270==_" Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_45058270==_ Content-Type: text/plain; charset="us-ascii"; format=flowed At 03:57 AM 8/4/00, julian_satran@il.ibm.com wrote: >I usually can distill .ps files to get the .pdf, but in this case, the >fonts were not correct. Can you distill the file or (if this is a PowerPoint) >print it with the fonts? I don't know what the "netiquette" is on this reflector with respect to what I consider a modestly sized attachment (313 KB), but if I've irked anyone, flame me and I won't do it again. --=====================_45058270==_ Content-Type: application/pdf; name="IETF48-iSCSI.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="IETF48-iSCSI.pdf" JVBERi0xLjMNJeLjz9MNCjY0IDAgb2JqDTw8IA0vTGluZWFyaXplZCAxIA0vTyA2NiANL0ggWyAx MTc1IDM3MCBdIA0vTCAzMjA1NTggDS9FIDg2NzY1IA0vTiAxNSANL1QgMzE5MTYwIA0+PiANZW5k b2JqDSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB4cmVmDTY0IDM3IA0wMDAwMDAwMDE2IDAwMDAwIG4NCjAwMDAwMDEwODggMDAwMDAgbg0KMDAw MDAwMTU0NSAwMDAwMCBuDQowMDAwMDAxNzUzIDAwMDAwIG4NCjAwMDAwMDE5NTIgMDAwMDAgbg0K MDAwMDAwMTk5MSAwMDAwMCBuDQowMDAwMDAyMDEzIDAwMDAwIG4NCjAwMDAwMDM4NzYgMDAwMDAg bg0KMDAwMDAwMzg5OCAwMDAwMCBuDQowMDAwMDA2Mjg4IDAwMDAwIG4NCjAwMDAwMDYzMTAgMDAw MDAgbg0KMDAwMDAwODcxOSAwMDAwMCBuDQowMDAwMDA4NzQxIDAwMDAwIG4NCjAwMDAwMTEyMzIg MDAwMDAgbg0KMDAwMDAxMTI1NCAwMDAwMCBuDQowMDAwMDEzNzM1IDAwMDAwIG4NCjAwMDAwMTM3 NTcgMDAwMDAgbg0KMDAwMDAxNjEzMyAwMDAwMCBuDQowMDAwMDE2NTE1IDAwMDAwIG4NCjAwMDAw MTY3MzcgMDAwMDAgbg0KMDAwMDAxNzI3OCAwMDAwMCBuDQowMDAwMDE3NDkxIDAwMDAwIG4NCjAw MDAwMTc2OTYgMDAwMDAgbg0KMDAwMDAxNzczNCAwMDAwMCBuDQowMDAwMDE4Mjc2IDAwMDAwIG4N CjAwMDAwMTgyOTggMDAwMDAgbg0KMDAwMDAyMDYwMCAwMDAwMCBuDQowMDAwMDIwNjIyIDAwMDAw IG4NCjAwMDAwMjMwMzggMDAwMDAgbg0KMDAwMDA0NjA2NiAwMDAwMCBuDQowMDAwMDU2ODQ0IDAw MDAwIG4NCjAwMDAwODM3MTYgMDAwMDAgbg0KMDAwMDA4Mzg1NSAwMDAwMCBuDQowMDAwMDg2Mjg3 IDAwMDAwIG4NCjAwMDAwODYzMzAgMDAwMDAgbg0KMDAwMDAwMTE3NSAwMDAwMCBuDQowMDAwMDAx NTIzIDAwMDAwIG4NCnRyYWlsZXINPDwNL1NpemUgMTAxDS9JbmZvIDYzIDAgUiANL1Jvb3QgNjUg MCBSIA0vUHJldiAzMTkxNTAgDS9JRFs8NzUzZmE0Y2RkN2E3Y2RiOTE2ZmFlOTRhOWQyNTU3NWY+ PDc1M2ZhNGNkZDdhN2NkYjkxNmZhZTk0YTlkMjU1NzVmPl0NPj4Nc3RhcnR4cmVmDTANJSVFT0YN ICAgIA02NSAwIG9iag08PCANL1R5cGUgL0NhdGFsb2cgDS9QYWdlcyA1MCAwIFIgDS9KVCA2MiAw IFIgDS9QYWdlTGFiZWxzIDQ4IDAgUiANPj4gDWVuZG9iag05OSAwIG9iag08PCAvUyAzNjUgL0wg NDY0IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTAwIDAgUiA+PiANc3RyZWFtDQpIiWJg YGBmYGCZzsDGwGD3iEGQAQEEGViAoiwMHBVIgm1bGdQZuB8wMLD/YLjBvIzRjsGzwceB1YE9gfkC kw5D2F0IuIDQoTK1lSWq8KZSlcGkUBHFzBP3yjSZPEMCWpcrV95Ywqka6CQ2p/d50gSVqa4cUUXm c7scaKSMgS0tLa2joyEtLaOjo4MNxKY2h1DgAENDm4HllAiQVgRiVXD4iDHwM5YwHWHoYGB4psDA soCBQWgZA4OvBgOjcQIDy0zGDMYJHDKsP7gLmBiaGBg4XgD1ZZ37tMBXxkHWZSuDKBMX4xXWE/sV JA7AAlyfgeVGEJBmAmJvgAADAJZPj0cNZW5kc3RyZWFtDWVuZG9iag0xMDAgMCBvYmoNMjU1IA1l bmRvYmoNNjYgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDQ5IDAgUiANL1Jlc291cmNl cyA2NyAwIFIgDS9Db250ZW50cyBbIDcwIDAgUiA3MiAwIFIgNzQgMCBSIDc2IDAgUiA3OCAwIFIg ODAgMCBSIDg5IDAgUiA5MSAwIFIgXSANL1JvdGF0ZSA5MCANL01lZGlhQm94IFsgMCAwIDYxMiA3 OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTY3IDAgb2JqDTw8IA0v UHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUMiA4MSAwIFIgL1RUNCA4MyAwIFIg L1RUNiA4NyAwIFIgPj4gDS9QYXR0ZXJuIDw8IC9QMSA5OCAwIFIgPj4gDS9FeHRHU3RhdGUgPDwg L0dTMSA5NSAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczggNjggMCBSIC9DczEwIDg2IDAgUiA+ PiANPj4gDWVuZG9iag02OCAwIG9iag1bIA0vSUNDQmFzZWQgOTYgMCBSIA1dDWVuZG9iag02OSAw IG9iag0xNzg1IA1lbmRvYmoNNzAgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0 aCA2OSAwIFIgPj4gDXN0cmVhbQ0KSImUV02LJTcMvM+v6HMgWtuS/HHeQyCQQJhDDiGnzQcJ2wvJ HPL3U3I/92s7jZ1hwZ59dEnVbblU+vDxLW+f3jZH2XmsJcTNb2+fvrx8+ObVb7+/vfjtj+2FhYTz liRSwa7k4/Z18oWKbH//+vLbCxfSki4PhNw/IEI5pUkEY6AdA3XkeAZR/FTiImbpYkZHrHHCMypF 1nlMzxaTHyGTo1RmiKRUVBYh4zVk9uS9TFgCETpAJC6LFEEqQh6Q4kmVF5DcQyIlz3NaHDqId55K CfM0fLx8PjGJvC4w0kN8IPZ+AalvI/7EJJLk56+joceEQEkXebSWnOiJSVQWkMg9hJnc4rLEVCHl hGQKoczfJmmPESZ1ZZ4nV2rKJyZTjDMpMEzlpqlhlGstTTFFB0ymkmeKgqddPZ14nk4UCnH+3YqL A6aQhAXG12qL5/EkIc0zFTHJObid3zoVnNlcrEymOkxWcmEuRnbVsKbzfIoj4BaYPGCUWOZ6Y8Vp mHY+wTmKYS45Vmk9RinlBUZrvbWLHbyFmcuUFU0HieTdXKbs/A3SZCoET5znkmPn32Miqaww9Qvk kxt7Sm6uOXYuPQbP5rmA2DfuMBLIywJTiwDa3CCJFt/5EN0nQgPJ7BYETZR4ri/GuQsaA7Tg/2Hi icnkdK4vlqHDJKbg84x9yiRlFTX2UTNT1JmahJzR7adPFMZ5zr6radcjXykUdKYq7GDd/EIPhrdg BzeXdMKAPezc1NNwgHvzi7s+nAkHPJWmURn+jRd3e6gOZlg6vyrtvgpZYOrS4m4P14EFPo9nGFZP rixu/0BE4ezS+y4/R1g7XmF4wESKc/cwahloUVk0zkEyOcHZhcXlHqQZsgwyC0zfATjD2cnsbv+n z3CBsVtYlLGdcUmwOiuM7zDi4Oxk8dWG9iwO1s4trNBgA8QzSX6f3RAPaycrTG9rJHCd295jnyTk +r+5tettmmACDbLIM9hBwVAqbjEVDbZTMKdqXIxeg70VgbVbeJTRRgtmV+cWmMGuC4ZXHxdz0TAW COZZDmH+PsP0IZhoo1vkGYYcwZCb4mow6ocpyVZP7xvaUNPk5jd7GA0Fgy7Pi3ocQAWDrvrFXDQM uopBN6bF/BXygMGzC9+Sg/QYTLrerzAVciISwcvMET52EIy5wnPFyZ57CLze1Ncow9mlxUm4Wlah 9W7FVOump6eYYTG/zKk67aNihpU0ExPFxBp5LjfZ+T4qJtZUZnKjEc5OZ+KimEYDz2ThL/gMnE3e MH0U+GSFyBUF0/OZH7cvLx8+vuXt4+vm6r/Xj9/DM36Lv/4EY9iafzbvtu+2n3522y/LeK8vPyCr wLugeW0RzQifO9oNqJs9/MhasyHY8TmgIg79KEZ0Saj9jhAMLeJNEcrDkn4+gyomiVJ/UVj5UFEF o6L98ttXR36IrORnfrRrrZupZMsPx2l1fKz2UpXIEHK/Sduiq8LX4q4ZERgvnFVMNrF1RBLFCxH0 C61b7IiwFcexIviDSB9yv0nboit8X9KDiJ2PoZi0J2KSKyeRiJoW2/RZB66ajbYKWtZBpA+536Rt 0a2qYzqIBKiioZQkyJUIPIA8iWRiMMBmTz+J1C5zrOhKDyJ9yP0mbYuuqDV7MyNy/BXhJlk6InCk 4SSCAmSbQFx92SeRKo/HKql9kT7kfpO2RW+kP583HrpPIXZEkMA9icDBggE2O/4nker1j1XQta4i 0kLuN2lbdDtGfhCJuAf2zXGz8pWI6Wd+EhHyVX9wIS5EfJXyYxWMHgeRPuR+k7ZFt8IOx61BIZvC oZbIu44IHrkQ0UMI0cw7IsHk/1jVPa7vEHK/Sdui21X38Fifq4A7E6gMdE8E7KOeRFJ9S2wFrexJ RKxjHKtCTQ4ifcj9Jm2L3sTPiBxiF2FccuArEfRYeRKBOIEBtuKvREzJ6qLhoatDwP0maYutNqKV SgNK4+vlRH1JR8NCnzRw2gn5sWW+0kiVR12VH5d3CLnfpG3RFUqTHkRwwwxVYDLjlQimguieRKwB s21Jr0SKKeqxaus0Q8j9Jm2LruZT/UFEqwmImPw0d0RidQyNCB4zIgqdvxCxgaytOJIHkT7kfpO2 RVeUnL3g5+pgagspOMaeCLpCfhLBW4KBzbbpSiRUInWFwXgQ6UPuN2lbdEXJoRFWIrCwMFsR9pdd RyTUga0RKbWfYuPrnQliinqsrdEMEfebrC24ouYkVR6wIMkbCo27uzLBo5PFxgPG3kOtzN/7cOGh JqjHqukhIkPI/SZti46Tw4sdRCALQCVnWhAuRGCOfHgSgTKBAbbAVyLJBPVY9WEBhoj7TdYWXFFy /sEDclkMFdBAOx6w9u7JAzMlCGDzeuWRTU+PVUv9IP8KMABhWyNEDWVuZHN0cmVhbQ1lbmRvYmoN NzEgMCBvYmoNMjMxMiANZW5kb2JqDTcyIDAgb2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9M ZW5ndGggNzEgMCBSID4+IA1zdHJlYW0NCkiJbJdNji0nDIXnvYoaZ1ACzO8yMsoComQUIiXZv5Rz bLgFdVHruV7Txv4wxpifJOluLV/FhdvlcPWf6NvtQ75STbdv4foLI+UOjiPl9p4jSeLtCmfJ3SpH /vzl59eff6Ca7gjV7Ord0tXyXSXw01K4/v3j57fr7x93i0vXlBnyv9///jLZD26n9VTb7cRAwu0T Z6W77CByx/qA1LuAAJ9aVpBwDZF9GRi7wX5wOm2n5nRdxHB3kIJZ5UYkVwx/p/zB8A6eAMBvaSuI 5GvKHGZEdqP94HjaTy1gZZ4ood7iOavB0orCgVgelHCn5vVbvF9QkgZFZRYxlJfRfnA87acWsUZD yXfiLOpGv6JgRWFBidDz+s2yosDMlDmOqLyM9oPjaT8h8bhGosBF4yxEMG8oyP8VJWOhXr9xQ6nl mjIjNoayG+0Hx9N+QupxjUQJwOSsdIcdBSfALSgV2+/1K2lBiY5bYxI7YCS7zX7w+5h3ujSSIAsT Z+GA1Y0ExaA+JN7hTHj9hrKS+HpNybQ2lN1oPzh+7AddGlA8TiZzncXHbSjwnRcUzCxev76tKBKv KXM1kpfNfvD7mI9YooIUnPJ6leBvFxYQQSBDrA8IylTDB/NWjNiuKXOdHLvJ/u31MZ6xPMVA8fWc E+4WNwwk5oqBMkUMzJMFI7O0mszANIzdZP/2+hgvXBwxBGHmnHij7iwUyPywUCDtvX7qSqHZQYES Nhg2e/3b42O53XZycc5D4ZTMbN8QovsoMgLNAoEN/iAwAPyHHBgAm7X+7U9Q0DLtQhOLAQD+IolT cJLqAcAUuRNYJRMCEXkAeEpVsC4owm6vf3uUZsmTEa2iSYnoRGaQcDUHAtVjStoxFSTDkpS4468p C06IUawm+7dPqbgyMwcS6qwmA0pKloYp4Y7hhGGaHusQ5fBawB6OxEJussRRuXaj/eBXKvaUOYL0 itlIRHcUi7mxngOJaXqEOmoJbVrVH5LMi9ZkSTMim9F+8CtoiRyYMiqKVCPxOH+clG8v7YRiqh7x tjuu3LIWrlQ1Q1SWPGroy2o/eJaK3Q4YQVURx5HYGnKfs+rt0ollqHrE3K5+dDZrPcdlcU1Zil39 b6v94FkqdtpxRO4QjAXnOXEWjlU5spiqR9R5afOb1lsOiXhNWXCPGMtutR88S2l3qBxBxAYLoshZ EeerHVlMFZeE9jL8lo0laD1XWWobLLvVfvAsBZudOYLiFI0louRiktzVH1GoWS8EnQ0eYy9Lm5i1 mpqsbgZlM9kPXqVgw3UEN2KuyhHY95aI6+OYtkMTNwR7XsbdrxxZ71mV1c2AbCb7wasUbHa6sjCM hoELAlNwNuRUWociQw33jU+VlaJeQ1SfB8Nqr387RC8JOgyg6XVKgHKDcJaISnGs7kMTv/Ix1PhI WRGaXbGUNdh9/zLZv51id4CMATSswSjQL+LRkXCiypHCNBHiAPeNb5SFgv37lFVGKHaT/dupZFzX 2B4UmTgo4s0XWMJZakcK02SQSWGvhIcilGvKGuOg2Ez2b6eSsatohVBeJBoFXhTYxIRW1h8pTBNR Tkn4ibI0HSVqJVVZUxoUm8n+7VQy9hcDER1uLkqBJxaaNbSytztiDFWEma8kRtuvGEmLqMqaRyl9 2ezfbiVjh1F6csQWVwVBVXGOk9BqHQ/JUGWkQVD5JllBilxTsrgpyMtmP/iVzPcoR4DojATYyKCS GdlyIjFVRLsAofJNspI0JVFZ6wjJy2Y/+EWzhDcGR7BpgwQbqbPgqRxJTJUBJwmXvpCwZH1kmzHZ bfaDX0nI2sYRJHEwEkGt5SycqnYkMVVGHAiF/fJKErSEmmyjfrxs9oNfSfwbRxpy3Ug8OmnOQkL6 I4mpIuIeLQ8DjzbnIVEEik9AdoP94BTnH8HCCKpM1u4cjRLaXVwDmcE7YQxVhlt4JlHaV4yUrimb HwF52ewHv5I4K2AEfUQ1koK2GrMKE+pIYqoMNxAKnyYrSfHXlC2Movqy2Q9+UVBwqDiC688ZSUKy cRYP2ekNM1QR8UQSvlFWkqolVSWPq5HsNvvBr0TWGY6gIRgkaBIqZ7HuHElMlRFHMPCJK0lzLKtD xlFcXzb7wa/gCLnGEbRIwUj83TJn4e3ZjiSmyogDAZ+UVhJfr49Mo5i8bPaDX4m8iTACXR+NxKG1 5izeTCeSoYqI4wbhJ8vywuOOfGSeMdlt9oNfibyZOYLnWuaTC90CGmsUidK0ozmQmCojLvopfiVJ cn3kbN9fNvvBrwg7FY7g7VqNJKOtZrli53IkMVVG3OunrK/eltv1kSNhXyb7wS2OFBo3dEooOWWA 0B9nsZFLJxBTRcA9QLDPbQOximayhUGy2+wHvyJJ3x4ZJac4I8EDtXIWjlc5kpgqA97048qHBI+K a5TVl51+8CXCXh4jha8O887nKmfhLdo+3t3FH30yeq/dRsZBYTA7pmB/6QjPAKFRcNZqGmjS1aio 0eywEWI2/8Go1/Bkx7uKgdVcY2I9gaVfzlHXoOTdRsM1T9eCpMODTnflr4/RhNU3HUlYbdBZbcR8 xDOyb66P/2CnDh1cWTbWIZenjKMmvk32g9tpHcUE64oKgvgkVhg/wv8BQelZQNBYJf3kDUR4QZiE 8QGym+wHt9N6wrkoyUCQSMJZ2JYdBJUHb60JkrUo45PW+8pF3g8mI1opA9lN9oPbaT0hY1lpCRL0 HsI60aTHFUS0s58g6J55SVTVfkAyC5DJOAri22Q/uJ3WE3KNKyOI/S/zRRs3ENSd8AHhbcx70+li H5DKe8okGs0BspvsB7fT+oQGCCCLzsI9kjcQOHAPSLCmJuj2PyCNl4NJvIgM5GWyH9xO69xGGSAZ 54Axx8mqKwieMq4+INGavLh3Et5zU0zGUQ7fJvvB7bTOxA52apDIjSmOIurdBgKVBSSpGm/kDSTw ajCZ3Di+L5P94HZa51H3SRQEZY4FCg+ftoOAnpflACm6Snza+jzykZ2myeTzANlN9oPbaX0WP4JY sct4CtYgKwgqfnxAmr6B8Gnrg1ErmYoURl19GewHp9N2QqXhuoCBSuP1cCK/ZlX/9X8BBgAsCQXB DWVuZHN0cmVhbQ1lbmRvYmoNNzMgMCBvYmoNMjMzMSANZW5kb2JqDTc0IDAgb2JqDTw8IC9GaWx0 ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzMgMCBSID4+IA1zdHJlYW0NCkiJnJc/z2UnDsb7+ylO nYIFzN96upUSKZoixSpVsltEYaTsFPv193kM3APnUkWjMXq5YP/wMcZ+/fUKYk308Uq2mBqv4kx2 wqGIXP/99+uX69vLGpfTNWWUeH3/7dsrhmoctqRSTA5yNejyJoR4xexMrXL9+dYes5isMzFkI9xV rUmJM//54fXzCyC+mGRvEDEJBBhyXEGqu6aMMNpBdpXtYHZqjznqAQkSTcjc5U0sG0gyZQHBMoJE E1cQb/M1ZYxugOwq28Hs1B5z1gMSRKCBu4KRHSSYUm4QnBIEGEJeQbyCqIxpemRX2Q5mp/aYK9Z6 BXH4LWFXMmI3EG9qukGq8SDAIHUFCf6aMubBsWtsB6tTeUTMhawcUuEs7io418bhjA1pclSHQPAc xPmFI5ZryphLB3mobAezUzu+HA7WQbKp2JUtQi74BcQh9P0NIsaBAIOXFQROmjLWwbFrbAerU3lE yLnBEXGfuMsbmzaOjBC8OaKxAMDg4spR6jVlrNMhu8p2MDu1R4Qcz0eQYGzmLjG1bCCI/AUEwUaQ hD8WEEEATslb0UF2le1gdmqPCDkekCAeZ+Qu3KsdREwoNwhSEwgwlLyCMEZVJDci9aGwHYxO3RF5 pvYbgzzjJWNXNvDkiuFwGd8YDp87AYBjriuIMKN2mfz0yK60HQxP/RGZpmRHFGQacdxVoWlF4UTI NwpyE9Inx+zcghLVKSqZIRTlobQdDE/9EbkmDZRkIndxbXArCk7kF5SAdU7HJCtKZnrvMoXhlYfS djA89UcEHs9IFJio3AUPpg0F8b+iIEEpSkK2X1EK82qXCb7pKLvSdjA89eO+6RmJ4oHJXdH4HQU3 wC4oSFHidJS4oATLT9MlvkAn2XW2g91bvdWjkQRRGLkLF6xsJEgG5SZxSFIg4OjzSuKYW7tkWHeU XWk7GL71ez0aUBxuJmOdycduKLCdFhTszE5HPv43ioRryjRS20NnO9i91QccUUEybnm5MtOxX0CE L0YoNwjSVMWAfStGYHLtEvXQ4NhVtk+rt/KE4ykGkq/jHrxOYcNAYK4YSFPEwL7lCQ6JqbXLVMeT 81DZPq3eyjMPRwy8aJV78FqnlQKR7xcKhL3ToawUGh0USGGDYdPXPi3emqvpNxf33GduSYz2DSHY 90J6oHZH4AO/EegA/kcMDIBNW/u0h7cXbxImHEtfAuAXidyCm1QOAH0hvwROyYCAR24ArYIomBcU YdfXPi1K7cGTHIteRUCtxwhiOXsi0HUMyX5NBcGwBCUL9SkzbkinWFW2T5vCaj5xAgVv0GBASklS sQXVvT9h9JUO5xDlcJrAbo7orilzGJlrV9oOdqXgmzJGEF4hdRLRL4rDGJznQNJXOrg6aAqtmtVv kqQNjMocp0c2pe1gV1ASWTAlZBQpnQR9UuWmZJzUE0pf6uDv/saxR1lRikaIypxGDn1obQfLKOdR pmEGWUUsZ0KtiH3uKsbGE8tY6uDz/vSzTVlZarmmzLk//U+t7WBZUDt7yxlUvr6z4D5H7sK1ykeW vtTB63y0Ocb1lUMgXlPm4gbLrrUdLAsaCpa4qKKQbTsLvMhdaEJKPbL0pXgktJbhmDcWr/lcZS51 sOxa28GyoMmSxBkkp9BZAlIuNokp7ojCleWC01ng0fdrJ5E0m3ZZ7HTKprIdrAq6zqgzeBFTUQ7P ujejUc3HsB0r8UJoJ8FuZeVI+s6qLHY6ZFPZDlYl42OjfRS6sWPggcAW3A05pdaxkK522s/UtWhO 5RqiuDQYVn3t0yBqSdBhAkWvVQKkG7gzB2SKY3YfK/Gn9phsUlaE2p9YyuL7e/9Q2T6N4usAGRMo WH2nQL2IpiPiRuUjRV8JF3uYr35v7Fi/T1lkuGJX2T6NSsJzjc+DJBMGRTDswCLuUj1S9JV0Mil6 l3BT+HxNWUIYFJvK9mlUEr4qSiGkFwmdAh0FPmJEKeuOFH0lvByjcAiyFB05aCZVWWIcFJvK9mlU Er4vJgIq3JSVAi0WijWUssYeMcZSuJldEr3tVoyoSVRlSSOVPnS2T7OS8IWRelLAJy4KgqxiLTeh 1DpekrGUngZBYU+ygmS5pmRyU5CHznawK4n9KGeAaDsJsBFBOdGz+UTSl8LbGQiFPclKUpVEZSnD JQ+d7WAXxRJ6DM7gow0SfEjdBUv5SNKX0uEk4dEXEqast6zTJ7vOdrArEVFbOYMg9p1EkGu5C7eq Hkn6UnocCJn18kriNYV2WUf+eOhsB7sS+RtnKmK9kzhU0tyFgHRHkr4UHncoeeh4lDk3iSJQvB2y K2wHo7j/cBZmkGWSVucolFDu4hlIdN4JYyylu4V3Eql9xYjxmrK64ZCHznawK5G7PGZQR5ROklFW Y1dmQB1J+lK6GwiZrclKkt01ZfUjqT50toNdJBRcKs7g+bOdJCLYuIuX7NTDjKXweCQJe5SVpGhK Vcnr2kl2ne1gVwLzDGdQEAwSFAmFu5h3jiR9KT0OZ2AIK0m1TKtDhpFcHzrbwa7gCtnKGZRIvpM4 UxN3ofesR5K+lB4HAoYYVxJXrreMI5k8dLaDXQl8iTCDtS50EovSmrv4Mp1IxlJ4HC8IhyRLh8cv 8pZp+mTX2Q52JfBl5gzatcSWC9UCCmskiVy1ojmQ9KX0uOiQ3UoS5XrLWb4/dLaDXRFWKpxB71o6 SUJZzXTFyuVI0pfS406HvHa9NdXrLUfAPlS2g1lcKRRuqJSQcvIAoT3uYiEXTyB9KRzuAILvXDeQ ntG6rH6Q7Drbwa5I1N4jIeVk20nQoBbuwvXKR5K+lA6vOtj8JkFTcY20+tDTDrZEWMtjJrPr6NbZ rnIXetF6sm75slwOvSqUj2EY/8eX7+X68vWy/Idq6fr65ScQ/RNEf4yL/D80PdeP179+tdfvL/Sp WrkkXDp+mAbziBVCo6UQAsJqKX0FCn4FFAVMttLCn6+visfTVp1WPF7DrANv0t/Ee5hqH7zd+O1y /l7y8zziu4o4v65Qp4OfHSai69mFH6WG4fAeQ3dUlaF+X9o+tQW8Uwy5BBKhsmjZ4HEPRnurL/e1 GRf4ubJ9KmNwhMAJmFV4ehu6GD8l3drzkh+S+ub/AgwAx5UKWw1lbmRzdHJlYW0NZW5kb2JqDTc1 IDAgb2JqDTI0MTMgDWVuZG9iag03NiAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVu Z3RoIDc1IDAgUiA+PiANc3RyZWFtDQpIiWxXSY4tuQ3c1ylq7YWggZqO4VUfwHCvmg3Yvj/gCFJZ Sr0SPpD1wUdGSOKg0FeNI8w5vluPYbT6rV9SYxAp3y3lUNL3X18iM8igoYTcYWBMLIxJYeQKy5// +IphIhxf/ICvzO///evvX676G01khJppkIA/hO8hdsbk0MeGr4XAaSwShz9d9TeaSA/N+GpIYxp+ CylNBJXQZeOLLT9mfnNe+KerXuBEWuhZYIGvOAF+q4yS0OMmKALoMbptY+GfnnpBE6mhN1p6SNHx JaTJqBoafnnwcyN+T/zO54BOV73AiUgYkRbkqjlBCcWW1ULLm8DOftS6tuEEp6te4FBNYQgtM8Ts BKgtW1YP8tpBtB3I2objn556QRPJWI9b5kpxClIYNYLsDfBU8M1GUJ8KPV31AieScLqwZK9sEqBJ OqNmKONFYIePY+U2nhScrnqBE2H70JJQx2aZM1RW3oihyCbohO6TrTCKF+mnq97gYIlMSkajOOVE ozBxI4USXwTcQe9G8OCfnnpDK0gGk5LRKF5Vs4fKxI0c8k4xjx34daxzcoLTVW9wBdXEpGRBITsB Ko6JGxgkrxxXNnAX+86+CE5XvcGVHrJRolO8rCb+x8QN9NArx8Ly6Zlt3EdaBKer3uBKC9koW6gr x+h6SxymwCvHxdaePBEL//TUG1qpoVhV4TRRe8TH6TUEYZS8Mmzl32ZdeXD8w1NvYEVCwYRvGdUQ HT6jjBGEGdzKD3xiD7dhSWgL/XDUG1YpQSItuB2ao6NJmgXNuNEjl81JBviy0A9HvWGVjCsNloJW zAsdEQNX3g80iwXQYitPT2VuN73BFAyPQQtaYzgwWgPnNiYOemMPW3a22o9PTg9PvYGBqGZa0BhO CG8cWJvE3PDdlp6KJXaV5OmpNzAcUGUacGDJCbHXhEzhSLu84IFbbXa2VS+nn96gkNvGLCDXySsI GUJxNFRD3xn1UVBHtrw+8Ien3sBQmM1ygY7wEkLxckkFMC90gjcrybqq8fDTGxJ6ahhbQ8E6dg3s BrRhe2XVhkytyermOfbDU29gGAnDCNEPK6sQK8xVxUFseKazlqdyHP1w1BsWJtoYbpkrqbiiuWNM ildSs8HbsadVkaej3rAwkKcVEWTjyiqaAmOwcYi/0prs5ImOzTv66ag3MFwo08oIV280dYhLp2ZG DVxuGz5y9dar9Tma01FvYLBEaIwmFLsOz/JgFC5nSL8F79NRTNDW/hTl6ao3OFy4UWih3DULbv0W 03eHCshtE5jIgRZe5WkEH656g4NkiIOWjPJ1gmHF16Fjct4EnTUvJjlZoU5wuuoNDqInZVrQH2sH VLyMymjtTWAjXkwS1rJy8OGqNzjIwtRoEdSwE1DyMgqzQzZBtR2YHq95VeiHq97gIGyzZQVNIk5A ycsoTI9Xkq18iunxmp4cnK56g4M0L0aJPvG66pS8jKro8RcBJ0MxRQUJsghOV73B4W0hRknN6wTU vIzCEHkl2S6UYk2M0bAITle9weFxJFZX1LxOQM3rUfOV5Gw7KJaJ5779cNUbHB53YnU17YCdINqy MEpeSTapbIJHav/Bf3vqDQ1v08qyqpS8jo9WsVUN3HcvfEIXUw28up3gdNUbXErWl61S8zpBRCVn RE0b/IugTnZYHnN1mxOcrnqDS1xch4Wat9HS0CqoiJ443jaB3YzZLgNZz9JPV73BgRwZhwWtEp1g oJIZldC96Yeg2w7sGmaZOMHpqjc4HB9EFyyCSnYCal5GZauch8Cu32yXZZkrBx+ueoNDAVBmt4pW yU5A0cuognPZBPYizabIy1hV+uGqNzhU8BhumcMJKHsZhWkim0AInWNf7ewEp6ve4NDk0yjRKuIE VL6MwjSJm8Bu+WSvx1KfIzpd9QaHMTUbLRS/TlBsZnZMFXkl2Q4n2aXAN6ITnK56g4MlWl1RBDsB RTCjME1eSWZ2TfqX51n64ag3MMzxaFVFEezwyZq2Y66XV4ojyzMJj6k8s/rDVW9wuIkic9Iog52A g5lREyXzQ8D5zK6da1w4wemqNzjcpYk5acneO7CAvLHycLemvQNIZhKYRs9zFemHq97gIs+VFoph Jxim3nAR4kQ2gcn0aDI9P+/SD1e9wBXoGZY21DAKuRoBOqUxCtf4LlJMHRL4PHo0y4erXuAKFFnG XYtXG/LoBNTEjEK+dpHyBoj2ROQ8WkX64aoXuAJFmQct1TQbCSiLGYWK+xmlLreiN7OsKj099QJW oIfFCPE3Ozx1sQfNseHZvrHwm58EH456wcJEtNqA/EIVOzplMYPQ7uLo/4Eftpu3H0qnJvvDmfrf f3/98f03F8HDi9aIOa0yPvH0QrnBKZ59ERTPDMKs8TX8c61C4nZEfbVuf6S8V4HBGG1kPufwRtML X8F7rDZapvWfryBa5nuwVrMlROI6JjMm/nrAVc418CJYK8JS4J+++Y/eFTfZ6L7niT2rHUvBsUCC IHARFiyyJepBGCr+M5OvctoqLc9zsgOmKXi+ihz+dNXfaIK7HwMKBmgAglUQz+pbnnHDs4Gn9dpc kuvTU3+DSWUB0gBaWzzu0wms1vk42egmJGa1b1tn8+Gqv9HQOigZGkowhceYWBjDp8mGtyfTLLYH mT/wb1f9jYaWQD3QIGwdwuM9y3RBDPax4X3d9uggicOfrvobTYTvEhoq5uw0/IZZPBFUMGc3vl0w 00TjXGLr01UvcFC/mLMCC3zFCfBbZZSgOjeBPciGvTnmc/qnp17QIF0xZmnpGLOOjwfKZFSFQpMf /MzOH/bkGPM5oNNVL3AifJbQglw1J0B72LIaFNomsLMfpri4DSc4XfUCh2rCnKVlYs46AWrLlsVn ySaItgNZ23D801MvaFAaWI9b5koxWqQwiq+SH/xh/TWyEdSnQk9XvcBhuuN0Ycle2STgpcsoPkte BHb49rAZ8qTgdNULnAjbhxZc/U45OeMRNVy0PQSd0H2yFcYjtj5c9QYHS2RScFd3p5ycmYzis+RF wB10mxTjwT899YZWkAwmBVd196rCXVqZuMFXycavtgF7cfCcnOB01RscJn1kUjD5m1cVrtPGxA0M kleOTY12e3H0uZ6VH656gyt8ldCCTvGywlXZmLiBHnrlWFg+3UR1f7TWh6ve4ApfJbQ0vEqcAF1v icMUeOW42NqTJ2Lhn556Qyt8lNCC05Rh+Dg9PDwHRgkz/H8BBgC8r/3gDWVuZHN0cmVhbQ1lbmRv YmoNNzcgMCBvYmoNMjQwMyANZW5kb2JqDTc4IDAgb2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2Rl IC9MZW5ndGggNzcgMCBSID4+IA1zdHJlYW0NCkiJbFe5ji27DcznKzp20Na+xDczYAPGDRwYjrwE hvmA5xf4911FSketHiWaAQ9ZZIvFRV//+t2Xu1tIl7tLzzhrbtdvf//lK/V413KVVm7v0iVfKeY7 QrGEeufUrv9Aku5YMiTtzk4lPdw9wKjerkQIDN4XwjeipzLQN0U5YcV4J0dJv1MxdH/3okbdLXTH sEvhJ9Q40DdFOWHFcKcESXR3CgMdFu1uK/DaK6GTRu77xP6oyQkm+js1SvwdmwG7GzdYWsdFL+ym YYdObFcH9qYpJzA4yoGScAdzCG1cWOnEXPBVQ/dRE+sNfteUExguKDMNuDBvDvGtHpnCldb0gAdu 7s3yauCbnpygkNvCLCDX3hiEDIEcBWyoK6OkINBb0LxO+E1TTmAgZtFcZFDW4MvNkCJgHugEL0rJ PNi46ckJKRQlWokFhDXsfLMaekISF3hi0Dl75c289k1TTmAh300doh5GVtMdmKuMi1jwTGeOkzmG vinKCSukuzWT9JHUiFqGUUEpL/Sg8HrtfjByV5QTVogoZUra3UdWURQNZdLRKR5p9XrzRMfHG/qu KCewgE6hNOoo5qrwHkHSqt3xEb1j9FqreV7NrignMEicA7ESOkkxeNKDVv2OLkx4646p6DfUScpd VU5wwaOmKfEoapXUjhvwV3XuDmU5aOxgKfVBT3XwUpUTXEDTaJQE0NccNCVfdR5lvRxUch6NdjDU HOyqcoLzHZVNCepjfAHqo9EqoLSXA23xyWsK4sjBS1VOcB6to1CSwGFzUMBGWqF3pOUg6xc4rbEw GPpSlROcr3fQrKBIkjnImG60Qvd4JFnpE5t2Nz9zsKvKCc4XTC9KUCfGq4o60cxl1PjDATtDrDxT nw52VTnB+YzvpqSCg+YgosxphSbySLIOlKhFjNYwHOyqcoLzCdyjpGGQmYOASjer/khy0C+Imok5 b1+qcoLzGMHKq64XbA6choVW8kiy0y/QD8j1g//UlBOaD+AEJBnGI8coFY2qYd498AkddWvg6DYH u6qc4LzXuiwZs3jk2IHJAVZdG/9wkDsrLLQ+qs0c7KpygvMMrkKCaYxfICkoFTCiera35UAnY9Bh gIZjDl6qcoKDc2QcEpSKMwcNTKaVR/X6j4OqX6BjmDQxB7uqnOBwfVi6IElgsjlAqSRaBWXOdKDj N+iwjH3k4KUqJzgQoAZKUCrBHGB0N1pF3MtykJneEKqW82DpS1VOcGBwaybpzRxw7aUVuklaDhKh g6ujnM3BrionOBR5V5colWQOuPnSCt3ELQc65X3XasvzinZVOcGhTfVCCZdfcxC1Z1Z0lfRIsl6O 16EQ06DpS1VOcJA45RWXYHPAJZhW6CaPJDO7uvrHOEm6K8oJDH3cKau4BBu816Kt6OvxkWJHevrE a4qzV79U5QSHSeSYk8I12BywMdOqgzIfB+zPrNo+2oU52FXlBIdZ6pmT4vW9AwmcFzIPs9WvL8DK TAe6o4c+SPpSlROc471SwmXYHDTd3jAIcSPLga7pTtf00AZJX6pygIvYZ0htbMMgclYHqJRCK4zx RVJ0HTqwfjR3lpeqHOAiNrKAWYtXG/JoDrgT0wr5WiTlBHD6RGQ/GiR9qcoBLmKjDI2SrDsbHXAt phUY92mltm45K+Y0WLprygEsYh9O6hB/g8FzLzaj3hY8y9dFnmEmeFOUAxY6onID6xdYbOhci2mE ck+G/iv08Llh6YE62esf9tT//vPrL9cvDIKX57QQgx803vHk4HKBc3m2ILg80wi9xmL484giuaUI fpWqf1J8RoHG6LRlznt4osnBX8R7LBdKutafReA08/XWUtMQfv/jt3b9+HntPn7++NOXv/5w+evf kHQg/u/y7vrj9de/uesf+o5xyR4amPoMlzNjBI+of345xUoWbsbYa9XC8whG9A4Tfi0cg3VEh/fS VTh4+SzJ+Kd7+wCnvFFSWCrsTLEN+F1VvqMlLAqF/rg49AHvGBJ2CxcWfFycI70n/FNVvqOlzCcI BVh8CJbxXd6bTW8LXmvFzrmfvVXlOxoi0bZTOHy9wuMiM228ZmfCa6nbyVoz+F1VvqOhgLTpFO6d VeErhidtQDO34LVV2cnuaPC7qnxHS9iT2XIKBm90evmY7AFvnILduj7C115r51y/36pygMOujJ6T IIFuMQcZKx6tEgj6cWDTws7Uw3Cwq8oBDpsu2g4l6GXBHGCZ8LTKWOjSx4HOOzs5Ys3BrioHuITX Vg+UIFnNHERsLrQqWOiWA53Yds431ltVDnCoNTQMSkC2ZA4COh2tKha65aDMvSOPnvtWlAMYJc5B wsE7UuxRn7RCb0gLXjcmO7mkmYNdVQ5wWHxQNpTgrTFSTBLSCo3bLQe689mZZ/t5qcoBDsMGnY8S /GI5weAtDKuxmD4ObGu1M+exFrxU5QSXOCRN0i0nnf2SVvj+lWLbu+3MZfT7l6qc4CIWBbIKk71b VjB6K5nXkMGVYns52DnfiC9NOaGh2UeSirPXkoLhW0m8FnUJnvj69LEz1zYc7KpygosYn6QV5mod H4D/SL2WwOPlQB9vduY+8HdNOaFF7AiaE8QyPgDrhOYtg8gLX1+fduY+P2BXlRNcxB6gOamo9KYO 4s3FoBXw+IPPeTzPMnfjXVNOYBENFQOiBJAhGDySH2BUtelPeH0Y8kCDGeCbnpygIoZBoAQ10gwc A6PAqIHBCzzam4QnrnbAb5pyAosYZYUbA8tywKtFbwtbHz12ljiZv9TkBBO93mHBX3ZcAnPCwqiD uAu7+mueJc24N005gcFRVYdYKCwL0MZVwrS7hd70KatnyWMT3hTlBIXraeoOy5BloXEXhxH2pZVQ PirnOd8Ju6KcsJDbpksa5o8loXEVh1HAa26he523emKKD/hNU05gXCNJIfA0WirA3cgvxqReSQXV rnmWUUu7opywuNiSQSgyP4LHbyAZ6rA8gtcniJ2gyYDfNOUExtVdc4HtZQSPioBNBmEXesnXPEsf nWZTlBMUJM7ZduyNRY3bLoyKzqTPC/YaR3WjTnc9OUF9Vm/uugbOXRdVgh6eFyV1QyC2n3HvanKC 4qOQeUjcdHV3w8DJgVZ4kSxGaonqgdfEgN8V5QTGR20oKunN4DnjaNX1/THhY77mWeN4uL5U5QSH ce4LJdx1VYKRX5y/qmN5LgfZX/OsaZTUS1W+w/1fgAEAwqj7uQ1lbmRzdHJlYW0NZW5kb2JqDTc5 IDAgb2JqDTIyOTggDWVuZG9iag04MCAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVu Z3RoIDc5IDAgUiA+PiANc3RyZWFtDQpIiWxXS64tsQmcv1X0OAPL/88yMsoComQUIiXZv5QqsNvX fa0ncZ84UGU3YOBPjt6lWJ+aoxu+Pf/6k1t3rYan+eBSjtD88y9/vCu1Pku2kp///f3fv0zlzwUu DJcqNcn1agTNdU+v6JLfBD0+S7YaJ8FpKje40F321GTXoxFU1zO9kot1E4z+LNlamgSnqdzgQnM5 U1Nc60ZQXO/0yi7Gl6CG9CzZ8LMRnKZygwvV5U5NdXXeILsR6VWc3zeocTxLtj4mwWkqN7hQXNGo NHxgI0huaOSq8z9ukPlxTPK7G8FpKje4kDU14OlKXgTem9fom6DmZ8nux0vw01RucCG5qlEZrsw0 jc5r5JobO02rXkDPH+qEPw3lBhYibg1N8S7PJA3Oa9w6EnnDj/Is2WOfBKep3OBCwKmoCS7PEHsX 8DmbH0jkl6CF8CzZ07rBaSo3uAAv/K8WVAp+gaYOF/E5W+BRNwGMlux51vHHVG5wJO/UJNS6EXTU Or2CpvsiyFrBKnspk+A0lRscP59SZtS6ETTUOr0i8DZB0QpW2eus44+p3OCYAJWaglo3AlSKp1dy eMNegpaeJXubVfAxlRscNN5TU12YNyiodXplJP0mGEqgsvd1g9NUbnAswUwNfpk3yKh1eqEk9w1Y Wq8c6wanqdzg+IhoVDpq3QgSap1euP++QY9awSbHrIOPqdzg8AyGaJox0zSi1umFZzFuAkWmeI9/ 2skNC6940JAMlLqho1A0bEjAnaPMyyVHWMc/TeUGhz4UGZIKvxlhvpf0Ql/aOcq0WXLEWcYfU7nB oZNGhgSH6RYSkFeGjZ31R4S7FrDKkeYNPqZyg/MepU5NRPiMoOND0wu2O8J8yV6ZZxl/TOUCl8ZA HlODQulFCZpmeYu47Y4wv/sryyyCj6lc4NJAg+4dmoxSNwI0jkgvxGsHmZ/llXXd4DSVC1waaGuR GjRobwQFCPRCxuUXv6TnlWueOC3lApZGRSJTg795wXtPp+L26et4Xtlf8G0nF6Q0CnKYGrDOo2fU uPmMefT/wC5rl192SJwS9E9s5fnvP/787fk3z2AFaHLMkewElAvnRsenrXYKtHiNKz6/HeKv8xjZ b0OkVxv6J4X3GOHhP6P+iSIXnjTQ2TWYyJpozBGPBH2QWXVR+4f/FLOjYSDTiBHhIcqS4FERuYhQ AoPBBD5vOPq6T8FvNeHpwQksZgR12RKhoNP1ZmcLfuFm/FrR+WwmwNEyrlPRa5WnBKsHnt7rfTQT fODMbTLPUv+aym+0jNmgkg8nKWPCex4J44SPGz7xkTKZ83jhf5rKb7RcOGdRgXeZYAX3CsF87CsZ fOYTZRLlbfAfU/mNhpMgLlTgSweFx4cs9An6Hi34yvIzmWeRf03lN1rO3FyoQAY2hW8uDPpwcdnw uo6YxDIy4U9T+Y2GFwvdl4qCl1g/PnptTANOaT4zhq/biMnc88Q/TeUChwVCp7eKXhurEWAMafTK 86FRgqDbiMk8S/hrKhe4nLm3UIMqiEaAQSrQq+gAvwh0GzGJDz8JTlO5wOXMvYUaBKsbAQqv0It7 yybI7NImy5zpv6ZygUOt4fmkBsmWjQDD7KAX95ZNwMJSUeIs3o+hXMCo8R4aNNswQ4xhXOPGrWXD N8VXWdLK0NNULnAYBlE21GC9mCFmEtKLe8smGCxdk2U9Px9TucDljC6ulPjFYoJmW3mszmJ6CTii LFnKfIs/pnKDy/DKphkWEzTbymN17i2bQLcRk2VN9B9TucEl7i3UIDoWFTzajZnXubdsAt1GTM43 6GspN7TEtYUaJJcFBS23MfE615aNr8uIydJmt/6Yyg0ucW2hBtPBvAD+x9Tr6Dk/LqDLiMkyJv5p KTe0xK2FGpxlXiBjr6QXZogfF+isX5NljRsfU7nBJW4t1DRUeleC5NiBO5eWFz95lq/J6ucsdlrK DSxxZ0HnjUiGaPAIfoRT00d/wTOwKvDATPDDTm5QifsKNaiRbuBoGBVOXFc2eGLpmsSnnfCHpdzA EhcWaBLLcsKrx+gbu+jJVda0Mn+byQ0mBf2GFX/54hKYHRZO3FM2tm4fJuua4E9LuYGBqCkhBgqL AqzxKQfXgY2uq4fJWuZseBjKDQqfpysdhiGLAm7KmhjcUF70rHuHyTrXp9NQbliR+wk16D8WBMSf 0/fgerLRdekwiS4+4Q9LuYFxhmQKIU+ThQK5m3hjLicbPuVnyTpr6TSUGxanWmYQiizMw+M3JBnH 3B+Hz+NZkrOuwR+WcgPjvK6xwPQyD4+KgE9Bwm70Wp4l65gvzWEoN6jIzaToPBwsizqnXThV7UkL 3eYciOZnnZ52coOK3Dio4axr4Jx1g47oZaekTgjEDuOd7X+ayQ0qcqeAJnPS1dkNDadwI+BiuDNS S1QFVokJfxrKDSxy/aiqGd3g2ePoxRUlvvCpPEu2lCfBaSo3OLTzUKnhrKsatPzqw9M8y3MTlPAs 2fIsqY+p3OAiX1ZqOO0aAaddegWk7iaoOuGobGXe4GMqNziMVEkp0Q5sMWocd+mF1uw3gU7hJlud VfsxlRscRsLsqUE/s6g0jrv0Shh3N4GO4SZbm4/xx1RucBhqMUFDg75rUWkcd+mVMda8BFXHcJOt z0HqYyo3OIzleEGg4bhrBBx36YVq2TeoOoabbH1WwcdUbnBYLIpGhQOvEXDgpVdFKm8CLV6T/O5G cJrKDQ6bUdGocOBdBN6b19hpynVhye7HS/DTVG5w2OyqRoUDrxFw4KVXQypvgv5M0cOqgtNQbmDY SytjUvg4GjzmaI0bOvVOUrySz5I9ztf5Yyo3uIA+zJgUjrtGwLUqwgttboe4Ba1glT2tG5ymcoML 9nKhV6PWKzWV8y68Ao+6CXQMN8mdUAk+pnKDI3mnhvOuEXDepVfQdF8EOoab7GXOJR9TucHx8ykl GrI3Ag689IrA2wQ6h5vsa534mMoNjglQqSmodSOoOoQ0JATesJdA53CTvc0q+JjKDQ4a76nhyGsE HHnphc5dN8FQApW9rxucpnKDYwlmajj2GgHHXnqhJHmD/wswACxz/AkNZW5kc3RyZWFtDWVuZG9i ag04MSAwIG9iag08PCANL1R5cGUgL0ZvbnQgDS9TdWJ0eXBlIC9UcnVlVHlwZSANL0ZpcnN0Q2hh ciAzMiANL0xhc3RDaGFyIDExNSANL1dpZHRocyBbIDI3OCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAg MCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIA0wIDAgNjY3IDAgMCAwIDAg MCA3MjIgMjc4IDAgMCA1NTYgODMzIDAgMCAwIDAgNzIyIDAgMCAwIDAgMCAwIDAgDTAgMCAwIDAg MCAwIDAgNTU2IDU1NiA1MDAgMCA1NTYgMjc4IDAgNTU2IDIyMiAwIDAgMCAwIDU1NiAwIDAgMCAN MzMzIDUwMCBdIA0vRW5jb2RpbmcgL1dpbkFuc2lFbmNvZGluZyANL0Jhc2VGb250IC9GUE1ORkor QXJpYWwgDS9Gb250RGVzY3JpcHRvciA4NSAwIFIgDT4+IA1lbmRvYmoNODIgMCBvYmoNPDwgDS9U eXBlIC9Gb250RGVzY3JpcHRvciANL0FzY2VudCA4OTEgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQg LTIxNiANL0ZsYWdzIDk2IA0vRm9udEJCb3ggWyAtNDk4IC0zMDcgMTEyMCAxMDIzIF0gDS9Gb250 TmFtZSAvRlBNTk1BK1RpbWVzTmV3Um9tYW4sSXRhbGljIA0vSXRhbGljQW5nbGUgLTE1IA0vU3Rl bVYgMCANL0ZvbnRGaWxlMiA5MiAwIFIgDT4+IA1lbmRvYmoNODMgMCBvYmoNPDwgDS9UeXBlIC9G b250IA0vU3VidHlwZSAvVHJ1ZVR5cGUgDS9GaXJzdENoYXIgMzIgDS9MYXN0Q2hhciAxNTAgDS9X aWR0aHMgWyAyNTAgMCAwIDAgMCAwIDAgMCAzMzMgMzMzIDAgMCAyNTAgMzMzIDI1MCAyNzggMCA1 MDAgMCA1MDAgNTAwIDAgMCANMCA1MDAgMCAwIDAgMCAwIDAgNTAwIDAgNjExIDYxMSA2NjcgNzIy IDYxMSA2MTEgMCAwIDMzMyAwIDAgNTU2IA04MzMgNjY3IDcyMiA2MTEgMCA2MTEgNTAwIDU1NiAw IDAgMCAwIDAgMCAwIDAgMCAwIDAgMCA1MDAgNTAwIDQ0NCANNTAwIDQ0NCAyNzggNTAwIDUwMCAy NzggMjc4IDQ0NCAyNzggNzIyIDUwMCA1MDAgNTAwIDUwMCAzODkgMzg5IA0yNzggNTAwIDQ0NCA2 NjcgNDQ0IDQ0NCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgDTAgMCAw IDAgMCAwIDAgMCA1MDAgXSANL0VuY29kaW5nIC9XaW5BbnNpRW5jb2RpbmcgDS9CYXNlRm9udCAv RlBNTk1BK1RpbWVzTmV3Um9tYW4sSXRhbGljIA0vRm9udERlc2NyaXB0b3IgODIgMCBSIA0+PiAN ZW5kb2JqDTg0IDAgb2JqDTw8IA0vVHlwZSAvRm9udERlc2NyaXB0b3IgDS9Bc2NlbnQgODkxIA0v Q2FwSGVpZ2h0IDAgDS9EZXNjZW50IC0yMTYgDS9GbGFncyAzMiANL0ZvbnRCQm94IFsgLTU2OCAt MzA3IDIwMjggMTAwNyBdIA0vRm9udE5hbWUgL0ZQTU9LSStUaW1lc05ld1JvbWFuIA0vSXRhbGlj QW5nbGUgMCANL1N0ZW1WIDAgDS9Gb250RmlsZTIgOTQgMCBSIA0+PiANZW5kb2JqDTg1IDAgb2Jq DTw8IA0vVHlwZSAvRm9udERlc2NyaXB0b3IgDS9Bc2NlbnQgOTA1IA0vQ2FwSGVpZ2h0IDAgDS9E ZXNjZW50IC0yMTEgDS9GbGFncyAzMiANL0ZvbnRCQm94IFsgLTY2NSAtMzI1IDIwMjggMTAzNyBd IA0vRm9udE5hbWUgL0ZQTU5GSitBcmlhbCANL0l0YWxpY0FuZ2xlIDAgDS9TdGVtViAwIA0vRm9u dEZpbGUyIDkzIDAgUiANPj4gDWVuZG9iag04NiAwIG9iag1bIA0vUGF0dGVybiA2OCAwIFIgDV0N ZW5kb2JqDTg3IDAgb2JqDTw8IA0vVHlwZSAvRm9udCANL1N1YnR5cGUgL1RydWVUeXBlIA0vRmly c3RDaGFyIDMyIA0vTGFzdENoYXIgMTUwIA0vV2lkdGhzIFsgMjUwIDAgMCAwIDAgMCA3NzggMCAz MzMgMzMzIDAgMCAyNTAgMzMzIDAgMjc4IDAgMCAwIDAgMCAwIDAgMCA1MDAgDTAgMjc4IDAgMCA1 NjQgMCA0NDQgMCA3MjIgMCA2NjcgNzIyIDYxMSA1NTYgMCA3MjIgMzMzIDM4OSAwIDYxMSANODg5 IDcyMiA3MjIgNTU2IDAgNjY3IDU1NiA2MTEgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgNDQ0IDUw MCA0NDQgDTUwMCA0NDQgMzMzIDUwMCA1MDAgMjc4IDI3OCA1MDAgMjc4IDc3OCA1MDAgNTAwIDUw MCA1MDAgMzMzIDM4OSANMjc4IDUwMCA1MDAgNzIyIDUwMCA1MDAgNDQ0IDAgMCAwIDAgMCAwIDAg MCAwIDAgMCAwIDAgMCAwIDAgMCAwIA0wIDAgMCAwIDAgMzMzIDQ0NCA0NDQgMCA1MDAgXSANL0Vu Y29kaW5nIC9XaW5BbnNpRW5jb2RpbmcgDS9CYXNlRm9udCAvRlBNT0tJK1RpbWVzTmV3Um9tYW4g DS9Gb250RGVzY3JpcHRvciA4NCAwIFIgDT4+IA1lbmRvYmoNODggMCBvYmoNMjIyNCANZW5kb2Jq DTg5IDAgb2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODggMCBSID4+IA1zdHJl YW0NCkiJbFfNqjW5DdzfpzjrLBrZlv/W2Q+EWeQBwmT1aSCT94eUJP/1ibnQB3SlqrZLVrl//v23 H3oa9c969vD577/+/OHCD7f4qSE/mcJHfpjKQ1w/JdeHSvn8QiQ/1DTSHooWKenJUavKwyUg4gSR P+vZ2yB4p8oNjvgJ0SO9OUF8ctGq+nDcBIasj/X67zy5YVF6QtFIfzo7engKaVV7UtvoOX/ms4f5 +u9UucFRfCIhUlBHTkBPYa3qT+JNUMNnPjtW7gTvVLnBUXiiSoKXaS4JyIvKFulJtAmwrfPZ01jB V6rc4IieqJJgO5tLgu1r+loRuVvhjq1YT86D4J0qF7jU+8NGmZ7ashFUvGhCFVa7FdZ9X8+cBsE7 VS5wqTf0cUOEn8pOULDJWgW9tsi6LetZ5greqXKBS72ijzWSn0pOkIGgVeg4Xvh46fWscwGvTLmA pV7QyBrBL094Ii3Kz3770j/r2Rb4zpMLUuoZPawRsI5XZ5xxr+nj1f+DPEYr7jw0Tg72E2v+/PXH zz8/f+o7+AH0Z4/jJV6AcuHc6Nja4m+RMAi0CNvvL/GP8RpMOxHtVbv9pLBeI3z0z6lPFLnwpJ5w AjSCronOHDEktAadVSY1mhZHH6vKWMLvf//tB4PxQVcbVMT/xMgSCgsEjFAUUKopaHShvc1lZfyv JEwgvMiv/8P59fP7uYCmC/D/BpospTlCpTYwI2mko0O/MbPxeoP0uMVpo0O+UuUGlzXHI1jKIIgc bNeobIK2G7DWhX9myg0taw4iTDgCjh+eWrzK+9Tx63F+ytyhd6rc4HKwIaGREidB1H2tBJ/ZBCXv AcB9EZypcoNDhHXPOEAmJ1A39Srv8XMCxHgO4a9UucFlsimkkdwmQYpe1Q+N2VYAazhs5CtVbnCI sG1ahE4WqR2jW6vCOCvHjNQrwjbar0y5oXG3MacRjhM/Fa/qh8axmA/m857wlSo3OES4eaT7nlV4 jwkXx9E+XKTlPtzQCd6pcoNDhEwUHPM2CZi8qh8am5Ezj2VM/DNTbmiMExg90oYCGBumm+Ys/GbH bNykynCRr1S5wSFCJgrjNjEJmL2q8kEQ9l2Q2yI4U+UGx9WHGyJtaFzsoJbKY7iNq4iegtr12dJo 0q9UucEhQiZKxnVlEnDzqrol8GtUrXVs1CQ4U+UGxwXOxBapbRKQKQcHbcvJdd9BUPQct5AWwZkq NzhEQtRIgUc4AS7X0atwo3jfNysX26jZpe9UucGBChtmkcqTgEw55PImsAa1gaq3pIl/ZsoNDZFQ NAKTGxozZPKqEjd+slcPtox51/lKlRsc1C7NI5UmgXkgWjDQJoh6DErvQ4hJcKbKDQ6RaKLAUofG sFTyqnxoHHQFxexyneN3ptzQcOBq9MjwN1QF0w0n5pDYDhiM13TgRXCmyg0OkWii9OlvGCuFvSof EpOegpK1SdVxneCdKjc4zLxaPDL8DVXugZhaW+PabAWpmtJhEZypcoNjzUEENjH8DZPdPRAuTgeB rSCGobQTvFPlBsea45Hhb6hyD+x2A5oEdgAK5aH0JDhT5QaHCOumwanJP5MquQnCbHmL7CPCzAbD f+C/M+WGxnoZ8UjmiZ8AplV9a1yztmeubK208M9MuaEhwrZlMGj/qMP9qTWtwrscEtsEyr6A6QVf qXKDQ4TII5kmAT5fraofEidbAKfRSZPgTJUbHK6POXpkfKjiCqs3K3zQwgM3QdT2zPbUTnKCd6rc 4BAhEwWWypMAvWZV7ZA46ObkEK2T+iI4U+UGhxt8Lh5pQwO4uCmnOZuA9N31k1M7aWrwTpUbHCJk qsChaRJw86q2RfYhynYWciuL4EyVG1yq7m+I1DYJyJTj0y6L3Sg4mxJ14Z+ZckNDxOwNn7FxaIwP 1+hVh1vaDNXribeq478z5YaWitsbIpUnPplu+TRLPbogsKtp5roIzlS5wSFi9pbhz0NieHjxqsMt 9Q1BYAtIw2u+MuWGlrK7GyKVJj6ZbOU0SzeB1LMdhbQIzlS5wSFi9paxd0Nh7Cd51WGWxb47UrWj QLNH36lyg0vs9oZIKZMgmHBoEFyzJoG5TLJ7F/e4CM5UucEhEvFyJcOgh8YwcfYqnTuTINoK2Fpp LuCdKTe0BH8uHilx4gcTDuclbvxgC0jWSaUvgjNVbnCIxKYR+PPQGB7evCrTJrD7RLJbC+ep8TtV bnAJM4o8ktskSNGrelsE2T48kt0q9LBNgjNVbnBJcxApNO2twFBtYzHoy0EAZH054M/3fyfKDSxp jkeGu6HKHRAOvhXOTffejIBDW/hnptzQkuZoJCyvJDfArkNh4ytyNENmmlP0nSk3NESIPDKtktz/ UNW3wNmaPyZToYdFcKbKDS7pofdId0Wgjflfx8scAttHR4xpHGUj+EqVGxwiZHsWodIkMP9DVTsE NpuP5Ee5LoIzVW5wEb1UPNLtQ5VxPsz/OtrhkDjpCoLZTcpjkH6lyg0OEdJTgQOZaBKwfhyiqh0a W/uEWmxWLPwzU25osdktUyP6ueX4RFqlORvfjm/Ihh/bIjhT5QaHSDBRMKCKE1Qb9VpVD41J2yeY 1eiscIJ3qtzgYrUhr5HKk4BYqzRnEXC3Ffg0pbwIzlS5wSESTBR4RHSCYj2vVWVLwPZFFmzaxT7m xFeq3OAi7Ll5pNIkIFMOFr414KoaU2ebFnURnKlyg0Mkmiow1CEyDJW8Km8NdPyDoKoScd7av1Ll BhdBFT1SyiQIplw5zdJvWuTzIvMiOFPlBoeI2RsuZ2GIjA1lrzrMktlWwDbxeLjxV6rc4CK7vSFS 4iQIplw93dIvc5SCDaS6CM5UucEhYvaGC3gYIsPEm1cdbmmTiOy0xTC79J0pN7SY3N0QyW3iR/Kq wywZM5Ns2KUFfqbJDQoR9zbMj6FvhERa1Q6jVGAHxZXcP0yTeS3mTs1NfwJ+/vpDs8NH/5D9PwEG AE4A+7wNZW5kc3RyZWFtDWVuZG9iag05MCAwIG9iag0yMzM4IA1lbmRvYmoNOTEgMCBvYmoNPDwg L0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA5MCAwIFIgPj4gDXN0cmVhbQ0KSImMV01vJbcR vL9fMccnA5ptks2v624cxEYSONYDcghykBe7iQJTcXaT+J/496a6ezjD95ZyBAOStt1dRbKaXZwT l7BScUtKZXUUl3biGNZUwpJCWTOV5UeNeJJIXV3RSPFr9lYVS0Tk41cnWmv1C34W1p9l+fz++YvU NoOLkmORlPJG4Fmq6krpICiCnKr8zHnHHzPbDC1KDiJMq2PDd2tOVhX5wM9BoKP+xP82guvUNoOL bi3FIsl3Ai/nmmklfxBghfgZdB9cd4Ixtc3gEGE5M3aQyQhoLWRVkQ4CW7v3G40RXKe2GVyUKovE 0gmCt6o6aMy6A0pKk3aCMbXN4BBhPTQPnTSS61pEuYzFDBrr4ZTqhGCT4CazzdC44nQtwr7jh2RV ddDYy9JLjspSdoIxtc3gEOFikWpnlgtkkirJOQicbiBKl5aydelNapvBIUIqSlhD6QRMVlUHjeVo CvO2jY4/ZrYZGuMGeouUTYGM45Aqydnxi16z4pUAZ2AE16ltBocIqSi8Bu4EzFaVeSDQHZCeEJed YExtMzjOa0wWKZvGSS9qyrz6QwK5PLTmKj9L2Jr0JrXN4BAhFSWugToBF6vKhwQlyw4yZpAdVCcY U9sMjtOaiDWSSycgVS5iGvJOkHQHSe5xcWEnGFPbDA4R5yWSVr+JHKGTVaV0EES5AJmTHlTv0uvU NoMDFQ5MI5k7AalyyOWDQBtUB2oucccfM9sMDRGXJJIhk+EzZLKq5A/8oEt3uo3cT+g6tc3goHYq FsnUCdQD0YKODgIv1yDVugnRCcbUNoNDxKsosNRNY1gqWVUcNHayg6R2ud/j68w2Q8OFy94im7+h yqluuDGDxHrBYLyqA+8EY2qbwSHiVZTa/Q1jJbFVxUFikluQojSpOK4RXKe2GRxmXk4W2fwNVeaB mFqHxrnoDkJWpd1OMKa2GRxLDiKwic3fMNnNA+HiNBDoDrzblDaC69Q2g2PJscjmb6gyD6z6AuoE egESxU3pTjCmthkcIiyHBqemlJSAzARhtnyIbCNCzQbDf8O/zmwzNJbHiEUid/wAMKmqh8Y5SnvG zNpKO/6Y2WZoiLAeGQzaawTvp1KkCmsZJNYJFG0D3QtuUtsMDhEii0TqBExWVQeJg26Aw9ZJnWBM bTM4PB+jt0ixM8MTVl5WCU/OMEjspT2j/pROMoLr1DaDQ4RUFFgqdwL0mlaVQWInhxOd106qO8GY 2mZweMHHZJGyaQAXV+Uk5yAgWTuXop3UNbhObTM4REhVgUNTJ+BiVeUQ2YYo612IJe0EY2qbwYVs /oZILp2AVDke7TLpi4KjKpF3/DGzzdAQUXuLEfZm+AkyWdXgljpD5XlirWr415lthhaS2RsimTs+ qW5xNEu5uiDQp2nkvBOMqW0Gh4jaW4Q/bxLDw5NVDW4pKwSBbiBsXnOT2WZoIZq7IZKp45PKlkaz NBMINepVCDvBmNpmcIiovUWc3aYwzpOsajDLpN8dIetVoN6j16ltBhfY7A2RlDqBU+HQIHhmdQJ1 maDvLq5+JxhT2wwOEY/FpQiD3jSGibNVydzpBF53wNpKfQPXmW2GFuDPySLJd3ynwuG++APf6QaC dlKqO8GY2mZwiPgiEfjzpjE8vFhVpINA3xNBXy0cu8bXqW0GFzCjyCKxdILgraqWnSDqh0fQV4Vc tk4wprYZXJAcRBJ1e0swVD1YDPo0EABZFgf8vv7rxDYDC5Jjkc3dUGUOCAc/FI5Fzl6NgF3Z8cfM NkMLkiMRt3slmQFWGQoHviB7NWSmPkWvM9sMDREii3SrJPM/VNVD4KjN74OqUN1OMKa2GVyQS2+R aopAG/W/isUMAutHh/dhu8pKcJPaZnCIkJ6Zh0qdQP0PVWUQWG3ek13lvBOMqW0G59FLySJVP1QZ 90P9r6IdBomD7MCp3YS4DdKb1DaDQ4TkVuBCBuoELB+HqCqDxto+LiedFTv+mNlmaL7oK1Mi8rll +ERSJTkHvl5fFxXfl51gTG0zOEScioIBlYwg66iXqjxoTNI+Tq1GZoURXKe2GZzPOuQlkrkTEEuV 5OwEXHUHNk0p7gRjapvBIeJUFHiEN4KkPS9V6ZCA9YvM6bTzdZsTN6ltBudhz8UimToBqXKw8EMD zqIxVdZpkXeCMbXN4BDxqgoMdRMZhkpWFQ8NZPyDIIsSvr/ab1LbDM6DylskpU7gVLk0mqW9tMjm ReSdYExtMzhE1N7wOHObyDhQtqrBLJl1B6wTjzc3vkltMzjPZm+IJN8JnCqXR7e0xxwFpwMp7wRj apvBIaL2hge420SGiRerGtxSJxHpbfOud+l1Zpuh+WDuhkgsHd+TVQ1myZiZpMMu7OBjWptBIWLe hvmx6eshkVSVwSgFeHl490d9ktuHaVCvxdzJscgvh1+fPpweTm8vpzeXi1/ccvmISlflky0s9/0P WuRyevliwNsHZ4FnyqXtX5bHT9mEfIRhFZf3p3v90y+Xn09/OX/z9g/L9x8+f3j8dIeOOr//+/L7 xx+Wpzv07fn5DmPgvPzu8e4+rvX8dOcwhc8f7/56+fb09eX0L/0wKrjSxu5lhRkPDQ+fh8PIJv68 PMs2fn3xMTu8EYOuXRZIsrTz493lH0Lzp18BYHHoAv64FodvOd198tXkq9Z7803ann7ARuN5ubvH fe0b7NuWYMIy5F/73gkB2T+E4S4MtksBX2D3/Q+slhNaBZ0hI8FhiS/qomrgXbbv++nh3cM3i2ze 6TbC9tkoiaxfqvrr8psxdY7txSrjcn+U6Pmef3kV+i8vA+MvebEMwLaNak31iA6/w4U/Y3X//O+H T8vl3XdLe/zppydE+fz8Nz3C/7eA1+K8cK64s/K6vcdDBP1xrPE4ajTG1xd0RDz/1iQ3OH2FyIyU D6CXljatfEGEgEb8QoN7Pd1XcG2ZL+iQvMtf6CA3ARRcXs1hqWjq1Jsa86xCmvvtN7pcTCg7h97G RJVrr/dV/uvzhY75In/KCX/7nx+fHp+Xh0fV8d+f7jAmz/aP5z5IALnYk9jjLZ9h2x6fnQQj3Kah J6fGioGCGfrm3WdHy/vPWH9Vu69S/uY7gzAwmbDy3k7Lz8srUN9+dfqfAAMALkQDXg1lbmRzdHJl YW0NZW5kb2JqDTkyIDAgb2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjI5Mzcg L0xlbmd0aDEgMzU4MTYgPj4gDXN0cmVhbQ0KSIlcVgt0jVcW/vY+/39vmhDRkOudGzcJ8vAIFa9E KgkpTSURS6LG5Cby8AgxQhNLWo8ZRBRV4lVttVoqNDcIYaaVajqUWpYahLGUmVKqFK2ySO6ZndtZ s9r59/rX2uecff7z7b2/vc8PAtAaC6GQMnZcn6jsMRl7gYosmX0pp9BZFJ0U0x5Ydhmg7Tnziu06 4b0NsiZja0leUX5hXWpNPeCVDpgx+TNK8zpP/nY7ENoApDUU5DqnfFZZ5iPfc8megQUy0arMOkEO XCfj4ILC4pJteybdkXEd4Dt6xqwcJ7ULnAKULpdxeqGzpMhrd6tk2R8n9vaZzsLcndG2o0D5u4Jn a9GsOcX6Z1lB+cKW9aI/5RYtnvPpVKCLAXjXm6sE1YsIlLeLmo0OgL4m782W1z1aN5nT4XAX6H+p WNm9/r/vr08IdmIV+aAMi5GIKGzHCUxHEVJRjWG4RxcwCoZYvYJeiEMzAsiJkRQto1Ww6ROy8rK+ xTfA2IRFeIC5OI8c/B0WbKb+CMYgfIUYnQ9/sxEDsRTr9T9hNQbgAzTqy9qNJLyHRhpG49RCMxYT MB8LsJJsFEaDaAFCBUMJPkE9+z1Ti1ZIxktIRwbysd8gOdNECqrpnIqXkzJQQc9Rvd4Nu6AKRSSe p4Ecrg+jG8IwAEMxHH/BOmzEBepNMaqfcQg28cmJQ+RLAdSdjui3ECiSjEmCdCUqsQsncZICKZ37 qCzzI/dN+GKWICxDBc7hPnnTBCrhOrXHPVxP0/t0g+yOlnMSMFpwl2GDeLcDB1CPzyQmjdSVUmgD 3TWKzajmRe4z7qs6QN9HG8E6HgWYiddQLrl5G0dxCd/iMRnkRW3pKPflS8rXeNu0aehlLQxAHzwv 0SrBMiwXOSQ7viA79aT+VEzn2Zfb8Ax+lav4B1WuatS/je90vN6pP5eY34IVDpFQpElWyyRrqyV3 u/ExalGH4/ge9/CzRHIaVVAN1dIjbsd7+JzRZDaa9/RW3QQfiXYIItBXpL9EcBReECwzsVky9SVO Sc08wRPqTIPpVVpGK2gVradK+oZ+4aV8mq+oSvWRcqnjBhlRxjSzwrxqSbU63ZXuzXqMeOcv3x4g vImVGOYKF+cIJ96SOO7FQRwRbI/wVOLiL94G01BKoxJaQItoNb1LFzmJp/EsLlKkuiqH6qGWG4FG lXHGuGTONyvcoe5M3RstvPEWNgwV3Bkif0SenDJfpELiUI2/SbaOCWtvCZsf4qmcxpJnH2pPQdSD EkXGS9YzaDI5qYDK6H2qokt0l/24A3fn1byO3+ev+Ts1W72ptqh96qxyG9r0MaNExpiZ4m+V+cAy 3lJuHWHNtu7w+qo5rPl48xV3K3d7dw/3OPef3X/VGXqefkVv0zv0Hl2t6z2VqoS7XYVfdpEe6C2V MwYvYrLgn47ZwskVWIM3RHaID/uwHw3CuDP4GlfwjcgN3JTM3vb49BBN4lMHclA/4Us0TaJsyqMi mu+RxbSRNtEWctERqqcTdJYuUCNdFfmFHtFjfpb9uQ9HcwKP4rGcxjmcy0X8Gm/kLfwhH+TD/IVk +Txf4OvsVl0kE4kqSf1BTZaIlKpFaps6qP6hzqlGdU09ltgYkqMgw2GEGEOMfGOJcdXsKXGaYk4z 3xE5avGxTLNUW/ZZTlpuWi3WntYka4r1Q+teq5ZKqcZaqdLfPMK4ndSLXxaUij7n/fQmneK9xh32 pUyar8CRRoRwPBk3uFyFUKwqoc5Sx6/jBVYSQ1/eyqOE3S1PmlRxf+FhunnWaE87AF5KBdJvTgt/ xojNchxGiG5EW7yhp6OWbFJRuXqT1MJCGkP1UkP5PJu/N5qUnzD0mroovLkhtT+AKi0nMYnDhW0x eAcBGCz5vIJSsnNvTMQmtVwyHYSOCDNmmNLD6YHai11cyeW8X3/JwA/S9yYaowjGVen7YQik2/hY sJ3gs1xOtYaFttFYwdBFeQk/jiGYtyJXzSWDF/JPRiMu8mCeqCLogdFPyW0oeVqCTLpNXthNlfyY grCeFor31+k2X0cxfiLNzWo1F9BxOkYBHE4jVF+4+RplC5pg3DVt5MXRUkcW4dUN3qXyaAvOmkfV ZSNZHYBBn1I0Nyk7J1CyGqTvIMTyWLV2n9PxSGCt1xo+zT9KdGbjom5QkYbTGP209ulpttFaVWhm 6AfuMnMJxyLPvGWNQSnHS4c4LXdRNcLoR+4kcQ+UmSESKZux5ulTTkVXvkcPUUKrpTqCxZN06RzV yKedYmvK3TRcboEnXCVdM1nNlT5zAA3C9gXS2/05R+6ZAkoDyy1heO6DzcKG+8ZUlMrfQwo+kdu0 SrRu5gdxcXHDY2OGDR0yeFD0cwP6R/Xr26d3ZER4WK+ePUJDgh3dg+yB3bp26dypYwdbQPt2/s+2 9Wvj27qVj/czXlaLaSgmRCQ6RmbZXaFZLiPUkZQU2TJ2OGXC+ZuJLJddpkb+3sZlz/KY2X9vGSeW ef9nGferZdz/LMnPPgzDIiPsiQ6761SCw15HE1MzRH89wZFpd93x6MkefY1Hby16UJBssCd2KEiw uyjLnugaOa9gRWJWgnyuxsc73hGf6x0ZgRpvH1F9RHPZHEU1ZIslj8K2xCE1DK/WAsrVyZGQ6Oro SGhB4FIhic4prpTUjMSEzkFBmZERLorPcWS74BjhahPuMUG85xiXJd5l9Rxjn9riDSrsNRH1K1bW +SE7K/w/tFcNVFTHFb7vzXu7RFGx/oM/S7aAgohoVSRaF3WpShKjAbJwiF0IWiMmmNjGaKuhNdae 9RejqUZNTWrSHmiaFT1x0Ur8iaBt1WpKPMdU2x7tSaxFrcfYmBKm3519b12wbUzPqfLtnbl37p07 M3funRdb7i4vLfEFRWkRz9E9DfNODvZZcqnvnS6Mf2WSb2W0NEEEvH2fdHE3EFjpCu6Y4YuWJvJv URFsQFdPyvUHcjH1at7EvhlwhN3npYQXNdvtZY5/nit4n3uie25gnh/nER8I0szFiXXx8Z56+SeK 97oC+T53YnBCgruodHL/XT0pMHPx7n4eV7/2kvShu+K6h3dzV9duViO2S3RjdkSmWmo4t/JmRrZT Y4/cUxEFQdcTLnjic2MhWfwzO4sCT2RhGP4VadAKluMYngzeN8kfiMtmPusHzaQ4tyvwCeHY3S1/ a88ptTiOpLhPiJscHJH4gtxuB9PSgqmpHBfOSThI+Ph11R+VPvS5kH7VvSDOBYLto0d8UCvKzsCe Jybyqa4KeagMnWDVDF+476KyhDryZKQVBXU/Sw7akl4FLKmyJRF1vxvhu4f4I6BXMCY58tctrncP 79zsoNb7v4hnh+V5j7rzZhT7XN6A39rbvPx2vbA8KyKzWsEek3wiQbdaeoJQUkRiSWQwd3yxQSMJ fw4VyeUhZwxCUXE0V24wzj8l/FvUKTHxHpVC8jprKXJHzXIzmJ3Wvv9Au34792IDAg4byXpefnEg 0KmdLBdpJxDIdbtyA/5AaUhWlbldce5APZ54wcACr98+0ZDctyohmLu6CIuYq2Wnq3KNr6vENi89 FlPRara2xdzCt4szup6LasdYrT+3dBs19Ioo0ZLx4fMSUOioofWOsVSmvaN1gWy1XiMTDdLuN+dQ g07yCnijoefVx8r9GL8EWAq4gMmAB5gKfB/4GHgYeAA6S4CvwsbLwDGm4Dc6S6jUuCi3AafNQgqY TfII2meAk2YTrUX/15j/oFgj95mF8rixUDY4auQBtJsgX4Jxp0DZxmnY62ospHXonzMuaoR13AZ/ EXgh6LWKAdRFH0vnxAA5UvgpyyB5Ta/R5kJvODBarGEepYB69LFtWyE/jv4Q6PjQ3wF+T7Snw76b xwHjMGYgaDpsp8JuC+T5zMfYoViPG36HgBLImsRIWqOPpBYxUn7LyKee1rpf5XXzmu01Kf/DPt0F 2GXbnmiE/buDO759Ic7Dp9+DPgtkYi2t+gl6y8ig+Qa17XX0pFUM51mce422BYg1yqmfc4DcCB+n mHtoFPqMWUAx9G8Y2+QZcZM8kKU5XqYN4E/RMxFjo6hO/y5dcuDrFutNx3wmxwn2bb2KhXK1bzro QOMv8j20uZ/kHKB1svZpG++Ncw2lQ3805roKP1qMhVoA+A58qwOq2R/Mn4E99+Pc92qFbbWw0x2x 921gGNa1NAx5ETFcDV4Oxg2MIXrBmudMFD3DsRcN63xsnLOh9r4GL7gaagRuwZdk4BCwFHofgmaA /zDoTMRiI8aP5HhFXFwPx6Z8i2MD8f478Mew72oNiG+OsfC90Zboc+inQCXwooPoNQs/wBh1Xzhm 2U/LdgvHFseMTa3YOKrX4t3M6+S4sqi6excoRfmAtXNsRSjuHce+opdxp5luoqkcs2wzQptUPhjH 9xFnmxyhlj98P5E3zit6mQqtWB9nU2svjkToGvkOZEsdfWi7MRKxH8IdSKHe4gZy0Hns4VM0je+x sYle0VdQT+cVysBZToetLR3oZoazWZsHewexn43GCdoCutlo1u83mjXTrJWXjRbtoFmrL+P23bQj 7LFMGdGyL8v/X6B/YNbiDV8r/2o2S2k00waslZxXtOGAy6bg1wFVQGpMmrY5pkILOQsoDnFzE6g0 PJRtehBzB2mC0Uvl7yTwCxyE/dcoz3iVFuK7tYso0JIdtTRfFOCOYi79A1rOYPugCyJxFI61bJve FUsWteO1Az3GOZ/zrk3V3UNetaivXT+bvsa1gfMz1wfO0YxwvMrXI3G5ATXkwzvx2T5O5e2o+HwJ NjM7xmUUvcCUawvnd64tmH8W5t8OW2/w+lV+RI7jHMl5Dnf+IXt8RxrRr9EOID/sVXn4BBXb9xrg e/4xZA9aeQR5mPaofFhJjzsKqUiMoYdUPppCs8xT5FI1yKqpRp38mcpluE92LVV1tFmui9TRAfJG OJ/JoyrfHJL1fD9V3UT9NHdoPczjlKDyykL6lbqHfAc/pWzMVSB+gZzbKheBlynGI/eCL65SsZKd pUFiEfQMuZFroniKklR9PCsrxQSaoHRXSI/xKer2m6gVlj01BtTchJjEW8Dhp0MqFxRzjFBXOx/z 2Tsr5H7nLHnYUU6N5iNYTxn9GWs5ofYgJBvVPrBuH5nOe+HMl+vFLdmGMb9VYJ0KWa/2A3sUvReq NvObAjYdlbRJ7QfrLKePYnzyEsOcS885PsM8mMscj1oyUTaYE2W1yq0O1LjFWOcg1LZYGs9x73xG SjFInrTrsGigJLFUvmEmyJ3YuyEWP4XzPr9J+L3Bbwjzl1z7ZUjpnMY7rRN5GEYK4rKC5oidwErq Zu7EWyQkX1RvhWYaLAxZI5bifRN+n/AboUDdl0r5pllHqXzHlA+Yg+8+zuMIcmkhckmOc5X8uaHT CMTcKOz3dGA24LX6hy0cCUM7FR6j6ZDn6/+gFrRL0P6m3iC+pzfQGH4Hivdls3hBHtWr5TLxGB0Q J+VpvT+9q8fAj/fkZ+J9KtKuUaOookNiKt5NC6hJHJOXxBF5Qe9M0/RxcrvYRRViuTwhnqXp4mnY W09HxY/ldbFWrhObEKOf0GHxG7nCyKJ3jc6wdYEatR/SVv3vtNXxIMVhvhxlv4rWwX5vheWIE+hF Q/lq426fy/QkirX8LW7nL/tq+2n7aPu3FrXM8o/XzXaVHsYYU2gakfwDkBSmbTNwJsWc11XO8iL3 xCAXPU7jII8n+vwGsAftaoy9BVxC+3kggPaPgH8CrwNPY9xNmBkFDEL/G0Y8PW/lmUqMTwNvPgC9 z8+gPxDtLLRPAP2IWj8CfQaYgPZtAPxW00IB0A06GCd5rkyLdw3jtwHn0H4N9NEwr3U32l0sug/Y CCwDhqv3a4d3yf+B/tt6dK+0Qx3K7FhTvhT13hNtV4Ps8/8iatWWkruotQ/2OqL8+U81rx1F/DRY /6keKXj/bq93hCcEmjZM0brBQ0aEBfHJI6pyeoj99BPgbeAUABv4HQToYr/+NqXQIAyur+udoLRC dRMnWo3RWeHG7tT0EX/M6YTH3zVAFyFRT4PDWrsHDxtxPaczGBrM7iUNEP9iveqD2jiu+O6epJMA wUl8CWNr9cGdAdkI3QlEMKATINxaxV84GeQv3DZ24jppnGCSprYjk4SZmEmcmaTph9vababtUOOa QzIgbGJ7pjPt5A9POp380c5kGtehbvpB0nZSD00M6tsVE7dppv2nd3rvd/ve7723u7d3t4KnhQrT wnC6lZbEKoUMkoQ08oBsBTkMcgPEAp3JoLdB3gfJgZiQUxhL3zxJrwhn8SF8GpJ8E71sxXoRTZlS ZpIiKYEMXCYGwrlruDJddUDN5q5l9q86AN08gQeZ4TXhWVzJ6sPcTqRDmp4FaOCQgWnhqKzNo1fO 42oPx7Q/z67SvjcH02KA3CATU4IueOug0N8yrUq7Oic8zU7UakVTVJd3a+4sjHP3/UB4LyM3aGXQ ZPfh4BVhGKZklOtiZguqEvNt26UWMdy8TfUx3NirFrMUPVoBgG5TNqpOuaufk9KqxmLS9ZqTUSPt qvMyJGxHWu4DvVxu11xy0y5VkpWwapHrtSKon80t6zXyeq2oNaip35LPyZfk12WTWW4Gr9qiVrXW tba0Ci65EhJerJVbZNOcMMxOJFuRLtESyjpPv0JJIW3UYFR/zVA+7BPsRBRI5TQyIE6IZMAyYSG+ 88C3nQ9C4bf0gvPUp/r8ge1sSKl0ncbBxyYllXZ7INvvZ9wBTXXDZLCFlZr+3BZVVdZpsYLce0IK wZ+y3CJgAPCPENKqeSAyE+1R3QyDraqTZWrQeBMWJc+vaCbW/OymMEOYSA5ezQWgF63RfEqjqvoU rQXqL+oFChS3KdVedfQKlMJCip1IgYE10Ai1BC1Ri3CWTJCr5A1iOitMCFeFNwTTI8B6URCoEBSi whZhQDCXxJrIAtzcAdBnQd4GEVAQdBTkEd6agDWE0RbQkBER8E6Al11F2SrmnoFPeNjzgYW0kCYL cBpwQhZ9VQSjRqxjgjGyYYKsqLISXgROh1WPFZKjxI/CyI7buY5wXa2vCttfDNufDdsfDNuTYfuO sP0zYfu6sL02bI9JpAl5kJ1UM43vcP1TrrdyvU5f5bH/xWO/4rF/zWN/0mP/ksf+eY99wGPv9thj dtyBW5AdtXPdyPUapvHSxZLeEmS7ipdQL7ILkzC15YiS8rQSpllSllaiANa0+zKNVRELbNYweM0g 4yCmFRQQNTE7RiH2FsAfIT++D/BCWqmnWfyTPIyznLFyPIYUFoV/iNxYBvwBGuftV1GI4/dX8Ltp /0MQ9h0GMRv+Nuz+oQgU0HiRx9NKA7gfSoceozEHPgQ1mflBVMNpcVgiDKMrYf60+wydw17kJqyJ LipP0iWIl9P0Qy1rxWn6j5osGU/Td5UshtbvwHc6TedD0NIL6TuheXozdJL+UskSPE1/oVyn1+Ws CYgzIU68oPAk591gBP6Z0F76DeUMfTmfe7SGk56ByRzXS+nTMKQh/zw9DGnu9z9G9+ZT7fHzHtx7 i7f6oD8AWzRu3KywxKV0Y+gB2qOM067Qddrh30tbKdin6T018zTi57Ua/Dy83g2Dg57U+cfp2tA4 vTcyh3+GRDwKEtAbxJT4qHhQPCAmRF1sEZvF9aJP9IplVqdVshZbi6wFVqvVYjVZYZtpLcvmbuiw 4cGozCIxsJiYNvFriTANin1SCLYS2JIZpUKCJPo6jUggkRVz242WQMKwbd3VP4nxqSROGNe+iBJf 8Bi3+/xZXLBtp2H2d2LDmUCJHZ0uIBvkuSxGO/qzOMciRqoNZ1c/fLOwPvJCNcPkyAvJJKp4POqK Ojsc9/R0f4rat6IDdw9X4N8P1xrj64m+fuPcmqShsovcmmTC2Njn2d0/S46To/HuWXKMQbJ/FveQ 4/HtzI57upMf02BBHQMaLOpjeVoKuRkNVneK0/bmaRSigSYzYLQxRDmN4jFGg2XGeJPjNN49SSnn mA6jcc4ZNx3Oc2TOufUvHLOEbnHOLbPEy1VySk0NUEI1jDLpqwHCZI2Pu7fddfvz7uN593Hu/vJd t5Z3n8u7z4E78H869nf+L0b8YF8nTmztn7SizmTX7jxWSIc7+DpwTLWfqL6EVwu/RoWBpFHg7zQK /Z0oGnUFpDYc3GMpMixgE0EYfYPX9VT1JROCKWf0IjDbV1zrY+tjzAXLmbmKwVyy4nI9tcELRcZW XBKYHVAE1nFDH6zLQ3Gjfh+AvzuJXPGD3fBbgUE4hoaGBgePDLEDApS+hNG+bWf/pKLEjap93clA 3HWw+8h/GT9KGPUQFGVBohg3dAgaHAzwuEBgKH8BudnlJ48jeRunosDgx3bM8g6yLAEMU5rN/Sbj Xs2/ulMBzaUEtFn4vzU86dQYOYkHj7BoyJXPMMizwvPN/lnBCV9LEUUvErxoEbPkgF6KzKZFARWI pkWMqqwW8yLsO/GmjO3Vt+Cu3G5batssfdDWu9SGonAt3QEVavQ6vA4ZFLxK0B2PcO2ObkYfIY/p GrxKnl/eJIyaTyEJtc2IVoxLbQ5nFg9nStCPcBbHM+aCDbYsvm+qpMtc5eicJRJyBaBE78LSvLS0 IN2C5RBqxHuwqKxVmqSIE1daRAspL8PDvifCta/M4NOu2aGXLi3vlN658Nwo7szhFlzrP3nuzaWx 23OwY/UubyKU1++asUF9h63U8Sn19YKSrhIzNRPzKujeu5NbWS9u7/mPbkQqKivKJZHgtc2RZmdT GNf7ntDqXskuP8C7YT5aehO6sfza8vLPl39FR3/8JklCN6Aff1rOYQldRGUoqBehsrL3o8U4VYyL HZfwEWQiUzOoo7Cq/OG/50f/5/kFFNzz6IL0eqhRhEpN4bUKm4Bws6ZWlJdZxPJEsMZiahTEDbV1 0edHfusL18rOEmujzVER6OhUk9MqzH0z/irZj3Nwj1dn0DMEZwXHtJlUmR4+xarM90rzKLgEwxK9 EbK/bunDOpwbGYG4eO4P0NcMKkTu2X9SXa3BTVxn9N69u5J2V4+VtLKklWRbu5Ise/XCNnZsa6IF ijthiD1teTkgQqe8wdgOGDoQ03YSxxNIKSShoUBCidOQ2EnJVDWVHwT/IGkmnXSambSZpDNtOjGU lHimTQVtIJL7rQxJqh3de/fuanXv+c53zrfAkGO/MvBAgj1jrNd8d4FQbsEvm8uLUWR9YSNqW7qu rq0t16a38NV3PTn3N2IB9Alq1+Qt6PvkHCIEdL+WIiJF6UMK8ySLJAZnYWET+FNA484/AP7636Qz 6SEmoQ4NXC4HACuYWIqGNdQcc+TWd5hXdbsagEWuZrYiH3pbk4wSXkVtofZSe8hFCwOuJ7ix20ab bBDme3ImIzYs4qFmMiIJD4PpWeGVygvnEi4C+QUbJsgUFWy2q2CSUPR5J3ABScSisT4fYkxGM8KT JApskig+V2XDtjyp0XgB+2EOu7Gfm6Ci+GPYAkiYqqZVYRqrN7Mz6Rvp8k6EwuyQNaEOCJdVu8Pd 4kHCjVn1C3wDCbNYmJ5v9W1mUR/GBpdY4YbYNJUPZ5kGEUU21oTJ6tLr/ob1XLFo/bZU5wrK11wU iQSlZgv+FrP19pnvxcI1xnCY4h2ViX0kfB8vOpWoObQesPrG3DVyFrBKoHfGkTg3rbXb/ZlsvDu+ JzIQPxI/Gf8F97LnfHySmjSOceOeS3HrevQgpjaJ/SLFUKzNXEsMpIKExGfjI/HX4zdcRloURUqc IEcgFIUcxlZ5ggCqUNL5LPwUfgZxFBTOcGplVL1ycxMLYnDwgmbHSTu2X8LPoRTiAGae2IAP7lyy CldNASmS6C2yDumyky0UZqHtmy3ovCvMZGZnHS3J2ZnsghTK9vXhvqyKF36FT9PdLKnQW+Dnncxx V1IN9fpNEdzy2b5VG7/ZuzrcOPzdvU8+fm7r9qO3Dx1cpDaEJUkYaA8/0N85Ql0JhHcs39a55TC/ Z9+Pd64bWaKe7Ru4/XisMqrUm5h297t7N/w0CyWTBpheYZYjDirxDk218vgAj22UydyAmuhmfjc3 xAwZfkf+TDiWZ82buH6OXsHhzRxmdPV2exsN0GtpGGAzBQJsIogxmghvsJgp3oAY5qqBh33wHMve MvOimTWZeRPL8WaLkSaQQDYrO4lPwELM1PAYMZkYxOepMxrHRTlEG6KMhUxS5+AyRVE5xHJcHt+6 wLMIsQyTJ9ExFh7NshOAvonCY2aLxczzk6QWsfA8XuMN1igvGNwGK/fWOL7+f9QuqPs/wp4kxEIo zpRNW93/L5hQyyNBvwbdP6GbQRlVTwFVmEV2uB984wrWU2DIlFAZyIehhEdPCxt8IAFU9aG+vizE FuMGo0IUZ7AZExwkV0rF50MvbHtqe6mijiSfLI7jY8zyLx59uPQ03vkE2VkqFR8DremAeHwCHHej EFDrDajWAN5tQovRvsM86B700l6m1dJSc59lWc0KvBH3Mv2hA6lH8KDnkdBg3aHYGf6U5aT/VOR4 3XOpEWHUPxx6OXw+9RqeMk9ZJoRL/uux6rDHjGhiK3dq5ecORv3cYAmOIptgo2wTUERH8fMab2Or WIr1LkiMkgn8GfLBnCXj2uD6oesjF+2S6gHPCnTHaIo3QfAKep8Fjs9mZu3uljJAuhwg4HfFV1Lb HPmaEzTNsx7Ewl1hNOiaMc9xKreh678Xht85d60r89tnjl546u2d+3Znu3YF6gLHfza066HTa6n/ bL2wdvjfbz7a9+Gm7iPtg9Mv9uz6tRB+qXtzf2/XkpYVM/deH+oe/HnPinHgzpug5KPMDiSDcpzX 7Eav0ef2/d70gY9hYlahkeTn3ssJ9ka53Dv0/i9aRPQ0XqI/ZD6oIqfpc+QV9kWZ9lavI5uDh1m6 2hgQz3ItEgEb7tNcSkcta1yfDGQCnQESkFL2N6xL3Hms5WJlo+7R+EhHMtYZo2Le5JlxvHDeJcCg QRkKxZl5swZxuJopzgBuZdiy2XKxk8VKUxO6qw46Ql8iWcbQaKAN5GtOhrdGrOuxI1bVu2blwspg SlEWjD598mKstTS1f21dGlwu3Ybv/0EeK12ZJZ3dITEgtrvlRH3tmqOv9nQtXvXXkdL7bapugmob ouZOI0RvKCMXxse0HxEXqqA9JGiSOcUQMpoVnFQySqfyoNKjHFSOKCeVSeXv1TereSbIKEwoFWyQ U6GlgaXySrk7sFHeHNor9ssvye+5/hh8X/lTyBmRU2LKtSBA16KYL+lPBugazdvaGNGcrY3OsOIQ Q4oCDiFXcw4+wAWCwTzl05bJwcpAgMWmAOt3+QJ+xeVSgrIYDMqKQ3E5KufNLxQWw4rTycqIBPx+ jmNNRLbLlIyUoEsM0Y5IyoVd+bnpHN/a6MqTxb9RDsqa16cHvzwn58m9Y0ifQXdmUB4v1ixYE1ob bTiJOyGp86RjLHJYkVH1BHmArC3rSyGrFlT1pqoWrqpZPQkgoFlQDwGOTFqXfhjoRmrSlcMK0kHD wPN1F4XW9OmXZ0arkDYyQjptTKfL8gLcQLrJ9mFSzpyKBjDahvqy0zYH5/Oo7L1NZeGhq7aYbM5F 9/PFf/DuRbUBgeeE0sOHkp7GNF/q4Zf19ZK64dI+vJLZcftEpzfqCvjDYb8zVrX7lclMs6c6QYXD JHuC7ijlip8gMvcucGIcHCOIYqgZn9L2GukKmnLEUplwJtWZWtu4veFAQ2/LE6nj3LPRM6kXuF/W jqZy9Bg3Fb6Ucq6KXaYpuTkejzklMeDEPhTAsXi8UvKJkuRjF4aSCWddAjcn5FDDwkRS/gnETHZi ymmSm2Nx6Z6oTxJYtiZeDy8NuYwZmyfwYlQDLxAGze7QzWg6J3igx+6xltjF+Me+PFmqWR2SHsnX pGnpDxKR4Kax//Fd7rFRXFcYnzuz65273t3ZnZ31PmburL07L3tfNl7HXuTYQygBEkxpS0JT4kaE p5SW4jrhGYojErCwIlSqhKZpS2l4yEF94DcEkrShkapKFKWpovaPJJVcAgUDUZGqCnvpmRk7GGjr lefOvTM7s7rf73znnFBTfQIlxlD3SAtORBMt3tOoG0VmMoQVnpnODPQop6gEPFIKtVvfGtSi1tgz aMTs+UgO5maolHH6lY5xO5gzmY7LN7sux61cYT0Nkg2clqhYezusdEJJbn9gzSoO7xXfE7Rl7nz0 1wa0X89B+zVQjKtjtz9raXkCerIMLLbB4rBipI2U0QKrqBN+p+mrT0VKc1I1JR3+7Z8DFX/NnKpZ xYSnIn2nIrMdJTxtuOAtaNZpIYDGt7/zVP87O57d8Eb3V39Z5vxLJSMUM/6ZbO4InZlHLvzh+V6l pfzzbz34489fOVadd+vqkt4lz53N5X+4cu3YulhIpf0hSetlmjbUqZmp8/Rw7/pu362VgTOHd+xj rLp33+2/uQ+Ds+jUflNRgvMr5wefrFhf+ax3c+UWeU/w1eBxaoQa8vmP8e/zdAWH6DHUYWJWOcA2 6tVMZIwOj4bWxDBlRTEjD9C9YLQPDei9VrQOCSXqZgAAMXnJXPhoUTKF0iEJSWuNtTsd67UkmsqA /45DaE6Nt7dOXA2O2wUZ7BqT1vOM5bae6YzU3FjhSqcU8FdegUrMXnIfXqu43MqijfMH95xYtezi WN9fOgsbyzffOnab2nMNHfrT6h0PxGJKnfuZ8uKNrd9coD+9c/zMb3535fkXfnW0b/L7H6MjNwqC UACPfQ+62TcgnhJUjvrkFCXdvmQ2hoCcFeI2fXumTx+WK/xCgPiBToJESZKFCFStkXTen80j2s8K eSMiBGtPM7uoCgfRitMoShWgPsbh0qYCKogfSqcZREWYBUM8JyDBoj6PhaiQv4/6Lht5ARCH5wjT qFujWQmsC2asJNzLeyeYXcdlB3RoNYFsy93c93JtMZ0BIqtdfEQI0K57UHQ7LQPsrWI3p8AhyvrQ SRRG+Kcru6+V/3Fh6rf+L4tGmCg3pOIS1FH+a02ET8z9CfI/vv3Apx81AYPfK3/++ku3Do6sUGlf iNTtYoqrmvU6bRJ/VwzKbjzP7EQPX7jyd6u7+wB2nYNdb0b95jEzPhSnX4q/Ej8aZ/ZKe/RXpYO5 4+Lx3FuuYX5YGs1510ubJajO3JzALY4zjaZYckG0xVPhSFtCRBzU/lwwSHmygcBTLPEQBRJUvtjc /HaB5CsW0fQqN6kQX4xGbySI6MqirJohWSoYlCFJKYranM/S+QDHRbJ0NE9YpcVQlWBFv8dM8+0F D/JI/aIZjRVFIHuULCyKB/IHcnBuViWk4qHc9RydS5ToNyFo3+T6qdeCltSqLbWCFOu+SiFaVEC1 HmtuG54SVRzDm3uX4WUc9RXH8JQxhwJlmgJrtAxPAQhmAXDz4s2LVgWT6bg5mcmMFzoz4zMsTFh+ 13o/EJ35iavQPaK7BvuKc2pnu71wPwr+3s54TsqDyLSDkUnbfVA67aS/RmfxbpzSzTX2bbqma2n6 4Ou7erdn1T6DTz/49Au7w/HQl7a+d6VT7bl1yb9UNHhRvSY1dUR8zPnlqieRmlf/Czczdelrm8pC WzZTjJfb56USQmDfifJeAIuXanczc1YXtYxaPpVPNin5GG8RdR6IegKIKqBFw5EUF2prsDZ/N9Qz 7kAkcFA7rp11DYdGNQ8KBCgEegMvfr/FSxWJkip9Q6GwyiD6DC8RTiARl4pUWSEqxXEykQVC5EJe pfP+QCCi0lURltQbMgFWKI/pof/o+cRDe/QDlFavmdoybZPm1hIN1IecRYQMadDHcyRJCoTpIYhY JNRjeHM99FTdJydngeBgQBz5ybT8ZMYECOhP7kt6XR0gfdcX0rf+L+mpWTrfr/xeUH7nOUiBtltk 7hLc0dupbO4orc0obQv9o58NbN3Srr1scF/5ztDm6uZnfFOgcMIIi+p1kmhc4nNJa7TKxSWjz+2a +uyRbWW+VX9wfnn9txXV8Ki2a9T2MA2rWxIqr5b724yHOoJe0PZdimIWgbYy+sEQSnGlKqtFMLlw G5XkqunHoq+FB0WmpxrRmCE05hEfCxMexcGwQ94gCcXicRl7BYy9fIimEYuThhcHY2foMBUHbbx0 GIoXq+Mq4F14P3bjnlgJAz0DmSZrGG1swqamF61zM2o09eB38QX8Kb4Bd4IQ2ATYsKVnEuMoTtqR HZ4V2RA/tliWrmYEm3xlOzYFHxyq/HCIBtrtJ2MIemzKvD0brA5bo40AnkbAXudkexzQS86XIul2 3MPFnYuV/MzYZo9pe95jYuAGm4ZgXz0pOOzc/eeQdIcfu0YO/JfS+A5Fdp/Ulcmo9yPh0aAO9tjA VKF/V059EFguZiM16atyoqnDR/sqzWSuoF6pjjertuxyYRszZ10xoYRVFXHCI7smz3cl4+kwaP82 1LsF0N5A2GzE4ZhB849V7df69D7jCDVKjYge3UAsTRzRQ3habxYLLIvT1cYYqjBrqvMIpEesoacp Vx1ba2A2SPVTRtCoNhjjXF38LHCAmQUUT7EgZC1mo2ytHZjn7uhohyXraMJa2wuasNOuzPY4rmxt 44QTj9O1J7QgkJZbgxMTodL/2VPbaq1KchhyhdAWH7t9aSBUghbp40Hg3AJ+AEbnFZ0199SROux4 CuolzxeW7HQjjcyJyXBgmVgnJNPX5eYlIa+PrmU7lyb42iuiGGrdvX95W1FamAANKnmS38qU1tVr dUhVlVrx5ck/P6nEqjhFfEDekrMc9ixFedaAEg3o/GCcRsji7yhw/Dj6hkzzKbaJLBQfJl+XVpD1 1FDNR+RfxKtL7xN6HdlDRgijEZRLBUNtVD0cZOusIVdPGhjsIozdjGhZnWi8N0R4NWmZLkK26SKa QPR6p1sUeG9aEgVJEnPZbDopC8mkzIfDkqaqhEhsA8XQNEUTCTGJZGIOtCtyMghCHhmyGgW7eZh/ vGg3EzFStOfyXHs+IPyH8qqPbeK8w+975/jrbOd8d7lzLj7bd875M7YT2wkYXHJMlK9k5SO05Stp KaSkoSh8CCjQkLQESIYKG20HpVSlFFC7pcu6fBAokLF16zYhrXT8QytNQaPSkBZtXVG3icTsfc9O YKyaNJ10vzfvne+Ue57f73mealGXa4cz7RM7xJMoqVwiDoIU+te/CyrgVo3yarQz7dVs9rS38ABv 4YG4ak70JO+epCh4BdGbtGbbdOog7tATeouN3dJVXl9GcR5F9EDZBPOjEE/pwjHJkwMGlE+LJoOq vmMu7HwLhf77PKXeekQpdrsQWpKCTgCfMJcAMoWNyOTJJcbJjsX8KSRaRDDoJ6fmvk4rGAgEpxE9 7SQL655n5Wm+u3xZYKnLMvEra1l9yFtVMTp+O9Txd29NC5WbTZWtrfD5YVDOLrEW1d+9aJijmkz2 x9rGT9dFg5ykqjy94g2SufuhYdH4+Q2qGkQqkCx/gfxacZlUzLlP7t00uhDnwrBB22ZiBSbE1TAz AnPBo8y8khZiJ3HaRS1jt7sGXOReCCnGJlG6/QurISlsJSyS1c2XSW7MJezbIcGVYAaxDMdCwDJq ebmfx0OqJExRVismjpljLTwbCTFsCU/DBDtMztI4TpvmRvY7WVqrcZu4Tu4kZ+CGyVi/BbyBPZ1G 8fgGHt/AYy6xWKJ+3h+OpfUqletVKxWra/lFfAd/mO/ji/g9EQsr8ALLRx4YNYgv0SmewLFb0W+w EuduTTFGn861/8kT8/9JD5y10PCG8IEBEsCJywT98CGdzwNeB9/9kzs9z2an4IBtjpwQfHLuSnnu ka/KqlZbc0+gIRPmpHJoD65oohDSN0jxmRoVjxJYXCIntt09ZXhhvP/pVCCo6zwjVewh+7IxUgUE 2IWy6GWURWmggJ8M+hQkgQIW+PlokS6d4Z6hzCmd765TlhFPOhq4x0tWSk2e5pJWd6t3O7vL/aJ3 H3dIOmp8jT3hPi4NlIxIlzxlJoeZJZwpQIops0XAUdRpQynUpq2utmnz1qVtzX4f3nUZNHl2rUFD KmpA1wz4mkErrTYMQ2FoEzLszatdUbrxGx2M2zSut7GxHkOdtRnjkkTU4VCaUgI4ntaUC4rxfm7V gxT51YFPt0/kWj5/6w/N53LQ17lu5GLd6iNHV/U9tePUkaIN277c/XlOHj94a8NluPVf+7W1N4dG Pzn0xcqNPfD94e5rKKH+HingP1EPuJEGprVSI8ET8+Ru9TX5TfWM6axvyDQoW41WGMEMewr5XbMr LM+U5xY9Gdqtvkv8WB6yX5AvqxSv0BmnUkzPkkJWqxQKUQyPBBMIbgk4KaSaIYry8ALqBcFqlryB ODCnvF4nIBin2Sr5+XBI4Gk/jrkGKPRHQtcp7GsFgh1gOnnID+uyiRJkXjZfesDP6rrJ53WTL+gm qp3nkG7yyM3ivUEmw0+a2vuudjLXQiaTKZha5GmdGSbzrRk3fzY5UBOZHFl9+DVGweZGKCcRSkaE UnCK28mCeAZZDFW+C0xQI/jPPu3KjR9t+sHzgZpnrRNfUq0NlaOhzJpfb350Y3/zi3vmrCmqH3x5 /S92KrlDXRFfxKiqC86QhoMJf7xo4gNp5cCa5u1OgFC7hlA7hVALIL38ntZu5i3BbGQ+qI8sjK4E rWA32OHdGfuh8XjsR5HzwkhkJO48axwwEUY37+6JkWSwqspgY+2SjTJYJUrkSiUxoKhSoMpg8LAc x7KcrCgeADk0JWUYTsTFcBxCIBIBm42igFmRITBUsMkQx9IVGDUP/vCxtAcD4PbkK+tCFTM9BVNV 1w0YUI5gB1lsUdmC+2R1A4kgY7Xi0llswV6yBfzY2a4M3htiMshE60v8FrbwFv0SeksnC1nMkCQa eGzyYWPV+KARLWCv99x9/O+LZIEDzJSZ0rXxf868AzjVFEaeTBqDeZV7KMmYpuVZYmJRB5ggZkWe FGTDjfrF21BEfMYy8TW1UAyzPv+Yq26uDV7448e/7euqfHoDNbFcS/b+pr3dV0G8Cuncs4unRVyM WVVJlGgSO8jU0lhcg+p7PXtvSLmtR5YbVeILy8ihtu1mhB5w3LtpeALNv5nwcW3fvoqXEkSTvcnR VNxqb3O0FbfR7fYOR0fxbroz1hk/YX/LcaKYDoGIPR1bFlsvr4u9aN7t2BLvMe+P7I+9aTvuOE6/ nnoPfGDrc/QV99Jn4u8nzsPLtouOEbo/PpS4E/fw8SXUYluDfVVsWcJo5ARuoW2+YyHdFTcWx+xx gykkDZMezRpaV+L/qyyXkMRHMA4AyKBNpymVTgMLHWWsvb7KykqiEt066O9WfN3KMPzOoFcelQk5 r4C49JeWpXHVypRgOiHXyp0yKYuPRHsZLV7NXCW64cxuiH/4NhhFSoDDFroRaEI1uABrQBbWfNju KmQoZKzpsTtR7K4xYQp/N2IqYG6M/YUewwUtnIguGO4tsBFshpsFNlWYyNXpYAAf1emaVBIxAB0P JlwF3wYwCYxooAcx+I5Niqnvlb3d3sTVte749bPTk96lM4wOpxRxB1oUw8m9LS83wOjyjVd3Zlu2 BMWZshf+Y0FlT+87z82Z3nBtXdWSFYd+RxkVgSA9VblZWXXn8V2L53bkbr6zav2VVj5avBjhfxiA oiSaFDKMan4S954djepBO9RdMUMhQ1xEdhO+bpmgoRHCj0gzsAIZe1VnRqZBJdiEvx1p1ugy2oqm eN4pW9zWsmF4QuMAIV+CgLCyjFlUQmU09RmjG2AEpV6j8Xz1h/PV40vnnbBLTHeIb4t9yAkPE55z ikUURMW6/jy5cDLd5qGIRnFuEXGm1Z20kBEL6RPXcww7S8TOqDDbx6KFjBmtzY5n6YlsNjs5zaO7 6L81QlciilejaDE9+jFEkE5kHnK1WHplHTWEkh/q8N43rdCPVdgPKwzGtgAbU17ZkLuT0pbEbRM/ o8THIlIiAkuXbDu83K0W1eeOLZq1QHWPr/xpOFClqqXOFd8nf5nd8hz6opX3/kxeIXvQBK+GNzSX xWeWq2EX7Iq8Do+VvRo5Fu9NDUWpSmxVBBtXe5o/XfVvvqs1uInrCu/dXWmllXe1eqyelmRJq921 ZVvGlmTkh7T4Qf3EgDHEGBdKgIIDJCZpU5eQwmQYTyBDWoibCc3g0DTpg5IQO6SilECC84NfzY9M O0kzpZ1xZ4CMhs6EdPqy6LkrOTT9UY12z93H1V7td77vfIdM1/RVkRURb6qCj6hJHl/LwCDnHnJv dVPtDahCg8MKsBiXXB/Lt2QKkTRNgOLEZFms4FxKolGOuegVYm1TQM5TM5qDUKRIhGBUgqZDouwU RTmRv//pPHijRJ6q1zifT7CIaVUWBe5YxRXUSdAkRYhQgahfyedFDe4TMZh8NJYkREFsEKnviwhO HZ4bTolXyBmiljoCbW8AAKtPJgP4XresJAOHh1OzgbsBMtCYFt1imm28XsK8rNd6UceT1qo5PGke ANajp3wMlkqP1vJ5WIgeHaXjt9yZZcUfHbx9Lz4+GY/fA6T/VihJvk7p+LLGxz05AnN7EdkywHdb Br4EDJBwYxpfN0EKCQsLODMIkIYDettD9F+oGe6/0LJu80NXidT9j4gkbOr9W4Ry/9ZK+JSbH4op NT24uXHrzE8lm5vBxekjezNIhcvdzEQYI6mrSDPWCuq96y6KNVVworIq0n0yG4+7xGf2Dw30Tlw9 9fiu9nWi9IHWs2u2q3bf4XMd1LNLm8c4s1BhFgJjnt374tUr1vaf61oxNTGLvjGxQes7UNk2Upyb 7ho6+7s/jwwAdkQa557hBOEmJGTQhDE/MrGIMa8jNhkuV9JyubziqEFXkbQZoKmRPB7C3c3fUV0N 7kEPh8I+xBOESsBZT4jjnRzHh6VgJqzQDLfokywWLqbynBDMU0c0KwN4P8/8liFDDGK2e34DSeRG EsHBg2oakhymtZrSg6wH/HhOVwl4/DXuQ+6vHMXlUevFGOfmYmyeDL1VTphljVgsLI2Px28vw1sA F63jayrhCx4PI2rPfFGI/xt9MY1dHK7aaBJDCZBSaYxEyW0rqET9CJOisITbS/LNkEMfbHt+zf7n 8sU70y/Oooao4K4T49U7Bh66cnwsOz4nG04sDe7oO3nobPG9uUnaPSX6ODsj//PvzUdQ48tbds8c BT1ug3e/F3ivIk5bDbWIz6l4V0/UorhSr2aJLGoxZJWs+hx5PPysco78iXQxNC8JIXBfPtpr8Ckh 1XhURt9VjimvhymXQbfmc7aUHlx6AD6mZtU3VVIFhDivLY/otwMSy8RAMeb9Qg7ip1okmIkplIW4 4XzMq3AAUILLcUPcVo62ciGO5Hw1EYxd0AiXcsYh41bjo0b6sPEV4wXjNeOHRoPRWx3fqJdPYNfg 7TVCEcdCYRHefjwOCCB44RnhxrhuleOTmDFhYEw9MOYyEQKmBDFTRnERBQxjZXLYJR2HMkGyZAmJ B20P5d/90bdPzJ5D4eP79sqV1aFqa4J1BFLbr3Wtf2LH4Itf/+TQt16Zfgmpl8Y6srURNeioqnNa RN557HunT+96cnAn5D9QlN4A+Z8AV/S+doYJImfEa81ZQDhZ2Cxaui3J4p3Fk0wlLVpjExw2ppJ+ 1mfZw+6x3GT/aDHmxCFxqzjSRD+YFmlJplO9wd7Wkfrp1A/Rj5ynxdeJd1CevRh4Ozmf4jcQSEbo 8xSq8MCtLL5fn9SuxVLtWlSCQWXK6RSjkiw79rKItSSKch59rslqfUNiMOpsyjTI/pZ01Ek5MPco IkGFHDL4dblJqmIy+fufzAUzGazcFo+HtzjaVNkhQPGm5uU3HRacGWwa1tl4Jmk5xuLqmYaVd55J gmfu1FhqMTFDOAQH6SgJuOPXIOBpyAHeDzngh0X6tYCU9JfoioNmA1W/60d+bxtY7za28adfZSVI 7+TiEphtyAjh3lfEN1f4b3riVLFj8cUsLZMUUkfnqUknaknNJ8fHsQbHD6ADywIPxrtUov+vxDrS zQ9IjelM6FMIekPxZ5U2M2ePrI30nNIitUHlB0+u7x+YfPflg99Mr5G3W5gKqxh2p/x9maeLdzvq dwM9T/xrx7Yga+c828QdTzXUZrY99aeNrdNPzKD1EyO1TWhLzKX6RN7GxJYe19YUt73bP4Tex7qr Afcngfs+IkYUtbRVsMQ8gidGEybBRNqHTevMpGquia00twZ7mF5Tr7mHHTNtEkZip+gf06855uh3 YoKCX3u7nDJHKm05UwT8nMlsMhv8hMksVhHH/JqJzXL+gD/hp/x+S1SyMwbFYqlqsYohkRR9CtFL Ylq7eYCUP6yuzfEa/NAsj3ivHL/uWa7Ag/9YxFZ4sABszhWgLRqPF8ooETZM6xIkoJ16rTaDsMBS zFhgbGzOVI4MjuaKLD6eg1iqyLhVcizz2f0/tGeMCkBVdsz3mKc39xx9Riz8/uQLeeQ6NbGrY9PP H1t4YfzgwdSKXX9BU43h0UOtOwOf5R+dQSvPb2wdHni4vdpnq25+qbsm+TH0P8XZ4mrqBnC9EymX CAqWs7kuR+F36BiJ54yae5W7I0PQ9m5NrU5G8XmfNwo9Auy6gfLdmgs2D2y8kOzGl+u4VDiiKCGK 7OygoxIdIjuVDnBPIcUJmzDjb5ppl+oIFNXg96N59IjmkCTC6JfM1lCXqoSEdAu0D3myOOdeweVJ ShMaHBpQ7nxXyB3qYhv/UGbQvfGlwlLhS8rg7mMp3pZbWhQWl30Lwk4FIJleWOAXpg3CAt/2JVHK HoqGPwzrILFV9MOAjrQFc6TW5sxFInWbc2G869Ycy25pVPe+TJQiGZ0/5WZGB6ZkY9J6SxttLg3K RbME4TK/YCZDPmw/ub+3b8/Uli1tNaGmmD8mCozZEd/aF+bb33iDH+5orm1N973aM7ClXgopPjPn zTV2pvw91GRHsb9488zNkVWSV61KRFwuB8+YDUz6kZ01n5GvdbhXjX6nY3R0sC7aIHmFhIlnWDU1 2XqHAHBvFFfTceBXgmgnBhCrZU52nrX/wnHO9Wrn+a9dsF8NXg7Nd7L2CWGif0qY6j/d/8t+o81q DWX7nNlsn9WW7aOzYY/8H7arNbZt6wrzirJli6REkZQoWTYpWSJF6mFTsihZpGEziZ3Yk5WH5dq1 E+Wxrcv71SRNnLQb0nVpkq5pgaDFsDYdimwrlq3LmsSe2mxDMGQBunZYgQ4DhgFbgRlpi87oBhQB CtTqLinJyYb9uDzmNU1L9zvnexTOt1XRvmtJBE7KRYPvvdMXTTpGon435WFGbb32NlHJDYbxCHjB Pppmfo1mkE5EgT7XjqaNdhnXIvvkVVrnr6DBgeSJyJAt46ps9o5blLKkDN6Tgfxm6Z1xv8mTh02W XCJNd/oxuQxNqImoterFmr/PliD6S3D2Cmyhib6ZQc1kVOwmfENuc9Ca1U0OkubAwdoAFULaGC5L R1kzmzaCqAkcRNIqYgNu1tHMq32ZQRtobWIPhzIm2hNdr9GHDv9uj8pEx377arZv7uNnH3/34UIi +M2eTd8+cPrzPxa3p0ozo4cvbl2jfm1YqoU3lQemfvzcO8V9Olrclet9audOLJQkPUzYkxKz6sjE 2ZL+dTVR4eh10YQ0m/NemL7wARe6tHHLP06Vvqp945XlY8LR/tWJwR2l2FofDj2UDHn0Z3Cmc2CD sZ+adEzJl2V0d+vu9r3cvthc+xx3SjwVaysje0VbWTW1XaXhAsAWTySTCM3kRnpmJVXJlUAkBXoQ xIHjfDDEBIMhJInkknyqh0mleiJpuyOVdPqxYF4KBXtSJHOehjp5HXcIoSqIXsOFoCmQSRt6Lfd+ ynSt0Kym6hHVKp2qtQt13arxrFUNSlM/TYFUIB9kU2ww78ycaeTdBvWa7LsIIYctYeWVBgMMDUDJ bKlLJuwBfwJWZMXZNiXzadL1xG2YWCwigGJZ/EUGeq4c9FxvhLr7YSvAljHagR8v9MAFPcDfFmhm kOmuU8AMAFRj3tH7muqy1TWVrrNyvY0cjgjIW7cO1TZT+3Dh3YpidJ5kPRjhKfTz3XPT3UJv5Igv wHQJwzP+s/Gg8SIYiyR4SvC2XPhCBdSN1fnVW2uV8TYXRSTX0+q30r1C8gR4vphg/L74o/xf15b/ YD/xeIfUisZM9Xz4y49sPS0+BENkEDVi7AFN9T2mqVTCoNSEEQhmZ3AQiAAfK4+EZyVJkUs4cqi1 ir5sdOAOCXfjspvnwgzHhYMYF5fCHMme90FAb7jbD6F4Fay7hm53V0Hkl/I+ijOCKmeCpulZrgGe WY12iCZn8KH6HU37sgr3HGfjAnGO5eLO42f+K8fWadjAOFNjOYOAF/NlPsKqjZQ6U1pcXrxLNtC2 sH7AGiH3lsgmuGZeqVivnGcNnBkiLPgKLETP/HHeXWANd6HB5JDIH+Bm+gE8m9ZoBVAV/D6u6bKs a/k7DE24vQUtMrxleFDOBp4M8UHfSItPi8u6Lse12pHlNetcJEOmyuyudWpaEKbArQOdvk7MxKj2 w9pa22CLF2KkIB8uUBnDTWYz5lcOxqxqdBHqNmKbsFncHLuotHTIQBkRZ6VeRVkBK4w7cAJXCF4Q GUEQfdXuKOkW4PcS0pIokBArK+HE1edRYKAbURs6ncGraOL6BhFAs/yDeZFbpLWOKqoamPhJWmCF 9MqIkffumRRbWbJCS+O4EwhU1eaJw4laOfDGcVsQMgT8BC7zU8CFwyXeP+kHjxk0DCicIrYum6zP 8jn3h0cFf4nrehwurbY88u/vDq1Kh7p8VMBlQ3F805HUB+FEmPAQgRav9RBctaNfnP7XnzSJ76Xd dAeNOVrs9p3fR5HDuF9EYWKFPgfxQ078BHJiHzhiTOPdTEE1CHdWNbyqapCqE3PifiyATyDf8fyI dOTZIXUtO8Xag0JA7Eih8CV2QDOMH/J7NOYBwB6TYh4J4XjC41II3I4peF8tUQWdhlfKKH0lhPH7 +ajIRKMisAPEbvqcjCfGeDwxEBMZOx4DMHvCiNIn8XKCkeUEgbfKkpN7oSPWjSVkkuhQufN8Fby5 4F+MVplF8TdoL2ygZxHJhiIJcPNa5n3ZotRgVq5zqXULTZrcGEdLUZ2aKgeyMmsFDzh0DfZcXL57 D/Ln8tJ68i4EGRkqLS9C/hxo8medNyFx1uV0hVJNXfUjDfgRcgnAYbSu5NuONnKgbeBp8/qEabdc 5O3bVoNYcw5A3goYohiJ5MOOVosgfabM3o8lMRF1oJA0m0NnW19bfPRmP4VHkhEMzDuL+/seCU35 QjmaZjxsVo/sP6rEWalybufLYLyzRYiwGUic8o5XxgNEO+kURXtMLHaNj57+syR5xHLg3HRYBy8e r12yH9sRoP0hp9UXGyBr7oB90QVk4yvtCKAQHvBGhzqJTHZ+yn/O2p0hTMEMbCNmx7pGvLNSp9JV gtChSBfKU16GorxujOIkL0Xef3Cf8y1wE74yauCoQCEUuEW9R9moKtANJ9dOsRTnPD7eUDXT41he lIK4sa4hyqLRTGCIMiTGursu+Oq7dMQLd8Nw1zAfNMhg/ffuQOOvXCzcJepPz/sLlOFrOtemf11a /Oyflf+lUQh3E9KKSaKVBDhs8TJm/h+neQGQP9+IFBp2CfwfsgQruO2oXQnQLkj9BW7bRl0V+0LA HRYlthcGw9kZxs145Cn+jCpmu6OH0CtHPH7eIUAkIl9+1HIAutOybcR4KgCVtj8FZto349uIaWZL f0Wr6FsHHprYRe/27UnO4XO+k8kTA+fQZ5LPDJxbcwl9yfVS7tKa18BPiVfzP+m/WriqXdV/PnBl +PLIfP+CtjAq7M3tyu8ZRieQmeGJCfRc7uzw90bQRwonc8e0U8OPjV4utEpAKMTW9T50cLIl3F2u Fc1xnpQmlHIJITQHGFtFODWAFLNpj2dV2uEo30EcTCDAywqcX8Wpabw+yOj6IDKKlEf5sSIzNlYU sbHRUUhkTnkSCu6gXhwju8+HTZcUYASlajZIQJANl7pd/rtsk6u27MJBDVzVgGZGH69uRFTdCHZm D+pA3+gETmHwdf0tcAsZtaE3iq9PvD1Wt1NWiahW+Q/f1RrcxHVG792VtHp5tXruateSdvWW9bQs 2cjYeA3GYIPBxtiMsQUJEEgZE2JPSRtIMWSAYgolDTWEDoyd8ugEcNJCaWyXFjpJaWmmU2ba6Y9O f/SHm5ROVXc6btJOBtG7u8IoA61GV1f3Svqh75zvO+d45O0G+pF8dCrHqph8FOlkJrO7ba4Na3Ou j9CL6TY6sv6JwyrTZWlKFObnC3kKOfBCfgjdlzmuEoHkd00LrktdGhFl9mu2gULevACRBTMrTgw9 ywU7Lz8UvpkrkOdqQAt4jbl2tNdK/8CUM8luPadSNqBcdisbXbJoXq+1nOt9IA9xOYIpdo2w1JUZ N/op51ZXns58C2KEch7usz4eSSF46/UNjZs3ZuvTS+kVF97sWptYbHnJr9Po9c5cWnCObAz6kpF+ D4YbjKaq5LGvrl1+5mqlgxICDT/JODd9e5ohwp6KxTp8tNg43vnaIl5MV68twup9LU1L65uL+0ZI Uk9YYyvt4ePplC/1Ldi822hFwkdGR/585m9YfovAsUzoEdhTV/w9dnS9VefwGaXOCSFtm0SdUwtP Kt41XvKu4rmSedUmHYDDuKSqi1inW+fu4l+F++Kj7suhi+EZbCZoGIAD4dsQ36jb6N7Iy1Fhp1sJ Cpreqq7sYBD1zReDQlTOCXElJwDoi0EQH/dBFApUgCgLDFFQG/XE4rZYLB6LPg4L8dgzwoKE50CW m8LGrmfvxaV4GEVaF5MJHlM+lLd6eZMkLlaKDzFFAWPyjEQhYy4GY1J8iP2P+BDtKMzOzkefThDP yA+IvMOwRF3qCyni/4QIRD4puCLdw5+RF57yl4ocLrAO8ezDwz/+xUC62bXXTumM5myTJ99dnwjE vF92sNbKUPtEX9KTPv0j3sca3UENolMO0t9fmm3YVuxvo0hrRVWv9XAuFA+m9sA3V1XZnEz8N2/3 vHAJGx6iHYJK40dpoQFx5gbiTAVwgkmxWYvpcI0ev2h9l5ngrluuO37KaPqZPudh63FmzHqeuWQh aq31zhXWNucGba9lvZXQG41mv4HA1WrarzLYpvCDooU4sKo7QxxYlj1JjBMY4WRJ6ToEJAUBIvoM iMuyQEyj5c4CwCMXLIIJoAYfcNEPGITQpwpCHQ8UsDoegKaCNGKkF2VYoNLakKWT6mfxo9ISUskw ubYW/MZ4cfbI6OQ7kDt06Orl/pWnPnuu7cRnWOfp4h+uvfeNUzB87d3W/NZi//3NO+AFZKceuYvt +C9RFXwgDbvFVT3whPG8cdJ4q0Kds68CrWSrfWVVj+YFcg/5KnstPK29FZmuuseSy7ydoJfEUyDj FQEOK/zVaZIEDpZOOeykLWX3Leem4PdEMuxN+TqAHyaDEHDJKfyE6JXaIwxI4CM9LGdjWS7o1xvQ r0wsZGvCHEvFZvCDgECETmQJidcRZQvJm2jns4Toya4loEh0Ei8TE8QdQk3M4IuRI4n+kPP5plj0 tffTWVZ0W5qk96KlEh04V2aOhZyzhqXZGv0UXv+D3pLlV5pi71+YJFV4qLA2+kBxCENowjcVysNW aZRLzZB7hg1Erq8BPb724QJWRK3Cf4AQC8k94KghhLraJ4Y/a0XQLcHlg4aAP1++62pf3/7id/6e 7ki1OehMh64Y0eeb/Q9pD+/KvLTkS5nBHeua26oHf1uNj358YPuJoT8Wc47KYnE17fCYAwHVohF8 sNvGuYnQQ2t7/fDYr7Z29vz7MkIbxBHaHyG0BZCEraJew2kqG+PtcVVIKlMeuSktl8FegVfM71gm vZeDF0NX4lcTN0OGseC5xFUO3w4PBo8n8JXOdq4X4rn44mQrxOP6eLI2hJ8BMMkLlJ4ypPQ6qE3p zHwgypspr8Ak4lRYmMLHRCsI+F0uCX4IPZRgoyghOoUnRKPdoNeRVCosUBS4hWaeAI+CMGoc621q jsKoA91ZSgyi5fFmKGXUSZvIs+iORcAykJHQZpBtZUQuywymKJpK6dPT8GPwONghVEvTL/9gvvDY AZbgJZ/AS81KAEsqbc6VcC7HV8F2SFHtKChZQVoBVYb18XRTBJeukxsUlIafBf+oOO2gbZ60vijo tzDBavexQ8Ndrdu2z7z1ynMrBmjPijW5vcV/Lks1duw5j49+fnqNgxa0xkBAqzO17IKFn62pu7D5 DFy9s3v56pe/K64rDsysWtOyAy6TvHwQNUEO4RsB/xGbMaSBAlpdsAffpNukXxe5gl8zXWEusbrD 7Bj7KIqPqs6qMLfHA8Fy4a/hSAp0QMzGYx4MCskKWDEFx0WvLaDRQCIM0Zc8Hl6w8bzAe/RhgadS OlHXqcN1M5gIkOJfj9zjJVASdI4XM40ZXoxledGPlhctD48uKl0ZwEPAj/O3+fv8HP+I1yDtO3oz ytNpOQ3Ml9ovGp19OCu79CeSpMCDrmUrVS5HX5fQq05JmoMQtgbwBasjAREKLtiaJyojX+2AfW+8 d7KrRgh6mTgtqDBCazCb2Gz381XuKg1/dpo32QT7IrxrUZGF0a+0hAJLG+Juj1Wj1ZLilnNLu4fp /diuwYTFSOlQ9R8VUJL6BFU/Bd4X/WkIaS9HNWkNKhNjsJvqQ+qwwWc6i+NJ2ATXws1QBaegStQn 7oIUofZHCOcUvClm7HcZ2uDymw3YMXAXihZDUyeE8B5Zf5//E/8PHh/hT6L63eFV/IS2PjjGHnPe ZWTZzyDqp9ASvJkJ5g6DMa9Vz8AWuA01APUvifnzSO7n8/mHaN7NFpCsNDXMFpTXvDytJDLjvqA0 hWRGowLKoRRVymetQWMrLRUugWUzqLq0dK7DPmkMqF/sWNLmqn599eShlZsES5wONAY0w1tX91GV N2pO7OZZcoc56kIS/esje1tSQkPtN98QX3zba0zAlrf29ywJext+tzP7/BE1HkoiBm9ANdyuOgjc UDMN1MjADSH3K5pyF9Vz2Ock3sMdA59C3O+qB/0kbuJdPDaCiIS5AWmCKjVBAFelm4NspcvNqJ0q qEWa5HSqVPgpMIFBjdXwX8qrNTaK6wrfO7Pv9+zsvNeemX17d9mHd9eza693xzz9ANsFGgLFMQ1J CQiRhrSNEikVJTQqWylBaVLUAg1qVJIGGoqBxkJpE6ngygQ1rdQ/aVqUqiYiUt20akJREy+9d8a8 pPZHdvTN0Tw0M3u/c77zHWTNZFZAIiywckpgA8SQn5RJ4gYJyZ0KOGn3t3znIAR2NHx4gqzeUyu/ xf6OJVhjcpWdaFnkuybXDLbtfhZPqqweptAO6ZAxJC5cQ1KCM3rO1BosM1bTUqE1XzAmB5TKnDkT 1GpGFlvrdRiYNecCrDHYQJXs0f/pnaJYbQhu9Ah1+EQ44Ba6+HXq5rXVWraqvHzQteu5TZZvtf/R XJiaDFPBaGib8LSW0DI9DxPLkp2PfQ+rBXZA51G+1uFBfZ+jV+glgpXiquL6+g72ceYJ9gRzAfyH cX4xt75vh5McYdaDTQzZA+oMoaa6asRxJ6wlmqmx1GTqE+Ya+0nNHuqr12mnK5Gs1npZzlpi6nQi KfXnSqVFL5yx14ENkKRM10M0Xed9bonuR264TgdcLecWEo98Uv1nNPJItM4LZVpnKjI9Rk/Sz9Iv 0lYaTYe6pxSX9BzMxZUXgpLphnE4jW43YogxY7ZsRF2IpsoFSZeOSqQk9DsljubQS12P/drg8K6R D1F5VtKjwabxAFR5OJ5iawada/AkOG/cicbCW6YZt5A7mEWz4GI3udlEEMGYUGwMTI0yWwdOG57h 3TVmcdrrQyggdCIsTnAQ1aNJ/J28I87vds82e7JHu9tkNwhNI8+/lvS6qa51nWPrtFIy6w2MvHrl /pye3aBQLiY9LI+s13vi+dR9SYFRd5x6dIAlH1k48e1okJJ3ck/2JbLRSHX4evvDP+jFkUOw8rDk oTq3sF/TMvl4z3fbv9wXpbmlf/3Nu6txJmVRJrVQJiXAp/rA6xCm9EAlpXsQfJWNxL3kLyzvqpau cF94iCCrEehwuqDH67PzdjtUYkjnGGiXFbcczAebQTKI+s7r/iSPPTOWtql0xVA4KVH5iL/BEwqv 83v4A/w7vJUXU3JLAUMJfN0TqjQTY4nJxJsJS+INMoqLGCg4FXJlxXyO0aCEeBfqRejh48oe5YBy FEmpUlB0hVSmifDpZPdlHhe2kRLzqMDnAvOji8drUPfBlqH+8byAqM1DRK5pBtBFMEHHkTygrUcz NkSOIZVGI8KtJ0oYYpqFqkxLlMP3JL8vpHrdE6VoWg+Iz/wodDHOrxZ6hRw50hhe88ih0c9a6mm5 kuqUxOVppXtFqZRf/d4093vi8SMlJ1r16I2r1hG06hm4UXfzjCARDsYpEWk89yY93sYGcV36QXFL +n3RmmbyUp0dlCalzeld0k55e+YnyTMZd3AJzvRCbxlH5LeXmMtkBNkIpzvNi3qBC5fF9CyEPIi0 4jOZRAIZfntYkgSBdxOkxWqzUoKUEcOyO+9uukk3YvGsda+fgtQ0qekeeEVo8XvFTAtcEaeJZ3SX 1ArHx2KTMSI2Tean0lfC+G1IRnGcylRw0P25nnJYVyuFsB4eD5Phc4jVLFk9ZVK0yBDybAsL8wGE hQlEkFl3SFabhsSa+0XC5vi8MbIuVvsZCRUbh5ZriqqhFLt8lqqJEb9RfhuRd8+gpghpg8mbbc/g 0m4jkCRDTKdBaCXO0iwHyU0rXnuo32XR2Fysq+bv3LDjL1p8oL01a4/5o0KpYwmU60GbBR4i9y8E L53enmcpZzTOyJn+UnnJPd95qf1hlTizsBoe//c2hbPFlr3cPvZUhDiGncUbqL72IaZXQoe+1YtN gR3Y06AfNkKAhTEQhdiR7xdegj8VXk2/0n+yGRhEJRjgtqqPqrPC26rVGfWk10ZJiyCKRDqdaeiN up5SI4Qoyik9lErpjTRqk1RlaWvlDKCwWZeZXpcL2CsztWQul3Bb0kJDbb0YeSdCRGa9xNyKc3AV 0GFsStybwkaxQ5hr6H0j5YbeUWk0Vile3fus96TX4hUHu4VV0zCESRtFHfGDCUQEqiXsRj6YR9o7 jxnDbC3MGbvAx/OYv1tCGpi1OwJ1B5LZwHlDQicgcvIQVxrugLet3qLPo9k7W2UyQRJmXPQt+FqP hpgzqIPbikKw697oAwm+I1Pyh2MMGgN5tXHPWlr0MmEtGW1+WUtUVWbp4fv6aymVzypKTPR56PyP hYaVGx7kOsn95XLsB/sKGwKunJrwCc5AuPxC+5UxmcsNB78xmm0mYVf7n6PdHWxczSpcIPFZ9V++ gR4ihpnd0l5JPo2YrUJa/9KhIizyvRWnIApdQr9wjDhDnBPPpKa7Z8gZy0XhougdkjZK2yXSUizk 89aOTKdYFClLIZ9bkkmFJYdatNrsSFzdHgdnqbSqMyFgj81mkp1+dRq+pWtFSncHy35Kpggq5fk6 h0XwAHeUI8a5PdzPOVLhCugcyQ31Vofe1GBTG9MmNVKbJqO613KliP1NEfubIq5NDsnogeLR4kdF cry4p0goxUJRL5JFLKK1myI6YZboBDZC6MAw9XOgufA3rKGG+zFRA0G8NzyQ5Xzd9KNwN2RZjr4l p7dr0AaQohpN0LSiJqtoIw1Wk3AkInqz9d3aF4pum68vWYhlGzvbl947+FxZzg8kQl4H7bDabX5t aEuu6qsuZXqc5P7eB59vhwYPr35qXAlQbh9dUru6h/Sxt9ubrx/fmJeTutOad1hdkeH7G8QTR5bb 4pi/v8MLForwARKIuofQABCtULA0K2auXwP5NfPoz6gV1UJ9+mdLFF4YAou/hz4HrqJXffUOzAFA hBG+gnDJBHndhBU1ZNuf7oaj8P/h3GHC/U0AvOcB8P0WgMByAILfByC0+zbYXbfBofeIOwEIjwPQ gb5HTphQ0XEUIXb18yP5AACp9wFIo5hF35xDzynUAehGZVIWAaj8CgCtCkANne/9IwBI+YA+CMDA KADL0H0r0Hqs/CEAqy4DMDQMwMjzAKxxATD6X31hYAgA+imobxRTHR8ZxaN4FI/iUTyKR/EoHsWj eBQPBGZgYmAE9ytFgH1RIIsR2GdiZGMgCJihNDcPAwO/gKAQUL+YOJAvDRFWYlBRVVPX0AR2yhgY 9A0MjYwZTM3MkQ1wcXVz9/D08vbx9fMPCAwKDgkNC4+IjIqOwWfrRsIOoxZgYZgAJOUYBIBe5WNQ YNBk0GcwZDBhsGCwZ3Bj8GQIYAhhqGSYpiD8/z9QnQKDBtCnhgzGYHlHoLwPQxBDIkT+/2PskEH6 wTxo6OMDwKD+/xGvCg6GNKg5zOAIYIT6QBoIIWw2IMsMFMMsnEARM4ZAKJsJ6Ld2KJsZKD4VymYB sk9D2WwMZoyMbgG+fr6O2iGZuanFfqnlQfm5iXk6niWJOZnJ5EkBAzGAwZfBD4gdGbSBgZnJkMuQ ylAMFEllKAcGXj6Qn8iQBwxWT4YSICsHqCIZKJ7KkM5QCuQlMhSRaQY9dUFig3kKwycGG4YMBlZg iAsAU5IjMHqauA4AQ5gJFFGME4AyHCxAFogHoxnSmISA2uEAPdrtgYDBAZj2pnGAjDnDYcBcDI19 ppmzgp4mMMfz23zlkOQAq14sv2MviN7kff/nH9a/PRzfOAyAXFB6AJsMEGAABff5sQplbmRzdHJl YW0NZW5kb2JqDTkzIDAgb2JqDTw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTA2ODcg L0xlbmd0aDEgMjIxMDQgPj4gDXN0cmVhbQ0KSIlcVAl4VNUV/s+9782EZBICBrKhvOGRKFkEQpEt DYFkAgoJ2YAJBZkhCUmQkAlr0LDJEjrsFFNAxFJEghR8oQECBQsuaD8MoWhVrGUTCypI2u9DVGRe zwyUQt/53nvnnnvW/557QABCsRASuaMKeqZM8jh3A6NPszSnuNLtWXdw70qgIAKgJcWzZ2r1njOz ee8LwNJrsqescs7ju24CVhugGmVT504+cfb4j0BSF6DjvvJSd8n7w77NAcZobPNUOQs6VnZYA9iO 8Lp7eeXMmrEHp5Xw+gIQMXBqVbEbRdfeAtI4fkRmpbvG0+6A7Tecj8H62jR3ZenGy7HPAIXNvP7B UzVjJufNT+FG/75neqnn2z3VlwA7xw++ox5GNL8x6k5EK/GIAswr/F71/30V5lX/vv8vvmHr5nsv 0IA9VIE9+DPepja2ehOH0IQPEIlMbEEtNqAOFoxjya+Rz6SyfANFm03oiW2czza0sO5YzMdhdKYo 82sswFL5EVstZaS7YQhyUYVVNNKchfE4ryxGP4zENHhooek0V5vrzdewA4fkB+YdhCAGxUwt5nfq Z+YXSGaLl7AJ52l9u/1I5ygLWfMVTMdmOUEhs8z8iTOwYw7noCAbLXRMJLL3UlyhKKqVGexlu2mY 77JWF0xAOTbjMPWlYcKujjezzRZ05hg17HUT9uEAUzOO4nOyqW3ma2YbopGEp7meJpyiY9J3Z5Fv MCOmMko9MIB3qvAW3sdp0um4qFJtaoqarj5vfowI9MZoznYnW/6Tbon5TAvkCSXLHIowxmWdH228 h4sUQz1pFI0RPUSV2CqnI4gj9mYqQQXjvZG9n6NEOiBsolVuV3Yrty2P+i6YYXwi8XgZr+A4hXKl Gs2gF+kT+lJkiIniZXFJblB2KWesbq76WVRiFXbjFnWk/pRHv6JyqqU6WkebqIVO01UxRBSK58QN WS6r5VFlKFOBMkNZrC5TV1iu+py+d31/9d0yU8xlyON+WMTZv4StXNkhtOIs03lcIpVCKIxJIzuN pheY5tMq+j010C5q4iin6RJ9Tf+mm3RbgMkiYoVddGPSxXQxR2wQW0Qr02lxTfwoI2U3mSj7ylRZ JKs4qzq5lmm/vKjEKK2KyTinqPXqq2qDult9W22z2KwvBiHow5+330m4c84H33JfvW+fr8m8iE58 hjGMQlekcvZupil83vXccW/iI7IxdjGUQGk0kpGZSFOommoYySW0mXYEct9LRxilT+kG5xwqugRy flL0FUPFKKZnRamoFmvFetEkPhE/SasMke1lJ5kgh8kJslTOlHNlvTTkh/If8pL8Xv7MZCrBSlel mxKvJCrDlInKLGWrckW5oo5XT6pfWYItlZZllmbLv6xPWdOsudY86wTrGusB68dBLu7Od7AfB/HA QxfkIumQ+7Fa9FGixSlxivt5IkpktuBOFQ20XMyjJtFdrbEMEoMoB21KPGN9QrwqvheDZDaNoAJM Eb3verNEKG/wL1V5B9eVI1zbKfZcY7HRfHHDYsM+ghjAMd+TvZREeRKfy/NkVbbh70owRdJ1sVPm chccVdJUJ+xyC/bKapqH/cLB0+l20Eru4xx6g+dCIaXQD9KEFDncRf3kl1iM58RnuM73eDl+SyVK GVajD9XiCl7nW9FDnWZJsHSiv4gKxSseoSYIZRdXN4C6k1QjsIQmyM2WG+IsZqFVCcY5+QfOvlXs ldlKm5pP5XwD5mEZqs1FmKs6lTNUBkljEKdc4OlWK1MUO/8X8FQZzzPtAN/uwzwHhshslkRx54zk vhjNE2Iz00aeEwp3UAXf8bE8xU6hyVIomlGmhhFPHUA56cvHOPN1bDLLMM1cj2SeB3VmLXtswFdY gwZa6nsBHjzGN+ccjVSzRKuaZSYLrzgrCkT9w+fLaMdRFL5h2osspKl/glf5FAUYbK40/8bd/QRP 2E2YhGdwmav8jiMMl8fQx5cjGs0s6eF6zyPP3Gl2pWCUm1MxCkeww6rCbU1MHzIkfXDaL1MHDRzQ v1/fX/RJ6d2r55PJSYkJPZ54PD6uu97NrnV97NEusTHRUZGdO0U80rFDePuwUFtIcLsgq0VVpCAk OfQsl2bEuwwlXh8+PNm/1t0scD8gcBkai7Ie1jE0V0BNe1gznTUn/59m+l3N9PuaFK6lIjU5SXPo mtGSqWvNNC7PyfyqTL1IM64H+OwAvzbAhzJvt7OB5ogqz9QMcmkOI2t2udfhymR3jSHBGXpGaXBy EhqDQ5gNYc6I1D2NFJlGAUZEOgY2CgSFclJGjJ7pMKL1TH8GhoxzuEuM3DynIzPWbi9KTjIoo1if ZEAfarRPDKggIxDGsGQY1kAYrcJfDVZojUnHvCubwzHJlWgr0Uvc452GdBf5Y3RI5LiZRuTzl6P+ t2TnHTOcdQ/uxkqvI6pC8y+93jrN+F2e88Fdu/9bVMQ+2FbEZbm8WRx6JYM4okDjaGJpkdOgpRxS 81fir+pufaW6wy9xTdGMdvpQvdw7xcVHE+M1kD/Xvi8mJv2QeQExDs1b6NTtxuBYvcid2aUxAt78 uX+MTteiH95JTmoM73AX2Maw9vcYW+iDTOn9vQAXUPdzI/LvI0v+jPSnuSEMrVjjTJw619Tf/ynt D29xf1bjp4jYyijhE6kw2mW4vOED/XK/vaHGheua9ya4A/Tr1x6WuO9JLHHhN+Fn/X1yv9V4/7+8 kZhoJCT4W8SawWfKOaYF1n2Tk2Y3C133hGv8Y/iQy9i6iwb2ZPjtdv8Br2hOxyReGAvznHfX/+G+ WoOruqrwOufsc+6FggTihUKGkhBSoBASEnlFkMsrBSLQEPK6pRIeVkpAsNiKTlsuEx7hkmhFYVKg NIkgMcHhAqkNTJXATBupU9COoWrx0UdmbONY2qGdKZQcv7XPPpebA9Ng1T9m7pdvr7Vfa6+99tr7 JNOKpBMUzBhTGtXLuKbVrQkUck3YrYl1L0tFJDcTP2QDUf+9sV+/hIGJc1bnRLWBn1H9dac+ryA1 Lz9UkjwnUqZ8m7ekm+TUT47VqVI0cVaJkaSrkp5kyFoE5dJYYxZK+kRFGn6WDOpVLT4/olJqtOTc aELZXOd/ae+UlDvs1GJf4V6SbnZTZkZzxnSXv9xN7mZen4gBg3EJ5i0JRSK9u9Uh1JwJ5ylCxNOS kpTkWVEqxMlMw6/Fbp3MKE2KBuGyWdwA8eeolNitYZIql+KPozN9bC4SXSSSm5qcGymLLG+xwytS kxNSI6f0c/q5yIY5ZW7gtNindyVFc6tK4avVWk4676nvK10LaVYCXTvWlZ2QI3c57s/saykV39MK Uf11+prYSAFgnm8ofccsohJtB4X0RnqCYQyloDhKj6JtI+QZ4NPcF+0Lgb8CU4EiYIjSLQCWAwUs o+0p7osxNvA4kjdSyD+M1ptF9g3Mt9dso4eBgyjXi7epwZpC6yAfQr8zgmgSt0GfvVYj1UB/APUr oTsILoFch/JS9MtU5V6+anzrgAEL+tEYZ5da70jjLE0UG+03sZZSjDkf2I45HgDnAnlokwieCezQ 2qhSa7PrUQ+mCsy/g/XAbMVzMc421E9HvxGQK1AeAjsscD8gBRilH6Up+hfpRXAG1l/srBtoo9W8 5tiaYL+y6VY4NubFA3P+EkjVp9gd4F5xtnlR4cE8I5vC4HIgCcjXX6V14qukwV/PmB1kMPxE7Ke/ ANPEKloIWYOdBWYz7WMZWCCx0b4hDlCtcZUmo+571l6sA1+y/HLUP6YM/R+UbqXRZsTXbIy/BTiI Mf8u42EVLcH848DZokPG0HagCnO97/qJfQN5C/Z1Meb61M8x3EgFwP3YlzCwlu3B/Bnsc953rahr Ctq+gzZLGdAPksDaOSa5D/fHWGkqDutvMtWjTTX8+jewAAJsgwsZZwqoexnjDAYsYCgwDugA6oFy IAd4ARiFuQnzGjJeETMcmzI+EBtmG3wI22TMOms4KPfTOTN1aiyeJ8U6SuUKKTwmnxeOWdhy3B2b zxTHjMsyvss57rUPeJ0cUzHG2ROddD/bIM8gYstlPnewmc/DXr2QKsH7EMcVHLNsn8vsF4416ROc CcVT49aaKc8I2CBKVbFe4bLrixivpkMYs8xagZxSS3PFt/F2/SGtEFdotjGaxpmZ0GE9aBvVO2mx H+9a7OUiyM94uIbha9fWmK1YZxP82U7PwqffEu36cNGumWaT/a5J2nmzSX9Klm9hL7RWp46ZEV/3 7+o/D/RLZhNyZpP9ntlu21jPbj4Tvk4tE0h2GfoTQBi4zz9Gq/GXay2+QkqwiK4C60WQcswgTRKt 2J8A8jzOAvSF5pt0xqimnaLd/qMWprDeTtt9AVqO749+PJd+iSoYPD54Q1wcdYs5byy57Marlznn q5gaBrZw/i4ovKPwMfAR4ugnmjPHJM7P8n5Ajga2O/FqX4vF53k6DN7lxqcnTss98dnHG5delncL 8rt7TmHHTnf9nB85x3GO5DzHecZt7+W4/hG9EXHMefhVCqlzPVxhPmx8S5195GHsd7FtW7n2EavZ bjAG2A1WFsp/AEz7CNa9KXanlthd6j4d7d6ljp7ucu9RM5vWqXx2SOabD+nH8h4tkvb1so7RZvM6 9h05UNpbq84g/Am7y0UZfL6PqrCOwcYOnEfogaXsE7kXRHfzvcB3orEHfua7qJoqjDfwXuC+2dRf 3hfTqRi2n5c63KnMrDOLqd7qpCxRiFzbSqt4r3gdbA/vvf8x6usPIE+003jxM7QJUG+0q5U+CNIR GRfctxyvH/jCt5J8iNmFaMPj1ck+QRqg/HFI+kL2x1uE44t9gTGtAC2W74lOes4spGKcoTpfmOqs Qpy5ADVgjMPoV8i2oN8QeV/voQdxviqRmyqRc0jGf8i+bjRhPZuQ1wEjDB810d1mGD4sl2ufLZwc u4PPj9FI93KMWHuQh/k9sYciYgzNscqpGrpqE3kS8+6CbivObybO7k70H6byNmHundBz3+n8luE3 Ap8XX5ASrbB8B5C0gd8pmN94l+qM+VSJOJ7h3wM/bKN03BcaYu8eYLwDKT+lUOVA6hIc1lKMBHpS 6rPpNb3RuAtxy3foKbGFHhFFlGWMp8GiP6WL3+GsfkL7jX60TLxC+0ULVbEsEmmUgXeu0Yy3Jesv 0gOs11+DXEMhMRX9K+mbYhltNI4j9n5PvcXD2Gv0M7+POBmB/h9iXAXtbQoZRThb21H+xD7K7eQc zXYxQ8yldNkvDtJWFx6b9Tz4bT72FPZyuZu9sDVmp2vjbeyT6+Rx0Y/biP00lci+DKQ53JWvV1MT UKv/iWYZC+i7WgMSzAHK1TqAAwo/p7mSjwP5uOMnaE8A48QEegHYgvJY8K+AY46Mt9sEegPYhrHP gk/ydwFDn0kTmaE7CNQAv3Hr4sFz3U4fDzOJusvPU5ihXbVvMLzt4eeJmG+imAZ/AojFpxnWZgr5 Hsf+jYT+HozpkTFPlnie1vRkT0/QLlKm9KGDYPwa3f0AD7wDXI7jZGZ1N/xH9n0eYH83Aw9J//6T AiqGvqBdouHgInCR8RhtYkBOh1zq+lO7ilhjNNCPpD62f44esUJ4x03z6r2yd197kvWTdDgebhzE 4mE3bWWI6WgPeGX/edrKsF5C3Uu3yuJIDwjRfcY+aRPJGPPI1iLcmYA+ArYOkX2qGDH5Is4ywG1l /75UzZBnF9Cb6RFGrH4C8jcQ59eJ7FfMKevd/XH3xbs/sC8oLgAh3BUXKBNcAJ7hciy+Vb7oFvP5 TrzHZM4lHZ42N8/EzbNxke+a24/5/wScnVeANuDl//VcnGU4RyRwnriMd8h0vCPb8T55kCqIbiCX fJoB/BR5aAn4dehwe3eNBvqi3B+6b4CfJbr+EcqPQt/uwNZFEtWqd+Vg6H6h+vrVeAVO/+u/JrqG iLp2zOl/vRFYg/IHwJMo/xl8FlyD9u+h31bwOaf+xjLIjwMvQu6EvBYoQflpcAA8FkgEBqD/Xga/ R275Dv2v8+2/P+6U8WZZCTuHgU+Dn/B+Q9wxu/vZA3u/Ndz974lN9S1xKzt+wDfTW3j3ReO/fT7r G8dl7Kf6w4MwlfoZg+h9wAYMGob/GcAiYBnwA+A5wJLtWLMe2AycAa7ImqAx6MTu7GALaJekk2vW ZklxuSMufUiKJ4tLHV6Q7/DseU6zHKfZ+C856nEzHR451uEBaVlh5t59s1pnDDQG0m//xXjVxjZ1 neFzzvXHdYKJbcC4xPG9jpNLEwMOJuCEhPg62IRisaQhY3GWQPgIpWUTTE5AmzR2sxZpaJSgdWOD SU1UVahahbhxuswJSEnJ1q1pN6qNMam0hXb8GD8YparoaAnZc64NDI1Ju9fP+7znfZ/zec/9MMDI PljKfkuKKCUSGRIWEB1ggiUfUQXXSJkSHpwQTIQKTKBkJ5FmJwWasTvDsQI2y24SF7bBP9mNXIbd GJnrDA/GNrBPyBlgAhDYJzg/Zh/jnXsVt3MRbBQYBCaAC8BNwMKu4ryC8yP2EVQfkhAQBbYCg8AE cBOwsg9hHewD/nAwLPejAGMfwDrYZUzrMmwRvhApe5+9j6H9JROpDY8ZTjCUd6TyvLOwOO+43OEs +3PmToWUZX8fkYPSUKyKXSQ6wNDZRTR+kchAC9AN7AMs8C7Bu0Q04BgwBOgA/nnAOgCZTQPvApdI FaACLYDI3sugmyy7kFEapZib/Yn9nizEov6R/cHgd9lbBr/Dfmfw22AfeJq9lfFJJFaIPEEdB9gB DiFvZm+OlLmk2ZiTTWB5JNgQEAWaga3AAGBhE6w0s1NyoZGzZBrPPYllyHWDT5FXRKI+J6nKWuwx mRtl9Rp4MIPyoMJU5fgJFLlRjv4EHjfKC0fgcaN8rx8eN8q39sPjRtn5HDxulI6t8LhRmtvgwWTZ y78pWyxFmvdQOVbEDmCVDmCVDmCVDhATO8BPcsfEx/bLTGUlVuykGqyolLRxqp2jWivVXqFaD9UO Uq2favVU20K1INW8VPNRTaXaWVqDpdCo+sYjxVrVQ7Vpqp2mWppqCtXKqVZGNZlG1CzzZ55aYVDC oJEYv6/AaxrCRRijHyvqx7b247afgL0AzBolFSK5NCd+wse5dKQymisvWx3eG1vPplBxCpdhilwB TLhAU9hGU2hkCg0UwUaBrcAkcBOYBSxQl2LgA4Ytgg0BUWAr8APgJmAxhnMTYGRvfohnjIGF8oNu 5iU2hbMUp5/51RKH1xF0rBcGvLTIR5t9sz4WIW58rxOXU3RmqX30C/u/vrATW8zGjrIBUoILcSzP A5k7JVKW/iKjnJViC+jPic+EXUdriULLwTX49OPllcQrcq4mXvY6OJzxbka1ooyyRBqnc3mtUemO 95p03ZtlcP/hPSv9Tc6aaEb6KyKvj0oXvYelt0NZEZFzSpaCxmVDOuatkU5PG9J+JE5mpIOcRqXv e5ukPV4j0ZNLbEmjpBZJrUqHtB7txb3bJTWNNkelqHeLVJ9TreR1RqUqDCGYcysx2Aqv0WnAZzT4 9UiW7laXWI9b263N1lXWsHWJ1W+VrCXWYut80SU6xLniHLFAFEWLaBKZSMT52dmrahBfqGS+xcGJ v4/wJ8TwHYxbGOO5RkVGNhB9npBkyU2NNKlP7iDJ7bJ+e1MgSwue7tDNgUaqu5Ik2dao1wSTWets qx4JJnVryzfbhyk9mkJUZz/KUtLWnqWzPHSoWHetbR8jlDoPvVjM+clDL6ZSxOPeH/VEXQ3O2nXx x5juvA0+PDyP+CX68eSmdv1XJSk9zJ3ZklRSf2mT3Nk+Rj+jnybiY/QWp1T7mNBAP0u08rjQEE+l klm62dARmd6CDjvmlqETfUTmOiKLvpzuZE5XjvrQlXGCzmYj5Yau3GYzdCbKdcPpskR8uKzM0CyU SdrQpBfK/6mZLoemvNzQuDUybWim3RrX6A2GxOuFxOc1JHQR8RoSL11kSDY/lITyksMPJIeNngT6 UOPNaexX72vsV6EJ/r9HT2MwSEfqUjs6Ez2BRHcg0QN06z/ev9uja9tleXhHiidkXVC6t+/YzXlb j54K9MT1HYG4PFzX+Zh0J0/XBeLDpDPR1j7cqfbEM3VqXSKwLZ4aaWqpjjzS1+EHfVW3PKaxFt5Y Ne+rKfKYdISnm3hfEd5XhPfVpDYZfRFjj7e0D4ukMbW2M8cjrLAA+7W72J9qdDv2NRibt87vOVg8 buJ/GguDKX1OoFG3Azy1NLY0xlO4p3hqLsJF+ZTnYJ2/eJy+lk85EHYGGkmwty/dRzyJZ+O5XxoH Qr19fMFzNpj+XwdyCV3dFk/3EpLUKzcl9ejTHe3DViui3XxK+ur7scLCRHZ2MhdchuBqHhSEB0Ie q+cxmy0v/O/r35fntfwu0NjZEar6aC9JpwTdl2xjeBS0dWCunR3t4/hc4q+HdAoTTNMgTd9vwxg2 yfmEz/c+evvyXn4devOcq4Uq6fvL8eBAHTyq8KCCMePE28VKGt9g9JrFmmUn1HnEbLomkAKr6Rol T4gW8zUmnGPLiY2eoMuIJ+i4XT9T/zXH5/UbZ+pJFL7jLszyKr/T7yyHwWOR3JWFybuqmXxFZNMk 7wH/kUzXzePEifdOJelXW0ymdYHNgV2BtO0Fm+XZRX3mfbZ04fPm5wsti902wbO40ucusdnmuXyV lRUVxFviY5RJPp+TiB7F0lauzFm0pMQnUxnL1RWs6+TD6jLGdXvjjc9vGOPicNWG6h0z9bUhZ63T VUtdC2uXV9Eu0rXC6Q+73QvmW6zczmUB6g9HVq1aWa0oAcwhHGlgOf84U157J73rmUMD39DePHLv Jbqmv2ZDct0PX753mX57i7K2Y3Xbz47cO20eT431bDm1YvE57Znh7uVCq9O9a+NTeyu+GrLOqdmz rvW7y/nb4VWseClWoZDsHiP22Ul18bwF1SbBZysYKnivgBWYGSsURbMoW62WLs1O7awwN0XsLXUB tKRrjmynsr3F3m3fZzfVpTzBru84bgf51B1dt/lUMfVofW1XyLgmNLjCiakCAdhXz7Mvz5+fsZjH Z06xji/XsZGZjWh8AkPrx6gE8tNfU4YvWzO6G6lZU23wiuocL63K8ZMVOQ6U57jEl2PPIoPVkN1R LZuPmc+YBUHGHhsgQ0QnphBRSQu5Qj4lZpeM4DF092+mqwa4afMMf98nyZatXzu2ZDly5L84jh2I Ezs/ZqFWS/kJIYTCJSMFl9C1DEKBOAwGu/VIt4ZA2yt0d1BW2AGF8Te6ACk0TenBsdyua8fBlS23 5lqOtelG7+rBdWk2oIn3yUm7niR/n/TqZH3P+7zP8+p1cqg1nzhcGue6sKulWzOddePpb5mK15Kq iBkruHSFGrg/B7+rwaMIflcKxHUWIowdBWgvCcl+dFznzYiYooTp/5QQ/5GepMPko3zOvVfQh/hx /34D37gPAJOAnyfCTfo2gATagQppcjO7nX2PJSxsPVsvEKVkMVfGLyWWkZu5LXwPRzOIopNcNd+E GohHzTrdyD3CW/ehXxN7zXvpE8Rxs8mOBJ6PUchBUYhmOS5G0XhKs4uFxVCHCNG0xcowHMfzIqAt qM3eZUf2AXQCcLDiHOWl+2GFbmUtVq/ObmMgM4BaAA8ZHEH9kNEtAgReoUOEYj9qectLtVFdFEH1 oxN9NoMRijiaHk3XufDSs25FzOK5+7uTkTRwpVJ1deL3NreYzfZQ06M9zw72THcZQ0UMixuDxa0I i9u7gM09AHRuCKDcUG1tbStualgcC+OYQeP/nuWtxtW8wHG5v1zwJfkyX5Lrx9OaJF9Zk5+en4av TktOZra1M5MGmTQ06hBKcnUN9GGGwgC07YNBuCwmKVVwBaTemWjpnVhKDTz46pV5i/YT39yfQ37w oIq89cCL9QscxtWECw+4gB8u0AU7w0N7tfq4topep5H2/tynfXZ3Ao93+/wlCZtx7ilJiFOjMDXi +N/6PKHJOL5fnBqNuL4RT4r5+ep87xJmubpO7bRs4bcK3dYdwqvcSaFfuM3/UxB5lvXaBIfNJtgE 1mIvRD63ZDXZbSLHUi6LRZLdSpEsA5/fkDDgcgkCTxeF+AOmtDfYEewKEkG/a4q1gR+ccEWxtKYN bcX1rIy4DCkzcjQlZvgyFjMsY9AmJ3v46VHqWXHQkLTv5B0Y1aRbaV1ICuIMm32GATfM5JPD527q biVp8ytJOz54XU2Kfgc+NHw4p3ITbf2eOMqSXBAgpqOSUCBgw5fzEhnwHUYvDP75Z+/faAw3L8iN Xmle/8Npvoa/w8Pdexe+emQiRg00vbf1wJCnOLhw00QGVjz/Ui1jHt9ExGu2zl293cjd/NxtUiUf AmFQAz36yxbOElE4d6SUi0RwZTlrCmdE6iNpLh1p59ZE2mIvcNtLX5P2u09yzmPKqfAF5Z3woHIt /KHzkzD9qAQ1WXNFyyKJJJksqyfnlbXQrdFV9JroZraH/RN7j7sXtdUkeEiK5cGEXOlzuFaUbihF pWo5n+J38Qf5HE8d5Hv5OzzB8yoh96NTuuTa41BVM5hdYq1UCaZ0pbgSFPuC/WiZLpboICSGvKFY qDdEhSqShuxpRYFELHk5iQ4lYVIudvnLg5dM10xIM6VMyFRRi7Uog41JTGeiY+nsaN3455+DVDY1 ksqOj2BnKsfRDB7xjlObtymQhkaRFJtMAX+oKlFdXZPfqhI4F36TueQhFMdZkiWn0yHJgRBhMvMI T+OVxk1E3VNvt/denLtxXtXa4R/D+Owd27Z6zrjWX9+549Qi0SL7L6ryk4MblleuW7P69ZDnl81z fte98LmFDp5zB4ut66fNbM24Mi826CvnT99y90H3zFr4SVgVw43l89qWNc38Kc7gdpxBzVBO7OhD +mlIsUKQqqJmU1RKO6MhTfOrcfURtUPbrZlmFNRJde4F0gJ3mk5zS4W09IS7nX6GWy2sl9a7L2sf scPysPJpwZfyl8pnnltaTlO8VLlQ7ohRKUGnFgiLqFXUsOdr8r7Iik6eNCFQqJrM0OpUecYVvM5A kdGZNqaLIZmfQFscxIlihC5DuBsegmfgXUhqMAWbIAGVork1kyWGnaZRHB8dEccx9EYq8J5vEvLI 4zDI+ALYL6rjlUXIKYKAv4RwyAa8RgX4TXDa8Tc7zz7Zm9Envnr34lqUaH5l8+nfbtp8Grvr17ua dr2/ceLOxNBv4N5LzS9e/eD6H6/iAl+Uu01kMevd4Ko+18JCTZ1VMEteUrBEbitok/ej/cRr3FHx qJulOcXajtYQ7dQmtoPr4o6x5y0XrOdZVsLO9BkieP8KYYOwTSAEaJC1PpZ31zbQgX31ELiFXdYC BIHBZmlXGbNLJRlVgEKQ9xfitwgyUQ1CbLewXnUGr5mhZk6ZkbmiMDGY98xMFv90NiwJNOQFHuYu 17ZmO0eznQZKWbvRS5WL6RG8G1BlMFZQNkgKbAk7RkuSzSEDoEkuEnVnPXd+Pzzxn84vdr7xsdar bHt8x6mjz7e/DLvlt65BD7Sehui53sOFa5/5w42hK7/AzGrAzCrCKDkxs27qT2lAdaJmIk2lLc3M 08RaaoPlaYYWsWWLqMT+EXXfMeY2V9hnKBXqw/ZG98PqY/blymJ1pX2de6W6xbTFOYbGXCKQoMDJ 8iKpTeqQCEkVdouHRCSKZKFqNQMDRAvcU4CBknXOqGdLSSRxBndgbs1obopDCWPUPUaVa1CT4mLQ rAcjCQO6JjNhVooSU7yKNo6PLBQz0ehYJtpoEGt8JA8abkUyddCob/tUDwoznd8CJ4J4JbA5zD7J wAz6QvkaJ54YKPvX219M3IGOj/8KefjNbeu57h+9ND6MHmNrW3b+/CRskY+8CTXMaxaGJ25O3BO9 vQOr4Z7ts1YfM/rNAiy1XdQNIIM+vchhgYJSrsQUXelQ9rMHuJMc7ebC3BnlskIqxurCbi3hoTmC FVQrdKKoo4AkTMB60AEduQKdlItJQKBfwXw72ldRm8i3pVZVS+zG/3XEpVyEA8AHxqAVfyVglcMN HP5KEOuyuL3ALRhuOowuLJu04fXP2qr/j/1qj22qCuPfuff09p7b1+3r9t52L7s+ZjsBuw4FJ+0Q xniNCQEjdvJQwA3UrQOjshkX0WHiAzEGNJoBEh/RyJxNxFdcjJKYkGx/GKNGA4pRE2OyGPQPdanf aQtEkRD9w2hyT/s797vnnvae73G+73d8qltiVknG9Khi2QS35ArhmSeZGBoiScx8eSStzU3NaaTl TSmMKR5S/iZ/vXtsZMQbvO+OZbnQlamVCyYmxKce6tuabrvO84zStn7jQ79txtUM46HjO8xOPnj1 DdBwmQ5/IB2lzeJC8U0HFY8WT2YjASMdkN12t0+0EHBVWaw+m2KPsmzT7HSRkXFGWIfGNQykZ6dH tSlN6NUOaKNaUaOa4ItWSDlOnsL3QR1M4r6j0OFf1FneRckSJ+9LYqpJ8ipezi94Bimr75Sc1qhT soeIQ0bFATVPDkGyq0zZ+T7S/MiJSrlG8ruHC/eM3/HK0sKOrZ0Pt2CK+XFv1+Gnp9cJB4d3rnpk cPot3De70eUtJR5vhcFs1wq2hx1go2ycnWBTzAqslvWye9lIZegkKzKllmEesFJBZJJ4DwHJIlFF skYtQEfoATpKx+lJKo3TKSoAraOTeEdph3xGQ+TqnKKjZqW45uCRne/zNjf5RdRid6FQoN9PTPzq p7FfP4OKX77Bva3B8azXIkpe4QX1qHpK/NY7Jf7slSiytezlNkf6LpXsVyf1k3pRp3Wyz+nTPOgg ImkOxeG0OyO2kpdsBL+2Dp17Isi9pE/pQq9+QB/Vx3Wqi0KTX6s4ynOeowJnnHS6pVyV0U0YrdxT mR/O+UmT3EyRFasiSmrMLTlDxKV4Kv5KDGHyQ5/1ocv8syvluOS0UrS6hw/t+GL9wU5VKSS2tvc/ T2P7jizsXZ4anO4XHrjt1ta9x6ffxiXdj7v0GNrEDfdlr5rpJSol9TRNr6Gr6Ga6nUrMLTOZObxu 5gBRJjZe+XBbsoY9MpHDdV7iFcLuC8ajZ9H7Z+MRc/bp/NfoMq4gxuKckt9A/XDYWeL8XXnOyFGV szvOilrcf2hed+aGG+fNn3/Vjb4aGjvY1z73+fiizPr89Efcpxmsaq/i+meRT7M7adgXnsuWsAWR NeFN4QH2CNsVec77UuN7ooMFgnpg1tLGjwOWkLBaENQUUfScnGM5JWfL2XOOHrmH9Sg9th57j6MQ K8Rd8VgkHrl0dmStcr3t5tjNDdvrtyNhflx52r63YV/jE7MOKy/an40fbngt9kFMqz6K7NZTM2et HI/aFRqsi/mpbUZ1kOf2qlojY6ww1hlHjAlDchm1xu3GCYPWGo8agvGWsBprDeA0VeWnM5VMIlEn KhEIT3Q+Lc2v2RqnO03IjFz1tmqhuspvpVUzbLVBEowYWa+eNpAZjlkjCZz5etWcyQRJBFP8VzGs I+tT4ykhk7o3JaRUrLoRqIu4wieAZGAFblvj8jOlo285EsQf8h2lcOTV43SyUnT7sIAkMc7yJbqS /7rMFCtEEYM0G7+spt7ia4y5VY/qVUUp7KgLAWuwhojlMuxqfHh7ibM+BOF6h12+VAmRhjhTpCQN Qa1azcM5yU8Z5Y7wU0AiOTTE81Ef39Bd3iu0MvWJx+IzBGSjpaysnav1AX5UQKLkK9HVzJjrwZ0D dzZHHz/25IrWKxOPrRp8Z6171N7fPdCjaTNDu97dt6b72ODEp+Tqqq35TQuurtejqcVDHYvuaqhN tu/coq/Mrbyivqraq0SaWgdya0euexnOtm3/PZD8X0OQEYVzEClSsvHzIQ38c8jzABi+S9lfhu0r APvnZTgmAZzbz4dr9x+htgN4xkyYMGHChAkTJkyYMGHChAkTJkyY+P8DBCDAmw9ELpEgQoKLNvGi M6KlPtkIMHMWl5r/+HjBwjZoXwxLl/GbaxGr11z8rf9io9CLvQYqqipBHcyHxbAcVsJNsBm64Tbo LxbxOR9vh2U4vgE2wS04ni8Wi6fO/1Ss/Fft4qaU8Z2kMteOPams0I6fsiyhpHMPUoYjOkQqsgBO aKnIIo4vqcgU5Y0VWUL57rbO5R1tS5Kt+e4N2y4kQxt0og068LoEktAKebTEBtiG2m+CLbADpQ04 dqFZf3ccNbOo2GXQ5hbURIWZaG2gdmUaVy5wA5A9+ESmKPG7M1fYLHjQSGfbn82ZwQZZ9F2/zP/m uLxM3F+xqvDkUxu/nJ6/ztXykxySS7MPnYon+HVs6YlPfjkyvUWdK/OY5XYu/fPvAwBUTB9WCmVu ZHN0cmVhbQ1lbmRvYmoNOTQgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAy Njc4MSAvTGVuZ3RoMSA0Mjg0MCA+PiANc3RyZWFtDQpIiVxVC3RNVxr+/r3PuTeSegZJMXKTKw+S iKhnKKnkRohHvMVQuUESj3A9i9Gi6rESr1qea5EyVhotw6VDUTMNC1NUvZ+jHlPvGcaodszC3fMl M53VzvnWOevfe//77O//97+/DQFQHXOgkd27X1LLvKzBy4GPT7O314gir+/41KAnwOahgNweMW2K 6/ueV+dy7BrgPJzvKyhqO7U8CQjaCNiTC8bNyC8tm5EFJC8GhhUXjvKO/Cr4L3Qvd/HTppAdoU1r lwE16rLdpLBoyvR/rkrewXY7oG7KuAkjvFKxfCawLJzt9CLvdF/I5FrbyMdPf9d4b9GoNnd7HwXK cskn2zdh8hTy5lOWWjnumzTKdzY//hkQNQWoGW4vJaseiODbSK9EQ8Dc4nub7/1Ad/PSHgt3YIy5 qUM5+3f/fYForMZHaIInkoxDqEB3fIy3kI2V6IpT2IEamCEnYMGNdGxBtERAIQNhYmMdrmAoJuEO biIOWbgudfgfD3yoj/bmAb9ZWGT20SsYadiO/TJO+iGJdqZKkHiuvMxUIAxx5qS5zNYG3JEmZicy ad1FbcRiNj5EHYzBcfOyMoPIQ7nMkgeIRC5KrFZWsRmLDtiNC5JFqydm2Jer7cY4ztosYVJhbph7 +KMlGMU/vY9FZLwLFaq5TrM3woUYvIle8HL0N7gioZKsU02s6WLWsbccT1W8Oqqd5BGPbhiOJdjE bFzEbfwgIdJaNshW4ow8ti+TWxamYibragOzV45t2CfJkqzCVBizFYamGMCxZSjj+p/htGRJjlTI QV1mtwh0NnVNPXPPGDTDYDL8CAe5xjNpQR+uoKP0FKuxNcVu+WouIxyJ9TiNM+RxnXn/Ac+lGXFL vadmm0Fmi7lDLkGIQDv0wRBMwDS8g99yVw/hMP4hL1Q1ep6yjtgz7SdmBXMbgy7k3pve/fjvEu7S LuwlLjLK2uJiFO2kl/SVAlkmq2WvXJEryqEi1UT1UPv1CX3NamPbJoV/qo/GXNeNQSjkDrzHbK9g vFtwBMeknsRIIiO6yPk/qg4qndisTqnrer5eZr20FwRuBv4aeGGK4WSVdWUepuJTZuHvUp8cmsoY mSzfkfly9XtdQ9fSbt1av6X76xy9SK/UX+lvrEnWVuuq3c322lud3sD4wBmTZT5gLgQO8opFAlqh Lesnn9U0lvx8xCTMwlwUYynrZQU2Yivj/hLHcAHf4m/cAUgkOY/m6kWsuvmylFgn2+SgHJFjckt+ rISKIuJUG9VZpakMVaDmEyvVaXVR3deN9Ag9W88hSvUefcWCZVnGbklk2iV2ueOEM86Z6cwL+vrl o1fNXuW8uh5AoEHg14HVgYOBe2agmUH+0UhEczJdSJbrWINlxKesxD04iq9xqYrrU1Fis+LDxc1q SOCudZau0o3oKX2IAcQgGUJ4JU8KidkyR96XefKBLJFVVVjL2MrkE9lDfC77iQtyQ+7KQ3mqWMRK s5qjVaxKUu0ZaZrqqnqrvkSBmkD41CQ1jTtUrj5T+9RFHaqjdaL26ol6nd6uD+nz+l+WshKsJKuj NdAqsOZZp6wz1mXrhR1he+xCu9Q+5GjoaOUY4BjjWOvY4bjveOl0OLOdec5ZzvNOExRNtfoT496N nz9JjlMy2a5rTVc3eC7Ctc9eKAOYMYfqr8fppfqsnS9PtEuuSrEerceazTpDPdcTZKD6UqJ0hJ2i 87EYRraqW+qZumfVk/7qgcRZH8rnaoJOU47KRexzVj1rnn0fUJeQot6VCnVEz9PzzB+QYpfKDbtU nYHLuqlCcYOneqFaw0nfqNGqBIOtVvYLjGbeP7GnM9+d1CJpps9bpbij3ep7eSKrqRonpbvVRL2t 2stWKu4raYxHMhE+WYVU+UK+lb0Q2aLLpYd6jbvlV9WlLa+xkzpSzutg5FRylBhVT7LVEzVAH3Cc 1q1FqBJnMVO0tGDt/PQEMJ4nYKWKpaZ5qCbnpCXCsYZ6/yxwoFKx7ct2Cetsk05AX7TAMHUCKTwb d4jBWICW2M8aXIQWai1mmTkykrrfk/qpsFfGIElCqJZh5Dab90V9FUUtHM5Vn1P/j1P1s+Qx3hEX T1YF4qzKkcWWh8qUS/0tIUZiGFvrscKx2z6H3hIGWK5AKav8Gt7mnfMd12+AjuQ3BJusBLJ2UZkn csb6QCZSiQU4IQrvknMnnvNsK5PKu9qMYYSjeUf14J14DKPNGqRx7/qaeaYEw80mMxQF6Ge2UH+n mV1og4V2jhpox1utqLHH5DDvoz9LCXU7E1epR9ESjofEdjLqZH+BYusStbOzWWwuoB7zEcUM5fEW vY0iPGbeMnUF3gj0UjtNhvbxhrqBPqbcREgwCs04Ku8BlDltas8cNLbLUlNTO3d6s2OHlPbt2rZp 3eqNlsktkponJsQ3axoXGxPdxB0V6Ypo/KtGDRu8Hh5Wv25ondq1atao/lpIcLUgp8O2tBIkeNwZ uS5/TK7finFnZiZWtt1ednh/1pHrd7Er45c+fldulZvrl56p9Mz/P8/U/3im/s9Tark6omNigsvj dvlPprtde2VIn8G0l6S7c1z+R1V2zyp7eZVdnXZkJCe4POGF6S6/5Lo8/oxphcWe3HT+bmdIcJo7 bVRwYgJ2BofQDKHlD3P7dkpYJ6kyVJgnZadCUHWS8jdwp3v8r7vTKxn4dbTHO9Kf3WewJ71hZGRO YoJf0ka48/xwd/HXjK9ywb9ZrxrYKI4r/HZn984hNj6bf9uQOy9nZJ+N+akNZwpcYt/FYBriH8yd 67RnMBHGTULFT0obBaPEkBzQhrSNCCIIoTZFuA1rIK1pJWRUIZRWQKvKoPy0TUloS5tAhKASRPX2 e7O3x/mghVa1/N2bmTdv5s2bb+bN1sppTFet6ZbTeLt4NbTd218+mNgx4KEV8UB2p9HZ0R41RUeM 58gLYN46c8I3P554u4rB82uj29K1hSIRntjl5Woisc1r7m+Mpmt9/BuLYQxT9UfiiQgm3oEQNjR7 MZfaG4uaSi8m9PI6eE326lYZYW6Jr/GaDxiPGKsTa+LYmIKESU2bfEcKCkLHrQ+pIOxNtEQNn7mw 0Ih11BX1j6VE06ajk0LeSSM1FeX9njw7rP2jc5OF7Jz0wqqUTpZkdy41NKXiqrBHxiLQwfSu9MKT qIE1zeWfVXMpsXIuuuEvpsDK7MR+dJkP1MYTnhq0e9je1P0ew5u4Qdh/49NPRrZ0JFtcfs8N4iKz JEU06J2yGQiYZWVMEHctdhQ+LpD1qoryjQOqaaz1eCEQPnocse2I1VQi+D4fb+/2gRCtQMXsaYza dS+tKDxCocpAzFTjrBl0NOOWsabH0aTM4wZ4fIz4y2KcmVWS+s/1jB8TXl1jKuP/g3qVrW9oNhoa 26LecCKejG1Dy4iarZ+b0iVLiq1AwE3Nj0gtMkC9prYoN+Bf90eMcFe8HkcNPppjaqOiUI3ZJbVQ yKHA3/bUyFyJZvNYmt8l+d854M4CgWWL4o2Ynni9/Rsb5fPdp9GA9RlbSXHbLLkmsyYwsj5vRH2E e9kJAYe1ErWhpS2RGDVCF8FllUhEDG8kEU90DFg9Kwyvx0gcF1ERTawNx53tH7B+vr3QjOyIYRGr lZoKfiOS2zccpuUe+nz9cJOnRm5W2p+e4woqRVxSHeB1KuqoV8P3E/A11yGqdwWR4b5OjdC1ANPR vkt7gfzo/zTqzZC71CAJtC8GPgPKgWbAC6wAosAS4DmgEX1N4Ns8hgOxk9rdX6EO/TR59FYqBhaj bGgfUZm2jnwo13Md880Wk6kM5WLoSt2T0fe0dYn16Fcs+7XCbh31QL8A9QeBfPdOKoTMBcagvQDj HGSfIRvESV6rdRXljfBjEcqfQ0bgax3kErQvRXk+kAObL6pBayXKeSjPR2zyUM4GwrC7yTbonwMf O6Efi7rKfTFvDmQh98WYpeKCUqjswXvkAvVrLTQW+tESWDev2VkT+88+/RtE2L902P5JsK/qbd/u gJqBVWK23KstybXuVc/QWrHfuoay4RpLYYb7Ak3B+j4BglonTXJPtv4KHxfpx6gK9SxgogSPuZe2 iusUgi7geg286aQF6kwoqqxb6rdosstPj2K9iDdNg+8x5h64MBX9mqV9J03RLlEByiFGFtGfU3FC bLD3DZC1iPuVLLI+xRi1DIxzHDgJ+wmYv5JjwPuutA73oe9l6J4F1oEjk4AJ0G+XHIYN22Oeh3kO ex/IIzkIMPeAWQ6S++PgQQcy/ockxgMTgDkAz/sa8AvgMeB73Afjjkf/KfDjeeYMc5P5wdyQ/Aef JGd5H9chNswx+8z8UH2SXgLGAuV40G9Nogx95XnhfWSf+Szw2Mwt5owjoS+xea9c5XUyp9KkoZfL ueUZZG6lyVLmPksRkmsoVQepmjlrx9qR0ocwn0c+E450/OHzKc8IpOimMRw73ndHOrFIyf3kh26J /i49qs2k5eIU+N+O8uOQcxCfffIMXtW+Tx+rvaS6B6kce8ln9/UMuZvhHlLWYLxBxLJEO0OvSzmk FmtDiq73WZf1PvV5G045XWZCGbR1LBnpuv+2/X+Bel7voydR/ps+ZFnaEL2KtZL778oMwOtItB8B eoCyrICyO6tbGXAvIw94cx14Rgvh2y9Ec7RBWqiNk+fOj/ZlGLtS66Z5sBP4ynlZLKMDrj76ghjC PmIu9Ty9wODxIdemeJTJuTu5JKXD17tIPgM5jpRnKmj9QZ6roPVHeSaD1rAtKci5ge9nmR9I3s15 Dl9TvHyDSsSNNH5m8DSNn/Ng58nkZZoczTKZW3Kccwqb8ZxreP3yfmyV50nec9AdcfpnypT9IRpQ D1nvy3v4DLU55xqYCfih/2XyHsE9jP3mnLnTanc9a7WLxVY71vlT1zbIa9ZRdZrVn8qpfpqVvMsK nFzKcdLPUFEqj/ppafI+83M+1Q4ih9t5dIzMn3+hifo1ebfNkv7yOeQzWIl7bxry+D+sW1o+PS1e JhI4l9wOjjSyTsuiceJPuHMX03qxz/qd2CXvoLAYppgI4AzDFjGbqKtUpNdRA2xIjsd9ILmN/Xdp 4CffBfWoY6+ce5n33nWLcoBp+hXcR63oc0iu1S/v8d00leMgbTcgr2Asd4DyNZUCyT5+afMU3gsy HrgD02KRzM0LeExXk+RsrrSZbd3KyqcgQ3+TqjG/X85VTzVZQSrRW60r8l2RT4+J0zRD1NNDKBdI 3m9DjipFvqxHfgTER8AwuOmx6zJXS2ndlPl+s8zn2XolLZfvCda5aIqrlKYzNAO6OFWINzHOM+DV LZTfsiz5Pvg95fHcaI8k3yf8TlDlefkt7N6hCj5j7IPMN+zPHvDtHD3EOdF9ADEcxWdQURDvomQe zEddhfxOGl5JthXZUvGp71Kr1LXQh+oJ9bB6wurmd6B4j74qfoD9O0w+0Yb8fQq5cR5y+GLE6jcU FWdRLkb7PmAj3n7rKVfLpU5xEf1mQbcWdmcwxgHoGVth8wHkWzRf/Iq6xCDeBxf5jUA+bQPkE0Ad 1So/pm71JnW7qpGT51lvyPEZ660vSxxA3ryYtE1C+urgbj5vwtvuLv5KX9P9ZB/v4h+PweNKO/TR NMolsj4A/LYcblR3Uh+wX30Pfb9Em5SDeObvpYhyCdibxE+oXsp+oBFnrEp5DpiuVdHPgC0ol0Oe AA7bddoDvA/0YuyTkEdd+FRgqI+Az5Bo2wfsBn7t6NLBc92tPR16IY2sv009DOW69U9GZn9tC1Vj vmptPhFDXKZXGK7N1ObeSG0C94M2BWNm1DHPLO1tWnMvf+4F5RzNkDG0EbqfNd4v+Oxyfv5/jXe/ wP5uBp6QPlzBfWxzaLRynoohWyFbxQb6BgP1CtRjTjyV6+Aa4yB9V7an9s9uB1dIcM7OaM+sZ+7r verqUbx10+DwIMWHV+lFhrYQ/YHMetY79CLDdQq6U3fWtR/dA214o+yRPpHkWEbdtZSeYqhT4WuB tNnBSNXP4VsD4L7SPod2MuTZBdRj1MVI6atoOyMtrtUcV8wp9c7+OPuSuT/wL6SdBdrwnj1LMyCb IR92ZIrfyftiBOcbbb6n6nyXXMroc/tM/Iv7qo2N4jjDMzvnvTuO9R2OIYAxc/b64IyP2BxNzccl t2fOUPus2AkUbLfK+QOHiIBsCiVqateglhSapHYLBQINdtKYVrFdr/eAnAMtVqomShQVV6rUqkJg Wv71R52kpUprcJ+ZO/OhGjWR0j+90/M8M+/7zjuzs7O7M3eeDTwr98v5/wQ8O+8D7wLv/K/7ong/ iHeEB5B71LWkXH0Ye8/NBMfVmx8QMpkNfQDfBTx5k+Mo/w7lJqAI5TdhOw49CEWayVuwT+E7gqV0 65RtIfbvhBwEkONWW6rtzRvAs6kcN88T8q8/pLEn1X7yRWA9fFiFk2cArN7JnwNRtJnO8wPUd0F/ hfqGVK5JlG9eA74LxIBjKZ38HiD8TvTxe7EfmeEc+rnq/c4fn1bT54zQtP7HGeKz6NpPpfecOabv /3/T6bPEDCrnIT1+9a7x3O+Mc49i/ThTf0JGpkbZtUR5edBIQosekmr5C4PSYS1cFPwFu6YMkKWE w3DVmpcjPVessrJ04YurUoXEsuXBq5FZ7Ar5K6CwK+wq8adaJfwPBSciGgyUfYu4KSWc9GInawIK MdgfEwVLgj0X2Qfwv8/eI1tls/csbU4QCd9lb5Is7HjPsbNpz9lE5pwgiezGJFAyCh4DxoEJwEZa 2U9JJ9AFDAHYgoE5UAxUCwvrZ/0YZx/au8HFQCvQBdjIJvYG7M8IZj9j20k+2r7IjpC50BfYYamv QxdCX4N9MfRV1IX2pOsnocJ/Im1/GfV50ONpPQZ7DvQo6kJ/lK7vxUdBtNuT1l6221rMPZHF8HuB EoChdASlI5i6I+JrDabs22yH7GkYGoTuTCmmq8PK0+U96kg8uCDYiyntwNR3YOY6MHMdxAZX+3RM eypmOWtHTDti2hHTjlkpYbvR326xfMAewAswzPtuzLuwm+BRYEzavwPuBnpFjT2LeSzEqA6x7Zaf Y5FtS6w2guHz7ClMtcGeSizIDXbdqTlniYUIzUyrW8S2SG9LwjlbWFsSC3NTiqhnIpmsmXwTUEg2 uAD4AhAFbKzZKijmb7HHyE4HMTJ5p9LJOm2dGbaSKM26yIKkBu9WTrLYchJCQCGPh2hpg7PNuc/J PE6vs8RpOGucGa2sk3UxxlkxC7NqFmcZyalRy75mJcTYoK5Z2e3qdZmuUdeYK8NUR9UxdVydUDO8 aolqqDVqg9qm7lO71V7V2a1225UGV5trn4t5XF5Xictw1bgyuJ32Rg6wJvHVAHuANqAbsGGO47B7 2ZNAHHcjjql4UuxBwQQ1DzCG8jg0AzU34tyIc8PqhtUNKwELTw3QALSlveptz3QbET8hPIDYrmTC mom5HQdPiBJQiZqGmoaahqgxZRIj9IC9QA3ApG0cwKoBT/tK0v4GQJX+CRkz7TNEW2XSaFw6WkjN QtpbSLsLqREKR4JGPigrKyuux31xf7zP1qq3+lr9rX22ar3aV+2v7rOF9bAv7A/32Yr1Yl+xv7jP xnXu437eZ+uqGqq6WHWpyhavaq3qrGKluHUJq6gkKDXfJ/SstWBhsNQdeQRnQ0ri4B7gKsCIG8yB YiAMtAIZypC0DsI6COsgqQbiQAZaDYpXDJinfcLeI32iJPzKPX6Gix+w1qysjlThtRsHegCG3APw D8joVGlI2k3wuLRXp+N7pV1EcWC6nXgJ1svXXT0ew3oSBuJAG5BBLrEt5CqA7GAOtAFDgI3V47+F bVEG8R9QBljA0FbM5WTePGxpsuY4PBGPMhtrQcM2WPBxyYckhyUXGJmV2o1K7ZeV2vOV2lIUFD+J wHFEcp7himhnIlp1RCuMaMj2IMkjmjJXsiqY/kXyY5IDRnae9kme9nGe9mGe9kqetitPeyRPtFuE Z1hTsiW7BNOjkislLzFcXHuHa1u4Vsq1iEZPUfROyiQvlpwjmH50xh11E+d5+hGJIhO1QoU8qRAp dMoKRSC3rNAGyE0rdAryTyt0mF+gn1D5aaM3rILrPDKX/o1W2ET947R+SCtIP3QCug16moSoD/q6 Fdov4n+C9idQf43kO0T8q6RGtuuhFdL+Srrdj61AE3o9aQW+gV5PkIDs9ZgVuA7rYStwCPJDK7AD 0mX5xAC3W6FlPDKHbiMFiohtJj5FjKQq3eOXkHkHdEOqcbkVEK2iooMkXWfpKyBLxSgvUJ3UyO64 pcuLzCW6TLGI6HLQOcQnNZO65eA1ki/VYen7kUU947vO/xE6Ly6c/J26rVP8zxdwfZtR/ROtsPr5 b0fEdFn8UiBJfef4b/Tz/NcFSbrZ4qOBpAOOi4GkQs/yYUyyiViFnuNDgW18UJfePh1e3Oqe0HJ+ Uq/nL/tQt/j+wAUxDLITV7wZ7rrAo7wq1M/X+5IUbiOEzoxZfI3+Nb4a5lVJWpHo5ysKkmIoJcjR f44vQ49LdDmUL5e+pTxM7PTrRsC+x95k32x/3L7WvtK+3O6159oX2bMdWQ6PI9Mx2zHL4XCoDptD cRBHdnJq3MC2npJs1SNEtQm2ybJHEQwSb3+FOhQ8O+YDLKbENpZRMytGYpvKzNKiWNI+9YS5qihm Omq+UjtM6ffrUDOVg0lKNtVigQrTgRwza13tCKG0+MBLOULbD7xUV0dj5mgziTV5zRsbcR2zHq83 M/Sy+WTe3vD8cNajc1avj85ADWkuuvObX3T3b36ueTS2sdZ8I7fODIrCVG5dzNyw0fvV2hFll9Ja Hh1R2oTU1Y7Q55Rd5U8IO30uWnc7jOQrbQjDnrwtFZYg+SKM5NOEDKuSYVim+eXR4fz8VNDbtEIE Yfm8LYO2pXIVoAvkqhGCMGUxKZC5CpTFIgzrIZXMfXey2YS6ZTL3bCKTLRJBwz4fQgI+ETJc6kPA sK9UuvvvuHVfajh1xCf78dE62Q+ld2L8qRisgnSM4kBM0ef5ayn7DME00Xh5a3N5i17eoJe3AA3m C3ufnm/ua/J6h7deFg6vyZY0NDU/LbSxxbyst0TNrXrUO9zYPIO7Wbgb9egwaS7fVDvcbLRErUaj sVxvjNYlTneui93T16Hbfa3rnCFZp0i2TvR1OjaDOybcp0Vf/ya76oOiuq74ve9rWZaF/V4eoLBv s7uF5WM/2FVghbcr+LWCRKyylpWVKigaEJfVoBjWjyQNEwNpO4RpnEhbxVjHuogxO2olTWOTxnZ0 0iadsX/Etk5bk27TThk6E2XtuQ9r0+nb+zvnvnvfvnPeueece26QyAoSWZPipCQruD6Ag82tUxko EFretsCnKUUmxEMk3xQKGFR7aqXgqDHlPpd/hUGwbSnsoUSWOZBQAshUmb/MT6YgOslUNgznPJ7K fa7GlH8Fv/l4SgXDanMA2VFuw876Jy0ajfYTxGJ2oP2xXGmsH4LW1BJMrHh6c2vCl/A1JMRIfQiT 5Yg9vpa3iqoZ3y0f1esb8o34Tvou+NhYLATDmhnhlkC1C73CkDAinBQuCByZaGu9LPpOCl8IdAy8 CffD1VAvyYwBh0Zu+2NRciEQEAUsiLPH7Mtb/QL6JlS9GCr0MqQFmAFuQAuART8D+mvAHwH/BDDo KNDvAH4ImCYjdBld1pC7s55IDNlJ0smlXdMOj2tpEvjWzgXesnmBNzQtcJ/flQv8Yp07058DBThG V4B+CLgD+AzwJYClXbRLenlswWtDURS1Y1AfwU0/IVF7P7ZDBxNz90ftdkRAHBxWAB614//1e4Sj MQSmgAUBBg9Jo1HytxjhT56DfRKBBUjNjGSocYrCV6nrUKzKqJmLiGWS1PVLNMqUkc5bGPEZHDsD 8xSicTGS4114C8q1q+Z8874m1ayvcd6H6qCvegjE6TCpTWoLEEj96GER/c5DkUUPUBHzDsgKPvo9 +4DdhZxoGVqNQ6IVBe8GKVUQc9nyfEWRITu/iPdzK0t686HodfcGDuRzjAsHSf2qM1QSLpZmayqL RbfNEVxu65BFCiLFEdf2ZXtc/cvuFGVlKe1artblLy7IUlIlHJfEa0WhtkBXW1tAM6XlZY4KGXYX lHCl9lqtXy53jiNqHDb8JP30paUrzLQ8SQ2JCtXKWwaDSuEEKyWxYxo1WLmfQLlSi99HPCqmfv52 QV1h0GjMUybxUVHLF1qx9XDEgz1Xa6Z6C/cUUlAIlIn59b52vpcf4kf4k/wFfoa/xX/Kf8Fn8vwa yJ1Hp00bNsM+0zQbbkzNhknbq5pvathe/6dG1WyKtDkwKLGrKqVK1aVmJa4xVmHAi9nl9kOq97Ba UwVNguoXTgcOw8r2hXF4bx826TmOkhkMbpd3iZHjzILVJlFPpXeJFegSr9tl0Os4mcHo9VTarDaL F7jVLHB6nUFrJU9DB/7N6PGpTS2JntDLKxsiOovl1DPrz24bfL/v9Ls//nuF8FzHof1jryaHhhOL DcXpI4MHQ4FNIeFXxzqXPTswHKuL0Tstsrr0u8M7WoKr84+/EOru+XriwMA/Du94ftm5zSuOd3VP tP/h2kej5U/ls4qasbZVWwaqnQPz/KXJgw2TW3f9wEXKhQ3pNdQg+wrSomrRPKY+o6ZeyHpJTWWO y9VoHGvBkzPlb2YLzRzm4roNW4hjhlPzPsl6YDmnA4FJsN5qs1IeFVpCDKPXGRdT1OBr20dPYNfc wTeaTHlrDqV7LWs7X8XDv8Fe/KinpP6v6bEbn1wYPvM90KEcdNgo6VAlPlXMlGSsYmkQrgYltFB/ yDNBgSLOwYkczcX1raf+Xwkc1noMRoNGr0Iyj9erAaOXU+Xj20dOpG/96+DJRhMfHGS3lQQ7v53e /3H6wzTusTR8jnfd+DgxPEk06Emfg8PHB8iIWkRbiAoZ3zPQcmOEv83TcoxkDJOToUGXNWKWgqnO 0Rfq43pan8QlUAzmtOdQOXzuCVAKojXcOB9OgU73NFXgP8Yqohnu03q8khuYBdljL1lwj56uPrlM prBodM7qoDfQNZI+VyqMNGuVcp282u1cEW3vmiJr1ILjVCscbWhUJxZRbHzRNu8Qi0nBl6BpRKlw M47gUTyBb2MOgqryLRRnSACAl4eJjSpSQCUP1pr0phaKnX9AGV8jb6Yf3aNqwfI0Wi/KEXuzsMsL Bk/SNlFJ0TqKgtdDnCpQEheKuiLaQUfoPfQEfZfm6Kv4PHWTSeLeqU+JJBJkEE91vhdZKXycDjvG ZkzVpvXN+HP2lS83sj8i+XDw0V/oYXYHxLoVufEB8UoIFthd6C6x9boPCHFFPCueF88/Yolbh91n c0/nnbFMZ13Ke9t61XYj84bit0qDDGViTknlyW0GpTHPorRkB/HL+Kjy+eyzKLsGVWOodvHqr7Xj b9ja3N2oG++kuqzdth3ug3jQtq900D3CjLBxWTzjiPqIZkQ3YhhnxjK+qx7TvG6YtJ63nXcnmcsZ 9xWfZd3Pvm+77yqWKeW2alSFl7rY+gyUlWdjJKIyFsGJ5CLHlhGmVS7yy6EolGNRggP6KtwpqpBH 9FCiJ+KZ8Nz1MB7zNZigcS0qwTlipsMoGkeNtJGvvIL/hu+jxy6UgvwU7ptL3ZtNSe5NHAkTX1o+ ILrsFYsFtYHJ0FtMrLkDFcoWdeBSXUkHKtc4OrDAFHXgxRlA7IayDlShBoKkmk3amQ7DhfvCkMbQ Xmz9ryNCniLMLNjI2FeTlVH7n1RFHBi/9P3wL8+e+mD3uUTV2jtTP929cQA7nxX3dXbGPU5vS/Px Z3Yfsa6kzh2b2Hhs5uLetW/s+lZTZ9/IzYGt0c1Tn+w+tG7n/n3rKndUpP+84nTk8OsHNq2q6iZe /OgeHqUTSAHxF5iieXKeWqTs8o7yEzzFiUiWJWoUOaKeFuWVo/oJPaW/BmfAXPQRZCcpF4DBUk8y gR27Fj6GfIv2K31sKvf7ywGlFf4A4XRCugXMawMLvQD4OxThv2Pa6Ki0ew9OcTKIh8vsOqadoZjr 9FqyUdM2ctJ6FBdrqj0IxRHVDGQC3UZ3YdfnOJalICohHLEDixCXNMIqXIRp3CaHXAJB1ZbRKqVS iEv4zYXDfXulLqoL99lhn5/7N91VH9vEeYfvvbN9d47te8+OP+L3fOe78zkO/jpwTBo1JJeJED7G kgJJocWifHSgQrc4tBlsY6HSNEOmiU5oDMrKmFi2VrJWCgs1sHbtVLFBpbYTjHVrJ/ijjYIgG9sy OprZ23tnvqRqcu59L699lvU8z+/3PL+CZfZmuZovzdY2e968qO0rZlbgH7D2v59QD2OnV4gMMTXB MpxOkuaA+JCeb83qXXqfvk7/Jvtddtw1Hj6G2DjjcnsoGKDssmQDnMfndNhtFKRJSqbiqVEf8FXA qQnnoib8QyvUCgOpRksD2EIQ8RT96pUI4CIgUorDpwOTKb0Cnn4tbkRTeopMnSXdRBaLve2k/CPT cG8VqpOWwRaLWLXtWUxK9VZH1/R0weqG5kVYnhq82x6xpMNBgXEhrSHkjhBsmI4AV9AZAYyAF6Iu XCxazGoRqPPntz2gzXvE5uZZ8qUdDvpBus+10HOXd8RKGx7d1JGR5/T85mfvrl/c/fj3u/13+AfG t8b3F3d2jh5YoWvbhNjclUuLv9ixeuuSF7b8dO9Md9ZURLbbVATEee44xlsgJFA2EOPhODd0iqzU Lzv8nA+G+TBCQijikHGIOqHlze2kvrrV2pMZaz/RUj+OxuvHYbF+HLSOT/itzfgh9LW6uQb85e3c Um4RXCL2yWu4R+FA42rxKW4z3CKOwN22kmeMK8GSd6+4RzrMHYaH+MPiae40fD18WnyHuwB/F7kg fsh9AK9zU3BKvM39G96O3BZTLLcMkZIIgCSSREQUBdbjRGxACKIAQ9KI8fONyL9D5GAUioKg8LCR H+IBDzmPp0KeN3hSxJ4gSpFxghgCu4GZ4CYMFwM5yh8IMAzLCBXwmcFy+Bly3GPwFVI/2ScCsULe MDxRw9PvuemhPD+Pbh2zyrYpjN0yFMbZy/IOswDwOoM9q9phpjA79pFSwZMJJUv2XW8nQwScBvDN z68luOvtDroD/+H6LxbuD6rDBSDTVhvzy3msnTaQA4H6P2Y/ayCpl6v/XKs8vKE2MNCU6wR/UcEH 7YWV1WuPtCe+MnkDnLvc1yxlaU3jQvp+29rZg3sesWuaLSOn1gE3Gat+ZHroUpy6OayNGDFrfIEV syBLZqmsdIA7JB7jjnlPca95GxgRBIJgF/UN/47A96ixwIvUgXCZOkuxLspjIyOLqTWUPctAPoaw 1donSATAGZybl52KvmBPCBSokFcm+ORxCGCF6p7Y5/6xm3RXqKzR0siSZezYYB4sv8IDie/iST5s 4MzMdkRDgAtJITK0RNu0sW4tw8unvwQLt4aL2GOKZnEWZwozk13TN2YwjKbtnLcqMupHDhetheMN 8YDmQGyacPnxwjTZ08AZdKcJ4t5sVPeTYZyLfWrcrEuc/rx3crFNjZp+4o2Z9Zmbh+G3/V6SOid/ UvrzrpHpg9++sFP6cu3m2dorp8dOga7X9++b40WN4Qb71lruvVN7a5euVGr/eL74UuPES5+d+c87 YNXZxQEf0nE1qrgad9q/SAQImXjPeLwBNUS+A38A/wDtI3CksQQP+g75z6PzkUuQCfHexohI0X5Q Cu8RyQTjkBAhK7SE3LIalJukhMfjJpsSgQDBCB19XkB4oTfq1b2G1+5dopoVGe7KGyqIqmBIPape VSlVDuIcd0RZXwfVHNCWY78ext0P42vNFObFt7cnkxacQljk/FBrjIucMAjCfrxEeGkQIF/T4F0k n3uOKBSxXou5vIlUnjfjopqXozacaWmH3IwBJHhI4Aip5gZjAaF5eY5MYHtZ8Fb5rdqzH44OToF5 tXdvPrZda5O3U9tGoyltrPbGxdonb1zaIIBFIAiawMKIqdc5BGH7JUYvR/zNGDDym4WvCYf1l0Nl /ax+Nc8MNg05huhRZpTd7dhN72P2sWxMQhFZ0SSUlFVG9ngkFjG0TJKSA9ECRCRQcTuM5IjxZIZI wzSZrpAXjcWpVBJrYTyCpgQhwrBlhnGUu+hRmiRoSPfRFD2SKaeSUjqLH9gWLkeRga4gCq3szw/h sETlCahosSOtV0+DkmXy2BCxdM1h7uMqNkwcaS2sb+Begbea1TSw5eCPmZMbnL5BwH+B+mbyYI0p vGyqMMereFyJqzLfiEc4U6v4jJIt4O8jb+oZ34EymPNMc6tD0zwe74qB2mWYeGhy+xa9szvx7Ox1 XU9Gg+HYKt3m55r9uXmJJ+1kdUrNPFNLbBTURK37seZgNNu5q1bWgtDYSBWfExNa7Y9b+/0cZkLG TEiYiTQgXk1kK0A02rRN81kb6zyepQ4mzyTPJf9EXUxes11zztpmneyQfcgxirnZbd/t2Ie5YWgn O4ekZZerAuKGm0F0REJBWXFgcsyTFjtyeCQUkFVRQnFZTaYSTsZlw+EEqC63O5gm1DiRgAkyYTKm NeMZLhBkmpOJMtECiBa9xWgZarG1PO9wSDToo8GvaUCbHd9JeBQxciTzQAXMWAnmVqFY/diaAuBf C/f4wHTw1jyEC6LOSPXujnnBrQNnHsCbPGBmMqSq8o1BHEhzOb98vxjukoLfB8c+Hehzaxpo7ln4 qduJo8jc6hl9VTzkdkqYaervbjXc8+RTmInry75ay/ct1WqDm+Umb0jT5ka/Tm2r39cur1uTMOvB 7CYvYhZ6iN8axaHeo73v917ttfl6jwjG/H58S3ol1CArioQEWWmVUEZWeiTUKSukhJyy6pMQklVc HWlZzUtogazir1RjMdS5YEFDg5PMpNOCgBivTyENBVxRQFTRlSHlqPK+clVxKBUyaoRh7xO9b/ZS 0V7Q26Mp+f7WJ1rJ1iOL1n8USi6HM8MmwLA4bGFc7bjvlPhVR/iu4dXjLza9+4j5zaHT4f8cpPL/ B/nOI2CcHMHoJnWdXGjpHMOb0vXqr/SV8abqmPXW3P/RXfWxbZRn/H3eS+JzPuzzxcnd+ez67uyz z0lsX52P4iZdjmVtyTqaaDD6AVmpqFiAQpOQtmm7TdnaLjTboNPGCmwLHSpdu1YqNGsJg62ZNNBY +0c0TVAoEpkErINFaFMHG02dPe85gRRtf/h9X/ssy+/veZ7fx7UX5oHHJ3Qlhj4UnNdgX28Rbkm4 ccvso59iDz8r3LWoEvct+hqrRCNarSGsRJTscXp0x2pW9M2BLS18VKW6IUdVUTeUqAp6zBtVA3pM DGAj87JiePv4YX6a5+Z4sPlu/k6e28RP8lM8x2/W+vRhfVrnbL1bv1PnJvUpnTJY1yKWKIj9/S64 bssWAayvN/8PIgsY0qHPXBrhcMEwr+svdp79kXt2WXfuL1wA7xYjp53WlSJsqt4UpFukPml/5Qn/ pFkqymCbjklDfPG6EfeitXJYqFUoUDvoBGl3EIITXLljKlaVNxI2vF6bd/hH+Cf5U3zpOf4tfg6v HDLfQpswHTcn6NJxffoptOVr3+5ho4kyxa6Kbry/vS3gDuSCRAkhtbwyVBluhYpytUJpJTigba64 D/SgMFV/CkFZ0V0vY8lRcnPhPEwXXCTkjiODX92q6A1aY1KKq1kXmNKki8a1ex7/7fd72pYq0brb Wz5/KzdWBAeZBnnwOcRGI99xVIFgVCIaOMY6+jW6k45qj2vHtee1SjAm4GGn0bel5Sv0jiUUe4DT jdplamCFUR5VBT2mRTViEwdj21/DAYGGY5TjyUnYSifo752KWgnxKh/TN/fMkxaCceUK0xAm12/3 MCzYAA2wAZI4/XodrknMD5ErGC0lP9YHr77beJtZ4wrx3VvXaUJlbu9dP/1mL+z0FA6aN2iD3H1M hE2oc3bNnrwlWhPMbMe7GoSU/RPvasPvnMt+GXyEl3xKleVP+etKbI+4AlZkN8jboFe+P7tLPgRP ZM/Lb8iX4X25qkpG91Vmr7K5FrnFXi1ztXZSTthcmVxqSxJXT1L4rpUsl/Jys9Jst+e6cr1kN9kh 71IG7VFyQN5vP04O2cfJUftw7pncBekVeTL3pvS6PJWbkd6T31Omcx+Sj6WPbPMm6JRWZTfCBum2 7L3SkPKy/JL9qvyq/Y78ju3zR1WvbmhRNaQbmahquUTI6zHBVRo9qibRWcmyQSBIZIWAIsssPHzO zgZtWbKzMjpk/O9SSFEk6uV5Qmw7afH27Tj9SjZjaJp+WH9GZ6M6rZfpY04OcoA1fOWs4B9b6g7v J67rwx52wGbOFrCQLiMuMCOLmPkRfj5F8Bgi2EH+lChx+Pv7UX467ljvqFkhWNkOxUXIy3IgLwti nvByXpqYmzoj5SU7mGdOjRRfGwDnQgfWEddTaAKbBWARYyx6DNyqa1dUs9suWDZ6uKBvzS2Yov4O b8Nwdh16OrM7e23SXhervfavku2zO74RrTPNJm2A27HRiiTNq5dK3Lezo588GL36XeSWuXfm3iv9 JfZUEkadNaMiiI8AUKer+REKYoRCkqarb6geqn6MvkXnqKfaMESBSZehM+kyOFbPWJDVMyaKAaDU EI2gKBo4b085/uRJKPd6gaohXvRyrA6OT7wlENAEW3AEThhLMZMsoElOgZaCw6npFE1VB93v6bpt wKQBhmJtfqoYQYostGCY8fAu46T2YtFmZkaK9SLoF/JurTxCGzNvAx3rHcsrKmIK2kle7CJfFDeR jeI2cq+4W/wJHIcX4Ix4Hj4G8QMKzOttIGgq+rG2zxM6d2x8idhO8X+O11a1ixNzl89idzjhPDue nt9Udzur5FFS2PGi4xfzYq2Yp0INvpR8NX52uiKPPzNV3P59JpinTiBPFuLm/M7ag/Rw2B1N18lH 7LPt4hpMFfq4Faz0cJE1RXz222qiCzuEdUTritZIa+mXZj2cb6HmVw+UfGH2N590wKmVDdVeHBxy E6bQIUyhlUQlzzpLD4nHPMfLjwslO2GXZwQe8pR08FUW4WqsMq/cFuWyHCWcwGmczTlcKdcZmU86 WsSJ0EigTfBqXur3Rr3U2xleFB9vFvoXcs5M+4yrHTlQ/WZFIpSoTvgqA2migpyGoAdPtaV4Esqr 0qBQXES+Jk2kElwWo1X/LRxFZHwMNzpbl7WweBUQmLiIAQE96QzwsLewu/B+4XJh75vnPjr7wIGH 7x8/958DD2BU3Fb4c+F8oRcehjbouPBs58ixwouFX40/BHVwI9xx4iGGDXJuSb3rKxpg7fMkg1f9 4fLmbGa7PKgOhr9u9WUeDXt2yc/Ff21dUi+F34iXKUkhYyXyZj7ZatmZjcl7kn2Z4UzFywRC4VR4 Tfg15ZJaesyCP8Zfl96Iv568aL0fLws7sYjF+xgZGhBVPXoMqbJGj5GI1lAXsdpjXTE0t56aOkyb NZT38CIJCSE75IT6QqWhzsx8CUgGnMwzGfpkZjIzleEyDWD4fWPpzATsHNeZ5a5fe13ovLkDezzB pf+mutvMhqIBn0F5z/bMoMaLRWVjSTSeksKyaSVSUqIR4mFckkpdI5hqrHFREu28FR3BEqSHWGuJ sURrxcJECTBSxd5GT9BD+gdggE1Z/f9gQNbxObxhkDmFZK3rEZqQFw0PHAknbm669gLqZlBF3YR/ nP3TwUt/WDpwY/OXI72Hbtp3a2M33VPYPhxF3bwhOshtZac1p3cfnfKtLi//+fD6Q2uqsZ7Msfdi PS3SBOeciREDxP2Jl2IvpbnO+C/SVI5KmbvjnBe8ZsJcTdbDNrotvgf20AejD2o7jCFzFEa0x9In 4IT5XOLF9Fy8pkzbB9+L70s+EX8ajtCj8VPpc+mL9gfpuXSVSGohREUL67R0eWa5fXf8nmx5HU/D YaiJqn7dIKalEvRsPnRrLCLEHNpgxuMGhSDa0/hJqlFPXeppDL3dnjs93EHPYQ/1EPVkuGkCfuD4 c1YkEqZ+nw+A8KKOPXB6fTPbHL2rmeindNqFckj1M0ILOC19/+W7amPbOOv489yLz87d2Wc7cc6x fXe2Y9fOxS+J33KOGxvS9S1h7dQsbUNMgjo1E21parqIbQo1fcvaspG9l4rRwUYV1KJUXV/SdtC0 Y4wKoXXwhQ9IDVJB2lhYkLJ92ZLwv3M7BB+wfM//7rnHlu653//3kr2TJbNpc8DV8JPMN69iN/pS DYWyHhBUVUdDooaGxH003BfF+XkB4FCuJGBxwd0kzE9YQRWBbbFDawJYCfNYmFUnBHr8N21JUYdL rE0KyqFYMJHCbRIM8UBrCgWbk0p7CiMV4IANwOAKQKJiKOlVFFqZu8BpeGZl4UK9FplZmbtcb9An nC5cErSkYAPCxDWeBBX1+7GBk/+HIwa7XI24/T6SAEj048uvLGdSCi8J3nBvxkCU4cTwP//8hx++ cRaLw8f3frHa6bXcevf0ofwO4ikC4+Wx/8ZV8RdPjM+El58+uo0jXsJTBw+cdoI/q678laKBRzuI rSW34+VWbMM2giWRjYqgKK1uwpsIiz0/g9eW7mQ7sk2khxoSh9xDTUMeE83TVtQym6f2s/v5/dYx 26g0Ko8mRpPHzEfZCX7Cetg2oU5RUynBwaf4NJ/xpXxpXwbsEBGjFEmRo9FYqgt3EUUq6U5KSTnp X51enVnPr2/pY/v5rUJ/tF/1yVgmPCk548n2iX3uvqbt7YOpwfRgZjA7kLOSLBt1sp5okFXyndFk vuKoOI81n2ROJn6UnErMRm62/FadzS/k6x82d3jQXsIzjd/HBD6AMb6GZsieEp851eb1+PbKHkm6 5tNn0u5T9S2qWuCs9RxnVbkWKxW2GMUUxEvgZiNtZDBSbyHO4ZIUSGMsh3F4BgdLQsJ+w07ctWPF Pm2/ayftM8TEFfmcpArQmfoC+XQc34h/El8Bkiuty5Ti78MFieJKPAnUR8XfxmuRhtdisQbyclnd B8JTWZxfAuJbqmgJtaZCBtfpdg8GgLRq1X0eEj5eBDzPg8PXz8pY2AfnBgtmm5OMMxJmWy0pFLXp ROiEgUnCZV2MSyGWa1VXCUCLNmu0JeQAajQnTDrgVYMCjaGmXgB9AH4ZvIllB7uTHxF2qFR5exlS k4r2IcNZcqxo06ikTUslbYZT2I7twTgRDJga9DgpEQZP6r4xYGKC9pRE1FC+KtwchpwFMctIWTny bMhRPjf4+DNq14e/PtHzydudafmdJrePCYWatl3aPf58Lr9q+c0Xe+d+ufvJjsYmfx1oozrx+jcO PNKV6hnfueelR07dtdBFKYE/eOH54cMD7TtbpXf2/6DvhT9l3HJCT6hdoJLnDZW8XcoP4AFiwDcg 7cK7iF2+XZI54S/6N/lP0q96pugzHobAPsmlp66ARWfBICMGkUwINrN/hpgtOS1YRaVGa9Fhg7/b jKYRhWaIyBWzJdDoklVJpzerfhtJgjQkvS5R0jUiglzE7FvKY5DPhM8WyzqPScCGbEZffoG1pWHv 1HtCAWKbseOoxGbgeDD/d0P2lgogY8Jt4bbu8EGhnMGwvrfB/6EX3ZDBdjupn9rCrFMe6bsBpiux dFN3YG8MRdIbmbBA9y7f6mvO5z5ffOC2KM7q3D2Iu3RPsWFlnjxGTqN2tJp4tLSX9R5PEY4tWexQ ZK1anLJcriMdqmMcjaeOohPsiYzJ53DlhWK1SFm8vXSv6SHloUBvvlQ85jPXWRkFBTbgnroN7IZM T647v2H1VnaEPWI5XHeYtfW5DrkIuThUJIbNKZQuxKOx9HXsAafHrcxetmhchNU4wzvkMwK3mSNK MAxzpGKUMY7iCqJuZ6OstkkcEveKZEI8IBLi92QBCyGJSRZKBaLQSo3GqjEilom2JmfItSU7xcZn Yzg2HEIpnuPS6dR1PIKagd5nS/VWDYXkUDU0GaJKoYUQUQ3h0HWiGzGoAV6VrDXM4JGS5ElobUzJ qimgfFWGFBi8wODNDGa6u7q/fT8NVCrq16AvVUHvabWwdD/CCRANUHFx6V5ZmN9XnK/AXdWu6UtU NaE7G/hqRjOvy3R6g7Qz15HtIEwWc52ZMEE4DRCmDKuBnfQ5vcjhtMm8FweCnbTmRR3mtIIzadbh FbzYGoAhbyp49c6s9bXR4ara0tLyfV3ZMERE6Gdo5m0Xig7IFWB6UAU6+2IbPBp4trkLglEuW7Wc Ag+rZwlOL3MlltVEhdUa4fCC+pWaWK0OXlYuotc6qHVQLVAtX2aIBx8IECEToytjJp3LZnM13TM1 NNbX5gxeaHRBwtQNVi6ba6ipKPwGiAWmiHXPNmdXDz0tRX//8dYtxVCYSIRDifOnn3q40+uoa7QJ XENhdGdbHr/aumlNf0fv4T1298Fvdbet+W5/87GdgUBrPt6ejvVPRuWvqkeWbx/qrGf4Qscra17E 5YK7dVhbPwRdsPL5yj3yKv0ccgEyFkpPCmH8pumy6RLzkUzR4W6+nFXCT5Bj1FFygjpDnjUz6xic N9ev4r/ilOrXiI0cojwuJPixfzvBeEIWVmqT6UmaGKar9DRN0v/gXAiJzRwn8Jv5UX6Sp6ownOdJ xAu8wifhdJa/wzM8wP9KIcMPh271GMBSASpG0AQoLZUrNZ2oFO2N2qfzX+BPDeRE3ArJMmGFlBTc VCd6kVtkOa8ZrmTKr2A36/Ein8mjIGSkFYP2DXu8T4dAGaABLwHsCVPb+ZrFXRVK2e2u/zgVE+48 curZP/7sxNnNP++3KaK3xYqdsdQe7euvvfZYJhMhPrv6rw8WX67m8+SlH69vEoKjS5Glv7Snfnfj /K889cDKa2GHNwLP+PHBi2YKO4J6p3+nJZZGQX1/G/mtNOF19lFb6C2mPmabZ5uXGaHH6Cqq+i96 3lXuKHPob7Qlh9fhfvFR71BwWBz2jokV73HHc85J+6R4BgzvdPAtfBO/x7zn/tB8z/uRsohFE7HR 8W+qqwe2iesOv3dn353/5HI+2+c4/ndHco7jc/xn2CEOAR84JNSQ2kCABDeNBahdpU7EXmEt6kS6 lKFANWeVyppqI6DRdmPSCBCYQWvJ1rWUjqnRNnW0U5VMyrpJLFKmRWgSStl7Z290tt799O7eyef7 3vf9vm8vf8p3ShxtWm6iLSL85cMFIKLhQzseeADWiChCriCNSgSQOEmUclJBGpEmkF2dlmalOWlB WpbqpKc888hC3RJkA41C5l1kCnFRO/ikJ0aapDs+M8yay2bCHOFAFKigAEbABJgGs2ABGPAJAlz4 ZuNYI5FrhFONsLECzSq/TEFAcZRIRSmV0lPpNenrxPerNqFU7FsaKhVXi0OLRQ14RUktLRU1sVnk kWh0dHTADhxiEIQACQrmMuAaki5E0WvWpJ7jsHudRbRGJJ69xCVrcXUQFiGFsCYScaABX+3bLVVi 2qo0JDPy3bEf/h3CmRM/j4XWey2mpqaNBzfsODe+//F1cfjE1fcgNX8XsuU+f8RvP+LzZvafO/8g HX4B95Tuh4vIe34Ptcw2GEb2/eHsld7eeAQjvlkJxwuRF3Uv6k/qRiMXI7MRWo2MRggQEYJ2Zbd+ N9OvnKbprTQUI+uMvcY9xtd1bwfPRujZyLJCiCIQpRsIPBNSpS1dYlZ8UnzK+Kx4VJwCU+IF+jr9 QdDkZ6wt5k2819pt97QIm9xeT7cP3WbShexARuj5QjAU8pEmHzBJZhErPG8vCKPCRYH0CRMCIdxr zVHoWa8EwnFcf9GboNLh9LEqIZHMr5aGkMDjD2rViI1LmI+cRkjAPeJlo1/RMS2yn2kVgaJDhwAt izCoD2lMrCYOMNSBYcRJFIcPRZGpqi7ySBcTj6hYVUeHvilhwZ6rhhNxKz2aOb3w7/deyCJKNip1 0NJWLwmuNtOXy2Gq60BkYEt++tn80z0bHrz/Puzt++mPNGY++Pxcr9vSVLwN73aPJLNf//CjP2HU tiOG7iKngQ14iDTKDMMNQ84CKNg+IfVO0Y3U350UVHfSh2E0pjNxxoeJ69PeVCCunc4Hw3EX5TQM WJ8Uhh37GvKNNCQNFG1gzHr7Y9Q48Qp1wnySO+75MfGzhqvWPxKf1n/GrRD/Iq086rkMR3NMgS4w I6jNjht+RX9Yv0ybdZCue5kgDRh2CsGebjf0EL2GrK+f6DfsJ0rEuHXcOWk9bzhvrDBXDdPGW8Tf iAXzitHGzNEQ0HM0IdI4rU7TOvrbOhuICnb8rFY+yQ/bj9mn7PN2nd3u+oMO6ioP5xCvdbj1WXG5 q27lk7qYyfSEC7pkC03fYYSAK1kvwEPCMaEskMKKzTbKwCgzwRBRpszMMyTHqAz6C8w0s8BQzAXW rgPj6O1WyJDKR1mVzbEkYDlWZMllFrL4SQzoZbJpb7om+chL9K0Wsd4Xh1BZQpaBw/wvYQlQSpZk BPfwQ3bUwzWjgQUhqdnEjg5QHILpgRkKQIIoDmo2Q0uoJS3S0ujHTE1Js9qWrEODweoQSNLVgrf6 ZVd15qpeq82M1ZmxOjNoM5U1JO2cM+kULck6NLQd/X/Nf9BKVR2/oyY2PBYbWfJXM8Jn8ODBE/uO t/nsH73+5r1/Xnvjg9UT8Cd6znmgfdcYsf7Oc88deN42/hcIP70H6d9e6Bxo7lBfQn0kCwB5VP8K UKBH3XhKnLRP+slustu81XmcPG7Wv6GDkbZj0gQ1QU8xU4Yz3BnLdJuBoziaGA4OK4SbYWe8zKtr 4IyXrpCM6mvyTnlvegmvpVl2QCWHrGQ02MpbKIY2cgj0Ctx5pYzsY4W4fxkGlQrk1LpAK+TrLdyr 9fWwGQN4pVCIa7Wzs1pTqWptjmlVFdxSfIKFGPZhdoSdZedYinWGbpAUSdcyYRWpPpQBVzTj2IXK F0OLJS0MdHWtlrpSq8g4RhRF0xZebrEJftnul4WAG7TYmt2wpihYRgAa8CtJoR0nhcRa5LY0u6Up flXwkcmyr7XDt9zyxl2rn7cGNjsvXx64WnxmoDPudazN+Hz+sOr+B7l99a3RNaHm5kD3fmLf1q7x dw93t3V4E9I3rNbY059s3gpIsOHLHvLPqL+vB4+BQeKv6nd4IfcD/2Q7Cdq4PHEkeGQXAYJUmNp5 StSl1mXzh9Yd9o/ky7qyfszxckM5cXLj2Jbytu9mX3O81jCZreiu62ccMw2347e3zebn8gv55byr UbSv5RK2dl9e/zaTaU+5gEC2SxkXcKZ5C1fP1plNRoPBarUZGGTkeRlDYDOncFVNvCk1JV+Ub8qk XIFnVHZAGZUgPyVdlG5KpFRbqlW0UsJL+IkMzKjobEZFpzI5G7RVIHONSVCTaZiukDHV7MwYI06Y c446Cec7xO8BBQxkH+hCl4wU7dwBd4RC9X3vklFkCLzomAR9ZFRt4KLwULQcnYqS0UQyTI72w37Z F4AB/JweR2O8HIDZwEhgNjAX0AUOi/loXs2fRS9Bn9c2k8kcz7Pl0z2wJyYKsF4YET5GQlQh3lGt kymYikXJHEnkSAhIjiRI/I+cnjiu19Cd5DP78jfg88iGGS+NNyjKfQVbTC2YLJUWOaV4H8lNCdmM RbQnl7ilmvKsfoF1KMUtlbgVNNBNSIuQDM18LM1LxNBgaWUJNS88l+dlPMd72MI7tHyDBsT1v1Hn 6La9nVuaE26PowHq/fLXYmtj8RhJbfJn/WE56N8j97uhe73XDbYl+kSwGaZEsEGfcoNcW58b7FT6 Rdjd0OOGu1v2uuGevZ5OF1ruWg+2xzIi3JZJtKtEWkRKsVHX5YaPR3a4wa7WHSLY4ki7NQtcTUaP Dhpx/vcJIg69pCWmISymRU06VWOYQ3sgwfE4Ii1f4jUvNQj9tQSDlM2hOWiqqalmqLT44tC+NW/d okUg9NXugmvQAkfVYLf4IfXVGZon+vf97uxY4dcKS1J6sl75Vsdv3uzuDfmk6H/YLt/YNs4ygN97 /nP2+ey7i//c2Y7vzvY5zvnis5P4Yl/i1ufUTbokTk3bJE2hrdcGhjagTiSmqRssQ5qKEEoN05jU TiQS0jTBl5BVIx9AClM18QGN8AEhxBcQ01jVVYtEGYi1Lu97TvoHETl+Xr9nRcrzPu/v+T29rQ8O nb307Juf//rVaQ+rE+cLqgGCU0u1QmPmwtHhzr9z+dGlX9342XDh2l/BrPLa4ndvmg6nm4uQDuex 1uq7gT4jwEqE3eZwe1snli/+cGFohOdT4+6L4qCYPIdfef7yjxfGVy6vnxm/98rw6VRePvzysUIo ZIdNBfNC+v4Dmt0I9h8zVzIzOllqQgmgU3Tfaqldsm+Wdkq7JZvqBI1Ss9RCW2YJSC5eEdhtG22y iawipKcSpCIwU8m4IvRt23ymltTTWrUg6DUgpUcwLDZgJ2CXYFmGDPOyu02CTRLQZItcJ39H2klU 6qksFpc1MdvINrOtrH01287im1kAYZPdye5m7dlm8S1obMxnZ9EEdd/qpihCnqJjh85WZo3uEIVO 3arLQKTX4XKmon29jnAvIFwRIoawuj8ugeUVaNrwdqiARRztTq3wdPcZW4SQtYTNSVi6BnehSB9s QosD9Uvfqc62on4fmTc7h4PmEGkTa/nBZ6eCxkRn9FAywNNiJJjzgR7H2v0Ll4/Of8n8aeeXC3DY kuV0HzMLaj86lysc7/Se00RZ9pOleduhrtFBvyhDhSPgyXiwBPZnM9yWQVNuyW15Q96THZLckHET vcmIB0NDBSuWRrsxm+/GZMqKphaOFOCJ+acSXkXogeeUDlclIV6jwpS/7QROA8MSFOHvIdtu4DYQ WraO6CiYdEW3PUdR3rBX5k3V4NFeZGS00OZBgwdNvsW3+Q1+j3fwW8mtn1jngyTlDjoU6NZ3uv0O Egfygtk/Hag21s2EuUfD6v7Egu6Q/2GirTynDxKtZMbGMpny2LfDg9XOkSNa1E0Ikd5+Hwg41tCD ciYz1onfl+YNmNlIeQ48/fqAFKZlmMcHFzsT4KrjKsyjgn387psR4AwDFf0bRlH3qluw2E21obbV t31vxzZUpwQ/rKo2Bu7sqraIqz8tVdNCfy3sj7gz4aikUERoG/jMHmY/cRQR/YCh1/3AjyaPgUw3 heakbtNUjovA3Mmi2JYALYGmtCHtSTZpK6P+Po6ypc4i1yvXmfvlWebol2sf1e/ClEEyQ0ZXKmhV KRusBVtIqEeTCNMr+OhYqpcWe4HgiyIMggNXgHII0/pkUvdXyBVCw/+T2361XFZhCld/s/HF04Px SJR9Os5roUcZvmo9zqjljnTvK7c/HE8mh7zEQmrhB/j331DjVpYBxmKYnYLVWsTHzAe0IRp4j5MB 8PWa+3Wy7WlT1+lr7PWea+K68Q5JGmEjcp45z54Xv8ZcYi+J13H3beGOiK+6X/G9b3ufvoXfou+w n/a4KmyFr4glqWJM0CvkN2lXDs8wUkrqyxklUGKIIDMHTjCnJHuSWQAL9EfMPxnHU+wx8T33e+Tf SAfnDjFiTBSP4uO008PSfm+EitGCT3SetM3ZTzoWmVPsKb8zTMdigngStzM0wNkev58JixEhrEGy pRMk7hZIBLZ0ciSdq+rCSA3LYR4/w8iSGJAALok0Aw0ADwCAAzjPiibtB/Y0TpMMw5NFDOO2wSfm DE/91uMhnTDv4TBPevLUKoXvUWCX+guFt6gdCqdyHLfOAz4iGsCALMTkXA7TGG1T29F2NUdDA6ta W8O1ZsnYBi+8E3/rG1YBLcPhtY76+iyz8hla3j0LEfmQi2X0qFIOwxLKoZbNGZB/5Ss+jVd932Ju XnHtLzD4BX7/hjJ3ALPTfb+Cnt0kCHhnV1aWoYmeXQFnrR9sGVu2hhDmwcdmADZTsR9qAfyNmfD8 +2kDRxjxGB4UWIPuBnc3UDD8HBa2Nct0YQC7L3Rc1E/1Ql9ajwedToLwW/0WEWEENVKA+MB1QVx8 nMTHb01RrngfWDvx9ert2xcSeTl8uHOkL9rf+XtYq3e0iWTQQ/ukSDDDAsaxdq/1h1oPRQViuCTh 2tifOn98MZ7zkbIMgn5uGDzT2V0s8UCWWQ8X/4JtfH0yyiZRlR+CTKZhlQex679oczvcHmfjrPlg ooCiOWqMFQC35V0aaXDA5Bpck2txbW4DfpGgFIGYSgBFcKaTgbS36hcCtSCGEU4SA7KX2v8zlAVY fazQpkCDAk2qRbWpDWqPclBboccA2+18lfIjpEK1sUYGSNQnKXqQoxfDhclOpaJFfCIf6WcB61j7 vDpfilnEtJnXJ63G073Lzjw0/wVQM3sLxO7ipyHb6iJgF5EZqF7QXoQCICkCv43fu5EoKsIgXJie xIwiTE4lWEXgoAPcSKqKkN+2eW8kq4owARfm4eRcul49JczVXEqxbhpKvwsjUpPzC0R5wJEaoEgP 4bQ7iMmJwTzPkYsQngwrx/MSaEmbEi5tA92ki4qmyqV8EbSKm0W8iPZC9YWqPDMj1ht1fLXeruNY nanjdeTHgVCh3jy9uI2fgdflZX4bLL2Krox6AF3mLnKID7uhPIsIjKHprIxmNPiqW3cHaS9CMPbQ Lg78IiFTtDeV7JOpeC/w0Qlf6nG/gHqhAshiWKqWXvwfydgvY2tyIwju0cE93CYes48nwD0MGks9 2a8Oz78UfGZt+qnleMhLjhzqlP1jcY60R9Pz+nMzOB4cnegMzhgeR3zg+Ih+MhsenO6MVYYiFt7T NAio+CdLdF9m6fwL09Nzoy91np+XQlBGOCbJNsD3WpqpH/OonWnLUOCFOAH3Bs3YQLETPDMSleXo 2Bw498ZAtw3A2qGgT/4L1s4wcJmjOvRJl46qJq839Kbe0tu6I2sHprVehZ82deemvqvjmzpowo0d 3RZzhRSB7qqlogjyVMKlCL7/Ml72sVGbdxz3Y/vefC/2+d59l7Md353PZ99d7i6EmHSNIQRIUkgY FeSAwEFRpwEbSRDlpUBOXaFlZcupTKUsFWFb2apObQJ/dKFauqxlG4iusGnrxP7o/kGMqJzaTagS Ylz22BdeNGnTTnke+2zH1j1f/37fz7dbaJCiQh0tc2JqcVM0tzSCCPmChVFQS0wQSNJFBPwxS8UK Jq2AtA5ax61XrbhVR8uwVGiIpVipTypJgxJelirSpIQhEiWhkl5uNviaSKXmOl7K/z9e0sEQZsbj ISwQASZz0MQ8EB9qPzAE/yBewlIF/5UtoY6PH3xUqwXQ86NXe3Zyfpc9t6TW5tEKBL545d7n7C5d Pu+yHOTKefWqH/asfeJgbf86NmRQJdkL9h4aeqHWMOBvgPos3waePruCMdRBkc65G9gFqA6JNIB+ rZMu+8DP/O/5fwMu2y42XLeZ6b8TYIWt07/OdwQctx0jr4ctrJZfgLMdUMNxFvzWd5lBNRZ0Wak4 YgnErXYa11dQhu2/F+qKg2v63IeX8EG8gk/iZvy2Q4MnNcc4NLiOaEdPUF5F3RmWV1Z1jpd7JpNr eib7Vq8/54h2nWPxrq+v759GHHMzCA4HOzfT2tpa7Oj/JcJgeQRHvFh+lpoNP/YVFmhRh6WqoUgL aKDjrgQajySIuDnhJr0c/KUMB/w2uBe0wD2Pk+JAGIOTzx7gkJAJTnX/efiBlQv0HgslBB39mnsP usd8gDjgOkDv8+8J7olYB4rQB/UUaYtQbjUMh08Pkfb5EJnXo6N5PiC2tAQa9SxIz0dBFLl2eMdz V0euHvjGoY/XLNixZPyFLYe/uRybOP3SxPP/Kp995Z3Dd/cubj998FLtszMf3TlegrrN3a11Y+9D 3UREBbh2mGjT9VCpLmoDdcyNH1VAm9Le1qNsULa7tyu7rfvd+5UXrWcts9a7NmdTW3+h2LyzGdfa QNaKJSXaA3t36GijB3ZwUUBEvleMIktRWk5ieIZqAS1FswW1ZOIueyjoyudYokKgJaJMTBAY8TmH Gpgb5rg+fpBHyzxAeIqf5Gf4a7yJLy36sGfeop6gjCIaruo2ZcCsO/AwAWAuSm+yhmZcdoHFaY03 JxyJpvgCS54DWSecCrYWDuTsGQ6y7QNxjMwGwWNAxuIFX4vBsz6vxVhp8UGXLPgfC2mmen3lFxpr r3dTFDCJ5aO939049PLg290tyXxA7alxoYWix0cJ0WAcNNtc31qz7cnVG7X+pmwMU4c/3b9l54t/ qo6N+Mh0bXZTIRqPA789tw3bWmwKukZqb+8SFvWvevbCH4dWBWm9ypbWunEEqtWgU7n2LkMy0bUU BtKGfXJBsS+Nauly+s3kmTTexDTx7alWuZfSGI3vTa2Q+8k+phjt49enNsu7qK3MVn5X6iA1xIxE h/gR+QjzPfkN8jXmjehr/Oup0/Jb/p8yP4+8I1/wfyD/Xv6rfFu+J6e49O747uSo56TnpHcmbVnj AY1WF8QPcR4/wkEyymICIwGpSNuFeEPQYjG7wmGEZV26ugLCggpAS6AMJgAGPk/kKF+fD/2V76rv Cx/m61A6RowyHhpeWYUSwzap06XBI9X2+7rQtDrfI4OxpCcQCyQ4JOmBU9wvcED0SlxdVN0kYSlB VVtl5BG36HpBCc312IIY3rkQq0sKS0mPMNiOYKG7lve0NniDG17uOvIH4P1ILSUWLfiOuK198MxP drdtxCbuPdufj8TjlF2FtrWz959XZkGc4yKx+1nwLuyaH/z6wkwBepYTivYLqFcSeVPbBUGDzKN5 UkM18gXcoqXA5hRg9VIxyOaoIMJAmIiKSxHCnnJ7OQrgwbIe+SgHcBQxDLFAdtlsBhpMhxk2BVKI G+Y/lgNlrsKhCEdBlpnhrnEmriTpBA+X8SGNDN8YGjZWkaoOVwfcdepQEeoR5A3rnuLTDaUe7+b5 YN5D/pOKn9q9f+GK5piwzkf70k0e55Ina/KyxhBhcgoMKxLAh0188kmHIrZ0eqVNta6nRGgYMb/h /M+c+Vqknuy2zd1A/wxXJ4fZNMmmhBSUpjOaXVWSdjXoLTrWJ8aoH8RMhIVIElKpMFgoF8xkYQpw 2kvwfb/ivOK6GLsY/4vwaey6chO/KdyMzSp2ul0ZUL6dPqSMglF0FCv7ykw5XI4cS49mnCQgUQKz OcwRQrnUeFmwRjC/l474G0JSWDllO0WMcSeEEzE7LTuTSrfSW9hc2CftU4663hImCrewmxGHZM1F kWk0CliQhZlsCsjnkenMFGA0dyoYDU2HowzLAIrhGJTRT4am/frJRpqOCU47TorGxhQFv0My2VQO QUzxlIU5DHPbFLZM8/qz0ThtRz+mAaCv8n/jv+AxfgrzavZBEpTIQbJCYuQUaNFCIhPKsFZgVcZF UBIHxbKIcWKTiIrvw5yYB9y5ngf6r6wO3zGY4/5AR//5OR4MFNUsdJjzcwDuwh5avQHPQ5vTaeQG VZ1H0oAK/YmA+BNz2r1Opx3GOiPMFYMIdftOFWY1qnqnWt83do2ajCQllqPcZgvrhthqlqwR+FpG I4glaYqAB4WpMwy8ue2e5SvqK/e9JD5QhEALXz94MDQOxtFxbNz+Q2fFV2Eq4UrkVONJYTztgM4I kReB5gkvs2eFbOwVZSw2ppgGirpfupNcSLUlQyrQCBWFI6zHQUJldIgIEWoGHlKMYVMdVJRud3H6 BM31fFg1NiE1NjV367xHFeobmJduvedRlaCnfi+6fi8SRlGNho+gVYWj9f/5UiNJeBmpYpQTPsep 3+BLjXbC5zjhNXAE3cZA5P/1gWtThCXoFuZ7EYyngXotGvYiuAu63egRNma4v25ROu+hFT6xd+Oy tRy7+dUr03ue3sn7Ak6ej5ze2rluS+2zdHrs+ZaVBTdFO7CJ2qUT27vTrUkps/yZHx86FSUYsPz4 91ernZsqi9R1Q68HSFdQZ23v3D/+zXbVxrZx1vF7zi/34rc7n1/uOb+dc7473+zYWexLYitarqwv ydaSaGhlLbJSKBUUCa0JKvvSKN4HllQCBVD50A2UahJ0CAFp0xd3ETSdAPUDrJWg0VYhrR/C2EQz lamrJq1O+D92MxVBIj//u8ePlcvfv7c/Pey+RiWotStUYOsDZ4e/NokmaXokdVo8ja9Gr8Za+APM LKbQSQWN+8cDk/7JwCcyZNWobMquWFTGiguRJZI4g1zRPncLJZwkcvXRNPL6bbYY8sVuRN/rqP6R SOLPlK+F7jpFFaSuVE4tpegUhZDb7clFJiTUlBAlCdKStCrdlO5IXulQ8lcnt9NAuxOjhcb9BiQ2 gDwE6/Y6ETphA95aRyB2VMc3nuwDf+/Y/HQBiZWoJnb0frDScQPDFjV7AFRuED2ztlbJZ58STa25 s/TCEz8a/E5v3HJf2/zr7vZvDjxl5b92uDJ5mP5mNnZ01DgC3aIh+7Zdpyid+rtjI5OYsWoSVi6Z 7qpvMFNXRzOjqkdhpXGSi7Ljad3UWBPtYNLsTtWnp9gW2uVIPKXrIATeVDEY5H28z5dViWUGqSWE QugYWkQ3kBuRsUMPYyUXDk9IP5ToJixLkov0R33UIeiP8dbsfxsACAB0ChpFxo6N7uhByP755NHh r5BIhsRkSElSgpgQUkmqM3W8/DLgEtjXMdH+wbhHs7dbBobA2NlHjYQ703YdDmVjGTO4+VHvd0/s 2jdVTA6Ooh0HRgrffrZ20HWqfWtxT1LUpt5qfuHA95vo9I7+BNLbrzUnBvbSzBcHaR36KUI/N6Cf Kl13uPCX+Bfkr2AXbm29c95n9xAOfjVqR3BE0bgePiuq4ZysYlWpczW+Hq7JNq4rz7Bj3E5+l7wL jylH2Z+yp7mfKa8mFnt+Sb3B/px7Hb+uvJH4PXuRu8Rfki/jN5WVxGrPLfkB/0D+TOld5BD5K8v9 h6qdWniyW9NWt+7Z062m2a2a1q2i2KmOg5PVUM8JSB3T9DHPCfVlz/fEhR6uzlb5qlxL/Mm7mn1H Yeb5k/Icdg2GR2VakiNpiUqoaSrMi+lwa+sVp8gpWJUx7uP4CMfxCUXJcSxcsYzX43az4EBSGFyC 8irYJ7dQyglP8kjgc/wif4n/G+/hZ7gEQY/geMtn2Cvs26yLneHwcWUFJSiV4uB5Q+EqR54bpzr1 fL9NymW/TXGrHM210NVLQg9q9nS7AadIvRSSqllCPizAGDN9v0GkS2nL72MAm3xf2SB1Wt7oho0O yAgD57ruMecpyZ2LAtjIBhJWH18hhDRgdNvWww7mCmga1P0ir8YCIywI8mWoXA7iQWvrDmg0D8Xh pRqrgkjDq5PmqY6UStloV0YlqZPQDc3ORr2QaZCGDIOkevTbpGlFb63FWV9PFRWqES25uWJtXonl M2K/65RuqFrfppcODKWCXMin624xvfvhRy7PQFngWKKSga11zwXAadHVcy5cIl99csSOmRDERN1U S5Olo9yx0of6h/lP9U/zfnLgvGR3zl1PZKrZUsn6+kAK40xCE0pu3kgZRaNmPB8/Gz8rnzVYnz6Y GzTHqb1oHzPG7sntNvfl91nzTFNoij/Q5/PzVrP0qnCKHNZXhCv6lfzV0nX9ev5d/d38zVKG8rgZ b9Qd53TG5PJey44/LTwtTnieY/bLz1knfQvCvHwSn9Tm9XmjWYrPca/E5wxXgDuAXhJeEt2ANOiS rvOIAawJcTEtqFo2rVJWMU2F+GA6lMHpdAaguszmTbW1NeM4sp5TWYblmJyVj1hWHrqsm30sF2FZ DnwBR3O8HuF5Xcvl+mQckWVsGRqGUAuo5ql8dgXdBWim0d3lDAqJ5E6gguAKfCgkCBB0VYomm4gq whGAvryCvgWiy6JfOKG8Aw+by+V96sPQER6C2bkLq9QRS2sh1ok6ifIERmcw+h2+gd8DLflxrgyk SVxWQzoSdKQTgPv8VX0FCZRBRYE3focvTxrIMZoGbYA1XeBmzDL7JpCHBSPjYehAzfy9PJ2Hj16E j+bPMJ3pdcJCTQtRlmCplmMtWavWTYuxDvV+7lcb9wuNKaxstNchOU09YgxsKbABb8vrCpgYeREK EQIpxMlg/IEy/Oi3e73RDWvAqS63gsAtdptk7OM7hf9Ht/9dGYEdZoc7NJxCZHiaJrGkUSAMNISI f4SEnWWoEmFfqhZ/rERIuXc+XtNJiXbuzkW7hCQ/hI/eLh1Nwr4uGbfp+egeaa4uOwOoCa7yhz9W ZTM2jC6MpiPszWsRs4ayX7Y237b+sfmJvnk7NTQMLHWnk5li+9/o13PD8aBL111xQYtE2x+jzwZU KU3reuDow3/RY+3LLnqsEgDwJCjK9U9g7ZDriLMV3sWjhcBCcEGcM+aqa761+G3zdoULlQxe9+X8 0/xx3/v9TLJeCh0ccJdGPCPCiDhkjORr1b76mG9cGBd3p8eMvflnq059P96vT9SPM7O+WWFWnI3N xn/CLAqL4ll5xUgHPSEhJIaKGSEjZooWb8XLdV6oP88dHJiouw3gAkxaZrVq8z6/vyLzHMNgo2pX K7YeXoiVRSTa/kAg5k/N4Ik0Spf1F7VZjdYWNKRhvVSqVXo/tiyzMgH/4IyNbI+H0THD5Gw9Ytu6 P2aafRV/pFLxQ69lzh+vmDr2DZWdJwyZd/mrjB1KomQmUyyXS5JAD4ERiSJxl5K7F/X2ptMp3g8Z 5eKLMRQr6S0UXFYxwkTJ/ILt4CV8B9/DbrJBXAWv0ANUhWLQN87bJRMYuExVUGWFvkbVqDq9bzn7 FyBD4UEDpgahXWgUpjYgu3XR3th2DcgqnUUYbpBRtRPiCNhhDJkLznShTS6QHK7NlOW7wnoDjpXX yQKvRrkBO8J/2K/22KauM/6de69jX9v3Ycf29SNp7Njx+xVDEm5evhQIIeAkQChP8xCkkBEoCa9S DQltoqyttE7tVqqKLZ0mbWVdYQzkpus0bWsq9Y9NQhOb+GMTaEs3JjUdqhiaOuLsO7YZYSqjnfjT 1/6d851zzz33ft93vldpKH/5I6T0BrlT7Dwlyp3Hp6ZoN2WY0mNnwFk88uP5PI04YzCGp/0dMOEh Nqqmybl/vs2rCi0VkL5xEXs79hpfZ8kKmkfOOuksDmiv1SpiVqdZTVm9E5tWSrXTiIp9JCzR3W4W JLXJK9G4dfUnkqqnliOpGewKAt4QSjO0xAh6KSw4Z6HPYeZTinVYhJQ6SznyeQRVRv4tCAXrEVmW VAsirtnV2rIZOsqdlcYeOy1gbmq1drXVYFfDaZsaQVgMDpUvbeZQI5oFYVczFPhmhb4dQR+/YLln zPdf/13WkPtuUMO/W860UVO/G4f1tQ6HYve1ZOhsKESdQWlMU/A2Gq895HzE5zc5Fq/sbQyS1uZA 87rj00O9anEw4arVnn15aSJR/F3AE9z0i3N9q7vQFdQpzozcuGfPTre9Hh2Bs3H8B8XJY81sIGAT FSU/NbXZ4gwxgYDOVn907s5oG43h5mIPewu9QYaMam/yMqpKjH0rwtQuTDp2tX5Vd7KG4Xmd1eAy uPmYzR3kA9aAOxhbRFqtLZ7l1j38HuOI60n3Ts+e+NOGY8ZjrqPuQ56n488Zn3O9Cq/yp92vxN6F yws/rPFjQI3F4tGokZSSNxfN+OKZSsYXNHhdbnc6arThgngsVsr1YlF8JOrmOaMhjr0Lw6TBX8n6 QtT2RPzaUMqv1ksLFcXtoqHO86KRXDPeNDLbjQeMfzeyxuNZfoDfxrP8cSwyRK0+9nvJSyTvhJfx vrgtTlLxbJyJuxYsPOv7PlYMsX5M3nLT+bHp2VtYUOXHZvuXDS/9C2Rzs9OxsmWiEZKSJRrmhR3s qZU+MMrciyxkjJpZ7LOzs1J6Viot5tW3bSRYihhm8qY9kfBd+41Fb2iMkWhT2Mm7ii+0nl/dsaot 7VPDxseWBxYX35Z8LllZgMchVB9aVsyQTyNhK28SMH9z+sTsnf0nv7Y0Hl3gkLo3TjAXG5J+s2zG cxDBqDCK58BOtmmLrAbOyU1wE8KEeJab5PQTChGUw0Jz6yBskAbtrIdTxFppK7dGusZdlvRwW5YX 2x+jJWxAy+h0l4yPmThRkgIsZ2NZjjUxnETMoiKwEiNygzqiSwvmGnmbRKQ0YYzSu0w3iMAx3Vqc JckJ/JrkoEDSgiYcEFjBnVKyyoDCKuakqQUYwrgcynfLLrT/1lju1nS/nL+NWruVn5bxh350dryz 1KiokpLzxCoPgTn4qeNTTiLPYGL1j0pHXR+MxzDXLvk9ce6yxqOXY9PYcPSQCUhIGh0FHKo0OfeH gkPlwjZKXi3YVO6AlZLfKFhVzmmn5I2CHUmpRF6Q7nca6BE2EtbXQnyNVNf+Np+d+DLU4tktpjtX me3FKzs6az1cuIaF2ddI/8hKRTYRV/GvATbq8mf6ik13rvjj3t1ouXMfk/c5CyMCC/XvAMOGNCOA W0d2c8t7MdX6sF++DancDPLna/Fxln/9kfOT91cAA0vY3exa3V5wQAK+ooUJiJxTafI0hBsNFlNY aywoFs1UAIUFNoVmJTU1NJ1oYjHoRTXJ0/46urf3JLFBPCGyIp3jufbzNmJzJVOT5NBF39CmshXl ZmZRK9hUDCibw4wO/2V1kIpKVPy8clpEj/qCjMNuq5z+ps+eJj0bVhl5QYhbI119bUtGTzKbhzWT yWyKOyJduUWPf+lZ3d5IcleHXxClrnh62aF1u94KBtu3dNeJotwRa+4dXzfyFszN3ZUCYWEKgPse IM8Hmk80M0CYKBvBEipPveMv2WHyCcrKDQNaPe9Cz6OTeRsUBM3G1qFwdO2S0qCcwMOJorgouzx1 PyNR8MFvSReU5JCbzc/ccyEVxqkfqKV+nlq+v1Ff5q2tzKi+5uPdTW6TWTRZ3ZZwd0O0fcnejR3s cKqrJdjSIEl6vjORqQuODR3ZoVFtFqfYtfABKJCCF7SuM54zybOpydQHqb+lap4RDyvPiycVzumq CwHhJJ8hanYWolrABAWrZjY1Z+vaBxNESjQkTiTYREnFr4dI6D2uXbI32E/YWTvlS3Klm+crlzJ1 Oz+bH59Ba5uZxj/lar5Oxyh/97NXUaPuAfMHh7NGk2B0OBzRzlzb43tPkZ3rc0ajWXAoFlR169LR k8WpqJrvQkUaDJ2xdO/4+pFzgWhiuMMvCgZDdyzdcxiVDf+5jjwczA/LYJOIH98D1494owyd++HQ Jyv4+T0YtpfBXy7D9FEZwlYA8QyAHASw/KiM2lYAmxXAnizDgXs696M5jwN4vg5Qv7WMhp8CNF4C 8P/piyN4AyDcBxDFd8XwO5IvAaSjAAtlgBaUQesnAIs2AHR0A3QGALrx27NFgMUFgCW7AJbZAHoy ACt+DdD3KcAq3Ksfn1mDMnoC12xAnjcdqOKR4o0yNp9DXCljy5H/D/nnq6iiiiqqqKKKKqqooooq qqiiiiqqmA9ggAC9bMBSirgRNfDQiy21PJjMAJIM1rvzHqgH8EIjBCAYCkcgFk9AKt2cWQAtrW3z N1i6rGd574q+laty/QODq9esHVr3xPoNGzdt3pJ/wBsLFy/BhYd/2KO6OLiMbRJ54UDENggRQFZg EbRDF2RhMfRAHwzCEDwDL8E34RVvrdflrfPWz80BlNaHIX7f+iXQC6tgDezA9S/PXz/35//1Q0l6 wXP9O9e/ff309dMVbX2Ri33oCgM8WdmXxXMAFZpD2laha5AK0xPC8TgThs4KzaBsdlVoFufHKzSH 9GsVugbpX/UM5gZWrogNjewbPtg/fHTNU/t27P+8cyjpQcjBAKyEFRBDiY/APhiGg9CP7VGU6FM4 3gH7kRqG3XAYRnE0/rmfetTrUGI6Gf69T8AwymdgBYaQADAVOALFzvJLAEOECRSwjBOAMhwsQBaI B6MZ0piA+YgRDtCjyR4IGByAqWEaB8iYMxx8zMXQ2GKaOWur19wV8fw2XzkkOcCqFz2WPQSiN3k/ 4P5d8rdHwIqDjwGUZRkhcQoActbIgQplbmRzdHJlYW0NZW5kb2JqDTk1IDAgb2JqDTw8IA0vVHlw ZSAvRXh0R1N0YXRlIA0vU0EgZmFsc2UgDS9TTSAwLjAyIA0vT1AgZmFsc2UgDS9vcCBmYWxzZSAN L09QTSAxIA0vQkcyIC9EZWZhdWx0IA0vVUNSMiAvRGVmYXVsdCANL1RSMiAvRGVmYXVsdCANPj4g DWVuZG9iag05NiAwIG9iag08PCAvTiAzIC9BbHRlcm5hdGUgL0RldmljZVJHQiAvTGVuZ3RoIDIz MzAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSInsTndU02kW/X4lPSGQhNAh1EAg 9BJAamihSJEuCBgChBJqAAFBEFGKoohIEwQRBRQUR0BsSBEdXSxjmRl1DgyHUUbWZWZELDO6jOvu 8ezuPzP/7t5z3nv3vPvd+76jk9GwEgaAcGfTBDoAAF6rD2sAnwH6NP+5X3oZDXPA//G/DOgzHmXk Bgcw0ITPdeRT/VFEzUTDwqQ/5okXZQs/UWOBO59lbWZrbc01NzZjuYkS4qSpOawgT1eWMD01XZrF ys6IE4pYXFb22u6zjD9t/O//MAoSJYiyRJI1R6hYlCeWJLL46ZJ4cY44XcISS1hr1/517LOEP2n7 N+SuWT8Setd7wIgxBrK3GAD5602A0kkAiWpbU6DwiEjWxze+hFAQsDbDNJ6mieKyP0VA/5kK1/7e ssWJH338oGCWUJqV+w8N/b1hABFQAQMoAXWgDfQBF5gDG2APnIE78Ab+IBhEgE1ACJJAGsgCeaAI lIIKUA32ggOgGbSDLtAD+sEgGAYXwBVwFdwAt8F98AjMggXwHCyDV+AdBEE4iALRISVIA9KFDCFz iAc5Qu6QLxQERUCxUCIkgaRQEbQdqobqoGaoA+qBTkPnoCvQdegO9A00By1CP0NvYQQmwwxYDdaD TWAe7AL7wMFwNJwIZ8IFcDm8B26EO+ET8BB8Bb4B34dn4efwCgIQEsJENBEuwkP4iD8SiSQgWUgx UoU0IJ1IPzKKTCNfIbPIEvIGxaJ0lIVyUXvUCw1BhWgmWozWoM3ocXQInUS/QufQZfQ9hoJRxRhi 7DACTDgmEZOHqcA0YLoxZzFTmPuYBcwrLBbLxLKxNlgvbAQ2GVuIrcEewg5gx7B3sPPYFRwOp4Qz xDng/HFxuBxcBa4JdwJ3GXcXt4B7jSfhNfDmeA98JF6CL8M34Hvxl/B38U/x7wiyBF2CHcGfEE/Y QqgldBFGCbcIC4R3RDkim+hADCYmE0uJjcR+4hTxMfEXEomkRbIlBZLEpG2kRtIp0jXSHOkNmUbm kPnkKLKUvId8jDxG/ob8C4VC0aM4UyIpOZQ9lB7KBOU7ymsZuoyxjEAmXqZEpkVmSOauzAsqgapL daFuohZQG6hnqLeoS7IEWT1ZvmycbLFsi+w52YeyK3J0OTM5f7k0uRq5Xrnrcs9oOJoezZ0WTyun HaFN0ObpCF2bzqcL6dvpXfQp+gIDy2AzBIxkRjXjJGOGsSxPk7eUD5XPl2+Rvyg/y0SYekwBM5VZ yxxkPmC+VVBTcFEQKVQq9CvcVVhVVFF0VhQpVikOKN5XfKvEUnJXSlHapzSs9EQZVeYoByrnKbcp TykvqTBU7FWEKlUqgyrfqsKqHNUg1ULVI6o3VVfU1NU81TLUmtQm1JbUmerO6snq9eqX1Bc16BqO GmKNeo3LGj+y5FkurFRWI2uStaypqumlKdXs0JzRfKfF1grRKtMa0HqiTdTmaSdo12uPay/raOj4 6RTp9Ol8q0vQ5ekm6R7UndZd1WPrhent1BvWe8ZWZAvYBew+9mN9ir6TfqZ+p/49A6wBzyDF4JDB bQ7MseIkcVo4twxhQ2tDseEhwztGGCNbI4lRp9FDLpnrws3l9nHnjJnGvsZlxsPGL0x0TCJN9plM m7w3tTJNNe0yfWRGM/M2KzMbNfvZnGMuNG8xv2dBsfCwKLEYsfjJ0tBSZNlm+bUV3crPaqfVuNVv 1jbWWdb91os2OjaxNq02D3kMXgCvhnfNFmPraltie8H2jZ21XY7doN3f7Ln2Kfa99s/WsdeJ1nWt m3fQcohz6HCYdWQ5xjoedpx10nSKc+p0+t5Z2zneudv5qYuBS7LLCZcXrqauWa5nXVf5dvyt/DE3 xM3Trcptxp3mHuLe7P6dh5ZHokefx7KnlWeh55gXxsvHa5/XQ4GaQCjoESx723hv9Z70Ifts8Gn2 +d6X45vlO+oH+3n77fd7vF53vWT9sD/wF/jv938SwA7IDDgfiA0MCGwJ/CHILKgoaHoDfUPMht4N r4Jdg2uDH4Xoh0hDxkOpoVGhPaGrYW5hdWGz4SbhW8NvRChHiCNGInGRoZHdkSsb3Tce2LgQZRVV EfUgmh2dH319k/Km1E0XY6gxcTFnYjGxYbG9sb/G+cd1xq1sFmxu3bws5AsPCp/HO8fXxy+KHER1 oqcJDgl1Cc8SHRL3Jy4mOSU1JC2J+eJm8U/JXsntyasp/inHUj6khqUOpOHTYtPOSWiSFMlkunp6 fvqdDMOMiozZTLvMA5nLWT5Z3dlQdnT2SA4jJyPnplRfukM6l+uY25L7Oi8070y+XL4k/+YWzpbK LU8LPAqOFqKFwsLxIs2i0qK5rS5bO4qh4s3F4yXaJeUlC9s8tx0vJZamlP6lzLSsruzl9rDto+Vq 5dvK53d47uirkKnIqni4035n+y50l3jXTKVFZVPl+6r4qi+rTasbqn+tEdZ8udtsd+PuD3sS9szU Wte27cXulex9sM9p3/E6ubqCuvn9fvuH6ln1VfUvD8QcuN5g2dB+kHhQenC20bdxpEmnaW/Tr81J zfdbXFsGWlVbK1tXD8Ufutvm3NbfrtZe3f72sPjw1x2eHUOdep0NR7BHco/80BXaNX2Ud7SnW7m7 uvu3Y5Jjs8eDjk/22PT09Kr21vbBfdK+xRNRJ26fdDs50s/t7xhgDlSfAqekp348HXv6waDP4PgZ 3pn+L3S/aD1LP1s1BA1tGVoeThqeHYkYuXPO+9z4qP3o2fPG549d0LzQclH+Yu0l4qXySx8uF1xe GcsYW7qSeGV+PGb80UT4xL3JwMmZKZ+pa1c9rk5Mu0xfvuZw7cJ1u78HzgAx/8m5yjjKt8s2y7bM Ncy1zTXNtc42zrbPN8+40DnQutE80b7SP9LB00TTxtRJ1MvVTtXR1lXW2Ndc1+DYZNjo2WzZ8dp2 2vvbgNwF3IrdEN2W3hzeot8p36/gNuC94UThzOJT4tvjY+Pr5HPk/OWE5g3mlucf56noMui86Ubp 0Opb6uXrcOv77IbtEe2c7ijutO9A78zwWPDl8XLx//KM8xnzp/Q09ML1UPXe9m32+/eK+Bn4qPk4 +cf6V/rn+3f8B/yY/Sn9uv5L/tz/bf//B8OWeQplbmRzdHJlYW0NZW5kb2JqDTk3IDAgb2JqDTw8 IA0vUHJvY1NldCBbIC9QREYgXSANPj4gDWVuZG9iag05OCAwIG9iag08PCAvVHlwZSAvUGF0dGVy biAvTWF0cml4IFsgMCAwLjQ4IC0wLjQ4IDAgMCAwIF0gL1BhdHRlcm5UeXBlIDEgL1Jlc291cmNl cyA5NyAwIFIgDS9QYWludFR5cGUgMiAvVGlsaW5nVHlwZSAxIC9CQm94IFsgMCAwIDggOCBdIC9Y U3RlcCA4IC9ZU3RlcCA4IC9MZW5ndGggNjYgDS9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJl YW0NCkiJKuQyVMhU4DJQMFCwAMKiVK5whTygmJeCoUIWEJcrGBoo+CpExxoopHCZAFXlAkkLhRyu YK5ALoAAAwCCrQxXCmVuZHN0cmVhbQ1lbmRvYmoNMSAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9Q YXJlbnQgNDkgMCBSIA0vUmVzb3VyY2VzIDIgMCBSIA0vQ29udGVudHMgMyAwIFIgDS9Sb3RhdGUg OTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSAN Pj4gDWVuZG9iag0yIDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwg L1RUNCA4MyAwIFIgL1RUNiA4NyAwIFIgL1RUNyA0MyAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dT MSA5NSAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczggNjggMCBSID4+IA0+PiANZW5kb2JqDTMg MCBvYmoNPDwgL0xlbmd0aCAxNTg2OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpI iZRXzc4kqRG891P02ZJZfpK/sw8+Wz74AVZey9Kykj3vLzkyga7KogUejQTqb8iIKEiSyF/+8qO8 f/3xtqZYh7H69HbvH7/+8frlr39373/9eLn3v9+vQIZCeWdKpmKOxqX3n7OrptL7v/98/fYK1cSa bwt80QuITMl5g8AKolIQrbFhFxLxp5oOmFVhJmtCTBudKZoU4h7TBcYMAzJbk+suIkdTIx0g0x2y OOMcbVQiwquAZEI9UHiSCBoh1ZkYwyGk6JBksgt7WcGrEGedqdXvaUL/+PKJycbFQwzpEOdNcO4Q Il9D7hOTDWW3/5zodYz3JscDT5SUo/iJyaYeQlLQISEYe7gsKUtI/YQU433df02OOoaCibbueYpI i+ETU0xKu1LAMaIt5hkTg+TSNqbGR0wxtewqClZbOZ30OZ1Exqf9vlWbHjHVkD/EOMm29DmeTCaW XRXhktO1ffY6V5zZvlhxmVIxJRrr98WIrxrG/Dmfag3iDjHlERNNoH294eTkmHk+3lqT/L7kcKbp mGhyOcREybd5sb1jmH2Z4qRRIck4uy9TfP4cMsuU986Esi85fP46JplIpxjZgfLRFpzJdl9z+Fx0 DNaWfQHhPVYx5I2jQ4wkAWrzDMnmsM+96F4R0Rva3QIfs8lhX19YswJNHrXg/4tJn5hibNzXF2ZQ MTkY78pOfS6G6gk1adQSTIq7auJLwWu/XVEDznO3r1y7Bl+txsddVQkW1s0d6sHjK4KFm8txoyA4 2Lmtpwke7s0d7vrjTILHqrxFDfBv4XC3H9kRAiydO6W2zsJAMHX5cLcf1yEQfF7YxYTojK2H2/8Q EuHs8s9d/pBg7cIpJjxikkl79/CsZZBl6uHhfJTMkOHs/OFyP0ozyjLEHGL0CxAKnB3t7vbyzoQK Y3ewKM/nLNQMq3OKcSqGLJwdHXbt8TyThbWzByv0sAHkgqHyc3aDHKwdnWK0rSEfpG/7GftEvsiv vbXTNo3QgXo68DzsIKEpJXvoih62k9CnxnRovR72lgjW7uBRnjaa0Ltae4h52HVC8+rSoS96tAWE fjZ4v/+eR/dB6GiTPfA8mhxCk5vTqTHSzRQVzqefa9qQ08bub/ajNSQ0umGf1M8GlNDoRnfoix6N bkSjm/Kh//LlEYO1B99SPOkYdLrOnWIk5BORDbzMPsIlFYI2l8K+4hQXdAi83tbXxABnlw8nYSWt /Hy7I7pauz29iB4W/cteqo0aFT0s5V0xiehYU9iXm2KdRkXHmuuu3MQEZxd3xSWiG/VhVxb+88J7 yDTYmYrnC1NGJcJU8MhgyT/ef7zsG/9wgr27gy7LTxuSGuTtBY9aAeBNxOX7/YNnEn44+I3KTydO E4t/f/32p9ffmBSW4kZapfzBJNkbKfweZ1EfCeo7uwJsmm7AwkwKN2oILy2ySxc3PN/FTc445nZI 6jt34NPoIwry4FaATdMN2P7VqPpYiD+5O3OuyK2L2RvYMZ4IaXwxE5fcPhKUdWYF2O5kAxSXUpiD CbwQRUoxF9zyixmPOW4BphjvzFK4+4hCP5gVYNN0AxbpL9zcJGIpX/Y7d5Ine3ITbkLiKSvuzLWs j5TnVyvApukGbB15ZiUtuF9S3DixG3eU2szGWnFXrol9JJT7zq0Am6YbsM72RMODzXlRkAiKHIeG VmmSZyk9mNyd3EmJ6iPBpwu5BmwPvoHr5MBReyQz8I4pbhzbjRuOj7mLZNzF7bmQ9XEct4Zrimtg wgp25twzI0uJuZj5QbmYa0+1qlPNiY/tY8QF79wKsD34Bi46Rd83PRkP31iQDYqeDeKHHqfI2YZJ ZRufwByjH3VNA7Yn4QB2EflZRQAsF69HStwFoOTmmwAnKYdJpZzLIkBG7q26AAXYFsaB7JI4A1YA s4yAyvbhrqDIYzMVeKBAQXjkXZUTkDHO0q4B28I4kOEAylDgTeIAj3S5K0DDdFNAspGYPP74UcA9 yBxjnHugANvCOJDxgtmhwKFWIQCf5+4KkpBNBVEyCRMpBZ5LXR9jHEVeA7aFcSB7BA4FVvKmEjju CnCaNwVJ7hGmpBRIb9bHWes1XlsIB7B3MxHxfHLe1AiKuwAc5k1AlqPEVJSAyFWvjzGPN1YDtoVx IMPVjUQMuedNAsddgZfbNxUUSedYBexSIFkgSVBGIdBwbeEbuNwrDn6sLAjIYLjzOyk+g5/bHfAn +0jDwuWvj/yadwUKsC2MA9kTrnVXgL3igPpIQyuFdypwkoYSfFMQxPn1MdlRiDVgWxgHsk+41l0B 8gVtmgWSVOuhQLb7UuAlDeVz7grEY/UxuVEMNGBbGAcyjHqIUpEDW2cEwBUpBVlq31QA0wF/jYnT 8VIg/U8fkx/FQAO2hXEge7xZQwHKLAfgK+tdQZLyPxXAerACknS8FEhr2Eeuv12BAmwL40D2qJND AV4aDsDnKQVR3r+pIIpjxsV2SoE4rz4mmnugANvCOJGttH5QgP+JHIDPUwpI3v6pIIuLwxQc3RRk tiF9TDSqgQZsC+NExnM5FJSeOPg8paC/QFNB6Q4WO6kUVL6HfUyjHGi8thBO4DAT0eeeN/g6JcCp RKxy3eWJuwkQSyJDSnMDFFxb+CYuSZfC/HywCCiPNLSzt2L+bHt7Z8VXXvxe2h0ZU547oADbwjiR IzK3K+DkRkDVaSg27FLgJA3F59wV9C3oezDtsAZsC+NETqjUXQFf8LeDZVVpKBb0UhB6MxF0GnLb M8dUR0HUgG1hvJDtUMC1jiO82JxLQpZWb0qAK8nEE6fjJSFLFyRjtqMaPBDbwjmhKyplFg1c8TkC HjbfNSSc3Gdhpn4QsPv5rqFINyRjdqMePBDbwgloi99wETQ08LvHEciWLxrGwhx6OqLfuWvgwjvH PJ3yA7EtnMRfgt8O2SEaHL/9HIF8+aahL8xeriQmUhq6O5AxDwkPwLZQEn8IfnukR5eQZRMdzGz5 JqEvxJ3gPMDklQQK7znmMOrCA7EtnOSll4U0OzSw/eMIJMw3DX1htnIdMFmlIdb3HPO4FA/AtlCS k+xAawG5IoE9MEfUrxk5FqYqNSEVScxLQu4+kUTC3AaN2BZOsj07Ej6ra+BOABHefs/IvjBlqYyY stIguyCbMB+IB15bGHFBC8wtrkocCgL2hyPc93zsC1OStwFTvCtgjzbH/ElIjdgWzsDfgd8FmF0D t4Yc4b8nZF+YYndsUR7FS4N37znmMrNBI7aFM/CX4HfFznYN7HX8m4sr0BcJfV2iblvRxikJQeqz jLnObVCAbWEM/B3oBRxyI7ECvCgkAciWVcFYl0K37mhllYIodlHGYkeB1IBtYQz8Gfjtxe+ygoLs 4QBkyxcFfR3sd+3WPSsFqbznWKZx1oBtYQy+90NwwkNBRu5wAHLli4K+LllxzMmKf74UFHrPsfgh QOG1hTA4SQv4qFi7ABjSyAGFK8sioK+D++amAVO8u9ZUxS3KWPyoCRqwLYzBSlrASuahAKZcAuq3 RBzr0EFy44Qp3BXwtsxx+hWN1xZC/F2yImFnugBsDgdA6bc87Otg/7l3xKSMe3cJQejnBii4tvBh YyQneOr8yA4J8F+zsK9DuGX+qFuXTOwW+1jiKAYasC2M7Gpd4eMMQwFL5YAAylVBX4cTLKyAdPuW kxgEGUua90ABtoXR82fUntH8u9aeNLgvaRXwP7arXgeP3Ab2eYp9Ap/+f2p3AXKNy8N1qYKoCK64 188MSe2upIUBrj9bmhlJI4q0YTBxiWV6+RFQJQ1JbPY6rnjj4AtcRNc7Lfw0a8T4DCP3U4COwzVO sczb/Ajo8ZqxNd2BDXAcjIGr6JrXRAEvLCfgAn8p0HFIZSGWmdFuBc3JFdTYNRtugONgDFxGn7kd CpizOAE57EuBjkP176BAsvpbQdAsxNidJqMNcByMgcvofOCC5J7Ov3AC8viXAh2HJ40dED4tvhUk USBxbsGKNw5C39UWwFcBqJ04IfF9/RCg4/CuVwqoYsdHgFQmGrvXZLABjoPRN7UFsFQB329O8N9G 1HH4lamgiB8fBTVfM9qrvOGNg9BzFZ3lXlMBLGI4IX77UMehwIsUkMWOj4Durxm71cob4DgYPZG7 1ryiQJsvj0IWL/qhwMbxf6kgiR1vBd2V647JksEKOA5Gz2V0rfz5u3X1DVf5pUDHod4WH4bVhz0w E1rMegob4DgYUcY6635EQDPb4Jy/BOg4nGnNd9/zCIjtumPRLdgAx06ItrZZByj8xVwDp7t28uu4 xJbkbv0e/hyvO1qNvAGOnRDla8HPZJZoWQp/z6v+xa/jeK2R9fEJC3+RG6hRPbjhjZ0PpSvyRmXl kkifpOr3yHX+i17HMa2RHq9qftNLPWaxaybcAMdOiBXgF95IpY/SiHpk+fhFr+OY10mPsuKhRwdw ae7bQMbOIcePakp/BWRkDg6wx83oLv6R9xyJk01mZd5JAAN6QyvI1odN6H+pjVW+DZAtZBdpRySA /yMNjyE2qUljE/tKs/TsHzlZMwgtKhaHuvJNy6YUKoNX2YL3Q4oYFliowSqzc3p2rXnZ+0naZbus S7wPzXkmDI3JEtcGOFY6g4WDhBsG49BmBzq5nRyCcScvTklefPNwR6ZLjSl1414Ax0pnsLpqWAsD 8U/+zVy7nOhkDpqued1fqcol1kwaE5Qp8wI43mQGikpYmOEwDvRmosnc9OSNOcoTlaI8WA9zYXbQ mCxHbYBjpTNYtAHCDcNyKAuyNzdd+HAneaBTkuf64a7MjBppTuVeAMdKZ7DdfObEFo3Wf3PjxF7c WV6FlOWNeLg704LGNBuWFXCsdAaLhCFGQ91FXzTW4m/yJJ3ZJK9ao1V5Im9yL/WhxmSZaQMcG5/h ejlwZE9xBjuRN3eUnmxyNykJUhPHPdyBRYlGO+4Vbixchon0osxVncEe7NUe4PnyL+auVuur1VD6 XDOyb1DuBXBsfIaLNzLoppcfAc9JYw/6pnfSixk9TlG6E7e6jScwYw6W11bAsRMaMIqEVLsIyLgr GA9LvAWULq3YFODFcvgslvNVBEjM0ZuABXAcjIaMQqmYAiR8TEAp1RYFDQCPgiBNAe7x6jupBzXm mdpXwHEwGjKKxWYKwo/CCQF2eSuo2MNHQZKNxCfEV48aHFOcxpznHiyA42A0ZFTszhR45CpMwPL8 W0ERsqkgi5PwSYuCwFSnMWdL8ivgOBgNOWCiKXDiG3QVcVGA03wpKHKP8CmLAikINc5cv+KNg9CA 8fqbEfF80jdorPIiAIf5ElDlKPFpi4DMrKcxV3tjV8BxMBoyOkczIsoe8U0Bx1tBkNs3FTSxc+4C 9igQF4gJmiWCFW4cfIYb4rQh6q3aMKGC4c3vJfkYf3Fiw+I2GzamP418zVXBAjgORkMOCddaFWCv OKFvNnSSeKcCLzaUyS8F0fESaizOEvEKOA5GQw4F11oVwC8dTwSQJFubAtnuR0EQG8py3gqkxtJY vCWDFXAcjIYckLWzZOSIK8MJqIoWBVVy31SAoqNnfmjHR0Fka6axBEsGK+A4GA054M0yBUiznIBV 9reCIul/KkDpQQVJ7PgoSLyEGpl/VcECOA5GQw7Ik6YALw0nYHmLgizv31SQpWLGxfaLAqm8NJY0 92ABHAfjRGZzJwrwP5kTkjQ6j4Ikb/9UUKWKwyeirngUVJYhGkuybLACjoNxIuO5NAVNjYPlLQr0 BZoKmlaw2MlFQec91FgsHax44yCcwHEaMVT1DVa3CPCLEbtcd3niXgKkJJFQytyABW4cfBM3SZdC fh4sJrTNhm72VuSvThoXeeTf/EHaHYmlzh1YAMfBOJEznKsKaG5M6KsNpQx7FHixodQ5bwW6BboH sxxeAcfBOJELMrUq4AVHL+pWG0oJ+iiI2kzE1YZse2Ys3RLiCjgOxgfZmQLmOs4IUuY8Eqq0elMC qpKa+KEdHwlVuiCJ1Vk22BDHwTmhOzJlFQ3M+JyBGra+NRSc3D2wJj0IlPv1raFJNySxessHG+I4 OAHt8BtVRDINfPc4A2750GADa1Q7ot95a2DinbHOSnlDHAdn4krw28MdosHz7ecM+OVLgw6sQa4k PmnRoNWBxGoSNsBxUCYuBL8D7KESqmyiRzHbviToQNwJ+gCfsEhI8ZqxRssLG+I4OFOQXhbSnGlg +ccZMMyXBh1YnVwHfNyiIfdrxmqXYgMcB2Xy4g60FpArElgDc0b/dKQNLF1yQmlizEdC1ToxiYS5 DSviODiTU3cULEs1sBPAjOC+HakDS5XMiE9dNMguyCbMB2LDGwcjLmhDcYurkk1BxP5whv/2ow4s Rd4GfPJbAWu0GettyBVxHJyR68DvBkzVwNaQM8K3IXVgyVqxZXkUHw3BXzPWNt2wIo6DM3Il+N2x s6qBtU64mFyBfkjQcSVp2Yo2bpEQJT9LrH1uwwI4DsbIdaAX8PBGoQK8KEkmwC2nAhtXopbuaGUX BVnKRYnNWYJcAcfBGLkM/A5S71JBg3s4AW75UKDjUH53Ld3roqC0a8Y2C+cVcByMMWg/hErYFFR4 hxPglQ8FOq44qZiLk/r5UdDSNWMLJmDBGwdh9GIL1FG5qwAUpJkTGjPLIUDHofpm04BPfletpUu1 KLEFywkr4DgYoxNboJSspgBFuUzoX0a0cegg2TjhE98KuC0zznplxRsHIf5dXFGwMyoAm8MJUPrl Qx2H8p+9Iz5L4a5VQhT6uQEL3Dj4sDHiCX6UH+6QCeHThToO0x3589q61MRqUWPLlgxWwHEwsqr1 jccZTQGlckIE5alAx+EEGxWktX2rRQoEia3Me7AAjoMxcBldHc3fvatpcF/KKcCGwcQllunlR0CV NCSx2eu44o2DL3ARXe+08NOsEeMzjNxPAToO1zjFMm/zI6DHa8bWdAc2wHEwBq6ia14TBbywnIAL /KVAxyGVhVhmRrsVNCdXUGPXbLgBjoMxcBl95nYoYM7iBOSwLwU6DtW/gwLJ6m8FQbMQY3eajDbA cTAGLqPzgQuSezr/wgnI418KdByeNHZA+LT4VpBEgcS5BSveOAh9V1sAXwWgduKExPf1Q4COw7te KaCKHR8BUplo7F6TwQY4Dkbf1BbAUgV8vznBfxtRx+FXpoIifnwU1HzNaK/yhjcOQs9VdJZ7TQWw iOGE+O1DHYcCL1JAFjs+Arq/ZuxWK2+A42D0RO5a84oCbb48Clm86IcCG8f/pYIkdrwVdFeuOyZL BivgOBg9l9G18ufv1tU3XOWXAh2Helt8GFYf9sBMaDHrKWyA42BEGeus+xEBzWyDc/4SoONwpjXf fc8jILbrjkW3YAMcOyHa2mYdoPAXcw2c7trJr+MSW5K79Xv4c7zuaDXyBjh2QpSvBT+TWaJlKfw9 r/oXv47jtUbWxycs/EVuoEb14IY3dj6UrsgblZVLIn2Sqt8j1/kveh3HtEZ6vKr5TS/1mMWumXAD HDshVoBfeCOVPkoj6pHl4xe9jmNeJz3KioceHcCluW8DGTuHHD+qKf0VkJE5OMAeJ2NsUkTWLJ2h fYzxt59/tevnr8vxDx/7Xz9/h4x/QsZ/7A7+fXl3/ev64093/fsf7JLYr1amsARd/+e8bHJ0x20o Oq9V1BKsf3kZQQZZQc8eA3TvfxDeS8qWzFnQgL9mPfJYlijyUhc6darEFIV59g9eioHBHXgaeJ+f 9p+ff3NpE92srqXpz2ULxZH8n0s7XyTfpdqb3x3e/rmP7Us0vt3DzjGrrfcj0V5Dqh4C/s5dthx5 s8Y15OkpAaUH1s0ehta+Mi1gbuhVAt6L+HGUQFL7gq1p85A1iZArdSNT8t7Vblp60JunBNRMTDct mMlH0KkbqwE3s2+hu3WPulXwj6cE1NTlmz0cPfRaMGBsaKtJqfoLDH14SkCpjSuj7as86KyJqn8v G7pa55tb8/94SkANzKvVepihO+/ATEzEheZWQFhaYzP04SkBNaCOaPcHfVnA2NB5qfZXN348JaAG Ztxqvd/QTd+tAVB3LzpVH4qgSG5HH54SUAPSgPZ0NFQzA+4NfXHVxdY+HH14SkCNzn+gclrohIBC P0fPmwoqcUNaftCbpwSU2vicOtgIgEZt0oDKr1voaVPC2EbV01ECSUvtMHs4GZ/HgLmRKfjGTK7A DX14SkB5Fa9oAQudNAVmYyItNFPDh7tUHvTmKQGlbbXoPuqIcA1DJ62IGtC9+Ri6tWdyxa4Y+vCU gFLBMMzujrYCqgFjQ3MIYIkac1WQw1ECSW0fb/JDzkMDBoeYRS5cbuLSn5J6eEpAucpqlw9eKkL1 pRpgomyhOTt1JuBoXkFOTwkoFZA4/ZZ8rkXAZQF9Qyesuk/f70XeHCWQXJo2nLWRUdY1wKTsIl9Y bu++30Y+HCWQVHZ3s4eTkZMMmC/ZLkqv3I7k9eP0lIDqOl1pWjVcq4VWUfOrE0C7X/TkojOa17jK g948JaDULrBRCwyN668BWoE39OCqWT/6KqinpwSUTkLD7OFo6EIGjA3dcYiQkfjodYqHpwRUhzhV W+eOB521cOpYWDZ0Q2q0Mf0sF3rzlIDSCbIWtTsHMKDRIDRAsz+9aN7vxjLSV0E9PSWguk4LZvcH fVnA2NCccFoluowHvXlKQKmdYQ8OrkCjGWuAFpoNnXGMjc+evYKcnhJQvWmq0MZYCDQkhAY0XdKL Tki7llBR+7Xy+vCUgOpVi7naUEELnSygb+gL662T23LPB715SkB1fI7a0G6GTlzdja970P3GqitL apvrGA9PCaheWCnVHo7GtWbA3NBcb23c8XWKh6MEUtfmpoWyY0mLnBAw9Ks2MmU/t6O1h7w5SiB1 pHv/7ZD2sG+IefVH8r9gI1M8If9APh0lgHpijdKfbuDJfnTjwm5gcC9emWLV4/STwOmJZVVflxcX ZfdGjckPlzKsTO5x6gu8OUoA+ZZwgwge7BTYQTSOhwzVUQaetsUfPwmghuqttlZW3XmCLw/oG5jl uVAkVNdLH08JqHZz8ZqIydCdLRmZmnc018xWXr1rfTwloNrkhqs9DN14IAjA+Sz0xVVnz2iST0cJ JL/lfudJTgzIzKiHDGhhv60+Fn08JaAaJInaKqcMXZn2KGXpRTesObMprmQ+/SSAWudNVHsYuPCm ImBs4Ik1s4pWn4k+jhJI3gS8JZCcGVBZbBYZzNyqXxcn744SSA16VW3VUkbOLIlodGkjs1VlVo3i HfzjKQHVKpWh2n2hLw8YG7oh7zIbQPGR6OMpAdUq24tqhmzoxPYDUVE2NLtgvvAs3Q/x9JSAcm3j SkfRF5smpBB66EIXrDqxShfv4B9PCaiG6aP/Dqu8RCcP6Bs6Y72J1RTX0dG7pwRUyxQgqioTbRWD ECiQnXlDs8FC0eE6Wvn4eEpAtUTZpPYw9KSsQsD9FlPck1SQgMUnoo+fBJDrdVfvBCcG3FSGDrbO nRJz2yeij6cEVEOajl+dOS5DDwpYDCVpR4PMEpJ9Ivo4SiDpaDTM7kbuVN0IGBu5I52v2e2mk3w6 SiD5OOfDHcl51F9Mf1U7yUNG4l3sidkHoo+nBFTF8EH7NrTqHgboPdvQlYu2+1gtpz+eElB16Gep 3SjCib48oO9orro0VpFF3h0lkNTGpDi0uhpZv6YhQEvaRsZluTjK5eRneDpKIFWMHrTvRLKWXQ+4 byf/Da9aHq8yaOlP0kP756+f//z+98fljhcZvvtESXhXxWyitubZendigBbo9e5//VyAErhE76Ri Kp3R7yrUO/3iP+q4nHhK+jIcmgA+9X8KVNiwlyFR6sVwVYj5YsaMwb/7lqIh63NWPl0xn54SUDPz ndVmVEUnqBAGzA2NG3d3pMHtGuPjKIE0MxOlJl5EIydN0oFMfMlU4TdT+O7pQW+eElAzMVNqZn0B erIfjdszxdC8cnfhfrjI+HhKQM2LiVAxwxgaKoQBY0M3rjpVf4GhD08JqHkxZSomr4XO2o8mStWL 5pWbtuOuMz6eElD6zqrLqJgXDY0RUQO0GaQXza2Yk2t3ofHxlIDSQ5hm9wd9WcDY0BwfJuXGvNOD 3jwloAZal9pNfwwNJaIB6GQvmrs8WfSnC42PpwTUQMOlPR2tAjUx4N7Q7CGz2Np9XDs9JaC8JHiB MHRCQKGfo+eNyzITN8SFxsdTAmp0fo7WveRoKBENqPy6heYB4pJhW5x8OEogjcYzUHs4GZ/HgLmR BxY9JjMwL/ThKQE1GjNHG0x50ElTYDYm0kIzNUbPvisLvXlKQHkj9LYIdGK11caJ9F9ojlWjdt8V Qx+eElCjcGBRuzvaCujsvLQLXZF7LFFjrgpyOEogjcIhq11sXkbGFDYHh7JF5sCGKUGfT0k9PCWg XCe5alL0dXN2nJOj5EJnLLozAYdLjY+nBJSqO5x+w7Yt9GUBfUOzYVPiYr8XeXOUQBoYQtXGWRsZ ZV0DMJO+ZM6Cvft+G/lwlEBSFd3NHk4e1GjzpmRzsl2UXrkdyevH6SkB5drfJwFDQ1jeF3XmQnPM 7BnNa1zlQW+eElBqF9ioBYbuVMM64aQNPbhq1o++CurpKQGlc9Ywezi6UcNrwNjQFKEQa/jodYqH pwSU2hgrdFYsDxpzh458ZUM3pEYb089yoTdPCSifaX3CBbpyWroLh6eF5v22gbavgnp6SkB1Fe9m 9wd9WcDY0IWrpiRFZ1rozVMCSm0Mnm1wDAW6cDK9Kyephc44xsZnz15BTk8JqKUdoScMDQmhASq9 y4tOSDsbDfu18vrwlIDqqv9NlrYHnSygb+gL67UJrt3zQW+eElAdn6M2tJuhE1d34+sedL+x6sqS 2uY6xsNTAqoXVkq1h6NxrRkwNzTXWxt3fJ3i4SiB1DPnoY4lLXJCgOr8tpEBtixp7SFvjhJIHene f3viAPDn54aYV38k/ws2MsUT8g/k01ECqCfWKP3pBp7sRzcu7AYG9+KVKVY9Tj8JnJ5YVvV1eXFR dm/UmPxwKcPK5B6nvsCbowSQbwk3iODBToEdRON4yFAdZeBpW/zxkwBqqN5qa2XlTKsBlwf0Dczy XCgSquulj6cEVLu5eE3EZOjOloxMzTuaa2Yrr961Pp4SUG1yw9Uehm48EATgfBb64qqzZzTJp6ME kt9yv/MkJwZkZtRDBrSw31Yfiz6eElANkkRtlVOGrkx7lLL0ohvWnNkUVzKffhJArfMmqj0MXHhT ETA28MSaWUWrz0QfRwkkbwLeEkjODKgsNosMZm7Vr4uTd0cJpAa9qrZqKSNnlkQ0urSR2aoyq0bx Dv7xlIBqlcpQ7b7QlweMDd2Qd5kNoPhI9PGU/5Fd7rqW5DYUzfsr6g+s9yOezIANDDqYwHDmSQzQ wIwD/765N6U60iEaYF/eS65SqSRy06FqYXtRzZAMHdl+ICrygWYXTAE2t/UR70hxqKVtltJRdGDT hBRCD93ojFVHVum8OvhXpDhUxfTRnm6Vl+i4EtqBTlhvZDXFdVzoM1IcqiYKEFWVkb6KQQgUyM50 oNlgoehwHa18fEWKQ9VI2aR+N/SgrELC/BRT3JOYcQDzmoi+4sSBll5f6p3gyIRJZbjA1rlj5Nle E9FXpDhUxTHtj84cwdCdAhZDSTzRILOEpDURfQWKI+lo1M1vRm5U3UjoB7nhOIfR7KaTfAeKI61x bg13JKdeHkx/RTvJS8bBC+yJaQ1EX5HiUAXDB/1paNU9TNB7dqALF233sdiZ/ooUhypdX0v9ShFO dFgJ7URz1bmyimzyGSiOpD4mxa7V1cj6NhUJWtIOMi5L4CiX4vqGd6A4UsHoQX9GkrXsroQ5F/kP RJX8RuVOT/+L+tH+/P3Hb89/fiy5s4oMn32jxD2rYDZRX8/ZfnZkghbo/exff/zll/+O55efz/2A n7/8/Ud8/vrE59/6m6n97n96B56/Pf/4Z3j+9erjQXGVGx/0WfDPH4Gk1c1igrSJWBm+sGDbIl4U kq3byrDEghsNP6bAJerSM7eR+29bbxaq0tBXpDjUSNwN9aeh4+R2acI80DwuZnE2ib4jxaEGqpf6 UOEbjY7VxzpWhuYhN7slyVekONSI/ENJrEZADx7dPte5MjSvplmUAUNfkeJQI/B11B95o4Ml2LEx NOuJWVzCjT4ixaFG4EcomecX6M4qMcI6gURbFTSLimvoK1IcSp+Jo1MwXRoaskUTIk/SRrN0my2z LfQVKQ7VJ6uD+u1FR0sY+YNmwzGL3rbRR6Q4VEcFU79qlTB0pQjRz10OdP20yrrJV6A4UkcFoz8X WaskErR5HGR2drN7nvmKFIdSH2ViVRNDp8yEdqCtV9qsm+eLPiLFoTp+UN+KJNCI1AQrowttKsps LfvsXZHiUB3Lpz9edLCEeaCp/czWOl/0ESkO1SufudoR0PgqmlC5hI2mYjVb2z57V6Q4VC9UmauJ Ag3hogmNG7fR1Nlma9835ooUh+qF9Wy1fkNHS8Dn3ui+B4Q62gs+4sSBeqaeX2oFYLsRo3OY22CO M2br3Gu+IsWhVFRhCFkSS9FhckoZg3PnQtsMZrYFa8RfkeJQPXFyqqiGGx0toR3otKfHFvsLPuLE gXq0VmhSFuDBYXCJ3Q3msGu2pb3mK1IcqqPT0J8L3TnCDjSmD7pyO2hRdgx9RYpDqR9VVK2xwdCp p2cGitKN7vHZtpX6oo9IcaiGrq6+DTtAo+1rgo1DGz0o4GlbXUfvjhSH0qFsmj9edLCE+UFjIN22 tfKij0hxKPUhYNdgCbT+hISkr/VBp7SFEldq6CtSHKp1yu6qK1joQl0+MyetV12OZ9s29qqvSHEo 9YP5vW50tISRP+jan23bjC/6iBSHao0DTlVhtdCZE9AsusgPuudn276E6VekOJSqR4xl6s+FTpzb pur0A83eYhaVztBXpDiU+hglVZjWF41ZUxPaB23Dqdme8os+IsWhmspm+FOPv6GjRmpC09f6oK2Y 0vZNvgLFkZoqePPHSw6WMA9ysYbIrS7pRR+R4lDqJ62wDd/I0BDGmqBDwYFuRNP2fc2vQHGkpsOU nsEGaQ9/agFG/NCX+oD7eLbtzdZ8B4oDqR/M7xscLX4c4MlmSLvk6R0njqM/4D7qr7NxB++rpuH2 Lm6L+dm2b/AVKA7UcD7pW6vRLetaGbGD4SCn+Wzbh0mPr0hxqAY1q35mMyM6rgR0u40u9dkWPXqh z0hxKD0iqNd6FrOhG+s5TlQ+0K08246wVn1HikPplcGWqz8NXdmFkDAPtFU6gKPV/684cSD1cUbW 9EkwzhCubj3Ak9tBO9LY6DNSHEorCPq5liqcckUX9nvUMlyDhcbWbYu+QfQdKQ6lJXWaPzY6rIR5 oPOzzChjg884cSD1UUC0chcDZxYYlPZ6gEt7th3VbvdXpDiUNq8Gf7AGKlrlFBMqi+RGt/RsO9o6 eHekOJT6wfy+0XEljAM94rPtWDXpK1IcSrs5ipPKhmzoyOKF5l8O9OzPtmOsVd+R4lAqb9AX1Z+G DuybUEPhgx4sHMvOteo7UhxK/ah+D+ztRKeVgOa/0ZSlZmdIG31GikOp3kPzUmGZ6KsebExQiXKg y3i2ndGkx1ekOFSNvAHqj40OK2EeaPYUszOljT4jxaHUh95TpV0MPagHIcXrgR5E0868Vn1HikNV 08g6eECzKlp3DgkxUNRu9OSG0M5i0uMrUhxK/WB+3+i4EsYHjf19bc0bfUaKQ+nslXUe0UksG1qV hLY+jGpFxflG5/K8tq1V35HiUDqKdvOnoVVQMUHv2vygy3xeu2TNV6Q4lPoRvsbFYui4EvqBbvV5 7djkM1AcqWiFha8vZ+RCKaCbqG/1IQ/uhNm5zt4dKQ5VGiuD+tPQqo+7JcyN/gNRJb9RGRqp47+o qX/+/uO3RwXUg3985o0Q9wz1kdhV3Of1zMSEQuHLZ/76Izz4t6UqJFTHuFQUOEAuz2pyVMHoauvv 3B+FQcoGajdsD2j6EgZMkd/wA7SEjPmr24qq+iXw8KiMTBBfEUvG8VobHiJ6jFkMMoa+IsWhRuIu FFwKouPkNvXOd9rojPNhthTTIV+R4lDq46OWyE9saHz1PngINrqix5jFjLTRR6Q41Ij8Q0k8mEAP 3TpNmDy6G924IbSl7VVfkeJQI/B11B95o4Ml2HEx9CCatvT+oo9IcSj18RFK5ukFurMsjMBvstB6 vJ5tMX4Z+ooUh9Jn4uiUwsoFNAZJTYg8SRsd0XTNltkW+ooUh1I/mN9edLSEkT/oTDQtNM1GH5Hi UPp9C/yqTc7QenOQoP3iQFduCG3d5CtQHEkP5TB/LrKWRSRkfasPmfrXbM1rxr0jxaHUj1X9xqpj 6JSZ0A70IHoYer7oI1IcquMH9bsWEkMjUhO0XdQXnQJ6rtla9tm7IsWhOpZPf7zoYAnzQEe0GLO1 zhd9RIpD9cpnlsGbADS+iiZULmGjM7bCbG377F2R4lAdJ6U+BefG0JE1eTRu3EZX9BizGHYNfUWK Q/XCeqZ+nxsdLWH0DxrnmaaO9oKPOHGgniniK26oge1GDFzh/IIHaqnZOvear0hxqJ44/NSw5lMI LYyXY3BUWegcADXbQl0964oUh+qof+pHDk+GjpbQDjTuIE2L/QUfceJAPbIZVlRwAw/OgAMl/gPO qKRmW9prviLFoXrgYFrRdwzdqTTG5CC70ZXbQYuyY+grUhxKfcjKim650dCdE1/rg+7oWmZbqS/6 iBSHaujq6hdKY6DR9jUhUjxv9MChM9vqOnp3pDiUTmLT/PGigyXMD7oEdi3a1sqLPiLFodSHYtWR ryy0/oQEnUsOdGLXosVKDX1FikPp1ItxqeoKFrpwnpqZunyjWTnMtrFXfUWKQ6kfzMfIZ+hoCRgK N7r2Z9s244s+IsWhWuNMU1VYLXT+P9lljwNLbgPhfE8xJ1jo/yd2bsBw4BM4Wwbe+wcuFqUe9hAL 9Fu+V/qao5bIIoeejcnVoWf93OdMx+69lRJQ8I+zM94HXTiqwWAmh2ZvsadWumtNnVICCnHWeHGa NHSpXDC+aC1C9zlLfdBOKQE14O013jj+hs5QYgG8ff+irZjyOS/5JZRAGvDXFq+HnGzBduRmDZFb 3cqDdkoJKMQFFXboNzK0GmMs0Jniix5E8znvNX8JJZBGwY8an6HWXuO9OTTsdYYEA8/1uc85LOe3 UAIIcbJ4XnA2/XLgzWbI57Gnb50EDv5H7yP+uhp38b5imd7ewx25fu5zXvBLKAE09HwytlaDLZuo jLqDyZHL/tznXGY9fpQSUEPdLOLKZkZ0Pgu0211065/71B590F4pAYUjovUaZ7EaerCe64mqDj3a 5z5XOlm/lRJQuDK65Yi3oTu7kC7YDm2VTsHZ6v+PTgIIsZ6RM38SrGdIr2534M3t4HOVddFeKQGF CqL9HKVKTznQjf1ea5leg4PWrbtP7RtEv5USUCip2+J10eks2A5dP+ex2rpgr5MAQqwFBJW7Gbiy wGhp7w7cxuc+V7fb/aOUgELzGhov1kCgYae4oLNIXvQon/tc4xy8t1ICCnGyeF50PguWQ6/8uc91 atKPUgIK3VyLE2xDNXRm8dLm3xx6z899rnWyfisloGBvtC8i3oZO7JvqhtIXvVg4znOfrN9KCSjE GfFM7O1El7NAm/9F05bac6dy0V4pAQW/p80LxrIwhh8cXACL4tBtfe5zZ7MeP0oJqJ55AxCvi05n wXZo9hR77lIu2isloBCr34PTboZe9INqxbtDL6L53PVk/VZKQHXzyBg81LMCjZ3TBTnR1F705obw uZtZjx+lBBTiZPG86HwWrC9a9/d59nrRXikBhdmrYh7BJFYNDSeB1qejWoM5v+jaPs9znKzfSgko jKLT4m1oGCouwF3bX3Tbn+d5bM2PUgIKcdYYutwMnc+C6dCjf57numQvlEBqqLAa48cZudEKYBPx q77kxZ2w5z5n762UgGqDlQHxNjT88bQF+6L/p6pWH1VVjzT1j4ylf//3j/98YKA++h/f+UZIeAdi XThh7ut5Z+GCRuPLd/4LPwhzYzqu59//+CcdqzqpqVNTA3fpC9rn9DqaYW1u59+5TWCqo03zmmMH +OuPf7us9R+12nzpP6vPz+rrGXwLDdLUOteej7DL9yOs9aCdUgJq6SzJeEzv4rFAT/PzfXlw/Jn8 EUogLS2QTf3xncUy79yz90ZmHz7HfeTHajulBNQq/Jyv+UCbKxZsl/TI30va+oN2SgmopbWX8Xiy 1iI0tTh/0Z1Z5+UK4o9SAgpxtVeZ41TDqG4DC7rLuuXTdr5l/EcpAbVg4/CqkViCDa1VeS7+/atq nTKexoN2SgkoxPrOkei0vmPN3KwcF12uMfl24x+lBBTiaq+yiXJvdqm5uW+vKr56Oamf+cMJJYBW 4tf1A5NaUeitChg4jWMtnaN6CSWAVuKJHPkMqnuzZeOv8zfjZXY4v1zgSygBhLjai+admPQ646/7 N2N1UDob0Ki1OzA5nQTOSrz3o9CskKuWcSVWgcudtNrr5eJfQgkgbJHWKsQ9H3DFQIEvkV3CPA9z +sHjpZPAQdzsPfMmrA5maVP9cjsT7mc3DtfpJHAQJ31P5Wj2TI5L/dCXS182eU1+BscjlADCxdF/ QGy2Z6uvh75Q9jhKDgaZE+kpoG+hBBDiZi96BlKdUaBvLuPCjFM9w8EBO6EE0Fzc+6EzzxlHlyUy XMY8voPN5M4EL50EDmqqnhXEZ0ZC59Y8qm9SOtnqlHs3+oCdUAJoTp7u0a5rH/Tiq/oWZddisE7M Ui/YCSWAEOt1RPNO9Qy5yxJxHUoPgvnPRI9wZlwnlAA6fmP04+C32gPqXX+yiWtkBY87wbyEEkCI u71o3IzVgMAwuO7ECzfYQcatEy+dBM4crNGIzVBuOMtMvWtNNn32Nc4XJPgllACCkWQ70S9xwDr9 Qe8aE29yZ9Z9XKyTScAgZivBz7j56jQCO1u/5Wc25tu5zccAvIUSQIirvegMn5XNHnrXk+hZetWs Rzll4qWTwKFVRzzPCAN939S7lqTnFuByjsXhOp0EDmL9gkN98eFWzWP4jqT3Hlyein6xTiYBc4YW xNuKD2zSpt71Iys7rJn97MJLJgGDWC/IQOXoBzstDdeOBvtnG3Yo1gU7oQQQ5je90oj7zZdtZvp2 ZNWs0Wv2US/YCSWAEDd70bwZs89MlqwLnnolGvuGHrcDdkIJIMRaNgfMkWWcrc9MFtkLHsyYFqjX s8cvoQTQLPzeiE9DytZnFrvCBXfNuNLX91stX0IJoFn4P4jnzZh9ZnHZBTf9bJUeU432ATuhBBBi TX2m25CSNRqdJL5gFuBKR9HOiPkWSgDNTG8z0+1IyRrN5qe54MKMeYzbulynk8CZ2aa9dBsSbF2m Xs/S5bKw11p4mPcFO6EEEGI9z/jD+giEyxIZLmG20MoG3bp9u7dQAmgm3kDEdR9wt0T2N+PO41DZ SFvLF+yEEkAzsWIgHjdjbTT4oVpfHjCIhe2uPfk6mQQMYq1viK2N4BNrm9mJ1fBip+Zbxjh3hOCX UAJobBbkWU4jgV4bzbaZ7gErl1UolYt1MgmYsdk/EPebr7YZnHz942JtH1gm6poX7IQSQIj1DYit j+DaawY7M58HrFtbWOXrcWtvoQTQWNwRVA9rJNDrjkG/XcZVM86bB+5UzLdQAgixfkPE/Was33jr 5ODAmnFmQa7n1r2FEkBj8cwhtmaCZqBnchee0AtmC830l7WamXgLJYDG4i1BT7FmAr3aOujXt/5o MQOYZaKWy3U6CRy0Pb3WiPtNWH3oriwCl6vUWnnx9sU6mQTMmCxDiNc8WPXNu/qW1JYmmlmDyp4X 7IQSQGOybsK6WCeB6ZiWiGtJ3IBMc6k3mtyXTgIHJkgLPWKb4aDXXrmb70haIlG3F8Hzcp1OAmcM 9hHEMx+ujlK7+YbUhp60xCJR+rpgJ5QAQsw+Ms4MBxu6LJH5rT7mI5JdvXZOxEsoAQRbrN0fsc1w 0OtP2d03JC29iSdJi8XlOp0EzujWR8YZ4aBXOwO9a0hND1li29BScbhOJ4GDmG1knhEOc8myPFw/ agXfTKHpJPsSSYCMZi1knvEN+m5JfHuRAgnj4aTf1aOa4Z7y0j+0V/39X9Xmj/6nWpxKbH+56cv9 eV0dzfim3xe7519c0Gb5psbfs7eewc3atNd60E4pAXV+RF80g4YutXx/paH18mye9D3rQ3ZCCSTE 6tG7mhEjqxvhAjelbs4km2dyj7sfL6UE1CqcKro244vWOocFbk7dvEGbbWufMepHKQGFuNurxpN1 0VyWn1S1Q6mhWOcFF+2UElCIq71qb0MrggvcsLp5izYL1S7loF9KCaiVbczUi3fRvXOBG1dtKxY9 7k7jQTulBBRiDppqTC66ai7w7S5rvUt/LrrctfuDdkoJKMTVXrW7xlvNCfVuZt0cIhaHS02ddvQl lABayYZNmMl9wH1T74ZW3QH14eMkfsBOKAG0Ek/kUHNywBWJ4K/d1Lo4By4WrNX7BTuhBNBKvEVD ZySCF6/zSn5uXYvbS5+77NC9dRI4K/HeD/Umh8t5NPnBVbto+nMupl3WBTuhBBC2SGvVUG9ywJxH sx9c7TzM+X/Sq6a3bSOI3v0reJSAKuV+k0BRIE0DtD20RaJb0YMsM7ZSrQjQstX01/fNzK609LqN kVwijvP4OFzOzJuXjiPxFrhY8SC28pyQE2Y7qpJ2Cq/jhF06jcRb4GLFg5ikzNNywrxB3Khi4cu8 vCQEbpPQJeszA8aKCI1D/+FNtq1B3KhO0inEXGVB9XwcJhMXwFgRIbbyoGRbg7hRnbRTiDVn3FJL B68zcQGMFRFiOntvs2314kZ1Ek8h5vL1LCZpM5/jYsWDmcqSZ7Nr9eJGTSlSHZtA7/NBJ+ICGCsi xE4e5HPC2jC+kChpC89zIqT1Yw6MFRFiakfvktuElHeSSKFQVAjUrVzKKlXbDBgrIig5DRDEOiRi dqOm1KfAxtLzquvTzJwDY0WE2MmDfM6Y/CgWhkKduOE8K4jPc2KGixUPYprR3iW72Vt2ScAX0kQF hkHT+fQFmXgGjBURdlOWE58MJ/Dk64AvhIk7mc2rdz7TFrBY0SBmKfHJbgJPPhQLsrmMn2A5X950 fVoA5sBYESE28iCxmzAHJPbAF5rEO4vjRdfrNCZmuFjxYB9kJQnJbQJPbhT4QpICm0AuYSqLxFvg YsWDmL4gYpfzJTMKy1QoEvW9ODbfukxbwGJFg5isJmKxmvCKtOQCX+iRjB2emS6dwgwWKxrE1CAw nOI0ey0q40s58qyf1ktRdJm4AMaKKBhuacQu58syE0o5kmkm/sd5k4kLYKyIEFt5UMgZs86E0rTS wtvytJVyS8QFMFZEiGls+j6ZzV6JzoTStHrPGfMK5Ew64xkwVkRB8/dGnARJic50pWulPqNVgb9d npYzYKyIguYLxCFnzDrTlbaV+gzEvGPSop2IC2CsiBCz22yzILUiNOQkLsQ8gA1vFLZPxDNgrIiC 4t0mtFmRWhGavrStXnPGXMa2y7wFLlY8QYndbLMgYa1TjC9sqwx2YzQXc5+JC2CsiBCz5VRJRwDs JJHCt3qWUMMCbV3yrjNgrIhCK95VcT8ysZNELt6VvSmIWUitVZm4AMaKKLQ8MRD7nDEJDV6U5suZ GIya5c6e8y1gsaJBTPMNscgIPjHJTN/yNMy0gfLV3qceYeIZMFZEvueBHHQSEuBJaHrxdGdi4uUp 1OpMW8BiReN71g/ELudLMoPKp59MK+fAY8J0IRMXwFgRIaYnIBYdQdtTBr3ifM7EdLSap7xJ29oc GCsi3/GJYHqIkABPJwZ8X2RsKGPVc8GliTkHxooIMX1DxC5nTN+4J+dQEFPGigeySV03B8aKyHdc c4hFTCAGVJO95grNxCyhivdLY2SZmANjReQ77hJoiogJ8LTWAd9d5g8NMxDzmDA68xa4WPFA9qit EbucMO2hveEhkHmJ1RhuvD7TFrBY0fjAYwhxFxIt7c29KSXJdpSo4hmk+5CJC2CsiHzguYnVRZQE S0eQRApJ4gNQvFxSRzPvDBcrHixBNOgRi4cDnrSyt6Ui0YjE3O6YOGTeAhcrHu9ZRxAHlXjJSvW2 FCTrqdJaHhLadZm4AMaKCDHriE8eDmtoJ4mEy/SRPaKV1rOpImbAWBFhLSb1RyweDnh6ld6VgkSj t+VKomGReQtcrHi8Ex3xycIBT+sM8IUgWSqylmWDRkXiLXCx4kHMMhKShYMv6SSPQo+sxjcj0jYl OwPFisRbkZCQ7BvwTpK4aBERNu/f/HrFxcn7LpWqwvakOvohrZqGq/dXP6yvvl2vbaOa9QfcZ2jt QrWs8kVL/UlaowxGYYvb1pEthzjb/C8ljtNp8Q7rLXjWp6vFb9cfh+1x9zgs1x8ppTQIfdKEFfCW pwL/rH98cgfSCjktDeHosRSt8kXbKNeRCjhaX1SPk6G8+L05F0rju7Z13fdC5c9UNNhh8Vb5oqBC sYGLkon8NrCroPlj8fswfRinuES9LDaH7dAs/1z/MktQBcrLNat8gT+R2rXod6VRJriSDP8zL/Ks HdKS3/J+i8a14ExZ4fxAssqXJyT48+E4TNvxcFjCqC5whs1pt983cdkthuHYjFMz/L0dhptm+8CI aRoOx+Z+ib5YHJcrvRinze3QpNuHm3t5wxbvw7MIH2d1efTlyTiNm2Y4bK73Q3M7jafj3dMv98zB aEvHXR8LV83LziZTfP5kftrd3jXXSPO0uznefcNHsqM3Puzo8mGJxlvQVbPfHIfD9lP1aZ+pPU0D BaowK73/f4O66jJJWXScutJSda8flyv4zMVmt99cyyVljp/97ljn+cxJG1y2CoPrC0vwcn9x0Jwj trtUCNBYPui3hw3lGBbXlKNb7AcJm8fNtFxBfCl5DPbF+HAvL9Ps5Q8J+Jh+018zaPzQpPvT/2/l Z5TH5LsS5pP8tW7QZ74itiWsDsp83Wc8szwdHm1HXIs34/3xBU1hIKredv6LP9X5/s/3xLvh4X5o Tnco+MdhajaP3PZSZnS1X5rFMzPumSO0Wr0K3vmvO8IzS3WE4Zw7ZItzf93cjuNNcxiOp3H6q9nu jrt/MJcx5w4v6QjrLPYu/XXT50LytC0cD8U34+F2uD/uxkODmXycxv0LCgA0rzrbVr364qTO9z8t AM91+H7YPkw0N0D0dn317wAmMdZBCmVuZHN0cmVhbQ1lbmRvYmoNNCAwIG9iag08PCANL1R5cGUg L1BhZ2UgDS9QYXJlbnQgNDkgMCBSIA0vUmVzb3VyY2VzIDUgMCBSIA0vQ29udGVudHMgNiAwIFIg DS9Sb3RhdGUgOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYx MiA3OTIgXSANPj4gDWVuZG9iag01IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSAN L0ZvbnQgPDwgL1RUNCA4MyAwIFIgL1RUNiA4NyAwIFIgL1RUNyA0MyAwIFIgPj4gDS9FeHRHU3Rh dGUgPDwgL0dTMSA5NSAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczggNjggMCBSID4+IA0+PiAN ZW5kb2JqDTYgMCBvYmoNPDwgL0xlbmd0aCAxNTgwOCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiAN c3RyZWFtDQpIiZRXzc4kqRG891P02ZJZfpK/sw8+Wz74AVZey9Kykj3vLzkyga7KogUejQTqb8iI KEiSyF/+8qO8f/3xtqZYh7H69HbvH7/+8frlr39373/9eLn3v9+vQIZCeWdKpmKOxqX3n7OrptL7 v/98/fYK1cSabwt80QuITMl5g8AKolIQrbFhFxLxp5oOmFVhJmtCTBudKZoU4h7TBcYMAzJbk+su IkdTIx0g0x2yOOMcbVQiwquAZEI9UHiSCBoh1ZkYwyGk6JBksgt7WcGrEGedqdXvaUL/+PKJycbF QwzpEOdNcO4QIl9D7hOTDWW3/5zodYz3JscDT5SUo/iJyaYeQlLQISEYe7gsKUtI/YQU433df02O OoaCibbueYpIi+ETU0xKu1LAMaIt5hkTg+TSNqbGR0wxtewqClZbOZ30OZ1Exqf9vlWbHjHVkD/E OMm29DmeTCaWXRXhktO1ffY6V5zZvlhxmVIxJRrr98WIrxrG/Dmfag3iDjHlERNNoH294eTkmHk+ 3lqT/L7kcKbpmGhyOcREybd5sb1jmH2Z4qRRIck4uy9TfP4cMsuU986Esi85fP46JplIpxjZgfLR FpzJdl9z+Fx0DNaWfQHhPVYx5I2jQ4wkAWrzDMnmsM+96F4R0Rva3QIfs8lhX19YswJNHrXg/4tJ n5hibNzXF2ZQMTkY78pOfS6G6gk1adQSTIq7auJLwWu/XVEDznO3r1y7Bl+txsddVQkW1s0d6sHj K4KFm8txoyA42Lmtpwke7s0d7vrjTILHqrxFDfBv4XC3H9kRAiydO6W2zsJAMHX5cLcf1yEQfF7Y xYTojK2H2/8QEuHs8s9d/pBg7cIpJjxikkl79/CsZZBl6uHhfJTMkOHs/OFyP0ozyjLEHGL0CxAK nB3t7vbyzoQKY3ewKM/nLNQMq3OKcSqGLJwdHXbt8TyThbWzByv0sAHkgqHyc3aDHKwdnWK0rSEf pG/7GftEvsivvbXTNo3QgXo68DzsIKEpJXvoih62k9CnxnRovR72lgjW7uBRnjaa0Ltae4h52HVC 8+rSoS96tAWEfjZ4v/+eR/dB6GiTPfA8mhxCk5vTqTHSzRQVzqefa9qQ08bub/ajNSQ0umGf1M8G lNDoRnfoix6NbkSjm/Kh//LlEYO1B99SPOkYdLrOnWIk5BORDbzMPsIlFYI2l8K+4hQXdAi83tbX xABnlw8nYSWt/Hy7I7pauz29iB4W/cteqo0aFT0s5V0xiehYU9iXm2KdRkXHmuuu3MQEZxd3xSWi G/VhVxb+88J7yDTYmYrnC1NGJcJU8MhgyT/ef7zsG/9wgr27gy7LTxuSGuTtBY9aAeBNxOX7/YNn En44+I3KTydOE4t/f/32p9ffmBSW4kZapfzBJNkbKfweZ1EfCeo7uwJsmm7AwkwKN2oILy2ySxc3 PN/FTc445nZI6jt34NPoIwry4FaATdMN2P7VqPpYiD+5O3OuyK2L2RvYMZ4IaXwxE5fcPhKUdWYF 2O5kAxSXUpiDCbwQRUoxF9zyixmPOW4BphjvzFK4+4hCP5gVYNN0AxbpL9zcJGIpX/Y7d5Ine3IT bkLiKSvuzLWsj5TnVyvApukGbB15ZiUtuF9S3DixG3eU2szGWnFXrol9JJT7zq0Am6YbsM72RMOD zXlRkAiKHIeGVmmSZyk9mNyd3EmJ6iPBpwu5BmwPvoHr5MBReyQz8I4pbhzbjRuOj7mLZNzF7bmQ 9XEct4Zrimtgwgp25twzI0uJuZj5QbmYa0+1qlPNiY/tY8QF79wKsD34Bi46Rd83PRkP31iQDYqe DeKHHqfI2YZJZRufwByjH3VNA7Yn4QB2EflZRQAsF69HStwFoOTmmwAnKYdJpZzLIkBG7q26AAXY FsaB7JI4A1YAs4yAyvbhrqDIYzMVeKBAQXjkXZUTkDHO0q4B28I4kOEAylDgTeIAj3S5K0DDdFNA spGYPP74UcA9yBxjnHugANvCOJDxgtmhwKFWIQCf5+4KkpBNBVEyCRMpBZ5LXR9jHEVeA7aFcSB7 BA4FVvKmEjjuCnCaNwVJ7hGmpBRIb9bHWes1XlsIB7B3MxHxfHLe1AiKuwAc5k1AlqPEVJSAyFWv jzGPN1YDtoVxIMPVjUQMuedNAsddgZfbNxUUSedYBexSIFkgSVBGIdBwbeEbuNwrDn6sLAjIYLjz Oyk+g5/bHfAn+0jDwuWvj/yadwUKsC2MA9kTrnVXgL3igPpIQyuFdypwkoYSfFMQxPn1MdlRiDVg WxgHsk+41l0B8gVtmgWSVOuhQLb7UuAlDeVz7grEY/UxuVEMNGBbGAcyjHqIUpEDW2cEwBUpBVlq 31QA0wF/jYnT8VIg/U8fkx/FQAO2hXEge7xZQwHKLAfgK+tdQZLyPxXAerACknS8FEhr2Eeuv12B AmwL40D2qJNDAV4aDsDnKQVR3r+pIIpjxsV2SoE4rz4mmnugANvCOJGttH5QgP+JHIDPUwpI3v6p IIuLwxQc3RRktiF9TDSqgQZsC+NExnM5FJSeOPg8paC/QFNB6Q4WO6kUVL6HfUyjHGi8thBO4DAT 0eeeN/g6JcCpRKxy3eWJuwkQSyJDSnMDFFxb+CYuSZfC/HywCCiPNLSzt2L+bHt7Z8VXXvxe2h0Z U547oADbwjiRIzK3K+DkRkDVaSg27FLgJA3F59wV9C3oezDtsAZsC+NETqjUXQFf8LeDZVVpKBb0 UhB6MxF0GnLbM8dUR0HUgG1hvJDtUMC1jiO82JxLQpZWb0qAK8nEE6fjJSFLFyRjtqMaPBDbwjmh KyplFg1c8TkCHjbfNSSc3Gdhpn4QsPv5rqFINyRjdqMePBDbwgloi99wETQ08LvHEciWLxrGwhx6 OqLfuWvgwjvHPJ3yA7EtnMRfgt8O2SEaHL/9HIF8+aahL8xeriQmUhq6O5AxDwkPwLZQEn8Ifnuk R5eQZRMdzGz5JqEvxJ3gPMDklQQK7znmMOrCA7EtnOSll4U0OzSw/eMIJMw3DX1htnIdMFmlIdb3 HPO4FA/AtlCSk+xAawG5IoE9MEfUrxk5FqYqNSEVScxLQu4+kUTC3AaN2BZOsj07Ej6ra+BOABHe fs/IvjBlqYyYstIguyCbMB+IB15bGHFBC8wtrkocCgL2hyPc93zsC1OStwFTvCtgjzbH/ElIjdgW zsDfgd8FmF0Dt4Yc4b8nZF+YYndsUR7FS4N37znmMrNBI7aFM/CX4HfFznYN7HX8m4sr0BcJfV2i blvRxikJQeqzjLnObVCAbWEM/B3oBRxyI7ECvCgkAciWVcFYl0K37mhllYIodlHGYkeB1IBtYQz8 Gfjtxe+ygoLs4QBkyxcFfR3sd+3WPSsFqbznWKZx1oBtYQy+90NwwkNBRu5wAHLli4K+LllxzMmK f74UFHrPsfghQOG1hTA4SQv4qFi7ABjSyAGFK8sioK+D++amAVO8u9ZUxS3KWPyoCRqwLYzBSlrA SuahAKZcAuq3RBzr0EFy44Qp3BXwtsxx+hWN1xZC/F2yImFnugBsDgdA6bc87Otg/7l3xKSMe3cJ QejnBii4tvBhYyQneOr8yA4J8F+zsK9DuGX+qFuXTOwW+1jiKAYasC2M7Gpd4eMMQwFL5YAAylVB X4cTLKyAdPuWkxgEGUua90ABtoXR82fUntH8u9aeNLgvaRXwP7arXgeP3Ab2eYp9Ap/+f2p3AXKN y8N1qYKoCK64188MSe2upIUBrj9bmhlJI4q0YTBxiWV6+RFQJQ1JbPY6rnjj4AtcRNc7Lfw0a8T4 DCP3U4COwzVOsczb/Ajo8ZqxNd2BDXAcjIGr6JrXRAEvLCfgAn8p0HFIZSGWmdFuBc3JFdTYNRtu gONgDFxGn7kdCpizOAE57EuBjkP176BAsvpbQdAsxNidJqMNcByMgcvofOCC5J7Ov3AC8viXAh2H J40dED4tvhUkUSBxbsGKNw5C39UWwFcBqJ04IfF9/RCg4/CuVwqoYsdHgFQmGrvXZLABjoPRN7UF sFQB329O8N9G1HH4lamgiB8fBTVfM9qrvOGNg9BzFZ3lXlMBLGI4IX77UMehwIsUkMWOj4Durxm7 1cob4DgYPZG71ryiQJsvj0IWL/qhwMbxf6kgiR1vBd2V647JksEKOA5Gz2V0rfz5u3X1DVf5pUDH od4WH4bVhz0wE1rMegob4DgYUcY6635EQDPb4Jy/BOg4nGnNd9/zCIjtumPRLdgAx06ItrZZByj8 xVwDp7t28uu4xJbkbv0e/hyvO1qNvAGOnRDla8HPZJZoWQp/z6v+xa/jeK2R9fEJC3+RG6hRPbjh jZ0PpSvyRmXlkkifpOr3yHX+i17HMa2RHq9qftNLPWaxaybcAMdOiBXgF95IpY/SiHpk+fhFr+OY 10mPsuKhRwdwae7bQMbOIcePakp/BWRkDg6wx83oLv6R9xyJk01mZd5JAAN6QyvI1odN6H+pjVW+ DZAtZBdpRySA/yMNjyE2qUljE/tKs/TsHzlZMwgtKhaHuvJNy6YUKoNX2YL3Q4oYFliowSqzc3p2 rXnZ+0naZbusS7wPzXkmDI3JEtcGOFY6g4WDhBsG49BmBzq5nRyCcScvTklefPNwR6ZLjSl1414A x0pnsLpqWAsD8U/+zVy7nOhkDpqued1fqcol1kwaE5Qp8wI43mQGikpYmOEwDvRmosnc9OSNOcoT laI8WA9zYXbQmCxHbYBjpTNYtAHCDcNyKAuyNzdd+HAneaBTkuf64a7MjBppTuVeAMdKZ7DdfObE Fo3Wf3PjxF7cWV6FlOWNeLg704LGNBuWFXCsdAaLhCFGQ91FXzTW4m/yJJ3ZJK9ao1V5Im9yL/Wh xmSZaQMcG5/hejlwZE9xBjuRN3eUnmxyNykJUhPHPdyBRYlGO+4Vbixchon0osxVncEe7NUe4Pny L+auVuur1VD6XDOyb1DuBXBsfIaLNzLoppcfAc9JYw/6pnfSixk9TlG6E7e6jScwYw6W11bAsRMa MIqEVLsIyLgrGA9LvAWULq3YFODFcvgslvNVBEjM0ZuABXAcjIaMQqmYAiR8TEAp1RYFDQCPgiBN Ae7x6jupBzXmmdpXwHEwGjKKxWYKwo/CCQF2eSuo2MNHQZKNxCfEV48aHFOcxpznHiyA42A0ZFTs zhR45CpMwPL8W0ERsqkgi5PwSYuCwFSnMWdL8ivgOBgNOWCiKXDiG3QVcVGA03wpKHKP8CmLAikI Nc5cv+KNg9CA8fqbEfF80jdorPIiAIf5ElDlKPFpi4DMrKcxV3tjV8BxMBoyOkczIsoe8U0Bx1tB kNs3FTSxc+4C9igQF4gJmiWCFW4cfIYb4rQh6q3aMKGC4c3vJfkYf3Fiw+I2GzamP418zVXBAjgO RkMOCddaFWCvOKFvNnSSeKcCLzaUyS8F0fESaizOEvEKOA5GQw4F11oVwC8dTwSQJFubAtnuR0EQ G8py3gqkxtJYvCWDFXAcjIYckLWzZOSIK8MJqIoWBVVy31SAoqNnfmjHR0Fka6axBEsGK+A4GA05 4M0yBUiznIBV9reCIul/KkDpQQVJ7PgoSLyEGpl/VcECOA5GQw7Ik6YALw0nYHmLgizv31SQpWLG xfaLAqm8NJY092ABHAfjRGZzJwrwP5kTkjQ6j4Ikb/9UUKWKwyeirngUVJYhGkuybLACjoNxIuO5 NAVNjYPlLQr0BZoKmlaw2MlFQec91FgsHax44yCcwHEaMVT1DVa3CPCLEbtcd3niXgKkJJFQytyA BW4cfBM3SZdCfh4sJrTNhm72VuSvThoXeeTf/EHaHYmlzh1YAMfBOJEznKsKaG5M6KsNpQx7FHix odQ5bwW6BboHsxxeAcfBOJELMrUq4AVHL+pWG0oJ+iiI2kzE1YZse2Ys3RLiCjgOxgfZmQLmOs4I UuY8Eqq0elMCqpKa+KEdHwlVuiCJ1Vk22BDHwTmhOzJlFQ3M+JyBGra+NRSc3D2wJj0IlPv1raFJ NySxessHG+I4OAHt8BtVRDINfPc4A2750GADa1Q7ot95a2DinbHOSnlDHAdn4krw28MdosHz7ecM +OVLgw6sQa4kPmnRoNWBxGoSNsBxUCYuBL8D7KESqmyiRzHbviToQNwJ+gCfsEhI8ZqxRssLG+I4 OFOQXhbSnGlg+ccZMMyXBh1YnVwHfNyiIfdrxmqXYgMcB2Xy4g60FpArElgDc0b/dKQNLF1yQmli zEdC1ToxiYS5DSviODiTU3cULEs1sBPAjOC+HakDS5XMiE9dNMguyCbMB2LDGwcjLmhDcYurkk1B xP5whv/2ow4sRd4GfPJbAWu0GettyBVxHJyR68DvBkzVwNaQM8K3IXVgyVqxZXkUHw3BXzPWNt2w Io6DM3Il+N2xs6qBtU64mFyBfkjQcSVp2Yo2bpEQJT9LrH1uwwI4DsbIdaAX8PBGoQK8KEkmwC2n AhtXopbuaGUXBVnKRYnNWYJcAcfBGLkM/A5S71JBg3s4AW75UKDjUH53Ld3roqC0a8Y2C+cVcByM MWg/hErYFFR4hxPglQ8FOq44qZiLk/r5UdDSNWMLJmDBGwdh9GIL1FG5qwAUpJkTGjPLIUDHofpm 04BPfletpUu1KLEFywkr4DgYoxNboJSspgBFuUzoX0a0cegg2TjhE98KuC0zznplxRsHIf5dXFGw MyoAm8MJUPrlQx2H8p+9Iz5L4a5VQhT6uQEL3Dj4sDHiCX6UH+6QCeHThToO0x3589q61MRqUWPL lgxWwHEwsqr1jccZTQGlckIE5alAx+EEGxWktX2rRQoEia3Me7AAjoMxcBldHc3fvatpcF/KKcCG wcQllunlR0CVNCSx2eu44o2DL3ARXe+08NOsEeMzjNxPAToO1zjFMm/zI6DHa8bWdAc2wHEwBq6i a14TBbywnIAL/KVAxyGVhVhmRrsVNCdXUGPXbLgBjoMxcBl95nYoYM7iBOSwLwU6DtW/gwLJ6m8F QbMQY3eajDbAcTAGLqPzgQuSezr/wgnI418KdByeNHZA+LT4VpBEgcS5BSveOAh9V1sAXwWgduKE xPf1Q4COw7teKaCKHR8BUplo7F6TwQY4Dkbf1BbAUgV8vznBfxtRx+FXpoIifnwU1HzNaK/yhjcO Qs9VdJZ7TQWwiOGE+O1DHYcCL1JAFjs+Arq/ZuxWK2+A42D0RO5a84oCbb48Clm86IcCG8f/pYIk drwVdFeuOyZLBivgOBg9l9G18ufv1tU3XOWXAh2Helt8GFYf9sBMaDHrKWyA42BEGeus+xEBzWyD c/4SoONwpjXffc8jILbrjkW3YAMcOyHa2mYdoPAXcw2c7trJr+MSW5K79Xv4c7zuaDXyBjh2QpSv BT+TWaJlKfw9r/oXv47jtUbWxycs/EVuoEb14IY3dj6UrsgblZVLIn2Sqt8j1/kveh3HtEZ6vKr5 TS/1mMWumXADHDshVoBfeCOVPkoj6pHl4xe9jmNeJz3KioceHcCluW8DGTuHHD+qKf0VkJE5OMAe J2NsUkTWLJ2hfYzxt59/tevnr8vxDx/7Xz9/h4x/QsZ/7A7+fXl3/ev64093/fsf7JLYr1amsARd /+e8bHJ0x20oOq9V1BKsf3kZQQZZQc8eA3TvfxDeS8qWzFnQgL9mPfJYlijyUhc6darEFIV59g9e ioHBHXgaeJ+f9p+ff3NpE92srqXpz2ULxZH8n0s7XyTfpdqb3x3e/rmP7Us0vt3DzjGrrfcj0V5D qh4C/s5dthx5s8Y15OkpAaUH1s0ehta+Mi1gbuhVAt6L+HGUQFL7gq1p85A1iZArdSNT8t7Vblp6 0JunBNRMTDctmMlH0KkbqwE3s2+hu3WPulXwj6cE1NTlmz0cPfRaMGBsaKtJqfoLDH14SkCpjSuj 7as86KyJqn8vG7pa55tb8/94SkANzKvVepihO+/ATEzEheZWQFhaYzP04SkBNaCOaPcHfVnA2NB5 qfZXN348JaAGZtxqvd/QTd+tAVB3LzpVH4qgSG5HH54SUAPSgPZ0NFQzA+4NfXHVxdY+HH14SkCN zn+gclrohIBCP0fPmwoqcUNaftCbpwSU2vicOtgIgEZt0oDKr1voaVPC2EbV01ECSUvtMHs4GZ/H gLmRKfjGTK7ADX14SkB5Fa9oAQudNAVmYyItNFPDh7tUHvTmKQGlbbXoPuqIcA1DJ62IGtC9+Ri6 tWdyxa4Y+vCUgFLBMMzujrYCqgFjQ3MIYIkac1WQw1ECSW0fb/JDzkMDBoeYRS5cbuLSn5J6eEpA ucpqlw9eKkL1pRpgomyhOTt1JuBoXkFOTwkoFZA4/ZZ8rkXAZQF9Qyesuk/f70XeHCWQXJo2nLWR UdY1wKTsIl9Ybu++30Y+HCWQVHZ3s4eTkZMMmC/ZLkqv3I7k9eP0lIDqOl1pWjVcq4VWUfOrE0C7 X/TkojOa17jKg948JaDULrBRCwyN668BWoE39OCqWT/6KqinpwSUTkLD7OFo6EIGjA3dcYiQkfjo dYqHpwRUhzhVW+eOB521cOpYWDZ0Q2q0Mf0sF3rzlIDSCbIWtTsHMKDRIDRAsz+9aN7vxjLSV0E9 PSWguk4LZvcHfVnA2NCccFoluowHvXlKQKmdYQ8OrkCjGWuAFpoNnXGMjc+evYKcnhJQvWmq0MZY CDQkhAY0XdKLTki7llBR+7Xy+vCUgOpVi7naUEELnSygb+gL662T23LPB715SkB1fI7a0G6GTlzd ja970P3GqitLapvrGA9PCaheWCnVHo7GtWbA3NBcb23c8XWKh6MEUtfmpoWyY0mLnBAw9Ks2MmU/ t6O1h7w5SiB1pHv/7ZD2sG+IefVH8r9gI1M8If9APh0lgHpijdKfbuDJfnTjwm5gcC9emWLV4/ST wOmJZVVflxcXZfdGjckPlzKsTO5x6gu8OUoA+ZZwgwge7BTYQTSOhwzVUQaetsUfPwmghuqttlZW 3XmCLw/oG5jluVAkVNdLH08JqHZz8ZqIydCdLRmZmnc018xWXr1rfTwloNrkhqs9DN14IAjA+Sz0 xVVnz2iST0cJJL/lfudJTgzIzKiHDGhhv60+Fn08JaAaJInaKqcMXZn2KGXpRTesObMprmQ+/SSA WudNVHsYuPCmImBs4Ik1s4pWn4k+jhJI3gS8JZCcGVBZbBYZzNyqXxcn744SSA16VW3VUkbOLIlo dGkjs1VlVo3iHfzjKQHVKpWh2n2hLw8YG7oh7zIbQPGR6OMpAdUq24tqhmzoxPYDUVE2NLtgvvAs 3Q/x9JSAcm3jSkfRF5smpBB66EIXrDqxShfv4B9PCaiG6aP/Dqu8RCcP6Bs6Y72J1RTX0dG7pwRU yxQgqioTbRWDECiQnXlDs8FC0eE6Wvn4eEpAtUTZpPYw9KSsQsD9FlPck1SQgMUnoo+fBJDrdVfv BCcG3FSGDrbOnRJz2yeij6cEVEOajl+dOS5DDwpYDCVpR4PMEpJ9Ivo4SiDpaDTM7kbuVN0IGBu5 I52v2e2mk3w6SiD5OOfDHcl51F9Mf1U7yUNG4l3sidkHoo+nBFTF8EH7NrTqHgboPdvQlYu2+1gt pz+eElB16Gep3SjCib48oO9orro0VpFF3h0lkNTGpDi0uhpZv6YhQEvaRsZluTjK5eRneDpKIFWM HrTvRLKWXQ+4byf/Da9aHq8yaOlP0kP756+f//z+98fljhcZvvtESXhXxWyitubZendigBbo9e5/ /VyAErhE76RiKp3R7yrUO/3iP+q4nHhK+jIcmgA+9X8KVNiwlyFR6sVwVYj5YsaMwb/7lqIh63NW Pl0xn54SUDPzndVmVEUnqBAGzA2NG3d3pMHtGuPjKIE0MxOlJl5EIydN0oFMfMlU4TdT+O7pQW+e ElAzMVNqZn0BerIfjdszxdC8cnfhfrjI+HhKQM2LiVAxwxgaKoQBY0M3rjpVf4GhD08JqHkxZSom r4XO2o8mStWL5pWbtuOuMz6eElD6zqrLqJgXDY0RUQO0GaQXza2Yk2t3ofHxlIDSQ5hm9wd9WcDY 0BwfJuXGvNOD3jwloAZal9pNfwwNJaIB6GQvmrs8WfSnC42PpwTUQMOlPR2tAjUx4N7Q7CGz2Np9 XDs9JaC8JHiBMHRCQKGfo+eNyzITN8SFxsdTAmp0fo7WveRoKBENqPy6heYB4pJhW5x8OEogjcYz UHs4GZ/HgLmRBxY9JjMwL/ThKQE1GjNHG0x50ElTYDYm0kIzNUbPvisLvXlKQHkj9LYIdGK11caJ 9F9ojlWjdt8VQx+eElCjcGBRuzvaCujsvLQLXZF7LFFjrgpyOEogjcIhq11sXkbGFDYHh7JF5sCG KUGfT0k9PCWgXCe5alL0dXN2nJOj5EJnLLozAYdLjY+nBJSqO5x+w7Yt9GUBfUOzYVPiYr8XeXOU QBoYQtXGWRsZZV0DMJO+ZM6Cvft+G/lwlEBSFd3NHk4e1GjzpmRzsl2UXrkdyevH6SkB5drfJwFD Q1jeF3XmQnPM7BnNa1zlQW+eElBqF9ioBYbuVMM64aQNPbhq1o++CurpKQGlc9Ywezi6UcNrwNjQ FKEQa/jodYqHpwSU2hgrdFYsDxpzh458ZUM3pEYb089yoTdPCSifaX3CBbpyWroLh6eF5v22gbav gnp6SkB1Fe9m9wd9WcDY0IWrpiRFZ1rozVMCSm0Mnm1wDAW6cDK9Kyephc44xsZnz15BTk8JqKUd oScMDQmhASq9y4tOSDsbDfu18vrwlIDqqv9NlrYHnSygb+gL67UJrt3zQW+eElAdn6M2tJuhE1d3 4+sedL+x6sqS2uY6xsNTAqoXVkq1h6NxrRkwNzTXWxt3fJ3i4SiB1DPnoY4lLXJCgOr8tpEBtixp 7SFvjhJIHenef3viAPDn54aYV38k/ws2MsUT8g/k01ECqCfWKP3pBp7sRzcu7AYG9+KVKVY9Tj8J nJ5YVvV1eXFRdm/UmPxwKcPK5B6nvsCbowSQbwk3iODBToEdRON4yFAdZeBpW/zxkwBqqN5qa2Xl TKsBlwf0DczyXCgSquulj6cEVLu5eE3EZOjOloxMzTuaa2Yrr961Pp4SUG1yw9Uehm48EATgfBb6 4qqzZzTJp6MEkt9yv/MkJwZkZtRDBrSw31Yfiz6eElANkkRtlVOGrkx7lLL0ohvWnNkUVzKffhJA rfMmqj0MXHhTETA28MSaWUWrz0QfRwkkbwLeEkjODKgsNosMZm7Vr4uTd0cJpAa9qrZqKSNnlkQ0 urSR2aoyq0bxDv7xlIBqlcpQ7b7QlweMDd2Qd5kNoPhI9PGU/5Fd7rqW5DYUzfsr6g+s9yOezIAN DDqYwHDmSQzQwIwD/765N6U60iEaYF/eS65SqSRy06FqYXtRzZAMHdl+ICrygWYXTAE2t/UR70hx qKVtltJRdGDThBRCD93ojFVHVum8OvhXpDhUxfTRnm6Vl+i4EtqBTlhvZDXFdVzoM1IcqiYKEFWV kb6KQQgUyM50oNlgoehwHa18fEWKQ9VI2aR+N/SgrELC/BRT3JOYcQDzmoi+4sSBll5f6p3gyIRJ ZbjA1rlj5NleE9FXpDhUxTHtj84cwdCdAhZDSTzRILOEpDURfQWKI+lo1M1vRm5U3UjoB7nhOIfR 7KaTfAeKI61xbg13JKdeHkx/RTvJS8bBC+yJaQ1EX5HiUAXDB/1paNU9TNB7dqALF233sdiZ/ooU hypdX0v9ShFOdFgJ7URz1bmyimzyGSiOpD4mxa7V1cj6NhUJWtIOMi5L4CiX4vqGd6A4UsHoQX9G krXsroQ5F/kPRJX8RuVOT/+L+tH+/P3Hb89/fiy5s4oMn32jxD2rYDZRX8/ZfnZkghbo/exff/zl l/+O55efz/2An7/8/Ud8/vrE59/6m6n97n96B56/Pf/4Z3j+9erjQXGVGx/0WfDPH4Gk1c1igrSJ WBm+sGDbIl4Ukq3byrDEghsNP6bAJerSM7eR+29bbxaq0tBXpDjUSNwN9aeh4+R2acI80DwuZnE2 ib4jxaEGqpf6UOEbjY7VxzpWhuYhN7slyVekONSI/ENJrEZADx7dPte5MjSvplmUAUNfkeJQI/B1 1B95o4Ml2LExNOuJWVzCjT4ixaFG4EcomecX6M4qMcI6gURbFTSLimvoK1IcSp+Jo1MwXRoaskUT Ik/SRrN0my2zLfQVKQ7VJ6uD+u1FR0sY+YNmwzGL3rbRR6Q4VEcFU79qlTB0pQjRz10OdP20yrrJ V6A4UkcFoz8XWaskErR5HGR2drN7nvmKFIdSH2ViVRNDp8yEdqCtV9qsm+eLPiLFoTp+UN+KJNCI 1AQrowttKspsLfvsXZHiUB3Lpz9edLCEeaCp/czWOl/0ESkO1SufudoR0PgqmlC5hI2mYjVb2z57 V6Q4VC9UmauJAg3hogmNG7fR1Nlma9835ooUh+qF9Wy1fkNHS8Dn3ui+B4Q62gs+4sSBeqaeX2oF YLsRo3OY22COM2br3Gu+IsWhVFRhCFkSS9FhckoZg3PnQtsMZrYFa8RfkeJQPXFyqqiGGx0toR3o tKfHFvsLPuLEgXq0VmhSFuDBYXCJ3Q3msGu2pb3mK1IcqqPT0J8L3TnCDjSmD7pyO2hRdgx9RYpD qR9VVK2xwdCpp2cGitKN7vHZtpX6oo9IcaiGrq6+DTtAo+1rgo1DGz0o4GlbXUfvjhSH0qFsmj9e dLCE+UFjIN22tfKij0hxKPUhYNdgCbT+hISkr/VBp7SFEldq6CtSHKp1yu6qK1joQl0+MyetV12O Z9s29qqvSHEo9YP5vW50tISRP+jan23bjC/6iBSHao0DTlVhtdCZE9AsusgPuudn276E6VekOJSq R4xl6s+FTpzbpur0A83eYhaVztBXpDiU+hglVZjWF41ZUxPaB23Dqdme8os+IsWhmspm+FOPv6Gj RmpC09f6oK2Y0vZNvgLFkZoqePPHSw6WMA9ysYbIrS7pRR+R4lDqJ62wDd/I0BDGmqBDwYFuRNP2 fc2vQHGkpsOUnsEGaQ9/agFG/NCX+oD7eLbtzdZ8B4oDqR/M7xscLX4c4MlmSLvk6R0njqM/4D7q r7NxB++rpuH2Lm6L+dm2b/AVKA7UcD7pW6vRLetaGbGD4SCn+Wzbh0mPr0hxqAY1q35mMyM6rgR0 u40u9dkWPXqhz0hxKD0iqNd6FrOhG+s5TlQ+0K08246wVn1HikPplcGWqz8NXdmFkDAPtFU6gKPV /684cSD1cUbW9EkwzhCubj3Ak9tBO9LY6DNSHEorCPq5liqcckUX9nvUMlyDhcbWbYu+QfQdKQ6l JXWaPzY6rIR5oPOzzChjg884cSD1UUC0chcDZxYYlPZ6gEt7th3VbvdXpDiUNq8Gf7AGKlrlFBMq i+RGt/RsO9o6eHekOJT6wfy+0XEljAM94rPtWDXpK1IcSrs5ipPKhmzoyOKF5l8O9OzPtmOsVd+R 4lAqb9AX1Z+GDuybUEPhgx4sHMvOteo7UhxK/ah+D+ztRKeVgOa/0ZSlZmdIG31GikOp3kPzUmGZ 6KsebExQiXKgy3i2ndGkx1ekOFSNvAHqj40OK2EeaPYUszOljT4jxaHUh95TpV0MPagHIcXrgR5E 0868Vn1HikNV08g6eECzKlp3DgkxUNRu9OSG0M5i0uMrUhxK/WB+3+i4EsYHjf19bc0bfUaKQ+ns lXUe0UksG1qVhLY+jGpFxflG5/K8tq1V35HiUDqKdvOnoVVQMUHv2vygy3xeu2TNV6Q4lPoRvsbF Yui4EvqBbvV57djkM1AcqWiFha8vZ+RCKaCbqG/1IQ/uhNm5zt4dKQ5VGiuD+tPQqo+7JcyN/gNR Jb9RGRqp47+oqX/+/uO3RwXUg3985o0Q9wz1kdhV3Of1zMSEQuHLZ/76Izz4t6UqJFTHuFQUOEAu z2pyVMHoauvv3B+FQcoGajdsD2j6EgZMkd/wA7SEjPmr24qq+iXw8KiMTBBfEUvG8VobHiJ6jFkM Moa+IsWhRuIuFFwKouPkNvXOd9rojPNhthTTIV+R4lDq46OWyE9saHz1PngINrqix5jFjLTRR6Q4 1Ij8Q0k8mEAP3TpNmDy6G924IbSl7VVfkeJQI/B11B95o4Ml2HEx9CCatvT+oo9IcSj18RFK5ukF urMsjMBvstB6vJ5tMX4Z+ooUh9Jn4uiUwsoFNAZJTYg8SRsd0XTNltkW+ooUh1I/mN9edLSEkT/o TDQtNM1GH5HiUPp9C/yqTc7QenOQoP3iQFduCG3d5CtQHEkP5TB/LrKWRSRkfasPmfrXbM1rxr0j xaHUj1X9xqpj6JSZ0A70IHoYer7oI1IcquMH9bsWEkMjUhO0XdQXnQJ6rtla9tm7IsWhOpZPf7zo YAnzQEe0GLO1zhd9RIpD9cpnlsGbADS+iiZULmGjM7bCbG377F2R4lAdJ6U+BefG0JE1eTRu3EZX 9BizGHYNfUWKQ/XCeqZ+nxsdLWH0DxrnmaaO9oKPOHGgniniK26oge1GDFzh/IIHaqnZOvear0hx qJ44/NSw5lMILYyXY3BUWegcADXbQl0964oUh+qof+pHDk+GjpbQDjTuIE2L/QUfceJAPbIZVlRw Aw/OgAMl/gPOqKRmW9prviLFoXrgYFrRdwzdqTTG5CC70ZXbQYuyY+grUhxKfcjKim650dCdE1/r g+7oWmZbqS/6iBSHaujq6hdKY6DR9jUhUjxv9MChM9vqOnp3pDiUTmLT/PGigyXMD7oEdi3a1sqL PiLFodSHYtWRryy0/oQEnUsOdGLXosVKDX1FikPp1ItxqeoKFrpwnpqZunyjWTnMtrFXfUWKQ6kf zMfIZ+hoCRgKN7r2Z9s244s+IsWhWuNMU1VYLXT+P9lljwNLbgPhfE8xJ1jo/yd2bsBw4BM4Wwbe +wcuFqUe9hAL9Fu+V/qao5bIIoeejcnVoWf93OdMx+69lRJQ8I+zM94HXTiqwWAmh2ZvsadWumtN nVICCnHWeHGaNHSpXDC+aC1C9zlLfdBOKQE14O013jj+hs5QYgG8ff+irZjyOS/5JZRAGvDXFq+H nGzBduRmDZFb3cqDdkoJKMQFFXboNzK0GmMs0Jniix5E8znvNX8JJZBGwY8an6HWXuO9OTTsdYYE A8/1uc85LOe3UAIIcbJ4XnA2/XLgzWbI57Gnb50EDv5H7yP+uhp38b5imd7ewx25fu5zXvBLKAE0 9HwytlaDLZuojLqDyZHL/tznXGY9fpQSUEPdLOLKZkZ0Pgu0211065/71B590F4pAYUjovUaZ7Ea erCe64mqDj3a5z5XOlm/lRJQuDK65Yi3oTu7kC7YDm2VTsHZ6v+PTgIIsZ6RM38SrGdIr2534M3t 4HOVddFeKQGFCqL9HKVKTznQjf1ea5leg4PWrbtP7RtEv5USUCip2+J10eks2A5dP+ex2rpgr5MA QqwFBJW7GbiywGhp7w7cxuc+V7fb/aOUgELzGhov1kCgYae4oLNIXvQon/tc4xy8t1ICCnGyeF50 PguWQ6/8uc91atKPUgIK3VyLE2xDNXRm8dLm3xx6z899rnWyfisloGBvtC8i3oZO7JvqhtIXvVg4 znOfrN9KCSjEGfFM7O1El7NAm/9F05bac6dy0V4pAQW/p80LxrIwhh8cXACL4tBtfe5zZ7MeP0oJ qJ55AxCvi05nwXZo9hR77lIu2isloBCr34PTboZe9INqxbtDL6L53PVk/VZKQHXzyBg81LMCjZ3T BTnR1F705obwuZtZjx+lBBTiZPG86HwWrC9a9/d59nrRXikBhdmrYh7BJFYNDSeB1qejWoM5v+ja Ps9znKzfSgkojKLT4m1oGCouwF3bX3Tbn+d5bM2PUgIKcdYYutwMnc+C6dCjf57numQvlEBqqLAa 48cZudEKYBPxq77kxZ2w5z5n762UgGqDlQHxNjT88bQF+6L/p6pWH1VVjzT1j4ylf//3j/98YKA+ +h/f+UZIeAdiXThh7ut5Z+GCRuPLd/4LPwhzYzqu59//+CcdqzqpqVNTA3fpC9rn9DqaYW1u59+5 TWCqo03zmmMH+OuPf7us9R+12nzpP6vPz+rrGXwLDdLUOteej7DL9yOs9aCdUgJq6SzJeEzv4rFA T/PzfXlw/Jn8EUogLS2QTf3xncUy79yz90ZmHz7HfeTHajulBNQq/Jyv+UCbKxZsl/TI30va+oN2 SgmopbWX8Xiy1iI0tTh/0Z1Z5+UK4o9SAgpxtVeZ41TDqG4DC7rLuuXTdr5l/EcpAbVg4/CqkViC Da1VeS7+/atqnTKexoN2SgkoxPrOkei0vmPN3KwcF12uMfl24x+lBBTiaq+yiXJvdqm5uW+vKr56 Oamf+cMJJYBW4tf1A5NaUeitChg4jWMtnaN6CSWAVuKJHPkMqnuzZeOv8zfjZXY4v1zgSygBhLja i+admPQ646/7N2N1UDob0Ki1OzA5nQTOSrz3o9CskKuWcSVWgcudtNrr5eJfQgkgbJHWKsQ9H3DF QIEvkV3CPA9z+sHjpZPAQdzsPfMmrA5maVP9cjsT7mc3DtfpJHAQJ31P5Wj2TI5L/dCXS182eU1+ BscjlADCxdF/QGy2Z6uvh75Q9jhKDgaZE+kpoG+hBBDiZi96BlKdUaBvLuPCjFM9w8EBO6EE0Fzc +6EzzxlHlyUyXMY8voPN5M4EL50EDmqqnhXEZ0ZC59Y8qm9SOtnqlHs3+oCdUAJoTp7u0a5rH/Ti q/oWZddisE7MUi/YCSWAEOt1RPNO9Qy5yxJxHUoPgvnPRI9wZlwnlAA6fmP04+C32gPqXX+yiWtk BY87wbyEEkCIu71o3IzVgMAwuO7ECzfYQcatEy+dBM4crNGIzVBuOMtMvWtNNn32Nc4XJPgllACC kWQ70S9xwDr9Qe8aE29yZ9Z9XKyTScAgZivBz7j56jQCO1u/5Wc25tu5zccAvIUSQIirvegMn5XN HnrXk+hZetWsRzll4qWTwKFVRzzPCAN939S7lqTnFuByjsXhOp0EDmL9gkN98eFWzWP4jqT3Hlye in6xTiYBc4YWxNuKD2zSpt71Iys7rJn97MJLJgGDWC/IQOXoBzstDdeOBvtnG3Yo1gU7oQQQ5je9 0oj7zZdtZvp2ZNWs0Wv2US/YCSWAEDd70bwZs89MlqwLnnolGvuGHrcDdkIJIMRaNgfMkWWcrc9M FtkLHsyYFqjXs8cvoQTQLPzeiE9DytZnFrvCBXfNuNLX91stX0IJoFn4P4jnzZh9ZnHZBTf9bJUe U432ATuhBBBiTX2m25CSNRqdJL5gFuBKR9HOiPkWSgDNTG8z0+1IyRrN5qe54MKMeYzbulynk8CZ 2aa9dBsSbF2mXs/S5bKw11p4mPcFO6EEEGI9z/jD+giEyxIZLmG20MoG3bp9u7dQAmgm3kDEdR9w t0T2N+PO41DZSFvLF+yEEkAzsWIgHjdjbTT4oVpfHjCIhe2uPfk6mQQMYq1viK2N4BNrm9mJ1fBi p+Zbxjh3hOCXUAJobBbkWU4jgV4bzbaZ7gErl1UolYt1MgmYsdk/EPebr7YZnHz942JtH1gm6poX 7IQSQIj1DYitj+DaawY7M58HrFtbWOXrcWtvoQTQWNwRVA9rJNDrjkG/XcZVM86bB+5UzLdQAgix fkPE/Was33jr5ODAmnFmQa7n1r2FEkBj8cwhtmaCZqBnchee0AtmC830l7WamXgLJYDG4i1BT7Fm Ar3aOujXt/5oMQOYZaKWy3U6CRy0Pb3WiPtNWH3oriwCl6vUWnnx9sU6mQTMmCxDiNc8WPXNu/qW 1JYmmlmDyp4X7IQSQGOybsK6WCeB6ZiWiGtJ3IBMc6k3mtyXTgIHJkgLPWKb4aDXXrmb70haIlG3 F8Hzcp1OAmcM9hHEMx+ujlK7+YbUhp60xCJR+rpgJ5QAQsw+Ms4MBxu6LJH5rT7mI5JdvXZOxEso AQRbrN0fsc1w0OtP2d03JC29iSdJi8XlOp0EzujWR8YZ4aBXOwO9a0hND1li29BScbhOJ4GDmG1k nhEOc8myPFw/agXfTKHpJPsSSYCMZi1knvEN+m5JfHuRAgnj4aTf1aOa4Z7y0j+0V/39X9Xmj/6n WpxKbH+56cv9eV0dzfim3xe7519c0Gb5psbfs7eewc3atNd60E4pAXV+RF80g4YutXx/paH18mye 9D3rQ3ZCCSTE6tG7mhEjqxvhAjelbs4km2dyj7sfL6UE1CqcKro244vWOocFbk7dvEGbbWufMepH KQGFuNurxpN10VyWn1S1Q6mhWOcFF+2UElCIq71qb0MrggvcsLp5izYL1S7loF9KCaiVbczUi3fR vXOBG1dtKxY97k7jQTulBBRiDppqTC66ai7w7S5rvUt/LrrctfuDdkoJKMTVXrW7xlvNCfVuZt0c IhaHS02ddvQllABayYZNmMl9wH1T74ZW3QH14eMkfsBOKAG0Ek/kUHNywBWJ4K/d1Lo4By4WrNX7 BTuhBNBKvEVDZySCF6/zSn5uXYvbS5+77NC9dRI4K/HeD/Umh8t5NPnBVbto+nMupl3WBTuhBBC2 SGvVUG9ywJxHsx9c7TzM+X/Sq2XHbSMI3vcreJQCKJ73kECQQ2wHSIDkYt2CHLS71JqJRjIkOsb+ SL431d0z0nDnkE188bLhUrE4j66uvByZt8Klhge1k/fEIpjjqM7eKbyeBfu8Gpm3wqWGBzVZWaDh hHmjpFHNxld4eUiIfE1in6PPApgaIlwc+o9gS2yNkkZNtk4h5lMW9cDLYQtxBUwNEWonL8qxNUoa Ndk7hdiwYkVXOgZTiCtgaohQ09oHV2JrkDRqsnkKMR/fwGaSJ/MlLjU86Klsea6k1iBp1NYm1XMI DKEsdCaugKkhQu3lRaEINpbxlUXJtQjcJ2IeP5bA1BChpusYfE6bsPJehFQORQeBbisfZZ1P2wKY GiI4OTUQ1CZmYk6jtvanyMEy8Kgbcs9cAlNDhNrLi0JRTHkUA0PlTnzhAjtIKH1igUsND2rq0cHn uDk4TknAV9ZEBwyNpg95B5l4AUwNEWZTtpOQAyfwlOuAr4yJbzKH1+BDoa1gqaFBzVYSctwEnnIo BmR7az/RsV6edEMeAJbA1BChtvIiiZsIB2T2wFeexDOL50E3mNwmFrjU8GAeZCeJOW0CT2kU+MqS IodAPsJ0LDJvhUsND2raQdS+6KUwishUORLde0lsQflCW8FSQ4OaoiZqiZrIijTkAl/5kbQd7pk+ r8IClhoa1HRBEDglaQ5GXCbUdhTYP12QQ9EX4gqYGqJo+Uqj9kUv20ys7Ui6meQfH2whroCpIULt 5EWxKGafiXVopYFXcbeV45aJK2BqiFBT2wxDDpuDFp+JdWgNgRXzCORtXuMFMDVE0fB+o86GpMVn +jq10j2jUYH3rnTLBTA1RNHwA+pYFLPP9HVspXsGYp4xadDOxBUwNUSoOW2qYkhKjIaSxI2YG7Dl icINmXgBTA1R1DzbRFUcSYnRDHVsDYYV8zF2feGtcKnhiVripiqGhLFOM76KrdLYrTV8mIdCXAFT Q4SaI6fOPgJgL0Kq3BrYQi0btPM5uy6AqSGKSrKr5vvIxF6E3LIrZ1MQs5E6pwtxBUwNUVTcMVCH opiMBh9K/eVKDEbDdueueitYamhQU39DLTaCLSabGRR3w0IbSa8JId8RJl4AU0MUBm7I0WQjAZ6M ZpBMdyUmXu5CyhTaCpYamjCwf6D2RS/ZDE4+/Sm0sg7cJmwfC3EFTA0RanoDavERXHtSMGjWcyWm pTXc5W2e1pbA1BCFnlcE3UOMBHhaMeCHSrElxXrgA5c75hKYGiLUtIeofVFMezxQcqiISbHmhmzz rVsCU0MUej5zqMVMYAZ0JgfDJ7QQs4Vqni+tlWFiCUwNUej5lsBTxEyAp7EO+P7Wf6iZgZjbhDWF t8Klhge2R9catS+CaQ4dLDeBwkus1vLFGwptBUsNTYjchlD3MdPS3DzY2pJcT0I19yAzxEJcAVND FCL3TYwu4iQYOqIIqSyJF0DzcEk3mnkXuNTwYAiiRo9aMhzw5JWDqx2JWiT6ds/EsfBWuNTwhMA+ gjrqzEtRanC1IblAJ01xkzC+L8QVMDVEqNlHQs5wGEN7ERJv3UfmCCVXz+UTsQCmhghjMbk/aslw wNOnDL42JGq9ik8SNYvCW+FSwxO8+EjIEQ54GmeArwzJ0SFTbBvUKjJvhUsND2q2kZgjHHJJLzoq P3IGe0akKotdgFJDEpxYSMzxDXgvIm5eRITdh7e/3vHh5HmXjqrG9KR7+kNedR7vPtz9sL17s926 TnfbPX5naezCadmUB0X3k7xGW7RChZ9tE0cOSbblXxKO1VEwv+3D3YYeMdlvv9yt3o2X6enYPZ12 h8t6+wdpyx0xZHMgtOP2wH+279ofQWIsEg1MZMCAtCkPqtMBfu1kRtIDVok08hqwLpKkSMx3Svn+ e+ELVz7q9Mh8m/JQ8eH0gZBEJf48XX2eIcbfVr9MxymtcZJWuzXMYXWYLvMac/uqe+SPWP++/fnf 9BuFte1hFgv9/1X1laVWvXmxK1pUvz/u7g9jN+736w0u2Wpab/S3ZvUwjce5m9Knw5jwtJun0/Hy qi9w8DOlwld+QWFp1t1cP0AN8gW/nrrTuaOV91CPM7WSffDYB7s6dKdPrxffI8XYaL5SfGF5KV65 2+p7vhM/jrv583m8dA+7Y3c/djgmp/P42H2Z5o+nz3O32+/Hh3k6PnVyY2hvFLwIV4MJ442wl9WY jvN4Pn0az7v76bCGM6+m+bn7e70xHh1mdcTR7Fen7jg+nWb5b+wtP5yOtI6Xcc41Xno/Pp+Ojx0v 3fYbeWV/e2XegPnjbub113zyw+r4uJvxEffP6Dwffur2oL3fXaaHbv/5+PDqzbA0K4feft1mXFma zbC3D3G8GW8P4+7cHXbP45lW/LSv9L5oPTqSSo/9yA+qc8bDvjBwaQMzGNSr9Fr6NeTK35rEwaMd iMvN1Xx2VrSgrxHjkWOUca2YV+q4/v6lDtxJ0jG9Vggs3Ngh/G8h1983QnhByhO2bz7vjpdPp/P8 5nE8TH+N52ec8vnL6fwn6Xy/vftnAIUrpSoKZW5kc3RyZWFtDWVuZG9iag03IDAgb2JqDTw8IA0v VHlwZSAvUGFnZSANL1BhcmVudCA0OSAwIFIgDS9SZXNvdXJjZXMgOCAwIFIgDS9Db250ZW50cyA5 IDAgUiANL1JvdGF0ZSA5MCANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAw IDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTggMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4 dCBdIA0vRm9udCA8PCAvVFQ0IDgzIDAgUiAvVFQ2IDg3IDAgUiAvVFQ3IDQzIDAgUiA+PiANL0V4 dEdTdGF0ZSA8PCAvR1MxIDk1IDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzOCA2OCAwIFIgPj4g DT4+IA1lbmRvYmoNOSAwIG9iag08PCAvTGVuZ3RoIDE1NzEzIC9GaWx0ZXIgL0ZsYXRlRGVjb2Rl ID4+IA1zdHJlYW0NCkiJlFfNziSpEbz3U/TZkll+kr+zDz5bPvgBVl7L0rKSPe8vOTKBrsqiBR6N BOpvyIgoSJLIX/7yo7x//fG2pliHsfr0du8fv/7x+uWvf3fvf/14ufe/369AhkJ5Z0qmYo7Gpfef s6um0vu//3z99grVxJpvC3zRC4hMyXmDwAqiUhCtsWEXEvGnmg6YVWEma0JMG50pmhTiHtMFxgwD MluT6y4iR1MjHSDTHbI44xxtVCLCq4BkQj1QeJIIGiHVmRjDIaTokGSyC3tZwasQZ52p1e9pQv/4 8onJxsVDDOkQ501w7hAiX0PuE5MNZbf/nOh1jPcmxwNPlJSj+InJph5CUtAhIRh7uCwpS0j9hBTj fd1/TY46hoKJtu55ikiL4RNTTEq7UsAxoi3mGROD5NI2psZHTDG17CoKVls5nfQ5nUTGp/2+VZse MdWQP8Q4ybb0OZ5MJpZdFeGS07V99jpXnNm+WHGZUjElGuv3xYivGsb8OZ9qDeIOMeURE02gfb3h 5OSYeT7eWpP8vuRwpumYaHI5xETJt3mxvWOYfZnipFEhyTi7L1N8/hwyy5T3zoSyLzl8/jommUin GNmB8tEWnMl2X3P4XHQM1pZ9AeE9VjHkjaNDjCQBavMMyeawz73oXhHRG9rdAh+zyWFfX1izAk0e teD/i0mfmGJs3NcXZlAxORjvyk59LobqCTVp1BJMirtq4kvBa79dUQPOc7evXLsGX63Gx11VCRbW zR3qweMrgoWby3GjIDjYua2nCR7uzR3u+uNMgseqvEUN8G/hcLcf2RECLJ07pbbOwkAwdflwtx/X IRB8XtjFhOiMrYfb/xAS4ezyz13+kGDtwikmPGKSSXv38KxlkGXq4eF8lMyQ4ez84XI/SjPKMsQc YvQLEAqcHe3u9vLOhApjd7Aoz+cs1Ayrc4pxKoYsnB0ddu3xPJOFtbMHK/SwAeSCofJzdoMcrB2d YrStIR+kb/sZ+0S+yK+9tdM2jdCBejrwPOwgoSkle+iKHraT0KfGdGi9HvaWCNbu4FGeNprQu1p7 iHnYdULz6tKhL3q0BYR+Nni//55H90HoaJM98DyaHEKTm9OpMdLNFBXOp59r2pDTxu5v9qM1JDS6 YZ/UzwaU0OhGd+iLHo1uRKOb8qH/8uURg7UH31I86Rh0us6dYiTkE5ENvMw+wiUVgjaXwr7iFBd0 CLze1tfEAGeXDydhJa38fLsjulq7Pb2IHhb9y16qjRoVPSzlXTGJ6FhT2JebYp1GRcea667cxARn F3fFJaIb9WFXFv7zwnvINNiZiucLU0YlwlTwyGDJP95/vOwb/3CCvbuDLstPG5Ia5O0Fj1oB4E3E 5fv9g2cSfjj4jcpPJ04Ti39//fan19+YFJbiRlql/MEk2Rsp/B5nUR8J6ju7AmyabsDCTAo3aggv LbJLFzc838VNzjjmdkjqO3fg0+gjCvLgVoBN0w3Y/tWo+liIP7k7c67IrYvZG9gxnghpfDETl9w+ EpR1ZgXY7mQDFJdSmIMJvBBFSjEX3PKLGY85bgGmGO/MUrj7iEI/mBVg03QDFukv3NwkYilf9jt3 kid7chNuQuIpK+7MtayPlOdXK8Cm6QZsHXlmJS24X1LcOLEbd5TazMZacVeuiX0klPvOrQCbphuw zvZEw4PNeVGQCIoch4ZWaZJnKT2Y3J3cSYnqI8GnC7kGbA++gevkwFF7JDPwjiluHNuNG46PuYtk 3MXtuZD1cRy3hmuKa2DCCnbm3DMjS4m5mPlBuZhrT7WqU82Jj+1jxAXv3AqwPfgGLjpF3zc9GQ/f WJANip4N4ocep8jZhkllG5/AHKMfdU0DtifhAHYR+VlFACwXr0dK3AWg5OabACcph0mlnMsiQEbu rboABdgWxoHskjgDVgCzjIDK9uGuoMhjMxV4oEBBeORdlROQMc7SrgHbwjiQ4QDKUOBN4gCPdLkr QMN0U0CykZg8/vhRwD3IHGOce6AA28I4kPGC2aHAoVYhAJ/n7gqSkE0FUTIJEykFnktdH2McRV4D toVxIHsEDgVW8qYSOO4KcJo3BUnuEaakFEhv1sdZ6zVeWwgHsHczEfF8ct7UCIq7ABzmTUCWo8RU lIDIVa+PMY83VgO2hXEgw9WNRAy5500Cx12Bl9s3FRRJ51gF7FIgWSBJUEYh0HBt4Ru43CsOfqws CMhguPM7KT6Dn9sd8Cf7SMPC5a+P/Jp3BQqwLYwD2ROudVeAveKA+khDK4V3KnCShhJ8UxDE+fUx 2VGINWBbGAeyT7jWXQHyBW2aBZJU66FAtvtS4CUN5XPuCsRj9TG5UQw0YFsYBzKMeohSkQNbZwTA FSkFWWrfVADTAX+NidPxUiD9Tx+TH8VAA7aFcSB7vFlDAcosB+Ar611BkvI/FcB6sAKSdLwUSGvY R66/XYECbAvjQPaok0MBXhoOwOcpBVHev6kgimPGxXZKgTivPiaae6AA28I4ka20flCA/4kcgM9T Ckje/qkgi4vDFBzdFGS2IX1MNKqBBmwL40TGczkUlJ44+DyloL9AU0HpDhY7qRRUvod9TKMcaLy2 EE7gMBPR5543+DolwKlErHLd5Ym7CRBLIkNKcwMUXFv4Ji5Jl8L8fLAIKI80tLO3Yv5se3tnxVde /F7aHRlTnjugANvCOJEjMrcr4ORGQNVpKDbsUuAkDcXn3BX0Leh7MO2wBmwL40ROqNRdAV/wt4Nl VWkoFvRSEHozEXQactszx1RHQdSAbWG8kO1QwLWOI7zYnEtCllZvSoArycQTp+MlIUsXJGO2oxo8 ENvCOaErKmUWDVzxOQIeNt81JJzcZ2GmfhCw+/muoUg3JGN2ox48ENvCCWiL33ARNDTwu8cRyJYv GsbCHHo6ot+5a+DCO8c8nfIDsS2cxF+C3w7ZIRocv/0cgXz5pqEvzF6uJCZSGro7kDEPCQ/AtlAS fwh+e6RHl5BlEx3MbPkmoS/EneA8wOSVBArvOeYw6sIDsS2c5KWXhTQ7NLD94wgkzDcNfWG2ch0w WaUh1vcc87gUD8C2UJKT7EBrAbkigT0wR9SvGTkWpio1IRVJzEtC7j6RRMLcBo3YFk6yPTsSPqtr 4E4AEd5+z8i+MGWpjJiy0iC7IJswH4gHXlsYcUELzC2uShwKAvaHI9z3fOwLU5K3AVO8K2CPNsf8 SUiN2BbOwN+B3wWYXQO3hhzhvydkX5hid2xRHsVLg3fvOeYys0EjtoUz8Jfgd8XOdg3sdfybiyvQ Fwl9XaJuW9HGKQlB6rOMuc5tUIBtYQz8HegFHHIjsQK8KCQByJZVwViXQrfuaGWVgih2UcZiR4HU gG1hDPwZ+O3F77KCguzhAGTLFwV9Hex37dY9KwWpvOdYpnHWgG1hDL73Q3DCQ0FG7nAAcuWLgr4u WXHMyYp/vhQUes+x+CFA4bWFMDhJC/ioWLsAGNLIAYUryyKgr4P75qYBU7y71lTFLcpY/KgJGrAt jMFKWsBK5qEAplwC6rdEHOvQQXLjhCncFfC2zHH6FY3XFkL8XbIiYWe6AGwOB0Dptzzs62D/uXfE pIx7dwlB6OcGKLi28GFjJCd46vzIDgnwX7Owr0O4Zf6oW5dM7Bb7WOIoBhqwLYzsal3h4wxDAUvl gADKVUFfhxMsrIB0+5aTGAQZS5r3QAG2hdHzZ9Se0fy71p40uC9pFfA/tqteB4/cBvZ5in0Cn/5/ ancBco3Lw3WpgqgIrrjXzwxJ7a6khQGuP1uaGUkjirRhMHGJZXr5EVAlDUls9jqueOPgC1xE1zst /DRrxPgMI/dTgI7DNU6xzNv8COjxmrE13YENcByMgavomtdEAS8sJ+ACfynQcUhlIZaZ0W4FzckV 1Ng1G26A42AMXEafuR0KmLM4ATnsS4GOQ/XvoECy+ltB0CzE2J0mow1wHIyBy+h84ILkns6/cALy +JcCHYcnjR0QPi2+FSRRIHFuwYo3DkLf1RbAVwGonTgh8X39EKDj8K5XCqhix0eAVCYau9dksAGO g9E3tQWwVAHfb07w30bUcfiVqaCIHx8FNV8z2qu84Y2D0HMVneVeUwEsYjghfvtQx6HAixSQxY6P gO6vGbvVyhvgOBg9kbvWvKJAmy+PQhYv+qHAxvF/qSCJHW8F3ZXrjsmSwQo4DkbPZXSt/Pm7dfUN V/mlQMeh3hYfhtWHPTATWsx6ChvgOBhRxjrrfkRAM9vgnL8E6Dicac133/MIiO26Y9Et2ADHToi2 tlkHKPzFXAOnu3by67jEluRu/R7+HK87Wo28AY6dEOVrwc9klmhZCn/Pq/7Fr+N4rZH18QkLf5Eb qFE9uOGNnQ+lK/JGZeWSSJ+k6vfIdf6LXscxrZEer2p+00s9ZrFrJtwAx06IFeAX3kilj9KIemT5 +EWv45jXSY+y4qFHB3Bp7ttAxs4hx49qSn8FZGQODrDHzegu/pH3HImTTWZl3kkAA3pDK8jWh03o f6mNVb4NkC1kF2lHJID/Iw2PITapSWMT+0qz9OwfOVkzCC0qFoe68k3LphQqg1fZgvdDihgWWKjB KrNzenatedn7Sdplu6xLvA/NeSYMjckS1wY4VjqDhYOEGwbj0GYHOrmdHIJxJy9OSV5883BHpkuN KXXjXgDHSmewumpYCwPxT/7NXLuc6GQOmq553V+pyiXWTBoTlCnzAjjeZAaKSliY4TAO9Gaiydz0 5I05yhOVojxYD3NhdtCYLEdtgGOlM1i0AcINw3IoC7I3N134cCd5oFOS5/rhrsyMGmlO5V4Ax0pn sN185sQWjdZ/c+PEXtxZXoWU5Y14uDvTgsY0G5YVcKx0BouEIUZD3UVfNNbib/Ikndkkr1qjVXki b3Iv9aHGZJlpAxwbn+F6OXBkT3EGO5E3d5SebHI3KQlSE8c93IFFiUY77hVuLFyGifSizFWdwR7s 1R7g+fIv5q5W66vVUPpcM7JvUO4FcGx8hos3Muimlx8Bz0ljD/qmd9KLGT1OUboTt7qNJzBjDpbX VsCxExowioRUuwjIuCsYD0u8BZQurdgU4MVy+CyW81UESMzRm4AFcByMhoxCqZgCJHxMQCnVFgUN AI+CIE0B7vHqO6kHNeaZ2lfAcTAaMorFZgrCj8IJAXZ5K6jYw0dBko3EJ8RXjxocU5zGnOceLIDj YDRkVOzOFHjkKkzA8vxbQRGyqSCLk/BJi4LAVKcxZ0vyK+A4GA05YKIpcOIbdBVxUYDTfCkoco/w KYsCKQg1zly/4o2D0IDx+psR8XzSN2is8iIAh/kSUOUo8WmLgMyspzFXe2NXwHEwGjI6RzMiyh7x TQHHW0GQ2zcVNLFz7gL2KBAXiAmaJYIVbhx8hhvitCHqrdowoYLhze8l+Rh/cWLD4jYbNqY/jXzN VcECOA5GQw4J11oVYK84oW82dJJ4pwIvNpTJLwXR8RJqLM4S8Qo4DkZDDgXXWhXALx1PBJAkW5sC 2e5HQRAbynLeCqTG0li8JYMVcByMhhyQtbNk5IgrwwmoihYFVXLfVICio2d+aMdHQWRrprEESwYr 4DgYDTngzTIFSLOcgFX2t4Ii6X8qQOlBBUns+ChIvIQamX9VwQI4DkZDDsiTpgAvDSdgeYuCLO/f VJClYsbF9osCqbw0ljT3YAEcB+NEZnMnCvA/mROSNDqPgiRv/1RQpYrDJ6KueBRUliEaS7JssAKO g3Ei47k0BU2Ng+UtCvQFmgqaVrDYyUVB5z3UWCwdrHjjIJzAcRoxVPUNVrcI8IsRu1x3eeJeAqQk kVDK3IAFbhx8EzdJl0J+HiwmtM2GbvZW5K9OGhd55N/8QdodiaXOHVgAx8E4kTOcqwpobkzoqw2l DHsUeLGh1DlvBboFugezHF4Bx8E4kQsytSrgBUcv6lYbSgn6KIjaTMTVhmx7ZizdEuIKOA7GB9mZ AuY6zghS5jwSqrR6UwKqkpr4oR0fCVW6IInVWTbYEMfBOaE7MmUVDcz4nIEatr41FJzcPbAmPQiU +/WtoUk3JLF6ywcb4jg4Ae3wG1VEMg189zgDbvnQYANrVDui33lrYOKdsc5KeUMcB2fiSvDbwx2i wfPt5wz45UuDDqxBriQ+adGg1YHEahI2wHFQJi4EvwPsoRKqbKJHMdu+JOhA3An6AJ+wSEjxmrFG ywsb4jg4U5BeFtKcaWD5xxkwzJcGHVidXAd83KIh92vGapdiAxwHZfLiDrQWkCsSWANzRv90pA0s XXJCaWLMR0LVOjGJhLkNK+I4OJNTdxQsSzWwE8CM4L4dqQNLlcyIT100yC7IJswHYsMbByMuaENx i6uSTUHE/nCG//ajDixF3gZ88lsBa7QZ623IFXEcnJHrwO8GTNXA1pAzwrchdWDJWrFleRQfDcFf M9Y23bAijoMzciX43bGzqoG1TriYXIF+SNBxJWnZijZukRAlP0usfW7DAjgOxsh1oBfw8EahArwo SSbALacCG1eilu5oZRcFWcpFic1ZglwBx8EYuQz8DlLvUkGDezgBbvlQoONQfnct3euioLRrxjYL 5xVwHIwxaD+EStgUVHiHE+CVDwU6rjipmIuT+vlR0NI1YwsmYMEbB2H0YgvUUbmrABSkmRMaM8sh QMeh+mbTgE9+V62lS7UosQXLCSvgOBijE1uglKymAEW5TOhfRrRx6CDZOOET3wq4LTPOemXFGwch /l1cUbAzKgCbwwlQ+uVDHYfyn70jPkvhrlVCFPq5AQvcOPiwMeIJfpQf7pAJ4dOFOg7THfnz2rrU xGpRY8uWDFbAcTCyqvWNxxlNAaVyQgTlqUDH4QQbFaS1fatFCgSJrcx7sACOgzFwGV0dzd+9q2lw X8opwIbBxCWW6eVHQJU0JLHZ67jijYMvcBFd77Tw06wR4zOM3E8BOg7XOMUyb/MjoMdrxtZ0BzbA cTAGrqJrXhMFvLCcgAv8pUDHIZWFWGZGuxU0J1dQY9dsuAGOgzFwGX3mdihgzuIE5LAvBToO1b+D AsnqbwVBsxBjd5qMNsBxMAYuo/OBC5J7Ov/CCcjjXwp0HJ40dkD4tPhWkESBxLkFK944CH1XWwBf BaB24oTE9/VDgI7Du14poIodHwFSmWjsXpPBBjgORt/UFsBSBXy/OcF/G1HH4VemgiJ+fBTUfM1o r/KGNw5Cz1V0lntNBbCI4YT47UMdhwIvUkAWOz4Cur9m7FYrb4DjYPRE7lrzigJtvjwKWbzohwIb x/+lgiR2vBV0V647JksGK+A4GD2X0bXy5+/W1Tdc5ZcCHYd6W3wYVh/2wExoMespbIDjYEQZ66z7 EQHNbINz/hKg43CmNd99zyMgtuuORbdgAxw7IdraZh2g8BdzDZzu2smv4xJbkrv1e/hzvO5oNfIG OHZClK8FP5NZomUp/D2v+he/juO1RtbHJyz8RW6gRvXghjd2PpSuyBuVlUsifZKq3yPX+S96Hce0 Rnq8qvlNL/WYxa6ZcAMcOyFWgF94I5U+SiPqkeXjF72OY14nPcqKhx4dwKW5bwMZO4ccP6op/RWQ kTk4wB4nY2xSRNYsnaF9jPG3n3+16+evy/EPH/tfP3+HjH9Cxn/sDv59eXf96/rjT3f9+x/sktiv VqawBF3/57xscnTHbSg6r1XUEqx/eRlBBllBzx4DdO9/EN5LypbMWdCAv2Y98liWKPJSFzp1qsQU hXn2D16KgcEdeBp4n5/2n59/c2kT3ayupenPZQvFkfyfSztfJN+l2pvfHd7+uY/tSzS+3cPOMaut 9yPRXkOqHgL+zl22HHmzxjXk6SkBpQfWzR6G1r4yLWBu6FUC3ov4cZRAUvuCrWnzkDWJkCt1I1Py 3tVuWnrQm6cE1ExMNy2YyUfQqRurATezb6G7dY+6VfCPpwTU1OWbPRw99FowYGxoq0mp+gsMfXhK QKmNK6PtqzzorImqfy8bulrnm1vz/3hKQA3Mq9V6mKE778BMTMSF5lZAWFpjM/ThKQE1oI5o9wd9 WcDY0Hmp9lc3fjwloAZm3Gq939BN360BUHcvOlUfiqBIbkcfnhJQA9KA9nQ0VDMD7g19cdXF1j4c fXhKQI3Of6ByWuiEgEI/R8+bCipxQ1p+0JunBJTa+Jw62AiARm3SgMqvW+hpU8LYRtXTUQJJS+0w ezgZn8eAuZEp+MZMrsANfXhKQHkVr2gBC500BWZjIi00U8OHu1Qe9OYpAaVtteg+6ohwDUMnrYga 0L35GLq1Z3LFrhj68JSAUsEwzO6OtgKqAWNDcwhgiRpzVZDDUQJJbR9v8kPOQwMGh5hFLlxu4tKf knp4SkC5ymqXD14qQvWlGmCibKE5O3Um4GheQU5PCSgVkDj9lnyuRcBlAX1DJ6y6T9/vRd4cJZBc mjactZFR1jXApOwiX1hu777fRj4cJZBUdnezh5ORkwyYL9kuSq/cjuT14/SUgOo6XWlaNVyrhVZR 86sTQLtf9OSiM5rXuMqD3jwloNQusFELDI3rrwFagTf04KpZP/oqqKenBJROQsPs4WjoQgaMDd1x iJCR+Oh1ioenBFSHOFVb544HnbVw6lhYNnRDarQx/SwXevOUgNIJsha1OwcwoNEgNECzP71o3u/G MtJXQT09JaC6Tgtm9wd9WcDY0JxwWiW6jAe9eUpAqZ1hDw6uQKMZa4AWmg2dcYyNz569gpyeElC9 aarQxlgINCSEBjRd0otOSLuWUFH7tfL68JSA6lWLudpQQQudLKBv6AvrrZPbcs8HvXlKQHV8jtrQ boZOXN2Nr3vQ/caqK0tqm+sYD08JqF5YKdUejsa1ZsDc0FxvbdzxdYqHowRS1+amhbJjSYucEDD0 qzYyZT+3o7WHvDlKIHWke//tkPawb4h59Ufyv2AjUzwh/0A+HSWAemKN0p9u4Ml+dOPCbmBwL16Z YtXj9JPA6YllVV+XFxdl90aNyQ+XMqxM7nHqC7w5SgD5lnCDCB7sFNhBNI6HDNVRBp62xR8/CaCG 6q22VlbdeYIvD+gbmOW5UCRU10sfTwmodnPxmojJ0J0tGZmadzTXzFZevWt9PCWg2uSGqz0M3Xgg CMD5LPTFVWfPaJJPRwkkv+V+50lODMjMqIcMaGG/rT4WfTwloBokidoqpwxdmfYoZelFN6w5symu ZD79JIBa501Uexi48KYiYGzgiTWzilafiT6OEkjeBLwlkJwZUFlsFhnM3KpfFyfvjhJIDXpVbdVS Rs4siWh0aSOzVWVWjeId/OMpAdUqlaHafaEvDxgbuiHvMhtA8ZHo4ykB1Srbi2qGbOjE9gNRUTY0 u2C+8CzdD/H0lIBybeNKR9EXmyakEHroQhesOrFKF+/gH08JqIbpo/8Oq7xEJw/oGzpjvYnVFNfR 0bunBFTLFCCqKhNtFYMQKJCdeUOzwULR4Tpa+fh4SkC1RNmk9jD0pKxCwP0WU9yTVJCAxSeij58E kOt1V+8EJwbcVIYOts6dEnPbJ6KPpwRUQ5qOX505LkMPClgMJWlHg8wSkn0i+jhKIOloNMzuRu5U 3QgYG7kjna/Z7aaTfDpKIPk458MdyXnUX0x/VTvJQ0biXeyJ2Qeij6cEVMXwQfs2tOoeBug929CV i7b7WC2nP54SUHXoZ6ndKMKJvjyg72iuujRWkUXeHSWQ1MakOLS6Glm/piFAS9pGxmW5OMrl5Gd4 OkogVYwetO9EspZdD7hvJ/8Nr1oerzJo6U/SQ/vnr5///P73x+WOFxm++0RJeFfFbKK25tl6d2KA Fuj17n/9XIASuETvpGIqndHvKtQ7/eI/6riceEr6MhyaAD71fwpU2LCXIVHqxXBViPlixozBv/uW oiHrc1Y+XTGfnhJQM/Od1WZURSeoEAbMDY0bd3ekwe0a4+MogTQzE6UmXkQjJ03SgUx8yVThN1P4 7ulBb54SUDMxU2pmfQF6sh+N2zPF0Lxyd+F+uMj4eEpAzYuJUDHDGBoqhAFjQzeuOlV/gaEPTwmo eTFlKiavhc7ajyZK1YvmlZu2464zPp4SUPrOqsuomBcNjRFRA7QZpBfNrZiTa3eh8fGUgNJDmGb3 B31ZwNjQHB8m5ca804PePCWgBlqX2k1/DA0logHoZC+auzxZ9KcLjY+nBNRAw6U9Ha0CNTHg3tDs IbPY2n1cOz0loLwkeIEwdEJAoZ+j543LMhM3xIXGx1MCanR+jta95GgoEQ2o/LqF5gHikmFbnHw4 SiCNxjNQezgZn8eAuZEHFj0mMzAv9OEpATUaM0cbTHnQSVNgNibSQjM1Rs++Kwu9eUpAeSP0tgh0 YrXVxon0X2iOVaN23xVDH54SUKNwYFG7O9oK6Oy8tAtdkXssUWOuCnI4SiCNwiGrXWxeRsYUNgeH skXmwIYpQZ9PST08JaBcJ7lqUvR1c3ack6PkQmcsujMBh0uNj6cElKo7nH7Dti30ZQF9Q7NhU+Ji vxd5c5RAGhhC1cZZGxllXQMwk75kzoK9+34b+XCUQFIV3c0eTh7UaPOmZHOyXZReuR3J68fpKQHl 2t8nAUNDWN4XdeZCc8zsGc1rXOVBb54SUGoX2KgFhu5UwzrhpA09uGrWj74K6ukpAaVz1jB7OLpR w2vA2NAUoRBr+Oh1ioenBJTaGCt0ViwPGnOHjnxlQzekRhvTz3KhN08JKJ9pfcIFunJauguHp4Xm /baBtq+CenpKQHUV72b3B31ZwNjQhaumJEVnWujNUwJKbQyebXAMBbpwMr0rJ6mFzjjGxmfPXkFO TwmopR2hJwwNCaEBKr3Li05IOxsN+7Xy+vCUgOqq/02WtgedLKBv6AvrtQmu3fNBb54SUB2foza0 m6ETV3fj6x50v7HqypLa5jrGw1MCqhdWSrWHo3GtGTA3NNdbG3d8neLhKIHUM+ehjiUtckKA6vy2 kQG2LGntIW+OEkgd6d5/e+IA8OfnhphXfyT/CzYyxRPyD+TTUQKoJ9Yo/ekGnuxHNy7sBgb34pUp Vj1OPwmcnlhW9XV5cVF2b9SY/HApw8rkHqe+wJujBJBvCTeI4MFOgR1E43jIUB1l4Glb/PGTAGqo 3mprZeVMqwGXB/QNzPJcKBKq66WPpwRUu7l4TcRk6M6WjEzNO5prZiuv3rU+nhJQbXLD1R6GbjwQ BOB8FvriqrNnNMmnowSS33K/8yQnBmRm1EMGtLDfVh+LPp4SUA2SRG2VU4auTHuUsvSiG9ac2RRX Mp9+EkCt8yaqPQxceFMRMDbwxJpZRavPRB9HCSRvAt4SSM4MqCw2iwxmbtWvi5N3RwmkBr2qtmop I2eWRDS6tJHZqjKrRvEO/vGUgGqVylDtvtCXB4wN3ZB3mQ2g+Ej08ZT/kV3uupbkNhTN+yvqD6z3 I57MgA0MOpjAcOZJDNDAjAP/vrk3pTrSIRpgX95LrlKpJHLToWphe1HNkAwd2X4gKvKBZhdMATa3 9RHvSHGopW2W0lF0YNOEFEIP3eiMVUdW6bw6+FekOFTF9NGebpWX6LgS2oFOWG9kNcV1XOgzUhyq JgoQVZWRvopBCBTIznSg2WCh6HAdrXx8RYpD1UjZpH439KCsQsL8FFPck5hxAPOaiL7ixIGWXl/q neDIhElluMDWuWPk2V4T0VekOFTFMe2PzhzB0J0CFkNJPNEgs4SkNRF9BYoj6WjUzW9GblTdSOgH ueE4h9HsppN8B4ojrXFuDXckp14eTH9FO8lLxsEL7IlpDURfkeJQBcMH/Wlo1T1M0Ht2oAsXbfex 2Jn+ihSHKl1fS/1KEU50WAntRHPVubKKbPIZKI6kPibFrtXVyPo2FQla0g4yLkvgKJfi+oZ3oDhS wehBf0aSteyuhDkX+Q9ElfxG5U5P/4v60f78/cdvz39+LLmzigyffaPEPatgNlFfz9l+dmSCFuj9 7F9//OWX/47nl5/P/YCfv/z9R3z++sTn3/qbqf3uf3oHnr89//hneP716uNBcZUbH/RZ8M8fgaTV zWKCtIlYGb6wYNsiXhSSrdvKsMSCGw0/psAl6tIzt5H7b1tvFqrS0FekONRI3A31p6Hj5HZpwjzQ PC5mcTaJviPFoQaql/pQ4RuNjtXHOlaG5iE3uyXJV6Q41Ij8Q0msRkAPHt0+17kyNK+mWZQBQ1+R 4lAj8HXUH3mjgyXYsTE064lZXMKNPiLFoUbgRyiZ5xfozioxwjqBRFsVNIuKa+grUhxKn4mjUzBd GhqyRRMiT9JGs3SbLbMt9BUpDtUnq4P67UVHSxj5g2bDMYvettFHpDhURwVTv2qVMHSlCNHPXQ50 /bTKuslXoDhSRwWjPxdZqyQStHkcZHZ2s3ue+YoUh1IfZWJVE0OnzIR2oK1X2qyb54s+IsWhOn5Q 34ok0IjUBCujC20qymwt++xdkeJQHcunP150sIR5oKn9zNY6X/QRKQ7VK5+52hHQ+CqaULmEjaZi NVvbPntXpDhUL1SZq4kCDeGiCY0bt9HU2WZr3zfmihSH6oX1bLV+Q0dLwOfe6L4HhDraCz7ixIF6 pp5fagVguxGjc5jbYI4zZuvca74ixaFUVGEIWRJL0WFyShmDc+dC2wxmtgVrxF+R4lA9cXKqqIYb HS2hHei0p8cW+ws+4sSBerRWaFIW4MFhcIndDeawa7alveYrUhyqo9PQnwvdOcIONKYPunI7aFF2 DH1FikOpH1VUrbHB0KmnZwaK0o3u8dm2lfqij0hxqIaurr4NO0Cj7WuCjUMbPSjgaVtdR++OFIfS oWyaP150sIT5QWMg3ba18qKPSHEo9SFg12AJtP6EhKSv9UGntIUSV2roK1IcqnXK7qorWOhCXT4z J61XXY5n2zb2qq9IcSj1g/m9bnS0hJE/6NqfbduML/qIFIdqjQNOVWG10JkT0Cy6yA+652fbvoTp V6Q4lKpHjGXqz4VOnNum6vQDzd5iFpXO0FekOJT6GCVVmNYXjVlTE9oHbcOp2Z7yiz4ixaGaymb4 U4+/oaNGakLT1/qgrZjS9k2+AsWRmip488dLDpYwD3KxhsitLulFH5HiUOonrbAN38jQEMaaoEPB gW5E0/Z9za9AcaSmw5SewQZpD39qAUb80Jf6gPt4tu3N1nwHigOpH8zvGxwtfhzgyWZIu+TpHSeO oz/gPuqvs3EH76um4fYubov52bZv8BUoDtRwPulbq9Et61oZsYPhIKf5bNuHSY+vSHGoBjWrfmYz IzquBHS7jS712RY9eqHPSHEoPSKo13oWs6Eb6zlOVD7QrTzbjrBWfUeKQ+mVwZarPw1d2YWQMA+0 VTqAo9X/rzhxIPVxRtb0STDOEK5uPcCT20E70tjoM1IcSisI+rmWKpxyRRf2e9QyXIOFxtZti75B 9B0pDqUldZo/NjqshHmg87PMKGODzzhxIPVRQLRyFwNnFhiU9nqAS3u2HdVu91ekOJQ2rwZ/sAYq WuUUEyqL5Ea39Gw72jp4d6Q4lPrB/L7RcSWMAz3is+1YNekrUhxKuzmKk8qGbOjI4oXmXw707M+2 Y6xV35HiUCpv0BfVn4YO7JtQQ+GDHiwcy8616jtSHEr9qH4P7O1Ep5WA5r/RlKVmZ0gbfUaKQ6ne Q/NSYZnoqx5sTFCJcqDLeLad0aTHV6Q4VI28AeqPjQ4rYR5o9hSzM6WNPiPFodSH3lOlXQw9qAch xeuBHkTTzrxWfUeKQ1XTyDp4QLMqWncOCTFQ1G705IbQzmLS4ytSHEr9YH7f6LgSxgeN/X1tzRt9 RopD6eyVdR7RSSwbWpWEtj6MakXF+Ubn8ry2rVXfkeJQOop286ehVVAxQe/a/KDLfF67ZM1XpDiU +hG+xsVi6LgS+oFu9Xnt2OQzUBypaIWFry9n5EIpoJuob/UhD+6E2bnO3h0pDlUaK4P609Cqj7sl zI3+A1Elv1EZGqnjv6ipf/7+47dHBdSDf3zmjRD3DPWR2FXc5/XMxIRC4ctn/vojPPi3pSokVMe4 VBQ4QC7PanJUwehq6+/cH4VBygZqN2wPaPoSBkyR3/ADtISM+avbiqr6JfDwqIxMEF8RS8bxWhse InqMWQwyhr4ixaFG4i4UXAqi4+Q29c532uiM82G2FNMhX5HiUOrjo5bIT2xofPU+eAg2uqLHmMWM tNFHpDjUiPxDSTyYQA/dOk2YPLob3bghtKXtVV+R4lAj8HXUH3mjgyXYcTH0IJq29P6ij0hxKPXx EUrm6QW6syyMwG+y0Hq8nm0xfhn6ihSH0mfi6JTCygU0BklNiDxJGx3RdM2W2Rb6ihSHUj+Y3150 tISRP+hMNC00zUYfkeJQ+n0L/KpNztB6c5Cg/eJAV24Ibd3kK1AcSQ/lMH8uspZFJGR9qw+Z+tds zWvGvSPFodSPVf3GqmPolJnQDvQgehh6vugjUhyq4wf1uxYSQyNSE7Rd1BedAnqu2Vr22bsixaE6 lk9/vOhgCfNAR7QYs7XOF31EikP1ymeWwZsANL6KJlQuYaMztsJsbfvsXZHiUB0npT4F58bQkTV5 NG7cRlf0GLMYdg19RYpD9cJ6pn6fGx0tYfQPGueZpo72go84caCeKeIrbqiB7UYMXOH8ggdqqdk6 95qvSHGonjj81LDmUwgtjJdjcFRZ6BwANdtCXT3rihSH6qh/6kcOT4aOltAONO4gTYv9BR9x4kA9 shlWVHADD86AAyX+A86opGZb2mu+IsWheuBgWtF3DN2pNMbkILvRldtBi7Jj6CtSHEp9yMqKbrnR 0J0TX+uD7uhaZlupL/qIFIdq6OrqF0pjoNH2NSFSPG/0wKEz2+o6enekOJROYtP88aKDJcwPugR2 LdrWyos+IsWh1Idi1ZGvLLT+hASdSw50YteixUoNfUWKQ+nUi3Gp6goWunCempm6fKNZOcy2sVd9 RYpDqR/Mx8hn6GgJGAo3uvZn2zbjiz4ixaFa40xTVVgtdP4/2WWPA0tuA+F8TzEnWOj/J3ZuwHDg EzhbBt77By4WpR72EAv0W75X+pqjlsgih56NydWhZ/3c50zH7r2VElDwj7Mz3gddOKrBYCaHZm+x p1a6a02dUgIKcdZ4cZo0dKlcML5oLUL3OUt90E4pATXg7TXeOP6GzlBiAbx9/6KtmPI5L/kllEAa 8NcWr4ecbMF25GYNkVvdyoN2SgkoxAUVdug3MrQaYyzQmeKLHkTzOe81fwklkEbBjxqfodZe4705 NOx1hgQDz/W5zzks57dQAghxsnhecDb9cuDNZsjnsadvnQQO/kfvI/66GnfxvmKZ3t7DHbl+7nNe 8EsoATT0fDK2VoMtm6iMuoPJkcv+3OdcZj1+lBJQQ90s4spmRnQ+C7TbXXTrn/vUHn3QXikBhSOi 9RpnsRp6sJ7riaoOPdrnPlc6Wb+VElC4MrrliLehO7uQLtgObZVOwdnq/49OAgixnpEzfxKsZ0iv bnfgze3gc5V10V4pAYUKov0cpUpPOdCN/V5rmV6Dg9atu0/tG0S/lRJQKKnb4nXR6SzYDl0/57Ha umCvkwBCrAUElbsZuLLAaGnvDtzG5z5Xt9v9o5SAQvMaGi/WQKBhp7igs0he9Cif+1zjHLy3UgIK cbJ4XnQ+C5ZDr/y5z3Vq0o9SAgrdXIsTbEM1dGbx0ubfHHrPz32udbJ+KyWgYG+0LyLehk7sm+qG 0he9WDjOc5+s30oJKMQZ8Uzs7USXs0Cb/0XTltpzp3LRXikBBb+nzQvGsjCGHxxcAIvi0G197nNn sx4/SgmonnkDEK+LTmfBdmj2FHvuUi7aKyWgEKvfg9Nuhl70g2rFu0Mvovnc9WT9VkpAdfPIGDzU swKNndMFOdHUXvTmhvC5m1mPH6UEFOJk8bzofBasL1r393n2etFeKQGF2atiHsEkVg0NJ4HWp6Na gzm/6No+z3OcrN9KCSiMotPibWgYKi7AXdtfdNuf53lszY9SAgpx1hi63Aydz4Lp0KN/nue6ZC+U QGqosBrjxxm50QpgE/GrvuTFnbDnPmfvrZSAaoOVAfE2NPzxtAX7ov+nqlYfVVWPNPWPjKV///eP /3xgoD76H9/5Rkh4B2JdOGHu63ln4YJG48t3/gs/CHNjOq7n3//4Jx2rOqmpU1MDd+kL2uf0Opph bW7n37lNYKqjTfOaYwf4649/u6z1H7XafOk/q8/P6usZfAsN0tQ6156PsMv3I6z1oJ1SAmrpLMl4 TO/isUBP8/N9eXD8mfwRSiAtLZBN/fGdxTLv3LP3RmYfPsd95MdqO6UE1Cr8nK/5QJsrFmyX9Mjf S9r6g3ZKCailtZfxeLLWIjS1OH/RnVnn5Qrij1ICCnG1V5njVMOobgMLusu65dN2vmX8RykBtWDj 8KqRWIINrVV5Lv79q2qdMp7Gg3ZKCSjE+s6R6LS+Y83crBwXXa4x+XbjH6UEFOJqr7KJcm92qbm5 b68qvno5qZ/5wwklgFbi1/UDk1pR6K0KGDiNYy2do3oJJYBW4okc+Qyqe7Nl46/zN+Nldji/XOBL KAGEuNqL5p2Y9Drjr/s3Y3VQOhvQqLU7MDmdBM5KvPej0KyQq5ZxJVaBy5202uvl4l9CCSBskdYq xD0fcMVAgS+RXcI8D3P6weOlk8BB3Ow98yasDmZpU/1yOxPuZzcO1+kkcBAnfU/laPZMjkv90JdL XzZ5TX4GxyOUAMLF0X9AbLZnq6+HvlD2OEoOBpkT6Smgb6EEEOJmL3oGUp1RoG8u48KMUz3DwQE7 oQTQXNz7oTPPGUeXJTJcxjy+g83kzgQvnQQOaqqeFcRnRkLn1jyqb1I62eqUezf6gJ1QAmhOnu7R rmsf9OKr+hZl12KwTsxSL9gJJYAQ63VE8071DLnLEnEdSg+C+c9Ej3BmXCeUADp+Y/Tj4LfaA+pd f7KJa2QFjzvBvIQSQIi7vWjcjNWAwDC47sQLN9hBxq0TL50Ezhys0YjNUG44y0y9a002ffY1zhck +CWUAIKRZDvRL3HAOv1B7xoTb3Jn1n1crJNJwCBmK8HPuPnqNAI7W7/lZzbm27nNxwC8hRJAiKu9 6Ayflc0eeteT6Fl61axHOWXipZPAoVVHPM8IA33f1LuWpOcW4HKOxeE6nQQOYv2CQ33x4VbNY/iO pPceXJ6KfrFOJgFzhhbE24oPbNKm3vUjKzusmf3swksmAYNYL8hA5egHOy0N144G+2cbdijWBTuh BBDmN73SiPvNl21m+nZk1azRa/ZRL9gJJYAQN3vRvBmzz0yWrAueeiUa+4YetwN2QgkgxFo2B8yR ZZytz0wW2QsezJgWqNezxy+hBNAs/N6IT0PK1mcWu8IFd8240tf3Wy1fQgmgWfg/iOfNmH1mcdkF N/1slR5TjfYBO6EEEGJNfabbkJI1Gp0kvmAW4EpH0c6I+RZKAM1MbzPT7UjJGs3mp7ngwox5jNu6 XKeTwJnZpr10GxJsXaZez9LlsrDXWniY9wU7oQQQYj3P+MP6CITLEhkuYbbQygbdun27t1ACaCbe QMR1H3C3RPY3487jUNlIW8sX7IQSQDOxYiAeN2NtNPihWl8eMIiF7a49+TqZBAxirW+IrY3gE2ub 2YnV8GKn5lvGOHeE4JdQAmhsFuRZTiOBXhvNtpnuASuXVSiVi3UyCZix2T8Q95uvthmcfP3jYm0f WCbqmhfshBJAiPUNiK2P4NprBjsznwesW1tY5etxa2+hBNBY3BFUD2sk0OuOQb9dxlUzzpsH7lTM t1ACCLF+Q8T9ZqzfeOvk4MCacWZBrufWvYUSQGPxzCG2ZoJmoGdyF57QC2YLzfSXtZqZeAslgMbi LUFPsWYCvdo66Ne3/mgxA5hlopbLdToJHLQ9vdaI+01YfeiuLAKXq9RaefH2xTqZBMyYLEOI1zxY 9c27+pbUliaaWYPKnhfshBJAY7JuwrpYJ4HpmJaIa0ncgExzqTea3JdOAgcmSAs9YpvhoNdeuZvv SFoiUbcXwfNynU4CZwz2EcQzH66OUrv5htSGnrTEIlH6umAnlABCzD4yzgwHG7oskfmtPuYjkl29 dk7ESygBBFus3R+xzXDQ60/Z3TckLb2JJ0mLxeU6nQTO6NZHxhnhoFc7A71rSE0PWWLb0FJxuE4n gYOYbWSeEQ5zybI8XD9qBd9Moekk+xJJgIxmLWSe8Q36bkl8e5ECCePhpN/Vo5rhnvLSP7RX/f1f 1eaP/qdanEpsf7npy/15XR3N+KbfF7vnX1zQZvmmxt+zt57Bzdq013rQTikBdX5EXzSDhi61fH+l ofXybJ70PetDdkIJJMTq0buaESOrG+ECN6VuziSbZ3KPux8vpQTUKpwqujbji9Y6hwVuTt28QZtt a58x6kcpAYW426vGk3XRXJafVLVDqaFY5wUX7ZQSUIirvWpvQyuCC9ywunmLNgvVLuWgX0oJqJVt zNSLd9G9c4EbV20rFj3uTuNBO6UEFGIOmmpMLrpqLvDtLmu9S38uuty1+4N2SgkoxNVetbvGW80J 9W5m3RwiFodLTZ129CWUAFrJhk2YyX3AfVPvhlbdAfXh4yR+wE4oAbQST+RQc3LAFYngr93UujgH Lhas1fsFO6EE0Eq8RUNnJIIXr/NKfm5di9tLn7vs0L11Ejgr8d4P9SaHy3k0+cFVu2j6cy6mXdYF O6EEELZIa9VQb3LAnEezH1ztPMz5f9KrZbeNIwje+RV7pADLmcduzy4Q5GDHlyBIgJg3IweaWktM 1JQhUjH8I/neVHfPkLOZwHLgi6iWisXibE9XVz6OzFvhuOFB3dvnpCJY46jP3mm8gwoe8mlk3grH DQ9qsTKS5UR5k6VRr8ZXeHVJSHpN0pijzwLIDREujvyDYomtydJoyNZpxNplyU96HLEQV0BuiFD3 9kE5tiZLoyF7pxEHVezkSicKhbgCckOEWs6e+hJbydJoyOZpxNq+pGaSN/MljhsezFS1vL6kVrI0 GmuTGjUEEpWDzsQVkBsi1IN9EBXBISq+sii7FqRzIuX1Ywnkhgi1XEcactqElY8mpHIoaQS5rdrK PnfbAsgNEZxcBgjqkDKxptFY+1PSYEm66lKemUsgN0SoB/sgKoolj2JhqNxJLxypg1CZEwscNzyo ZUbTkOPm1GtKAr6yJmkwDJqR8hNU4gWQGyLspmonlAMn8JLrgK+MSW+yhlcaqNBWMG5oUKuVUI6b wEsOxYIcL+Mn9apXN13KC8ASyA0R6mgfZHET4UDMHvjKk3RnGXTRpZDHxALHDQ/2QXWSlNMm8JJG ga8sKWkI1BaWtsi8FY4bHtTyBFEPRa+EUUSmypHk3ltiIzcU2grGDQ1qiZqoLWoiK8qSC3zlRzZ2 dGYO+RQWMG5oUMsFQeC0pDkFcxmq7YjUP3uyphgLcQXkhihFvdKoh6JXbSbVdmTTzPLPQLEQV0Bu iFD39kGpKFafSXVolYXX6bS1dsvEFZAbItQyNmnKYXPy5jOpDq1EqlhXoCHmM14AuSFKQZ836mxI 3nxmrFOr3DNZFfTZlWm5AHJDlIL+gjoVxeozYx1b5Z6BWHdMWbQzcQXkhgi1pk1XDMmZ0UiSuBDr AI66UfRTJl4AuSFKXneb5IojOTOaqY6tFFSxtnE/Ft4Kxw1P8hY3XTEkrHVe8VVstcEeY9Bmngpx BeSGCLVGTp99BMDRhFS5ldRCoxp0P+TsugByQ5ScZVev91GJBxNyya6aTUGsRtr3vhBXQG6IktOJ gZqKYjEafFGZL2diMAa1u/6st4JxQ4Na5htqsxE8YrGZyek0LLRJ9AaifEeUeAHkhogmHcgpZCMB Xoxmskx3JhZenUIuFNoKxg0NTeofqIeiV2wGnS8vhdbOQcdEHFMhroDcEKGWT0BtPoJrLwomr3rO xHK0Qad8zNvaEsgNEY16IpgeZiTAy4kBP1WKoyj2kzZcnphLIDdEqOUZoh6KYnnGkySHilgUex3I Md+6JZAbIhq151CbmcAMpCenoB1aiNVCve6XMdoysQRyQ0Sj3hJ4ipkJ8LLWAT9e5o8MMxDrmIih 8FY4bnhge3KtUQ9FsOyhU9QhUHiFNUa9eFOhrWDc0FDSMYR6TJlW9uYp1pbUjyLU6wwKUyrEFZAb Iko6N7G6mJNg6UgmpLIkPQCvy6XcaOVd4LjhwRIkgx61ZTjgxSunvnYkGZGY26MSp8Jb4bjhIVIf QZ185pUoNfW1IfUkneZ0SIRhLMQVkBsi1OojlDMc1tDRhKTL9LE9wtnV63NHLIDcEGEtFvdHbRkO ePkq01Abkoxep50kw6LwVjhueGgwH6Ec4YCXdQb4ypB6aTKntiGjIvNWOG54UKuNpBzhkEtG01H5 UR/wzITUZbELEDck1JuFpBzfgB9MxMWLhLB7+/qXlTan7rvSqh7bkx/lRbzqcV69Xb3arL7bbPrO d5sPeF+UtQvdcl1+cXI/xWt8xCh0eNuGNXJYsi0/RbjrNjv8yzlcxM2n1frV9rjfdTzv7raH/ZGP V5s/RFmeh5St4Rrv6HU46Mvmx/9+I0SmIjJIvBu76/zqOk/YMCgFWWX8hFMSjXoGWden1ffODeMP RkRf+LZnKjQeuEQR25eK8vVErZM/flq9W7+dj8f9w+F4FV+mdbc93HS7h8Nh3p30j79vfvqyajzw l9OIvWyhWs7w/wg+s9SCVaaj8kDQLvJAfn643R9edNun0918OO13WxGquo/z7ulxf/r87EmHHlfS ufCNmgtLqzmcm0gPef36gRkKjy/QD8fj9nbGb6ft8c+j6j5tb5/vjgCtwdO3nnNhaRojnBsjWl/8 +ngzP8433c18v79CMlz/NT+iR2j9+eraS6v8fXUdhoA/nPTfd3N3eOL386OhD7do27uZ5+dbKEax 6fiNj+PM0j4Obfr1Bgof5+NHtPVZ6kXlc+cfJbD0U3v+X62vEDRHPwnNu/Vv8+5Bzrh/GXHGz54a ll0EWGpP7WsFnQnaA/PnZrD+fbPd3XUPHzpcue79v+Zad7dFG3c2F68l5cETMQCNaTgzuV6peH/Y 8/a++/B00BGzvceNBfH21PHT8dS9n2sqLMig0j5Y7/njPZ7U4TTfCOLNZvXPAKJpcsQKZW5kc3Ry ZWFtDWVuZG9iag0xMCAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNDkgMCBSIA0vUmVz b3VyY2VzIDExIDAgUiANL0NvbnRlbnRzIDEyIDAgUiANL1JvdGF0ZSA5MCANL01lZGlhQm94IFsg MCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTExIDAg b2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUNCA4MyAwIFIgL1RU NiA4NyAwIFIgL1RUNyA0MyAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA5NSAwIFIgPj4gDS9D b2xvclNwYWNlIDw8IC9DczggNjggMCBSID4+IA0+PiANZW5kb2JqDTEyIDAgb2JqDTw8IC9MZW5n dGggMTU5NjMgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImUV83OJKkRvPdT9NmS WX6Sv7MPPls++AFWXsvSspI97y85MoGuyqIFHo0E6m/IiChIkshf/vKjvH/98bamWIex+vR27x+/ /vH65a9/d+9//Xi597/fr0CGQnlnSqZijsal95+zq6bS+7//fP32CtXEmm8LfNELiEzJeYPACqJS EK2xYRcS8aeaDphVYSZrQkwbnSmaFOIe0wXGDAMyW5PrLiJHUyMdINMdsjjjHG1UIsKrgGRCPVB4 kggaIdWZGMMhpOiQZLILe1nBqxBnnanV72lC//jyicnGxUMM6RDnTXDuECJfQ+4Tkw1lt/+c6HWM 9ybHA0+UlKP4icmmHkJS0CEhGHu4LClLSP2EFON93X9NjjqGgom27nmKSIvhE1NMSrtSwDGiLeYZ E4Pk0jamxkdMMbXsKgpWWzmd9DmdRMan/b5Vmx4x1ZA/xDjJtvQ5nkwmll0V4ZLTtX32Olec2b5Y cZlSMSUa6/fFiK8axvw5n2oN4g4x5RETTaB9veHk5Jh5Pt5ak/y+5HCm6ZhocjnERMm3ebG9Y5h9 meKkUSHJOLsvU3z+HDLLlPfOhLIvOXz+OiaZSKcY2YHy0RacyXZfc/hcdAzWln0B4T1WMeSNo0OM JAFq8wzJ5rDPveheEdEb2t0CH7PJYV9fWLMCTR614P+LSZ+YYmzc1xdmUDE5GO/KTn0uhuoJNWnU EkyKu2riS8Frv11RA85zt69cuwZfrcbHXVUJFtbNHerB4yuChZvLcaMgONi5racJHu7NHe7640yC x6q8RQ3wb+Fwtx/ZEQIsnTults7CQDB1+XC3H9chEHxe2MWE6Iyth9v/EBLh7PLPXf6QYO3CKSY8 YpJJe/fwrGWQZerh4XyUzJDh7Pzhcj9KM8oyxBxi9AsQCpwd7e728s6ECmN3sCjP5yzUDKtzinEq hiycHR127fE8k4W1swcr9LAB5IKh8nN2gxysHZ1itK0hH6Rv+xn7RL7Ir7210zaN0IF6OvA87CCh KSV76IoetpPQp8Z0aL0e9pYI1u7gUZ42mtC7WnuIedh1QvPq0qEverQFhH42eL//nkf3Qehokz3w PJocQpOb06kx0s0UFc6nn2vakNPG7m/2ozUkNLphn9TPBpTQ6EZ36IsejW5Eo5vyof/y5RGDtQff UjzpGHS6zp1iJOQTkQ28zD7CJRWCNpfCvuIUF3QIvN7W18QAZ5cPJ2Elrfx8uyO6Wrs9vYgeFv3L XqqNGhU9LOVdMYnoWFPYl5tinUZFx5rrrtzEBGcXd8Ulohv1YVcW/vPCe8g02JmK5wtTRiXCVPDI YMk/3n+87Bv/cIK9u4Muy08bkhrk7QWPWgHgTcTl+/2DZxJ+OPiNyk8nThOLf3/99qfX35gUluJG WqX8wSTZGyn8HmdRHwnqO7sCbJpuwMJMCjdqCC8tsksXNzzfxU3OOOZ2SOo7d+DT6CMK8uBWgE3T Ddj+1aj6WIg/uTtzrsiti9kb2DGeCGl8MROX3D4SlHVmBdjuZAMUl1KYgwm8EEVKMRfc8osZjzlu AaYY78xSuPuIQj+YFWDTdAMW6S/c3CRiKV/2O3eSJ3tyE25C4ikr7sy1rI+U51crwKbpBmwdeWYl LbhfUtw4sRt3lNrMxlpxV66JfSSU+86tAJumG7DO9kTDg815UZAIihyHhlZpkmcpPZjcndxJieoj wacLuQZsD76B6+TAUXskM/COKW4c240bjo+5i2Tcxe25kPVxHLeGa4prYMIKdubcMyNLibmY+UG5 mGtPtapTzYmP7WPEBe/cCrA9+AYuOkXfNz0ZD99YkA2Kng3ihx6nyNmGSWUbn8Acox91TQO2J+EA dhH5WUUALBevR0rcBaDk5psAJymHSaWcyyJARu6tugAF2BbGgeySOANWALOMgMr24a6gyGMzFXig QEF45F2VE5AxztKuAdvCOJDhAMpQ4E3iAI90uStAw3RTQLKRmDz++FHAPcgcY5x7oADbwjiQ8YLZ ocChViEAn+fuCpKQTQVRMgkTKQWeS10fYxxFXgO2hXEgewQOBVbyphI47gpwmjcFSe4RpqQUSG/W x1nrNV5bCAewdzMR8Xxy3tQIirsAHOZNQJajxFSUgMhVr48xjzdWA7aFcSDD1Y1EDLnnTQLHXYGX 2zcVFEnnWAXsUiBZIElQRiHQcG3hG7jcKw5+rCwIyGC48zspPoOf2x3wJ/tIw8Llr4/8mncFCrAt jAPZE651V4C94oD6SEMrhXcqcJKGEnxTEMT59THZUYg1YFsYB7JPuNZdAfIFbZoFklTroUC2+1Lg JQ3lc+4KxGP1MblRDDRgWxgHMox6iFKRA1tnBMAVKQVZat9UANMBf42J0/FSIP1PH5MfxUADtoVx IHu8WUMByiwH4CvrXUGS8j8VwHqwApJ0vBRIa9hHrr9dgQJsC+NA9qiTQwFeGg7A5ykFUd6/qSCK Y8bFdkqBOK8+Jpp7oADbwjiRrbR+UID/iRyAz1MKSN7+qSCLi8MUHN0UZLYhfUw0qoEGbAvjRMZz ORSUnjj4PKWgv0BTQekOFjupFFS+h31MoxxovLYQTuAwE9Hnnjf4OiXAqUSsct3libsJEEsiQ0pz AxRcW/gmLkmXwvx8sAgojzS0s7di/mx7e2fFV178XtodGVOeO6AA28I4kSMytyvg5EZA1WkoNuxS 4CQNxefcFfQt6Hsw7bAGbAvjRE6o1F0BX/C3g2VVaSgW9FIQejMRdBpy2zPHVEdB1IBtYbyQ7VDA tY4jvNicS0KWVm9KgCvJxBOn4yUhSxckY7ajGjwQ28I5oSsqZRYNXPE5Ah423zUknNxnYaZ+ELD7 +a6hSDckY3ajHjwQ28IJaIvfcBE0NPC7xxHIli8axsIcejqi37lr4MI7xzyd8gOxLZzEX4LfDtkh Ghy//RyBfPmmoS/MXq4kJlIaujuQMQ8JD8C2UBJ/CH57pEeXkGUTHcxs+SahL8Sd4DzA5JUECu85 5jDqwgOxLZzkpZeFNDs0sP3jCCTMNw19YbZyHTBZpSHW9xzzuBQPwLZQkpPsQGsBuSKBPTBH1K8Z ORamKjUhFUnMS0LuPpFEwtwGjdgWTrI9OxI+q2vgTgAR3n7PyL4wZamMmLLSILsgmzAfiAdeWxhx QQvMLa5KHAoC9ocj3Pd87AtTkrcBU7wrYI82x/xJSI3YFs7A34HfBZhdA7eGHOG/J2RfmGJ3bFEe xUuDd+855jKzQSO2hTPwl+B3xc52Dex1/JuLK9AXCX1dom5b0cYpCUHqs4y5zm1QgG1hDPwd6AUc ciOxArwoJAHIllXBWJdCt+5oZZWCKHZRxmJHgdSAbWEM/Bn47cXvsoKC7OEAZMsXBX0d7Hft1j0r Bam851imcdaAbWEMvvdDcMJDQUbucABy5YuCvi5ZcczJin++FBR6z7H4IUDhtYUwOEkL+KhYuwAY 0sgBhSvLIqCvg/vmpgFTvLvWVMUtylj8qAkasC2MwUpawErmoQCmXALqt0Qc69BBcuOEKdwV8LbM cfoVjdcWQvxdsiJhZ7oAbA4HQOm3POzrYP+5d8SkjHt3CUHo5wYouLbwYWMkJ3jq/MgOCfBfs7Cv Q7hl/qhbl0zsFvtY4igGGrAtjOxqXeHjDEMBS+WAAMpVQV+HEyysgHT7lpMYBBlLmvdAAbaF0fNn 1J7R/LvWnjS4L2kV8D+2q14Hj9wG9nmKfQKf/n9qdwFyjcvDdamCqAiuuNfPDEntrqSFAa4/W5oZ SSOKtGEwcYllevkRUCUNSWz2Oq544+ALXETXOy38NGvE+Awj91OAjsM1TrHM2/wI6PGasTXdgQ1w HIyBq+ia10QBLywn4AJ/KdBxSGUhlpnRbgXNyRXU2DUbboDjYAxcRp+5HQqYszgBOexLgY5D9e+g QLL6W0HQLMTYnSajDXAcjIHL6HzgguSezr9wAvL4lwIdhyeNHRA+Lb4VJFEgcW7BijcOQt/VFsBX AaidOCHxff0QoOPwrlcKqGLHR4BUJhq712SwAY6D0Te1BbBUAd9vTvDfRtRx+JWpoIgfHwU1XzPa q7zhjYPQcxWd5V5TASxiOCF++1DHocCLFJDFjo+A7q8Zu9XKG+A4GD2Ru9a8okCbL49CFi/6ocDG 8X+pIIkdbwXdleuOyZLBCjgORs9ldK38+bt19Q1X+aVAx6HeFh+G1Yc9MBNazHoKG+A4GFHGOut+ REAz2+CcvwToOJxpzXff8wiI7bpj0S3YAMdOiLa2WQco/MVcA6e7dvLruMSW5G79Hv4crztajbwB jp0Q5WvBz2SWaFkKf8+r/sWv43itkfXxCQt/kRuoUT244Y2dD6Ur8kZl5ZJIn6Tq98h1/otexzGt kR6van7TSz1msWsm3ADHTogV4BfeSKWP0oh6ZPn4Ra/jmNdJj7LioUcHcGnu20DGziHHj2pKfwVk ZA4OsMfN6C7+kfcciZNNZmXeSQADekMryNaHTeh/qY1Vvg2QLWQXaUckgP8jDY8hNqlJYxP7SrP0 7B85WTMILSoWh7ryTcumFCqDV9mC90OKGBZYqMEqs3N6dq152ftJ2mW7rEu8D815JgyNyRLXBjhW OoOFg4QbBuPQZgc6uZ0cgnEnL05JXnzzcEemS40pdeNeAMdKZ7C6algLA/FP/s1cu5zoZA6arnnd X6nKJdZMGhOUKfMCON5kBopKWJjhMA70ZqLJ3PTkjTnKE5WiPFgPc2F20JgsR22AY6UzWLQBwg3D cigLsjc3XfhwJ3mgU5Ln+uGuzIwaaU7lXgDHSmew3XzmxBaN1n9z48Re3FlehZTljXi4O9OCxjQb lhVwrHQGi4QhRkPdRV801uJv8iSd2SSvWqNVeSJvci/1ocZkmWkDHBuf4Xo5cGRPcQY7kTd3lJ5s cjcpCVITxz3cgUWJRjvuFW4sXIaJ9KLMVZ3BHuzVHuD58i/mrlbrq9VQ+lwzsm9Q7gVwbHyGizcy 6KaXHwHPSWMP+qZ30osZPU5RuhO3uo0nMGMOltdWwLETGjCKhFS7CMi4KxgPS7wFlC6t2BTgxXL4 LJbzVQRIzNGbgAVwHIyGjEKpmAIkfExAKdUWBQ0Aj4IgTQHu8eo7qQc15pnaV8BxMBoyisVmCsKP wgkBdnkrqNjDR0GSjcQnxFePGhxTnMac5x4sgONgNGRU7M4UeOQqTMDy/FtBEbKpIIuT8EmLgsBU pzFnS/Ir4DgYDTlgoilw4ht0FXFRgNN8KShyj/ApiwIpCDXOXL/ijYPQgPH6mxHxfNI3aKzyIgCH +RJQ5SjxaYuAzKynMVd7Y1fAcTAaMjpHMyLKHvFNAcdbQZDbNxU0sXPuAvYoEBeICZolghVuHHyG G+K0Ieqt2jChguHN7yX5GH9xYsPiNhs2pj+NfM1VwQI4DkZDDgnXWhVgrzihbzZ0kninAi82lMkv BdHxEmoszhLxCjgORkMOBddaFcAvHU8EkCRbmwLZ7kdBEBvKct4KpMbSWLwlgxVwHIyGHJC1s2Tk iCvDCaiKFgVVct9UgKKjZ35ox0dBZGumsQRLBivgOBgNOeDNMgVIs5yAVfa3giLpfypA6UEFSez4 KEi8hBqZf1XBAjgORkMOyJOmAC8NJ2B5i4Is799UkKVixsX2iwKpvDSWNPdgARwH40RmcycK8D+Z E5I0Oo+CJG//VFClisMnoq54FFSWIRpLsmywAo6DcSLjuTQFTY2D5S0K9AWaCppWsNjJRUHnPdRY LB2seOMgnMBxGjFU9Q1WtwjwixG7XHd54l4CpCSRUMrcgAVuHHwTN0mXQn4eLCa0zYZu9lbkr04a F3nk3/xB2h2Jpc4dWADHwTiRM5yrCmhuTOirDaUMexR4saHUOW8FugW6B7McXgHHwTiRCzK1KuAF Ry/qVhtKCfooiNpMxNWGbHtmLN0S4go4DsYH2ZkC5jrOCFLmPBKqtHpTAqqSmvihHR8JVbogidVZ NtgQx8E5oTsyZRUNzPicgRq2vjUUnNw9sCY9CJT79a2hSTcksXrLBxviODgB7fAbVUQyDXz3OANu +dBgA2tUO6LfeWtg4p2xzkp5QxwHZ+JK8NvDHaLB8+3nDPjlS4MOrEGuJD5p0aDVgcRqEjbAcVAm LgS/A+yhEqpsokcx274k6EDcCfoAn7BISPGasUbLCxviODhTkF4W0pxpYPnHGTDMlwYdWJ1cB3zc oiH3a8Zql2IDHAdl8uIOtBaQKxJYA3NG/3SkDSxdckJpYsxHQtU6MYmEuQ0r4jg4k1N3FCxLNbAT wIzgvh2pA0uVzIhPXTTILsgmzAdiwxsHIy5oQ3GLq5JNQcT+cIb/9qMOLEXeBnzyWwFrtBnrbcgV cRyckevA7wZM1cDWkDPCtyF1YMlasWV5FB8NwV8z1jbdsCKOgzNyJfjdsbOqgbVOuJhcgX5I0HEl admKNm6RECU/S6x9bsMCOA7GyHWgF/DwRqECvChJJsAtpwIbV6KW7mhlFwVZykWJzVmCXAHHwRi5 DPwOUu9SQYN7OAFu+VCg41B+dy3d66KgtGvGNgvnFXAcjDFoP4RK2BRUeIcT4JUPBTquOKmYi5P6 +VHQ0jVjCyZgwRsHYfRiC9RRuasAFKSZExozyyFAx6H6ZtOAT35XraVLtSixBcsJK+A4GKMTW6CU rKYARblM6F9GtHHoINk44RPfCrgtM856ZcUbByH+XVxRsDMqAJvDCVD65UMdh/KfvSM+S+GuVUIU +rkBC9w4+LAx4gl+lB/ukAnh04U6DtMd+fPautTEalFjy5YMVsBxMLKq9Y3HGU0BpXJCBOWpQMfh BBsVpLV9q0UKBImtzHuwAI6DMXAZXR3N372raXBfyinAhsHEJZbp5UdAlTQksdnruOKNgy9wEV3v tPDTrBHjM4zcTwE6Dtc4xTJv8yOgx2vG1nQHNsBxMAauomteEwW8sJyAC/ylQMchlYVYZka7FTQn V1Bj12y4AY6DMXAZfeZ2KGDO4gTksC8FOg7Vv4MCyepvBUGzEGN3mow2wHEwBi6j84ELkns6/8IJ yONfCnQcnjR2QPi0+FaQRIHEuQUr3jgIfVdbAF8FoHbihMT39UOAjsO7Ximgih0fAVKZaOxek8EG OA5G39QWwFIFfL85wX8bUcfhV6aCIn58FNR8zWiv8oY3DkLPVXSWe00FsIjhhPjtQx2HAi9SQBY7 PgK6v2bsVitvgONg9ETuWvOKAm2+PApZvOiHAhvH/6WCJHa8FXRXrjsmSwYr4DgYPZfRtfLn79bV N1zllwIdh3pbfBhWH/bATGgx6ylsgONgRBnrrPsRAc1sg3P+EqDjcKY1333PIyC2645Ft2ADHDsh 2tpmHaDwF3MNnO7aya/jEluSu/V7+HO87mg18gY4dkKUrwU/k1miZSn8Pa/6F7+O47VG1scnLPxF bqBG9eCGN3Y+lK7IG5WVSyJ9kqrfI9f5L3odx7RGeryq+U0v9ZjFrplwAxw7IVaAX3gjlT5KI+qR 5eMXvY5jXic9yoqHHh3ApblvAxk7hxw/qin9FZCROTjAHidjbFJE1iydoX2M8beff7Xr56/L8Q8f +18/f4eMf0LGf+wO/n15d/3r+uNPd/37H+yS2K9WprAEXf/nvGxydMdtKDqvVdQSrH95GUEGWUHP HgN0738Q3kvKlsxZ0IC/Zj3yWJYo8lIXOnWqxBSFefYPXoqBwR14Gnifn/afn39zaRPdrK6l6c9l C8WR/J9LO18k36Xam98d3v65j+1LNL7dw84xq633I9FeQ6oeAv7OXbYcebPGNeTpKQGlB9bNHobW vjItYG7oVQLei/hxlEBS+4KtafOQNYmQK3UjU/Le1W5aetCbpwTUTEw3LZjJR9CpG6sBN7Nvobt1 j7pV8I+nBNTU5Zs9HD30WjBgbGirSan6Cwx9eEpAqY0ro+2rPOisiap/Lxu6WuebW/P/eEpADcyr 1XqYoTvvwExMxIXmVkBYWmMz9OEpATWgjmj3B31ZwNjQean2Vzd+PCWgBmbcar3f0E3frQFQdy86 VR+KoEhuRx+eElAD0oD2dDRUMwPuDX1x1cXWPhx9eEpAjc5/oHJa6ISAQj9Hz5sKKnFDWn7Qm6cE lNr4nDrYCIBGbdKAyq9b6GlTwthG1dNRAklL7TB7OBmfx4C5kSn4xkyuwA19eEpAeRWvaAELnTQF ZmMiLTRTw4e7VB705ikBpW216D7qiHANQyetiBrQvfkYurVncsWuGPrwlIBSwTDM7o62AqoBY0Nz CGCJGnNVkMNRAkltH2/yQ85DAwaHmEUuXG7i0p+SenhKQLnKapcPXipC9aUaYKJsoTk7dSbgaF5B Tk8JKBWQOP2WfK5FwGUBfUMnrLpP3+9F3hwlkFyaNpy1kVHWNcCk7CJfWG7vvt9GPhwlkFR2d7OH k5GTDJgv2S5Kr9yO5PXj9JSA6jpdaVo1XKuFVlHzqxNAu1/05KIzmte4yoPePCWg1C6wUQsMjeuv AVqBN/Tgqlk/+iqop6cElE5Cw+zhaOhCBowN3XGIkJH46HWKh6cEVIc4VVvnjgedtXDqWFg2dENq tDH9LBd685SA0gmyFrU7BzCg0SA0QLM/vWje78Yy0ldBPT0loLpOC2b3B31ZwNjQnHBaJbqMB715 SkCpnWEPDq5AoxlrgBaaDZ1xjI3Pnr2CnJ4SUL1pqtDGWAg0JIQGNF3Si05Iu5ZQUfu18vrwlIDq VYu52lBBC50soG/oC+utk9tyzwe9eUpAdXyO2tBuhk5c3Y2ve9D9xqorS2qb6xgPTwmoXlgp1R6O xrVmwNzQXG9t3PF1ioejBFLX5qaFsmNJi5wQMPSrNjJlP7ejtYe8OUogdaR7/+2Q9rBviHn1R/K/ YCNTPCH/QD4dJYB6Yo3Sn27gyX5048JuYHAvXpli1eP0k8DpiWVVX5cXF2X3Ro3JD5cyrEzuceoL vDlKAPmWcIMIHuwU2EE0jocM1VEGnrbFHz8JoIbqrbZWVt15gi8P6BuY5blQJFTXSx9PCah2c/Ga iMnQnS0ZmZp3NNfMVl69a308JaDa5IarPQzdeCAIwPks9MVVZ89okk9HCSS/5X7nSU4MyMyohwxo Yb+tPhZ9PCWgGiSJ2iqnDF2Z9ihl6UU3rDmzKa5kPv0kgFrnTVR7GLjwpiJgbOCJNbOKVp+JPo4S SN4EvCWQnBlQWWwWGczcql8XJ++OEkgNelVt1VJGziyJaHRpI7NVZVaN4h384ykB1SqVodp9oS8P GBu6Ie8yG0DxkejjKQHVKtuLaoZs6MT2A1FRNjS7YL7wLN0P8fSUgHJt40pH0RebJqQQeuhCF6w6 sUoX7+AfTwmohumj/w6rvEQnD+gbOmO9idUU19HRu6cEVMsUIKoqE20VgxAokJ15Q7PBQtHhOlr5 +HhKQLVE2aT2MPSkrELA/RZT3JNUkIDFJ6KPnwSQ63VX7wQnBtxUhg62zp0Sc9snoo+nBFRDmo5f nTkuQw8KWAwlaUeDzBKSfSL6OEog6Wg0zO5G7lTdCBgbuSOdr9ntppN8Okog+Tjnwx3JedRfTH9V O8lDRuJd7InZB6KPpwRUxfBB+za06h4G6D3b0JWLtvtYLac/nhJQdehnqd0owom+PKDvaK66NFaR Rd4dJZDUxqQ4tLoaWb+mIUBL2kbGZbk4yuXkZ3g6SiBVjB6070Syll0PuG8n/w2vWh6vMmjpT9JD ++evn//8/vfH5Y4XGb77REl4V8Vsorbm2Xp3YoAW6PXuf/1cgBK4RO+kYiqd0e8q1Dv94j/quJx4 SvoyHJoAPvV/ClTYsJchUerFcFWI+WLGjMG/+5aiIetzVj5dMZ+eElAz853VZlRFJ6gQBswNjRt3 d6TB7Rrj4yiBNDMTpSZeRCMnTdKBTHzJVOE3U/ju6UFvnhJQMzFTamZ9AXqyH43bM8XQvHJ34X64 yPh4SkDNi4lQMcMYGiqEAWNDN646VX+BoQ9PCah5MWUqJq+FztqPJkrVi+aVm7bjrjM+nhJQ+s6q y6iYFw2NEVEDtBmkF82tmJNrd6Hx8ZSA0kOYZvcHfVnA2NAcHyblxrzTg948JaAGWpfaTX8MDSWi AehkL5q7PFn0pwuNj6cE1EDDpT0drQI1MeDe0Owhs9jafVw7PSWgvCR4gTB0QkChn6PnjcsyEzfE hcbHUwJqdH6O1r3kaCgRDaj8uoXmAeKSYVucfDhKII3GM1B7OBmfx4C5kQcWPSYzMC/04SkBNRoz RxtMedBJU2A2JtJCMzVGz74rC715SkB5I/S2CHRitdXGifRfaI5Vo3bfFUMfnhJQo3BgUbs72gro 7Ly0C12ReyxRY64KcjhKII3CIatdbF5GxhQ2B4eyRebAhilBn09JPTwloFwnuWpS9HVzdpyTo+RC Zyy6MwGHS42PpwSUqjucfsO2LfRlAX1Ds2FT4mK/F3lzlEAaGELVxlkbGWVdAzCTvmTOgr37fhv5 cJRAUhXdzR5OHtRo86Zkc7JdlF65Hcnrx+kpAeXa3ycBQ0NY3hd15kJzzOwZzWtc5UFvnhJQahfY qAWG7lTDOuGkDT24ataPvgrq6SkBpXPWMHs4ulHDa8DY0BShEGv46HWKh6cElNoYK3RWLA8ac4eO fGVDN6RGG9PPcqE3Twkon2l9wgW6clq6C4enheb9toG2r4J6ekpAdRXvZvcHfVnA2NCFq6YkRWda 6M1TAkptDJ5tcAwFunAyvSsnqYXOOMbGZ89eQU5PCailHaEnDA0JoQEqvcuLTkg7Gw37tfL68JSA 6qr/TZa2B50soG/oC+u1Ca7d80FvnhJQHZ+jNrSboRNXd+PrHnS/serKktrmOsbDUwKqF1ZKtYej ca0ZMDc011sbd3yd4uEogdQz56GOJS1yQoDq/LaRAbYsae0hb44SSB3p3n974gDw5+eGmFd/JP8L NjLFE/IP5NNRAqgn1ij96Qae7Ec3LuwGBvfilSlWPU4/CZyeWFb1dXlxUXZv1Jj8cCnDyuQep77A m6MEkG8JN4jgwU6BHUTjeMhQHWXgaVv88ZMAaqjeamtl5UyrAZcH9A3M8lwoEqrrpY+nBFS7uXhN xGTozpaMTM07mmtmK6/etT6eElBtcsPVHoZuPBAE4HwW+uKqs2c0yaejBJLfcr/zJCcGZGbUQwa0 sN9WH4s+nhJQDZJEbZVThq5Me5Sy9KIb1pzZFFcyn34SQK3zJqo9DFx4UxEwNvDEmllFq89EH0cJ JG8C3hJIzgyoLDaLDGZu1a+Lk3dHCaQGvaq2aikjZ5ZENLq0kdmqMqtG8Q7+8ZSAapXKUO2+0JcH jA3dkHeZDaD4SPTxlP+RXe66luQ2FM37K+oPrPcjnsyADQw6mMBw5kkM0MCMA/++uTelOtIhGmBf 3kuuUqkkctOhamF7Uc2QDB3ZfiAq8oFmF0wBNrf1Ee9IcailbZbSUXRg04QUQg/d6IxVR1bpvDr4 V6Q4VMX00Z5ulZfouBLagU5Yb2Q1xXVc6DNSHKomChBVlZG+ikEIFMjOdKDZYKHocB2tfHxFikPV SNmkfjf0oKxCwvwUU9yTmHEA85qIvuLEgZZeX+qd4MiESWW4wNa5Y+TZXhPRV6Q4VMUx7Y/OHMHQ nQIWQ0k80SCzhKQ1EX0FiiPpaNTNb0ZuVN1I6Ae54TiH0eymk3wHiiOtcW4NdySnXh5Mf0U7yUvG wQvsiWkNRF+R4lAFwwf9aWjVPUzQe3agCxdt97HYmf6KFIcqXV9L/UoRTnRYCe1Ec9W5sops8hko jqQ+JsWu1dXI+jYVCVrSDjIuS+Aol+L6hnegOFLB6EF/RpK17K6EORf5D0SV/EblTk//i/rR/vz9 x2/Pf34subOKDJ99o8Q9q2A2UV/P2X52ZIIW6P3sX3/85Zf/jueXn8/9gJ+//P1HfP76xOff+pup /e5/egeevz3/+Gd4/vXq40FxlRsf9Fnwzx+BpNXNYoK0iVgZvrBg2yJeFJKt28qwxIIbDT+mwCXq 0jO3kftvW28WqtLQV6Q41EjcDfWnoePkdmnCPNA8LmZxNom+I8WhBqqX+lDhG42O1cc6VobmITe7 JclXpDjUiPxDSaxGQA8e3T7XuTI0r6ZZlAFDX5HiUCPwddQfeaODJdixMTTriVlcwo0+IsWhRuBH KJnnF+jOKjHCOoFEWxU0i4pr6CtSHEqfiaNTMF0aGrJFEyJP0kazdJstsy30FSkO1Serg/rtRUdL GPmDZsMxi9620UekOFRHBVO/apUwdKUI0c9dDnT9tMq6yVegOFJHBaM/F1mrJBK0eRxkdnaze575 ihSHUh9lYlUTQ6fMhHagrVfarJvniz4ixaE6flDfiiTQiNQEK6MLbSrKbC377F2R4lAdy6c/XnSw hHmgqf3M1jpf9BEpDtUrn7naEdD4KppQuYSNpmI1W9s+e1ekOFQvVJmriQIN4aIJjRu30dTZZmvf N+aKFIfqhfVstX5DR0vA597ovgeEOtoLPuLEgXqmnl9qBWC7EaNzmNtgjjNm69xrviLFoVRUYQhZ EkvRYXJKGYNz50LbDGa2BWvEX5HiUD1xcqqohhsdLaEd6LSnxxb7Cz7ixIF6tFZoUhbgwWFwid0N 5rBrtqW95itSHKqj09CfC905wg40pg+6cjtoUXYMfUWKQ6kfVVStscHQqadnBorSje7x2baV+qKP SHGohq6uvg07QKPta4KNQxs9KOBpW11H744Uh9KhbJo/XnSwhPlBYyDdtrXyoo9IcSj1IWDXYAm0 /oSEpK/1Qae0hRJXaugrUhyqdcruqitY6EJdPjMnrVddjmfbNvaqr0hxKPWD+b1udLSEkT/o2p9t 24wv+ogUh2qNA05VYbXQmRPQLLrID7rnZ9u+hOlXpDiUqkeMZerPhU6c26bq9APN3mIWlc7QV6Q4 lPoYJVWY1heNWVMT2gdtw6nZnvKLPiLFoZrKZvhTj7+ho0ZqQtPX+qCtmNL2Tb4CxZGaKnjzx0sO ljAPcrGGyK0u6UUfkeJQ6ietsA3fyNAQxpqgQ8GBbkTT9n3Nr0BxpKbDlJ7BBmkPf2oBRvzQl/qA +3i27c3WfAeKA6kfzO8bHC1+HODJZki75OkdJ46jP+A+6q+zcQfvq6bh9i5ui/nZtm/wFSgO1HA+ 6Vur0S3rWhmxg+Egp/ls24dJj69IcagGNat+ZjMjOq4EdLuNLvXZFj16oc9IcSg9IqjXehazoRvr OU5UPtCtPNuOsFZ9R4pD6ZXBlqs/DV3ZhZAwD7RVOoCj1f+vOHEg9XFG1vRJMM4Qrm49wJPbQTvS 2OgzUhxKKwj6uZYqnHJFF/Z71DJcg4XG1m2LvkH0HSkOpSV1mj82OqyEeaDzs8woY4PPOHEg9VFA tHIXA2cWGJT2eoBLe7Yd1W73V6Q4lDavBn+wBipa5RQTKovkRrf0bDvaOnh3pDiU+sH8vtFxJYwD PeKz7Vg16StSHEq7OYqTyoZs6MjiheZfDvTsz7ZjrFXfkeJQKm/QF9Wfhg7sm1BD4YMeLBzLzrXq O1IcSv2ofg/s7USnlYDmv9GUpWZnSBt9RopDqd5D81JhmeirHmxMUIlyoMt4tp3RpMdXpDhUjbwB 6o+NDithHmj2FLMzpY0+I8Wh1IfeU6VdDD2oByHF64EeRNPOvFZ9R4pDVdPIOnhAsypadw4JMVDU bvTkhtDOYtLjK1IcSv1gft/ouBLGB439fW3NG31GikPp7JV1HtFJLBtalYS2PoxqRcX5RufyvLat Vd+R4lA6inbzp6FVUDFB79r8oMt8XrtkzVekOJT6Eb7GxWLouBL6gW71ee3Y5DNQHKlohYWvL2fk Qimgm6hv9SEP7oTZuc7eHSkOVRorg/rT0KqPuyXMjf4DUSW/URkaqeO/qKl//v7jt0cF1IN/fOaN EPcM9ZHYVdzn9czEhELhy2f++iM8+LelKiRUx7hUFDhALs9qclTB6Grr79wfhUHKBmo3bA9o+hIG TJHf8AO0hIz5q9uKqvol8PCojEwQXxFLxvFaGx4ieoxZDDKGviLFoUbiLhRcCqLj5Db1znfa6Izz YbYU0yFfkeJQ6uOjlshPbGh89T54CDa6oseYxYy00UekONSI/ENJPJhAD906TZg8uhvduCG0pe1V X5HiUCPwddQfeaODJdhxMfQgmrb0/qKPSHEo9fERSubpBbqzLIzAb7LQeryebTF+GfqKFIfSZ+Lo lMLKBTQGSU2IPEkbHdF0zZbZFvqKFIdSP5jfXnS0hJE/6Ew0LTTNRh+R4lD6fQv8qk3O0HpzkKD9 4kBXbght3eQrUBxJD+Uwfy6ylkUkZH2rD5n612zNa8a9I8Wh1I9V/caqY+iUmdAO9CB6GHq+6CNS HKrjB/W7FhJDI1ITtF3UF50Ceq7ZWvbZuyLFoTqWT3+86GAJ80BHtBiztc4XfUSKQ/XKZ5bBmwA0 voomVC5hozO2wmxt++xdkeJQHSelPgXnxtCRNXk0btxGV/QYsxh2DX1FikP1wnqmfp8bHS1h9A8a 55mmjvaCjzhxoJ4p4ituqIHtRgxc4fyCB2qp2Tr3mq9IcaieOPzUsOZTCC2Ml2NwVFnoHAA120Jd PeuKFIfqqH/qRw5Pho6W0A407iBNi/0FH3HiQD2yGVZUcAMPzoADJf4DzqikZlvaa74ixaF64GBa 0XcM3ak0xuQgu9GV20GLsmPoK1IcSn3IyopuudHQnRNf64Pu6FpmW6kv+ogUh2ro6uoXSmOg0fY1 IVI8b/TAoTPb6jp6d6Q4lE5i0/zxooMlzA+6BHYt2tbKiz4ixaHUh2LVka8stP6EBJ1LDnRi16LF Sg19RYpD6dSLcanqCha6cJ6ambp8o1k5zLaxV31FikOpH8zHyGfoaAkYCje69mfbNuOLPiLFoVrj TFNVWC10/j/ZZY8DS24D4XxPMSdY6P8ndm7AcOATOFsG3vsHLhalHvYQC/Rbvlf6mqOWyCKHno3J 1aFn/dznTMfuvZUSUPCPszPeB104qsFgJodmb7GnVrprTZ1SAgpx1nhxmjR0qVwwvmgtQvc5S33Q TikBNeDtNd44/obOUGIBvH3/oq2Y8jkv+SWUQBrw1xavh5xswXbkZg2RW93Kg3ZKCSjEBRV26Dcy tBpjLNCZ4oseRPM57zV/CSWQRsGPGp+h1l7jvTk07HWGBAPP9bnPOSznt1ACCHGyeF5wNv1y4M1m yOexp2+dBA7+R+8j/road/G+Ypne3sMduX7uc17wSygBNPR8MrZWgy2bqIy6g8mRy/7c51xmPX6U ElBD3SziymZGdD4LtNtddOuf+9QefdBeKQGFI6L1GmexGnqwnuuJqg492uc+VzpZv5USULgyuuWI t6E7u5Au2A5tlU7B2er/j04CCLGekTN/EqxnSK9ud+DN7eBzlXXRXikBhQqi/RylSk850I39XmuZ XoOD1q27T+0bRL+VElAoqdviddHpLNgOXT/nsdq6YK+TAEKsBQSVuxm4ssBoae8O3MbnPle32/2j lIBC8xoaL9ZAoGGnuKCzSF70KJ/7XOMcvLdSAgpxsnhedD4LlkOv/LnPdWrSj1ICCt1cixNsQzV0 ZvHS5t8ces/Pfa51sn4rJaBgb7QvIt6GTuyb6obSF71YOM5zn6zfSgkoxBnxTOztRJezQJv/RdOW 2nOnctFeKQEFv6fNC8ayMIYfHFwAi+LQbX3uc2ezHj9KCaieeQMQr4tOZ8F2aPYUe+5SLtorJaAQ q9+D026GXvSDasW7Qy+i+dz1ZP1WSkB188gYPNSzAo2d0wU50dRe9OaG8LmbWY8fpQQU4mTxvOh8 FqwvWvf3efZ60V4pAYXZq2IewSRWDQ0ngdano1qDOb/o2j7Pc5ys30oJKIyi0+JtaBgqLsBd2190 25/neWzNj1ICCnHWGLrcDJ3PgunQo3+e57pkL5RAaqiwGuPHGbnRCmAT8au+5MWdsOc+Z++tlIBq g5UB8TY0/PG0Bfui/6eqVh9VVY809Y+MpX//94//fGCgPvof3/lGSHgHYl04Ye7reWfhgkbjy3f+ Cz8Ic2M6ruff//gnHas6qalTUwN36Qva5/Q6mmFtbuffuU1gqqNN85pjB/jrj3+7rPUftdp86T+r z8/q6xl8Cw3S1DrXno+wy/cjrPWgnVICauksyXhM7+KxQE/z8315cPyZ/BFKIC0tkE398Z3FMu/c s/dGZh8+x33kx2o7pQTUKvycr/lAmysWbJf0yN9L2vqDdkoJqKW1l/F4stYiNLU4f9GdWeflCuKP UgIKcbVXmeNUw6huAwu6y7rl03a+ZfxHKQG1YOPwqpFYgg2tVXku/v2rap0ynsaDdkoJKMT6zpHo tL5jzdysHBddrjH5duMfpQQU4mqvsolyb3apublvryq+ejmpn/nDCSWAVuLX9QOTWlHorQoYOI1j LZ2jegklgFbiiRz5DKp7s2Xjr/M342V2OL9c4EsoAYS42ovmnZj0OuOv+zdjdVA6G9CotTswOZ0E zkq896PQrJCrlnElVoHLnbTa6+XiX0IJIGyR1irEPR9wxUCBL5FdwjwPc/rB46WTwEHc7D3zJqwO ZmlT/XI7E+5nNw7X6SRwECd9T+Vo9kyOS/3Ql0tfNnlNfgbHI5QAwsXRf0Bstmerr4e+UPY4Sg4G mRPpKaBvoQQQ4mYvegZSnVGgby7jwoxTPcPBATuhBNBc3PuhM88ZR5clMlzGPL6DzeTOBC+dBA5q qp4VxGdGQufWPKpvUjrZ6pR7N/qAnVACaE6e7tGuax/04qv6FmXXYrBOzFIv2AklgBDrdUTzTvUM ucsScR1KD4L5z0SPcGZcJ5QAOn5j9OPgt9oD6l1/solrZAWPO8G8hBJAiLu9aNyM1YDAMLjuxAs3 2EHGrRMvnQTOHKzRiM1QbjjLTL1rTTZ99jXOFyT4JZQAgpFkO9EvccA6/UHvGhNvcmfWfVysk0nA IGYrwc+4+eo0Ajtbv+VnNubbuc3HALyFEkCIq73oDJ+VzR5615PoWXrVrEc5ZeKlk8ChVUc8zwgD fd/Uu5ak5xbgco7F4TqdBA5i/YJDffHhVs1j+I6k9x5cnop+sU4mAXOGFsTbig9s0qbe9SMrO6yZ /ezCSyYBg1gvyEDl6Ac7LQ3Xjgb7Zxt2KNYFO6EEEOY3vdKI+82XbWb6dmTVrNFr9lEv2AklgBA3 e9G8GbPPTJasC556JRr7hh63A3ZCCSDEWjYHzJFlnK3PTBbZCx7MmBao17PHL6EE0Cz83ohPQ8rW Zxa7wgV3zbjS1/dbLV9CCaBZ+D+I582YfWZx2QU3/WyVHlON9gE7oQQQYk19ptuQkjUanSS+YBbg SkfRzoj5FkoAzUxvM9PtSMkazeanueDCjHmM27pcp5PAmdmmvXQbEmxdpl7P0uWysNdaeJj3BTuh BBBiPc/4w/oIhMsSGS5httDKBt26fbu3UAJoJt5AxHUfcLdE9jfjzuNQ2UhbyxfshBJAM7FiIB43 Y200+KFaXx4wiIXtrj35OpkEDGKtb4itjeATa5vZidXwYqfmW8Y4d4Tgl1ACaGwW5FlOI4FeG822 me4BK5dVKJWLdTIJmLHZPxD3m6+2GZx8/eNibR9YJuqaF+yEEkCI9Q2IrY/g2msGOzOfB6xbW1jl 63Frb6EE0FjcEVQPayTQ645Bv13GVTPOmwfuVMy3UAIIsX5DxP1mrN946+TgwJpxZkGu59a9hRJA Y/HMIbZmgmagZ3IXntALZgvN9Je1mpl4CyWAxuItQU+xZgK92jro17f+aDEDmGWilst1OgkctD29 1oj7TVh96K4sAper1Fp58fbFOpkEzJgsQ4jXPFj1zbv6ltSWJppZg8qeF+yEEkBjsm7CulgngemY lohrSdyATHOpN5rcl04CByZICz1im+Gg1165m+9IWiJRtxfB83KdTgJnDPYRxDMfro5Su/mG1Iae tMQiUfq6YCeUAELMPjLODAcbuiyR+a0+5iOSXb12TsRLKAEEW6zdH7HNcNDrT9ndNyQtvYknSYvF 5TqdBM7o1kfGGeGgVzsDvWtITQ9ZYtvQUnG4TieBg5htZJ4RDnPJsjxcP2oF30yh6ST7EkmAjGYt ZJ7xDfpuSXx7kQIJ4+Gk39WjmuGe8tI/tFf9/V/V5o/+p1qcSmx/uenL/XldHc34pt8Xu+dfXNBm +abG37O3nsHN2rTXetBOKQF1fkRfNIOGLrV8f6Wh9fJsnvQ960N2QgkkxOrRu5oRI6sb4QI3pW7O JJtnco+7Hy+lBNQqnCq6NuOL1jqHBW5O3bxBm21rnzHqRykBhbjbq8aTddFclp9UtUOpoVjnBRft lBJQiKu9am9DK4IL3LC6eYs2C9Uu5aBfSgmolW3M1It30b1zgRtXbSsWPe5O40E7pQQUYg6aakwu umou8O0ua71Lfy663LX7g3ZKCSjE1V61u8ZbzQn1bmbdHCIWh0tNnXb0JZQAWsmGTZjJfcB9U++G Vt0B9eHjJH7ATigBtBJP5FBzcsAVieCv3dS6OAcuFqzV+wU7oQTQSrxFQ2ckghev80p+bl2L20uf u+zQvXUSOCvx3g/1JofLeTT5wVW7aPpzLqZd1gU7oQQQtkhr1VBvcsCcR7MfXO08zPl/0qtgx5Ea iN7zFX3sIIVt2227LQESLBxAAiFtbiyHnpme3bBxMiQ9s+yP8L28qrITN14tg/Yy6cq8vC7b5Xr1 0nYk3gIXKx7EvbzH54TZjqqkncJrOWGbdiPxFrhY8SAmKXM0nDCvFzeqWPgyLw8Jnq+JH5L1WQBj RYSLQ/9wJttWL25UJ+kUYq4yrwJvh8nEBTBWRIh7eVGyrV7cqE7aKcSaM+7oSnunM3EBjBURYtp7 12fb6sSN6iSeQszl61hM0mS+xMWKBz2VJa/PrtWJGzWlSA1sAp3LG52IC2CsiBBbeZHLCWvD+EKi 5Fo47hM+jR9LYKyIENN1dDa5TUj5IIkUCkWFQLeVS1mlalsAY0UEJacGglj7RMxu1JT65NlYOh51 XeqZS2CsiBBbeZHLGZMfxcBQqBNfOMcK4nKfWOBixYOYerSzyW6Gnl0S8IU0UYGh0QwunSATL4Cx IsJsynLikuEEnnwd8IUw8U1m8+qsy7QFLFY0iFlKXLKbwJMPxYBsru3H95wvT7ouDQBLYKyIEBt5 kdhNmAMSe+ALTeKZxfKg63RqEwtcrHgwD7KS+OQ2gSc3CnwhSZ5NIJcwlUXiLXCx4kFMJ4jY5nzJ jMIyFYpE914cm+tspi1gsaJBTFYTsVhNeEUacoEv9EjaDvdMm3ZhAYsVDWK6IDCc4jSDFpVxpRw5 1s/eSVEMmbgAxorIG77SiG3Ol2XGl3Ik3Uz8j3UmExfAWBEh7uVFPmfMOuNL00oDb8fdVsotERfA WBEhprbpQjKbQYnO+NK0OscZ8whkTdrjBTBWRF7zeSNOgqREZ4bStdI9o1GBzy53ywUwVkRe8wNi nzNmnRlK20r3DMQ8Y9KgnYgLYKyIELPb7LIgdSI05CSuxNyADU8UfUjEC2CsiLzi2cZ3WZE6EZpQ 2lanOWMu437IvAUuVjxeid3ssiBhrFOML2yrNHZjNBdzyMQFMFZEiNlyqqQjAA6SSOFbHUuoYYHu bfKuC2CsiHwn3lXxfWRiK4lcvSt7UxCzkPa9ysQFMFZEvuOOgdjljElosFDqLxdiMGqWu/6SbwGL FQ1i6m+IRUZwxCQzoeNumGk95audS3eEiRfAWBG5wA3Z6yQkwJPQBPF0F2Li5S7U6UxbwGJF4wLr B2Kb8yWZQeXTR6aVfeA2YQafiQtgrIgQ0xsQi47g2lMGQXE+F2LaWs1d3qRpbQmMFZEbeEfQPURI gKcdAz4UGRvKWAUuuNQxl8BYESGmM0Rsc8Z0xoGcQ0FMGStuyCbduiUwVkRu4JpDLGICMaCaDJor NBOzhCqeL42RYWIJjBWRG/iWQFNETICnsQ744dp/qJmBmNuE0Zm3wMWKB7JH1xqxzQnTHBoMN4HM S6zG8MULmbaAxYrGeW5DiAefaGluDqaUpH6gRBX3IB18Ji6AsSJynvsmRhdREgwdXhIpJIk3QPFw STeaeRe4WPFgCKJGj1g8HPCklaEvFYlaJPr2wMQ+8xa4WPE4xzqC2KvES1Yq9KUg9Y4qreMmoe2Q iQtgrIgQs4645OEwhg6SiL92H5kjOrl6faqIBTBWRBiLSf0Ri4cDnpYSbClI1Ho7riRqFpm3wMWK x1nREZcsHPA0zgBfCFJPRdaxbFCrSLwFLlY8iFlGfLJw8CWD5FHoUa9xZkTapWQXoFiRuF4kxCf7 BryVJK5aRITNq5e/rLg4ed6lUlWYntRAH6RVp2n1avXddvViu+0b1Wzv8TtDYxeqZZMfOrqfpDXK oBV2+Nk2suUQZ5v/UuLYna7rm+3takOPuOXb96v21XQ+746HczMe7prb4+Ew3c4Ur7d/UJ6pO7ok FPTLnlsFf2y//zQBUvc5dQ1xCRicNvmha5QdSCksjTgqYPcod94bzpdS7SjJr7rODt8In7vwkQLA C27yQ8GHqgQhJRhl2f6ybOwvGH9rv23OknizW6NTtuc1/u3bZsT3c3O8b7Yvfy2X0+wFd3iXPt9g wc369+1PyGWDQWTAoWM/+HX6+jrDu7w77ObdOB9PvEtjM4+nN9P8jC3SGnehU2G5Rf93Yy4s1cZc 64HSR6Y/zs0Oiz1iffvd03T3nBydxsSNjvx5OWaWKsfhmmOQw5Mk4xr9p53Gw9zMx+bhdHzarfv2 bsLXoW1usNPvd3czvnq7xgGZVvb+adzt8d24xuvam92efjN/4IMs16k8Lc/iaNMDMsQdVh4NV2k0 ltA9a6GGfo11ymdJ0qPf9yDOyyyKppdlnqen6TTuF2X4NZY9tMc1GlJ7mi6rfPucBRiNVxpqNJ+x gCtJsYDNv4qp61Nz+dgCbqZ5nk5yEiOdwPxBTu9uumvu8WUjzYdv1eVOmQu55ivcHsFNZTAdxpv9 swrVoEC9G8znFeqFpSpUe01RDnD7dmquVx+5nh8fHo5nLBMFezM1lDg9Pp6paB1V6IeP9JRqd73Q X7eVSKa/ptvHGaTY7vXGfTm0MY4Hebrj2n83TQ/NuN83M/Li12y/kMzdlTtk7hjX0KyWfnia9uOM rB/G23fTfG7oheiCg9ws21L26J1FPrv58gKhVbzP6Qmnd0YLJEpKhd5Fr3ldrgh5Tm924+F2er1+ xtla5TBI44D+82z7T5zthaU8W8X6b8LwMRnt0l39GeccHyO26s9HUgiH2xkntKa/1xttSVoU7RV2 aSksL1iG+AdHARx4437YrqwKlEzjMMd4eAlFk/nGsocyNB7cr/4ZAJ//Cc8KZW5kc3RyZWFtDWVu ZG9iag0xMyAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNDkgMCBSIA0vUmVzb3VyY2Vz IDE0IDAgUiANL0NvbnRlbnRzIDE1IDAgUiANL1JvdGF0ZSA5MCANL01lZGlhQm94IFsgMCAwIDYx MiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTE0IDAgb2JqDTw8 IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUNCA4MyAwIFIgL1RUNiA4NyAw IFIgL1RUNyA0MyAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA5NSAwIFIgPj4gDS9Db2xvclNw YWNlIDw8IC9DczggNjggMCBSID4+IA0+PiANZW5kb2JqDTE1IDAgb2JqDTw8IC9MZW5ndGggMTU3 NjIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImUV83OJKkRvPdT9NmSWX6Sv7MP Pls++AFWXsvSspI97y85MoGuyqIFHo0E6m/IiChIkshf/vKjvH/98bamWIex+vR27x+//vH65a9/ d+9//Xi597/fr0CGQnlnSqZijsal95+zq6bS+7//fP32CtXEmm8LfNELiEzJeYPACqJSEK2xYRcS 8aeaDphVYSZrQkwbnSmaFOIe0wXGDAMyW5PrLiJHUyMdINMdsjjjHG1UIsKrgGRCPVB4kggaIdWZ GMMhpOiQZLILe1nBqxBnnanV72lC//jyicnGxUMM6RDnTXDuECJfQ+4Tkw1lt/+c6HWM9ybHA0+U lKP4icmmHkJS0CEhGHu4LClLSP2EFON93X9NjjqGgom27nmKSIvhE1NMSrtSwDGiLeYZE4Pk0jam xkdMMbXsKgpWWzmd9DmdRMan/b5Vmx4x1ZA/xDjJtvQ5nkwmll0V4ZLTtX32Olec2b5YcZlSMSUa 6/fFiK8axvw5n2oN4g4x5RETTaB9veHk5Jh5Pt5ak/y+5HCm6ZhocjnERMm3ebG9Y5h9meKkUSHJ OLsvU3z+HDLLlPfOhLIvOXz+OiaZSKcY2YHy0RacyXZfc/hcdAzWln0B4T1WMeSNo0OMJAFq8wzJ 5rDPveheEdEb2t0CH7PJYV9fWLMCTR614P+LSZ+YYmzc1xdmUDE5GO/KTn0uhuoJNWnUEkyKu2ri S8Frv11RA85zt69cuwZfrcbHXVUJFtbNHerB4yuChZvLcaMgONi5racJHu7NHe7640yCx6q8RQ3w b+Fwtx/ZEQIsnTults7CQDB1+XC3H9chEHxe2MWE6Iyth9v/EBLh7PLPXf6QYO3CKSY8YpJJe/fw rGWQZerh4XyUzJDh7Pzhcj9KM8oyxBxi9AsQCpwd7e728s6ECmN3sCjP5yzUDKtzinEqhiycHR12 7fE8k4W1swcr9LAB5IKh8nN2gxysHZ1itK0hH6Rv+xn7RL7Ir7210zaN0IF6OvA87CChKSV76Ioe tpPQp8Z0aL0e9pYI1u7gUZ42mtC7WnuIedh1QvPq0qEverQFhH42eL//nkf3Qehokz3wPJocQpOb 06kx0s0UFc6nn2vakNPG7m/2ozUkNLphn9TPBpTQ6EZ36IsejW5Eo5vyof/y5RGDtQffUjzpGHS6 zp1iJOQTkQ28zD7CJRWCNpfCvuIUF3QIvN7W18QAZ5cPJ2Elrfx8uyO6Wrs9vYgeFv3LXqqNGhU9 LOVdMYnoWFPYl5tinUZFx5rrrtzEBGcXd8Ulohv1YVcW/vPCe8g02JmK5wtTRiXCVPDIYMk/3n+8 7Bv/cIK9u4Muy08bkhrk7QWPWgHgTcTl+/2DZxJ+OPiNyk8nThOLf3/99qfX35gUluJGWqX8wSTZ Gyn8HmdRHwnqO7sCbJpuwMJMCjdqCC8tsksXNzzfxU3OOOZ2SOo7d+DT6CMK8uBWgE3TDdj+1aj6 WIg/uTtzrsiti9kb2DGeCGl8MROX3D4SlHVmBdjuZAMUl1KYgwm8EEVKMRfc8osZjzluAaYY78xS uPuIQj+YFWDTdAMW6S/c3CRiKV/2O3eSJ3tyE25C4ikr7sy1rI+U51crwKbpBmwdeWYlLbhfUtw4 sRt3lNrMxlpxV66JfSSU+86tAJumG7DO9kTDg815UZAIihyHhlZpkmcpPZjcndxJieojwacLuQZs D76B6+TAUXskM/COKW4c240bjo+5i2Tcxe25kPVxHLeGa4prYMIKdubcMyNLibmY+UG5mGtPtapT zYmP7WPEBe/cCrA9+AYuOkXfNz0ZD99YkA2Kng3ihx6nyNmGSWUbn8Acox91TQO2J+EAdhH5WUUA LBevR0rcBaDk5psAJymHSaWcyyJARu6tugAF2BbGgeySOANWALOMgMr24a6gyGMzFXigQEF45F2V E5AxztKuAdvCOJDhAMpQ4E3iAI90uStAw3RTQLKRmDz++FHAPcgcY5x7oADbwjiQ8YLZocChViEA n+fuCpKQTQVRMgkTKQWeS10fYxxFXgO2hXEgewQOBVbyphI47gpwmjcFSe4RpqQUSG/Wx1nrNV5b CAewdzMR8Xxy3tQIirsAHOZNQJajxFSUgMhVr48xjzdWA7aFcSDD1Y1EDLnnTQLHXYGX2zcVFEnn WAXsUiBZIElQRiHQcG3hG7jcKw5+rCwIyGC48zspPoOf2x3wJ/tIw8Llr4/8mncFCrAtjAPZE651 V4C94oD6SEMrhXcqcJKGEnxTEMT59THZUYg1YFsYB7JPuNZdAfIFbZoFklTroUC2+1LgJQ3lc+4K xGP1MblRDDRgWxgHMox6iFKRA1tnBMAVKQVZat9UANMBf42J0/FSIP1PH5MfxUADtoVxIHu8WUMB yiwH4CvrXUGS8j8VwHqwApJ0vBRIa9hHrr9dgQJsC+NA9qiTQwFeGg7A5ykFUd6/qSCKY8bFdkqB OK8+Jpp7oADbwjiRrbR+UID/iRyAz1MKSN7+qSCLi8MUHN0UZLYhfUw0qoEGbAvjRMZzORSUnjj4 PKWgv0BTQekOFjupFFS+h31MoxxovLYQTuAwE9Hnnjf4OiXAqUSsct3libsJEEsiQ0pzAxRcW/gm LkmXwvx8sAgojzS0s7di/mx7e2fFV178XtodGVOeO6AA28I4kSMytyvg5EZA1WkoNuxS4CQNxefc FfQt6Hsw7bAGbAvjRE6o1F0BX/C3g2VVaSgW9FIQejMRdBpy2zPHVEdB1IBtYbyQ7VDAtY4jvNic S0KWVm9KgCvJxBOn4yUhSxckY7ajGjwQ28I5oSsqZRYNXPE5Ah423zUknNxnYaZ+ELD7+a6hSDck Y3ajHjwQ28IJaIvfcBE0NPC7xxHIli8axsIcejqi37lr4MI7xzyd8gOxLZzEX4LfDtkhGhy//RyB fPmmoS/MXq4kJlIaujuQMQ8JD8C2UBJ/CH57pEeXkGUTHcxs+SahL8Sd4DzA5JUECu855jDqwgOx LZzkpZeFNDs0sP3jCCTMNw19YbZyHTBZpSHW9xzzuBQPwLZQkpPsQGsBuSKBPTBH1K8ZORamKjUh FUnMS0LuPpFEwtwGjdgWTrI9OxI+q2vgTgAR3n7PyL4wZamMmLLSILsgmzAfiAdeWxhxQQvMLa5K HAoC9ocj3Pd87AtTkrcBU7wrYI82x/xJSI3YFs7A34HfBZhdA7eGHOG/J2RfmGJ3bFEexUuDd+85 5jKzQSO2hTPwl+B3xc52Dex1/JuLK9AXCX1dom5b0cYpCUHqs4y5zm1QgG1hDPwd6AUcciOxArwo JAHIllXBWJdCt+5oZZWCKHZRxmJHgdSAbWEM/Bn47cXvsoKC7OEAZMsXBX0d7Hft1j0rBam851im cdaAbWEMvvdDcMJDQUbucABy5YuCvi5ZcczJin++FBR6z7H4IUDhtYUwOEkL+KhYuwAY0sgBhSvL IqCvg/vmpgFTvLvWVMUtylj8qAkasC2MwUpawErmoQCmXALqt0Qc69BBcuOEKdwV8LbMcfoVjdcW QvxdsiJhZ7oAbA4HQOm3POzrYP+5d8SkjHt3CUHo5wYouLbwYWMkJ3jq/MgOCfBfs7CvQ7hl/qhb l0zsFvtY4igGGrAtjOxqXeHjDEMBS+WAAMpVQV+HEyysgHT7lpMYBBlLmvdAAbaF0fNn1J7R/LvW njS4L2kV8D+2q14Hj9wG9nmKfQKf/n9qdwFyjcvDdamCqAiuuNfPDEntrqSFAa4/W5oZSSOKtGEw cYllevkRUCUNSWz2Oq544+ALXETXOy38NGvE+Awj91OAjsM1TrHM2/wI6PGasTXdgQ1wHIyBq+ia 10QBLywn4AJ/KdBxSGUhlpnRbgXNyRXU2DUbboDjYAxcRp+5HQqYszgBOexLgY5D9e+gQLL6W0HQ LMTYnSajDXAcjIHL6HzgguSezr9wAvL4lwIdhyeNHRA+Lb4VJFEgcW7BijcOQt/VFsBXAaidOCHx ff0QoOPwrlcKqGLHR4BUJhq712SwAY6D0Te1BbBUAd9vTvDfRtRx+JWpoIgfHwU1XzPaq7zhjYPQ cxWd5V5TASxiOCF++1DHocCLFJDFjo+A7q8Zu9XKG+A4GD2Ru9a8okCbL49CFi/6ocDG8X+pIIkd bwXdleuOyZLBCjgORs9ldK38+bt19Q1X+aVAx6HeFh+G1Yc9MBNazHoKG+A4GFHGOut+REAz2+Cc vwToOJxpzXff8wiI7bpj0S3YAMdOiLa2WQco/MVcA6e7dvLruMSW5G79Hv4crztajbwBjp0Q5WvB z2SWaFkKf8+r/sWv43itkfXxCQt/kRuoUT244Y2dD6Ur8kZl5ZJIn6Tq98h1/otexzGtkR6van7T Sz1msWsm3ADHTogV4BfeSKWP0oh6ZPn4Ra/jmNdJj7LioUcHcGnu20DGziHHj2pKfwVkZA4OsMfN 6C7+kfcciZNNZmXeSQADekMryNaHTeh/qY1Vvg2QLWQXaUckgP8jDY8hNqlJYxP7SrP07B85WTMI LSoWh7ryTcumFCqDV9mC90OKGBZYqMEqs3N6dq152ftJ2mW7rEu8D815JgyNyRLXBjhWOoOFg4Qb BuPQZgc6uZ0cgnEnL05JXnzzcEemS40pdeNeAMdKZ7C6algLA/FP/s1cu5zoZA6arnndX6nKJdZM GhOUKfMCON5kBopKWJjhMA70ZqLJ3PTkjTnKE5WiPFgPc2F20JgsR22AY6UzWLQBwg3DcigLsjc3 XfhwJ3mgU5Ln+uGuzIwaaU7lXgDHSmew3XzmxBaN1n9z48Re3FlehZTljXi4O9OCxjQblhVwrHQG i4QhRkPdRV801uJv8iSd2SSvWqNVeSJvci/1ocZkmWkDHBuf4Xo5cGRPcQY7kTd3lJ5scjcpCVIT xz3cgUWJRjvuFW4sXIaJ9KLMVZ3BHuzVHuD58i/mrlbrq9VQ+lwzsm9Q7gVwbHyGizcy6KaXHwHP SWMP+qZ30osZPU5RuhO3uo0nMGMOltdWwLETGjCKhFS7CMi4KxgPS7wFlC6t2BTgxXL4LJbzVQRI zNGbgAVwHIyGjEKpmAIkfExAKdUWBQ0Aj4IgTQHu8eo7qQc15pnaV8BxMBoyisVmCsKPwgkBdnkr qNjDR0GSjcQnxFePGhxTnMac5x4sgONgNGRU7M4UeOQqTMDy/FtBEbKpIIuT8EmLgsBUpzFnS/Ir 4DgYDTlgoilw4ht0FXFRgNN8KShyj/ApiwIpCDXOXL/ijYPQgPH6mxHxfNI3aKzyIgCH+RJQ5Sjx aYuAzKynMVd7Y1fAcTAaMjpHMyLKHvFNAcdbQZDbNxU0sXPuAvYoEBeICZolghVuHHyGG+K0Ieqt 2jChguHN7yX5GH9xYsPiNhs2pj+NfM1VwQI4DkZDDgnXWhVgrzihbzZ0kninAi82lMkvBdHxEmos zhLxCjgORkMOBddaFcAvHU8EkCRbmwLZ7kdBEBvKct4KpMbSWLwlgxVwHIyGHJC1s2TkiCvDCaiK FgVVct9UgKKjZ35ox0dBZGumsQRLBivgOBgNOeDNMgVIs5yAVfa3giLpfypA6UEFSez4KEi8hBqZ f1XBAjgORkMOyJOmAC8NJ2B5i4Is799UkKVixsX2iwKpvDSWNPdgARwH40RmcycK8D+ZE5I0Oo+C JG//VFClisMnoq54FFSWIRpLsmywAo6DcSLjuTQFTY2D5S0K9AWaCppWsNjJRUHnPdRYLB2seOMg nMBxGjFU9Q1WtwjwixG7XHd54l4CpCSRUMrcgAVuHHwTN0mXQn4eLCa0zYZu9lbkr04aF3nk3/xB 2h2Jpc4dWADHwTiRM5yrCmhuTOirDaUMexR4saHUOW8FugW6B7McXgHHwTiRCzK1KuAFRy/qVhtK CfooiNpMxNWGbHtmLN0S4go4DsYH2ZkC5jrOCFLmPBKqtHpTAqqSmvihHR8JVbogidVZNtgQx8E5 oTsyZRUNzPicgRq2vjUUnNw9sCY9CJT79a2hSTcksXrLBxviODgB7fAbVUQyDXz3OANu+dBgA2tU O6LfeWtg4p2xzkp5QxwHZ+JK8NvDHaLB8+3nDPjlS4MOrEGuJD5p0aDVgcRqEjbAcVAmLgS/A+yh Eqpsokcx274k6EDcCfoAn7BISPGasUbLCxviODhTkF4W0pxpYPnHGTDMlwYdWJ1cB3zcoiH3a8Zq l2IDHAdl8uIOtBaQKxJYA3NG/3SkDSxdckJpYsxHQtU6MYmEuQ0r4jg4k1N3FCxLNbATwIzgvh2p A0uVzIhPXTTILsgmzAdiwxsHIy5oQ3GLq5JNQcT+cIb/9qMOLEXeBnzyWwFrtBnrbcgVcRyckevA 7wZM1cDWkDPCtyF1YMlasWV5FB8NwV8z1jbdsCKOgzNyJfjdsbOqgbVOuJhcgX5I0HEladmKNm6R ECU/S6x9bsMCOA7GyHWgF/DwRqECvChJJsAtpwIbV6KW7mhlFwVZykWJzVmCXAHHwRi5DPwOUu9S QYN7OAFu+VCg41B+dy3d66KgtGvGNgvnFXAcjDFoP4RK2BRUeIcT4JUPBTquOKmYi5P6+VHQ0jVj CyZgwRsHYfRiC9RRuasAFKSZExozyyFAx6H6ZtOAT35XraVLtSixBcsJK+A4GKMTW6CUrKYARblM 6F9GtHHoINk44RPfCrgtM856ZcUbByH+XVxRsDMqAJvDCVD65UMdh/KfvSM+S+GuVUIU+rkBC9w4 +LAx4gl+lB/ukAnh04U6DtMd+fPautTEalFjy5YMVsBxMLKq9Y3HGU0BpXJCBOWpQMfhBBsVpLV9 q0UKBImtzHuwAI6DMXAZXR3N372raXBfyinAhsHEJZbp5UdAlTQksdnruOKNgy9wEV3vtPDTrBHj M4zcTwE6Dtc4xTJv8yOgx2vG1nQHNsBxMAauomteEwW8sJyAC/ylQMchlYVYZka7FTQnV1Bj12y4 AY6DMXAZfeZ2KGDO4gTksC8FOg7Vv4MCyepvBUGzEGN3mow2wHEwBi6j84ELkns6/8IJyONfCnQc njR2QPi0+FaQRIHEuQUr3jgIfVdbAF8FoHbihMT39UOAjsO7Ximgih0fAVKZaOxek8EGOA5G39QW wFIFfL85wX8bUcfhV6aCIn58FNR8zWiv8oY3DkLPVXSWe00FsIjhhPjtQx2HAi9SQBY7PgK6v2bs VitvgONg9ETuWvOKAm2+PApZvOiHAhvH/6WCJHa8FXRXrjsmSwYr4DgYPZfRtfLn79bVN1zllwId h3pbfBhWH/bATGgx6ylsgONgRBnrrPsRAc1sg3P+EqDjcKY1333PIyC2645Ft2ADHDsh2tpmHaDw F3MNnO7aya/jEluSu/V7+HO87mg18gY4dkKUrwU/k1miZSn8Pa/6F7+O47VG1scnLPxFbqBG9eCG N3Y+lK7IG5WVSyJ9kqrfI9f5L3odx7RGeryq+U0v9ZjFrplwAxw7IVaAX3gjlT5KI+qR5eMXvY5j Xic9yoqHHh3ApblvAxk7hxw/qin9FZCROTjAHidjbFJE1iydoX2M8beff7Xr56/L8Q8f+18/f4eM f0LGf+wO/n15d/3r+uNPd/37H+yS2K9WprAEXf/nvGxydMdtKDqvVdQSrH95GUEGWUHPHgN0738Q 3kvKlsxZ0IC/Zj3yWJYo8lIXOnWqxBSFefYPXoqBwR14Gnifn/afn39zaRPdrK6l6c9lC8WR/J9L O18k36Xam98d3v65j+1LNL7dw84xq633I9FeQ6oeAv7OXbYcebPGNeTpKQGlB9bNHobWvjItYG7o VQLei/hxlEBS+4KtafOQNYmQK3UjU/Le1W5aetCbpwTUTEw3LZjJR9CpG6sBN7Nvobt1j7pV8I+n BNTU5Zs9HD30WjBgbGirSan6Cwx9eEpAqY0ro+2rPOisiap/Lxu6WuebW/P/eEpADcyr1XqYoTvv wExMxIXmVkBYWmMz9OEpATWgjmj3B31ZwNjQean2Vzd+PCWgBmbcar3f0E3frQFQdy86VR+KoEhu Rx+eElAD0oD2dDRUMwPuDX1x1cXWPhx9eEpAjc5/oHJa6ISAQj9Hz5sKKnFDWn7Qm6cElNr4nDrY CIBGbdKAyq9b6GlTwthG1dNRAklL7TB7OBmfx4C5kSn4xkyuwA19eEpAeRWvaAELnTQFZmMiLTRT w4e7VB705ikBpW216D7qiHANQyetiBrQvfkYurVncsWuGPrwlIBSwTDM7o62AqoBY0NzCGCJGnNV kMNRAkltH2/yQ85DAwaHmEUuXG7i0p+SenhKQLnKapcPXipC9aUaYKJsoTk7dSbgaF5BTk8JKBWQ OP2WfK5FwGUBfUMnrLpP3+9F3hwlkFyaNpy1kVHWNcCk7CJfWG7vvt9GPhwlkFR2d7OHk5GTDJgv 2S5Kr9yO5PXj9JSA6jpdaVo1XKuFVlHzqxNAu1/05KIzmte4yoPePCWg1C6wUQsMjeuvAVqBN/Tg qlk/+iqop6cElE5Cw+zhaOhCBowN3XGIkJH46HWKh6cEVIc4VVvnjgedtXDqWFg2dENqtDH9LBd6 85SA0gmyFrU7BzCg0SA0QLM/vWje78Yy0ldBPT0loLpOC2b3B31ZwNjQnHBaJbqMB715SkCpnWEP Dq5AoxlrgBaaDZ1xjI3Pnr2CnJ4SUL1pqtDGWAg0JIQGNF3Si05Iu5ZQUfu18vrwlIDqVYu52lBB C50soG/oC+utk9tyzwe9eUpAdXyO2tBuhk5c3Y2ve9D9xqorS2qb6xgPTwmoXlgp1R6OxrVmwNzQ XG9t3PF1ioejBFLX5qaFsmNJi5wQMPSrNjJlP7ejtYe8OUogdaR7/+2Q9rBviHn1R/K/YCNTPCH/ QD4dJYB6Yo3Sn27gyX5048JuYHAvXpli1eP0k8DpiWVVX5cXF2X3Ro3JD5cyrEzuceoLvDlKAPmW cIMIHuwU2EE0jocM1VEGnrbFHz8JoIbqrbZWVt15gi8P6BuY5blQJFTXSx9PCah2c/GaiMnQnS0Z mZp3NNfMVl69a308JaDa5IarPQzdeCAIwPks9MVVZ89okk9HCSS/5X7nSU4MyMyohwxoYb+tPhZ9 PCWgGiSJ2iqnDF2Z9ihl6UU3rDmzKa5kPv0kgFrnTVR7GLjwpiJgbOCJNbOKVp+JPo4SSN4EvCWQ nBlQWWwWGczcql8XJ++OEkgNelVt1VJGziyJaHRpI7NVZVaN4h384ykB1SqVodp9oS8PGBu6Ie8y G0DxkejjKQHVKtuLaoZs6MT2A1FRNjS7YL7wLN0P8fSUgHJt40pH0RebJqQQeuhCF6w6sUoX7+Af Twmohumj/w6rvEQnD+gbOmO9idUU19HRu6cEVMsUIKoqE20VgxAokJ15Q7PBQtHhOlr5+HhKQLVE 2aT2MPSkrELA/RZT3JNUkIDFJ6KPnwSQ63VX7wQnBtxUhg62zp0Sc9snoo+nBFRDmo5fnTkuQw8K WAwlaUeDzBKSfSL6OEog6Wg0zO5G7lTdCBgbuSOdr9ntppN8Okog+Tjnwx3JedRfTH9VO8lDRuJd 7InZB6KPpwRUxfBB+za06h4G6D3b0JWLtvtYLac/nhJQdehnqd0owom+PKDvaK66NFaRRd4dJZDU xqQ4tLoaWb+mIUBL2kbGZbk4yuXkZ3g6SiBVjB6070Syll0PuG8n/w2vWh6vMmjpT9JD++evn//8 /vfH5Y4XGb77REl4V8Vsorbm2Xp3YoAW6PXuf/1cgBK4RO+kYiqd0e8q1Dv94j/quJx4SvoyHJoA PvV/ClTYsJchUerFcFWI+WLGjMG/+5aiIetzVj5dMZ+eElAz853VZlRFJ6gQBswNjRt3d6TB7Rrj 4yiBNDMTpSZeRCMnTdKBTHzJVOE3U/ju6UFvnhJQMzFTamZ9AXqyH43bM8XQvHJ34X64yPh4SkDN i4lQMcMYGiqEAWNDN646VX+BoQ9PCah5MWUqJq+FztqPJkrVi+aVm7bjrjM+nhJQ+s6qy6iYFw2N EVEDtBmkF82tmJNrd6Hx8ZSA0kOYZvcHfVnA2NAcHyblxrzTg948JaAGWpfaTX8MDSWiAehkL5q7 PFn0pwuNj6cE1EDDpT0drQI1MeDe0Owhs9jafVw7PSWgvCR4gTB0QkChn6PnjcsyEzfEhcbHUwJq dH6O1r3kaCgRDaj8uoXmAeKSYVucfDhKII3GM1B7OBmfx4C5kQcWPSYzMC/04SkBNRozRxtMedBJ U2A2JtJCMzVGz74rC715SkB5I/S2CHRitdXGifRfaI5Vo3bfFUMfnhJQo3BgUbs72gro7Ly0C12R eyxRY64KcjhKII3CIatdbF5GxhQ2B4eyRebAhilBn09JPTwloFwnuWpS9HVzdpyTo+RCZyy6MwGH S42PpwSUqjucfsO2LfRlAX1Ds2FT4mK/F3lzlEAaGELVxlkbGWVdAzCTvmTOgr37fhv5cJRAUhXd zR5OHtRo86Zkc7JdlF65Hcnrx+kpAeXa3ycBQ0NY3hd15kJzzOwZzWtc5UFvnhJQahfYqAWG7lTD OuGkDT24ataPvgrq6SkBpXPWMHs4ulHDa8DY0BShEGv46HWKh6cElNoYK3RWLA8ac4eOfGVDN6RG G9PPcqE3Twkon2l9wgW6clq6C4enheb9toG2r4J6ekpAdRXvZvcHfVnA2NCFq6YkRWda6M1TAkpt DJ5tcAwFunAyvSsnqYXOOMbGZ89eQU5PCailHaEnDA0JoQEqvcuLTkg7Gw37tfL68JSA6qr/TZa2 B50soG/oC+u1Ca7d80FvnhJQHZ+jNrSboRNXd+PrHnS/serKktrmOsbDUwKqF1ZKtYejca0ZMDc0 11sbd3yd4uEogdQz56GOJS1yQoDq/LaRAbYsae0hb44SSB3p3n974gDw5+eGmFd/JP8LNjLFE/IP 5NNRAqgn1ij96Qae7Ec3LuwGBvfilSlWPU4/CZyeWFb1dXlxUXZv1Jj8cCnDyuQep77Am6MEkG8J N4jgwU6BHUTjeMhQHWXgaVv88ZMAaqjeamtl5UyrAZcH9A3M8lwoEqrrpY+nBFS7uXhNxGTozpaM TM07mmtmK6/etT6eElBtcsPVHoZuPBAE4HwW+uKqs2c0yaejBJLfcr/zJCcGZGbUQwa0sN9WH4s+ nhJQDZJEbZVThq5Me5Sy9KIb1pzZFFcyn34SQK3zJqo9DFx4UxEwNvDEmllFq89EH0cJJG8C3hJI zgyoLDaLDGZu1a+Lk3dHCaQGvaq2aikjZ5ZENLq0kdmqMqtG8Q7+8ZSAapXKUO2+0JcHjA3dkHeZ DaD4SPTxlP+RXe66luQ2FM37K+oPrPcjnsyADQw6mMBw5kkM0MCMA/++uTelOtIhGmBf3kuuUqkk ctOhamF7Uc2QDB3ZfiAq8oFmF0wBNrf1Ee9IcailbZbSUXRg04QUQg/d6IxVR1bpvDr4V6Q4VMX0 0Z5ulZfouBLagU5Yb2Q1xXVc6DNSHKomChBVlZG+ikEIFMjOdKDZYKHocB2tfHxFikPVSNmkfjf0 oKxCwvwUU9yTmHEA85qIvuLEgZZeX+qd4MiESWW4wNa5Y+TZXhPRV6Q4VMUx7Y/OHMHQnQIWQ0k8 0SCzhKQ1EX0FiiPpaNTNb0ZuVN1I6Ae54TiH0eymk3wHiiOtcW4NdySnXh5Mf0U7yUvGwQvsiWkN RF+R4lAFwwf9aWjVPUzQe3agCxdt97HYmf6KFIcqXV9L/UoRTnRYCe1Ec9W5sops8hkojqQ+JsWu 1dXI+jYVCVrSDjIuS+Aol+L6hnegOFLB6EF/RpK17K6EORf5D0SV/EblTk//i/rR/vz9x2/Pf34s ubOKDJ99o8Q9q2A2UV/P2X52ZIIW6P3sX3/85Zf/jueXn8/9gJ+//P1HfP76xOff+pup/e5/egee vz3/+Gd4/vXq40FxlRsf9Fnwzx+BpNXNYoK0iVgZvrBg2yJeFJKt28qwxIIbDT+mwCXq0jO3kftv W28WqtLQV6Q41EjcDfWnoePkdmnCPNA8LmZxNom+I8WhBqqX+lDhG42O1cc6VobmITe7JclXpDjU iPxDSaxGQA8e3T7XuTI0r6ZZlAFDX5HiUCPwddQfeaODJdixMTTriVlcwo0+IsWhRuBHKJnnF+jO KjHCOoFEWxU0i4pr6CtSHEqfiaNTMF0aGrJFEyJP0kazdJstsy30FSkO1Serg/rtRUdLGPmDZsMx i9620UekOFRHBVO/apUwdKUI0c9dDnT9tMq6yVegOFJHBaM/F1mrJBK0eRxkdnaze575ihSHUh9l YlUTQ6fMhHagrVfarJvniz4ixaE6flDfiiTQiNQEK6MLbSrKbC377F2R4lAdy6c/XnSwhHmgqf3M 1jpf9BEpDtUrn7naEdD4KppQuYSNpmI1W9s+e1ekOFQvVJmriQIN4aIJjRu30dTZZmvfN+aKFIfq hfVstX5DR0vA597ovgeEOtoLPuLEgXqmnl9qBWC7EaNzmNtgjjNm69xrviLFoVRUYQhZEkvRYXJK GYNz50LbDGa2BWvEX5HiUD1xcqqohhsdLaEd6LSnxxb7Cz7ixIF6tFZoUhbgwWFwid0N5rBrtqW9 5itSHKqj09CfC905wg40pg+6cjtoUXYMfUWKQ6kfVVStscHQqadnBorSje7x2baV+qKPSHGohq6u vg07QKPta4KNQxs9KOBpW11H744Uh9KhbJo/XnSwhPlBYyDdtrXyoo9IcSj1IWDXYAm0/oSEpK/1 Qae0hRJXaugrUhyqdcruqitY6EJdPjMnrVddjmfbNvaqr0hxKPWD+b1udLSEkT/o2p9t24wv+ogU h2qNA05VYbXQmRPQLLrID7rnZ9u+hOlXpDiUqkeMZerPhU6c26bq9APN3mIWlc7QV6Q4lPoYJVWY 1heNWVMT2gdtw6nZnvKLPiLFoZrKZvhTj7+ho0ZqQtPX+qCtmNL2Tb4CxZGaKnjzx0sOljAPcrGG yK0u6UUfkeJQ6ietsA3fyNAQxpqgQ8GBbkTT9n3Nr0BxpKbDlJ7BBmkPf2oBRvzQl/qA+3i27c3W fAeKA6kfzO8bHC1+HODJZki75OkdJ46jP+A+6q+zcQfvq6bh9i5ui/nZtm/wFSgO1HA+6Vur0S3r Whmxg+Egp/ls24dJj69IcagGNat+ZjMjOq4EdLuNLvXZFj16oc9IcSg9IqjXehazoRvrOU5UPtCt PNuOsFZ9R4pD6ZXBlqs/DV3ZhZAwD7RVOoCj1f+vOHEg9XFG1vRJMM4Qrm49wJPbQTvS2OgzUhxK Kwj6uZYqnHJFF/Z71DJcg4XG1m2LvkH0HSkOpSV1mj82OqyEeaDzs8woY4PPOHEg9VFAtHIXA2cW GJT2eoBLe7Yd1W73V6Q4lDavBn+wBipa5RQTKovkRrf0bDvaOnh3pDiU+sH8vtFxJYwDPeKz7Vg1 6StSHEq7OYqTyoZs6MjiheZfDvTsz7ZjrFXfkeJQKm/QF9Wfhg7sm1BD4YMeLBzLzrXqO1IcSv2o fg/s7USnlYDmv9GUpWZnSBt9RopDqd5D81JhmeirHmxMUIlyoMt4tp3RpMdXpDhUjbwB6o+NDith Hmj2FLMzpY0+I8Wh1IfeU6VdDD2oByHF64EeRNPOvFZ9R4pDVdPIOnhAsypadw4JMVDUbvTkhtDO YtLjK1IcSv1gft/ouBLGB439fW3NG31GikPp7JV1HtFJLBtalYS2PoxqRcX5RufyvLatVd+R4lA6 inbzp6FVUDFB79r8oMt8XrtkzVekOJT6Eb7GxWLouBL6gW71ee3Y5DNQHKlohYWvL2fkQimgm6hv 9SEP7oTZuc7eHSkOVRorg/rT0KqPuyXMjf4DUSW/URkaqeO/qKl//v7jt0cF1IN/fOaNEPcM9ZHY Vdzn9czEhELhy2f++iM8+LelKiRUx7hUFDhALs9qclTB6Grr79wfhUHKBmo3bA9o+hIGTJHf8AO0 hIz5q9uKqvol8PCojEwQXxFLxvFaGx4ieoxZDDKGviLFoUbiLhRcCqLj5Db1znfa6IzzYbYU0yFf keJQ6uOjlshPbGh89T54CDa6oseYxYy00UekONSI/ENJPJhAD906TZg8uhvduCG0pe1VX5HiUCPw ddQfeaODJdhxMfQgmrb0/qKPSHEo9fERSubpBbqzLIzAb7LQeryebTF+GfqKFIfSZ+LolMLKBTQG SU2IPEkbHdF0zZbZFvqKFIdSP5jfXnS0hJE/6Ew0LTTNRh+R4lD6fQv8qk3O0HpzkKD94kBXbght 3eQrUBxJD+Uwfy6ylkUkZH2rD5n612zNa8a9I8Wh1I9V/caqY+iUmdAO9CB6GHq+6CNSHKrjB/W7 FhJDI1ITtF3UF50Ceq7ZWvbZuyLFoTqWT3+86GAJ80BHtBiztc4XfUSKQ/XKZ5bBmwA0voomVC5h ozO2wmxt++xdkeJQHSelPgXnxtCRNXk0btxGV/QYsxh2DX1FikP1wnqmfp8bHS1h9A8a55mmjvaC jzhxoJ4p4ituqIHtRgxc4fyCB2qp2Tr3mq9IcaieOPzUsOZTCC2Ml2NwVFnoHAA120JdPeuKFIfq qH/qRw5Pho6W0A407iBNi/0FH3HiQD2yGVZUcAMPzoADJf4DzqikZlvaa74ixaF64GBa0XcM3ak0 xuQgu9GV20GLsmPoK1IcSn3IyopuudHQnRNf64Pu6FpmW6kv+ogUh2ro6uoXSmOg0fY1IVI8b/TA oTPb6jp6d6Q4lE5i0/zxooMlzA+6BHYt2tbKiz4ixaHUh2LVka8stP6EBJ1LDnRi16LFSg19RYpD 6dSLcanqCha6cJ6ambp8o1k5zLaxV31FikOpH8zHyGfoaAkYCje69mfbNuOLPiLFoVrjTFNVWC10 /j/ZZY8DS24D4XxPMSdY6P8ndm7AcOATOFsG3vsHLhalHvYQC/Rbvlf6mqOWyCKHno3J1aFn/dzn TMfuvZUSUPCPszPeB104qsFgJodmb7GnVrprTZ1SAgpx1nhxmjR0qVwwvmgtQvc5S33QTikBNeDt Nd44/obOUGIBvH3/oq2Y8jkv+SWUQBrw1xavh5xswXbkZg2RW93Kg3ZKCSjEBRV26DcytBpjLNCZ 4oseRPM57zV/CSWQRsGPGp+h1l7jvTk07HWGBAPP9bnPOSznt1ACCHGyeF5wNv1y4M1myOexp2+d BA7+R+8j/road/G+Ypne3sMduX7uc17wSygBNPR8MrZWgy2bqIy6g8mRy/7c51xmPX6UElBD3Szi ymZGdD4LtNtddOuf+9QefdBeKQGFI6L1GmexGnqwnuuJqg492uc+VzpZv5USULgyuuWIt6E7u5Au 2A5tlU7B2er/j04CCLGekTN/EqxnSK9ud+DN7eBzlXXRXikBhQqi/RylSk850I39XmuZXoOD1q27 T+0bRL+VElAoqdviddHpLNgOXT/nsdq6YK+TAEKsBQSVuxm4ssBoae8O3MbnPle32/2jlIBC8xoa L9ZAoGGnuKCzSF70KJ/7XOMcvLdSAgpxsnhedD4LlkOv/LnPdWrSj1ICCt1cixNsQzV0ZvHS5t8c es/Pfa51sn4rJaBgb7QvIt6GTuyb6obSF71YOM5zn6zfSgkoxBnxTOztRJezQJv/RdOW2nOnctFe KQEFv6fNC8ayMIYfHFwAi+LQbX3uc2ezHj9KCaieeQMQr4tOZ8F2aPYUe+5SLtorJaAQq9+D026G XvSDasW7Qy+i+dz1ZP1WSkB188gYPNSzAo2d0wU50dRe9OaG8LmbWY8fpQQU4mTxvOh8FqwvWvf3 efZ60V4pAYXZq2IewSRWDQ0ngdano1qDOb/o2j7Pc5ys30oJKIyi0+JtaBgqLsBd219025/neWzN j1ICCnHWGLrcDJ3PgunQo3+e57pkL5RAaqiwGuPHGbnRCmAT8au+5MWdsOc+Z++tlIBqg5UB8TY0 /PG0Bfui/6eqVh9VVY809Y+MpX//94//fGCgPvof3/lGSHgHYl04Ye7reWfhgkbjy3f+Cz8Ic2M6 ruff//gnHas6qalTUwN36Qva5/Q6mmFtbuffuU1gqqNN85pjB/jrj3+7rPUftdp86T+rz8/q6xl8 Cw3S1DrXno+wy/cjrPWgnVICauksyXhM7+KxQE/z8315cPyZ/BFKIC0tkE398Z3FMu/cs/dGZh8+ x33kx2o7pQTUKvycr/lAmysWbJf0yN9L2vqDdkoJqKW1l/F4stYiNLU4f9GdWeflCuKPUgIKcbVX meNUw6huAwu6y7rl03a+ZfxHKQG1YOPwqpFYgg2tVXku/v2rap0ynsaDdkoJKMT6zpHotL5jzdys HBddrjH5duMfpQQU4mqvsolyb3apublvryq+ejmpn/nDCSWAVuLX9QOTWlHorQoYOI1jLZ2jegkl gFbiiRz5DKp7s2Xjr/M342V2OL9c4EsoAYS42ovmnZj0OuOv+zdjdVA6G9CotTswOZ0Ezkq896PQ rJCrlnElVoHLnbTa6+XiX0IJIGyR1irEPR9wxUCBL5FdwjwPc/rB46WTwEHc7D3zJqwOZmlT/XI7 E+5nNw7X6SRwECd9T+Vo9kyOS/3Ql0tfNnlNfgbHI5QAwsXRf0Bstmerr4e+UPY4Sg4GmRPpKaBv oQQQ4mYvegZSnVGgby7jwoxTPcPBATuhBNBc3PuhM88ZR5clMlzGPL6DzeTOBC+dBA5qqp4VxGdG QufWPKpvUjrZ6pR7N/qAnVACaE6e7tGuax/04qv6FmXXYrBOzFIv2AklgBDrdUTzTvUMucsScR1K D4L5z0SPcGZcJ5QAOn5j9OPgt9oD6l1/solrZAWPO8G8hBJAiLu9aNyM1YDAMLjuxAs32EHGrRMv nQTOHKzRiM1QbjjLTL1rTTZ99jXOFyT4JZQAgpFkO9EvccA6/UHvGhNvcmfWfVysk0nAIGYrwc+4 +eo0Ajtbv+VnNubbuc3HALyFEkCIq73oDJ+VzR5615PoWXrVrEc5ZeKlk8ChVUc8zwgDfd/Uu5ak 5xbgco7F4TqdBA5i/YJDffHhVs1j+I6k9x5cnop+sU4mAXOGFsTbig9s0qbe9SMrO6yZ/ezCSyYB g1gvyEDl6Ac7LQ3Xjgb7Zxt2KNYFO6EEEOY3vdKI+82XbWb6dmTVrNFr9lEv2AklgBA3e9G8GbPP TJasC556JRr7hh63A3ZCCSDEWjYHzJFlnK3PTBbZCx7MmBao17PHL6EE0Cz83ohPQ8rWZxa7wgV3 zbjS1/dbLV9CCaBZ+D+I582YfWZx2QU3/WyVHlON9gE7oQQQYk19ptuQkjUanSS+YBbgSkfRzoj5 FkoAzUxvM9PtSMkazeanueDCjHmM27pcp5PAmdmmvXQbEmxdpl7P0uWysNdaeJj3BTuhBBBiPc/4 w/oIhMsSGS5httDKBt26fbu3UAJoJt5AxHUfcLdE9jfjzuNQ2UhbyxfshBJAM7FiIB43Y200+KFa Xx4wiIXtrj35OpkEDGKtb4itjeATa5vZidXwYqfmW8Y4d4Tgl1ACaGwW5FlOI4FeG822me4BK5dV KJWLdTIJmLHZPxD3m6+2GZx8/eNibR9YJuqaF+yEEkCI9Q2IrY/g2msGOzOfB6xbW1jl63Frb6EE 0FjcEVQPayTQ645Bv13GVTPOmwfuVMy3UAIIsX5DxP1mrN946+TgwJpxZkGu59a9hRJAY/HMIbZm gmagZ3IXntALZgvN9Je1mpl4CyWAxuItQU+xZgK92jro17f+aDEDmGWilst1OgkctD291oj7TVh9 6K4sAper1Fp58fbFOpkEzJgsQ4jXPFj1zbv6ltSWJppZg8qeF+yEEkBjsm7CulgngemYlohrSdyA THOpN5rcl04CByZICz1im+Gg1165m+9IWiJRtxfB83KdTgJnDPYRxDMfro5Su/mG1IaetMQiUfq6 YCeUAELMPjLODAcbuiyR+a0+5iOSXb12TsRLKAEEW6zdH7HNcNDrT9ndNyQtvYknSYvF5TqdBM7o 1kfGGeGgVzsDvWtITQ9ZYtvQUnG4TieBg5htZJ4RDnPJsjxcP2oF30yh6ST7EkmAjGYtZJ7xDfpu SXx7kQIJ4+Gk39WjmuGe8tI/tFf9/V/V5o/+p1qcSmx/uenL/XldHc34pt8Xu+dfXNBm+abG37O3 nsHN2rTXetBOKQF1fkRfNIOGLrV8f6Wh9fJsnvQ960N2QgkkxOrRu5oRI6sb4QI3pW7OJJtnco+7 Hy+lBNQqnCq6NuOL1jqHBW5O3bxBm21rnzHqRykBhbjbq8aTddFclp9UtUOpoVjnBRftlBJQiKu9 am9DK4IL3LC6eYs2C9Uu5aBfSgmolW3M1It30b1zgRtXbSsWPe5O40E7pQQUYg6aakwuumou8O0u a71Lfy663LX7g3ZKCSjE1V61u8ZbzQn1bmbdHCIWh0tNnXb0JZQAWsmGTZjJfcB9U++GVt0B9eHj JH7ATigBtBJP5FBzcsAVieCv3dS6OAcuFqzV+wU7oQTQSrxFQ2ckghev80p+bl2L20ufu+zQvXUS OCvx3g/1JofLeTT5wVW7aPpzLqZd1gU7oQQQtkhr1VBvcsCcR7MfXO08zPl/0qumt5EiiN79K+aE bIksPf1RPY0QEl8H0IKQ1jfEwQTvrlF6vIodof0j/F5eVXXbPellCeQSu5KXN9U11fXqlXIU3gaX Ox7EXp8Ta8JiR8eincobJOFQqlF4G1zueBCzlBEvJ8Ib1Y2OInyVV5aEKNckTsX6LIC5I8LF4T+Q q7Y1qhu1RTqVWLosjknK4SpxA8wdEWKvDyq2NaobtUU7ldhKxoavdCRbiRtg7ogQc+3JV9tK6kZt EU8llvYlEZOymS9xuePBTBXJ89W1krpR14rUJCaQqBa6EDfA3BEhDvogqglbJ/hGovRakMyJWNaP JTB3RIj5OlIobhNSPmkijUJxI/BtlVYeS7ctgLkjgpLzAEFsYyEWN+pafYpiLElWXSozcwnMHRHi oA+imjH7USwMjTrJhSNREKpzYoHLHQ9intEUit1MXlwS8I00cYNh0ExU3qAQL4C5I8JuKnJCxXAC z74O+EaY5CaLeaVAlbaB5Y4GsUgJFbsJPPtQLMjuOn6il3xl06WyACyBuSNC7PRBajdhDljsgW80 SXaWIIsu2TImFrjc8WAfFCWJxW0Cz24U+EaSophAaWFui8Lb4HLHg5jfIOJQ82UzCsvUKBLfe3Vs ZEKlbWC5o0HMVhOxWk14RV5ygW/0SMeOzMxQqrCA5Y4GMV8QGE51msmqylArRyT66UmbYqrEDTB3 RNHJlUYcar4iM7GVI51m6n8CuUrcAHNHhNjrg2LNWHQmtqaVF14j01bbrRA3wNwRIeaxSamYzTSq zsTWtBJJxrICBVdqvADmjihaed+IiyCNqjNT61r5nvGqIO+uTssFMHdE0coXxLFmLDoztbaV7xmI ZcfkRbsQN8DcESEWt2mqIBkVGnYSV2IZwE42Cp8K8QKYO6I4ym4TTVUko0KTWttKVjKWNvZT5W1w ueOJo9pNUwUJa90o+Ma26mB3zkozp0rcAHNHhFgs51h0BMBJE2l8K4mEOhFoH4p3XQBzRxSNetdR 7qMQB03k6l3Fm4JYhNT7sRI3wNwRRSMTAzHVjFlocFCeLxdiMFqRO3/Jt4HljgYxzzfEKiN4xSwz ycg0rLSR87VE5Y4I8QKYOyJKMpCjLUICPAtNUk93IWZemULGVtoGljsaSqIfiEPNl2UGnc8flVbr IGPCTbESN8DcESHmJyBWHcG15wzSKPlciLm0Vqa8K9vaEpg7IpqkIpgeKiTAc8WAT03GjjMekzRc mZhLYO6IEPM7RBxqxvyOEzuHhpgzHmUgu3LrlsDcEdEkPYdYxQRiwD2ZrHRoJRYJHWW/dE6XiSUw d0Q0yS2BpqiYAM9rHfDTdf7wMAOxjAlnK2+Dyx0PZI+vNeJQE+Y9NDkZApWXWZ2Ti5cqbQPLHQ1F GUOIp1hoeW9OrpUkP3Gio8wgm2IlboC5I6IocxOriyoJlo6oiTSSJAUYZbnkGy28C1zueLAE8aBH rB4OeNbK5FtF4hGJuT0Jcay8DS53PESiI4jjWHjZSiXfCpIn7jQjQ8KGqRI3wNwRIRYdoeLhsIZO mki8Th/dI4xePV86YgHMHRHWYlZ/xOrhgOejpNAKEo9eI53Ew6LyNrjc8VBQHaFi4YDndQb4RpA8 N5kR2eBRUXgbXO54EIuMxGLh4EsmzaPRI2/xzpjUlGQXoNyRkFcJicW+AR80iasWMeHw6pufVtKc su9yq47YnsaJP1ir7verV6uvt6vPtls/jMP2Nf7P8dqFbrmpXwzfT9aa0WEUGvzbNovlUGdbf3Li NyiPQWdvb1f8DV5v++fql/XL45vDxr+w6/nTYfdwfrufz4fb3Vl/d9zcYF6t52E3/z6c9rcP94fz +82v2x/4DGVyUhER5vcyRuRj++3/J8eRYz2yZbM4DTfl0wwjKjxhUYK2jAkl5wNLQTkJI8fjk31h TJi+3P7BZPSR+lU6dDL4OPHclkq/Sq3kNPPwdnca8sPd+fDubj+8fphvz4fjfPp8o4+65m04Xzyq fuFfweDx0B8tOjOZkvtHM7YsdLC8N/VLS+MxMvwU9J1LliVhW17uaX86lWLPw28Ph7vfD/Oboa/x B3Jlbq7BM3O90DzKFbepJovbwdVt++P4qCmeUFvoq2FhfGa+F5rHtb12A6aWFveYN7CC6/3GrYd3 u/vdNTzv74d5/+Z4PuzOKL9fHza4LngJp3/vb8e9ieu8bPD/2tYXlr6vr5U3JJX/6lmVd+z9TKJn Vv5K87jy0tbr+Tjvhyek40cMaI9trkvnqZlcGR737LUHRvehnj3Od0+pmA/IjGx6ZsWuNP+cp/Ha q48S/WTYz7f3799J8BfmcEBy6+839kVc/4x3D8D25at/71Wf2ESmJ/TqxyTswtL26iiq69L0WLz0 DpKe68fDfMi7u+GQMY0zTri7HMnDhq5PG8BpzWOwmYBysO+2q2B4aU7QX3DCEUFCabgJQfccCPDr 1d8DALdpyPkKZW5kc3RyZWFtDWVuZG9iag0xNiAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJl bnQgNDkgMCBSIA0vUmVzb3VyY2VzIDE3IDAgUiANL0NvbnRlbnRzIDE4IDAgUiANL1JvdGF0ZSA5 MCANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+ PiANZW5kb2JqDTE3IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwg L1RUNCA4MyAwIFIgL1RUNiA4NyAwIFIgL1RUNyA0MyAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dT MSA5NSAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczggNjggMCBSID4+IA0+PiANZW5kb2JqDTE4 IDAgb2JqDTw8IC9MZW5ndGggMTYxNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0K SImUV83OJKkRvPdT9NmSWX6Sv7MPPls++AFWXsvSspI97y85MoGuyqIFHo0E6m/IiChIkshf/vKj vH/98bamWIex+vR27x+//vH65a9/d+9//Xi597/fr0CGQnlnSqZijsal95+zq6bS+7//fP32CtXE mm8LfNELiEzJeYPACqJSEK2xYRcS8aeaDphVYSZrQkwbnSmaFOIe0wXGDAMyW5PrLiJHUyMdINMd sjjjHG1UIsKrgGRCPVB4kggaIdWZGMMhpOiQZLILe1nBqxBnnanV72lC//jyicnGxUMM6RDnTXDu ECJfQ+4Tkw1lt/+c6HWM9ybHA0+UlKP4icmmHkJS0CEhGHu4LClLSP2EFON93X9NjjqGgom27nmK SIvhE1NMSrtSwDGiLeYZE4Pk0jamxkdMMbXsKgpWWzmd9DmdRMan/b5Vmx4x1ZA/xDjJtvQ5nkwm ll0V4ZLTtX32Olec2b5YcZlSMSUa6/fFiK8axvw5n2oN4g4x5RETTaB9veHk5Jh5Pt5ak/y+5HCm 6ZhocjnERMm3ebG9Y5h9meKkUSHJOLsvU3z+HDLLlPfOhLIvOXz+OiaZSKcY2YHy0RacyXZfc/hc dAzWln0B4T1WMeSNo0OMJAFq8wzJ5rDPveheEdEb2t0CH7PJYV9fWLMCTR614P+LSZ+YYmzc1xdm UDE5GO/KTn0uhuoJNWnUEkyKu2riS8Frv11RA85zt69cuwZfrcbHXVUJFtbNHerB4yuChZvLcaMg ONi5racJHu7NHe7640yCx6q8RQ3wb+Fwtx/ZEQIsnTults7CQDB1+XC3H9chEHxe2MWE6Iyth9v/ EBLh7PLPXf6QYO3CKSY8YpJJe/fwrGWQZerh4XyUzJDh7Pzhcj9KM8oyxBxi9AsQCpwd7e728s6E CmN3sCjP5yzUDKtzinEqhiycHR127fE8k4W1swcr9LAB5IKh8nN2gxysHZ1itK0hH6Rv+xn7RL7I r7210zaN0IF6OvA87CChKSV76IoetpPQp8Z0aL0e9pYI1u7gUZ42mtC7WnuIedh1QvPq0qEverQF hH42eL//nkf3Qehokz3wPJocQpOb06kx0s0UFc6nn2vakNPG7m/2ozUkNLphn9TPBpTQ6EZ36Ise jW5Eo5vyof/y5RGDtQffUjzpGHS6zp1iJOQTkQ28zD7CJRWCNpfCvuIUF3QIvN7W18QAZ5cPJ2El rfx8uyO6Wrs9vYgeFv3LXqqNGhU9LOVdMYnoWFPYl5tinUZFx5rrrtzEBGcXd8Ulohv1YVcW/vPC e8g02JmK5wtTRiXCVPDIYMk/3n+87Bv/cIK9u4Muy08bkhrk7QWPWgHgTcTl+/2DZxJ+OPiNyk8n ThOLf3/99qfX35gUluJGWqX8wSTZGyn8HmdRHwnqO7sCbJpuwMJMCjdqCC8tsksXNzzfxU3OOOZ2 SOo7d+DT6CMK8uBWgE3TDdj+1aj6WIg/uTtzrsiti9kb2DGeCGl8MROX3D4SlHVmBdjuZAMUl1KY gwm8EEVKMRfc8osZjzluAaYY78xSuPuIQj+YFWDTdAMW6S/c3CRiKV/2O3eSJ3tyE25C4ikr7sy1 rI+U51crwKbpBmwdeWYlLbhfUtw4sRt3lNrMxlpxV66JfSSU+86tAJumG7DO9kTDg815UZAIihyH hlZpkmcpPZjcndxJieojwacLuQZsD76B6+TAUXskM/COKW4c240bjo+5i2Tcxe25kPVxHLeGa4pr YMIKdubcMyNLibmY+UG5mGtPtapTzYmP7WPEBe/cCrA9+AYuOkXfNz0ZD99YkA2Kng3ihx6nyNmG SWUbn8Acox91TQO2J+EAdhH5WUUALBevR0rcBaDk5psAJymHSaWcyyJARu6tugAF2BbGgeySOANW ALOMgMr24a6gyGMzFXigQEF45F2VE5AxztKuAdvCOJDhAMpQ4E3iAI90uStAw3RTQLKRmDz++FHA PcgcY5x7oADbwjiQ8YLZocChViEAn+fuCpKQTQVRMgkTKQWeS10fYxxFXgO2hXEgewQOBVbyphI4 7gpwmjcFSe4RpqQUSG/Wx1nrNV5bCAewdzMR8Xxy3tQIirsAHOZNQJajxFSUgMhVr48xjzdWA7aF cSDD1Y1EDLnnTQLHXYGX2zcVFEnnWAXsUiBZIElQRiHQcG3hG7jcKw5+rCwIyGC48zspPoOf2x3w J/tIw8Llr4/8mncFCrAtjAPZE651V4C94oD6SEMrhXcqcJKGEnxTEMT59THZUYg1YFsYB7JPuNZd AfIFbZoFklTroUC2+1LgJQ3lc+4KxGP1MblRDDRgWxgHMox6iFKRA1tnBMAVKQVZat9UANMBf42J 0/FSIP1PH5MfxUADtoVxIHu8WUMByiwH4CvrXUGS8j8VwHqwApJ0vBRIa9hHrr9dgQJsC+NA9qiT QwFeGg7A5ykFUd6/qSCKY8bFdkqBOK8+Jpp7oADbwjiRrbR+UID/iRyAz1MKSN7+qSCLi8MUHN0U ZLYhfUw0qoEGbAvjRMZzORSUnjj4PKWgv0BTQekOFjupFFS+h31MoxxovLYQTuAwE9Hnnjf4OiXA qUSsct3libsJEEsiQ0pzAxRcW/gmLkmXwvx8sAgojzS0s7di/mx7e2fFV178XtodGVOeO6AA28I4 kSMytyvg5EZA1WkoNuxS4CQNxefcFfQt6Hsw7bAGbAvjRE6o1F0BX/C3g2VVaSgW9FIQejMRdBpy 2zPHVEdB1IBtYbyQ7VDAtY4jvNicS0KWVm9KgCvJxBOn4yUhSxckY7ajGjwQ28I5oSsqZRYNXPE5 Ah423zUknNxnYaZ+ELD7+a6hSDckY3ajHjwQ28IJaIvfcBE0NPC7xxHIli8axsIcejqi37lr4MI7 xzyd8gOxLZzEX4LfDtkhGhy//RyBfPmmoS/MXq4kJlIaujuQMQ8JD8C2UBJ/CH57pEeXkGUTHcxs +SahL8Sd4DzA5JUECu855jDqwgOxLZzkpZeFNDs0sP3jCCTMNw19YbZyHTBZpSHW9xzzuBQPwLZQ kpPsQGsBuSKBPTBH1K8ZORamKjUhFUnMS0LuPpFEwtwGjdgWTrI9OxI+q2vgTgAR3n7PyL4wZamM mLLSILsgmzAfiAdeWxhxQQvMLa5KHAoC9ocj3Pd87AtTkrcBU7wrYI82x/xJSI3YFs7A34HfBZhd A7eGHOG/J2RfmGJ3bFEexUuDd+855jKzQSO2hTPwl+B3xc52Dex1/JuLK9AXCX1dom5b0cYpCUHq s4y5zm1QgG1hDPwd6AUcciOxArwoJAHIllXBWJdCt+5oZZWCKHZRxmJHgdSAbWEM/Bn47cXvsoKC 7OEAZMsXBX0d7Hft1j0rBam851imcdaAbWEMvvdDcMJDQUbucABy5YuCvi5ZcczJin++FBR6z7H4 IUDhtYUwOEkL+KhYuwAY0sgBhSvLIqCvg/vmpgFTvLvWVMUtylj8qAkasC2MwUpawErmoQCmXALq t0Qc69BBcuOEKdwV8LbMcfoVjdcWQvxdsiJhZ7oAbA4HQOm3POzrYP+5d8SkjHt3CUHo5wYouLbw YWMkJ3jq/MgOCfBfs7CvQ7hl/qhbl0zsFvtY4igGGrAtjOxqXeHjDEMBS+WAAMpVQV+HEyysgHT7 lpMYBBlLmvdAAbaF0fNn1J7R/LvWnjS4L2kV8D+2q14Hj9wG9nmKfQKf/n9qdwFyjcvDdamCqAiu uNfPDEntrqSFAa4/W5oZSSOKtGEwcYllevkRUCUNSWz2Oq544+ALXETXOy38NGvE+Awj91OAjsM1 TrHM2/wI6PGasTXdgQ1wHIyBq+ia10QBLywn4AJ/KdBxSGUhlpnRbgXNyRXU2DUbboDjYAxcRp+5 HQqYszgBOexLgY5D9e+gQLL6W0HQLMTYnSajDXAcjIHL6HzgguSezr9wAvL4lwIdhyeNHRA+Lb4V JFEgcW7BijcOQt/VFsBXAaidOCHxff0QoOPwrlcKqGLHR4BUJhq712SwAY6D0Te1BbBUAd9vTvDf RtRx+JWpoIgfHwU1XzPaq7zhjYPQcxWd5V5TASxiOCF++1DHocCLFJDFjo+A7q8Zu9XKG+A4GD2R u9a8okCbL49CFi/6ocDG8X+pIIkdbwXdleuOyZLBCjgORs9ldK38+bt19Q1X+aVAx6HeFh+G1Yc9 MBNazHoKG+A4GFHGOut+REAz2+CcvwToOJxpzXff8wiI7bpj0S3YAMdOiLa2WQco/MVcA6e7dvLr uMSW5G79Hv4crztajbwBjp0Q5WvBz2SWaFkKf8+r/sWv43itkfXxCQt/kRuoUT244Y2dD6Ur8kZl 5ZJIn6Tq98h1/otexzGtkR6van7TSz1msWsm3ADHTogV4BfeSKWP0oh6ZPn4Ra/jmNdJj7LioUcH cGnu20DGziHHj2pKfwVkZA4OsMfN6C7+kfcciZNNZmXeSQADekMryNaHTeh/qY1Vvg2QLWQXaUck gP8jDY8hNqlJYxP7SrP07B85WTMILSoWh7ryTcumFCqDV9mC90OKGBZYqMEqs3N6dq152ftJ2mW7 rEu8D815JgyNyRLXBjhWOoOFg4QbBuPQZgc6uZ0cgnEnL05JXnzzcEemS40pdeNeAMdKZ7C6algL A/FP/s1cu5zoZA6arnndX6nKJdZMGhOUKfMCON5kBopKWJjhMA70ZqLJ3PTkjTnKE5WiPFgPc2F2 0JgsR22AY6UzWLQBwg3DcigLsjc3XfhwJ3mgU5Ln+uGuzIwaaU7lXgDHSmew3XzmxBaN1n9z48Re 3FlehZTljXi4O9OCxjQblhVwrHQGi4QhRkPdRV801uJv8iSd2SSvWqNVeSJvci/1ocZkmWkDHBuf 4Xo5cGRPcQY7kTd3lJ5scjcpCVITxz3cgUWJRjvuFW4sXIaJ9KLMVZ3BHuzVHuD58i/mrlbrq9VQ +lwzsm9Q7gVwbHyGizcy6KaXHwHPSWMP+qZ30osZPU5RuhO3uo0nMGMOltdWwLETGjCKhFS7CMi4 KxgPS7wFlC6t2BTgxXL4LJbzVQRIzNGbgAVwHIyGjEKpmAIkfExAKdUWBQ0Aj4IgTQHu8eo7qQc1 5pnaV8BxMBoyisVmCsKPwgkBdnkrqNjDR0GSjcQnxFePGhxTnMac5x4sgONgNGRU7M4UeOQqTMDy /FtBEbKpIIuT8EmLgsBUpzFnS/Ir4DgYDTlgoilw4ht0FXFRgNN8KShyj/ApiwIpCDXOXL/ijYPQ gPH6mxHxfNI3aKzyIgCH+RJQ5SjxaYuAzKynMVd7Y1fAcTAaMjpHMyLKHvFNAcdbQZDbNxU0sXPu AvYoEBeICZolghVuHHyGG+K0Ieqt2jChguHN7yX5GH9xYsPiNhs2pj+NfM1VwQI4DkZDDgnXWhVg rzihbzZ0kninAi82lMkvBdHxEmoszhLxCjgORkMOBddaFcAvHU8EkCRbmwLZ7kdBEBvKct4KpMbS WLwlgxVwHIyGHJC1s2TkiCvDCaiKFgVVct9UgKKjZ35ox0dBZGumsQRLBivgOBgNOeDNMgVIs5yA Vfa3giLpfypA6UEFSez4KEi8hBqZf1XBAjgORkMOyJOmAC8NJ2B5i4Is799UkKVixsX2iwKpvDSW NPdgARwH40RmcycK8D+ZE5I0Oo+CJG//VFClisMnoq54FFSWIRpLsmywAo6DcSLjuTQFTY2D5S0K 9AWaCppWsNjJRUHnPdRYLB2seOMgnMBxGjFU9Q1WtwjwixG7XHd54l4CpCSRUMrcgAVuHHwTN0mX Qn4eLCa0zYZu9lbkr04aF3nk3/xB2h2Jpc4dWADHwTiRM5yrCmhuTOirDaUMexR4saHUOW8FugW6 B7McXgHHwTiRCzK1KuAFRy/qVhtKCfooiNpMxNWGbHtmLN0S4go4DsYH2ZkC5jrOCFLmPBKqtHpT AqqSmvihHR8JVbogidVZNtgQx8E5oTsyZRUNzPicgRq2vjUUnNw9sCY9CJT79a2hSTcksXrLBxvi ODgB7fAbVUQyDXz3OANu+dBgA2tUO6LfeWtg4p2xzkp5QxwHZ+JK8NvDHaLB8+3nDPjlS4MOrEGu JD5p0aDVgcRqEjbAcVAmLgS/A+yhEqpsokcx274k6EDcCfoAn7BISPGasUbLCxviODhTkF4W0pxp YPnHGTDMlwYdWJ1cB3zcoiH3a8Zql2IDHAdl8uIOtBaQKxJYA3NG/3SkDSxdckJpYsxHQtU6MYmE uQ0r4jg4k1N3FCxLNbATwIzgvh2pA0uVzIhPXTTILsgmzAdiwxsHIy5oQ3GLq5JNQcT+cIb/9qMO LEXeBnzyWwFrtBnrbcgVcRyckevA7wZM1cDWkDPCtyF1YMlasWV5FB8NwV8z1jbdsCKOgzNyJfjd sbOqgbVOuJhcgX5I0HEladmKNm6RECU/S6x9bsMCOA7GyHWgF/DwRqECvChJJsAtpwIbV6KW7mhl FwVZykWJzVmCXAHHwRi5DPwOUu9SQYN7OAFu+VCg41B+dy3d66KgtGvGNgvnFXAcjDFoP4RK2BRU eIcT4JUPBTquOKmYi5P6+VHQ0jVjCyZgwRsHYfRiC9RRuasAFKSZExozyyFAx6H6ZtOAT35XraVL tSixBcsJK+A4GKMTW6CUrKYARblM6F9GtHHoINk44RPfCrgtM856ZcUbByH+XVxRsDMqAJvDCVD6 5UMdh/KfvSM+S+GuVUIU+rkBC9w4+LAx4gl+lB/ukAnh04U6DtMd+fPautTEalFjy5YMVsBxMLKq 9Y3HGU0BpXJCBOWpQMfhBBsVpLV9q0UKBImtzHuwAI6DMXAZXR3N372raXBfyinAhsHEJZbp5UdA lTQksdnruOKNgy9wEV3vtPDTrBHjM4zcTwE6Dtc4xTJv8yOgx2vG1nQHNsBxMAauomteEwW8sJyA C/ylQMchlYVYZka7FTQnV1Bj12y4AY6DMXAZfeZ2KGDO4gTksC8FOg7Vv4MCyepvBUGzEGN3mow2 wHEwBi6j84ELkns6/8IJyONfCnQcnjR2QPi0+FaQRIHEuQUr3jgIfVdbAF8FoHbihMT39UOAjsO7 Ximgih0fAVKZaOxek8EGOA5G39QWwFIFfL85wX8bUcfhV6aCIn58FNR8zWiv8oY3DkLPVXSWe00F sIjhhPjtQx2HAi9SQBY7PgK6v2bsVitvgONg9ETuWvOKAm2+PApZvOiHAhvH/6WCJHa8FXRXrjsm SwYr4DgYPZfRtfLn79bVN1zllwIdh3pbfBhWH/bATGgx6ylsgONgRBnrrPsRAc1sg3P+EqDjcKY1 333PIyC2645Ft2ADHDsh2tpmHaDwF3MNnO7aya/jEluSu/V7+HO87mg18gY4dkKUrwU/k1miZSn8 Pa/6F7+O47VG1scnLPxFbqBG9eCGN3Y+lK7IG5WVSyJ9kqrfI9f5L3odx7RGeryq+U0v9ZjFrplw Axw7IVaAX3gjlT5KI+qR5eMXvY5jXic9yoqHHh3ApblvAxk7hxw/qin9FZCROTjAHidjbFJE1iyd oX2M8beff7Xr56/L8Q8f+18/f4eMf0LGf+wO/n15d/3r+uNPd/37H+yS2K9WprAEXf/nvGxydMdt KDqvVdQSrH95GUEGWUHPHgN0738Q3kvKlsxZ0IC/Zj3yWJYo8lIXOnWqxBSFefYPXoqBwR14Gnif n/afn39zaRPdrK6l6c9lC8WR/J9LO18k36Xam98d3v65j+1LNL7dw84xq633I9FeQ6oeAv7OXbYc ebPGNeTpKQGlB9bNHobWvjItYG7oVQLei/hxlEBS+4KtafOQNYmQK3UjU/Le1W5aetCbpwTUTEw3 LZjJR9CpG6sBN7Nvobt1j7pV8I+nBNTU5Zs9HD30WjBgbGirSan6Cwx9eEpAqY0ro+2rPOisiap/ Lxu6WuebW/P/eEpADcyr1XqYoTvvwExMxIXmVkBYWmMz9OEpATWgjmj3B31ZwNjQean2Vzd+PCWg Bmbcar3f0E3frQFQdy86VR+KoEhuRx+eElAD0oD2dDRUMwPuDX1x1cXWPhx9eEpAjc5/oHJa6ISA Qj9Hz5sKKnFDWn7Qm6cElNr4nDrYCIBGbdKAyq9b6GlTwthG1dNRAklL7TB7OBmfx4C5kSn4xkyu wA19eEpAeRWvaAELnTQFZmMiLTRTw4e7VB705ikBpW216D7qiHANQyetiBrQvfkYurVncsWuGPrw lIBSwTDM7o62AqoBY0NzCGCJGnNVkMNRAkltH2/yQ85DAwaHmEUuXG7i0p+SenhKQLnKapcPXipC 9aUaYKJsoTk7dSbgaF5BTk8JKBWQOP2WfK5FwGUBfUMnrLpP3+9F3hwlkFyaNpy1kVHWNcCk7CJf WG7vvt9GPhwlkFR2d7OHk5GTDJgv2S5Kr9yO5PXj9JSA6jpdaVo1XKuFVlHzqxNAu1/05KIzmte4 yoPePCWg1C6wUQsMjeuvAVqBN/Tgqlk/+iqop6cElE5Cw+zhaOhCBowN3XGIkJH46HWKh6cEVIc4 VVvnjgedtXDqWFg2dENqtDH9LBd685SA0gmyFrU7BzCg0SA0QLM/vWje78Yy0ldBPT0loLpOC2b3 B31ZwNjQnHBaJbqMB715SkCpnWEPDq5AoxlrgBaaDZ1xjI3Pnr2CnJ4SUL1pqtDGWAg0JIQGNF3S i05Iu5ZQUfu18vrwlIDqVYu52lBBC50soG/oC+utk9tyzwe9eUpAdXyO2tBuhk5c3Y2ve9D9xqor S2qb6xgPTwmoXlgp1R6OxrVmwNzQXG9t3PF1ioejBFLX5qaFsmNJi5wQMPSrNjJlP7ejtYe8OUog daR7/+2Q9rBviHn1R/K/YCNTPCH/QD4dJYB6Yo3Sn27gyX5048JuYHAvXpli1eP0k8DpiWVVX5cX F2X3Ro3JD5cyrEzuceoLvDlKAPmWcIMIHuwU2EE0jocM1VEGnrbFHz8JoIbqrbZWVt15gi8P6BuY 5blQJFTXSx9PCah2c/GaiMnQnS0ZmZp3NNfMVl69a308JaDa5IarPQzdeCAIwPks9MVVZ89okk9H CSS/5X7nSU4MyMyohwxoYb+tPhZ9PCWgGiSJ2iqnDF2Z9ihl6UU3rDmzKa5kPv0kgFrnTVR7GLjw piJgbOCJNbOKVp+JPo4SSN4EvCWQnBlQWWwWGczcql8XJ++OEkgNelVt1VJGziyJaHRpI7NVZVaN 4h384ykB1SqVodp9oS8PGBu6Ie8yG0DxkejjKQHVKtuLaoZs6MT2A1FRNjS7YL7wLN0P8fSUgHJt 40pH0RebJqQQeuhCF6w6sUoX7+AfTwmohumj/w6rvEQnD+gbOmO9idUU19HRu6cEVMsUIKoqE20V gxAokJ15Q7PBQtHhOlr5+HhKQLVE2aT2MPSkrELA/RZT3JNUkIDFJ6KPnwSQ63VX7wQnBtxUhg62 zp0Sc9snoo+nBFRDmo5fnTkuQw8KWAwlaUeDzBKSfSL6OEog6Wg0zO5G7lTdCBgbuSOdr9ntppN8 Okog+Tjnwx3JedRfTH9VO8lDRuJd7InZB6KPpwRUxfBB+za06h4G6D3b0JWLtvtYLac/nhJQdehn qd0owom+PKDvaK66NFaRRd4dJZDUxqQ4tLoaWb+mIUBL2kbGZbk4yuXkZ3g6SiBVjB6070Syll0P uG8n/w2vWh6vMmjpT9JD++evn//8/vfH5Y4XGb77REl4V8Vsorbm2Xp3YoAW6PXuf/1cgBK4RO+k Yiqd0e8q1Dv94j/quJx4SvoyHJoAPvV/ClTYsJchUerFcFWI+WLGjMG/+5aiIetzVj5dMZ+eElAz 853VZlRFJ6gQBswNjRt3d6TB7Rrj4yiBNDMTpSZeRCMnTdKBTHzJVOE3U/ju6UFvnhJQMzFTamZ9 AXqyH43bM8XQvHJ34X64yPh4SkDNi4lQMcMYGiqEAWNDN646VX+BoQ9PCah5MWUqJq+FztqPJkrV i+aVm7bjrjM+nhJQ+s6qy6iYFw2NEVEDtBmkF82tmJNrd6Hx8ZSA0kOYZvcHfVnA2NAcHyblxrzT g948JaAGWpfaTX8MDSWiAehkL5q7PFn0pwuNj6cE1EDDpT0drQI1MeDe0Owhs9jafVw7PSWgvCR4 gTB0QkChn6PnjcsyEzfEhcbHUwJqdH6O1r3kaCgRDaj8uoXmAeKSYVucfDhKII3GM1B7OBmfx4C5 kQcWPSYzMC/04SkBNRozRxtMedBJU2A2JtJCMzVGz74rC715SkB5I/S2CHRitdXGifRfaI5Vo3bf FUMfnhJQo3BgUbs72gro7Ly0C12ReyxRY64KcjhKII3CIatdbF5GxhQ2B4eyRebAhilBn09JPTwl oFwnuWpS9HVzdpyTo+RCZyy6MwGHS42PpwSUqjucfsO2LfRlAX1Ds2FT4mK/F3lzlEAaGELVxlkb GWVdAzCTvmTOgr37fhv5cJRAUhXdzR5OHtRo86Zkc7JdlF65Hcnrx+kpAeXa3ycBQ0NY3hd15kJz zOwZzWtc5UFvnhJQahfYqAWG7lTDOuGkDT24ataPvgrq6SkBpXPWMHs4ulHDa8DY0BShEGv46HWK h6cElNoYK3RWLA8ac4eOfGVDN6RGG9PPcqE3Twkon2l9wgW6clq6C4enheb9toG2r4J6ekpAdRXv ZvcHfVnA2NCFq6YkRWda6M1TAkptDJ5tcAwFunAyvSsnqYXOOMbGZ89eQU5PCailHaEnDA0JoQEq vcuLTkg7Gw37tfL68JSA6qr/TZa2B50soG/oC+u1Ca7d80FvnhJQHZ+jNrSboRNXd+PrHnS/serK ktrmOsbDUwKqF1ZKtYejca0ZMDc011sbd3yd4uEogdQz56GOJS1yQoDq/LaRAbYsae0hb44SSB3p 3n974gDw5+eGmFd/JP8LNjLFE/IP5NNRAqgn1ij96Qae7Ec3LuwGBvfilSlWPU4/CZyeWFb1dXlx UXZv1Jj8cCnDyuQep77Am6MEkG8JN4jgwU6BHUTjeMhQHWXgaVv88ZMAaqjeamtl5UyrAZcH9A3M 8lwoEqrrpY+nBFS7uXhNxGTozpaMTM07mmtmK6/etT6eElBtcsPVHoZuPBAE4HwW+uKqs2c0yaej BJLfcr/zJCcGZGbUQwa0sN9WH4s+nhJQDZJEbZVThq5Me5Sy9KIb1pzZFFcyn34SQK3zJqo9DFx4 UxEwNvDEmllFq89EH0cJJG8C3hJIzgyoLDaLDGZu1a+Lk3dHCaQGvaq2aikjZ5ZENLq0kdmqMqtG 8Q7+8ZSAapXKUO2+0JcHjA3dkHeZDaD4SPTxlP+RXe66luQ2FM37K+oPrPcjnsyADQw6mMBw5kkM 0MCMA/++uTelOtIhGmBf3kuuUqkkctOhamF7Uc2QDB3ZfiAq8oFmF0wBNrf1Ee9IcailbZbSUXRg 04QUQg/d6IxVR1bpvDr4V6Q4VMX00Z5ulZfouBLagU5Yb2Q1xXVc6DNSHKomChBVlZG+ikEIFMjO dKDZYKHocB2tfHxFikPVSNmkfjf0oKxCwvwUU9yTmHEA85qIvuLEgZZeX+qd4MiESWW4wNa5Y+TZ XhPRV6Q4VMUx7Y/OHMHQnQIWQ0k80SCzhKQ1EX0FiiPpaNTNb0ZuVN1I6Ae54TiH0eymk3wHiiOt cW4NdySnXh5Mf0U7yUvGwQvsiWkNRF+R4lAFwwf9aWjVPUzQe3agCxdt97HYmf6KFIcqXV9L/UoR TnRYCe1Ec9W5sops8hkojqQ+JsWu1dXI+jYVCVrSDjIuS+Aol+L6hnegOFLB6EF/RpK17K6EORf5 D0SV/EblTk//i/rR/vz9x2/Pf34subOKDJ99o8Q9q2A2UV/P2X52ZIIW6P3sX3/85Zf/jueXn8/9 gJ+//P1HfP76xOff+pup/e5/egeevz3/+Gd4/vXq40FxlRsf9Fnwzx+BpNXNYoK0iVgZvrBg2yJe FJKt28qwxIIbDT+mwCXq0jO3kftvW28WqtLQV6Q41EjcDfWnoePkdmnCPNA8LmZxNom+I8WhBqqX +lDhG42O1cc6VobmITe7JclXpDjUiPxDSaxGQA8e3T7XuTI0r6ZZlAFDX5HiUCPwddQfeaODJdix MTTriVlcwo0+IsWhRuBHKJnnF+jOKjHCOoFEWxU0i4pr6CtSHEqfiaNTMF0aGrJFEyJP0kazdJst sy30FSkO1Serg/rtRUdLGPmDZsMxi9620UekOFRHBVO/apUwdKUI0c9dDnT9tMq6yVegOFJHBaM/ F1mrJBK0eRxkdnaze575ihSHUh9lYlUTQ6fMhHagrVfarJvniz4ixaE6flDfiiTQiNQEK6MLbSrK bC377F2R4lAdy6c/XnSwhHmgqf3M1jpf9BEpDtUrn7naEdD4KppQuYSNpmI1W9s+e1ekOFQvVJmr iQIN4aIJjRu30dTZZmvfN+aKFIfqhfVstX5DR0vA597ovgeEOtoLPuLEgXqmnl9qBWC7EaNzmNtg jjNm69xrviLFoVRUYQhZEkvRYXJKGYNz50LbDGa2BWvEX5HiUD1xcqqohhsdLaEd6LSnxxb7Cz7i xIF6tFZoUhbgwWFwid0N5rBrtqW95itSHKqj09CfC905wg40pg+6cjtoUXYMfUWKQ6kfVVStscHQ qadnBorSje7x2baV+qKPSHGohq6uvg07QKPta4KNQxs9KOBpW11H744Uh9KhbJo/XnSwhPlBYyDd trXyoo9IcSj1IWDXYAm0/oSEpK/1Qae0hRJXaugrUhyqdcruqitY6EJdPjMnrVddjmfbNvaqr0hx KPWD+b1udLSEkT/o2p9t24wv+ogUh2qNA05VYbXQmRPQLLrID7rnZ9u+hOlXpDiUqkeMZerPhU6c 26bq9APN3mIWlc7QV6Q4lPoYJVWY1heNWVMT2gdtw6nZnvKLPiLFoZrKZvhTj7+ho0ZqQtPX+qCt mNL2Tb4CxZGaKnjzx0sOljAPcrGGyK0u6UUfkeJQ6ietsA3fyNAQxpqgQ8GBbkTT9n3Nr0BxpKbD lJ7BBmkPf2oBRvzQl/qA+3i27c3WfAeKA6kfzO8bHC1+HODJZki75OkdJ46jP+A+6q+zcQfvq6bh 9i5ui/nZtm/wFSgO1HA+6Vur0S3rWhmxg+Egp/ls24dJj69IcagGNat+ZjMjOq4EdLuNLvXZFj16 oc9IcSg9IqjXehazoRvrOU5UPtCtPNuOsFZ9R4pD6ZXBlqs/DV3ZhZAwD7RVOoCj1f+vOHEg9XFG 1vRJMM4Qrm49wJPbQTvS2OgzUhxKKwj6uZYqnHJFF/Z71DJcg4XG1m2LvkH0HSkOpSV1mj82OqyE eaDzs8woY4PPOHEg9VFAtHIXA2cWGJT2eoBLe7Yd1W73V6Q4lDavBn+wBipa5RQTKovkRrf0bDva Onh3pDiU+sH8vtFxJYwDPeKz7Vg16StSHEq7OYqTyoZs6MjiheZfDvTsz7ZjrFXfkeJQKm/QF9Wf hg7sm1BD4YMeLBzLzrXqO1IcSv2ofg/s7USnlYDmv9GUpWZnSBt9RopDqd5D81JhmeirHmxMUIly oMt4tp3RpMdXpDhUjbwB6o+NDithHmj2FLMzpY0+I8Wh1IfeU6VdDD2oByHF64EeRNPOvFZ9R4pD VdPIOnhAsypadw4JMVDUbvTkhtDOYtLjK1IcSv1gft/ouBLGB439fW3NG31GikPp7JV1HtFJLBta lYS2PoxqRcX5RufyvLatVd+R4lA6inbzp6FVUDFB79r8oMt8XrtkzVekOJT6Eb7GxWLouBL6gW71 ee3Y5DNQHKlohYWvL2fkQimgm6hv9SEP7oTZuc7eHSkOVRorg/rT0KqPuyXMjf4DUSW/URkaqeO/ qKl//v7jt0cF1IN/fOaNEPcM9ZHYVdzn9czEhELhy2f++iM8+LelKiRUx7hUFDhALs9qclTB6Grr 79wfhUHKBmo3bA9o+hIGTJHf8AO0hIz5q9uKqvol8PCojEwQXxFLxvFaGx4ieoxZDDKGviLFoUbi LhRcCqLj5Db1znfa6IzzYbYU0yFfkeJQ6uOjlshPbGh89T54CDa6oseYxYy00UekONSI/ENJPJhA D906TZg8uhvduCG0pe1VX5HiUCPwddQfeaODJdhxMfQgmrb0/qKPSHEo9fERSubpBbqzLIzAb7LQ eryebTF+GfqKFIfSZ+LolMLKBTQGSU2IPEkbHdF0zZbZFvqKFIdSP5jfXnS0hJE/6Ew0LTTNRh+R 4lD6fQv8qk3O0HpzkKD94kBXbght3eQrUBxJD+Uwfy6ylkUkZH2rD5n612zNa8a9I8Wh1I9V/caq Y+iUmdAO9CB6GHq+6CNSHKrjB/W7FhJDI1ITtF3UF50Ceq7ZWvbZuyLFoTqWT3+86GAJ80BHtBiz tc4XfUSKQ/XKZ5bBmwA0voomVC5hozO2wmxt++xdkeJQHSelPgXnxtCRNXk0btxGV/QYsxh2DX1F ikP1wnqmfp8bHS1h9A8a55mmjvaCjzhxoJ4p4ituqIHtRgxc4fyCB2qp2Tr3mq9IcaieOPzUsOZT CC2Ml2NwVFnoHAA120JdPeuKFIfqqH/qRw5Pho6W0A407iBNi/0FH3HiQD2yGVZUcAMPzoADJf4D zqikZlvaa74ixaF64GBa0XcM3ak0xuQgu9GV20GLsmPoK1IcSn3IyopuudHQnRNf64Pu6FpmW6kv +ogUh2ro6uoXSmOg0fY1IVI8b/TAoTPb6jp6d6Q4lE5i0/zxooMlzA+6BHYt2tbKiz4ixaHUh2LV ka8stP6EBJ1LDnRi16LFSg19RYpD6dSLcanqCha6cJ6ambp8o1k5zLaxV31FikOpH8zHyGfoaAkY Cje69mfbNuOLPiLFoVrjTFNVWC10/j/ZZY8DS24D4XxPMSdY6P8ndm7AcOATOFsG3vsHLhalHvYQ C/Rbvlf6mqOWyCKHno3J1aFn/dznTMfuvZUSUPCPszPeB104qsFgJodmb7GnVrprTZ1SAgpx1nhx mjR0qVwwvmgtQvc5S33QTikBNeDtNd44/obOUGIBvH3/oq2Y8jkv+SWUQBrw1xavh5xswXbkZg2R W93Kg3ZKCSjEBRV26DcytBpjLNCZ4oseRPM57zV/CSWQRsGPGp+h1l7jvTk07HWGBAPP9bnPOSzn t1ACCHGyeF5wNv1y4M1myOexp2+dBA7+R+8j/road/G+Ypne3sMduX7uc17wSygBNPR8MrZWgy2b qIy6g8mRy/7c51xmPX6UElBD3SziymZGdD4LtNtddOuf+9QefdBeKQGFI6L1GmexGnqwnuuJqg49 2uc+VzpZv5USULgyuuWIt6E7u5Au2A5tlU7B2er/j04CCLGekTN/EqxnSK9ud+DN7eBzlXXRXikB hQqi/RylSk850I39XmuZXoOD1q27T+0bRL+VElAoqdviddHpLNgOXT/nsdq6YK+TAEKsBQSVuxm4 ssBoae8O3MbnPle32/2jlIBC8xoaL9ZAoGGnuKCzSF70KJ/7XOMcvLdSAgpxsnhedD4LlkOv/LnP dWrSj1ICCt1cixNsQzV0ZvHS5t8ces/Pfa51sn4rJaBgb7QvIt6GTuyb6obSF71YOM5zn6zfSgko xBnxTOztRJezQJv/RdOW2nOnctFeKQEFv6fNC8ayMIYfHFwAi+LQbX3uc2ezHj9KCaieeQMQr4tO Z8F2aPYUe+5SLtorJaAQq9+D026GXvSDasW7Qy+i+dz1ZP1WSkB188gYPNSzAo2d0wU50dRe9OaG 8LmbWY8fpQQU4mTxvOh8FqwvWvf3efZ60V4pAYXZq2IewSRWDQ0ngdano1qDOb/o2j7Pc5ys30oJ KIyi0+JtaBgqLsBd219025/neWzNj1ICCnHWGLrcDJ3PgunQo3+e57pkL5RAaqiwGuPHGbnRCmAT 8au+5MWdsOc+Z++tlIBqg5UB8TY0/PG0Bfui/6eqVh9VVY809Y+MpX//94//fGCgPvof3/lGSHgH Yl04Ye7reWfhgkbjy3f+Cz8Ic2M6ruff//gnHas6qalTUwN36Qva5/Q6mmFtbuffuU1gqqNN85pj B/jrj3+7rPUftdp86T+rz8/q6xl8Cw3S1DrXno+wy/cjrPWgnVICauksyXhM7+KxQE/z8315cPyZ /BFKIC0tkE398Z3FMu/cs/dGZh8+x33kx2o7pQTUKvycr/lAmysWbJf0yN9L2vqDdkoJqKW1l/F4 stYiNLU4f9GdWeflCuKPUgIKcbVXmeNUw6huAwu6y7rl03a+ZfxHKQG1YOPwqpFYgg2tVXku/v2r ap0ynsaDdkoJKMT6zpHotL5jzdysHBddrjH5duMfpQQU4mqvsolyb3apublvryq+ejmpn/nDCSWA VuLX9QOTWlHorQoYOI1jLZ2jegklgFbiiRz5DKp7s2Xjr/M342V2OL9c4EsoAYS42ovmnZj0OuOv +zdjdVA6G9CotTswOZ0Ezkq896PQrJCrlnElVoHLnbTa6+XiX0IJIGyR1irEPR9wxUCBL5FdwjwP c/rB46WTwEHc7D3zJqwOZmlT/XI7E+5nNw7X6SRwECd9T+Vo9kyOS/3Ql0tfNnlNfgbHI5QAwsXR f0Bstmerr4e+UPY4Sg4GmRPpKaBvoQQQ4mYvegZSnVGgby7jwoxTPcPBATuhBNBc3PuhM88ZR5cl MlzGPL6DzeTOBC+dBA5qqp4VxGdGQufWPKpvUjrZ6pR7N/qAnVACaE6e7tGuax/04qv6FmXXYrBO zFIv2AklgBDrdUTzTvUMucsScR1KD4L5z0SPcGZcJ5QAOn5j9OPgt9oD6l1/solrZAWPO8G8hBJA iLu9aNyM1YDAMLjuxAs32EHGrRMvnQTOHKzRiM1QbjjLTL1rTTZ99jXOFyT4JZQAgpFkO9EvccA6 /UHvGhNvcmfWfVysk0nAIGYrwc+4+eo0Ajtbv+VnNubbuc3HALyFEkCIq73oDJ+VzR5615PoWXrV rEc5ZeKlk8ChVUc8zwgDfd/Uu5ak5xbgco7F4TqdBA5i/YJDffHhVs1j+I6k9x5cnop+sU4mAXOG FsTbig9s0qbe9SMrO6yZ/ezCSyYBg1gvyEDl6Ac7LQ3Xjgb7Zxt2KNYFO6EEEOY3vdKI+82XbWb6 dmTVrNFr9lEv2AklgBA3e9G8GbPPTJasC556JRr7hh63A3ZCCSDEWjYHzJFlnK3PTBbZCx7MmBao 17PHL6EE0Cz83ohPQ8rWZxa7wgV3zbjS1/dbLV9CCaBZ+D+I582YfWZx2QU3/WyVHlON9gE7oQQQ Yk19ptuQkjUanSS+YBbgSkfRzoj5FkoAzUxvM9PtSMkazeanueDCjHmM27pcp5PAmdmmvXQbEmxd pl7P0uWysNdaeJj3BTuhBBBiPc/4w/oIhMsSGS5httDKBt26fbu3UAJoJt5AxHUfcLdE9jfjzuNQ 2UhbyxfshBJAM7FiIB43Y200+KFaXx4wiIXtrj35OpkEDGKtb4itjeATa5vZidXwYqfmW8Y4d4Tg l1ACaGwW5FlOI4FeG822me4BK5dVKJWLdTIJmLHZPxD3m6+2GZx8/eNibR9YJuqaF+yEEkCI9Q2I rY/g2msGOzOfB6xbW1jl63Frb6EE0FjcEVQPayTQ645Bv13GVTPOmwfuVMy3UAIIsX5DxP1mrN94 6+TgwJpxZkGu59a9hRJAY/HMIbZmgmagZ3IXntALZgvN9Je1mpl4CyWAxuItQU+xZgK92jro17f+ aDEDmGWilst1OgkctD291oj7TVh96K4sAper1Fp58fbFOpkEzJgsQ4jXPFj1zbv6ltSWJppZg8qe F+yEEkBjsm7CulgngemYlohrSdyATHOpN5rcl04CByZICz1im+Gg1165m+9IWiJRtxfB83KdTgJn DPYRxDMfro5Su/mG1IaetMQiUfq6YCeUAELMPjLODAcbuiyR+a0+5iOSXb12TsRLKAEEW6zdH7HN cNDrT9ndNyQtvYknSYvF5TqdBM7o1kfGGeGgVzsDvWtITQ9ZYtvQUnG4TieBg5htZJ4RDnPJsjxc P2oF30yh6ST7EkmAjGYtZJ7xDfpuSXx7kQIJ4+Gk39WjmuGe8tI/tFf9/V/V5o/+p1qcSmx/uenL /XldHc34pt8Xu+dfXNBm+abG37O3nsHN2rTXetBOKQF1fkRfNIOGLrV8f6Wh9fJsnvQ960N2Qgkk xOrRu5oRI6sb4QI3pW7OJJtnco+7Hy+lBNQqnCq6NuOL1jqHBW5O3bxBm21rnzHqRykBhbjbq8aT ddFclp9UtUOpoVjnBRftlBJQiKu9am9DK4IL3LC6eYs2C9Uu5aBfSgmolW3M1It30b1zgRtXbSsW Pe5O40E7pQQUYg6aakwuumou8O0ua71Lfy663LX7g3ZKCSjE1V61u8ZbzQn1bmbdHCIWh0tNnXb0 JZQAWsmGTZjJfcB9U++GVt0B9eHjJH7ATigBtBJP5FBzcsAVieCv3dS6OAcuFqzV+wU7oQTQSrxF Q2ckghev80p+bl2L20ufu+zQvXUSOCvx3g/1JofLeTT5wVW7aPpzLqZd1gU7oQQQtkhr1VBvcsCc R7MfXO08zPl/0qumt5HciN79K3gKWsEq2x9ssjsIAiSTHDZAcoh128mhY/fY2hElRS3D8N/IL86r V6TMBmc3BvZid0lPr4vFqnpVMRyRN8OFgge21ff45DDX0SZqp/L2dLiP0Yi8GS4UPLBFypwMJ+T1 uo02FL7EyyHBs0z8EFefFTAURCgc+cJ1aW31uo22UTqVmFnmm5Hh6BJxBgwFEWyrL4prq9dttI3a qcQtPa6lpL1rE3EGDAURbIm9s2ltdbqNtlE8lZjp6ygmcTJf40LBg55KybNpa3W6jXa5SA1cAp1L gY7EGTAURLB7fZFLDrcd8ZlEaVk49gkfx481MBREsKUcXR+3TUj5oI5kCiWJINXKVG5itq2AoSCC kksDgd36SMxttMv1yXOxdBx1XeyZa2AoiGD3+iKXPJZ9FANDpk4sOEcFcalPrHCh4IEtPdr1cd0c Lbck4DNpkgRDoxlcvEESr4ChIMJsSjlxceEEXvY64DNhYiVzeXW9S7QZLBQ0sCklLq6bwMseigG5 e28/3tJfTrouDgBrYCiIYHf6Il03sRyI2AOfaRJnlp6Drmtjm1jhQsGDeZBK4uO2Cbxso8BnkuS5 BDKFJS0ib4YLBQ9suUHYffJXllGsTJkiSd3rxubqPtFmsFDQwJZVE7aumtgVZcgFPtMjbTvsmX2M wgoWChrYUiBYOHXTHFtVGZfLkaN+WqdJMSTiDBgKIt+xpGH3yV/KjM/lSLuZ7j+96xJxBgwFEWyr L/LJY+qMz5dWGXhrdltNt0icAUNBBFvaphvjsjk2qjM+X1qdo8ccgfouxngFDAWRb3nfsKMgNaoz Q761Sp3JqMC7S91yBQwFkW/5ANsnj6kzQ762Sp2BmDOmDNqROAOGggg2t806CVKtQiObxDsxG3DH icKOkXgFDAWRbzjb+DopUq1CM+Zrq2vpMdPYDok3w4WCxze6btZJkDDWNcRna6s29q5rmcxjIs6A oSCCzZWziToC4KCOZHuro4R2FGjbx911BQwFka91d21YjyTu1ZH33ZW7KYgppNY2iTgDhoLI1+wY sF3yWIQGB5X+ciMGY0u5szd/M1goaGBLf4OtMoIrFpkZa3bDROvF39a5WCMkXgFDQeRGNmTfRiEB XoRm1J3uRiy87EJ1m2gzWCho3Ej9gN0nf0VmkPnyL9FqHNgmusEn4gwYCiLY8gbYqiMoe/FgbOjP jVhC27LLd3FaWwNDQeQGRgTdQ4UEeIkY8GPmcSceNyMTLnbMNTAURLDlDmH3yWO541E2h4xYPG7Y kLtYdWtgKIjcwJyDrWICMZCcHFtmaCKmhDacL7tOh4k1MBREbmCVQFNUTICXsQ744b3/SDMDMdtE 1ybeDBcKHsielDXsPjksc+jYsQkkXmHtOhbemGgzWChonGcbgj34SCtz89jlkmQHcbRhD2pHn4gz YCiInGffxOiiSoKhw6sjmSQxAA2HS6lo8q5woeDBECSNHrbucMCLVo42VyRpkejbA4l94s1woeBx jjoC2zeRV1ap0eaCZJ1kWs0m0fZDIs6AoSCCTR1xcYfDGDqoI/69++gcUWvp2ZgRK2AoiDAWi/rD 1h0OeDnK2OeCJK23ZiZJs0i8GS4UPK5XHXFxhQNexhngM0GykmQ1ZUNaReTNcKHggU0Z8XGFw14y qB+ZHtkWdyakdXR2BQoFibMqIT6ub8D36sS7Fgmhuf/0jzsmJ+ddSdUG01MzyD/Rqst8d3/3593d 97udNY3ZfcHvOhm7kC3b9FBLfYrWNB1aYY2f7QJXDt1s019xvDa7B3xV1yjE3etd9ekUwnR8XL4z YV6W6WnG03Vavi4Gn+LpadnsfhJfY4d0USy24LBsF/y3+8tHqXAQnw7SoH+PQ2+26QEf9ZhSB8ya EI1mRCzlJIxU9P717g913Q9/VCqXqFrZJgezjf8zIiQnmMTHoAdvJQTb9Ph69yM93wyVOG9+Y/45 L+fTcZnh92U2/GKzbarzeX98WszpC+7s/gee6WG6XN7M5l+7v8GDbUNxQCSU290i7Rjp6Wj2x/11 P11Pl+3r/nE2L8f9f15micwHAtO2zEK7Doy84uMxuXEUMXHvMXEak93zDH83SPBKnN6gOqsr4gD7 dNlgg6jEcfNMxMyvD+cF+aqP8Qez0YcDfoG/5hrxJsKUcEaQYc3Hq8RXyfV35oFXgyeJNyO9++37 5T3c5df4uSIaUsIbw4+Oj9+Z0/m6Px2ng3mcrhOv7RIv+POGhP8v8Khd1/vmVwU+cRSBt7fAx0P8 PdYOk+9lmVE5p9+Xflph9vAzPqArNB1U0MKJFh1vrD/kqPRg7DHb9JDTWEiRRQiiq2hn4PqxCpsG 8xUijJZTHfe4rbE6vxz4f7ry34wbaytj1JpoLbgXW32Ve+mrxWyQRrjhvrpscPoBF03Us34/q/X1 exIs83WDWbSKfAmjL3l9Js3pkL/5/hOsobr/QQn2/PntQ6QgMGf1K3EKiflc8Ts91HFV2egkt9K+ ZR5GD17a6/P+YePA8IyewIcFdZ1OhIDCMfn0DUGTs7JvTIcFb/OIAxFGfxh/Nm0wNlVP30rRb1w9 5Gf0cOtXXv2NJrt6nrhp0onb2B4+Pc8PX3mLx8MbfIbQyVHxehxVStzs2Sb1m4scUWp4QZyH6qzA 0+Vq9vz+qB8IH0LJjx4kLywDqj9CQJ/nN4E8no7/VV6JtryM4Vy3h8bdzl99rMNalKsb+rLD5vGz P1vjt5/nNd5waujG4Saauf5gIhD3/iRd7nzYT0fEAw9zQCecpGuZ8LJczfJyPkuspsOB/ZCtEO78 dXdnWwzS2OKcxTEsTve71mx7zAW+dzI7fJHZ4VvO9i2XnV8SRx9l6zHpubm+ndGY/v1yRdyP5uH5 dJI0P5n90/GEbnWeLlOYr/MFzWu5edhjoMPU5PyInLLRQ8xHsrL9kocDqs7Va/3+mQCeXq6H/RGt cn+UJDGPl+nL1Zwu4P9pfrjKZyG5I5ObsZLgbXSmk6EJ6w6d+d8AdC9JcwplbmRzdHJlYW0NZW5k b2JqDTE5IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA0OSAwIFIgDS9SZXNvdXJjZXMg MjAgMCBSIA0vQ29udGVudHMgMjEgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEy IDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMjAgMCBvYmoNPDwg DS9Qcm9jU2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQ0IDgzIDAgUiAvVFQ2IDg3IDAg UiAvVFQ3IDQzIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDk1IDAgUiA+PiANL0NvbG9yU3Bh Y2UgPDwgL0NzOCA2OCAwIFIgPj4gDT4+IA1lbmRvYmoNMjEgMCBvYmoNPDwgL0xlbmd0aCAxNjAy NiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZRXzc4kqRG891P02ZJZfpK/sw8+ Wz74AVZey9Kykj3vLzkyga7KogUejQTqb8iIKEiSyF/+8qO8f/3xtqZYh7H69HbvH7/+8frlr393 73/9eLn3v9+vQIZCeWdKpmKOxqX3n7OrptL7v/98/fYK1cSabwt80QuITMl5g8AKolIQrbFhFxLx p5oOmFVhJmtCTBudKZoU4h7TBcYMAzJbk+suIkdTIx0g0x2yOOMcbVQiwquAZEI9UHiSCBoh1ZkY wyGk6JBksgt7WcGrEGedqdXvaUL/+PKJycbFQwzpEOdNcO4QIl9D7hOTDWW3/5zodYz3JscDT5SU o/iJyaYeQlLQISEYe7gsKUtI/YQU433df02OOoaCibbueYpIi+ETU0xKu1LAMaIt5hkTg+TSNqbG R0wxtewqClZbOZ30OZ1Exqf9vlWbHjHVkD/EOMm29DmeTCaWXRXhktO1ffY6V5zZvlhxmVIxJRrr 98WIrxrG/Dmfag3iDjHlERNNoH294eTkmHk+3lqT/L7kcKbpmGhyOcREybd5sb1jmH2Z4qRRIck4 uy9TfP4cMsuU986Esi85fP46JplIpxjZgfLRFpzJdl9z+Fx0DNaWfQHhPVYx5I2jQ4wkAWrzDMnm sM+96F4R0Rva3QIfs8lhX19YswJNHrXg/4tJn5hibNzXF2ZQMTkY78pOfS6G6gk1adQSTIq7auJL wWu/XVEDznO3r1y7Bl+txsddVQkW1s0d6sHjK4KFm8txoyA42Lmtpwke7s0d7vrjTILHqrxFDfBv 4XC3H9kRAiydO6W2zsJAMHX5cLcf1yEQfF7YxYTojK2H2/8QEuHs8s9d/pBg7cIpJjxikkl79/Cs ZZBl6uHhfJTMkOHs/OFyP0ozyjLEHGL0CxAKnB3t7vbyzoQKY3ewKM/nLNQMq3OKcSqGLJwdHXbt 8TyThbWzByv0sAHkgqHyc3aDHKwdnWK0rSEfpG/7GftEvsivvbXTNo3QgXo68DzsIKEpJXvoih62 k9CnxnRovR72lgjW7uBRnjaa0Ltae4h52HVC8+rSoS96tAWEfjZ4v/+eR/dB6GiTPfA8mhxCk5vT qTHSzRQVzqefa9qQ08bub/ajNSQ0umGf1M8GlNDoRnfoix6NbkSjm/Kh//LlEYO1B99SPOkYdLrO nWIk5BORDbzMPsIlFYI2l8K+4hQXdAi83tbXxABnlw8nYSWt/Hy7I7pauz29iB4W/cteqo0aFT0s 5V0xiehYU9iXm2KdRkXHmuuu3MQEZxd3xSWiG/VhVxb+88J7yDTYmYrnC1NGJcJU8MhgyT/ef7zs G/9wgr27gy7LTxuSGuTtBY9aAeBNxOX7/YNnEn44+I3KTydOE4t/f/32p9ffmBSW4kZapfzBJNkb KfweZ1EfCeo7uwJsmm7AwkwKN2oILy2ySxc3PN/FTc445nZI6jt34NPoIwry4FaATdMN2P7VqPpY iD+5O3OuyK2L2RvYMZ4IaXwxE5fcPhKUdWYF2O5kAxSXUpiDCbwQRUoxF9zyixmPOW4BphjvzFK4 +4hCP5gVYNN0AxbpL9zcJGIpX/Y7d5Ine3ITbkLiKSvuzLWsj5TnVyvApukGbB15ZiUtuF9S3Dix G3eU2szGWnFXrol9JJT7zq0Am6YbsM72RMODzXlRkAiKHIeGVmmSZyk9mNyd3EmJ6iPBpwu5BmwP voHr5MBReyQz8I4pbhzbjRuOj7mLZNzF7bmQ9XEct4Zrimtgwgp25twzI0uJuZj5QbmYa0+1qlPN iY/tY8QF79wKsD34Bi46Rd83PRkP31iQDYqeDeKHHqfI2YZJZRufwByjH3VNA7Yn4QB2EflZRQAs F69HStwFoOTmmwAnKYdJpZzLIkBG7q26AAXYFsaB7JI4A1YAs4yAyvbhrqDIYzMVeKBAQXjkXZUT kDHO0q4B28I4kOEAylDgTeIAj3S5K0DDdFNAspGYPP74UcA9yBxjnHugANvCOJDxgtmhwKFWIQCf 5+4KkpBNBVEyCRMpBZ5LXR9jHEVeA7aFcSB7BA4FVvKmEjjuCnCaNwVJ7hGmpBRIb9bHWes1XlsI B7B3MxHxfHLe1AiKuwAc5k1AlqPEVJSAyFWvjzGPN1YDtoVxIMPVjUQMuedNAsddgZfbNxUUSedY BexSIFkgSVBGIdBwbeEbuNwrDn6sLAjIYLjzOyk+g5/bHfAn+0jDwuWvj/yadwUKsC2MA9kTrnVX gL3igPpIQyuFdypwkoYSfFMQxPn1MdlRiDVgWxgHsk+41l0B8gVtmgWSVOuhQLb7UuAlDeVz7grE Y/UxuVEMNGBbGAcyjHqIUpEDW2cEwBUpBVlq31QA0wF/jYnT8VIg/U8fkx/FQAO2hXEge7xZQwHK LAfgK+tdQZLyPxXAerACknS8FEhr2Eeuv12BAmwL40D2qJNDAV4aDsDnKQVR3r+pIIpjxsV2SoE4 rz4mmnugANvCOJGttH5QgP+JHIDPUwpI3v6pIIuLwxQc3RRktiF9TDSqgQZsC+NExnM5FJSeOPg8 paC/QFNB6Q4WO6kUVL6HfUyjHGi8thBO4DAT0eeeN/g6JcCpRKxy3eWJuwkQSyJDSnMDFFxb+CYu SZfC/HywCCiPNLSzt2L+bHt7Z8VXXvxe2h0ZU547oADbwjiRIzK3K+DkRkDVaSg27FLgJA3F59wV 9C3oezDtsAZsC+NETqjUXQFf8LeDZVVpKBb0UhB6MxF0GnLbM8dUR0HUgG1hvJDtUMC1jiO82JxL QpZWb0qAK8nEE6fjJSFLFyRjtqMaPBDbwjmhKyplFg1c8TkCHjbfNSSc3Gdhpn4QsPv5rqFINyRj dqMePBDbwgloi99wETQ08LvHEciWLxrGwhx6OqLfuWvgwjvHPJ3yA7EtnMRfgt8O2SEaHL/9HIF8 +aahL8xeriQmUhq6O5AxDwkPwLZQEn8IfnukR5eQZRMdzGz5JqEvxJ3gPMDklQQK7znmMOrCA7Et nOSll4U0OzSw/eMIJMw3DX1htnIdMFmlIdb3HPO4FA/AtlCSk+xAawG5IoE9MEfUrxk5FqYqNSEV ScxLQu4+kUTC3AaN2BZOsj07Ej6ra+BOABHefs/IvjBlqYyYstIguyCbMB+IB15bGHFBC8wtrkoc CgL2hyPc93zsC1OStwFTvCtgjzbH/ElIjdgWzsDfgd8FmF0Dt4Yc4b8nZF+YYndsUR7FS4N37znm MrNBI7aFM/CX4HfFznYN7HX8m4sr0BcJfV2iblvRxikJQeqzjLnObVCAbWEM/B3oBRxyI7ECvCgk AciWVcFYl0K37mhllYIodlHGYkeB1IBtYQz8Gfjtxe+ygoLs4QBkyxcFfR3sd+3WPSsFqbznWKZx 1oBtYQy+90NwwkNBRu5wAHLli4K+LllxzMmKf74UFHrPsfghQOG1hTA4SQv4qFi7ABjSyAGFK8si oK+D++amAVO8u9ZUxS3KWPyoCRqwLYzBSlrASuahAKZcAuq3RBzr0EFy44Qp3BXwtsxx+hWN1xZC /F2yImFnugBsDgdA6bc87Otg/7l3xKSMe3cJQejnBii4tvBhYyQneOr8yA4J8F+zsK9DuGX+qFuX TOwW+1jiKAYasC2M7Gpd4eMMQwFL5YAAylVBX4cTLKyAdPuWkxgEGUua90ABtoXR82fUntH8u9ae NLgvaRXwP7arXgeP3Ab2eYp9Ap/+f2p3AXKNy8N1qYKoCK64188MSe2upIUBrj9bmhlJI4q0YTBx iWV6+RFQJQ1JbPY6rnjj4AtcRNc7Lfw0a8T4DCP3U4COwzVOsczb/Ajo8ZqxNd2BDXAcjIGr6JrX RAEvLCfgAn8p0HFIZSGWmdFuBc3JFdTYNRtugONgDFxGn7kdCpizOAE57EuBjkP176BAsvpbQdAs xNidJqMNcByMgcvofOCC5J7Ov3AC8viXAh2HJ40dED4tvhUkUSBxbsGKNw5C39UWwFcBqJ04IfF9 /RCg4/CuVwqoYsdHgFQmGrvXZLABjoPRN7UFsFQB329O8N9G1HH4lamgiB8fBTVfM9qrvOGNg9Bz FZ3lXlMBLGI4IX77UMehwIsUkMWOj4Durxm71cob4DgYPZG71ryiQJsvj0IWL/qhwMbxf6kgiR1v Bd2V647JksEKOA5Gz2V0rfz5u3X1DVf5pUDHod4WH4bVhz0wE1rMegob4DgYUcY6635EQDPb4Jy/ BOg4nGnNd9/zCIjtumPRLdgAx06ItrZZByj8xVwDp7t28uu4xJbkbv0e/hyvO1qNvAGOnRDla8HP ZJZoWQp/z6v+xa/jeK2R9fEJC3+RG6hRPbjhjZ0PpSvyRmXlkkifpOr3yHX+i17HMa2RHq9qftNL PWaxaybcAMdOiBXgF95IpY/SiHpk+fhFr+OY10mPsuKhRwdwae7bQMbOIcePakp/BWRkDg6wx83o Lv6R9xyJk01mZd5JAAN6QyvI1odN6H+pjVW+DZAtZBdpRySA/yMNjyE2qUljE/tKs/TsHzlZMwgt KhaHuvJNy6YUKoNX2YL3Q4oYFliowSqzc3p2rXnZ+0naZbusS7wPzXkmDI3JEtcGOFY6g4WDhBsG 49BmBzq5nRyCcScvTklefPNwR6ZLjSl1414Ax0pnsLpqWAsD8U/+zVy7nOhkDpqued1fqcol1kwa E5Qp8wI43mQGikpYmOEwDvRmosnc9OSNOcoTlaI8WA9zYXbQmCxHbYBjpTNYtAHCDcNyKAuyNzdd +HAneaBTkuf64a7MjBppTuVeAMdKZ7DdfObEFo3Wf3PjxF7cWV6FlOWNeLg704LGNBuWFXCsdAaL hCFGQ91FXzTW4m/yJJ3ZJK9ao1V5Im9yL/WhxmSZaQMcG5/hejlwZE9xBjuRN3eUnmxyNykJUhPH PdyBRYlGO+4Vbixchon0osxVncEe7NUe4PnyL+auVuur1VD6XDOyb1DuBXBsfIaLNzLoppcfAc9J Yw/6pnfSixk9TlG6E7e6jScwYw6W11bAsRMaMIqEVLsIyLgrGA9LvAWULq3YFODFcvgslvNVBEjM 0ZuABXAcjIaMQqmYAiR8TEAp1RYFDQCPgiBNAe7x6jupBzXmmdpXwHEwGjKKxWYKwo/CCQF2eSuo 2MNHQZKNxCfEV48aHFOcxpznHiyA42A0ZFTszhR45CpMwPL8W0ERsqkgi5PwSYuCwFSnMWdL8ivg OBgNOWCiKXDiG3QVcVGA03wpKHKP8CmLAikINc5cv+KNg9CA8fqbEfF80jdorPIiAIf5ElDlKPFp i4DMrKcxV3tjV8BxMBoyOkczIsoe8U0Bx1tBkNs3FTSxc+4C9igQF4gJmiWCFW4cfIYb4rQh6q3a MKGC4c3vJfkYf3Fiw+I2GzamP418zVXBAjgORkMOCddaFWCvOKFvNnSSeKcCLzaUyS8F0fESaizO EvEKOA5GQw4F11oVwC8dTwSQJFubAtnuR0EQG8py3gqkxtJYvCWDFXAcjIYckLWzZOSIK8MJqIoW BVVy31SAoqNnfmjHR0Fka6axBEsGK+A4GA054M0yBUiznIBV9reCIul/KkDpQQVJ7PgoSLyEGpl/ VcECOA5GQw7Ik6YALw0nYHmLgizv31SQpWLGxfaLAqm8NJY092ABHAfjRGZzJwrwP5kTkjQ6j4Ik b/9UUKWKwyeirngUVJYhGkuybLACjoNxIuO5NAVNjYPlLQr0BZoKmlaw2MlFQec91FgsHax44yCc wHEaMVT1DVa3CPCLEbtcd3niXgKkJJFQytyABW4cfBM3SZdCfh4sJrTNhm72VuSvThoXeeTf/EHa HYmlzh1YAMfBOJEznKsKaG5M6KsNpQx7FHixodQ5bwW6BboHsxxeAcfBOJELMrUq4AVHL+pWG0oJ +iiI2kzE1YZse2Ys3RLiCjgOxgfZmQLmOs4IUuY8Eqq0elMCqpKa+KEdHwlVuiCJ1Vk22BDHwTmh OzJlFQ3M+JyBGra+NRSc3D2wJj0IlPv1raFJNySxessHG+I4OAHt8BtVRDINfPc4A2750GADa1Q7 ot95a2DinbHOSnlDHAdn4krw28MdosHz7ecM+OVLgw6sQa4kPmnRoNWBxGoSNsBxUCYuBL8D7KES qmyiRzHbviToQNwJ+gCfsEhI8ZqxRssLG+I4OFOQXhbSnGlg+ccZMMyXBh1YnVwHfNyiIfdrxmqX YgMcB2Xy4g60FpArElgDc0b/dKQNLF1yQmlizEdC1ToxiYS5DSviODiTU3cULEs1sBPAjOC+HakD S5XMiE9dNMguyCbMB2LDGwcjLmhDcYurkk1BxP5whv/2ow4sRd4GfPJbAWu0GettyBVxHJyR68Dv BkzVwNaQM8K3IXVgyVqxZXkUHw3BXzPWNt2wIo6DM3Il+N2xs6qBtU64mFyBfkjQcSVp2Yo2bpEQ JT9LrH1uwwI4DsbIdaAX8PBGoQK8KEkmwC2nAhtXopbuaGUXBVnKRYnNWYJcAcfBGLkM/A5S71JB g3s4AW75UKDjUH53Ld3roqC0a8Y2C+cVcByMMWg/hErYFFR4hxPglQ8FOq44qZiLk/r5UdDSNWML JmDBGwdh9GIL1FG5qwAUpJkTGjPLIUDHofpm04BPfletpUu1KLEFywkr4DgYoxNboJSspgBFuUzo X0a0cegg2TjhE98KuC0zznplxRsHIf5dXFGwMyoAm8MJUPrlQx2H8p+9Iz5L4a5VQhT6uQEL3Dj4 sDHiCX6UH+6QCeHThToO0x3589q61MRqUWPLlgxWwHEwsqr1jccZTQGlckIE5alAx+EEGxWktX2r RQoEia3Me7AAjoMxcBldHc3fvatpcF/KKcCGwcQllunlR0CVNCSx2eu44o2DL3ARXe+08NOsEeMz jNxPAToO1zjFMm/zI6DHa8bWdAc2wHEwBq6ia14TBbywnIAL/KVAxyGVhVhmRrsVNCdXUGPXbLgB joMxcBl95nYoYM7iBOSwLwU6DtW/gwLJ6m8FQbMQY3eajDbAcTAGLqPzgQuSezr/wgnI418KdBye NHZA+LT4VpBEgcS5BSveOAh9V1sAXwWgduKExPf1Q4COw7teKaCKHR8BUplo7F6TwQY4Dkbf1BbA UgV8vznBfxtRx+FXpoIifnwU1HzNaK/yhjcOQs9VdJZ7TQWwiOGE+O1DHYcCL1JAFjs+Arq/ZuxW K2+A42D0RO5a84oCbb48Clm86IcCG8f/pYIkdrwVdFeuOyZLBivgOBg9l9G18ufv1tU3XOWXAh2H elt8GFYf9sBMaDHrKWyA42BEGeus+xEBzWyDc/4SoONwpjXffc8jILbrjkW3YAMcOyHa2mYdoPAX cw2c7trJr+MSW5K79Xv4c7zuaDXyBjh2QpSvBT+TWaJlKfw9r/oXv47jtUbWxycs/EVuoEb14IY3 dj6UrsgblZVLIn2Sqt8j1/kveh3HtEZ6vKr5TS/1mMWumXADHDshVoBfeCOVPkoj6pHl4xe9jmNe Jz3KioceHcCluW8DGTuHHD+qKf0VkJE5OMAeJ2NsUkTWLJ2hfYzxt59/tevnr8vxDx/7Xz9/h4x/ QsZ/7A7+fXl3/ev64093/fsf7JLYr1amsARd/+e8bHJ0x20oOq9V1BKsf3kZQQZZQc8eA3TvfxDe S8qWzFnQgL9mPfJYlijyUhc6darEFIV59g9eioHBHXgaeJ+f9p+ff3NpE92srqXpz2ULxZH8n0s7 XyTfpdqb3x3e/rmP7Us0vt3DzjGrrfcj0V5Dqh4C/s5dthx5s8Y15OkpAaUH1s0ehta+Mi1gbuhV At6L+HGUQFL7gq1p85A1iZArdSNT8t7Vblp60JunBNRMTDctmMlH0KkbqwE3s2+hu3WPulXwj6cE 1NTlmz0cPfRaMGBsaKtJqfoLDH14SkCpjSuj7as86KyJqn8vG7pa55tb8/94SkANzKvVepihO+/A TEzEheZWQFhaYzP04SkBNaCOaPcHfVnA2NB5qfZXN348JaAGZtxqvd/QTd+tAVB3LzpVH4qgSG5H H54SUAPSgPZ0NFQzA+4NfXHVxdY+HH14SkCNzn+gclrohIBCP0fPmwoqcUNaftCbpwSU2vicOtgI gEZt0oDKr1voaVPC2EbV01ECSUvtMHs4GZ/HgLmRKfjGTK7ADX14SkB5Fa9oAQudNAVmYyItNFPD h7tUHvTmKQGlbbXoPuqIcA1DJ62IGtC9+Ri6tWdyxa4Y+vCUgFLBMMzujrYCqgFjQ3MIYIkac1WQ w1ECSW0fb/JDzkMDBoeYRS5cbuLSn5J6eEpAucpqlw9eKkL1pRpgomyhOTt1JuBoXkFOTwkoFZA4 /ZZ8rkXAZQF9Qyesuk/f70XeHCWQXJo2nLWRUdY1wKTsIl9Ybu++30Y+HCWQVHZ3s4eTkZMMmC/Z Lkqv3I7k9eP0lIDqOl1pWjVcq4VWUfOrE0C7X/TkojOa17jKg948JaDULrBRCwyN668BWoE39OCq WT/6KqinpwSUTkLD7OFo6EIGjA3dcYiQkfjodYqHpwRUhzhVW+eOB521cOpYWDZ0Q2q0Mf0sF3rz lIDSCbIWtTsHMKDRIDRAsz+9aN7vxjLSV0E9PSWguk4LZvcHfVnA2NCccFoluowHvXlKQKmdYQ8O rkCjGWuAFpoNnXGMjc+evYKcnhJQvWmq0MZYCDQkhAY0XdKLTki7llBR+7Xy+vCUgOpVi7naUEEL nSygb+gL662T23LPB715SkB1fI7a0G6GTlzdja970P3GqitLapvrGA9PCaheWCnVHo7GtWbA3NBc b23c8XWKh6MEUtfmpoWyY0mLnBAw9Ks2MmU/t6O1h7w5SiB1pHv/7ZD2sG+IefVH8r9gI1M8If9A Ph0lgHpijdKfbuDJfnTjwm5gcC9emWLV4/STwOmJZVVflxcXZfdGjckPlzKsTO5x6gu8OUoA+ZZw gwge7BTYQTSOhwzVUQaetsUfPwmghuqttlZW3XmCLw/oG5jluVAkVNdLH08JqHZz8ZqIydCdLRmZ mnc018xWXr1rfTwloNrkhqs9DN14IAjA+Sz0xVVnz2iST0cJJL/lfudJTgzIzKiHDGhhv60+Fn08 JaAaJInaKqcMXZn2KGXpRTesObMprmQ+/SSAWudNVHsYuPCmImBs4Ik1s4pWn4k+jhJI3gS8JZCc GVBZbBYZzNyqXxcn744SSA16VW3VUkbOLIlodGkjs1VlVo3iHfzjKQHVKpWh2n2hLw8YG7oh7zIb QPGR6OMpAdUq24tqhmzoxPYDUVE2NLtgvvAs3Q/x9JSAcm3jSkfRF5smpBB66EIXrDqxShfv4B9P CaiG6aP/Dqu8RCcP6Bs6Y72J1RTX0dG7pwRUyxQgqioTbRWDECiQnXlDs8FC0eE6Wvn4eEpAtUTZ pPYw9KSsQsD9FlPck1SQgMUnoo+fBJDrdVfvBCcG3FSGDrbOnRJz2yeij6cEVEOajl+dOS5DDwpY DCVpR4PMEpJ9Ivo4SiDpaDTM7kbuVN0IGBu5I52v2e2mk3w6SiD5OOfDHcl51F9Mf1U7yUNG4l3s idkHoo+nBFTF8EH7NrTqHgboPdvQlYu2+1gtpz+eElB16Gep3SjCib48oO9orro0VpFF3h0lkNTG pDi0uhpZv6YhQEvaRsZluTjK5eRneDpKIFWMHrTvRLKWXQ+4byf/Da9aHq8yaOlP0kP756+f//z+ 98fljhcZvvtESXhXxWyitubZendigBbo9e5//VyAErhE76RiKp3R7yrUO/3iP+q4nHhK+jIcmgA+ 9X8KVNiwlyFR6sVwVYj5YsaMwb/7lqIh63NWPl0xn54SUDPzndVmVEUnqBAGzA2NG3d3pMHtGuPj KIE0MxOlJl5EIydN0oFMfMlU4TdT+O7pQW+eElAzMVNqZn0BerIfjdszxdC8cnfhfrjI+HhKQM2L iVAxwxgaKoQBY0M3rjpVf4GhD08JqHkxZSomr4XO2o8mStWL5pWbtuOuMz6eElD6zqrLqJgXDY0R UQO0GaQXza2Yk2t3ofHxlIDSQ5hm9wd9WcDY0BwfJuXGvNOD3jwloAZal9pNfwwNJaIB6GQvmrs8 WfSnC42PpwTUQMOlPR2tAjUx4N7Q7CGz2Np9XDs9JaC8JHiBMHRCQKGfo+eNyzITN8SFxsdTAmp0 fo7WveRoKBENqPy6heYB4pJhW5x8OEogjcYzUHs4GZ/HgLmRBxY9JjMwL/ThKQE1GjNHG0x50ElT YDYm0kIzNUbPvisLvXlKQHkj9LYIdGK11caJ9F9ojlWjdt8VQx+eElCjcGBRuzvaCujsvLQLXZF7 LFFjrgpyOEogjcIhq11sXkbGFDYHh7JF5sCGKUGfT0k9PCWgXCe5alL0dXN2nJOj5EJnLLozAYdL jY+nBJSqO5x+w7Yt9GUBfUOzYVPiYr8XeXOUQBoYQtXGWRsZZV0DMJO+ZM6Cvft+G/lwlEBSFd3N Hk4e1GjzpmRzsl2UXrkdyevH6SkB5drfJwFDQ1jeF3XmQnPM7BnNa1zlQW+eElBqF9ioBYbuVMM6 4aQNPbhq1o++CurpKQGlc9Ywezi6UcNrwNjQFKEQa/jodYqHpwSU2hgrdFYsDxpzh458ZUM3pEYb 089yoTdPCSifaX3CBbpyWroLh6eF5v22gbavgnp6SkB1Fe9m9wd9WcDY0IWrpiRFZ1rozVMCSm0M nm1wDAW6cDK9Kyephc44xsZnz15BTk8JqKUdoScMDQmhASq9y4tOSDsbDfu18vrwlIDqqv9NlrYH nSygb+gL67UJrt3zQW+eElAdn6M2tJuhE1d34+sedL+x6sqS2uY6xsNTAqoXVkq1h6NxrRkwNzTX Wxt3fJ3i4SiB1DPnoY4lLXJCgOr8tpEBtixp7SFvjhJIHenef3viAPDn54aYV38k/ws2MsUT8g/k 01ECqCfWKP3pBp7sRzcu7AYG9+KVKVY9Tj8JnJ5YVvV1eXFRdm/UmPxwKcPK5B6nvsCbowSQbwk3 iODBToEdRON4yFAdZeBpW/zxkwBqqN5qa2XlTKsBlwf0DczyXCgSquulj6cEVLu5eE3EZOjOloxM zTuaa2Yrr961Pp4SUG1yw9Uehm48EATgfBb64qqzZzTJp6MEkt9yv/MkJwZkZtRDBrSw31Yfiz6e ElANkkRtlVOGrkx7lLL0ohvWnNkUVzKffhJArfMmqj0MXHhTETA28MSaWUWrz0QfRwkkbwLeEkjO DKgsNosMZm7Vr4uTd0cJpAa9qrZqKSNnlkQ0urSR2aoyq0bxDv7xlIBqlcpQ7b7QlweMDd2Qd5kN oPhI9PGU/5Fd7rqW5DYUzfsr6g+s9yOezIANDDqYwHDmSQzQwIwD/765N6U60iEaYF/eS65SqSRy 06FqYXtRzZAMHdl+ICrygWYXTAE2t/UR70hxqKVtltJRdGDThBRCD93ojFVHVum8OvhXpDhUxfTR nm6Vl+i4EtqBTlhvZDXFdVzoM1IcqiYKEFWVkb6KQQgUyM50oNlgoehwHa18fEWKQ9VI2aR+N/Sg rELC/BRT3JOYcQDzmoi+4sSBll5f6p3gyIRJZbjA1rlj5NleE9FXpDhUxTHtj84cwdCdAhZDSTzR ILOEpDURfQWKI+lo1M1vRm5U3UjoB7nhOIfR7KaTfAeKI61xbg13JKdeHkx/RTvJS8bBC+yJaQ1E X5HiUAXDB/1paNU9TNB7dqALF233sdiZ/ooUhypdX0v9ShFOdFgJ7URz1bmyimzyGSiOpD4mxa7V 1cj6NhUJWtIOMi5L4CiX4vqGd6A4UsHoQX9GkrXsroQ5F/kPRJX8RuVOT/+L+tH+/P3Hb89/fiy5 s4oMn32jxD2rYDZRX8/ZfnZkghbo/exff/zll/+O55efz/2An7/8/Ud8/vrE59/6m6n97n96B56/ Pf/4Z3j+9erjQXGVGx/0WfDPH4Gk1c1igrSJWBm+sGDbIl4Ukq3byrDEghsNP6bAJerSM7eR+29b bxaq0tBXpDjUSNwN9aeh4+R2acI80DwuZnE2ib4jxaEGqpf6UOEbjY7VxzpWhuYhN7slyVekONSI /ENJrEZADx7dPte5MjSvplmUAUNfkeJQI/B11B95o4Ml2LExNOuJWVzCjT4ixaFG4EcomecX6M4q McI6gURbFTSLimvoK1IcSp+Jo1MwXRoaskUTIk/SRrN0my2zLfQVKQ7VJ6uD+u1FR0sY+YNmwzGL 3rbRR6Q4VEcFU79qlTB0pQjRz10OdP20yrrJV6A4UkcFoz8XWaskErR5HGR2drN7nvmKFIdSH2Vi VRNDp8yEdqCtV9qsm+eLPiLFoTp+UN+KJNCI1AQrowttKspsLfvsXZHiUB3Lpz9edLCEeaCp/czW Ol/0ESkO1SufudoR0PgqmlC5hI2mYjVb2z57V6Q4VC9UmauJAg3hogmNG7fR1Nlma9835ooUh+qF 9Wy1fkNHS8Dn3ui+B4Q62gs+4sSBeqaeX2oFYLsRo3OY22COM2br3Gu+IsWhVFRhCFkSS9FhckoZ g3PnQtsMZrYFa8RfkeJQPXFyqqiGGx0toR3otKfHFvsLPuLEgXq0VmhSFuDBYXCJ3Q3msGu2pb3m K1IcqqPT0J8L3TnCDjSmD7pyO2hRdgx9RYpDqR9VVK2xwdCpp2cGitKN7vHZtpX6oo9IcaiGrq6+ DTtAo+1rgo1DGz0o4GlbXUfvjhSH0qFsmj9edLCE+UFjIN22tfKij0hxKPUhYNdgCbT+hISkr/VB p7SFEldq6CtSHKp1yu6qK1joQl0+MyetV12OZ9s29qqvSHEo9YP5vW50tISRP+jan23bjC/6iBSH ao0DTlVhtdCZE9AsusgPuudn276E6VekOJSqR4xl6s+FTpzbpur0A83eYhaVztBXpDiU+hglVZjW F41ZUxPaB23Dqdme8os+IsWhmspm+FOPv6GjRmpC09f6oK2Y0vZNvgLFkZoqePPHSw6WMA9ysYbI rS7pRR+R4lDqJ62wDd/I0BDGmqBDwYFuRNP2fc2vQHGkpsOUnsEGaQ9/agFG/NCX+oD7eLbtzdZ8 B4oDqR/M7xscLX4c4MlmSLvk6R0njqM/4D7qr7NxB++rpuH2Lm6L+dm2b/AVKA7UcD7pW6vRLeta GbGD4SCn+Wzbh0mPr0hxqAY1q35mMyM6rgR0u40u9dkWPXqhz0hxKD0iqNd6FrOhG+s5TlQ+0K08 246wVn1HikPplcGWqz8NXdmFkDAPtFU6gKPV/684cSD1cUbW9EkwzhCubj3Ak9tBO9LY6DNSHEor CPq5liqcckUX9nvUMlyDhcbWbYu+QfQdKQ6lJXWaPzY6rIR5oPOzzChjg884cSD1UUC0chcDZxYY lPZ6gEt7th3VbvdXpDiUNq8Gf7AGKlrlFBMqi+RGt/RsO9o6eHekOJT6wfy+0XEljAM94rPtWDXp K1IcSrs5ipPKhmzoyOKF5l8O9OzPtmOsVd+R4lAqb9AX1Z+GDuybUEPhgx4sHMvOteo7UhxK/ah+ D+ztRKeVgOa/0ZSlZmdIG31GikOp3kPzUmGZ6KsebExQiXKgy3i2ndGkx1ekOFSNvAHqj40OK2Ee aPYUszOljT4jxaHUh95TpV0MPagHIcXrgR5E0868Vn1HikNV08g6eECzKlp3DgkxUNRu9OSG0M5i 0uMrUhxK/WB+3+i4EsYHjf19bc0bfUaKQ+nslXUe0UksG1qVhLY+jGpFxflG5/K8tq1V35HiUDqK dvOnoVVQMUHv2vygy3xeu2TNV6Q4lPoRvsbFYui4EvqBbvV57djkM1AcqWiFha8vZ+RCKaCbqG/1 IQ/uhNm5zt4dKQ5VGiuD+tPQqo+7JcyN/gNRJb9RGRqp47+oqX/+/uO3RwXUg3985o0Q9wz1kdhV 3Of1zMSEQuHLZ/76Izz4t6UqJFTHuFQUOEAuz2pyVMHoauvv3B+FQcoGajdsD2j6EgZMkd/wA7SE jPmr24qq+iXw8KiMTBBfEUvG8VobHiJ6jFkMMoa+IsWhRuIuFFwKouPkNvXOd9rojPNhthTTIV+R 4lDq46OWyE9saHz1PngINrqix5jFjLTRR6Q41Ij8Q0k8mEAP3TpNmDy6G924IbSl7VVfkeJQI/B1 1B95o4Ml2HEx9CCatvT+oo9IcSj18RFK5ukFurMsjMBvstB6vJ5tMX4Z+ooUh9Jn4uiUwsoFNAZJ TYg8SRsd0XTNltkW+ooUh1I/mN9edLSEkT/oTDQtNM1GH5HiUPp9C/yqTc7QenOQoP3iQFduCG3d 5CtQHEkP5TB/LrKWRSRkfasPmfrXbM1rxr0jxaHUj1X9xqpj6JSZ0A70IHoYer7oI1IcquMH9bsW EkMjUhO0XdQXnQJ6rtla9tm7IsWhOpZPf7zoYAnzQEe0GLO1zhd9RIpD9cpnlsGbADS+iiZULmGj M7bCbG377F2R4lAdJ6U+BefG0JE1eTRu3EZX9BizGHYNfUWKQ/XCeqZ+nxsdLWH0DxrnmaaO9oKP OHGgniniK26oge1GDFzh/IIHaqnZOvear0hxqJ44/NSw5lMILYyXY3BUWegcADXbQl0964oUh+qo f+pHDk+GjpbQDjTuIE2L/QUfceJAPbIZVlRwAw/OgAMl/gPOqKRmW9prviLFoXrgYFrRdwzdqTTG 5CC70ZXbQYuyY+grUhxKfcjKim650dCdE1/rg+7oWmZbqS/6iBSHaujq6hdKY6DR9jUhUjxv9MCh M9vqOnp3pDiUTmLT/PGigyXMD7oEdi3a1sqLPiLFodSHYtWRryy0/oQEnUsOdGLXosVKDX1FikPp 1ItxqeoKFrpwnpqZunyjWTnMtrFXfUWKQ6kfzMfIZ+hoCRgKN7r2Z9s244s+IsWhWuNMU1VYLXT+ P9lljwNLbgPhfE8xJ1jo/yd2bsBw4BM4Wwbe+wcuFqUe9hAL9Fu+V/qao5bIIoeejcnVoWf93OdM x+69lRJQ8I+zM94HXTiqwWAmh2ZvsadWumtNnVICCnHWeHGaNHSpXDC+aC1C9zlLfdBOKQE14O01 3jj+hs5QYgG8ff+irZjyOS/5JZRAGvDXFq+HnGzBduRmDZFb3cqDdkoJKMQFFXboNzK0GmMs0Jni ix5E8znvNX8JJZBGwY8an6HWXuO9OTTsdYYEA8/1uc85LOe3UAIIcbJ4XnA2/XLgzWbI57Gnb50E Dv5H7yP+uhp38b5imd7ewx25fu5zXvBLKAE09HwytlaDLZuojLqDyZHL/tznXGY9fpQSUEPdLOLK ZkZ0Pgu0211065/71B590F4pAYUjovUaZ7EaerCe64mqDj3a5z5XOlm/lRJQuDK65Yi3oTu7kC7Y Dm2VTsHZ6v+PTgIIsZ6RM38SrGdIr2534M3t4HOVddFeKQGFCqL9HKVKTznQjf1ea5leg4PWrbtP 7RtEv5USUCip2+J10eks2A5dP+ex2rpgr5MAQqwFBJW7GbiywGhp7w7cxuc+V7fb/aOUgELzGhov 1kCgYae4oLNIXvQon/tc4xy8t1ICCnGyeF50PguWQ6/8uc91atKPUgIK3VyLE2xDNXRm8dLm3xx6 z899rnWyfisloGBvtC8i3oZO7JvqhtIXvVg4znOfrN9KCSjEGfFM7O1El7NAm/9F05bac6dy0V4p AQW/p80LxrIwhh8cXACL4tBtfe5zZ7MeP0oJqJ55AxCvi05nwXZo9hR77lIu2isloBCr34PTboZe 9INqxbtDL6L53PVk/VZKQHXzyBg81LMCjZ3TBTnR1F705obwuZtZjx+lBBTiZPG86HwWrC9a9/d5 9nrRXikBhdmrYh7BJFYNDSeB1qejWoM5v+jaPs9znKzfSgkojKLT4m1oGCouwF3bX3Tbn+d5bM2P UgIKcdYYutwMnc+C6dCjf57numQvlEBqqLAa48cZudEKYBPxq77kxZ2w5z5n762UgGqDlQHxNjT8 8bQF+6L/p6pWH1VVjzT1j4ylf//3j/98YKA++h/f+UZIeAdiXThh7ut5Z+GCRuPLd/4LPwhzYzqu 59//+CcdqzqpqVNTA3fpC9rn9DqaYW1u59+5TWCqo03zmmMH+OuPf7us9R+12nzpP6vPz+rrGXwL DdLUOteej7DL9yOs9aCdUgJq6SzJeEzv4rFAT/PzfXlw/Jn8EUogLS2QTf3xncUy79yz90ZmHz7H feTHajulBNQq/Jyv+UCbKxZsl/TI30va+oN2SgmopbWX8Xiy1iI0tTh/0Z1Z5+UK4o9SAgpxtVeZ 41TDqG4DC7rLuuXTdr5l/EcpAbVg4/CqkViCDa1VeS7+/atqnTKexoN2SgkoxPrOkei0vmPN3Kwc F12uMfl24x+lBBTiaq+yiXJvdqm5uW+vKr56Oamf+cMJJYBW4tf1A5NaUeitChg4jWMtnaN6CSWA VuKJHPkMqnuzZeOv8zfjZXY4v1zgSygBhLjai+admPQ646/7N2N1UDob0Ki1OzA5nQTOSrz3o9Cs kKuWcSVWgcudtNrr5eJfQgkgbJHWKsQ9H3DFQIEvkV3CPA9z+sHjpZPAQdzsPfMmrA5maVP9cjsT 7mc3DtfpJHAQJ31P5Wj2TI5L/dCXS182eU1+BscjlADCxdF/QGy2Z6uvh75Q9jhKDgaZE+kpoG+h BBDiZi96BlKdUaBvLuPCjFM9w8EBO6EE0Fzc+6EzzxlHlyUyXMY8voPN5M4EL50EDmqqnhXEZ0ZC 59Y8qm9SOtnqlHs3+oCdUAJoTp7u0a5rH/Tiq/oWZddisE7MUi/YCSWAEOt1RPNO9Qy5yxJxHUoP gvnPRI9wZlwnlAA6fmP04+C32gPqXX+yiWtkBY87wbyEEkCIu71o3IzVgMAwuO7ECzfYQcatEy+d BM4crNGIzVBuOMtMvWtNNn32Nc4XJPgllACCkWQ70S9xwDr9Qe8aE29yZ9Z9XKyTScAgZivBz7j5 6jQCO1u/5Wc25tu5zccAvIUSQIirvegMn5XNHnrXk+hZetWsRzll4qWTwKFVRzzPCAN939S7lqTn FuByjsXhOp0EDmL9gkN98eFWzWP4jqT3Hlyein6xTiYBc4YWxNuKD2zSpt71Iys7rJn97MJLJgGD WC/IQOXoBzstDdeOBvtnG3Yo1gU7oQQQ5je90oj7zZdtZvp2ZNWs0Wv2US/YCSWAEDd70bwZs89M lqwLnnolGvuGHrcDdkIJIMRaNgfMkWWcrc9MFtkLHsyYFqjXs8cvoQTQLPzeiE9DytZnFrvCBXfN uNLX91stX0IJoFn4P4jnzZh9ZnHZBTf9bJUeU432ATuhBBBiTX2m25CSNRqdJL5gFuBKR9HOiPkW SgDNTG8z0+1IyRrN5qe54MKMeYzbulynk8CZ2aa9dBsSbF2mXs/S5bKw11p4mPcFO6EEEGI9z/jD +giEyxIZLmG20MoG3bp9u7dQAmgm3kDEdR9wt0T2N+PO41DZSFvLF+yEEkAzsWIgHjdjbTT4oVpf HjCIhe2uPfk6mQQMYq1viK2N4BNrm9mJ1fBip+Zbxjh3hOCXUAJobBbkWU4jgV4bzbaZ7gErl1Uo lYt1MgmYsdk/EPebr7YZnHz942JtH1gm6poX7IQSQIj1DYitj+DaawY7M58HrFtbWOXrcWtvoQTQ WNwRVA9rJNDrjkG/XcZVM86bB+5UzLdQAgixfkPE/Was33jr5ODAmnFmQa7n1r2FEkBj8cwhtmaC ZqBnchee0AtmC830l7WamXgLJYDG4i1BT7FmAr3aOujXt/5oMQOYZaKWy3U6CRy0Pb3WiPtNWH3o riwCl6vUWnnx9sU6mQTMmCxDiNc8WPXNu/qW1JYmmlmDyp4X7IQSQGOybsK6WCeB6ZiWiGtJ3IBM c6k3mtyXTgIHJkgLPWKb4aDXXrmb70haIlG3F8Hzcp1OAmcM9hHEMx+ujlK7+YbUhp60xCJR+rpg J5QAQsw+Ms4MBxu6LJH5rT7mI5JdvXZOxEsoAQRbrN0fsc1w0OtP2d03JC29iSdJi8XlOp0EzujW R8YZ4aBXOwO9a0hND1li29BScbhOJ4GDmG1knhEOc8myPFw/agXfTKHpJPsSSYCMZi1knvEN+m5J fHuRAgnj4aTf1aOa4Z7y0j+0V/39X9Xmj/6nWpxKbH+56cv9eV0dzfim3xe7519c0Gb5psbfs7ee wc3atNd60E4pAXV+RF80g4YutXx/paH18mye9D3rQ3ZCCSTE6tG7mhEjqxvhAjelbs4km2dyj7sf L6UE1CqcKro244vWOocFbk7dvEGbbWufMepHKQGFuNurxpN10VyWn1S1Q6mhWOcFF+2UElCIq71q b0MrggvcsLp5izYL1S7loF9KCaiVbczUi3fRvXOBG1dtKxY97k7jQTulBBRiDppqTC66ai7w7S5r vUt/LrrctfuDdkoJKMTVXrW7xlvNCfVuZt0cIhaHS02ddvQllABayYZNmMl9wH1T74ZW3QH14eMk fsBOKAG0Ek/kUHNywBWJ4K/d1Lo4By4WrNX7BTuhBNBKvEVDZySCF6/zSn5uXYvbS5+77NC9dRI4 K/HeD/Umh8t5NPnBVbto+nMupl3WBTuhBBC2SGvVUG9ywJxHsx9c7TzM+X/aq6bJkRqG3udX+Ng5 ZGl/tN2uojgAe1gOHNjcgEPvTM9uwziZSnqA/SP8Xp4kO3FjPqvgshNllWdZlvT0cjoybuWXGhzY Ts4JJWCWozpzp+AOHPCQs5FxK7/U4MAmKvO0nDBuEDWqmfgKLi8JgdskjFn6bBxTA4TGof/wtsjW IGrUZOoUYK6yoCOnwxbgyjE1QLCdHJRlaxA1ajJ3CrDhiHtq6eBNAa4cUwMEm3LvXZGtXtSoyeQp wFy+nskkb+Zbv9TgYKYy5bmiWr2oUVuT1Mgi0PuS6AxcOaYGCPYgB/kSsLHsX1GUtIXnORHy+rF1 TA0QbGpHP2S1CSofJZCKoagQqFu5lHWuto1jaoDA5DRAYJuQgVmN2pqfAgtLz6uuzzNz65gaINiD HORLxKRHsTBU7MQN55lBfJkTG7/U4MCmGe2HLDejY5UE/4qaqMAwaEafX5CBN46pAcJuynTis+CE P+k6+FfExJ3M4tUPvsBWbqmBgc1U4rPchD/pUCzI9jZ+guN4edP1eQHYOqYGCLaVg0RuQhwQ2cO/ 4iTeWQZedL3JY2Ljlxoc7IPMJCGrTfiTGoV/RUmBRSCXMJVFxq38UoMDm14Q9lDiJTEKyVQxEvW9 KDbfDwW2cksNDGySmrBFakIr0pIL/4qPZOzwzBxyFjZuqYGBTQ0CwSlKMxphGV/TkWf+dF6KYizA lWNqgILlloY9lHiZZkJNRzLNRP8M3hbgyjE1QLCdHBRKxMwzoRattPD2PG2l3DJw5ZgaINg0Nn3M YjNq4ZlQi1bvOWJegQabc7xxTA1QMPzesDMhaeGZsVat1Ge0KvDblWm5cUwNUDD8AXYoETPPjLVs pT4DMO+YtGhn4MoxNUCwWW32hZB6IRpSEjdgHsCWNwoXM/DGMTVAQfNuE/rCSL0QTaxlqzccMZex Gwtu5ZcanKBFbvaFkLDWafavZKsMdmsNF3MswJVjaoBgs+TUmUfgOEoglW71TKGWCdoNWbtuHFMD FHrRrpr7kYEHCeSmXVmbApiJ1DldgCvH1ACFnicGbF8iJqLBRWm+XIGBaJju3DXeyi01MLBpvsEW GsETE83EnqdhgQ0Ur/E+9wgDbxxTA+QjD+RgMpHAn4gmiqa7AhMuT6HeFNjKLTUwPjJ/wB5KvEQz qHz6U2AlDzwm7BgKcOWYGiDYdAJs4RG0PUUQNcdzBabUGp7yNm9rW8fUAPmRM4LpIUQCf8oY/GMV saWIdeSCyxNz65gaINj0hrCHEjG9cSTlUAFTxJoHss1dt3VMDZAfueZgC5mADKgmo+EKLcBMoZr3 S2tlmdg6pgbIj9wl4BQhE/jTWgf/8TZ/aJgBmMeENQW38ksNDmiP2hr2UAKmPTRaHgIFl1Ct5caL BbZySw2MDzyGYI8hw9LeHG1NSW6kQDXPIBNDAa4cUwPkA89NrC7CJFg6ggRSURInQPNySR3NuBu/ 1OBgCaJBD1s0HPyJK6OrGYlGJOb2yMCh4FZ+qcHxnnkEdtAZl6RUdDUhOU+V1vOQMMNYgCvH1ADB Zh7xWcNhDR0lkHCbPrJH9NJ6LlfExjE1QFiLif1hi4aDP10lDjUh0ejtuZJoWBTcyi81OH4QHvFZ wsGf1hn4V4TkqMh6pg0aFRm38ksNDmymkZAlHHTJKHFUfOQM3oxA+xzsxik1IN4JhYQs3+A/SBA3 LiJA9faLr++4OHnfpVLV2J70SH+Iq87z3du7zw93nxwOTml1eMTvLK1dqJZ9+dBTfxLXaItR2ONn h8SSQ5Rt+ZcCR3Z67MeH+7s9f4zq8PPdtx1ute4Qcze9ny87DI1OLWwfd3usTJ2S//0g1qyelkf+ YlanRzXhFm/fyK/uTylNx4fd94ev6IZ5rvpMMXSm4yHDfw5f/l9HI12hpMuA0CKWtX350CvtsTs4 2dd0xItRvvg9OEeUnp4S8ylyNX52+IHw/BWPWAf6c18+VHjoBADS1RKnWutrqjGoOdVvjsu6TLvw ynTrcjqqX3d7M/TdA+7XzU/LT/P5o3o8n9LO0+UX9l5PZ76oepo+zme1nlROMHJiek+J5Nd019NQ fDitW6fz+3mtfruTy/xVckChYC1EvknOv03JFeX3KaFwS0qcpOT1L/P9S5WMV0N3el7xqmOHL6cn 9TCtk1rP0/HyiOvjkdU3h0Pz0DrQFQYkJX+gwHiFMkobjIHY/6PLWPo17iJ/axCH6ewADBBpn9vz 9vl5Dyf14zw/I9vrfLz/qKYVRZrw3KFbcLlXtjsuYr7s9shMJ4ZauJD30Gidep7P+RfrpXpp2j3w 0nKcvZ4MJuGTX46X09Nyv6zzQ5UwAhq65XKh9E4XtST5ZqZ+GroHFNcs7lVJXQuqOiW/1XfdtK7T /Ycd5mmHk1CL64eZm2+HeunwON/tFOp1OaoLGpVPQm11z9OZjnrmH073P87r5Z+8oPVQUhEj6+9f 0P3pC95AqhfUPNFJH14H4+05NSeVXnM+Tu+eZgzi+xP3Jj+UokmD61zUh9PxdF6O71VuNWqw14c7 S4xglXeQsHggyul+6Ekk8kx/pJn+R4Fi5TQjdo460N+9BH1EaHjSl1kaIR+J5SSaUZkBiXJypO6J gfT1yL9pfWfBFB7rz380F694zVy8JdpKXb1dp/XlUmoW62Gp2TwLbafqYYayW8og1R3Nx3YicprY e7cpkO3DoyV9pK/amtfXmr9IdLciuOS64ClNsxuzSr17WdXxtEooeBFHlI65MTC1gx1oq95b2lH7 TRn8QeYiFu0QzSZz9WTPBUjNVmpO6iuiqsCPWLDkOK0j2F9O+20Aqu8FAwplbmRzdHJlYW0NZW5k b2JqDTIyIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA0OSAwIFIgDS9SZXNvdXJjZXMg MjMgMCBSIA0vQ29udGVudHMgMjQgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEy IDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMjMgMCBvYmoNPDwg DS9Qcm9jU2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQ0IDgzIDAgUiAvVFQ2IDg3IDAg UiAvVFQ3IDQzIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDk1IDAgUiA+PiANL0NvbG9yU3Bh Y2UgPDwgL0NzOCA2OCAwIFIgPj4gDT4+IA1lbmRvYmoNMjQgMCBvYmoNPDwgL0xlbmd0aCAxNTg1 NSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZRXzc4kqRG891P02ZJZfpK/sw8+ Wz74AVZey9Kykj3vLzkyga7KogUejQTqb8iIKEiSyF/+8qO8f/3xtqZYh7H69HbvH7/+8frlr393 73/9eLn3v9+vQIZCeWdKpmKOxqX3n7OrptL7v/98/fYK1cSabwt80QuITMl5g8AKolIQrbFhFxLx p5oOmFVhJmtCTBudKZoU4h7TBcYMAzJbk+suIkdTIx0g0x2yOOMcbVQiwquAZEI9UHiSCBoh1ZkY wyGk6JBksgt7WcGrEGedqdXvaUL/+PKJycbFQwzpEOdNcO4QIl9D7hOTDWW3/5zodYz3JscDT5SU o/iJyaYeQlLQISEYe7gsKUtI/YQU433df02OOoaCibbueYpIi+ETU0xKu1LAMaIt5hkTg+TSNqbG R0wxtewqClZbOZ30OZ1Exqf9vlWbHjHVkD/EOMm29DmeTCaWXRXhktO1ffY6V5zZvlhxmVIxJRrr 98WIrxrG/Dmfag3iDjHlERNNoH294eTkmHk+3lqT/L7kcKbpmGhyOcREybd5sb1jmH2Z4qRRIck4 uy9TfP4cMsuU986Esi85fP46JplIpxjZgfLRFpzJdl9z+Fx0DNaWfQHhPVYx5I2jQ4wkAWrzDMnm sM+96F4R0Rva3QIfs8lhX19YswJNHrXg/4tJn5hibNzXF2ZQMTkY78pOfS6G6gk1adQSTIq7auJL wWu/XVEDznO3r1y7Bl+txsddVQkW1s0d6sHjK4KFm8txoyA42Lmtpwke7s0d7vrjTILHqrxFDfBv 4XC3H9kRAiydO6W2zsJAMHX5cLcf1yEQfF7YxYTojK2H2/8QEuHs8s9d/pBg7cIpJjxikkl79/Cs ZZBl6uHhfJTMkOHs/OFyP0ozyjLEHGL0CxAKnB3t7vbyzoQKY3ewKM/nLNQMq3OKcSqGLJwdHXbt 8TyThbWzByv0sAHkgqHyc3aDHKwdnWK0rSEfpG/7GftEvsivvbXTNo3QgXo68DzsIKEpJXvoih62 k9CnxnRovR72lgjW7uBRnjaa0Ltae4h52HVC8+rSoS96tAWEfjZ4v/+eR/dB6GiTPfA8mhxCk5vT qTHSzRQVzqefa9qQ08bub/ajNSQ0umGf1M8GlNDoRnfoix6NbkSjm/Kh//LlEYO1B99SPOkYdLrO nWIk5BORDbzMPsIlFYI2l8K+4hQXdAi83tbXxABnlw8nYSWt/Hy7I7pauz29iB4W/cteqo0aFT0s 5V0xiehYU9iXm2KdRkXHmuuu3MQEZxd3xSWiG/VhVxb+88J7yDTYmYrnC1NGJcJU8MhgyT/ef7zs G/9wgr27gy7LTxuSGuTtBY9aAeBNxOX7/YNnEn44+I3KTydOE4t/f/32p9ffmBSW4kZapfzBJNkb KfweZ1EfCeo7uwJsmm7AwkwKN2oILy2ySxc3PN/FTc445nZI6jt34NPoIwry4FaATdMN2P7VqPpY iD+5O3OuyK2L2RvYMZ4IaXwxE5fcPhKUdWYF2O5kAxSXUpiDCbwQRUoxF9zyixmPOW4BphjvzFK4 +4hCP5gVYNN0AxbpL9zcJGIpX/Y7d5Ine3ITbkLiKSvuzLWsj5TnVyvApukGbB15ZiUtuF9S3Dix G3eU2szGWnFXrol9JJT7zq0Am6YbsM72RMODzXlRkAiKHIeGVmmSZyk9mNyd3EmJ6iPBpwu5BmwP voHr5MBReyQz8I4pbhzbjRuOj7mLZNzF7bmQ9XEct4Zrimtgwgp25twzI0uJuZj5QbmYa0+1qlPN iY/tY8QF79wKsD34Bi46Rd83PRkP31iQDYqeDeKHHqfI2YZJZRufwByjH3VNA7Yn4QB2EflZRQAs F69HStwFoOTmmwAnKYdJpZzLIkBG7q26AAXYFsaB7JI4A1YAs4yAyvbhrqDIYzMVeKBAQXjkXZUT kDHO0q4B28I4kOEAylDgTeIAj3S5K0DDdFNAspGYPP74UcA9yBxjnHugANvCOJDxgtmhwKFWIQCf 5+4KkpBNBVEyCRMpBZ5LXR9jHEVeA7aFcSB7BA4FVvKmEjjuCnCaNwVJ7hGmpBRIb9bHWes1XlsI B7B3MxHxfHLe1AiKuwAc5k1AlqPEVJSAyFWvjzGPN1YDtoVxIMPVjUQMuedNAsddgZfbNxUUSedY BexSIFkgSVBGIdBwbeEbuNwrDn6sLAjIYLjzOyk+g5/bHfAn+0jDwuWvj/yadwUKsC2MA9kTrnVX gL3igPpIQyuFdypwkoYSfFMQxPn1MdlRiDVgWxgHsk+41l0B8gVtmgWSVOuhQLb7UuAlDeVz7grE Y/UxuVEMNGBbGAcyjHqIUpEDW2cEwBUpBVlq31QA0wF/jYnT8VIg/U8fkx/FQAO2hXEge7xZQwHK LAfgK+tdQZLyPxXAerACknS8FEhr2Eeuv12BAmwL40D2qJNDAV4aDsDnKQVR3r+pIIpjxsV2SoE4 rz4mmnugANvCOJGttH5QgP+JHIDPUwpI3v6pIIuLwxQc3RRktiF9TDSqgQZsC+NExnM5FJSeOPg8 paC/QFNB6Q4WO6kUVL6HfUyjHGi8thBO4DAT0eeeN/g6JcCpRKxy3eWJuwkQSyJDSnMDFFxb+CYu SZfC/HywCCiPNLSzt2L+bHt7Z8VXXvxe2h0ZU547oADbwjiRIzK3K+DkRkDVaSg27FLgJA3F59wV 9C3oezDtsAZsC+NETqjUXQFf8LeDZVVpKBb0UhB6MxF0GnLbM8dUR0HUgG1hvJDtUMC1jiO82JxL QpZWb0qAK8nEE6fjJSFLFyRjtqMaPBDbwjmhKyplFg1c8TkCHjbfNSSc3Gdhpn4QsPv5rqFINyRj dqMePBDbwgloi99wETQ08LvHEciWLxrGwhx6OqLfuWvgwjvHPJ3yA7EtnMRfgt8O2SEaHL/9HIF8 +aahL8xeriQmUhq6O5AxDwkPwLZQEn8IfnukR5eQZRMdzGz5JqEvxJ3gPMDklQQK7znmMOrCA7Et nOSll4U0OzSw/eMIJMw3DX1htnIdMFmlIdb3HPO4FA/AtlCSk+xAawG5IoE9MEfUrxk5FqYqNSEV ScxLQu4+kUTC3AaN2BZOsj07Ej6ra+BOABHefs/IvjBlqYyYstIguyCbMB+IB15bGHFBC8wtrkoc CgL2hyPc93zsC1OStwFTvCtgjzbH/ElIjdgWzsDfgd8FmF0Dt4Yc4b8nZF+YYndsUR7FS4N37znm MrNBI7aFM/CX4HfFznYN7HX8m4sr0BcJfV2iblvRxikJQeqzjLnObVCAbWEM/B3oBRxyI7ECvCgk AciWVcFYl0K37mhllYIodlHGYkeB1IBtYQz8Gfjtxe+ygoLs4QBkyxcFfR3sd+3WPSsFqbznWKZx 1oBtYQy+90NwwkNBRu5wAHLli4K+LllxzMmKf74UFHrPsfghQOG1hTA4SQv4qFi7ABjSyAGFK8si oK+D++amAVO8u9ZUxS3KWPyoCRqwLYzBSlrASuahAKZcAuq3RBzr0EFy44Qp3BXwtsxx+hWN1xZC /F2yImFnugBsDgdA6bc87Otg/7l3xKSMe3cJQejnBii4tvBhYyQneOr8yA4J8F+zsK9DuGX+qFuX TOwW+1jiKAYasC2M7Gpd4eMMQwFL5YAAylVBX4cTLKyAdPuWkxgEGUua90ABtoXR82fUntH8u9ae NLgvaRXwP7arXgeP3Ab2eYp9Ap/+f2p3AXKNy8N1qYKoCK64188MSe2upIUBrj9bmhlJI4q0YTBx iWV6+RFQJQ1JbPY6rnjj4AtcRNc7Lfw0a8T4DCP3U4COwzVOsczb/Ajo8ZqxNd2BDXAcjIGr6JrX RAEvLCfgAn8p0HFIZSGWmdFuBc3JFdTYNRtugONgDFxGn7kdCpizOAE57EuBjkP176BAsvpbQdAs xNidJqMNcByMgcvofOCC5J7Ov3AC8viXAh2HJ40dED4tvhUkUSBxbsGKNw5C39UWwFcBqJ04IfF9 /RCg4/CuVwqoYsdHgFQmGrvXZLABjoPRN7UFsFQB329O8N9G1HH4lamgiB8fBTVfM9qrvOGNg9Bz FZ3lXlMBLGI4IX77UMehwIsUkMWOj4Durxm71cob4DgYPZG71ryiQJsvj0IWL/qhwMbxf6kgiR1v Bd2V647JksEKOA5Gz2V0rfz5u3X1DVf5pUDHod4WH4bVhz0wE1rMegob4DgYUcY6635EQDPb4Jy/ BOg4nGnNd9/zCIjtumPRLdgAx06ItrZZByj8xVwDp7t28uu4xJbkbv0e/hyvO1qNvAGOnRDla8HP ZJZoWQp/z6v+xa/jeK2R9fEJC3+RG6hRPbjhjZ0PpSvyRmXlkkifpOr3yHX+i17HMa2RHq9qftNL PWaxaybcAMdOiBXgF95IpY/SiHpk+fhFr+OY10mPsuKhRwdwae7bQMbOIcePakp/BWRkDg6wx83o Lv6R9xyJk01mZd5JAAN6QyvI1odN6H+pjVW+DZAtZBdpRySA/yMNjyE2qUljE/tKs/TsHzlZMwgt KhaHuvJNy6YUKoNX2YL3Q4oYFliowSqzc3p2rXnZ+0naZbusS7wPzXkmDI3JEtcGOFY6g4WDhBsG 49BmBzq5nRyCcScvTklefPNwR6ZLjSl1414Ax0pnsLpqWAsD8U/+zVy7nOhkDpqued1fqcol1kwa E5Qp8wI43mQGikpYmOEwDvRmosnc9OSNOcoTlaI8WA9zYXbQmCxHbYBjpTNYtAHCDcNyKAuyNzdd +HAneaBTkuf64a7MjBppTuVeAMdKZ7DdfObEFo3Wf3PjxF7cWV6FlOWNeLg704LGNBuWFXCsdAaL hCFGQ91FXzTW4m/yJJ3ZJK9ao1V5Im9yL/WhxmSZaQMcG5/hejlwZE9xBjuRN3eUnmxyNykJUhPH PdyBRYlGO+4Vbixchon0osxVncEe7NUe4PnyL+auVuur1VD6XDOyb1DuBXBsfIaLNzLoppcfAc9J Yw/6pnfSixk9TlG6E7e6jScwYw6W11bAsRMaMIqEVLsIyLgrGA9LvAWULq3YFODFcvgslvNVBEjM 0ZuABXAcjIaMQqmYAiR8TEAp1RYFDQCPgiBNAe7x6jupBzXmmdpXwHEwGjKKxWYKwo/CCQF2eSuo 2MNHQZKNxCfEV48aHFOcxpznHiyA42A0ZFTszhR45CpMwPL8W0ERsqkgi5PwSYuCwFSnMWdL8ivg OBgNOWCiKXDiG3QVcVGA03wpKHKP8CmLAikINc5cv+KNg9CA8fqbEfF80jdorPIiAIf5ElDlKPFp i4DMrKcxV3tjV8BxMBoyOkczIsoe8U0Bx1tBkNs3FTSxc+4C9igQF4gJmiWCFW4cfIYb4rQh6q3a MKGC4c3vJfkYf3Fiw+I2GzamP418zVXBAjgORkMOCddaFWCvOKFvNnSSeKcCLzaUyS8F0fESaizO EvEKOA5GQw4F11oVwC8dTwSQJFubAtnuR0EQG8py3gqkxtJYvCWDFXAcjIYckLWzZOSIK8MJqIoW BVVy31SAoqNnfmjHR0Fka6axBEsGK+A4GA054M0yBUiznIBV9reCIul/KkDpQQVJ7PgoSLyEGpl/ VcECOA5GQw7Ik6YALw0nYHmLgizv31SQpWLGxfaLAqm8NJY092ABHAfjRGZzJwrwP5kTkjQ6j4Ik b/9UUKWKwyeirngUVJYhGkuybLACjoNxIuO5NAVNjYPlLQr0BZoKmlaw2MlFQec91FgsHax44yCc wHEaMVT1DVa3CPCLEbtcd3niXgKkJJFQytyABW4cfBM3SZdCfh4sJrTNhm72VuSvThoXeeTf/EHa HYmlzh1YAMfBOJEznKsKaG5M6KsNpQx7FHixodQ5bwW6BboHsxxeAcfBOJELMrUq4AVHL+pWG0oJ +iiI2kzE1YZse2Ys3RLiCjgOxgfZmQLmOs4IUuY8Eqq0elMCqpKa+KEdHwlVuiCJ1Vk22BDHwTmh OzJlFQ3M+JyBGra+NRSc3D2wJj0IlPv1raFJNySxessHG+I4OAHt8BtVRDINfPc4A2750GADa1Q7 ot95a2DinbHOSnlDHAdn4krw28MdosHz7ecM+OVLgw6sQa4kPmnRoNWBxGoSNsBxUCYuBL8D7KES qmyiRzHbviToQNwJ+gCfsEhI8ZqxRssLG+I4OFOQXhbSnGlg+ccZMMyXBh1YnVwHfNyiIfdrxmqX YgMcB2Xy4g60FpArElgDc0b/dKQNLF1yQmlizEdC1ToxiYS5DSviODiTU3cULEs1sBPAjOC+HakD S5XMiE9dNMguyCbMB2LDGwcjLmhDcYurkk1BxP5whv/2ow4sRd4GfPJbAWu0GettyBVxHJyR68Dv BkzVwNaQM8K3IXVgyVqxZXkUHw3BXzPWNt2wIo6DM3Il+N2xs6qBtU64mFyBfkjQcSVp2Yo2bpEQ JT9LrH1uwwI4DsbIdaAX8PBGoQK8KEkmwC2nAhtXopbuaGUXBVnKRYnNWYJcAcfBGLkM/A5S71JB g3s4AW75UKDjUH53Ld3roqC0a8Y2C+cVcByMMWg/hErYFFR4hxPglQ8FOq44qZiLk/r5UdDSNWML JmDBGwdh9GIL1FG5qwAUpJkTGjPLIUDHofpm04BPfletpUu1KLEFywkr4DgYoxNboJSspgBFuUzo X0a0cegg2TjhE98KuC0zznplxRsHIf5dXFGwMyoAm8MJUPrlQx2H8p+9Iz5L4a5VQhT6uQEL3Dj4 sDHiCX6UH+6QCeHThToO0x3589q61MRqUWPLlgxWwHEwsqr1jccZTQGlckIE5alAx+EEGxWktX2r RQoEia3Me7AAjoMxcBldHc3fvatpcF/KKcCGwcQllunlR0CVNCSx2eu44o2DL3ARXe+08NOsEeMz jNxPAToO1zjFMm/zI6DHa8bWdAc2wHEwBq6ia14TBbywnIAL/KVAxyGVhVhmRrsVNCdXUGPXbLgB joMxcBl95nYoYM7iBOSwLwU6DtW/gwLJ6m8FQbMQY3eajDbAcTAGLqPzgQuSezr/wgnI418KdBye NHZA+LT4VpBEgcS5BSveOAh9V1sAXwWgduKExPf1Q4COw7teKaCKHR8BUplo7F6TwQY4Dkbf1BbA UgV8vznBfxtRx+FXpoIifnwU1HzNaK/yhjcOQs9VdJZ7TQWwiOGE+O1DHYcCL1JAFjs+Arq/ZuxW K2+A42D0RO5a84oCbb48Clm86IcCG8f/pYIkdrwVdFeuOyZLBivgOBg9l9G18ufv1tU3XOWXAh2H elt8GFYf9sBMaDHrKWyA42BEGeus+xEBzWyDc/4SoONwpjXffc8jILbrjkW3YAMcOyHa2mYdoPAX cw2c7trJr+MSW5K79Xv4c7zuaDXyBjh2QpSvBT+TWaJlKfw9r/oXv47jtUbWxycs/EVuoEb14IY3 dj6UrsgblZVLIn2Sqt8j1/kveh3HtEZ6vKr5TS/1mMWumXADHDshVoBfeCOVPkoj6pHl4xe9jmNe Jz3KioceHcCluW8DGTuHHD+qKf0VkJE5OMAeJ2NsUkTWLJ2hfYzxt59/tevnr8vxDx/7Xz9/h4x/ QsZ/7A7+fXl3/ev64093/fsf7JLYr1amsARd/+e8bHJ0x20oOq9V1BKsf3kZQQZZQc8eA3TvfxDe S8qWzFnQgL9mPfJYlijyUhc6darEFIV59g9eioHBHXgaeJ+f9p+ff3NpE92srqXpz2ULxZH8n0s7 XyTfpdqb3x3e/rmP7Us0vt3DzjGrrfcj0V5Dqh4C/s5dthx5s8Y15OkpAaUH1s0ehta+Mi1gbuhV At6L+HGUQFL7gq1p85A1iZArdSNT8t7Vblp60JunBNRMTDctmMlH0KkbqwE3s2+hu3WPulXwj6cE 1NTlmz0cPfRaMGBsaKtJqfoLDH14SkCpjSuj7as86KyJqn8vG7pa55tb8/94SkANzKvVepihO+/A TEzEheZWQFhaYzP04SkBNaCOaPcHfVnA2NB5qfZXN348JaAGZtxqvd/QTd+tAVB3LzpVH4qgSG5H H54SUAPSgPZ0NFQzA+4NfXHVxdY+HH14SkCNzn+gclrohIBCP0fPmwoqcUNaftCbpwSU2vicOtgI gEZt0oDKr1voaVPC2EbV01ECSUvtMHs4GZ/HgLmRKfjGTK7ADX14SkB5Fa9oAQudNAVmYyItNFPD h7tUHvTmKQGlbbXoPuqIcA1DJ62IGtC9+Ri6tWdyxa4Y+vCUgFLBMMzujrYCqgFjQ3MIYIkac1WQ w1ECSW0fb/JDzkMDBoeYRS5cbuLSn5J6eEpAucpqlw9eKkL1pRpgomyhOTt1JuBoXkFOTwkoFZA4 /ZZ8rkXAZQF9Qyesuk/f70XeHCWQXJo2nLWRUdY1wKTsIl9Ybu++30Y+HCWQVHZ3s4eTkZMMmC/Z Lkqv3I7k9eP0lIDqOl1pWjVcq4VWUfOrE0C7X/TkojOa17jKg948JaDULrBRCwyN668BWoE39OCq WT/6KqinpwSUTkLD7OFo6EIGjA3dcYiQkfjodYqHpwRUhzhVW+eOB521cOpYWDZ0Q2q0Mf0sF3rz lIDSCbIWtTsHMKDRIDRAsz+9aN7vxjLSV0E9PSWguk4LZvcHfVnA2NCccFoluowHvXlKQKmdYQ8O rkCjGWuAFpoNnXGMjc+evYKcnhJQvWmq0MZYCDQkhAY0XdKLTki7llBR+7Xy+vCUgOpVi7naUEEL nSygb+gL662T23LPB715SkB1fI7a0G6GTlzdja970P3GqitLapvrGA9PCaheWCnVHo7GtWbA3NBc b23c8XWKh6MEUtfmpoWyY0mLnBAw9Ks2MmU/t6O1h7w5SiB1pHv/7ZD2sG+IefVH8r9gI1M8If9A Ph0lgHpijdKfbuDJfnTjwm5gcC9emWLV4/STwOmJZVVflxcXZfdGjckPlzKsTO5x6gu8OUoA+ZZw gwge7BTYQTSOhwzVUQaetsUfPwmghuqttlZW3XmCLw/oG5jluVAkVNdLH08JqHZz8ZqIydCdLRmZ mnc018xWXr1rfTwloNrkhqs9DN14IAjA+Sz0xVVnz2iST0cJJL/lfudJTgzIzKiHDGhhv60+Fn08 JaAaJInaKqcMXZn2KGXpRTesObMprmQ+/SSAWudNVHsYuPCmImBs4Ik1s4pWn4k+jhJI3gS8JZCc GVBZbBYZzNyqXxcn744SSA16VW3VUkbOLIlodGkjs1VlVo3iHfzjKQHVKpWh2n2hLw8YG7oh7zIb QPGR6OMpAdUq24tqhmzoxPYDUVE2NLtgvvAs3Q/x9JSAcm3jSkfRF5smpBB66EIXrDqxShfv4B9P CaiG6aP/Dqu8RCcP6Bs6Y72J1RTX0dG7pwRUyxQgqioTbRWDECiQnXlDs8FC0eE6Wvn4eEpAtUTZ pPYw9KSsQsD9FlPck1SQgMUnoo+fBJDrdVfvBCcG3FSGDrbOnRJz2yeij6cEVEOajl+dOS5DDwpY DCVpR4PMEpJ9Ivo4SiDpaDTM7kbuVN0IGBu5I52v2e2mk3w6SiD5OOfDHcl51F9Mf1U7yUNG4l3s idkHoo+nBFTF8EH7NrTqHgboPdvQlYu2+1gtpz+eElB16Gep3SjCib48oO9orro0VpFF3h0lkNTG pDi0uhpZv6YhQEvaRsZluTjK5eRneDpKIFWMHrTvRLKWXQ+4byf/Da9aHq8yaOlP0kP756+f//z+ 98fljhcZvvtESXhXxWyitubZendigBbo9e5//VyAErhE76RiKp3R7yrUO/3iP+q4nHhK+jIcmgA+ 9X8KVNiwlyFR6sVwVYj5YsaMwb/7lqIh63NWPl0xn54SUDPzndVmVEUnqBAGzA2NG3d3pMHtGuPj KIE0MxOlJl5EIydN0oFMfMlU4TdT+O7pQW+eElAzMVNqZn0BerIfjdszxdC8cnfhfrjI+HhKQM2L iVAxwxgaKoQBY0M3rjpVf4GhD08JqHkxZSomr4XO2o8mStWL5pWbtuOuMz6eElD6zqrLqJgXDY0R UQO0GaQXza2Yk2t3ofHxlIDSQ5hm9wd9WcDY0BwfJuXGvNOD3jwloAZal9pNfwwNJaIB6GQvmrs8 WfSnC42PpwTUQMOlPR2tAjUx4N7Q7CGz2Np9XDs9JaC8JHiBMHRCQKGfo+eNyzITN8SFxsdTAmp0 fo7WveRoKBENqPy6heYB4pJhW5x8OEogjcYzUHs4GZ/HgLmRBxY9JjMwL/ThKQE1GjNHG0x50ElT YDYm0kIzNUbPvisLvXlKQHkj9LYIdGK11caJ9F9ojlWjdt8VQx+eElCjcGBRuzvaCujsvLQLXZF7 LFFjrgpyOEogjcIhq11sXkbGFDYHh7JF5sCGKUGfT0k9PCWgXCe5alL0dXN2nJOj5EJnLLozAYdL jY+nBJSqO5x+w7Yt9GUBfUOzYVPiYr8XeXOUQBoYQtXGWRsZZV0DMJO+ZM6Cvft+G/lwlEBSFd3N Hk4e1GjzpmRzsl2UXrkdyevH6SkB5drfJwFDQ1jeF3XmQnPM7BnNa1zlQW+eElBqF9ioBYbuVMM6 4aQNPbhq1o++CurpKQGlc9Ywezi6UcNrwNjQFKEQa/jodYqHpwSU2hgrdFYsDxpzh458ZUM3pEYb 089yoTdPCSifaX3CBbpyWroLh6eF5v22gbavgnp6SkB1Fe9m9wd9WcDY0IWrpiRFZ1rozVMCSm0M nm1wDAW6cDK9Kyephc44xsZnz15BTk8JqKUdoScMDQmhASq9y4tOSDsbDfu18vrwlIDqqv9NlrYH nSygb+gL67UJrt3zQW+eElAdn6M2tJuhE1d34+sedL+x6sqS2uY6xsNTAqoXVkq1h6NxrRkwNzTX Wxt3fJ3i4SiB1DPnoY4lLXJCgOr8tpEBtixp7SFvjhJIHenef3viAPDn54aYV38k/ws2MsUT8g/k 01ECqCfWKP3pBp7sRzcu7AYG9+KVKVY9Tj8JnJ5YVvV1eXFRdm/UmPxwKcPK5B6nvsCbowSQbwk3 iODBToEdRON4yFAdZeBpW/zxkwBqqN5qa2XlTKsBlwf0DczyXCgSquulj6cEVLu5eE3EZOjOloxM zTuaa2Yrr961Pp4SUG1yw9Uehm48EATgfBb64qqzZzTJp6MEkt9yv/MkJwZkZtRDBrSw31Yfiz6e ElANkkRtlVOGrkx7lLL0ohvWnNkUVzKffhJArfMmqj0MXHhTETA28MSaWUWrz0QfRwkkbwLeEkjO DKgsNosMZm7Vr4uTd0cJpAa9qrZqKSNnlkQ0urSR2aoyq0bxDv7xlIBqlcpQ7b7QlweMDd2Qd5kN oPhI9PGU/5Fd7rqW5DYUzfsr6g+s9yOezIANDDqYwHDmSQzQwIwD/765N6U60iEaYF/eS65SqSRy 06FqYXtRzZAMHdl+ICrygWYXTAE2t/UR70hxqKVtltJRdGDThBRCD93ojFVHVum8OvhXpDhUxfTR nm6Vl+i4EtqBTlhvZDXFdVzoM1IcqiYKEFWVkb6KQQgUyM50oNlgoehwHa18fEWKQ9VI2aR+N/Sg rELC/BRT3JOYcQDzmoi+4sSBll5f6p3gyIRJZbjA1rlj5NleE9FXpDhUxTHtj84cwdCdAhZDSTzR ILOEpDURfQWKI+lo1M1vRm5U3UjoB7nhOIfR7KaTfAeKI61xbg13JKdeHkx/RTvJS8bBC+yJaQ1E X5HiUAXDB/1paNU9TNB7dqALF233sdiZ/ooUhypdX0v9ShFOdFgJ7URz1bmyimzyGSiOpD4mxa7V 1cj6NhUJWtIOMi5L4CiX4vqGd6A4UsHoQX9GkrXsroQ5F/kPRJX8RuVOT/+L+tH+/P3Hb89/fiy5 s4oMn32jxD2rYDZRX8/ZfnZkghbo/exff/zll/+O55efz/2An7/8/Ud8/vrE59/6m6n97n96B56/ Pf/4Z3j+9erjQXGVGx/0WfDPH4Gk1c1igrSJWBm+sGDbIl4Ukq3byrDEghsNP6bAJerSM7eR+29b bxaq0tBXpDjUSNwN9aeh4+R2acI80DwuZnE2ib4jxaEGqpf6UOEbjY7VxzpWhuYhN7slyVekONSI /ENJrEZADx7dPte5MjSvplmUAUNfkeJQI/B11B95o4Ml2LExNOuJWVzCjT4ixaFG4EcomecX6M4q McI6gURbFTSLimvoK1IcSp+Jo1MwXRoaskUTIk/SRrN0my2zLfQVKQ7VJ6uD+u1FR0sY+YNmwzGL 3rbRR6Q4VEcFU79qlTB0pQjRz10OdP20yrrJV6A4UkcFoz8XWaskErR5HGR2drN7nvmKFIdSH2Vi VRNDp8yEdqCtV9qsm+eLPiLFoTp+UN+KJNCI1AQrowttKspsLfvsXZHiUB3Lpz9edLCEeaCp/czW Ol/0ESkO1SufudoR0PgqmlC5hI2mYjVb2z57V6Q4VC9UmauJAg3hogmNG7fR1Nlma9835ooUh+qF 9Wy1fkNHS8Dn3ui+B4Q62gs+4sSBeqaeX2oFYLsRo3OY22COM2br3Gu+IsWhVFRhCFkSS9FhckoZ g3PnQtsMZrYFa8RfkeJQPXFyqqiGGx0toR3otKfHFvsLPuLEgXq0VmhSFuDBYXCJ3Q3msGu2pb3m K1IcqqPT0J8L3TnCDjSmD7pyO2hRdgx9RYpDqR9VVK2xwdCpp2cGitKN7vHZtpX6oo9IcaiGrq6+ DTtAo+1rgo1DGz0o4GlbXUfvjhSH0qFsmj9edLCE+UFjIN22tfKij0hxKPUhYNdgCbT+hISkr/VB p7SFEldq6CtSHKp1yu6qK1joQl0+MyetV12OZ9s29qqvSHEo9YP5vW50tISRP+jan23bjC/6iBSH ao0DTlVhtdCZE9AsusgPuudn276E6VekOJSqR4xl6s+FTpzbpur0A83eYhaVztBXpDiU+hglVZjW F41ZUxPaB23Dqdme8os+IsWhmspm+FOPv6GjRmpC09f6oK2Y0vZNvgLFkZoqePPHSw6WMA9ysYbI rS7pRR+R4lDqJ62wDd/I0BDGmqBDwYFuRNP2fc2vQHGkpsOUnsEGaQ9/agFG/NCX+oD7eLbtzdZ8 B4oDqR/M7xscLX4c4MlmSLvk6R0njqM/4D7qr7NxB++rpuH2Lm6L+dm2b/AVKA7UcD7pW6vRLeta GbGD4SCn+Wzbh0mPr0hxqAY1q35mMyM6rgR0u40u9dkWPXqhz0hxKD0iqNd6FrOhG+s5TlQ+0K08 246wVn1HikPplcGWqz8NXdmFkDAPtFU6gKPV/684cSD1cUbW9EkwzhCubj3Ak9tBO9LY6DNSHEor CPq5liqcckUX9nvUMlyDhcbWbYu+QfQdKQ6lJXWaPzY6rIR5oPOzzChjg884cSD1UUC0chcDZxYY lPZ6gEt7th3VbvdXpDiUNq8Gf7AGKlrlFBMqi+RGt/RsO9o6eHekOJT6wfy+0XEljAM94rPtWDXp K1IcSrs5ipPKhmzoyOKF5l8O9OzPtmOsVd+R4lAqb9AX1Z+GDuybUEPhgx4sHMvOteo7UhxK/ah+ D+ztRKeVgOa/0ZSlZmdIG31GikOp3kPzUmGZ6KsebExQiXKgy3i2ndGkx1ekOFSNvAHqj40OK2Ee aPYUszOljT4jxaHUh95TpV0MPagHIcXrgR5E0868Vn1HikNV08g6eECzKlp3DgkxUNRu9OSG0M5i 0uMrUhxK/WB+3+i4EsYHjf19bc0bfUaKQ+nslXUe0UksG1qVhLY+jGpFxflG5/K8tq1V35HiUDqK dvOnoVVQMUHv2vygy3xeu2TNV6Q4lPoRvsbFYui4EvqBbvV57djkM1AcqWiFha8vZ+RCKaCbqG/1 IQ/uhNm5zt4dKQ5VGiuD+tPQqo+7JcyN/gNRJb9RGRqp47+oqX/+/uO3RwXUg3985o0Q9wz1kdhV 3Of1zMSEQuHLZ/76Izz4t6UqJFTHuFQUOEAuz2pyVMHoauvv3B+FQcoGajdsD2j6EgZMkd/wA7SE jPmr24qq+iXw8KiMTBBfEUvG8VobHiJ6jFkMMoa+IsWhRuIuFFwKouPkNvXOd9rojPNhthTTIV+R 4lDq46OWyE9saHz1PngINrqix5jFjLTRR6Q41Ij8Q0k8mEAP3TpNmDy6G924IbSl7VVfkeJQI/B1 1B95o4Ml2HEx9CCatvT+oo9IcSj18RFK5ukFurMsjMBvstB6vJ5tMX4Z+ooUh9Jn4uiUwsoFNAZJ TYg8SRsd0XTNltkW+ooUh1I/mN9edLSEkT/oTDQtNM1GH5HiUPp9C/yqTc7QenOQoP3iQFduCG3d 5CtQHEkP5TB/LrKWRSRkfasPmfrXbM1rxr0jxaHUj1X9xqpj6JSZ0A70IHoYer7oI1IcquMH9bsW EkMjUhO0XdQXnQJ6rtla9tm7IsWhOpZPf7zoYAnzQEe0GLO1zhd9RIpD9cpnlsGbADS+iiZULmGj M7bCbG377F2R4lAdJ6U+BefG0JE1eTRu3EZX9BizGHYNfUWKQ/XCeqZ+nxsdLWH0DxrnmaaO9oKP OHGgniniK26oge1GDFzh/IIHaqnZOvear0hxqJ44/NSw5lMILYyXY3BUWegcADXbQl0964oUh+qo f+pHDk+GjpbQDjTuIE2L/QUfceJAPbIZVlRwAw/OgAMl/gPOqKRmW9prviLFoXrgYFrRdwzdqTTG 5CC70ZXbQYuyY+grUhxKfcjKim650dCdE1/rg+7oWmZbqS/6iBSHaujq6hdKY6DR9jUhUjxv9MCh M9vqOnp3pDiUTmLT/PGigyXMD7oEdi3a1sqLPiLFodSHYtWRryy0/oQEnUsOdGLXosVKDX1FikPp 1ItxqeoKFrpwnpqZunyjWTnMtrFXfUWKQ6kfzMfIZ+hoCRgKN7r2Z9s244s+IsWhWuNMU1VYLXT+ P9lljwNLbgPhfE8xJ1jo/yd2bsBw4BM4Wwbe+wcuFqUe9hAL9Fu+V/qao5bIIoeejcnVoWf93OdM x+69lRJQ8I+zM94HXTiqwWAmh2ZvsadWumtNnVICCnHWeHGaNHSpXDC+aC1C9zlLfdBOKQE14O01 3jj+hs5QYgG8ff+irZjyOS/5JZRAGvDXFq+HnGzBduRmDZFb3cqDdkoJKMQFFXboNzK0GmMs0Jni ix5E8znvNX8JJZBGwY8an6HWXuO9OTTsdYYEA8/1uc85LOe3UAIIcbJ4XnA2/XLgzWbI57Gnb50E Dv5H7yP+uhp38b5imd7ewx25fu5zXvBLKAE09HwytlaDLZuojLqDyZHL/tznXGY9fpQSUEPdLOLK ZkZ0Pgu0211065/71B590F4pAYUjovUaZ7EaerCe64mqDj3a5z5XOlm/lRJQuDK65Yi3oTu7kC7Y Dm2VTsHZ6v+PTgIIsZ6RM38SrGdIr2534M3t4HOVddFeKQGFCqL9HKVKTznQjf1ea5leg4PWrbtP 7RtEv5USUCip2+J10eks2A5dP+ex2rpgr5MAQqwFBJW7GbiywGhp7w7cxuc+V7fb/aOUgELzGhov 1kCgYae4oLNIXvQon/tc4xy8t1ICCnGyeF50PguWQ6/8uc91atKPUgIK3VyLE2xDNXRm8dLm3xx6 z899rnWyfisloGBvtC8i3oZO7JvqhtIXvVg4znOfrN9KCSjEGfFM7O1El7NAm/9F05bac6dy0V4p AQW/p80LxrIwhh8cXACL4tBtfe5zZ7MeP0oJqJ55AxCvi05nwXZo9hR77lIu2isloBCr34PTboZe 9INqxbtDL6L53PVk/VZKQHXzyBg81LMCjZ3TBTnR1F705obwuZtZjx+lBBTiZPG86HwWrC9a9/d5 9nrRXikBhdmrYh7BJFYNDSeB1qejWoM5v+jaPs9znKzfSgkojKLT4m1oGCouwF3bX3Tbn+d5bM2P UgIKcdYYutwMnc+C6dCjf57numQvlEBqqLAa48cZudEKYBPxq77kxZ2w5z5n762UgGqDlQHxNjT8 8bQF+6L/p6pWH1VVjzT1j4ylf//3j/98YKA++h/f+UZIeAdiXThh7ut5Z+GCRuPLd/4LPwhzYzqu 59//+CcdqzqpqVNTA3fpC9rn9DqaYW1u59+5TWCqo03zmmMH+OuPf7us9R+12nzpP6vPz+rrGXwL DdLUOteej7DL9yOs9aCdUgJq6SzJeEzv4rFAT/PzfXlw/Jn8EUogLS2QTf3xncUy79yz90ZmHz7H feTHajulBNQq/Jyv+UCbKxZsl/TI30va+oN2SgmopbWX8Xiy1iI0tTh/0Z1Z5+UK4o9SAgpxtVeZ 41TDqG4DC7rLuuXTdr5l/EcpAbVg4/CqkViCDa1VeS7+/atqnTKexoN2SgkoxPrOkei0vmPN3Kwc F12uMfl24x+lBBTiaq+yiXJvdqm5uW+vKr56Oamf+cMJJYBW4tf1A5NaUeitChg4jWMtnaN6CSWA VuKJHPkMqnuzZeOv8zfjZXY4v1zgSygBhLjai+admPQ646/7N2N1UDob0Ki1OzA5nQTOSrz3o9Cs kKuWcSVWgcudtNrr5eJfQgkgbJHWKsQ9H3DFQIEvkV3CPA9z+sHjpZPAQdzsPfMmrA5maVP9cjsT 7mc3DtfpJHAQJ31P5Wj2TI5L/dCXS182eU1+BscjlADCxdF/QGy2Z6uvh75Q9jhKDgaZE+kpoG+h BBDiZi96BlKdUaBvLuPCjFM9w8EBO6EE0Fzc+6EzzxlHlyUyXMY8voPN5M4EL50EDmqqnhXEZ0ZC 59Y8qm9SOtnqlHs3+oCdUAJoTp7u0a5rH/Tiq/oWZddisE7MUi/YCSWAEOt1RPNO9Qy5yxJxHUoP gvnPRI9wZlwnlAA6fmP04+C32gPqXX+yiWtkBY87wbyEEkCIu71o3IzVgMAwuO7ECzfYQcatEy+d BM4crNGIzVBuOMtMvWtNNn32Nc4XJPgllACCkWQ70S9xwDr9Qe8aE29yZ9Z9XKyTScAgZivBz7j5 6jQCO1u/5Wc25tu5zccAvIUSQIirvegMn5XNHnrXk+hZetWsRzll4qWTwKFVRzzPCAN939S7lqTn FuByjsXhOp0EDmL9gkN98eFWzWP4jqT3Hlyein6xTiYBc4YWxNuKD2zSpt71Iys7rJn97MJLJgGD WC/IQOXoBzstDdeOBvtnG3Yo1gU7oQQQ5je90oj7zZdtZvp2ZNWs0Wv2US/YCSWAEDd70bwZs89M lqwLnnolGvuGHrcDdkIJIMRaNgfMkWWcrc9MFtkLHsyYFqjXs8cvoQTQLPzeiE9DytZnFrvCBXfN uNLX91stX0IJoFn4P4jnzZh9ZnHZBTf9bJUeU432ATuhBBBiTX2m25CSNRqdJL5gFuBKR9HOiPkW SgDNTG8z0+1IyRrN5qe54MKMeYzbulynk8CZ2aa9dBsSbF2mXs/S5bKw11p4mPcFO6EEEGI9z/jD +giEyxIZLmG20MoG3bp9u7dQAmgm3kDEdR9wt0T2N+PO41DZSFvLF+yEEkAzsWIgHjdjbTT4oVpf HjCIhe2uPfk6mQQMYq1viK2N4BNrm9mJ1fBip+Zbxjh3hOCXUAJobBbkWU4jgV4bzbaZ7gErl1Uo lYt1MgmYsdk/EPebr7YZnHz942JtH1gm6poX7IQSQIj1DYitj+DaawY7M58HrFtbWOXrcWtvoQTQ WNwRVA9rJNDrjkG/XcZVM86bB+5UzLdQAgixfkPE/Was33jr5ODAmnFmQa7n1r2FEkBj8cwhtmaC ZqBnchee0AtmC830l7WamXgLJYDG4i1BT7FmAr3aOujXt/5oMQOYZaKWy3U6CRy0Pb3WiPtNWH3o riwCl6vUWnnx9sU6mQTMmCxDiNc8WPXNu/qW1JYmmlmDyp4X7IQSQGOybsK6WCeB6ZiWiGtJ3IBM c6k3mtyXTgIHJkgLPWKb4aDXXrmb70haIlG3F8Hzcp1OAmcM9hHEMx+ujlK7+YbUhp60xCJR+rpg J5QAQsw+Ms4MBxu6LJH5rT7mI5JdvXZOxEsoAQRbrN0fsc1w0OtP2d03JC29iSdJi8XlOp0EzujW R8YZ4aBXOwO9a0hND1li29BScbhOJ4GDmG1knhEOc8myPFw/agXfTKHpJPsSSYCMZi1knvEN+m5J fHuRAgnj4aTf1aOa4Z7y0j+0V/39X9Xmj/6nWpxKbH+56cv9eV0dzfim3xe7519c0Gb5psbfs7ee wc3atNd60E4pAXV+RF80g4YutXx/paH18mye9D3rQ3ZCCSTE6tG7mhEjqxvhAjelbs4km2dyj7sf L6UE1CqcKro244vWOocFbk7dvEGbbWufMepHKQGFuNurxpN10VyWn1S1Q6mhWOcFF+2UElCIq71q b0MrggvcsLp5izYL1S7loF9KCaiVbczUi3fRvXOBG1dtKxY97k7jQTulBBRiDppqTC66ai7w7S5r vUt/LrrctfuDdkoJKMTVXrW7xlvNCfVuZt0cIhaHS02ddvQllABayYZNmMl9wH1T74ZW3QH14eMk fsBOKAG0Ek/kUHNywBWJ4K/d1Lo4By4WrNX7BTuhBNBKvEVDZySCF6/zSn5uXYvbS5+77NC9dRI4 K/HeD/Umh8t5NPnBVbto+nMupl3WBTuhBBC2SGvVUG9ywJxHsx9c7TzM+X/Kq2DHbRuI3v0VPMoF 3EqiREpAUaBNe2gPbYH1rehBa3PXak0psLXZ5kf6vX0zJO1R2ACbS+JJnh/Hw+G8ebEckVfgfMaD uAnn2JQw29EqamfgbTnhNlYj8gqcz3gQk5QZWk6Y1wY3WrHwJV5eEiw/E9tF67MC+owID4f+w+hk W21wo3WUzkDMXWarnsuhE7EA+owIcRMOirbVBjdaR+0MxDVnXNKTtqZOxALoMyLEVHvTJNtqghut o3gGYm5fw2ISN/M1zmc8mKkseU1yrSa4US1FqmMTaEwqdCQWQJ8RIW7DQSYlXGvGC4kKz8LwnLBx /VgDfUaEmJ6jaaPbhJR3IRGhUNQI9Fq5lavYbSugz4ig5DRAENc2ErMb1VKfLBtLw6uuiTNzDfQZ EeI2HGRSxuRHsTAIdeIHZ1hBTJoTK5zPeBDTjDZttJt9wy4JeCFN1GAYNJ2JN8jEK6DPiLCbspyY aDiBJ18HvBAmfslsXk1rEq2A+YwGMUuJiXYTePKhWJD1ffzYhvPlTdfEBWAN9BkRYh0OCnYT5oDE HnihSbyztLzomjqOiRXOZzzYB1lJbHSbwJMbBV5IkmUTyC1MbRF5Bc5nPIjpBhG3KV8yo7BMQpHo 3QfHZso20QqYz2gQk9VEHKwmvCItucALPQpjh2dmG6uwgvmMBjE9EBjO4DT7OqiMkXJkWD8bE5qi S8QC6DMiq/lJI25TviwzVspRmGbB/7RGJ2IB9BkR4iYcZFPGrDNWmlZaeEuetqHdIrEA+owIMY1N 00ez2VdBZ6w0rcZwxrwCtTrWeAX0GZGt+b4RR0Gqgs500rXSO6NVge8uTcsV0GdEtuYPiG3KmHWm k7aV3hmIecekRTsSC6DPiBCz2yyTIJVBaMhJ3Il5AGveKJo+Eq+APiOyFe82tkyKVAah6aVtNTVn zG3cdIlX4HzGY6tgN8skSFjrKsYL2xoGu9Y1N3OfiAXQZ0SI2XJWUUcA7EIiwrcallDNAt200buu gD4jsmXwrhW/RyZuQyJ378reFMQspE1TJWIB9BmRLXliIDYpYxIa/FCaLzdiMNYsd80tXwHzGQ1i mm+Ig4zgiklm+pKnYaK1lG9tTHwjTLwC+ozI9DyQbR2FBHgSmj54uhsx8fIUKutEK2A+ozE96wfi NuVLMoPOp78SbagDjwnd2UQsgD4jQkwnIA46gmdPGfQV53MjptLWPOV13NbWQJ8RmY4rgukRhAR4 qhjwvchYU8ZVzw0XJ+Ya6DMixHSHiNuUMd1xT85BEFPGFQ9kHV/dGugzItNxzyEOYgIxoJ7sa+7Q RMwSWvF+qXVYJtZAnxGZjl8JNCWICfC01gHf3ecPDTMQ85jQdeIVOJ/xQPboWSNuU8K0h/aah0Di JVat+eH1iVbAfEZjLI8hxJ2NtLQ391pKUtNRohXPoLq3iVgAfUZkLM9NrC5BSbB02JCIkCQuQMXL Jb1o5l3hfMaDJYgGPeLg4YAnrewbqUg0IjG3Oya2iVfgfMZjDOsIYltFXrJSfSMFqTHUaSUPibrt ErEA+owIMeuIiR4Oa2gXErH36RP2iDI8vSZ2xAroMyKsxaT+iIOHA55+St9KQaLRW3In0bBIvALn Mx7TBh0x0cIBT+sM8EKQGmqykmWDRkXkFTif8SBmGbHRwsGXdCEPoUdNjTsj0jImuwL5jMQ0QUJs tG/AtyGJuxYRoXp49+uGm5P3XWrVCttT1dFfpFUXt3nY/LDffLPfN6pS+yd8T9PahW7ZpQ8lvU/S mkpjFJb42t6z5QjONv1JiaM6ZYn/P2x29BHTY/+6KX67HN3FHdXRnccP7vJRbfd/UYJxLJqoEPSV hmcE/7X/8XPf/N+jyT/ivN392yXlUXIG/775xH8/f0LJb1QeQL+2uv1a+Bic9UexbDFnitN2h52n cGp68Y/uMvI/Ts9gOjnvtn/uf3lTQl9Ah0u06RJryGyPFXKXPpSqMthomrBFVj36iG6Ru4R/3K1Y 35Zl232HKoDP3PhIC60FX/wg+PA+QUjp+lAScysJhieVf3x49/CzctPweHZXNX96qfOTOszeD9Px qvBHbA/83pqoYp2bO2kb6zw8P4PGb/GUC7fdQfWK63V4xhFPl9mrcRqXcdjCSRXLfFHLrJbh8uwW NeNUFW+AT6E0s+ukH0THXB3gw1kdx+syTodF7d/9jnynyR2WcQsXWMw4u5iub7mEmtyLhWivLuFL S39jyUp/r1IZu3F/crFEMwquXq4o2eNHNaRiLKdYoi2+Zgt1OM3z1fE/XdFzBdVN1opvRNxDMfr3 Z7TgtNwu1v3jDi/LOE9qvBL/Qjc8IOI2ELf7ad35U7zdK653nJ53w8UNdJ/bHUS98Or1NB5OqV8U /jf1EQ7mPPdfZaXoOFH8kEFxCkf3YTy43dVdqBPm6Yx6PC34OJzP6v3FHdwRR3OmxHZviSrkRqej pLrglj0NH5x6dG66Z/KWVmga/bWtdfdJK7y9C24Eb+iC79X1PPKPeh2n4/yqvDuchmm8htagi+LO QI3+du49bs2tq9nei2DEBaH/bZFu6XVcTuOkHucXlIW//dN+898AU9anfwplbmRzdHJlYW0NZW5k b2JqDTI1IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA0OSAwIFIgDS9SZXNvdXJjZXMg MjYgMCBSIA0vQ29udGVudHMgMjcgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEy IDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMjYgMCBvYmoNPDwg DS9Qcm9jU2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQ0IDgzIDAgUiAvVFQ2IDg3IDAg UiAvVFQ3IDQzIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDk1IDAgUiA+PiANL0NvbG9yU3Bh Y2UgPDwgL0NzOCA2OCAwIFIgPj4gDT4+IA1lbmRvYmoNMjcgMCBvYmoNPDwgL0xlbmd0aCAxNjAx MyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZRXzc4kqRG891P02ZJZfpK/sw8+ Wz74AVZey9Kykj3vLzkyga7KogUejQTqb8iIKEiSyF/+8qO8f/3xtqZYh7H69HbvH7/+8frlr393 73/9eLn3v9+vQIZCeWdKpmKOxqX3n7OrptL7v/98/fYK1cSabwt80QuITMl5g8AKolIQrbFhFxLx p5oOmFVhJmtCTBudKZoU4h7TBcYMAzJbk+suIkdTIx0g0x2yOOMcbVQiwquAZEI9UHiSCBoh1ZkY wyGk6JBksgt7WcGrEGedqdXvaUL/+PKJycbFQwzpEOdNcO4QIl9D7hOTDWW3/5zodYz3JscDT5SU o/iJyaYeQlLQISEYe7gsKUtI/YQU433df02OOoaCibbueYpIi+ETU0xKu1LAMaIt5hkTg+TSNqbG R0wxtewqClZbOZ30OZ1Exqf9vlWbHjHVkD/EOMm29DmeTCaWXRXhktO1ffY6V5zZvlhxmVIxJRrr 98WIrxrG/Dmfag3iDjHlERNNoH294eTkmHk+3lqT/L7kcKbpmGhyOcREybd5sb1jmH2Z4qRRIck4 uy9TfP4cMsuU986Esi85fP46JplIpxjZgfLRFpzJdl9z+Fx0DNaWfQHhPVYx5I2jQ4wkAWrzDMnm sM+96F4R0Rva3QIfs8lhX19YswJNHrXg/4tJn5hibNzXF2ZQMTkY78pOfS6G6gk1adQSTIq7auJL wWu/XVEDznO3r1y7Bl+txsddVQkW1s0d6sHjK4KFm8txoyA42Lmtpwke7s0d7vrjTILHqrxFDfBv 4XC3H9kRAiydO6W2zsJAMHX5cLcf1yEQfF7YxYTojK2H2/8QEuHs8s9d/pBg7cIpJjxikkl79/Cs ZZBl6uHhfJTMkOHs/OFyP0ozyjLEHGL0CxAKnB3t7vbyzoQKY3ewKM/nLNQMq3OKcSqGLJwdHXbt 8TyThbWzByv0sAHkgqHyc3aDHKwdnWK0rSEfpG/7GftEvsivvbXTNo3QgXo68DzsIKEpJXvoih62 k9CnxnRovR72lgjW7uBRnjaa0Ltae4h52HVC8+rSoS96tAWEfjZ4v/+eR/dB6GiTPfA8mhxCk5vT qTHSzRQVzqefa9qQ08bub/ajNSQ0umGf1M8GlNDoRnfoix6NbkSjm/Kh//LlEYO1B99SPOkYdLrO nWIk5BORDbzMPsIlFYI2l8K+4hQXdAi83tbXxABnlw8nYSWt/Hy7I7pauz29iB4W/cteqo0aFT0s 5V0xiehYU9iXm2KdRkXHmuuu3MQEZxd3xSWiG/VhVxb+88J7yDTYmYrnC1NGJcJU8MhgyT/ef7zs G/9wgr27gy7LTxuSGuTtBY9aAeBNxOX7/YNnEn44+I3KTydOE4t/f/32p9ffmBSW4kZapfzBJNkb KfweZ1EfCeo7uwJsmm7AwkwKN2oILy2ySxc3PN/FTc445nZI6jt34NPoIwry4FaATdMN2P7VqPpY iD+5O3OuyK2L2RvYMZ4IaXwxE5fcPhKUdWYF2O5kAxSXUpiDCbwQRUoxF9zyixmPOW4BphjvzFK4 +4hCP5gVYNN0AxbpL9zcJGIpX/Y7d5Ine3ITbkLiKSvuzLWsj5TnVyvApukGbB15ZiUtuF9S3Dix G3eU2szGWnFXrol9JJT7zq0Am6YbsM72RMODzXlRkAiKHIeGVmmSZyk9mNyd3EmJ6iPBpwu5BmwP voHr5MBReyQz8I4pbhzbjRuOj7mLZNzF7bmQ9XEct4Zrimtgwgp25twzI0uJuZj5QbmYa0+1qlPN iY/tY8QF79wKsD34Bi46Rd83PRkP31iQDYqeDeKHHqfI2YZJZRufwByjH3VNA7Yn4QB2EflZRQAs F69HStwFoOTmmwAnKYdJpZzLIkBG7q26AAXYFsaB7JI4A1YAs4yAyvbhrqDIYzMVeKBAQXjkXZUT kDHO0q4B28I4kOEAylDgTeIAj3S5K0DDdFNAspGYPP74UcA9yBxjnHugANvCOJDxgtmhwKFWIQCf 5+4KkpBNBVEyCRMpBZ5LXR9jHEVeA7aFcSB7BA4FVvKmEjjuCnCaNwVJ7hGmpBRIb9bHWes1XlsI B7B3MxHxfHLe1AiKuwAc5k1AlqPEVJSAyFWvjzGPN1YDtoVxIMPVjUQMuedNAsddgZfbNxUUSedY BexSIFkgSVBGIdBwbeEbuNwrDn6sLAjIYLjzOyk+g5/bHfAn+0jDwuWvj/yadwUKsC2MA9kTrnVX gL3igPpIQyuFdypwkoYSfFMQxPn1MdlRiDVgWxgHsk+41l0B8gVtmgWSVOuhQLb7UuAlDeVz7grE Y/UxuVEMNGBbGAcyjHqIUpEDW2cEwBUpBVlq31QA0wF/jYnT8VIg/U8fkx/FQAO2hXEge7xZQwHK LAfgK+tdQZLyPxXAerACknS8FEhr2Eeuv12BAmwL40D2qJNDAV4aDsDnKQVR3r+pIIpjxsV2SoE4 rz4mmnugANvCOJGttH5QgP+JHIDPUwpI3v6pIIuLwxQc3RRktiF9TDSqgQZsC+NExnM5FJSeOPg8 paC/QFNB6Q4WO6kUVL6HfUyjHGi8thBO4DAT0eeeN/g6JcCpRKxy3eWJuwkQSyJDSnMDFFxb+CYu SZfC/HywCCiPNLSzt2L+bHt7Z8VXXvxe2h0ZU547oADbwjiRIzK3K+DkRkDVaSg27FLgJA3F59wV 9C3oezDtsAZsC+NETqjUXQFf8LeDZVVpKBb0UhB6MxF0GnLbM8dUR0HUgG1hvJDtUMC1jiO82JxL QpZWb0qAK8nEE6fjJSFLFyRjtqMaPBDbwjmhKyplFg1c8TkCHjbfNSSc3Gdhpn4QsPv5rqFINyRj dqMePBDbwgloi99wETQ08LvHEciWLxrGwhx6OqLfuWvgwjvHPJ3yA7EtnMRfgt8O2SEaHL/9HIF8 +aahL8xeriQmUhq6O5AxDwkPwLZQEn8IfnukR5eQZRMdzGz5JqEvxJ3gPMDklQQK7znmMOrCA7Et nOSll4U0OzSw/eMIJMw3DX1htnIdMFmlIdb3HPO4FA/AtlCSk+xAawG5IoE9MEfUrxk5FqYqNSEV ScxLQu4+kUTC3AaN2BZOsj07Ej6ra+BOABHefs/IvjBlqYyYstIguyCbMB+IB15bGHFBC8wtrkoc CgL2hyPc93zsC1OStwFTvCtgjzbH/ElIjdgWzsDfgd8FmF0Dt4Yc4b8nZF+YYndsUR7FS4N37znm MrNBI7aFM/CX4HfFznYN7HX8m4sr0BcJfV2iblvRxikJQeqzjLnObVCAbWEM/B3oBRxyI7ECvCgk AciWVcFYl0K37mhllYIodlHGYkeB1IBtYQz8Gfjtxe+ygoLs4QBkyxcFfR3sd+3WPSsFqbznWKZx 1oBtYQy+90NwwkNBRu5wAHLli4K+LllxzMmKf74UFHrPsfghQOG1hTA4SQv4qFi7ABjSyAGFK8si oK+D++amAVO8u9ZUxS3KWPyoCRqwLYzBSlrASuahAKZcAuq3RBzr0EFy44Qp3BXwtsxx+hWN1xZC /F2yImFnugBsDgdA6bc87Otg/7l3xKSMe3cJQejnBii4tvBhYyQneOr8yA4J8F+zsK9DuGX+qFuX TOwW+1jiKAYasC2M7Gpd4eMMQwFL5YAAylVBX4cTLKyAdPuWkxgEGUua90ABtoXR82fUntH8u9ae NLgvaRXwP7arXgeP3Ab2eYp9Ap/+f2p3AXKNy8N1qYKoCK64188MSe2upIUBrj9bmhlJI4q0YTBx iWV6+RFQJQ1JbPY6rnjj4AtcRNc7Lfw0a8T4DCP3U4COwzVOsczb/Ajo8ZqxNd2BDXAcjIGr6JrX RAEvLCfgAn8p0HFIZSGWmdFuBc3JFdTYNRtugONgDFxGn7kdCpizOAE57EuBjkP176BAsvpbQdAs xNidJqMNcByMgcvofOCC5J7Ov3AC8viXAh2HJ40dED4tvhUkUSBxbsGKNw5C39UWwFcBqJ04IfF9 /RCg4/CuVwqoYsdHgFQmGrvXZLABjoPRN7UFsFQB329O8N9G1HH4lamgiB8fBTVfM9qrvOGNg9Bz FZ3lXlMBLGI4IX77UMehwIsUkMWOj4Durxm71cob4DgYPZG71ryiQJsvj0IWL/qhwMbxf6kgiR1v Bd2V647JksEKOA5Gz2V0rfz5u3X1DVf5pUDHod4WH4bVhz0wE1rMegob4DgYUcY6635EQDPb4Jy/ BOg4nGnNd9/zCIjtumPRLdgAx06ItrZZByj8xVwDp7t28uu4xJbkbv0e/hyvO1qNvAGOnRDla8HP ZJZoWQp/z6v+xa/jeK2R9fEJC3+RG6hRPbjhjZ0PpSvyRmXlkkifpOr3yHX+i17HMa2RHq9qftNL PWaxaybcAMdOiBXgF95IpY/SiHpk+fhFr+OY10mPsuKhRwdwae7bQMbOIcePakp/BWRkDg6wx83o Lv6R9xyJk01mZd5JAAN6QyvI1odN6H+pjVW+DZAtZBdpRySA/yMNjyE2qUljE/tKs/TsHzlZMwgt KhaHuvJNy6YUKoNX2YL3Q4oYFliowSqzc3p2rXnZ+0naZbusS7wPzXkmDI3JEtcGOFY6g4WDhBsG 49BmBzq5nRyCcScvTklefPNwR6ZLjSl1414Ax0pnsLpqWAsD8U/+zVy7nOhkDpqued1fqcol1kwa E5Qp8wI43mQGikpYmOEwDvRmosnc9OSNOcoTlaI8WA9zYXbQmCxHbYBjpTNYtAHCDcNyKAuyNzdd +HAneaBTkuf64a7MjBppTuVeAMdKZ7DdfObEFo3Wf3PjxF7cWV6FlOWNeLg704LGNBuWFXCsdAaL hCFGQ91FXzTW4m/yJJ3ZJK9ao1V5Im9yL/WhxmSZaQMcG5/hejlwZE9xBjuRN3eUnmxyNykJUhPH PdyBRYlGO+4Vbixchon0osxVncEe7NUe4PnyL+auVuur1VD6XDOyb1DuBXBsfIaLNzLoppcfAc9J Yw/6pnfSixk9TlG6E7e6jScwYw6W11bAsRMaMIqEVLsIyLgrGA9LvAWULq3YFODFcvgslvNVBEjM 0ZuABXAcjIaMQqmYAiR8TEAp1RYFDQCPgiBNAe7x6jupBzXmmdpXwHEwGjKKxWYKwo/CCQF2eSuo 2MNHQZKNxCfEV48aHFOcxpznHiyA42A0ZFTszhR45CpMwPL8W0ERsqkgi5PwSYuCwFSnMWdL8ivg OBgNOWCiKXDiG3QVcVGA03wpKHKP8CmLAikINc5cv+KNg9CA8fqbEfF80jdorPIiAIf5ElDlKPFp i4DMrKcxV3tjV8BxMBoyOkczIsoe8U0Bx1tBkNs3FTSxc+4C9igQF4gJmiWCFW4cfIYb4rQh6q3a MKGC4c3vJfkYf3Fiw+I2GzamP418zVXBAjgORkMOCddaFWCvOKFvNnSSeKcCLzaUyS8F0fESaizO EvEKOA5GQw4F11oVwC8dTwSQJFubAtnuR0EQG8py3gqkxtJYvCWDFXAcjIYckLWzZOSIK8MJqIoW BVVy31SAoqNnfmjHR0Fka6axBEsGK+A4GA054M0yBUiznIBV9reCIul/KkDpQQVJ7PgoSLyEGpl/ VcECOA5GQw7Ik6YALw0nYHmLgizv31SQpWLGxfaLAqm8NJY092ABHAfjRGZzJwrwP5kTkjQ6j4Ik b/9UUKWKwyeirngUVJYhGkuybLACjoNxIuO5NAVNjYPlLQr0BZoKmlaw2MlFQec91FgsHax44yCc wHEaMVT1DVa3CPCLEbtcd3niXgKkJJFQytyABW4cfBM3SZdCfh4sJrTNhm72VuSvThoXeeTf/EHa HYmlzh1YAMfBOJEznKsKaG5M6KsNpQx7FHixodQ5bwW6BboHsxxeAcfBOJELMrUq4AVHL+pWG0oJ +iiI2kzE1YZse2Ys3RLiCjgOxgfZmQLmOs4IUuY8Eqq0elMCqpKa+KEdHwlVuiCJ1Vk22BDHwTmh OzJlFQ3M+JyBGra+NRSc3D2wJj0IlPv1raFJNySxessHG+I4OAHt8BtVRDINfPc4A2750GADa1Q7 ot95a2DinbHOSnlDHAdn4krw28MdosHz7ecM+OVLgw6sQa4kPmnRoNWBxGoSNsBxUCYuBL8D7KES qmyiRzHbviToQNwJ+gCfsEhI8ZqxRssLG+I4OFOQXhbSnGlg+ccZMMyXBh1YnVwHfNyiIfdrxmqX YgMcB2Xy4g60FpArElgDc0b/dKQNLF1yQmlizEdC1ToxiYS5DSviODiTU3cULEs1sBPAjOC+HakD S5XMiE9dNMguyCbMB2LDGwcjLmhDcYurkk1BxP5whv/2ow4sRd4GfPJbAWu0GettyBVxHJyR68Dv BkzVwNaQM8K3IXVgyVqxZXkUHw3BXzPWNt2wIo6DM3Il+N2xs6qBtU64mFyBfkjQcSVp2Yo2bpEQ JT9LrH1uwwI4DsbIdaAX8PBGoQK8KEkmwC2nAhtXopbuaGUXBVnKRYnNWYJcAcfBGLkM/A5S71JB g3s4AW75UKDjUH53Ld3roqC0a8Y2C+cVcByMMWg/hErYFFR4hxPglQ8FOq44qZiLk/r5UdDSNWML JmDBGwdh9GIL1FG5qwAUpJkTGjPLIUDHofpm04BPfletpUu1KLEFywkr4DgYoxNboJSspgBFuUzo X0a0cegg2TjhE98KuC0zznplxRsHIf5dXFGwMyoAm8MJUPrlQx2H8p+9Iz5L4a5VQhT6uQEL3Dj4 sDHiCX6UH+6QCeHThToO0x3589q61MRqUWPLlgxWwHEwsqr1jccZTQGlckIE5alAx+EEGxWktX2r RQoEia3Me7AAjoMxcBldHc3fvatpcF/KKcCGwcQllunlR0CVNCSx2eu44o2DL3ARXe+08NOsEeMz jNxPAToO1zjFMm/zI6DHa8bWdAc2wHEwBq6ia14TBbywnIAL/KVAxyGVhVhmRrsVNCdXUGPXbLgB joMxcBl95nYoYM7iBOSwLwU6DtW/gwLJ6m8FQbMQY3eajDbAcTAGLqPzgQuSezr/wgnI418KdBye NHZA+LT4VpBEgcS5BSveOAh9V1sAXwWgduKExPf1Q4COw7teKaCKHR8BUplo7F6TwQY4Dkbf1BbA UgV8vznBfxtRx+FXpoIifnwU1HzNaK/yhjcOQs9VdJZ7TQWwiOGE+O1DHYcCL1JAFjs+Arq/ZuxW K2+A42D0RO5a84oCbb48Clm86IcCG8f/pYIkdrwVdFeuOyZLBivgOBg9l9G18ufv1tU3XOWXAh2H elt8GFYf9sBMaDHrKWyA42BEGeus+xEBzWyDc/4SoONwpjXffc8jILbrjkW3YAMcOyHa2mYdoPAX cw2c7trJr+MSW5K79Xv4c7zuaDXyBjh2QpSvBT+TWaJlKfw9r/oXv47jtUbWxycs/EVuoEb14IY3 dj6UrsgblZVLIn2Sqt8j1/kveh3HtEZ6vKr5TS/1mMWumXADHDshVoBfeCOVPkoj6pHl4xe9jmNe Jz3KioceHcCluW8DGTuHHD+qKf0VkJE5OMAeJ2NsUkTWLJ2hfYzxt59/tevnr8vxDx/7Xz9/h4x/ QsZ/7A7+fXl3/ev64093/fsf7JLYr1amsARd/+e8bHJ0x20oOq9V1BKsf3kZQQZZQc8eA3TvfxDe S8qWzFnQgL9mPfJYlijyUhc6darEFIV59g9eioHBHXgaeJ+f9p+ff3NpE92srqXpz2ULxZH8n0s7 XyTfpdqb3x3e/rmP7Us0vt3DzjGrrfcj0V5Dqh4C/s5dthx5s8Y15OkpAaUH1s0ehta+Mi1gbuhV At6L+HGUQFL7gq1p85A1iZArdSNT8t7Vblp60JunBNRMTDctmMlH0KkbqwE3s2+hu3WPulXwj6cE 1NTlmz0cPfRaMGBsaKtJqfoLDH14SkCpjSuj7as86KyJqn8vG7pa55tb8/94SkANzKvVepihO+/A TEzEheZWQFhaYzP04SkBNaCOaPcHfVnA2NB5qfZXN348JaAGZtxqvd/QTd+tAVB3LzpVH4qgSG5H H54SUAPSgPZ0NFQzA+4NfXHVxdY+HH14SkCNzn+gclrohIBCP0fPmwoqcUNaftCbpwSU2vicOtgI gEZt0oDKr1voaVPC2EbV01ECSUvtMHs4GZ/HgLmRKfjGTK7ADX14SkB5Fa9oAQudNAVmYyItNFPD h7tUHvTmKQGlbbXoPuqIcA1DJ62IGtC9+Ri6tWdyxa4Y+vCUgFLBMMzujrYCqgFjQ3MIYIkac1WQ w1ECSW0fb/JDzkMDBoeYRS5cbuLSn5J6eEpAucpqlw9eKkL1pRpgomyhOTt1JuBoXkFOTwkoFZA4 /ZZ8rkXAZQF9Qyesuk/f70XeHCWQXJo2nLWRUdY1wKTsIl9Ybu++30Y+HCWQVHZ3s4eTkZMMmC/Z Lkqv3I7k9eP0lIDqOl1pWjVcq4VWUfOrE0C7X/TkojOa17jKg948JaDULrBRCwyN668BWoE39OCq WT/6KqinpwSUTkLD7OFo6EIGjA3dcYiQkfjodYqHpwRUhzhVW+eOB521cOpYWDZ0Q2q0Mf0sF3rz lIDSCbIWtTsHMKDRIDRAsz+9aN7vxjLSV0E9PSWguk4LZvcHfVnA2NCccFoluowHvXlKQKmdYQ8O rkCjGWuAFpoNnXGMjc+evYKcnhJQvWmq0MZYCDQkhAY0XdKLTki7llBR+7Xy+vCUgOpVi7naUEEL nSygb+gL662T23LPB715SkB1fI7a0G6GTlzdja970P3GqitLapvrGA9PCaheWCnVHo7GtWbA3NBc b23c8XWKh6MEUtfmpoWyY0mLnBAw9Ks2MmU/t6O1h7w5SiB1pHv/7ZD2sG+IefVH8r9gI1M8If9A Ph0lgHpijdKfbuDJfnTjwm5gcC9emWLV4/STwOmJZVVflxcXZfdGjckPlzKsTO5x6gu8OUoA+ZZw gwge7BTYQTSOhwzVUQaetsUfPwmghuqttlZW3XmCLw/oG5jluVAkVNdLH08JqHZz8ZqIydCdLRmZ mnc018xWXr1rfTwloNrkhqs9DN14IAjA+Sz0xVVnz2iST0cJJL/lfudJTgzIzKiHDGhhv60+Fn08 JaAaJInaKqcMXZn2KGXpRTesObMprmQ+/SSAWudNVHsYuPCmImBs4Ik1s4pWn4k+jhJI3gS8JZCc GVBZbBYZzNyqXxcn744SSA16VW3VUkbOLIlodGkjs1VlVo3iHfzjKQHVKpWh2n2hLw8YG7oh7zIb QPGR6OMpAdUq24tqhmzoxPYDUVE2NLtgvvAs3Q/x9JSAcm3jSkfRF5smpBB66EIXrDqxShfv4B9P CaiG6aP/Dqu8RCcP6Bs6Y72J1RTX0dG7pwRUyxQgqioTbRWDECiQnXlDs8FC0eE6Wvn4eEpAtUTZ pPYw9KSsQsD9FlPck1SQgMUnoo+fBJDrdVfvBCcG3FSGDrbOnRJz2yeij6cEVEOajl+dOS5DDwpY DCVpR4PMEpJ9Ivo4SiDpaDTM7kbuVN0IGBu5I52v2e2mk3w6SiD5OOfDHcl51F9Mf1U7yUNG4l3s idkHoo+nBFTF8EH7NrTqHgboPdvQlYu2+1gtpz+eElB16Gep3SjCib48oO9orro0VpFF3h0lkNTG pDi0uhpZv6YhQEvaRsZluTjK5eRneDpKIFWMHrTvRLKWXQ+4byf/Da9aHq8yaOlP0kP756+f//z+ 98fljhcZvvtESXhXxWyitubZendigBbo9e5//VyAErhE76RiKp3R7yrUO/3iP+q4nHhK+jIcmgA+ 9X8KVNiwlyFR6sVwVYj5YsaMwb/7lqIh63NWPl0xn54SUDPzndVmVEUnqBAGzA2NG3d3pMHtGuPj KIE0MxOlJl5EIydN0oFMfMlU4TdT+O7pQW+eElAzMVNqZn0BerIfjdszxdC8cnfhfrjI+HhKQM2L iVAxwxgaKoQBY0M3rjpVf4GhD08JqHkxZSomr4XO2o8mStWL5pWbtuOuMz6eElD6zqrLqJgXDY0R UQO0GaQXza2Yk2t3ofHxlIDSQ5hm9wd9WcDY0BwfJuXGvNOD3jwloAZal9pNfwwNJaIB6GQvmrs8 WfSnC42PpwTUQMOlPR2tAjUx4N7Q7CGz2Np9XDs9JaC8JHiBMHRCQKGfo+eNyzITN8SFxsdTAmp0 fo7WveRoKBENqPy6heYB4pJhW5x8OEogjcYzUHs4GZ/HgLmRBxY9JjMwL/ThKQE1GjNHG0x50ElT YDYm0kIzNUbPvisLvXlKQHkj9LYIdGK11caJ9F9ojlWjdt8VQx+eElCjcGBRuzvaCujsvLQLXZF7 LFFjrgpyOEogjcIhq11sXkbGFDYHh7JF5sCGKUGfT0k9PCWgXCe5alL0dXN2nJOj5EJnLLozAYdL jY+nBJSqO5x+w7Yt9GUBfUOzYVPiYr8XeXOUQBoYQtXGWRsZZV0DMJO+ZM6Cvft+G/lwlEBSFd3N Hk4e1GjzpmRzsl2UXrkdyevH6SkB5drfJwFDQ1jeF3XmQnPM7BnNa1zlQW+eElBqF9ioBYbuVMM6 4aQNPbhq1o++CurpKQGlc9Ywezi6UcNrwNjQFKEQa/jodYqHpwSU2hgrdFYsDxpzh458ZUM3pEYb 089yoTdPCSifaX3CBbpyWroLh6eF5v22gbavgnp6SkB1Fe9m9wd9WcDY0IWrpiRFZ1rozVMCSm0M nm1wDAW6cDK9Kyephc44xsZnz15BTk8JqKUdoScMDQmhASq9y4tOSDsbDfu18vrwlIDqqv9NlrYH nSygb+gL67UJrt3zQW+eElAdn6M2tJuhE1d34+sedL+x6sqS2uY6xsNTAqoXVkq1h6NxrRkwNzTX Wxt3fJ3i4SiB1DPnoY4lLXJCgOr8tpEBtixp7SFvjhJIHenef3viAPDn54aYV38k/ws2MsUT8g/k 01ECqCfWKP3pBp7sRzcu7AYG9+KVKVY9Tj8JnJ5YVvV1eXFRdm/UmPxwKcPK5B6nvsCbowSQbwk3 iODBToEdRON4yFAdZeBpW/zxkwBqqN5qa2XlTKsBlwf0DczyXCgSquulj6cEVLu5eE3EZOjOloxM zTuaa2Yrr961Pp4SUG1yw9Uehm48EATgfBb64qqzZzTJp6MEkt9yv/MkJwZkZtRDBrSw31Yfiz6e ElANkkRtlVOGrkx7lLL0ohvWnNkUVzKffhJArfMmqj0MXHhTETA28MSaWUWrz0QfRwkkbwLeEkjO DKgsNosMZm7Vr4uTd0cJpAa9qrZqKSNnlkQ0urSR2aoyq0bxDv7xlIBqlcpQ7b7QlweMDd2Qd5kN oPhI9PGU/5Fd7rqW5DYUzfsr6g+s9yOezIANDDqYwHDmSQzQwIwD/765N6U60iEaYF/eS65SqSRy 06FqYXtRzZAMHdl+ICrygWYXTAE2t/UR70hxqKVtltJRdGDThBRCD93ojFVHVum8OvhXpDhUxfTR nm6Vl+i4EtqBTlhvZDXFdVzoM1IcqiYKEFWVkb6KQQgUyM50oNlgoehwHa18fEWKQ9VI2aR+N/Sg rELC/BRT3JOYcQDzmoi+4sSBll5f6p3gyIRJZbjA1rlj5NleE9FXpDhUxTHtj84cwdCdAhZDSTzR ILOEpDURfQWKI+lo1M1vRm5U3UjoB7nhOIfR7KaTfAeKI61xbg13JKdeHkx/RTvJS8bBC+yJaQ1E X5HiUAXDB/1paNU9TNB7dqALF233sdiZ/ooUhypdX0v9ShFOdFgJ7URz1bmyimzyGSiOpD4mxa7V 1cj6NhUJWtIOMi5L4CiX4vqGd6A4UsHoQX9GkrXsroQ5F/kPRJX8RuVOT/+L+tH+/P3Hb89/fiy5 s4oMn32jxD2rYDZRX8/ZfnZkghbo/exff/zll/+O55efz/2An7/8/Ud8/vrE59/6m6n97n96B56/ Pf/4Z3j+9erjQXGVGx/0WfDPH4Gk1c1igrSJWBm+sGDbIl4Ukq3byrDEghsNP6bAJerSM7eR+29b bxaq0tBXpDjUSNwN9aeh4+R2acI80DwuZnE2ib4jxaEGqpf6UOEbjY7VxzpWhuYhN7slyVekONSI /ENJrEZADx7dPte5MjSvplmUAUNfkeJQI/B11B95o4Ml2LExNOuJWVzCjT4ixaFG4EcomecX6M4q McI6gURbFTSLimvoK1IcSp+Jo1MwXRoaskUTIk/SRrN0my2zLfQVKQ7VJ6uD+u1FR0sY+YNmwzGL 3rbRR6Q4VEcFU79qlTB0pQjRz10OdP20yrrJV6A4UkcFoz8XWaskErR5HGR2drN7nvmKFIdSH2Vi VRNDp8yEdqCtV9qsm+eLPiLFoTp+UN+KJNCI1AQrowttKspsLfvsXZHiUB3Lpz9edLCEeaCp/czW Ol/0ESkO1SufudoR0PgqmlC5hI2mYjVb2z57V6Q4VC9UmauJAg3hogmNG7fR1Nlma9835ooUh+qF 9Wy1fkNHS8Dn3ui+B4Q62gs+4sSBeqaeX2oFYLsRo3OY22COM2br3Gu+IsWhVFRhCFkSS9FhckoZ g3PnQtsMZrYFa8RfkeJQPXFyqqiGGx0toR3otKfHFvsLPuLEgXq0VmhSFuDBYXCJ3Q3msGu2pb3m K1IcqqPT0J8L3TnCDjSmD7pyO2hRdgx9RYpDqR9VVK2xwdCpp2cGitKN7vHZtpX6oo9IcaiGrq6+ DTtAo+1rgo1DGz0o4GlbXUfvjhSH0qFsmj9edLCE+UFjIN22tfKij0hxKPUhYNdgCbT+hISkr/VB p7SFEldq6CtSHKp1yu6qK1joQl0+MyetV12OZ9s29qqvSHEo9YP5vW50tISRP+jan23bjC/6iBSH ao0DTlVhtdCZE9AsusgPuudn276E6VekOJSqR4xl6s+FTpzbpur0A83eYhaVztBXpDiU+hglVZjW F41ZUxPaB23Dqdme8os+IsWhmspm+FOPv6GjRmpC09f6oK2Y0vZNvgLFkZoqePPHSw6WMA9ysYbI rS7pRR+R4lDqJ62wDd/I0BDGmqBDwYFuRNP2fc2vQHGkpsOUnsEGaQ9/agFG/NCX+oD7eLbtzdZ8 B4oDqR/M7xscLX4c4MlmSLvk6R0njqM/4D7qr7NxB++rpuH2Lm6L+dm2b/AVKA7UcD7pW6vRLeta GbGD4SCn+Wzbh0mPr0hxqAY1q35mMyM6rgR0u40u9dkWPXqhz0hxKD0iqNd6FrOhG+s5TlQ+0K08 246wVn1HikPplcGWqz8NXdmFkDAPtFU6gKPV/684cSD1cUbW9EkwzhCubj3Ak9tBO9LY6DNSHEor CPq5liqcckUX9nvUMlyDhcbWbYu+QfQdKQ6lJXWaPzY6rIR5oPOzzChjg884cSD1UUC0chcDZxYY lPZ6gEt7th3VbvdXpDiUNq8Gf7AGKlrlFBMqi+RGt/RsO9o6eHekOJT6wfy+0XEljAM94rPtWDXp K1IcSrs5ipPKhmzoyOKF5l8O9OzPtmOsVd+R4lAqb9AX1Z+GDuybUEPhgx4sHMvOteo7UhxK/ah+ D+ztRKeVgOa/0ZSlZmdIG31GikOp3kPzUmGZ6KsebExQiXKgy3i2ndGkx1ekOFSNvAHqj40OK2Ee aPYUszOljT4jxaHUh95TpV0MPagHIcXrgR5E0868Vn1HikNV08g6eECzKlp3DgkxUNRu9OSG0M5i 0uMrUhxK/WB+3+i4EsYHjf19bc0bfUaKQ+nslXUe0UksG1qVhLY+jGpFxflG5/K8tq1V35HiUDqK dvOnoVVQMUHv2vygy3xeu2TNV6Q4lPoRvsbFYui4EvqBbvV57djkM1AcqWiFha8vZ+RCKaCbqG/1 IQ/uhNm5zt4dKQ5VGiuD+tPQqo+7JcyN/gNRJb9RGRqp47+oqX/+/uO3RwXUg3985o0Q9wz1kdhV 3Of1zMSEQuHLZ/76Izz4t6UqJFTHuFQUOEAuz2pyVMHoauvv3B+FQcoGajdsD2j6EgZMkd/wA7SE jPmr24qq+iXw8KiMTBBfEUvG8VobHiJ6jFkMMoa+IsWhRuIuFFwKouPkNvXOd9rojPNhthTTIV+R 4lDq46OWyE9saHz1PngINrqix5jFjLTRR6Q41Ij8Q0k8mEAP3TpNmDy6G924IbSl7VVfkeJQI/B1 1B95o4Ml2HEx9CCatvT+oo9IcSj18RFK5ukFurMsjMBvstB6vJ5tMX4Z+ooUh9Jn4uiUwsoFNAZJ TYg8SRsd0XTNltkW+ooUh1I/mN9edLSEkT/oTDQtNM1GH5HiUPp9C/yqTc7QenOQoP3iQFduCG3d 5CtQHEkP5TB/LrKWRSRkfasPmfrXbM1rxr0jxaHUj1X9xqpj6JSZ0A70IHoYer7oI1IcquMH9bsW EkMjUhO0XdQXnQJ6rtla9tm7IsWhOpZPf7zoYAnzQEe0GLO1zhd9RIpD9cpnlsGbADS+iiZULmGj M7bCbG377F2R4lAdJ6U+BefG0JE1eTRu3EZX9BizGHYNfUWKQ/XCeqZ+nxsdLWH0DxrnmaaO9oKP OHGgniniK26oge1GDFzh/IIHaqnZOvear0hxqJ44/NSw5lMILYyXY3BUWegcADXbQl0964oUh+qo f+pHDk+GjpbQDjTuIE2L/QUfceJAPbIZVlRwAw/OgAMl/gPOqKRmW9prviLFoXrgYFrRdwzdqTTG 5CC70ZXbQYuyY+grUhxKfcjKim650dCdE1/rg+7oWmZbqS/6iBSHaujq6hdKY6DR9jUhUjxv9MCh M9vqOnp3pDiUTmLT/PGigyXMD7oEdi3a1sqLPiLFodSHYtWRryy0/oQEnUsOdGLXosVKDX1FikPp 1ItxqeoKFrpwnpqZunyjWTnMtrFXfUWKQ6kfzMfIZ+hoCRgKN7r2Z9s244s+IsWhWuNMU1VYLXT+ P9lljwNLbgPhfE8xJ1jo/yd2bsBw4BM4Wwbe+wcuFqUe9hAL9Fu+V/qao5bIIoeejcnVoWf93OdM x+69lRJQ8I+zM94HXTiqwWAmh2ZvsadWumtNnVICCnHWeHGaNHSpXDC+aC1C9zlLfdBOKQE14O01 3jj+hs5QYgG8ff+irZjyOS/5JZRAGvDXFq+HnGzBduRmDZFb3cqDdkoJKMQFFXboNzK0GmMs0Jni ix5E8znvNX8JJZBGwY8an6HWXuO9OTTsdYYEA8/1uc85LOe3UAIIcbJ4XnA2/XLgzWbI57Gnb50E Dv5H7yP+uhp38b5imd7ewx25fu5zXvBLKAE09HwytlaDLZuojLqDyZHL/tznXGY9fpQSUEPdLOLK ZkZ0Pgu0211065/71B590F4pAYUjovUaZ7EaerCe64mqDj3a5z5XOlm/lRJQuDK65Yi3oTu7kC7Y Dm2VTsHZ6v+PTgIIsZ6RM38SrGdIr2534M3t4HOVddFeKQGFCqL9HKVKTznQjf1ea5leg4PWrbtP 7RtEv5USUCip2+J10eks2A5dP+ex2rpgr5MAQqwFBJW7GbiywGhp7w7cxuc+V7fb/aOUgELzGhov 1kCgYae4oLNIXvQon/tc4xy8t1ICCnGyeF50PguWQ6/8uc91atKPUgIK3VyLE2xDNXRm8dLm3xx6 z899rnWyfisloGBvtC8i3oZO7JvqhtIXvVg4znOfrN9KCSjEGfFM7O1El7NAm/9F05bac6dy0V4p AQW/p80LxrIwhh8cXACL4tBtfe5zZ7MeP0oJqJ55AxCvi05nwXZo9hR77lIu2isloBCr34PTboZe 9INqxbtDL6L53PVk/VZKQHXzyBg81LMCjZ3TBTnR1F705obwuZtZjx+lBBTiZPG86HwWrC9a9/d5 9nrRXikBhdmrYh7BJFYNDSeB1qejWoM5v+jaPs9znKzfSgkojKLT4m1oGCouwF3bX3Tbn+d5bM2P UgIKcdYYutwMnc+C6dCjf57numQvlEBqqLAa48cZudEKYBPxq77kxZ2w5z5n762UgGqDlQHxNjT8 8bQF+6L/p6pWH1VVjzT1j4ylf//3j/98YKA++h/f+UZIeAdiXThh7ut5Z+GCRuPLd/4LPwhzYzqu 59//+CcdqzqpqVNTA3fpC9rn9DqaYW1u59+5TWCqo03zmmMH+OuPf7us9R+12nzpP6vPz+rrGXwL DdLUOteej7DL9yOs9aCdUgJq6SzJeEzv4rFAT/PzfXlw/Jn8EUogLS2QTf3xncUy79yz90ZmHz7H feTHajulBNQq/Jyv+UCbKxZsl/TI30va+oN2SgmopbWX8Xiy1iI0tTh/0Z1Z5+UK4o9SAgpxtVeZ 41TDqG4DC7rLuuXTdr5l/EcpAbVg4/CqkViCDa1VeS7+/atqnTKexoN2SgkoxPrOkei0vmPN3Kwc F12uMfl24x+lBBTiaq+yiXJvdqm5uW+vKr56Oamf+cMJJYBW4tf1A5NaUeitChg4jWMtnaN6CSWA VuKJHPkMqnuzZeOv8zfjZXY4v1zgSygBhLjai+admPQ646/7N2N1UDob0Ki1OzA5nQTOSrz3o9Cs kKuWcSVWgcudtNrr5eJfQgkgbJHWKsQ9H3DFQIEvkV3CPA9z+sHjpZPAQdzsPfMmrA5maVP9cjsT 7mc3DtfpJHAQJ31P5Wj2TI5L/dCXS182eU1+BscjlADCxdF/QGy2Z6uvh75Q9jhKDgaZE+kpoG+h BBDiZi96BlKdUaBvLuPCjFM9w8EBO6EE0Fzc+6EzzxlHlyUyXMY8voPN5M4EL50EDmqqnhXEZ0ZC 59Y8qm9SOtnqlHs3+oCdUAJoTp7u0a5rH/Tiq/oWZddisE7MUi/YCSWAEOt1RPNO9Qy5yxJxHUoP gvnPRI9wZlwnlAA6fmP04+C32gPqXX+yiWtkBY87wbyEEkCIu71o3IzVgMAwuO7ECzfYQcatEy+d BM4crNGIzVBuOMtMvWtNNn32Nc4XJPgllACCkWQ70S9xwDr9Qe8aE29yZ9Z9XKyTScAgZivBz7j5 6jQCO1u/5Wc25tu5zccAvIUSQIirvegMn5XNHnrXk+hZetWsRzll4qWTwKFVRzzPCAN939S7lqTn FuByjsXhOp0EDmL9gkN98eFWzWP4jqT3Hlyein6xTiYBc4YWxNuKD2zSpt71Iys7rJn97MJLJgGD WC/IQOXoBzstDdeOBvtnG3Yo1gU7oQQQ5je90oj7zZdtZvp2ZNWs0Wv2US/YCSWAEDd70bwZs89M lqwLnnolGvuGHrcDdkIJIMRaNgfMkWWcrc9MFtkLHsyYFqjXs8cvoQTQLPzeiE9DytZnFrvCBXfN uNLX91stX0IJoFn4P4jnzZh9ZnHZBTf9bJUeU432ATuhBBBiTX2m25CSNRqdJL5gFuBKR9HOiPkW SgDNTG8z0+1IyRrN5qe54MKMeYzbulynk8CZ2aa9dBsSbF2mXs/S5bKw11p4mPcFO6EEEGI9z/jD +giEyxIZLmG20MoG3bp9u7dQAmgm3kDEdR9wt0T2N+PO41DZSFvLF+yEEkAzsWIgHjdjbTT4oVpf HjCIhe2uPfk6mQQMYq1viK2N4BNrm9mJ1fBip+Zbxjh3hOCXUAJobBbkWU4jgV4bzbaZ7gErl1Uo lYt1MgmYsdk/EPebr7YZnHz942JtH1gm6poX7IQSQIj1DYitj+DaawY7M58HrFtbWOXrcWtvoQTQ WNwRVA9rJNDrjkG/XcZVM86bB+5UzLdQAgixfkPE/Was33jr5ODAmnFmQa7n1r2FEkBj8cwhtmaC ZqBnchee0AtmC830l7WamXgLJYDG4i1BT7FmAr3aOujXt/5oMQOYZaKWy3U6CRy0Pb3WiPtNWH3o riwCl6vUWnnx9sU6mQTMmCxDiNc8WPXNu/qW1JYmmlmDyp4X7IQSQGOybsK6WCeB6ZiWiGtJ3IBM c6k3mtyXTgIHJkgLPWKb4aDXXrmb70haIlG3F8Hzcp1OAmcM9hHEMx+ujlK7+YbUhp60xCJR+rpg J5QAQsw+Ms4MBxu6LJH5rT7mI5JdvXZOxEsoAQRbrN0fsc1w0OtP2d03JC29iSdJi8XlOp0EzujW R8YZ4aBXOwO9a0hND1li29BScbhOJ4GDmG1knhEOc8myPFw/agXfTKHpJPsSSYCMZi1knvEN+m5J fHuRAgnj4aTf1aOa4Z7y0j+0V/39X9Xmj/6nWpxKbH+56cv9eV0dzfim3xe7519c0Gb5psbfs7ee wc3atNd60E4pAXV+RF80g4YutXx/paH18mye9D3rQ3ZCCSTE6tG7mhEjqxvhAjelbs4km2dyj7sf L6UE1CqcKro244vWOocFbk7dvEGbbWufMepHKQGFuNurxpN10VyWn1S1Q6mhWOcFF+2UElCIq71q b0MrggvcsLp5izYL1S7loF9KCaiVbczUi3fRvXOBG1dtKxY97k7jQTulBBRiDppqTC66ai7w7S5r vUt/LrrctfuDdkoJKMTVXrW7xlvNCfVuZt0cIhaHS02ddvQllABayYZNmMl9wH1T74ZW3QH14eMk fsBOKAG0Ek/kUHNywBWJ4K/d1Lo4By4WrNX7BTuhBNBKvEVDZySCF6/zSn5uXYvbS5+77NC9dRI4 K/HeD/Umh8t5NPnBVbto+nMupl3WBTuhBBC2SGvVUG9ywJxHsx9c7TzM+X/Kq2W5bSMI3vUVOJIH OrvYF1CVSqryuOSQHMybnQMjQSISLqiQUFT+EX9vemZ2yUWtbMkXmyM2G4PZ2enpVI7EW+BixYPY ynNCTpjtqE7aKbyOE3apGom3wMWKBzFJmaflhHmDuFHNwpd5eUkIfE1Cl6zPAhgrIlwc+sKbbFuD uNE2SacQc5cF3XM5TCYugLEiQmzlQcm2BnGjbdJOIW45Y0VXOvg2ExfAWBEhptp7m22rFzfaJvEU Ym5fz2KSNvMlLlY8mKkseTa7Vi9u1JQi1bEJ9D4XOhEXwFgRIXbyIJ8Tbg3jC4mSa+F5ToS0fiyB sSJCTNfRu+Q2IeWdJFIoFDUC3VZuZZ26bQGMFRGUnAYI4jYkYnajptSnwMbS86rr08xcAmNFhNjJ g3zOmPwoFoZCnfjCeVYQn+fEAhcrHsQ0o71LdrO37JKAL6SJGgyDpvPpBJl4AYwVEXZTlhOfDCfw 5OuAL4SJbzKbV+98pi1gsaJBzFLik90EnnwoFmRzHT/Bcr686fq0ACyBsSJCbORBYjdhDkjsgS80 iXcWx4uub9OYWOBixYN9kJUkJLcJPLlR4AtJCmwCuYWpLRJvgYsVD2I6QcQu50tmFJapUCS69+LY vHKZtoDFigYxWU3EYjXhFWnJBb7QIxk7PDNdqsICFisaxHRBYDjFafatqIwv5cizflovTdFl4gIY K6Jg+EojdjlflplQypFMM/E/zptMXABjRYTYyoNCzph1JpSmlRZexdNW2i0RF8BYESGmsen7ZDZ7 LToTStPqPWfMK5AzqcYLYKyIQsvnjTgJkhad6UrXSveMVgU+uzwtF8BYEYWWPyAOOWPWma60rXTP QMw7Ji3aibgAxooIMbtNlQVJidCQk7gS8wA2vFHYPhEvgLEiCpp3m6CyIikRmr60rb7ljLmNbZd5 C1yseIIWu6myIGGt04wvbKsMdmNabuY+ExfAWBEhZsupk44A2EkihW/1LKGGBdq65F0XwFgRBSXe VfN9ZGIniVy9K3tTELOQWqszcQGMFVFQPDEQ+5wxCQ1elObLhRiMLcudveRbwGJFg5jmG2KRERwx yUyveBpm2kD5tt6nO8LEC2CsiHzPAzm0SUiAJ6HpxdNdiImXp5BqM20BixWN71k/ELucL8kMOp/+ y7RSBx4TpguZuADGiggxPQGx6AiuPWXQa87nQkylbXnKm7StLYGxIvIdVwTTQ4QEeKoY8H2RsaGM dc8NlybmEhgrIsR0hohdzpjOuCfnUBBTxpoHskm3bgmMFZHvuOcQi5hADKgn+5Y7NBOzhGreL42R ZWIJjBWR7/iWQFNETICntQ747jp/aJiBmMeEaTNvgYsVD2SPrjVilxOmPbQ3PAQyL7Eawxevz7QF LFY0PvAYQtyFREt7c29KSbIdJap5BrV9yMQFMFZEPvDcxOoiSoKlI0gihSRxATQvl3SjmXeBixUP liAa9IjFwwFPWtnbUpFoRGJud0wcMm+BixWP96wjiINOvGSlelsKkvXUaYqHROu6TFwAY0WEmHXE Jw+HNbSTRMJ1+sgeoeTq2dQRC2CsiLAWk/ojFg8HPL1K70pBotGruJNoWGTeAhcrHu9ER3yycMDT OgN8IUiWmkyxbNCoSLwFLlY8iFlGQrJw8CWd5FHokW1xZkSqUrILUKxIvBUJCcm+Ae8kiasWEWHz /uffb7g5ed+lVtXYnnRH/5FWnYab9zc/bW++225to5vtPX5naO1Ct2zyByWTyfpGG8xChd9tI3sO sbb5X8oc5VEK39/ebOgjxsf2+Wb1x+luOA13zd1wGP8bTp+a9fZvyjDNRZ8kgn5ieUjwf9tfvvTL Fx9NBhLP21x/rSgPxRl8fvMTP3/5CYovafkAelt9eVsYGTzrw2peY9Cs9usNlp7V0ExP8a/hNPIf pwcw7Yc4rP/c/vamhL6F7sWsN1hzW6StcRUwxkDJyeKBl+p8XN0ep/ndx/Wbq7T4Bbon5O5poe89 dtdN/qAa7bFKWVlfdY8Gpu7h9uSiUhrfK+W6H4TKX6hIf0MAVfpQUGEmgIuSiXIK9nIKuBh8Ctv9 0NweY9xNd1SzNS78CoU7N7vT0JzHh2m8H29309wcp8On5nk/HoYmnQrVSvlrqQyXKh026pVohep5 nPfj1Mx43HnePaBZ8dfdG+qCZQL6rfWyLvlQ3lqSC8sbS8Jv+zSN/z4NzbiGV1vdUWeF1TDNqMhw au6Pp2bmb/bpm+Yw3uOX981uUSEdXrgEUqIPXCPU3K6o/uOZ6zNO4zzuZuLfPTDTKyWCoPOkfbVC 9isVyiRlgTTPW3Jvl6nlr+UK8grV8MFrTMe5obPfUYW61RF//bzetA5ryGpH5XKrg3y1Pz497KVe v25vjG45CW80r0CaVqANpIKypDl8T3P45fwt1iMNiahOuL0OWispXyp8bs5Pj4/H0zzSjBiQ/u6A 6zBNw+08Hqdz84iTviRnHT2h8ZAF2qFScor2p+6V5EhBFcZilVxPyX1YnYfzmYaVXh2n5oyiHNAO 8fGAoYW7N86XHEJPRI1t0XmdSzkYBcsL9pzDK/1itX/XQai+PGm+qXMy3Sudc21+gmA+bHenh2F+ 2xEgC7y91T0/yStcrKDT21vaN0P/9ROwjhYe80J7XDu6TR0th8GNez2M6Q5zcB5kgknGDPluvaGu PnCQevv2n83xfpPPzDpDj0bW5ImuWQeIz2tZY3FSVi37ZnH7Uio8jo58E9fYXlfFdfy4WmOpoSnl 6I2ef4Qc5cyQA8gdyqpCvm0WDd11XhL7fwAYezE3CmVuZHN0cmVhbQ1lbmRvYmoNMjggMCBvYmoN PDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDUxIDAgUiANL1Jlc291cmNlcyAyOSAwIFIgDS9Db250 ZW50cyAzMCAwIFIgDS9Sb3RhdGUgOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9w Qm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0yOSAwIG9iag08PCANL1Byb2NTZXQgWyAv UERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDQgODMgMCBSIC9UVDYgODcgMCBSIC9UVDcgNDMgMCBS ID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgOTUgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAvQ3M4IDY4 IDAgUiA+PiANPj4gDWVuZG9iag0zMCAwIG9iag08PCAvTGVuZ3RoIDE1NzUwIC9GaWx0ZXIgL0Zs YXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJlFfNziSpEbz3U/TZkll+kr+zDz5bPvgBVl7L0rKSPe8v OTKBrsqiBR6NBOpvyIgoSJLIX/7yo7x//fG2pliHsfr0du8fv/7x+uWvf3fvf/14ufe/369AhkJ5 Z0qmYo7Gpfefs6um0vu//3z99grVxJpvC3zRC4hMyXmDwAqiUhCtsWEXEvGnmg6YVWEma0JMG50p mhTiHtMFxgwDMluT6y4iR1MjHSDTHbI44xxtVCLCq4BkQj1QeJIIGiHVmRjDIaTokGSyC3tZwasQ Z52p1e9pQv/48onJxsVDDOkQ501w7hAiX0PuE5MNZbf/nOh1jPcmxwNPlJSj+InJph5CUtAhIRh7 uCwpS0j9hBTjfd1/TY46hoKJtu55ikiL4RNTTEq7UsAxoi3mGROD5NI2psZHTDG17CoKVls5nfQ5 nUTGp/2+VZseMdWQP8Q4ybb0OZ5MJpZdFeGS07V99jpXnNm+WHGZUjElGuv3xYivGsb8OZ9qDeIO MeURE02gfb3h5OSYeT7eWpP8vuRwpumYaHI5xETJt3mxvWOYfZnipFEhyTi7L1N8/hwyy5T3zoSy Lzl8/jommUinGNmB8tEWnMl2X3P4XHQM1pZ9AeE9VjHkjaNDjCQBavMMyeawz73oXhHRG9rdAh+z yWFfX1izAk0eteD/i0mfmGJs3NcXZlAxORjvyk59LobqCTVp1BJMirtq4kvBa79dUQPOc7evXLsG X63Gx11VCRbWzR3qweMrgoWby3GjIDjYua2nCR7uzR3u+uNMgseqvEUN8G/hcLcf2RECLJ07pbbO wkAwdflwtx/XIRB8XtjFhOiMrYfb/xAS4ezyz13+kGDtwikmPGKSSXv38KxlkGXq4eF8lMyQ4ez8 4XI/SjPKMsQcYvQLEAqcHe3u9vLOhApjd7Aoz+cs1Ayrc4pxKoYsnB0ddu3xPJOFtbMHK/SwAeSC ofJzdoMcrB2dYrStIR+kb/sZ+0S+yK+9tdM2jdCBejrwPOwgoSkle+iKHraT0KfGdGi9HvaWCNbu 4FGeNprQu1p7iHnYdULz6tKhL3q0BYR+Nni//55H90HoaJM98DyaHEKTm9OpMdLNFBXOp59r2pDT xu5v9qM1JDS6YZ/UzwaU0OhGd+iLHo1uRKOb8qH/8uURg7UH31I86Rh0us6dYiTkE5ENvMw+wiUV gjaXwr7iFBd0CLze1tfEAGeXDydhJa38fLsjulq7Pb2IHhb9y16qjRoVPSzlXTGJ6FhT2JebYp1G Rcea667cxARnF3fFJaIb9WFXFv7zwnvINNiZiucLU0YlwlTwyGDJP95/vOwb/3CCvbuDLstPG5Ia 5O0Fj1oB4E3E5fv9g2cSfjj4jcpPJ04Ti39//fan19+YFJbiRlql/MEk2Rsp/B5nUR8J6ju7Amya bsDCTAo3aggvLbJLFzc838VNzjjmdkjqO3fg0+gjCvLgVoBN0w3Y/tWo+liIP7k7c67IrYvZG9gx nghpfDETl9w+EpR1ZgXY7mQDFJdSmIMJvBBFSjEX3PKLGY85bgGmGO/MUrj7iEI/mBVg03QDFukv 3NwkYilf9jt3kid7chNuQuIpK+7MtayPlOdXK8Cm6QZsHXlmJS24X1LcOLEbd5TazMZacVeuiX0k lPvOrQCbphuwzvZEw4PNeVGQCIoch4ZWaZJnKT2Y3J3cSYnqI8GnC7kGbA++gevkwFF7JDPwjilu HNuNG46PuYtk3MXtuZD1cRy3hmuKa2DCCnbm3DMjS4m5mPlBuZhrT7WqU82Jj+1jxAXv3AqwPfgG LjpF3zc9GQ/fWJANip4N4ocep8jZhkllG5/AHKMfdU0DtifhAHYR+VlFACwXr0dK3AWg5OabACcp h0mlnMsiQEburboABdgWxoHskjgDVgCzjIDK9uGuoMhjMxV4oEBBeORdlROQMc7SrgHbwjiQ4QDK UOBN4gCPdLkrQMN0U0CykZg8/vhRwD3IHGOce6AA28I4kPGC2aHAoVYhAJ/n7gqSkE0FUTIJEykF nktdH2McRV4DtoVxIHsEDgVW8qYSOO4KcJo3BUnuEaakFEhv1sdZ6zVeWwgHsHczEfF8ct7UCIq7 ABzmTUCWo8RUlIDIVa+PMY83VgO2hXEgw9WNRAy5500Cx12Bl9s3FRRJ51gF7FIgWSBJUEYh0HBt 4Ru43CsOfqwsCMhguPM7KT6Dn9sd8Cf7SMPC5a+P/Jp3BQqwLYwD2ROudVeAveKA+khDK4V3KnCS hhJ8UxDE+fUx2VGINWBbGAeyT7jWXQHyBW2aBZJU66FAtvtS4CUN5XPuCsRj9TG5UQw0YFsYBzKM eohSkQNbZwTAFSkFWWrfVADTAX+NidPxUiD9Tx+TH8VAA7aFcSB7vFlDAcosB+Ar611BkvI/FcB6 sAKSdLwUSGvYR66/XYECbAvjQPaok0MBXhoOwOcpBVHev6kgimPGxXZKgTivPiaae6AA28I4ka20 flCA/4kcgM9TCkje/qkgi4vDFBzdFGS2IX1MNKqBBmwL40TGczkUlJ44+DyloL9AU0HpDhY7qRRU vod9TKMcaLy2EE7gMBPR5543+DolwKlErHLd5Ym7CRBLIkNKcwMUXFv4Ji5Jl8L8fLAIKI80tLO3 Yv5se3tnxVde/F7aHRlTnjugANvCOJEjMrcr4ORGQNVpKDbsUuAkDcXn3BX0Leh7MO2wBmwL40RO qNRdAV/wt4NlVWkoFvRSEHozEXQactszx1RHQdSAbWG8kO1QwLWOI7zYnEtCllZvSoArycQTp+Ml IUsXJGO2oxo8ENvCOaErKmUWDVzxOQIeNt81JJzcZ2GmfhCw+/muoUg3JGN2ox48ENvCCWiL33AR NDTwu8cRyJYvGsbCHHo6ot+5a+DCO8c8nfIDsS2cxF+C3w7ZIRocv/0cgXz5pqEvzF6uJCZSGro7 kDEPCQ/AtlASfwh+e6RHl5BlEx3MbPkmoS/EneA8wOSVBArvOeYw6sIDsS2c5KWXhTQ7NLD94wgk zDcNfWG2ch0wWaUh1vcc87gUD8C2UJKT7EBrAbkigT0wR9SvGTkWpio1IRVJzEtC7j6RRMLcBo3Y Fk6yPTsSPqtr4E4AEd5+z8i+MGWpjJiy0iC7IJswH4gHXlsYcUELzC2uShwKAvaHI9z3fOwLU5K3 AVO8K2CPNsf8SUiN2BbOwN+B3wWYXQO3hhzhvydkX5hid2xRHsVLg3fvOeYys0EjtoUz8Jfgd8XO dg3sdfybiyvQFwl9XaJuW9HGKQlB6rOMuc5tUIBtYQz8HegFHHIjsQK8KCQByJZVwViXQrfuaGWV gih2UcZiR4HUgG1hDPwZ+O3F77KCguzhAGTLFwV9Hex37dY9KwWpvOdYpnHWgG1hDL73Q3DCQ0FG 7nAAcuWLgr4uWXHMyYp/vhQUes+x+CFA4bWFMDhJC/ioWLsAGNLIAYUryyKgr4P75qYBU7y71lTF LcpY/KgJGrAtjMFKWsBK5qEAplwC6rdEHOvQQXLjhCncFfC2zHH6FY3XFkL8XbIiYWe6AGwOB0Dp tzzs62D/uXfEpIx7dwlB6OcGKLi28GFjJCd46vzIDgnwX7Owr0O4Zf6oW5dM7Bb7WOIoBhqwLYzs al3h4wxDAUvlgADKVUFfhxMsrIB0+5aTGAQZS5r3QAG2hdHzZ9Se0fy71p40uC9pFfA/tqteB4/c BvZ5in0Cn/5/ancBco3Lw3WpgqgIrrjXzwxJ7a6khQGuP1uaGUkjirRhMHGJZXr5EVAlDUls9jqu eOPgC1xE1zst/DRrxPgMI/dTgI7DNU6xzNv8COjxmrE13YENcByMgavomtdEAS8sJ+ACfynQcUhl IZaZ0W4FzckV1Ng1G26A42AMXEafuR0KmLM4ATnsS4GOQ/XvoECy+ltB0CzE2J0mow1wHIyBy+h8 4ILkns6/cALy+JcCHYcnjR0QPi2+FSRRIHFuwYo3DkLf1RbAVwGonTgh8X39EKDj8K5XCqhix0eA VCYau9dksAGOg9E3tQWwVAHfb07w30bUcfiVqaCIHx8FNV8z2qu84Y2D0HMVneVeUwEsYjghfvtQ x6HAixSQxY6PgO6vGbvVyhvgOBg9kbvWvKJAmy+PQhYv+qHAxvF/qSCJHW8F3ZXrjsmSwQo4DkbP ZXSt/Pm7dfUNV/mlQMeh3hYfhtWHPTATWsx6ChvgOBhRxjrrfkRAM9vgnL8E6Dicac133/MIiO26 Y9Et2ADHToi2tlkHKPzFXAOnu3by67jEluRu/R7+HK87Wo28AY6dEOVrwc9klmhZCn/Pq/7Fr+N4 rZH18QkLf5EbqFE9uOGNnQ+lK/JGZeWSSJ+k6vfIdf6LXscxrZEer2p+00s9ZrFrJtwAx06IFeAX 3kilj9KIemT5+EWv45jXSY+y4qFHB3Bp7ttAxs4hx49qSn8FZGQODrDHzegu/pH3HImTTWZl3kkA A3pDK8jWh03of6mNVb4NkC1kF2lHJID/Iw2PITapSWMT+0qz9OwfOVkzCC0qFoe68k3LphQqg1fZ gvdDihgWWKjBKrNzenatedn7Sdplu6xLvA/NeSYMjckS1wY4VjqDhYOEGwbj0GYHOrmdHIJxJy9O SV5883BHpkuNKXXjXgDHSmewumpYCwPxT/7NXLuc6GQOmq553V+pyiXWTBoTlCnzAjjeZAaKSliY 4TAO9Gaiydz05I05yhOVojxYD3NhdtCYLEdtgGOlM1i0AcINw3IoC7I3N134cCd5oFOS5/rhrsyM GmlO5V4Ax0pnsN185sQWjdZ/c+PEXtxZXoWU5Y14uDvTgsY0G5YVcKx0BouEIUZD3UVfNNbib/Ik ndkkr1qjVXkib3Iv9aHGZJlpAxwbn+F6OXBkT3EGO5E3d5SebHI3KQlSE8c93IFFiUY77hVuLFyG ifSizFWdwR7s1R7g+fIv5q5W66vVUPpcM7JvUO4FcGx8hos3Muimlx8Bz0ljD/qmd9KLGT1OUboT t7qNJzBjDpbXVsCxExowioRUuwjIuCsYD0u8BZQurdgU4MVy+CyW81UESMzRm4AFcByMhoxCqZgC JHxMQCnVFgUNAI+CIE0B7vHqO6kHNeaZ2lfAcTAaMorFZgrCj8IJAXZ5K6jYw0dBko3EJ8RXjxoc U5zGnOceLIDjYDRkVOzOFHjkKkzA8vxbQRGyqSCLk/BJi4LAVKcxZ0vyK+A4GA05YKIpcOIbdBVx UYDTfCkoco/wKYsCKQg1zly/4o2D0IDx+psR8XzSN2is8iIAh/kSUOUo8WmLgMyspzFXe2NXwHEw GjI6RzMiyh7xTQHHW0GQ2zcVNLFz7gL2KBAXiAmaJYIVbhx8hhvitCHqrdowoYLhze8l+Rh/cWLD 4jYbNqY/jXzNVcECOA5GQw4J11oVYK84oW82dJJ4pwIvNpTJLwXR8RJqLM4S8Qo4DkZDDgXXWhXA Lx1PBJAkW5sC2e5HQRAbynLeCqTG0li8JYMVcByMhhyQtbNk5IgrwwmoihYFVXLfVICio2d+aMdH QWRrprEESwYr4DgYDTngzTIFSLOcgFX2t4Ii6X8qQOlBBUns+ChIvIQamX9VwQI4DkZDDsiTpgAv DSdgeYuCLO/fVJClYsbF9osCqbw0ljT3YAEcB+NEZnMnCvA/mROSNDqPgiRv/1RQpYrDJ6KueBRU liEaS7JssAKOg3Ei47k0BU2Ng+UtCvQFmgqaVrDYyUVB5z3UWCwdrHjjIJzAcRoxVPUNVrcI8IsR u1x3eeJeAqQkkVDK3IAFbhx8EzdJl0J+HiwmtM2GbvZW5K9OGhd55N/8QdodiaXOHVgAx8E4kTOc qwpobkzoqw2lDHsUeLGh1DlvBboFugezHF4Bx8E4kQsytSrgBUcv6lYbSgn6KIjaTMTVhmx7Zizd EuIKOA7GB9mZAuY6zghS5jwSqrR6UwKqkpr4oR0fCVW6IInVWTbYEMfBOaE7MmUVDcz4nIEatr41 FJzcPbAmPQiU+/WtoUk3JLF6ywcb4jg4Ae3wG1VEMg189zgDbvnQYANrVDui33lrYOKdsc5KeUMc B2fiSvDbwx2iwfPt5wz45UuDDqxBriQ+adGg1YHEahI2wHFQJi4EvwPsoRKqbKJHMdu+JOhA3An6 AJ+wSEjxmrFGywsb4jg4U5BeFtKcaWD5xxkwzJcGHVidXAd83KIh92vGapdiAxwHZfLiDrQWkCsS WANzRv90pA0sXXJCaWLMR0LVOjGJhLkNK+I4OJNTdxQsSzWwE8CM4L4dqQNLlcyIT100yC7IJswH YsMbByMuaENxi6uSTUHE/nCG//ajDixF3gZ88lsBa7QZ623IFXEcnJHrwO8GTNXA1pAzwrchdWDJ WrFleRQfDcFfM9Y23bAijoMzciX43bGzqoG1TriYXIF+SNBxJWnZijZukRAlP0usfW7DAjgOxsh1 oBfw8EahArwoSSbALacCG1eilu5oZRcFWcpFic1ZglwBx8EYuQz8DlLvUkGDezgBbvlQoONQfnct 3euioLRrxjYL5xVwHIwxaD+EStgUVHiHE+CVDwU6rjipmIuT+vlR0NI1YwsmYMEbB2H0YgvUUbmr ABSkmRMaM8shQMeh+mbTgE9+V62lS7UosQXLCSvgOBijE1uglKymAEW5TOhfRrRx6CDZOOET3wq4 LTPOemXFGwch/l1cUbAzKgCbwwlQ+uVDHYfyn70jPkvhrlVCFPq5AQvcOPiwMeIJfpQf7pAJ4dOF Og7THfnz2rrUxGpRY8uWDFbAcTCyqvWNxxlNAaVyQgTlqUDH4QQbFaS1fatFCgSJrcx7sACOgzFw GV0dzd+9q2lwX8opwIbBxCWW6eVHQJU0JLHZ67jijYMvcBFd77Tw06wR4zOM3E8BOg7XOMUyb/Mj oMdrxtZ0BzbAcTAGrqJrXhMFvLCcgAv8pUDHIZWFWGZGuxU0J1dQY9dsuAGOgzFwGX3mdihgzuIE 5LAvBToO1b+DAsnqbwVBsxBjd5qMNsBxMAYuo/OBC5J7Ov/CCcjjXwp0HJ40dkD4tPhWkESBxLkF K944CH1XWwBfBaB24oTE9/VDgI7Du14poIodHwFSmWjsXpPBBjgORt/UFsBSBXy/OcF/G1HH4Vem giJ+fBTUfM1or/KGNw5Cz1V0lntNBbCI4YT47UMdhwIvUkAWOz4Cur9m7FYrb4DjYPRE7lrzigJt vjwKWbzohwIbx/+lgiR2vBV0V647JksGK+A4GD2X0bXy5+/W1Tdc5ZcCHYd6W3wYVh/2wExoMesp bIDjYEQZ66z7EQHNbINz/hKg43CmNd99zyMgtuuORbdgAxw7IdraZh2g8BdzDZzu2smv4xJbkrv1 e/hzvO5oNfIGOHZClK8FP5NZomUp/D2v+he/juO1RtbHJyz8RW6gRvXghjd2PpSuyBuVlUsifZKq 3yPX+S96Hce0Rnq8qvlNL/WYxa6ZcAMcOyFWgF94I5U+SiPqkeXjF72OY14nPcqKhx4dwKW5bwMZ O4ccP6op/RWQkTk4wB4nY2xSRNYsnaF9jPG3n3+16+evy/EPH/tfP3+HjH9Cxn/sDv59eXf96/rj T3f9+x/sktivVqawBF3/57xscnTHbSg6r1XUEqx/eRlBBllBzx4DdO9/EN5LypbMWdCAv2Y98liW KPJSFzp1qsQUhXn2D16KgcEdeBp4n5/2n59/c2kT3ayupenPZQvFkfyfSztfJN+l2pvfHd7+uY/t SzS+3cPOMaut9yPRXkOqHgL+zl22HHmzxjXk6SkBpQfWzR6G1r4yLWBu6FUC3ov4cZRAUvuCrWnz kDWJkCt1I1Py3tVuWnrQm6cE1ExMNy2YyUfQqRurATezb6G7dY+6VfCPpwTU1OWbPRw99FowYGxo q0mp+gsMfXhKQKmNK6PtqzzorImqfy8bulrnm1vz/3hKQA3Mq9V6mKE778BMTMSF5lZAWFpjM/Th KQE1oI5o9wd9WcDY0Hmp9lc3fjwloAZm3Gq939BN360BUHcvOlUfiqBIbkcfnhJQA9KA9nQ0VDMD 7g19cdXF1j4cfXhKQI3Of6ByWuiEgEI/R8+bCipxQ1p+0JunBJTa+Jw62AiARm3SgMqvW+hpU8LY RtXTUQJJS+0wezgZn8eAuZEp+MZMrsANfXhKQHkVr2gBC500BWZjIi00U8OHu1Qe9OYpAaVtteg+ 6ohwDUMnrYga0L35GLq1Z3LFrhj68JSAUsEwzO6OtgKqAWNDcwhgiRpzVZDDUQJJbR9v8kPOQwMG h5hFLlxu4tKfknp4SkC5ymqXD14qQvWlGmCibKE5O3Um4GheQU5PCSgVkDj9lnyuRcBlAX1DJ6y6 T9/vRd4cJZBcmjactZFR1jXApOwiX1hu777fRj4cJZBUdnezh5ORkwyYL9kuSq/cjuT14/SUgOo6 XWlaNVyrhVZR86sTQLtf9OSiM5rXuMqD3jwloNQusFELDI3rrwFagTf04KpZP/oqqKenBJROQsPs 4WjoQgaMDd1xiJCR+Oh1ioenBFSHOFVb544HnbVw6lhYNnRDarQx/SwXevOUgNIJsha1OwcwoNEg NECzP71o3u/GMtJXQT09JaC6Tgtm9wd9WcDY0JxwWiW6jAe9eUpAqZ1hDw6uQKMZa4AWmg2dcYyN z569gpyeElC9aarQxlgINCSEBjRd0otOSLuWUFH7tfL68JSA6lWLudpQQQudLKBv6AvrrZPbcs8H vXlKQHV8jtrQboZOXN2Nr3vQ/caqK0tqm+sYD08JqF5YKdUejsa1ZsDc0FxvbdzxdYqHowRS1+am hbJjSYucEDD0qzYyZT+3o7WHvDlKIHWke//tkPawb4h59Ufyv2AjUzwh/0A+HSWAemKN0p9u4Ml+ dOPCbmBwL16ZYtXj9JPA6YllVV+XFxdl90aNyQ+XMqxM7nHqC7w5SgD5lnCDCB7sFNhBNI6HDNVR Bp62xR8/CaCG6q22VlbdeYIvD+gbmOW5UCRU10sfTwmodnPxmojJ0J0tGZmadzTXzFZevWt9PCWg 2uSGqz0M3XggCMD5LPTFVWfPaJJPRwkkv+V+50lODMjMqIcMaGG/rT4WfTwloBokidoqpwxdmfYo ZelFN6w5symuZD79JIBa501Uexi48KYiYGzgiTWzilafiT6OEkjeBLwlkJwZUFlsFhnM3KpfFyfv jhJIDXpVbdVSRs4siWh0aSOzVWVWjeId/OMpAdUqlaHafaEvDxgbuiHvMhtA8ZHo4ykB1Srbi2qG bOjE9gNRUTY0u2C+8CzdD/H0lIBybeNKR9EXmyakEHroQhesOrFKF+/gH08JqIbpo/8Oq7xEJw/o GzpjvYnVFNfR0bunBFTLFCCqKhNtFYMQKJCdeUOzwULR4Tpa+fh4SkC1RNmk9jD0pKxCwP0WU9yT VJCAxSeij58EkOt1V+8EJwbcVIYOts6dEnPbJ6KPpwRUQ5qOX505LkMPClgMJWlHg8wSkn0i+jhK IOloNMzuRu5U3QgYG7kjna/Z7aaTfDpKIPk458MdyXnUX0x/VTvJQ0biXeyJ2Qeij6cEVMXwQfs2 tOoeBug929CVi7b7WC2nP54SUHXoZ6ndKMKJvjyg72iuujRWkUXeHSWQ1MakOLS6Glm/piFAS9pG xmW5OMrl5Gd4OkogVYwetO9EspZdD7hvJ/8Nr1oerzJo6U/SQ/vnr5///P73x+WOFxm++0RJeFfF bKK25tl6d2KAFuj17n/9XIASuETvpGIqndHvKtQ7/eI/6riceEr6MhyaAD71fwpU2LCXIVHqxXBV iPlixozBv/uWoiHrc1Y+XTGfnhJQM/Od1WZURSeoEAbMDY0bd3ekwe0a4+MogTQzE6UmXkQjJ03S gUx8yVThN1P47ulBb54SUDMxU2pmfQF6sh+N2zPF0Lxyd+F+uMj4eEpAzYuJUDHDGBoqhAFjQzeu OlV/gaEPTwmoeTFlKiavhc7ajyZK1YvmlZu2464zPp4SUPrOqsuomBcNjRFRA7QZpBfNrZiTa3eh 8fGUgNJDmGb3B31ZwNjQHB8m5ca804PePCWgBlqX2k1/DA0logHoZC+auzxZ9KcLjY+nBNRAw6U9 Ha0CNTHg3tDsIbPY2n1cOz0loLwkeIEwdEJAoZ+j543LMhM3xIXGx1MCanR+jta95GgoEQ2o/LqF 5gHikmFbnHw4SiCNxjNQezgZn8eAuZEHFj0mMzAv9OEpATUaM0cbTHnQSVNgNibSQjM1Rs++Kwu9 eUpAeSP0tgh0YrXVxon0X2iOVaN23xVDH54SUKNwYFG7O9oK6Oy8tAtdkXssUWOuCnI4SiCNwiGr XWxeRsYUNgeHskXmwIYpQZ9PST08JaBcJ7lqUvR1c3ack6PkQmcsujMBh0uNj6cElKo7nH7Dti30 ZQF9Q7NhU+Jivxd5c5RAGhhC1cZZGxllXQMwk75kzoK9+34b+XCUQFIV3c0eTh7UaPOmZHOyXZRe uR3J68fpKQHl2t8nAUNDWN4XdeZCc8zsGc1rXOVBb54SUGoX2KgFhu5UwzrhpA09uGrWj74K6ukp AaVz1jB7OLpRw2vA2NAUoRBr+Oh1ioenBJTaGCt0ViwPGnOHjnxlQzekRhvTz3KhN08JKJ9pfcIF unJauguHp4Xm/baBtq+CenpKQHUV72b3B31ZwNjQhaumJEVnWujNUwJKbQyebXAMBbpwMr0rJ6mF zjjGxmfPXkFOTwmopR2hJwwNCaEBKr3Li05IOxsN+7Xy+vCUgOqq/02WtgedLKBv6AvrtQmu3fNB b54SUB2foza0m6ETV3fj6x50v7HqypLa5jrGw1MCqhdWSrWHo3GtGTA3NNdbG3d8neLhKIHUM+eh jiUtckKA6vy2kQG2LGntIW+OEkgd6d5/e+IA8OfnhphXfyT/CzYyxRPyD+TTUQKoJ9Yo/ekGnuxH Ny7sBgb34pUpVj1OPwmcnlhW9XV5cVF2b9SY/HApw8rkHqe+wJujBJBvCTeI4MFOgR1E43jIUB1l 4Glb/PGTAGqo3mprZeVMqwGXB/QNzPJcKBKq66WPpwRUu7l4TcRk6M6WjEzNO5prZiuv3rU+nhJQ bXLD1R6GbjwQBOB8FvriqrNnNMmnowSS33K/8yQnBmRm1EMGtLDfVh+LPp4SUA2SRG2VU4auTHuU svSiG9ac2RRXMp9+EkCt8yaqPQxceFMRMDbwxJpZRavPRB9HCSRvAt4SSM4MqCw2iwxmbtWvi5N3 RwmkBr2qtmopI2eWRDS6tJHZqjKrRvEO/vGUgGqVylDtvtCXB4wN3ZB3mQ2g+Ej08ZT/kV3uupbk NhTN+yvqD6z3I57MgA0MOpjAcOZJDNDAjAP/vrk3pTrSIRpgX95LrlKpJHLToWphe1HNkAwd2X4g KvKBZhdMATa39RHvSHGopW2W0lF0YNOEFEIP3eiMVUdW6bw6+FekOFTF9NGebpWX6LgS2oFOWG9k NcV1XOgzUhyqJgoQVZWRvopBCBTIznSg2WCh6HAdrXx8RYpD1UjZpH439KCsQsL8FFPck5hxAPOa iL7ixIGWXl/qneDIhElluMDWuWPk2V4T0VekOFTFMe2PzhzB0J0CFkNJPNEgs4SkNRF9BYoj6WjU zW9GblTdSOgHueE4h9HsppN8B4ojrXFuDXckp14eTH9FO8lLxsEL7IlpDURfkeJQBcMH/Wlo1T1M 0Ht2oAsXbfex2Jn+ihSHKl1fS/1KEU50WAntRHPVubKKbPIZKI6kPibFrtXVyPo2FQla0g4yLkvg KJfi+oZ3oDhSwehBf0aSteyuhDkX+Q9ElfxG5U5P/4v60f78/cdvz39+LLmzigyffaPEPatgNlFf z9l+dmSCFuj97F9//OWX/47nl5/P/YCfv/z9R3z++sTn3/qbqf3uf3oHnr89//hneP716uNBcZUb H/RZ8M8fgaTVzWKCtIlYGb6wYNsiXhSSrdvKsMSCGw0/psAl6tIzt5H7b1tvFqrS0FekONRI3A31 p6Hj5HZpwjzQPC5mcTaJviPFoQaql/pQ4RuNjtXHOlaG5iE3uyXJV6Q41Ij8Q0msRkAPHt0+17ky NK+mWZQBQ1+R4lAj8HXUH3mjgyXYsTE064lZXMKNPiLFoUbgRyiZ5xfozioxwjqBRFsVNIuKa+gr UhxKn4mjUzBdGhqyRRMiT9JGs3SbLbMt9BUpDtUnq4P67UVHSxj5g2bDMYvettFHpDhURwVTv2qV MHSlCNHPXQ50/bTKuslXoDhSRwWjPxdZqyQStHkcZHZ2s3ue+YoUh1IfZWJVE0OnzIR2oK1X2qyb 54s+IsWhOn5Q34ok0IjUBCujC20qymwt++xdkeJQHcunP150sIR5oKn9zNY6X/QRKQ7VK5+52hHQ +CqaULmEjaZiNVvbPntXpDhUL1SZq4kCDeGiCY0bt9HU2WZr3zfmihSH6oX1bLV+Q0dLwOfe6L4H hDraCz7ixIF6pp5fagVguxGjc5jbYI4zZuvca74ixaFUVGEIWRJL0WFyShmDc+dC2wxmtgVrxF+R 4lA9cXKqqIYbHS2hHei0p8cW+ws+4sSBerRWaFIW4MFhcIndDeawa7alveYrUhyqo9PQnwvdOcIO NKYPunI7aFF2DH1FikOpH1VUrbHB0KmnZwaK0o3u8dm2lfqij0hxqIaurr4NO0Cj7WuCjUMbPSjg aVtdR++OFIfSoWyaP150sIT5QWMg3ba18qKPSHEo9SFg12AJtP6EhKSv9UGntIUSV2roK1IcqnXK 7qorWOhCXT4zJ61XXY5n2zb2qq9IcSj1g/m9bnS0hJE/6NqfbduML/qIFIdqjQNOVWG10JkT0Cy6 yA+652fbvoTpV6Q4lKpHjGXqz4VOnNum6vQDzd5iFpXO0FekOJT6GCVVmNYXjVlTE9oHbcOp2Z7y iz4ixaGaymb4U4+/oaNGakLT1/qgrZjS9k2+AsWRmip488dLDpYwD3KxhsitLulFH5HiUOonrbAN 38jQEMaaoEPBgW5E0/Z9za9AcaSmw5SewQZpD39qAUb80Jf6gPt4tu3N1nwHigOpH8zvGxwtfhzg yWZIu+TpHSeOoz/gPuqvs3EH76um4fYubov52bZv8BUoDtRwPulbq9Et61oZsYPhIKf5bNuHSY+v SHGoBjWrfmYzIzquBHS7jS712RY9eqHPSHEoPSKo13oWs6Eb6zlOVD7QrTzbjrBWfUeKQ+mVwZar Pw1d2YWQMA+0VTqAo9X/rzhxIPVxRtb0STDOEK5uPcCT20E70tjoM1IcSisI+rmWKpxyRRf2e9Qy XIOFxtZti75B9B0pDqUldZo/NjqshHmg87PMKGODzzhxIPVRQLRyFwNnFhiU9nqAS3u2HdVu91ek OJQ2rwZ/sAYqWuUUEyqL5Ea39Gw72jp4d6Q4lPrB/L7RcSWMAz3is+1YNekrUhxKuzmKk8qGbOjI 4oXmXw707M+2Y6xV35HiUCpv0BfVn4YO7JtQQ+GDHiwcy8616jtSHEr9qH4P7O1Ep5WA5r/RlKVm Z0gbfUaKQ6neQ/NSYZnoqx5sTFCJcqDLeLad0aTHV6Q4VI28AeqPjQ4rYR5o9hSzM6WNPiPFodSH 3lOlXQw9qAchxeuBHkTTzrxWfUeKQ1XTyDp4QLMqWncOCTFQ1G705IbQzmLS4ytSHEr9YH7f6LgS xgeN/X1tzRt9RopD6eyVdR7RSSwbWpWEtj6MakXF+Ubn8ry2rVXfkeJQOop286ehVVAxQe/a/KDL fF67ZM1XpDiU+hG+xsVi6LgS+oFu9Xnt2OQzUBypaIWFry9n5EIpoJuob/UhD+6E2bnO3h0pDlUa K4P609Cqj7slzI3+A1Elv1EZGqnjv6ipf/7+47dHBdSDf3zmjRD3DPWR2FXc5/XMxIRC4ctn/voj PPi3pSokVMe4VBQ4QC7PanJUwehq6+/cH4VBygZqN2wPaPoSBkyR3/ADtISM+avbiqr6JfDwqIxM EF8RS8bxWhseInqMWQwyhr4ixaFG4i4UXAqi4+Q29c532uiM82G2FNMhX5HiUOrjo5bIT2xofPU+ eAg2uqLHmMWMtNFHpDjUiPxDSTyYQA/dOk2YPLob3bghtKXtVV+R4lAj8HXUH3mjgyXYcTH0IJq2 9P6ij0hxKPXxEUrm6QW6syyMwG+y0Hq8nm0xfhn6ihSH0mfi6JTCygU0BklNiDxJGx3RdM2W2Rb6 ihSHUj+Y3150tISRP+hMNC00zUYfkeJQ+n0L/KpNztB6c5Cg/eJAV24Ibd3kK1AcSQ/lMH8uspZF JGR9qw+Z+tdszWvGvSPFodSPVf3GqmPolJnQDvQgehh6vugjUhyq4wf1uxYSQyNSE7Rd1BedAnqu 2Vr22bsixaE6lk9/vOhgCfNAR7QYs7XOF31EikP1ymeWwZsANL6KJlQuYaMztsJsbfvsXZHiUB0n pT4F58bQkTV5NG7cRlf0GLMYdg19RYpD9cJ6pn6fGx0tYfQPGueZpo72go84caCeKeIrbqiB7UYM XOH8ggdqqdk695qvSHGonjj81LDmUwgtjJdjcFRZ6BwANdtCXT3rihSH6qh/6kcOT4aOltAONO4g TYv9BR9x4kA9shlWVHADD86AAyX+A86opGZb2mu+IsWheuBgWtF3DN2pNMbkILvRldtBi7Jj6CtS HEp9yMqKbrnR0J0TX+uD7uhaZlupL/qIFIdq6OrqF0pjoNH2NSFSPG/0wKEz2+o6enekOJROYtP8 8aKDJcwPugR2LdrWyos+IsWh1Idi1ZGvLLT+hASdSw50YteixUoNfUWKQ+nUi3Gp6goWunCempm6 fKNZOcy2sVd9RYpDqR/Mx8hn6GgJGAo3uvZn2zbjiz4ixaFa40xTVVgtdP4/2WWPA0tuA+F8TzEn WOj/J3ZuwHDgEzhbBt77By4WpR72EAv0W75X+pqjlsgih56NydWhZ/3c50zH7r2VElDwj7Mz3gdd OKrBYCaHZm+xp1a6a02dUgIKcdZ4cZo0dKlcML5oLUL3OUt90E4pATXg7TXeOP6GzlBiAbx9/6Kt mPI5L/kllEAa8NcWr4ecbMF25GYNkVvdyoN2SgkoxAUVdug3MrQaYyzQmeKLHkTzOe81fwklkEbB jxqfodZe4705NOx1hgQDz/W5zzks57dQAghxsnhecDb9cuDNZsjnsadvnQQO/kfvI/66GnfxvmKZ 3t7DHbl+7nNe8EsoATT0fDK2VoMtm6iMuoPJkcv+3OdcZj1+lBJQQ90s4spmRnQ+C7TbXXTrn/vU Hn3QXikBhSOi9RpnsRp6sJ7riaoOPdrnPlc6Wb+VElC4MrrliLehO7uQLtgObZVOwdnq/49OAgix npEzfxKsZ0ivbnfgze3gc5V10V4pAYUKov0cpUpPOdCN/V5rmV6Dg9atu0/tG0S/lRJQKKnb4nXR 6SzYDl0/57HaumCvkwBCrAUElbsZuLLAaGnvDtzG5z5Xt9v9o5SAQvMaGi/WQKBhp7igs0he9Cif +1zjHLy3UgIKcbJ4XnQ+C5ZDr/y5z3Vq0o9SAgrdXIsTbEM1dGbx0ubfHHrPz32udbJ+KyWgYG+0 LyLehk7sm+qG0he9WDjOc5+s30oJKMQZ8Uzs7USXs0Cb/0XTltpzp3LRXikBBb+nzQvGsjCGHxxc AIvi0G197nNnsx4/SgmonnkDEK+LTmfBdmj2FHvuUi7aKyWgEKvfg9Nuhl70g2rFu0Mvovnc9WT9 VkpAdfPIGDzUswKNndMFOdHUXvTmhvC5m1mPH6UEFOJk8bzofBasL1r393n2etFeKQGF2atiHsEk Vg0NJ4HWp6Nagzm/6No+z3OcrN9KCSiMotPibWgYKi7AXdtfdNuf53lszY9SAgpx1hi63Aydz4Lp 0KN/nue6ZC+UQGqosBrjxxm50QpgE/GrvuTFnbDnPmfvrZSAaoOVAfE2NPzxtAX7ov+nqlYfVVWP NPWPjKV///eP/3xgoD76H9/5Rkh4B2JdOGHu63ln4YJG48t3/gs/CHNjOq7n3//4Jx2rOqmpU1MD d+kL2uf0OpphbW7n37lNYKqjTfOaYwf4649/u6z1H7XafOk/q8/P6usZfAsN0tQ6156PsMv3I6z1 oJ1SAmrpLMl4TO/isUBP8/N9eXD8mfwRSiAtLZBN/fGdxTLv3LP3RmYfPsd95MdqO6UE1Cr8nK/5 QJsrFmyX9MjfS9r6g3ZKCailtZfxeLLWIjS1OH/RnVnn5Qrij1ICCnG1V5njVMOobgMLusu65dN2 vmX8RykBtWDj8KqRWIINrVV5Lv79q2qdMp7Gg3ZKCSjE+s6R6LS+Y83crBwXXa4x+XbjH6UEFOJq r7KJcm92qbm5b68qvno5qZ/5wwklgFbi1/UDk1pR6K0KGDiNYy2do3oJJYBW4okc+Qyqe7Nl46/z N+Nldji/XOBLKAGEuNqL5p2Y9Drjr/s3Y3VQOhvQqLU7MDmdBM5KvPej0KyQq5ZxJVaBy5202uvl 4l9CCSBskdYqxD0fcMVAgS+RXcI8D3P6weOlk8BB3Ow98yasDmZpU/1yOxPuZzcO1+kkcBAnfU/l aPZMjkv90JdLXzZ5TX4GxyOUAMLF0X9AbLZnq6+HvlD2OEoOBpkT6Smgb6EEEOJmL3oGUp1RoG8u 48KMUz3DwQE7oQTQXNz7oTPPGUeXJTJcxjy+g83kzgQvnQQOaqqeFcRnRkLn1jyqb1I62eqUezf6 gJ1QAmhOnu7Rrmsf9OKr+hZl12KwTsxSL9gJJYAQ63VE8071DLnLEnEdSg+C+c9Ej3BmXCeUADp+ Y/Tj4LfaA+pdf7KJa2QFjzvBvIQSQIi7vWjcjNWAwDC47sQLN9hBxq0TL50Ezhys0YjNUG44y0y9 a002ffY1zhck+CWUAIKRZDvRL3HAOv1B7xoTb3Jn1n1crJNJwCBmK8HPuPnqNAI7W7/lZzbm27nN xwC8hRJAiKu96Ayflc0eeteT6Fl61axHOWXipZPAoVVHPM8IA33f1LuWpOcW4HKOxeE6nQQOYv2C Q33x4VbNY/iOpPceXJ6KfrFOJgFzhhbE24oPbNKm3vUjKzusmf3swksmAYNYL8hA5egHOy0N144G +2cbdijWBTuhBBDmN73SiPvNl21m+nZk1azRa/ZRL9gJJYAQN3vRvBmzz0yWrAueeiUa+4YetwN2 QgkgxFo2B8yRZZytz0wW2QsezJgWqNezxy+hBNAs/N6IT0PK1mcWu8IFd8240tf3Wy1fQgmgWfg/ iOfNmH1mcdkFN/1slR5TjfYBO6EEEGJNfabbkJI1Gp0kvmAW4EpH0c6I+RZKAM1MbzPT7UjJGs3m p7ngwox5jNu6XKeTwJnZpr10GxJsXaZez9LlsrDXWniY9wU7oQQQYj3P+MP6CITLEhkuYbbQygbd un27t1ACaCbeQMR1H3C3RPY3487jUNlIW8sX7IQSQDOxYiAeN2NtNPihWl8eMIiF7a49+TqZBAxi rW+IrY3gE2ub2YnV8GKn5lvGOHeE4JdQAmhsFuRZTiOBXhvNtpnuASuXVSiVi3UyCZix2T8Q95uv thmcfP3jYm0fWCbqmhfshBJAiPUNiK2P4NprBjsznwesW1tY5etxa2+hBNBY3BFUD2sk0OuOQb9d xlUzzpsH7lTMt1ACCLF+Q8T9ZqzfeOvk4MCacWZBrufWvYUSQGPxzCG2ZoJmoGdyF57QC2YLzfSX tZqZeAslgMbiLUFPsWYCvdo66Ne3/mgxA5hlopbLdToJHLQ9vdaI+01YfeiuLAKXq9RaefH2xTqZ BMyYLEOI1zxY9c27+pbUliaaWYPKnhfshBJAY7JuwrpYJ4HpmJaIa0ncgExzqTea3JdOAgcmSAs9 YpvhoNdeuZvvSFoiUbcXwfNynU4CZwz2EcQzH66OUrv5htSGnrTEIlH6umAnlABCzD4yzgwHG7os kfmtPuYjkl29dk7ESygBBFus3R+xzXDQ60/Z3TckLb2JJ0mLxeU6nQTO6NZHxhnhoFc7A71rSE0P WWLb0FJxuE4ngYOYbWSeEQ5zybI8XD9qBd9Moekk+xJJgIxmLWSe8Q36bkl8e5ECCePhpN/Vo5rh nvLSP7RX/f1f1eaP/qdanEpsf7npy/15XR3N+KbfF7vnX1zQZvmmxt+zt57Bzdq013rQTikBdX5E XzSDhi61fH+lofXybJ70PetDdkIJJMTq0buaESOrG+ECN6VuziSbZ3KPux8vpQTUKpwqujbji9Y6 hwVuTt28QZtta58x6kcpAYW426vGk3XRXJafVLVDqaFY5wUX7ZQSUIirvWpvQyuCC9ywunmLNgvV LuWgX0oJqJVtzNSLd9G9c4EbV20rFj3uTuNBO6UEFGIOmmpMLrpqLvDtLmu9S38uuty1+4N2Sgko xNVetbvGW80J9W5m3RwiFodLTZ129CWUAFrJhk2YyX3AfVPvhlbdAfXh4yR+wE4oAbQST+RQc3LA FYngr93UujgHLhas1fsFO6EE0Eq8RUNnJIIXr/NKfm5di9tLn7vs0L11Ejgr8d4P9SaHy3k0+cFV u2j6cy6mXdYFO6EEELZIa9VQb3LAnEezH1ztPMz5f9KrZbmRGwbe/RVzlFPlZEjOgJyqVA55XHLI IatbKoexPV5r15Rckpyt/H0aAGlhlvuoJJeVsGo3QYJEo8txFF6Dyw0P4kHXiTVhsaOuaKfyjpLw WE6j8BpcbngQs5QRDyfCG9WNOhG+yitDQpRnElOxPitgbojwcPgHCtW2RnWjvkinEssti26S4wiV 2ABzQ4R40IWKbY3qRn3RTiX2knHPTzqSr8QGmBsixHz2NFTbSupGfRFPJZbrSyImZTJf43LDg54q kjdU10rqRoMVqSQmkKgedCE2wNwQIR51IaoJ+yB4I1H6LEj6RCzjxxqYGyLE/BxpLG4TUp40EaNQ fBH4tcpVduW2rYC5IYKScwNB7GMhFjcarD5FMZYkoy6VnrkG5oYI8agLUc2Y/SgGBqNO8uBIFIRq n1jhcsODmHs0jcVuToO4JOCNNPEFQ6NJVCooxCtgbogwm4qcUDGcwLOvA94Ik7xkMa80UqU1sNzQ IBYpoWI3gWcfigE5XNpPHCRfmXSpDABrYG6IEAddSO0mzAGLPfBGk2RmGWXQJV/axAqXGx7Mg6Ik sbhN4NmNAm8kKYoJlCvM16LwGlxueBBzBRGPNV82o7BMRpH43atjo36stAaWGxrEbDURq9WEV+Qh F3ijR9p2pGeO5RRWsNzQIOYHAsOpTnPyqjJk5YhEPwfSS5EqsQHmhigGedKIx5qvyEy0cqTdTP3P SKESG2BuiBAPulCsGYvORGtaeeDtpdvqdSvEBpgbIsTcNmkqZnNyqjPRmlYiyVhGoDGUM14Bc0MU vdQbcREkpzqTrGvld8ajgtSudssVMDdE0csXxLFmLDqTrG3ldwZimTF50C7EBpgbIsTiNvsqSL0K DTuJC7E04CATxTAV4hUwN0TRyWwT+6pIvQrNZG0reclYrvGQKq/B5YYnOrWbfRUkjHVO8Ma2amMP wctlniqxAeaGCLFYTld0BMCkiRjfSiKhQQR6GIt3XQFzQxR79a5O3qMQj5rIxbuKNwWxCOkwuEps gLkhir10DMRUM2ahwUa5v7wSg9GL3A2v+RpYbmgQc39DrDKCErPMTL10w0obOV9PVN6IEK+AuSGi SRpy9EVIgGehmdTTvRIzr3Sh3ldaA8sNDU2iH4jHmi/LDG4+f1RaPQdpEyHFSmyAuSFCzCsgVh3B s+cMJif5vBLz0Xrp8qFMa2tgbogoyYmge6iQAM8nBvxkMg6csZvkwpWOuQbmhggx1xDxWDPmGk/s HAwxZ+ykIYfy6tbA3BBRkjuHWMUEYsB3cvJyQyuxSKiT+TIEHSbWwNwQUZJXAk1RMQGexzrg06X/ cDMDsbSJ4CuvweWGB7LHzxrxWBPmOXQK0gQqL7OGIA9vqrQGlhsaitKGEKdYaHlunoKVpCFxok56 kJ9iJTbA3BBRlL6J0UWVBENH1ESMJMkBOBku+UUL7wqXGx4MQdzoEauHA561chqsInGLRN9OQhwr r8HlhodIdARxdIWXrdQ0WEEaiG9aL03Cj6kSG2BuiBCLjlDxcBhDkyYSL91H54hen95QbsQKmBsi jMWs/ojVwwHPW5lGK0jcenu5SdwsKq/B5YaHRtURKhYOeB5ngDeCNPAl60U2uFUUXoPLDQ9ikZFY LBx8SdI8jB4NHjVj0r4kuwLlhoQGlZBY7BvwoyZx0SIm7N789NuVXE6Zd/mqOkxPLvEHa9VxuXpz 9eP26rvtduhct33A3wUeu3BbbuqXnt8na40LaIU9/mybxXKos63/cuI4nR7z8fZOvw3d9sPVH5vt 44KVTtfoFpvna4wum8P+tHT7l3y7HHf7t53+dCc/PS55uf5z+yvvoPRNKhJyA9JBmoh8bH/+z9zY b6z79ewUU3dTPvvO4XgTpiQIi5tw3rxbOU3ZoWyO9/U9dpp+2L5jMvrC4VU6XGPwcd5ZdtJjV+DS r4EZN78vp2dO/9TNx6Wbn06H7vRy+265O3fnw2VP17ro53fgHcY6jx672sK/TfyVpc3cXTL3kjmX Ic+7fff8cnw+oACHBy7L80fF2J06Tr7vbjBdTWnkGtabUghd0Etz+/J0jYPbvC/lTVLeoOWd797v Dx+elvu3qOj+/PWaerwNF9HI/t+JVBZ7IpcDkW9pVcmPM+2W/Xz7xAWWg9h+c/XRTXBOt3+ej28X lB1nQJvDNc4rbrr73eluPt53Hx5n/WH5aznibGjDh7ST/7p/mZ86ORAmv9SJJK/d/uFwzPN5d9h3 u3P3OCOT28PLGfmc59P7bn44L8fujGra9MIlPS33a2lRUOxwuf/qlRxGaLL38esF+FIjemWxBXDS O9mJ1SZhe5Bf1+NyF+f9fVMd3sYv26thhBnAX8KVYjhGQjC73c1I6sLQNB+4aX46Q1YuHuM+82jk ELUUJ7yY/f18Phz/rutOSVqeR0MfMHjouhjFvvWedN1/BgD0LG8uCmVuZHN0cmVhbQ1lbmRvYmoN MzEgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDUxIDAgUiANL1Jlc291cmNlcyAzMiAw IFIgDS9Db250ZW50cyAzMyAwIFIgDS9Sb3RhdGUgOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzky IF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0zMiAwIG9iag08PCANL1By b2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDQgODMgMCBSIC9UVDYgODcgMCBSIC9U VDcgNDMgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgOTUgMCBSID4+IA0vQ29sb3JTcGFjZSA8 PCAvQ3M4IDY4IDAgUiA+PiANPj4gDWVuZG9iag0zMyAwIG9iag08PCAvTGVuZ3RoIDE1ODI2IC9G aWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJlFfNziSpEbz3U/TZkll+kr+zDz5bPvgB Vl7L0rKSPe8vOTKBrsqiBR6NBOpvyIgoSJLIX/7yo7x//fG2pliHsfr0du8fv/7x+uWvf3fvf/14 ufe/369AhkJ5Z0qmYo7Gpfefs6um0vu//3z99grVxJpvC3zRC4hMyXmDwAqiUhCtsWEXEvGnmg6Y VWEma0JMG50pmhTiHtMFxgwDMluT6y4iR1MjHSDTHbI44xxtVCLCq4BkQj1QeJIIGiHVmRjDIaTo kGSyC3tZwasQZ52p1e9pQv/48onJxsVDDOkQ501w7hAiX0PuE5MNZbf/nOh1jPcmxwNPlJSj+InJ ph5CUtAhIRh7uCwpS0j9hBTjfd1/TY46hoKJtu55ikiL4RNTTEq7UsAxoi3mGROD5NI2psZHTDG1 7CoKVls5nfQ5nUTGp/2+VZseMdWQP8Q4ybb0OZ5MJpZdFeGS07V99jpXnNm+WHGZUjElGuv3xYiv Gsb8OZ9qDeIOMeURE02gfb3h5OSYeT7eWpP8vuRwpumYaHI5xETJt3mxvWOYfZnipFEhyTi7L1N8 /hwyy5T3zoSyLzl8/jommUinGNmB8tEWnMl2X3P4XHQM1pZ9AeE9VjHkjaNDjCQBavMMyeawz73o XhHRG9rdAh+zyWFfX1izAk0eteD/i0mfmGJs3NcXZlAxORjvyk59LobqCTVp1BJMirtq4kvBa79d UQPOc7evXLsGX63Gx11VCRbWzR3qweMrgoWby3GjIDjYua2nCR7uzR3u+uNMgseqvEUN8G/hcLcf 2RECLJ07pbbOwkAwdflwtx/XIRB8XtjFhOiMrYfb/xAS4ezyz13+kGDtwikmPGKSSXv38KxlkGXq 4eF8lMyQ4ez84XI/SjPKMsQcYvQLEAqcHe3u9vLOhApjd7Aoz+cs1Ayrc4pxKoYsnB0ddu3xPJOF tbMHK/SwAeSCofJzdoMcrB2dYrStIR+kb/sZ+0S+yK+9tdM2jdCBejrwPOwgoSkle+iKHraT0KfG dGi9HvaWCNbu4FGeNprQu1p7iHnYdULz6tKhL3q0BYR+Nni//55H90HoaJM98DyaHEKTm9OpMdLN FBXOp59r2pDTxu5v9qM1JDS6YZ/UzwaU0OhGd+iLHo1uRKOb8qH/8uURg7UH31I86Rh0us6dYiTk E5ENvMw+wiUVgjaXwr7iFBd0CLze1tfEAGeXDydhJa38fLsjulq7Pb2IHhb9y16qjRoVPSzlXTGJ 6FhT2JebYp1GRcea667cxARnF3fFJaIb9WFXFv7zwnvINNiZiucLU0YlwlTwyGDJP95/vOwb/3CC vbuDLstPG5Ia5O0Fj1oB4E3E5fv9g2cSfjj4jcpPJ04Ti39//fan19+YFJbiRlql/MEk2Rsp/B5n UR8J6ju7AmyabsDCTAo3aggvLbJLFzc838VNzjjmdkjqO3fg0+gjCvLgVoBN0w3Y/tWo+liIP7k7 c67IrYvZG9gxnghpfDETl9w+EpR1ZgXY7mQDFJdSmIMJvBBFSjEX3PKLGY85bgGmGO/MUrj7iEI/ mBVg03QDFukv3NwkYilf9jt3kid7chNuQuIpK+7MtayPlOdXK8Cm6QZsHXlmJS24X1LcOLEbd5Ta zMZacVeuiX0klPvOrQCbphuwzvZEw4PNeVGQCIoch4ZWaZJnKT2Y3J3cSYnqI8GnC7kGbA++gevk wFF7JDPwjiluHNuNG46PuYtk3MXtuZD1cRy3hmuKa2DCCnbm3DMjS4m5mPlBuZhrT7WqU82Jj+1j xAXv3AqwPfgGLjpF3zc9GQ/fWJANip4N4ocep8jZhkllG5/AHKMfdU0DtifhAHYR+VlFACwXr0dK 3AWg5OabACcph0mlnMsiQEburboABdgWxoHskjgDVgCzjIDK9uGuoMhjMxV4oEBBeORdlROQMc7S rgHbwjiQ4QDKUOBN4gCPdLkrQMN0U0CykZg8/vhRwD3IHGOce6AA28I4kPGC2aHAoVYhAJ/n7gqS kE0FUTIJEykFnktdH2McRV4DtoVxIHsEDgVW8qYSOO4KcJo3BUnuEaakFEhv1sdZ6zVeWwgHsHcz EfF8ct7UCIq7ABzmTUCWo8RUlIDIVa+PMY83VgO2hXEgw9WNRAy5500Cx12Bl9s3FRRJ51gF7FIg WSBJUEYh0HBt4Ru43CsOfqwsCMhguPM7KT6Dn9sd8Cf7SMPC5a+P/Jp3BQqwLYwD2ROudVeAveKA +khDK4V3KnCShhJ8UxDE+fUx2VGINWBbGAeyT7jWXQHyBW2aBZJU66FAtvtS4CUN5XPuCsRj9TG5 UQw0YFsYBzKMeohSkQNbZwTAFSkFWWrfVADTAX+NidPxUiD9Tx+TH8VAA7aFcSB7vFlDAcosB+Ar 611BkvI/FcB6sAKSdLwUSGvYR66/XYECbAvjQPaok0MBXhoOwOcpBVHev6kgimPGxXZKgTivPiaa e6AA28I4ka20flCA/4kcgM9TCkje/qkgi4vDFBzdFGS2IX1MNKqBBmwL40TGczkUlJ44+DyloL9A U0HpDhY7qRRUvod9TKMcaLy2EE7gMBPR5543+DolwKlErHLd5Ym7CRBLIkNKcwMUXFv4Ji5Jl8L8 fLAIKI80tLO3Yv5se3tnxVde/F7aHRlTnjugANvCOJEjMrcr4ORGQNVpKDbsUuAkDcXn3BX0Leh7 MO2wBmwL40ROqNRdAV/wt4NlVWkoFvRSEHozEXQactszx1RHQdSAbWG8kO1QwLWOI7zYnEtCllZv SoArycQTp+MlIUsXJGO2oxo8ENvCOaErKmUWDVzxOQIeNt81JJzcZ2GmfhCw+/muoUg3JGN2ox48 ENvCCWiL33ARNDTwu8cRyJYvGsbCHHo6ot+5a+DCO8c8nfIDsS2cxF+C3w7ZIRocv/0cgXz5pqEv zF6uJCZSGro7kDEPCQ/AtlASfwh+e6RHl5BlEx3MbPkmoS/EneA8wOSVBArvOeYw6sIDsS2c5KWX hTQ7NLD94wgkzDcNfWG2ch0wWaUh1vcc87gUD8C2UJKT7EBrAbkigT0wR9SvGTkWpio1IRVJzEtC 7j6RRMLcBo3YFk6yPTsSPqtr4E4AEd5+z8i+MGWpjJiy0iC7IJswH4gHXlsYcUELzC2uShwKAvaH I9z3fOwLU5K3AVO8K2CPNsf8SUiN2BbOwN+B3wWYXQO3hhzhvydkX5hid2xRHsVLg3fvOeYys0Ej toUz8Jfgd8XOdg3sdfybiyvQFwl9XaJuW9HGKQlB6rOMuc5tUIBtYQz8HegFHHIjsQK8KCQByJZV wViXQrfuaGWVgih2UcZiR4HUgG1hDPwZ+O3F77KCguzhAGTLFwV9Hex37dY9KwWpvOdYpnHWgG1h DL73Q3DCQ0FG7nAAcuWLgr4uWXHMyYp/vhQUes+x+CFA4bWFMDhJC/ioWLsAGNLIAYUryyKgr4P7 5qYBU7y71lTFLcpY/KgJGrAtjMFKWsBK5qEAplwC6rdEHOvQQXLjhCncFfC2zHH6FY3XFkL8XbIi YWe6AGwOB0Dptzzs62D/uXfEpIx7dwlB6OcGKLi28GFjJCd46vzIDgnwX7Owr0O4Zf6oW5dM7Bb7 WOIoBhqwLYzsal3h4wxDAUvlgADKVUFfhxMsrIB0+5aTGAQZS5r3QAG2hdHzZ9Se0fy71p40uC9p FfA/tqteB4/cBvZ5in0Cn/5/ancBco3Lw3WpgqgIrrjXzwxJ7a6khQGuP1uaGUkjirRhMHGJZXr5 EVAlDUls9jqueOPgC1xE1zst/DRrxPgMI/dTgI7DNU6xzNv8COjxmrE13YENcByMgavomtdEAS8s J+ACfynQcUhlIZaZ0W4FzckV1Ng1G26A42AMXEafuR0KmLM4ATnsS4GOQ/XvoECy+ltB0CzE2J0m ow1wHIyBy+h84ILkns6/cALy+JcCHYcnjR0QPi2+FSRRIHFuwYo3DkLf1RbAVwGonTgh8X39EKDj 8K5XCqhix0eAVCYau9dksAGOg9E3tQWwVAHfb07w30bUcfiVqaCIHx8FNV8z2qu84Y2D0HMVneVe UwEsYjghfvtQx6HAixSQxY6PgO6vGbvVyhvgOBg9kbvWvKJAmy+PQhYv+qHAxvF/qSCJHW8F3ZXr jsmSwQo4DkbPZXSt/Pm7dfUNV/mlQMeh3hYfhtWHPTATWsx6ChvgOBhRxjrrfkRAM9vgnL8E6Dic ac133/MIiO26Y9Et2ADHToi2tlkHKPzFXAOnu3by67jEluRu/R7+HK87Wo28AY6dEOVrwc9klmhZ Cn/Pq/7Fr+N4rZH18QkLf5EbqFE9uOGNnQ+lK/JGZeWSSJ+k6vfIdf6LXscxrZEer2p+00s9ZrFr JtwAx06IFeAX3kilj9KIemT5+EWv45jXSY+y4qFHB3Bp7ttAxs4hx49qSn8FZGQODrDHzegu/pH3 HImTTWZl3kkAA3pDK8jWh03of6mNVb4NkC1kF2lHJID/Iw2PITapSWMT+0qz9OwfOVkzCC0qFoe6 8k3LphQqg1fZgvdDihgWWKjBKrNzenatedn7Sdplu6xLvA/NeSYMjckS1wY4VjqDhYOEGwbj0GYH OrmdHIJxJy9OSV5883BHpkuNKXXjXgDHSmewumpYCwPxT/7NXLuc6GQOmq553V+pyiXWTBoTlCnz AjjeZAaKSliY4TAO9Gaiydz05I05yhOVojxYD3NhdtCYLEdtgGOlM1i0AcINw3IoC7I3N134cCd5 oFOS5/rhrsyMGmlO5V4Ax0pnsN185sQWjdZ/c+PEXtxZXoWU5Y14uDvTgsY0G5YVcKx0BouEIUZD 3UVfNNbib/Ikndkkr1qjVXkib3Iv9aHGZJlpAxwbn+F6OXBkT3EGO5E3d5SebHI3KQlSE8c93IFF iUY77hVuLFyGifSizFWdwR7s1R7g+fIv5q5W66vVUPpcM7JvUO4FcGx8hos3Muimlx8Bz0ljD/qm d9KLGT1OUboTt7qNJzBjDpbXVsCxExowioRUuwjIuCsYD0u8BZQurdgU4MVy+CyW81UESMzRm4AF cByMhoxCqZgCJHxMQCnVFgUNAI+CIE0B7vHqO6kHNeaZ2lfAcTAaMorFZgrCj8IJAXZ5K6jYw0dB ko3EJ8RXjxocU5zGnOceLIDjYDRkVOzOFHjkKkzA8vxbQRGyqSCLk/BJi4LAVKcxZ0vyK+A4GA05 YKIpcOIbdBVxUYDTfCkoco/wKYsCKQg1zly/4o2D0IDx+psR8XzSN2is8iIAh/kSUOUo8WmLgMys pzFXe2NXwHEwGjI6RzMiyh7xTQHHW0GQ2zcVNLFz7gL2KBAXiAmaJYIVbhx8hhvitCHqrdowoYLh ze8l+Rh/cWLD4jYbNqY/jXzNVcECOA5GQw4J11oVYK84oW82dJJ4pwIvNpTJLwXR8RJqLM4S8Qo4 DkZDDgXXWhXALx1PBJAkW5sC2e5HQRAbynLeCqTG0li8JYMVcByMhhyQtbNk5IgrwwmoihYFVXLf VICio2d+aMdHQWRrprEESwYr4DgYDTngzTIFSLOcgFX2t4Ii6X8qQOlBBUns+ChIvIQamX9VwQI4 DkZDDsiTpgAvDSdgeYuCLO/fVJClYsbF9osCqbw0ljT3YAEcB+NEZnMnCvA/mROSNDqPgiRv/1RQ pYrDJ6KueBRUliEaS7JssAKOg3Ei47k0BU2Ng+UtCvQFmgqaVrDYyUVB5z3UWCwdrHjjIJzAcRox VPUNVrcI8IsRu1x3eeJeAqQkkVDK3IAFbhx8EzdJl0J+HiwmtM2GbvZW5K9OGhd55N/8QdodiaXO HVgAx8E4kTOcqwpobkzoqw2lDHsUeLGh1DlvBboFugezHF4Bx8E4kQsytSrgBUcv6lYbSgn6KIja TMTVhmx7ZizdEuIKOA7GB9mZAuY6zghS5jwSqrR6UwKqkpr4oR0fCVW6IInVWTbYEMfBOaE7MmUV Dcz4nIEatr41FJzcPbAmPQiU+/WtoUk3JLF6ywcb4jg4Ae3wG1VEMg189zgDbvnQYANrVDui33lr YOKdsc5KeUMcB2fiSvDbwx2iwfPt5wz45UuDDqxBriQ+adGg1YHEahI2wHFQJi4EvwPsoRKqbKJH Mdu+JOhA3An6AJ+wSEjxmrFGywsb4jg4U5BeFtKcaWD5xxkwzJcGHVidXAd83KIh92vGapdiAxwH ZfLiDrQWkCsSWANzRv90pA0sXXJCaWLMR0LVOjGJhLkNK+I4OJNTdxQsSzWwE8CM4L4dqQNLlcyI T100yC7IJswHYsMbByMuaENxi6uSTUHE/nCG//ajDixF3gZ88lsBa7QZ623IFXEcnJHrwO8GTNXA 1pAzwrchdWDJWrFleRQfDcFfM9Y23bAijoMzciX43bGzqoG1TriYXIF+SNBxJWnZijZukRAlP0us fW7DAjgOxsh1oBfw8EahArwoSSbALacCG1eilu5oZRcFWcpFic1ZglwBx8EYuQz8DlLvUkGDezgB bvlQoONQfnct3euioLRrxjYL5xVwHIwxaD+EStgUVHiHE+CVDwU6rjipmIuT+vlR0NI1YwsmYMEb B2H0YgvUUbmrABSkmRMaM8shQMeh+mbTgE9+V62lS7UosQXLCSvgOBijE1uglKymAEW5TOhfRrRx 6CDZOOET3wq4LTPOemXFGwch/l1cUbAzKgCbwwlQ+uVDHYfyn70jPkvhrlVCFPq5AQvcOPiwMeIJ fpQf7pAJ4dOFOg7THfnz2rrUxGpRY8uWDFbAcTCyqvWNxxlNAaVyQgTlqUDH4QQbFaS1fatFCgSJ rcx7sACOgzFwGV0dzd+9q2lwX8opwIbBxCWW6eVHQJU0JLHZ67jijYMvcBFd77Tw06wR4zOM3E8B Og7XOMUyb/MjoMdrxtZ0BzbAcTAGrqJrXhMFvLCcgAv8pUDHIZWFWGZGuxU0J1dQY9dsuAGOgzFw GX3mdihgzuIE5LAvBToO1b+DAsnqbwVBsxBjd5qMNsBxMAYuo/OBC5J7Ov/CCcjjXwp0HJ40dkD4 tPhWkESBxLkFK944CH1XWwBfBaB24oTE9/VDgI7Du14poIodHwFSmWjsXpPBBjgORt/UFsBSBXy/ OcF/G1HH4VemgiJ+fBTUfM1or/KGNw5Cz1V0lntNBbCI4YT47UMdhwIvUkAWOz4Cur9m7FYrb4Dj YPRE7lrzigJtvjwKWbzohwIbx/+lgiR2vBV0V647JksGK+A4GD2X0bXy5+/W1Tdc5ZcCHYd6W3wY Vh/2wExoMespbIDjYEQZ66z7EQHNbINz/hKg43CmNd99zyMgtuuORbdgAxw7IdraZh2g8BdzDZzu 2smv4xJbkrv1e/hzvO5oNfIGOHZClK8FP5NZomUp/D2v+he/juO1RtbHJyz8RW6gRvXghjd2PpSu yBuVlUsifZKq3yPX+S96Hce0Rnq8qvlNL/WYxa6ZcAMcOyFWgF94I5U+SiPqkeXjF72OY14nPcqK hx4dwKW5bwMZO4ccP6op/RWQkTk4wB4nY2xSRNYsnaF9jPG3n3+16+evy/EPH/tfP3+HjH9Cxn/s Dv59eXf96/rjT3f9+x/sktivVqawBF3/57xscnTHbSg6r1XUEqx/eRlBBllBzx4DdO9/EN5LypbM WdCAv2Y98liWKPJSFzp1qsQUhXn2D16KgcEdeBp4n5/2n59/c2kT3ayupenPZQvFkfyfSztfJN+l 2pvfHd7+uY/tSzS+3cPOMaut9yPRXkOqHgL+zl22HHmzxjXk6SkBpQfWzR6G1r4yLWBu6FUC3ov4 cZRAUvuCrWnzkDWJkCt1I1Py3tVuWnrQm6cE1ExMNy2YyUfQqRurATezb6G7dY+6VfCPpwTU1OWb PRw99FowYGxoq0mp+gsMfXhKQKmNK6PtqzzorImqfy8bulrnm1vz/3hKQA3Mq9V6mKE778BMTMSF 5lZAWFpjM/ThKQE1oI5o9wd9WcDY0Hmp9lc3fjwloAZm3Gq939BN360BUHcvOlUfiqBIbkcfnhJQ A9KA9nQ0VDMD7g19cdXF1j4cfXhKQI3Of6ByWuiEgEI/R8+bCipxQ1p+0JunBJTa+Jw62AiARm3S gMqvW+hpU8LYRtXTUQJJS+0wezgZn8eAuZEp+MZMrsANfXhKQHkVr2gBC500BWZjIi00U8OHu1Qe 9OYpAaVtteg+6ohwDUMnrYga0L35GLq1Z3LFrhj68JSAUsEwzO6OtgKqAWNDcwhgiRpzVZDDUQJJ bR9v8kPOQwMGh5hFLlxu4tKfknp4SkC5ymqXD14qQvWlGmCibKE5O3Um4GheQU5PCSgVkDj9lnyu RcBlAX1DJ6y6T9/vRd4cJZBcmjactZFR1jXApOwiX1hu777fRj4cJZBUdnezh5ORkwyYL9kuSq/c juT14/SUgOo6XWlaNVyrhVZR86sTQLtf9OSiM5rXuMqD3jwloNQusFELDI3rrwFagTf04KpZP/oq qKenBJROQsPs4WjoQgaMDd1xiJCR+Oh1ioenBFSHOFVb544HnbVw6lhYNnRDarQx/SwXevOUgNIJ sha1OwcwoNEgNECzP71o3u/GMtJXQT09JaC6Tgtm9wd9WcDY0JxwWiW6jAe9eUpAqZ1hDw6uQKMZ a4AWmg2dcYyNz569gpyeElC9aarQxlgINCSEBjRd0otOSLuWUFH7tfL68JSA6lWLudpQQQudLKBv 6AvrrZPbcs8HvXlKQHV8jtrQboZOXN2Nr3vQ/caqK0tqm+sYD08JqF5YKdUejsa1ZsDc0Fxvbdzx dYqHowRS1+amhbJjSYucEDD0qzYyZT+3o7WHvDlKIHWke//tkPawb4h59Ufyv2AjUzwh/0A+HSWA emKN0p9u4Ml+dOPCbmBwL16ZYtXj9JPA6YllVV+XFxdl90aNyQ+XMqxM7nHqC7w5SgD5lnCDCB7s FNhBNI6HDNVRBp62xR8/CaCG6q22VlbdeYIvD+gbmOW5UCRU10sfTwmodnPxmojJ0J0tGZmadzTX zFZevWt9PCWg2uSGqz0M3XggCMD5LPTFVWfPaJJPRwkkv+V+50lODMjMqIcMaGG/rT4WfTwloBok idoqpwxdmfYoZelFN6w5symuZD79JIBa501Uexi48KYiYGzgiTWzilafiT6OEkjeBLwlkJwZUFls FhnM3KpfFyfvjhJIDXpVbdVSRs4siWh0aSOzVWVWjeId/OMpAdUqlaHafaEvDxgbuiHvMhtA8ZHo 4ykB1Srbi2qGbOjE9gNRUTY0u2C+8CzdD/H0lIBybeNKR9EXmyakEHroQhesOrFKF+/gH08JqIbp o/8Oq7xEJw/oGzpjvYnVFNfR0bunBFTLFCCqKhNtFYMQKJCdeUOzwULR4Tpa+fh4SkC1RNmk9jD0 pKxCwP0WU9yTVJCAxSeij58EkOt1V+8EJwbcVIYOts6dEnPbJ6KPpwRUQ5qOX505LkMPClgMJWlH g8wSkn0i+jhKIOloNMzuRu5U3QgYG7kjna/Z7aaTfDpKIPk458MdyXnUX0x/VTvJQ0biXeyJ2Qei j6cEVMXwQfs2tOoeBug929CVi7b7WC2nP54SUHXoZ6ndKMKJvjyg72iuujRWkUXeHSWQ1MakOLS6 Glm/piFAS9pGxmW5OMrl5Gd4OkogVYwetO9EspZdD7hvJ/8Nr1oerzJo6U/SQ/vnr5///P73x+WO Fxm++0RJeFfFbKK25tl6d2KAFuj17n/9XIASuETvpGIqndHvKtQ7/eI/6riceEr6MhyaAD71fwpU 2LCXIVHqxXBViPlixozBv/uWoiHrc1Y+XTGfnhJQM/Od1WZURSeoEAbMDY0bd3ekwe0a4+MogTQz E6UmXkQjJ03SgUx8yVThN1P47ulBb54SUDMxU2pmfQF6sh+N2zPF0Lxyd+F+uMj4eEpAzYuJUDHD GBoqhAFjQzeuOlV/gaEPTwmoeTFlKiavhc7ajyZK1YvmlZu2464zPp4SUPrOqsuomBcNjRFRA7QZ pBfNrZiTa3eh8fGUgNJDmGb3B31ZwNjQHB8m5ca804PePCWgBlqX2k1/DA0logHoZC+auzxZ9KcL jY+nBNRAw6U9Ha0CNTHg3tDsIbPY2n1cOz0loLwkeIEwdEJAoZ+j543LMhM3xIXGx1MCanR+jta9 5GgoEQ2o/LqF5gHikmFbnHw4SiCNxjNQezgZn8eAuZEHFj0mMzAv9OEpATUaM0cbTHnQSVNgNibS QjM1Rs++Kwu9eUpAeSP0tgh0YrXVxon0X2iOVaN23xVDH54SUKNwYFG7O9oK6Oy8tAtdkXssUWOu CnI4SiCNwiGrXWxeRsYUNgeHskXmwIYpQZ9PST08JaBcJ7lqUvR1c3ack6PkQmcsujMBh0uNj6cE lKo7nH7Dti30ZQF9Q7NhU+Jivxd5c5RAGhhC1cZZGxllXQMwk75kzoK9+34b+XCUQFIV3c0eTh7U aPOmZHOyXZReuR3J68fpKQHl2t8nAUNDWN4XdeZCc8zsGc1rXOVBb54SUGoX2KgFhu5UwzrhpA09 uGrWj74K6ukpAaVz1jB7OLpRw2vA2NAUoRBr+Oh1ioenBJTaGCt0ViwPGnOHjnxlQzekRhvTz3Kh N08JKJ9pfcIFunJauguHp4Xm/baBtq+CenpKQHUV72b3B31ZwNjQhaumJEVnWujNUwJKbQyebXAM BbpwMr0rJ6mFzjjGxmfPXkFOTwmopR2hJwwNCaEBKr3Li05IOxsN+7Xy+vCUgOqq/02WtgedLKBv 6AvrtQmu3fNBb54SUB2foza0m6ETV3fj6x50v7HqypLa5jrGw1MCqhdWSrWHo3GtGTA3NNdbG3d8 neLhKIHUM+ehjiUtckKA6vy2kQG2LGntIW+OEkgd6d5/e+IA8OfnhphXfyT/CzYyxRPyD+TTUQKo J9Yo/ekGnuxHNy7sBgb34pUpVj1OPwmcnlhW9XV5cVF2b9SY/HApw8rkHqe+wJujBJBvCTeI4MFO gR1E43jIUB1l4Glb/PGTAGqo3mprZeVMqwGXB/QNzPJcKBKq66WPpwRUu7l4TcRk6M6WjEzNO5pr Ziuv3rU+nhJQbXLD1R6GbjwQBOB8FvriqrNnNMmnowSS33K/8yQnBmRm1EMGtLDfVh+LPp4SUA2S RG2VU4auTHuUsvSiG9ac2RRXMp9+EkCt8yaqPQxceFMRMDbwxJpZRavPRB9HCSRvAt4SSM4MqCw2 iwxmbtWvi5N3RwmkBr2qtmopI2eWRDS6tJHZqjKrRvEO/vGUgGqVylDtvtCXB4wN3ZB3mQ2g+Ej0 8ZT/kV3uupbkNhTN+yvqD6z3I57MgA0MOpjAcOZJDNDAjAP/vrk3pTrSIRpgX95LrlKpJHLToWph e1HNkAwd2X4gKvKBZhdMATa39RHvSHGopW2W0lF0YNOEFEIP3eiMVUdW6bw6+FekOFTF9NGebpWX 6LgS2oFOWG9kNcV1XOgzUhyqJgoQVZWRvopBCBTIznSg2WCh6HAdrXx8RYpD1UjZpH439KCsQsL8 FFPck5hxAPOaiL7ixIGWXl/qneDIhElluMDWuWPk2V4T0VekOFTFMe2PzhzB0J0CFkNJPNEgs4Sk NRF9BYoj6WjUzW9GblTdSOgHueE4h9HsppN8B4ojrXFuDXckp14eTH9FO8lLxsEL7IlpDURfkeJQ BcMH/Wlo1T1M0Ht2oAsXbfex2Jn+ihSHKl1fS/1KEU50WAntRHPVubKKbPIZKI6kPibFrtXVyPo2 FQla0g4yLkvgKJfi+oZ3oDhSwehBf0aSteyuhDkX+Q9ElfxG5U5P/4v60f78/cdvz39+LLmzigyf faPEPatgNlFfz9l+dmSCFuj97F9//OWX/47nl5/P/YCfv/z9R3z++sTn3/qbqf3uf3oHnr89//hn eP716uNBcZUbH/RZ8M8fgaTVzWKCtIlYGb6wYNsiXhSSrdvKsMSCGw0/psAl6tIzt5H7b1tvFqrS 0FekONRI3A31p6Hj5HZpwjzQPC5mcTaJviPFoQaql/pQ4RuNjtXHOlaG5iE3uyXJV6Q41Ij8Q0ms RkAPHt0+17kyNK+mWZQBQ1+R4lAj8HXUH3mjgyXYsTE064lZXMKNPiLFoUbgRyiZ5xfozioxwjqB RFsVNIuKa+grUhxKn4mjUzBdGhqyRRMiT9JGs3SbLbMt9BUpDtUnq4P67UVHSxj5g2bDMYvettFH pDhURwVTv2qVMHSlCNHPXQ50/bTKuslXoDhSRwWjPxdZqyQStHkcZHZ2s3ue+YoUh1IfZWJVE0On zIR2oK1X2qyb54s+IsWhOn5Q34ok0IjUBCujC20qymwt++xdkeJQHcunP150sIR5oKn9zNY6X/QR KQ7VK5+52hHQ+CqaULmEjaZiNVvbPntXpDhUL1SZq4kCDeGiCY0bt9HU2WZr3zfmihSH6oX1bLV+ Q0dLwOfe6L4HhDraCz7ixIF6pp5fagVguxGjc5jbYI4zZuvca74ixaFUVGEIWRJL0WFyShmDc+dC 2wxmtgVrxF+R4lA9cXKqqIYbHS2hHei0p8cW+ws+4sSBerRWaFIW4MFhcIndDeawa7alveYrUhyq o9PQnwvdOcIONKYPunI7aFF2DH1FikOpH1VUrbHB0KmnZwaK0o3u8dm2lfqij0hxqIaurr4NO0Cj 7WuCjUMbPSjgaVtdR++OFIfSoWyaP150sIT5QWMg3ba18qKPSHEo9SFg12AJtP6EhKSv9UGntIUS V2roK1IcqnXK7qorWOhCXT4zJ61XXY5n2zb2qq9IcSj1g/m9bnS0hJE/6NqfbduML/qIFIdqjQNO VWG10JkT0Cy6yA+652fbvoTpV6Q4lKpHjGXqz4VOnNum6vQDzd5iFpXO0FekOJT6GCVVmNYXjVlT E9oHbcOp2Z7yiz4ixaGaymb4U4+/oaNGakLT1/qgrZjS9k2+AsWRmip488dLDpYwD3KxhsitLulF H5HiUOonrbAN38jQEMaaoEPBgW5E0/Z9za9AcaSmw5SewQZpD39qAUb80Jf6gPt4tu3N1nwHigOp H8zvGxwtfhzgyWZIu+TpHSeOoz/gPuqvs3EH76um4fYubov52bZv8BUoDtRwPulbq9Et61oZsYPh IKf5bNuHSY+vSHGoBjWrfmYzIzquBHS7jS712RY9eqHPSHEoPSKo13oWs6Eb6zlOVD7QrTzbjrBW fUeKQ+mVwZarPw1d2YWQMA+0VTqAo9X/rzhxIPVxRtb0STDOEK5uPcCT20E70tjoM1IcSisI+rmW KpxyRRf2e9QyXIOFxtZti75B9B0pDqUldZo/NjqshHmg87PMKGODzzhxIPVRQLRyFwNnFhiU9nqA S3u2HdVu91ekOJQ2rwZ/sAYqWuUUEyqL5Ea39Gw72jp4d6Q4lPrB/L7RcSWMAz3is+1YNekrUhxK uzmKk8qGbOjI4oXmXw707M+2Y6xV35HiUCpv0BfVn4YO7JtQQ+GDHiwcy8616jtSHEr9qH4P7O1E p5WA5r/RlKVmZ0gbfUaKQ6neQ/NSYZnoqx5sTFCJcqDLeLad0aTHV6Q4VI28AeqPjQ4rYR5o9hSz M6WNPiPFodSH3lOlXQw9qAchxeuBHkTTzrxWfUeKQ1XTyDp4QLMqWncOCTFQ1G705IbQzmLS4ytS HEr9YH7f6LgSxgeN/X1tzRt9RopD6eyVdR7RSSwbWpWEtj6MakXF+Ubn8ry2rVXfkeJQOop286eh VVAxQe/a/KDLfF67ZM1XpDiU+hG+xsVi6LgS+oFu9Xnt2OQzUBypaIWFry9n5EIpoJuob/UhD+6E 2bnO3h0pDlUaK4P609Cqj7slzI3+A1Elv1EZGqnjv6ipf/7+47dHBdSDf3zmjRD3DPWR2FXc5/XM xIRC4ctn/vojPPi3pSokVMe4VBQ4QC7PanJUwehq6+/cH4VBygZqN2wPaPoSBkyR3/ADtISM+avb iqr6JfDwqIxMEF8RS8bxWhseInqMWQwyhr4ixaFG4i4UXAqi4+Q29c532uiM82G2FNMhX5HiUOrj o5bIT2xofPU+eAg2uqLHmMWMtNFHpDjUiPxDSTyYQA/dOk2YPLob3bghtKXtVV+R4lAj8HXUH3mj gyXYcTH0IJq29P6ij0hxKPXxEUrm6QW6syyMwG+y0Hq8nm0xfhn6ihSH0mfi6JTCygU0BklNiDxJ Gx3RdM2W2Rb6ihSHUj+Y3150tISRP+hMNC00zUYfkeJQ+n0L/KpNztB6c5Cg/eJAV24Ibd3kK1Ac SQ/lMH8uspZFJGR9qw+Z+tdszWvGvSPFodSPVf3GqmPolJnQDvQgehh6vugjUhyq4wf1uxYSQyNS E7Rd1BedAnqu2Vr22bsixaE6lk9/vOhgCfNAR7QYs7XOF31EikP1ymeWwZsANL6KJlQuYaMztsJs bfvsXZHiUB0npT4F58bQkTV5NG7cRlf0GLMYdg19RYpD9cJ6pn6fGx0tYfQPGueZpo72go84caCe KeIrbqiB7UYMXOH8ggdqqdk695qvSHGonjj81LDmUwgtjJdjcFRZ6BwANdtCXT3rihSH6qh/6kcO T4aOltAONO4gTYv9BR9x4kA9shlWVHADD86AAyX+A86opGZb2mu+IsWheuBgWtF3DN2pNMbkILvR ldtBi7Jj6CtSHEp9yMqKbrnR0J0TX+uD7uhaZlupL/qIFIdq6OrqF0pjoNH2NSFSPG/0wKEz2+o6 enekOJROYtP88aKDJcwPugR2LdrWyos+IsWh1Idi1ZGvLLT+hASdSw50YteixUoNfUWKQ+nUi3Gp 6goWunCempm6fKNZOcy2sVd9RYpDqR/Mx8hn6GgJGAo3uvZn2zbjiz4ixaFa40xTVVgtdP4/2WWP A0tuA+F8TzEnWOj/J3ZuwHDgEzhbBt77By4WpR72EAv0W75X+pqjlsgih56NydWhZ/3c50zH7r2V ElDwj7Mz3gddOKrBYCaHZm+xp1a6a02dUgIKcdZ4cZo0dKlcML5oLUL3OUt90E4pATXg7TXeOP6G zlBiAbx9/6KtmPI5L/kllEAa8NcWr4ecbMF25GYNkVvdyoN2SgkoxAUVdug3MrQaYyzQmeKLHkTz Oe81fwklkEbBjxqfodZe4705NOx1hgQDz/W5zzks57dQAghxsnhecDb9cuDNZsjnsadvnQQO/kfv I/66GnfxvmKZ3t7DHbl+7nNe8EsoATT0fDK2VoMtm6iMuoPJkcv+3OdcZj1+lBJQQ90s4spmRnQ+ C7TbXXTrn/vUHn3QXikBhSOi9RpnsRp6sJ7riaoOPdrnPlc6Wb+VElC4MrrliLehO7uQLtgObZVO wdnq/49OAgixnpEzfxKsZ0ivbnfgze3gc5V10V4pAYUKov0cpUpPOdCN/V5rmV6Dg9atu0/tG0S/ lRJQKKnb4nXR6SzYDl0/57HaumCvkwBCrAUElbsZuLLAaGnvDtzG5z5Xt9v9o5SAQvMaGi/WQKBh p7igs0he9Cif+1zjHLy3UgIKcbJ4XnQ+C5ZDr/y5z3Vq0o9SAgrdXIsTbEM1dGbx0ubfHHrPz32u dbJ+KyWgYG+0LyLehk7sm+qG0he9WDjOc5+s30oJKMQZ8Uzs7USXs0Cb/0XTltpzp3LRXikBBb+n zQvGsjCGHxxcAIvi0G197nNnsx4/SgmonnkDEK+LTmfBdmj2FHvuUi7aKyWgEKvfg9Nuhl70g2rF u0Mvovnc9WT9VkpAdfPIGDzUswKNndMFOdHUXvTmhvC5m1mPH6UEFOJk8bzofBasL1r393n2etFe KQGF2atiHsEkVg0NJ4HWp6Nagzm/6No+z3OcrN9KCSiMotPibWgYKi7AXdtfdNuf53lszY9SAgpx 1hi63Aydz4Lp0KN/nue6ZC+UQGqosBrjxxm50QpgE/GrvuTFnbDnPmfvrZSAaoOVAfE2NPzxtAX7 ov+nqlYfVVWPNPWPjKV///eP/3xgoD76H9/5Rkh4B2JdOGHu63ln4YJG48t3/gs/CHNjOq7n3//4 Jx2rOqmpU1MDd+kL2uf0OpphbW7n37lNYKqjTfOaYwf4649/u6z1H7XafOk/q8/P6usZfAsN0tQ6 156PsMv3I6z1oJ1SAmrpLMl4TO/isUBP8/N9eXD8mfwRSiAtLZBN/fGdxTLv3LP3RmYfPsd95Mdq O6UE1Cr8nK/5QJsrFmyX9MjfS9r6g3ZKCailtZfxeLLWIjS1OH/RnVnn5Qrij1ICCnG1V5njVMOo bgMLusu65dN2vmX8RykBtWDj8KqRWIINrVV5Lv79q2qdMp7Gg3ZKCSjE+s6R6LS+Y83crBwXXa4x +XbjH6UEFOJqr7KJcm92qbm5b68qvno5qZ/5wwklgFbi1/UDk1pR6K0KGDiNYy2do3oJJYBW4okc +Qyqe7Nl46/zN+Nldji/XOBLKAGEuNqL5p2Y9Drjr/s3Y3VQOhvQqLU7MDmdBM5KvPej0KyQq5Zx JVaBy5202uvl4l9CCSBskdYqxD0fcMVAgS+RXcI8D3P6weOlk8BB3Ow98yasDmZpU/1yOxPuZzcO 1+kkcBAnfU/laPZMjkv90JdLXzZ5TX4GxyOUAMLF0X9AbLZnq6+HvlD2OEoOBpkT6Smgb6EEEOJm L3oGUp1RoG8u48KMUz3DwQE7oQTQXNz7oTPPGUeXJTJcxjy+g83kzgQvnQQOaqqeFcRnRkLn1jyq b1I62eqUezf6gJ1QAmhOnu7Rrmsf9OKr+hZl12KwTsxSL9gJJYAQ63VE8071DLnLEnEdSg+C+c9E j3BmXCeUADp+Y/Tj4LfaA+pdf7KJa2QFjzvBvIQSQIi7vWjcjNWAwDC47sQLN9hBxq0TL50Ezhys 0YjNUG44y0y9a002ffY1zhck+CWUAIKRZDvRL3HAOv1B7xoTb3Jn1n1crJNJwCBmK8HPuPnqNAI7 W7/lZzbm27nNxwC8hRJAiKu96Ayflc0eeteT6Fl61axHOWXipZPAoVVHPM8IA33f1LuWpOcW4HKO xeE6nQQOYv2CQ33x4VbNY/iOpPceXJ6KfrFOJgFzhhbE24oPbNKm3vUjKzusmf3swksmAYNYL8hA 5egHOy0N144G+2cbdijWBTuhBBDmN73SiPvNl21m+nZk1azRa/ZRL9gJJYAQN3vRvBmzz0yWrAue eiUa+4YetwN2QgkgxFo2B8yRZZytz0wW2QsezJgWqNezxy+hBNAs/N6IT0PK1mcWu8IFd8240tf3 Wy1fQgmgWfg/iOfNmH1mcdkFN/1slR5TjfYBO6EEEGJNfabbkJI1Gp0kvmAW4EpH0c6I+RZKAM1M bzPT7UjJGs3mp7ngwox5jNu6XKeTwJnZpr10GxJsXaZez9LlsrDXWniY9wU7oQQQYj3P+MP6CITL EhkuYbbQygbdun27t1ACaCbeQMR1H3C3RPY3487jUNlIW8sX7IQSQDOxYiAeN2NtNPihWl8eMIiF 7a49+TqZBAxirW+IrY3gE2ub2YnV8GKn5lvGOHeE4JdQAmhsFuRZTiOBXhvNtpnuASuXVSiVi3Uy CZix2T8Q95uvthmcfP3jYm0fWCbqmhfshBJAiPUNiK2P4NprBjsznwesW1tY5etxa2+hBNBY3BFU D2sk0OuOQb9dxlUzzpsH7lTMt1ACCLF+Q8T9ZqzfeOvk4MCacWZBrufWvYUSQGPxzCG2ZoJmoGdy F57QC2YLzfSXtZqZeAslgMbiLUFPsWYCvdo66Ne3/mgxA5hlopbLdToJHLQ9vdaI+01YfeiuLAKX q9RaefH2xTqZBMyYLEOI1zxY9c27+pbUliaaWYPKnhfshBJAY7JuwrpYJ4HpmJaIa0ncgExzqTea 3JdOAgcmSAs9YpvhoNdeuZvvSFoiUbcXwfNynU4CZwz2EcQzH66OUrv5htSGnrTEIlH6umAnlABC zD4yzgwHG7oskfmtPuYjkl29dk7ESygBBFus3R+xzXDQ60/Z3TckLb2JJ0mLxeU6nQTO6NZHxhnh oFc7A71rSE0PWWLb0FJxuE4ngYOYbWSeEQ5zybI8XD9qBd9Moekk+xJJgIxmLWSe8Q36bkl8e5EC CePhpN/Vo5rhnvLSP7RX/f1f1eaP/qdanEpsf7npy/15XR3N+KbfF7vnX1zQZvmmxt+zt57Bzdq0 13rQTikBdX5EXzSDhi61fH+lofXybJ70PetDdkIJJMTq0buaESOrG+ECN6VuziSbZ3KPux8vpQTU Kpwqujbji9Y6hwVuTt28QZtta58x6kcpAYW426vGk3XRXJafVLVDqaFY5wUX7ZQSUIirvWpvQyuC C9ywunmLNgvVLuWgX0oJqJVtzNSLd9G9c4EbV20rFj3uTuNBO6UEFGIOmmpMLrpqLvDtLmu9S38u uty1+4N2SgkoxNVetbvGW80J9W5m3RwiFodLTZ129CWUAFrJhk2YyX3AfVPvhlbdAfXh4yR+wE4o AbQST+RQc3LAFYngr93UujgHLhas1fsFO6EE0Eq8RUNnJIIXr/NKfm5di9tLn7vs0L11Ejgr8d4P 9SaHy3k0+cFVu2j6cy6mXdYFO6EEELZIa9VQb3LAnEezH1ztPMz5f9KrYDlyGoje8xU6zlCVQpJl yaIoDiwc4MCBzI3l4CTOMmxkp5yBrf0RvpfX3dJYXhWwVVxImn15brVa/frlcmTeCpcaHsROvhNK wmxHTdZO4e054T5XI/NWuNTwICYp87ScMG8QN2pY+AovLwmBn0kYsvXZAVNDhIdD/+C7YluDuFGb pVOIucuCiVyOrhBXwNQQIXbyoWxbg7hRm7VTiC1nrOlJB28LcQVMDRFiqr13xbZ6caM2i6cQc/t6 FpO8me9xqeHBTGXJc8W1enGjXS1SA5tA70uhM3EFTA0R4l4+5EvCtmN8JVHyLDzPiZDXjz0wNUSI 6Tn6PrtNSPkgiVQKRY1Ar5Vb2eRu2wFTQwQlpwGC2IZMzG60q/UpsLH0vOr6PDP3wNQQIe7lQ75k TH4UC0OlTvzgPCuIL3Nih0sND2Ka0b7PdjM6dknAV9JEDYZBM/h8g0y8A6aGCLspy4nPhhN48nXA V8LEL5nNq+99oa1gqaFBzFLis90EnnwoFuRuGz/Bcb686fq8AOyBqSFC3MmHxG7CHJDYA19pEu8s PS+63uYxscOlhgf7ICtJyG4TeHKjwFeSFNgEcgtTW2TeCpcaHsR0g4j7ki+ZUVimSpHo3Ytj87ov tBUsNTSIyWoiFqsJr0hLLvCVHsnY4ZnZ5yrsYKmhQUwPBIZTnGa0ojK+liPP+um8NMVQiCtgaohC x08acV/yZZkJtRzJNBP/0/uuEFfA1BAhdvKhUDJmnQm1aaWFV/O0lXbLxBUwNUSIaWz6mM1mNKIz oTat3nPGvAL1Xa7xDpgaomD5vhFnQTKiM0PtWumd0arAd1em5Q6YGqJg+RfEoWTMOjPUtpXeGYh5 x6RFOxNXwNQQIWa3qYsgaREachIbMQ/gjjcKFzPxDpgaomB4twm6KJIWoYm1bfWWM+Y2dkPhrXCp 4QlG7KYugoS1zjC+sq0y2LvOcjPHQlwBU0OEmC2nyToC4CCJVL7Vs4R2LNCuz951B0wNUdDiXQ2/ RybuJZHNu7I3BTELqXOmEFfA1BAFzRMDsS8Zk9DgoDRfrsRgtCx37ppvBUsNDWKab4hFRnDFJDNR 8zQstIHytd7nN8LEO2BqiHzkgRxsFhLgSWiieLorMfHyFNK20Faw1ND4yPqBuC/5ksyg8+lHoZU6 8JjohlCIK2BqiBDTFxCLjuDZUwbRcD5XYiqt5Snf5W1tD0wNkR+4IpgeIiTAU8WAj1XGHWVsIjdc nph7YGqIENMdIu5LxnTHkZxDRUwZGx7IXX51e2BqiPzAPYdYxARiQD0ZLXdoIWYJNbxfdp0sE3tg aoj8wK8EmiJiAjytdcAP2/yhYQZiHhOdLbwVLjU8kD161oj7kjDtobHjIVB4ibXr+OHFQlvBUkPj A48hxEPItLQ3x66WJDdQooZnkI2hEFfA1BD5wHMTq4soCZaOIIlUksQFMLxc0otm3h0uNTxYgmjQ IxYPBzxpZXS1ItGIxNwemDgU3gqXGh7vWUcQB5N5yUpFVwuS89RpmoeE7YdCXAFTQ4SYdcRnD4c1 dJBEwjZ9ZI/Q8vRc7ogdMDVEWItJ/RGLhwOejhL7WpBo9GruJBoWhbfCpYbH96IjPls44GmdAb4S JEdNplk2aFRk3gqXGh7ELCMhWzj4kkHyqPTIWdwZkeqc7A6UGhLvREJCtm/A95LEpkVEqO7e/HTD zcn7LrWqwfZkBvpBWrVON3c3355uvjydnDLq9IS/62jtQrfcll80vU/SGtNhFGr82Smx5RBnW/5L iaM6WuPfH8Bz+nBz+Hl6WP6c1o/H0++UUZ6DPkvCLeCOhwL/OH23/wMkFUpSluzcoG7zT62MQ408 JGDAsIsYspQTn5nzoBS+1rofvhEi/y+nu1IZjQ0w4sHz+XCSjmgoS42S4Ty/HO4mZDc+q+cjGvQw HW9RkcOf0/OrWp7Ueuzof8oJ1LhOaprH+x30Ud1/VMdfTz8ilVta6vAEcG7+hg5b4c53b+5++Kqp gqbTI/nyC/6XgaGzeO4GV2qizoX4p+Nb0jS429vyS83gyPDF6+n9dvogp5/WdVnVvFzOT2ecdTg8 jJfzMqu/jre2xyA+3Msp1+X9NKuHZZ6nBwK8qudpfFSXpT76gA7G0Zn/+iVrcp1xfBCkdMQoP4zz o3oaz89/rBMT/FdN8B6cCaatCRf3swtzpfmkMPGaLnUIpZsztZzpOr1exvVSqmIPD0f4gMMVwtH8 +KpepvnxjO4wh/mdQmHPs9Tn9IV8x20X0HNXvKzLO7Cj2WY1qvtPy6wexllN5fvTo6L+KWTDRhYl aSaZpw/06WWeqIMvv9GfU56G8/SH83xGcuUbR3eg2yxZSg36rRxempeIX5EncvqMFnbGYPLBrP6/ 69poPrkuu508X9fjeBnVZR3n13TEvD6cOdVPLy7SxdG1jXRZVNv7aX9BWyNYId5K/7Quct0oxft5 +TCrl+U8X1S+b3on+DeqNyezo22vinN9mlb19pAv5zw/f1Tn9LKslxG0T7jD54Xa6AXzid/c2yOT fn+6+XsASgyBlwplbmRzdHJlYW0NZW5kb2JqDTM0IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1Bh cmVudCA1MSAwIFIgDS9SZXNvdXJjZXMgMzUgMCBSIA0vQ29udGVudHMgMzYgMCBSIA0vUm90YXRl IDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0g DT4+IA1lbmRvYmoNMzUgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8 PCAvVFQ0IDgzIDAgUiAvVFQ2IDg3IDAgUiAvVFQ3IDQzIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAv R1MxIDk1IDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzOCA2OCAwIFIgPj4gDT4+IA1lbmRvYmoN MzYgMCBvYmoNPDwgL0xlbmd0aCAxNTgzMCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFt DQpIiZRXzc4kqRG891P02ZJZfpK/sw8+Wz74AVZey9Kykj3vLzkyga7KogUejQTqb8iIKEiSyF/+ 8qO8f/3xtqZYh7H69HbvH7/+8frlr39373/9eLn3v9+vQIZCeWdKpmKOxqX3n7OrptL7v/98/fYK 1cSabwt80QuITMl5g8AKolIQrbFhFxLxp5oOmFVhJmtCTBudKZoU4h7TBcYMAzJbk+suIkdTIx0g 0x2yOOMcbVQiwquAZEI9UHiSCBoh1ZkYwyGk6JBksgt7WcGrEGedqdXvaUL/+PKJycbFQwzpEOdN cO4QIl9D7hOTDWW3/5zodYz3JscDT5SUo/iJyaYeQlLQISEYe7gsKUtI/YQU433df02OOoaCibbu eYpIi+ETU0xKu1LAMaIt5hkTg+TSNqbGR0wxtewqClZbOZ30OZ1Exqf9vlWbHjHVkD/EOMm29Dme TCaWXRXhktO1ffY6V5zZvlhxmVIxJRrr98WIrxrG/Dmfag3iDjHlERNNoH294eTkmHk+3lqT/L7k cKbpmGhyOcREybd5sb1jmH2Z4qRRIck4uy9TfP4cMsuU986Esi85fP46JplIpxjZgfLRFpzJdl9z +Fx0DNaWfQHhPVYx5I2jQ4wkAWrzDMnmsM+96F4R0Rva3QIfs8lhX19YswJNHrXg/4tJn5hibNzX F2ZQMTkY78pOfS6G6gk1adQSTIq7auJLwWu/XVEDznO3r1y7Bl+txsddVQkW1s0d6sHjK4KFm8tx oyA42Lmtpwke7s0d7vrjTILHqrxFDfBv4XC3H9kRAiydO6W2zsJAMHX5cLcf1yEQfF7YxYTojK2H 2/8QEuHs8s9d/pBg7cIpJjxikkl79/CsZZBl6uHhfJTMkOHs/OFyP0ozyjLEHGL0CxAKnB3t7vby zoQKY3ewKM/nLNQMq3OKcSqGLJwdHXbt8TyThbWzByv0sAHkgqHyc3aDHKwdnWK0rSEfpG/7GftE vsivvbXTNo3QgXo68DzsIKEpJXvoih62k9CnxnRovR72lgjW7uBRnjaa0Ltae4h52HVC8+rSoS96 tAWEfjZ4v/+eR/dB6GiTPfA8mhxCk5vTqTHSzRQVzqefa9qQ08bub/ajNSQ0umGf1M8GlNDoRnfo ix6NbkSjm/Kh//LlEYO1B99SPOkYdLrOnWIk5BORDbzMPsIlFYI2l8K+4hQXdAi83tbXxABnlw8n YSWt/Hy7I7pauz29iB4W/cteqo0aFT0s5V0xiehYU9iXm2KdRkXHmuuu3MQEZxd3xSWiG/VhVxb+ 88J7yDTYmYrnC1NGJcJU8MhgyT/ef7zsG/9wgr27gy7LTxuSGuTtBY9aAeBNxOX7/YNnEn44+I3K TydOE4t/f/32p9ffmBSW4kZapfzBJNkbKfweZ1EfCeo7uwJsmm7AwkwKN2oILy2ySxc3PN/FTc44 5nZI6jt34NPoIwry4FaATdMN2P7VqPpYiD+5O3OuyK2L2RvYMZ4IaXwxE5fcPhKUdWYF2O5kAxSX UpiDCbwQRUoxF9zyixmPOW4BphjvzFK4+4hCP5gVYNN0AxbpL9zcJGIpX/Y7d5Ine3ITbkLiKSvu zLWsj5TnVyvApukGbB15ZiUtuF9S3DixG3eU2szGWnFXrol9JJT7zq0Am6YbsM72RMODzXlRkAiK HIeGVmmSZyk9mNyd3EmJ6iPBpwu5BmwPvoHr5MBReyQz8I4pbhzbjRuOj7mLZNzF7bmQ9XEct4Zr imtgwgp25twzI0uJuZj5QbmYa0+1qlPNiY/tY8QF79wKsD34Bi46Rd83PRkP31iQDYqeDeKHHqfI 2YZJZRufwByjH3VNA7Yn4QB2EflZRQAsF69HStwFoOTmmwAnKYdJpZzLIkBG7q26AAXYFsaB7JI4 A1YAs4yAyvbhrqDIYzMVeKBAQXjkXZUTkDHO0q4B28I4kOEAylDgTeIAj3S5K0DDdFNAspGYPP74 UcA9yBxjnHugANvCOJDxgtmhwKFWIQCf5+4KkpBNBVEyCRMpBZ5LXR9jHEVeA7aFcSB7BA4FVvKm EjjuCnCaNwVJ7hGmpBRIb9bHWes1XlsIB7B3MxHxfHLe1AiKuwAc5k1AlqPEVJSAyFWvjzGPN1YD toVxIMPVjUQMuedNAsddgZfbNxUUSedYBexSIFkgSVBGIdBwbeEbuNwrDn6sLAjIYLjzOyk+g5/b HfAn+0jDwuWvj/yadwUKsC2MA9kTrnVXgL3igPpIQyuFdypwkoYSfFMQxPn1MdlRiDVgWxgHsk+4 1l0B8gVtmgWSVOuhQLb7UuAlDeVz7grEY/UxuVEMNGBbGAcyjHqIUpEDW2cEwBUpBVlq31QA0wF/ jYnT8VIg/U8fkx/FQAO2hXEge7xZQwHKLAfgK+tdQZLyPxXAerACknS8FEhr2Eeuv12BAmwL40D2 qJNDAV4aDsDnKQVR3r+pIIpjxsV2SoE4rz4mmnugANvCOJGttH5QgP+JHIDPUwpI3v6pIIuLwxQc 3RRktiF9TDSqgQZsC+NExnM5FJSeOPg8paC/QFNB6Q4WO6kUVL6HfUyjHGi8thBO4DAT0eeeN/g6 JcCpRKxy3eWJuwkQSyJDSnMDFFxb+CYuSZfC/HywCCiPNLSzt2L+bHt7Z8VXXvxe2h0ZU547oADb wjiRIzK3K+DkRkDVaSg27FLgJA3F59wV9C3oezDtsAZsC+NETqjUXQFf8LeDZVVpKBb0UhB6MxF0 GnLbM8dUR0HUgG1hvJDtUMC1jiO82JxLQpZWb0qAK8nEE6fjJSFLFyRjtqMaPBDbwjmhKyplFg1c 8TkCHjbfNSSc3Gdhpn4QsPv5rqFINyRjdqMePBDbwgloi99wETQ08LvHEciWLxrGwhx6OqLfuWvg wjvHPJ3yA7EtnMRfgt8O2SEaHL/9HIF8+aahL8xeriQmUhq6O5AxDwkPwLZQEn8IfnukR5eQZRMd zGz5JqEvxJ3gPMDklQQK7znmMOrCA7EtnOSll4U0OzSw/eMIJMw3DX1htnIdMFmlIdb3HPO4FA/A tlCSk+xAawG5IoE9MEfUrxk5FqYqNSEVScxLQu4+kUTC3AaN2BZOsj07Ej6ra+BOABHefs/IvjBl qYyYstIguyCbMB+IB15bGHFBC8wtrkocCgL2hyPc93zsC1OStwFTvCtgjzbH/ElIjdgWzsDfgd8F mF0Dt4Yc4b8nZF+YYndsUR7FS4N37znmMrNBI7aFM/CX4HfFznYN7HX8m4sr0BcJfV2iblvRxikJ QeqzjLnObVCAbWEM/B3oBRxyI7ECvCgkAciWVcFYl0K37mhllYIodlHGYkeB1IBtYQz8Gfjtxe+y goLs4QBkyxcFfR3sd+3WPSsFqbznWKZx1oBtYQy+90NwwkNBRu5wAHLli4K+LllxzMmKf74UFHrP sfghQOG1hTA4SQv4qFi7ABjSyAGFK8sioK+D++amAVO8u9ZUxS3KWPyoCRqwLYzBSlrASuahAKZc Auq3RBzr0EFy44Qp3BXwtsxx+hWN1xZC/F2yImFnugBsDgdA6bc87Otg/7l3xKSMe3cJQejnBii4 tvBhYyQneOr8yA4J8F+zsK9DuGX+qFuXTOwW+1jiKAYasC2M7Gpd4eMMQwFL5YAAylVBX4cTLKyA dPuWkxgEGUua90ABtoXR82fUntH8u9aeNLgvaRXwP7arXgeP3Ab2eYp9Ap/+f2p3AXKNy8N1qYKo CK64188MSe2upIUBrj9bmhlJI4q0YTBxiWV6+RFQJQ1JbPY6rnjj4AtcRNc7Lfw0a8T4DCP3U4CO wzVOsczb/Ajo8ZqxNd2BDXAcjIGr6JrXRAEvLCfgAn8p0HFIZSGWmdFuBc3JFdTYNRtugONgDFxG n7kdCpizOAE57EuBjkP176BAsvpbQdAsxNidJqMNcByMgcvofOCC5J7Ov3AC8viXAh2HJ40dED4t vhUkUSBxbsGKNw5C39UWwFcBqJ04IfF9/RCg4/CuVwqoYsdHgFQmGrvXZLABjoPRN7UFsFQB329O 8N9G1HH4lamgiB8fBTVfM9qrvOGNg9BzFZ3lXlMBLGI4IX77UMehwIsUkMWOj4Durxm71cob4DgY PZG71ryiQJsvj0IWL/qhwMbxf6kgiR1vBd2V647JksEKOA5Gz2V0rfz5u3X1DVf5pUDHod4WH4bV hz0wE1rMegob4DgYUcY6635EQDPb4Jy/BOg4nGnNd9/zCIjtumPRLdgAx06ItrZZByj8xVwDp7t2 8uu4xJbkbv0e/hyvO1qNvAGOnRDla8HPZJZoWQp/z6v+xa/jeK2R9fEJC3+RG6hRPbjhjZ0PpSvy RmXlkkifpOr3yHX+i17HMa2RHq9qftNLPWaxaybcAMdOiBXgF95IpY/SiHpk+fhFr+OY10mPsuKh Rwdwae7bQMbOIcePakp/BWRkDg6wx83oLv6R9xyJk01mZd5JAAN6QyvI1odN6H+pjVW+DZAtZBdp RySA/yMNjyE2qUljE/tKs/TsHzlZMwgtKhaHuvJNy6YUKoNX2YL3Q4oYFliowSqzc3p2rXnZ+0na ZbusS7wPzXkmDI3JEtcGOFY6g4WDhBsG49BmBzq5nRyCcScvTklefPNwR6ZLjSl1414Ax0pnsLpq WAsD8U/+zVy7nOhkDpqued1fqcol1kwaE5Qp8wI43mQGikpYmOEwDvRmosnc9OSNOcoTlaI8WA9z YXbQmCxHbYBjpTNYtAHCDcNyKAuyNzdd+HAneaBTkuf64a7MjBppTuVeAMdKZ7DdfObEFo3Wf3Pj xF7cWV6FlOWNeLg704LGNBuWFXCsdAaLhCFGQ91FXzTW4m/yJJ3ZJK9ao1V5Im9yL/WhxmSZaQMc G5/hejlwZE9xBjuRN3eUnmxyNykJUhPHPdyBRYlGO+4Vbixchon0osxVncEe7NUe4PnyL+auVuur 1VD6XDOyb1DuBXBsfIaLNzLoppcfAc9JYw/6pnfSixk9TlG6E7e6jScwYw6W11bAsRMaMIqEVLsI yLgrGA9LvAWULq3YFODFcvgslvNVBEjM0ZuABXAcjIaMQqmYAiR8TEAp1RYFDQCPgiBNAe7x6jup BzXmmdpXwHEwGjKKxWYKwo/CCQF2eSuo2MNHQZKNxCfEV48aHFOcxpznHiyA42A0ZFTszhR45CpM wPL8W0ERsqkgi5PwSYuCwFSnMWdL8ivgOBgNOWCiKXDiG3QVcVGA03wpKHKP8CmLAikINc5cv+KN g9CA8fqbEfF80jdorPIiAIf5ElDlKPFpi4DMrKcxV3tjV8BxMBoyOkczIsoe8U0Bx1tBkNs3FTSx c+4C9igQF4gJmiWCFW4cfIYb4rQh6q3aMKGC4c3vJfkYf3Fiw+I2GzamP418zVXBAjgORkMOCdda FWCvOKFvNnSSeKcCLzaUyS8F0fESaizOEvEKOA5GQw4F11oVwC8dTwSQJFubAtnuR0EQG8py3gqk xtJYvCWDFXAcjIYckLWzZOSIK8MJqIoWBVVy31SAoqNnfmjHR0Fka6axBEsGK+A4GA054M0yBUiz nIBV9reCIul/KkDpQQVJ7PgoSLyEGpl/VcECOA5GQw7Ik6YALw0nYHmLgizv31SQpWLGxfaLAqm8 NJY092ABHAfjRGZzJwrwP5kTkjQ6j4Ikb/9UUKWKwyeirngUVJYhGkuybLACjoNxIuO5NAVNjYPl LQr0BZoKmlaw2MlFQec91FgsHax44yCcwHEaMVT1DVa3CPCLEbtcd3niXgKkJJFQytyABW4cfBM3 SZdCfh4sJrTNhm72VuSvThoXeeTf/EHaHYmlzh1YAMfBOJEznKsKaG5M6KsNpQx7FHixodQ5bwW6 BboHsxxeAcfBOJELMrUq4AVHL+pWG0oJ+iiI2kzE1YZse2Ys3RLiCjgOxgfZmQLmOs4IUuY8Eqq0 elMCqpKa+KEdHwlVuiCJ1Vk22BDHwTmhOzJlFQ3M+JyBGra+NRSc3D2wJj0IlPv1raFJNySxessH G+I4OAHt8BtVRDINfPc4A2750GADa1Q7ot95a2DinbHOSnlDHAdn4krw28MdosHz7ecM+OVLgw6s Qa4kPmnRoNWBxGoSNsBxUCYuBL8D7KESqmyiRzHbviToQNwJ+gCfsEhI8ZqxRssLG+I4OFOQXhbS nGlg+ccZMMyXBh1YnVwHfNyiIfdrxmqXYgMcB2Xy4g60FpArElgDc0b/dKQNLF1yQmlizEdC1Tox iYS5DSviODiTU3cULEs1sBPAjOC+HakDS5XMiE9dNMguyCbMB2LDGwcjLmhDcYurkk1BxP5whv/2 ow4sRd4GfPJbAWu0GettyBVxHJyR68DvBkzVwNaQM8K3IXVgyVqxZXkUHw3BXzPWNt2wIo6DM3Il +N2xs6qBtU64mFyBfkjQcSVp2Yo2bpEQJT9LrH1uwwI4DsbIdaAX8PBGoQK8KEkmwC2nAhtXopbu aGUXBVnKRYnNWYJcAcfBGLkM/A5S71JBg3s4AW75UKDjUH53Ld3roqC0a8Y2C+cVcByMMWg/hErY FFR4hxPglQ8FOq44qZiLk/r5UdDSNWMLJmDBGwdh9GIL1FG5qwAUpJkTGjPLIUDHofpm04BPflet pUu1KLEFywkr4DgYoxNboJSspgBFuUzoX0a0cegg2TjhE98KuC0zznplxRsHIf5dXFGwMyoAm8MJ UPrlQx2H8p+9Iz5L4a5VQhT6uQEL3Dj4sDHiCX6UH+6QCeHThToO0x3589q61MRqUWPLlgxWwHEw sqr1jccZTQGlckIE5alAx+EEGxWktX2rRQoEia3Me7AAjoMxcBldHc3fvatpcF/KKcCGwcQllunl R0CVNCSx2eu44o2DL3ARXe+08NOsEeMzjNxPAToO1zjFMm/zI6DHa8bWdAc2wHEwBq6ia14TBbyw nIAL/KVAxyGVhVhmRrsVNCdXUGPXbLgBjoMxcBl95nYoYM7iBOSwLwU6DtW/gwLJ6m8FQbMQY3ea jDbAcTAGLqPzgQuSezr/wgnI418KdByeNHZA+LT4VpBEgcS5BSveOAh9V1sAXwWgduKExPf1Q4CO w7teKaCKHR8BUplo7F6TwQY4Dkbf1BbAUgV8vznBfxtRx+FXpoIifnwU1HzNaK/yhjcOQs9VdJZ7 TQWwiOGE+O1DHYcCL1JAFjs+Arq/ZuxWK2+A42D0RO5a84oCbb48Clm86IcCG8f/pYIkdrwVdFeu OyZLBivgOBg9l9G18ufv1tU3XOWXAh2Helt8GFYf9sBMaDHrKWyA42BEGeus+xEBzWyDc/4SoONw pjXffc8jILbrjkW3YAMcOyHa2mYdoPAXcw2c7trJr+MSW5K79Xv4c7zuaDXyBjh2QpSvBT+TWaJl Kfw9r/oXv47jtUbWxycs/EVuoEb14IY3dj6UrsgblZVLIn2Sqt8j1/kveh3HtEZ6vKr5TS/1mMWu mXADHDshVoBfeCOVPkoj6pHl4xe9jmNeJz3KioceHcCluW8DGTuHHD+qKf0VkJE5OMAeJ2NsUkTW LJ2hfYzxt59/tevnr8vxDx/7Xz9/h4x/QsZ/7A7+fXl3/ev64093/fsf7JLYr1amsARd/+e8bHJ0 x20oOq9V1BKsf3kZQQZZQc8eA3TvfxDeS8qWzFnQgL9mPfJYlijyUhc6darEFIV59g9eioHBHXga eJ+f9p+ff3NpE92srqXpz2ULxZH8n0s7XyTfpdqb3x3e/rmP7Us0vt3DzjGrrfcj0V5Dqh4C/s5d thx5s8Y15OkpAaUH1s0ehta+Mi1gbuhVAt6L+HGUQFL7gq1p85A1iZArdSNT8t7Vblp60JunBNRM TDctmMlH0KkbqwE3s2+hu3WPulXwj6cE1NTlmz0cPfRaMGBsaKtJqfoLDH14SkCpjSuj7as86KyJ qn8vG7pa55tb8/94SkANzKvVepihO+/ATEzEheZWQFhaYzP04SkBNaCOaPcHfVnA2NB5qfZXN348 JaAGZtxqvd/QTd+tAVB3LzpVH4qgSG5HH54SUAPSgPZ0NFQzA+4NfXHVxdY+HH14SkCNzn+gclro hIBCP0fPmwoqcUNaftCbpwSU2vicOtgIgEZt0oDKr1voaVPC2EbV01ECSUvtMHs4GZ/HgLmRKfjG TK7ADX14SkB5Fa9oAQudNAVmYyItNFPDh7tUHvTmKQGlbbXoPuqIcA1DJ62IGtC9+Ri6tWdyxa4Y +vCUgFLBMMzujrYCqgFjQ3MIYIkac1WQw1ECSW0fb/JDzkMDBoeYRS5cbuLSn5J6eEpAucpqlw9e KkL1pRpgomyhOTt1JuBoXkFOTwkoFZA4/ZZ8rkXAZQF9Qyesuk/f70XeHCWQXJo2nLWRUdY1wKTs Il9Ybu++30Y+HCWQVHZ3s4eTkZMMmC/ZLkqv3I7k9eP0lIDqOl1pWjVcq4VWUfOrE0C7X/TkojOa 17jKg948JaDULrBRCwyN668BWoE39OCqWT/6KqinpwSUTkLD7OFo6EIGjA3dcYiQkfjodYqHpwRU hzhVW+eOB521cOpYWDZ0Q2q0Mf0sF3rzlIDSCbIWtTsHMKDRIDRAsz+9aN7vxjLSV0E9PSWguk4L ZvcHfVnA2NCccFoluowHvXlKQKmdYQ8OrkCjGWuAFpoNnXGMjc+evYKcnhJQvWmq0MZYCDQkhAY0 XdKLTki7llBR+7Xy+vCUgOpVi7naUEELnSygb+gL662T23LPB715SkB1fI7a0G6GTlzdja970P3G qitLapvrGA9PCaheWCnVHo7GtWbA3NBcb23c8XWKh6MEUtfmpoWyY0mLnBAw9Ks2MmU/t6O1h7w5 SiB1pHv/7ZD2sG+IefVH8r9gI1M8If9APh0lgHpijdKfbuDJfnTjwm5gcC9emWLV4/STwOmJZVVf lxcXZfdGjckPlzKsTO5x6gu8OUoA+ZZwgwge7BTYQTSOhwzVUQaetsUfPwmghuqttlZW3XmCLw/o G5jluVAkVNdLH08JqHZz8ZqIydCdLRmZmnc018xWXr1rfTwloNrkhqs9DN14IAjA+Sz0xVVnz2iS T0cJJL/lfudJTgzIzKiHDGhhv60+Fn08JaAaJInaKqcMXZn2KGXpRTesObMprmQ+/SSAWudNVHsY uPCmImBs4Ik1s4pWn4k+jhJI3gS8JZCcGVBZbBYZzNyqXxcn744SSA16VW3VUkbOLIlodGkjs1Vl Vo3iHfzjKQHVKpWh2n2hLw8YG7oh7zIbQPGR6OMpAdUq24tqhmzoxPYDUVE2NLtgvvAs3Q/x9JSA cm3jSkfRF5smpBB66EIXrDqxShfv4B9PCaiG6aP/Dqu8RCcP6Bs6Y72J1RTX0dG7pwRUyxQgqioT bRWDECiQnXlDs8FC0eE6Wvn4eEpAtUTZpPYw9KSsQsD9FlPck1SQgMUnoo+fBJDrdVfvBCcG3FSG DrbOnRJz2yeij6cEVEOajl+dOS5DDwpYDCVpR4PMEpJ9Ivo4SiDpaDTM7kbuVN0IGBu5I52v2e2m k3w6SiD5OOfDHcl51F9Mf1U7yUNG4l3sidkHoo+nBFTF8EH7NrTqHgboPdvQlYu2+1gtpz+eElB1 6Gep3SjCib48oO9orro0VpFF3h0lkNTGpDi0uhpZv6YhQEvaRsZluTjK5eRneDpKIFWMHrTvRLKW XQ+4byf/Da9aHq8yaOlP0kP756+f//z+98fljhcZvvtESXhXxWyitubZendigBbo9e5//VyAErhE 76RiKp3R7yrUO/3iP+q4nHhK+jIcmgA+9X8KVNiwlyFR6sVwVYj5YsaMwb/7lqIh63NWPl0xn54S UDPzndVmVEUnqBAGzA2NG3d3pMHtGuPjKIE0MxOlJl5EIydN0oFMfMlU4TdT+O7pQW+eElAzMVNq Zn0BerIfjdszxdC8cnfhfrjI+HhKQM2LiVAxwxgaKoQBY0M3rjpVf4GhD08JqHkxZSomr4XO2o8m StWL5pWbtuOuMz6eElD6zqrLqJgXDY0RUQO0GaQXza2Yk2t3ofHxlIDSQ5hm9wd9WcDY0BwfJuXG vNOD3jwloAZal9pNfwwNJaIB6GQvmrs8WfSnC42PpwTUQMOlPR2tAjUx4N7Q7CGz2Np9XDs9JaC8 JHiBMHRCQKGfo+eNyzITN8SFxsdTAmp0fo7WveRoKBENqPy6heYB4pJhW5x8OEogjcYzUHs4GZ/H gLmRBxY9JjMwL/ThKQE1GjNHG0x50ElTYDYm0kIzNUbPvisLvXlKQHkj9LYIdGK11caJ9F9ojlWj dt8VQx+eElCjcGBRuzvaCujsvLQLXZF7LFFjrgpyOEogjcIhq11sXkbGFDYHh7JF5sCGKUGfT0k9 PCWgXCe5alL0dXN2nJOj5EJnLLozAYdLjY+nBJSqO5x+w7Yt9GUBfUOzYVPiYr8XeXOUQBoYQtXG WRsZZV0DMJO+ZM6Cvft+G/lwlEBSFd3NHk4e1GjzpmRzsl2UXrkdyevH6SkB5drfJwFDQ1jeF3Xm QnPM7BnNa1zlQW+eElBqF9ioBYbuVMM64aQNPbhq1o++CurpKQGlc9Ywezi6UcNrwNjQFKEQa/jo dYqHpwSU2hgrdFYsDxpzh458ZUM3pEYb089yoTdPCSifaX3CBbpyWroLh6eF5v22gbavgnp6SkB1 Fe9m9wd9WcDY0IWrpiRFZ1rozVMCSm0Mnm1wDAW6cDK9Kyephc44xsZnz15BTk8JqKUdoScMDQmh ASq9y4tOSDsbDfu18vrwlIDqqv9NlrYHnSygb+gL67UJrt3zQW+eElAdn6M2tJuhE1d34+sedL+x 6sqS2uY6xsNTAqoXVkq1h6NxrRkwNzTXWxt3fJ3i4SiB1DPnoY4lLXJCgOr8tpEBtixp7SFvjhJI Henef3viAPDn54aYV38k/ws2MsUT8g/k01ECqCfWKP3pBp7sRzcu7AYG9+KVKVY9Tj8JnJ5YVvV1 eXFRdm/UmPxwKcPK5B6nvsCbowSQbwk3iODBToEdRON4yFAdZeBpW/zxkwBqqN5qa2XlTKsBlwf0 DczyXCgSquulj6cEVLu5eE3EZOjOloxMzTuaa2Yrr961Pp4SUG1yw9Uehm48EATgfBb64qqzZzTJ p6MEkt9yv/MkJwZkZtRDBrSw31Yfiz6eElANkkRtlVOGrkx7lLL0ohvWnNkUVzKffhJArfMmqj0M XHhTETA28MSaWUWrz0QfRwkkbwLeEkjODKgsNosMZm7Vr4uTd0cJpAa9qrZqKSNnlkQ0urSR2aoy q0bxDv7xlIBqlcpQ7b7QlweMDd2Qd5kNoPhI9PGU/5Fd7rqW5DYUzfsr6g+s9yOezIANDDqYwHDm SQzQwIwD/765N6U60iEaYF/eS65SqSRy06FqYXtRzZAMHdl+ICrygWYXTAE2t/UR70hxqKVtltJR dGDThBRCD93ojFVHVum8OvhXpDhUxfTRnm6Vl+i4EtqBTlhvZDXFdVzoM1IcqiYKEFWVkb6KQQgU yM50oNlgoehwHa18fEWKQ9VI2aR+N/SgrELC/BRT3JOYcQDzmoi+4sSBll5f6p3gyIRJZbjA1rlj 5NleE9FXpDhUxTHtj84cwdCdAhZDSTzRILOEpDURfQWKI+lo1M1vRm5U3UjoB7nhOIfR7KaTfAeK I61xbg13JKdeHkx/RTvJS8bBC+yJaQ1EX5HiUAXDB/1paNU9TNB7dqALF233sdiZ/ooUhypdX0v9 ShFOdFgJ7URz1bmyimzyGSiOpD4mxa7V1cj6NhUJWtIOMi5L4CiX4vqGd6A4UsHoQX9GkrXsroQ5 F/kPRJX8RuVOT/+L+tH+/P3Hb89/fiy5s4oMn32jxD2rYDZRX8/ZfnZkghbo/exff/zll/+O55ef z/2An7/8/Ud8/vrE59/6m6n97n96B56/Pf/4Z3j+9erjQXGVGx/0WfDPH4Gk1c1igrSJWBm+sGDb Il4Ukq3byrDEghsNP6bAJerSM7eR+29bbxaq0tBXpDjUSNwN9aeh4+R2acI80DwuZnE2ib4jxaEG qpf6UOEbjY7VxzpWhuYhN7slyVekONSI/ENJrEZADx7dPte5MjSvplmUAUNfkeJQI/B11B95o4Ml 2LExNOuJWVzCjT4ixaFG4EcomecX6M4qMcI6gURbFTSLimvoK1IcSp+Jo1MwXRoaskUTIk/SRrN0 my2zLfQVKQ7VJ6uD+u1FR0sY+YNmwzGL3rbRR6Q4VEcFU79qlTB0pQjRz10OdP20yrrJV6A4UkcF oz8XWaskErR5HGR2drN7nvmKFIdSH2ViVRNDp8yEdqCtV9qsm+eLPiLFoTp+UN+KJNCI1AQrowtt KspsLfvsXZHiUB3Lpz9edLCEeaCp/czWOl/0ESkO1SufudoR0PgqmlC5hI2mYjVb2z57V6Q4VC9U mauJAg3hogmNG7fR1Nlma9835ooUh+qF9Wy1fkNHS8Dn3ui+B4Q62gs+4sSBeqaeX2oFYLsRo3OY 22COM2br3Gu+IsWhVFRhCFkSS9FhckoZg3PnQtsMZrYFa8RfkeJQPXFyqqiGGx0toR3otKfHFvsL PuLEgXq0VmhSFuDBYXCJ3Q3msGu2pb3mK1IcqqPT0J8L3TnCDjSmD7pyO2hRdgx9RYpDqR9VVK2x wdCpp2cGitKN7vHZtpX6oo9IcaiGrq6+DTtAo+1rgo1DGz0o4GlbXUfvjhSH0qFsmj9edLCE+UFj IN22tfKij0hxKPUhYNdgCbT+hISkr/VBp7SFEldq6CtSHKp1yu6qK1joQl0+MyetV12OZ9s29qqv SHEo9YP5vW50tISRP+jan23bjC/6iBSHao0DTlVhtdCZE9AsusgPuudn276E6VekOJSqR4xl6s+F Tpzbpur0A83eYhaVztBXpDiU+hglVZjWF41ZUxPaB23Dqdme8os+IsWhmspm+FOPv6GjRmpC09f6 oK2Y0vZNvgLFkZoqePPHSw6WMA9ysYbIrS7pRR+R4lDqJ62wDd/I0BDGmqBDwYFuRNP2fc2vQHGk psOUnsEGaQ9/agFG/NCX+oD7eLbtzdZ8B4oDqR/M7xscLX4c4MlmSLvk6R0njqM/4D7qr7NxB++r puH2Lm6L+dm2b/AVKA7UcD7pW6vRLetaGbGD4SCn+Wzbh0mPr0hxqAY1q35mMyM6rgR0u40u9dkW PXqhz0hxKD0iqNd6FrOhG+s5TlQ+0K08246wVn1HikPplcGWqz8NXdmFkDAPtFU6gKPV/684cSD1 cUbW9EkwzhCubj3Ak9tBO9LY6DNSHEorCPq5liqcckUX9nvUMlyDhcbWbYu+QfQdKQ6lJXWaPzY6 rIR5oPOzzChjg884cSD1UUC0chcDZxYYlPZ6gEt7th3VbvdXpDiUNq8Gf7AGKlrlFBMqi+RGt/Rs O9o6eHekOJT6wfy+0XEljAM94rPtWDXpK1IcSrs5ipPKhmzoyOKF5l8O9OzPtmOsVd+R4lAqb9AX 1Z+GDuybUEPhgx4sHMvOteo7UhxK/ah+D+ztRKeVgOa/0ZSlZmdIG31GikOp3kPzUmGZ6KsebExQ iXKgy3i2ndGkx1ekOFSNvAHqj40OK2EeaPYUszOljT4jxaHUh95TpV0MPagHIcXrgR5E0868Vn1H ikNV08g6eECzKlp3DgkxUNRu9OSG0M5i0uMrUhxK/WB+3+i4EsYHjf19bc0bfUaKQ+nslXUe0Uks G1qVhLY+jGpFxflG5/K8tq1V35HiUDqKdvOnoVVQMUHv2vygy3xeu2TNV6Q4lPoRvsbFYui4EvqB bvV57djkM1AcqWiFha8vZ+RCKaCbqG/1IQ/uhNm5zt4dKQ5VGiuD+tPQqo+7JcyN/gNRJb9RGRqp 47+oqX/+/uO3RwXUg3985o0Q9wz1kdhV3Of1zMSEQuHLZ/76Izz4t6UqJFTHuFQUOEAuz2pyVMHo auvv3B+FQcoGajdsD2j6EgZMkd/wA7SEjPmr24qq+iXw8KiMTBBfEUvG8VobHiJ6jFkMMoa+IsWh RuIuFFwKouPkNvXOd9rojPNhthTTIV+R4lDq46OWyE9saHz1PngINrqix5jFjLTRR6Q41Ij8Q0k8 mEAP3TpNmDy6G924IbSl7VVfkeJQI/B11B95o4Ml2HEx9CCatvT+oo9IcSj18RFK5ukFurMsjMBv stB6vJ5tMX4Z+ooUh9Jn4uiUwsoFNAZJTYg8SRsd0XTNltkW+ooUh1I/mN9edLSEkT/oTDQtNM1G H5HiUPp9C/yqTc7QenOQoP3iQFduCG3d5CtQHEkP5TB/LrKWRSRkfasPmfrXbM1rxr0jxaHUj1X9 xqpj6JSZ0A70IHoYer7oI1IcquMH9bsWEkMjUhO0XdQXnQJ6rtla9tm7IsWhOpZPf7zoYAnzQEe0 GLO1zhd9RIpD9cpnlsGbADS+iiZULmGjM7bCbG377F2R4lAdJ6U+BefG0JE1eTRu3EZX9BizGHYN fUWKQ/XCeqZ+nxsdLWH0DxrnmaaO9oKPOHGgniniK26oge1GDFzh/IIHaqnZOvear0hxqJ44/NSw 5lMILYyXY3BUWegcADXbQl0964oUh+qof+pHDk+GjpbQDjTuIE2L/QUfceJAPbIZVlRwAw/OgAMl /gPOqKRmW9prviLFoXrgYFrRdwzdqTTG5CC70ZXbQYuyY+grUhxKfcjKim650dCdE1/rg+7oWmZb qS/6iBSHaujq6hdKY6DR9jUhUjxv9MChM9vqOnp3pDiUTmLT/PGigyXMD7oEdi3a1sqLPiLFodSH YtWRryy0/oQEnUsOdGLXosVKDX1FikPp1ItxqeoKFrpwnpqZunyjWTnMtrFXfUWKQ6kfzMfIZ+ho CRgKN7r2Z9s244s+IsWhWuNMU1VYLXT+P9lljwNLbgPhfE8xJ1jo/yd2bsBw4BM4Wwbe+wcuFqUe 9hAL9Fu+V/qao5bIIoeejcnVoWf93OdMx+69lRJQ8I+zM94HXTiqwWAmh2ZvsadWumtNnVICCnHW eHGaNHSpXDC+aC1C9zlLfdBOKQE14O013jj+hs5QYgG8ff+irZjyOS/5JZRAGvDXFq+HnGzBduRm DZFb3cqDdkoJKMQFFXboNzK0GmMs0Jniix5E8znvNX8JJZBGwY8an6HWXuO9OTTsdYYEA8/1uc85 LOe3UAIIcbJ4XnA2/XLgzWbI57Gnb50EDv5H7yP+uhp38b5imd7ewx25fu5zXvBLKAE09HwytlaD LZuojLqDyZHL/tznXGY9fpQSUEPdLOLKZkZ0Pgu0211065/71B590F4pAYUjovUaZ7EaerCe64mq Dj3a5z5XOlm/lRJQuDK65Yi3oTu7kC7YDm2VTsHZ6v+PTgIIsZ6RM38SrGdIr2534M3t4HOVddFe KQGFCqL9HKVKTznQjf1ea5leg4PWrbtP7RtEv5USUCip2+J10eks2A5dP+ex2rpgr5MAQqwFBJW7 GbiywGhp7w7cxuc+V7fb/aOUgELzGhov1kCgYae4oLNIXvQon/tc4xy8t1ICCnGyeF50PguWQ6/8 uc91atKPUgIK3VyLE2xDNXRm8dLm3xx6z899rnWyfisloGBvtC8i3oZO7JvqhtIXvVg4znOfrN9K CSjEGfFM7O1El7NAm/9F05bac6dy0V4pAQW/p80LxrIwhh8cXACL4tBtfe5zZ7MeP0oJqJ55AxCv i05nwXZo9hR77lIu2isloBCr34PTboZe9INqxbtDL6L53PVk/VZKQHXzyBg81LMCjZ3TBTnR1F70 5obwuZtZjx+lBBTiZPG86HwWrC9a9/d59nrRXikBhdmrYh7BJFYNDSeB1qejWoM5v+jaPs9znKzf SgkojKLT4m1oGCouwF3bX3Tbn+d5bM2PUgIKcdYYutwMnc+C6dCjf57numQvlEBqqLAa48cZudEK YBPxq77kxZ2w5z5n762UgGqDlQHxNjT88bQF+6L/p6pWH1VVjzT1j4ylf//3j/98YKA++h/f+UZI eAdiXThh7ut5Z+GCRuPLd/4LPwhzYzqu59//+CcdqzqpqVNTA3fpC9rn9DqaYW1u59+5TWCqo03z mmMH+OuPf7us9R+12nzpP6vPz+rrGXwLDdLUOteej7DL9yOs9aCdUgJq6SzJeEzv4rFAT/PzfXlw /Jn8EUogLS2QTf3xncUy79yz90ZmHz7HfeTHajulBNQq/Jyv+UCbKxZsl/TI30va+oN2SgmopbWX 8Xiy1iI0tTh/0Z1Z5+UK4o9SAgpxtVeZ41TDqG4DC7rLuuXTdr5l/EcpAbVg4/CqkViCDa1VeS7+ /atqnTKexoN2SgkoxPrOkei0vmPN3KwcF12uMfl24x+lBBTiaq+yiXJvdqm5uW+vKr56Oamf+cMJ JYBW4tf1A5NaUeitChg4jWMtnaN6CSWAVuKJHPkMqnuzZeOv8zfjZXY4v1zgSygBhLjai+admPQ6 46/7N2N1UDob0Ki1OzA5nQTOSrz3o9CskKuWcSVWgcudtNrr5eJfQgkgbJHWKsQ9H3DFQIEvkV3C PA9z+sHjpZPAQdzsPfMmrA5maVP9cjsT7mc3DtfpJHAQJ31P5Wj2TI5L/dCXS182eU1+BscjlADC xdF/QGy2Z6uvh75Q9jhKDgaZE+kpoG+hBBDiZi96BlKdUaBvLuPCjFM9w8EBO6EE0Fzc+6EzzxlH lyUyXMY8voPN5M4EL50EDmqqnhXEZ0ZC59Y8qm9SOtnqlHs3+oCdUAJoTp7u0a5rH/Tiq/oWZddi sE7MUi/YCSWAEOt1RPNO9Qy5yxJxHUoPgvnPRI9wZlwnlAA6fmP04+C32gPqXX+yiWtkBY87wbyE EkCIu71o3IzVgMAwuO7ECzfYQcatEy+dBM4crNGIzVBuOMtMvWtNNn32Nc4XJPgllACCkWQ70S9x wDr9Qe8aE29yZ9Z9XKyTScAgZivBz7j56jQCO1u/5Wc25tu5zccAvIUSQIirvegMn5XNHnrXk+hZ etWsRzll4qWTwKFVRzzPCAN939S7lqTnFuByjsXhOp0EDmL9gkN98eFWzWP4jqT3Hlyein6xTiYB c4YWxNuKD2zSpt71Iys7rJn97MJLJgGDWC/IQOXoBzstDdeOBvtnG3Yo1gU7oQQQ5je90oj7zZdt Zvp2ZNWs0Wv2US/YCSWAEDd70bwZs89MlqwLnnolGvuGHrcDdkIJIMRaNgfMkWWcrc9MFtkLHsyY FqjXs8cvoQTQLPzeiE9DytZnFrvCBXfNuNLX91stX0IJoFn4P4jnzZh9ZnHZBTf9bJUeU432ATuh BBBiTX2m25CSNRqdJL5gFuBKR9HOiPkWSgDNTG8z0+1IyRrN5qe54MKMeYzbulynk8CZ2aa9dBsS bF2mXs/S5bKw11p4mPcFO6EEEGI9z/jD+giEyxIZLmG20MoG3bp9u7dQAmgm3kDEdR9wt0T2N+PO 41DZSFvLF+yEEkAzsWIgHjdjbTT4oVpfHjCIhe2uPfk6mQQMYq1viK2N4BNrm9mJ1fBip+Zbxjh3 hOCXUAJobBbkWU4jgV4bzbaZ7gErl1UolYt1MgmYsdk/EPebr7YZnHz942JtH1gm6poX7IQSQIj1 DYitj+DaawY7M58HrFtbWOXrcWtvoQTQWNwRVA9rJNDrjkG/XcZVM86bB+5UzLdQAgixfkPE/Was 33jr5ODAmnFmQa7n1r2FEkBj8cwhtmaCZqBnchee0AtmC830l7WamXgLJYDG4i1BT7FmAr3aOujX t/5oMQOYZaKWy3U6CRy0Pb3WiPtNWH3oriwCl6vUWnnx9sU6mQTMmCxDiNc8WPXNu/qW1JYmmlmD yp4X7IQSQGOybsK6WCeB6ZiWiGtJ3IBMc6k3mtyXTgIHJkgLPWKb4aDXXrmb70haIlG3F8Hzcp1O AmcM9hHEMx+ujlK7+YbUhp60xCJR+rpgJ5QAQsw+Ms4MBxu6LJH5rT7mI5JdvXZOxEsoAQRbrN0f sc1w0OtP2d03JC29iSdJi8XlOp0EzujWR8YZ4aBXOwO9a0hND1li29BScbhOJ4GDmG1knhEOc8my PFw/agXfTKHpJPsSSYCMZi1knvEN+m5JfHuRAgnj4aTf1aOa4Z7y0j+0V/39X9Xmj/6nWpxKbH+5 6cv9eV0dzfim3xe7519c0Gb5psbfs7eewc3atNd60E4pAXV+RF80g4YutXx/paH18mye9D3rQ3ZC CSTE6tG7mhEjqxvhAjelbs4km2dyj7sfL6UE1CqcKro244vWOocFbk7dvEGbbWufMepHKQGFuNur xpN10VyWn1S1Q6mhWOcFF+2UElCIq71qb0MrggvcsLp5izYL1S7loF9KCaiVbczUi3fRvXOBG1dt KxY97k7jQTulBBRiDppqTC66ai7w7S5rvUt/LrrctfuDdkoJKMTVXrW7xlvNCfVuZt0cIhaHS02d dvQllABayYZNmMl9wH1T74ZW3QH14eMkfsBOKAG0Ek/kUHNywBWJ4K/d1Lo4By4WrNX7BTuhBNBK vEVDZySCF6/zSn5uXYvbS5+77NC9dRI4K/HeD/Umh8t5NPnBVbto+nMupl3WBTuhBBC2SGvVUG9y wJxHsx9c7TzM+X/Kq2C5bRuI3v0VPFKdcUOAwIKc6fTQtD300OkkusU90BLjqDEkR5LT8Y/0e/t2 F5DAok2mh8Re+elxCezu25eOI/EWuFjxIHb6nJATFjtqknYqr5eEfTqNxFvgYsWDmKWMeDkR3qBu 1IjwZV5ZEoK0SRiS9VkAY0WExuE/UJ9ta1A3apN0KrFUWTCjHEefiQtgrIgQO31Qsq1B3ahN2qnE VjLuuKUD2UxcAGNFhJjPnly2raRu1CbxVGIpXxIxSZv5EhcrHsxUkTyXXSupG+1LkRrEBBLlg07E BTBWRIi9PohywrYXfCFR2hYkcyKk9WMJjBURYm5H8sltQsoHTaRQKC4E7lYpZZOqbQGMFRGUnAcI YhsSsbjRvtSnIMaSZNWlNDOXwFgRIfb6IMoZsx/FwlCokzQciYJQnhMLXKx4EPOMJp/s5ujEJQFf SBMXGAbNQOkGhXgBjBURdlORE0qGE3j2dcAXwiSdLOaVPGXaAhYrGsQiJZTsJvDsQ7Eg99fxE5zk K5supQVgCYwVEeJeH6R2E+aAxR74QpNkZ/Gy6JJNY2KBixUP9kFRkpDcJvDsRoEvJCmICZQS5rJI vAUuVjyI+QYR+5wvm1FYpkKRuO/VsVHnM20BixUNYraaiNVqwivykgt8oUc6dmRm+nQKC1isaBBz g8BwqtMcraoMlXJEop+OtCiGTFwAY0UUemlpxD7nKzITSjnSaab+x1OfiQtgrIgQO31QyBmLzoTS tPLC28m01XJLxAUwVkSIeWzSmMzmaFRnQmlaiSRjWYF8n854AYwVUbBy34iTIBnVmaF0rdxnvCrI 3eVpuQDGiihY+QVxyBmLzgylbeU+A7HsmLxoJ+ICGCsixOI2uyxInQoNO4krsQzgXjYKNybiBTBW RMHIbhO6rEidCs1Y2laykrGUsRsyb4GLFU8waje7LEhY64zgC9uqg73vrRTzmIkLYKyIEIvlNElH ABw0kcK3kkhoLwLtfPKuC2CsiEKn3tVIPwqx10Su3lW8KYhFSJ0zmbgAxooodDIxEFPOmIUGL8rz 5UIMRity5y75FrBY0SDm+YZYZQRXzDIzdjINM23gfC1R6hEhXgBjRUSjDORgk5AAz0Izqqe7EDOv TKHOZtoCFisaGkU/EPucL8sMKp9/ZFo9BxkT/RAycQGMFRFifgJi1RG0PWcwGsnnQsxHa2XK92lb WwJjRUSDnAimhwoJ8HxiwI9Fxj1nbEYpuDQxl8BYESHmO0Tsc8Z8xyM7h4KYMzYykPvUdUtgrIho kJpDrGICMeCaHK1UaCYWCTWyX/a9LhNLYKyIaJAugaaomADPax3ww3X+8DADsYyJ3mbeAhcrHsge tzVinxPmPXTsZQhkXmbte2m8MdMWsFjRUJAxhHgIiZb35rEvJckNnKiRGWTHkIkLYKyIKMjcxOqi SoKlI2gihSTJARhZLrmjhXeBixUPliAe9IjVwwHPWjm6UpF4RGJuD0IcMm+BixUPkegI4mASL1up 0ZWC5IgrrZMhYf2QiQtgrIgQi45Q8nBYQwdNJFynj+4RnbaeSxWxAMaKCGsxqz9i9XDA86uMvhQk Hr2dVBIPi8xb4GLFQ151hJKFA57XGeALQXJcZJ3IBo+KxFvgYsWDWGQkJAsHXzJoHoUeOYs7Y9Iu JbsAxYqEnEpISPYNeK9JXLWICZu3r3+9keKUfZdL1WB7MgP/YK06zjdvb35Y37xar11jmvV7fK/n tQvVcpt/6bg/WWtMj1HY4WvrKJZDnW3+nxPH6XQQv/VGfwPyz5v2zbw5fJ6PL81duznsz9/erVbr PzjBNBYpKcQtvuNkRsiP9Y//9VVkG3K2ln3e0Nymn11jcDgDdhzIghlxWpyrnIXkJ6lxVt91nR++ VzL6wqtnOhQh+DitqO/mmYpT7oy85rv252n3+Hycm/MK19MeVrfYBtvmw7TfPsonc7OdzlODuNnO U/owwTYfm9Xv61+QA0IzDp5fX55jLym/a6fPh9122m/m5q/VrXWQjXa7wrBvmXcz7Zt7PON4eHqa t839S3Oejg/z+aTM62+UL1zzTqTI54hcTDtPm0/PuxVetD0qw5v1Gn8ZWqX44rlbiA+xaiwO/v8e 94WlPO/b4iD4N5KyWn/Ay86n3cO+mXax2Z2a86GZ99P949xMzWk+nXaHPX92Ok8vDdcNH8GFZNS3 PzzNx+m8w1u69rCfHpvp1DxqxDcztvsH/giEitk/gH79+rcESh8q9KPcQT5tydpqC+Be4rTbn/Fv 3r6akdH94+70Yd5+tZyd4b0BTm55rMsT/VLvXgjKEzUybti8XLqWrnWRCoMPOKI6pvPKtwc0Iaqk nVFvvpWuRPTSnJ6fng7HMx//fDwejvr6P61vnMEQc9SQQ0Gj6jA6qLn1FjMKAxSD5z0Pnn9PGZUd ePLVReD/UQT7w3n3frfBDeKupbO4FY7z0yPuHB8d50/POG65fk7KY5JhihOICXKZkurYrQxfSQqV 2Tk0TZ3UWLToXas9E1rMqnwUvEvbxvAmiTfQh0IhesxseebfAwDO+l2oCmVuZHN0cmVhbQ1lbmRv YmoNMzcgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDUxIDAgUiANL1Jlc291cmNlcyAz OCAwIFIgDS9Db250ZW50cyAzOSAwIFIgDS9Sb3RhdGUgOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIg NzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0zOCAwIG9iag08PCAN L1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDQgODMgMCBSIC9UVDYgODcgMCBS IC9UVDcgNDMgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgOTUgMCBSID4+IA0vQ29sb3JTcGFj ZSA8PCAvQ3M4IDY4IDAgUiA+PiANPj4gDWVuZG9iag0zOSAwIG9iag08PCAvTGVuZ3RoIDE1ODQ3 IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJlFfNziSpEbz3U/TZkll+kr+zDz5b PvgBVl7L0rKSPe8vOTKBrsqiBR6NBOpvyIgoSJLIX/7yo7x//fG2pliHsfr0du8fv/7x+uWvf3fv f/14ufe/369AhkJ5Z0qmYo7Gpfefs6um0vu//3z99grVxJpvC3zRC4hMyXmDwAqiUhCtsWEXEvGn mg6YVWEma0JMG50pmhTiHtMFxgwDMluT6y4iR1MjHSDTHbI44xxtVCLCq4BkQj1QeJIIGiHVmRjD IaTokGSyC3tZwasQZ52p1e9pQv/48onJxsVDDOkQ501w7hAiX0PuE5MNZbf/nOh1jPcmxwNPlJSj +InJph5CUtAhIRh7uCwpS0j9hBTjfd1/TY46hoKJtu55ikiL4RNTTEq7UsAxoi3mGROD5NI2psZH TDG17CoKVls5nfQ5nUTGp/2+VZseMdWQP8Q4ybb0OZ5MJpZdFeGS07V99jpXnNm+WHGZUjElGuv3 xYivGsb8OZ9qDeIOMeURE02gfb3h5OSYeT7eWpP8vuRwpumYaHI5xETJt3mxvWOYfZnipFEhyTi7 L1N8/hwyy5T3zoSyLzl8/jommUinGNmB8tEWnMl2X3P4XHQM1pZ9AeE9VjHkjaNDjCQBavMMyeaw z73oXhHRG9rdAh+zyWFfX1izAk0eteD/i0mfmGJs3NcXZlAxORjvyk59LobqCTVp1BJMirtq4kvB a79dUQPOc7evXLsGX63Gx11VCRbWzR3qweMrgoWby3GjIDjYua2nCR7uzR3u+uNMgseqvEUN8G/h cLcf2RECLJ07pbbOwkAwdflwtx/XIRB8XtjFhOiMrYfb/xAS4ezyz13+kGDtwikmPGKSSXv38Kxl kGXq4eF8lMyQ4ez84XI/SjPKMsQcYvQLEAqcHe3u9vLOhApjd7Aoz+cs1Ayrc4pxKoYsnB0ddu3x PJOFtbMHK/SwAeSCofJzdoMcrB2dYrStIR+kb/sZ+0S+yK+9tdM2jdCBejrwPOwgoSkle+iKHraT 0KfGdGi9HvaWCNbu4FGeNprQu1p7iHnYdULz6tKhL3q0BYR+Nni//55H90HoaJM98DyaHEKTm9Op MdLNFBXOp59r2pDTxu5v9qM1JDS6YZ/UzwaU0OhGd+iLHo1uRKOb8qH/8uURg7UH31I86Rh0us6d YiTkE5ENvMw+wiUVgjaXwr7iFBd0CLze1tfEAGeXDydhJa38fLsjulq7Pb2IHhb9y16qjRoVPSzl XTGJ6FhT2JebYp1GRcea667cxARnF3fFJaIb9WFXFv7zwnvINNiZiucLU0YlwlTwyGDJP95/vOwb /3CCvbuDLstPG5Ia5O0Fj1oB4E3E5fv9g2cSfjj4jcpPJ04Ti39//fan19+YFJbiRlql/MEk2Rsp /B5nUR8J6ju7AmyabsDCTAo3aggvLbJLFzc838VNzjjmdkjqO3fg0+gjCvLgVoBN0w3Y/tWo+liI P7k7c67IrYvZG9gxnghpfDETl9w+EpR1ZgXY7mQDFJdSmIMJvBBFSjEX3PKLGY85bgGmGO/MUrj7 iEI/mBVg03QDFukv3NwkYilf9jt3kid7chNuQuIpK+7MtayPlOdXK8Cm6QZsHXlmJS24X1LcOLEb d5TazMZacVeuiX0klPvOrQCbphuwzvZEw4PNeVGQCIoch4ZWaZJnKT2Y3J3cSYnqI8GnC7kGbA++ gevkwFF7JDPwjiluHNuNG46PuYtk3MXtuZD1cRy3hmuKa2DCCnbm3DMjS4m5mPlBuZhrT7WqU82J j+1jxAXv3AqwPfgGLjpF3zc9GQ/fWJANip4N4ocep8jZhkllG5/AHKMfdU0DtifhAHYR+VlFACwX r0dK3AWg5OabACcph0mlnMsiQEburboABdgWxoHskjgDVgCzjIDK9uGuoMhjMxV4oEBBeORdlROQ Mc7SrgHbwjiQ4QDKUOBN4gCPdLkrQMN0U0CykZg8/vhRwD3IHGOce6AA28I4kPGC2aHAoVYhAJ/n 7gqSkE0FUTIJEykFnktdH2McRV4DtoVxIHsEDgVW8qYSOO4KcJo3BUnuEaakFEhv1sdZ6zVeWwgH sHczEfF8ct7UCIq7ABzmTUCWo8RUlIDIVa+PMY83VgO2hXEgw9WNRAy5500Cx12Bl9s3FRRJ51gF 7FIgWSBJUEYh0HBt4Ru43CsOfqwsCMhguPM7KT6Dn9sd8Cf7SMPC5a+P/Jp3BQqwLYwD2ROudVeA veKA+khDK4V3KnCShhJ8UxDE+fUx2VGINWBbGAeyT7jWXQHyBW2aBZJU66FAtvtS4CUN5XPuCsRj 9TG5UQw0YFsYBzKMeohSkQNbZwTAFSkFWWrfVADTAX+NidPxUiD9Tx+TH8VAA7aFcSB7vFlDAcos B+Ar611BkvI/FcB6sAKSdLwUSGvYR66/XYECbAvjQPaok0MBXhoOwOcpBVHev6kgimPGxXZKgTiv Piaae6AA28I4ka20flCA/4kcgM9TCkje/qkgi4vDFBzdFGS2IX1MNKqBBmwL40TGczkUlJ44+Dyl oL9AU0HpDhY7qRRUvod9TKMcaLy2EE7gMBPR5543+DolwKlErHLd5Ym7CRBLIkNKcwMUXFv4Ji5J l8L8fLAIKI80tLO3Yv5se3tnxVde/F7aHRlTnjugANvCOJEjMrcr4ORGQNVpKDbsUuAkDcXn3BX0 Leh7MO2wBmwL40ROqNRdAV/wt4NlVWkoFvRSEHozEXQactszx1RHQdSAbWG8kO1QwLWOI7zYnEtC llZvSoArycQTp+MlIUsXJGO2oxo8ENvCOaErKmUWDVzxOQIeNt81JJzcZ2GmfhCw+/muoUg3JGN2 ox48ENvCCWiL33ARNDTwu8cRyJYvGsbCHHo6ot+5a+DCO8c8nfIDsS2cxF+C3w7ZIRocv/0cgXz5 pqEvzF6uJCZSGro7kDEPCQ/AtlASfwh+e6RHl5BlEx3MbPkmoS/EneA8wOSVBArvOeYw6sIDsS2c 5KWXhTQ7NLD94wgkzDcNfWG2ch0wWaUh1vcc87gUD8C2UJKT7EBrAbkigT0wR9SvGTkWpio1IRVJ zEtC7j6RRMLcBo3YFk6yPTsSPqtr4E4AEd5+z8i+MGWpjJiy0iC7IJswH4gHXlsYcUELzC2uShwK AvaHI9z3fOwLU5K3AVO8K2CPNsf8SUiN2BbOwN+B3wWYXQO3hhzhvydkX5hid2xRHsVLg3fvOeYy s0EjtoUz8Jfgd8XOdg3sdfybiyvQFwl9XaJuW9HGKQlB6rOMuc5tUIBtYQz8HegFHHIjsQK8KCQB yJZVwViXQrfuaGWVgih2UcZiR4HUgG1hDPwZ+O3F77KCguzhAGTLFwV9Hex37dY9KwWpvOdYpnHW gG1hDL73Q3DCQ0FG7nAAcuWLgr4uWXHMyYp/vhQUes+x+CFA4bWFMDhJC/ioWLsAGNLIAYUryyKg r4P75qYBU7y71lTFLcpY/KgJGrAtjMFKWsBK5qEAplwC6rdEHOvQQXLjhCncFfC2zHH6FY3XFkL8 XbIiYWe6AGwOB0Dptzzs62D/uXfEpIx7dwlB6OcGKLi28GFjJCd46vzIDgnwX7Owr0O4Zf6oW5dM 7Bb7WOIoBhqwLYzsal3h4wxDAUvlgADKVUFfhxMsrIB0+5aTGAQZS5r3QAG2hdHzZ9Se0fy71p40 uC9pFfA/tqteB4/cBvZ5in0Cn/5/ancBco3Lw3WpgqgIrrjXzwxJ7a6khQGuP1uaGUkjirRhMHGJ ZXr5EVAlDUls9jqueOPgC1xE1zst/DRrxPgMI/dTgI7DNU6xzNv8COjxmrE13YENcByMgavomtdE AS8sJ+ACfynQcUhlIZaZ0W4FzckV1Ng1G26A42AMXEafuR0KmLM4ATnsS4GOQ/XvoECy+ltB0CzE 2J0mow1wHIyBy+h84ILkns6/cALy+JcCHYcnjR0QPi2+FSRRIHFuwYo3DkLf1RbAVwGonTgh8X39 EKDj8K5XCqhix0eAVCYau9dksAGOg9E3tQWwVAHfb07w30bUcfiVqaCIHx8FNV8z2qu84Y2D0HMV neVeUwEsYjghfvtQx6HAixSQxY6PgO6vGbvVyhvgOBg9kbvWvKJAmy+PQhYv+qHAxvF/qSCJHW8F 3ZXrjsmSwQo4DkbPZXSt/Pm7dfUNV/mlQMeh3hYfhtWHPTATWsx6ChvgOBhRxjrrfkRAM9vgnL8E 6Dicac133/MIiO26Y9Et2ADHToi2tlkHKPzFXAOnu3by67jEluRu/R7+HK87Wo28AY6dEOVrwc9k lmhZCn/Pq/7Fr+N4rZH18QkLf5EbqFE9uOGNnQ+lK/JGZeWSSJ+k6vfIdf6LXscxrZEer2p+00s9 ZrFrJtwAx06IFeAX3kilj9KIemT5+EWv45jXSY+y4qFHB3Bp7ttAxs4hx49qSn8FZGQODrDHzegu /pH3HImTTWZl3kkAA3pDK8jWh03of6mNVb4NkC1kF2lHJID/Iw2PITapSWMT+0qz9OwfOVkzCC0q Foe68k3LphQqg1fZgvdDihgWWKjBKrNzenatedn7Sdplu6xLvA/NeSYMjckS1wY4VjqDhYOEGwbj 0GYHOrmdHIJxJy9OSV5883BHpkuNKXXjXgDHSmewumpYCwPxT/7NXLuc6GQOmq553V+pyiXWTBoT lCnzAjjeZAaKSliY4TAO9Gaiydz05I05yhOVojxYD3NhdtCYLEdtgGOlM1i0AcINw3IoC7I3N134 cCd5oFOS5/rhrsyMGmlO5V4Ax0pnsN185sQWjdZ/c+PEXtxZXoWU5Y14uDvTgsY0G5YVcKx0BouE IUZD3UVfNNbib/Ikndkkr1qjVXkib3Iv9aHGZJlpAxwbn+F6OXBkT3EGO5E3d5SebHI3KQlSE8c9 3IFFiUY77hVuLFyGifSizFWdwR7s1R7g+fIv5q5W66vVUPpcM7JvUO4FcGx8hos3Muimlx8Bz0lj D/qmd9KLGT1OUboTt7qNJzBjDpbXVsCxExowioRUuwjIuCsYD0u8BZQurdgU4MVy+CyW81UESMzR m4AFcByMhoxCqZgCJHxMQCnVFgUNAI+CIE0B7vHqO6kHNeaZ2lfAcTAaMorFZgrCj8IJAXZ5K6jY w0dBko3EJ8RXjxocU5zGnOceLIDjYDRkVOzOFHjkKkzA8vxbQRGyqSCLk/BJi4LAVKcxZ0vyK+A4 GA05YKIpcOIbdBVxUYDTfCkoco/wKYsCKQg1zly/4o2D0IDx+psR8XzSN2is8iIAh/kSUOUo8WmL gMyspzFXe2NXwHEwGjI6RzMiyh7xTQHHW0GQ2zcVNLFz7gL2KBAXiAmaJYIVbhx8hhvitCHqrdow oYLhze8l+Rh/cWLD4jYbNqY/jXzNVcECOA5GQw4J11oVYK84oW82dJJ4pwIvNpTJLwXR8RJqLM4S 8Qo4DkZDDgXXWhXALx1PBJAkW5sC2e5HQRAbynLeCqTG0li8JYMVcByMhhyQtbNk5IgrwwmoihYF VXLfVICio2d+aMdHQWRrprEESwYr4DgYDTngzTIFSLOcgFX2t4Ii6X8qQOlBBUns+ChIvIQamX9V wQI4DkZDDsiTpgAvDSdgeYuCLO/fVJClYsbF9osCqbw0ljT3YAEcB+NEZnMnCvA/mROSNDqPgiRv /1RQpYrDJ6KueBRUliEaS7JssAKOg3Ei47k0BU2Ng+UtCvQFmgqaVrDYyUVB5z3UWCwdrHjjIJzA cRoxVPUNVrcI8IsRu1x3eeJeAqQkkVDK3IAFbhx8EzdJl0J+HiwmtM2GbvZW5K9OGhd55N/8Qdod iaXOHVgAx8E4kTOcqwpobkzoqw2lDHsUeLGh1DlvBboFugezHF4Bx8E4kQsytSrgBUcv6lYbSgn6 KIjaTMTVhmx7ZizdEuIKOA7GB9mZAuY6zghS5jwSqrR6UwKqkpr4oR0fCVW6IInVWTbYEMfBOaE7 MmUVDcz4nIEatr41FJzcPbAmPQiU+/WtoUk3JLF6ywcb4jg4Ae3wG1VEMg189zgDbvnQYANrVDui 33lrYOKdsc5KeUMcB2fiSvDbwx2iwfPt5wz45UuDDqxBriQ+adGg1YHEahI2wHFQJi4EvwPsoRKq bKJHMdu+JOhA3An6AJ+wSEjxmrFGywsb4jg4U5BeFtKcaWD5xxkwzJcGHVidXAd83KIh92vGapdi AxwHZfLiDrQWkCsSWANzRv90pA0sXXJCaWLMR0LVOjGJhLkNK+I4OJNTdxQsSzWwE8CM4L4dqQNL lcyIT100yC7IJswHYsMbByMuaENxi6uSTUHE/nCG//ajDixF3gZ88lsBa7QZ623IFXEcnJHrwO8G TNXA1pAzwrchdWDJWrFleRQfDcFfM9Y23bAijoMzciX43bGzqoG1TriYXIF+SNBxJWnZijZukRAl P0usfW7DAjgOxsh1oBfw8EahArwoSSbALacCG1eilu5oZRcFWcpFic1ZglwBx8EYuQz8DlLvUkGD ezgBbvlQoONQfnct3euioLRrxjYL5xVwHIwxaD+EStgUVHiHE+CVDwU6rjipmIuT+vlR0NI1Ywsm YMEbB2H0YgvUUbmrABSkmRMaM8shQMeh+mbTgE9+V62lS7UosQXLCSvgOBijE1uglKymAEW5TOhf RrRx6CDZOOET3wq4LTPOemXFGwch/l1cUbAzKgCbwwlQ+uVDHYfyn70jPkvhrlVCFPq5AQvcOPiw MeIJfpQf7pAJ4dOFOg7THfnz2rrUxGpRY8uWDFbAcTCyqvWNxxlNAaVyQgTlqUDH4QQbFaS1fatF CgSJrcx7sACOgzFwGV0dzd+9q2lwX8opwIbBxCWW6eVHQJU0JLHZ67jijYMvcBFd77Tw06wR4zOM 3E8BOg7XOMUyb/MjoMdrxtZ0BzbAcTAGrqJrXhMFvLCcgAv8pUDHIZWFWGZGuxU0J1dQY9dsuAGO gzFwGX3mdihgzuIE5LAvBToO1b+DAsnqbwVBsxBjd5qMNsBxMAYuo/OBC5J7Ov/CCcjjXwp0HJ40 dkD4tPhWkESBxLkFK944CH1XWwBfBaB24oTE9/VDgI7Du14poIodHwFSmWjsXpPBBjgORt/UFsBS BXy/OcF/G1HH4VemgiJ+fBTUfM1or/KGNw5Cz1V0lntNBbCI4YT47UMdhwIvUkAWOz4Cur9m7FYr b4DjYPRE7lrzigJtvjwKWbzohwIbx/+lgiR2vBV0V647JksGK+A4GD2X0bXy5+/W1Tdc5ZcCHYd6 W3wYVh/2wExoMespbIDjYEQZ66z7EQHNbINz/hKg43CmNd99zyMgtuuORbdgAxw7IdraZh2g8Bdz DZzu2smv4xJbkrv1e/hzvO5oNfIGOHZClK8FP5NZomUp/D2v+he/juO1RtbHJyz8RW6gRvXghjd2 PpSuyBuVlUsifZKq3yPX+S96Hce0Rnq8qvlNL/WYxa6ZcAMcOyFWgF94I5U+SiPqkeXjF72OY14n PcqKhx4dwKW5bwMZO4ccP6op/RWQkTk4wB4nY2xSRNYsnaF9jPG3n3+16+evy/EPH/tfP3+HjH9C xn/sDv59eXf96/rjT3f9+x/sktivVqawBF3/57xscnTHbSg6r1XUEqx/eRlBBllBzx4DdO9/EN5L ypbMWdCAv2Y98liWKPJSFzp1qsQUhXn2D16KgcEdeBp4n5/2n59/c2kT3ayupenPZQvFkfyfSztf JN+l2pvfHd7+uY/tSzS+3cPOMaut9yPRXkOqHgL+zl22HHmzxjXk6SkBpQfWzR6G1r4yLWBu6FUC 3ov4cZRAUvuCrWnzkDWJkCt1I1Py3tVuWnrQm6cE1ExMNy2YyUfQqRurATezb6G7dY+6VfCPpwTU 1OWbPRw99FowYGxoq0mp+gsMfXhKQKmNK6PtqzzorImqfy8bulrnm1vz/3hKQA3Mq9V6mKE778BM TMSF5lZAWFpjM/ThKQE1oI5o9wd9WcDY0Hmp9lc3fjwloAZm3Gq939BN360BUHcvOlUfiqBIbkcf nhJQA9KA9nQ0VDMD7g19cdXF1j4cfXhKQI3Of6ByWuiEgEI/R8+bCipxQ1p+0JunBJTa+Jw62AiA Rm3SgMqvW+hpU8LYRtXTUQJJS+0wezgZn8eAuZEp+MZMrsANfXhKQHkVr2gBC500BWZjIi00U8OH u1Qe9OYpAaVtteg+6ohwDUMnrYga0L35GLq1Z3LFrhj68JSAUsEwzO6OtgKqAWNDcwhgiRpzVZDD UQJJbR9v8kPOQwMGh5hFLlxu4tKfknp4SkC5ymqXD14qQvWlGmCibKE5O3Um4GheQU5PCSgVkDj9 lnyuRcBlAX1DJ6y6T9/vRd4cJZBcmjactZFR1jXApOwiX1hu777fRj4cJZBUdnezh5ORkwyYL9ku Sq/cjuT14/SUgOo6XWlaNVyrhVZR86sTQLtf9OSiM5rXuMqD3jwloNQusFELDI3rrwFagTf04KpZ P/oqqKenBJROQsPs4WjoQgaMDd1xiJCR+Oh1ioenBFSHOFVb544HnbVw6lhYNnRDarQx/SwXevOU gNIJsha1OwcwoNEgNECzP71o3u/GMtJXQT09JaC6Tgtm9wd9WcDY0JxwWiW6jAe9eUpAqZ1hDw6u QKMZa4AWmg2dcYyNz569gpyeElC9aarQxlgINCSEBjRd0otOSLuWUFH7tfL68JSA6lWLudpQQQud LKBv6AvrrZPbcs8HvXlKQHV8jtrQboZOXN2Nr3vQ/caqK0tqm+sYD08JqF5YKdUejsa1ZsDc0Fxv bdzxdYqHowRS1+amhbJjSYucEDD0qzYyZT+3o7WHvDlKIHWke//tkPawb4h59Ufyv2AjUzwh/0A+ HSWAemKN0p9u4Ml+dOPCbmBwL16ZYtXj9JPA6YllVV+XFxdl90aNyQ+XMqxM7nHqC7w5SgD5lnCD CB7sFNhBNI6HDNVRBp62xR8/CaCG6q22VlbdeYIvD+gbmOW5UCRU10sfTwmodnPxmojJ0J0tGZma dzTXzFZevWt9PCWg2uSGqz0M3XggCMD5LPTFVWfPaJJPRwkkv+V+50lODMjMqIcMaGG/rT4WfTwl oBokidoqpwxdmfYoZelFN6w5symuZD79JIBa501Uexi48KYiYGzgiTWzilafiT6OEkjeBLwlkJwZ UFlsFhnM3KpfFyfvjhJIDXpVbdVSRs4siWh0aSOzVWVWjeId/OMpAdUqlaHafaEvDxgbuiHvMhtA 8ZHo4ykB1Srbi2qGbOjE9gNRUTY0u2C+8CzdD/H0lIBybeNKR9EXmyakEHroQhesOrFKF+/gH08J qIbpo/8Oq7xEJw/oGzpjvYnVFNfR0bunBFTLFCCqKhNtFYMQKJCdeUOzwULR4Tpa+fh4SkC1RNmk 9jD0pKxCwP0WU9yTVJCAxSeij58EkOt1V+8EJwbcVIYOts6dEnPbJ6KPpwRUQ5qOX505LkMPClgM JWlHg8wSkn0i+jhKIOloNMzuRu5U3QgYG7kjna/Z7aaTfDpKIPk458MdyXnUX0x/VTvJQ0biXeyJ 2Qeij6cEVMXwQfs2tOoeBug929CVi7b7WC2nP54SUHXoZ6ndKMKJvjyg72iuujRWkUXeHSWQ1Mak OLS6Glm/piFAS9pGxmW5OMrl5Gd4OkogVYwetO9EspZdD7hvJ/8Nr1oerzJo6U/SQ/vnr5///P73 x+WOFxm++0RJeFfFbKK25tl6d2KAFuj17n/9XIASuETvpGIqndHvKtQ7/eI/6riceEr6MhyaAD71 fwpU2LCXIVHqxXBViPlixozBv/uWoiHrc1Y+XTGfnhJQM/Od1WZURSeoEAbMDY0bd3ekwe0a4+Mo gTQzE6UmXkQjJ03SgUx8yVThN1P47ulBb54SUDMxU2pmfQF6sh+N2zPF0Lxyd+F+uMj4eEpAzYuJ UDHDGBoqhAFjQzeuOlV/gaEPTwmoeTFlKiavhc7ajyZK1YvmlZu2464zPp4SUPrOqsuomBcNjRFR A7QZpBfNrZiTa3eh8fGUgNJDmGb3B31ZwNjQHB8m5ca804PePCWgBlqX2k1/DA0logHoZC+auzxZ 9KcLjY+nBNRAw6U9Ha0CNTHg3tDsIbPY2n1cOz0loLwkeIEwdEJAoZ+j543LMhM3xIXGx1MCanR+ jta95GgoEQ2o/LqF5gHikmFbnHw4SiCNxjNQezgZn8eAuZEHFj0mMzAv9OEpATUaM0cbTHnQSVNg NibSQjM1Rs++Kwu9eUpAeSP0tgh0YrXVxon0X2iOVaN23xVDH54SUKNwYFG7O9oK6Oy8tAtdkXss UWOuCnI4SiCNwiGrXWxeRsYUNgeHskXmwIYpQZ9PST08JaBcJ7lqUvR1c3ack6PkQmcsujMBh0uN j6cElKo7nH7Dti30ZQF9Q7NhU+Jivxd5c5RAGhhC1cZZGxllXQMwk75kzoK9+34b+XCUQFIV3c0e Th7UaPOmZHOyXZReuR3J68fpKQHl2t8nAUNDWN4XdeZCc8zsGc1rXOVBb54SUGoX2KgFhu5Uwzrh pA09uGrWj74K6ukpAaVz1jB7OLpRw2vA2NAUoRBr+Oh1ioenBJTaGCt0ViwPGnOHjnxlQzekRhvT z3KhN08JKJ9pfcIFunJauguHp4Xm/baBtq+CenpKQHUV72b3B31ZwNjQhaumJEVnWujNUwJKbQye bXAMBbpwMr0rJ6mFzjjGxmfPXkFOTwmopR2hJwwNCaEBKr3Li05IOxsN+7Xy+vCUgOqq/02Wtged LKBv6AvrtQmu3fNBb54SUB2foza0m6ETV3fj6x50v7HqypLa5jrGw1MCqhdWSrWHo3GtGTA3NNdb G3d8neLhKIHUM+ehjiUtckKA6vy2kQG2LGntIW+OEkgd6d5/e+IA8OfnhphXfyT/CzYyxRPyD+TT UQKoJ9Yo/ekGnuxHNy7sBgb34pUpVj1OPwmcnlhW9XV5cVF2b9SY/HApw8rkHqe+wJujBJBvCTeI 4MFOgR1E43jIUB1l4Glb/PGTAGqo3mprZeVMqwGXB/QNzPJcKBKq66WPpwRUu7l4TcRk6M6WjEzN O5prZiuv3rU+nhJQbXLD1R6GbjwQBOB8FvriqrNnNMmnowSS33K/8yQnBmRm1EMGtLDfVh+LPp4S UA2SRG2VU4auTHuUsvSiG9ac2RRXMp9+EkCt8yaqPQxceFMRMDbwxJpZRavPRB9HCSRvAt4SSM4M qCw2iwxmbtWvi5N3RwmkBr2qtmopI2eWRDS6tJHZqjKrRvEO/vGUgGqVylDtvtCXB4wN3ZB3mQ2g +Ej08ZT/kV3uupbkNhTN+yvqD6z3I57MgA0MOpjAcOZJDNDAjAP/vrk3pTrSIRpgX95LrlKpJHLT oWphe1HNkAwd2X4gKvKBZhdMATa39RHvSHGopW2W0lF0YNOEFEIP3eiMVUdW6bw6+FekOFTF9NGe bpWX6LgS2oFOWG9kNcV1XOgzUhyqJgoQVZWRvopBCBTIznSg2WCh6HAdrXx8RYpD1UjZpH439KCs QsL8FFPck5hxAPOaiL7ixIGWXl/qneDIhElluMDWuWPk2V4T0VekOFTFMe2PzhzB0J0CFkNJPNEg s4SkNRF9BYoj6WjUzW9GblTdSOgHueE4h9HsppN8B4ojrXFuDXckp14eTH9FO8lLxsEL7IlpDURf keJQBcMH/Wlo1T1M0Ht2oAsXbfex2Jn+ihSHKl1fS/1KEU50WAntRHPVubKKbPIZKI6kPibFrtXV yPo2FQla0g4yLkvgKJfi+oZ3oDhSwehBf0aSteyuhDkX+Q9ElfxG5U5P/4v60f78/cdvz39+LLmz igyffaPEPatgNlFfz9l+dmSCFuj97F9//OWX/47nl5/P/YCfv/z9R3z++sTn3/qbqf3uf3oHnr89 //hneP716uNBcZUbH/RZ8M8fgaTVzWKCtIlYGb6wYNsiXhSSrdvKsMSCGw0/psAl6tIzt5H7b1tv FqrS0FekONRI3A31p6Hj5HZpwjzQPC5mcTaJviPFoQaql/pQ4RuNjtXHOlaG5iE3uyXJV6Q41Ij8 Q0msRkAPHt0+17kyNK+mWZQBQ1+R4lAj8HXUH3mjgyXYsTE064lZXMKNPiLFoUbgRyiZ5xfoziox wjqBRFsVNIuKa+grUhxKn4mjUzBdGhqyRRMiT9JGs3SbLbMt9BUpDtUnq4P67UVHSxj5g2bDMYve ttFHpDhURwVTv2qVMHSlCNHPXQ50/bTKuslXoDhSRwWjPxdZqyQStHkcZHZ2s3ue+YoUh1IfZWJV E0OnzIR2oK1X2qyb54s+IsWhOn5Q34ok0IjUBCujC20qymwt++xdkeJQHcunP150sIR5oKn9zNY6 X/QRKQ7VK5+52hHQ+CqaULmEjaZiNVvbPntXpDhUL1SZq4kCDeGiCY0bt9HU2WZr3zfmihSH6oX1 bLV+Q0dLwOfe6L4HhDraCz7ixIF6pp5fagVguxGjc5jbYI4zZuvca74ixaFUVGEIWRJL0WFyShmD c+dC2wxmtgVrxF+R4lA9cXKqqIYbHS2hHei0p8cW+ws+4sSBerRWaFIW4MFhcIndDeawa7alveYr Uhyqo9PQnwvdOcIONKYPunI7aFF2DH1FikOpH1VUrbHB0KmnZwaK0o3u8dm2lfqij0hxqIaurr4N O0Cj7WuCjUMbPSjgaVtdR++OFIfSoWyaP150sIT5QWMg3ba18qKPSHEo9SFg12AJtP6EhKSv9UGn tIUSV2roK1IcqnXK7qorWOhCXT4zJ61XXY5n2zb2qq9IcSj1g/m9bnS0hJE/6NqfbduML/qIFIdq jQNOVWG10JkT0Cy6yA+652fbvoTpV6Q4lKpHjGXqz4VOnNum6vQDzd5iFpXO0FekOJT6GCVVmNYX jVlTE9oHbcOp2Z7yiz4ixaGaymb4U4+/oaNGakLT1/qgrZjS9k2+AsWRmip488dLDpYwD3Kxhsit LulFH5HiUOonrbAN38jQEMaaoEPBgW5E0/Z9za9AcaSmw5SewQZpD39qAUb80Jf6gPt4tu3N1nwH igOpH8zvGxwtfhzgyWZIu+TpHSeOoz/gPuqvs3EH76um4fYubov52bZv8BUoDtRwPulbq9Et61oZ sYPhIKf5bNuHSY+vSHGoBjWrfmYzIzquBHS7jS712RY9eqHPSHEoPSKo13oWs6Eb6zlOVD7QrTzb jrBWfUeKQ+mVwZarPw1d2YWQMA+0VTqAo9X/rzhxIPVxRtb0STDOEK5uPcCT20E70tjoM1IcSisI +rmWKpxyRRf2e9QyXIOFxtZti75B9B0pDqUldZo/NjqshHmg87PMKGODzzhxIPVRQLRyFwNnFhiU 9nqAS3u2HdVu91ekOJQ2rwZ/sAYqWuUUEyqL5Ea39Gw72jp4d6Q4lPrB/L7RcSWMAz3is+1YNekr UhxKuzmKk8qGbOjI4oXmXw707M+2Y6xV35HiUCpv0BfVn4YO7JtQQ+GDHiwcy8616jtSHEr9qH4P 7O1Ep5WA5r/RlKVmZ0gbfUaKQ6neQ/NSYZnoqx5sTFCJcqDLeLad0aTHV6Q4VI28AeqPjQ4rYR5o 9hSzM6WNPiPFodSH3lOlXQw9qAchxeuBHkTTzrxWfUeKQ1XTyDp4QLMqWncOCTFQ1G705IbQzmLS 4ytSHEr9YH7f6LgSxgeN/X1tzRt9RopD6eyVdR7RSSwbWpWEtj6MakXF+Ubn8ry2rVXfkeJQOop2 86ehVVAxQe/a/KDLfF67ZM1XpDiU+hG+xsVi6LgS+oFu9Xnt2OQzUBypaIWFry9n5EIpoJuob/Uh D+6E2bnO3h0pDlUaK4P609Cqj7slzI3+A1Elv1EZGqnjv6ipf/7+47dHBdSDf3zmjRD3DPWR2FXc 5/XMxIRC4ctn/vojPPi3pSokVMe4VBQ4QC7PanJUwehq6+/cH4VBygZqN2wPaPoSBkyR3/ADtISM +avbiqr6JfDwqIxMEF8RS8bxWhseInqMWQwyhr4ixaFG4i4UXAqi4+Q29c532uiM82G2FNMhX5Hi UOrjo5bIT2xofPU+eAg2uqLHmMWMtNFHpDjUiPxDSTyYQA/dOk2YPLob3bghtKXtVV+R4lAj8HXU H3mjgyXYcTH0IJq29P6ij0hxKPXxEUrm6QW6syyMwG+y0Hq8nm0xfhn6ihSH0mfi6JTCygU0BklN iDxJGx3RdM2W2Rb6ihSHUj+Y3150tISRP+hMNC00zUYfkeJQ+n0L/KpNztB6c5Cg/eJAV24Ibd3k K1AcSQ/lMH8uspZFJGR9qw+Z+tdszWvGvSPFodSPVf3GqmPolJnQDvQgehh6vugjUhyq4wf1uxYS QyNSE7Rd1BedAnqu2Vr22bsixaE6lk9/vOhgCfNAR7QYs7XOF31EikP1ymeWwZsANL6KJlQuYaMz tsJsbfvsXZHiUB0npT4F58bQkTV5NG7cRlf0GLMYdg19RYpD9cJ6pn6fGx0tYfQPGueZpo72go84 caCeKeIrbqiB7UYMXOH8ggdqqdk695qvSHGonjj81LDmUwgtjJdjcFRZ6BwANdtCXT3rihSH6qh/ 6kcOT4aOltAONO4gTYv9BR9x4kA9shlWVHADD86AAyX+A86opGZb2mu+IsWheuBgWtF3DN2pNMbk ILvRldtBi7Jj6CtSHEp9yMqKbrnR0J0TX+uD7uhaZlupL/qIFIdq6OrqF0pjoNH2NSFSPG/0wKEz 2+o6enekOJROYtP88aKDJcwPugR2LdrWyos+IsWh1Idi1ZGvLLT+hASdSw50YteixUoNfUWKQ+nU i3Gp6goWunCempm6fKNZOcy2sVd9RYpDqR/Mx8hn6GgJGAo3uvZn2zbjiz4ixaFa40xTVVgtdP4/ 2WWPA0tuA+F8TzEnWOj/J3ZuwHDgEzhbBt77By4WpR72EAv0W75X+pqjlsgih56NydWhZ/3c50zH 7r2VElDwj7Mz3gddOKrBYCaHZm+xp1a6a02dUgIKcdZ4cZo0dKlcML5oLUL3OUt90E4pATXg7TXe OP6GzlBiAbx9/6KtmPI5L/kllEAa8NcWr4ecbMF25GYNkVvdyoN2SgkoxAUVdug3MrQaYyzQmeKL HkTzOe81fwklkEbBjxqfodZe4705NOx1hgQDz/W5zzks57dQAghxsnhecDb9cuDNZsjnsadvnQQO /kfvI/66GnfxvmKZ3t7DHbl+7nNe8EsoATT0fDK2VoMtm6iMuoPJkcv+3OdcZj1+lBJQQ90s4spm RnQ+C7TbXXTrn/vUHn3QXikBhSOi9RpnsRp6sJ7riaoOPdrnPlc6Wb+VElC4MrrliLehO7uQLtgO bZVOwdnq/49OAgixnpEzfxKsZ0ivbnfgze3gc5V10V4pAYUKov0cpUpPOdCN/V5rmV6Dg9atu0/t G0S/lRJQKKnb4nXR6SzYDl0/57HaumCvkwBCrAUElbsZuLLAaGnvDtzG5z5Xt9v9o5SAQvMaGi/W QKBhp7igs0he9Cif+1zjHLy3UgIKcbJ4XnQ+C5ZDr/y5z3Vq0o9SAgrdXIsTbEM1dGbx0ubfHHrP z32udbJ+KyWgYG+0LyLehk7sm+qG0he9WDjOc5+s30oJKMQZ8Uzs7USXs0Cb/0XTltpzp3LRXikB Bb+nzQvGsjCGHxxcAIvi0G197nNnsx4/SgmonnkDEK+LTmfBdmj2FHvuUi7aKyWgEKvfg9Nuhl70 g2rFu0Mvovnc9WT9VkpAdfPIGDzUswKNndMFOdHUXvTmhvC5m1mPH6UEFOJk8bzofBasL1r393n2 etFeKQGF2atiHsEkVg0NJ4HWp6Nagzm/6No+z3OcrN9KCSiMotPibWgYKi7AXdtfdNuf53lszY9S Agpx1hi63Aydz4Lp0KN/nue6ZC+UQGqosBrjxxm50QpgE/GrvuTFnbDnPmfvrZSAaoOVAfE2NPzx tAX7ov+nqlYfVVWPNPWPjKV///eP/3xgoD76H9/5Rkh4B2JdOGHu63ln4YJG48t3/gs/CHNjOq7n 3//4Jx2rOqmpU1MDd+kL2uf0OpphbW7n37lNYKqjTfOaYwf4649/u6z1H7XafOk/q8/P6usZfAsN 0tQ6156PsMv3I6z1oJ1SAmrpLMl4TO/isUBP8/N9eXD8mfwRSiAtLZBN/fGdxTLv3LP3RmYfPsd9 5MdqO6UE1Cr8nK/5QJsrFmyX9MjfS9r6g3ZKCailtZfxeLLWIjS1OH/RnVnn5Qrij1ICCnG1V5nj VMOobgMLusu65dN2vmX8RykBtWDj8KqRWIINrVV5Lv79q2qdMp7Gg3ZKCSjE+s6R6LS+Y83crBwX Xa4x+XbjH6UEFOJqr7KJcm92qbm5b68qvno5qZ/5wwklgFbi1/UDk1pR6K0KGDiNYy2do3oJJYBW 4okc+Qyqe7Nl46/zN+Nldji/XOBLKAGEuNqL5p2Y9Drjr/s3Y3VQOhvQqLU7MDmdBM5KvPej0KyQ q5ZxJVaBy5202uvl4l9CCSBskdYqxD0fcMVAgS+RXcI8D3P6weOlk8BB3Ow98yasDmZpU/1yOxPu ZzcO1+kkcBAnfU/laPZMjkv90JdLXzZ5TX4GxyOUAMLF0X9AbLZnq6+HvlD2OEoOBpkT6Smgb6EE EOJmL3oGUp1RoG8u48KMUz3DwQE7oQTQXNz7oTPPGUeXJTJcxjy+g83kzgQvnQQOaqqeFcRnRkLn 1jyqb1I62eqUezf6gJ1QAmhOnu7Rrmsf9OKr+hZl12KwTsxSL9gJJYAQ63VE8071DLnLEnEdSg+C +c9Ej3BmXCeUADp+Y/Tj4LfaA+pdf7KJa2QFjzvBvIQSQIi7vWjcjNWAwDC47sQLN9hBxq0TL50E zhys0YjNUG44y0y9a002ffY1zhck+CWUAIKRZDvRL3HAOv1B7xoTb3Jn1n1crJNJwCBmK8HPuPnq NAI7W7/lZzbm27nNxwC8hRJAiKu96Ayflc0eeteT6Fl61axHOWXipZPAoVVHPM8IA33f1LuWpOcW 4HKOxeE6nQQOYv2CQ33x4VbNY/iOpPceXJ6KfrFOJgFzhhbE24oPbNKm3vUjKzusmf3swksmAYNY L8hA5egHOy0N144G+2cbdijWBTuhBBDmN73SiPvNl21m+nZk1azRa/ZRL9gJJYAQN3vRvBmzz0yW rAueeiUa+4YetwN2QgkgxFo2B8yRZZytz0wW2QsezJgWqNezxy+hBNAs/N6IT0PK1mcWu8IFd824 0tf3Wy1fQgmgWfg/iOfNmH1mcdkFN/1slR5TjfYBO6EEEGJNfabbkJI1Gp0kvmAW4EpH0c6I+RZK AM1MbzPT7UjJGs3mp7ngwox5jNu6XKeTwJnZpr10GxJsXaZez9LlsrDXWniY9wU7oQQQYj3P+MP6 CITLEhkuYbbQygbdun27t1ACaCbeQMR1H3C3RPY3487jUNlIW8sX7IQSQDOxYiAeN2NtNPihWl8e MIiF7a49+TqZBAxirW+IrY3gE2ub2YnV8GKn5lvGOHeE4JdQAmhsFuRZTiOBXhvNtpnuASuXVSiV i3UyCZix2T8Q95uvthmcfP3jYm0fWCbqmhfshBJAiPUNiK2P4NprBjsznwesW1tY5etxa2+hBNBY 3BFUD2sk0OuOQb9dxlUzzpsH7lTMt1ACCLF+Q8T9ZqzfeOvk4MCacWZBrufWvYUSQGPxzCG2ZoJm oGdyF57QC2YLzfSXtZqZeAslgMbiLUFPsWYCvdo66Ne3/mgxA5hlopbLdToJHLQ9vdaI+01Yfeiu LAKXq9RaefH2xTqZBMyYLEOI1zxY9c27+pbUliaaWYPKnhfshBJAY7JuwrpYJ4HpmJaIa0ncgExz qTea3JdOAgcmSAs9YpvhoNdeuZvvSFoiUbcXwfNynU4CZwz2EcQzH66OUrv5htSGnrTEIlH6umAn lABCzD4yzgwHG7oskfmtPuYjkl29dk7ESygBBFus3R+xzXDQ60/Z3TckLb2JJ0mLxeU6nQTO6NZH xhnhoFc7A71rSE0PWWLb0FJxuE4ngYOYbWSeEQ5zybI8XD9qBd9Moekk+xJJgIxmLWSe8Q36bkl8 e5ECCePhpN/Vo5rhnvLSP7RX/f1f1eaP/qdanEpsf7npy/15XR3N+KbfF7vnX1zQZvmmxt+zt57B zdq013rQTikBdX5EXzSDhi61fH+lofXybJ70PetDdkIJJMTq0buaESOrG+ECN6VuziSbZ3KPux8v pQTUKpwqujbji9Y6hwVuTt28QZtta58x6kcpAYW426vGk3XRXJafVLVDqaFY5wUX7ZQSUIirvWpv QyuCC9ywunmLNgvVLuWgX0oJqJVtzNSLd9G9c4EbV20rFj3uTuNBO6UEFGIOmmpMLrpqLvDtLmu9 S38uuty1+4N2SgkoxNVetbvGW80J9W5m3RwiFodLTZ129CWUAFrJhk2YyX3AfVPvhlbdAfXh4yR+ wE4oAbQST+RQc3LAFYngr93UujgHLhas1fsFO6EE0Eq8RUNnJIIXr/NKfm5di9tLn7vs0L11Ejgr 8d4P9SaHy3k0+cFVu2j6cy6mXdYFO6EEELZIa9VQb3LAnEezH1ztPMz5f9KrpseNIoje91fM0UYy TH9PSwgJAgeQ4LK+BQ4Te5I42h4vtkPI74AfzKuqbrtnO18Sl9iVfX5TU11dr14uR+atcKnhQWzl OaEkzHZUZe0UXscJu1yNzFvhUsODmKTM03LCvEHcqGLhK7y8JAS+JmHI1mcBTA0RLg79wZtiW4O4 UZ2lU4i5y4KKXA5TiCtgaogQW3lQtq1B3KjO2inEmjPu6UoHrwtxBUwNEWKqvbfFtnpxozqLpxBz +3oWk7yZL3Gp4cFMZcmzxbV6caOmFqmBTaD3pdCZuAKmhgixkwf5krA2jK8kSq6F5zkR8vqxBKaG CDFdR++y24SUD5JIpVDUCHRbuZVV7rYFMDVEUHIaIIh1yMTsRk2tT4GNpedV1+eZuQSmhgixkwf5 kjH5USwMlTrxhfOsIL7MiQUuNTyIaUZ7l+1mtOySgK+kiRoMg2bw+QSZeAFMDRF2U5YTnw0n8OTr gK+EiW8ym1fvfKGtYKmhQcxS4rPdBJ58KBZkcxs/wXK+vOn6vAAsgakhQmzkQWI3YQ5I7IGvNIl3 FseLrtd5TCxwqeHBPshKErLbBJ7cKPCVJAU2gdzC1BaZt8KlhgcxnSBiV/IlMwrLVCkS3XtxbL53 hbaCpYYGMVlNxGI14RVpyQW+0iMZOzwzXa7CApYaGsR0QWA4xWlGLSrjaznyrJ/WS1MMhbgCpoYo GL7SiF3Jl2Um1HIk00z8j/OmEFfA1BAhtvKgUDJmnQm1aaWFt+dpK+2WiStgaogQ09j0MZvNqERn Qm1aveeMeQVyJtd4AUwNUdB83oizICnRmaF2rXTPaFXgsyvTcgFMDVHQ/AVxKBmzzgy1baV7BmLe MWnRzsQVMDVEiNlt9kWQehEachI3Yh7AhjcKGzPxApgaoqB4twl9UaRehCbWttVrzpjb2A6Ft8Kl hicosZt9ESSsdYrxlW2VwW6M5maOhbgCpoYIMVtOlXUEwEESqXyrZwk1LNDWZe+6AKaGKPTiXRXf RyZ2ksjNu7I3BTELqbWqEFfA1BCFnicGYl8yJqHBi9J8uRKDUbPc2Wu+FSw1NIhpviEWGcERk8zE nqdhoQ2Ur/Y+3xEmXgBTQ+QjD+Sgs5AAT0ITxdNdiYmXp1CvC20FSw2Nj6wfiF3Jl2QGnU8fhVbq wGPCDKEQV8DUECGmJyAWHcG1pwyi4nyuxFRazVPe5G1tCUwNkR+4IpgeIiTAU8WAj1XGhjJWkRsu T8wlMDVEiOkMEbuSMZ1xJOdQEVPGigeyybduCUwNkR+45xCLmEAMqCej5g4txCyhivdLY2SZWAJT Q+QHviXQFBET4GmtA364zR8aZiDmMWF04a1wqeGB7NG1RuxKwrSHRsNDoPASqzF88WKhrWCpofGB xxDiIWRa2pujqSXJDpSo4hmkYyjEFTA1RD7w3MTqIkqCpSNIIpUkcQEUL5d0o5l3gUsND5YgGvSI xcMBT1oZba1INCIxtwcmDoW3wqWGx3vWEcRBZV6yUtHWgmQ9dVrPQ0K7oRBXwNQQIWYd8dnDYQ0d JJFwmz6yR/Ry9WzuiAUwNURYi0n9EYuHA55eJbpakGj09txJNCwKb4VLDY93oiM+WzjgaZ0BvhIk S03Ws2zQqMi8FS41PIhZRkK2cPAlg+RR6ZHVODMi7XOyC1BqSLwVCQnZvgHvJImbFhFhd//stztu Tt53qVUVtic10Adp1Wm6u7/7YXv3zXZrO9VtX+J3htYudMumfOnpfpLWKINR2ONn28SWQ5xt+ZcS 77vtDn/qe1zE7bu756vv9/vD5bBG8VfHeXzo0rR7Pc6Hczqv/9j+Qlnm2eizTGzwa8uDgj+2P36e BMmHkrwm2zd0m/zZd8pj8/BB04qjIqpHuXNtcr7v7r7tezd8t31DRP4TVbhSoSHBRdkledlwfW0t r72d/r6sAVp1u2NK47z/QKY9ZYgHlC94QIy0r6pOaXRV7HOyxP3RPDWJFOzqpnypaSyuux3cNc9I XFRh9BnnOc3ji4epexxPY1rDGa6mNVbW1WU6dfP06nhBzdXqMHLxFYp/7vAu3V/TvD+eunyCG2xI A7oRR8UP8deHYHTyU97Ohz/fTh3VBI3Nz7Cr+XwA3+cPUNMiFjAnFwf4yZp84OyuLE/PDtcGXM9X v46Pjwcc2bCaX33JWUFqvu4hgc1RfekpXQmeHtKtfuUSjXM3PhzG82F+1SXkGFflAnQvcQ7ny+n2 l5GLi3fB/xzmy1FOaftV0wBWuIfuxfvL1P0zH0/SASN3wMO/64120NbV/bP7n7txvz9N5/P0RW1s MPz0gMH0/9r4RvO0Qu76FloR4wqv+WLq3p6nPRfk8vpw2lNTX95385ioFNS2426Hd+jWeDTVg3+5 O86X0/FhLel8vAuthVmkoj3twvptPjVCrwR1Ayqe+uQhy+jjm1mOn4cJz5LrGOEXSaOcbxrf03tT 8j9t76xTqJfpHK/YmFJfw6FuLEmyp0H/kgb9h3MbNHZTlLXO7ZqCTDYllT5Nb6bdZdr/vpqPl+6Q Hh+mNM30H+uSBlYDLKXWQTOgl5KFMZAu5MRp/DcA+huQSgplbmRzdHJlYW0NZW5kb2JqDTQwIDAg b2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1MSAwIFIgDS9SZXNvdXJjZXMgNDEgMCBSIA0v Q29udGVudHMgNDIgMCBSIA0vUm90YXRlIDkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0v Q3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNNDEgMCBvYmoNPDwgDS9Qcm9jU2V0 IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQ0IDgzIDAgUiAvVFQ2IDg3IDAgUiAvVFQ3IDQz IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDk1IDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0Nz OCA2OCAwIFIgPj4gDT4+IA1lbmRvYmoNNDIgMCBvYmoNPDwgL0xlbmd0aCAxNTc1MSAvRmlsdGVy IC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZRXzc4kqRG891P02ZJZfpK/sw8+Wz74AVZey9Ky kj3vLzkyga7KogUejQTqb8iIKEiSyF/+8qO8f/3xtqZYh7H69HbvH7/+8frlr39373/9eLn3v9+v QIZCeWdKpmKOxqX3n7OrptL7v/98/fYK1cSabwt80QuITMl5g8AKolIQrbFhFxLxp5oOmFVhJmtC TBudKZoU4h7TBcYMAzJbk+suIkdTIx0g0x2yOOMcbVQiwquAZEI9UHiSCBoh1ZkYwyGk6JBksgt7 WcGrEGedqdXvaUL/+PKJycbFQwzpEOdNcO4QIl9D7hOTDWW3/5zodYz3JscDT5SUo/iJyaYeQlLQ ISEYe7gsKUtI/YQU433df02OOoaCibbueYpIi+ETU0xKu1LAMaIt5hkTg+TSNqbGR0wxtewqClZb OZ30OZ1Exqf9vlWbHjHVkD/EOMm29DmeTCaWXRXhktO1ffY6V5zZvlhxmVIxJRrr98WIrxrG/Dmf ag3iDjHlERNNoH294eTkmHk+3lqT/L7kcKbpmGhyOcREybd5sb1jmH2Z4qRRIck4uy9TfP4cMsuU 986Esi85fP46JplIpxjZgfLRFpzJdl9z+Fx0DNaWfQHhPVYx5I2jQ4wkAWrzDMnmsM+96F4R0Rva 3QIfs8lhX19YswJNHrXg/4tJn5hibNzXF2ZQMTkY78pOfS6G6gk1adQSTIq7auJLwWu/XVEDznO3 r1y7Bl+txsddVQkW1s0d6sHjK4KFm8txoyA42Lmtpwke7s0d7vrjTILHqrxFDfBv4XC3H9kRAiyd O6W2zsJAMHX5cLcf1yEQfF7YxYTojK2H2/8QEuHs8s9d/pBg7cIpJjxikkl79/CsZZBl6uHhfJTM kOHs/OFyP0ozyjLEHGL0CxAKnB3t7vbyzoQKY3ewKM/nLNQMq3OKcSqGLJwdHXbt8TyThbWzByv0 sAHkgqHyc3aDHKwdnWK0rSEfpG/7GftEvsivvbXTNo3QgXo68DzsIKEpJXvoih62k9CnxnRovR72 lgjW7uBRnjaa0Ltae4h52HVC8+rSoS96tAWEfjZ4v/+eR/dB6GiTPfA8mhxCk5vTqTHSzRQVzqef a9qQ08bub/ajNSQ0umGf1M8GlNDoRnfoix6NbkSjm/Kh//LlEYO1B99SPOkYdLrOnWIk5BORDbzM PsIlFYI2l8K+4hQXdAi83tbXxABnlw8nYSWt/Hy7I7pauz29iB4W/cteqo0aFT0s5V0xiehYU9iX m2KdRkXHmuuu3MQEZxd3xSWiG/VhVxb+88J7yDTYmYrnC1NGJcJU8MhgyT/ef7zsG/9wgr27gy7L TxuSGuTtBY9aAeBNxOX7/YNnEn44+I3KTydOE4t/f/32p9ffmBSW4kZapfzBJNkbKfweZ1EfCeo7 uwJsmm7AwkwKN2oILy2ySxc3PN/FTc445nZI6jt34NPoIwry4FaATdMN2P7VqPpYiD+5O3OuyK2L 2RvYMZ4IaXwxE5fcPhKUdWYF2O5kAxSXUpiDCbwQRUoxF9zyixmPOW4BphjvzFK4+4hCP5gVYNN0 AxbpL9zcJGIpX/Y7d5Ine3ITbkLiKSvuzLWsj5TnVyvApukGbB15ZiUtuF9S3DixG3eU2szGWnFX rol9JJT7zq0Am6YbsM72RMODzXlRkAiKHIeGVmmSZyk9mNyd3EmJ6iPBpwu5BmwPvoHr5MBReyQz 8I4pbhzbjRuOj7mLZNzF7bmQ9XEct4Zrimtgwgp25twzI0uJuZj5QbmYa0+1qlPNiY/tY8QF79wK sD34Bi46Rd83PRkP31iQDYqeDeKHHqfI2YZJZRufwByjH3VNA7Yn4QB2EflZRQAsF69HStwFoOTm mwAnKYdJpZzLIkBG7q26AAXYFsaB7JI4A1YAs4yAyvbhrqDIYzMVeKBAQXjkXZUTkDHO0q4B28I4 kOEAylDgTeIAj3S5K0DDdFNAspGYPP74UcA9yBxjnHugANvCOJDxgtmhwKFWIQCf5+4KkpBNBVEy CRMpBZ5LXR9jHEVeA7aFcSB7BA4FVvKmEjjuCnCaNwVJ7hGmpBRIb9bHWes1XlsIB7B3MxHxfHLe 1AiKuwAc5k1AlqPEVJSAyFWvjzGPN1YDtoVxIMPVjUQMuedNAsddgZfbNxUUSedYBexSIFkgSVBG IdBwbeEbuNwrDn6sLAjIYLjzOyk+g5/bHfAn+0jDwuWvj/yadwUKsC2MA9kTrnVXgL3igPpIQyuF dypwkoYSfFMQxPn1MdlRiDVgWxgHsk+41l0B8gVtmgWSVOuhQLb7UuAlDeVz7grEY/UxuVEMNGBb GAcyjHqIUpEDW2cEwBUpBVlq31QA0wF/jYnT8VIg/U8fkx/FQAO2hXEge7xZQwHKLAfgK+tdQZLy PxXAerACknS8FEhr2Eeuv12BAmwL40D2qJNDAV4aDsDnKQVR3r+pIIpjxsV2SoE4rz4mmnugANvC OJGttH5QgP+JHIDPUwpI3v6pIIuLwxQc3RRktiF9TDSqgQZsC+NExnM5FJSeOPg8paC/QFNB6Q4W O6kUVL6HfUyjHGi8thBO4DAT0eeeN/g6JcCpRKxy3eWJuwkQSyJDSnMDFFxb+CYuSZfC/HywCCiP NLSzt2L+bHt7Z8VXXvxe2h0ZU547oADbwjiRIzK3K+DkRkDVaSg27FLgJA3F59wV9C3oezDtsAZs C+NETqjUXQFf8LeDZVVpKBb0UhB6MxF0GnLbM8dUR0HUgG1hvJDtUMC1jiO82JxLQpZWb0qAK8nE E6fjJSFLFyRjtqMaPBDbwjmhKyplFg1c8TkCHjbfNSSc3Gdhpn4QsPv5rqFINyRjdqMePBDbwglo i99wETQ08LvHEciWLxrGwhx6OqLfuWvgwjvHPJ3yA7EtnMRfgt8O2SEaHL/9HIF8+aahL8xeriQm Uhq6O5AxDwkPwLZQEn8IfnukR5eQZRMdzGz5JqEvxJ3gPMDklQQK7znmMOrCA7EtnOSll4U0OzSw /eMIJMw3DX1htnIdMFmlIdb3HPO4FA/AtlCSk+xAawG5IoE9MEfUrxk5FqYqNSEVScxLQu4+kUTC 3AaN2BZOsj07Ej6ra+BOABHefs/IvjBlqYyYstIguyCbMB+IB15bGHFBC8wtrkocCgL2hyPc93zs C1OStwFTvCtgjzbH/ElIjdgWzsDfgd8FmF0Dt4Yc4b8nZF+YYndsUR7FS4N37znmMrNBI7aFM/CX 4HfFznYN7HX8m4sr0BcJfV2iblvRxikJQeqzjLnObVCAbWEM/B3oBRxyI7ECvCgkAciWVcFYl0K3 7mhllYIodlHGYkeB1IBtYQz8Gfjtxe+ygoLs4QBkyxcFfR3sd+3WPSsFqbznWKZx1oBtYQy+90Nw wkNBRu5wAHLli4K+LllxzMmKf74UFHrPsfghQOG1hTA4SQv4qFi7ABjSyAGFK8sioK+D++amAVO8 u9ZUxS3KWPyoCRqwLYzBSlrASuahAKZcAuq3RBzr0EFy44Qp3BXwtsxx+hWN1xZC/F2yImFnugBs DgdA6bc87Otg/7l3xKSMe3cJQejnBii4tvBhYyQneOr8yA4J8F+zsK9DuGX+qFuXTOwW+1jiKAYa sC2M7Gpd4eMMQwFL5YAAylVBX4cTLKyAdPuWkxgEGUua90ABtoXR82fUntH8u9aeNLgvaRXwP7ar XgeP3Ab2eYp9Ap/+f2p3AXKNy8N1qYKoCK64188MSe2upIUBrj9bmhlJI4q0YTBxiWV6+RFQJQ1J bPY6rnjj4AtcRNc7Lfw0a8T4DCP3U4COwzVOsczb/Ajo8ZqxNd2BDXAcjIGr6JrXRAEvLCfgAn8p 0HFIZSGWmdFuBc3JFdTYNRtugONgDFxGn7kdCpizOAE57EuBjkP176BAsvpbQdAsxNidJqMNcByM gcvofOCC5J7Ov3AC8viXAh2HJ40dED4tvhUkUSBxbsGKNw5C39UWwFcBqJ04IfF9/RCg4/CuVwqo YsdHgFQmGrvXZLABjoPRN7UFsFQB329O8N9G1HH4lamgiB8fBTVfM9qrvOGNg9BzFZ3lXlMBLGI4 IX77UMehwIsUkMWOj4Durxm71cob4DgYPZG71ryiQJsvj0IWL/qhwMbxf6kgiR1vBd2V647JksEK OA5Gz2V0rfz5u3X1DVf5pUDHod4WH4bVhz0wE1rMegob4DgYUcY6635EQDPb4Jy/BOg4nGnNd9/z CIjtumPRLdgAx06ItrZZByj8xVwDp7t28uu4xJbkbv0e/hyvO1qNvAGOnRDla8HPZJZoWQp/z6v+ xa/jeK2R9fEJC3+RG6hRPbjhjZ0PpSvyRmXlkkifpOr3yHX+i17HMa2RHq9qftNLPWaxaybcAMdO iBXgF95IpY/SiHpk+fhFr+OY10mPsuKhRwdwae7bQMbOIcePakp/BWRkDg6wx83oLv6R9xyJk01m Zd5JAAN6QyvI1odN6H+pjVW+DZAtZBdpRySA/yMNjyE2qUljE/tKs/TsHzlZMwgtKhaHuvJNy6YU KoNX2YL3Q4oYFliowSqzc3p2rXnZ+0naZbusS7wPzXkmDI3JEtcGOFY6g4WDhBsG49BmBzq5nRyC cScvTklefPNwR6ZLjSl1414Ax0pnsLpqWAsD8U/+zVy7nOhkDpqued1fqcol1kwaE5Qp8wI43mQG ikpYmOEwDvRmosnc9OSNOcoTlaI8WA9zYXbQmCxHbYBjpTNYtAHCDcNyKAuyNzdd+HAneaBTkuf6 4a7MjBppTuVeAMdKZ7DdfObEFo3Wf3PjxF7cWV6FlOWNeLg704LGNBuWFXCsdAaLhCFGQ91FXzTW 4m/yJJ3ZJK9ao1V5Im9yL/WhxmSZaQMcG5/hejlwZE9xBjuRN3eUnmxyNykJUhPHPdyBRYlGO+4V bixchon0osxVncEe7NUe4PnyL+auVuur1VD6XDOyb1DuBXBsfIaLNzLoppcfAc9JYw/6pnfSixk9 TlG6E7e6jScwYw6W11bAsRMaMIqEVLsIyLgrGA9LvAWULq3YFODFcvgslvNVBEjM0ZuABXAcjIaM QqmYAiR8TEAp1RYFDQCPgiBNAe7x6jupBzXmmdpXwHEwGjKKxWYKwo/CCQF2eSuo2MNHQZKNxCfE V48aHFOcxpznHiyA42A0ZFTszhR45CpMwPL8W0ERsqkgi5PwSYuCwFSnMWdL8ivgOBgNOWCiKXDi G3QVcVGA03wpKHKP8CmLAikINc5cv+KNg9CA8fqbEfF80jdorPIiAIf5ElDlKPFpi4DMrKcxV3tj V8BxMBoyOkczIsoe8U0Bx1tBkNs3FTSxc+4C9igQF4gJmiWCFW4cfIYb4rQh6q3aMKGC4c3vJfkY f3Fiw+I2GzamP418zVXBAjgORkMOCddaFWCvOKFvNnSSeKcCLzaUyS8F0fESaizOEvEKOA5GQw4F 11oVwC8dTwSQJFubAtnuR0EQG8py3gqkxtJYvCWDFXAcjIYckLWzZOSIK8MJqIoWBVVy31SAoqNn fmjHR0Fka6axBEsGK+A4GA054M0yBUiznIBV9reCIul/KkDpQQVJ7PgoSLyEGpl/VcECOA5GQw7I k6YALw0nYHmLgizv31SQpWLGxfaLAqm8NJY092ABHAfjRGZzJwrwP5kTkjQ6j4Ikb/9UUKWKwyei rngUVJYhGkuybLACjoNxIuO5NAVNjYPlLQr0BZoKmlaw2MlFQec91FgsHax44yCcwHEaMVT1DVa3 CPCLEbtcd3niXgKkJJFQytyABW4cfBM3SZdCfh4sJrTNhm72VuSvThoXeeTf/EHaHYmlzh1YAMfB OJEznKsKaG5M6KsNpQx7FHixodQ5bwW6BboHsxxeAcfBOJELMrUq4AVHL+pWG0oJ+iiI2kzE1YZs e2Ys3RLiCjgOxgfZmQLmOs4IUuY8Eqq0elMCqpKa+KEdHwlVuiCJ1Vk22BDHwTmhOzJlFQ3M+JyB Gra+NRSc3D2wJj0IlPv1raFJNySxessHG+I4OAHt8BtVRDINfPc4A2750GADa1Q7ot95a2DinbHO SnlDHAdn4krw28MdosHz7ecM+OVLgw6sQa4kPmnRoNWBxGoSNsBxUCYuBL8D7KESqmyiRzHbviTo QNwJ+gCfsEhI8ZqxRssLG+I4OFOQXhbSnGlg+ccZMMyXBh1YnVwHfNyiIfdrxmqXYgMcB2Xy4g60 FpArElgDc0b/dKQNLF1yQmlizEdC1ToxiYS5DSviODiTU3cULEs1sBPAjOC+HakDS5XMiE9dNMgu yCbMB2LDGwcjLmhDcYurkk1BxP5whv/2ow4sRd4GfPJbAWu0GettyBVxHJyR68DvBkzVwNaQM8K3 IXVgyVqxZXkUHw3BXzPWNt2wIo6DM3Il+N2xs6qBtU64mFyBfkjQcSVp2Yo2bpEQJT9LrH1uwwI4 DsbIdaAX8PBGoQK8KEkmwC2nAhtXopbuaGUXBVnKRYnNWYJcAcfBGLkM/A5S71JBg3s4AW75UKDj UH53Ld3roqC0a8Y2C+cVcByMMWg/hErYFFR4hxPglQ8FOq44qZiLk/r5UdDSNWMLJmDBGwdh9GIL 1FG5qwAUpJkTGjPLIUDHofpm04BPfletpUu1KLEFywkr4DgYoxNboJSspgBFuUzoX0a0cegg2Tjh E98KuC0zznplxRsHIf5dXFGwMyoAm8MJUPrlQx2H8p+9Iz5L4a5VQhT6uQEL3Dj4sDHiCX6UH+6Q CeHThToO0x3589q61MRqUWPLlgxWwHEwsqr1jccZTQGlckIE5alAx+EEGxWktX2rRQoEia3Me7AA joMxcBldHc3fvatpcF/KKcCGwcQllunlR0CVNCSx2eu44o2DL3ARXe+08NOsEeMzjNxPAToO1zjF Mm/zI6DHa8bWdAc2wHEwBq6ia14TBbywnIAL/KVAxyGVhVhmRrsVNCdXUGPXbLgBjoMxcBl95nYo YM7iBOSwLwU6DtW/gwLJ6m8FQbMQY3eajDbAcTAGLqPzgQuSezr/wgnI418KdByeNHZA+LT4VpBE gcS5BSveOAh9V1sAXwWgduKExPf1Q4COw7teKaCKHR8BUplo7F6TwQY4Dkbf1BbAUgV8vznBfxtR x+FXpoIifnwU1HzNaK/yhjcOQs9VdJZ7TQWwiOGE+O1DHYcCL1JAFjs+Arq/ZuxWK2+A42D0RO5a 84oCbb48Clm86IcCG8f/pYIkdrwVdFeuOyZLBivgOBg9l9G18ufv1tU3XOWXAh2Helt8GFYf9sBM aDHrKWyA42BEGeus+xEBzWyDc/4SoONwpjXffc8jILbrjkW3YAMcOyHa2mYdoPAXcw2c7trJr+MS W5K79Xv4c7zuaDXyBjh2QpSvBT+TWaJlKfw9r/oXv47jtUbWxycs/EVuoEb14IY3dj6UrsgblZVL In2Sqt8j1/kveh3HtEZ6vKr5TS/1mMWumXADHDshVoBfeCOVPkoj6pHl4xe9jmNeJz3KioceHcCl uW8DGTuHHD+qKf0VkJE5OMAeJ2NsUkTWLJ2hfYzxt59/tevnr8vxDx/7Xz9/h4x/QsZ/7A7+fXl3 /ev64093/fsf7JLYr1amsARd/+e8bHJ0x20oOq9V1BKsf3kZQQZZQc8eA3TvfxDeS8qWzFnQgL9m PfJYlijyUhc6darEFIV59g9eioHBHXgaeJ+f9p+ff3NpE92srqXpz2ULxZH8n0s7XyTfpdqb3x3e /rmP7Us0vt3DzjGrrfcj0V5Dqh4C/s5dthx5s8Y15OkpAaUH1s0ehta+Mi1gbuhVAt6L+HGUQFL7 gq1p85A1iZArdSNT8t7Vblp60JunBNRMTDctmMlH0KkbqwE3s2+hu3WPulXwj6cE1NTlmz0cPfRa MGBsaKtJqfoLDH14SkCpjSuj7as86KyJqn8vG7pa55tb8/94SkANzKvVepihO+/ATEzEheZWQFha YzP04SkBNaCOaPcHfVnA2NB5qfZXN348JaAGZtxqvd/QTd+tAVB3LzpVH4qgSG5HH54SUAPSgPZ0 NFQzA+4NfXHVxdY+HH14SkCNzn+gclrohIBCP0fPmwoqcUNaftCbpwSU2vicOtgIgEZt0oDKr1vo aVPC2EbV01ECSUvtMHs4GZ/HgLmRKfjGTK7ADX14SkB5Fa9oAQudNAVmYyItNFPDh7tUHvTmKQGl bbXoPuqIcA1DJ62IGtC9+Ri6tWdyxa4Y+vCUgFLBMMzujrYCqgFjQ3MIYIkac1WQw1ECSW0fb/JD zkMDBoeYRS5cbuLSn5J6eEpAucpqlw9eKkL1pRpgomyhOTt1JuBoXkFOTwkoFZA4/ZZ8rkXAZQF9 Qyesuk/f70XeHCWQXJo2nLWRUdY1wKTsIl9Ybu++30Y+HCWQVHZ3s4eTkZMMmC/ZLkqv3I7k9eP0 lIDqOl1pWjVcq4VWUfOrE0C7X/TkojOa17jKg948JaDULrBRCwyN668BWoE39OCqWT/6KqinpwSU TkLD7OFo6EIGjA3dcYiQkfjodYqHpwRUhzhVW+eOB521cOpYWDZ0Q2q0Mf0sF3rzlIDSCbIWtTsH MKDRIDRAsz+9aN7vxjLSV0E9PSWguk4LZvcHfVnA2NCccFoluowHvXlKQKmdYQ8OrkCjGWuAFpoN nXGMjc+evYKcnhJQvWmq0MZYCDQkhAY0XdKLTki7llBR+7Xy+vCUgOpVi7naUEELnSygb+gL662T 23LPB715SkB1fI7a0G6GTlzdja970P3GqitLapvrGA9PCaheWCnVHo7GtWbA3NBcb23c8XWKh6ME UtfmpoWyY0mLnBAw9Ks2MmU/t6O1h7w5SiB1pHv/7ZD2sG+IefVH8r9gI1M8If9APh0lgHpijdKf buDJfnTjwm5gcC9emWLV4/STwOmJZVVflxcXZfdGjckPlzKsTO5x6gu8OUoA+ZZwgwge7BTYQTSO hwzVUQaetsUfPwmghuqttlZW3XmCLw/oG5jluVAkVNdLH08JqHZz8ZqIydCdLRmZmnc018xWXr1r fTwloNrkhqs9DN14IAjA+Sz0xVVnz2iST0cJJL/lfudJTgzIzKiHDGhhv60+Fn08JaAaJInaKqcM XZn2KGXpRTesObMprmQ+/SSAWudNVHsYuPCmImBs4Ik1s4pWn4k+jhJI3gS8JZCcGVBZbBYZzNyq Xxcn744SSA16VW3VUkbOLIlodGkjs1VlVo3iHfzjKQHVKpWh2n2hLw8YG7oh7zIbQPGR6OMpAdUq 24tqhmzoxPYDUVE2NLtgvvAs3Q/x9JSAcm3jSkfRF5smpBB66EIXrDqxShfv4B9PCaiG6aP/Dqu8 RCcP6Bs6Y72J1RTX0dG7pwRUyxQgqioTbRWDECiQnXlDs8FC0eE6Wvn4eEpAtUTZpPYw9KSsQsD9 FlPck1SQgMUnoo+fBJDrdVfvBCcG3FSGDrbOnRJz2yeij6cEVEOajl+dOS5DDwpYDCVpR4PMEpJ9 Ivo4SiDpaDTM7kbuVN0IGBu5I52v2e2mk3w6SiD5OOfDHcl51F9Mf1U7yUNG4l3sidkHoo+nBFTF 8EH7NrTqHgboPdvQlYu2+1gtpz+eElB16Gep3SjCib48oO9orro0VpFF3h0lkNTGpDi0uhpZv6Yh QEvaRsZluTjK5eRneDpKIFWMHrTvRLKWXQ+4byf/Da9aHq8yaOlP0kP756+f//z+98fljhcZvvtE SXhXxWyitubZendigBbo9e5//VyAErhE76RiKp3R7yrUO/3iP+q4nHhK+jIcmgA+9X8KVNiwlyFR 6sVwVYj5YsaMwb/7lqIh63NWPl0xn54SUDPzndVmVEUnqBAGzA2NG3d3pMHtGuPjKIE0MxOlJl5E IydN0oFMfMlU4TdT+O7pQW+eElAzMVNqZn0BerIfjdszxdC8cnfhfrjI+HhKQM2LiVAxwxgaKoQB Y0M3rjpVf4GhD08JqHkxZSomr4XO2o8mStWL5pWbtuOuMz6eElD6zqrLqJgXDY0RUQO0GaQXza2Y k2t3ofHxlIDSQ5hm9wd9WcDY0BwfJuXGvNOD3jwloAZal9pNfwwNJaIB6GQvmrs8WfSnC42PpwTU QMOlPR2tAjUx4N7Q7CGz2Np9XDs9JaC8JHiBMHRCQKGfo+eNyzITN8SFxsdTAmp0fo7WveRoKBEN qPy6heYB4pJhW5x8OEogjcYzUHs4GZ/HgLmRBxY9JjMwL/ThKQE1GjNHG0x50ElTYDYm0kIzNUbP visLvXlKQHkj9LYIdGK11caJ9F9ojlWjdt8VQx+eElCjcGBRuzvaCujsvLQLXZF7LFFjrgpyOEog jcIhq11sXkbGFDYHh7JF5sCGKUGfT0k9PCWgXCe5alL0dXN2nJOj5EJnLLozAYdLjY+nBJSqO5x+ w7Yt9GUBfUOzYVPiYr8XeXOUQBoYQtXGWRsZZV0DMJO+ZM6Cvft+G/lwlEBSFd3NHk4e1GjzpmRz sl2UXrkdyevH6SkB5drfJwFDQ1jeF3XmQnPM7BnNa1zlQW+eElBqF9ioBYbuVMM64aQNPbhq1o++ CurpKQGlc9Ywezi6UcNrwNjQFKEQa/jodYqHpwSU2hgrdFYsDxpzh458ZUM3pEYb089yoTdPCSif aX3CBbpyWroLh6eF5v22gbavgnp6SkB1Fe9m9wd9WcDY0IWrpiRFZ1rozVMCSm0Mnm1wDAW6cDK9 Kyephc44xsZnz15BTk8JqKUdoScMDQmhASq9y4tOSDsbDfu18vrwlIDqqv9NlrYHnSygb+gL67UJ rt3zQW+eElAdn6M2tJuhE1d34+sedL+x6sqS2uY6xsNTAqoXVkq1h6NxrRkwNzTXWxt3fJ3i4SiB 1DPnoY4lLXJCgOr8tpEBtixp7SFvjhJIHenef3viAPDn54aYV38k/ws2MsUT8g/k01ECqCfWKP3p Bp7sRzcu7AYG9+KVKVY9Tj8JnJ5YVvV1eXFRdm/UmPxwKcPK5B6nvsCbowSQbwk3iODBToEdRON4 yFAdZeBpW/zxkwBqqN5qa2XlTKsBlwf0DczyXCgSquulj6cEVLu5eE3EZOjOloxMzTuaa2Yrr961 Pp4SUG1yw9Uehm48EATgfBb64qqzZzTJp6MEkt9yv/MkJwZkZtRDBrSw31Yfiz6eElANkkRtlVOG rkx7lLL0ohvWnNkUVzKffhJArfMmqj0MXHhTETA28MSaWUWrz0QfRwkkbwLeEkjODKgsNosMZm7V r4uTd0cJpAa9qrZqKSNnlkQ0urSR2aoyq0bxDv7xlIBqlcpQ7b7QlweMDd2Qd5kNoPhI9PGU/5Fd 7rqW5DYUzfsr6g+s9yOezIANDDqYwHDmSQzQwIwD/765N6U60iEaYF/eS65SqSRy06FqYXtRzZAM Hdl+ICrygWYXTAE2t/UR70hxqKVtltJRdGDThBRCD93ojFVHVum8OvhXpDhUxfTRnm6Vl+i4EtqB TlhvZDXFdVzoM1IcqiYKEFWVkb6KQQgUyM50oNlgoehwHa18fEWKQ9VI2aR+N/SgrELC/BRT3JOY cQDzmoi+4sSBll5f6p3gyIRJZbjA1rlj5NleE9FXpDhUxTHtj84cwdCdAhZDSTzRILOEpDURfQWK I+lo1M1vRm5U3UjoB7nhOIfR7KaTfAeKI61xbg13JKdeHkx/RTvJS8bBC+yJaQ1EX5HiUAXDB/1p aNU9TNB7dqALF233sdiZ/ooUhypdX0v9ShFOdFgJ7URz1bmyimzyGSiOpD4mxa7V1cj6NhUJWtIO Mi5L4CiX4vqGd6A4UsHoQX9GkrXsroQ5F/kPRJX8RuVOT/+L+tH+/P3Hb89/fiy5s4oMn32jxD2r YDZRX8/ZfnZkghbo/exff/zll/+O55efz/2An7/8/Ud8/vrE59/6m6n97n96B56/Pf/4Z3j+9erj QXGVGx/0WfDPH4Gk1c1igrSJWBm+sGDbIl4Ukq3byrDEghsNP6bAJerSM7eR+29bbxaq0tBXpDjU SNwN9aeh4+R2acI80DwuZnE2ib4jxaEGqpf6UOEbjY7VxzpWhuYhN7slyVekONSI/ENJrEZADx7d Pte5MjSvplmUAUNfkeJQI/B11B95o4Ml2LExNOuJWVzCjT4ixaFG4EcomecX6M4qMcI6gURbFTSL imvoK1IcSp+Jo1MwXRoaskUTIk/SRrN0my2zLfQVKQ7VJ6uD+u1FR0sY+YNmwzGL3rbRR6Q4VEcF U79qlTB0pQjRz10OdP20yrrJV6A4UkcFoz8XWaskErR5HGR2drN7nvmKFIdSH2ViVRNDp8yEdqCt V9qsm+eLPiLFoTp+UN+KJNCI1AQrowttKspsLfvsXZHiUB3Lpz9edLCEeaCp/czWOl/0ESkO1Suf udoR0PgqmlC5hI2mYjVb2z57V6Q4VC9UmauJAg3hogmNG7fR1Nlma9835ooUh+qF9Wy1fkNHS8Dn 3ui+B4Q62gs+4sSBeqaeX2oFYLsRo3OY22COM2br3Gu+IsWhVFRhCFkSS9FhckoZg3PnQtsMZrYF a8RfkeJQPXFyqqiGGx0toR3otKfHFvsLPuLEgXq0VmhSFuDBYXCJ3Q3msGu2pb3mK1IcqqPT0J8L 3TnCDjSmD7pyO2hRdgx9RYpDqR9VVK2xwdCpp2cGitKN7vHZtpX6oo9IcaiGrq6+DTtAo+1rgo1D Gz0o4GlbXUfvjhSH0qFsmj9edLCE+UFjIN22tfKij0hxKPUhYNdgCbT+hISkr/VBp7SFEldq6CtS HKp1yu6qK1joQl0+MyetV12OZ9s29qqvSHEo9YP5vW50tISRP+jan23bjC/6iBSHao0DTlVhtdCZ E9AsusgPuudn276E6VekOJSqR4xl6s+FTpzbpur0A83eYhaVztBXpDiU+hglVZjWF41ZUxPaB23D qdme8os+IsWhmspm+FOPv6GjRmpC09f6oK2Y0vZNvgLFkZoqePPHSw6WMA9ysYbIrS7pRR+R4lDq J62wDd/I0BDGmqBDwYFuRNP2fc2vQHGkpsOUnsEGaQ9/agFG/NCX+oD7eLbtzdZ8B4oDqR/M7xsc LX4c4MlmSLvk6R0njqM/4D7qr7NxB++rpuH2Lm6L+dm2b/AVKA7UcD7pW6vRLetaGbGD4SCn+Wzb h0mPr0hxqAY1q35mMyM6rgR0u40u9dkWPXqhz0hxKD0iqNd6FrOhG+s5TlQ+0K08246wVn1HikPp lcGWqz8NXdmFkDAPtFU6gKPV/684cSD1cUbW9EkwzhCubj3Ak9tBO9LY6DNSHEorCPq5liqcckUX 9nvUMlyDhcbWbYu+QfQdKQ6lJXWaPzY6rIR5oPOzzChjg884cSD1UUC0chcDZxYYlPZ6gEt7th3V bvdXpDiUNq8Gf7AGKlrlFBMqi+RGt/RsO9o6eHekOJT6wfy+0XEljAM94rPtWDXpK1IcSrs5ipPK hmzoyOKF5l8O9OzPtmOsVd+R4lAqb9AX1Z+GDuybUEPhgx4sHMvOteo7UhxK/ah+D+ztRKeVgOa/ 0ZSlZmdIG31GikOp3kPzUmGZ6KsebExQiXKgy3i2ndGkx1ekOFSNvAHqj40OK2EeaPYUszOljT4j xaHUh95TpV0MPagHIcXrgR5E0868Vn1HikNV08g6eECzKlp3DgkxUNRu9OSG0M5i0uMrUhxK/WB+ 3+i4EsYHjf19bc0bfUaKQ+nslXUe0UksG1qVhLY+jGpFxflG5/K8tq1V35HiUDqKdvOnoVVQMUHv 2vygy3xeu2TNV6Q4lPoRvsbFYui4EvqBbvV57djkM1AcqWiFha8vZ+RCKaCbqG/1IQ/uhNm5zt4d KQ5VGiuD+tPQqo+7JcyN/gNRJb9RGRqp47+oqX/+/uO3RwXUg3985o0Q9wz1kdhV3Of1zMSEQuHL Z/76Izz4t6UqJFTHuFQUOEAuz2pyVMHoauvv3B+FQcoGajdsD2j6EgZMkd/wA7SEjPmr24qq+iXw 8KiMTBBfEUvG8VobHiJ6jFkMMoa+IsWhRuIuFFwKouPkNvXOd9rojPNhthTTIV+R4lDq46OWyE9s aHz1PngINrqix5jFjLTRR6Q41Ij8Q0k8mEAP3TpNmDy6G924IbSl7VVfkeJQI/B11B95o4Ml2HEx 9CCatvT+oo9IcSj18RFK5ukFurMsjMBvstB6vJ5tMX4Z+ooUh9Jn4uiUwsoFNAZJTYg8SRsd0XTN ltkW+ooUh1I/mN9edLSEkT/oTDQtNM1GH5HiUPp9C/yqTc7QenOQoP3iQFduCG3d5CtQHEkP5TB/ LrKWRSRkfasPmfrXbM1rxr0jxaHUj1X9xqpj6JSZ0A70IHoYer7oI1IcquMH9bsWEkMjUhO0XdQX nQJ6rtla9tm7IsWhOpZPf7zoYAnzQEe0GLO1zhd9RIpD9cpnlsGbADS+iiZULmGjM7bCbG377F2R 4lAdJ6U+BefG0JE1eTRu3EZX9BizGHYNfUWKQ/XCeqZ+nxsdLWH0DxrnmaaO9oKPOHGgniniK26o ge1GDFzh/IIHaqnZOvear0hxqJ44/NSw5lMILYyXY3BUWegcADXbQl0964oUh+qof+pHDk+GjpbQ DjTuIE2L/QUfceJAPbIZVlRwAw/OgAMl/gPOqKRmW9prviLFoXrgYFrRdwzdqTTG5CC70ZXbQYuy Y+grUhxKfcjKim650dCdE1/rg+7oWmZbqS/6iBSHaujq6hdKY6DR9jUhUjxv9MChM9vqOnp3pDiU TmLT/PGigyXMD7oEdi3a1sqLPiLFodSHYtWRryy0/oQEnUsOdGLXosVKDX1FikPp1ItxqeoKFrpw npqZunyjWTnMtrFXfUWKQ6kfzMfIZ+hoCRgKN7r2Z9s244s+IsWhWuNMU1VYLXT+P9lljwNLbgPh fE8xJ1jo/yd2bsBw4BM4Wwbe+wcuFqUe9hAL9Fu+V/qao5bIIoeejcnVoWf93OdMx+69lRJQ8I+z M94HXTiqwWAmh2ZvsadWumtNnVICCnHWeHGaNHSpXDC+aC1C9zlLfdBOKQE14O013jj+hs5QYgG8 ff+irZjyOS/5JZRAGvDXFq+HnGzBduRmDZFb3cqDdkoJKMQFFXboNzK0GmMs0Jniix5E8znvNX8J JZBGwY8an6HWXuO9OTTsdYYEA8/1uc85LOe3UAIIcbJ4XnA2/XLgzWbI57Gnb50EDv5H7yP+uhp3 8b5imd7ewx25fu5zXvBLKAE09HwytlaDLZuojLqDyZHL/tznXGY9fpQSUEPdLOLKZkZ0Pgu02110 65/71B590F4pAYUjovUaZ7EaerCe64mqDj3a5z5XOlm/lRJQuDK65Yi3oTu7kC7YDm2VTsHZ6v+P TgIIsZ6RM38SrGdIr2534M3t4HOVddFeKQGFCqL9HKVKTznQjf1ea5leg4PWrbtP7RtEv5USUCip 2+J10eks2A5dP+ex2rpgr5MAQqwFBJW7GbiywGhp7w7cxuc+V7fb/aOUgELzGhov1kCgYae4oLNI XvQon/tc4xy8t1ICCnGyeF50PguWQ6/8uc91atKPUgIK3VyLE2xDNXRm8dLm3xx6z899rnWyfisl oGBvtC8i3oZO7JvqhtIXvVg4znOfrN9KCSjEGfFM7O1El7NAm/9F05bac6dy0V4pAQW/p80LxrIw hh8cXACL4tBtfe5zZ7MeP0oJqJ55AxCvi05nwXZo9hR77lIu2isloBCr34PTboZe9INqxbtDL6L5 3PVk/VZKQHXzyBg81LMCjZ3TBTnR1F705obwuZtZjx+lBBTiZPG86HwWrC9a9/d59nrRXikBhdmr Yh7BJFYNDSeB1qejWoM5v+jaPs9znKzfSgkojKLT4m1oGCouwF3bX3Tbn+d5bM2PUgIKcdYYutwM nc+C6dCjf57numQvlEBqqLAa48cZudEKYBPxq77kxZ2w5z5n762UgGqDlQHxNjT88bQF+6L/p6pW H1VVjzT1j4ylf//3j/98YKA++h/f+UZIeAdiXThh7ut5Z+GCRuPLd/4LPwhzYzqu59//+Ccdqzqp qVNTA3fpC9rn9DqaYW1u59+5TWCqo03zmmMH+OuPf7us9R+12nzpP6vPz+rrGXwLDdLUOteej7DL 9yOs9aCdUgJq6SzJeEzv4rFAT/PzfXlw/Jn8EUogLS2QTf3xncUy79yz90ZmHz7HfeTHajulBNQq /Jyv+UCbKxZsl/TI30va+oN2SgmopbWX8Xiy1iI0tTh/0Z1Z5+UK4o9SAgpxtVeZ41TDqG4DC7rL uuXTdr5l/EcpAbVg4/CqkViCDa1VeS7+/atqnTKexoN2SgkoxPrOkei0vmPN3KwcF12uMfl24x+l BBTiaq+yiXJvdqm5uW+vKr56Oamf+cMJJYBW4tf1A5NaUeitChg4jWMtnaN6CSWAVuKJHPkMqnuz ZeOv8zfjZXY4v1zgSygBhLjai+admPQ646/7N2N1UDob0Ki1OzA5nQTOSrz3o9CskKuWcSVWgcud tNrr5eJfQgkgbJHWKsQ9H3DFQIEvkV3CPA9z+sHjpZPAQdzsPfMmrA5maVP9cjsT7mc3DtfpJHAQ J31P5Wj2TI5L/dCXS182eU1+BscjlADCxdF/QGy2Z6uvh75Q9jhKDgaZE+kpoG+hBBDiZi96BlKd UaBvLuPCjFM9w8EBO6EE0Fzc+6EzzxlHlyUyXMY8voPN5M4EL50EDmqqnhXEZ0ZC59Y8qm9SOtnq lHs3+oCdUAJoTp7u0a5rH/Tiq/oWZddisE7MUi/YCSWAEOt1RPNO9Qy5yxJxHUoPgvnPRI9wZlwn lAA6fmP04+C32gPqXX+yiWtkBY87wbyEEkCIu71o3IzVgMAwuO7ECzfYQcatEy+dBM4crNGIzVBu OMtMvWtNNn32Nc4XJPgllACCkWQ70S9xwDr9Qe8aE29yZ9Z9XKyTScAgZivBz7j56jQCO1u/5Wc2 5tu5zccAvIUSQIirvegMn5XNHnrXk+hZetWsRzll4qWTwKFVRzzPCAN939S7lqTnFuByjsXhOp0E DmL9gkN98eFWzWP4jqT3Hlyein6xTiYBc4YWxNuKD2zSpt71Iys7rJn97MJLJgGDWC/IQOXoBzst DdeOBvtnG3Yo1gU7oQQQ5je90oj7zZdtZvp2ZNWs0Wv2US/YCSWAEDd70bwZs89MlqwLnnolGvuG HrcDdkIJIMRaNgfMkWWcrc9MFtkLHsyYFqjXs8cvoQTQLPzeiE9DytZnFrvCBXfNuNLX91stX0IJ oFn4P4jnzZh9ZnHZBTf9bJUeU432ATuhBBBiTX2m25CSNRqdJL5gFuBKR9HOiPkWSgDNTG8z0+1I yRrN5qe54MKMeYzbulynk8CZ2aa9dBsSbF2mXs/S5bKw11p4mPcFO6EEEGI9z/jD+giEyxIZLmG2 0MoG3bp9u7dQAmgm3kDEdR9wt0T2N+PO41DZSFvLF+yEEkAzsWIgHjdjbTT4oVpfHjCIhe2uPfk6 mQQMYq1viK2N4BNrm9mJ1fBip+Zbxjh3hOCXUAJobBbkWU4jgV4bzbaZ7gErl1UolYt1MgmYsdk/ EPebr7YZnHz942JtH1gm6poX7IQSQIj1DYitj+DaawY7M58HrFtbWOXrcWtvoQTQWNwRVA9rJNDr jkG/XcZVM86bB+5UzLdQAgixfkPE/Was33jr5ODAmnFmQa7n1r2FEkBj8cwhtmaCZqBnchee0Atm C830l7WamXgLJYDG4i1BT7FmAr3aOujXt/5oMQOYZaKWy3U6CRy0Pb3WiPtNWH3oriwCl6vUWnnx 9sU6mQTMmCxDiNc8WPXNu/qW1JYmmlmDyp4X7IQSQGOybsK6WCeB6ZiWiGtJ3IBMc6k3mtyXTgIH JkgLPWKb4aDXXrmb70haIlG3F8Hzcp1OAmcM9hHEMx+ujlK7+YbUhp60xCJR+rpgJ5QAQsw+Ms4M Bxu6LJH5rT7mI5JdvXZOxEsoAQRbrN0fsc1w0OtP2d03JC29iSdJi8XlOp0EzujWR8YZ4aBXOwO9 a0hND1li29BScbhOJ4GDmG1knhEOc8myPFw/agXfTKHpJPsSSYCMZi1knvEN+m5JfHuRAgnj4aTf 1aOa4Z7y0j+0V/39X9Xmj/6nWpxKbH+56cv9eV0dzfim3xe7519c0Gb5psbfs7eewc3atNd60E4p AXV+RF80g4YutXx/paH18mye9D3rQ3ZCCSTE6tG7mhEjqxvhAjelbs4km2dyj7sfL6UE1CqcKro2 44vWOocFbk7dvEGbbWufMepHKQGFuNurxpN10VyWn1S1Q6mhWOcFF+2UElCIq71qb0MrggvcsLp5 izYL1S7loF9KCaiVbczUi3fRvXOBG1dtKxY97k7jQTulBBRiDppqTC66ai7w7S5rvUt/LrrctfuD dkoJKMTVXrW7xlvNCfVuZt0cIhaHS02ddvQllABayYZNmMl9wH1T74ZW3QH14eMkfsBOKAG0Ek/k UHNywBWJ4K/d1Lo4By4WrNX7BTuhBNBKvEVDZySCF6/zSn5uXYvbS5+77NC9dRI4K/HeD/Umh8t5 NPnBVbto+nMupl3WBTuhBBC2SGvVUG9ywJxHsx9c7TzM+X/Sq6a3bSOI3v0reJQPrsnlfpBA0cKN eynSHGLdmh5UmbZVaCVDkpv63/fNm115mQUSFLlYHPnpcbicmTcvHUfiLXCx4kFs9T4hJ0w72iXt VF7HhF06jcRb4GLFg1ikzMtyQt6gbrSj8GVeLgmBbRKGZH1mwFgRoXHkH77PtjWoGzVJOpWYVRa6 kcfRZ+ICGCsixFZvlGxrUDdqknYqsWHGrbR08CYTF8BYESGWs/c221avbtQk8VRilq+nmKTNfI6L FQ9mKiXPZtfq1Y32pUgNNIHe54NOxAUwVkSInd7I54RNT3whUdoWnnMipPVjDowVEWJpR++S24SU D5pIoVBSCNKtLOUuVdsMGCsiKLkMEMQmJGK60b7Up0Bj6bnq+jQz58BYESF2eiOfMxY/ioWhUCc2 nKeC+DwnZrhY8SCWGe1dspujpUsCvpAmKTAMmsGnN0jiGTBWRNhNKSc+GU7gxdcBXwgTO5nm1Tuf aQtYrGgQU0p8spvAiw/Fgty/jZ9gmS83XZ8WgDkwVkSIe72R2k2YAxF74AtN4s7iuOh6k8bEDBcr HuyDVJKQ3Cbw4kaBLyQp0ASyhKUsEm+BixUPYnmDiF3OV8woLFOhSNL36th86zJtAYsVDWKxmojV asIrypILfKFHOnY4M106hRksVjSIpUFgONVpjkZVxpdy5Kmf1mtRDJm4AMaKKPRsacQu50uZCaUc 6TRT/+N8n4kLYKyIEFu9UcgZU2dCaVpl4W05bbXcEnEBjBURYhmbfkxmc+xUZ0JpWr1nxlyBXJ/O eAaMFVEwfN+IkyB1qjND6Vqlz2RV4LvL03IGjBVRMLxAHHLG1JmhtK3SZyDmjimLdiIugLEiQky3 2WZBalVoxEm8EXMA99wo7JiIZ8BYEYWOu01osyK1KjRjaVu9YcYsYztk3gIXK57Qqd1ssyBhreuI L2yrDva+NyzmMRMXwFgRIabl7JKOADhoIoVv9ZTQngJtXfKuM2CsiEKr3rVjP5LYaSJv3pXeFMQU Umu7TFwAY0UUWk4MxD5nLEKDB5X5ciYGo6Hc2XO+BSxWNIhlviFWGcErFpkZW07DTBskX+N96hES z4CxIvIjB3IwSUiAF6EZ1dOdiYWXU6g1mbaAxYrGj9QPxC7nKzKDypePTKvnwDHRDyETF8BYESGW OyBWHUHbSwZjx3zOxHK0hlO+T9vaHBgrIj/wRDA9VEiAlxMDfiwy7iXjbmTBpYk5B8aKCLG8Q8Qu ZyzveBTnUBBLxh0Hcp+6bg6MFZEfWHOIVUwgBlKTo2GFZmJKaMf9su91mZgDY0XkB3YJNEXFBHhZ 64Af3uaPDDMQc0z0JvMWuFjxQPakrRG7nLDsoWPPIZB5hbXv2Xhjpi1gsaLxgWMI8RASrezNY19K kh0k0Y4zyIwhExfAWBH5wLmJ1UWVBEtH0EQKSeIBdFwupaPJO8PFigdLkAx6xOrhgBetHG2pSDIi MbcHEofMW+BixeM9dQRx6BKvWKnRloJkvVRayyFh3JCJC2CsiBBTR3zycFhDB00kvE0f3SNabT2b KmIGjBUR1mJRf8Tq4YCXRxldKUgyeltWkgyLzFvgYsXjneqITxYOeFlngC8EyUqRtZQNGRWJt8DF igcxZSQkCwdfMmgehR5Zg3cmpG1KdgaKFYm3KiEh2TfgnSbxpkVC2Ny9+3DB4uS+K6XaYXvqBvkQ rTpMF3cXvywvrpdL23TN8gG/62XtQrVc5YtW+lO0pusxClv8bBlpOdTZ5r+S+BWOp8VDLNcXcoWG X36+WHzYn5rX6dQ877eb49N030y7/cvj0+Xyb0kzDUefdEIoLCcFP5a3X/09Eg85cSOWb2iu0mfb dDinAesOFKIbcXCSNo9F7tMyScnvx7Z1w09K5r9yCpkO9Qg+yS2mB/b5geUKD/x+/7jZNdfN6uX0 NO1Om/XqtNnLF8dp/XLYnF6r1FtJGXfLF/IVnJpM786gxMY2pf/VpI0oFrzrVb4oaSx63w5OXx5S dcLF/Dtm/cfi3T5euh+GxfPlFbaIxXY6TdvX5vg8rTcPr81mt96+3G92j83q0su/t83+0EzbSX+E 5zw2+4fm8s/lb0gFDEZO4/ZcCOlm6EfebDd9bo4vf0VwucXmdMJrvd+vL1HSixf9koyfFnc3Hz7e 3H66JPG3Dm3grLbfe2hnmi8Pzb4dmuOr/jg9bo6n6dCsmufD/mGznZrPm9NTc/f+5u7nb1Zob1BK ASI7K9H/W5hnlroyu3MrMtvl9C/aaHVYRbzbw/Hb+WHrMxaX35dfZqnz46hYfGR/rLbT9fTv83a1 Y3i83sRnVBfKAMerwwJV1Y3yOrSs2uHceF2qqv0J1WgW0/H6eFqla5TzornfMFg94uGPdS19+eTW ORlJ3/nkZ5b6ye35zfRaSLe/31zfve7WT83Hab3/ZzpwSvy6vPhvAG74X1wKZW5kc3RyZWFtDWVu ZG9iag00MyAwIG9iag08PCANL1R5cGUgL0ZvbnQgDS9TdWJ0eXBlIC9UeXBlMCANL0Jhc2VGb250 IC9GUE5EQkYrTW9ub3R5cGVTb3J0cyANL0VuY29kaW5nIC9JZGVudGl0eS1IIA0vRGVzY2VuZGFu dEZvbnRzIFsgNDYgMCBSIF0gDT4+IA1lbmRvYmoNNDQgMCBvYmoNPDwgDS9UeXBlIC9Gb250RGVz Y3JpcHRvciANL0FzY2VudCA4MDEgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQgLTIwMCANL0ZsYWdz IDQgDS9Gb250QkJveCBbIDAgLTIwMSA5NzkgODAxIF0gDS9Gb250TmFtZSAvRlBOREJGK01vbm90 eXBlU29ydHMgDS9JdGFsaWNBbmdsZSAwIA0vU3RlbVYgMCANL0ZvbnRGaWxlMiA0NSAwIFIgDT4+ IA1lbmRvYmoNNDUgMCBvYmoNPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyOTkxIC9M ZW5ndGgxIDQ3NjQgPj4gDXN0cmVhbQ0KSIl8Vgt0TVca/v59ztk3D5HrkRRRTpwkyE1kPEKQkEpu HjeYJA3uDR335kFQEWk8p9WoKm6iypQyxRisoKbWuRhiuiyUKVPTVTo6SnXartV2aqlhTHVNpyrz n+MxzFp1/vzZ/3P/j/3vvS4IQDgaocBZObdB/3n7ggss2QhoDVPqps5ctePyCkA6AdWY+vSCKTEL QqxzhrHNlprqQNWh79deBzpWMT+4hgXtRzjKgIhBzCfUzGyY/+7RzDPMewFl0NOzKgNdvV13sj8j bZwZmF+nJDui2D+X7fW6+uq62uVv+oCoZkAs1FZzFscY41Cu7YMKtH3BdqCY21es1dJZnyXVtuOR X9tPfI/2+p+nTQ74SaMBWCs2IA/5pCEb85CDPngCLnyFoViONPTGfu5wX6RiIeVDpVdRoR3F73BC bBAHMA7l7N0HoxDEBvwWUUhGPmpRp+3HYGQxnY/btIK209fCI2qhIRoJbJOMZqyHiQu4QkMpi+pp Na2nrXSd9+7JnsNtzxIsxBG8jS/FCIShPWIZklg+G/vY8xuaSsvpFF3mPAYjk7Meyx7jUc55r8Um 7GXf4ziJUzhPDupLRQyzOZc1Ilb4GMrFM+Kgdp337AqDK7Sy6s/7jORdSriyiWjAXCzgCXsN23AM 7+ISrlM49SYXw0AqoxkMq2k3vUOfiwSRLJ4Qs8R8sUc+hg7cNxfvNpL3CaACNZiGWXgGzzMsxot4 CWt4119zlr9h2I09OIz38QE+wT9xg6IokffPoElUQ89yxhtpE22jA3SGPma4JcJFjlijtKg7tRht newlD8oPOKLOZ5lpg9X3cdyJCdyLiZjEMJtjN2AO1/Ms1/MCVmIzQtwhCw7iBMNZ7ugljn8NN3AT 3xGRQpL7FklOiuWa+3FGw8lDY6mExtFEeoqqaBr3s55eYXiVNnCGLQw7KET7GA7RW9yXk3w+F+mG 6CZ6inQxRGSJQvEkQ6WYJ5aIl8Rqsc6GP4sz4qw4J/4ubinhSh9lsDJcqVIalG3KaXWIOlTdon6o XlQv2XBZva310DI0r7ZYW6d9pH0iO8kquVUekH+Vn8qv5X8cy3gOE3iSeTp5FuvxHE9nE1Zxx1uw CwcYrJrfx19wER/jS4Yr+AdXGkPdSac8GwrJR+U0k+ubQ3NpEZ/zGj4Hq75ddIyO0x/pb/QpfU43 6F/0Pf1IPwqIGJ6srlxrD662l0gTgxiyRAFPWrl4SlSJWp6OxQzLxSuiRewUu8URcVycEKfFJfGZ uC5+UCKVaCVJcSmDlHRlklKvNCkrlc3KFmW78qbyjnKW4YLygwq1WF2ovqxe0KARz8BgbbL2grZV O6R9pV2TkTJGdpOPy/6yUHrkODlDLpKvyVPyO4fiiHWkOAodyxwnuRubeAoe/uoxBjswiV7nG+0C oRcOqRNRrfQQS8mHt5VbNr5OfbAIh6mjOgRF2EUuLKLJtB4eTODbu4mq+M5d5DnLwl46gh5KEg3D RxD8glzAWuqOI8Ih9mgv0y6s4Fn9N89qOmqVw8qHylvCixl8x/bwWzSEzimTqJZOKhGigntzkbkM Pvlr1ttJXZAproq9+D108SvxC/U5pV92+qCBA/r/LK1faooruW+f3kmJCUaveL1nj8e7x3Xr2uWx 2JjOnTp2cEa3j2oXGREe5pCaqghCitvI8+tmkt9Uk4yCglSLNwIsCDwg8Js6i/IetjF1v22mP2yZ zZZT/s8y+45l9n1Lcup8UVNTdLehm+/lGnorlZd4mV6Za/h086pNj7FpNclmopiJj2cP3d2lJlc3 ya+7zby5NUG3P5f3C0VG5Bg51RGpKQhFRDIZyZSZZ9SFKG8E2YTIcw8LCYRFcVamx8h1m4VGrpWC qSS6A1VmcYnXnRsXH+9LTTEpp9KoMGGMMqNdtgly7DCmzDEddhh9mlUOmvRQytFgc6sTFX5Xuyqj KjDJayoBnxWjg8vMN3LN/IVfdElNaaWWMq8ZntNKKPMegqetMVTYmJvrY8ug4g4Glz1snppSVOqN 53wMd7NuJVjqtXNjc+qSxuEtmVXAnVKqDbcl8U/XzXBjlFETnO7nY+gWNFG6IH5vN0/2obbP4HHr wTKvEW+OjDN8gdzuoc4Ili7YV5itFz6sSU0JOTvc6WGoffRdol3Ug0T1fZ1N2eYWxVnfayJZGRmF fPimXqlzJl7DFIkZ1r/qDAQrM9iMPx9xr6ZxZ/xB5zCrxVqi09CDN8FHbFz95mFJ4K5EJjpvwiKt Qbg/TKy/R5sul5mcbM2AI4cPjTMbYfPpqSlzzSKjzqmbRdwyFHvZyTcsjVseH2+dX1NrNiqYMRtL vHd4HRVxe5Gd5vKZwm9pjt7TxIyzNI33NPfd/QYP6n5YP9BizLCk+3/RzthO7pphJsU+Ql19R88X w62HVC0xWOxNCgSb4pL8wWYfH00eX7JgMM/Q84L+YKC1rbHC0J1GMFRUFKxz+++V1Np2tCnOzG72 1RA31RxodwNWTDntdjUQFm/9MnKcs7N48HtP3hVJRjGUbU6j3mFijL2eRxS/cf1kMZpkKbJkGXpp S5Hh2AOPYwnGqVV4UZ7HfIcHjY55KLBXD+s8qGFMYHQz+hknOzxtbbwOYCyQneGWHZEvw1CgfIt8 dTo6qaMxVt2MQo1fSelBkRzLNhaWolB6MVqWY4xcxutaxmJkyz+g2qbfwHiWu2ULx69AtfYtMrU/ IVG7zXFKOcZ4+LVVvF8684k2X2DF1nYiW3sDbm0Oumkb4ZIbWbeUsdjGctYP1Z7HKP5tm6ktgeCc suRofvP3YwL34ZeyAOm8DpS38OR/ya//1zaLOA7g77vny13TbSZxlmDckthpsbMmK85Zo9vTlKWR 0BkXV9NqXV03sQ6hW4pfWVl1Q0Smj8gEv4BVKNSxjdhNaYdoLQPxB38Q/aWdX/CHgSDCflEEu9T3 JRUE9wco5uF1H+7u4bnnuXvy3B1XQCk3siKMpKGacZvBZ+hxs7jbGkdaRJGWW5C2dlIzDhpOCT2G FUOR0vJF3EJFuZlxM7J2GNdQm5VDkrrFD3XyJIYM5wLO2P14hjpUDjmnl+2txdskzDPb84g4x/gM nLB5L93/ZqoFbe5ZbNcJ9scJjDpLWFB70cX4uyrhDvUYdqjHMWW4JzmOS3w3JhDlOd01vfhaHcAu VUavdRYDKohWt8JzKhyjSzhkcDzzhltlP1VrfRVTr3LcNnJcf8YWxptUFJ+qjzHK+AXfq26208G2 31fzWM/7KHZGOA+/gRR5VCAbcTGJFHlUoBFycFiWUZFlb7VENsu/WDikvRynP8u7J9ce/56+pUVa oNP0Lr1Dr9DzNEaP0l4apN30EA3QA9RHu2hGzE/Xwmf14NXD1nq4qx7sepDeccbv6ILJ03s0QT69 TEfpCD1Hh+gpepKGaR/toVKuPdQQavDfVP7ryn9N+S8o/1nlP6H8g8rfr/xHlN+v/PuV36s26Ot1 XK/X1+lrdUQ36bU6rIN6jV6lA1prV9taau55RL4yN4T8nnjlt2LzjAjc219xmjOiEs4jf19mFkLN HT3WtPGfv8jfMyJfeHoWMfHVRyp2XsVGFMvyRRb5psg/r/xaUURMF9Cef/ilwXW4wiXrl7pi4fbh YoaNlD7QyPR1PViPZ2RjINxVGowm+jJNwZGtFWZm04nIWPScDTGFRk4bq7jCWE2mqq2zrdNU2ahV rTGLj5WqyFg6ET0nplaqgiwOsRvK5bL5SJvvNVfC4D5VofVD24fyxYy8dRoB+xMuPlnL1JJJIFm9 GLyIbSbdlEqEEqEbmAg2uRS35pY8B38gbs+ZeaJn+SdXOG/xmjd76ywhxxGAtMaD4pKQBc65h7nr meCOzBXJgQO/dHRgG5NNKadBWi2Xb5SLcjHCZOFyC5tv/Y8dhf/NYV6dL8VOBGtTvmS8Grdz+KvO JMxCwNTTj/uzfbuvuvNXRHXtxMmhz0+beOr4kdHl5eo+9Y07zGzjX4uIPwUYALGIOlkKZW5kc3Ry ZWFtDWVuZG9iag00NiAwIG9iag08PCANL1R5cGUgL0ZvbnQgDS9TdWJ0eXBlIC9DSURGb250VHlw ZTIgDS9CYXNlRm9udCAvRlBOREJGK01vbm90eXBlU29ydHMgDS9Gb250RGVzY3JpcHRvciA0NCAw IFIgDS9DSURTeXN0ZW1JbmZvIDw8IC9SZWdpc3RyeSAoQWRvYmUpL09yZGVyaW5nIChJZGVudGl0 eSkvU3VwcGxlbWVudCAwID4+IA0vRFcgMTAwMCANL1cgWyA4OCBbIDc4OCBdIF0gDT4+IA1lbmRv YmoNNDcgMCBvYmoNPDwgDS9TIC9EIA0+PiANZW5kb2JqDTQ4IDAgb2JqDTw8IA0vTnVtcyBbIDAg NDcgMCBSIF0gDT4+IA1lbmRvYmoNNDkgMCBvYmoNPDwgDS9UeXBlIC9QYWdlcyANL0tpZHMgWyA2 NiAwIFIgMSAwIFIgNCAwIFIgNyAwIFIgMTAgMCBSIDEzIDAgUiAxNiAwIFIgMTkgMCBSIDIyIDAg UiAyNSAwIFIgDV0gDS9Db3VudCAxMCANL1BhcmVudCA1MCAwIFIgDT4+IA1lbmRvYmoNNTAgMCBv YmoNPDwgDS9UeXBlIC9QYWdlcyANL0tpZHMgWyA0OSAwIFIgNTEgMCBSIF0gDS9Db3VudCAxNSAN Pj4gDWVuZG9iag01MSAwIG9iag08PCANL1R5cGUgL1BhZ2VzIA0vS2lkcyBbIDI4IDAgUiAzMSAw IFIgMzQgMCBSIDM3IDAgUiA0MCAwIFIgXSANL0NvdW50IDUgDS9QYXJlbnQgNTAgMCBSIA0+PiAN ZW5kb2JqDTUyIDAgb2JqDTw8IA0vRHQgKEQ6MjAwMDA4MDMxMDM4MTIpDS9KVE0gKERpc3RpbGxl cikNPj4gDWVuZG9iag01MyAwIG9iag0vVGhpcyANZW5kb2JqDTU0IDAgb2JqDTw8IA0vQ1AgKERp c3RpbGxlcikNL0ZpIDUzIDAgUiANPj4gDWVuZG9iag01NSAwIG9iag08PCANL1BDTSAvRGV2aWNl Q01ZSyANPj4gDWVuZG9iag01NiAwIG9iag08PCANL1IgWyAxMjAwIDEyMDAgXSANPj4gDWVuZG9i ag01NyAwIG9iag08PCANL0NvIDU1IDAgUiANL0pURiAwIA0vTUIgWyAwIDAgNjEyIDc5MiBdIA0v UiA1NiAwIFIgDS9XIFsgMCAxNCBdIA0+PiANZW5kb2JqDTU4IDAgb2JqDTw8IA0vRmkgWyA1NCAw IFIgXSANL1AgWyA1NyAwIFIgXSANPj4gDWVuZG9iag01OSAwIG9iag08PCANL0N0IChTaG9ydEVk Z2VGZWVkKQ0vRG0gWyA2MTIgNzkyIDYxMiA3OTIgXSANPj4gDWVuZG9iag02MCAwIG9iag08PCAN L01lIDU5IDAgUiANPj4gDWVuZG9iag02MSAwIG9iag08PCANL0QgWyA1OCAwIFIgXSANL01TIDYw IDAgUiANL1R5cGUgL0pvYlRpY2tldENvbnRlbnRzIA0+PiANZW5kb2JqDTYyIDAgb2JqDTw8IA0v QSBbIDUyIDAgUiBdIA0vQ24gWyA2MSAwIFIgXSANL1YgMS4xMDAwMSANPj4gDWVuZG9iag02MyAw IG9iag08PCANL0NyZWF0aW9uRGF0ZSAoRDoyMDAwMDgwMzEwMzgxMikNL1Byb2R1Y2VyIChBY3Jv YmF0IERpc3RpbGxlciA0LjAgZm9yIFdpbmRvd3MpDS9BdXRob3IgKFlPRUxMRSkNL0NyZWF0b3Ig KFBzY3JpcHQuZGxsIFZlcnNpb24gNS4wKQ0vVGl0bGUgKE1pY3Jvc29mdCBQb3dlclBvaW50IC0g SUVURjQ4LWlTQ1NJLnBwdCkNL01vZERhdGUgKEQ6MjAwMDA4MDMxMDM4MTQtMDcnMDAnKQ0+PiAN ZW5kb2JqDXhyZWYNMCA2NCANMDAwMDAwMDAwMCA2NTUzNSBmDQowMDAwMDg2NjEzIDAwMDAwIG4N CjAwMDAwODY3NjUgMDAwMDAgbg0KMDAwMDA4NjkyMyAwMDAwMCBuDQowMDAwMTAyODY3IDAwMDAw IG4NCjAwMDAxMDMwMTkgMDAwMDAgbg0KMDAwMDEwMzE3NyAwMDAwMCBuDQowMDAwMTE5MDYwIDAw MDAwIG4NCjAwMDAxMTkyMTIgMDAwMDAgbg0KMDAwMDExOTM3MCAwMDAwMCBuDQowMDAwMTM1MTU4 IDAwMDAwIG4NCjAwMDAxMzUzMTMgMDAwMDAgbg0KMDAwMDEzNTQ3MiAwMDAwMCBuDQowMDAwMTUx NTExIDAwMDAwIG4NCjAwMDAxNTE2NjYgMDAwMDAgbg0KMDAwMDE1MTgyNSAwMDAwMCBuDQowMDAw MTY3NjYzIDAwMDAwIG4NCjAwMDAxNjc4MTggMDAwMDAgbg0KMDAwMDE2Nzk3NyAwMDAwMCBuDQow MDAwMTg0MjI4IDAwMDAwIG4NCjAwMDAxODQzODMgMDAwMDAgbg0KMDAwMDE4NDU0MiAwMDAwMCBu DQowMDAwMjAwNjQ0IDAwMDAwIG4NCjAwMDAyMDA3OTkgMDAwMDAgbg0KMDAwMDIwMDk1OCAwMDAw MCBuDQowMDAwMjE2ODg5IDAwMDAwIG4NCjAwMDAyMTcwNDQgMDAwMDAgbg0KMDAwMDIxNzIwMyAw MDAwMCBuDQowMDAwMjMzMjkyIDAwMDAwIG4NCjAwMDAyMzM0NDcgMDAwMDAgbg0KMDAwMDIzMzYw NiAwMDAwMCBuDQowMDAwMjQ5NDMyIDAwMDAwIG4NCjAwMDAyNDk1ODcgMDAwMDAgbg0KMDAwMDI0 OTc0NiAwMDAwMCBuDQowMDAwMjY1NjQ4IDAwMDAwIG4NCjAwMDAyNjU4MDMgMDAwMDAgbg0KMDAw MDI2NTk2MiAwMDAwMCBuDQowMDAwMjgxODY4IDAwMDAwIG4NCjAwMDAyODIwMjMgMDAwMDAgbg0K MDAwMDI4MjE4MiAwMDAwMCBuDQowMDAwMjk4MTA1IDAwMDAwIG4NCjAwMDAyOTgyNjAgMDAwMDAg bg0KMDAwMDI5ODQxOSAwMDAwMCBuDQowMDAwMzE0MjQ2IDAwMDAwIG4NCjAwMDAzMTQzODUgMDAw MDAgbg0KMDAwMDMxNDU5MiAwMDAwMCBuDQowMDAwMzE3NjcyIDAwMDAwIG4NCjAwMDAzMTc4OTIg MDAwMDAgbg0KMDAwMDMxNzkyMyAwMDAwMCBuDQowMDAwMzE3OTY3IDAwMDAwIG4NCjAwMDAzMTgx MTEgMDAwMDAgbg0KMDAwMDMxODE4NSAwMDAwMCBuDQowMDAwMzE4Mjk1IDAwMDAwIG4NCjAwMDAz MTgzNTkgMDAwMDAgbg0KMDAwMDMxODM4MiAwMDAwMCBuDQowMDAwMzE4NDM0IDAwMDAwIG4NCjAw MDAzMTg0NzYgMDAwMDAgbg0KMDAwMDMxODUxOCAwMDAwMCBuDQowMDAwMzE4NjA3IDAwMDAwIG4N CjAwMDAzMTg2NjIgMDAwMDAgbg0KMDAwMDMxODczMSAwMDAwMCBuDQowMDAwMzE4NzY3IDAwMDAw IG4NCjAwMDAzMTg4NDQgMDAwMDAgbg0KMDAwMDMxODkxMSAwMDAwMCBuDQp0cmFpbGVyDTw8DS9T aXplIDY0DS9JRFs8NzUzZmE0Y2RkN2E3Y2RiOTE2ZmFlOTRhOWQyNTU3NWY+PDc1M2ZhNGNkZDdh N2NkYjkxNmZhZTk0YTlkMjU1NzVmPl0NPj4Nc3RhcnR4cmVmDTE3Mw0lJUVPRg0= --=====================_45058270==_ Content-Type: text/plain; charset="us-ascii"; format=flowed Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org --=====================_45058270==_-- From owner-ips@ECE.cmu.edu Fri Aug 04 10:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22602; 4 Aug 2000 6:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA03244 for ips-outgoing; Fri, 4 Aug 2000 05:08:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA03224 for ; Fri, 4 Aug 2000 05:08:11 -0400 (EDT) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id DAA24203; Fri, 4 Aug 2000 03:07:57 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id DAA26082; Fri, 4 Aug 2000 03:07:56 -0600 (MDT) Received: from sun.com (hobo65.Central.Sun.COM [129.147.8.65]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id CAA23007; Fri, 4 Aug 2000 02:54:11 -0600 (MDT) Message-ID: <398A89BF.95543187@sun.com> Date: Fri, 04 Aug 2000 03:15:43 -0600 From: "Mark A. Carlson" Reply-To: mark.carlson@sun.com Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: Requirements specification References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > > David, > > The one additional requirement is availability/fault-tolerance. Could you elaborate on this and tell us how multiple connections between the same two NICs enhance availability? Also, can you state what the availability requirements are? > > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). The serious implications that I think everyone is concerned about is the complexity that we are introducing into the protocol and the question of whether this really provides the performance that is asserted. > > Storage traffic requirements will always exceed those of many other > applications. > > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. Chances are that each LUN could be owned by separate initiators and require authentication to separate principals anyway. I think that this is the requirement we need to keep in mind. -- mark From owner-ips@ECE.cmu.edu Fri Aug 04 11:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22927; 4 Aug 2000 7:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA05211 for ips-outgoing; Fri, 4 Aug 2000 06:33:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ms4.hinet.net (root@ms4.hinet.net [168.95.4.40]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA05194 for ; Fri, 4 Aug 2000 06:33:16 -0400 (EDT) Received: from kite ([210.241.235.94]) by ms4.hinet.net (8.8.8/8.8.8) with SMTP id SAA12752; Fri, 4 Aug 2000 18:21:53 +0800 (CST) To: J@ms4.hinet.net From: MtvStar HyJ Subject: t|AFHIGHH Date: Fri, 04 Aug 2000 18:18:05 +0800 Message-Id: <36742.762564814816900.6970@localhost> MIME-Version: 1.0 Content-Type: text/html; charset=big5 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id GAB05211 =AC=A1=B0=CA=AA=AF=A5J=B3=F8 =20 =20
=20
=AC=A1=B0=CA=AA=AF=A5J=B3=F8 = V4.0
3D"=B5=D8=A4H=A5=FE=B2y=AD=B5=
=20
=20

 

 =20
=20 =20
=20
<= font face=3D"Arial, Helvetica, sans-serif">=A1=DD=A1=B1=A4C=20 =BCL =A4K =A6=DE =A1=B1=A1=DD
=20 =20
=20

=20 =A1=BD =A7=DA=C4=B1=B1o =A4p=BD=DE= =B9=EF yuki~ =A4@=B0=C6=BEl=B1=A1=A5=BC=A4F=AA=BA=BC=CB=A4l~~ (Cindy)=20

=20

=20

=20
8=A4=EB1=A4=E9=AA=BA=A7=B9=A5=FE=AET=BC=D6=B0=DA=A1I=AC=DD=A4p=BD=DE=A4@=B0= =C6=BA=CA=A4=B6=A8=EC=A6=BA=AA=BA=BC=CB=A4l=A1I=A4S=A8S=A8=C6=A6w=B1=C6=AD= =D3=B0=AA=A5D=BC=BD=A6b=A4=A4=B6=A1~=20 =AFu=ACO=AA=BA=A1I=B5=B2=AAG8=A4=EB2=A4=E9=AA=BA=AET=BC=D6= =B7s=BBD=AC=DD=A8=ECyuki=A6b=A4=D6=A6~=A7L=B9=CE=BF=FD=BCv=A1A=B3X=B0=DD=AE= =C9=A7N=B1o=ADn=A9R=B3=E9~~ (=B7=C6=B3=B7=B7=C6=B3=B7~~)=A1Ayuki=C1=F6=B5M=A7=A4=A6b=A4p=BD=DE=A8=AD=AE= =C7~=20 but=A4p=BD=DE=A4@=A5y=B8=DC=A4]=A8S=C1=BF=A1I=A4p=BD=DE=A4= @=A6V=B6W=B7d=AF=BA=A1B=B6W=AC=A1=BC=E2~ =AD=B1=B9=EFyuki=B4N=A4=A3=B5o=A4= @=BBy~~~ =A4=C0=A9=FA=B4N=ACO=BEl=B1=A1=A5=BC=A4F=A1I=ABu~~~~~ ~~ =A4p=BD=DE =A5[=AAo=A7a=A1I=A1I=20
=20 =20
=20

=20 =A1=BD =BD=D6=AA=BA=BAt=B0=DB=B7|= (=A7A)(=A9p )=C5=A5=A4F=B3=CCHIGH=A1H(=A4pK)=20

=20

=20

=20
=B4=BB=B0=B2=A8=BA=BB=F2=A6h=B3=F5=BAt=B0=DB=B7|=A1A=A7A=C5=A5=A4F=B4X=B3= =F5=A1H=A1H=AD=FE=B3=F5=C5=FD=A7Ahigh=A8=EC=B3=CC=B0=AA=C2I=A1H=A1H=20
=20 =20
=20

=20 =A1=BD =A7A=BB{=AC=B0=BD=D6=B3=CC= =A6=B3=B8=EA=AE=E6=B7=ED"=A8k=A4H=B1=FE=A4=E2"=A1H=A1H(=A4p=BC_)=20

=20

=20

=20
=BD=D0= =A6U=A6=EC=B5o=AA=ED=A6=DB=A4v=AA=BA=B7N=A8=A3=A4=A3=ADn=A7@=A4=D3=A6h=A7= =E5=B5=FB!^^=20
=20 =20
=20

=20 =A1=BD =A6p=AAG=A5i=A5H=A9M=A6=DB= =A4v=A3x=B0=B8=B9=B3=BER=A6b=A4@=B0_3=A4=D1~(SHIRLEY)=20

=20

=20

=20
=A6p=AAG=A7A=A5i=A5H=A9M=A6=DB=A4v=A3x=B0=B8=B9=B3=BER=A6b=A4@=B0_3=A4=D1= =A1A=A7A=B3=CC=A7=C6=B1=E6=A5L=B3=AD=A7A=B0=B5=A4=B0=BB=F2=B0=C7=A1H=20 =BD=D0=A4j=AEa=A8=D3=BB=A1=BB=A1=A3t!
=20 =20
=20

=20 =A1=BD =AA=F7=AB=B0=AAZ~~2000=A6~= =A4=A7=C5=CA (=A4p=A5H)=20

=20

=20

=20
=A5H=ABe=A6]=AC=B0=ADn=A4W=BD=D2~~~=A5i=ACO=A7=DA=A4@=AA=BD=AB=DC=B7Q=AC=DD= =A1K=A9=D2=A5H=A8S=AC=DD=A8=EC=AA=F7=AB=B0=AAZ=A9M=A4=A4=A4s=AC=FC=C1J=A5= D=BAt=AA=BA2000=A6~=A4=A7=C5=CA=A1A=AB=DC=B0=AA=BF=B3TVBS=AD=AB=BC=BD=A4F= ^-^=AA=F7=AB=B0=AAZ=AFu=AA=BA =AB=DC=AB=D3~~=A4=A4=A4s=AC=FC=C1J=A4]=AB=DC=AC=FC~~=BC@=B1=A1=A4]=A4=A3=BF= =F9=A1A=C1=F6=B5M=B5=B2=A7=BD=ACO=B4d=B6=CB=AA=BA~~=A6=FD=A5L=AD=CC=AD=C7= =AA=BA=B7R=B1=A1=B4N=B9=B3=B9=EF=B8=DC=A9=D2=B4=A3=A8=EC=AA=BA=C4=FA=C0=EB= =A4@=BC=CB=B7P=A4H=A1I=C4=FA=C0=EB=AA=BA=A5=CD=A9R=C1=F6 =B5u=BC=C8~~=20 =A6=FD=BFU=BFN=AA=BA=A8=BA=A4@=C0=FE=B6=A1=ACO=B3=CC=AC=FC= =C4R=AA=BA~~
=A2=A8=20 =A8=D3=A5h=B0Q=BD=D7 =A2=A9=20 =20
=20
<= font face=3D"Arial, Helvetica, sans-serif">=A1=DD=A1=B1=AC=A1=20 =B0=CA =AA=AF =A5J =A1=B1=A1=DD
=20 =20
=A1=BD =A7d=A9v=BE=CB=BA= t=B0=DB=B7|
=20
=20

=A4=E9=B4=C1=A1= G8/19
=AE=C9=B6=A1=A1GPM 19:00
=A6a=C2I=A1G=B0=AA=B6=AF=A4=A4=A4s=C5=E9=A8|=B3=F5<= /font>

 =A4= =E9=B4=C1=A1G8/26
=AE=C9=B6=A1=A1GPM 19:00
=A6a=C2I=A1G=A5x=A5_=B0=EA=A4=F7=AC=F6=A9=C0=C0]
=B0=E2=B2=BC=BD=D0=AC=A2=A6~=A5N=B0=E2=B2=BC=A8t=B2=CE= 2341-9898
=20 =20
=A1=BD =A4=FD=A7=BB=AE= =A68=A4=EB=A8=EC9=A4=EB=B3=A1=B8=A8=A1B=A5=FE=A5x=A8=B5=B0j=BAt=A5X
=20 =20
=20

=A4=E9=B4=C1=A1= G8/5(=A4=BB)
=AE=C9=B6=A1=A1Gpm 2:30
=A6a=C2I=A1G=AB=CC=AAF=A4=D3=A5=AD=ACv=A6=CA=B3f=B8= =DB=AB~=AE=D1=A9=B1

 =B3= =A1=B8=A8=A8=B5=BAt1
=A4=E9=B4=C1=A1G8/6(=A4=E9)
=AE=C9=B6=A1=A1Gpm 3:00~4:30
=A6a=C2I=A1G=A5x=AAF=A5=AC=B9A=B3=A1=B8=A8=AB=CE (=B6= =B7=AA=F9=B2=BC)
=A5x=AAF=BF=A4=A9=B5=A5=AD=B6m=AE=E7=B6=E9=A4=BB=BEF1= 23=B8=B9 (089)561211
=20 =20
=A1=BD =BEG=A4=A4=B0=F2= =A1m=AFu=AAB=A4=CD=A1n=C3=B1=A6W=B7|
=20 =20 =20
=20

=A4=E9=B4=C1=A1= G8/4 (=A4=AD)
=AE=C9=B6=A1=A1G19:00
=A6a=C2I=A1G=B0=AA=B6=AF=BA~=AF=AB=A6=CA=B3f=

=A4= =E9=B4=C1=A1G8/5 (=A4=BB)
=AE=C9=B6=A1=A1G17:00
=A6a=C2I=A1G=A5x=A4=A4=A4=A4=A4=CD=A6=CA=B3f
=A4= =E9=B4=C1=A1G8/6 (=A4=E9)
=AE=C9=B6=A1=A1G19:00
=A6a=C2I=A1G=A5x=A5_=AF=C3=AC=F9=AF=C3=AC=F9=A4@=BC=D3=
=20 =20
=A1=BD =BCB=ADi=E5_=B3= =CC=B7s=C3=B1=A6W=B7|=A6=E6=B5{
=20 =20
=20

=A4=E9=B4=C1=A1= G8/12=A1]=A4=BB=A1^
=AE=C9=B6=A1=A1G13:00
=A6a=C2I=A1G=AA=F7=A5=DB=B0=F3(=A9=BE=A7=B5=AAF=B8=F4= =A9=B1)
=AE=C9=B6=A1=A1G15:30
=A6a=C2I=A1G=AE=E7=B6=E9=B7R=B6R(=AE=E7=B6=E9=A5=AB= =A4=A4=A4s=B8=F4939=B8=B9)
=AE=C9=B6=A1=A1G16:30
=A6a=C2I=A1G=A4j=B7=CB =B8=F3=A5@=AC=F6=B0=DB=A4=F9= (=A4j=BC=ED=B5o)
=AE=C9=B6=A1=A1G15:30
=A6a=C2I=A1G=A5=AD=C2=ED =B8=F3=A5@=AC=F6=B0=DB=A4=F9= (=A4j=BC=ED=B5o)
=AE=C9=B6=A1=A1G19:00
=A6a=C2I=A1G=B7s=A6=CB =B8=F3=A5@=AC=F6=B0=DB=A4=F9= (=A4j=BC=ED=B5o)

 =A4= =E9=B4=C1=A1G8/13=A1]=A4=E9=A1^
=AE=C9=B6=A1=A1G13:30
=A6a=C2I=A1G=B7s=A6=CB =AA=B4=BA=C0=B0=DB=A4=F9=C3=B1= =B0=DB=B7|
=AE=C9=B6=A1=A1G17:30
=A6a=C2I=A1G=A5x=A4=A4 =AA=B4=BA=C0=B0=DB=A4=F9=C3=B1= =B0=DB=B7|
=20 =20
=A1=BD =C0b=A1I=A4C=A4= i=A1E=B1i=A6t=A9_=C2=DD=AD=BA=C3=B1=B7|
=20 =20
=A4= =E9=B4=C1=A1G89=A6~8=A4=EB5=A4=E9=A1@=ACP=B4=C1=A4=BB
=AE=C9=B6=A1=A1G14=A1G00
=A6a=C2I=A1G=A6=E8=AA=F9=CBm=B2^=A8=E0=B0=DB=A4=F9=A6= =E6
=20 =20
=A1=BD MTV=AEL=A4=E9=B0= =AA=AEp=AD=B5=BC=D6=B7|
=20 =20
=A4= =E9=B4=C1=A1G8/12=A1B8/13
=AE=C9=B6=A1=A1GPM7:00
=A6a=C2I=A1G=A5x=A5_=A5=AB=A5=DF=C5=E9=A8|=B3=F5
=BAt=A5X=A1G=A4=FD=A4O=A7=BB=A1B=A5=AD=A4=AB=B0=ED=A1= B=A7=F5=AA=B3=A1B=AAL=BE=E5=B0=F6=A1B=AF=AB=B8=DC=A1B=AC=DB=A4t=A4C=C3u=A1= B=B1i=BE_=C0=AE
=A1@=A1@=A1@=C1=C2=BE^=BEW=A1B=AD=EC=A4l=A4=D6=A4k=BF= =DF=A1B=AE]=BFP=AB=BA
=20 =20
=A1=BD =A9=EC=A9=D4=AE= w=B3=CC=B7s=AC=A1=B0=CA=A6=E6=B5{
=20 =20 =20 =20
=A4= =E9=B4=C1=A1G8/5
=AE=C9=B6=A1=A1GPM 14:00-15:00
=A6a=C2I=A1G=B5=D8=AF=C7=AB=C2=A8q
=AE=C9=B6=A1=A1GPM 17:00-18:00
=A6a=C2I=A1G=B7s=A6=CB=A1@=AF=BA=B6=C6=A6=BF=B4=F2KTV= 		=A4= =E9=B4=C1=A1G8/6
=AE=C9=B6=A1=A1GPM 17:00-18:00
=A6a=C2I=A1G=B0=AA=B6=AF=A1@=B7s=A5=FA=A4T=B6V=A6=CA=B3= f
=AE=C9=B6=A1=A1GPM 23:00-24:00
=A6a=C2I=A1G=B0=AA=B6=AF=A1@=A8=DC=C3=B9=AC=F6PUB
=A4= =E9=B4=C1=A1G8/8
=AE=C9=B6=A1=A1GPM 19:00-20:00
=A6a=C2I=A1G=A5x=ABn=A1@=B7s=A5=FA=A4T=B6V=A6=CA=B3f<= /font>		 =A4= =E9=B4=C1=A1G8/9
=AE=C9=B6=A1=A1GPM 19:00-20:00
=A6a=C2I=A1G=A5x=A4=A4=A1@=A4j=A4Y=A6=CA=B3f
=AE=C9=B6=A1=A1GPM 23:00-24:00
=A6a=C2I=A1G=A5x=A4=A4=A1@=AA=FC=A9f=A9fPUB
=A4= =E9=B4=C1=A1G8/11
=AE=C9=B6=A1=A1GPM 19:00-20:00
=A6a=C2I=A1G=A4=A4=BE=FA=A1@=BB=B7=AAF=A6=CA=B3f
=20 =20
=A1=BD =A5i=A8=E0=AEa=B1= =DA=A6X=B0=DB=B9=CE=A1m=A4=D1=A4=D1=A4=D1=C2=C5=A1n=BAt=B0=DB=B7| =
=20 =20
=A4= =E9=B4=C1=A1G8/4
=AE=C9=B6=A1=A1G=B1=DF=A4W
=A6a=C2I=A1G=A5x=A5_101
=B8=D4=B1=A1=BD=D0=AC=A2=A1G=B5=D8=AF=C7=B0=DB=A4=F9=A1= @2788-2456
=20 =20
=A1=BD =BF=BD=A8=C8=B0= a=A1m=BF=BD=C1=A5=A1n=BAt=B0=DB=B7|
=20 =20
=A4= =E9=B4=C1=A1G8=A4=EB26=A4=E9
=AE=C9=B6=A1=A1GPM7:00
=A6a=C2I=A1G=A5x=A5_=A5=AB=A5=DF=C5=E9=A8|=B3=F5
=20 =20
=A1=BD =A4=AD=A4=EB=A4= =D1=A5=FE=AC=D9pub=A8=B5=B0j=BAt=B0=DB
=20 =20 =20
=A4= =E9=B4=C1=A1G8/04(=A4=AD)
=AE=C9=B6=A1=A1Gpm22:00~23:00
=A6a=C2I=A1G=A5x=A5_ @Live		 =A4= =E9=B4=C1=A1G8/23(=A4T)
=AE=C9=B6=A1=A1Gpm22:00~23:00
=A6a=C2I=A1G=A5x=ABn =AA=F8=BAa=A5@=AC=C9
=A4= =E9=B4=C1=A1G8/24(=A5|)
=AE=C9=B6=A1=A1Gpm22:00~23:00
=A6a=C2I=A1G=B0=AA=B6=AF =C2=C5=A6=E2=A8g=B7Q<= /td>
=20 =20
=A1=BD =A1=BD =A4=AD=A4= =EB=A4=D1=BAt=B0=DB=B7|
=20 =20 =20
=A4= =E9=B4=C1=A1G2000/08/26=20
=AE=C9=B6=A1=A1GPM7:10
=A6a=C2I =B0=AA=B6=AF=A4=A4=A4s=C5=E9=A8|=B3=F5 		=A4= =E9=B4=C1=A1G08/19
=AE=C9=B6=A1=A1GPM6:10
=A6a=C2I =B9=FC=A4=C6=C5=E9=A8|=B3=F5
=A4= =E9=B4=C1=A1G08/12
=AE=C9=B6=A1=A1GPM7:10
=A6a=C2I =A5x=A5_=A4=A4=A4s=A8=AC=B2y=B3=F5
=A2=A8=20 =A7=F3=A6h=A6=E6=B5{ =A2=A9=20 =20
=20
<= font face=3D"Arial, Helvetica, sans-serif">=A1=DD=A1=B1=A6n=20 =B1d =A8=EC =BFN =B3=F8 =A1=B1=A1=DD
=20 =20
=20

=20 =A1=BD=A1y=BB=B6=B9=B3=C0]=A1z=B5= o=BFN=B7s=B3f=A8=EC=A1I =20

=20

=20

=20
LUNA= -SEA =A4=EB=A4=A7=AE=FC=A1y=A8g=A1z=BF=C3=B9=F5=ABO=C5@=B5{=A6=A1=A1=CF=AE=E0=A5= =AC=A1A=A7=D6=C2I=A8=D3=A5h=A4U=B8=FC=A7a=A1=E3=20 =A1i=A4U=B8=FC=A5h=A1j=20
=20 =20
=20

=20 =A1=BD =B7s=A5@=A5N=A4C=A4i=ADn=B9= L=AD=B5=BC=D6=B1=A1=A4H=B8` =20

=20

=20

=20
=AEL=A4= =E9=AA=A2=AA=A2=B1=A1=B8=DC=BA=F8=BA=F8=A1A=AE=F6=BA=A9=A4C=A4i=C5=E5=B3=DF= =B3s=B3s=20 =B7s=A5@=A5N=AA=BA=B1=A1=A4H=B8`=A1A=A7=D6=BB=BC=AD=B5=BC= =D6=B6=C7=B1=A1=B7N =A4=A3=BA=DE=A7A=A6=B3=A8S=A6=B3=B1=A1=A4H=A1A=A4=A3=BA=DE=A7A=ACO=A4=A3=AC= O=B1=A1=A4H=A1A=A4=B5=A6~=A4C=A4i=B3=A3=A5i=A5H=A5=FA=A9=FA=A5=BF=A4j=AA=BA= =B9L=B1=A1=A4H=B8`=A1I =B5=D8=A4H=A5=FE=B2y=AD=B5=BC=D6=BA=F4=C1=DC=A7A=A4W=BA=F4=AA=B1=B9C=C0=B8= =AE=B3=C2=A7=AA=AB=A1A=A4@=B0_=B7P=A8=FC=B7s=A5@=A5N=AA=BA=B1=A1=A4H=B8`=A1= =E3=20 =A1i=AE=B3=BC=FA=AB~=A5h=A1j=20
=20 =20
=20

=20 =A1=BD =B6W=A1=A8=A8g=A1=A8=A4=E9= =A5=BB=B5=F8=C4=B1=A8t=A4j=B9=CE=A1m=A4=EB=A4=A7=AE=FC=A1n =

=20

=20

=20
=A4=DF=A7=DE=A4@=C5=E9=ACO=A4=EB=A4=A7=AE=FC=A4=DF=C6F=BBP=C2=DF=C5=E9=A6= X=A6=D3=AC=B0=A4@=AE=C9=AFu=A5=BF=B9=EA=A4O=AA=BA=AEi=B2{=A1A=20 2000=A6~=A1yLUNACY =A8g=A1z=B2=D7=A9=F3=B5n=B3=F5=A1I=A1I= =A7=D6=A7=D6=B0=D1=A5[=B0=DD=B5=AA=B9C=C0=B8=B4=FA=C5=E7=A4@=A4U=A1A=A5=FE= =B3=A1=B5=AA=B9=EF=A4F=A1A=C1=D9=A6=B3=BE=F7=B7|=B1o=A8=ECLuna Sea=AA=BA=B1M=C4=DD=C3=D8=AB~=B3=E1=A1I=20 =A1i=AE=B3=BC=FA=AB~=A5h=A1j=20
=20 =20
=20

=20 =A1=BD =B7R=A6b=A4C=A4i=ACP=B3t=B0= t=A1A=BCW=B1j=B1z=AA=BA=AE=F6=BA=A9=AB=FC=BC=C6 =20

=20

=20

=20
=A5u=ADn=A6b8=A4=EB3=A4=E9=A2w10=A4=E9=A4C=A4i=B1=A1=A4H=B6g=AC=A1=B0=CA=B4= =C1=B6=A1=A1A=A8=CF=A5=CE=A5=F4=A6=F3=A4@=B6=B5=A5e=ACP=C5]=AAk=AB=CE=AA=BA= =A5e=ACP=AAA=B0=C8=A1A=A7Y=A5i=A6=DC=A4C=A4i=A4j=BD=FC=BDL=A8=C9=A8=FC=A5= u=A6=B3=B1=A1=A4H=B8`=B1M=C4=DD =AA=BA=BD=FC=BDL=A6=A1=C3=D8=BC=FA=AC=A1=B0=CA=A1CE=A5@=A5N=B1=A1=A4H=B4N= =A6=B3=A5i=AF=E0=C4=B9=B1o=B1=A1=A4H=B9=EF=A7=D9=A1B=B9=EF=BF=F6...=A1A=A4= =A3=AA=E1=A4@=A4=F2=BF=FA=A1A=A4@=BC=CB=B9L=AD=D3=AE=F6=BA=A9=A4C=A4i=B1=A1= =A4H=A9]=A1I=20 =A1i=AE=B3=BC=FA=AB~=A5h=A1j=20
=A2=A8 =A7=F3=A6h=A6n=B1d=20 =A2=A9=20 =20
=20
<= font face=3D"Arial, Helvetica, sans-serif" color=3D"#333333">=A1=DD=A1=B1=B7|=20 =AD=FB =B6=D2 =B6=B0 =A1=B1=A1=DD
=20
=20
=20

=A7A=C1=D9=A8S=A5[=A4JMtvStar=B7|=AD=FB=AC=B0=B6=DC=A1H= =A1I=A6=A8=AC=B0=B7|=AD=FB=A1A=B1z=B1N=A8=C9=A6=B3=C0u=A5=FD=B0=D1=BBP=A7= =DA=AD=CC=C1|=BF=EC=AA=BA=AC=A1=B0=CA=A1A=A8=C3=A9w=A6=AC=A8=ECMtvStar=AD= =B5=BC=D6 =B6g=B3=F8=A1F=B0=A3=A6=B9=A4=A7=A5~=A1A=C1=D9=A6=B3=BE=F7=B7|=B0=D1=A5[=A5H=AB=E1=B1M=AC=B0=B7= |=AD=FB=A9=D2=C1|=BF=EC=AA=BA=A6U=B6=B5=AC=A1=B0=CA(=A6p=A1G=B7s=BAq=B5o=AA= =ED=B7|=A1B=B0O=AA=CC=B7|=B5=A5=B5=A5)=A1A=B3o=BC=CB=A7K=BF=FA=A4S=A6n=B1= d=AA=BA=A8=C6=A1A =A7A=AB=E7=AF=E0=A9=F1=B9L=A9O=A1I=A7=D6=C2I=A8=D3=A5[=20 =A4J=A1I=A1i=A5[=A4J=B7|=AD=FB=A1j
 
 

=20
=20
=20

=A8=FA=AE=F8=ADq=BE\=A1Ghttp://www.mtvstar.com.tw/member/
=AA=A9=C5v=A9=D2=A6=B3=A1G=B5=D8=A4H=A5= =FE=B2y=B0=D3=B7~=AA=D1=A5=F7=A6=B3=AD=AD=A4=BD=A5q
Copyright @ 1998-2000 Chinese Global Business Network. All righ= ts reserved.=20
=A5=BB=BA=F4=AF=B8=A9=D2=A6=B3=AD=B5=BC=D6=A1B=A8v=B9=B3=C5v=A4= =CE=B5=DB=A7@=C5v=A1A=A7=A1=C4=DD=A6U=A4=BD=A5q=A9=D2=A6=B3 =ABD=B8g=A4=B9= =B3\=A1A=BD=D0=A4=C5=A5=F4=B7N=C2=E0=B8=FC
=C5w=B1N=B1z=AA=BA=B7N=A8=A3=A8=A3 E-Mail =B5=B9=A7=DA=AD=CC
Please let us know your valuable comments regarding our website= . Click=20 in !!!

From owner-ips@ECE.cmu.edu Fri Aug 04 11:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22922; 4 Aug 2000 7:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA05135 for ips-outgoing; Fri, 4 Aug 2000 06:30:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA05115 for ; Fri, 4 Aug 2000 06:30:04 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id MAA97752 for ; Fri, 4 Aug 2000 12:29:32 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id MAA107540 for ; Fri, 4 Aug 2000 12:29:32 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256931.0039A073 ; Fri, 4 Aug 2000 12:29:26 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 4 Aug 2000 13:27:07 +0300 Subject: IETF48 - foils available Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk After hearing several complaints - I've placed a new PS file at the same address. I hope it works better (although I can tell only that it is twice as large!). Julo ---------------------- Forwarded by Julian Satran/Haifa/IBM on 04/08/2000 13:24 --------------------------- Julian Satran 03/08/2000 13:16 To: ips@ece.cmu.edu cc: From: Julian Satran/Haifa/IBM@IBMIL Subject: IETF48 - foils available The slides I presented at IETF-48 are available at: htpp://www.haifa.il.ibm.com/satran/IETF48-iSCSI.ps Hopefully they will appear on IPS site (with Dave Nagle's kind help). Regards, Julo From owner-ips@ECE.cmu.edu Fri Aug 04 16:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24132; 4 Aug 2000 12:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA13861 for ips-outgoing; Fri, 4 Aug 2000 11:18:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA12833 for ; Fri, 4 Aug 2000 10:51:38 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 37DC41BBE; Fri, 4 Aug 2000 07:51:36 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA19274; Fri, 4 Aug 2000 07:50:45 -0700 (PDT) Message-Id: <4.2.2.20000804065510.00b42900@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 04 Aug 2000 07:07:11 -0700 To: mark.carlson@sun.com, julian_satran@il.ibm.com From: Michael Krause Subject: Re: Requirements specification Cc: ips@ece.cmu.edu In-Reply-To: <398A89BF.95543187@sun.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 03:15 AM 8/4/00 -0600, Mark A. Carlson wrote: >julian_satran@il.ibm.com wrote: > > > > David, > > > > The one additional requirement is availability/fault-tolerance. > >Could you elaborate on this and tell us how multiple connections >between the same two NICs enhance availability? Availability is a function of the paths and components in use: - Multi-port NICs provide protection from cable disconnect (the most common form of failure) at the NIC level. Multi-port also allows the control data to be shared within the NIC making fail-over and performance enhancements simpler to implement. There are many proof-of-concepts as well as products running today that have demonstrated these values. - A given port is capable of surviving failure within the intermediate fabric, e.g. a cable disconnect between intermediate switches or a switch failure. Support for multiple connections allows traffic to be segregated to different paths and thus survive this type of failure. >Also, can you state what the availability requirements are? - Application transparent survival all forms of fabric failures. - Application transparent fail-over to alternative paths, alternative hardware, etc. - Ability to deal with hot-plug / hot-removal of fabric elements in an application transparent way. For example, if a hot-plug event occurs and a new path is established which happens to provide higher bandwidth or better QoS, the session layer would be able to add a new connection to flow over that path. The connection arbitration policies could be adjusted to skew the operations to that path since it provides the better performance, etc. to the application. All of this can occur without involving the application itself being accomplished within the iSCSI session layer. > > Your arguments about performance are valid. However I doubt that there will > > be enough incentives - beyond price - to develop things for high end > > controllers and servers. > > > > Enabling multiple connections brings those applications the performance > > required without any serious implications to the rest of the "family" > (as I outlined > > in Pittsburgh controllers and servers that don't need multiple > connections/session don't have to implement them). > >The serious implications that I think everyone is concerned about is the >complexity that we are introducing into the protocol and the question of >whether this really provides the performance that is asserted. Suggestion is for those that advocate to show proof of concept / modeling that shows the benefit. We can debate all we want but demonstration is the key to showing the benefits. If we can get agreement on the basic operational model, I'm sure several people will step up to prove it one way or the other. > > Storage traffic requirements will always exceed those of many other > > applications. > > > > As for the "one-connection-per-LU" we covered this solution in long > > discussions > > and even several full fledged implementation - as it is compelingly simple. > > However the resource consumption is unjustifiably high and the security > > problems are > > even worse (the LUs "viewed" by an initiator depend on who he says he is) > > than > > in the current draft. > >Chances are that each LUN could be owned by separate initiators and require >authentication to separate principals anyway. I think that this is the >requirement we need to keep in mind. Mike From owner-ips@ECE.cmu.edu Fri Aug 04 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24479; 4 Aug 2000 13:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16612 for ips-outgoing; Fri, 4 Aug 2000 12:23:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16459 for ; Fri, 4 Aug 2000 12:18:50 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA01288; Fri, 4 Aug 2000 09:15:54 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA08582; Fri, 4 Aug 2000 09:13:13 -0700 From: "Douglas Otis" To: , Subject: RE: Requirements specification Date: Fri, 4 Aug 2000 09:20:40 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Fri Aug 04 21:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25252; 4 Aug 2000 16:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA25290 for ips-outgoing; Fri, 4 Aug 2000 15:46:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA21971 for ; Fri, 4 Aug 2000 14:37:53 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id LAA09789 for ; Fri, 4 Aug 2000 11:37:52 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.151.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA27587 for ; Fri, 4 Aug 2000 11:37:51 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA15308; Fri, 4 Aug 2000 11:37:51 -0700 (PDT) Message-Id: <200008041837.LAA15308@ha10nwk.EBay.Sun.COM> Date: Fri, 4 Aug 2000 11:37:51 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Requirements specification To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: S8p2gDBy0DKeFB1QDKEg4Q== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > The one additional requirement is availability/fault-tolerance. I agree that we need to establish the availability requirements of IPS, but I see them as orthogonal to the connection discussion. IP already has the wonderful rerouting properties to be fault resilient as well as link level technology to do failover. From an IPS perspective the host adapters and the storage adapters are the weak links. This to do a reasonable HA solution you will need multiple adapters on both ends which will imply initiator/target pairs. So I don't see how having multiple TCP connections per session will increase availability. Existing HA solutions that layer on top of FC or parallel SCSI could just as easily layer on top of IPS. > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). I am not sure that I understand your point here. As has been mentioned by others, one performance concern is how to maximize utilization of the available link bandwidth with a higher level protocol, fundementally you can never get 100%. Part of the concern is that most existing TCP implementations are not scaling up as fast as the link layer protocols. To handle this situation people consider some form of aggregation of TCP links to relieve the pressure on the implementation side. The question becomes what is the appropriate way to aggregate. If we run multiple LUNs over the same session then we will push the TCP implementation to its limits. However, if we run a session per LUN, the bandwidth pressures will be dramatically less on TCP, we will more easily utilize existing and future link layer bandwidth, and simplify the multiplexing by leaving it in the TCP layer. > Storage traffic requirements will always exceed those of many other > applications. Storage will always exceed thoses of *some* other applications, it is easy to find numerous applications that exceed storage requirements. I am not clear on the point, but we need to make sure that we clearly define what the storage traffic requirements are. > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. I don't agree that the security problems are worse. For any session, whether it is a session per target or a session per LUN will need to do full security negotiation. Realitive to the expected lifetime of the sessions the time and complexity will be negligible. If we adequately solve it for one model it should trivially apply to the other. I also don't agree on the resource consumption being too high. In terms of memory resources the necessary data buffering is dependant on the aggregate flow between the target and initiator which will be independent of how many connections were used to get the data to the other side. There will be more connections, but within typical TCP implementations the size and complexity of control blocks is minimal. The ability to switch and manage many connections rapidly is a long solved problem. It would be illumninating to hear if Adaptec has found that their connection per LUN is resource consuming. -David From owner-ips@ECE.cmu.edu Fri Aug 04 21:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25232; 4 Aug 2000 16:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA25273 for ips-outgoing; Fri, 4 Aug 2000 15:45:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pallas.veritas.com (pallas.veritas.com [204.177.156.25]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA20017 for ; Fri, 4 Aug 2000 13:49:25 -0400 (EDT) Received: from megami.veritas.com (megami.veritas.com [192.203.46.101]) by pallas.veritas.com (8.9.1a/8.9.1) with SMTP id KAA23003; Fri, 4 Aug 2000 10:54:27 -0700 (PDT) Received: from localhost (1132 bytes) by megami.veritas.com via sendmail with P:stdio/R:smart_host/T:smtp (sender: ) (ident using unix) id for ; Fri, 4 Aug 2000 10:49:17 -0700 (PDT) (Smail-3.2.0.101 1997-Dec-17 #4 built 1999-Aug-24) Date: Fri, 4 Aug 2000 10:49:17 -0700 From: Ravi Venkat To: julian_satran@il.ibm.com Cc: Peter Johansson , ips@ece.cmu.edu Subject: Re: IETF48 - foils available Message-ID: <20000804104917.F26718@megami.veritas.com> Reply-To: ravindra@veritas.com References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0us In-Reply-To: ; from julian_satran@il.ibm.com on Fri, Aug 04, 2000 at 05:01:33AM +0300 Sender: owner-ips@ece.cmu.edu Precedence: bulk On Fri, Aug 04, 2000 at 05:01:33AM +0300, julian_satran@il.ibm.com wrote: > > > Ready To Transfer - it is a "prompt" for data from target to initiator. > > Julo > > Peter Johansson on 03/08/2000 20:43:51 > > Please respond to Peter Johansson I am not able to read the pdf nor the ps file on Solaris. Is it just me? Can you or someone point to an alternate location of the doc? Thanks in advance, -- Ravi From owner-ips@ECE.cmu.edu Fri Aug 04 21:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25234; 4 Aug 2000 16:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA25275 for ips-outgoing; Fri, 4 Aug 2000 15:45:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA20896 for ; Fri, 4 Aug 2000 14:13:00 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel1.hp.com (Postfix) with ESMTP id 224061504; Fri, 4 Aug 2000 11:08:32 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id LAA22179; Fri, 4 Aug 2000 11:08:28 -0700 (PDT) X-OpenMail-Hops: 1 Date: Fri, 4 Aug 2000 11:08:21 -0700 Message-Id: Subject: RE: Re: Requirements specification MIME-Version: 1.0 To: julian_satran@il.ibm.com, mark.carlson@sun.com Cc: ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk The primary reason to have multiple connections per session are cases where the bandwidth requirements for a related set of commands (ordered) exceeds the capabilities of a single link. I am sure there is a desire for this in high end apps (tapes, remote mirroring etc) regardless of the current size of the fat pipe. The other arguments do sound a little weak to me. A reasonable implementation should be able to drive a single connection at link speed - so to have features of the spec that accomodate poor implementations is perhaps not the best thing to do. Coming hardware accelerators will only make this more prevelent. Also if the commands are not related, i.e. going to different LUNs in the same physical device, you could always open multiple unrelated sessions to that device. How many sessions to open etc would be an interesting implementation problem. Also as soon as the spec specifies mechanisms to recover sessions with a single connection, one does not need multiple connections for recovery. We have always seen demand from customers/envrionments where there is a desire to transfer a single stream (like ftp) faster than the fastest currently available link i.e. like the case in the first para. It would be reasonable to accomodate that. We have to put more work into the spec to show how the multiple connections per session would work in terms of an architecture/ paper design. I think this would alleviate the fears of complexity (I think it will not be too complex - or it might show that it is indeed too complex). Somesh -----Original Message----- From: mark.carlson@sun.com [mailto:mark.carlson@sun.com] Sent: Friday, August 04, 2000 2:16 AM To: mark.carlson@sun.com; julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: FW: Re: Requirements specification julian_satran@il.ibm.com wrote: > > David, > > The one additional requirement is availability/fault-tolerance. Could you elaborate on this and tell us how multiple connections between the same two NICs enhance availability? Also, can you state what the availability requirements are? > > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). The serious implications that I think everyone is concerned about is the complexity that we are introducing into the protocol and the question of whether this really provides the performance that is asserted. > > Storage traffic requirements will always exceed those of many other > applications. > > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. Chances are that each LUN could be owned by separate initiators and require authentication to separate principals anyway. I think that this is the requirement we need to keep in mind. -- mark From owner-ips@ECE.cmu.edu Fri Aug 04 21:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25247; 4 Aug 2000 16:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA25282 for ips-outgoing; Fri, 4 Aug 2000 15:45:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA20926 for ; Fri, 4 Aug 2000 14:13:40 -0400 (EDT) From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA47176 for ; Fri, 4 Aug 2000 14:11:24 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id MAA82610 for ; Fri, 4 Aug 2000 12:13:34 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256931.00641D81 ; Fri, 4 Aug 2000 12:13:31 -0600 X-Lotus-FromDomain: IBMUS To: ips@ece.cmu.edu Message-ID: <87256931.0062B234.00@d53mta03h.boulder.ibm.com> Date: Fri, 4 Aug 2000 10:58:15 -0700 Subject: RE: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Two points: 1. The iSCSI presenters seem to imply that having 'n' connections through the IP fabric will automatically give you 'n' (or close to 'n') times the bandwidth if you round-robin your packets across the connections. This is probably true only in the very ideal case and in the real world, the situation is far less rosy to make such an assertion so confidently. 2. Fault-tolerance currently is (and should be) layered above the SCSI transport layer. There are enough solutions from several vendors in the market which deal with this, so there is no use in reinventing the wheel. Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" principles, I would appeal to the iSCSI "design team" to forgo multiple connections per session and use specialized solutions for remote mirroring and remote tape backup (the apps that require multiple connections). Sincerely, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose "Douglas Otis" @ece.cmu.edu on 08/04/2000 09:20:40 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, cc: Subject: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Fri Aug 04 21:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25257; 4 Aug 2000 16:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA25255 for ips-outgoing; Fri, 4 Aug 2000 15:45:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA19518 for ; Fri, 4 Aug 2000 13:35:54 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by atlrel2.hp.com (Postfix) with ESMTP id AD240C26; Fri, 4 Aug 2000 13:35:53 -0400 (EDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id KAA14346; Fri, 4 Aug 2000 10:35:53 -0700 (PDT) X-OpenMail-Hops: 1 Date: Fri, 4 Aug 2000 10:35:44 -0700 Message-Id: Subject: RE: RE: Requirements specification MIME-Version: 1.0 To: dotis@sanlight.net, ips@ece.cmu.edu, julian_satran@il.ibm.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk The way it is currently defined allows implementors to implement one connection per lun or one connection per (arbitrary) connection of luns. I think this is more flexible as it would allow either implementation or migration over time from one form to the other. Somesh -----Original Message----- From: dotis@sanlight.net [mailto:dotis@sanlight.net] Sent: Friday, August 04, 2000 9:21 AM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: FW: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Sat Aug 05 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27907; 5 Aug 2000 3:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA07833 for ips-outgoing; Sat, 5 Aug 2000 02:05:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from storeage_nt.store-age.com ([199.203.178.211]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA00704 for ; Fri, 4 Aug 2000 18:38:36 -0400 (EDT) Received: from e4u5e0 (ras5-p132.hfa.netvision.net.il [62.0.149.132]) by storeage_nt.store-age.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id P8MP59SK; Sat, 5 Aug 2000 01:36:45 +0200 Message-ID: <008f01bffe6c$98fffea0$35bf003e@e4u5e0> From: "Nelson Nahum" To: , References: <87256931.0062B234.00@d53mta03h.boulder.ibm.com> Subject: Re: Requirements specification Date: Sat, 5 Aug 2000 01:34:07 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree that the multiple connections per session should be removed. After all, the parallel SCSI and the FC SCSI uses one "SCSI session" per channel. Today Load Balancing and Fault Tolerance accross multiple channels are handle at a higher level (Dynamic Multi Pathing at the host level) and this should continue to work with the iSCSI as its works with FC or parallel SCSI. The question of using "fat pipe" can be addressed at a lower level (e.g. Infiniband). So why to complicate the iSCSI standard and implementation with something that is already solved at different levels? Nelson Nahum CTO & R&D Manager StoreAge Networking Technologies ----- Original Message ----- From: To: Sent: Friday, August 04, 2000 7:58 PM Subject: RE: Requirements specification > > > > Two points: > > 1. The iSCSI presenters seem to imply that having 'n' connections through > the IP > fabric will automatically give you 'n' (or close to 'n') times the > bandwidth if you > round-robin your packets across the connections. This is probably true only > in the > very ideal case and in the real world, the situation is far less rosy to > make such > an assertion so confidently. > > 2. Fault-tolerance currently is (and should be) layered above the SCSI > transport layer. There are > enough solutions from several vendors in the market which deal with this, > so there is no use in > reinventing the wheel. > > Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" > principles, > I would appeal to the iSCSI "design team" to forgo multiple connections per > session and use specialized > solutions for remote mirroring and remote tape backup (the apps that > require multiple > connections). > > Sincerely, > Prasenjit > > Prasenjit Sarkar > Research Staff Member > IBM Almaden Research > San Jose > > > "Douglas Otis" @ece.cmu.edu on 08/04/2000 09:20:40 AM > > Sent by: owner-ips@ece.cmu.edu > > > To: Julian Satran/Haifa/IBM@IBMIL, > cc: > Subject: RE: Requirements specification > > > > Julo, > > You comments are based on several assumptions reflecting your present > architecture. Your implementation is done at the controller rather than a > device. You also assume authentication is done at the controller. Each > LUN > could belong to a different authority and be an independent (virtual) > device > managed through LDAP. If you bring the interface to the device, you can > obtain the required scaling that is otherwise difficult at the controller > as > with your architecture. By combining everything into a single connection, > you do not improve reliability, scalability, availability or fault > tolerance. > > Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Thursday, August 03, 2000 7:37 PM > To: ips@ece.cmu.edu > Subject: Re: Requirements specification > > > > > David, > > The one additional requirement is availability/fault-tolerance. > > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). > > Storage traffic requirements will always exceed those of many other > applications. > > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. > > Regards, > Julo > > > > David Robinson on 04/08/2000 02:43:11 > > Please respond to David Robinson > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Requirements specification > > > > > To further elaborate on my comments in Pittsburgh on multiple > connections per link and connections per LUN vs per target. > > The current requirements specify that the protocol must support > multiple connections per session. So far the only justification > for this that I have clearly heard is performance, current and future > systems will demand bandwidth that will require aggregation. Is there > any other reason for multiple connections? > > My challenge to this requirement is that it is fundementally a link > and transport layer issue that is being exposed to the session layer > due to a perception that current link/transport implementations are not > adequate to meet perceived demand. The key question here is if this > is a "physics" issue that can't be solved with better implementations > or just bad implementations? I am leaning towards the latter. I expect > that if this protocol is a success, a number of highly tuned adapters > using tricks such as hardware assist will be developed. Those doing > the development will have direct control over the quality of the > implementation. Furthermore, the performance critical environments > are likely to be local in nature so preassure to create necessary > switches and routers will also exist. > > The advantages of limiting a single connection per session should be > a simplification in the connection management and error handling. From > the earliest drafts we have already seen restrictions of individual > command/data/status sequences to a single connection to better handle > ordering issues. I forsee further restrictions possibly being > required to cover handling of lost connections when sequences are > received out of across multiple connections. Similarily Steve's > comments on security management of multiple connections is of concern. > > The second area that I brought up was the requirement of one session > per initiator target pair instead of one per LUN (i.e. SEP). I am willing > to accept the design constraint that a single target must address > 10,000 LUNs which can be done with a connection per LUN. However, > statements of scaling much higher into the areas where 64K port > limitations appear I think is not reasonable. Given the bandwidth > available on today's and near future drives that will easily > exceed 100MBps I can't imagine designing and deploying storage systems > with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps > networks this will be a horrible throughput bottleneck that will > get worse as storage adapters appear to be gaining bandwidth faster than > networks. Therefore requiring greater than 10,000 doesn't seem necessary. > > >From the performance perspective, a connection per LUN also makes sense. > SCSI command flows are already being constrained to a single connection > in the current proposal for ordering reasons, so the number of > concurrent outstanding requests per LUN is a manageable number. The > concurrency desired by multiple connections per session in the > existing draft will naturally occur with a connection per LUN. As > each TCP connection is a unique flow existing link layer hardware > that tries to preserve ordering based on a "flow" (likely IP/port pairs) > will give the desired performance properties. Both my objections and > the requirements for multiple connections I question above become moot. > > >From a connection management, command ordering, and error recover > perspective things should also get simplier. Ordering is obviously > maintained and the sender can now recover from connection errors > based on a smaller context and possibly use TCP layer information > to determine what responses were received (ACK windows?). > > To summarize I would like to see the requirements changed to reflect > a maximum of 64K LUNs per IP node, require only one transport layer > connection per session, and define a session to be an initiator/LUN > pair. > > -David > > > > > > > From owner-ips@ECE.cmu.edu Sat Aug 05 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27909; 5 Aug 2000 3:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA07856 for ips-outgoing; Sat, 5 Aug 2000 02:05:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA04709 for ; Fri, 4 Aug 2000 22:17:45 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 61DC91206; Fri, 4 Aug 2000 19:17:44 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA07296; Fri, 4 Aug 2000 19:16:53 -0700 (PDT) Message-Id: <4.2.2.20000804185615.00b69100@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 04 Aug 2000 19:02:22 -0700 To: psarkar@almaden.ibm.com, ips@ece.cmu.edu From: Michael Krause Subject: RE: Requirements specification In-Reply-To: <87256931.0062B234.00@d53mta03h.boulder.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 10:58 AM 8/4/00 -0700, psarkar@almaden.ibm.com wrote: >Two points: > >1. The iSCSI presenters seem to imply that having 'n' connections through >the IP fabric will automatically give you 'n' (or close to 'n') times the >bandwidth if you round-robin your packets across the connections. This is >probably true only in the very ideal case and in the real world, the >situation is far less rosy to make such an assertion so confidently. Not true that this has to be the ideal world - there are numerous applications that maintain high-bandwidth through multiple connections on the same path or across several paths. Also, network routers often implement RED / WRED schemes which may only impact a subset of the sessions connections. >2. Fault-tolerance currently is (and should be) layered above the SCSI >transport layer. There are enough solutions from several vendors in the >market which deal with this, >so there is no use in reinventing the wheel. Fault tolerance is a highest level of availability there are other levels that allow one to recover from a variety of failure types without requiring to implement to this level. Most of these can be simply implemented below the iSCSI level. Also, dynamic changes in the network composition can be dealt with without resorting to high-level changes to the applications. The objective of improving the availability is to not be required to use this overly complex solutions but to create simple, low-implementation costs solutions that will benefit the majority of the customers. >Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" >principles,I would appeal to the iSCSI "design team" to forgo multiple >connections per >session and use specialized solutions for remote mirroring and remote tape >backup (the apps that require multiple connections). Multiple connections can be done using a KISS principle and more robust solutions can be built on top of this solution to expand the functional capabilities without requiring the architecture to be radically permuted in the process. Mike From owner-ips@ECE.cmu.edu Sat Aug 05 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29509; 5 Aug 2000 8:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA20526 for ips-outgoing; Sat, 5 Aug 2000 07:48:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA20512 for ; Sat, 5 Aug 2000 07:48:02 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA73000 for ; Sat, 5 Aug 2000 13:47:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id NAA139574 for ; Sat, 5 Aug 2000 13:47:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.0040C444 ; Sat, 5 Aug 2000 13:47:25 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 14:45:44 +0300 Subject: RE: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, The current architecture is good for the whole spectrum. If you are intent on using it for a disk drive you can do so and fill with 0 the fields you are not interested in. You don't have to implement the functions that are intended for controllers. The controller/drive scaling controversy is certainly outside the scope of iSCSI. Regards, Julo "Douglas Otis" on 04/08/2000 19:20:40 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Sat Aug 05 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29511; 5 Aug 2000 8:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA20891 for ips-outgoing; Sat, 5 Aug 2000 08:16:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA20877 for ; Sat, 5 Aug 2000 08:15:57 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id OAA93840 for ; Sat, 5 Aug 2000 14:15:25 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id OAA20946 for ; Sat, 5 Aug 2000 14:15:25 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.00435212 ; Sat, 5 Aug 2000 14:15:19 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 15:13:40 +0300 Subject: Re: IETF48 - foils available Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk There is now also a PDF version: http://www.haifa.il.ibm.com/satran/IETF48-iSCSI.pdf I hope you can see this. Julo Ravi Venkat on 04/08/2000 20:49:17 Please respond to ravindra@veritas.com To: julian_satran%ibmil.RSCS@DEVM.DE.IBM.COM cc: Peter Johansson , ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: IETF48 - foils available On Fri, Aug 04, 2000 at 05:01:33AM +0300, julian_satran@il.ibm.com wrote: > > > Ready To Transfer - it is a "prompt" for data from target to initiator. > > Julo > > Peter Johansson on 03/08/2000 20:43:51 > > Please respond to Peter Johansson I am not able to read the pdf nor the ps file on Solaris. Is it just me? Can you or someone point to an alternate location of the doc? Thanks in advance, -- Ravi From owner-ips@ECE.cmu.edu Sat Aug 05 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29507; 5 Aug 2000 8:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA20748 for ips-outgoing; Sat, 5 Aug 2000 08:05:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA20734 for ; Sat, 5 Aug 2000 08:05:38 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA23066 for ; Sat, 5 Aug 2000 14:05:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id OAA17830 for ; Sat, 5 Aug 2000 14:05:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.0042616D ; Sat, 5 Aug 2000 14:05:03 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 15:03:19 +0300 Subject: RE: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Prasanjit, I will not answer the first point - it is a truism. You will have to have a decent load balnacing scheme. And the draft does not allude at linear scaling. As for the second there are many other applications that can benefit from several connections for both speed and availability. FC failed to realize that error recovery is NOT done at SCSI layer in most cases (parallel SCSI connections don't fail) and had to introduce it afterwards. Many other benefits of several connections are the potential to add connections as you grow without having to abandon old connections, separate bandwidth management from storage management etc. Backup/mirroring are not the only application that will gain from using several connections. Taking the view that the any complexity is bad is shortsighted in this context. Julo Please respond to psarkar@almaden.ibm.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Requirements specification Two points: 1. The iSCSI presenters seem to imply that having 'n' connections through the IP fabric will automatically give you 'n' (or close to 'n') times the bandwidth if you round-robin your packets across the connections. This is probably true only in the very ideal case and in the real world, the situation is far less rosy to make such an assertion so confidently. 2. Fault-tolerance currently is (and should be) layered above the SCSI transport layer. There are enough solutions from several vendors in the market which deal with this, so there is no use in reinventing the wheel. Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" principles, I would appeal to the iSCSI "design team" to forgo multiple connections per session and use specialized solutions for remote mirroring and remote tape backup (the apps that require multiple connections). Sincerely, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose "Douglas Otis" @ece.cmu.edu on 08/04/2000 09:20:40 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, cc: Subject: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Sat Aug 05 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29743; 5 Aug 2000 9:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA21238 for ips-outgoing; Sat, 5 Aug 2000 08:44:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA21224 for ; Sat, 5 Aug 2000 08:44:04 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA274072 for ; Sat, 5 Aug 2000 14:43:32 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id OAA23836 for ; Sat, 5 Aug 2000 14:43:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.0045D84C ; Sat, 5 Aug 2000 14:42:53 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 15:41:10 +0300 Subject: Re: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, The resource consumption I was referring to is not the buffer space but rather state maintenance; the common figure I was given is 6k/socket. Considering also the limited number of sockets in the OS environments used by today controllers it did not seem a neat solution. And as it was pointed out already from this design you can go to one connection/LU while the reverse does not hold. There is also a structural advantage of connecting to a target not a LU - the LU map may change while you are working. If you leave all the multiplexing to TCP - including using several physical connections I am not aware of any good technology to do it today (as TCP traffic is bound to the end-point IP addresses). We could decide to throw back the balancing need to the TCP implementers but I am not sure that this will get us very far. Availability is not an issue for parallel SCSI and it is not there (there are no recovery mechanisms on SCSI) ; FCP designer found out about this quite late in the game and moved to FCP-2 (that introduces a fairly complex recovery mechanism). Julo David Robinson on 04/08/2000 21:37:51 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Requirements specification > The one additional requirement is availability/fault-tolerance. I agree that we need to establish the availability requirements of IPS, but I see them as orthogonal to the connection discussion. IP already has the wonderful rerouting properties to be fault resilient as well as link level technology to do failover. From an IPS perspective the host adapters and the storage adapters are the weak links. This to do a reasonable HA solution you will need multiple adapters on both ends which will imply initiator/target pairs. So I don't see how having multiple TCP connections per session will increase availability. Existing HA solutions that layer on top of FC or parallel SCSI could just as easily layer on top of IPS. > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). I am not sure that I understand your point here. As has been mentioned by others, one performance concern is how to maximize utilization of the available link bandwidth with a higher level protocol, fundementally you can never get 100%. Part of the concern is that most existing TCP implementations are not scaling up as fast as the link layer protocols. To handle this situation people consider some form of aggregation of TCP links to relieve the pressure on the implementation side. The question becomes what is the appropriate way to aggregate. If we run multiple LUNs over the same session then we will push the TCP implementation to its limits. However, if we run a session per LUN, the bandwidth pressures will be dramatically less on TCP, we will more easily utilize existing and future link layer bandwidth, and simplify the multiplexing by leaving it in the TCP layer. > Storage traffic requirements will always exceed those of many other > applications. Storage will always exceed thoses of *some* other applications, it is easy to find numerous applications that exceed storage requirements. I am not clear on the point, but we need to make sure that we clearly define what the storage traffic requirements are. > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. I don't agree that the security problems are worse. For any session, whether it is a session per target or a session per LUN will need to do full security negotiation. Realitive to the expected lifetime of the sessions the time and complexity will be negligible. If we adequately solve it for one model it should trivially apply to the other. I also don't agree on the resource consumption being too high. In terms of memory resources the necessary data buffering is dependant on the aggregate flow between the target and initiator which will be independent of how many connections were used to get the data to the other side. There will be more connections, but within typical TCP implementations the size and complexity of control blocks is minimal. The ability to switch and manage many connections rapidly is a long solved problem. It would be illumninating to hear if Adaptec has found that their connection per LUN is resource consuming. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05342; 9 Aug 2000 1:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16457 for ips-outgoing; Tue, 8 Aug 2000 23:51:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA20734 for ; Sat, 5 Aug 2000 08:05:38 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id OAA23066 for ; Sat, 5 Aug 2000 14:05:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id OAA17830 for ; Sat, 5 Aug 2000 14:05:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.0042616D ; Sat, 5 Aug 2000 14:05:03 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 15:03:19 +0300 Subject: RE: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Prasanjit, I will not answer the first point - it is a truism. You will have to have a decent load balnacing scheme. And the draft does not allude at linear scaling. As for the second there are many other applications that can benefit from several connections for both speed and availability. FC failed to realize that error recovery is NOT done at SCSI layer in most cases (parallel SCSI connections don't fail) and had to introduce it afterwards. Many other benefits of several connections are the potential to add connections as you grow without having to abandon old connections, separate bandwidth management from storage management etc. Backup/mirroring are not the only application that will gain from using several connections. Taking the view that the any complexity is bad is shortsighted in this context. Julo Please respond to psarkar@almaden.ibm.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Requirements specification Two points: 1. The iSCSI presenters seem to imply that having 'n' connections through the IP fabric will automatically give you 'n' (or close to 'n') times the bandwidth if you round-robin your packets across the connections. This is probably true only in the very ideal case and in the real world, the situation is far less rosy to make such an assertion so confidently. 2. Fault-tolerance currently is (and should be) layered above the SCSI transport layer. There are enough solutions from several vendors in the market which deal with this, so there is no use in reinventing the wheel. Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" principles, I would appeal to the iSCSI "design team" to forgo multiple connections per session and use specialized solutions for remote mirroring and remote tape backup (the apps that require multiple connections). Sincerely, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose "Douglas Otis" @ece.cmu.edu on 08/04/2000 09:20:40 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, cc: Subject: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05262; 9 Aug 2000 1:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16983 for ips-outgoing; Wed, 9 Aug 2000 00:05:38 -0400 (EDT) Date: Wed, 9 Aug 2000 00:05:38 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090405.AAA16983@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28299 for ; Sat, 5 Aug 2000 15:31:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA39702 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id VAA14510 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.006B2EFB ; Sat, 5 Aug 2000 21:30:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 22:29:04 +0300 Subject: food for thought Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Multiple TCP connections per session is one of those things that is very easy to remove but hard to add as an afterthought. Julo From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05322; 9 Aug 2000 1:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16305 for ips-outgoing; Tue, 8 Aug 2000 23:48:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA12833 for ; Fri, 4 Aug 2000 10:51:38 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 37DC41BBE; Fri, 4 Aug 2000 07:51:36 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA19274; Fri, 4 Aug 2000 07:50:45 -0700 (PDT) Message-Id: <4.2.2.20000804065510.00b42900@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 04 Aug 2000 07:07:11 -0700 To: mark.carlson@sun.com, julian_satran@il.ibm.com From: Michael Krause Subject: Re: Requirements specification Cc: ips@ece.cmu.edu In-Reply-To: <398A89BF.95543187@sun.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 03:15 AM 8/4/00 -0600, Mark A. Carlson wrote: >julian_satran@il.ibm.com wrote: > > > > David, > > > > The one additional requirement is availability/fault-tolerance. > >Could you elaborate on this and tell us how multiple connections >between the same two NICs enhance availability? Availability is a function of the paths and components in use: - Multi-port NICs provide protection from cable disconnect (the most common form of failure) at the NIC level. Multi-port also allows the control data to be shared within the NIC making fail-over and performance enhancements simpler to implement. There are many proof-of-concepts as well as products running today that have demonstrated these values. - A given port is capable of surviving failure within the intermediate fabric, e.g. a cable disconnect between intermediate switches or a switch failure. Support for multiple connections allows traffic to be segregated to different paths and thus survive this type of failure. >Also, can you state what the availability requirements are? - Application transparent survival all forms of fabric failures. - Application transparent fail-over to alternative paths, alternative hardware, etc. - Ability to deal with hot-plug / hot-removal of fabric elements in an application transparent way. For example, if a hot-plug event occurs and a new path is established which happens to provide higher bandwidth or better QoS, the session layer would be able to add a new connection to flow over that path. The connection arbitration policies could be adjusted to skew the operations to that path since it provides the better performance, etc. to the application. All of this can occur without involving the application itself being accomplished within the iSCSI session layer. > > Your arguments about performance are valid. However I doubt that there will > > be enough incentives - beyond price - to develop things for high end > > controllers and servers. > > > > Enabling multiple connections brings those applications the performance > > required without any serious implications to the rest of the "family" > (as I outlined > > in Pittsburgh controllers and servers that don't need multiple > connections/session don't have to implement them). > >The serious implications that I think everyone is concerned about is the >complexity that we are introducing into the protocol and the question of >whether this really provides the performance that is asserted. Suggestion is for those that advocate to show proof of concept / modeling that shows the benefit. We can debate all we want but demonstration is the key to showing the benefits. If we can get agreement on the basic operational model, I'm sure several people will step up to prove it one way or the other. > > Storage traffic requirements will always exceed those of many other > > applications. > > > > As for the "one-connection-per-LU" we covered this solution in long > > discussions > > and even several full fledged implementation - as it is compelingly simple. > > However the resource consumption is unjustifiably high and the security > > problems are > > even worse (the LUs "viewed" by an initiator depend on who he says he is) > > than > > in the current draft. > >Chances are that each LUN could be owned by separate initiators and require >authentication to separate principals anyway. I think that this is the >requirement we need to keep in mind. Mike From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05254; 9 Aug 2000 1:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16901 for ips-outgoing; Wed, 9 Aug 2000 00:04:57 -0400 (EDT) Date: Wed, 9 Aug 2000 00:04:57 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090404.AAA16901@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05258; 9 Aug 2000 1:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16514 for ips-outgoing; Tue, 8 Aug 2000 23:52:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA21224 for ; Sat, 5 Aug 2000 08:44:04 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id OAA274072 for ; Sat, 5 Aug 2000 14:43:32 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id OAA23836 for ; Sat, 5 Aug 2000 14:43:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.0045D84C ; Sat, 5 Aug 2000 14:42:53 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 15:41:10 +0300 Subject: Re: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, The resource consumption I was referring to is not the buffer space but rather state maintenance; the common figure I was given is 6k/socket. Considering also the limited number of sockets in the OS environments used by today controllers it did not seem a neat solution. And as it was pointed out already from this design you can go to one connection/LU while the reverse does not hold. There is also a structural advantage of connecting to a target not a LU - the LU map may change while you are working. If you leave all the multiplexing to TCP - including using several physical connections I am not aware of any good technology to do it today (as TCP traffic is bound to the end-point IP addresses). We could decide to throw back the balancing need to the TCP implementers but I am not sure that this will get us very far. Availability is not an issue for parallel SCSI and it is not there (there are no recovery mechanisms on SCSI) ; FCP designer found out about this quite late in the game and moved to FCP-2 (that introduces a fairly complex recovery mechanism). Julo David Robinson on 04/08/2000 21:37:51 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Requirements specification > The one additional requirement is availability/fault-tolerance. I agree that we need to establish the availability requirements of IPS, but I see them as orthogonal to the connection discussion. IP already has the wonderful rerouting properties to be fault resilient as well as link level technology to do failover. From an IPS perspective the host adapters and the storage adapters are the weak links. This to do a reasonable HA solution you will need multiple adapters on both ends which will imply initiator/target pairs. So I don't see how having multiple TCP connections per session will increase availability. Existing HA solutions that layer on top of FC or parallel SCSI could just as easily layer on top of IPS. > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). I am not sure that I understand your point here. As has been mentioned by others, one performance concern is how to maximize utilization of the available link bandwidth with a higher level protocol, fundementally you can never get 100%. Part of the concern is that most existing TCP implementations are not scaling up as fast as the link layer protocols. To handle this situation people consider some form of aggregation of TCP links to relieve the pressure on the implementation side. The question becomes what is the appropriate way to aggregate. If we run multiple LUNs over the same session then we will push the TCP implementation to its limits. However, if we run a session per LUN, the bandwidth pressures will be dramatically less on TCP, we will more easily utilize existing and future link layer bandwidth, and simplify the multiplexing by leaving it in the TCP layer. > Storage traffic requirements will always exceed those of many other > applications. Storage will always exceed thoses of *some* other applications, it is easy to find numerous applications that exceed storage requirements. I am not clear on the point, but we need to make sure that we clearly define what the storage traffic requirements are. > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. I don't agree that the security problems are worse. For any session, whether it is a session per target or a session per LUN will need to do full security negotiation. Realitive to the expected lifetime of the sessions the time and complexity will be negligible. If we adequately solve it for one model it should trivially apply to the other. I also don't agree on the resource consumption being too high. In terms of memory resources the necessary data buffering is dependant on the aggregate flow between the target and initiator which will be independent of how many connections were used to get the data to the other side. There will be more connections, but within typical TCP implementations the size and complexity of control blocks is minimal. The ability to switch and manage many connections rapidly is a long solved problem. It would be illumninating to hear if Adaptec has found that their connection per LUN is resource consuming. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05340; 9 Aug 2000 1:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16985 for ips-outgoing; Wed, 9 Aug 2000 00:05:40 -0400 (EDT) Date: Wed, 9 Aug 2000 00:05:40 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090405.AAA16985@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28299 for ; Sat, 5 Aug 2000 15:31:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA39702 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id VAA14510 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.006B2EFB ; Sat, 5 Aug 2000 21:30:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 22:29:04 +0300 Subject: food for thought Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Multiple TCP connections per session is one of those things that is very easy to remove but hard to add as an afterthought. Julo ------_=_NextPart_000_01C001B6.CDDD4A22-- ------_=_NextPart_000_01C001B6.F202508C-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05332; 9 Aug 2000 1:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16413 for ips-outgoing; Tue, 8 Aug 2000 23:51:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from storeage_nt.store-age.com ([199.203.178.211]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA00704 for ; Fri, 4 Aug 2000 18:38:36 -0400 (EDT) Received: from e4u5e0 (ras5-p132.hfa.netvision.net.il [62.0.149.132]) by storeage_nt.store-age.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id P8MP59SK; Sat, 5 Aug 2000 01:36:45 +0200 Message-ID: <008f01bffe6c$98fffea0$35bf003e@e4u5e0> From: "Nelson Nahum" To: , References: <87256931.0062B234.00@d53mta03h.boulder.ibm.com> Subject: Re: Requirements specification Date: Sat, 5 Aug 2000 01:34:07 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree that the multiple connections per session should be removed. After all, the parallel SCSI and the FC SCSI uses one "SCSI session" per channel. Today Load Balancing and Fault Tolerance accross multiple channels are handle at a higher level (Dynamic Multi Pathing at the host level) and this should continue to work with the iSCSI as its works with FC or parallel SCSI. The question of using "fat pipe" can be addressed at a lower level (e.g. Infiniband). So why to complicate the iSCSI standard and implementation with something that is already solved at different levels? Nelson Nahum CTO & R&D Manager StoreAge Networking Technologies ----- Original Message ----- From: To: Sent: Friday, August 04, 2000 7:58 PM Subject: RE: Requirements specification > > > > Two points: > > 1. The iSCSI presenters seem to imply that having 'n' connections through > the IP > fabric will automatically give you 'n' (or close to 'n') times the > bandwidth if you > round-robin your packets across the connections. This is probably true only > in the > very ideal case and in the real world, the situation is far less rosy to > make such > an assertion so confidently. > > 2. Fault-tolerance currently is (and should be) layered above the SCSI > transport layer. There are > enough solutions from several vendors in the market which deal with this, > so there is no use in > reinventing the wheel. > > Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" > principles, > I would appeal to the iSCSI "design team" to forgo multiple connections per > session and use specialized > solutions for remote mirroring and remote tape backup (the apps that > require multiple > connections). > > Sincerely, > Prasenjit > > Prasenjit Sarkar > Research Staff Member > IBM Almaden Research > San Jose > > > "Douglas Otis" @ece.cmu.edu on 08/04/2000 09:20:40 AM > > Sent by: owner-ips@ece.cmu.edu > > > To: Julian Satran/Haifa/IBM@IBMIL, > cc: > Subject: RE: Requirements specification > > > > Julo, > > You comments are based on several assumptions reflecting your present > architecture. Your implementation is done at the controller rather than a > device. You also assume authentication is done at the controller. Each > LUN > could belong to a different authority and be an independent (virtual) > device > managed through LDAP. If you bring the interface to the device, you can > obtain the required scaling that is otherwise difficult at the controller > as > with your architecture. By combining everything into a single connection, > you do not improve reliability, scalability, availability or fault > tolerance. > > Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Thursday, August 03, 2000 7:37 PM > To: ips@ece.cmu.edu > Subject: Re: Requirements specification > > > > > David, > > The one additional requirement is availability/fault-tolerance. > > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). > > Storage traffic requirements will always exceed those of many other > applications. > > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. > > Regards, > Julo > > > > David Robinson on 04/08/2000 02:43:11 > > Please respond to David Robinson > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Requirements specification > > > > > To further elaborate on my comments in Pittsburgh on multiple > connections per link and connections per LUN vs per target. > > The current requirements specify that the protocol must support > multiple connections per session. So far the only justification > for this that I have clearly heard is performance, current and future > systems will demand bandwidth that will require aggregation. Is there > any other reason for multiple connections? > > My challenge to this requirement is that it is fundementally a link > and transport layer issue that is being exposed to the session layer > due to a perception that current link/transport implementations are not > adequate to meet perceived demand. The key question here is if this > is a "physics" issue that can't be solved with better implementations > or just bad implementations? I am leaning towards the latter. I expect > that if this protocol is a success, a number of highly tuned adapters > using tricks such as hardware assist will be developed. Those doing > the development will have direct control over the quality of the > implementation. Furthermore, the performance critical environments > are likely to be local in nature so preassure to create necessary > switches and routers will also exist. > > The advantages of limiting a single connection per session should be > a simplification in the connection management and error handling. From > the earliest drafts we have already seen restrictions of individual > command/data/status sequences to a single connection to better handle > ordering issues. I forsee further restrictions possibly being > required to cover handling of lost connections when sequences are > received out of across multiple connections. Similarily Steve's > comments on security management of multiple connections is of concern. > > The second area that I brought up was the requirement of one session > per initiator target pair instead of one per LUN (i.e. SEP). I am willing > to accept the design constraint that a single target must address > 10,000 LUNs which can be done with a connection per LUN. However, > statements of scaling much higher into the areas where 64K port > limitations appear I think is not reasonable. Given the bandwidth > available on today's and near future drives that will easily > exceed 100MBps I can't imagine designing and deploying storage systems > with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps > networks this will be a horrible throughput bottleneck that will > get worse as storage adapters appear to be gaining bandwidth faster than > networks. Therefore requiring greater than 10,000 doesn't seem necessary. > > >From the performance perspective, a connection per LUN also makes sense. > SCSI command flows are already being constrained to a single connection > in the current proposal for ordering reasons, so the number of > concurrent outstanding requests per LUN is a manageable number. The > concurrency desired by multiple connections per session in the > existing draft will naturally occur with a connection per LUN. As > each TCP connection is a unique flow existing link layer hardware > that tries to preserve ordering based on a "flow" (likely IP/port pairs) > will give the desired performance properties. Both my objections and > the requirements for multiple connections I question above become moot. > > >From a connection management, command ordering, and error recover > perspective things should also get simplier. Ordering is obviously > maintained and the sender can now recover from connection errors > based on a smaller context and possibly use TCP layer information > to determine what responses were received (ACK windows?). > > To summarize I would like to see the requirements changed to reflect > a maximum of 64K LUNs per IP node, require only one transport layer > connection per session, and define a session to be an initiator/LUN > pair. > > -David > > > > > > > From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05287; 9 Aug 2000 1:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16385 for ips-outgoing; Tue, 8 Aug 2000 23:50:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA21971 for ; Fri, 4 Aug 2000 14:37:53 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id LAA09789 for ; Fri, 4 Aug 2000 11:37:52 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.151.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA27587 for ; Fri, 4 Aug 2000 11:37:51 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA15308; Fri, 4 Aug 2000 11:37:51 -0700 (PDT) Message-Id: <200008041837.LAA15308@ha10nwk.EBay.Sun.COM> Date: Fri, 4 Aug 2000 11:37:51 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Requirements specification To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: S8p2gDBy0DKeFB1QDKEg4Q== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > The one additional requirement is availability/fault-tolerance. I agree that we need to establish the availability requirements of IPS, but I see them as orthogonal to the connection discussion. IP already has the wonderful rerouting properties to be fault resilient as well as link level technology to do failover. From an IPS perspective the host adapters and the storage adapters are the weak links. This to do a reasonable HA solution you will need multiple adapters on both ends which will imply initiator/target pairs. So I don't see how having multiple TCP connections per session will increase availability. Existing HA solutions that layer on top of FC or parallel SCSI could just as easily layer on top of IPS. > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). I am not sure that I understand your point here. As has been mentioned by others, one performance concern is how to maximize utilization of the available link bandwidth with a higher level protocol, fundementally you can never get 100%. Part of the concern is that most existing TCP implementations are not scaling up as fast as the link layer protocols. To handle this situation people consider some form of aggregation of TCP links to relieve the pressure on the implementation side. The question becomes what is the appropriate way to aggregate. If we run multiple LUNs over the same session then we will push the TCP implementation to its limits. However, if we run a session per LUN, the bandwidth pressures will be dramatically less on TCP, we will more easily utilize existing and future link layer bandwidth, and simplify the multiplexing by leaving it in the TCP layer. > Storage traffic requirements will always exceed those of many other > applications. Storage will always exceed thoses of *some* other applications, it is easy to find numerous applications that exceed storage requirements. I am not clear on the point, but we need to make sure that we clearly define what the storage traffic requirements are. > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. I don't agree that the security problems are worse. For any session, whether it is a session per target or a session per LUN will need to do full security negotiation. Realitive to the expected lifetime of the sessions the time and complexity will be negligible. If we adequately solve it for one model it should trivially apply to the other. I also don't agree on the resource consumption being too high. In terms of memory resources the necessary data buffering is dependant on the aggregate flow between the target and initiator which will be independent of how many connections were used to get the data to the other side. There will be more connections, but within typical TCP implementations the size and complexity of control blocks is minimal. The ability to switch and manage many connections rapidly is a long solved problem. It would be illumninating to hear if Adaptec has found that their connection per LUN is resource consuming. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05309; 9 Aug 2000 1:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16342 for ips-outgoing; Tue, 8 Aug 2000 23:49:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA19518 for ; Fri, 4 Aug 2000 13:35:54 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by atlrel2.hp.com (Postfix) with ESMTP id AD240C26; Fri, 4 Aug 2000 13:35:53 -0400 (EDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id KAA14346; Fri, 4 Aug 2000 10:35:53 -0700 (PDT) X-OpenMail-Hops: 1 Date: Fri, 4 Aug 2000 10:35:44 -0700 Message-Id: Subject: RE: RE: Requirements specification MIME-Version: 1.0 To: dotis@sanlight.net, ips@ece.cmu.edu, julian_satran@il.ibm.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk The way it is currently defined allows implementors to implement one connection per lun or one connection per (arbitrary) connection of luns. I think this is more flexible as it would allow either implementation or migration over time from one form to the other. Somesh -----Original Message----- From: dotis@sanlight.net [mailto:dotis@sanlight.net] Sent: Friday, August 04, 2000 9:21 AM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: FW: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05327; 9 Aug 2000 1:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16315 for ips-outgoing; Tue, 8 Aug 2000 23:49:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16459 for ; Fri, 4 Aug 2000 12:18:50 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA01288; Fri, 4 Aug 2000 09:15:54 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA08582; Fri, 4 Aug 2000 09:13:13 -0700 From: "Douglas Otis" To: , Subject: RE: Requirements specification Date: Fri, 4 Aug 2000 09:20:40 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05260; 9 Aug 2000 1:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16476 for ips-outgoing; Tue, 8 Aug 2000 23:52:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA20877 for ; Sat, 5 Aug 2000 08:15:57 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id OAA93840 for ; Sat, 5 Aug 2000 14:15:25 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id OAA20946 for ; Sat, 5 Aug 2000 14:15:25 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.00435212 ; Sat, 5 Aug 2000 14:15:19 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 15:13:40 +0300 Subject: Re: IETF48 - foils available Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk There is now also a PDF version: http://www.haifa.il.ibm.com/satran/IETF48-iSCSI.pdf I hope you can see this. Julo Ravi Venkat on 04/08/2000 20:49:17 Please respond to ravindra@veritas.com To: julian_satran%ibmil.RSCS@DEVM.DE.IBM.COM cc: Peter Johansson , ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: IETF48 - foils available On Fri, Aug 04, 2000 at 05:01:33AM +0300, julian_satran@il.ibm.com wrote: > > > Ready To Transfer - it is a "prompt" for data from target to initiator. > > Julo > > Peter Johansson on 03/08/2000 20:43:51 > > Please respond to Peter Johansson I am not able to read the pdf nor the ps file on Solaris. Is it just me? Can you or someone point to an alternate location of the doc? Thanks in advance, -- Ravi From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05302; 9 Aug 2000 1:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16350 for ips-outgoing; Tue, 8 Aug 2000 23:49:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pallas.veritas.com (pallas.veritas.com [204.177.156.25]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA20017 for ; Fri, 4 Aug 2000 13:49:25 -0400 (EDT) Received: from megami.veritas.com (megami.veritas.com [192.203.46.101]) by pallas.veritas.com (8.9.1a/8.9.1) with SMTP id KAA23003; Fri, 4 Aug 2000 10:54:27 -0700 (PDT) Received: from localhost (1132 bytes) by megami.veritas.com via sendmail with P:stdio/R:smart_host/T:smtp (sender: ) (ident using unix) id for ; Fri, 4 Aug 2000 10:49:17 -0700 (PDT) (Smail-3.2.0.101 1997-Dec-17 #4 built 1999-Aug-24) Date: Fri, 4 Aug 2000 10:49:17 -0700 From: Ravi Venkat To: julian_satran@il.ibm.com Cc: Peter Johansson , ips@ece.cmu.edu Subject: Re: IETF48 - foils available Message-ID: <20000804104917.F26718@megami.veritas.com> Reply-To: ravindra@veritas.com References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0us In-Reply-To: ; from julian_satran@il.ibm.com on Fri, Aug 04, 2000 at 05:01:33AM +0300 Sender: owner-ips@ece.cmu.edu Precedence: bulk On Fri, Aug 04, 2000 at 05:01:33AM +0300, julian_satran@il.ibm.com wrote: > > > Ready To Transfer - it is a "prompt" for data from target to initiator. > > Julo > > Peter Johansson on 03/08/2000 20:43:51 > > Please respond to Peter Johansson I am not able to read the pdf nor the ps file on Solaris. Is it just me? Can you or someone point to an alternate location of the doc? Thanks in advance, -- Ravi From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05313; 9 Aug 2000 1:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16891 for ips-outgoing; Wed, 9 Aug 2000 00:04:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05338; 9 Aug 2000 1:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16373 for ips-outgoing; Tue, 8 Aug 2000 23:50:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA20926 for ; Fri, 4 Aug 2000 14:13:40 -0400 (EDT) From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA47176 for ; Fri, 4 Aug 2000 14:11:24 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id MAA82610 for ; Fri, 4 Aug 2000 12:13:34 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256931.00641D81 ; Fri, 4 Aug 2000 12:13:31 -0600 X-Lotus-FromDomain: IBMUS To: ips@ece.cmu.edu Message-ID: <87256931.0062B234.00@d53mta03h.boulder.ibm.com> Date: Fri, 4 Aug 2000 10:58:15 -0700 Subject: RE: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Two points: 1. The iSCSI presenters seem to imply that having 'n' connections through the IP fabric will automatically give you 'n' (or close to 'n') times the bandwidth if you round-robin your packets across the connections. This is probably true only in the very ideal case and in the real world, the situation is far less rosy to make such an assertion so confidently. 2. Fault-tolerance currently is (and should be) layered above the SCSI transport layer. There are enough solutions from several vendors in the market which deal with this, so there is no use in reinventing the wheel. Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" principles, I would appeal to the iSCSI "design team" to forgo multiple connections per session and use specialized solutions for remote mirroring and remote tape backup (the apps that require multiple connections). Sincerely, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose "Douglas Otis" @ece.cmu.edu on 08/04/2000 09:20:40 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, cc: Subject: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05300; 9 Aug 2000 1:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16867 for ips-outgoing; Wed, 9 Aug 2000 00:04:20 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090404.AAA16867@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f To: owner-ips@ece.cmu.edu Subject: BOUNCE ips@ece.cmu.edu: Approval required: Date: Sat, 5 Aug 2000 12:31:25 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk >From bassoon@ece.cmu.edu Sat Aug 5 15:31:21 2000 Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28299 for ; Sat, 5 Aug 2000 15:31:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA39702 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id VAA14510 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.006B2EFB ; Sat, 5 Aug 2000 21:30:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 22:29:04 +0300 Subject: food for thought Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Multiple TCP connections per session is one of those things that is very easy to remove but hard to add as an afterthought. Julo ------_=_NextPart_000_01C001B6.CDDD4A22-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05289; 9 Aug 2000 1:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17569 for ips-outgoing; Wed, 9 Aug 2000 00:18:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28299 for ; Sat, 5 Aug 2000 15:31:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA39702 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id VAA14510 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.006B2EFB ; Sat, 5 Aug 2000 21:30:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 22:29:04 +0300 Subject: food for thought Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Multiple TCP connections per session is one of those things that is very easy to remove but hard to add as an afterthought. Julo --30173A61.965794676/onet2.cup.hp.com-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05256; 9 Aug 2000 1:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16800 for ips-outgoing; Wed, 9 Aug 2000 00:03:10 -0400 (EDT) From: owner-ips@ece.cmu.edu Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA09806; Sun, 6 Aug 2000 01:12:42 -0400 (EDT) Date: Sun, 6 Aug 2000 01:12:42 -0400 (EDT) Message-Id: <200008060512.BAA09806@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f To: owner-ips@ece.cmu.edu Subject: BOUNCE ips@ece.cmu.edu: Approval required: Sender: owner-ips@ece.cmu.edu Precedence: bulk >From bassoon@ece.cmu.edu Sun Aug 6 01:12:37 2000 Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05311; 9 Aug 2000 1:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17088 for ips-outgoing; Wed, 9 Aug 2000 00:06:44 -0400 (EDT) Date: Wed, 9 Aug 2000 00:06:44 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090406.AAA17088@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28299 for ; Sat, 5 Aug 2000 15:31:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA39702 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id VAA14510 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.006B2EFB ; Sat, 5 Aug 2000 21:30:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 22:29:04 +0300 Subject: food for thought Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Multiple TCP connections per session is one of those things that is very easy to remove but hard to add as an afterthought. Julo ------_=_NextPart_000_01C001B6.CDDD4A22-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05279; 9 Aug 2000 1:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16986 for ips-outgoing; Wed, 9 Aug 2000 00:05:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA21911 for ; Mon, 7 Aug 2000 14:21:15 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id LAA27181 for ; Mon, 7 Aug 2000 11:21:13 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Mon, 7 Aug 2000 11:21:12 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF02EEA04F@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: multiple TCP connections Date: Mon, 7 Aug 2000 11:21:04 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks to all who answered my "help me figure out what I don't know" questions regarding multiple TCP connections. To summarize my current understanding: To my surprise, neither TCP, IP, nor Ethernet have mechanisms for striping traffic across multiple interfaces when that traffic is part of a single TCP connection. (There may be an exception for ethernet in the letter of 802.3ad, but in terms of practical implementations available in the near future, there's no striping avaiable for a single TCP connection.) However, there are mechanisms for striping traffic from multiple TCP connections. We are given the constraint that we can't change TCP. If we assume a requirement to stripe traffic between a single muli-homed host and a single LUN on a multi-homed device across multiple paths, then we need to turn that traffic into traffic over multiple TCP connections. One way of doing this would be to use one TCP connection per SCSI task. (Think HTTP.) To minimize the costly overhead of this approach, we'd probably have to consider something like Transactional TCP. Moveover, this would not increase the transfer speed for a single command. I don't find this approach very palatable. Another way of distributing the traffic over multiple TCP connections is to create the notion of a session, and bind multiple TCP connections to that session. Architecturally this looks yucky, but I prefer sessions over the alternative of one connection per command. So I agree that if we accept the requirement as stated above, then sessions are the logical answer. Discussion of requirements: I happen to agree with the requirement to stripe the traffic between a single muli-homed host and a single LUN on a multi-homed device across multiple paths. For a small, cheap iSCSI implementations near the device level, I'd like to stick with 1 Gb ethernet even as 10 Gb becomes available, in order to minimize costs. Customers really like multiple paths to their devices, so device-level implementations will require two interfaces for this reason alone. Finally, note that media data rates are always increasing, and we're not that many years away from hitting 1 Gb/s. We can argue whether the storage interconnect really needs to match the media rate, since once the drive starts seeking the average data rate drops by orders of magnitude. Regardless of the technical merits, I think interface speed sells in the storage market, so I think customers will continue to demand interface speeds that match the media rate. Consequently I think that low-end iSCSI devices will have a requirement for interface speeds in excess of 2 Gb/s within the next few years, and would best meet this requirement by striping across two 1 Gb ethernet interfaces in an iSCSI session. Complaints about current definition of an iSCSI session: I have one concern about the current definition of an iSCSI session. I'd prefer a striping mechanism that would support striping of the data transfer for a single command across multiple interfaces. As the spec stands now, all messages for a single command have allegiance to one TCP connection, and therefor the all the data transfer for that command must occur over one TCP connection. Given the current architecture of iSCSI, I see the need for this allegiance, since IMHO hardware DMA acceleration requires access to the state of the iSCSI command. However, if we adopt VI/TCP as the enabler for hardware DMA acceleration, the data transfer state is (almost) self-contained in the VI segment header, and I think it becomes feasible to remove the requirement that the data transfer maintain allegiance to the TCP connection that originated the SCSI command. Then we could stripe the data DMA traffic across multiple interfaces, while still keeping all the control and status traffic on the originating TCP connection. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05277; 9 Aug 2000 1:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16810 for ips-outgoing; Wed, 9 Aug 2000 00:03:23 -0400 (EDT) From: owner-ips@ece.cmu.edu Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA28304; Sat, 5 Aug 2000 15:31:25 -0400 (EDT) Date: Sat, 5 Aug 2000 15:31:25 -0400 (EDT) Message-Id: <200008051931.PAA28304@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f To: owner-ips@ece.cmu.edu Subject: BOUNCE ips@ece.cmu.edu: Approval required: Sender: owner-ips@ece.cmu.edu Precedence: bulk >From bassoon@ece.cmu.edu Sat Aug 5 15:31:21 2000 Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28299 for ; Sat, 5 Aug 2000 15:31:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA39702 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id VAA14510 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.006B2EFB ; Sat, 5 Aug 2000 21:30:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 22:29:04 +0300 Subject: food for thought Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Multiple TCP connections per session is one of those things that is very easy to remove but hard to add as an afterthought. Julo From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05334; 9 Aug 2000 1:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16365 for ips-outgoing; Tue, 8 Aug 2000 23:50:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA20896 for ; Fri, 4 Aug 2000 14:13:00 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel1.hp.com (Postfix) with ESMTP id 224061504; Fri, 4 Aug 2000 11:08:32 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id LAA22179; Fri, 4 Aug 2000 11:08:28 -0700 (PDT) X-OpenMail-Hops: 1 Date: Fri, 4 Aug 2000 11:08:21 -0700 Message-Id: Subject: RE: Re: Requirements specification MIME-Version: 1.0 To: julian_satran@il.ibm.com, mark.carlson@sun.com Cc: ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk The primary reason to have multiple connections per session are cases where the bandwidth requirements for a related set of commands (ordered) exceeds the capabilities of a single link. I am sure there is a desire for this in high end apps (tapes, remote mirroring etc) regardless of the current size of the fat pipe. The other arguments do sound a little weak to me. A reasonable implementation should be able to drive a single connection at link speed - so to have features of the spec that accomodate poor implementations is perhaps not the best thing to do. Coming hardware accelerators will only make this more prevelent. Also if the commands are not related, i.e. going to different LUNs in the same physical device, you could always open multiple unrelated sessions to that device. How many sessions to open etc would be an interesting implementation problem. Also as soon as the spec specifies mechanisms to recover sessions with a single connection, one does not need multiple connections for recovery. We have always seen demand from customers/envrionments where there is a desire to transfer a single stream (like ftp) faster than the fastest currently available link i.e. like the case in the first para. It would be reasonable to accomodate that. We have to put more work into the spec to show how the multiple connections per session would work in terms of an architecture/ paper design. I think this would alleviate the fears of complexity (I think it will not be too complex - or it might show that it is indeed too complex). Somesh -----Original Message----- From: mark.carlson@sun.com [mailto:mark.carlson@sun.com] Sent: Friday, August 04, 2000 2:16 AM To: mark.carlson@sun.com; julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: FW: Re: Requirements specification julian_satran@il.ibm.com wrote: > > David, > > The one additional requirement is availability/fault-tolerance. Could you elaborate on this and tell us how multiple connections between the same two NICs enhance availability? Also, can you state what the availability requirements are? > > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). The serious implications that I think everyone is concerned about is the complexity that we are introducing into the protocol and the question of whether this really provides the performance that is asserted. > > Storage traffic requirements will always exceed those of many other > applications. > > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. Chances are that each LUN could be owned by separate initiators and require authentication to separate principals anyway. I think that this is the requirement we need to keep in mind. -- mark From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05252; 9 Aug 2000 1:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16850 for ips-outgoing; Wed, 9 Aug 2000 00:04:09 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090404.AAA16850@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f To: owner-ips@ece.cmu.edu Subject: BOUNCE ips@ece.cmu.edu: Approval required: Date: Sat, 5 Aug 2000 22:12:42 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk >From bassoon@ece.cmu.edu Sun Aug 6 01:12:37 2000 Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- ------_=_NextPart_000_01C001B6.C7EE9030-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05336; 9 Aug 2000 1:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17419 for ips-outgoing; Wed, 9 Aug 2000 00:14:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- --6A141A29.965794463/onet2.cup.hp.com-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05381; 9 Aug 2000 1:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16966 for ips-outgoing; Wed, 9 Aug 2000 00:05:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16240 for ; Mon, 7 Aug 2000 12:12:27 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA01584; Mon, 7 Aug 2000 09:09:03 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA18390; Mon, 7 Aug 2000 09:06:28 -0700 From: "Douglas Otis" To: , Subject: RE: Requirements specification Date: Mon, 7 Aug 2000 09:14:30 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, As your architecture is based on using a controller to aggregate data rather than a switch, you are making choices detrimental to an architecture that brings the interface closer to the device. This is reflected in choices for configuration, authentication and protocol. Your architecture is very close to a Fibre-Channel gateway. Alter solicitation within Fibre-Channel, and there is not a significant difference (Spoofing solicitations within the gateway would be one means). As an example, you require data successfully delivered be retained for possible later solicitation with a controller based error recovery. Two means of delivering data and error recovery is just one example of added complexity due to an inability to scale data handling. Although read-channels and data densities improve at a steady pace, as they have for the past quarter century, mechanics of the drives have not. Today's drives can deliver 320 Mbits/second of data on the outside cylinders. The physical size of the drive in conjunction with the number of heads and disks all have substantial impact in a competitive market with respect to power and cost. The cost/volume trend takes us to a single larger disk which paradoxically increases access time as read channel data rates increase. You optimize to take advantage of the burst performance of the read channel with added complexities attempting to time or stage such transfers through your architectural restrictions where the device becomes part of this fabric. Is it logical to design a system where everything is aimed at taking advantage of the high momentary data rate offered by the read channel, or by offering the same throughput using more devices where each interface bandwidth is 'restricted' with respect to these read channel data rates? The advantage of such an approach is found with respect to smaller random traffic. With more devices, redundancy is easily achieved and parallel access offers a means of performance improvement by spreading activity over more devices. In this case, the switch provides bandwidth aggregation and each device would only see their traffic, but the client could see the traffic of hundreds of these devices. Regardless of the nature of the traffic, performance would be more uniform and control could be left at the client. An 8ms access + latency figure in the high cost drives restricts the number of 'independent' operations that average 64k byte to 100 per second or 52 Mbit per second. Forgoing the peak data rate, such an architecture of 'restricted' drives would scale whereas the controller based approach does not and is vulnerable. An independent nexus at the LUN is the only design that offers the required scaling and configuration flexibility. Switch and client aggregation makes sense in cost, performance, capacity, reliability, and scalability. Protocol overhead should be addressed in the protocol itself and not by means of controller aggregation. There are substantial improvements to be made in the protocol area without the use of intervening controllers. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Saturday, August 05, 2000 4:46 AM To: ips@ece.cmu.edu Subject: RE: Requirements specification Doug, The current architecture is good for the whole spectrum. If you are intent on using it for a disk drive you can do so and fill with 0 the fields you are not interested in. You don't have to implement the functions that are intended for controllers. The controller/drive scaling controversy is certainly outside the scope of iSCSI. Regards, Julo "Douglas Otis" on 04/08/2000 19:20:40 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05379; 9 Aug 2000 1:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16442 for ips-outgoing; Tue, 8 Aug 2000 23:51:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA20512 for ; Sat, 5 Aug 2000 07:48:02 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA73000 for ; Sat, 5 Aug 2000 13:47:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id NAA139574 for ; Sat, 5 Aug 2000 13:47:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.0040C444 ; Sat, 5 Aug 2000 13:47:25 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 14:45:44 +0300 Subject: RE: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, The current architecture is good for the whole spectrum. If you are intent on using it for a disk drive you can do so and fill with 0 the fields you are not interested in. You don't have to implement the functions that are intended for controllers. The controller/drive scaling controversy is certainly outside the scope of iSCSI. Regards, Julo "Douglas Otis" on 04/08/2000 19:20:40 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Wed Aug 09 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05372; 9 Aug 2000 1:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA16977 for ips-outgoing; Wed, 9 Aug 2000 00:05:34 -0400 (EDT) Date: Wed, 9 Aug 2000 00:05:34 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090405.AAA16977@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- ------_=_NextPart_000_01C001B6.C7EE9030-- ------_=_NextPart_000_01C001B6.EC05487E-- From owner-ips@ECE.cmu.edu Wed Aug 09 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05370; 9 Aug 2000 1:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16427 for ips-outgoing; Tue, 8 Aug 2000 23:51:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA04709 for ; Fri, 4 Aug 2000 22:17:45 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 61DC91206; Fri, 4 Aug 2000 19:17:44 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA07296; Fri, 4 Aug 2000 19:16:53 -0700 (PDT) Message-Id: <4.2.2.20000804185615.00b69100@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 04 Aug 2000 19:02:22 -0700 To: psarkar@almaden.ibm.com, ips@ece.cmu.edu From: Michael Krause Subject: RE: Requirements specification In-Reply-To: <87256931.0062B234.00@d53mta03h.boulder.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 10:58 AM 8/4/00 -0700, psarkar@almaden.ibm.com wrote: >Two points: > >1. The iSCSI presenters seem to imply that having 'n' connections through >the IP fabric will automatically give you 'n' (or close to 'n') times the >bandwidth if you round-robin your packets across the connections. This is >probably true only in the very ideal case and in the real world, the >situation is far less rosy to make such an assertion so confidently. Not true that this has to be the ideal world - there are numerous applications that maintain high-bandwidth through multiple connections on the same path or across several paths. Also, network routers often implement RED / WRED schemes which may only impact a subset of the sessions connections. >2. Fault-tolerance currently is (and should be) layered above the SCSI >transport layer. There are enough solutions from several vendors in the >market which deal with this, >so there is no use in reinventing the wheel. Fault tolerance is a highest level of availability there are other levels that allow one to recover from a variety of failure types without requiring to implement to this level. Most of these can be simply implemented below the iSCSI level. Also, dynamic changes in the network composition can be dealt with without resorting to high-level changes to the applications. The objective of improving the availability is to not be required to use this overly complex solutions but to create simple, low-implementation costs solutions that will benefit the majority of the customers. >Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" >principles,I would appeal to the iSCSI "design team" to forgo multiple >connections per >session and use specialized solutions for remote mirroring and remote tape >backup (the apps that require multiple connections). Multiple connections can be done using a KISS principle and more robust solutions can be built on top of this solution to expand the functional capabilities without requiring the architecture to be radically permuted in the process. Mike From owner-ips@ECE.cmu.edu Wed Aug 09 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05365; 9 Aug 2000 1:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16326 for ips-outgoing; Tue, 8 Aug 2000 23:49:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA18397 for ; Fri, 4 Aug 2000 13:12:22 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA01306; Fri, 4 Aug 2000 10:09:29 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA08718; Fri, 4 Aug 2000 10:06:52 -0700 From: "Douglas Otis" To: "David Robinson" , Subject: RE: Requirements specification Date: Fri, 4 Aug 2000 10:14:21 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: <200008032343.QAA13951@ha10nwk.EBay.Sun.COM> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Although read-channels and data densities improve at a steady pace, as they have for the past quarter century, the mechanics of the drives have not. Today's drives can deliver 320 Mbits/second of data on the outside cylinders. Improvement of the mechanics comes at a high price with respect to power and cost. There are many trade-offs made in this struggle including the physical size of the drive and the number of heads and disks all with substantial impact in a highly competitive market. The cost/volume trend takes us to a single disk which increases access time as read channel data rate increases. Would it be logical to design a system where everything is aimed at taking advantage of the high momentary data rate offered by the read channel, or by offering the same throughput using more drives where each drive's interface bandwidth is restricted with respect to these read channel data rates? The advantage of such an approach is found with respect to smaller random traffic. With more devices, redundancy is easily achieved and parallel access offers a means of performance improvement by spreading activity over more devices. Remember the switch provides bandwidth aggregation and is not found in the individual device. Each device would only see their traffic. The client could see the traffic of hundreds of these devices. Regardless of the nature of the traffic, performance would be more uniform. An 8ms access + latency figure in the high cost drives restricts the number of 'independent' operations that average 64k byte to 100 per second or 52 Mbit per second. Such an architecture of 'restricted' drives would scale whereas the controller based approach does not and is vulnerable. An independent nexus at the LUN is the only design that offers the required scaling and configuration flexibility. Other than your comment about keeping up with the read channel, I would tend to agree. Several Fast Ethernet disks combined at a 1 Gbit Ethernet client makes sense in cost, performance, capacity, reliability, and scalability. The protocol overhead should be addressed in the protocol itself. There are substantial improvements to be made in the protocol area. If the desire is to maintain the existing architecture, then Fibre-Channel encapsulation provides that function as well and solves many of the protocol issues plaguing iSCSI or SEP attempts. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of David Robinson Sent: Thursday, August 03, 2000 4:43 PM To: ips@ece.cmu.edu Subject: Requirements specification The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. -David From owner-ips@ECE.cmu.edu Wed Aug 09 06:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05565; 9 Aug 2000 2:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17147 for ips-outgoing; Wed, 9 Aug 2000 00:07:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail03-oak.pilot.net (mail-oak-3.pilot.net [198.232.147.18]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA27559 for ; Tue, 8 Aug 2000 15:34:27 -0400 (EDT) Received: from msd-gw.hitachi.com (root@msd.hitachi.com [206.189.5.161]) by mail03-oak.pilot.net with ESMTP id LAA22064 for ; Tue, 8 Aug 2000 11:56:42 -0700 (PDT) Received: from hicam-msd.hitachi.com (thunder [137.168.16.3]) by msd-gw.hitachi.com with ESMTP id LAA03731 for ; Tue, 8 Aug 2000 11:51:55 -0700 (PDT) Received: from SANJAYS-DPC.hicam-msd.hitachi.com (sanjays-dpc [192.48.127.192]) by hicam-msd.hitachi.com with ESMTP id LAA15791 for ; Tue, 8 Aug 2000 11:51:55 -0700 (PDT) Message-Id: <4.3.0.20000808113651.00b786f0@thunder> X-Sender: sanjays@thunder X-Mailer: QUALCOMM Windows Eudora Version 4.3 Date: Tue, 08 Aug 2000 12:02:34 -0700 To: ips@ece.cmu.edu From: Sanjay Sreenath Subject: archive for this mailing list? Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello, I was trying to reach the archive for this mailing list. The one @ http://ips.pdl.cs.cmu.edu/mail/maillist.html sees to be down. Could some one please point me to the correct URL if it has moved. From owner-ips@ECE.cmu.edu Wed Aug 09 06:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05572; 9 Aug 2000 2:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA18217 for ips-outgoing; Wed, 9 Aug 2000 00:43:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA18200 for ; Wed, 9 Aug 2000 00:42:50 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id AAA00378; Wed, 9 Aug 2000 00:42:29 -0400 (EDT) Date: Wed, 9 Aug 2000 00:42:29 -0400 (EDT) From: Scott Bradner Message-Id: <200008090442.AAA00378@newdev.harvard.edu> To: ips@ece.cmu.edu, randy_haagens@hp.com Subject: Re: Multiple TCP connections Sender: owner-ips@ece.cmu.edu Precedence: bulk > This doesn't solve > > the problem, however, because it would result in all the frames of a session > taking the same link in a trunk. That's not what we want. out of orderpackets in TCP can trigger fast retransmit which, in spite of its name, slows down TCP sessions Scott From owner-ips@ECE.cmu.edu Wed Aug 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05672; 9 Aug 2000 2:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA19391 for ips-outgoing; Wed, 9 Aug 2000 01:31:24 -0400 (EDT) Date: Wed, 9 Aug 2000 01:31:24 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090531.BAA19391@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28299 for ; Sat, 5 Aug 2000 15:31:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA39702 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id VAA14510 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.006B2EFB ; Sat, 5 Aug 2000 21:30:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 22:29:04 +0300 Subject: food for thought Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Multiple TCP connections per session is one of those things that is very easy to remove but hard to add as an afterthought. Julo ------_=_NextPart_000_01C001B6.CDDD4A22-- From owner-ips@ECE.cmu.edu Wed Aug 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05686; 9 Aug 2000 2:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA19356 for ips-outgoing; Wed, 9 Aug 2000 01:30:41 -0400 (EDT) Date: Wed, 9 Aug 2000 01:30:41 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090530.BAA19356@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- ------_=_NextPart_000_01C001B6.C7EE9030-- From owner-ips@ECE.cmu.edu Wed Aug 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05679; 9 Aug 2000 2:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA19336 for ips-outgoing; Wed, 9 Aug 2000 01:30:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- From owner-ips@ECE.cmu.edu Wed Aug 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05663; 9 Aug 2000 2:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17065 for ips-outgoing; Wed, 9 Aug 2000 00:06:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (truffle.Stanford.EDU [171.64.73.147]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA29069 for ; Mon, 7 Aug 2000 17:04:47 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id GAA03985; Tue, 8 Aug 2000 06:05:19 -0700 To: Subject: Multiple connections & design complexity References: <87256931.0062B234.00@d53mta03h.boulder.ibm.com> <008f01bffe6c$98fffea0$35bf003e@e4u5e0> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 08 Aug 2000 06:05:19 -0700 In-Reply-To: "Nelson Nahum"'s message of "Sat, 5 Aug 2000 01:34:07 +0200" Message-ID: Lines: 34 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk There has been significant comment that using multiple connections per session adds significantly to complexity of the design. This message is an attempt to describe the complexity exactly. I hope that you'll agree with me that the complexity is not onerous. There are two ways in which multiple connections are used in iSCSI: 1) Multiple simultaneous TCP connections for bandwidth 2) Multiple TCP connections for fault tolerance/recovery (i.e. when one TCP connection in a session dies, another one starts up) The features in iSCSI that come out of #1: Command Reference Number Expected Command Reference Number (not strictly necessary if the window never shrinks) Max Command Reference Number Session ID The features in iSCSI that come out of #2: Status Reference Number Expected Status Reference Number Session ID Recovery bit in opcode field Connection IDs for TCP connections And the following iSCSI opcode has been suggested by some for recovery: Terminate TCP connection -Costa From owner-ips@ECE.cmu.edu Wed Aug 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05677; 9 Aug 2000 2:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA19374 for ips-outgoing; Wed, 9 Aug 2000 01:30:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28299 for ; Sat, 5 Aug 2000 15:31:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA39702 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id VAA14510 for ; Sat, 5 Aug 2000 21:30:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256932.006B2EFB ; Sat, 5 Aug 2000 21:30:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 5 Aug 2000 22:29:04 +0300 Subject: food for thought Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Multiple TCP connections per session is one of those things that is very easy to remove but hard to add as an afterthought. Julo From owner-ips@ECE.cmu.edu Wed Aug 09 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05939; 9 Aug 2000 3:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17138 for ips-outgoing; Wed, 9 Aug 2000 00:07:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA10570 for ; Tue, 8 Aug 2000 12:17:28 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 9366EDCC; Tue, 8 Aug 2000 09:14:50 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id JAA25818; Tue, 8 Aug 2000 09:14:03 -0700 (PDT) Message-Id: <4.2.2.20000808090343.00af9cf0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 08 Aug 2000 09:09:52 -0700 To: psarkar@almaden.ibm.com From: Michael Krause Subject: RE: Requirements specification Cc: ips@ece.cmu.edu In-Reply-To: <87256934.0079F9A3.00@d53mta03h.boulder.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 03:09 PM 8/7/00 -0700, psarkar@almaden.ibm.com wrote: >Mike, > >I think we need some convergence here. > >There is no denial that there are applications that can take >advantage of multiple connections. However, none of them >are particularly simple (they go way beyond round-robin and >a lot require router assistance as you say) and the point I do not believe they are complex nor that there is a lot of router assistance, if any, required - they do require the ability to take input from other policy making entities who have this knowledge and then execute a policy algorithm. This policy may be opaquely communicated via a request to establish a new connection on a given path or to issue operations using a particular weight round-robin algorithm. All of this is outside of the iSCSI specification and all that is needed within iSCSI is the mechanism to communicate the results of these external policy decisions. I believe there is a functional layering (really lateral than vertical) but that this does not require multiple layers within iSCSI itself - the policy management, etc. can be implemented outside of iSCSI (a peer functional component) that is completely optional for an implementation (it would use a default of 1 connection per session). Those that want to take advantage of these capabilities would be able to do so without necessarily having to rework their implementation other than to take action input through this defined interface. Mike From owner-ips@ECE.cmu.edu Wed Aug 09 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05932; 9 Aug 2000 3:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17096 for ips-outgoing; Wed, 9 Aug 2000 00:06:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA03014 for ; Mon, 7 Aug 2000 18:58:59 -0400 (EDT) From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id RAA41608; Mon, 7 Aug 2000 17:52:28 -0500 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id QAA124364; Mon, 7 Aug 2000 16:58:58 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256934.007E3E7F ; Mon, 7 Aug 2000 16:58:55 -0600 X-Lotus-FromDomain: IBMUS To: Michael Krause cc: ips@ece.cmu.edu Message-ID: <87256934.0079F9A3.00@d53mta03h.boulder.ibm.com> Date: Mon, 7 Aug 2000 15:09:13 -0700 Subject: RE: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, I think we need some convergence here. There is no denial that there are applications that can take advantage of multiple connections. However, none of them are particularly simple (they go way beyond round-robin and a lot require router assistance as you say) and the point being made here is that a reasonable place for managing the complexity of multiple connections is above the transport layer. This also does not mean that you need to change application software - inserting SCSI filter drivers to handle multiple connections can make this transparent to applications. In summary, this is a layering issue - you can have one iSCSI layer handling multiple connections, availability, etc. or you could put the multiple connections layer on top of a simpler iSCSI layer. Either way, the application is unaware of what is going on. Furthermore, if you did not want multiple connections, you could just remove the multiple connections layer. I personally prefer the simpler iSCSI layer approach as it would cater to the common case. Regards, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose Michael Krause @ece.cmu.edu on 08/04/2000 07:02:22 PM Sent by: owner-ips@ece.cmu.edu To: Prasenjit Sarkar/Almaden/IBM@IBMUS, ips@ece.cmu.edu cc: Subject: RE: Requirements specification At 10:58 AM 8/4/00 -0700, psarkar@almaden.ibm.com wrote: >Two points: > >1. The iSCSI presenters seem to imply that having 'n' connections through >the IP fabric will automatically give you 'n' (or close to 'n') times the >bandwidth if you round-robin your packets across the connections. This is >probably true only in the very ideal case and in the real world, the >situation is far less rosy to make such an assertion so confidently. Not true that this has to be the ideal world - there are numerous applications that maintain high-bandwidth through multiple connections on the same path or across several paths. Also, network routers often implement RED / WRED schemes which may only impact a subset of the sessions connections. >2. Fault-tolerance currently is (and should be) layered above the SCSI >transport layer. There are enough solutions from several vendors in the >market which deal with this, >so there is no use in reinventing the wheel. Fault tolerance is a highest level of availability there are other levels that allow one to recover from a variety of failure types without requiring to implement to this level. Most of these can be simply implemented below the iSCSI level. Also, dynamic changes in the network composition can be dealt with without resorting to high-level changes to the applications. The objective of improving the availability is to not be required to use this overly complex solutions but to create simple, low-implementation costs solutions that will benefit the majority of the customers. >Based on the "Keep It Simple and Stupid" and "Optimize for the Common Case" >principles,I would appeal to the iSCSI "design team" to forgo multiple >connections per >session and use specialized solutions for remote mirroring and remote tape >backup (the apps that require multiple connections). Multiple connections can be done using a KISS principle and more robust solutions can be built on top of this solution to expand the functional capabilities without requiring the architecture to be radically permuted in the process. Mike From owner-ips@ECE.cmu.edu Wed Aug 09 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05930; 9 Aug 2000 3:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA20958 for ips-outgoing; Wed, 9 Aug 2000 02:36:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA20943 for ; Wed, 9 Aug 2000 02:36:12 -0400 (EDT) Received: from beppo.feral.com (beppo [192.67.166.79]) by feral.com (8.9.3/8.9.3) with ESMTP id XAA23408 for ; Tue, 8 Aug 2000 23:36:11 -0700 Date: Tue, 8 Aug 2000 23:36:15 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: ips@ece.cmu.edu Subject: mail spammage... In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk I'm getting all sorts of duplicated mail and grot from owner-ips. It doesn't bode well for the acceptance of the efforts of this group to suffer such a problem.. :-)... From owner-ips@ECE.cmu.edu Wed Aug 09 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06403; 9 Aug 2000 4:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17123 for ips-outgoing; Wed, 9 Aug 2000 00:07:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA13611 for ; Tue, 8 Aug 2000 02:10:49 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA305838 for ; Tue, 8 Aug 2000 08:10:18 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id IAA47650 for ; Tue, 8 Aug 2000 08:10:14 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256935.0021E19C ; Tue, 8 Aug 2000 08:10:04 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 8 Aug 2000 09:08:23 +0300 Subject: RE: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, Again the current protocol allows you to do what you want - e.g., build a "virtual target"/LU if this is the way you think things will scale. The paradox of decreased performance per drive due to the increase in recording density is not lost to the storage industry and the major techniques through which it attempts to mitigate it are caching and striping. The numbers you quote are pure drive numbers. For the drive-to-controller cache you might use a "lightweight iSCSI" (software only) or some other mechanism. >From controller to host - once you use one of the boosting techniques (caching, stripping) you will need fast channels. The protocol looks is very simple (multiplexing LU is just another field). You can use it also with a initiator-LU scheme but if we settle for this design we can't use it in larger controllers. Regards, Julo "Douglas Otis" on 07/08/2000 19:14:30 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Requirements specification Julo, As your architecture is based on using a controller to aggregate data rather than a switch, you are making choices detrimental to an architecture that brings the interface closer to the device. This is reflected in choices for configuration, authentication and protocol. Your architecture is very close to a Fibre-Channel gateway. Alter solicitation within Fibre-Channel, and there is not a significant difference (Spoofing solicitations within the gateway would be one means). As an example, you require data successfully delivered be retained for possible later solicitation with a controller based error recovery. Two means of delivering data and error recovery is just one example of added complexity due to an inability to scale data handling. Although read-channels and data densities improve at a steady pace, as they have for the past quarter century, mechanics of the drives have not. Today's drives can deliver 320 Mbits/second of data on the outside cylinders. The physical size of the drive in conjunction with the number of heads and disks all have substantial impact in a competitive market with respect to power and cost. The cost/volume trend takes us to a single larger disk which paradoxically increases access time as read channel data rates increase. You optimize to take advantage of the burst performance of the read channel with added complexities attempting to time or stage such transfers through your architectural restrictions where the device becomes part of this fabric. Is it logical to design a system where everything is aimed at taking advantage of the high momentary data rate offered by the read channel, or by offering the same throughput using more devices where each interface bandwidth is 'restricted' with respect to these read channel data rates? The advantage of such an approach is found with respect to smaller random traffic. With more devices, redundancy is easily achieved and parallel access offers a means of performance improvement by spreading activity over more devices. In this case, the switch provides bandwidth aggregation and each device would only see their traffic, but the client could see the traffic of hundreds of these devices. Regardless of the nature of the traffic, performance would be more uniform and control could be left at the client. An 8ms access + latency figure in the high cost drives restricts the number of 'independent' operations that average 64k byte to 100 per second or 52 Mbit per second. Forgoing the peak data rate, such an architecture of 'restricted' drives would scale whereas the controller based approach does not and is vulnerable. An independent nexus at the LUN is the only design that offers the required scaling and configuration flexibility. Switch and client aggregation makes sense in cost, performance, capacity, reliability, and scalability. Protocol overhead should be addressed in the protocol itself and not by means of controller aggregation. There are substantial improvements to be made in the protocol area without the use of intervening controllers. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Saturday, August 05, 2000 4:46 AM To: ips@ece.cmu.edu Subject: RE: Requirements specification Doug, The current architecture is good for the whole spectrum. If you are intent on using it for a disk drive you can do so and fill with 0 the fields you are not interested in. You don't have to implement the functions that are intended for controllers. The controller/drive scaling controversy is certainly outside the scope of iSCSI. Regards, Julo "Douglas Otis" on 04/08/2000 19:20:40 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Requirements specification Julo, You comments are based on several assumptions reflecting your present architecture. Your implementation is done at the controller rather than a device. You also assume authentication is done at the controller. Each LUN could belong to a different authority and be an independent (virtual) device managed through LDAP. If you bring the interface to the device, you can obtain the required scaling that is otherwise difficult at the controller as with your architecture. By combining everything into a single connection, you do not improve reliability, scalability, availability or fault tolerance. Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Thursday, August 03, 2000 7:37 PM To: ips@ece.cmu.edu Subject: Re: Requirements specification David, The one additional requirement is availability/fault-tolerance. Your arguments about performance are valid. However I doubt that there will be enough incentives - beyond price - to develop things for high end controllers and servers. Enabling multiple connections brings those applications the performance required without any serious implications to the rest of the "family" (as I outlined in Pittsburgh controllers and servers that don't need multiple connections/session don't have to implement them). Storage traffic requirements will always exceed those of many other applications. As for the "one-connection-per-LU" we covered this solution in long discussions and even several full fledged implementation - as it is compelingly simple. However the resource consumption is unjustifiably high and the security problems are even worse (the LUs "viewed" by an initiator depend on who he says he is) than in the current draft. Regards, Julo David Robinson on 04/08/2000 02:43:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Requirements specification To further elaborate on my comments in Pittsburgh on multiple connections per link and connections per LUN vs per target. The current requirements specify that the protocol must support multiple connections per session. So far the only justification for this that I have clearly heard is performance, current and future systems will demand bandwidth that will require aggregation. Is there any other reason for multiple connections? My challenge to this requirement is that it is fundementally a link and transport layer issue that is being exposed to the session layer due to a perception that current link/transport implementations are not adequate to meet perceived demand. The key question here is if this is a "physics" issue that can't be solved with better implementations or just bad implementations? I am leaning towards the latter. I expect that if this protocol is a success, a number of highly tuned adapters using tricks such as hardware assist will be developed. Those doing the development will have direct control over the quality of the implementation. Furthermore, the performance critical environments are likely to be local in nature so preassure to create necessary switches and routers will also exist. The advantages of limiting a single connection per session should be a simplification in the connection management and error handling. From the earliest drafts we have already seen restrictions of individual command/data/status sequences to a single connection to better handle ordering issues. I forsee further restrictions possibly being required to cover handling of lost connections when sequences are received out of across multiple connections. Similarily Steve's comments on security management of multiple connections is of concern. The second area that I brought up was the requirement of one session per initiator target pair instead of one per LUN (i.e. SEP). I am willing to accept the design constraint that a single target must address 10,000 LUNs which can be done with a connection per LUN. However, statements of scaling much higher into the areas where 64K port limitations appear I think is not reasonable. Given the bandwidth available on today's and near future drives that will easily exceed 100MBps I can't imagine designing and deploying storage systems with over 10,000 LUNs but only one network adapter. Even with 10+ Gbps networks this will be a horrible throughput bottleneck that will get worse as storage adapters appear to be gaining bandwidth faster than networks. Therefore requiring greater than 10,000 doesn't seem necessary. >>From the performance perspective, a connection per LUN also makes sense. SCSI command flows are already being constrained to a single connection in the current proposal for ordering reasons, so the number of concurrent outstanding requests per LUN is a manageable number. The concurrency desired by multiple connections per session in the existing draft will naturally occur with a connection per LUN. As each TCP connection is a unique flow existing link layer hardware that tries to preserve ordering based on a "flow" (likely IP/port pairs) will give the desired performance properties. Both my objections and the requirements for multiple connections I question above become moot. >>From a connection management, command ordering, and error recover perspective things should also get simplier. Ordering is obviously maintained and the sender can now recover from connection errors based on a smaller context and possibly use TCP layer information to determine what responses were received (ACK windows?). To summarize I would like to see the requirements changed to reflect a maximum of 64K LUNs per IP node, require only one transport layer connection per session, and define a session to be an initiator/LUN pair. -David From owner-ips@ECE.cmu.edu Wed Aug 09 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06398; 9 Aug 2000 4:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17087 for ips-outgoing; Wed, 9 Aug 2000 00:06:42 -0400 (EDT) Date: Wed, 9 Aug 2000 00:06:42 -0400 (EDT) From: owner-ips@ece.cmu.edu Message-Id: <200008090406.AAA17087@ece.cmu.edu> X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09800 for ; Sun, 6 Aug 2000 01:12:36 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5636BD41 for ; Sat, 5 Aug 2000 22:12:31 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id XAA25025 for ; Sat, 5 Aug 2000 23:12:30 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Sat, 05 Aug 2000 23:12:29 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 23:12:29 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E9728@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: Re: Multiple TCP connections Date: Sat, 5 Aug 2000 23:12:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01BFFF64.E9DA3EF0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: text/plain; charset="iso-8859-1" This memo recaps some of the reasons for the iSCSI design committee's chosing multiple TCP connections and the session concept. Also discussed is the question of whether TCP connections should be related directly to LUNs. We chose to support multple TCP connections in order to benefit from concurrency in the fabric (primarily) and also in end node implementations (hardware and software). This is related to the stated requirement for bandwidth aggregation. The notion is that no matter how fast an individual link (100 Mbps, 1Gbps or 10 Gbps), it will always be desirable to build end nodes and fabrics that can use multiple links, in parallel, for aggregated bandwidth. The existence of the 802.3ad link aggregation standard is evidence that the Ethernet community values bandwidth aggregation. Unfortunately, 802.3ad-compliant networks will achieve parallel flows on link trunks only for traffic from different "conversations" (see Pat Thaler's memo dated 8/03). Our understanding is that for today, at least, all level-2 and -3 switches will forward the frames from a single TCP connection over the same link of a multilink trunk. This is because the hash key used to assign a frame to a trunk is based on a combination of the MAC and IP source and destination addresses, plus the TCP source and destination port numbers. (The more sophisicated the switch, the more of these values may be used in the hash key.) For a single TCP connection, all of these values remain identical for all of the frames in that connection. Hence, all of the frames of that connection will take the same route through the L3 or L2 switched Ethernet fabric. Pat alludes to the possibility of discriminating at the session layer, where session-layer connection IDs would in fact be different. This doesn't solve the problem, however, because it would result in all the frames of a session taking the same link in a trunk. That's not what we want. Our understanding is that to leverage existing infrastructure, and achieve parallel flows through the Ethernet fabric, we must use different TCP connections (therefore different port number), at the very least. This practice will allow L4 switches to assign different TCP conversations to different 802.3ad links. While we're at it, it's helpful also to use different IP and MAC addresses, so that L3 and L2 switches also will do the right thing. For a moment, assume that the IP/Ethernet fabric were able to support multi-link concurrency for a single TCP stream. Then, the question of in-order arrival occurs. Unquestionably, in-order arrival would be preferable, as it would ease the TCP segment re-assembly process. Arguably, however, out-of-order arrival could be reasonably handled by a TCP hardware engine, provided that the time skew of the arrivals was tightly controlled. (This limits the amount of memory required for the reassembly buffer.) On the other hand, early hardware implementations of TCP will likely assume in-order arrival for the fast-path implementation, and escalate to firmware for handling out-of-order segment arrival, which should normally happen only in the error case (dropped segment or IP route change). Allowing the routine arrival of segments out of order is probably not a wise choice. Alternatively, it's conceivable that switches could be designed that would distribute TCP frames across multiple links, while maintaining the order of their reassembly at the receiving switch ports. Note that the end nodes, with their multiple links, would also have to participate in such a new aggregation protocol. This new class of switches, if they were to emerge, would make it feasible to consider limiting iSCSI sessions to a single TCP/IP connection, at least for local area Ethernet fabrics. Similar developments would be required in wide-area switching. Even assuming these developments, one possible problem would remain: the TCP engine at the two ends of the link would have to handle the aggregated traffic of several links. Aggregating TCP connections into a session allows us to deploy multiple TCP engines, typically one per IP address, and requires only that the TCP engine implementation scales with the link speed. The next question is, given multiple TCP connections per end node, how many should we support? The iSCSI design committee concluded that the right answer was "several". Consider the case of a multiport storage controller (or host computer). To use each of the ports, we certainly need one TCP connection per host per port at a minimum. If 100 host computers, each with 16 connections to the Ethernet fabric, share a storage array that also has 16 connections to the Ethernet fabric, then the storage array needs to support 1600 connections, which is reasonable. If the hosts actually use one connection group (aka "session") for writes, and a second one for reads, in order to allow reads that are unordered with respect to those writes, then 3200 connections are needed. Still reasonable for a large storage array. Some have suggested a single connection per LU. This might be reasonable for a disk that contains only a single LU. But a storage controller contains today 1024 LUs, and in the future, perhaps 10,000 LUs. Sometimes an LU will be shared between multiple hosts, meaning that the number of connections per LU will be greater than one. Assume that 128 hosts are arranged in 16 clusters of 8 hosts, running a distributed file or database system between them. Then each LU will have to support 8 host connections. Assume further that a second connection per host is needed for asynchronous reads. 16 connections per LU, or 160,000 connections in total. If each connection state record is 64B (a totally wild guess), this amounts to 10 MB of memory needed for state records. As a point of comparison, first-generation TCP hardware acclerators are planned with support for approximately 1000 connections. If this weren't bad enough, it turns ot that one (or two, in the case of asynch reads) connection per LU isn't enough to meet performance requirements. While the large number of TCP connections required for the many LUs certainly will deliver enough aggregate throughput for unrelated traffic, only one (or two) connections are available for a single LU. Bear in mind that for storage controllers, writes to an LU really are writes to storage controller memroy, and not to disk. (A background process destages data to disk, typically at a much lower rate that data is delivered to cache, due the benefits that the cache provides, which are too nuanced to go into here.) Today's storage controllers can absorb write bursts at typically 1 GB (that's Gigabyte) per second, which would require the aggregation of 8 1 Gbps Ethernet links. By the time 10 GbE emerges, storage controller bandwidth will have scaled up to the 10 GBps range. Conclusion: one (or two) TCP connections per LU is both too many (resulting in too much memory devoted to state records) and too few (insufficient bandwidth for high-speed IO to controller cache). Decoupling the number of TCP connections from the number of LUs is the necessary result. If you still don't buy this argument, consider the evolution to object-based storage, where SCSI LUs are replaced by objects. Objects may be used for the same purposes that LUs are today (to contain a file system, for example); or they may be used to contain file subtrees, individual files, or even file extents. They will be much more numerous than LUs. iSCSI allows the host to bind n TCP connections together into an iSCSI session, which provides the SCSI "transport" function. The connections of this session typically will use n different Ethernet interfaces and their respective TCP engines. The session is connected to an abstract iSCSI "target", which is a collection of SCSI LUNs named by a url. Within the session, thousands of IOs may be outstanding at a given time, involving perhaps 1600 or so LUs (128 hosts are organized into 16 clusters; the 10,000 LUs are divided among the 16 clusters of hosts.) Because the iSCSI session is a SCSI transport, we've chosen to support ordered command delivery within the iSCSI session. SCSI requires this functionality of any transport, so that the SCSI attributes "ordered" and "simple" will have some meaning. This mechanism dates to the SCSI bus (which is a link), which always delivers commands in order. Under the assumption of in-order command delivery, the SCSI device server can meaningfully use the task attributes to control the order of task execution. (Actually, the SCSI SAM-2 equivocates on whether ordered command delivery is a requirement; this is probably a compromise to permit support FCP-1, which doesn't-support ordered command delivery, to be a legal SCSI transport. Notably, FCP-2 has adopted a command numbering scheme similar to our own, for in-order command delivery.) Command ordering is accomplished by numbering the commands. Command numbering has two additional benefits: (1) We can apply flow control to command delivery, in order to prevent the hosts from overruning the storage array; (2) We can know, through a cumulative acknowledgement mechanism, that a command has been received at the storage controller. A similar mechanism is used for reponse message delivery, so that the target can know that its response (status) message was received at the initiator, and that command retry will not be subsequently attempted by the host. This permits the target to discard its command replay buffer. Sequencing of commands was chosen by the design committee after lengthy consideration of an alternative: numbering every iSCSI session-layer PDU. The latter approach actually would have made recovery after TCP connection failure a lot easier, at least conceptually, since it would be handled at the iSCSI PDU (message) level, and not at the higher SCSI task (command) level. But there was a problem in the implementation: the central iSCSI session layer would need to be involved in numbering every iSCSI PDU sent by any of the iSCSI/TCP engines. This would require an undersirable amount of communication between these engines. The method we've chosen requires only that commands be numbered as they leave the SCSI layer, and similarly, that response window variables be updated only when response messages are returned to the SCSI layer. This assures that iSCSI code in the host will run only when SCSI code runs, during startIO and completion processing. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com <> ------_=_NextPart_000_01BFFF64.E9DA3EF0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01BFFF64.E9DA3EF0-- ------_=_NextPart_000_01C001B6.C7EE9030-- From owner-ips@ECE.cmu.edu Wed Aug 09 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06340; 9 Aug 2000 4:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA17066 for ips-outgoing; Wed, 9 Aug 2000 00:06:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA29217 for ; Mon, 7 Aug 2000 17:08:33 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e77L8Xw09777 for ; Mon, 7 Aug 2000 16:08:33 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA20253; Mon, 7 Aug 00 16:06:25 -0500 Message-Id: <10008072106.AA20253@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Requirements specification In-Reply-To: Message from David Robinson of "Thu, 03 Aug 2000 16:43:11 PDT." <200008032343.QAA13951@ha10nwk.EBay.Sun.COM> Date: Mon, 07 Aug 2000 16:07:38 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > The current requirements specify that the protocol must support > multiple connections per session. So far the only justification > for this that I have clearly heard is performance, current and future > systems will demand bandwidth that will require aggregation. Is there > any other reason for multiple connections? ST attempted to provide support for ``striping'' at a protocol level which is somewhat comparable to performance scaling application of the multiple connections per session proposal for iSCSI. Basically, it was a failure. It handled one very specific class of application (not very well). As others here have mentioned, the error handling was horrendous. I'd say we spent WAY too long trying to iron the bugs out, and it STILL had square wheels AND unaddressed reachable states. When we did SST, we specifically prohibited use of ST striping both because it wouldn't work well, and because the SCSI stacks of high end platforms already support LUN-level striping which was perfectly adequate. The high end applications that needed more than a single channel's worth of data, were always signed up for additional system integration work to make a less general technology do the trick. In order to guarantee the target performance levels, they had to perform a complete system analysis and engineering effort anyway. Everybody else seemed more interested in the best performance which was available `inexpensively'. That corresponds to using a single (general purpose) channel, and scaling with channel technology. We concluded that as long as channel speeds were increasing briskly it didn't make sense to spend any effort on a low level striping scheme. If network technology generations start coming years apart, it might make sense again. A similar argument applies to availability (multipathing). High end SCSI stacks do it, and the relatively smaller set of applications that need the feature will be engineered/integrated end to end for the requirement. The integrators needn't sling code, there are off-the-shelf components to do it, but they DO have to make sure that the technology is applied properly. They can't just expect it to work under any circumstances. > The second area that I brought up was the requirement of one session > per initiator target pair instead of one per LUN (i.e. SEP). I am > willing to accept the design constraint that a single target must > address 10,000 LUNs which can be done with a connection per > LUN. However, statements of scaling much higher into the areas where > 64K port limitations appear I think is not reasonable. The ST/SST experience is on the opposite side of this argument. The main issue is that when you move from a bounded storage configuration to a storage on a generalized network, you have to assume a leap in the scale of connectivity. That's one of primary advantages using a network. If you do not use all reasonable techniques at your disposal to reduce the number of connections, you will run out of connection resources. On the initiator end, iSCSI is unlikely to be the only substantive consumer of connection resources on a channel. It's bad platform citizenship for a protocol to assume that it can consume the majority of any shared resource. On the target end, storage targets tend to be a cost sensitive business, and more connections is more state, so more cost. If the standard specifies one LUN per connection, the clear expectation is that (multi LUN) targets must support many (initiators * LUNs) connections. The other crunch point for connections in ST was that the hardware acceleration included connection specific behavior. There is some increment of hardware complexity (not necessarily design complexity) per supported connection. If iSCSI hardware acceleration has this same characteristic, conserving connections will probably be even more important. Steph From owner-ips@ECE.cmu.edu Wed Aug 09 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08133; 9 Aug 2000 11:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA20091 for ips-outgoing; Wed, 9 Aug 2000 10:05:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA20057 for ; Wed, 9 Aug 2000 10:04:48 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA27898 for ; Wed, 9 Aug 2000 16:04:13 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id QAA20302 for ; Wed, 9 Aug 2000 16:04:13 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256936.004D493A ; Wed, 9 Aug 2000 16:04:10 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 9 Aug 2000 09:28:49 +0300 Subject: Re: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, It would be nice if you would share with us: what are the "wonderful rerouting properties" you are referring to ? what is the link technology to do failover ? I am sure only about myself but I guess that many of us are ignorant on the above. As for your argument about where the weak points are - debatable as it may be it seems only to show that multiple connections are desirable. You seem to associate a HA or port with an initiator while the mapping can be more complex. As for availability - several connections and a failover mechanism will make a solution more reliable. The question is who is doing the failover. Your first argument seems to imply that IP will make it through some magic that I don't know about. Your last argument says that we should leave it to SCSI (every adapter is a different SCSI initiator) and we know that means percolate to the application layer - as SCSI has traditionally no recovery mechanism. I have to admit that I am utterly confused by your line of thought. Regards, Julo David Robinson on 04/08/2000 21:37:51 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Requirements specification > The one additional requirement is availability/fault-tolerance. I agree that we need to establish the availability requirements of IPS, but I see them as orthogonal to the connection discussion. IP already has the wonderful rerouting properties to be fault resilient as well as link level technology to do failover. From an IPS perspective the host adapters and the storage adapters are the weak links. This to do a reasonable HA solution you will need multiple adapters on both ends which will imply initiator/target pairs. So I don't see how having multiple TCP connections per session will increase availability. Existing HA solutions that layer on top of FC or parallel SCSI could just as easily layer on top of IPS. > Your arguments about performance are valid. However I doubt that there will > be enough incentives - beyond price - to develop things for high end > controllers and > servers. > > Enabling multiple connections brings those applications the performance > required > without any serious implications to the rest of the "family" (as I outlined > in Pittsburgh > controllers and servers that don't need multiple connections/session don't > have to implement them). I am not sure that I understand your point here. As has been mentioned by others, one performance concern is how to maximize utilization of the available link bandwidth with a higher level protocol, fundementally you can never get 100%. Part of the concern is that most existing TCP implementations are not scaling up as fast as the link layer protocols. To handle this situation people consider some form of aggregation of TCP links to relieve the pressure on the implementation side. The question becomes what is the appropriate way to aggregate. If we run multiple LUNs over the same session then we will push the TCP implementation to its limits. However, if we run a session per LUN, the bandwidth pressures will be dramatically less on TCP, we will more easily utilize existing and future link layer bandwidth, and simplify the multiplexing by leaving it in the TCP layer. > Storage traffic requirements will always exceed those of many other > applications. Storage will always exceed thoses of *some* other applications, it is easy to find numerous applications that exceed storage requirements. I am not clear on the point, but we need to make sure that we clearly define what the storage traffic requirements are. > As for the "one-connection-per-LU" we covered this solution in long > discussions > and even several full fledged implementation - as it is compelingly simple. > However the resource consumption is unjustifiably high and the security > problems are > even worse (the LUs "viewed" by an initiator depend on who he says he is) > than > in the current draft. I don't agree that the security problems are worse. For any session, whether it is a session per target or a session per LUN will need to do full security negotiation. Realitive to the expected lifetime of the sessions the time and complexity will be negligible. If we adequately solve it for one model it should trivially apply to the other. I also don't agree on the resource consumption being too high. In terms of memory resources the necessary data buffering is dependant on the aggregate flow between the target and initiator which will be independent of how many connections were used to get the data to the other side. There will be more connections, but within typical TCP implementations the size and complexity of control blocks is minimal. The ability to switch and manage many connections rapidly is a long solved problem. It would be illumninating to hear if Adaptec has found that their connection per LUN is resource consuming. -David From owner-ips@ECE.cmu.edu Wed Aug 09 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08135; 9 Aug 2000 11:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA20090 for ips-outgoing; Wed, 9 Aug 2000 10:05:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA20059 for ; Wed, 9 Aug 2000 10:04:48 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA08282 for ; Wed, 9 Aug 2000 16:04:12 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id QAA20300 for ; Wed, 9 Aug 2000 16:04:12 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256936.004D497C ; Wed, 9 Aug 2000 16:04:10 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 9 Aug 2000 09:44:33 +0300 Subject: RE: multiple TCP connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, Thanks for your excellent summary. It is hard to believe but there are no good trunking mechanisms in the TCP/IP world and you argument about enabling the use of several 1Gbs connections without waiting for the 10Gbs and enabling any topology that has the required bandwidth end-to-end where very much on our mind when designing the protocol. Julo Stephen Byan on 07/08/2000 21:21:04 Please respond to Stephen Byan To: "'ips@ece.cmu.edu'" cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: multiple TCP connections Thanks to all who answered my "help me figure out what I don't know" questions regarding multiple TCP connections. To summarize my current understanding: To my surprise, neither TCP, IP, nor Ethernet have mechanisms for striping traffic across multiple interfaces when that traffic is part of a single TCP connection. (There may be an exception for ethernet in the letter of 802.3ad, but in terms of practical implementations available in the near future, there's no striping avaiable for a single TCP connection.) However, there are mechanisms for striping traffic from multiple TCP connections. We are given the constraint that we can't change TCP. If we assume a requirement to stripe traffic between a single muli-homed host and a single LUN on a multi-homed device across multiple paths, then we need to turn that traffic into traffic over multiple TCP connections. One way of doing this would be to use one TCP connection per SCSI task. (Think HTTP.) To minimize the costly overhead of this approach, we'd probably have to consider something like Transactional TCP. Moveover, this would not increase the transfer speed for a single command. I don't find this approach very palatable. Another way of distributing the traffic over multiple TCP connections is to create the notion of a session, and bind multiple TCP connections to that session. Architecturally this looks yucky, but I prefer sessions over the alternative of one connection per command. So I agree that if we accept the requirement as stated above, then sessions are the logical answer. Discussion of requirements: I happen to agree with the requirement to stripe the traffic between a single muli-homed host and a single LUN on a multi-homed device across multiple paths. For a small, cheap iSCSI implementations near the device level, I'd like to stick with 1 Gb ethernet even as 10 Gb becomes available, in order to minimize costs. Customers really like multiple paths to their devices, so device-level implementations will require two interfaces for this reason alone. Finally, note that media data rates are always increasing, and we're not that many years away from hitting 1 Gb/s. We can argue whether the storage interconnect really needs to match the media rate, since once the drive starts seeking the average data rate drops by orders of magnitude. Regardless of the technical merits, I think interface speed sells in the storage market, so I think customers will continue to demand interface speeds that match the media rate. Consequently I think that low-end iSCSI devices will have a requirement for interface speeds in excess of 2 Gb/s within the next few years, and would best meet this requirement by striping across two 1 Gb ethernet interfaces in an iSCSI session. Complaints about current definition of an iSCSI session: I have one concern about the current definition of an iSCSI session. I'd prefer a striping mechanism that would support striping of the data transfer for a single command across multiple interfaces. As the spec stands now, all messages for a single command have allegiance to one TCP connection, and therefor the all the data transfer for that command must occur over one TCP connection. Given the current architecture of iSCSI, I see the need for this allegiance, since IMHO hardware DMA acceleration requires access to the state of the iSCSI command. However, if we adopt VI/TCP as the enabler for hardware DMA acceleration, the data transfer state is (almost) self-contained in the VI segment header, and I think it becomes feasible to remove the requirement that the data transfer maintain allegiance to the TCP connection that originated the SCSI command. Then we could stripe the data DMA traffic across multiple interfaces, while still keeping all the control and status traffic on the originating TCP connection. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Wed Aug 09 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08131; 9 Aug 2000 11:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA22587 for ips-outgoing; Wed, 9 Aug 2000 10:35:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA22562 for ; Wed, 9 Aug 2000 10:35:31 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id HAA24370 for ; Wed, 9 Aug 2000 07:35:30 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Wed, 9 Aug 2000 07:35:26 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF02EEA052@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Multiple connections & design complexity Date: Wed, 9 Aug 2000 07:35:25 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa wrote: > There has been significant comment that using multiple connections > per session adds significantly to complexity of the design. > This message is an attempt to describe the complexity exactly. > I hope that you'll agree with me that the complexity is not onerous. I agree the complexity doesn't look too bad. My first impresssion was that the striping functionality was in the wrong architectural layer, and so the session concept in the draft seemed bogus. I'm now convinced that it must be implemented in the application protocol, i.e. iSCSI or perhaps VI/TCP. It doesn't look hard to implement. It even may have some advantages, since the obvious implementation implies round-robin scheduling between initiators, which provides some level of fairness in command-scheduling which might otherwise be lacking. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Wed Aug 09 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08394; 9 Aug 2000 12:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA23670 for ips-outgoing; Wed, 9 Aug 2000 10:55:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA23654 for ; Wed, 9 Aug 2000 10:55:17 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id HAA27358 for ; Wed, 9 Aug 2000 07:55:16 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Wed, 9 Aug 2000 07:55:13 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF02EEA053@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Requirements specification Date: Wed, 9 Aug 2000 07:55:13 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph wrote: > We concluded that as long as channel speeds were increasing briskly it > didn't make sense to spend any effort on a low level striping scheme. > If network technology generations start coming years apart, it might > make sense again. I don't think network channel speeds are increasing briskly. Ethernet was 10 Mb/s in 1980, 100 Mb/s in 1990, and 1 Gb/s in 2000. These are slow infrastructural step functions. Note that continued increase in storage capacity is based partly on a continued increase in media bandwidth. Also part of the modest increase in random access performance over the past decade has been based on a continued increase in media bandwidth (i.e. higher spindle RPM). In the 1990's there was a transition from shared bandwidth to switched bandwidth which helped mitigate the problem of the slow evolution of link speeds. I think storage interconnect bandwidth needs to be able to scale nicely between the step-function increases provided by new physical layers. We've already spent the scaling available by moving from shared bandwidth to switched bandwidth; link aggregation seems like the next necessary step. I argue that for storage it's not enough to aggregate links statistically, based on multiple flows. Instead it is necessary to aggregate links for a single flow. Consequently I now favor the iSCSI session concept. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Wed Aug 09 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08886; 9 Aug 2000 14:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA28186 for ips-outgoing; Wed, 9 Aug 2000 12:29:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imo-r11.mx.aol.com (imo-r11.mx.aol.com [152.163.225.65]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA28062 for ; Wed, 9 Aug 2000 12:27:02 -0400 (EDT) From: AndInc@aol.com Received: from AndInc@aol.com by imo-r11.mx.aol.com (mail_out_v27.12.) id l.63.9880a57 (4262); Wed, 9 Aug 2000 12:26:10 -0400 (EDT) Message-ID: <63.9880a57.26c2e022@aol.com> Date: Wed, 9 Aug 2000 12:26:10 EDT Subject: Re: Requirements specification To: steph@cs.uchicago.edu CC: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: AOL 4.0 for Windows 95 sub 42 Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Steven, On a technical basis, thank you for your summary. I think it's important to recall and contrast theory and practice when examining complex issues. Recalling my own experience with protocol successes and failures, I am reminded of how important products are to protocols. The protocols I've seen succeed are usually modest in scope, and tend to converge around a reasonable theory (the spec) and actual practice (the products). In my humble opinion, any addition of complexity to the iSCSI protocol will affect the practice, and perhaps the ultimate success of the protocol in the market. If the Adaptec proposal has a compelling strength, it that it exists and could be implemented by engineers that are not brain surgeons. Mike A From owner-ips@ECE.cmu.edu Wed Aug 09 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08959; 9 Aug 2000 14:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA00519 for ips-outgoing; Wed, 9 Aug 2000 13:23:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA00267 for ; Wed, 9 Aug 2000 13:17:40 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA02126; Wed, 9 Aug 2000 10:13:54 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA25404; Wed, 9 Aug 2000 10:11:19 -0700 From: "Douglas Otis" To: "David Robinson" , Subject: RE: Requirements specification Date: Wed, 9 Aug 2000 10:19:45 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <200008041837.LAA15308@ha10nwk.EBay.Sun.COM> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Be aware, three days of messages never posted to the reflector and can only be found in the archive that is now once again online. It is difficult to carry on discussions when messages are slow to post or go missing for several days and then only show up in the archive and not on the reflector as intended. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > David Robinson > Sent: Friday, August 04, 2000 11:38 AM > To: ips@ece.cmu.edu > Subject: Re: Requirements specification > > > The one additional requirement is availability/fault-tolerance. > > I agree that we need to establish the availability requirements > of IPS, but I see them as orthogonal to the connection discussion. I think we can reinvent the wheel in this case or understand how the present networking equipment provides solutions. > IP already has the wonderful rerouting properties to be fault > resilient as well as link level technology to do failover. From > an IPS perspective the host adapters and the storage adapters are > the weak links. This to do a reasonable HA solution you will need > multiple adapters on both ends which will imply initiator/target > pairs. This is the reinventing. If your intent is to provide a Fibre-Channel gateway, the only innovation required is to spoof and cache data solicitations. You're done; case closed. If you wish to create a standard that scales and provides the desired HA features by bring the IP interface to the device, then much of your architecture and assumptions must change. It is futile to develop a standard to compete with a Fibre-Channel gateway. There is nothing being offered by the iSCSI approach this is not already possible with a Fibre-Channel gateway. No additional drivers, interfaces, or system modifications are required beyond the Fibre-Channel gateway. iSCSI on the other hand, must start from scratch creating the hardware interface and drivers. > So I don't see how having multiple TCP connections per session > will increase availability. Existing HA solutions that layer on top > of FC or parallel SCSI could just as easily layer on top of IPS. I suspect this is a separate consideration. If you wish to make a Fibre-Channel gateway, then an OC-192 interface at the router will provide 40 Gbit of connectivity which should be plenty for any near term technology. Cisco has announced such a meld and this removes the perceived 1 Gbit Ethernet restriction. See: http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_061300b.htm > > Your arguments about performance are valid. However I doubt > that there will > > be enough incentives - beyond price - to develop things for high end > > controllers and > > servers. > > > > Enabling multiple connections brings those applications the performance > > required > > without any serious implications to the rest of the "family" > (as I outlined > > in Pittsburgh > > controllers and servers that don't need multiple > connections/session don't > > have to implement them). > > I am not sure that I understand your point here. As has been mentioned by > others, one performance concern is how to maximize utilization of the > available link bandwidth with a higher level protocol, fundamentally > you can never get 100%. Part of the concern is that most existing TCP > implementations are not scaling up as fast as the link layer protocols. > To handle this situation people consider some form of aggregation > of TCP links to relieve the pressure on the implementation side. The > question becomes what is the appropriate way to aggregate. If we run > multiple LUNs over the same session then we will push the TCP > implementation > to its limits. However, if we run a session per LUN, the bandwidth > pressures will be dramatically less on TCP, we will more easily > utilize existing and future link layer bandwidth, and simplify the > multiplexing by leaving it in the TCP layer. This also makes the proper assumption of taking the interface to the device. Otherwise we are making a gateway and if that is what you want, then it is being done already so why make competing standards? > > Storage traffic requirements will always exceed those of many other > > applications. > > Storage will always exceed thoses of *some* other applications, it is > easy to find numerous applications that exceed storage requirements. > I am not clear on the point, but we need to make sure that we clearly > define what the storage traffic requirements are. > > > As for the "one-connection-per-LU" we covered this solution in long > > discussions > > and even several full fledged implementation - as it is > compelingly simple. > > However the resource consumption is unjustifiably high and the security > > problems are > > even worse (the LUs "viewed" by an initiator depend on who he > says he is) > > than > > in the current draft. > > I don't agree that the security problems are worse. For any > session, whether > it is a session per target or a session per LUN will need to do full > security negotiation. Realitive to the expected lifetime of the sessions > the time and complexity will be negligible. If we adequately solve it for > one model it should trivially apply to the other. > > I also don't agree on the resource consumption being too high. In terms > of memory resources the necessary data buffering is dependant on the > aggregate flow between the target and initiator which will be > independent of how many connections were used to get the data to > the other side. There will be more connections, but within typical > TCP implementations the size and complexity of control blocks is > minimal. The ability to switch and manage many connections rapidly > is a long solved problem. It would be illumninating to hear if Adaptec > has found that their connection per LUN is resource consuming. > > -David > The only down-side I see with this is that TCP is not intended to be intermittent at high data rates. -Doug From owner-ips@ECE.cmu.edu Wed Aug 09 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09632; 9 Aug 2000 16:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA04345 for ips-outgoing; Wed, 9 Aug 2000 14:53:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA04341 for ; Wed, 9 Aug 2000 14:53:48 -0400 (EDT) From: psarkar@almaden.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA14114; Wed, 9 Aug 2000 14:51:38 -0400 Received: from d53mta03h.boulder.ibm.com (d53mta03h.boulder.ibm.com [9.99.142.3]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id MAA67032; Wed, 9 Aug 2000 12:53:46 -0600 Received: by d53mta03h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256936.0067C886 ; Wed, 9 Aug 2000 12:53:35 -0600 X-Lotus-FromDomain: IBMUS To: csapuntz@cisco.com cc: ips@ece.cmu.edu, csapuntz@cisco.com Message-ID: <87256936.006724AF.00@d53mta03h.boulder.ibm.com> Date: Wed, 9 Aug 2000 11:46:33 -0700 Subject: Re: Multiple connections & design complexity Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The complexity does not arise simply from the additional fields (which I agree are not onerous) but from the way the additional fields will be interpreted (mistakenly or otherwise) and the potential interoperability hassles different interpretations will cause. Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose csapuntz@cisco.com@ece.cmu.edu on 08/08/2000 06:05:19 AM Sent by: owner-ips@ece.cmu.edu To: cc: csapuntz@cisco.com Subject: Multiple connections & design complexity There has been significant comment that using multiple connections per session adds significantly to complexity of the design. This message is an attempt to describe the complexity exactly. I hope that you'll agree with me that the complexity is not onerous. There are two ways in which multiple connections are used in iSCSI: 1) Multiple simultaneous TCP connections for bandwidth 2) Multiple TCP connections for fault tolerance/recovery (i.e. when one TCP connection in a session dies, another one starts up) The features in iSCSI that come out of #1: Command Reference Number Expected Command Reference Number (not strictly necessary if the window never shrinks) Max Command Reference Number Session ID The features in iSCSI that come out of #2: Status Reference Number Expected Status Reference Number Session ID Recovery bit in opcode field Connection IDs for TCP connections And the following iSCSI opcode has been suggested by some for recovery: Terminate TCP connection -Costa From owner-ips@ECE.cmu.edu Wed Aug 09 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09630; 9 Aug 2000 16:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA04810 for ips-outgoing; Wed, 9 Aug 2000 15:04:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA04803 for ; Wed, 9 Aug 2000 15:04:31 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id MAA11110 for ; Wed, 9 Aug 2000 12:04:30 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Wed, 9 Aug 2000 12:04:29 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF02EEA057@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Multiple TCP connections Date: Wed, 9 Aug 2000 12:04:28 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Randy Haagens wrote: > If the hosts actually use > one connection group (aka "session") for writes, and a second > one for reads, > in order to allow reads that are unordered with respect to > those writes, > then 3200 connections are needed. Why would an initiator use multiple sessions instead of flipping the mode page bit that relaxes the SCSI coherency rule? Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Wed Aug 09 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09643; 9 Aug 2000 16:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA04415 for ips-outgoing; Wed, 9 Aug 2000 14:55:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA04411 for ; Wed, 9 Aug 2000 14:55:24 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id LAA11278 for ; Wed, 9 Aug 2000 11:55:23 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.143.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA03366 for ; Wed, 9 Aug 2000 11:55:21 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA05330; Wed, 9 Aug 2000 11:55:21 -0700 (PDT) Message-Id: <200008091855.LAA05330@ha10nwk.EBay.Sun.COM> Date: Wed, 9 Aug 2000 11:55:21 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Multiple TCP connections To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: Qf46JEYyQuGEBe3Tzhmv7w== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk [This mailing list is acting up so forgive me if this is a repeat] Randy provides a good summary of the "design team's" decision of why they thought multiple connections per session. My contention is that the argument is inverted that you need multiple paths through the fabric to get performance and current link layer technology (802.3ad) only provides concurrency based on TCP layer headers thus multiple connections are needed. Further given multiple connections per session you can't have a connection per LUN as there will not be enough connections based on simple math. If you work the argument backwards I see a different result. If you assume instead that we have one connection per LUN we should look at the number of concurrent connections possible. TCP with its 16 bit port number limit us to 64K ports therefore only 64K active connections per IP interface. Given the high bandwidth of existing drives, especially with the amount of cache appearing in controllers, just 10% of the possible connections active will saturate a link layer technology for the next few decades. Therefore to get to the range of that many LUNs and thus connections you will need multiple IP interfaces. Even with the existing draft proposal, no sane implementor would throttle 10K+ LUNs with a single IP interface. I would propose that the requirement be 64K active sessions. Given that requirement having a session per LUN makes sense. The next issue is performance. I agree that to get maximal use of a fabric you need to exploit concurrency. The question becomes where is the correct place to put the concurrency. If we follow the argument that we should have a session per LUN and the standard semantics of LUNs are in order request/response with minimal concurrency leading to a per connection performance requirement that is on par with today's link level technology and protocol implementations and will likely grow at the same rates. The throughput performance that is really a concern is the aggregate bandwidth of an initiator to multiple LUNs. With a session per LUN, each TCP connection can be placed on a different link layer channel (802.3ad) using TCP layer header information. Therefore the performance will scale with the link layer improvements using the existing link layer aggregation mechanism. Ultimately the initiator to LUN bandwidth will be a host memory to storage contoller cache memory copy, currently being designed interconnect technology like Infiniband is exactly designed for such types of copies, so as storage devices move towards a memory to memory model the interconnects will exist. The issue no longer becomes trying to figure out how to exploit multi-link concurrency for a single TCP stream, but what the contents of a single TCP stream is, given that we know existing link technology will allow multiple TCP streams to be passed concurrently. I assert that using a LUN is the natural level of concurrency and the performance demand of a single LUN can be met by existing TCP implementations and should scale over time. The numerical argument given claims that a single storage controller may have 160K concurrent connections. That is likely to be an extreme case with a poorly balanced set of hardware but I will grant it for the sake of argument. The argument is posed that this will be too expensive to maintain that much TCP state. The proposed cost is ~10MB of memory which today will at about $1 to the cost of a box containing 10,000 disk drives (~$1M assuming $100 drives). Not a compelling argument. Furthermore, if you multiplex multiple LUNs per connection you still need sufficient state to mux-demux requests which will be on the same order of magnitude as TCP state. So ultimately the "cost" argument is a wash. > Conclusion: one (or two) TCP connections per LU is both too many (resulting > in too much memory devoted to state records) and too few (insufficient > bandwidth for high-speed IO to controller cache). Decoupling the number of > TCP connections from the number of LUs is the necessary result. I don't buy the conclusion, the amount of memory devoted to state records is relatively small and is actually constant regardless of whether the mux-demux is done at the TCP layer or session layer. Also the driver for interconnect technology is memory to memory copying so the advances in storage technology will not likely outgrow the link layer. -David From owner-ips@ECE.cmu.edu Wed Aug 09 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09656; 9 Aug 2000 16:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA05130 for ips-outgoing; Wed, 9 Aug 2000 15:11:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA05121 for ; Wed, 9 Aug 2000 15:11:17 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id MAA19618 for ; Wed, 9 Aug 2000 12:11:13 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.151.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id MAA05697 for ; Wed, 9 Aug 2000 12:11:00 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id MAA14740; Wed, 9 Aug 2000 12:11:00 -0700 (PDT) Message-Id: <200008091911.MAA14740@ha10nwk.EBay.Sun.COM> Date: Wed, 9 Aug 2000 12:11:00 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Requirements specification To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: yhrfOgQ5h1lswODXS8o64g== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > It would be nice if you would share with us: > > what are the "wonderful rerouting properties" you are referring to ? > what is the link technology to do failover ? > > I am sure only about myself but I guess that many of us are ignorant on the > above. Walk up to any well designed IP network (some ISP's machine room) and randomly cut a network cable, everything just keeps working. Routers detect dead links and use another redundant link. IP technology is giving the Pentagon headaches, no longer does bombing one facility wipe out a command and control network. Cheap technology from Cisco defeats million dollar bombs! > As for availability - several connections and a failover mechanism will > make > a solution more reliable. The question is who is doing the failover. Doing failover between multiple parallel or FC SCSI channels is very well known and in wide practice today. Why is it then a requirement that we put it into iSCSI? -David From owner-ips@ECE.cmu.edu Wed Aug 09 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09878; 9 Aug 2000 17:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA06856 for ips-outgoing; Wed, 9 Aug 2000 15:38:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA06849 for ; Wed, 9 Aug 2000 15:38:06 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id PAA12269 for ; Wed, 9 Aug 2000 15:38:05 -0400 Message-ID: <012701c00236$e576c6b0$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: Subject: Ordering Issues for VI over TCP Date: Wed, 9 Aug 2000 15:20:36 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk This proposal pertains to the layering of VI on top of TCP and is being sent to the VIDF and IPS mailing lists. For proposed VI/TCP spec see: http://www.ietf.org/internet-drafts/draft-dicecco-vitcp-00.txt TCP provides a reliable, in order, transport service. So when VI is layered on top of TCP, the VI layer should see all data from the remote end arrive in order. However it has been proposed that some optimized implementations may want to merge the TCP and VI layers, and do this in such a way that incoming packets, which are sometimes not in order, can be fully processed as they arrive, and the data written to its ultimate destination without needing to be buffered pending arrival of any intervening packets necessary to do full in order processing. To fully exploit this out of order processing, it would be necessary to modify the VI API definition to allow optionally relax ordering. To this end I would offer the following proposal. This proposal is NOT offered with respect to the 1.1 revision currently under discussion, and would only be considered in the 2.0 time frame. I am sending this now in hopes of getting some very preliminary feedback as to whether it makes sense to proceed in this direction. Two additional bits are defined in the control field of all transmit descriptors. These are "un-ordered" and "half-ordered". These bits are hint bits in that an implementation is free to ignore them and would still be fully compliant and interoperable. The meaning of these bits is as follows. If the neither bit is set, then the corresponding operation (send, RDMA read, or RDMA write) will be fully ordered as is required currently by VI. This is true even with respect to other operations which may be un-ordered. If the half-ordered bit is set, then the operation will not be completed on the remote host until after all preceding operations are complete. However if a half-ordered operation is followed by an un-ordered operation, an implementation is free to reorder these two. Half ordered operations are useful to send completion messages which guarantee that previous operations have completed. If two un-ordered operations (with no full ordered operations between them) are done, then an implementation is free to reorder these. The following chart indicates whether ordering is required between two operations. Second Op. first op ordered half-ordered un-ordered ------------------------------------------------------------ ordered yes yes yes half-ordered yes yes no unordered yes yes no The only exception to the above is already provided for in the VI spec in that an RDMA read (without the fence bit set) is not guaranteed to be ordered with respect to a subsequent send or RDMA write. More than one of the fence bit, un-ordered bit, and half-ordered bit should never be set. Note that if an application does two unordered sends, R followed by S, and the remote end posts two receive descriptors, X and Y, then message R may end up in the buffer designated by Y, and S in that by X. There may be cases where the application may want to put sequence numbers in the application level messages, and put them back in order after receiving them out of order. The advantage to doing this at the application layer rather than the TCP layer is that zero copy receives can be done directly to the application and only pointer reordering is needed. I would further propose that there be no ordering requirements on the order in which RDMA read responses are written to memory on the requesting host. I am not sure that this is currently spelled out one way or the other in the VI spec. The only restriction here would be that posting multiple RDMA reads pointing to overlapping local receive buffers would be unpredictable. But this is not something that makes any sense to do anyway. (Does anyone disagree with this?) The above proposed semantics are reflected in the VI/TCP protocol as follows. In addition to the currently defined message types of SEND, RDMA_WRITE, and RDMA_READ_REQUEST, three new types are defined: SEND_UNORDERED, RDMA_WRITE_UNORDERED, and RDMA_READ_REQUEST_UNORDERED. An implementation may (but is not required to) use an unordered message type when the following two conditions are met: 1. The un-ordered bit was set in the corresponding transmit descriptor. 2. There are no ordered messages sent for which TCP ACK has not yet been received. The significance of the half-ordered bit is that it allows subsequent un-ordered messages to be sent with the un-ordered message type without having to wait for the associated TCP ACK. On the receiving end, with a RDMA_WRITE_UNORDERED message type, the contained data may immediately be written directly to the buffer even if previous VI segments are missing. With a RDMA_READ_REQUEST_UNORDERED, the rdma read may likewise be done immediately. On receiving a SEND_UNORDERED, The send may be delivered, but the implementation must behave consistently in the case of segmented sends (i.e. if a pair of sends are reordered at the receiver, all segments of each send must be consistently reordered). Reordering of sends will most likely make sense in the presence of short sends which fit in a single packet. Provided the above paragraph on RDMA read responses is acceptable, the any received VI segment with message type RDMA_READ_RESPONSE may have the data written directly to the receive buffer without waiting for previous packets to arrive. This proposal assumes that VI segments and TCP segments are aligned in that there is exactly one complete VI segment contained in each TCP segments. The mechanism for doing that is not discussed here. From owner-ips@ECE.cmu.edu Wed Aug 09 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10072; 9 Aug 2000 18:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA09238 for ips-outgoing; Wed, 9 Aug 2000 16:29:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ausadmmsrr203.amer.dell.com (ausadmmsrr203.us.dell.com [143.166.82.105]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id QAA09234 for ; Wed, 9 Aug 2000 16:29:39 -0400 (EDT) From: Dan_McConnell@Dell.com Received: from 143.166.99.78 by ausadmmsrr203.amer.dell.com with ESMTP (WorldSecure Server SMTP Relay(WSS) v4.5); Wed, 09 Aug 2000 11:50:25 -0500 X-Server-Uuid: cd3f99d2-8e46-4086-bbba-396330015d70 Received: by ausxc06.aus.amer.dell.com with Internet Mail Service (5.5.2650.21) id ; Wed, 9 Aug 2000 11:50:33 -0500 Message-ID: To: ips@ece.cmu.edu Subject: Translation vs Encapsulation Date: Wed, 9 Aug 2000 11:50:27 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Is the iSCSI layer being designed to encapsulate SCSI Commands or to actually pull out the SCSI command and re-issue them using iSCSI commands? Dan McConnell Enterprise Systems Group Storage Architecture and Technology Evaluation Dan_McConnell@Dell.com Office (512) 723-3287 Lab (512) 723-9173 Fax (512) 723-6657 From owner-ips@ECE.cmu.edu Wed Aug 09 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10085; 9 Aug 2000 18:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA12345 for ips-outgoing; Wed, 9 Aug 2000 17:38:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA12341 for ; Wed, 9 Aug 2000 17:38:13 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA02191 for ; Wed, 9 Aug 2000 14:34:24 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA26081 for ; Wed, 9 Aug 2000 14:31:50 -0700 From: "Douglas Otis" To: Subject: RE: Ordering Issues for VI over TCP Date: Wed, 9 Aug 2000 14:40:18 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-reply-to: <012701c00236$e576c6b0$cb08efd0@sachmo.giganet.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim Have you considered SCTP rather than TCP to allow the required alignment necessary for out of order processing? http://search.ietf.org/internet-drafts/draft-ietf-sigtran-sctp-13.txt Doug > This proposal assumes that VI segments and TCP segments > are aligned in that there is exactly one complete VI > segment contained in each TCP segments. The mechanism > for doing that is not discussed here. From owner-ips@ECE.cmu.edu Wed Aug 09 22:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10166; 9 Aug 2000 18:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA12837 for ips-outgoing; Wed, 9 Aug 2000 17:52:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA12828 for ; Wed, 9 Aug 2000 17:52:27 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e79LqRw02041 for ; Wed, 9 Aug 2000 16:52:27 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA22010; Wed, 9 Aug 00 16:50:21 -0500 Message-Id: <10008092150.AA22010@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Requirements specification In-Reply-To: Message from Stephen Byan of "Wed, 09 Aug 2000 07:55:13 PDT." <8133266FE373D11190CD00805FA768BF02EEA053@shrcmsg1.tdh.qntm.com> Date: Wed, 09 Aug 2000 16:52:21 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > I don't think network channel speeds are increasing briskly. Ethernet was 10 > Mb/s in 1980, 100 Mb/s in 1990, and 1 Gb/s in 2000. These are slow > infrastructural step functions. Past performance is not necessarily indicative of future results. The reason the generation cycle for edge channel technology (Ethernet) has not been faster over the past 20 years is lack of demand. In the 80s and 90s the giants were platform vendors, and the bulk of the smaller hardware companies also produced platform-related things. Not many of those companies left. On the other hand, there are zillions of I/O and networking companies now. Clearly, this is because the demand for networking has increased dramatically. This will inevitably increase the rate at which the technical hurdles to faster channel speeds are overcome. Pipe size is presently one of the least of these hurdles. T11.1 has seen various feasible proposals for physical layers which deliver a substantial fraction of typical platform memory bandwidth (which is the ultimate `speed limit'), for increasingly fast platforms. The most immediate hurdle is what iSCSI is addressing---efficient data delivery at faster speeds. There's no point in having a fast channel if you can't make use of it. Once appropriate hardware acceleration is available, the demand for faster channels will come as applications are developed to use it, and the physical layer technologies are close at hand. > In the 1990's there was a transition from shared bandwidth to switched > bandwidth which helped mitigate the problem of the slow evolution of link > speeds. The biggest problem this began to solve was how to introduce faster channels into the network infrastructure. If you have a shared medium, it is very difficult to support channels of different speeds. This is one of the reasons why faster FC is slow in coming, even though the physical medium technology has been around. If the medium is not shared, you can upgrade on a link by link basis. The other hurdle that has to be crossed once you allow channels of various speeds is congestion. Any infrastructure that lacks some form of congestion control (adaptive (TCP), or bandwidth reservation (ATM)) will not be able to support increasing link speeds effectively. This is another reason why faster FC is slow in coming. What iSCSI brings to the table (assuming it can carry it without dropping it :^), is hardware data handling with congestion control and media independence, hopefully in a way that other protocols besides storage can also use the hardware. This is the dynamite which will blast open the door to memory speed channels. Once you have a memory speed channel, there's no need to stripe. Steph From owner-ips@ECE.cmu.edu Thu Aug 10 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10692; 9 Aug 2000 20:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA14823 for ips-outgoing; Wed, 9 Aug 2000 18:57:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ganymede.or.intel.com (ganymede.or.intel.com [134.134.248.3]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA14819 for ; Wed, 9 Aug 2000 18:57:18 -0400 (EDT) Received: from SMTP (orsmsxvs02-1.jf.intel.com [192.168.65.201]) by ganymede.or.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.30 2000/06/08 18:25:35 dmccart Exp $) with SMTP id PAA06652 for ; Wed, 9 Aug 2000 15:57:16 -0700 (PDT) Received: from orsmsx29.jf.intel.com ([192.168.70.29]) by 192.168.70.201 (Norton AntiVirus for Internet Email Gateways 1.0) ; Wed, 09 Aug 2000 22:57:16 0000 (GMT) Received: by orsmsx29.jf.intel.com with Internet Mail Service (5.5.2650.21) id ; Wed, 9 Aug 2000 15:57:15 -0700 Message-ID: From: "Mesnier, Michael" To: "'Dan_McConnell@Dell.com'" , ips@ece.cmu.edu Subject: RE: Translation vs Encapsulation Date: Wed, 9 Aug 2000 15:57:11 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk iSCSI is an encapsulation protocol. Section 3 of http://www.ietf.org/internet-drafts/draft-satran-iscsi-01.txt goes into more detail. -- Mike Mesnier Enterprise Architecture Laboratory Intel Corporation -----Original Message----- From: Dan_McConnell@Dell.com [mailto:Dan_McConnell@Dell.com] Sent: Wednesday, August 09, 2000 9:50 AM To: ips@ece.cmu.edu Subject: Translation vs Encapsulation Is the iSCSI layer being designed to encapsulate SCSI Commands or to actually pull out the SCSI command and re-issue them using iSCSI commands? Dan McConnell Enterprise Systems Group Storage Architecture and Technology Evaluation Dan_McConnell@Dell.com Office (512) 723-3287 Lab (512) 723-9173 Fax (512) 723-6657 From owner-ips@ECE.cmu.edu Thu Aug 10 00:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10746; 9 Aug 2000 20:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA15255 for ips-outgoing; Wed, 9 Aug 2000 19:11:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA15250 for ; Wed, 9 Aug 2000 19:11:08 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA02210; Wed, 9 Aug 2000 16:07:19 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA26317; Wed, 9 Aug 2000 16:04:45 -0700 From: "Douglas Otis" To: "Stephen Bailey" , Subject: RE: Requirements specification Date: Wed, 9 Aug 2000 16:13:13 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-reply-to: <10008092150.AA22010@candide.cs.uchicago.edu> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Bailey > Sent: Wednesday, August 09, 2000 2:52 PM > To: ips@ece.cmu.edu > Subject: Re: Requirements specification > > > > I don't think network channel speeds are increasing briskly. > Ethernet was 10 > > Mb/s in 1980, 100 Mb/s in 1990, and 1 Gb/s in 2000. These are slow > > infrastructural step functions. You're comparing a times two rate to a times 10 rate. You should also add 10 Gb/s in 2001, and 40 Gb/s in 2002. > The most immediate hurdle is what iSCSI is addressing---efficient data > delivery at faster speeds. There's no point in having a fast channel > if you can't make use of it. Once appropriate hardware acceleration > is available, the demand for faster channels will come as applications > are developed to use it, and the physical layer technologies are close > at hand. iSCSI does not provide any solutions if you consider the bottleneck in this interface is the memory. iSCSI requires ordered delivery, multiple memory copies, split header aggregation, etc. All these things prevent iSCSI from being successful even with dedicated hardware. > > In the 1990's there was a transition from shared bandwidth to switched > > bandwidth which helped mitigate the problem of the slow > evolution of link > > speeds. > > The biggest problem this began to solve was how to introduce faster > channels into the network infrastructure. If you have a shared > medium, it is very difficult to support channels of different speeds. > This is one of the reasons why faster FC is slow in coming, even > though the physical medium technology has been around. If the medium > is not shared, you can upgrade on a link by link basis. > > The other hurdle that has to be crossed once you allow channels of > various speeds is congestion. Any infrastructure that lacks some form > of congestion control (adaptive (TCP), or bandwidth reservation (ATM)) > will not be able to support increasing link speeds effectively. This > is another reason why faster FC is slow in coming. > > What iSCSI brings to the table (assuming it can carry it without > dropping it :^), is hardware data handling with congestion control and > media independence, hopefully in a way that other protocols besides > storage can also use the hardware. This is the dynamite which will > blast open the door to memory speed channels. Once you have a memory > speed channel, there's no need to stripe. > > Steph Odd, memory is the bottleneck. A memory effective solution will be found, but do not expect iSCSI to lead. If you have a high speed channel, there is no reason not to aggregate slower devices. An interim cache will not improve reliability, scale ability, or availability. If this network is feeding a high end server, the server back plane will always offer higher performance and hence the best location for any additional memory and state information. Doug From owner-ips@ECE.cmu.edu Thu Aug 10 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11584; 9 Aug 2000 23:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA21533 for ips-outgoing; Wed, 9 Aug 2000 22:40:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA21528 for ; Wed, 9 Aug 2000 22:40:49 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Wed, 9 Aug 2000 22:40:18 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100E81@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: Towards Consensus on TCP Connections Date: Wed, 9 Aug 2000 22:40:17 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Let's see if we can make some progress towards consensus here. I see two open issues in the discussion on connection issues: (A) Should iSCSI require a TCP connection per LUN? (B) Should iSCSI have a session abstraction that binds multiple TCP connections into one iSCSI connection? On the first issue, (A), I have only seen one individual (David Robinson) advocating the "yes" position. Therefore, I believe there is rough consensus for "no" as the answer to (A). This does not forbid building a system in which each iSCSI target has a single LUN, but also allows multiple LUNs per target. In RFC 2119 terms, I think that this will come down to: - Initiators MUST support multiple LUNs/target - Targets MAY support multiple LUNs/target One of the things that needs to be done to the iSCSI draft is to clarify the requirements by reference to RFC 2119 and use of the capitalized terms found therein. NB: Now I'm going to give the game away by explaining what I just did in IETF terms. As WG co-chair, I have expressed my opinion that rough consensus exists; that opinion may be incorrect since WG co-chairs are not infallible by any means. If anyone other than the one individual noted above believes that "yes" is the best technical answer to (A), they should speak up on the list, especially if I've missed a message advocating "yes" from someone else in reviewing the traffic on the list. Despite Steve Byan's recent change of opinion, I do not believe that we have reached rough consensus on (B). Part of the reason for this is that I'm not sure everyone has clearly understood the consequences of a "no" on (B). Let me start from Costa's simple explanation: > There are two ways in which multiple connections are used in iSCSI: > > 1) Multiple simultaneous TCP connections for bandwidth > > 2) Multiple TCP connections for fault tolerance/recovery > (i.e. when one TCP connection in a session dies, > another one starts up) It is an established practice in the storage industry to achieve both 1) and 2) via the use of multiple SCSI connections, and evidenced by numerous products that do this for host access to storage. Hence a "no" answer to (B) does not preclude the use of multiple TCP connections for bandwidth or fault tolerance/recovery as long as one is willing to use multiple SCSI connections to do so. So, to resolve issue (B), we need to discuss whether there are compelling reasons for doing 1) and/or 2) within a single SCSI connection, as opposed to across multiple SCSI connections. This specific issue needs further discussion -- I invite everyone to have at it ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Aug 10 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11573; 9 Aug 2000 23:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA20508 for ips-outgoing; Wed, 9 Aug 2000 22:06:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA20501 for ; Wed, 9 Aug 2000 22:06:06 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Wed, 9 Aug 2000 22:05:35 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100E80@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: Mailing List Date: Wed, 9 Aug 2000 22:05:34 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk The mailing list maintainer (David Nagle) apologizes for the 2-day outage that we had earlier and the unfortunate consequences of convincing the mailer to behave itself and clear the backlog. While I'm inclined to leave things as they are, if anyone thinks that we should relocate the mailing list now, send me a short email directly (off the list) -- if I get enough of them, getting the list moved will get added to my ever-lengthening ToDo list :-). --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Aug 10 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12368; 10 Aug 2000 2:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA25307 for ips-outgoing; Thu, 10 Aug 2000 01:16:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25301 for ; Thu, 10 Aug 2000 01:16:11 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id D1C5098C; Wed, 9 Aug 2000 22:16:10 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id WAA12920; Wed, 9 Aug 2000 22:15:26 -0700 (PDT) Message-Id: <4.2.2.20000809210401.00b886a0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 09 Aug 2000 21:14:19 -0700 To: David Robinson , ips@ece.cmu.edu From: Michael Krause Subject: Re: Multiple TCP connections In-Reply-To: <200008091855.LAA05330@ha10nwk.EBay.Sun.COM> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:55 AM 8/9/00 -0700, David Robinson wrote: >[This mailing list is acting up so forgive me if this is a repeat] > >Randy provides a good summary of the "design team's" decision of why >they thought multiple connections per session. > >My contention is that the argument is inverted that you need multiple >paths through the fabric to get performance and current link layer >technology (802.3ad) only provides concurrency based on TCP layer >headers thus multiple connections are needed. Further given multiple >connections per session you can't have a connection per LUN as there >will not be enough connections based on simple math. > >If you work the argument backwards I see a different result. If you >assume instead that we have one connection per LUN we should look at >the number of concurrent connections possible. TCP with its 16 bit >port number limit us to 64K ports therefore only 64K active connections >per IP interface. This is per IP address and a given interface can have multiple IP addresses without much complexity. >Given the high bandwidth of existing drives, >especially with the amount of cache appearing in controllers, just 10% >of the possible connections active will saturate a link layer >technology for the next few decades. Therefore to get to the range of >that many LUNs and thus connections you will need multiple IP >interfaces. Even with the existing draft proposal, no sane implementor >would throttle 10K+ LUNs with a single IP interface. Most implementations would be expected to have multiple adapters / chipsets interfaces per storage devices and server - both for load balancing and availability. Also, it is unlikely that any scheduler in hardware is going to be able to handle 64K connections with the state being constantly updated (an on-chip cache will be needed and then the chip will need to access off-chip memory or host memory for the rest of the state structures). In any case, this situation will require high memory bandwidth for the state structures independent of the technology being used (TCP, InfiniBand, VI, etc.). >I would propose that the requirement be 64K active sessions. Given >that requirement having a session per LUN makes sense. Sessions per interface, per host, per endnode? >The next issue is performance. I agree that to get maximal use of a >fabric you need to exploit concurrency. The question becomes where is >the correct place to put the concurrency. If we follow the argument >that we should have a session per LUN and the standard semantics of >LUNs are in order request/response with minimal concurrency leading to >a per connection performance requirement that is on par with today's >link level technology and protocol implementations and will likely grow >at the same rates. The throughput performance that is really a concern >is the aggregate bandwidth of an initiator to multiple LUNs. With a >session per LUN, each TCP connection can be placed on a different link >layer channel (802.3ad) using TCP layer header information. Therefore >the performance will scale with the link layer improvements using the >existing link layer aggregation mechanism. Ultimately the initiator to >LUN bandwidth will be a host memory to storage contoller cache memory >copy, currently being designed interconnect technology like Infiniband >is exactly designed for such types of copies, so as storage devices >move towards a memory to memory model the interconnects will exist. Whether one uses InfiniBand is irrelevant - the issue is whether one uses RDMA semantics in your proposal or not. The work being discussed here does support the use of RDMA technology and thus would meet your requirements. InfiniBand's main priority was a PCI-X replacement and limited distance IPC. Some companies also envision using it as a backbone for other traffic types as well but there are architectural limitations with it being designed for the data center and the associated distances that may limit its value in storage solutions. >The issue no longer becomes trying to figure out how to exploit >multi-link concurrency for a single TCP stream, but what the contents >of a single TCP stream is, given that we know existing link technology >will allow multiple TCP streams to be passed concurrently. I assert >that using a LUN is the natural level of concurrency and the >performance demand of a single LUN can be met by existing TCP >implementations and should scale over time. > >The numerical argument given claims that a single storage controller >may have 160K concurrent connections. That is likely to be an extreme >case with a poorly balanced set of hardware but I will grant it for the >sake of argument. The argument is posed that this will be too expensive >to maintain that much TCP state. The proposed cost is ~10MB of memory >which today will at about $1 to the cost of a box containing 10,000 >disk drives (~$1M assuming $100 drives). Not a compelling argument. >Furthermore, if you multiplex multiple LUNs per connection you still >need sufficient state to mux-demux requests which will be on the same >order of magnitude as TCP state. So ultimately the "cost" argument is a >wash. > > > Conclusion: one (or two) TCP connections per LU is both too many (resulting > > in too much memory devoted to state records) and too few (insufficient > > bandwidth for high-speed IO to controller cache). Decoupling the number of > > TCP connections from the number of LUs is the necessary result. > >I don't buy the conclusion, the amount of memory devoted to state >records is relatively small and is actually constant regardless of >whether the mux-demux is done at the TCP layer or session layer. Also >the driver for interconnect technology is memory to memory copying >so the advances in storage technology will not likely outgrow the link >layer. Mike From owner-ips@ECE.cmu.edu Thu Aug 10 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12416; 10 Aug 2000 2:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA24857 for ips-outgoing; Thu, 10 Aug 2000 00:57:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (adsl-63-192-219-28.dsl.snfc21.pacbell.net [63.192.219.28]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA24850 for ; Thu, 10 Aug 2000 00:57:41 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id NAA06202; Thu, 10 Aug 2000 13:58:14 -0700 To: ips@ece.cmu.edu Subject: Re: Towards Consensus on TCP Connections References: <0F31E5C394DAD311B60C00E029101A0704100E81@corpmx9.isus.emc.com> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 10 Aug 2000 13:58:14 -0700 Message-ID: Lines: 30 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com writes: > > There are two ways in which multiple connections are used in iSCSI: > > > > 1) Multiple simultaneous TCP connections for bandwidth > > > > 2) Multiple TCP connections for fault tolerance/recovery > > (i.e. when one TCP connection in a session dies, > > another one starts up) > > It is an established practice in the storage industry > to achieve both 1) and 2) via the use of multiple > SCSI connections, and evidenced by numerous products > that do this for host access to storage. Hence a > "no" answer to (B) does not preclude the use of > multiple TCP connections for bandwidth or fault > tolerance/recovery as long as one is willing to use > multiple SCSI connections to do so. On recovery, a big concern was the tape backup issue. Do SCSI applications recover gracefully today from failed SCSI connections? My understanding was that many tape backups program abort the backup. Related to recovery, when a TCP/SCSI connection closes, what ramifications does it have on device state (like mode pages, PREVENT/ALLOW REMOVAL, RESERVE/RELEASE, etc.)? Where does SCSI specify this? Thanks, -Costa From owner-ips@ECE.cmu.edu Thu Aug 10 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12359; 10 Aug 2000 2:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA25294 for ips-outgoing; Thu, 10 Aug 2000 01:15:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25290 for ; Thu, 10 Aug 2000 01:15:55 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 28F8DAB8; Wed, 9 Aug 2000 22:15:51 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id WAA12894; Wed, 9 Aug 2000 22:15:06 -0700 (PDT) Message-Id: <4.2.2.20000809203606.00afd8d0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 09 Aug 2000 20:48:41 -0700 To: "Douglas Otis" , "Stephen Bailey" , From: Michael Krause Subject: RE: Requirements specification In-Reply-To: References: <10008092150.AA22010@candide.cs.uchicago.edu> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:13 PM 8/9/00 -0700, Douglas Otis wrote: > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Stephen Bailey > > Sent: Wednesday, August 09, 2000 2:52 PM > > To: ips@ece.cmu.edu > > Subject: Re: Requirements specification > > > > > > > I don't think network channel speeds are increasing briskly. > > Ethernet was 10 > > > Mb/s in 1980, 100 Mb/s in 1990, and 1 Gb/s in 2000. These are slow > > > infrastructural step functions. > >You're comparing a times two rate to a times 10 rate. You should also add >10 Gb/s in 2001, and 40 Gb/s in 2002. 10 GbE standard is finalized in March 2002. 100 Gbps optics have been demonstrated and one would expect to jump to that as the next wide-area / metro solution and possibly the data center backbone via a 100 GbE within 5-7 years after 10 GbE is finalized. These are fabrics that one attaches into via some speed relative to the adapter / chipset that is being used within the server or storage device. >iSCSI does not provide any solutions if you consider the bottleneck in this >interface is the memory. iSCSI requires ordered delivery, multiple memory >copies, split header aggregation, etc. All these things prevent iSCSI from >being successful even with dedicated hardware. All external secondary fabrics independent of the protocol are limited by the memory bandwidth - this has many components: coherency overheads, D-cache miss rates, actual data manipulation, etc. Use of iSCSI is not the issue per se but the impact of the protocol and its processing on that memory bandwidth. Even technologies like InfiniBand or VI are similarly limited by the memory bandwidth, have similar levels of complexity and resource requirements, and do not provide any solutions per se. The key is to off-load as much of the protocol processing as possible so that the application is the main consumer of the memory bandwidth. iSCSI does provide the ability to implement a protocol off-load solution. >Odd, memory is the bottleneck. A memory effective solution will be found, >but do not expect iSCSI to lead. If you have a high speed channel, there is >no reason not to aggregate slower devices. An interim cache will not >improve reliability, scale ability, or availability. If this network is >feeding a high end server, the server back plane will always offer higher >performance and hence the best location for any additional memory and state >information. Whether one integrates the server point of attachment with the memory controller is independent of this protocol discussion - the issue is how much memory is required to be accessed that is protocol-specific and whether this is best saved on off-chip memory at the server point of attachment or in the server memory. There are pros / cons for each given not all memory solutions provide the needed bandwidth nor latency (latency is critical for protocol state tables, etc.). From a spec point of view all of the above is implementation-specific but implementation impacts should be evaluated to determine whether a given feature within the specification is practical to implement or not with the technology expected in the product deployment time frame. Mike From owner-ips@ECE.cmu.edu Thu Aug 10 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14650; 10 Aug 2000 10:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA19085 for ips-outgoing; Thu, 10 Aug 2000 09:33:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-1.cisco.com (sj-msg-core-1.cisco.com [171.71.163.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA17897 for ; Thu, 10 Aug 2000 09:02:02 -0400 (EDT) From: fred@cisco.com Received: from irp-view7.cisco.com (irp-view7.cisco.com [171.69.25.24]) by sj-msg-core-1.cisco.com (8.9.3/8.9.1) with ESMTP id GAA11360; Thu, 10 Aug 2000 06:01:48 -0700 (PDT) Received: (fred@localhost) by irp-view7.cisco.com (8.8.8-Cisco List Logging/CISCO.WS.1.2) id GAA28725; Thu, 10 Aug 2000 06:01:31 -0700 (PDT) Date: Thu, 10 Aug 2000 06:01:31 -0700 (PDT) Message-Id: <200008101301.GAA28725@irp-view7.cisco.com> To: haagens@hp.com, ips@ece.cmu.edu Subject: draft-haagens-ips-iscsireqs-00.txt Sender: owner-ips@ece.cmu.edu Precedence: bulk I notice that you recently posted a new internet draft. A document that might be worth reading is http://www.ietf.org/ID-nits.html This document was put together by members of the IESG to help the community know what are the most common problems that we see in documents, whether from working groups or individual submissions, and pro-actively avoid them. This is an automatic email; please don't conclude that I have discovered something awful about your draft, as I haven't even read it yet. Rather, I'm just letting you know that there is a list of common issues that get raised over and over again in IESG reviews, and I'd like to save you the hassle by letting you see for yourself whether any of them apply. Let me know if you have any questions I can answer. From owner-ips@ECE.cmu.edu Thu Aug 10 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14648; 10 Aug 2000 10:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA19060 for ips-outgoing; Thu, 10 Aug 2000 09:33:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25888 for ; Thu, 10 Aug 2000 01:42:01 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA50970 for ; Thu, 10 Aug 2000 07:41:30 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id HAA34204 for ; Thu, 10 Aug 2000 07:41:30 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256937.001F42E0 ; Thu, 10 Aug 2000 07:41:27 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 10 Aug 2000 08:39:36 +0300 Subject: Re: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David writes: ___________________ ..... It is an established practice in the storage industry to achieve both 1) and 2) via the use of multiple SCSI connections, and evidenced by numerous products that do this for host access to storage. Hence a "no" answer to (B) does not preclude the use of multiple TCP connections for bandwidth or fault tolerance/recovery as long as one is willing to use multiple SCSI connections to do so. So, to resolve issue (B), we need to discuss whether there are compelling reasons for doing 1) and/or 2) within a single SCSI connection, as opposed to across multiple SCSI connections. This specific issue needs further discussion -- I invite everyone to have at it ... --David ___________________ That is a fair description of the status-quo except for the following points: i) a good number of mainframe class controllers aggregate bandwidth and links underneath the channel layer (the rough equivalent of SCSI) for reasons similar to those neatly stated in Michael Krause comment - anything above the channel layer is oblivious to the channel configuration (NB the aggregation scheme used there is even more radical than the one in the current iSCSI draft - even the allegiance mechanism is missing) ii) parallel SCSI buses have different failure modes than a SAN; with parallel buses at low speed - it is safe to assume that if a command fails it is the device/controller that is failing and retry is seldom an issue. As a result of this history applications and devices HANDLE BADLY recovery at SCSI level - the most notorious being the tapes. The FC community has realized this after the first version of FCP was out and gone a long way to enable recovery at the channel level (introduced command numbering to solve this recovery issue). Considering that failures in a complex network have a substantially higher probability than on parallel buses and that percolating them to the upper layers looks like a bad idea we had only few design choices - expect TCP to introduce a recovery mechanism or provide one of our own. Another way (and a more abstract one) to look at it is that the SCSI stack was designed with simple conceptual layers (device, transport, physical interconnect) and as the physical interconnect gets more complex you have to adjust the transport in order to maintain simplicity at the device (SCSI command) level. Julo From owner-ips@ECE.cmu.edu Thu Aug 10 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14643; 10 Aug 2000 10:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA17988 for ips-outgoing; Thu, 10 Aug 2000 09:04:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA17984 for ; Thu, 10 Aug 2000 09:04:31 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id GAA23176 for ; Thu, 10 Aug 2000 06:04:30 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Thu, 10 Aug 2000 06:04:27 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF02EEA059@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Towards Consensus on TCP Connections Date: Thu, 10 Aug 2000 06:04:27 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk David Black wrote: > (B) Should iSCSI have a session abstraction that > binds multiple TCP connections into one > iSCSI connection? [snip] > Let me start from Costa's simple explanation: > > > There are two ways in which multiple connections are used in iSCSI: > > > > 1) Multiple simultaneous TCP connections for bandwidth > > > > 2) Multiple TCP connections for fault tolerance/recovery > > (i.e. when one TCP connection in a session dies, > > another one starts up) > [snip] > So, to resolve issue (B), we need to discuss whether > there are compelling reasons for doing 1) and/or 2) > within a single SCSI connection, as opposed to across > multiple SCSI connections. This specific issue needs > further discussion -- I invite everyone to have at it ... As I've explained in a previous post, I see a need for low-cost bandwidth in excess of 1 Gb/s for low-end single-LUN iSCSI disks. This bandwidth should be available to a single iSCSI command from an initiator. I believe the only low-cost link technology available to us is 1 Gb/s ethernet over cat5 UTP (is this 1000baseT in IEEE-speak?). To get sufficient bandwidth at sufficiently low cost in early device-level implementations, I think the use of multiple TCP connections over 1 Gb/s ethernet links is the only feasible option. 10 Gb/s ethernet will be too expensive for the initial device-level implementations, especially since I think device-level customers will require redundant interfaces, so a device would have to bear the cost burden of two 10 Gb/s interfaces. So I think the requirements for multiple connections go beyond what is currently in draft-satran-iscsi-01.txt, which only allows load-sharing at the SCSI command-level. Note that for disks, load-sharing at the command-level could be obtained by multiple SCSI connections rather than multiple TCP connections. I think it is important to increase the bandwidth available to the data transfer for a single SCSI command, so an extension to the current multiple TCP connection proposal looks like the right answer. The extension I propose is to remove the allegiance to the initiating TCP connection for data transfers _only_. Not RTT's, not commands, not status, not task management function requests, etc. Only data transfers. Note that this probably implies something like the iSCSI over VI over TCP proposal, in order to decouple the adapter state needed for RDMA data transfer from the command state. When I consider tapes, the alternative of multiple SCSI connections looks even less appealing. Unlike disks, tape reads and writes are not idempotent, so SCSI-level error recovery is quite complex because the initiator needs to discover the current state of the tape. Multiple SCSI sessions don't directly address this problem; all they offer are redundant paths to the device. The iSCSI session concept looks good because it offers a way to retry individual information units and continue the execution of a SCSI command in the presence of link errors, without having to invoke SCSI-level tape error recovery. With respect to tape device bandwidth requirements, I don't have an informed opinion. However, I note that if the device-level requirement should exceed the speed of economically-feasible links, multiple SCSI connections do not solve the bandwidth aggregation problem, since the bandwidth must be applied to the data transfers for an ordered set of SCSI commands. Multiple SCSI connections do not preserve the ordering of commands. So if the tape device bandwidth requirements exceed that of a single link, multiple TCP connections would be required. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Thu Aug 10 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14663; 10 Aug 2000 10:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA19070 for ips-outgoing; Thu, 10 Aug 2000 09:33:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA26374 for ; Thu, 10 Aug 2000 02:03:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA139610 for ; Thu, 10 Aug 2000 08:02:49 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id IAA46676 for ; Thu, 10 Aug 2000 08:02:49 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256937.00213662 ; Thu, 10 Aug 2000 08:02:46 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 10 Aug 2000 09:00:58 +0300 Subject: Re: Requirements specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I did walk (more than once) into a complex network - but instead of looking at screens I took a look at logs - and they tell a different story than yours. A link outage at the middle of a network may result (and usually does) in a TCP connection failing. Sometimes this is handled by applications but not always. And yes - the network does not fail but this a statistical argument - some applications may loose hours of work as SCSI was designed with a very reliable interconnect in mind. A link outage at the end nodes always results in TCP connections getting dropped. And failover is handled badly at the SCSI layer (and FCP-2 goes a long way to avoid percolating errors to SCSI). Julo David Robinson on 09/08/2000 22:11:00 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Requirements specification > It would be nice if you would share with us: > > what are the "wonderful rerouting properties" you are referring to ? > what is the link technology to do failover ? > > I am sure only about myself but I guess that many of us are ignorant on the > above. Walk up to any well designed IP network (some ISP's machine room) and randomly cut a network cable, everything just keeps working. Routers detect dead links and use another redundant link. IP technology is giving the Pentagon headaches, no longer does bombing one facility wipe out a command and control network. Cheap technology from Cisco defeats million dollar bombs! > As for availability - several connections and a failover mechanism will > make > a solution more reliable. The question is who is doing the failover. Doing failover between multiple parallel or FC SCSI channels is very well known and in wide practice today. Why is it then a requirement that we put it into iSCSI? -David From owner-ips@ECE.cmu.edu Thu Aug 10 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14668; 10 Aug 2000 10:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA19079 for ips-outgoing; Thu, 10 Aug 2000 09:33:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA27645 for ; Thu, 10 Aug 2000 03:03:41 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id JAA12498 for ; Thu, 10 Aug 2000 09:03:09 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id JAA51564 for ; Thu, 10 Aug 2000 09:03:09 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256937.0026AE35 ; Thu, 10 Aug 2000 09:02:29 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 10 Aug 2000 09:54:36 +0300 Subject: 1 per LU or session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk There is a subtle argument against 1 connection/LU that plays in the network. Assuming that QoS will come into play in storage too (with SLAs and all the rest) and that in the network QoS decisions are based on flow it is not very network friendly to have a large number of TCP connections. With the current design "virtual" control units could aggregate - for an initiator - a number of LUs with similar QoS (and virtual CUs are also supported by our naming scheme). Julo From owner-ips@ECE.cmu.edu Thu Aug 10 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15459; 10 Aug 2000 13:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA25514 for ips-outgoing; Thu, 10 Aug 2000 12:02:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA25509 for ; Thu, 10 Aug 2000 12:02:50 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA44264 for ; Thu, 10 Aug 2000 18:02:19 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id SAA23626 for ; Thu, 10 Aug 2000 18:02:19 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256937.0058183C ; Thu, 10 Aug 2000 18:02:13 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 10 Aug 2000 18:54:37 +0300 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk We had this in draft 00! (Adelaide). It was deemed too complex by the team and had the perceived disadvantage of having to setup too much address-association information in the adapters (as every adapter has to be ready for any piece of data). A variant of it (easier palatable) was recently suggested by Kalman Meth (Kalman could you please re-post it?). Kalman's proposal could be extended to enable complete "data striping". If the teams shows enough interested we might attempt to work it through e-mail (a memo describing some details). It does not require more RDMA support than the current one. One side-effect of it is that control information runs on a single connection and command sequencing becomes unnecessary and flow control is a TCP window (I think). Julo Stephen Byan on 10/08/2000 16:04:27 Please respond to Stephen Byan To: "'ips@ece.cmu.edu'" cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Towards Consensus on TCP Connections David Black wrote: > (B) Should iSCSI have a session abstraction that > binds multiple TCP connections into one > iSCSI connection? [snip] > Let me start from Costa's simple explanation: > > > There are two ways in which multiple connections are used in iSCSI: > > > > 1) Multiple simultaneous TCP connections for bandwidth > > > > 2) Multiple TCP connections for fault tolerance/recovery > > (i.e. when one TCP connection in a session dies, > > another one starts up) > [snip] > So, to resolve issue (B), we need to discuss whether > there are compelling reasons for doing 1) and/or 2) > within a single SCSI connection, as opposed to across > multiple SCSI connections. This specific issue needs > further discussion -- I invite everyone to have at it ... As I've explained in a previous post, I see a need for low-cost bandwidth in excess of 1 Gb/s for low-end single-LUN iSCSI disks. This bandwidth should be available to a single iSCSI command from an initiator. I believe the only low-cost link technology available to us is 1 Gb/s ethernet over cat5 UTP (is this 1000baseT in IEEE-speak?). To get sufficient bandwidth at sufficiently low cost in early device-level implementations, I think the use of multiple TCP connections over 1 Gb/s ethernet links is the only feasible option. 10 Gb/s ethernet will be too expensive for the initial device-level implementations, especially since I think device-level customers will require redundant interfaces, so a device would have to bear the cost burden of two 10 Gb/s interfaces. So I think the requirements for multiple connections go beyond what is currently in draft-satran-iscsi-01.txt, which only allows load-sharing at the SCSI command-level. Note that for disks, load-sharing at the command-level could be obtained by multiple SCSI connections rather than multiple TCP connections. I think it is important to increase the bandwidth available to the data transfer for a single SCSI command, so an extension to the current multiple TCP connection proposal looks like the right answer. The extension I propose is to remove the allegiance to the initiating TCP connection for data transfers _only_. Not RTT's, not commands, not status, not task management function requests, etc. Only data transfers. Note that this probably implies something like the iSCSI over VI over TCP proposal, in order to decouple the adapter state needed for RDMA data transfer from the command state. When I consider tapes, the alternative of multiple SCSI connections looks even less appealing. Unlike disks, tape reads and writes are not idempotent, so SCSI-level error recovery is quite complex because the initiator needs to discover the current state of the tape. Multiple SCSI sessions don't directly address this problem; all they offer are redundant paths to the device. The iSCSI session concept looks good because it offers a way to retry individual information units and continue the execution of a SCSI command in the presence of link errors, without having to invoke SCSI-level tape error recovery. With respect to tape device bandwidth requirements, I don't have an informed opinion. However, I note that if the device-level requirement should exceed the speed of economically-feasible links, multiple SCSI connections do not solve the bandwidth aggregation problem, since the bandwidth must be applied to the data transfers for an ordered set of SCSI commands. Multiple SCSI connections do not preserve the ordering of commands. So if the tape device bandwidth requirements exceed that of a single link, multiple TCP connections would be required. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Thu Aug 10 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15486; 10 Aug 2000 13:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA26997 for ips-outgoing; Thu, 10 Aug 2000 12:42:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA26991 for ; Thu, 10 Aug 2000 12:42:00 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e7AGfxw17914 for ; Thu, 10 Aug 2000 11:41:59 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA22748; Thu, 10 Aug 00 11:39:51 -0500 Message-Id: <10008101639.AA22748@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Towards Consensus on TCP Connections In-Reply-To: Message from csapuntz@cisco.com of "10 Aug 2000 13:58:14 PDT." Date: Thu, 10 Aug 2000 11:41:50 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > On recovery, a big concern was the tape backup issue. Do SCSI > applications recover gracefully today from failed SCSI connections? > My understanding was that many tape backups program abort the backup. Tape is hard. The main reason is that when an error occurs on a {READ, WRITE} SEQUENTIAL, you don't really know what state the tape is in. Maybe the tape has advanced by the length of the failed operation Maybe not. Maybe the tape has been eaten. Maybe the tape has been ejected. It's really hard to do anything at ANY layer except go into heavy duty recovery (rewind and try again). There are two distinct applications of tape with different requirements, backup and streaming data recording. Backup is by far the most common application. Many backup applications don't attempt recovery because they assume that correcting the problem will probably require operator intervention. Amanda is an example of a backup application which does recovery correctly, and in essence, it operates a layer above the backup applications that actually touch the tape. It is responsible for buffering the data (on a disk), notifying the operator of the failure, and trying it again on some arranged schedule, or on operator request. The best thing you can do to improve tape behavior in either the backup or streaming application is to improve the reliability of the data transport, which is exactly what iSCSI does simply by using TCP. The problem FC had was that when you write an arbitrary amount of data, eventually you WILL get a media layer error and then you're lost. With FC error rates, this is usually only a problem for the streaming data application. Nonetheless, although the streaming data application is the minority, the customers are high profile and have huge installations. > Related to recovery, when a TCP/SCSI connection closes, what ramifications > does it have on device state (like mode pages, PREVENT/ALLOW REMOVAL, > RESERVE/RELEASE, etc.)? Where does SCSI specify this? This is a good question. FC sorta blew this one originally. Reservations did not even persist across hot plugs of uninvolved equipment in FC-AL. As a result, you have clustering software that rereserves every few seconds `just in case'. In the case of FC, the obvious solution was to reserve by node name (WWN which is unique to the device, as opposed to port name, which is unique to the attachment point). The mistake was drawing too direct an analogy between parallel SCSI and FC. Parallel SCSI had limited addressing and relatively stable topology, and FC had wider addressing and a much more dynamic topology. Currently FC is somewhat mired in backward compatability issues with respect to these recovery topics. Hopefully iSCSI will follow a more enlightened course. Steph From owner-ips@ECE.cmu.edu Thu Aug 10 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15652; 10 Aug 2000 14:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA27807 for ips-outgoing; Thu, 10 Aug 2000 13:01:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA27802 for ; Thu, 10 Aug 2000 13:01:22 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e7AH1Lw18541 for ; Thu, 10 Aug 2000 12:01:21 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA22772; Thu, 10 Aug 00 11:59:15 -0500 Message-Id: <10008101659.AA22772@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Towards Consensus on TCP Connections In-Reply-To: Message from Stephen Byan of "Thu, 10 Aug 2000 06:04:27 PDT." <8133266FE373D11190CD00805FA768BF02EEA059@shrcmsg1.tdh.qntm.com> Date: Thu, 10 Aug 2000 12:01:13 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > As I've explained in a previous post, I see a need for low-cost bandwidth in > excess of 1 Gb/s for low-end single-LUN iSCSI disks. This bandwidth should > be available to a single iSCSI command from an initiator. The gating factor for whether iSCSI succeeds is not going to be 200 MB/s instead of 100 MB/s out of a single LUN. If iSCSI works at ALL in a cost effective way that can be implemented in a disk, there'll be wild dancing in the streets and you'll all (or maybe your companies will) be rich beyond the dreams of avarice. The easier you can make it for the implementors, the more likely it will succeed. > When I consider tapes, the alternative of multiple SCSI connections looks > even less appealing. Unlike disks, tape reads and writes are not idempotent, > so SCSI-level error recovery is quite complex because the initiator needs to > discover the current state of the tape. I don't see how multiple TCP connections in an iSCSI session can solve this problem in a way that works for tapes either. Specifically, the time to discover that the path you're using has failed and switch to another one (O(seconds)) is going to be longer than the amount of data the tape has in it's buffer. In this case, it seems an upper layer failover mechanism could work just as well (poorly). Steph From owner-ips@ECE.cmu.edu Thu Aug 10 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15958; 10 Aug 2000 15:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA29967 for ips-outgoing; Thu, 10 Aug 2000 13:53:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA29963 for ; Thu, 10 Aug 2000 13:53:32 -0400 (EDT) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id KAA22792 for ; Thu, 10 Aug 2000 10:53:00 -0700 (PDT) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id KAA22122 for ; Thu, 10 Aug 2000 10:52:59 -0700 (PDT) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Thu, 10 Aug 2000 10:52:47 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C05B31C90@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: ips@ece.cmu.edu Subject: RE: Towards Consensus on TCP Connections Date: Thu, 10 Aug 2000 10:52:43 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > (A) Should iSCSI require a TCP connection per LUN? David Robinson is not the only one who advocates a session/LUN. I've stated publicly that I feel that this is the preferred method of operation and the current version of SEP mandates it. Julian has stated several times that this is the simplest method. That being said, I do appreciate the concerns regarding resource requirements. Randy's example used 64B for state as a wild guess, and a low one at that (our implementation is currently using about 0.5K/TCP connection), and that is before you factor in how you allocate send/receive buffers. I still purport that a vast majority of targets will have the resources to handle session/LUN, even those with many LUs because they have lots of memory. But, I agree that the protocol must not require it. That is why the next version of SEP will include the LUN field in the header. What I advocate is the ability to establish multiple sessions between initiator and target and assign 'n' LUs to each session, where 'n' is 1 to all. Somesh states "the way it [iSCSI] is currently defined allows implementors to implement one connection per lun or one connection per (arbitrary) connection of luns. I think this is more flexible as it would allow either implementation or migration over time from one form to the other." This is true, but the draft assumes one session per target and there is no provision to negotiate how many sessions and how to map LUNs to sessions. Another point, if the targets support ACLs for authorization purposes, it will know ahead of time what the maximum number of connections. Paul From owner-ips@ECE.cmu.edu Fri Aug 11 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17156; 10 Aug 2000 20:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA12148 for ips-outgoing; Thu, 10 Aug 2000 19:02:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA12141 for ; Thu, 10 Aug 2000 19:01:56 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 5491E10C8 for ; Thu, 10 Aug 2000 16:01:51 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with SMTP id RAA26216 for ; Thu, 10 Aug 2000 17:01:48 -0600 (MDT) Received: from 15.56.8.172 by xboibrg2.boi.hp.com (InterScan E-Mail VirusWall NT); Thu, 10 Aug 2000 17:01:25 -0600 (Mountain Daylight Time) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Thu, 10 Aug 2000 17:01:23 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E973E@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Subject: iSCSI Requirements Slides presented at IETF-48 Date: Thu, 10 Aug 2000 17:00:40 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C0031E.CDCC62C0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C0031E.CDCC62C0 Content-Type: text/plain; charset="iso-8859-1" Here are the slides that I presented in the requirements discusion for iSCSI, at IETF-48. I've included a couple of "bonus" slides that I might have presented, had there been time. Best regards to all, Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 <> <> ------_=_NextPart_000_01C0031E.CDCC62C0 Content-Type: application/octet-stream; name="Haagens, Randy (E-mail).vcf" Content-Disposition: attachment; filename="Haagens, Randy (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000607T194543Z END:VCARD ------_=_NextPart_000_01C0031E.CDCC62C0 Content-Type: application/octet-stream; name="iSCSI Requirements v0.4 slides.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="iSCSI Requirements v0.4 slides.pdf" JVBERi0xLjINJeLjz9MNCjYxIDAgb2JqDTw8IA0vTGluZWFyaXplZCAxIA0vTyA2MyANL0ggWyA3 MDIgMzMyIF0gDS9MIDYxMTM3IA0vRSA0NDkwIA0vTiAxNSANL1QgNTk3OTkgDT4+IA1lbmRvYmoN ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB4cmVmDTYxIDE1IA0wMDAwMDAwMDE2IDAwMDAwIG4NCjAwMDAwMDA2NDcgMDAwMDAgbg0KMDAw MDAwMTAzNCAwMDAwMCBuDQowMDAwMDAxMTkwIDAwMDAwIG4NCjAwMDAwMDEzNDkgMDAwMDAgbg0K MDAwMDAwMTUyOSAwMDAwMCBuDQowMDAwMDAyMTMyIDAwMDAwIG4NCjAwMDAwMDIzMTkgMDAwMDAg bg0KMDAwMDAwMjg3MyAwMDAwMCBuDQowMDAwMDAzMDYwIDAwMDAwIG4NCjAwMDAwMDM1MDUgMDAw MDAgbg0KMDAwMDAwNDA3MyAwMDAwMCBuDQowMDAwMDA0MjU5IDAwMDAwIG4NCjAwMDAwMDA3MDIg MDAwMDAgbg0KMDAwMDAwMTAxMyAwMDAwMCBuDQp0cmFpbGVyDTw8DS9TaXplIDc2DS9JbmZvIDYw IDAgUiANL1Jvb3QgNjIgMCBSIA0vUHJldiA1OTc4OSANL0lEWzxhNmY1YTRiY2RhNTVlODVhZDE4 ZjdiMzJmZjE1NzJhOD48YTZmNWE0YmNkYTU1ZTg1YWQxOGY3YjMyZmYxNTcyYTg+XQ0+Pg1zdGFy dHhyZWYNMA0lJUVPRg0gICAgIA02MiAwIG9iag08PCANL1R5cGUgL0NhdGFsb2cgDS9QYWdlcyA1 OCAwIFIgDT4+IA1lbmRvYmoNNzQgMCBvYmoNPDwgL1MgMjQ0IC9GaWx0ZXIgL0ZsYXRlRGVjb2Rl IC9MZW5ndGggNzUgMCBSID4+IA1zdHJlYW0NCkiJYmBgYGZgYNrHwMLAwHKSgZcBAXiBYqxAzNHA YAAWcPhQcrDzUGZQmCVzY2AnQ6i8TeqFdpMpDAwdfZ0ZGTs6GsCqDE4UKmopHEleqGXZ4nh0ctWD mxqHFY+cWPFMkuXVRWfFllKBJMMLU3naBK84dhx3CimccEYk2x2ipFmwr2UFgsXApOLBwMAowcAo KAGkhZRAXAYOIFMQaFUDgyASxeACkQMrhdA4nQsEegwMNxcCaW4gFgOL+DFwCbcKzuKVjj24Nb+q 7QBDXmGYQsPOE22FB/QZe0E2AAQYAPD8T4kNZW5kc3RyZWFtDWVuZG9iag03NSAwIG9iag0yMjYg DWVuZG9iag02MyAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNTcgMCBSIA0vUmVzb3Vy Y2VzIDY0IDAgUiANL0NvbnRlbnRzIDY4IDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAg MCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag02NCAwIG9i ag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNjYgMCBSIC9UVDQg NzAgMCBSIC9UVDYgNzEgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgNzMgMCBSID4+IA0vQ29s b3JTcGFjZSA8PCAvQ3M1IDY1IDAgUiA+PiANPj4gDWVuZG9iag02NSAwIG9iag1bIA0vQ2FsUkdC IDw8IC9XaGl0ZVBvaW50IFsgMC45NTA1IDEgMS4wODkgXSAvR2FtbWEgWyAyLjIyMjIxIDIuMjIy MjEgMi4yMjIyMSBdIA0vTWF0cml4IFsgMC40MTI0IDAuMjEyNiAwLjAxOTMgMC4zNTc2IDAuNzE1 MTkgMC4xMTkyIDAuMTgwNSAwLjA3MjIgMC45NTA1IF0gPj4gDQ1dDWVuZG9iag02NiAwIG9iag08 PCANL1R5cGUgL0ZvbnQgDS9TdWJ0eXBlIC9UcnVlVHlwZSANL0ZpcnN0Q2hhciAzMiANL0xhc3RD aGFyIDE2NyANL1dpZHRocyBbIDMxMCAwIDAgMCAwIDAgNjQyIDAgMjQwIDI0MCAwIDAgMzEwIDE5 MCAzMTAgNjQwIDYyMCA2MjAgNjIwIDYyMCA2MjAgDTYyMCA2MjAgNjIwIDYyMCA2MjAgMzEwIDMx MCAwIDAgMCAwIDAgNjQxIDU2NSA3MzkgNjUyIDUwNyA0NDQgMCANNzE2IDIxMCAyMTggMCAzMjcg ODk3IDg0MyA4NTQgNDcyIDAgNDg4IDU0NCA0NTAgNjYwIDU1NSAxMTA1IDAgMCANMCAzMDAgMCAz MDAgMCAwIDAgNTYwIDU2MCA0NzcgNTYwIDUxMCAyNDAgNTY3IDQ5MCAyMzQgMjM0IDQyNyAxOTkg DTcyNSA0OTAgNTQ3IDU2MCA1NjAgMjc4IDM3NSAyMDkgNDcyIDQzMyA3NTIgNDQzIDQ2MCA0ODAg MCAwIDAgMCANMCAwIDAgMCAwIDAgMTAwMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAyNTkgMzkw IDM5MCAwIDAgMCAwIDAgMCANMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgNjIwIF0gDS9FbmNvZGlu ZyAvV2luQW5zaUVuY29kaW5nIA0vQmFzZUZvbnQgL0Z1dHVyYS1MaWdodCANL0ZvbnREZXNjcmlw dG9yIDY3IDAgUiANPj4gDWVuZG9iag02NyAwIG9iag08PCANL1R5cGUgL0ZvbnREZXNjcmlwdG9y IA0vQXNjZW50IDEwMzYgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQgLTI3MCANL0ZsYWdzIDMyIA0v Rm9udEJCb3ggWyAtMTg5IC0yNzAgMTEyNiAxMDM2IF0gDS9Gb250TmFtZSAvRnV0dXJhLUxpZ2h0 IA0vSXRhbGljQW5nbGUgMCANL1N0ZW1WIDAgDT4+IA1lbmRvYmoNNjggMCBvYmoNPDwgL0xlbmd0 aCA0ODAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSIlUUstu2zAQvOsr9kgWJb18 ir61dYqmORRBrFtRFIJM22piO5FkpP37LklbQUBAFDma2ZldLVajg24EldfYVYtvawW7sVLQQ2W8 tAFqlzdnEUStEYZYbasvTbVoGk2kZlsRECRayA8Ea6SlA0qLSPiB8LRIXaBEhRqa8orKQvNa/WT3 LReqlobtuJWORS60loGB4r+au0ooOgT0kNh2Cc1Nlch+OesYV3TQceWkYnB35opcsyegC6YRyz1m PTJuZ+MGwSR3xiqpPDgVpLfZNOliPZdwoZTo16v1d3iIL2du5JL1Q97iIR6nMauTpsbUiPIkZUWZ NDizlDok5ayHLkljEd0M7XYS+5bX0rN2F48jF4YaIThKzXruqcQzVwS+By6njnj6CvVDfMk67z5F lNPf6Rrfl/iCjAUfgPYQvE6Nzd60mmNT17PDBy4ca8mIZsfNP7hti8tLYCWNqu3ML9mKlC78m54m q9nARcoSqWmeddNp+Ag/yKzKI6d5TbnCKxcp0Iko/kp5jBtYF/g0UHH4PHR7nmbcl9vYTechZkfN h8vvFd5yLJMPdhtfn+I0ifu2e2yHDaxOkjd/rhFUjoBzYivrnNmlzL8vmT/tn2V3OuRCX5vqvwAD AOENv7gKZW5kc3RyZWFtDWVuZG9iag02OSAwIG9iag08PCANL1R5cGUgL0ZvbnREZXNjcmlwdG9y IA0vQXNjZW50IDEwMTYgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQgLTI3MCANL0ZsYWdzIDMyIA0v Rm9udEJCb3ggWyAtMTc5IC0yNzAgMTE1MCAxMDE2IF0gDS9Gb250TmFtZSAvRnV0dXJhLUhlYXZ5 IA0vSXRhbGljQW5nbGUgMCANL1N0ZW1WIDAgDT4+IA1lbmRvYmoNNzAgMCBvYmoNPDwgDS9UeXBl IC9Gb250IA0vU3VidHlwZSAvVHJ1ZVR5cGUgDS9GaXJzdENoYXIgMzIgDS9MYXN0Q2hhciAxMjEg DS9XaWR0aHMgWyAyODkgMCAwIDAgMCAwIDAgMCAyNzEgMjcxIDAgMCAwIDMzMCAyODkgMCA1Nzgg NTc4IDU3OCAwIDU3OCAwIDAgMCANMCAwIDAgMCAwIDAgMCAwIDAgNjc2IDAgNjAxIDY0NiAwIDAg MCAwIDI1NCAwIDAgMCA4MzYgMCA3ODIgNTI4IA0wIDU1OCA1MjYgNDUxIDY2MCAwIDAgMCAwIDAg MCAwIDAgMCAwIDAgNTY1IDU2NSA0MjEgNTY1IDUwMSAzMDMgDTU2MiA1NDAgMjUzIDAgMCAyMzkg ODAyIDU0MCA1NDggNTY1IDU2NSAzNjcgNDEwIDI3NCA1MzUgNDg4IDAgNTQ5IA01MjAgXSANL0Vu Y29kaW5nIC9XaW5BbnNpRW5jb2RpbmcgDS9CYXNlRm9udCAvRnV0dXJhLUhlYXZ5IA0vRm9udERl c2NyaXB0b3IgNjkgMCBSIA0+PiANZW5kb2JqDTcxIDAgb2JqDTw8IA0vVHlwZSAvRm9udCANL1N1 YnR5cGUgL1RydWVUeXBlIA0vRmlyc3RDaGFyIDMyIA0vTGFzdENoYXIgMTUwIA0vV2lkdGhzIFsg MzA4IDAgMCAwIDAgMCAwIDAgMjg3IDI4NyAwIDAgMzA4IDIwOCAzMDggNTcwIDYxNiA2MTYgNjE2 IDYxNiA2MTYgDTYxNiA2MTYgMCA2MTYgNjE2IDMwOCAwIDYxNiAwIDYxNiA0ODggODAwIDY5OSA1 NjggNzE3IDY3OCA1MzQgNDc4IA04MjAgNzE0IDIzOSAwIDAgMzc1IDg5NiA4MzQgMCA1MDMgODYy IDU0MiA1NTkgNDcwIDY4NSA2MjQgMTA5MSAwIA0wIDAgMzE1IDAgMzE1IDAgNTAwIDAgNTc3IDU3 NyA0NzYgNTc3IDUxOSAyNzQgNTgwIDUyOCAyNjIgMjYyIDQ3MCANMjI3IDc2OCA1MjggNTYyIDU3 NyA1NzcgMzM1IDM5OSAyNDAgNTEyIDQ2MyA3NjAgNDk4IDQ5OCAwIDAgMCAwIA0wIDU5MiAwIDAg MCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAyODkgNDU3IDQ1NyAwIDUwMCBdIA0vRW5j b2RpbmcgL1dpbkFuc2lFbmNvZGluZyANL0Jhc2VGb250IC9GdXR1cmEtQm9vayANL0ZvbnREZXNj cmlwdG9yIDcyIDAgUiANPj4gDWVuZG9iag03MiAwIG9iag08PCANL1R5cGUgL0ZvbnREZXNjcmlw dG9yIA0vQXNjZW50IDEwMzUgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQgLTI3MCANL0ZsYWdzIDMy IA0vRm9udEJCb3ggWyAtMTc4IC0yNzAgMTE3MiAxMDM1IF0gDS9Gb250TmFtZSAvRnV0dXJhLUJv b2sgDS9JdGFsaWNBbmdsZSAwIA0vU3RlbVYgMCANPj4gDWVuZG9iag03MyAwIG9iag08PCANL1R5 cGUgL0V4dEdTdGF0ZSANL1NBIGZhbHNlIA0vU00gMC4wMiANL1RSIC9JZGVudGl0eSANPj4gDWVu ZG9iag0xIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1NyAwIFIgDS9SZXNvdXJjZXMg MiAwIFIgDS9Db250ZW50cyAzIDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIg NzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0yIDAgb2JqDTw8IA0v UHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUMiA2NiAwIFIgL1RUNCA3MCAwIFIg L1RUNiA3MSAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA3MyAwIFIgPj4gDS9Db2xvclNwYWNl IDw8IC9DczUgNjUgMCBSID4+IA0+PiANZW5kb2JqDTMgMCBvYmoNPDwgL0xlbmd0aCAxMzA3IC9G aWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJfFZNc9s2EL3rV+wR6Jg0AH7qmLh26k7r cSu1PTQ90BSlcEIRGpKyx6f+hybT39u3APThOOl4RiYB7L7dt28XvLwaM6pH0u5vrGeX7xaaNuNM U0uzJI/TkorM/ctSRVFhFA3NbD17u5xdLpcGRsv1DBtlrFJyP4rSJM5y0mUSlykttzMVK7euGIBf sLusZxE/GkPLp9mf4o1M47nYS53FWmykUXGCNxUbMcoizsTES7kgraSGX3FBJjwpJf9a/jhTBwBt 4jLJSrwsv3cYWpkjnE493H0lI10AYwPcTDQyMjATZJyvqMBWmiRnPnTBPvgp097Fr+yihFG/Cg/P 9AOeDLxiK0E6hxPIB28Amcep6LGYIisGAoXpkcJEUcI5ZHChcyqUYdoDf6o45VAGyna7ToJq0dbV QxsekQgwpmd6Lxa1RXK52DXv5QEtP6KBThTF/6JkeRkn55i+OPoIqnIP+jcyM7lHSoC0kDkKduXz XdxSO1JF22q3a/sN2TUtrha3UUKTpeXV/QX2PvEKTUPVjzs7TJ9D8SIdJ7pID3wrVZ7ynZ9BZ1DE 79BFJmyH7Aqx3zaXbzvONBE1eDbiI42THapNQ7Z30dHtvWRZ0V0zPdnh4whyfnpzhz3WkYxy8bOc Q4ERW99R5a1W9Ec4E+hDhHmWnEfqYjIp6yhYe9NKagTY2Y3dj5z6ZFeVU/GzjDLE/A8SKMVIiwBA 1VB/cJbthEghlXraD8145KZEN6FKyVzHICZP53FilD7W6ShOpRLP1b8yKpQG1FJGKUuTC1SIHQjS XLocsdYcTC6qTkal6MIJ2o9hGzplmjfYRX6E8ApxDbeuGSNO8EMz9E7UyWGF2p5PoGmnsFE5xBVH kQcnlv2nYi0ZnW48bgB9GBq6+uB9VH3fdAcKXsoVhYgzQ3mBspqTXNOTXJOjZkxagtHfuMwji9Lp bzfYyToCUhbSlzynKp6br/CcH3k2L3jO0QeofybY99oO9AinSIInGtYLsXV0GHH5ENYtJJCKmrXK 9LNemWgjGojz7mbhj0OLK7q6vVk4p+s2GPO4Yu5e2QalvuYLYxXt/SVfJ7qKF3Rl4p3k5q6m5ql6 5srMhROynVBzZlBysW45jJKLPjUDR4CEpBYWdaunV+I1c4j+K+JVp6rpM1KZnhs/zVqvCq+HiCV0 gW6CVu+roQornVSi6SLfXw8u+v0Yju3sxIdK5k3zBJ5a1iPPZz535uGZDkeR5/h1Lg20Z15NyvKo DH02rVjGb1ngwO6bNaMmwoFrtAyPTAqLgy+43WJYXV5PrrUmCkd7yb+nkwl3x8i5wk2w3PPI8I/N N8bV8cI9VD17UfWCC5qIvna1zOFGM0euaXZuOjU+nAFi3Fa9ax8WLIv0hc1qXzcrqi30eTbd9VkQ ySmI9EUQJXeSYqxq22ECurkxksNK0DUsuP7Ry60Qo7uFckiOeWSaB99v4LGzddWxaJ/alQuc22lo Ktp74kbJWggOendTbkA+BdceKpydsJ2K4CVAjN+4uf6nq645tRLDYNgwqdVj1XZ8dbe4r3FXfgI6 Loqc52k12b75TOOhsR7Z1OCqR0Ai3AzL704fNYcOOp97PCru3ffNwHMDk0LyqMXgs93oqrZtV6sO PS65w8Ohxs0a1Nevbppt008XiKTeDxwo2/1iFy8jKL7s4ZCz8Tlnora93bb40qxcQriEnK7WAypV ET93DU1u1vAXC9PR+9vaAV0vZ/8NALJaiogKZW5kc3RyZWFtDWVuZG9iag00IDAgb2JqDTw8IA0v VHlwZSAvUGFnZSANL1BhcmVudCA1NyAwIFIgDS9SZXNvdXJjZXMgNSAwIFIgDS9Db250ZW50cyA2 IDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsg MCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag01IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1Rl eHQgXSANL0ZvbnQgPDwgL1RUMiA2NiAwIFIgL1RUNCA3MCAwIFIgL1RUNiA3MSAwIFIgPj4gDS9F eHRHU3RhdGUgPDwgL0dTMSA3MyAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgNjUgMCBSID4+ IA0+PiANZW5kb2JqDTYgMCBvYmoNPDwgL0xlbmd0aCAxMDM2IC9GaWx0ZXIgL0ZsYXRlRGVjb2Rl ID4+IA1zdHJlYW0NCkiJnFXLbuQ2ELzrK/pIBZZM6u2js7tIHCwMYy0gh2wOsoYzVjwjCiJnvT7t R2w+ONWkPA8YGwTBAOJDVHdXsarn8p0tqbek/M/20eUv94o2NlI0UJRXadFQXfqhLCQldSZp1tE6 +rmNLts2w0ftOsKLJpUF+YekIk/LilSTp01B7S6SqfT7khPwAm/bPkp4mmXUPkd/iOu4SK/EPlZl qsQmzmSaYyXTTNi4TkvheKsSpGSsEFdcULbMpIz/bH+L5GsClaVNXjZYtO99DiWzQzpVhHR3XZyo Gjk2yFsKHScZPhOU+1hJjVdFnp/EUDXH4FmpQohPHKLBR+NqmbzQr5hliIpXOeC8ngAerJDkKi3E iM0CqDgRKCwOFOaScsZQIoSqqJYZ077wJ+sjhmahbJq2MagWQ989DMsUQJDDvdBncd8bgKvEpD/H r9mqQzbQiUsJT1xZ1aT5ec48KwqPOWT7dh3nWS4mPKu05jHPMG55F+AGPyrRY6xRQbecczinsP/6 3vixBAu8ny88YFaVOSUYVV28ki7V1QG0XEB/A8CiQYKPcVJAHabvtmSdmbuNpq7vtbUX1JvRmu2w 6txgRurGFU0G63Hjk7U/LdHzN7Lw0Wvc2KcQXe+M07Qa7BPH5r0c18hKsRYMd5bWZqbuUMCejyiA LlHhsB3cy0K9DNjUEVumjtiqM2xVwFYEbFz9HOqwL3FSi7F/9JudfRlREvvlEfcM8e6GeY7hxFKY ZQRketDuWeuRSXGz2W71bE9L+h90n5T00MVoDsJTcyWeYoUz+2k5wazoc8rLI+XZaQbxIU7Y8l/M du8v7Xlwj3T/7v6GnCG7nyYzO9I7PW8Yk3lA2kb8pXvnuQcv3m6VN1vCFtS0Myu9DX5+qy8PUp3q G5bN+H5bGAlAJ4NLzKBv1IWr2AyeY3hY8e3/m2pl+aa7BV2JO68LM4yeNOfbWxKzfcLchMVkOFkG j0h/rMIcIpwZ7eLpMYTATjhqxh/I7KRvnNWCb94H8egV+geSYIJOixYbJ5UIgg69kT5e33pP+UIs eBKDdUNYbpiQUpBZE1dW+KJhHIIvdmHH3w7yAFshEMxjvCWrNzs9uh+K8fgfodSZGJW4j/m/4LGb 9YqLQ+0XMTcZ8jqsvBBqztvwIxMvrJkO/vgKbEu1bu5WA0sNgg4xaNrug/FsaN8HW+Psej30Z1KW V28a8sKsNwtr6Po2ccZfT8KaLMXvnAeVkf7q9GgHplTQ3nJ5NzGL7w50CprN3sGmzGHfzZ6/nDto ECC3n5m+30BNkEXHzaYRnfPB9N//0W93x6vxzRtFfulg6BHdwsxPFhb2Yl2Re/QTTZO31z50wYft 0NNNOINaQ5yw1M7X8KGN/hkAr5UKswplbmRzdHJlYW0NZW5kb2JqDTcgMCBvYmoNPDwgDS9UeXBl IC9QYWdlIA0vUGFyZW50IDU3IDAgUiANL1Jlc291cmNlcyA4IDAgUiANL0NvbnRlbnRzIDkgMCBS IA0vUm90YXRlIC05MCANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAg NjEyIDc5MiBdIA0+PiANZW5kb2JqDTggMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvVGV4dCBd IA0vRm9udCA8PCAvRjIgNDYgMCBSIC9UVDIgNjYgMCBSIC9UVDQgNzAgMCBSIC9UVDggNDcgMCBS IC9UVDEwIDQ4IDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDczIDAgUiA+PiANL0NvbG9yU3Bh Y2UgPDwgL0NzNSA2NSAwIFIgPj4gDT4+IA1lbmRvYmoNOSAwIG9iag08PCAvTGVuZ3RoIDgwMjIg L0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImUV8uOXLcR3fdX3GW3kLliFd9LW3Hs GEgQZHoneCGNx7KEkRRIYxj++9ST5M3YhgIBmltNsljPU4fPX3zO293nDeTf57vT829vYXvz+QTb 2+0Uy57aVrP8ySlsNxXD9un+9NPp6+vp+fWKdOj604kW2h7SJv+FLcU9lw1a3Fvaru9PYQ/ye+AL WKDV693phj8Rt+uvp5fnry5p7+dfLpB3OL+5YNgjSWHH8+dL3fP5kX8q5w3CBUjv+S8b2lcIlx+u 35+CXwC4t5gbCde/yh0QcFwHSa/716vLDVS64w3dm8/3lxukY+ctia6bSkspxkUHVNbBXxlUxb9Z RaNDH360j9+27+gLSSstRXLHd5A/JNElfU/nD/RjIq/4om8oipyBF7eb2n/7giL/PUX13RY46L+S w9s/tpc/hO3HU6plx0a/xD2l7f0pNdgDy4XjnWFIhbIV8h4pPeQWHbk7pd4lKbHutWwpZ9GkClKC vdNaFz13p9vTKAcMnTemlnakG5GUAx2n1GLaHkgmU0hE/pXFsDfeHTjzDyfobafdldwWSfQnCq6t 5s1cYoFySWLai0jIe1KlPyZGPinqVab7U+lsvsoUrFSq2sHKkMTMp1hMYnWJHBm5eK+8Dq6ushcl cFhZlFCl3A7LuXJA52kKYV215+TGyOUZ3TG1LUs+p+2p8SXTt1RcnbqeEudkBIZ6ytRlyZfp0ojG bmGScMfmydFsUBe3mauYd02VZjKmXfdqnmPUGGBofGukxOIiBtPU5Sx2U9WlprCpKpDsJ6y7HAXU zcVUUdN2lrPVDHU8W4nJbgZBkYSRo0wyQwGKrF6hxof/iDGoFT1kMpI9GfvJh9IWfeSxnPf7YtHz bg+FL6XFXg5um+6koOabs9w6OCND4ho2TlRaopqSyhb0xDpHRlKXuvJ0ZVEys8n1lUauMWsVeikg gTQb4nWCVK5stpcRUjWvZcYyl6WXIct8nZcpn89LGbP+vFQ53T9bAHOUHHmLYAbJgXeQr3uHyWHv Ptfs3ek3e/e6Zd7cbrk3v3vm4OCeG3Z4YAxZPG6OOx5WByUPu2GWJ8URzVI2AM8SOvDQEi5wOcqh ZXZoFksrrsx2N9emtdW6eW6l14PlyEqzg+mz0lWEmaXdo6XBSr+Li7M1upWfdQ6JmkWUiLNsnaWd Nte1E/24NerQbo08brdGH9YZEAzrDSeGdwYjw3tDmREdxaARPBM9tuugeji9fmYMJQ2GEsMWecRm taMGsYvpyUpNyBXnCjHroH97++L279vtR2IK8fzwy4Xq9Pz49uOH7SqfHy/MVf5jyx/f/OaTfQzR FKQdMg9dntuxi0s5R7E10kBJJIHGMXZBssydqctEfEg0R2OTjs0JXa5bjl2rOVI46GjMu0pSD5nK IKLJFNqMTVqBRbIno+WbZeZ5CBpwVoYUrqr5iU0mRgZDALqYWyNDMG3cMTlUlTjLtDkkTb6vgoty NvWmuTbdo/T0ai6VNA1LdKi0aTh3XS/TL27SWKbf3OMljaikGt2vKuhhupqiT7EYSbQZjEyUZAzk 0lwx0HVdbjr7s95kmU6U2SIij4WUQSNmYmqqizfzYcqsKGOZG4Dps8rGBECnSQpKNGJ3dVkajChd UH0KfDZPEiUm6LABlaUMeBiJawk0SgwSIlYJYrR2TaDt6MtKJOdpqhrGlqEdi2qzy6nG6mpbDDr5 3PYIOp7cN7qmr77ryJyxIVnGm8cuxkNkbbvHnR8uaUkL98fMGt9al3HEQ1ImjPtGQ5Ij5zWRklar 1wxbISPFCiolxWmvNx6iaz3aUB31yjLf5/XM5/Msd5vJoxno+tkpHBEyanSSDfzRaGNd+5DPeoea 3tHAdu1ocDNr9L+ZPfDB3HL4MK8HulhUHHwsaAOaLKgDtyzoA9csKQP2NGcDFS2ljpmWcINUr4es o3+US+YML7spwaZMqi0TxbBqkmrMxUiAVWsu3dRpNecK6prWeqb8a060FzJRjDZbJVf3TDspM+VI s9NYtk5TmJ7rEtHlvDTyUK9tPm9XGBjGKUpM2xVFpm+KMtN3RaEZGwGpETmVRlwPk2wZs22MWeBH J83TBKD5FQ64jlkZrrnynOWv2nXMfnu5of48v73cEAac37x6fbmh+nT5UVe3b+gD+Ovn+08fdMf9 o09cjjSZV5Kk7/2TyN+SEe82eSdRLEoSqMtclNune1olXxYPELkfS5VSEgfYYjb2/N3l+u4w4ok7 bQWEmIKgBFENUvn1M9PZOUXyH+mNEpmSJBFPIgOd7zn//PrVuIRsNm5nd7w/sRIe3j1NdmUlLOtQ ZGPuxu/N9dgELkoR3rq4HsiWvhFQkFJKOZWCEJfOlWjxBH55/IlfCaSkChP7g1svzx8oi9RC5zvP VKxK4ruGjN93f2JH5LeBAA9vJvPNBomMLUawwChjt0A8PM0quWEEH5/SP+oEK8tQtCz/9oJYXjtv t1/9041fiyjG8gVFFEv9wiIyrvOlRcRg8H8XEQOOFRHKpMHIIXsgn9AfiU2LyB6R+pT6g+yEJQv0 yx/kJ9SZnPGcYv4keMKoTsSYHmO0XDDozLh9wqGxw4FDY+sLh8ZWDxwaKZgLh8YGBw6NtU0OjQS3 C4dGg1PjkkjttHJoLGnl0FjCgUPzG3Th0Ej+rRwaaZKuHBop4JNDY4orh+ZX4cKhMdYDh+acrhya H3kLh+ZH3sKhkVK9cmjuhpVDI5QDh2ZYWTi0vcksYjR4Vg6NAVcODb0fODT0snJo6PHAoYG8XTk0 kLeTQwNnti1iOkwvoMyuHBpqP3BoqPXAoYGSvXJoqPHAoYGyv3BoqOHAoaH0A4dmsF04tGHvmPxj WYnBPK28YWpXXjEuV9YxbVNWMm1X1jJ9U1YzfVfWM2OjpGjGTjjTjKxv7yuHnmkRPjayZnRtJNXo 3Ei60b1RE0YHR80YWxwFZWxy1JuRzVGPRkZHvRpZHfVsZNbL3bjuaAbmwaNTjCOPTjIOPRrN160P 5ax1qOv1BvZrvcHdLO9/N9vxwd0y+HCvHV08Ks7lLGgOTR5Uxy0L+sA1S8qAPc3ZQEVLqWOmJdwg 1esBKcMrh0bO8Lq7Hzh05IfIwqFjiAcOHSnhK4eO3GCz1iMzjIVDRwgrh44ABw4dqTxWDs3yyqGX dam75bw08lCvbT5vVxgYxilKTNsVRaZvijLTd0WhGRsBqRE5k+KBQ4+wv372P5TBmDNnsbUAv8Od cXDntHDnJFzZmPP57aP+5ow5OWM+31/g/HjgMsbGuEG+mI1xCf8OG7M0pygN8F5KkokegNOxhS5Q xx0V2LAXTdDkgYOhsEIiI01bxxGEcY+NDtpKQM+jzstRgg4VeRtSsJvK7AvJ+q5hDGa0akZ8lOhA t06BUgQqu71zWE46s5LLPAe6lRfLRWfW0IeKrV3lJoOBRzTa9csUU8SHhlohjPhFgVtQwZarvejs cDXMcN30yMnr3TWZ52YbBaTjYnsFt0V986nkvpdqkdLQ0Hux4oxcsTlgcYVijWdxh6KNZlmBbI1m SYPcLNCaVMhFYd2SDtk6z2ICOVnSXY6mj98zZHxG1xf1PiOYHGJ2hnC2qCzzBAhnTZ+SDwJaVSfj BZQAKyFm9akonAM/t0jM6pyWC4uaRur/XESWSHLx9TLXuVTZeTtulT2UY2hqq12OIE64bTYah+kI 6qq7hvT8YmvcdZbLEhpkPz1uKiTXlLl0RtARmsqWFH3DjJzxzGU7PadIUI4j40gAwEjmBYEo1er1 Qu+WWUv8iPkv5dWyW7eRRDGPlb+Cy6YB2ewHyebSUaJJZmF4IAFZOJsb+TpyIEsz8p0o/o354jmn qrt59WAFAQxZVLGb9ThV5xRDKFDjOnKMxLKeNKTyOaxA5ulpxTm5nvHWPuCHw9omVAYMqnYRli1t Sf/QXpuwHK8tWm+vHVw+Xhu8uFb7v3pe50ONrM6PGnidL5KVOnxKxupsKgmtk0uz3eZaKUYde1qq OhNLIdvILIUuwFMUEHVDbg+LKo0CHkJwWrEVvOqSAj38t+IygIGHI9iWIw3WfNaP6tep7FJeu4LP GpF2TQBnhbWp+KjjSdO0mtmR7XBp2HZ5aej28dLvzbkyD5rzZV5oZGWY1KjLrKk5KUhpKWvPuV51 zF/P8DzRoJVEnY9JnswOJwvHD5Ny/NlpDwXhuvM3bx+Qd+mHUbI8Jv4T7n78sZEZWWTcysd4PW92 3/cXv/K2lX6pY4DAoZDgKyqlwtF6qyRVfgyCLryF/ghPpYpf+B139fOufQQuM2ccVVFK+fl4Emoq QUFsH20nUL+q7tZOVT1Eng2iWgv+z9coKEhxO9UXgoES4WoAEYe+KoI0weMIOOkCgc7hY+nvhNaR t+uy5GVKc5fV/QSICbSXaUvBiTQBXipHOR1iLqUvii7mVNWqx0OoN3uRsvlox4TD1GTFE9lOIuaa rhx+VHsqYtPL/hfnJn2FEOI81PvVPuXqjJ6fpuq53j+lstmV709lEFb/MN/nUJ0f8+o8Lx+L1NW4 x7SqcNw0Bt03SsrGFqimFMUtb2vKU10ptSLYceZprReWMl2VSjmTrjJS65iKvAcYgZcYq7yvz3Mz I9QY624ZVc7HmoGy2nDj0fdln4oc4p2uUVxVoi9rVJblgA2gq6dI3RiWB0taDLncnub6fLwy8lnD HuOxcTo6iayzsu3mSTaJ9ctT1DCLY6DZafVaZeEa1CSafw1adN2aEzxKkCVl0/Iwo5OMiJZwVX2l GFhXl+NKzVmfayFBufJyrbOO2RUHWXTIipOsla4wykJqK8gwJY4hyO02HSE0697Z4JtFxqzwzqJy Vvjj60Rlaw9496B9sly0tlext/Yr51t7lvtb+5bvt/Yu/rX2L/638VDCa7OjhN9mS8lOnTwleW0u leS2uVWS3+ZaKU6be6V4bSyWyrapqYUvU7UBg0ut3q+4SZjHzY59koIkrahLQ82GojINcwEpIZtA odOK5zQsBYMC9+SHcpe2Q1st2St8WKbSRRw/ikBtMT5r0NqCq11bdD38aGktLb5+WSdAc0znQ3Nb x0eLSafLGrJOnzUlOpzWjNXnmtEHPPaMjkijiBeWLD/REZMnAQsXx0WFxI9veox87972kMau+6E/ Qe3dOxEVvJY8/q0egZaoMuG9+3Z36E8AC7frAS6HB8+n25t91SMc0ZwSzJNQ7qNRKKLk9Rn87i4+ HodAQgfo0yIFr/LE/YWi4fXFRW4HjoVHPYROmfXQq2kcO/QlIkWTQjiVJMTYkpA0lu7stHvXi/s9 5Iu7O3SXtz01pruRPxz65O7Udn29V8udxIkEDRnSg4kCyGqqVOmUVP3kmNTgftkd9ve7rz/1NUOr OhmCRhyYIYHnyShKEjrr4wtWn6AxzGyFTTPHsmUF7LfMEeovb3tGc9z2rAwKy2zdLeuUZU7WaZm0 ljlZnllHZW+yzIPll3SFZR4sv6h1Lau3PJOxb5m9dblwhWW2Pi08aZkN/EdKf8OzvFj4VzK0rBb+ QagW/mG28A9etvCfJwv/eTTxD7OFf+6ulmfJxL8Zc7Tw/3QnZH0IrCzD4AknpTaOZ68D8x8/9wGD 233Xnfdgj9ndY/TO7tMBM9q7y6u28QIVHO4TsIMZh67jndwpScao6jQLVf/R0solCKmcB+HijbW1 DPMPv5H2ljL+nyUZyIEw8a/jnFdKCLzkvTs/BVMmMOb5Dyekhegu+hPoHHdaObd8GsQ/+OA7uJbp wbeSrTy3bC3piMHdW+Yquj0TNrm73z71PrnLHmzolJFPsEBFchSoaprTvDomnkmgo954uvsMJzEs 3L//+6X7rmey3O99JM/vb7586k8gE8GVOE6G5O3I4OTT2DEiuP2YCPlbKO7qdXLJ5D7su12PT3kk VEhyvzuSHgMAxsD15P53Tf2BMS4OnuhzuepW71F/TiLFwyTR+jys0QKIxaGhKKDvb7+okjkwoMWB +yV8BH/Y885E33x23b18KbrDlf5ZUxPpEJx1V/s7lnJx52/07281NRhtHn5AqY8plHhKDelY84Ee EO3dZS/R4CY6wk5QV9SRiV+mKxPrccU/Q8DUnvjPizB7wH3yoq299MEi2hjw//Fld7PKjsC9IHez FwGsndxe/SjmtG1FV6NbLHOyTmf2nmVOhmeW1xjJo+XXTHreNk+kZ8s8WN/mxmJZveWZ7ACW2VuX R9KzZbY+HUjPljlYnok8tczB8ozy1LJGw7NZ5KlljoZnczbxP2cL//Ns4n+eTfzPk4n/eTLxb6V7 Hk38z6OJ/zmZ+J+Tif85Wvifo4n/OZj4n4OF/389w+usgfSECBbwujJAbAww6fR9uzvIZPdOyF1Y lMTZfdFHIQaM5Dtl6t0v1b4rf6mGnq99Xdc7KBFqPFRrHEbi+LOseiKupNkTooEyiklkojzEeeyw EervoFXqYj0TRmqBjjfiaEg4hDfPRf/i+Agy+YwH0Lv8fi3aNuB3dK88ybniyfWLjy/lBZQzJIGy Hk58jvL+ot+Rqul5pJHfRVLl/LrARhGDCTyIftPiYOIMpTiJCnvT5kVOGmauU5vmQdYpwzwYfoXl lXEYVm94FrKWcNvsrcsFm5bZ+vQk65RhDpZno6xThjlYnhF3ltUqdoiCfsMcLc+C9sm22brbyzpl mJN1epB1yjBb+A/Dtg0SzcI/zBb+KSsNv3w28a/jx7Ba+PeTiX+YLfxzY7A8Gy38Y4Oy8A+zhX/w gYH/5/bDhcAaZh57sh6G1FacouYvqOWx52Fwunei4D23ipub/SWJBlPF3WJ5aVvs4x3y/FS3WExn pChGjgLZYceFnQkvz+nl64sLP2D+Xnx87PDCFo2L8Cn9fW5L3diGZD2b3NXu8D9ddr70iQvHzf6+ OSzhw8lGcMo+LAk4ZJEWHwfmLC4csKOXoVAfFhYHC64+YmAEOqunMHJDVzkzynpCimsZJymhUKOX fntMd0slOHTy9bFnU+U76bURy2HIK9+NUf5TwsSMGDm6K+WtjsgVzPtZaFnHa1QjXUq6SgWBk2CE iWVO3V/7i19ZrFxOvUIVuAa6jobHWCsXIYhl+oPiLXWP9G2X/eaN1q/b3XzouMUCax/uuCsGJxpn gsbBIuGwq7Kcz8USk+4pR7GU7B8F9bc/E1S90Q5KYBhqTA2Rqs6gPeA5w9j3DAra626nUusE1Qvo PfwfkqcTxzpPL9kdDrtLBD67KxFpn5EbbC24jE83qucO2ylZRkJrmkWPPUmJ+/ufyka5bIKyyYO3 ixxah/qSj3dSzsXdHm4vb69Z7ijT5VD/TrU6O7EkV16AYkWW3N3aw2jd4IVPxyTNFCfpwNTmy1H8 IcKii8WT+Qd+aag4O2W8J9zoWA2mQFwf+cp7d37fY/QCiJzAlNWenl5ePVDMRe36WdgHTTom5s0P kVR2Dd+a7PRDIsdMmCEIQCf5iJ1OBzlPYH4a1rBtDJxmhhUw2LR6Dg3DGgynmDrDGDedwjTNmz7B GDddwhaVNz8K4/atsjZsG9P2yYkl3TambX+2jwk8t43Dtjey824bh21vuO9u2/y2P8LN28ZtVC/e APXitzG9DAakYdxGNFhzG9B5MfCc8zacYdtGc54NNMO4jWYl/W3j9q2jgeY8Gmh+qhgXr6AJvPIB XXCk58ZxcdSZfnba7YSdQGry/9d1JJZZJxOb0hUgx2iEM7IrjdQ/8OKbly/SyP0nyxvQ/RzpC3+q VZ3MbAP5AamBCFBXD56fHjvp/RF5nvbQ8hnEAh23uJtDP4JR7noq+PrH654sc91TzVVtIdo2pYh8 hWkmu+Wibf/ZcdLH7h4RzaJ3fIgY9eUBISQZ9EDsREJE3vEz/5/2KtiR28ihp81hvqKP0gLjqFhS qXRNjADxYb0LD5BLLoOGEyfoSRxnsIv8/b5HsmT1TFNpI/Fp5jVVLBbrFfmIqDURq/x8dWBWtHvR VRnVFcokkp+Qfz/6mkPh4wmyl7P6uT592Rrm8/R93oTlsWo7lOzt0bDF/elJy6NYd32eNIp7MLhU 7b1ZN3NsA1YWKq4WSuaUtCJIdTTnRRqGmB70n/Y1+iCKbvPkyPbhWsULCe3rcMBCoE4N4Dvb0yC2 sID8U4vVfJyd43jz7uY8WpwMw27ZnKVoOjw6R2t0hs2zrfMtt+EcVxW1HCgHM+cQ+tRS8rQo4L5V Ir36qp+711Rr0r28XATIYnJ30WJ0kcis5QN5eS2RUU4xZtbLZcAJPZ8RWv5OQrMxQYU2PhNq+Kd1 9mvUbbTFhWpBHkj7hxVzklNiuj/eSPNtlGTzVEhGqv71L0E59iP34Mj8GyGJB86sbd2gHbU5NWSc lAaVk+uXHqQ5OTtBo+QmDyCWrMEh0/Xj8RytwTk2v77Ot9wGc4mRmjRpwv7PiBjUlp+tjvDabPdL xTcNOgROsyXvKW3TyBFjuYqxaRpUAdg73OPsZyrCqYi+PQRbcFcOGX7I2IRXhtNTbHOFIt4Ykpby qAJLvaWsBdkAssZ8jA5PgEWFo32Ks1BmuBdDtgMXGmRHtEUITwuweXSED21Dx9jComlfW9j0sjkA 6XoWKE+UVGJbZEIqtsAUrHEZUoe6Yt1oG8ZTqiZRZYt5cbmOqeclE/INO5RJKfqMeaISP81J+/k1 /KNWGA+C8w3jkD69as5/nYF4b+iQkrQiPDTsh4hZmKiA06Ij6oNDGdaujrXuMkFqDis6chBEkXB4 4pA2rxDTFR69uzHgWxxvHNaBvm0Vg6RWVo8OSFjd0DGpruE49sDNzdkplItnwfJcKgTaUVgY1/gc rasbbq5N07RdtzE9IyWaArOX4chZ8Ge0HKEvqDJnfTOjqHDDWxhFCVf19sacSUgDx5txZNlttnHm +1j0YA4oTEQ/NGgv35eR3SCG+3SEb8Xuxa1p0XMmvpMG4FVv3nHVVJzWIzj8/cL4w9NkE3cqjXXW SYvl5a6/nV5M3f37t5qUL79BxIe7H+hD5SEb0+IlgR2jvS8s7v7R3/188+XdXfUlL3gNdy9vugMN T6JoXtAml/V+yjTxMWR2sXkofLEWnQ08+A+n0qnnX/ePP/W3EEXdf3s+1bc9CNIdHlvgt7gCSTOi BmESozBHeTM53T8+3h91oHuH1aV76G/Bh1l9le6XPqFpdo+NHBltBDeGR8/sQQJgoMOdSdHJNGMl X2DRGSkPPNVBpsUu6c1TkZiVC8Ih5UK3Fe6DEnNNtZMqXC11+LRu+7fVOjGlyzhGNs+GR1USYa0T PAemz/5g2ajtCMfW40zZrCrsZCpaIgxhUiy+hVln1U8yaftpqGg5wbeOZ31KbWUdX6Sy+jV0ZAyr CeGjK3NZqSuC00UDcDyKzXXrYRxfenYyaQulR7uovXeHzP+mIpN1EynhOivRJMJ3/zz88pFReRx1 vMls3QN5dWuf/aAmHOayKfNkkUkCi+rIyCRRFFrMI5NEUeAphZYcRIHfahBF1rJ32VT11QamyN+s Dz8wjdGqQhZEpjGKIloykXCRaYhi0LcQmYYoBtIwsqQoCgm5CVPETWjPiJs5RdzMQ8hNmCJuohlG 3IQp4qbUiJuwRNy0JxyZoiuWEnJTSsRNmUJuwhRxE7U/4qaMITfDzOaQm5JDboqE3BQJuSkp4iYs ETchPSJuwhRxMy0hN2EKtkInj7gJU5RB1P+ImzAF3PwPe89F3YYHx4201q2i7YtPEm3mAmSYJVRs bGuzymztcNUV2xvoKQiRuXvzbY/0Tt3h37/2Qr31oYeiaUprK9ymzEjoZMqrv2Lujr/2SAFVGnXL Y1+6D/bL6fRW/344fN/9qP/df/zGTP+jcpza73/08Dt33/er0oN6owYpxeawjDmD2mFelGRJwSL6 zpN2UVlmmwYEAxXln4mYpBpNoHQ5SmFFNVTrdlrZagRhz5OKT86knB58zE0EZ8/B6x6rCw52iygg YhcVr7e4I8tp6aC6IGohNpl1Hm/QxA54y6skXjaS+D2U321Bin6y9e8pglP3rv18f7Lff2+5eqJr oTeQlTQWsvO5rq0UFmkSHRav0bYLc50qv2qTRqBu58+jbvOQbB4VEsGBHiBWtrM25TRDEFHYOqza JGXOOrqqUtUGmExRTjOLEQHk5DTptMuPplGTZYsNqGMuUVRUQbUlkyrQ5s0QRbBtNTW96lH41x6h e9qGf7x5d7OJ80HldPb4C5tgC81AC81Qc2hL2mbbUDbP4MDCw/lJk4V7sgKDBz/bNb/6CsXjdY9B S7qXH9/qrFcyJx4IV1RULONWT9upKw0qhMeZ46AXzIkxaslMg77iHavERlXEO1b8HFpVF+9YZSco VoodYw6DWlQix8YchrRo64iNsVeVyrFxjFeqXI6NYxxPvEwJHhuHOBoVzrFxiKMhLWNbiuNRAR0b Y1YvaYfUS4o5vQw7lF6GHUbXZYfQddnhc60xnWvdYXOdd9hc5x0217LD5lpiNtdph8112mHz846P eyBpiorKJ/1tQLqa6smTNehvvj7c9yNF0wf7+8flTsyCR5Kz98vzTpzGzPNBh8t1rTiNdrWFLfNy I3YJ8Tm7MTsK76yolHpYsZ0j7MgpT3x9OHNij3A4WQ/M1hnMZcpWsA2hbySjl+ETsN29f510Kmqe HNk2XGvYXLaVolfR/BrCt76vYezjUfnXFrA72h6G/fk8Zh5P76jFKMsaIP9doyNoHm1B22wbytMG nTg0weFEL9c06P8PAO8NFE4KZW5kc3RyZWFtDWVuZG9iag0xMCAwIG9iag08PCANL1R5cGUgL1Bh Z2UgDS9QYXJlbnQgNTcgMCBSIA0vUmVzb3VyY2VzIDExIDAgUiANL0NvbnRlbnRzIDEyIDAgUiAN L1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYx MiA3OTIgXSANPj4gDWVuZG9iag0xMSAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0g DS9Gb250IDw8IC9UVDIgNjYgMCBSIC9UVDQgNzAgMCBSIC9UVDggNDcgMCBSID4+IA0vUGF0dGVy biA8PCAvUDEgMTUgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgNzMgMCBSID4+IA0vQ29sb3JT cGFjZSA8PCAvQ3M1IDY1IDAgUiA+PiANPj4gDWVuZG9iag0xMiAwIG9iag08PCAvTGVuZ3RoIDQ5 MTEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSIl8V01v3DgSvetX8CgF2wpZ/JB0 nHgGOxsgwSQtYA6ZHIJO23C2207iDozZv7H7g+cVSanbs66CAbdKjywWyVevSi+vHqLZPRiX/x52 zct/bp25eWicuTWNT30YzRDzTwzWbAay5vu+uW5ezc3LeSZMmq8bAGNvg8n/rAm+j8m40fdjMPOx sb3N7y0vwAbQedds+JHIzI/Nh/anLvRT+6NzsXftTUe297BsT+1DN/SxPfGr1BpnOwe/7T8M1Sdr u4/z68YuCzjqRx9HGPPPeQ1naV3OhbLcb5+6jRuwxg3Wje2+2xCmtSZmX5sBUPD+wocb2Ac/RVdc vGcXIybdfa4Pf5pf8UTwCshjO8sI7AcWFpn60N7hZcCueKFfcIp8A1dbU+LfXuHkX+NUvxjLh/6I DZs35sNHaz43frT9kEyKth9Hc2wCLgYmTX0K5tBscQZfcGlh6u24jEq+98lEh4G4t23zLeMMh96R odRHMmHgocB/f2HumpULfoh8gwPOz/LgzTruOmNwLGCBn0VMdIkzIxkL4jycrhgKZQoLmAQ4PhUR s2IglokvYlYKJOEGZchJoaSRL17EnOhz6JMYytCLy6VePDFgJIYSmbUiRmIoYJEMeTEUL5O2ZIOA kUxaYKJLJ5MWmEjaZGXSJiuTNlkBiJNMWmAiaeMokxaYSNo4iKQFJJKWNUsMJcmkjVEmLTBxuSCT FphI2uhl0gITSRtJJC0gkbQQaJG0wETSFpEXMcklSoBE2ndV/1GJtEqxrR3A+GwHwAmaF3B82WgB cvH1U66hpXy+6UYUd66QXPjvloebfX04Znzf2VIw8eZU6nwfrDcb17spLNXZRnb8of3vT1xiU/v1 K6ak9pBd3HYbCFf76S6/2+2XbqH3yD8INjoQ1DrQxLta8C17a//XzV/OXQXqeZwGQ/3g0sW6deFf f4NzblY2b+elsCNQRIgFRv6fMBRHamtx96js50pNnIGo5+gdoBfVPqxnHNYz9jhaDihiZMkUFora YuHklzbHx3LIt9ur7b/M9h5Njm8PPzqPwznd3t+ZOT/ed9xmfa3w/c2fS+zfGhoS69MaG9sUcleB hQnlGJxIoEMAM7KCpBA51XYNQS15WyEwGQgbT+gI8YZC8Rly1BjoJ06kBfXU0zqLsrOzT8plty6I zg3sWmJxl5FiaLUnYsIemrpqPavDurMFf9rzMFq0l39KdkCOsfWcHTQkBYtVgUW4iLAAh6rDImy1 uHyvTfZVkAWYqiaLsNOcu6rMIqwtbas+izApkaE70i4kTVWoBXjsddQrkaFRGrXIhiraApyqbouw 5jtW9RbhoM0OF+nwLKzxX+MJuiuN/4A1/idS+V9ERYadxn+gGv/xrO7LqvxHc6XxH7CyNDosjf+A Nf6j0dL4D1jjP9otHdX4H6PK/xhV/seg8j8Gjf/Rq/yPXuV/JJX/kVT+awSPTuV/dCr/o1X5H63K /zBp/A+Tyv8wqvwPo8r/MKj8D4PG/5BU/oek8j9Elf8hqvwPQeN/CCr/g1f5H7zK/0Aq/wNp/A9O 5X9wKv+DVfkfrMr/YGXMTyr//aTy348q//2o8t8PGv/9oPLfJ5X/Pqn891Hlv48a/31Q+e+Dyn/v Vf57r/Lfk8Z/Tyr/vVP5753Kf29V/nur8Z8mlf+ANf7TqPIfsMZ/7TxpUPkPWOM/JZX/gDX+43tJ mxxV/lNQ+Q9Y4z95lf+AtaVJ5T9gjf/4KtT4D1i9L6vxH6jGfzep/Aes8d+NKv8Ba74Hlf+ANf67 pPH/XWMN/22vGmdem1wMHg1/9/KsFHOTfsw2pqSYW+P6fZlQ6oPhDyeMSVybyeyWzj2hnoL+aAzx 6PkNF3f2GXIku1wgaFxR6A+tsyg7O/uk3EvVBct5LLG4y0h3y37TRJych6auWvd/WHe24A/Nq7l5 Oc+jcWa+xmlsRl4q/7OG7KXizsdmg7bMRjPvGjw4xD0/Nh/a9/tjt8EHYnvfbfDp0Z7wA7K1+2Ka Hb9P7d0DfrBue3/oNmiEgH+cX/OaGDUFM//MXm1I2X9x/dtVB1e+/aPtQLH2oXPU/tHleb/MzdMz PTYBXTxMSCIS/NBs4eWLaZzLVceX4wPRyXClGTMHtrh4jOG8HU1Mubgdq+nxDTdQdlROidZTAhuI T8ih1wS1A7fwgU+IdwAEO2j7vu/mLxynM/z3sGtCyqqY8L1gFzpGV7gYUpZEEWNWPY+xGIoQM/V5 LMugiDnRZxZAEROXy9InYiSGkkVPxEgMheVOhLwUSsxCJ2JeCiVmiRMx0WVNZgEL4rya9s9gf0/o yUwgKgexXrd1mapZ+h5yLlu3ZLW1Q0m97e1p3zmk79v96fH++7+XpMs5FVi9jCtcQOOFYEosOaVe zTmrMYDy0o6JTTYL3qoidJHm7+9/nFhCAmvG1H5f1sq5yZuFIy4TQ6pZzqyeUtG0muaE88IOl6PJ XyRYOXFINEa+ViZaMlwxjJ+qABDYgFgqlNshP5VtoBxAlgCP/B9Hx3XCok5wWD4XijI5Wu7guE7k EItZAnuuuhStKR+SxyXuavMsHgBWxPJNdswNzRgWmzsrNiITbjGgvrlQFGviVFtmxZwki89i8VA6 Q5aJtk6r1uKymnU5OodxGeSuuX5RNxbGXB7XjVX7YmNhmMpN1hCrXaIPozvvkudCGZadBVwjF706 bUwl4OK0WGVnK5SDX6dVa/VZ7bognQO5DPNia+uduWlgj+ud/W3nFb7Y+TdU5pym0eZKDYNFdKgp 8/sLc3euErUZiynXsdq0LEPXZkyDmfoinJVKgzXfWa80OGizc2pqcNAi06bGnIcKbLW4At+JBlst LoigijotMsrJr8Dqth3TSIO1pS3niAaTHNm7KvKlujyN80kbxGLvwMdSU0733z+hqPRTe7M3bzt0 m9wm5henxw4dSXtRaWoXNnJrvCZVMS/UxEN0x3BWk2oXxfCj7c/PVKRqMQNvcJnFFWLJ/GIUKVmA rBXrnGqtDqtdFqNzDJcRZiGpxWuVCk/uUicYpnBWSC7brCTF5g5koEWxVqvq2WpntVtmFiVc/C4i yQ2rpzNK2f0ys1qr32rXVcvMJaLLeC93uGol7/CiuNUZa3GrcVa77qGWm9UqtWg1yz7rxFLEFrdL fathrmjewjqzWovbasallfd0jucy2ry/b40bhtwk4psNqsKfOvhEYgL8n5K7Mcf45PNzHXoNOB+L BpOC5rvTYEQuw/Hi0+5ZmLTIQq+jXovM8/FpsNciI74mDdZ8O05IDQ7abMv1S4ODFpmVMeRPVOIC bJW4oC9RiQuwleN6quS81Pl4q5LHVckt7jUr+VUH6rb7u1OHqp3QsUM6Wkh7H9vDbYeKSe1/9p/5 YWrNm5vycDw9lfaAL4ciAlRkpNiHxXZTzt7D8rmxmNxP8RBu8YZcxXISOtSR8Ew/FfzA80rlKPs/ D8XHlE+9jnpS4Fo+ZNgnBQ582hqs+a7FSoaDNpuYzBoctMi0hR2TWYOtFpdlMmuwVeIidCAq6pTI aOT2RYOd5jyLuQbLSz9NQl5pWFaqOejP3VR42k1R6abefLorz59u9uXhWH/vTqb7i/UqWXLiCKJ3 fUUfuwlLdG29HFnCBw4YeyaCg7kMQpgBaQZLAoK/98ulelXXAOGYCUnZ2ZX1KiuXl2vwUJ+/ye/1 0f57QQNJ/qaIKVliqF252tAZTeX4LE4mSGV9VwoS4PFfAp+ji7ZlS+kFnFwh8ARAS4H41/2Xc7EG 3QLLW7coE7qZoc2UKehmh07GFFEhyyMWyXoQCfRyV/VVQOVICIdsUUoEvcAlgmJC2vSF6uAq6hiO htVYHPVSXMXdYknll1Zxl1hSUVJdVl1+HrgzLKnKBQyBO8KSqlzAwM+WNGYJRcVJs6AySwYDB9iC amkrT/e5pLJLKBwn04LKLqGwm2WNW0JhuBlcUI0TfIQpjkuhH5fqcYJTej+nYanOWTzfjBOa0xlJ OMjmvvlhaEMAGdxF081xri9LNNP5hLqiQEqpEUzL6rBJLQ6c7otqT4GVUpuUcS4qKXVqa8ukIqG2 KWSGLjeltilkTGUSWpdAhpLQJJBB7RLIUBqaxN5Qp2zXVCxTar+8etT9gLJ3L2VHRn8nZaAt5wml iakjF0WHccgEygmHJrOGF/LPKt6rfKMy+hEab53vstf4hVzOn9ALNn85yx3Xzv2FQoPqLunezv01 UfuEdu6viXrqr5G6ot8ptU8hSy0Ns3oxUU/rxUjtZ/Viop7Wi5HaTevFRDutFyO1ndWLiXpaL0Zq M6sXE3Vq63JWLybqab0Yqpt2Vi8m6mm9GKhHbaUtB1ExzBxuMLZvMH7UYCgd0GJeP3m5MJ+NRCJX 9ADRS4QaG9ILPAapbMF1aZrjOSAKPI1sV1HkuSguw4zV0ptiVCR61fYqHli7ZSp1NlXWDW0PZAhz u3r/CO6suURG0hllw190NibCllozuK+1tCFexie6OzvdNow60K3QYRwXXVKgOgMAKxpmeq3wZsBr 6e4C32AVQnb1DPu8yIgHu+xb1i2tgsCyNY8AKguftZ75cXBM7kBoiUEZJedjUmsDT62haukUEjTd q++hlhMk1HTeRTVPrSl1yjZfSUrtU6t5ak2pfQpZamOzeUBdpnDx1JpSlwlcniaUlNYkkHmeWlNq kzLOU2tKndq6orqXUttlZGNKPMapnNj4wSx5tTt+LfBSvjueuE49vr5G0c2u35MVbIT/MjPwFoLT oSTIUMqlj+3km82muP5I65punVZOUAPcn8VXhW3jOtfEklkCG6O4/rArmEdk529Fld+DRRjPJAJ8 FByCh91TgewMeXagbwPqUebfs4IZ+9sdY8fOmG6bChTmuWzQVJHXwEG81Rb8JDCxQVnPDyxV+VuW bpnNmPyOOYzPd+/0x2/4BoPMs5tz/HW6aGfHUnakuaK3d/rUw7OEjd3gus7hg2A732en3RbHy78U azrr8RaDCAycv8fZhEYVbH/3LvuzWINU8KiCB1e8wxpeKwMs455a2kC2MmF449qe8n+KNRG13SVs NS34O7/ja8Fs9K0IOM/98dNp7mheUPV3qm3w5siLTZ7d8s3BVfsbeXIujHh3ySsKFdfKh+PtQ77b 3V3ePXiBu91R2LT5fn+a0U7p2ugQBh2CulFJRQ1Nwjipfpea0tNHK1Mb7UtM/rQDKceGXtKtoTrG H0gWdJnYdDTqB4zBGMH6rDAgFHR9hPjujBNasGxEUR0f7jF+OnxaD9XuGI/0cN8zbSmNqqG6cOhk OYO046E3oidk+gueKPahE2m+QUnAfFZFk4bnKRW2K0M1qZP3kL10DHnZO67yakcl3oSWiuhqdq8u BFD0YrXKAt6UPVnCFgJHXlSgamN4iu3qw2oEFscKbYdM5qeITKWITEW1Sqt0tx7HFq4c1luqk1xv 4dJY7irNpRdPizr/g3ihzZ9fCM+PXSj6wHPIUih6KMqfCUXfch8dBaJg+v1W8nv/K9FVWmFVKius h6LLN66PLd8yb9HLFIPqXxH62BK5jy19WcIn2lEJW3SR5WsrQcjLCKSNkSVCjCyRYmTpiwpTbfQn GMRV5waKEEUloRNRqSSoVFCLWBN36jEsRJVvw6/HlENM+cWQcjgzzac/HFKuJXKVKm31qLQ5KW3V /1XafGn7yCP0PhF43tI1uKra0FSFbkvZQIuahn1bkgNKHlgC+0EliQzXyQYWaGXNg1GUGqtBJGLr Y4FkmMyA1KxKW45hSmTVOioltNDyb7oCGOUhLMoAW3qOfj2LyqcYKHTH/EHh0mzoFp3ckDTlVq7m WkjDzedddPm/K4PmjKmI5hPie5aD3DLhoxEIE5DtWmYd2KNG84zYp86J1rFbeTL7k+YqawZWqcMb q3FFJqMpzJX01FY12TpEGZmmExrbKhueV9WWuD9IEJOxx69uzufd8S7bnrLHryj277o1tVwdr1nj jmSo7Iky/Osaiu62DQ1RZsNxI+s60uqVYVwhmA2Hcec+2ih0B+U7GGJL70LVIFJjZtNH4tKwy8eW IUQt1wSqtzu9ExpO6TIcOa5ylOSYIvrrqCtKfbUFloVbW75g23i6YIsmR8fhmjCyRk7wY2smAa1l aL2xaGkx3xHlbTtY24VHXfcihQd5yf98cOiaUXC48uHgQOOYBAeNUCDSVNuOg6DABj8fFGo9UCUZ BsXXQVAYKpkLMTEnu7PMdZKV8zBBsWfzlR3FCbnEmp/aoi4ldlpuhQ3Z78KHd5HT15vaBYfT49iO G0vF0Phuu8QLjmeX/Nm9+gAQa0/+7HxLt24GwZw2P3Pu3dC5TTUw7YT4aVTw6ZtGMLbCkFWUo8Zq ZTynVldPmXBQusTSN25S6jVmHdZzqByiyDzSVIYdypuCMWwiAGkkUdpHxqMvetNbkN9inFaxSG2n Y1DxusSgCMRW2s3gfBGJvisY1cwQ/xZ9iT0BF5RdmeDfP+CJwQkPzJec7Q4fxP3lQOhOpbLi0XUK dXiOLd/T5RQ0vpZ2ZPrqz1Hyog8/4Q/D1K4eyuxodlDu86dd3P03ABn+5NwKZW5kc3RyZWFtDWVu ZG9iag0xMyAwIG9iag08PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvSW1hZ2UgL1dpZHRoIDgg L0hlaWdodCA4IC9CaXRzUGVyQ29tcG9uZW50IDEgDS9JbWFnZU1hc2sgdHJ1ZSAvTGVuZ3RoIDMw IC9GaWx0ZXIgL0NDSVRURmF4RGVjb2RlIC9EZWNvZGVQYXJtcyA8PCAvSyAtMSAvQ29sdW1ucyA4 ID4+ID4+IA1zdHJlYW0NCiOiOgQSpJJJWkklSSSStJJKkkklaSSVJJJKACACCmVuZHN0cmVhbQ1l bmRvYmoNMTQgMCBvYmoNPDwgDS9Qcm9jU2V0IFsgL1BERiAvSW1hZ2VCIF0gDS9YT2JqZWN0IDw8 IC9JbTEgMTMgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgNzMgMCBSID4+IA0vQ29sb3JTcGFj ZSA8PCAvQ3M1IDY1IDAgUiA+PiANPj4gDWVuZG9iag0xNSAwIG9iag08PCAvVHlwZSAvUGF0dGVy biAvTWF0cml4IFsgMCAtMC45NjAwMSAtMC45NjAwMSAwIDYxMiA3OTIgXSAvUGF0dGVyblR5cGUg MSANL1Jlc291cmNlcyAxNCAwIFIgL1BhaW50VHlwZSAxIC9UaWxpbmdUeXBlIDEgL0JCb3ggWyAw IDAgMSAxIF0gL1hTdGVwIDEgDS9ZU3RlcCAxIC9MZW5ndGggNzIgL0ZpbHRlciAvRmxhdGVEZWNv ZGUgPj4gDXN0cmVhbQ0KSInSdy42VUguVjAAw+JkLn33YEOF9GIuAwVDINQ1VChK5UrjMgTzgNKF QCZIpS6IMlRIzuXS98w1VHDJ5wrkAggwAISCEEYKZW5kc3RyZWFtDWVuZG9iag0xNiAwIG9iag08 PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNTcgMCBSIA0vUmVzb3VyY2VzIDE3IDAgUiANL0NvbnRl bnRzIDE4IDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9w Qm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0xNyAwIG9iag08PCANL1Byb2NTZXQgWyAv UERGIC9UZXh0IF0gDS9Gb250IDw8IC9GNCA0OSAwIFIgL1RUMiA2NiAwIFIgL1RUNCA3MCAwIFIg Pj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA3MyAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgNjUg MCBSID4+IA0+PiANZW5kb2JqDTE4IDAgb2JqDTw8IC9MZW5ndGggNjM1NCAvRmlsdGVyIC9GbGF0 ZURlY29kZSA+PiANc3RyZWFtDQpIidRX224cuRFF8jhfwaeAbXhavJP9KMt2LMNGlJ1B8iD4QSuN bG1GGq9mtAt90f5mThXZl9GFNhJsLhCg6e4ii3U5Vad4cLT14nwrNP9tz2cHf15o8Xk70+JKzGxo XRLR8493SsyjUeJ2NbucvVrODpZLg03LyxkEqVVO8D8lnG19EDrZNjmxvJ6pVvF3RQfQS+jE8nw2 p0frxfLX2ak0sdG+1VK8v2s0zpNrgQ/SKJW/q+bT8v1M9Vq0aZP1CS/L16xIKzPo1C7rPDlr5jq2 Vn5uXOvlqpkbbJMisK55hMhZO9GhI+mgJ6+zih9IRcKmm4vycC/e4clAK0S27WS/4nNj8IZDOlh/ g49ObvmgNwgVhfloIbL9iyOE9z0efhKqjUb8KrQSH8XpJyUuZl1sVRJd1wZhlG6NE75zFH+EfVHC 7oawm0Ahz/+V8Fq1XRBRGdpQIj8JjInZK9dqfPyyapAoCXMWx+J6c5Ff1+Jyg3BZeSsurra726sf 73arC3G+ucbXKK8bhWzc3Vydn+2uNjeDh9b61hvhYmgjcg+LYEjH5sBwg5TCUYcPQAaWwtwAr9zU qxFMHW+mf0oYp1uAz1rak13SynKesjOLo0bDIbk4zmlVbdDOi7luVTSWckuuJ01bgInbhpK1aQhB cgdvjNycE8as3Kx7DcolwxqSYw04ZXXbGC1/aeYEpSve4eRq8F87drz30LWdeeDhA7901FQe3nat Dn2uki1WfqEiCPJ+23QwMx/n5dlgYNKdfehicHnz8Q0FJModTI4IDDZb6NrclO+AqAZSz3e5pNoY tZ46y4aYrOubbgPDyK5uA3KE6kKSr2c6cbbL+xruG5UoIB6FgiBd93gYVvRywAbRW7NGhG6QX76A 0o67EJbgLDpkeuh6kFMtJn5nCE5VPE6CU5YTFR1tKkmIY3NKI8QQc0BMHCIhQX79uqagOkll0MBY I1EMHBSUXuconEZRDb4uLambIPYDWkZCYu6RCIAACaH/tzm3aCIwhQQ6kSk5H3EsYjfaRKAXJ7DI S0p1Jzc7wjJgByyLDw01JzqmA7g567o/RbXOoNtNzGQE6bFVdPmYs/Yf2b+W29tZS+2CzzpuCIdS LG+bgN+zm3zwFkIEiIRabrKsIG2fJYxPlLCAKvFBaYo+n7xX3KcLKLDy8CPVg5ZzwqGTppkTsMVj tXPqfNDaKweD5eIinfK3ZvnTc2ZYPZjxMBSlawrdNqCTKM2n/uD9gjYtoJT0hPWiH9NmCyl9oRpK XNpcUFSSZ2tBVUvfqWoT4l0kQJUl+snCVY55KV3b6s6kfx1rb6geUL+ldxvoGhs3EDc2bmrCKLan GveT7dok15o0RLM07P+xXt079d292iAsTo0Y+b/u1ibyeDZ06/I+6dYmdNQVh249rOjlGKBCGrv1 IB+7NS2ZNOthxSBWY6eebj81dLIhZuShyJlECLUGgwnxR3l3KDmbGcYZx8RrIs02wwqru7zCWUNb i3w91RjymbQCNjl47ffOiLRxzSeEQb6e2qBSVmGAJQxslpo301x+d4nri0NrAWJgTjPvDCusCtks YxNpK/L1VKPrz0B3gEpTmLQckeeONR/gevF6aoEpRuYxU2PuMBPKNszMI5vm9zFhWK7SFCQadhC/ 9pReimrUUOQ9pRMo8InKqMcVbQluekiW97iiLf29xHSQCecMIw2NfmhRl08ULMUPaXf4QeC5/5eL w/d1Fd1G3z1kxzjeWlyhBC44GJJLri99LVfczgM053Ye5CWK2xAdW6w+b4iPh6IsLurEoXTeM/hp iADd5Mn/KScNhhck1SE1Ie05+ftTDx69d/+JCBl0Q68ARY3bGiarnuVcKHc1Ffspje5caJWHuOfJ j8x3Rgq6nil2Azxo5YU44usbAYBWdPL6mvc5mmDyEj63zF/fnpAWu7OGFd1gQnLy4izPPBd5EBKn JnwaeIDmOBpRFBLY9+5x3PS+aDyh0e5o3nhoNS8bz7e0V42mmyU+axq/DF00oyzSPxGaOXO0p+0P 5FEyZc74NybJBc+4FCHyjsQXZQDpgn1UJ94M84dk7zF00VirfXzofDc6H3rnj+G9gV+B/Hp7RHjw 8gRepxyNeQ4HWeTlK0DHjVJKIMUkQzSk8DCBWo3u9whdLMnRgG0Rev+Wx92T/EamLPIse5gbxrwo fwodaXottScHfFdY8FAAJYYGWbpIBEmpNXL5gUXm5eNxco7rbgSwiWrRPjX6eeinSpQWehYIBpr2 q/1UXnEycRCfl+Qx2aDlMiP0iDyJ8oSPzRJE+6lZFpSMTqw79IF+KO6by+/VUhBK4PRRwdnxNqYn BYfZgAsucMFF1PVYaChdH8J3F9pxX2EnxwS2JPcLjyBo8JC/8o53pQbtgDQT/EOkjUBLpUGRJs/6 wb3yLUEBdXeQBz46nOa/l8xOYn/tYUboh2zfyz5eJj0mqTDB4PEbLo83b0DfLu9avnguGGKxOMzY Kd6d5Bvju9yRh6A8i1XUFoMmwZ1xSn4KrWHSIRJ2mQP6b8vtbApETF/CBwTiMb+lsY+F4u5NviVe 7a7OdlS5G5A7NbREeeRScPIvaCwOzt7jrZM0sVsMAVzhq+vMOG2KnX90rcqtv0T2aM3sEeUVfkHD PKZTt7zJYMYU5KyN4kFyHtu8PMvpz0Z+5nytduKYMNfJBybvW5s7YG9zUOYRH6e9hsSXAxwFJkbj lqvbUi0uEyw+9gQ79LJsatqj8pSpPEoubTTJH8iDLlezlT/fEa4t3TqITXaCTzl4C2CI5eUs5EMU HzOi4A9MEpi0bbRTL+QfSXCwXJq8nSgNU4bqrcwXyz06Jr4nY3DL/SX/NJqMnHs20snt1/yMASDS WGAR3C2vFKe+mROCSiMxAB5GSYSmzDsFC5iOy7E6DKdiUKGkzA0VzS1iZKh9ED4IaOd0Zhjk4jXj xw8fCE+Ue6yPOcS8fdB339fdgwx30wzfZe76cXtPXWpLiXBowddDzQoKHcrJ8ZyeekaZdtmu0CLz CDo28/B15v+vDMQNDSIJ6jHN7VaMUJGXXWbSJjyAgQ4xKTj5cc4saETpd1c8cH9uG5peSIdAT6Fn 4WkfNUB+C+U3lt/uZWMsFvffkYJ+glZ0raXuQ9c3XCYCNSG+Q3Q8UWN+Xsx+HldAROP1cIP4+wtx MxnDcWOB1IdEVwhF97LpZUMjL3TPqYhh2vNiDMFVKb4/L+aLUk1sa5YZulfUxDXdmi6kNbGr7VZ8 F6yIXc0y9bwMUPQVuyBWFbs0T1c1sarYpQlINamuWRYIpzWxrin3BO+auHa0o+eauIZ/mgZqltkq /mmGrkpr+EcTquGfhpeaZaqKf2qWz0tVV8U/xDX8Y4qv4R/iGv5rGFSxin+Ia/hXoYp/iKtn+xr+ Ia3hX7kq/iGu4V/ZKv4hrh1tavj/6+zV8tF9SOcxVPEc+l8bAebo0aC+E34INMAkEmAfXYd2PR8W ojPGfYMKcYH4Hiq0ML9ChTUxovS8mKiwJsX358VMhTWxrVnGVFgT13QzFdbErrabqbAmdjXL1POy TIU1sarYlamwJlYVu5gKa1Jds4ypsCbWNeVMhTVx7Wimwpq4hv9MhTVxDf9MhTVpDf+ZCmviGv4z FdbEFd2ZCmviGv4zFdbENfzXMJip8J/Ml0+PHLcRxWEf51P0kW14JP5rkn2UpRhwEAGKdoIcFjos RiNZ8uyutLtSkE/kr5lXRXbPyBJf4lsgQNuc100Wi78qVjGZ8V+vQibTtSfGf70K+3Kk/NersC8H yr9ehX3VM/57V2FwqV2F/1fXnrSBcQpaS4WC4LpehzFIIMc4awkYkwRPjJMWbXW034Qpa+Fbx0eM tVatw4BP/TJPG+ga+K6OQAiSwXEd1vjVOaJcvjLQBdpov2kGtPFx08xr48X0OtE60lX2m/t2NEWg 1v/sEMOkZWZE8Hvr5HxQn9ikhUotUp6M06NgPoxITcV8OI44CW/e7fFrNFcPo3fm3e0oGdXcqFtx AikCCvyd47B7prWPLVmmvDRP17OrpxXwVcCRr0diwVQQbBCZXiqKgGJEPGH1dk9Ir8M2LZXJm1Pl YeUsQ/GCmY964SZ1QNK94uWffuh7wM+S9GX7g/y71zrN2qVis76VbH/DXpHqzNt38rcIqnIhmKsR dZGAiVxlhn+M4Nhja1unWOurf6AO/kTu8XOjrg6DdWJ/hJN1z1p0xJYf6wgemXVfbQx8Zqfg1HEo 89lMbVSXEWfWIdCw+ukytpqg60QeUEnyqcu0EdirZrQx2KtGtvGynzrTOtJ1vs1eqPWlmDb7hbw4 i8e1XE5Tdfmzw4ggQJRPAt5ozf4gcY/wvqjKXZXar3eLm0MuYtji4TZc/JLTuc9yTZirh6dau6we nib1T3t7iucz1dHq4TZcPbyMq4frRIuH6zKrh5sZq4ebkcvbbT9tpmXEPJxqTWuVZbc0ItatWOOM 1Me7q0ru/W8j/Pp8BNPJtN9uKr5XYzJvD6t3L9UVXux9ZYfXoMNV7LJYd61jSVR61RzRQoQ51+vS yi/Xpw+ajyOuvLDqx5NeZI3j5s0Pyytxtl+u0ZJhW6HJxzMTYqwTfCP8cdlJroxWWGwQhhOE0wnC pBBGhXDrlMLSKEyVQtX2dTjgncm8rIOPkjWz+VTzQf3tHtAiKQzq0Mc/o+sadm82SOdpyjAMidOu Wdh8N+7eb7ZyDpPNZ7nVfC/C493O18+x8zK5+nk96mk5aoks3YlmqoPmLWc+Yy8J+Wx/GHAtY3Uo ctp3Ekyy362me3wzaa7L5jC8HGXfywz3H7CdIFdsqrDIFXx/GC6T+CeYV2s8plTDphES0vQHQpKv ZUsjZP1gicI010KwEbLqJ0LklXNCljUWQpL7gpCTCX1CQlJ7Y9SWo4UQau3mVxfbzbCT7UqYyP0W cLzwZjC/4ZaDe4fny+83V29P1YyE1/VBy6Fibh6Gl8vzR0El1XPSuKyzPeinAtcX0Ejp7kLsUeMf +VT+HDXfShBpyQ7yuCYFNe/tiEXmhkQ012PSEe7w5UWxHH+UndgkkKNf345SxeGyjCs5guSrU0nw frjUJFOCnEHEqUlRhMIGHEizN2n1ebH5uLyRpbnEGa2V6T9/GG5OtQIue3wyoTL4Vg0LFSffVaMg RFSk464K65joiFFeNkZUR2Z24iiikmX1kai+bxRqi9w3CqrvG1XkNiVi6BuFerEQo7IkiK6atNrv q2RivdmJGsm3UavwvhqJUf1pv9WMYQ9CitwG9bKbp+WysynUYL94KtV+MBe/oPXS8D2MTm8DCdd3 o2S0/Rg1PbUYrXHnpQ+gkelx5v8tLoPTdN2Jy64aa3nRVaVW66kSl0R0xCiNS6I6MrPGJVHJsvpI VN83qsYlUX3fKI1LIoa+UTUuiRqIURqXRCUT19K9r0bybawFR1eNxKj+tP249DN6rFaq//k4xBWb QsEsuB9zdloNIKzzGuChnAJcpsPET8YZ33/4cJSlIsrXq4dRa4x3tzd1Uj/VyqD/2fSNz1qfe2kH Jx7W67tVbhkFXdQCr47hmSj1nJc+JZ5kB2fMfpUvvu45ptqP+SjsXC/jmL1wKF/EqM1l8FnOYH0j uLm+EaekcVT14/mMS8+BN4rUHLOkl7M1lFd8gebMrvrx3Abt/9aa8vywI9pABFkMSB6no/5fTwTv odyTes5beEaP5sUdiqiMUkobDVRbVgqrvTRy3tweh0t0G8n49GoBJYSMKTzmcnM4P95ZbHgBG6K5 08Ls9qHOE5Z58vk80WJH0plMNq3Ana6U2K6UF7+OTvqdf9+PIFfny+bqOPxyU39/QGdZYGpTsFMt iat4kO0Us3+ou4MRCBbjR+k00GWo0Wu1+FGxwlkg9+JgZjlfN9mvL5uAzUvIRX2vBml78c1GCAGa fdF3NY35vojE3BOz5OW+6Pv2gFGihb49k8DdF0Pfnqix2BX7swaJyL4Y+196iZ6+GPv29Bd0UnH0 Rduz5u9ftYrS6BW9ECSqfXSyz1ZkYVNLRLgWEBLdM0JMYkjiVlK5dHcOuCMpmdfSOWbz6ey90KTh tfRbvjaH3rwZJYxxH2xbw5WXZwSMxHEb3ap0jymCGX6V5zrdXZv18uLJCECTeY4gn81Wgs8jzGoH +7usmcxUhz+mOvwxvzql+zWwao0ND5TmtpDlXKtXfa2x+2rsi7XG7quRfFtr7L4aiVFk2qCtZF+1 5FuvrWRftcQk94h86rSV7KlWW8m+6voz17RI1P6yNTESlXi5pkaiemKUJEciBmKUpkeiEsxrgiQq mTgwzGuO7KqeYV6zZF/tS45hXhNlV7UMc6gEczsTzCESk21hmEMlmNvMMIdKlk0Mc6gEczsxzO3E MLeRYA6RYG4Dwxwqwdx6hrmU8n3RMcytY5ijVSCYW8swt7YnSfvVx1zUPuZ+LgRzUfuY+zn3MRex j7mfE8Fc1D7mfp4I5qKSZSPBXNQ+5n4OBHNR+5h7VPVU7GPuZ0cwF7WPuUcT2sdc1P7EZSaYi9rH 3JdCMBe1j7kn9pbMMIdKMC+JYQ6VYF4mgjlEgnmJDHOoBPMSGOZQybKeYQ6VYF4cwxwqwbxYgjlE gnmeGeZQCea5MMyhkokzwzxnhnlODHOoBHPi4jwxzKESzHNkmEMlmOdAMIdIMM+eYQ6VYJ4dwxwq WdYyzKESzNPMMIdKME+FYA6RYJ4ywzyxFtQn1oKKSiZmLaioBPPEWlBRCeaEisRaUFEJ5om1oKL2 Mf9pt9lq3Fr8q0/o4obJafG1u95YVe73G1T4Ng27/WaLJx+G3b82l+bFiGYkm7txi4xkbh/krze3 +9vjcHFov34eUaWbdyM6rWz2I9KeOYyvdn/FjHMsYdii70jzsHumM9s4yyLyBH/oIsPLw7iVpsd8 /IQnj27C3NeVHgad6fHPcXDDDmXhozwV2QxmszqPTGG+G3fvN1tUQrLQFvE3ydzPNuZ7ER7vdr5+ DosKQK+fiy1u2bANZd2xrTtGv+R1y3hwsmc8FXMc0dplM1zI34SdbtHpra9/HtE6iC9QrMMX+AB/ F29Yh/AVb5TkF3e4mJo7nDhG3fEUKwU0KOZm3KKqNG9G9HWYdOuxM1kKHprNNWYXC64wBlvmQX14 ek/MRdmCSWT1v+w2bpB/OOngNTykEAMmX5feXuOjL2uAMBlkEZloGiNMtuxrjRImW2YX8jJVHbPM yoXAZEcmxzElYhlksjQIy1xmHndZrgUme2YZMjRVA7NskquByYFZFuVyYDKbO1D+ITP+EWWMf0lk zDK2sKP8Q2b8I28x/iEz/u3M+IfKDLeF8g+Z8W8z5R8yWzpR/iEz/u1E+YfM+LeR8Q+V8W8D5R8y 4996yj9kNrej/ENm/FtL+beW8m9tV/PzzPgXmfDv58L4F5nw7+dM+BeV8O/n9B++y6A1bhiIwrRH /QofpYKJLNmSfGxpCy0USuPbkkNwEkjZZEs3pH+/b8ayNqdHDqvxs2bGyqeRhvEvMuE/zBPjX2QW emT8i0z4D3Nk/ItM+JfnXCX8h3lg/ItM+A+zZ/yLTHyXmfEvMuE/lML4F5nwH1jWJVP+ITP+S6L8 Q2b8l4nxD5XxX0bKP2TGf4mUf8gsdKD8Q2b8l4HyD5nxXzzjHyrjP8+Uf8iM/1wo/5CZ70z5z5ny nxPlHzLjny13nij/kBn/eaT8Q2b858j4h8r4z4HyD5nxnwfKP2QW2lP+ITP+00z5h8z4T4XxD5Xx nzLlHzLjPyXKP2Tme6L8Q2b8p5HyD5nxzzhJkfIPmfGfAuUfMuM/DYx/qIz/5Bn/nxbTa83w+NtG 0hwK8+rXD93yZLzK59WgT/GpW1bTYxRit/wzB/vToa9K9q/rURft6UV+gz2tp2N3fV+fvjr0GfbR SWdpV4fya+/dzfIdHqeSkBP6J/ldPmuMcdiD+Fi2KN0vh0YHs3q0gPb8R4KO9uT6odhnh/4r2fN9 pz6vvo4dHDwY+Xdjk3h1Kx69+LLv3PLb9Choae567P9p0sD2vTy/WpawzcZdus3WVIp+uo7m+un4 vICOVfOQvF7ULtWe8Kk9+t0odkQ1sEc3WKwKnuLeaffZrw5LP9lHvB3RZ7alSRkFQpYGie5JhHlP Ajt4W5lvz3dwiDsbFhj/QgmKps/e1ocvjxj4bbEycnhW718Wc7jx3Z2JUbuCKehx/GTChP2rJorX 0VybGLIMp6FIxXnCBD0L6oRj04PeYo7qEELTHz7A5ywbQF5JRWPo4d6C7PqgR6fYep9qOlzUNKXc o/jsaYpZxrdpSmEcL1nW91uWcoKGS5JNviQpb7zJsUVosr8k2ERMH9EMIEDCTsuSYLOLnDAjLs1j l6X5K90IkupwNXIpbcrRFK0k1cRd4eJiN9T/ajZTLlGYtRlZlqQ60PnVd42kYWuYms+Wzpapzq5j dbuasxaIuRUIHcWiF/yiRUuqg2yIqW2vg73+KPtxsD96B2c27Lz9F2AA8hyTMgplbmRzdHJlYW0N ZW5kb2JqDTE5IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1NyAwIFIgDS9SZXNvdXJj ZXMgMjAgMCBSIA0vQ29udGVudHMgMjEgMCBSIA0vUm90YXRlIC05MCANL01lZGlhQm94IFsgMCAw IDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTIwIDAgb2Jq DTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUMiA2NiAwIFIgL1RUNCA3 MCAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA3MyAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9D czUgNjUgMCBSID4+IA0+PiANZW5kb2JqDTIxIDAgb2JqDTw8IC9MZW5ndGggMjkyMyAvRmlsdGVy IC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIidRX3W4buRm911PwkrOAJvwnB73qOkXrIMkGlY29 2OyFY8tepbEd2EoWeY7tO+Q1ew7J4ciy7JXTLdpCwGi+mSG//+8cPju49eL0Vuj8uz2dPfvrQouL 25kWKzGzoXdJRJ//vFNiHo0SN8vZ+ez7o9mzoyODRUfnM7xIvXIiX5RwtvdB6GT75MTR5Uz1Kj9X VEAhDOLodDbnrfXi6NfZT1L5TvteS/HiU6ehT34QeCCNUuW56n4+ejFTbRdh+mC4z/PZuE3dUemy 42rRaYWVB/wb5KKbuz7Jw04P2F38+abTpo/ytJtrI3+hTi1X606n3stl5/FpeVUffYKAXeqqZbYG t8n6JOaTLbRAK9OM0a4Y8+YE62Nv5UXnsoK5wWIpYt5oHvHKWQvfxj105B6889Wfv3OLhEVXZ/Xm i/gbTcSueGVh3fjFRWcgQQldvcJDJ2+zIqTMtZRZJSzD6bGFDiIqwzQzX5P5VffiYHEoXn36sF59 vO5sjgPeJ7kWRxBN0a/lzcVyLY7hoaFWx3hS619QK6yzg4UoCVwcoL5e4Oa9UH004lehlXglfvpZ ibOZM0OvjXBD7IMTRvfGiaD4CIW3uFd4Qz8EkS8svNAbFKrlpnCk+JByJIsjz68ZFC0vYXGSK1zg zNVoZesCm5BSlLxNvTXbNnz/HY3YUm0HxY88F5YY1kotNeloA9QvqTHKz9AcUQcoMY/0LUcDLKrI eaQwCev6wWw5fk+n6xWsVJYLaqMlU1Qtljed0fIzy97L1SnbyFVNsEk7FwRKWkVjx6JT2k5mDn2Q H7p5QHxWyHHkTtrJ+gZ9oLHtl3E7l/SQt0vOjj2ZhmrJJ35s5LvbL7hBJaIfnVwvL5vX6IxU/HB7 +W0jB4x3qXdPchv2BTTnttulX/MYqe1a8uQZAE6CVcfR8Tk3W+4rvsujICAE8zxmxBveyOvc3zet 8K1xvQ3CwQyYbGAxKsnT3Yecc2gs4aJlT1bngtso4cOrTjvkYw1PIxTDRQs7rq/q82WZVKfFBNUj WnbT5aelxtaydjQHMwtO/E5uEBvkz0R+NCanaDs6oakJkWMPXOAa5HI9Nd97MTa/N7n5L2cOcxk3 Vf7Q5JAbE7JyxJxJ3m+9S7nH2vom5wYswiUERac2XmbZW9YpRI8RPkl7rHQu539cOol13BQvuDz2 aXKqirvV7rM06Jy+cW0TFzOO4JlJinsEpBfDznNlNI/k2Ay5mEMZeTXHcQK92kQ53ZJFOiDZsAPJ FoddBqyzTsvl1ZrVB5jo2EDnXWDH5vtlweybXBvmXtfmnrAbPfFWArGDDO7d2+5OPdX6rR4jPMBH ODYFpMqYuTlcKrT7GhwHbAYs2hDYx7vgaBuE0CuIjlORlTV2QOYncvHquDt6P1V73XvIe6Nag2bz VPnDKD9Y7aOJpejBlXqT9jIx46RxuDrSpTtI+VSQxBbc9QkgaZJvir8dJ3UN1t44aRQKzFS1/9dI aZSn0811xPRxqDQeCt0TXf/voaXWbMG9wVJHfPNHYaX7d7HSpJDbYm+stGjfGL4ZK43Nbd+wrsoN 60zM/dmmxyTvt14Puc/a+ibnJixCQ7yNl2UqYNKoCbUmcY+1mkWb2tpJrHOn+DFCXnOrig8p3mex 1TmF4+JJLBmuckOT6X2VYX9yFU6asOewfzBdI1irRGXODmx+9MsQHh8BxliihYvDBh4NeiozbCFv /9F63ga93QK+fv3q5OqkIPMF8Da3D09/BaYxx3tLjfDCcKKjfX1E3noXhmFrcmaa4EuTvrzmiAjy IiONQ5P6cpaNZaZwsIpjHCj9eKDUiECI21bq1vjgAy+PwQQK1BpMHhS0Q/cSwT3DNy9ZQcjOJ0Cr obXe8V0L7SOANsZWDSzNe3Cm26h0btNXX3218DUUT13xNBVPmZJv8nUn6THB0LxGeqrsUFm5Jwy4 JEnxKO9ab+4uR90mV+q6rm7ydpPUz8ceKeLv6d5YrRnqjeVFfkj7XUpLPGEq9+a0djCEm/9JTguL E4OULyrTPZjqFeNZDS6NWmovlm2+OgKJlT1qrNz9SXwNaKaIRKTpdWrY4mOuaZ/BqXZB1vlA6HTM KfHTebWWJymCrnV/LF5/IiGK8vIdC10XYjAOD/IsMKLtgGB+1o1MGKPy8vj12068laGwDVf+3vEv yA36X9ywCozMbDfzI25YlxohVptc4lFOio2t8feoQ9qkWTDvMwcmONEDExP1FKKHPNghtU1sy6nX UxWiGTm1kTwnxWKDGhA8SZ50BvOHp10NkMEn+l6Adk47HQqziZynW/T9EaPuuYkYRA8SjK9AuFqu h41gH6OP5hGjMZD9XCzLiDyrE1GHFLc59AOIxojoArx6KFSY13mhGDkiiA0ab/PK6JB9wVc0SYHZ 5CaY3R0dU9gcGeZdLJi8+wMd21E80bqBcMsTRmProa4u3XaCmWKKXh7qzmrxRjTfkwNqQEHcRjzH Cnssnjrl+dri+Vi1lXga/A1h52HxP+HaBsWqmHI508AGlyaMqbIDVjhiEp0yG/Ke663No6mtb/Ke 643PndvWT3Lt7CJjvUvE++l9lQdbMFfjHMIZ2eQ91+ucy2n9KI/T9Q5amZhx2mWGPA5Y68b5zpZh 4/9AnkdqKa7RIoQHFAXOi5cdh+2dZ/nwhlcAhNuRvUakFi9BmUID1Thxpa+RPAnlm8jMcbyMaRcx jaMttII07YZtmq1gt955BliXYn2SgR51RGQQtzSNg37NURjl7YjfsHdwjVW7NthNUfhxifVR3ohO e/myc/D1GLqSfE2G4WTfGSnED/lIiBNsknl84NlH6jGjWNDl6Lu6t96Yq78RvLhWZV6C+zW3Hsr9 SWb1GFBgSvCSvmQvk/xnCbDDI44b0BdTOP1dmmuqI4tMjbB17su5z7wHOp1c34rzTtedHf6u+TzJ y24gzVf82MNRmEBfGYjXhRjXD9f52cnOiGo/GsIJSEN+/PF1xzNFL8Qrqh2KOUF+KZkR76B3BHFE LOclbQTsZon1GuZpGnvCkrMyhzvgL0vli7PyrJ8MS4P3rQinVIcxRIh7YMpC/mNJCbIxZh1kqKe5 QWpBxgm/rzrae1YEIeryN6CWQR4UxJ13moEykwlGtSS5yQRds/Q1cZMke0dqyDtxfl0MuKFalpw4 W/FJQA2HQnoyi/1Usb6UfWW5+bvrKxqO3F2f78yRalU/mvF8+Xl1yo7JKT8861Ip5SxfkW+vV6V4 zldgTkifk2PDw0VCpgZU2Djx6ElHHSsnPdsFzIRTpO8sHojFQQnmgioOc2cLaPXyOePoS2oMOBuh GhZk98k1l7sda11gq2N0pZwIiEwr1nuED/RldNChDQYqvqZfZpSel2+XnYHSz3Xl6XJnYk3jyCo1 vTlcy47gdoUSsexFltOKjWWzEbX6tGJraiaf1ccWw1HBjICYT1PaK2FjyiCUoQZEcItIiyFTaBSR cEB3244ktSe3OHQ+QGW/AoIK9UONKmMarc2IHUIjh9YUxKZrpV/KGGPZr1fnpfCzcwP3YyeedDzX jKWDsix8wLjNvefT/JpvDrCPufxBJzj+Spx4qsOxo7RF4vMobZmOv3TDvyqvgh2EQSj2KzviYQZ4 z+nOXo3x4pfpB9sCk5llTTxt0MCax+hr18njHw/BbdceYhm3HJviXFJMLT6kfK/yOF3MOzrxN4VZ v/Gtob0O7Fg3lmkKzztrOFAh8Xz/sOeBNyqgbrEEwk6tZwif2PLdoU20oqAwosNbc9aEXaAnemsF u1pdEouCXTFTOxuNp4KjWp1piBUcFS/ENokmxSzSYyk4ic2NvknC4tPVAio4C2Z25vwevHWTJNOL vEkGl7lJYsrWuu0DNxiyFK5jpsbj/x/Q/urbsUixBV/GtlyGD5mVvkMKZW5kc3RyZWFtDWVuZG9i ag0yMiAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNTcgMCBSIA0vUmVzb3VyY2VzIDIz IDAgUiANL0NvbnRlbnRzIDI0IDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIg NzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0yMyAwIG9iag08PCAN L1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNjYgMCBSIC9UVDQgNzAgMCBS IC9UVDYgNzEgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgNzMgMCBSID4+IA0vQ29sb3JTcGFj ZSA8PCAvQ3M1IDY1IDAgUiA+PiANPj4gDWVuZG9iag0yNCAwIG9iag08PCAvTGVuZ3RoIDExNzIg L0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImUVU1v4zYQvftXzJEqKoUURUk+FVun 3aboLoJaRQ+7e1BkJlFrS65EJ+tT/0V/b9+MHH80iwUKAxZJUfPmPc48Xi1GR81IRn5jM7t6uzT0 MM4MtTSzeZKVVDh5uExTXKSaBj+7n31fza6qKsVH1f0ML8pEZyR/mjKbuJxMaZMyo2oz04mWdc0A PMHbqpnFPExTqp5nH9SbKEvmahcZlxj1EKU6sZjpJFVjVCROBV7KFRkdGcRV31J6GGkdfap+nukX AJMmpXUlJtW1YBidHuFMNsHd1lFsCmA8ANcpH8UpPlNUSqy4wKvM2rMYpuAYPHJmCvErhyjxUbc6 DPb0E0YpouKVBZ2XHeCDGUDmSaY6LGZgxUCQMDtKaDVZ5uAQwuRU6JRlh36SuS6PJLTlDNRysbyJ 1/XeD3Qzjjs/RtUfHDE/RoRkEH76x7HkZWJfxTXzU9xyYvY3sk9z0HgfxVrYoADUJop5re0eqL8n BqdQDw8+jFR3K/rltyh2kHIiZoDkLMV4miJ7UVFPKk7DI1aalTjmPLujSuLRzTWt200b6tD2HbUC bNQWWeSqx348Rr+iuz0t37yLUwGsvjmEldI6xjXgoIVAIQSUH+lpTIiH0+pqNfhxxPILqYlIeiCi Jw7mjMNZNbkzEgWCWZwtzntY0bYewp6afrunj2rw937wXRPFGQJ7qQYKvUBl2P8xOsc618udsNJz wdSCK8yqfrNlGqUK7V27bgH5HMUsZxtZ/IdH6vyzsH3TCLwTeAg40qLvwtCvR9r0K7+mYwoumaOE 5B8VmaMvSkoduq/kQ5va2bwqmg8VWjPmVK7mcxyKDDMHt4i4BZ6o/PSCcFmW9uAy/ynLEwIf6bEs ucTegUGhduvQxluYQ6r6IdDKP7WNn7h8rQS1e+U90/Gp32E3Tj3WAYKmivznugnryKGtWb82yqQo YvGJGrUChQuFgllGfPRsIeiSQbK49vxhO1XDk8fOXA0syVztBYVuI4fdPe/AR4HaTpqMwqM8PbVT WFQkgAtFKNGx7THO4SAiJGyuFH7Gcl9dH/sKPiUGIVp8x66A1FKN3n8lxslU2AfOVE7VDefGYNx8 sVDohxXMBkHbJz/suWOqWtx5/BPn/BcsCGbwUYmVN1ygpeKMLbceh9xMu7vVeKj4/28TS5FvsURW Tt1QHcLQ3u0C+jfg5EhqvACqZacNAzeDOUzXJPkfDCzUSFoc4bNvduI0K7/1cDL2nC5+pXFxpvHB ZCQeINQU7UWY6Q45fXtpH8WFQXGnxlZNfKSpHdfUZOvteEnowgnXbVMHv5KxELmT4Z73YKXzsEh4 zCGpSQjJcwS9SYpjy5e4rdGKJtfJPKU8myc2PTV7lh2Pwx26/R+gzB1wFjjROWwI5Vzy+TIjXIj8 oC7iHbuIjWpzJ2vARCtNLx6+7AfGui9cU6k9OU52cU0V6q14HcR4rsGeDYFgiko4F6p5bINvwu4w 9V8rvJO16+Ki8JyAZEcQ7uSR5RX7tRNWpqbJeg07lbsEF6OsyCz+cXFL9eBxK3TBb7ZrOT7O5odq 9u8A+ldE3wplbmRzdHJlYW0NZW5kb2JqDTI1IDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVu dCA1NyAwIFIgDS9SZXNvdXJjZXMgMjYgMCBSIA0vQ29udGVudHMgMjcgMCBSIA0vUm90YXRlIC05 MCANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+ PiANZW5kb2JqDTI2IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwg L1RUMiA2NiAwIFIgL1RUNCA3MCAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA3MyAwIFIgPj4g DS9Db2xvclNwYWNlIDw8IC9DczUgNjUgMCBSID4+IA0+PiANZW5kb2JqDTI3IDAgb2JqDTw8IC9M ZW5ndGggNTIzMSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIidRX3W4bNxq911Pw klNAE/6Tg71qnGI3ge0GKxm5aHrh2LKrri0Ftpxun6N9h7zmnkNyZmRZVhKgu8DCgDUcDj9+v+d8 34ujey8u7oXOf/cXkxd/n2lxfT/RYikmNrQuiejzj3dKTKNR4m4xuZq8nE9ezOcGh+ZXE2ykVjmR /ynhbOuD0Mm2yYn57US1Kr9XvICL0In5xWTKR+vF/LfJT9LERvtWS/HmodG4T94IvJBGqfJeNT/P 30zUIEWYNhjKeTXpxVSJSheJy1mjFU4e8aeTs2bq2iRfN7qDdPH9XaNNG+VFM9VG/sI7tVxuGp1a LxeNx6dlq756wAJS6qlF1gaPyfokpqMu1EArMyijXVHm7TnOx9bK68blC6YGh6XosqBpxJazFrb1 MnSkDD75as8/KSLh0OqyPvwu/kEVIRVbFtr1X1w3BitcQlNXeOnkfb4IIXNDyKwSlu70EKGDiMow zDVeKo0hMr1Dj2avxclDg+Bslh/XsCTKu42Yn99dLzbibAX38ZIfkBpMq6OZKPGaHSGd3uDhV6Ha aMRvQitxIn76WYnLiTMd33l6xwijW+NEQNwM82z2JM+6tgsi/2OehRZ5GWyb06w4POutisav1nSB lrdwTJJL/INbVr2SQ87b0LVB+C7yVuuZxIMGL7+jCjsXW+R1cLpcO+RkyT7H+3H1grdF+Qm3RkQc yeQRqEWfxq2lpNYbk4+61ngr8BM7b/qkTkXU/LzRKCXkHgVe43+Qi8HV1sSWwYPjkjCW/6fQHhrm Oh2NpF8dLggCqYf6/IKJ1lBUSJHR2TbzSwpphHmIqstRvZ04hBcK1PVNvw4m+xxr5dqwvf668y7G 8XBZlGBWQbcT1GcLRw6C+3XMnsDaheyMfj2bME0nJqTW4G1sE7zqeal3Q0bu+MoA6SAzwvuhr59t iIslG+k2gsddg4jBbUwL1A0QychXJS9aZ1D4QBoFYX0aBLeV0u8lsClKbdLLhrj4vnnk9xzlYgts V4kPo61lHeEuWHUzMR2A2ozrartLCH/UdOm+ctwtQmQIjHfEIdNbnzIcy9nJWTP/dUwKCHYqC0ZE Aw6Efn3Tr5/NiKqcTp5g5ZDJ0Mw4JvpB/XSHfIB+iJEdotPpsbQcMOH+XwWH4Xgb9G4AfP365Hx1 XsjkOgdhQdQ1iOfemk6tC10ndPmpuK60qbiuXKWG4zWRwsvrjE8WSBEI5JEc2EwDWeJsRfLqZAVY eCyFJ2mihyxBkhyfITGq61OOugONhpzLSPe9CFFda33Hnx3X7gWJ3rc6w84TLPwf27rThehYIEOF duhCSiSLcrUqPzuCmZUtlChPfxOfA9A6Amu6cTsNdWZtNhr5CoSCm+AKoAVc9lwGupTxw6h2KJBq AdGh0vvxmTh9IF1FefuBvtAyU0ifX1BaK0RkxxMq9IJM6BHi+Oz0fSPeS1oBl7ry84E/YRswiiFW wc7BEEWfHTbEelMM2SK8nnAPsl6rrfG7BqTKcrPFXQO4CfJT6WUWz5QVEDlEj3XnyfVViB3i2rdL pbQ1SxsBRMc326IoGzNbIuHhUZMLY39JFBcZsJHdctGhkkDzy+qJoW+mHvPmXpWe9gO2jVEDiPCZ 84OZqtvy9dlqAyejfAJJ+HpRyuiyVo0Gc2dXJze4+hnUg0O63KpbEBjbgycQgaeA5nTrP3TscqqY EPcA8X7XFJ6x+rFfRsP+Qpv2pE20LqNx0iMal+rB6VJp54B1U+5lp3hZ0zai8L7Vl8axGfgmV+rC pN/gSmP3A+9/wyy9xb42ZYq4nWi0CUycsr7p164LLBus0W2F7fXXnbc2Y/hwflx/3Xnju9Im1vPj utZzWeM8nO229+u6y0mNNUiTIRnWX3le50iO5/t1j6qPeMqEPHeE0Hfa7Pny+Cc/RxIbTieElFgQ 09hJ2IFdK5X9uFqIdUNevWPhaCluG9bP+A5IjLcb1pdnbInR4h55oQm5G8JSrKOiYmrozg1NkBsg ts6DHxc4j3ZWNNrLY0yDRp41HD9P2ec62TZGCvFjHhJWjU4ylzPefeQ9pl8WnJ9/V2XrLYj7gzTC syhjmZ83FN2V5/PchAEwML7CRtqS7e7kn4UxQeA+kKNbZUoL9rgrMdWQGc4mis61AsbqIAh3Orm5 F1eNrpLRj+IavE/ytkEhSqEKpbZUgbbSEaelj6kfbvK7870e1b5XhIhERd69O23YHrVCnPDarqgT 5O8lMuID7u3pFB7LcYlbDrtbNFQOpet4FGgns7M9fvKK+5ejNqnzfhgzxviG3i9wdmCcQv5hFgk2 RAw1upGWOgapRZ0HVw2VvCwLIerxt43C/6NCedNG0ztmVMGoITLOj91ZZfHPqcxMrWN3xidxtS4s eifyPAWVLpd8g4RGWjv2HFrLB7yqL8r/+s16RaVRBOurvUFRQ5pr0/c0n5YXLJEc49eXTSq5m9eQ ZuVmWbLlaommJTc893WcQOyQgeiT0fK48Higy3ekcsd5y/pAW0CgbhuLF2J2VBw54xWvcyEL3Oox K2qWbw6LQbtEroQGud1io7fYb9iQ9rYaRlPkAoybb10ywSNsoC29gQ553/HiNe0y/epV+XbRGFz6 qZ68eJSWJvbXqThcl+t40Xi4LVcXQ0Hdc/nwZlOmqq5UoGaNMd84cqE/Nz0XlUkQuGnJBSZ3sYZM GMLuJCi63JR13HUmcd6tEFvLb6dxbWKBFxb7RVFh0c+GOlpbCLN3qjWFLmlbqZGCV0z1zfKqJHu2 r6M0kAfKkuNEnzJIx0LGpnT2gcBfZEczZImqBfkR2iE3GsOKul6gxedjksn+Qle67eb+W+i6jIYj XffrWfF0ysvqaAD0M15G44OW14Gr0aP044EdrBh4449jRtiBLPK0I96VgQ8/mqV1+udjG0qYq06w QYUycw46jt06+ld+ysGNrR+UnHJkCbWrN4ZN06Ftd2BXc0o6tO0OnVZ8f2jbHdJMPb8HbPEH9CL0 HNALHvcH9MK2OqAX4f/grj6kWci5dmBbHxKew39o+9DVjq3WoW3zvGZ7OjjcNubGk248dRUKNafe TKxvkesOpXw0NYR0VgGYLvMFu4lMa67utLbv13PrX+RasIy2mBlYkyH0TaF2PezqVBn0h39DTge5 mx7SV/glTYM+gK+brEpEw7KPNexIyaoKRF9BHLu4WN82hlR622QqXF8SLm2/D3BPhaU60u0DWeyu IP3NWOC1cjPMmIDyF8GiGePgBOSwinF6bggyyTI7QzC7Q9BI5Bne5fHZKftomIGRpwBtNPYxHcuT 84/86MnkyPYpYE4DlHs/OCZV5J+B8o0Gb0zZXSwvSKaupwzc50D7WzPWVudeSIeRuSmhWOb26RPj IetOJkGP9q9kgAtoY3e071IR9nZdGpy7wrLsZjsJgqD370tkRnawrA8nfB5SkUtwwfaQOsTF4hyL tEym1jHND0+m1uYxxpOPn8Tk/9hpT9LCt97hCQmdrB4Ex1AEz4942qH5rb1DF/0TLVz59iiPH2uW jZWrVZmqLtAhdMPL99AGF25F0JC6hVfoUwQYErKmMZFEHgew1JRDIeUxlUUVIT58qagcv/oP91XT GzdyRO/+FTySi0hgf7DJxp4c62LAu3BgL/YsyyNZ2NHI0YxtJH8j+cF5r6q7Sc1wqHHWcDZ7kabY zWZV9atXrw4v8LuFJ60/oa8zgr67Z661E3udbVww41o/s1Ag7IXZdvgRqmClD+H1TrpZstfZHo/z gc1wtFEhoffMFr898Eey18XunEqp9O1iJ+mkLqAzsMqQkVI8b+ZqJ3ppatFyU5KpX1sz6QJ+T82c UY2YOC3BsY/Bk0/cbOt3W4ylUici5ld3I7sM0vnbjuAcCL3QUXodixvEQhUS3HkJOyT0vNw00p52 CJ/NJqno+016zqZGcb1L1e5c7/aBeLLjZnJlfT9ChsYJeHniRSpjP3mx2HM5saRc4kXIWevwMQFJ ZL1GdsthkcPYG6YjYFxM9yjcdAALk/KBzeY8YcDX2+0thyDLshxEJ2DkAhtu91knWNFPiXaCDESP aWfSz7sgqki5B/B3T3JPaJnHg97xnUNVLJxAAV0c9P4TBYy26hllhLtn1snUNTKG2r4XtQnbtKPh Bxit0N/dMw+Fw7lG7XW2mS5+x7MJ+GIm5vE28pIMTkQtYjyxT7CPR+eG8MIoYkXrpQHOpFZ5ca9K 7g76TmZajHn15kDJaRHYjr7sUd68XoDmsJqxA+GcOwjpayBj4as9bv6qOQOaEvcd1AX40YbOQLND MeM+MgmkznVJAhlAeDzxBn9DvdoV5rK9CCKFrKOunNdFSKZrZeTD1IhafkoUiUR13hMg0zifcshM rtK2XhuZB+PAgWSviz2PhxPfNs6x+ZTXi51uVE4rxJYPT6ZzkotCbcV+M6UCB8KzWdt3Mv4dwWJi AjfI1twSYplFXK+zCJNnkTb2hoDkER2rXfUST+uLrP/aGOYlf6uHQO9xzjJ2+Ct+mAMVwrvWeAoH jPEmOyCakQKyuUDs7JH2OLEr9CeDix0gvfejcHlw2XJX4WeimGWl/Ix5qgvH+dlGkZA6b4UE9SV6 dnZO2v83Dh+OWl63/wlHLXDquT9h1DIYyYavGrXMMOANZew/y7T1WIstc4/pJWOFe0b7lMIFfcRp 5Wb7kX43rXDh6RLeeNzJ/7+Ch87wXyHgjYnc9AdR8KmBFdiMDe0p3Jz0euzYy8rb2VygfMt0+lKo fxQ1bwdD5J+m5q0V8fsVat6iF/p2jp7+V4L+KU6wnVcgJE4YbVUxShFF0I8UsiToZ3EBrmeV4ft2 FhSqdRJ7Pr9SFXzFgrGkzK2EZvarXHVS/fIiiwLKAYxV3tA9XB0bMwjKzasC3DCQ3MVe5gbAGkEk qTZ/z2muATuacCjif5/7BynBET0wADaK/NoFNr4WxYHgcMqB4kjc+xy6eqjfv39ogvacDoDbAiR9 vV8PXddN1JOXy32cpzTYUjenSsAu+6Rw6rw5Hw7KIIf7zaJQnKuaoI+tkFhrJ/a62FDkzk5m12wv 0JhxbRblcySGAtJB60UjBfpayxkU33cHyiC1iBcIYWDpOkiAzWYlEV3tbptYHqKUJdr92zIgLM4U els9an+BvYaMafjf96z1J67MOBbDDHl9nyjnZtvWBQSmOmuvYjKEMPB8i0KwiCzaSSEcEIZmNTXO VAlmkStSWiPlxUxav3kk02JIfpZiGBt+sqMMvqUYin3q+3YiJJMhCQqdIzWiY0A6DD7b62LbNjUY H4nf0T6p4bghpuPQcIqx0HAsJoa2PSJEvmnPsVEn0dN6ju0o7E9vOTYCoGEPRt/GfzOSxph+va0x /WpbMEkXxtsrdr6BgfUjf6AHvOcebx1fTUq5EyVExwGp6u0X6v17TgeuvqNG9vUlpZGH7MfDHuLG UfIA6qGuLh9WiSvYPKIvpCMRtXrc7RZFQu6R+WvNCWTAoYgEurzBrFmvRDC9l6Pe/qDO5CyCqs7A AhDaX5hOQ7qC1LeYOXBQqH8TM9aa2zP2EBBndWaxJw5d9shKmPzV9urWq19+xvDX19VPjeEQeMnz fP1R58PqEx20OJYfEaEX4aGu3TRoZT4/3Wi2Vg2Gy3TIGgFFhtpjJjIMsbrVgPW86nI2bW2XnbRR nbyGGy2/4GR44aRW727v6YfJj6t7fjTW140hGVW7D/p8VUGnkvVXjcVf5Itufm5Q3xgAV9XFas0A LG+WYclEB3Oy3aaRDpciyLYliVb9e61RwQMrVwILNwWI2Pov1VuwYuBNm/rhZiUUqUsVMga4V7Ik 6QOXVrvGUER/UDtp6+o5c90xdpztWExKtR0LasyibeMo0yfgQ1Hxcs/HnT7armxtXY7I9TmiBK7A FF6v0q/8oCFOz3j7uFHC564ZBDXybJ3gTF8jlqwTYKi5aSSC3WXDgHb6+j3T3Oe1Sk/THb/hd1fe 3s4X2pCrtzVeA2BJDcyiQpRTb/Xql4ap/Zl6YeBXAJBOazDUH6v0o7puLE6vPxFngT7xiKuGoNBT bwlmlHJaqvKPNVH6j4YkVyne76+n1TxBtkvIJkqtoPTtJZMXNMW2vmkUhbvqJZHg6wumyNc/Vht9 415udFcxS45FBvRaST2KeadP5SyHszjK4Szlnq3k19bpg4+wHUoi0TzFw3crbMadPNANlgWOFDiD PonHiBM/ELPl4bs1E1y2Mp+sh4eGDt4DDlFqH/yBJtszmwjkcxMEJPzW5nHSSrm55NKbJijvmFKi USra1ISTuCUsXao7poXP40sRgZ8RUhWx7iSfXjKGWz6jqz+qrqyeX13l17ZbvY4e17FfdNJG4pi/ Tp29/KT3t2O0EalyiF5aSycpMdpSYv1PpquXFFjpLAMJEhntNc0B/7qUo7whbds1PCAdisySWG6E dnC2pNam1zYH9aNVM0zIgkgccN5DQ1DfMEsA6O58FHRJOLgWZVAFtGiorZay5szZJBquZRVUeXxZ +u/SMu77+LKluFlaXjpbVPXSsl96u6X4WFr2S561R9dsjHy+tNwe9wt9nqJnabk97hdl4tLLMrYd Xw78vbRslg4HUo97dqjg6M2Ijb0BhqpfJNPHRkvMkuxXICG0pGQ9iBRbNW1SWwD138UN4JWHi8zt oTtdmmV//aHa7OPeOYFJ8teVqTfh/viy4H5pGbEdXxbcLy0vnS24X1r2S28L7peW/ZJn7dG1hPul 5fa4Xwn3S8vtcb8U90urZskzwf3Sslk6XHB/bPlvx5GfXkmzCzxQwJPi+/rhjn1A2ooMDI3+tUZ7 ZigzBotMpwQMUbGbyMGYO65Ps9CrWzZZ9BtKCEptfui6kTGA7ZF98SJtke6T98/KtbkuuaGE4Gls k7Z+wcbn6lfqKAp5sCg+CJAw9MVPDZy/4qDH/EQFBuEmAm+9Y7QY2SBLRemt2cY5GjSqVvTpw01q vLv53mi7SW/81+db1bgr1Qhftv/ODPKfAQBQgJPKCmVuZHN0cmVhbQ1lbmRvYmoNMjggMCBvYmoN PDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDU3IDAgUiANL1Jlc291cmNlcyAyOSAwIFIgDS9Db250 ZW50cyAzMCAwIFIgDS9Sb3RhdGUgLTkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5MiBdIA0vQ3Jv cEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMjkgMCBvYmoNPDwgDS9Qcm9jU2V0IFsg L1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQyIDY2IDAgUiAvVFQ0IDcwIDAgUiAvVFQ2IDcxIDAg UiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDczIDAgUiA+PiANL0NvbG9yU3BhY2UgPDwgL0NzNSA2 NSAwIFIgPj4gDT4+IA1lbmRvYmoNMzAgMCBvYmoNPDwgL0xlbmd0aCAxMTc1IC9GaWx0ZXIgL0Zs YXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJfFbLjttGELzrK/o4E5jUPPiSESRw5CBRkATGisnF9oGS RjJhiVRIyhud8g+xke9N9QxX0jpZYwFxnl3d1dU9O533Ka170v6vX0+mPyw17fqJppomNouTgvLU f9JEUZQbRZ2bbCfflZNpWRpcKrcTbBSxSsj/KEpsnGakCxsXCZWHiYqVX1cMwBPslutJxENjqLyf vBYvZBLPxEnqNNZiJ42KLWYqNqKXeZyKgZcyQVpJDbviGZlxpJR8W/40UQ8A2sTG2Bkm5UuPoZW5 wOkkwL2qZKRzYOyAmwonI2NgizSWMzgSbEZ5FudW2xtbOmdbPEp1MHXHpgpcbjbj4Ew/YmRgHVsW 1h5OIC7MADaLE9FgMUF0DAQqkwuVVpHlWFKY0BnlyjD94DFEcIlFjQ7UMkpA2hJszcR8uYj2VVg5 u44WfX9yF5DsAgI2kZPwi4xlRWxvoUC/SRIfagD561dpTYZIrDVxLg6YKSBbMwNQg2/qyWQcDVOp pQhfnScPzKnA3EMM3iZYTwqk+DcZ8fW7n6k/yyhDnpuh+pOOXXtse7eRvPKcsCP6dV8/n06/lhHf CzstzOBzqOomaqqD++b19NBu6m3turfeofKrUWtX6vJHHqRiwTaQwvX+tHEeaQnbRsw5gYVYLujj UHU7N3yiuiFG+ZZ9NoLo99rd98/oUB2PuGJF3exGQXoG9A0Ds6sDxSMHkkBBeqWg8BQ8p71rdsM7 YtOJ6DhsK9rV3h3ojVjOlx5xQaX3jRYvKeMjiUg4nlSsaF8f6uGNDHJ+KjPXctTX1HC6ffiJaBvJ +Wnceqjbhqr93u3qKiyu3RNJvwQXpAljS5+uBQ1V/57W7eFQNZzBXEw31VBN+6EaTj3z24NfKuev cKgZQZ8KwJNpL/7bG/dDURhf3gmi6PtapoiF5r7mIRv2qllLizDCkaNkJQ1fkrG6JlHfJjHH9RfM uhW069rTkdotLKNlsZrKuUfxEUmASiV8XEA1ftY/pZnkCmcfwc08rYU4HY9tN/TUdhsHgeRYcpJr dPPAMfVDV7OCZtwreKvZ0bbtaBUSMBP3rBot6g2kVu12ndtVF9LVfym4+mRufWJLd6wY0LluP7ju TNuuPXAmJYsb/iBeoIMCzgoIwNxySraSs1VJzT7uT527QBd4SNChjMIVQ1kyQ2fiBhKelfSa+mys 6X/A94xhlj6l8/BZSK2wBu1J7li9J/695K5MnefAeV+GMDnfhG4+S4a64ORKw/RH7raJuHNHHhix R2MX1TmUDq14kd8yX5Dbres+hfPoGGHrctLhaf3j0fG6Q/cbPXncsXWax2ginz8O/9thUMmsPYlC Rg2s9mdqnNtQ1VCNFrIgvBfcXca+w+IxD62G5nfsUIFO8IWaMLPxUfQ99qIHfuXmfD1HitEnvKEO /WPVngZWxd/92ETX78bdNSStxftx/XRAbvBYQiLeqdWJ13PRD814fjz4VPWom3809KPq0eIXVKQR LaRmWZuu8b10Ew34au4OGsuRtJ4Sn43xBEQKIbkP4SKKl3kKt975SRUOet2zX9+Xk38HANv6NwIK ZW5kc3RyZWFtDWVuZG9iag0zMSAwIG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNTkgMCBS IA0vUmVzb3VyY2VzIDMyIDAgUiANL0NvbnRlbnRzIDMzIDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRp YUJveCBbIDAgMCA2MTIgNzkyIF0gDS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9i ag0zMiAwIG9iag08PCANL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9GNCA0OSAw IFIgL1RUMiA2NiAwIFIgL1RUNCA3MCAwIFIgL1RUOCA0NyAwIFIgPj4gDS9FeHRHU3RhdGUgPDwg L0dTMSA3MyAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgNjUgMCBSID4+IA0+PiANZW5kb2Jq DTMzIDAgb2JqDTw8IC9MZW5ndGggNDgyMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFt DQpIiYSXW3PbOBKF3/Ur8AimVjTuAB8TZbLr1KTWO1LVPiR5cClKoinZztiepPLv5zRA0vLtpJKy SB6yATS+bnSfrG6i2t4oW//dbBcn/15b9eVmYdVeLXzqQ1E51p8YjFpmZ9T1bvF58WqzONlsHD7a fF5AKL0Jqv4xKvg+JmWL70tQm4uF6U19bmQAuUmD2mwXS7n0UW1+LN5rEzsbe6vV2787i/H0QeGB dsa056b7uHm7MJMV63rn/ICbzetqyBo327Sh2Tw775Y2915/6UIf9a5bOtcXrSwep37Qttpc5tRn b/2RLZvFllxF20z9IaYKPr78NF78VP/BlYN1SB7Wpje+dA53GGzAKi7xMOibOtBvG5gsCQMd/RWX yx6s1qotbr2C79/i4k+8kZ36oaxR79T7j0Z9WmCVdSuG0HvxcO5TUUPpc5JdefVi3JYwb4s3yovV iJnapHL7XvZk8mT1mEmz80zdEL1++W7p1Hp3/X2/3anXu8P+++76pzq7ur7tNn/KWmZkjJj8obAn Q++CCtnLUCn2mAbG8/fnVua5Df2QVP1jlM2md1hYjPcmNxEyzi24thtvVsvq+0G7DhsctXpzPT44 bw8uuqXs8P6ywzSwJRUezM6qpe2dER+8rtbtETbDsfXYOwBSf9QK97ZP+qrD0vSn9nTfLSNGuqzP 2gi2N08M4UulqRn/APNAQ2MIg6/frLuCzz90EyHiMey6xyqwqS71ETEWjPzAi2tx4kPXSSyoYJIE HZxX1+Kq0xACX392iM0IBBGUMmfskd62Z+eHedRpL52LspsyfMFPqphh+HC3hw+Gd0gLfqiR/3Dn XPWt3qzOHiJjs53GcL5P4Rdj2DwIUn5w8u5jemu46tN5mChbPQ9gxJPzAE+6EBkOqSpEP0ZH4yGN WcCYkYyCzelAgtNO/f77Sp2o326/7q4FhEFf7m6RshBm+o08cFqQLAIkYk6AtJWZgoxx2ZmalR56 fwzvMYJcfhQ/j7a+YOsRNPLSQ+/HMpOd4kR2J7lU8DbAGal1kHx49rVd/LzZy+qs3p53wE83PGAh 2RCPqW5et3c5IzX7p5e3u+vPWLHkw62EoNU79WGMKACqz06PUJ+hS1bcPzEPDx0z/zR12Um+CMZI Bny49JSPIm6/xqL0qkt6jUno08deRzbFUKPXvZERfuF4Z11LV/kp7mlO8U/mlIt6aFy08GyPLseU 4uH9R85v8T2ucC0IgmD9vcX1HOUS8rubJ4M8zXk61MP6F+uNQd6NyTwZ5+XRcd7Wi9DWQY5zQLZe rQGV+F9mtZQsaI0+myfXDo+JAfurpOejXARfs+Wc9YYpYp07ilh8o716Vyfk9e7T/ly9rClw0OPP Tsi4eewpj2EkARdkdqTDjGjzoCXcT1XYb/xHGgyD+DJEO6bi++dYelSgrF92NRDfLbuEH4eZZv2v ziJfwF+I1aLXp03zXQ3V9vBKDg8HXERpf8+rjupEfo7qjVpBTOuIkqdxAONYEVfWRWEhDjuE9xJ4 xA5E2WJQ4e+cf6/Su5/8Ebk46r0k2lbpWeOP0Vx1FuhgGQ9zicnOTydkseNpdd2VujwrddptJ4u8 2krAeH01ZyPEgasWSqgWWgA4K/hLnbevX4Rj8kMSqsYFhn5wDxb46LypCSklM5WvxU/nqeCckCqr o9tQ80mKyRU7+IfLkzNrW9OjOCNrJMkaCtuuRsjV5fhcKBQmb8dyJWdrjxdaJ+KarV8uGfUjNtZK ekwoyIHsxcKWttHt/oC1O1OqN/wgB8/FjML0xqSHLPnuUC2GcKd/fgGj9Vt5pY1xPOZhljFCqreV vWMDj/0fclap1Npn9H++SzFhPmxkK6ze3+7Pb0EMsJGKShBCQXh60kn9/d8u1WD6WZOAbJoHV7CD g+mi+gqk5eHx6ZbsEcWrQ03WWTIrtszVnQpS3t9O/v7rrjav5xdy2WAEIiD2/xfq8ui4QfRBTaXW NJgJYJ5f/Qw5VlaJjJ14Xg49VzG35+XasTHZs5k5yQJMZrZt5YzIgX1dD20mBzYz87yGwiiSeUE2 ZF6ozyKZl63p+HkZ7R1VLZtZklqZyZYZjzWIicyGDjWHEJnxjwBj/Ev8sZk5xj9Uxr+1lH9rKf84 yhn/1jD+zUD5h8z4lyxNZgaZ8c8YNJnybzLlvzWlTKZjR8Y/VMa/CZR/Eyj/xlP+IbOhHeP/f082 cnLgRUFgrGVtK6rlajzv0L046Qtw9jhp0q5x9rvYyn07tFoWDpt19XrsMKYHcogBBHk/t2ahfj7b +4kLzBz9YL2Q1gTFsa19rLS181kXUO8JT3EQF1/M98nXmhxVsMRByrUYDrGS2+5QUaNlcvP9Afc1 YNutFNBusjPetEHwYbt1WYY6zLde+BuN2Bok0wjtbruYZtDuD4tpfu1+mnszNN/VUbaLm3Gvioj1 D4qT1nqk5KZeSMqFo2rhZSf16jcp7or+dkD3iYoBFZoUaKhUULXtr6RcS3OLZ1OoFS060bkAKbkV eqt529pGeXwF/9zthpFKr3jxUiytK0JcgV7J8VMH9df0RqtOgOQz1ckQWbwPkYX7EFi0QyXBPngS 6xBJqA+ORTpUEuiDZXEOlQxrWJTjkhxyeEzOODwmR1wp5IQrhR1wJbPzDSo53lCOk9MNKjEc2dlW IjvaSmAnG1RysJG1PpWESyuJ0iBvjXGNzmDqO5K/a2lRCKOlRYaVMx8ZVCL8OxqPAckVZS5ScaiJ tzaFFlGBj6VwR0c3NXJ5NuzLnWExB8Mv0agk/U2Sh5fQ30qHU9PI1eVdhzeGrHdt41KNg7ZUX/Np Xap3beeIHIja9o7IgX0dplT5nBzYzJhlX/MUkQ372tVMRWTD5mV79rGtyepZ2dRsRWRLjOPQSmRm tp2fz6mlZiwiM4+j38hsZrkmrWfl1HPVs5nFmreIzPhHv8H4h8xse8q/hCf72lH+ITP+GUbIBox/ yIx/5BnGP2TGP5oR8jFUNnH0Iox/yIx/9BuMf8hs6ET5h8z4R7/B+IfM+Ee/wVXGP9oNxj9kxj/a EcY/ZGbbUv4hM/6NofxDZvwb86zmhoHxLzLh36EKJvyLTPh3Qyb8i0r4d0Ni/ItM+HcowAn/IrOh A+NfZMK/QxFO+BeZ8O8GR/gXlfDvUIcT/kUm/Dsp+9jYhvDvUD8S/kUm/DuU44R/kQn/js0axTrj HzLjH6Uu4x8y4x8VO/s4Uv5RsjP+ITP+pYtlM/OM/+Io/5AZ/8VS/iEz/oth/ENl/OeB8g+Z8Z8L 5R8ys50p/5AZ/zlR/iEz/pm7c6T8Q2b850D5h8z4z57xD5Xxnx3lHzLjP1vKP2Q2tKH8Q2b8p4Hy D5nxnwrjHyrjP2XKP2TGf0qUf8jMdqT8Q2b8p0D5h8z4Z5wkT/mHzPjHc8Y/nhP+X20WyxrXBv/a FXpHlbzAt7lYmCrcbBfoIkxSm+1iiSvn1ebH4r0+69D1ZH3dLZG19NWt/Dp9tb06qPVufPq9Qxeg 9x0au6y3HZKj3nUfN29hcQjFqyW6mzSozetq2YRBBpEruKQOov7YdUvprvRff+PKoWHRN22kW1Ut nbwJyqoNqsw+xyJrgTVT7YgJ/Q/v1bbbRnJEkTzyK/pxxgi50z33vDlygBVgAwYoIFjYeaDEkUWH F4UcSfAH7BckH5xzqqqHF1GKNgEWBkz1dHedup6q/kN69X00xjhFoDGKsqTsD6Pkj9z46eoq6HVo 1CDv9Tp18dHgLG8GizO1GK+yICbjD0+b8VeTLFO8IuvETflbwdIxHpXD8ccUzxL6Am8A+AIX8Bu9 kXkUNL3RVCG6wxeVucPTMeKOCyDleP0k63SM8TS5TfF8hNBxgGWEgofaZAXp1GCGNdIr6cWH+3NU F/MOhBD9r1cj7767UZ7rFF/kzJDVKJTVRJeojOVoOspDLbyT89m4wnmlbj2/HLYLtBIuc6WhuH37 DhJbSVc5QQDtxBEhbot8LnX2idsQkEFPUfLJjXwmUwD5qsD/VCXH4MvUnjK1EdrGQuuQMEjuVnPd wxe4hwG9rjLPRNfky4ekQah/hovo4F/2LuI/FEMIJTUgLiErUlYufQbIf3lH6BPAUGQCVwaFO6qr EAibXF18Zj4eAvnaRxREEQX8OorH9ICOwyHanwKJgVktSJcDUCkBihAZYzFATM8hMCFwQd4uB57z UrbJx48XpyaESgZ+jZEvGM4Yo/OeAg0Dn6++0JxxViUmfEkWKDGfXKRVMkWcksshSEj/EPGCP06I U2uQvi39xUcggVBn0E/kf3qfQtsaCC9FP5RvD335nE1fjroBeI6he4hXXEaErDlh7Ch5z2+4VNZC uYheiz5qPFc2A8+FSpP/bwsSXZP0d+4GdJXcpSSN2XpNp3jyGslk6a4RBLAM2QTr9XxhB76B6YrE /ZTW/LnfbLEWEqIYN/uWgvpKO6V7nS5mPYCTxUZ5aQw+yTP0rMLrr2lchT0zq8IY1CZN65xwo0eD ABFBTVIh5bfg3jFhFjQHOJ2bd0u2r4qUmMmBUhlbaZR3fpC+a+pPb0RJpFJ8/bNoiP3aeyddkf3C 2lhWxjaW16rgrts+CiezAYC4KzSASnQcswu4OfUblkso08aDj/ZxC32gBx2qhzcp0iLZ9twuTCFT RJrY74ca7d+74XyYfo+4uPep2tbpVZNzo6hjPdypJou5fl2nzOfeLVJp1LY81PMWsspkZlLcg1ky d9c/AJcnrr+zrdn9/dLkwChszXpbSpn4ZO1uoomd1EsZdf/TC0q73RmXHX5yG/vtZ6rYPzQsMT2B nhXNK+m5mmlw1/b7rds6i/CG+QB/Wj6sU65sL+aIi9dl07IKhKXb/dZyytYxtYYD/9QLD3BXEw/t Uh9R+OceYcz0m8NL+V7AbshN0dUO7eKuu91uVpLGDJQKvRNVoybx6PZxqBNMjJKPJyUyrmSePakR t3sw+dd6njrHyjkwpUtZi1GXiSWym8qHBjdIp+vBjFq+qZSVfY1A4pUnVWSRWhapZWK9+9exU7YY vjEz/1sy4+rd6GDcljH5q5Bo5b+mbBsZk6asqnyfNMNozgpn0hQULFaQ8ytduV9JONAlxwI1IZO6 GMgS6FBZ9LKNpwtxs4RBjn2wP5ZSbfByk8SjdvOHecp93tDaSET4YOtrHYu/myQDUHUkX/j1Xg9t WWWBZEDtEJ11H5MvA6Sz1Z20sajj3cwuz+H0NnmikvFT5+wvycCKlcCV25lyt/3JcY7l6LL2wbQU uzVH6qiqesR0WAmv2MKg+miEMJGqemzmxswxXcwzm6XSye6Z6tJODrmDsa+LgdMLe6Gx6fN11+PN 18iLhFk3S6Xt3AiVwsMN6em663VieNKXYJd6ZUTd75T1VAxfS8gMXdj9ja6EQPmIEgIVbN4s47HF 2hnOXcoC69z0Qil4euk+4EwdoRRkIdrckIuruONm6/mplMtUHlFrWfamvKqkdm5MnXVnwkwvN30A W9fJ9e4HfqOefbdK+Zyd7F0tA1psn/W+fZYHY45H1gRW3PRiekkdkPyfHpa9NsPFmG3Ry2DFtJXW glx4oGrJoo9zS1Fg7hsfN+z9IFjYg1chcvpcIQJHiTGSEUW04bIWBzACGU8RBiPrAHQ8j/zf4s4P Gnk7KJ5ZQ5syqwuM70jmPK4u05bATPw2ATwDJ/5qEIVt72xnbXr1nCVYG2jLlFMkj5xTvE7Ahdba TAILLxekhp1bpWwQJxj3yy4lTyPPoQhzjCvSIAvyVOhct5eLR/tC1qtYzq3SOSelbb9zX6XFFUpO clXn4K8pKoVkRzEGec8JKk/U1rX0gvnO9ZsThVPpLWIOtNZVF4cJzZljQqjKfZbmMWn0HooNpnhC ycwSdKxhc/W51qAk7yKeWgvH94vb1NdSlnK6YyIH1kwCi9NS+BnM0Im1NayVb5PJ5PlbzV6feKud e3vibdTo2+iFV2hZTJqXX6GZ8d9LT9ETLfRBN+iRv/KiM3hkBWvyAPwtaDW8EZ+P8HOL/0OYFC+j 1e0Eb6tQogqLE2NfwctQrLgFJgpQFWUSBu9Oz6B41hRAwKcFIZRmcvrxS/L5Z8hm8/nlmTm+9gKC KOa8Cgz45CVbPG2pXCiqIzMGargZJZefOd8QokTOFVE4em21F37WgrKSZMK1slI/iVwv7JN8/HgR BYc2CqZr/Ot+yUv+EUqEKKbaJPNZoZ759J6NpEYAjv3um/Cb/J61/4PfCfLb/J4Vb/a7CX+737MG fv+vXo+OeZvXOVm/yetDdbVgbFcU0B0qFZCaMRyAuTVGCWcZJUclouIL/CBgp3yCnq7x2KY6oXkh QmHpzQ1nipyDmtAwKK9sn41l+w5YxA4I3qR3yb2FdZmSXcZdSlMNNrBVGNjIrzMl5Zs0Er6Y/d2J fU/uC5xGm/+eufkoh9+qxtUZk9at4rqlN90SHv/PAJKA2PMKZW5kc3RyZWFtDWVuZG9iag0zNCAw IG9iag08PCANL1R5cGUgL1BhZ2UgDS9QYXJlbnQgNTkgMCBSIA0vUmVzb3VyY2VzIDM1IDAgUiAN L0NvbnRlbnRzIDM2IDAgUiANL1JvdGF0ZSAtOTAgDS9NZWRpYUJveCBbIDAgMCA2MTIgNzkyIF0g DS9Dcm9wQm94IFsgMCAwIDYxMiA3OTIgXSANPj4gDWVuZG9iag0zNSAwIG9iag08PCANL1Byb2NT ZXQgWyAvUERGIC9UZXh0IF0gDS9Gb250IDw8IC9UVDIgNjYgMCBSIC9UVDQgNzAgMCBSIC9UVDgg NDcgMCBSID4+IA0vRXh0R1N0YXRlIDw8IC9HUzEgNzMgMCBSID4+IA0vQ29sb3JTcGFjZSA8PCAv Q3M1IDY1IDAgUiA+PiANPj4gDWVuZG9iag0zNiAwIG9iag08PCAvTGVuZ3RoIDI0NjQgL0ZpbHRl ciAvRmxhdGVEZWNvZGUgPj4gDXN0cmVhbQ0KSImkV8tyG0cSvOMr+tijWIymH/M62vTGWg7bwTBw ccg6UCBI0caDIYCr0N9vZvVjhgQIAdpQCJye7q6srqyqznl7tavVYqeM/NstJm//MzPqfjcx6kFN XFP6TrW1/Kl9paatrdTn5eRu8uN88nY+t9g0v5tgoisrr+SnUt6VdaNM58rOq/l6UpWVvK8IwAFm 54vJlI/WqvmXyXv9Q+HLXj8Vpi6Nvi9sVTqMqtLqXdGWtd7zVaOVqQoDu/pfysanqio+zH+ZVAnA 2NJa12Mw/0kwTGUznPEB7vqmmJoWGPfArfWymFoLW8rgdQNHrNictk3ZOuNGtkxLW3yqTTD1B011 2Ly5jQ9f1c94srCOKQdraQXOhRHA+tLrDV56nI5ACKXPoXSVcjxLDROmUW1lGX7EUU5QmXwYRwf0 w+xq9k7Nlrvdw3ajrrabxfJxX8z/nvx7PhloJc9XMxWiNLvCzC94+FtVtP0FYVW/qfcfKnU7ca4m bF0hPI2y4BIs92VvSfyPbyLzXXB3GilPGeBq0IjNIKsSn6sx70Z8r0Lc5ldFg5hcSwDGviYHbFva jg7gt2VGZfxDVFhChjpfGssAvcQV0t4jVIWx+koSagbKQca7Q/imLv0IvW2+hd61ZWuPox9gumOI /YWI3vy/iAgxUOrG0w4LlSFuulOH7Mho3Rp69hqgfwEYqfSNYRbFXEJpSy7NjlNZlxYbUCFD74j0 Xf9cYHOt/3wtZXzvEcgIY5mIJ1KmkghUcvaXCVMLon53Heso2e9csm4A9PoZPEvH93LqcIiQ+u/1 r78Whp3u6mWIENjOD9ZNczJEHXxBc/SpLRgb7f/2g3SwbgB4ltgwPhR2418W9iHrNXlgOdtXqvmw kCPOQDpxTpMOGCQfWY/JdcC6P8Z6BEqsE+cbrHeGQTuX9Wg/sE7rJ1nvPBkk643/JuvRdGY9WD/F etewPZB217xGe3uM9v5i2r35Htr7S2knzCHtzZh2d4z2/lLavbmI9v4S2r25hPb+Utq9OYP2o9Ue W3wL9RA7PFaebvEtrhNVc/FBkFI1zq5CD56NLhWqCFwjXyAYUV800cif9cQz4rinOqbBCidkpfdh AQKwnri2ZQx4ETVY4HBMjHCdy3qac7ymKrq/mty9QQR72MMmsS43Jq9BGFth7CXToT+NlbEIUcLX TdwOPC6pJeUCvpjoA8JMMKl2awnWoYet43tO9+LiYFBcFgzH59pbbhxhRDdTkML8Ks/7KrgtJjwZ yXHkbWVDIGMcsYDshkgdeomSksAZ1kXwunseRy8RQKy428th6WErh4oqzknLWAk8Dwl478P+nGcj EZnuMocM6Y4rx2NXJYLi7NHKfK3VvLz3HYi2/kyp2Ip/x9TpgUp8KWdeaMQB9lyNeAT5AoVIPGMz 3qlajgrxu/FC87CNNOjz9GHLlLGtTW3wAn3omiYG8ix9iBMx/7O2Ol8gOied6myFiGM7L/6cqRCd l/ZwpkhE1bjaDkr3WyIxHv58kegax/NeLhJzCZ+nERGgC0s44WTmzxKJKLGLmY9AmfnzVKJrzmY+ AmTmzxGKfXMu89F6Zv5coXia+hNC8QLqKeAi9W1zqVC8gHrifA/1/cXUUymeoL47qhTPp55i8Xzq +0upj2LxO6o+9nvfny0VcTlQFFTmO6UixRlyy/Y9kddJFdm2CRpkJnIOR7a9xCzIvXzprqgEucFV oj6C2iMLeF0nGSVy0nbyRZJFkm2ThSBGATloTeL3I7FIfIqu0e6OMU5SkVv6+pmYHDxsRX9y3iWt SHvtCCH4DEOVP+qhiNE4vxr7kM8YpCGPHdWmSMdRHNOVGQKV1V5UE6skBWOcs9qjxSj2KCdZqm3N 3DjiZlCjdNNntTmOJKlD3vpaSn6dxjGAQrYR+n3tgpNpBxSsTey6PL8aWUwI8EVaa/zwCABt/JKA +TS5GqOPmB77F8cj/2J6Zv/SjuRfFP3Zv8HiwPTIvwTQDl86g38j9LQ7UDb4Fykc+QfSQMngX9yR /cPXRudH/mWLCcGKYMwORoTsYEiKwcfBg2TBejpXWykCWLBmNHamI36NRd1oZDhC/wljb1NKprF8 EjkjTmRN60Yj7n0+G3bGmymNLEcZh1tGMLTIW0z8DddfPEwYLEK3fTufQ3Wp+R3a3bRnbstPJZXE zyUj43g9tY5dcCoNEZu+hIuQWa6v1WMBSKM/F53e/reYQtzrh4JO6tuCK5a7Ap5p9XvBq1px3RLN W6+4r8Va1Ly++bhawqBWH7/uw8NuT4Ba5/U36wK1raO1/ScBLCr9hNeNvoci7/Un6c44Eo5fITfm P4n/TZ/9d23wf/+pQFitXqrNkq290fsvWyI2QERH0P+oOzgMz+Bfqz/yRG2celikSyB+VMSPzTXG djTmrSJfurYcBrZjYBeTOGaaSNrnsaSJrVtmExMBvaj2wwA7n83FbeEyziPDXMggruoHBFfF7yGb OifbYThIHMUkeZEaFmXiOymCQbmg5lJosVFCO9uuESZhjTFTj0sw5BA75LaeFqhdkIiuoPdckCbi 6hty4PW6mCJpsTdMbheMf8dc4sbw+4CsQEJs7oUnFZfcFFOyuVEfA6y63W7i08ev6mYj1M3fiPO4 W+bhnkdcxHfYZJbiy66nfIl/3/FtrdV6y4ceiY0075B59BiJO2Xuqb/0Oj7udjdhxX2au9sWNYtk TYNex+l94Zl4XFMn6A3yu0U6G/1XUSr1+3bscNMNidwEjxfbUCdrlId+2jCrEZrFDctHP2w3inWj t3HiXj5cQ3Hpm8U/3NtLTSE+iKdl7RphaMrqVDeMapOGYeETv3tpdRkYuC2zKOodr+e6hSZEOZi+ Goama6U2mo5CJY986Hxx7NGrerazPBaNZaBubEh5S0t2NOLe57NhpwniL46qnn8yDq97uTvSmKoA q4PDrAYO3DA4XhKmZWnXjYiSWBFNPXDkAkd/SIuxmmGvYhthpCVTF/Gleggi0zH74Lt+p3ax9wkf 4X/cBF5XZKaX/uT01+XnwvSZxMfQgAPOdpHgUVRjIxuK9E5yAg13v2XJyEL0xOVtMLG8Vdu4bvVV PUnzfRTI7bNSaodDx1LaEx0FLZnnJHVq1jWL9CGMwq94Y2KK4gTbO6l5FcLRQXOrkOStJDl78uY2 JGaHMzjeNCqASbCcXtBhm0YPj3KV2KOWgwt7uoBYxoVP4W1O6/dgmh85Hyp1Cy0g6oE9vmOSW4rU Lo1XaT5epHk+jldZWyB1RMnjs0Paahjn+diR83wcUx/9bwB5b7h5CmVuZHN0cmVhbQ1lbmRvYmoN MzcgMCBvYmoNPDwgDS9UeXBlIC9QYWdlIA0vUGFyZW50IDU5IDAgUiANL1Jlc291cmNlcyAzOCAw IFIgDS9Db250ZW50cyAzOSAwIFIgDS9Sb3RhdGUgLTkwIA0vTWVkaWFCb3ggWyAwIDAgNjEyIDc5 MiBdIA0vQ3JvcEJveCBbIDAgMCA2MTIgNzkyIF0gDT4+IA1lbmRvYmoNMzggMCBvYmoNPDwgDS9Q cm9jU2V0IFsgL1BERiAvVGV4dCBdIA0vRm9udCA8PCAvVFQyIDY2IDAgUiAvVFQ0IDcwIDAgUiAv VFQ2IDcxIDAgUiA+PiANL0V4dEdTdGF0ZSA8PCAvR1MxIDczIDAgUiA+PiANL0NvbG9yU3BhY2Ug PDwgL0NzNSA2NSAwIFIgPj4gDT4+IA1lbmRvYmoNMzkgMCBvYmoNPDwgL0xlbmd0aCAxMTA1IC9G aWx0ZXIgL0ZsYXRlRGVjb2RlID4+IA1zdHJlYW0NCkiJfFXLbuNGELzzK/o4E4T0PPjSKfB6FxsH XkBYMadsDjQ1khlLpMOh1tApf5HvTfWQkiw4CQzI0/Oo7q6ubt7c+YwaTzr8+Sa6+bzStPWRppYi mydpSUUW/mWporgwigYXbaIPVXRTVQaPqk2EgzJRKYUfRalNspx0aZMypWofqUSFfcUO2MBp1UQx L42h6jX6TdzKNFmIg9RZosVWGpVYWCoxwssiycTIW7kgraQGrviRzLxSSv5e/RKpkwNtEmPsAkb1 MfjQypzd6XRyt6xlrAv42MJvJpyMjQEWaWznCMQGzLjIk8Jq+wZLF4zFq0xPUF8ZqsTjbj0vjvQz VgboOLJAO91AXrDgbJGkosNmiuzYEahMz1RaRZZzyQChcyqUYfrBY8hAXbhTGUcgqrtlvKuPbqB7 7w/Oy+oPBszPgGAO/E+/qE5eJvY9rL2UZOboLwRv0qQQXzkLI1zTf3fDkfoNrWtZirEmP0poQwyu 3tPL0DfO+xaEIrktbXqsSrHbyQyP+9e224Yak3fTYi9jRnfdBELroX8JbGjEl1mK8V8X6Yl6cH8p Y34O0aQFdLGSGrpx2z3QJqACTgkuMpS3DtZRxhDQ4PxhqspupLajXe89p9Su7la4psU9bYZ6z9FO skLpoGUmDoIBY3m6SKxROij7nbD+lnGhlPgVxUVYXf24g1cowdHYT+FY0UsjvjuQCAaxu2m7er5F YKIQs8HXmnpkIxUtpAMWOzp0VztXDznwyQXCp9aD+ZBxKJxbnxNC/Mwqx20vvZHm5xTyRY7SLQGq RQsuEf4eJELZTR/K2Y1stpylFR2ZLFxV9AWnC/EBZTkH1W+kVigCi6YQ4R19E2E9XdCKPj/CZD8e SWnxTZ5CvZauhXRR+rzAkDAX6ZaXKiyudGEn6Rbi45db6geq59KGYLe0d81TWNZd6/fI8Ei+36Ey 47Tt6CXMngE1ZfIfd24/zYX3Cr0owRRX7ZOKT9CmEcOAANZudM0Ysm777v/E/u/9aFKuywPTm4u5 0yTPxWea2qyGZo5O8lwcIADiVHMoKeYQRppOpidrtHDMBwsxNeB8K54v8Tk3yH1o1qVELGjSCZDn JQOGd694J/rheRpl1Q+XqJtLLucESoBVCKAQd0tqnlzDgk3FM3clZ7BnR/Dsx8F12/GJXtCg7ePu yEWwHFG9dn8e6tFdudOXsaj1lTt0NEebiqV3DWrufb1F87Vb50fI+bBb06Ojg3drjCyQ1jWodylc zTunKN60jtaXKqV2/hpcTaSME7jldNDLoxumOdfVY8t7OXqfmxa9e+QvgcG3rKY73tKsV5babNGG H5SoDCMCRzNvVM3Hy5/+S43nkXSixL7RJCOuAqLbQY0tBD+4cag7v29BNWbM2vl2wOg6U/zm5TJ8 MKeaBNp4eK5WDzfV6oHlApIPoWFKCNSiYuOR6qFhDlLxJHkct9wEh2FC/1RF/wwAL20MCQplbmRz dHJlYW0NZW5kb2JqDTQwIDAgb2JqDTw8IA0vVHlwZSAvUGFnZSANL1BhcmVudCA1OSAwIFIgDS9S ZXNvdXJjZXMgNDEgMCBSIA0vQ29udGVudHMgNDIgMCBSIA0vUm90YXRlIC05MCANL01lZGlhQm94 IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3ggWyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTQx IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYgL1RleHQgXSANL0ZvbnQgPDwgL1RUMiA2NiAwIFIg L1RUNCA3MCAwIFIgL1RUOCA0NyAwIFIgPj4gDS9FeHRHU3RhdGUgPDwgL0dTMSA3MyAwIFIgPj4g DS9Db2xvclNwYWNlIDw8IC9DczUgNjUgMCBSID4+IA0+PiANZW5kb2JqDTQyIDAgb2JqDTw8IC9M ZW5ndGggNDIwMCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PiANc3RyZWFtDQpIiZxXTXPjNha861fg CKZGHIIfILm3xLO18VRSUa112ZrkQEu0rYxMeSjKqfn32/0eKMmxx0OnXCWTIPC+ul8DeH+xL8xq b5z87Vez9/+5cuZ2P3NmY2aZj/PKlIX8K/LEzMs0MX07u5n9tJy9Xy5TLFrezPChipPcyE9i8iwu vHFVFle5Wd7PkjiR8YQO+OJrs1zN5nzMCrP8a/bJJkXkithZ8/EQOfizW4MBmyaJjifRH8uPs2S0 4tI4TbMaL8sPYsgl6dGmy9XmoonmrowzexvlcWHbaJ6mcWWNw7CPa5uLzXnp4zJz2ZktV9IWnwqn pv5LUxUWd+vw8NX8jKcU1vEpg7Vxxm2U4g3OamTRYTC3e3GEeuXHemWJyZhLARPOmzJJWeNQLFTr mEuoz480m9vbvtWHZtjsOvPjNsri1A5t32HgsT06OgFTx7U38gNgyiLOc+OzLC7T0Vl5wqIKdeuj Mi7tLnIO/x6iBFnt9lHFGq7NTeSR9C7Cksz2katZUpbCbiL8XLEamb24iliMy9iYi919VCNK/W3E XLdWe6ZBQZ2gsh/oDxOGQ/i2ubqIHGcHm5eBA3NMS1itD8qm7JRBgOtegwXgqH0JC/tontP2LeK1 rY6ZBnna1uibTP5ywEJv224VuWy0sDabLnI5xtftQ4uIEH0bRobtV6zI7TvgAPJuOtNIkMsfNKDE BSYlWaahraI57WZY3A3RvLJ9ExUo8xbj+Gc20byMpYxIPWcd9eES49bc7/g5t2AhUTiQZTXW8gXB MhRU/DcxfIfI0nG0PxkmqBcczY5eLhXAG0UgWEURSiQltsLiXbc3kkCj/rsxWcHBHXFIa032ujXr NiJDtuCttEdD4sAqmDQQXaESvZrhTqeiiusNI3H2kW48vTu7Pkilm6156JFvRX6SHAOX400gg7Wt AOrH8SaMf9bR+Dzi59T/9bDl9JyrE9Rg8xDeW1rLrFkGSi7eC1MuSV5nF+AJIyctChGcUsNmqh05 g5rvze5Bud5qgzVD+MKMw+NDOzZE24WGQAQI4KtG/u/l7KTVFO+LK6OqeHWBLx/x8KdJqCV/GZeY X82nPxKznuVFJlKewJRBy9apSUHwlGL+0w9BzauX1dwnFA0oP0WEonEu5U4JrtVDcairi2eR5iXc i3e4LLk7pBw4Ov+7yxLNSpfZKIpPXApsn+zminpzgVIWohBBH5779aUTjZ3q19do5vSJ33Nv+Xe8 5W/05l/19u3cKgHk6M5XBjtu6b/tr0I/SHKZn+wuT2A5H4mDdj0jztVLThK0MJADcY9bv1e8Fj9r v//veSp5IsRQJynT+hs7/+6FhMQeXj1nR1HRm71cRMs/JYG0ZnmDbf68Hn/maJVsP+7HyvFP9pdf KPo16jQWB6eL2p9sO/8d47DqpThiWs4aaTD+649yKilP1k/l0eb1ZcK+PXavf4Va2rO+pr+XW/Z5 twaoRzcj1urm21jTydk57wnY+bfBHt2MaL+WDVOpPFdNRHs0HuB+JQWFmzm4dCLcJ+POf8e64O1r RzD+AeD5mwH3/wTw/I2A+28D/kp3j26mAl68DHj5MuD5GwH3bwE8fyPgnoDz6Dcd8iwTkgQ1V8hf l/Msq6Ss9Sv7csC8emFLzgrHfWB0WHAr+o7DomBUL2wg03bltCpiP30nSatSlfhNW0mGQ+bpoDNh K8lgnxo+bTfJ0DJJNXk3ydgw+cTdJK2lvpN3k7SueMaYup1o+aere6j/W+U9APAGeQ8ITFT4gMBk hQ8QTFP4AMF0hQ8YTJb4AMJkxR1BeKPkjiBMl9wRhGmqO4IwVXVHECap7gjCZNUdQZgou7jppMeb Ts2s5YcCJ+3mM8jgUXLK4nhJzLPjNcfxMrgwG17VCrunzpl73F5L3EoT3NnWmxu97m4itgkvuzvO xTW3ucXN0Vn97fGb2lZfGr2pytUTa8zuUfbrVicF+wdeNTNeCvNxJi6prV4VzeXi/H5bnC64aaGx N+t1D6u80Nb4ReQ5otxDrtNxKDZmSWI5exeluKTCcckw6NjbpqNnXWBQ1xRRoH5hrjyapjOtTFub brdWs2aFleex1afYSo3trnmMCuQmnnImzKog4QxGtyyfFCdlwnzLjzORdnWMp2EtSquJetvu5c6m v2GBvryLUsxDtEDnmDDroPln4IvU+klN/YkPWeDDhoGWVlAg+PeMGGmwBCju0AwbhlTYXWc2+6iC y2tJAEUFL0x43hHNCgiZ6yYMNd2KnCgZN1QeC1e9VH6neO/356Hl6TE0p5ENd0rQ9l4wj2HFWWOu Ik+LOWmoX9puQA7CB4/Qzc0Or7XtzXDXCvoGpOcxYg79Apb6uev04woJ1hIVEne2Mze9fvhCtA66 KDjsNKKBU0u7/TomIGEn0rHntW00oL7fRHMm/ggzDNTs+J7DOBGDteo4dKN1NLt+rYvDkt4EY0PE QIwsumMc1TjF7MPD6KZvwEiEyRRAlRBEp4uGcdJNMLxC0m60tY8cOWau9RuqwN4YnuSrSi4yUwVt IYVjHqAkCRQcHEJgD4fwNMgE1ZyQhbzfRSNPSJz7cd01QnO0tKL/wn6GHhWSiDRJx1gzKR3eAbg2 23q0E97N6D14PS2n01sgezJ5DFCKKVa+hBiCdo3DwfmqfULi6kRirzXpDlJX5FTYazYC+xiSj+gK qQTrG6PcYKZKwEIW6Ocde82pW+g7zIz7jC+5KRU8GUD+eaidg3bYZG7CJnO+OaS+5uZQFCVnh83B nwQ2CwL7r4hOqCDYuAgjG38R1WwacL9gzWwrbQ02bKSRcxncq4Y1vSqVWR0iAsCXfhQv5ihWWGXM Dv0DVuL4VBqOSQU/aHj5KbxUw1u32JhKwLQhwYBguw770vVXQ7nIySLzuxXeX6puLWTXYEnfmYVo nWiVRuc0lgLfLvXjpNm/R+LB5HOkg6yGw4MGso1IlTZ+Qgp/IkUeGuUuKGhPLmRB+6lDniwtyNru KKwi9JQwlFFELLVzQlDZbfM18to55HJPbathj9om7BzUZjDSmcsP3Ms5BVTZNtRN6scTGSuPMlZr tGyFGmIr/zoqITp5Lx0jkKu8VfYRQxm3dEhUNk68ZfmcNGmFsZI8F2mrbVh2c8MynJlWEl7yqxcR ka5vdNEalXc2WOijxMrSvca4f1L40p0YFFJZCGWpfiRiHE4hOOk0W/MQBdVgUGg82RCrs5Zzeg7U E5a2XOnCHefFrkvla+HT0yWwyo4xFW5sOmZkfpPzEsKSfZ94IxyHc1rLO2mHmno5d7FuTnRcZ6IE 5v7AWMmdmsnJsc3oC7/36MXHSBon6BVK66rkec+dqJoFqjZbKBDost10etT9rOe2vZxv3kUEnKcQ PdMokVfKzp126UEM4GDQPh2/b/WgYhpzvRsGrfa2DW5EL0q7+qwHIgUWBQa1SspVCPh0agh7rrQx 6LLeydkJnaPCgHMET68ePvS31/OpHPp0pBFFY7N373iMabTw3A9WGz2zdDrTDLq6ET0qsGtTrDeq FJLlWIpg8JF8LXgaDmk8rXtx2jfSsG/gWMYtJ7dbVkaPydwDCcRnNcr8/k95tey2cQTBe75ijstA VnZnZl++KUKAHBxAiHUx4MtKZKyNGFIgqQj6jXxxqrp7H+KbMCCTu8Pqx3RXV3N2eJkdd3TJJoYI Qfb8aGJkOZ5WLi/BBKcnRoZ1Ah/yUPO0FW8YircTaZ/J9BLbasWCFAXBOSFFspkwO5oJe71kTPaT DRueJQzqmqCk3NvydUI+neusnbqnBkFnTF8tM0RqFx9D2K3dXW0ri4xQDFDR5HCVfEeSxvh6xxck 7kVF6YzadqafqeRrvU6ZWFGKhvzvXho9Im3huzoFpjNTCr580+Gzdq39XsSoKGw3b2hro2LAPel7 E9AvVydGR7MWJLL7O2cB+udtzPGvOiAoG5C/jU0GGyZTHcLU7dN2/ahciE3tUyUXR9CVbDGYoZun jg/5YGm4P+ypu5E0/KmANpUwFjOZXcvFoI3lbob6Dlbfoido1U2Zet2OqDpae9GYvkAxuT9uZA27 dY3wTJHIjM4T+6X+xQ1YgaUir6T0GvdD5IdcZki4lHq9szwxQ1t7p1JK+mHj7HhCNs6QKLFc953l +G/9+FMWMR5iRo3FWeGqwOZBh/36M1vsl/v7Cgfv/2KzVddpdPKHzZay2Ypa/mOzOf5b6xjOZIxB NNwK99ztmi3Aqr3hnJ29ZXnbXgEpgV/UgWd37JVqr/1K/ruVPH5FGUFQ7VgO8Nj3psN1uhP0tukQ ORMh+DpeGZtWHiSH3/8thpBG6MAyRac4EnDlfLCJ+3UfuscQcqVPGZ6SVpe9L190rtz2QaSShB4b ARzH5hkPcPQMsVnNmTd0FOinjEpsgFfXi1Iy1PuOR0d9Zw1k/kzfB3B1/gi4Ol/UCDZe4H280Pvi Eu/jpd4X9J6T56T/dUVQ3i0ulCrDn7CQyhlebui1mrIvROPvJO6YfPuIXpRa9R36kdJR9KG/O8x8 BzNeilkQU3OyB7Xv1Ih5jpz7uqQLuF1kHp1aVQc7NQIZtRJiyVMXk1IsYbC3h+kQT9kr5WgAF/o9 9vaTEkdRT0sxLXnFIccqes6lx5QkHcpqqNk0lgfufAgMZYtr6SLz/HsiMkkjPNkNKy8/El70NZ3p wElNPfbeEELGt7wkf5LxYmYZlvycbrsIHi0kP6U/1HPVCN7S70uuSefnH74PEvNj/keV3Dlv6Gd7 Hyubced5z8xf5H12ifcd+tneZ+d4D13he10xFvGxzNnCHrxTGkccUMq/qUI2DWua9lF2MiyNsr1Q Nol+VYVr6lmkvZ17wiaCHzuq22ZCuYUlBs8Nzonsdc3DUvW1Pb5yppVNb0N37qix8YI3Wuem03aj bWqr13Kh75o5ZT11m61Dha5DmXgUKcYb+6VrNwSyXbEZ29bm1K0sV+PzVkNdPLupei24RfIieysF Nu99tnhUmfxOwej3SEVfF0KPFSVTJxbHHHJQLIZUVFuIh0jZpKswc72HmQNKx/emz2DmgFokyYCZ 4xnEHLfUos91kIB2gvAyS/JY7XtsqmAcEo+P5xOzRxRhCKwTo0cC81ChlVDzHmbekqIe074e0E8y sy8jCZ/sFk8Rsy8kwQN2egq8EGlAcqsOyrkRuXVX4HWsnH8Fw/Df0UNhuAKD76bW+fBZB799rwcp zRhxm9LyrG/VrNb6v5GOXqMJQFNrLqgxedEVk5su6G1CNmvBeQUZBI0c8H3qHt5ltXVV6nEcNZVc B+6dJJhrJTx3J3W+EtIUijEqIMI/7Pky+TxBwXApFXg92tg7dcCc6o4sySQ+WYGcZmMaivUQW6mx MaoIjkU0Qe3VgolI0fbk3Dt96ISHMsWOyWKhpx43reyz3WNQML+26sFc4gk9uUOSdLaezQpdrPrE FngttNi7gkQgMMMx2MUzDoFvFwPLalBFz63gQVLrA9GK5BXYyWIqKF5B88R9T9ZLxyEQ6Nam/2jv 34X13FtrP5u7ZrVq/+1+3S70tFuupvZsdSXhV5hfLI6WKUtODKHW5sVi2Y2RNxtLzeY/ezed6Qeb TTqPZpLoqXuajb9/n6A88mFM/D8AYLrhNgplbmRzdHJlYW0NZW5kb2JqDTQzIDAgb2JqDTw8IA0v VHlwZSAvUGFnZSANL1BhcmVudCA1OSAwIFIgDS9SZXNvdXJjZXMgNDQgMCBSIA0vQ29udGVudHMg NDUgMCBSIA0vUm90YXRlIC05MCANL01lZGlhQm94IFsgMCAwIDYxMiA3OTIgXSANL0Nyb3BCb3gg WyAwIDAgNjEyIDc5MiBdIA0+PiANZW5kb2JqDTQ0IDAgb2JqDTw8IA0vUHJvY1NldCBbIC9QREYg L1RleHQgXSANL0ZvbnQgPDwgL1RUMiA2NiAwIFIgL1RUNCA3MCAwIFIgL1RUNiA3MSAwIFIgPj4g DS9FeHRHU3RhdGUgPDwgL0dTMSA3MyAwIFIgPj4gDS9Db2xvclNwYWNlIDw8IC9DczUgNjUgMCBS ID4+IA0+PiANZW5kb2JqDTQ1IDAgb2JqDTw8IC9MZW5ndGggODM2IC9GaWx0ZXIgL0ZsYXRlRGVj b2RlID4+IA1zdHJlYW0NCkiJhFRNj9s2EL3rV8yRLCotP0RJe0zSIN2iQINYOQU5qDLXq8IreiXZ wZ761/uGlD/aAgkMWOSQfDPz3iPv3s2O+pl0/M19dvdho2k3Z5oGymxVlA3VLn5cqSivjaLJZ4/Z 2za7a1uDQ+1jhoWmUCXFP0WlLVxFurFFU1L7nKlCxbjiBDzBattnOQ+NofZb9kW8kWVxL45Su0KL nTSqsJipwohZ1oUTC4cqQVpJDVzxM5l1pJT82v6WqXMCbQpj7D0m7S8xh1bmkk6XKd3HTua6Ro4d 8jrhZW4MsEgjXKEQFzHzuipqq+0Nlq4Zi0dOJ6hPDNXg8LhdB6/0K0YG6FiyQDvvQF+YIdl9UYoR wRLdcSJQWV6otIos9+IAoSuqlWH6wWPsQNlLM6rhCsQfy5Of6GGej36W7V8MVl3AwBq4T/9QpmoK +3/IGznWpv5G4caJFgU7EQ4hsrRHxVAn7F5pO8x9OPnpNZavAeosYdXqujxzdUO7qi+wpmwA9Vnm DnLOfqYeQxGkBfJ48uMyhLHb08NH8uP2ICuwGIZxuWakxccjT9Ex4/BylHkJcn2isv3pNs8bFqAU 9Oy7cabHMEWcmBFYw7ij5albqBvXhHSIyWReiWHm8LB5t0ERRjzQGLZ+tVrsVF871TcOczet1kXN JRh2biohPNLWL36SeY3gMxxeimEcRumwlauJA8/l9CFFYRS+Db5namhBjPkCqBaHsGc1vg3ruRUH MB7dxPE2lVw0tokKaVvXV4WSm6ONYnuwT37d+y81ubH+2u5VztTh/S3JXf9yHCK9HcjDvWJC98O8 cP+nbj9saemmnV8umv2Qxo3UAlLUEOIwhSX0YZ9v/QGawTP/0fes7e+fkzVoCYmpdKtv7ZpEBL+m LC9cIO1GWlNBIY8vG6nHt4ZWR3w14hO+UA7/bNxFWstX47vXwbr16VgvbmrOKbTGWyEwnMx+n9Kr MaTigQ37vUo8sTDy5F/WXcPE7wiek2fPl4kfFLb6Mkeyu/MqCEk4bPsKnRxn3o75emxLf57BN8sa O3FFbMO3fJbfR3589/twWsFSrnRP4Mmnc474YtOaYQWb1+m6J0jmb4xMvW+zfwQYAK+wkHEKZW5k c3RyZWFtDWVuZG9iag00NiAwIG9iag08PCANL1R5cGUgL0ZvbnQgDS9TdWJ0eXBlIC9UeXBlMSAN L0ZpcnN0Q2hhciAxIA0vTGFzdENoYXIgNiANL1dpZHRocyBbIDg5MCA4OTAgODkwIDg5MCA4OTAg ODkwIF0gDS9FbmNvZGluZyA1NCAwIFIgDS9CYXNlRm9udCAvR09DSlBGK1RUM0FFbzAwIA0vRm9u dERlc2NyaXB0b3IgNTAgMCBSIA0+PiANZW5kb2JqDTQ3IDAgb2JqDTw8IA0vVHlwZSAvRm9udCAN L1N1YnR5cGUgL1RydWVUeXBlIA0vRmlyc3RDaGFyIDMyIA0vTGFzdENoYXIgMTQ4IA0vV2lkdGhz IFsgMjc4IDAgMCAwIDAgMCAwIDAgMzMzIDMzMyAwIDAgMjc4IDMzMyAyNzggMjc4IDU1NiA1NTYg NTU2IDU1NiA1NTYgDTAgMCAwIDU1NiAwIDAgMCAwIDAgMCAwIDAgNjY3IDY2NyA3MjIgNzIyIDY2 NyA2MTEgNzc4IDcyMiAyNzggNTAwIA0wIDU1NiA4MzMgNzIyIDc3OCA2NjcgNzc4IDcyMiA2Njcg NjExIDAgMCA5NDQgMCA2NjcgMCAwIDAgMCAwIDAgDTAgNTU2IDU1NiA1MDAgNTU2IDU1NiAyNzgg NTU2IDU1NiAyMjIgMCA1MDAgMjIyIDgzMyA1NTYgNTU2IDU1NiANMCAzMzMgNTAwIDI3OCA1NTYg NTAwIDcyMiA1MDAgNTAwIDUwMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIA0wIDAgMCAwIDAg MCAwIDAgMCAwIDAgMzMzIDMzMyBdIA0vRW5jb2RpbmcgL1dpbkFuc2lFbmNvZGluZyANL0Jhc2VG b250IC9BcmlhbE1UIA0vRm9udERlc2NyaXB0b3IgNTIgMCBSIA0+PiANZW5kb2JqDTQ4IDAgb2Jq DTw8IA0vVHlwZSAvRm9udCANL1N1YnR5cGUgL1RydWVUeXBlIA0vRmlyc3RDaGFyIDMyIA0vTGFz dENoYXIgMTQ2IA0vV2lkdGhzIFsgMjc4IDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAg MCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgDTAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAg MCAwIDAgMCAwIDAgMCAwIDAgMCA5NDQgMCAwIDAgMCAwIDAgMCANMCAwIDU1NiAwIDAgMCA1NTYg MCAwIDU1NiAwIDAgMCAwIDAgNTU2IDAgMCAwIDAgNTAwIDI3OCAwIDAgNzIyIA0wIDAgMCAwIDAg MCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMjIyIF0gDS9FbmNvZGlu ZyAvV2luQW5zaUVuY29kaW5nIA0vQmFzZUZvbnQgL0FyaWFsLUl0YWxpY01UIA0vRm9udERlc2Ny aXB0b3IgNTMgMCBSIA0+PiANZW5kb2JqDTQ5IDAgb2JqDTw8IA0vVHlwZSAvRm9udCANL1N1YnR5 cGUgL1R5cGUxIA0vRW5jb2RpbmcgNTUgMCBSIA0vQmFzZUZvbnQgL1N5bWJvbCANL1RvVW5pY29k ZSA1NiAwIFIgDT4+IA1lbmRvYmoNNTAgMCBvYmoNPDwgDS9UeXBlIC9Gb250RGVzY3JpcHRvciAN L0FzY2VudCAwIA0vQ2FwSGVpZ2h0IDAgDS9EZXNjZW50IDAgDS9GbGFncyA0IA0vRm9udEJCb3gg WyAwIDAgODAwIDcyMCBdIA0vRm9udE5hbWUgL0dPQ0pQRitUVDNBRW8wMCANL0l0YWxpY0FuZ2xl IDAgDS9TdGVtViAwIA0vQ2hhclNldCAoL2Nyb3NzdGFyMi9oZXhzdGFyMi9sb3plbmdlNi90cmlz dGFyMi9wZW50YXN0YXIyL2xvemVuZ2U0KQ0vRm9udEZpbGUzIDUxIDAgUiANPj4gDWVuZG9iag01 MSAwIG9iag08PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDc4OSAvU3VidHlwZSAvVHlw ZTFDID4+IA1zdHJlYW0NCkiJtJDpSxRxGMd31Z2ZItqUtnKyX8eMXRaZiWlY6lh22aF4gLm7jraa He6y2UGGtJ3OrhMVvQiicqODLUsrTVNEPDo3K6XsIIuCChMqInrWnomaehH9A8vz4rng+X6+j1YT FKDRarUjFq9KXrY6JSIjY07SImtk5J/ZTN/EYaNcrjCqMEzXoSNVmiW7Hb9crn/FMBrC9TAt+HBY 0PMQDa3VDg1hJ82Iji+wWzdvLhXtUess2//mjdYdlpIiS0ypvfhvb7OUlIr/b6L/yaq6B3QaWmXS BGok7faA6YHBgR1BUROGQ+zgFgOYHkMycMAT4NUQesAsgyiBsBA45Bngke/BZDSxKC5EAXnkCIYj h8IiFMeiGYVuCEd+PHLACSCAyLroQaeywQBzwfkVON7XgomUEqe06JDzCeikscYXaoAaH6VTKKyh /Ubh6641fHP31d0k9R23PE8qu51Pt74tUE/Ra46Zj649UnCk8GCxjOOkieYFs5nESAsGYzSL06uR uR9HHsx9nw9D5C9S3wWvl/Heu9j3hX1t68noIFntKedj5EgpqSgthXHRcAetBgzOSZ0qk1zJcmrT NabkallrF9t0vM5zmVyorj3eIDOfpZYlyKpU/rObpQiGQT3EKgJ1oqW9vf+2t2kMhH7u+jTAPim5 u+IGWdkw7/Rk2SQZ9+c6jI7cnTnbGOToD20vzj2sZIBydpqi2PmpczBkCklYkZmdsTw9IRR5PxJv ULIMdWV1jvp9DXuuVzTKoJPenvE2M95mT9cbFkYnvsKIWgKk8eFzCGJ7be2ZV0hmbfzJCXKOZKww 7jXtNe/OK2dUxpfXeqseqSYCnG35cWzqNjGvlIj2pF2oqWRQTwPjfFHeaGdu2NzGpaxxR+F6Gym2 5pWny2lSQtWs+j9/8K/N+rKrjup9l/Z4Ks7IQEnvzj5oY7paqx9/ZAfWPRM6idA55TyOVB2IvlDg FI9io90H3ZVudZLm7F+Go1mk09OQlkm2tLQq3r/Eg7EqMYg/ZlL4Jv8Q6nWg/ylRw8Po8SO/G34L MADcd19XCmVuZHN0cmVhbQ1lbmRvYmoNNTIgMCBvYmoNPDwgDS9UeXBlIC9Gb250RGVzY3JpcHRv ciANL0FzY2VudCA5MDUgDS9DYXBIZWlnaHQgMCANL0Rlc2NlbnQgLTIxMSANL0ZsYWdzIDMyIA0v Rm9udEJCb3ggWyAtMjIyIC0zMjUgMTA3MiAxMDM3IF0gDS9Gb250TmFtZSAvQXJpYWxNVCANL0l0 YWxpY0FuZ2xlIDAgDS9TdGVtViAwIA0+PiANZW5kb2JqDTUzIDAgb2JqDTw8IA0vVHlwZSAvRm9u dERlc2NyaXB0b3IgDS9Bc2NlbnQgOTA1IA0vQ2FwSGVpZ2h0IDAgDS9EZXNjZW50IC0yMTEgDS9G bGFncyA5NiANL0ZvbnRCQm94IFsgLTI2MiAtMzI1IDEwODIgMTAyNSBdIA0vRm9udE5hbWUgL0Fy aWFsLUl0YWxpY01UIA0vSXRhbGljQW5nbGUgLTE1IA0vU3RlbVYgMCANPj4gDWVuZG9iag01NCAw IG9iag08PCANL1R5cGUgL0VuY29kaW5nIA0vRGlmZmVyZW5jZXMgWyAxIC9wZW50YXN0YXIyIC9s b3plbmdlNiAvdHJpc3RhcjIgL2xvemVuZ2U0IC9jcm9zc3RhcjIgL2hleHN0YXIyIA1dIA0+PiAN ZW5kb2JqDTU1IDAgb2JqDTw8IA0vVHlwZSAvRW5jb2RpbmcgDS9EaWZmZXJlbmNlcyBbIDEgL2Fy cm93cmlnaHQgL2Fycm93bGVmdCBdIA0+PiANZW5kb2JqDTU2IDAgb2JqDTw8IC9GaWx0ZXIgL0Zs YXRlRGVjb2RlIC9MZW5ndGggMjIwID4+IA1zdHJlYW0NCkiJVFCxbsUgDNz5Co+v6kBgqxSxvC4Z Xls1bXcCToT0MIiQIX9fIOmTOnCS73z2GX4dXgdyGfhHCmbEDLMjm3ANWzIIEy6OQEiwzuSzami8 jsCLedzXjH6gOUDfM/5ZxDWnHS7j7qdwf+6egL8ni8nRApcv8f1TiHGL8Y4eKUMHSoHFmfHrTcc3 7bHIp7Xx4lwYLK5RG0yaFoS+E6qAVIBk/2tMHo5pPsqztYIUL1KxZjugEJ1iZcRfc51WT3uEMVtK JWe7v0WsoRzh44tiiDVDfexXgAEAkPFskQplbmRzdHJlYW0NZW5kb2JqDTU3IDAgb2JqDTw8IA0v VHlwZSAvUGFnZXMgDS9LaWRzIFsgNjMgMCBSIDEgMCBSIDQgMCBSIDcgMCBSIDEwIDAgUiAxNiAw IFIgMTkgMCBSIDIyIDAgUiAyNSAwIFIgMjggMCBSIA1dIA0vQ291bnQgMTAgDS9QYXJlbnQgNTgg MCBSIA0+PiANZW5kb2JqDTU4IDAgb2JqDTw8IA0vVHlwZSAvUGFnZXMgDS9LaWRzIFsgNTcgMCBS IDU5IDAgUiBdIA0vQ291bnQgMTUgDT4+IA1lbmRvYmoNNTkgMCBvYmoNPDwgDS9UeXBlIC9QYWdl cyANL0tpZHMgWyAzMSAwIFIgMzQgMCBSIDM3IDAgUiA0MCAwIFIgNDMgMCBSIF0gDS9Db3VudCA1 IA0vUGFyZW50IDU4IDAgUiANPj4gDWVuZG9iag02MCAwIG9iag08PCANL0NyZWF0aW9uRGF0ZSAo RDoyMDAwMDgxMDE1NDQ1NSkNL1Byb2R1Y2VyIChBY3JvYmF0IERpc3RpbGxlciA0LjAgZm9yIFdp bmRvd3MpDS9Nb2REYXRlIChEOjIwMDAwODEwMTU0NjA5LTA3JzAwJykNPj4gDWVuZG9iag14cmVm DTAgNjEgDTAwMDAwMDAwMDAgNjU1MzUgZg0KMDAwMDAwNDMzNyAwMDAwMCBuDQowMDAwMDA0NDkw IDAwMDAwIG4NCjAwMDAwMDQ2NDggMDAwMDAgbg0KMDAwMDAwNjAyOSAwMDAwMCBuDQowMDAwMDA2 MTgyIDAwMDAwIG4NCjAwMDAwMDYzNDAgMDAwMDAgbg0KMDAwMDAwNzQ1MCAwMDAwMCBuDQowMDAw MDA3NjAzIDAwMDAwIG4NCjAwMDAwMDc3ODUgMDAwMDAgbg0KMDAwMDAxNTg4MSAwMDAwMCBuDQow MDAwMDE2MDM3IDAwMDAwIG4NCjAwMDAwMTYyMjMgMDAwMDAgbg0KMDAwMDAyMTIwOSAwMDAwMCBu DQowMDAwMDIxNDM4IDAwMDAwIG4NCjAwMDAwMjE1NzggMDAwMDAgbg0KMDAwMDAyMTg3OCAwMDAw MCBuDQowMDAwMDIyMDM0IDAwMDAwIG4NCjAwMDAwMjIxOTIgMDAwMDAgbg0KMDAwMDAyODYyMSAw MDAwMCBuDQowMDAwMDI4Nzc3IDAwMDAwIG4NCjAwMDAwMjg5MjQgMDAwMDAgbg0KMDAwMDAzMTky MiAwMDAwMCBuDQowMDAwMDMyMDc4IDAwMDAwIG4NCjAwMDAwMzIyMzcgMDAwMDAgbg0KMDAwMDAz MzQ4NCAwMDAwMCBuDQowMDAwMDMzNjQwIDAwMDAwIG4NCjAwMDAwMzM3ODcgMDAwMDAgbg0KMDAw MDAzOTA5MyAwMDAwMCBuDQowMDAwMDM5MjQ5IDAwMDAwIG4NCjAwMDAwMzk0MDggMDAwMDAgbg0K MDAwMDA0MDY1OCAwMDAwMCBuDQowMDAwMDQwODE0IDAwMDAwIG4NCjAwMDAwNDA5ODQgMDAwMDAg bg0KMDAwMDA0NTg4MSAwMDAwMCBuDQowMDAwMDQ2MDM3IDAwMDAwIG4NCjAwMDAwNDYxOTYgMDAw MDAgbg0KMDAwMDA0ODczNSAwMDAwMCBuDQowMDAwMDQ4ODkxIDAwMDAwIG4NCjAwMDAwNDkwNTAg MDAwMDAgbg0KMDAwMDA1MDIzMCAwMDAwMCBuDQowMDAwMDUwMzg2IDAwMDAwIG4NCjAwMDAwNTA1 NDUgMDAwMDAgbg0KMDAwMDA1NDgyMCAwMDAwMCBuDQowMDAwMDU0OTc2IDAwMDAwIG4NCjAwMDAw NTUxMzUgMDAwMDAgbg0KMDAwMDA1NjA0NSAwMDAwMCBuDQowMDAwMDU2MjMzIDAwMDAwIG4NCjAw MDAwNTY3NjQgMDAwMDAgbg0KMDAwMDA1NzE5NiAwMDAwMCBuDQowMDAwMDU3MzA2IDAwMDAwIG4N CjAwMDAwNTc1NjkgMDAwMDAgbg0KMDAwMDA1ODQ0OSAwMDAwMCBuDQowMDAwMDU4NjMwIDAwMDAw IG4NCjAwMDAwNTg4MjAgMDAwMDAgbg0KMDAwMDA1ODk0NSAwMDAwMCBuDQowMDAwMDU5MDI5IDAw MDAwIG4NCjAwMDAwNTkzMjMgMDAwMDAgbg0KMDAwMDA1OTQ2NyAwMDAwMCBuDQowMDAwMDU5NTQx IDAwMDAwIG4NCjAwMDAwNTk2NTEgMDAwMDAgbg0KdHJhaWxlcg08PA0vU2l6ZSA2MQ0vSURbPGE2 ZjVhNGJjZGE1NWU4NWFkMThmN2IzMmZmMTU3MmE4PjxhNmY1YTRiY2RhNTVlODVhZDE4ZjdiMzJm ZjE1NzJhOD5dDT4+DXN0YXJ0eHJlZg0xNzMNJSVFT0YN ------_=_NextPart_000_01C0031E.CDCC62C0-- From owner-ips@ECE.cmu.edu Fri Aug 11 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17208; 10 Aug 2000 20:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA12405 for ips-outgoing; Thu, 10 Aug 2000 19:10:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA12398 for ; Thu, 10 Aug 2000 19:10:37 -0400 (EDT) From: hufferd@us.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id SAA27512; Thu, 10 Aug 2000 18:04:03 -0500 Received: from d53mta08h.boulder.ibm.com (d53mta08h.boulder.ibm.com [9.99.142.8]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id RAA157918; Thu, 10 Aug 2000 17:10:31 -0600 Received: by d53mta08h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256937.007F4C82 ; Thu, 10 Aug 2000 17:10:26 -0600 X-Lotus-FromDomain: IBMUS To: somesh_gupta@hp.com cc: ips@ece.cmu.edu, mankin@east.isi.edu, sob@harvard.edu Message-ID: <87256937.007F4A7D.00@d53mta08h.boulder.ibm.com> Date: Thu, 10 Aug 2000 16:07:52 -0700 Subject: Re: Request to exclude FC over IP from storage over IP working group charter Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I think that Somesh stated this very well and I agree the FC over IP is an important problem but a completely different problem. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 somesh_gupta@hp.com@ece.cmu.edu on 08/10/2000 03:54:37 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu, mankin@east.isi.edu, sob@harvard.edu cc: Subject: Request to exclude FC over IP from storage over IP working group charter Scott/Allsion, I would urge you to not include FC over IP in the charter of the SCSI over IP working group. Using (TCP)/IP to transport SCSI command blocks fits very naturally as part of the SCSI Architecture Model which allows for the use of different transports to transfer SCSI commands. This include the parallel SCSI, fibre-channel (FCP), SCI, and many others including a generic packet transport mechanism. In this case (TCP)/IP acts as a mechanism to transport an application level protocol as is intended - preserving the protocol layering principles and allowing for an architecturally clean solution. I have been impressed with the momentum behind this idea and saw a number of participants at the ietf who were there for this session (including myself). This specification will really drive the convergence to IP based networks. There is significant technical and commercial push for this idea. FC over IP is a proposal (at least based on the presentation at the ietf meeting in Pittsburg) to have IP act as a tunnel between two FC islands i.e. there is a fully functional fibre-channel island A and a fully functional fibre-channel island B, and the proposal is to connect them together using an IP based network (don't know if it is to make a super island or connect them as a "router" would connect two sub-nets). I think that the problems being solved and the motivations (from a technical perspective - not just a commercial perspective) are significantly different. 1. This is putting a transport over another transport using tunneling protocols. A lot of the issues will be about making one work while satisfying the contraints and using the capabilities of the other. Probably combines the timing and recovery mechanisms of the two transports in some horrible way. 2. The naming issues are entirely different. In storage over IP, each of the partcipating entities is an IP entity. In FC over IP, the participating entities are actually FC entities and will have some mechanism of figuring out that the other FC entity lives across a tunnel avaliable on the local network. If the proposal is to create a larger net, then again the FC port addresses will have to partitioned in some interesting way. 3. How would arbitrated loop work in such an environment of creating a large net, or will it connect only Fabric ports together. 4. Security issues are again completely different as the end nodes are really not IP entities. 5. The protocol encapsulation is also completely different. In Summary, putting scsi over ip is architecturally very clean and accomodated by the SAM architecture as well as networking layered architecture. The end systems communicating in this case are using TCP/IP to transfer application (SCSI) PDUs and the problems/solutions have a very good framework to work with. FC tunnels through IP is a commerically important problem to solve but is a completely different problem. I would like to keep both of them seperate so that each of them can be successful and provide useful specifications. Regards, Somesh Gupta Project Manager/Architect Storage over IP (for HP-UX servers) Hewlett-Packard From owner-ips@ECE.cmu.edu Fri Aug 11 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17151; 10 Aug 2000 20:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA11921 for ips-outgoing; Thu, 10 Aug 2000 18:54:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA11916 for ; Thu, 10 Aug 2000 18:54:50 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel1.hp.com (Postfix) with ESMTP id E0FCC12A3; Thu, 10 Aug 2000 15:54:49 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id PAA28773; Thu, 10 Aug 2000 15:54:49 -0700 (PDT) X-OpenMail-Hops: 1 Date: Thu, 10 Aug 2000 15:54:37 -0700 Message-Id: Subject: Request to exclude FC over IP from storage over IP working group charter MIME-Version: 1.0 To: ips@ece.cmu.edu, mankin@east.isi.edu, sob@harvard.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Scott/Allsion, I would urge you to not include FC over IP in the charter of the SCSI over IP working group. Using (TCP)/IP to transport SCSI command blocks fits very naturally as part of the SCSI Architecture Model which allows for the use of different transports to transfer SCSI commands. This include the parallel SCSI, fibre-channel (FCP), SCI, and many others including a generic packet transport mechanism. In this case (TCP)/IP acts as a mechanism to transport an application level protocol as is intended - preserving the protocol layering principles and allowing for an architecturally clean solution. I have been impressed with the momentum behind this idea and saw a number of participants at the ietf who were there for this session (including myself). This specification will really drive the convergence to IP based networks. There is significant technical and commercial push for this idea. FC over IP is a proposal (at least based on the presentation at the ietf meeting in Pittsburg) to have IP act as a tunnel between two FC islands i.e. there is a fully functional fibre-channel island A and a fully functional fibre-channel island B, and the proposal is to connect them together using an IP based network (don't know if it is to make a super island or connect them as a "router" would connect two sub-nets). I think that the problems being solved and the motivations (from a technical perspective - not just a commercial perspective) are significantly different. 1. This is putting a transport over another transport using tunneling protocols. A lot of the issues will be about making one work while satisfying the contraints and using the capabilities of the other. Probably combines the timing and recovery mechanisms of the two transports in some horrible way. 2. The naming issues are entirely different. In storage over IP, each of the partcipating entities is an IP entity. In FC over IP, the participating entities are actually FC entities and will have some mechanism of figuring out that the other FC entity lives across a tunnel avaliable on the local network. If the proposal is to create a larger net, then again the FC port addresses will have to partitioned in some interesting way. 3. How would arbitrated loop work in such an environment of creating a large net, or will it connect only Fabric ports together. 4. Security issues are again completely different as the end nodes are really not IP entities. 5. The protocol encapsulation is also completely different. In Summary, putting scsi over ip is architecturally very clean and accomodated by the SAM architecture as well as networking layered architecture. The end systems communicating in this case are using TCP/IP to transfer application (SCSI) PDUs and the problems/solutions have a very good framework to work with. FC tunnels through IP is a commerically important problem to solve but is a completely different problem. I would like to keep both of them seperate so that each of them can be successful and provide useful specifications. Regards, Somesh Gupta Project Manager/Architect Storage over IP (for HP-UX servers) Hewlett-Packard From owner-ips@ECE.cmu.edu Fri Aug 11 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17566; 10 Aug 2000 21:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA14713 for ips-outgoing; Thu, 10 Aug 2000 20:38:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA14707 for ; Thu, 10 Aug 2000 20:38:24 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id UAA07018; Thu, 10 Aug 2000 20:38:01 -0400 (EDT) Date: Thu, 10 Aug 2000 20:38:01 -0400 (EDT) From: Scott Bradner Message-Id: <200008110038.UAA07018@newdev.harvard.edu> To: somesh_gupta@hp.com Subject: Re: Request to exclude FC over IP from storage over IP working group charter Cc: ips@ece.cmu.edu, mankin@east.isi.edu Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh sez: > I would urge you to not include FC over IP in the charter of > the SCSI over IP working group. there is no option for FC over IP to ignore the same set of network issues that the ips working group will ahev to address - that is the issues inherent in real-world IP networks, congestion control, security, etc. (I do not see that the security issues are all that different.) It would be possible to fire up another working group to replicate dealing with those issues for FC over IP but it does seem like a significant waste of effort in addition it seems to me that assuming that the only possible FC over IP model is the group-of-devices-behind-a-gateway is short sighted - I would fully expect that vendors would quickly move the "gateway" into the drive so that individual drives could be connected and this looks quite like the rest of the work that ips is supposed to work on. scott From owner-ips@ECE.cmu.edu Fri Aug 11 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17571; 10 Aug 2000 21:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA14974 for ips-outgoing; Thu, 10 Aug 2000 20:50:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA14970 for ; Thu, 10 Aug 2000 20:50:07 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Thu, 10 Aug 2000 20:49:31 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100E96@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: DRAFT Pittsburgh ips Minutes Date: Thu, 10 Aug 2000 20:49:31 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Here are the draft minutes from the ips meeting in Pittsburgh. Many thanks to Costa and Howard for taking the notes, and especially to Costa for doing most of the editing that resulted in these minutes. Send corrections and clarifications directly to me or to the list. Also, would everyone who presented in Pittsburgh send me their slides for inclusion in the proceedings, unless they've already been sent me or posted to the list? I need PowerPoint or Postscript. Netiquette is generally to put large documents (like slides) on a web site and send a URL to the list rather than sending directly to the list. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- IP Storage BOF minutes - DRAFT Summer IETF 2000 Pittsburgh, PA Wednesday, August 2, 2000 --------------------------- Co-chairs: Steve Bellovin (smb@research.att.com) and David Black (black_david@emc.com) Area directors: Allison Mankin (mankin@isi.edu) and Scott Bradner (sob@harvard.edu) Attendance: ~100 people Agenda bashing - David Black, co-chair -------------- There were no comments on the agenda. Area director - Scott Bradner, Transport AD ------------- Intellectual property rights: If there is intellectual property in anything discussed in the WG, the person discussing it is responsible for saying that there is intellectual property involved. If what someone says is going to be constrained by intellectual property rights, they shouldn't say it, especially if they're not prepared to describe the resulting restrictions. A lot of work on iSCSI has been done in a very focused and very dedicated design team. From now on, this is a WG effort. Most of the work should be done on the WG's mailing list in order to: - minimize interim meetings - minimize phone calls - discuss issues on a common mailing list accessible to all Charter status and bashing - David Black, co-chair -------------------------- Concerns were raised about emphasis of TCP as the protocol. The charter should not and currently does not limit itself to TCP. Instead, it states a requirement for congestion control. Congestion control without TCP is possible, the ECM working group and SCTP are examples. Scott Bradner reminded the group that the charter comes from the IESG and IAB. This group is only suggesting a charter to the IESG/IAB. It was agreed that all specific references to Gigabit Ethernet should be removed from the charter (editor's note: there weren't any). The question was raised about why FC/IP may be worked on by ips. The answers were: similar application domains, same people interested in both technologies, maybe even similar network traffic patterns. The question was raised of how a charter and a requirements document differ. Scott Bradner answered that a charter is commands from the IESG to the working group, i.e., it's the general structure of what the working group is going to do and what it won't do. A requirements document lays out requirements for the area of investigations. It is expected to be longer, it is expected to go into detail, and it doesn't prescribe a specific solution. iSCSI requirements - Randy Haagens, Hewlett-Packard ------------------ iSCSI is a native SCSI transport on IP, not FCP on top of IP. TCP may be deficient for iSCSI use in a couple of areas. The TCP reassembly buffer is inefficient at higher speeds. The 16-bit checksum seems insufficient. These problems need to be addressed. A concern was raised that using multiple TCP connections adds a lot of complexity to the protocol that is not present in other IETF protocols, and that this is premature optimization. Randy responded that storage bandwidth needs (esp. at high end) will always outrun speeds on a single link, thus the need to use multiple links through the fabric (which multiple TCP connection enables). There was also a brief discussion of whether to use a TCP connection per LUN - the primary reason not to is scalability concerns for large storage controllers with lots of LUNs. Suggestions were made by a number of people that the requirements document should focus on the required functionality, not the required implementation. The requirements presentation spent time talking about the SMU concept from the SCSI Architecture Manual-2. Ralph Weber, secretary of T10, and technical editor of SAM-2, pointed out that SAM-2 is changing significantly in that area. T10 is also working on lifting the 64-bit limit on the size of target names. The need for multiple connections to a single LUN was questioned, since a LUN will not be able to use more than one network link (a gigabit link). Several people pointed out that this is not true, since many LUNs are backed by RAM, which can source data at full link rates. An issue was raised about whether gigabit performance should be required. It was agreed that the requirements should allow a gigabit or more, but not require it, and it was noted that the charter explicitly mentions small devices. ------------------ Question for consensus: All we have available is the 16-bit checksum. looking for direction on whether 16-bit checksum is adequate, or should be requirements that there will be strong checksum? Consensus: Need to investigate checksums stronger than 16-bits. SCTP was noted as having a 32-bit checksum. iSCSI spec - Julian Satran, IBM Research - Haifa ---------- General overview and summary of iSCSI. See the document for details. The issue of using multiple connections came up again. After some discussion, it was agreed to take this issue to the list for discussion and resolution. An issue was raised about whether iSCSI's error recovery on dropped TCP connections is necessary, as some other have no such error recovery scheme. FCP (the Fibre Channel encapsulation of SCSI) has discovered the hard way that it needs error recovery for dropped frames, which is now part of the FCP-2 effort. On the other hand, a parallel SCSI bus makes no attempt to recover from breaking the physical connection. It was pointed out that since iSCSI uses TCP, it has fewer cases to deal with than FCP, since it must only deal with connection drops, not packet loss. More discussion on the list is needed on error recovery, including scenarios that may require the target to drop data that TCP has successfully delivered (e.g., where the TCP connection breaks before the ACK or response is delivered and hence the initiator retries the operation via some other connection). There was some confusion on SCSI's ordering requirements. SCSI requires commands to be delivered to the target in the same order as the initiator issued them. However, the order in which the target executes the commands is also controlled by the queuing attributes on the device and command. There is an inconsistent use of words like should, may, and can in the document. For example, the ping response appears to be optional as currently worded. This is wrong and needs to be corrected. In general, the authors of the document should review RFC-2119 for definitions of terms like MUST, SHOULD, and MAY and use them appropriately to clarify the requirements on implementations. Security Discussion - led by Steve Bellovin, co-chair ------------------- The discussion started by asking about the need for encryption: It was generally stated by the audience that authentication and integrity seem most important. Speed is also a requirement, as initial link speeds of a gigabit are envisioned. Steve Bellovin noted that encryption generally runs faster than authentication algorithms and that there were encryption algorithms doing a gigabit in hardware already. It was noted that it is difficult to turn on IPSec in the middle of a connection. Consensus call -------------- There was a consensus call eventually. Should encryption be mandatory? The consensus in the room is that encryption of the stream should not be mandatory. A lot of people were concerned about the overhead it would imply on cheap devices. There was some discussion that secure negotiation was highly desirable, to prevent a man-in-the-middle from causing the security negotiation to result in no security. It was noted that most application of iSCSI expect a secure interconnect. Also, secure interconnects are least likely to be available at the low end, where people have the least ability to pay for a secure network. ------- Steve Bellovin asked the group about authentication. "Whom are you authenticating? Is it bi-directional or uni-directional? How many parties does the SCSI device have to authenticate. How do you administer this?" It was suggested that the host OS be authenticated and that the authentication should be bi-directional. Web servers could be used for administration. It was pointed out that the web server doesn't scale to large installations and that LDAP could be used to configure a group of machines. It was also pointed out that disks may not have room for a large number of keys and ACLs, which may necessitate the use of an authentication server. At the other end of the spectrum, it was observed that storage arrays are in many cases more powerful than the servers they connect to. It was decided that it is feasible have good random numbers from disks. Steve Bellovin asked if we can do public key calculations at login time? The general consensus is that even most disks have enough CPU power to do this at login time. A hard disk has about as much CPU as a cell phone, which is capable of doing public key calculations. The issue of providing customer support for badly configured security mechanism came up. How do you deal with a user that has lost his keys? Add back-door passwords? There were no good answers in the meeting. Consensus call -------------- Should public keys be mandatory for login authentication? (About half of the people voted for and half against) Scott Bradner suggested that this issue be remitted to the mailing list. Steve Bellovin pointed out that it's much harder to do something for security for UDP than TCP. TCP is easier because there's a steady, in-order context. For UDP, one have to do something like IPSec. Other parting words: There's no confidentiality without integrity checking. A separate session key is probably needed for each TCP connection. For some choices of crypto algorithms (e.g. RC4), it is important to make sure that you NEVER reuse the same key, as key reuse enables some relatively easy attacks on the crypto. ---------------- There was no iSCSI MIB discussion due to insufficient time. FC-over-IP - Elizabeth Rodriguez, Lucent ------------------ It has not been decided where FC-over-IP will be worked on in the IETF. The ipfc WG's time slot will be used to discuss FC-over-IP, but it is not an official work item of the ipfc WG. FC-over-IP is a gateway-to-gateway protocol. Multiple gateways can talk amongst themselves. This was described as complementary to iSCSI. See the draft for current technical details. Scott Bradner asked for input over about where the FC-over-IP work should be done. Send Scott mail privately (sob@harvard.edu) or on the ips mailing list. There was some discussion of a 10km limit in FibreChannel. Standard FibreChannel has a 10km limit on a single link based on emitter and detector characteristics. The actual protocol timeouts are several seconds which should go allow the protocol to go much further. One important timeout is R_A_TOV (resource allocation time-out value), maximum packet lifetime in a Fibre Channel fabric, which is in the seconds. Default time-out values in fibre-channel fabrics are between 1-10 seconds. Should be good for fairly long hauls. SEP - Paul von Stamwitz, Adaptec ---------------- SEP work is progressing. The one LUN per connection limitation has been removed and the packet format is being changed to resemble packetized SCSI. iSCSI/VI - Constantine Sapuntzakis, Cisco --------------------- This was a quick presentation to set up a bar/dinner BOF for further discussion. Protocol stack overhead is too high for IP-based protocols. To reduce the number of copies/page flips, the NIC must place data in the correct place in memory the first time. One way for the NIC to do this is to parse the upper-layer protocol. Another way is to add a generic RDMA protocol which works with multiple upper-layer protocols. VI is one good way of doing an RDMA abstraction. By running iSCSI over VI, we can leverage this generic infrastructure. From owner-ips@ECE.cmu.edu Fri Aug 11 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17556; 10 Aug 2000 21:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA14615 for ips-outgoing; Thu, 10 Aug 2000 20:34:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA14610 for ; Thu, 10 Aug 2000 20:34:35 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id RAA02605 for ; Thu, 10 Aug 2000 17:30:20 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id RAA30173 for ; Thu, 10 Aug 2000 17:27:46 -0700 From: "Douglas Otis" To: "Ips" Subject: Adding LUN as a field within the command structure. Date: Thu, 10 Aug 2000 17:36:26 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk I vote no. If you are making hardware to process TCP, this hardware is able to handle more than a few connections. There will *always* be a port sort operation to find the connection and associated state information. It is not more effective to require an additional sort to locate LUN as a second tier of state information. The ability of the client to index the entire volume of devices possible is remote and will be a larger constraint. The point of aggregation, gateway, which mandates this replicated operation by means of inclusion of LUN indicates the inability to scale this design due to this architectural bottle neck. The data path from the target to the point of aggregation is not guaranteed and, as this is an encapsulation, even error handling is not benefited by TCP as there can still be delivery errors. Is the desire to create such a constriction due to a lack of ability to map drives across IPs and ports? What rational justifies aggregation into something that does not scale and is not reliable? This design does not provide a clean separation between protocol layers as desired. Doug From owner-ips@ECE.cmu.edu Fri Aug 11 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17561; 10 Aug 2000 21:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA15449 for ips-outgoing; Thu, 10 Aug 2000 21:08:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA15441 for ; Thu, 10 Aug 2000 21:08:35 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id SAA02618; Thu, 10 Aug 2000 18:04:23 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id SAA30264; Thu, 10 Aug 2000 18:01:45 -0700 From: "Douglas Otis" To: , , , Subject: RE: Request to exclude FC over IP from storage over IP working group charter Date: Thu, 10 Aug 2000 18:10:25 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Should this FC over IP group wish to use IP as a means of encapsulation, I would like to understand the rational behind exclusion. There is nothing to suggest that iSCSI will not suffer from errors injected between the controller and target. I do not see iSCSI as the panacea you do. They would bring a greater level of expertise to these problems. As iSCSI advocates dedicated hardware, what prevents this hardware from implementing a tunnel end-point at the client? In fact, that is what iSCSI is. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > somesh_gupta@hp.com > Sent: Thursday, August 10, 2000 3:55 PM > To: ips@ece.cmu.edu; mankin@east.isi.edu; sob@harvard.edu > Subject: Request to exclude FC over IP from storage over IP working > group charter > > > Scott/Allsion, > > I would urge you to not include FC over IP in the charter of > the SCSI over IP working group. > > Using (TCP)/IP to transport SCSI command blocks fits very > naturally as part of the SCSI Architecture Model which allows > for the use of different transports to transfer SCSI > commands. This include the parallel SCSI, fibre-channel (FCP), > SCI, and many others including a generic packet transport > mechanism. > > In this case (TCP)/IP acts as a mechanism to transport an > application level protocol as is intended - preserving the > protocol layering principles and allowing for an architecturally > clean solution. > > I have been impressed with the momentum behind this idea and saw > a number of participants at the ietf who were there for this > session (including myself). This specification will really drive > the convergence to IP based networks. There is significant > technical and commercial push for this idea. > > FC over IP is a proposal (at least based on the presentation at > the ietf meeting in Pittsburg) to have IP act as a tunnel > between two FC islands i.e. there is a fully functional > fibre-channel island A and a fully functional fibre-channel > island B, and the proposal is to connect them together using > an IP based network (don't know if it is to make a super island > or connect them as a "router" would connect two sub-nets). > > I think that the problems being solved and the motivations (from > a technical perspective - not just a commercial perspective) > are significantly different. > > 1. This is putting a transport over another transport using > tunneling protocols. A lot of the issues will be about making > one work while satisfying the contraints and using the > capabilities of the other. Probably combines the timing and > recovery mechanisms of the two transports in some horrible way. You will find write data both solicited and unsolicited equally onerous. You could do better with control within the gateway and actually provide a cleaner solution. As it is now, even acknowledged data may be requested to be resent and errors will still plague the transport layer using the gateway model. iSCSI is not as clean or clear as you think. > 2. The naming issues are entirely different. In storage over IP, > each of the partcipating entities is an IP entity. The debate about LUN within the iSCSI header speaks differently. There would not be an IP entity but rather a gateway of some sort. This gateway is likely to have Fibre-Channel on one side as example and will carry very limited state information. > In FC over IP, > the participating entities are actually FC entities and will have > some mechanism of figuring out that the other FC entity lives > across a tunnel available on the local network. If the proposal > is to create a larger net, then again the FC port addresses will > have to partitioned in some interesting way. > > 3. How would arbitrated loop work in such an environment of creating > a large net, or will it connect only Fabric ports together. You could exclude AL if you wished. > 4. Security issues are again completely different as the end nodes are > really not IP entities. This is true for iSCSI. This is especially true if you consider the fact that the iSCSI gateway can not handle separate sessions for each device! > 5. The protocol encapsulation is also completely different. Is there something within the charter that excludes valid concepts because they are different? > In Summary, putting scsi over ip is architecturally very clean and > accomodated by the SAM architecture as well as networking layered > architecture. The end systems communicating in this case are using > TCP/IP to transfer application (SCSI) PDUs and the problems/solutions > have a very good framework to work with. > > FC tunnels through IP is a commerically important problem to solve > but is a completely different problem. > > I would like to keep both of them seperate so that each of them > can be successful and provide useful specifications. I see little difference between what is being called iSCSI and the more immediately relevant FC over IP that provides both directions of translation. You could extract iSCSI from one-half of this tunnel. It will more likely work and be understood. Doug From owner-ips@ECE.cmu.edu Fri Aug 11 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17582; 10 Aug 2000 21:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA16011 for ips-outgoing; Thu, 10 Aug 2000 21:28:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA16007 for ; Thu, 10 Aug 2000 21:28:40 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by atlrel2.hp.com (Postfix) with ESMTP id E2CAFE6D; Thu, 10 Aug 2000 21:28:39 -0400 (EDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id SAA15782; Thu, 10 Aug 2000 18:28:39 -0700 (PDT) X-OpenMail-Hops: 1 Date: Thu, 10 Aug 2000 18:28:23 -0700 Message-Id: Subject: RE: Request to exclude FC over IP from storage over IP working group charter MIME-Version: 1.0 To: somesh_gupta@hp.com, sob@harvard.edu Cc: ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Why would they do that and not just do a native implementation? This would require them to pay the price of two seperate protocols, and the bridging between them to get one (and having to deal with all loop issues etc). Also what you are mentioning is still a bridge (in a seperate device or in a chip in the drive). The Lucent proposal was for a tunnel. Storage over IP should not be looked as a bridge between TCP/IP and FC or parallel SCSI domains. It is a native transport for SCSI commands. There will exist bridges (externel or for some reason embedded) to bridge these 3 different transport protocols. Somesh > -----Original Message----- > From: sob@harvard.edu [mailto:sob@harvard.edu] > Sent: Thursday, August 10, 2000 6:16 PM > To: somesh_gupta@hp.com > Cc: ips@ece.cmu.edu > Subject: RE: Request to exclude FC over IP from storage over > IP working > group charter > > > > You are talking about (I am making an assumption here so > correct me if > > this is not the case) a bridge here (which typically would > mean storage > > on IP on one-side and FCP on the other). > > that is not correct > I assume that vendors will innovate and a logical innovation is to > move the FC/IP gateway into the drive itself > > Scott > From owner-ips@ECE.cmu.edu Fri Aug 11 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17730; 10 Aug 2000 22:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA16715 for ips-outgoing; Thu, 10 Aug 2000 21:53:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA16703 for ; Thu, 10 Aug 2000 21:53:22 -0400 (EDT) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA08346; Thu, 10 Aug 2000 18:52:50 -0700 (PDT) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA08468; Thu, 10 Aug 2000 18:52:49 -0700 (PDT) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Thu, 10 Aug 2000 18:52:35 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C05B31C93@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: "'Douglas Otis'" Cc: Ips Subject: RE: Adding LUN as a field within the command structure. Date: Thu, 10 Aug 2000 18:52:27 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug wrote: > I vote no. > If you are making hardware to process TCP, this hardware is able to handle > more than a few connections. There will *always* be a port sort operation > to find the connection and associated state information. It is not more > effective to require an additional sort to locate LUN as a second tier of > state information. There is another second tier of state information (or is it tertiary?) that would require sorting, and that is the tag. > The ability of the client to index the entire volume of > devices possible is remote and will be a larger constraint. If you map a group of LUNs to a connection, then the indexing is for that group only and not the entire volume. The feedback we received is that a connection/LUN is desirable, but making it a requirement is too restrictive. -Paul From owner-ips@ECE.cmu.edu Fri Aug 11 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17737; 10 Aug 2000 22:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA15662 for ips-outgoing; Thu, 10 Aug 2000 21:16:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA15657 for ; Thu, 10 Aug 2000 21:16:15 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id VAA07227; Thu, 10 Aug 2000 21:16:14 -0400 (EDT) Date: Thu, 10 Aug 2000 21:16:14 -0400 (EDT) From: Scott Bradner Message-Id: <200008110116.VAA07227@newdev.harvard.edu> To: somesh_gupta@hp.com Subject: RE: Request to exclude FC over IP from storage over IP working group charter Cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Precedence: bulk > You are talking about (I am making an assumption here so correct me if > this is not the case) a bridge here (which typically would mean storage > on IP on one-side and FCP on the other). that is not correct I assume that vendors will innovate and a logical innovation is to move the FC/IP gateway into the drive itself Scott From owner-ips@ECE.cmu.edu Fri Aug 11 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17728; 10 Aug 2000 22:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA15694 for ips-outgoing; Thu, 10 Aug 2000 21:17:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA15688 for ; Thu, 10 Aug 2000 21:17:11 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id VAA07234; Thu, 10 Aug 2000 21:17:10 -0400 (EDT) Date: Thu, 10 Aug 2000 21:17:10 -0400 (EDT) From: Scott Bradner Message-Id: <200008110117.VAA07234@newdev.harvard.edu> To: somesh_gupta@hp.com Subject: RE: Request to exclude FC over IP from storage over IP working group charter Cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Precedence: bulk > Also since > the end points are FC end points in FC over IP and they already have > protocols in place, the security issues (and > all others) and their solutions will be completely different. Internet compatable congestion control? Scott From owner-ips@ECE.cmu.edu Fri Aug 11 02:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17750; 10 Aug 2000 22:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA15588 for ips-outgoing; Thu, 10 Aug 2000 21:13:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA15583 for ; Thu, 10 Aug 2000 21:13:48 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel1.hp.com (Postfix) with ESMTP id 48ACD1067; Thu, 10 Aug 2000 18:13:47 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id SAA24106; Thu, 10 Aug 2000 18:13:46 -0700 (PDT) X-OpenMail-Hops: 1 Date: Thu, 10 Aug 2000 18:13:35 -0700 Message-Id: Subject: RE: Request to exclude FC over IP from storage over IP working group charter MIME-Version: 1.0 To: somesh_gupta@hp.com, sob@harvard.edu Cc: ips@ece.cmu.edu, mankin@east.isi.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: sob@harvard.edu [mailto:sob@harvard.edu] > Sent: Thursday, August 10, 2000 5:38 PM > To: somesh_gupta@hp.com > Cc: ips@ece.cmu.edu; mankin@east.isi.edu > Subject: Re: Request to exclude FC over IP from storage over > IP working > group charter > > > Somesh sez: > > I would urge you to not include FC over IP in the charter of > > the SCSI over IP working group. > > there is no option for FC over IP to ignore the same set of network > issues that the ips working group will ahev to address - that is the > issues inherent in real-world IP networks, congestion control, > security, etc. (I do not see that the security issues are all that > different.) I think all protocols at various levels for all sorts of applications using IP should address congestion control, security etc. Also since the end points are FC end points in FC over IP and they already have protocols in place, the security issues (and all others) and their solutions will be completely different. > > It would be possible to fire up another working group to replicate > dealing with those issues for FC over IP but it does seem like a > significant waste of effort > > in addition it seems to me that assuming that the only possible > FC over IP model is the group-of-devices-behind-a-gateway is > short sighted - I would fully expect that vendors would quickly > move the "gateway" into the drive so that individual drives > could be connected and this looks quite like the rest of the work that > ips is supposed to work on. I think there is some misunderstanding here between what I thought was presented by the Lucent presentor at the ietf and what I think you are saying. The presentation at the ietf addressed was regarding an IP tunnel between SAN islands which is what I am making all my points about. You are talking about (I am making an assumption here so correct me if this is not the case) a bridge here (which typically would mean storage on IP on one-side and FCP on the other). Again, I see all 3 as important, commercially and technically. I still think combining FC over IP in the working group with Storage over IP would waste more time than keeping them seperate. Keeping them seperate would let the interested parties focus more precisely on their set of requirements and problems and provide a relevant spec quickly. In my opinion storage over IP is very much TCP/IP centric as it is another important application over TCP/IP. FC over IP as discussed is much more FC centric using IP as a tunnel. The kind of expertise required and the people interested would be more different than the same. I know that it will be additional burden on some senior people in the ietf organization to track 2 working groups instead of 1. However, I think the participants at the grunt level (and the motivations and the expertise etc. etc) are so different that 2 is better than 1 (or so it seems to me). > > scott > Do appreciate your consideration. Somesh From owner-ips@ECE.cmu.edu Fri Aug 11 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18084; 10 Aug 2000 23:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA18033 for ips-outgoing; Thu, 10 Aug 2000 22:43:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA18027 for ; Thu, 10 Aug 2000 22:43:08 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id WAA07521; Thu, 10 Aug 2000 22:43:07 -0400 (EDT) Date: Thu, 10 Aug 2000 22:43:07 -0400 (EDT) From: Scott Bradner Message-Id: <200008110243.WAA07521@newdev.harvard.edu> To: paulv@corp.adaptec.com, sob@harvard.edu, somesh_gupta@hp.com Subject: RE: Request to exclude FC over IP from storage over IP working gr oup charter Cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Precedence: bulk > Are you suggesting (dare I ask...) > FCP/IP = IPS? no but I am sugesting that vendors will not limit their FC over IP to the case where there is a seperate gateway vendors are inventive beasts Scott From owner-ips@ECE.cmu.edu Fri Aug 11 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18079; 10 Aug 2000 23:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA18234 for ips-outgoing; Thu, 10 Aug 2000 22:50:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA18229 for ; Thu, 10 Aug 2000 22:50:07 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0FZ300A3YWLQLH@mta6.snfc21.pbi.net> for IPS@ECE.CMU.edu; Thu, 10 Aug 2000 19:37:03 -0700 (PDT) Date: Thu, 10 Aug 2000 19:36:26 -0700 From: Peter Johansson Subject: RE: Towards Consensus on TCP Connections In-reply-to: <8133266FE373D11190CD00805FA768BF02EEA059@shrcmsg1.tdh.qntm.com> X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000810193334.00bcee30@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 06:04 AM 8/10/00, Stephen Byan wrote: >I believe the only low-cost link technology available to us is 1 Gb/s >ethernet over cat5 UTP Has the WG examined gigabit Serial Bus, IEEE P1394b? I'm not suggesting you take the time and energy, now, to do this on the reflector. I understand that there MIGHT be other considerations that affect preferred medium choice for prototype implementations. But if you take the time to independently inspect P1394b, I think you might find its cost / performance characteristics suitable. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Fri Aug 11 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18097; 10 Aug 2000 23:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA17860 for ips-outgoing; Thu, 10 Aug 2000 22:37:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA17855 for ; Thu, 10 Aug 2000 22:37:18 -0400 (EDT) Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id TAA11506; Thu, 10 Aug 2000 19:36:42 -0700 (PDT) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by kingdom.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id TAA09274; Thu, 10 Aug 2000 19:36:41 -0700 (PDT) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Thu, 10 Aug 2000 19:36:27 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C05B31C94@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: "'Scott Bradner'" , somesh_gupta@hp.com Cc: ips@ece.cmu.edu Subject: RE: Request to exclude FC over IP from storage over IP working gr oup charter Date: Thu, 10 Aug 2000 19:36:23 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk [somesh] > > You are talking about (I am making an assumption here so correct me if > > this is not the case) a bridge here (which typically would mean storage > > on IP on one-side and FCP on the other). > [Scott] > that is not correct > I assume that vendors will innovate and a logical innovation is to > move the FC/IP gateway into the drive itself > Scott, I'm not sure where this is heading. You make an interesting point, one that was not brought up at the Pittsburgh meeting (which only went as far as transparent gateways bridging multiple FC subnets over IP.) If I understand, you can envision a native FC HBA carrying on a conversation, through an FC/IP gateway, with a drive directly attached to an IP network. Since the drive is IP capable, it makes sense that an IPS HBA should be able to converse with it as well. Let's close the loop. Why shouldn't the IPS HBA not be capable of conversing with a native FC drive through the gateway? Are you suggesting (dare I ask...) FCP/IP = IPS? -Paul From owner-ips@ECE.cmu.edu Fri Aug 11 10:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19970; 11 Aug 2000 6:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA26514 for ips-outgoing; Fri, 11 Aug 2000 04:29:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA26509 for ; Fri, 11 Aug 2000 04:29:27 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA27058 for ; Fri, 11 Aug 2000 10:28:54 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id KAA24628 for ; Fri, 11 Aug 2000 10:28:50 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256938.002E939E ; Fri, 11 Aug 2000 10:28:44 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 11 Aug 2000 10:25:35 +0300 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Very good point. Yest we considered firewire and where impressed by the SCSI serial bus as one of the better SCSI mappings. It has a great RDMA mechanism and a lean architecture. Unfortunately it is not IP and the community of users is more restricted. Julo Peter Johansson on 11/08/2000 05:36:26 Please respond to Peter Johansson To: IP Storage cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Towards Consensus on TCP Connections At 06:04 AM 8/10/00, Stephen Byan wrote: >I believe the only low-cost link technology available to us is 1 Gb/s >ethernet over cat5 UTP Has the WG examined gigabit Serial Bus, IEEE P1394b? I'm not suggesting you take the time and energy, now, to do this on the reflector. I understand that there MIGHT be other considerations that affect preferred medium choice for prototype implementations. But if you take the time to independently inspect P1394b, I think you might find its cost / performance characteristics suitable. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Fri Aug 11 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21449; 11 Aug 2000 11:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA14110 for ips-outgoing; Fri, 11 Aug 2000 10:19:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from web903.mail.yahoo.com (web903.mail.yahoo.com [128.11.23.78]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id AAA21222 for ; Fri, 11 Aug 2000 00:36:57 -0400 (EDT) Received: (qmail 18599 invoked by uid 60001); 11 Aug 2000 04:36:56 -0000 Message-ID: <20000811043656.18598.qmail@web903.mail.yahoo.com> Received: from [198.4.83.36] by web903.mail.yahoo.com; Thu, 10 Aug 2000 21:36:56 PDT Date: Thu, 10 Aug 2000 21:36:56 -0700 (PDT) From: Prasenjit Sarkar Subject: multiple connections tidbits To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk IBM's current multiple connection solution over SCSI/Fibre Channel involves layering a "multiple connections" SCSI filter driver over parallel scsi/fibre channel transport. The filter driver solution costs immensely much more than the transport driver (I'm costing the cost of the HBAs towards the transport as the FC/SCSI driver is given away for free). While cost cannot determine complexity, it gives some idea - the number of man-years spent on the product was also considerable. Additionally, I was given to understand that SCSI "mulitple connections" filter driver + simple iSCSI transport driver should work in principle and would be the shortest path. Feedback from other enterprise storage vendors (EMC, HP, Hitachi, Sun ......) about their current multiple connections solutions would be welcome. Regards, Prasenjit __________________________________________________ Do You Yahoo!? Kick off your party with Yahoo! Invites. http://invites.yahoo.com/ From owner-ips@ECE.cmu.edu Fri Aug 11 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21480; 11 Aug 2000 11:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA13558 for ips-outgoing; Fri, 11 Aug 2000 10:06:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA13540 for ; Fri, 11 Aug 2000 10:06:42 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id HAA04464 for ; Fri, 11 Aug 2000 07:06:40 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Fri, 11 Aug 2000 07:06:37 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF02EEA05E@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Towards Consensus on TCP Connections Date: Fri, 11 Aug 2000 07:06:36 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > The gating factor for whether iSCSI succeeds is not going to be 200 > MB/s instead of 100 MB/s out of a single LUN. In general, I agree. iSCSI can succeed in the high and midrange storage market without link aggregation for a single LUN. These markets can afford 10 Gb/s links. As a disk device level interface, iSCSI will not succeed unless it offers at least 2 Gb/s by around 2002, at very low cost for the link. Note that even Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA speed roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: http://serialata.org/F9pp.pdf. One can argue the technical merits, but from a marketing viewpoint, the disk industry (both suppliers and customers) has long held the view that interface speeds need to match the media data rate. iSCSI can try to make an argument that slower speeds are technically adequate, but this will increase the barriers to establishing iSCSI as a device interface. > If iSCSI works at ALL in a cost effective way that can be implemented > in a disk, there'll be wild dancing in the streets and you'll all (or > maybe your companies will) be rich beyond the dreams of avarice. > > The easier you can make it for the implementors, the more likely it > will succeed. Disk drive companies have implemented much more complex interfaces than iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP connections don't look very hard to implement. They just look like a wart. But I think a necessary one. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Fri Aug 11 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22238; 11 Aug 2000 14:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA21636 for ips-outgoing; Fri, 11 Aug 2000 13:32:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA21623 for ; Fri, 11 Aug 2000 13:32:07 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA02795; Fri, 11 Aug 2000 10:27:49 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA32608; Fri, 11 Aug 2000 10:25:12 -0700 From: "Douglas Otis" To: , Cc: Subject: RE: Request to exclude FC over IP from storage over IP working group charter Date: Fri, 11 Aug 2000 10:34:01 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > somesh_gupta@hp.com > Sent: Thursday, August 10, 2000 6:28 PM > To: somesh_gupta@hp.com; sob@harvard.edu > Cc: ips@ece.cmu.edu > Subject: RE: Request to exclude FC over IP from storage over IP working > group charter > > > Why would they do that and not just do a native implementation? > This would require them to pay the price of two seperate protocols, > and the bridging between them to get one (and having to deal with > all loop issues etc). Encapsulation would be a translation which then becomes native to the device. At least you would have a defined environment where their code already habitats. > Also what you are mentioning is still a bridge (in a seperate > device or in a chip in the drive). The Lucent proposal was > for a tunnel. A tunnel over IP however. It is nothing more than a protocol. > Storage over IP should not be looked as a bridge between TCP/IP > and FC or parallel SCSI domains. It is a native transport for SCSI > commands. There will exist bridges (externel or for some reason > embedded) to bridge these 3 different transport protocols. There should be a move in that direction, but unless this interface moves to the device, removing state information from the middle, as not done for iSCSI, is the proper means for creating this gateway which is really the intent of iSCSI. I see only disaster trying to extricate error situations where there are three or more locations of state information along the path that do not agree. How to signal a problem with combined sessions by means of a LUN field for multiple drives? Generate gateway state information and error handling? Stateless encapsulation is the proper approach for combining transports as iSCSI intends and so FC over IP is a superior solution. Doug From owner-ips@ECE.cmu.edu Fri Aug 11 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22243; 11 Aug 2000 14:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA20205 for ips-outgoing; Fri, 11 Aug 2000 12:57:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA20200 for ; Fri, 11 Aug 2000 12:57:46 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0FZ400C1QZTLAY@mta6.snfc21.pbi.net> for ips@ece.cmu.edu; Fri, 11 Aug 2000 09:44:10 -0700 (PDT) Date: Fri, 11 Aug 2000 09:43:40 -0700 From: Peter Johansson Subject: iSCSI In-reply-to: X-Sender: Celeborn@PacBell.net To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Message-id: <4.3.2.7.2.20000811093038.00bec2d0@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 10:25 AM 8/11/00, julian_satran@il.ibm.com wrote: >Very good point [about Serial Bus]. Yes, we considered firewire and where >impressed by the SCSI serial bus as one of the better SCSI mappings. It >has a great RDMA mechanism and a lean architecture. Unfortunately it is >not IP and the community of users is more restricted. Your answer suggests that you were looking at Serial Bus Protocol 2 (SBP-2), an transport layer for Serial Bus that may be used for SCSI. Thanks for the compliments on its design, but I understand that it is not IP---and therefore not suited to your needs. I was actually suggesting you look at Serial Bus, IEEE 1394 and IEEE P1394b. There is an IP protocol, specified by RFC 2734. This protocol is implemented in Windows ME and available for embedded applications from quite a few vendors. I know that iSCSI is intended to be transport-medium agnostic, to be based upon IP. If that's to be the case, I hope you continue to evaluate ALL the possible media: Ethernet, FCP, Serial Bus (especially P1394b) , UTP-5, etc., etc. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Fri Aug 11 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23074; 11 Aug 2000 17:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA26907 for ips-outgoing; Fri, 11 Aug 2000 16:06:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA26894 for ; Fri, 11 Aug 2000 16:06:10 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA02872; Fri, 11 Aug 2000 13:00:28 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA00572; Fri, 11 Aug 2000 12:57:55 -0700 From: "Douglas Otis" To: , Cc: "Stephen Byan" Subject: RE: Towards Consensus on TCP Connections Date: Fri, 11 Aug 2000 13:06:44 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <87256938.00694505.00@d53mta08h.boulder.ibm.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk John, Hard to imagine a Storage Controller unrelated to disk drives. Are we discussing a protocol or your product spec? Even the lowly IDE interface started out as a controller. The quagmire is the mess created by the stateful mid-stream bottleneck you call a controller. Here you are mixing protocols which you abhor. You are not using a network or protocol effectively if you refuse to allow this interface to be suitable at the end-point, the device. The fact that your architecture can not scale speaks loudly. If you wish to use this interface at a controller, it should assume an identical role to a device. Doug > -----Original Message----- > From: hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] > Sent: Friday, August 11, 2000 12:07 PM > To: ips@ece.cmu.edu > Cc: Stephen Byan; Douglas Otis > Subject: RE: Towards Consensus on TCP Connections > > I agree with most of what Stephen says. > > Now, do you think that now we can shelve the disk drive discussion and > focus more on the Storage Controller functions. > > When we started all this, I thought that we wanted to FIRST focus on the > interface to Storage Controllers, in order to avoid the quagmire > that Fibre > Channel got into when they tried to handle both controller (with Switched > interfaces) and Disk Interfaces with Loops. > > I would like to continue to focus on the Host to Storage Controller > approach which we were following. If there is some important item, which > will prevent its operation on disks, we should be aware of it, but our > focus should be FIRST on Storage Controllers. This current Disk interface > discussion, seems to be distracting from our goal to insure that the > proposal works well at the Storage Controller level. > > I think that it has been stated several times that the use of a > Session per > LUN etc. is within the current iSCSI proposal, and it has been stated a > number of times by folks that make their living creating Storage > Controllers that they do not think they can support a protocol that > REQUIRES a Session per LUN. > > So I would like to propose, again, that we get back on track and work > issues which apply first to Storage Controllers. > > > > . > . > . > John L. Hufferd > > Internet address: hufferd@us.ibm.com > > > > "Douglas Otis" @ece.cmu.edu on 08/11/2000 11:14:13 AM > > Sent by: owner-ips@ece.cmu.edu > > > To: "Stephen Byan" , > cc: > Subject: RE: Towards Consensus on TCP Connections > > > > Today's drives can deliver 320 Mbits/second of data on the outside > cylinders. Improvement of the mechanics comes at a high price > with respect > to power and cost. The cost/volume trend takes us to a single disk which > increases access time as read channel data rate increases. By offering > scaled throughput using more drives where each drive's interface bandwidth > is restricted with respect to read channel data rates provides a system > with > uniform and superior performance. The advantage of such an approach is > found with respect to smaller random traffic. With more devices, > redundancy > is easily achieved and parallel access offers a means of performance > improvement by spreading activity over more devices. The switch provides > bandwidth aggregation and is not found in the individual device. > > An 8ms access + latency figure in the high cost drives restricts > the number > of 'independent' operations that average 64k byte to 100 per second or 52 > Mbit per second. Such an architecture of 'restricted' drives would scale > whereas the solicitated burst approach does not. An independent nexus at > the LUN is the only design that offers required scaling and configuration > flexibility. Keeping up with the read channel is a wasted effort. In > time, > 1 Gbit Ethernet will be the practical solution about the time drives are 1 > inche in size. Several Fast Ethernet disks combined at a 1 Gbit Ethernet > client makes sense in cost, performance, capacity, reliability, and > scalability at this point in time. The protocol overhead should be > addressed. There are substantial improvements to be made to allow this > innovation using standard adapters. > > The power cost to use copper 1 Gbit is high. Firewire does not scale and > has a limited reach. Firewire also places scatter/gather on the drive > together with direct access. Doing such over a WAN will impose > significant > changes. Serial ATA is nothing more than IDE through a SERDES. The read > channel data rate is like a drug, just say no. It is hard not to buy > enough > dram to allow a proper buffer these days. Serial ATA removes all buffers. > Intel is just usurping any remaining electronics at the cost of > sensitivity > to a near by cell phone. Fewer drives with less electronics. What a good > idea? > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Stephen Byan > > Sent: Friday, August 11, 2000 7:07 AM > > To: 'ips@ece.cmu.edu' > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > > MB/s instead of 100 MB/s out of a single LUN. > > > > In general, I agree. iSCSI can succeed in the high and midrange storage > > market without link aggregation for a single LUN. These markets can > afford > > 10 Gb/s links. > > > > As a disk device level interface, iSCSI will not succeed unless > > it offers at > > least 2 Gb/s by around 2002, at very low cost for the link. Note that > even > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA > speed > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > http://serialata.org/F9pp.pdf. > > > > One can argue the technical merits, but from a marketing > > viewpoint, the disk > > industry (both suppliers and customers) has long held the view that > > interface speeds need to match the media data rate. iSCSI can try > > to make an > > argument that slower speeds are technically adequate, but this > > will increase > > the barriers to establishing iSCSI as a device interface. > > > > > If iSCSI works at ALL in a cost effective way that can be implemented > > > in a disk, there'll be wild dancing in the streets and you'll all (or > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > The easier you can make it for the implementors, the more likely it > > > will succeed. > > > > Disk drive companies have implemented much more complex interfaces than > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > > connections don't look very hard to implement. They just look like a > wart. > > But I think a necessary one. > > > > Regards, > > -Steve > > > > Steve Byan > > > > Design Engineer > > MS 1-3/E23 > > 333 South Street > > Shrewsbury, MA 01545 > > (508)770-3414 > > fax: (508)770-2604 > > > > > > From owner-ips@ECE.cmu.edu Fri Aug 11 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23076; 11 Aug 2000 17:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA26754 for ips-outgoing; Fri, 11 Aug 2000 16:04:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spectrum2.osicom.com (spectrum2.osicom.com [131.143.8.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA15117 for ; Fri, 11 Aug 2000 10:45:08 -0400 (EDT) Received: from aj.osicom.com (azure2.aj.osicom.com [131.143.44.35]) by spectrum2.osicom.com (8.9.2/8.9.2) with SMTP id KAA24382; Fri, 11 Aug 2000 10:46:31 -0400 (EDT) Received: from entradanetworks.com by aj.osicom.com (SMI-8.6/SMI-SVR4(azure2)) id KAA15480; Fri, 11 Aug 2000 10:44:57 -0400 Message-ID: <39941207.E481D007@entradanetworks.com> Date: Fri, 11 Aug 2000 10:47:35 -0400 From: "Robert B. Harmon" X-Mailer: Mozilla 4.74 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: hufferd@us.ibm.com CC: somesh_gupta@hp.com, ips@ece.cmu.edu, mankin@east.isi.edu, sob@harvard.edu Subject: Re: Request to exclude FC over IP from storage over IP workinggroup charter References: <87256937.007F4A7D.00@d53mta08h.boulder.ibm.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk If we are looking for concensus, I'd much rather have FC/IP here. FC/IP has gateway issues as part of the charter. An FC/IP bridge is the 1st step to a iSCSI/FCP gateway. hufferd@us.ibm.com wrote: > > I think that Somesh stated this very well and I agree the FC over IP is an > important problem but a completely different problem. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSD San Jose Ca > (408) 256-0403, Tie: 276-0403 > Internet address: hufferd@us.ibm.com > Notes address: John Hufferd/San Jose/IBM @ IBMUS > VM address: hufferd at IBMUSM54 > > somesh_gupta@hp.com@ece.cmu.edu on 08/10/2000 03:54:37 PM > > Sent by: owner-ips@ece.cmu.edu > > To: ips@ece.cmu.edu, mankin@east.isi.edu, sob@harvard.edu > cc: > Subject: Request to exclude FC over IP from storage over IP working group > charter > > Scott/Allsion, > > I would urge you to not include FC over IP in the charter of > the SCSI over IP working group. > > Using (TCP)/IP to transport SCSI command blocks fits very > naturally as part of the SCSI Architecture Model which allows > for the use of different transports to transfer SCSI > commands. This include the parallel SCSI, fibre-channel (FCP), > SCI, and many others including a generic packet transport > mechanism. > > In this case (TCP)/IP acts as a mechanism to transport an > application level protocol as is intended - preserving the > protocol layering principles and allowing for an architecturally > clean solution. > > I have been impressed with the momentum behind this idea and saw > a number of participants at the ietf who were there for this > session (including myself). This specification will really drive > the convergence to IP based networks. There is significant > technical and commercial push for this idea. > > FC over IP is a proposal (at least based on the presentation at > the ietf meeting in Pittsburg) to have IP act as a tunnel > between two FC islands i.e. there is a fully functional > fibre-channel island A and a fully functional fibre-channel > island B, and the proposal is to connect them together using > an IP based network (don't know if it is to make a super island > or connect them as a "router" would connect two sub-nets). > > I think that the problems being solved and the motivations (from > a technical perspective - not just a commercial perspective) > are significantly different. > > 1. This is putting a transport over another transport using > tunneling protocols. A lot of the issues will be about making > one work while satisfying the contraints and using the > capabilities of the other. Probably combines the timing and > recovery mechanisms of the two transports in some horrible way. > > 2. The naming issues are entirely different. In storage over IP, > each of the partcipating entities is an IP entity. In FC over IP, > the participating entities are actually FC entities and will have > some mechanism of figuring out that the other FC entity lives > across a tunnel avaliable on the local network. If the proposal > is to create a larger net, then again the FC port addresses will > have to partitioned in some interesting way. > > 3. How would arbitrated loop work in such an environment of creating > a large net, or will it connect only Fabric ports together. > > 4. Security issues are again completely different as the end nodes are > really not IP entities. > > 5. The protocol encapsulation is also completely different. > > In Summary, putting scsi over ip is architecturally very clean and > accomodated by the SAM architecture as well as networking layered > architecture. The end systems communicating in this case are using > TCP/IP to transfer application (SCSI) PDUs and the problems/solutions > have a very good framework to work with. > > FC tunnels through IP is a commerically important problem to solve > but is a completely different problem. > > I would like to keep both of them seperate so that each of them > can be successful and provide useful specifications. > > Regards, > Somesh Gupta > Project Manager/Architect > Storage over IP > (for HP-UX servers) > Hewlett-Packard From owner-ips@ECE.cmu.edu Fri Aug 11 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23078; 11 Aug 2000 17:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA26767 for ips-outgoing; Fri, 11 Aug 2000 16:04:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from over.ny.us.ibm.com (over.ny.us.ibm.com [32.97.182.111]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA26744 for ; Fri, 11 Aug 2000 16:04:14 -0400 (EDT) From: hufferd@us.ibm.com Received: from e33.esmtp.ibm.com (e33.esmtp.ibm.com [9.14.4.131]) by admin.ny.us.ibm.com. (8.9.3/8.9.3) with ESMTP id PAA38904 for ; Fri, 11 Aug 2000 15:13:52 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id OAA34756; Fri, 11 Aug 2000 14:03:34 -0500 Received: from d53mta08h.boulder.ibm.com (d53mta08h.boulder.ibm.com [9.99.142.8]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id NAA23678; Fri, 11 Aug 2000 13:10:02 -0600 Received: by d53mta08h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256938.00694699 ; Fri, 11 Aug 2000 13:09:53 -0600 X-Lotus-FromDomain: IBMUS To: ips@ece.cmu.edu cc: "Stephen Byan" , "Douglas Otis" Message-ID: <87256938.00694505.00@d53mta08h.boulder.ibm.com> Date: Fri, 11 Aug 2000 12:06:43 -0700 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with most of what Stephen says. Now, do you think that now we can shelve the disk drive discussion and focus more on the Storage Controller functions. When we started all this, I thought that we wanted to FIRST focus on the interface to Storage Controllers, in order to avoid the quagmire that Fibre Channel got into when they tried to handle both controller (with Switched interfaces) and Disk Interfaces with Loops. I would like to continue to focus on the Host to Storage Controller approach which we were following. If there is some important item, which will prevent its operation on disks, we should be aware of it, but our focus should be FIRST on Storage Controllers. This current Disk interface discussion, seems to be distracting from our goal to insure that the proposal works well at the Storage Controller level. I think that it has been stated several times that the use of a Session per LUN etc. is within the current iSCSI proposal, and it has been stated a number of times by folks that make their living creating Storage Controllers that they do not think they can support a protocol that REQUIRES a Session per LUN. So I would like to propose, again, that we get back on track and work issues which apply first to Storage Controllers. . . . John L. Hufferd Internet address: hufferd@us.ibm.com "Douglas Otis" @ece.cmu.edu on 08/11/2000 11:14:13 AM Sent by: owner-ips@ece.cmu.edu To: "Stephen Byan" , cc: Subject: RE: Towards Consensus on TCP Connections Today's drives can deliver 320 Mbits/second of data on the outside cylinders. Improvement of the mechanics comes at a high price with respect to power and cost. The cost/volume trend takes us to a single disk which increases access time as read channel data rate increases. By offering scaled throughput using more drives where each drive's interface bandwidth is restricted with respect to read channel data rates provides a system with uniform and superior performance. The advantage of such an approach is found with respect to smaller random traffic. With more devices, redundancy is easily achieved and parallel access offers a means of performance improvement by spreading activity over more devices. The switch provides bandwidth aggregation and is not found in the individual device. An 8ms access + latency figure in the high cost drives restricts the number of 'independent' operations that average 64k byte to 100 per second or 52 Mbit per second. Such an architecture of 'restricted' drives would scale whereas the solicitated burst approach does not. An independent nexus at the LUN is the only design that offers required scaling and configuration flexibility. Keeping up with the read channel is a wasted effort. In time, 1 Gbit Ethernet will be the practical solution about the time drives are 1 inche in size. Several Fast Ethernet disks combined at a 1 Gbit Ethernet client makes sense in cost, performance, capacity, reliability, and scalability at this point in time. The protocol overhead should be addressed. There are substantial improvements to be made to allow this innovation using standard adapters. The power cost to use copper 1 Gbit is high. Firewire does not scale and has a limited reach. Firewire also places scatter/gather on the drive together with direct access. Doing such over a WAN will impose significant changes. Serial ATA is nothing more than IDE through a SERDES. The read channel data rate is like a drug, just say no. It is hard not to buy enough dram to allow a proper buffer these days. Serial ATA removes all buffers. Intel is just usurping any remaining electronics at the cost of sensitivity to a near by cell phone. Fewer drives with less electronics. What a good idea? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Byan > Sent: Friday, August 11, 2000 7:07 AM > To: 'ips@ece.cmu.edu' > Subject: RE: Towards Consensus on TCP Connections > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > MB/s instead of 100 MB/s out of a single LUN. > > In general, I agree. iSCSI can succeed in the high and midrange storage > market without link aggregation for a single LUN. These markets can afford > 10 Gb/s links. > > As a disk device level interface, iSCSI will not succeed unless > it offers at > least 2 Gb/s by around 2002, at very low cost for the link. Note that even > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA speed > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > http://serialata.org/F9pp.pdf. > > One can argue the technical merits, but from a marketing > viewpoint, the disk > industry (both suppliers and customers) has long held the view that > interface speeds need to match the media data rate. iSCSI can try > to make an > argument that slower speeds are technically adequate, but this > will increase > the barriers to establishing iSCSI as a device interface. > > > If iSCSI works at ALL in a cost effective way that can be implemented > > in a disk, there'll be wild dancing in the streets and you'll all (or > > maybe your companies will) be rich beyond the dreams of avarice. > > > > The easier you can make it for the implementors, the more likely it > > will succeed. > > Disk drive companies have implemented much more complex interfaces than > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > connections don't look very hard to implement. They just look like a wart. > But I think a necessary one. > > Regards, > -Steve > > Steve Byan > > Design Engineer > MS 1-3/E23 > 333 South Street > Shrewsbury, MA 01545 > (508)770-3414 > fax: (508)770-2604 > From owner-ips@ECE.cmu.edu Fri Aug 11 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23413; 11 Aug 2000 18:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA00294 for ips-outgoing; Fri, 11 Aug 2000 17:20:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA00287 for ; Fri, 11 Aug 2000 17:20:02 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id XAA78986 for ; Fri, 11 Aug 2000 23:19:21 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id XAA27460 for ; Fri, 11 Aug 2000 23:19:22 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256938.00751D31 ; Fri, 11 Aug 2000 23:19:11 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 12 Aug 2000 00:17:27 +0300 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, I am not sure that I agree with your architecture statements but I like to play with numbers (as most of the fellows engineers on this list probably do). What would be in your opinion reasonable requirements for command and data transfer rates for the next 3-7 years? I would like to decouple that discussion from architecture - data rates can scale even in a shared architecture as mainframe channels have shown for years. I would rather like to understand if we can meet the data rates with reasonable latency. Julo "Douglas Otis" on 11/08/2000 21:14:13 Please respond to "Douglas Otis" To: "Stephen Byan" , ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Towards Consensus on TCP Connections Today's drives can deliver 320 Mbits/second of data on the outside cylinders. Improvement of the mechanics comes at a high price with respect to power and cost. The cost/volume trend takes us to a single disk which increases access time as read channel data rate increases. By offering scaled throughput using more drives where each drive's interface bandwidth is restricted with respect to read channel data rates provides a system with uniform and superior performance. The advantage of such an approach is found with respect to smaller random traffic. With more devices, redundancy is easily achieved and parallel access offers a means of performance improvement by spreading activity over more devices. The switch provides bandwidth aggregation and is not found in the individual device. An 8ms access + latency figure in the high cost drives restricts the number of 'independent' operations that average 64k byte to 100 per second or 52 Mbit per second. Such an architecture of 'restricted' drives would scale whereas the solicitated burst approach does not. An independent nexus at the LUN is the only design that offers required scaling and configuration flexibility. Keeping up with the read channel is a wasted effort. In time, 1 Gbit Ethernet will be the practical solution about the time drives are 1 inche in size. Several Fast Ethernet disks combined at a 1 Gbit Ethernet client makes sense in cost, performance, capacity, reliability, and scalability at this point in time. The protocol overhead should be addressed. There are substantial improvements to be made to allow this innovation using standard adapters. The power cost to use copper 1 Gbit is high. Firewire does not scale and has a limited reach. Firewire also places scatter/gather on the drive together with direct access. Doing such over a WAN will impose significant changes. Serial ATA is nothing more than IDE through a SERDES. The read channel data rate is like a drug, just say no. It is hard not to buy enough dram to allow a proper buffer these days. Serial ATA removes all buffers. Intel is just usurping any remaining electronics at the cost of sensitivity to a near by cell phone. Fewer drives with less electronics. What a good idea? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Byan > Sent: Friday, August 11, 2000 7:07 AM > To: 'ips@ece.cmu.edu' > Subject: RE: Towards Consensus on TCP Connections > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > MB/s instead of 100 MB/s out of a single LUN. > > In general, I agree. iSCSI can succeed in the high and midrange storage > market without link aggregation for a single LUN. These markets can afford > 10 Gb/s links. > > As a disk device level interface, iSCSI will not succeed unless > it offers at > least 2 Gb/s by around 2002, at very low cost for the link. Note that even > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA speed > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > http://serialata.org/F9pp.pdf. > > One can argue the technical merits, but from a marketing > viewpoint, the disk > industry (both suppliers and customers) has long held the view that > interface speeds need to match the media data rate. iSCSI can try > to make an > argument that slower speeds are technically adequate, but this > will increase > the barriers to establishing iSCSI as a device interface. > > > If iSCSI works at ALL in a cost effective way that can be implemented > > in a disk, there'll be wild dancing in the streets and you'll all (or > > maybe your companies will) be rich beyond the dreams of avarice. > > > > The easier you can make it for the implementors, the more likely it > > will succeed. > > Disk drive companies have implemented much more complex interfaces than > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > connections don't look very hard to implement. They just look like a wart. > But I think a necessary one. > > Regards, > -Steve > > Steve Byan > > Design Engineer > MS 1-3/E23 > 333 South Street > Shrewsbury, MA 01545 > (508)770-3414 > fax: (508)770-2604 > From owner-ips@ECE.cmu.edu Fri Aug 11 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23419; 11 Aug 2000 18:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA29894 for ips-outgoing; Fri, 11 Aug 2000 17:08:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA29890 for ; Fri, 11 Aug 2000 17:08:53 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id OAA25576; Fri, 11 Aug 2000 14:08:51 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Fri, 11 Aug 2000 14:08:49 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF02EEA066@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: ips@ece.cmu.edu Cc: Stephen Byan , Douglas Otis Subject: RE: Towards Consensus on TCP Connections Date: Fri, 11 Aug 2000 14:08:46 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] wrote: > Now, do you think that now we can shelve the disk drive discussion and > focus more on the Storage Controller functions. I hope no one thinks I'm trying to derail the Storage Controller portion of our discussions. I'm simply explaining why I advocate my position in favor of multiple TCP connections per iSCSI session. > So I would like to propose, again, that we get back on track and work > issues which apply first to Storage Controllers. OK. So do you favor or oppose multiple TCP connections per iSCSI session? Your position wasn't clear to me from your message. Thanks. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Sat Aug 12 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23875; 11 Aug 2000 20:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA04219 for ips-outgoing; Fri, 11 Aug 2000 19:45:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA04214 for ; Fri, 11 Aug 2000 19:45:34 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA02931; Fri, 11 Aug 2000 16:41:22 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA01141; Fri, 11 Aug 2000 16:38:47 -0700 From: "Douglas Otis" To: , Subject: RE: Towards Consensus on TCP Connections Date: Fri, 11 Aug 2000 16:47:37 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, An architecture that scales does not need to provide the entire bandwidth from a single device. If these devices are each handling 25 Mbytes per second, then 2 of them exceed Fibre-Channel. Next week you could use 4, and the week after that 8. The advantage of using a network without any practical architectural restriction with intelligence at the client. On the other hand, you want to design a single mid-point gateway to handle the entire bandwidth. To what end? You argue it can not process a TCP session per device. You will not be tracking mid-stream errors as this is more costly. You add to the burden of tracking the state of the individual device with this mid-stream state machine requiring additional sorting due to this merged protocol without taking advantage of TCP to aid this process. For your type of solution, a simple hardware based tunnel would be better. Do not include handling of the encapsulated protocol and at least it becomes understandable and more likely to manage the task. In other words, make it Fibre-Channel over IP and you have a chance with your architecture. The point to my statements was to indicate the device is able to handle an IP interface today using Fast Ethernet as the rate from the drive is relatively low. Not to warm the cockles of marketing pointing to latency, but at least they sell more drives overcoming this problem using scale. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Friday, August 11, 2000 2:17 PM > To: ips@ece.cmu.edu > Subject: RE: Towards Consensus on TCP Connections > > > > > Doug, > > I am not sure that I agree with your architecture statements but I like to > play with numbers (as most of the fellows engineers on this list probably > do). What would be in your opinion > reasonable requirements for command and data transfer rates for the next > 3-7 years? > > I would like to decouple that discussion from architecture - data > rates can > scale even in a shared > architecture as mainframe channels have shown for years. > > I would rather like to understand if we can meet the data rates with > reasonable latency. > > Julo > > "Douglas Otis" on 11/08/2000 21:14:13 > > Please respond to "Douglas Otis" > > To: "Stephen Byan" , ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Towards Consensus on TCP Connections > > > > > Today's drives can deliver 320 Mbits/second of data on the outside > cylinders. Improvement of the mechanics comes at a high price > with respect > to power and cost. The cost/volume trend takes us to a single disk which > increases access time as read channel data rate increases. By offering > scaled throughput using more drives where each drive's interface bandwidth > is restricted with respect to read channel data rates provides a system > with > uniform and superior performance. The advantage of such an approach is > found with respect to smaller random traffic. With more devices, > redundancy > is easily achieved and parallel access offers a means of performance > improvement by spreading activity over more devices. The switch provides > bandwidth aggregation and is not found in the individual device. > > An 8ms access + latency figure in the high cost drives restricts > the number > of 'independent' operations that average 64k byte to 100 per second or 52 > Mbit per second. Such an architecture of 'restricted' drives would scale > whereas the solicitated burst approach does not. An independent nexus at > the LUN is the only design that offers required scaling and configuration > flexibility. Keeping up with the read channel is a wasted effort. In > time, > 1 Gbit Ethernet will be the practical solution about the time drives are 1 > inche in size. Several Fast Ethernet disks combined at a 1 Gbit Ethernet > client makes sense in cost, performance, capacity, reliability, and > scalability at this point in time. The protocol overhead should be > addressed. There are substantial improvements to be made to allow this > innovation using standard adapters. > > The power cost to use copper 1 Gbit is high. Firewire does not scale and > has a limited reach. Firewire also places scatter/gather on the drive > together with direct access. Doing such over a WAN will impose > significant > changes. Serial ATA is nothing more than IDE through a SERDES. The read > channel data rate is like a drug, just say no. It is hard not to buy > enough > dram to allow a proper buffer these days. Serial ATA removes all buffers. > Intel is just usurping any remaining electronics at the cost of > sensitivity > to a near by cell phone. Fewer drives with less electronics. What a good > idea? > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Stephen Byan > > Sent: Friday, August 11, 2000 7:07 AM > > To: 'ips@ece.cmu.edu' > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > > MB/s instead of 100 MB/s out of a single LUN. > > > > In general, I agree. iSCSI can succeed in the high and midrange storage > > market without link aggregation for a single LUN. These markets can > afford > > 10 Gb/s links. > > > > As a disk device level interface, iSCSI will not succeed unless > > it offers at > > least 2 Gb/s by around 2002, at very low cost for the link. Note that > even > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA > speed > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > http://serialata.org/F9pp.pdf. > > > > One can argue the technical merits, but from a marketing > > viewpoint, the disk > > industry (both suppliers and customers) has long held the view that > > interface speeds need to match the media data rate. iSCSI can try > > to make an > > argument that slower speeds are technically adequate, but this > > will increase > > the barriers to establishing iSCSI as a device interface. > > > > > If iSCSI works at ALL in a cost effective way that can be implemented > > > in a disk, there'll be wild dancing in the streets and you'll all (or > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > The easier you can make it for the implementors, the more likely it > > > will succeed. > > > > Disk drive companies have implemented much more complex interfaces than > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > > connections don't look very hard to implement. They just look like a > wart. > > But I think a necessary one. > > > > Regards, > > -Steve > > > > Steve Byan > > > > Design Engineer > > MS 1-3/E23 > > 333 South Street > > Shrewsbury, MA 01545 > > (508)770-3414 > > fax: (508)770-2604 > > > > > > From owner-ips@ECE.cmu.edu Sat Aug 12 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23880; 11 Aug 2000 20:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA02920 for ips-outgoing; Fri, 11 Aug 2000 18:47:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA02915 for ; Fri, 11 Aug 2000 18:47:05 -0400 (EDT) From: hufferd@us.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA25416; Fri, 11 Aug 2000 18:40:02 -0400 Received: from d53mta08h.boulder.ibm.com (d53mta08h.boulder.ibm.com [9.99.142.8]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id QAA136516; Fri, 11 Aug 2000 16:45:45 -0600 Received: by d53mta08h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256938.007D06CF ; Fri, 11 Aug 2000 16:45:37 -0600 X-Lotus-FromDomain: IBMUS To: "Douglas Otis" cc: ips@ece.cmu.edu, "Stephen Byan" Message-ID: <87256938.007D0500.00@d53mta08h.boulder.ibm.com> Date: Fri, 11 Aug 2000 14:06:10 -0700 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, I do not think that such contentious statements are useful. If you think that Storage Controllers from EMC and IBM do not scale, then I think there are a number of Companies that would radically disagree. Again, I am asking that we first focus back on the original intent of Host to Storage Controllers. . . . John L. Hufferd Internet address: hufferd@us.ibm.com "Douglas Otis" on 08/11/2000 01:06:44 PM To: John Hufferd/San Jose/IBM@IBMUS, cc: "Stephen Byan" Subject: RE: Towards Consensus on TCP Connections John, Hard to imagine a Storage Controller unrelated to disk drives. Are we discussing a protocol or your product spec? Even the lowly IDE interface started out as a controller. The quagmire is the mess created by the stateful mid-stream bottleneck you call a controller. Here you are mixing protocols which you abhor. You are not using a network or protocol effectively if you refuse to allow this interface to be suitable at the end-point, the device. The fact that your architecture can not scale speaks loudly. If you wish to use this interface at a controller, it should assume an identical role to a device. Doug > -----Original Message----- > From: hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] > Sent: Friday, August 11, 2000 12:07 PM > To: ips@ece.cmu.edu > Cc: Stephen Byan; Douglas Otis > Subject: RE: Towards Consensus on TCP Connections > > I agree with most of what Stephen says. > > Now, do you think that now we can shelve the disk drive discussion and > focus more on the Storage Controller functions. > > When we started all this, I thought that we wanted to FIRST focus on the > interface to Storage Controllers, in order to avoid the quagmire > that Fibre > Channel got into when they tried to handle both controller (with Switched > interfaces) and Disk Interfaces with Loops. > > I would like to continue to focus on the Host to Storage Controller > approach which we were following. If there is some important item, which > will prevent its operation on disks, we should be aware of it, but our > focus should be FIRST on Storage Controllers. This current Disk interface > discussion, seems to be distracting from our goal to insure that the > proposal works well at the Storage Controller level. > > I think that it has been stated several times that the use of a > Session per > LUN etc. is within the current iSCSI proposal, and it has been stated a > number of times by folks that make their living creating Storage > Controllers that they do not think they can support a protocol that > REQUIRES a Session per LUN. > > So I would like to propose, again, that we get back on track and work > issues which apply first to Storage Controllers. > > > > . > . > . > John L. Hufferd > > Internet address: hufferd@us.ibm.com > > > > "Douglas Otis" @ece.cmu.edu on 08/11/2000 11:14:13 AM > > Sent by: owner-ips@ece.cmu.edu > > > To: "Stephen Byan" , > cc: > Subject: RE: Towards Consensus on TCP Connections > > > > Today's drives can deliver 320 Mbits/second of data on the outside > cylinders. Improvement of the mechanics comes at a high price > with respect > to power and cost. The cost/volume trend takes us to a single disk which > increases access time as read channel data rate increases. By offering > scaled throughput using more drives where each drive's interface bandwidth > is restricted with respect to read channel data rates provides a system > with > uniform and superior performance. The advantage of such an approach is > found with respect to smaller random traffic. With more devices, > redundancy > is easily achieved and parallel access offers a means of performance > improvement by spreading activity over more devices. The switch provides > bandwidth aggregation and is not found in the individual device. > > An 8ms access + latency figure in the high cost drives restricts > the number > of 'independent' operations that average 64k byte to 100 per second or 52 > Mbit per second. Such an architecture of 'restricted' drives would scale > whereas the solicitated burst approach does not. An independent nexus at > the LUN is the only design that offers required scaling and configuration > flexibility. Keeping up with the read channel is a wasted effort. In > time, > 1 Gbit Ethernet will be the practical solution about the time drives are 1 > inche in size. Several Fast Ethernet disks combined at a 1 Gbit Ethernet > client makes sense in cost, performance, capacity, reliability, and > scalability at this point in time. The protocol overhead should be > addressed. There are substantial improvements to be made to allow this > innovation using standard adapters. > > The power cost to use copper 1 Gbit is high. Firewire does not scale and > has a limited reach. Firewire also places scatter/gather on the drive > together with direct access. Doing such over a WAN will impose > significant > changes. Serial ATA is nothing more than IDE through a SERDES. The read > channel data rate is like a drug, just say no. It is hard not to buy > enough > dram to allow a proper buffer these days. Serial ATA removes all buffers. > Intel is just usurping any remaining electronics at the cost of > sensitivity > to a near by cell phone. Fewer drives with less electronics. What a good > idea? > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Stephen Byan > > Sent: Friday, August 11, 2000 7:07 AM > > To: 'ips@ece.cmu.edu' > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > > MB/s instead of 100 MB/s out of a single LUN. > > > > In general, I agree. iSCSI can succeed in the high and midrange storage > > market without link aggregation for a single LUN. These markets can > afford > > 10 Gb/s links. > > > > As a disk device level interface, iSCSI will not succeed unless > > it offers at > > least 2 Gb/s by around 2002, at very low cost for the link. Note that > even > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA > speed > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > http://serialata.org/F9pp.pdf. > > > > One can argue the technical merits, but from a marketing > > viewpoint, the disk > > industry (both suppliers and customers) has long held the view that > > interface speeds need to match the media data rate. iSCSI can try > > to make an > > argument that slower speeds are technically adequate, but this > > will increase > > the barriers to establishing iSCSI as a device interface. > > > > > If iSCSI works at ALL in a cost effective way that can be implemented > > > in a disk, there'll be wild dancing in the streets and you'll all (or > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > The easier you can make it for the implementors, the more likely it > > > will succeed. > > > > Disk drive companies have implemented much more complex interfaces than > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > > connections don't look very hard to implement. They just look like a > wart. > > But I think a necessary one. > > > > Regards, > > -Steve > > > > Steve Byan > > > > Design Engineer > > MS 1-3/E23 > > 333 South Street > > Shrewsbury, MA 01545 > > (508)770-3414 > > fax: (508)770-2604 > > > > > > From owner-ips@ECE.cmu.edu Sat Aug 12 01:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24061; 11 Aug 2000 20:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA04676 for ips-outgoing; Fri, 11 Aug 2000 20:05:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA04672 for ; Fri, 11 Aug 2000 20:05:41 -0400 (EDT) From: hufferd@us.ibm.com Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id UAA53668; Fri, 11 Aug 2000 20:03:31 -0400 Received: from d53mta08h.boulder.ibm.com (d53mta08h.boulder.ibm.com [9.99.142.8]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with SMTP id SAA162606; Fri, 11 Aug 2000 18:05:37 -0600 Received: by d53mta08h.boulder.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id 87256939.000081A8 ; Fri, 11 Aug 2000 18:05:31 -0600 X-Lotus-FromDomain: IBMUS To: Stephen Byan cc: ips@ece.cmu.edu, Stephen Byan , Douglas Otis Message-ID: <87256939.00008082.00@d53mta08h.boulder.ibm.com> Date: Fri, 11 Aug 2000 17:04:36 -0700 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen, That is a fair question. At this moment I think that Multiple TCP/IP Connections per session can be useful, but I am neither a rabid fan nor a rabid detractor. The people involved have put a lot of effort into understanding the problems and solutions that this approach both cause and solve. The biggest goal that we want to solve in the long run is how we can actually use the complete set of connections to both sending Commands & Data as well as receiving Data, on any connection. If we had that capability, I would be a Rabid Fan of Multiple TCP/IP Connections per session. We, IBM, and EMC, etc. have done this kind of thing on our large storage controller connected to S390 processors for years, and the results has been extremely valuable. The iSCSI team, however, gave up on this approach for the initial specifications, since it seemed to be very difficult to either explain or perhaps implement across multiple NIC Adapters. Regardless, they then attempted to keep as much as they could of the Multiple TCP/IP Connections per Session, but have the data return on the Command Connection. This is perhaps better then not having this capability, but looses some of the key value that we originally wanted. I expected that the IPS group would look at this carefully and see if they could either come up with a way to do it even better. That is, approach the goal of permitting the Storage Controller to respond on any Connection of a Session with the approprate Data and Status, if the preferred/default one is busy. I even hallucinated my own, perhaps not well thought out idea, that perhaps something like a RTT could be sent on another Session Connection (not the one on which the command was received), to alert that Initiator side that data from command xxxx was about to be sent on some specific Session Connection. (I said that it was not well thought out!) In this way, or some other better way, it might be possible to use all the Session Connections for Both Data and Commands. I can accept the fact that we are not completely there at this time, and can accept the fact that what we have now may be all we can do for the first release of this standard. I feel this is reasonable and possible, since it has a base on which to do even better things in subsequent versions of the standard. Having said that, let me put on my product hat and talk about what I think many of us will do as soon as possible. And that is to use a subset of the Multiple TCP/IP Connections per Session --- that is, one TCP/IP Connection per Session with possibly multiple Session to the same target (probably on different Host NICs). This will mean, however, that the Host systems will need to have the Session Balancing software which we and our competitors current have available to run on the host when they have multiple FC connections. Since IBM and others already have such code, it seems like the path of least resistance. The problem is, that we and others will have to keep this unique code around, which is independent of this Standard, and we and our competitors may have to write/maintain this code in addition to the iSCSI Device Drivers. If it was all in the Standard, then the Host customer could choose the approprate code that went with its iSCSI compatible NICs, and no other separate piece of code would have to be developed and maintained (or installed). I think that some companies will also attempt to get to the Multiple Conversations per Session, since it will give their customers the advantage of not having to install Storage Vendor specific software to Balance between the various Connections. I think this will be goodness. So let me net this: I think that Multiple Conversations per Session are useful, and can be even more useful when extended in the future. I think that using the subset of one Conversation per Session (with probable multiple session to a target) is something that many/most of the companies will use, since it is simpler and they already have the connection balancing software. But I think the end-game must be multiple conversations per session that carry commands and data independently. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 Stephen Byan @ece.cmu.edu on 08/11/2000 02:08:46 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Stephen Byan , Douglas Otis Subject: RE: Towards Consensus on TCP Connections hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] wrote: > Now, do you think that now we can shelve the disk drive discussion and > focus more on the Storage Controller functions. I hope no one thinks I'm trying to derail the Storage Controller portion of our discussions. I'm simply explaining why I advocate my position in favor of multiple TCP connections per iSCSI session. > So I would like to propose, again, that we get back on track and work > issues which apply first to Storage Controllers. OK. So do you favor or oppose multiple TCP connections per iSCSI session? Your position wasn't clear to me from your message. Thanks. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Sat Aug 12 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24083; 11 Aug 2000 21:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA04507 for ips-outgoing; Fri, 11 Aug 2000 19:59:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA04502 for ; Fri, 11 Aug 2000 19:59:30 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA02940; Fri, 11 Aug 2000 16:54:01 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA01181; Fri, 11 Aug 2000 16:51:28 -0700 From: "Douglas Otis" To: Cc: , "Stephen Byan" Subject: RE: Towards Consensus on TCP Connections Date: Fri, 11 Aug 2000 17:00:19 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <87256938.007D0500.00@d53mta08h.boulder.ibm.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk John, I would appreciate something a bit more specific. I was not making a statement about existing products bolted directly into the system back-plane. Over a network, less is known. Again, for your type of solution, you would be better off making a tunnel for the reasons mentioned. Discuss the details and not your association. Doug > -----Original Message----- > From: hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] > Sent: Friday, August 11, 2000 2:06 PM > To: Douglas Otis > Cc: ips@ece.cmu.edu; Stephen Byan > Subject: RE: Towards Consensus on TCP Connections > > > > > Doug, > I do not think that such contentious statements are useful. > > If you think that Storage Controllers from EMC and IBM do not > scale, then I > think there are a number of Companies that would radically disagree. > > Again, I am asking that we first focus back on the original intent of Host > to Storage Controllers. > > . > . > . > John L. Hufferd > > Internet address: hufferd@us.ibm.com > > > "Douglas Otis" on 08/11/2000 01:06:44 PM > > To: John Hufferd/San Jose/IBM@IBMUS, > cc: "Stephen Byan" > Subject: RE: Towards Consensus on TCP Connections > > > > John, > > Hard to imagine a Storage Controller unrelated to disk drives. Are we > discussing a protocol or your product spec? Even the lowly IDE interface > started out as a controller. The quagmire is the mess created by the > stateful mid-stream bottleneck you call a controller. Here you are mixing > protocols which you abhor. You are not using a network or protocol > effectively if you refuse to allow this interface to be suitable at the > end-point, the device. The fact that your architecture can not scale > speaks > loudly. If you wish to use this interface at a controller, it should > assume > an identical role to a device. > > Doug > From owner-ips@ECE.cmu.edu Sat Aug 12 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24324; 11 Aug 2000 21:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA05520 for ips-outgoing; Fri, 11 Aug 2000 20:38:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA05516 for ; Fri, 11 Aug 2000 20:38:48 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 11 Aug 2000 20:38:18 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100EA7@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: DRAFT Pittsburgh ips Minutes Date: Fri, 11 Aug 2000 20:38:16 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk An important clarification has been pointed out in the DRAFT minutes. The DRAFT minutes say: Consensus call -------------- There was a consensus call eventually. Should encryption be mandatory? The consensus in the room is that encryption of the stream should not be mandatory. A lot of people were concerned about the overhead it would imply on cheap devices. The consensus call question was actually "should implementation of encryption be mandatory"? Since the consensus was "no", implementation of encryption will not be required by the specification. Note that implementation of authentication and related data integrity measures (e.g., that make it impossible to insert data into or hijack an authenticated session) will be mandatory, per the AD (i.e., if they are not mandatory in the spec that the WG produces, the spec will be returned to the WG with instructions to make it so). --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sat Aug 12 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24335; 11 Aug 2000 21:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA06240 for ips-outgoing; Fri, 11 Aug 2000 21:08:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA06232 for ; Fri, 11 Aug 2000 21:08:54 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 11 Aug 2000 21:08:19 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100EA8@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: Towards Consensus on TCP Connections Date: Fri, 11 Aug 2000 21:08:19 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I do not think that such contentious statements are useful. Can we calm down a bit -- or at least leave refereeing attempts to folks like the co-chairs and ADs? Thanks in advance. It is the case that Douglas Otis's statement to which John Hufferd objected is seriously at odds with industry practice -- > If you wish to use this interface at a controller, it should > assume an identical role to a device. Even disk arrays that are all-Fibre Channel are often using FC-SW (switched) to talk to servers and FC-AL (arbitrated loop) to talk to drives. There are disk array products with Fibre Channel interfaces to servers that use SCSI and even SSA drives. Most mainframes still use ESCON as their storage interface, even though there hasn't been an ESCON disk drive produced for many years. So, I think the onus is on Douglas to explain why uniformity of controller and drive interfaces should be required, given the large number of product engineering decisions that have gone in the other direction. After all, this is the Internet *Engineering* Task Force :-). The overall discussion of whether the iSCSI protocol specification should include both arrays and individual drives is definitely germane and in scope for this list - there are words in the draft charter to that effect. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sat Aug 12 04:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25019; 12 Aug 2000 0:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA08593 for ips-outgoing; Fri, 11 Aug 2000 22:59:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA08588 for ; Fri, 11 Aug 2000 22:58:59 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 11 Aug 2000 22:58:27 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100EAB@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: exactly the same protocol ?? Date: Fri, 11 Aug 2000 22:58:26 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk With my WG co-chair hat off, it seems that Robert Snively and I have different notions of what "exactly the same protocol" means. It would have been nice if FC-AL and FC-SW were exactly the same protocol, as it would have saved EMC and a number of other companies a considerable amount of software/firmware development time and effort. FC-AL and FC-SW are different protocols at the FC-2 level that do not interoperate - a device that only speaks FC-AL cannot communicate with a device that only speaks FC-SW and vice-versa. FCP (which encapsulates SCSI for Fibre Channel) is an FC-4 level upper layer protocol that runs over both FC-AL and FC-SW. I don't believe that there are any disk drives that speak FC-SW. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sat Aug 12 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25303; 12 Aug 2000 1:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA09982 for ips-outgoing; Fri, 11 Aug 2000 23:57:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA07861 for ; Fri, 11 Aug 2000 22:21:18 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Fri, 11 Aug 2000 19:21:50 -0700 Message-ID: From: Robert Snively To: "'Black_David@emc.com'" , ips@ece.cmu.edu Subject: RE: Towards Consensus on TCP Connections Date: Fri, 11 Aug 2000 19:20:02 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Thank you for your calming words. At present, all the disk arrays that attach using Fibre Channel use exactly the same protocol that is used by the individual disk drives, i.e. FCP (or FCP-2). Some optional choices may be different (e.g., if more than one disk drive is attached, it is likely to use FC-AL-2 instead of the point-to-point FC transfer rules), but the protocol is identical. Disk arrays may have a slightly larger command set, including portions of SSC-2 and SES. Both devices may have out-of-band command capability as well, including SFF-8067 for disk drives and various management interfaces for RAID boxes. But the protocol is the same (FCP). Disk drives tend to have symmetrical behaviors across their multiple ports, while RAID boxes may have either symmetrical or asymmetrical behaviors. The protocol is still the same. Tape drives and storage subsystems may have ordering requirements. The protocol is still the same. The splendor of this is that it does not matter whether you are controller attached, bridge attached, or directly attached, no changes need to be made in the protocol. Similar behavior is true of parallel SCSI RAID boxes and disk drives. For this and a bunch of other reasons, we have been encouraging the use of the common packet structure and architecture implemented by packetized parallel SCSI, SBP-2, SSA, and FCP in the iSCSI. Bob Bob Snively Brocade Communications Phone 408 487 8135 1901 Guadalupe Parkway San Jose, CA 95131 Email rsnively@brocade.com > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, August 11, 2000 6:08 PM > To: ips@ece.cmu.edu > Subject: RE: Towards Consensus on TCP Connections > > > > I do not think that such contentious statements are useful. > > Can we calm down a bit -- or at least leave refereeing attempts > to folks like the co-chairs and ADs? Thanks in advance. > > It is the case that Douglas Otis's statement to which John Hufferd > objected is seriously at odds with industry practice -- > > > If you wish to use this interface at a controller, it should > > assume an identical role to a device. > > Even disk arrays that are all-Fibre Channel are often using FC-SW > (switched) to talk to servers and FC-AL (arbitrated loop) to > talk to drives. > There are disk array products with Fibre Channel interfaces > to servers > that use SCSI and even SSA drives. Most mainframes still use ESCON > as their storage interface, even though there hasn't been an ESCON > disk drive produced for many years. > > So, I think the onus is on Douglas to explain why uniformity > of controller > and drive interfaces should be required, given the large > number of product > engineering decisions that have gone in the other direction. > After all, > this > is the Internet *Engineering* Task Force :-). > > The overall discussion of whether the iSCSI protocol > specification should > include both arrays and individual drives is definitely > germane and in > scope for this list - there are words in the draft charter > to that effect. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ECE.cmu.edu Sat Aug 12 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26149; 12 Aug 2000 4:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA23178 for ips-outgoing; Fri, 11 Aug 2000 14:12:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA23163 for ; Fri, 11 Aug 2000 14:12:10 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA02820; Fri, 11 Aug 2000 11:08:01 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA32723; Fri, 11 Aug 2000 11:05:24 -0700 From: "Douglas Otis" To: "Stephen Byan" , Subject: RE: Towards Consensus on TCP Connections Date: Fri, 11 Aug 2000 11:14:13 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <8133266FE373D11190CD00805FA768BF02EEA05E@shrcmsg1.tdh.qntm.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Today's drives can deliver 320 Mbits/second of data on the outside cylinders. Improvement of the mechanics comes at a high price with respect to power and cost. The cost/volume trend takes us to a single disk which increases access time as read channel data rate increases. By offering scaled throughput using more drives where each drive's interface bandwidth is restricted with respect to read channel data rates provides a system with uniform and superior performance. The advantage of such an approach is found with respect to smaller random traffic. With more devices, redundancy is easily achieved and parallel access offers a means of performance improvement by spreading activity over more devices. The switch provides bandwidth aggregation and is not found in the individual device. An 8ms access + latency figure in the high cost drives restricts the number of 'independent' operations that average 64k byte to 100 per second or 52 Mbit per second. Such an architecture of 'restricted' drives would scale whereas the solicitated burst approach does not. An independent nexus at the LUN is the only design that offers required scaling and configuration flexibility. Keeping up with the read channel is a wasted effort. In time, 1 Gbit Ethernet will be the practical solution about the time drives are 1 inche in size. Several Fast Ethernet disks combined at a 1 Gbit Ethernet client makes sense in cost, performance, capacity, reliability, and scalability at this point in time. The protocol overhead should be addressed. There are substantial improvements to be made to allow this innovation using standard adapters. The power cost to use copper 1 Gbit is high. Firewire does not scale and has a limited reach. Firewire also places scatter/gather on the drive together with direct access. Doing such over a WAN will impose significant changes. Serial ATA is nothing more than IDE through a SERDES. The read channel data rate is like a drug, just say no. It is hard not to buy enough dram to allow a proper buffer these days. Serial ATA removes all buffers. Intel is just usurping any remaining electronics at the cost of sensitivity to a near by cell phone. Fewer drives with less electronics. What a good idea? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Byan > Sent: Friday, August 11, 2000 7:07 AM > To: 'ips@ece.cmu.edu' > Subject: RE: Towards Consensus on TCP Connections > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > MB/s instead of 100 MB/s out of a single LUN. > > In general, I agree. iSCSI can succeed in the high and midrange storage > market without link aggregation for a single LUN. These markets can afford > 10 Gb/s links. > > As a disk device level interface, iSCSI will not succeed unless > it offers at > least 2 Gb/s by around 2002, at very low cost for the link. Note that even > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA speed > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > http://serialata.org/F9pp.pdf. > > One can argue the technical merits, but from a marketing > viewpoint, the disk > industry (both suppliers and customers) has long held the view that > interface speeds need to match the media data rate. iSCSI can try > to make an > argument that slower speeds are technically adequate, but this > will increase > the barriers to establishing iSCSI as a device interface. > > > If iSCSI works at ALL in a cost effective way that can be implemented > > in a disk, there'll be wild dancing in the streets and you'll all (or > > maybe your companies will) be rich beyond the dreams of avarice. > > > > The easier you can make it for the implementors, the more likely it > > will succeed. > > Disk drive companies have implemented much more complex interfaces than > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > connections don't look very hard to implement. They just look like a wart. > But I think a necessary one. > > Regards, > -Steve > > Steve Byan > > Design Engineer > MS 1-3/E23 > 333 South Street > Shrewsbury, MA 01545 > (508)770-3414 > fax: (508)770-2604 > From owner-ips@ECE.cmu.edu Sat Aug 12 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26151; 12 Aug 2000 4:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA23820 for ips-outgoing; Fri, 11 Aug 2000 14:28:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA23815 for ; Fri, 11 Aug 2000 14:28:07 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA02833; Fri, 11 Aug 2000 11:23:58 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA00304; Fri, 11 Aug 2000 11:21:20 -0700 From: "Douglas Otis" To: "VonStamwitz, Paul" Cc: "Ips" Subject: RE: Adding LUN as a field within the command structure. Date: Fri, 11 Aug 2000 11:30:09 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <5B0C98C89640D311A319002048406A1C05B31C93@aimexc03.corp.adaptec.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Paul, As iSCSI advocates thousands of LUNs per session, then search for the LUN over a range multiplied by the number of outstanding tags unique for all LUNS as well. If TCP is in hardware, how is innovation helping? RTT uses just the Tag. What a mnemonic. RTT should mean Round Trip Time in an IETF specification. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > VonStamwitz, Paul > Sent: Thursday, August 10, 2000 6:52 PM > To: 'Douglas Otis' > Cc: Ips > Subject: RE: Adding LUN as a field within the command structure. > > > > Doug wrote: > > I vote no. > > > If you are making hardware to process TCP, this hardware is > able to handle > > more than a few connections. There will *always* be a port > sort operation > > to find the connection and associated state information. It is not more > > effective to require an additional sort to locate LUN as a > second tier of > > state information. > > There is another second tier of state information (or is it > tertiary?) that > would require sorting, and that is the tag. > > > The ability of the client to index the entire volume of > > devices possible is remote and will be a larger constraint. > > If you map a group of LUNs to a connection, then the indexing is for that > group only and not the entire volume. > > The feedback we received is that a connection/LUN is desirable, but making > it a requirement is too restrictive. > > -Paul > From owner-ips@ECE.cmu.edu Sat Aug 12 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26164; 12 Aug 2000 4:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA14485 for ips-outgoing; Sat, 12 Aug 2000 03:26:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA14479 for ; Sat, 12 Aug 2000 03:26:49 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA38424 for ; Sat, 12 Aug 2000 09:26:18 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id JAA37976 for ; Sat, 12 Aug 2000 09:26:18 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256939.0028DA4F ; Sat, 12 Aug 2000 09:26:13 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 12 Aug 2000 10:24:27 +0300 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Well - I understand the argument about a connection/LU and I did even implement it (as many others - see Paul's reply). You can aggregate at the TCP level but you have to aggregate somewhere to use effectively the bandwidth. Eliminating the CU is not a concern of this group and not a suggestion to be accepted lightly by the community (who will do caching, storage virtualization and management etc.) and I am confident most of the participants on this list do not want to discuss this subject. FC over IP as replacement for iSCSI is a poor suggestion - FC is a networking protocol and IP too. FCP over IP is a gateway solution. Julo "Douglas Otis" on 12/08/2000 02:47:37 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Towards Consensus on TCP Connections Julo, An architecture that scales does not need to provide the entire bandwidth from a single device. If these devices are each handling 25 Mbytes per second, then 2 of them exceed Fibre-Channel. Next week you could use 4, and the week after that 8. The advantage of using a network without any practical architectural restriction with intelligence at the client. On the other hand, you want to design a single mid-point gateway to handle the entire bandwidth. To what end? You argue it can not process a TCP session per device. You will not be tracking mid-stream errors as this is more costly. You add to the burden of tracking the state of the individual device with this mid-stream state machine requiring additional sorting due to this merged protocol without taking advantage of TCP to aid this process. For your type of solution, a simple hardware based tunnel would be better. Do not include handling of the encapsulated protocol and at least it becomes understandable and more likely to manage the task. In other words, make it Fibre-Channel over IP and you have a chance with your architecture. The point to my statements was to indicate the device is able to handle an IP interface today using Fast Ethernet as the rate from the drive is relatively low. Not to warm the cockles of marketing pointing to latency, but at least they sell more drives overcoming this problem using scale. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Friday, August 11, 2000 2:17 PM > To: ips@ece.cmu.edu > Subject: RE: Towards Consensus on TCP Connections > > > > > Doug, > > I am not sure that I agree with your architecture statements but I like to > play with numbers (as most of the fellows engineers on this list probably > do). What would be in your opinion > reasonable requirements for command and data transfer rates for the next > 3-7 years? > > I would like to decouple that discussion from architecture - data > rates can > scale even in a shared > architecture as mainframe channels have shown for years. > > I would rather like to understand if we can meet the data rates with > reasonable latency. > > Julo > > "Douglas Otis" on 11/08/2000 21:14:13 > > Please respond to "Douglas Otis" > > To: "Stephen Byan" , ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Towards Consensus on TCP Connections > > > > > Today's drives can deliver 320 Mbits/second of data on the outside > cylinders. Improvement of the mechanics comes at a high price > with respect > to power and cost. The cost/volume trend takes us to a single disk which > increases access time as read channel data rate increases. By offering > scaled throughput using more drives where each drive's interface bandwidth > is restricted with respect to read channel data rates provides a system > with > uniform and superior performance. The advantage of such an approach is > found with respect to smaller random traffic. With more devices, > redundancy > is easily achieved and parallel access offers a means of performance > improvement by spreading activity over more devices. The switch provides > bandwidth aggregation and is not found in the individual device. > > An 8ms access + latency figure in the high cost drives restricts > the number > of 'independent' operations that average 64k byte to 100 per second or 52 > Mbit per second. Such an architecture of 'restricted' drives would scale > whereas the solicitated burst approach does not. An independent nexus at > the LUN is the only design that offers required scaling and configuration > flexibility. Keeping up with the read channel is a wasted effort. In > time, > 1 Gbit Ethernet will be the practical solution about the time drives are 1 > inche in size. Several Fast Ethernet disks combined at a 1 Gbit Ethernet > client makes sense in cost, performance, capacity, reliability, and > scalability at this point in time. The protocol overhead should be > addressed. There are substantial improvements to be made to allow this > innovation using standard adapters. > > The power cost to use copper 1 Gbit is high. Firewire does not scale and > has a limited reach. Firewire also places scatter/gather on the drive > together with direct access. Doing such over a WAN will impose > significant > changes. Serial ATA is nothing more than IDE through a SERDES. The read > channel data rate is like a drug, just say no. It is hard not to buy > enough > dram to allow a proper buffer these days. Serial ATA removes all buffers. > Intel is just usurping any remaining electronics at the cost of > sensitivity > to a near by cell phone. Fewer drives with less electronics. What a good > idea? > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Stephen Byan > > Sent: Friday, August 11, 2000 7:07 AM > > To: 'ips@ece.cmu.edu' > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > > MB/s instead of 100 MB/s out of a single LUN. > > > > In general, I agree. iSCSI can succeed in the high and midrange storage > > market without link aggregation for a single LUN. These markets can > afford > > 10 Gb/s links. > > > > As a disk device level interface, iSCSI will not succeed unless > > it offers at > > least 2 Gb/s by around 2002, at very low cost for the link. Note that > even > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA > speed > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > http://serialata.org/F9pp.pdf. > > > > One can argue the technical merits, but from a marketing > > viewpoint, the disk > > industry (both suppliers and customers) has long held the view that > > interface speeds need to match the media data rate. iSCSI can try > > to make an > > argument that slower speeds are technically adequate, but this > > will increase > > the barriers to establishing iSCSI as a device interface. > > > > > If iSCSI works at ALL in a cost effective way that can be implemented > > > in a disk, there'll be wild dancing in the streets and you'll all (or > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > The easier you can make it for the implementors, the more likely it > > > will succeed. > > > > Disk drive companies have implemented much more complex interfaces than > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > > connections don't look very hard to implement. They just look like a > wart. > > But I think a necessary one. > > > > Regards, > > -Steve > > > > Steve Byan > > > > Design Engineer > > MS 1-3/E23 > > 333 South Street > > Shrewsbury, MA 01545 > > (508)770-3414 > > fax: (508)770-2604 > > > > > > From owner-ips@ECE.cmu.edu Sat Aug 12 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28594; 12 Aug 2000 13:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA03786 for ips-outgoing; Sat, 12 Aug 2000 12:32:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA03782 for ; Sat, 12 Aug 2000 12:32:12 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0FZ600CNCTW0HO@mta6.snfc21.pbi.net> for IPS@ECE.CMU.edu; Sat, 12 Aug 2000 09:31:13 -0700 (PDT) Date: Sat, 12 Aug 2000 09:28:37 -0700 From: Peter Johansson Subject: RE: Towards Consensus on TCP Connections In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000812091559.00bcdcd0@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed References: <8133266FE373D11190CD00805FA768BF02EEA05E@shrcmsg1.tdh.qntm.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:14 AM 8/11/00, Douglas Otis wrote: >Firewire does not scale and has a limited reach. Doug, I take exception to the first part of your statement, both because I believe that IEEE 1394 does scale and for another reason (see below). As for limited reach, I agree that point-to-point connections measured in the hundreds of meters are not adequate for a MAN / WAN. But I also don't think that consideration is relevant to iSCSI (from the protocol perspective), whose purpose is to permit a diversity of media solutions appropriate to each user's needs. >Firewire also places scatter/gather on the drive together with direct access. I think you're confusing SBP-2 (which does locate the scatter / gather chore in the disk drive) with IEEE 1394 (which is agnostic about such matters). SBP-2 cannot be "native" iSCSI any more than can FCP! It's my understanding that the charter of iSCSI is to be transport neutral. That is, iSCSI is supposed to run over anything---copper, fibre, tautly stretched string---that supports Internet protocol. Is this belief in error? If that assumption is correct, then when it comes time to compare and contrast the cost and efficiency of different media, IEEE 1394 ought to be in the matrix along with Ethernet, FC, UTP-5, etc., etc. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Sun Aug 13 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30817; 12 Aug 2000 22:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA17100 for ips-outgoing; Sat, 12 Aug 2000 21:37:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA17094 for ; Sat, 12 Aug 2000 21:37:29 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id SAA27419; Sat, 12 Aug 2000 18:37:26 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.189.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id SAA20087; Sat, 12 Aug 2000 18:33:40 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id SAA13626; Sat, 12 Aug 2000 18:33:39 -0700 (PDT) Message-ID: <3995F9DA.BA07F2A0@ebay.sun.com> Date: Sat, 12 Aug 2000 18:28:58 -0700 From: David Robinson X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: DRAFT Pittsburgh ips Minutes References: <0F31E5C394DAD311B60C00E029101A0704100EA7@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com wrote: > > An important clarification has been pointed out in the DRAFT minutes. > The DRAFT minutes say: > > Consensus call > -------------- > There was a consensus call eventually. Should encryption be mandatory? > > The consensus in the room is that encryption of the stream should > not be mandatory. A lot of people were concerned about the overhead > it would imply on cheap devices. > > The consensus call question was actually "should implementation > of encryption be mandatory"? Since the consensus was "no", > implementation of encryption will not be required by the specification. Point of order, the concensus call MUST be made on the mailing list, the minutes only reflect the concensus of the room, not the WG. > Note that implementation of authentication and related data integrity > measures (e.g., that make it impossible to insert data into or hijack > an authenticated session) will be mandatory, per the AD (i.e., if they > are not mandatory in the spec that the WG produces, the spec > will be returned to the WG with instructions to make it so). For those not in the room, as David said the question was not to allow the possiblity of privacy but if it would be manditory to implement in order to be conforming to the Standard. Everyone seemed to agree that regardless of the outcome of the consensus, there MUST be a mechanism to negotiate privacy. In looking at who in the room was in favor of making privacy manditory to implement, it was clear that most were those that have already struggled with this question in the NFSv4 WG where it was made manditory to implement. The primary motivating factor in the NFSv4 WG for this decision was to ensure that a customer who buys a conforming implentation would have assurances that privacy was available if needed. As with the discussions in Pitt most NFSv4 participants believe that by default most customers and probably most implentations will negotiate away privacy for performance reasons. However, it was strongly felt that if it was not manditory to implement it wouldn't be implemented. NFS has long been burned by the reputation that it is insecure even though there have been defacto standard ways to make it secure for years. The root cause was the lack of availablity of implementations. Having lived through this for a long time, I strongly caution the WG that failing to make the set of authentication/integrity/privacy manditory to implement will doom the IPS standard to use only on isolated networks and not achieve the promise described in the drafts as no one will trust it. In recent trade-rag articles about IPS, security is already one of the leading causes of doubt to the success of the standard. -David From owner-ips@ECE.cmu.edu Sun Aug 13 06:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31641; 13 Aug 2000 1:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA20481 for ips-outgoing; Sun, 13 Aug 2000 00:08:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA20476 for ; Sun, 13 Aug 2000 00:08:27 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id GAA38906 for ; Sun, 13 Aug 2000 06:07:56 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id GAA24722 for ; Sun, 13 Aug 2000 06:07:55 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693A.0016AF18 ; Sun, 13 Aug 2000 06:07:46 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 13 Aug 2000 06:53:42 +0300 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk IEEE 1394 is a legitimate contender for the local area - and provided that prices are kept low by the competition in the consumer space - may make iSCSI and interesting control scheme for a wide variety of devices - including but not limited to storage. Julo Peter Johansson on 12/08/2000 19:28:37 Please respond to Peter Johansson To: IP Storage cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Towards Consensus on TCP Connections At 11:14 AM 8/11/00, Douglas Otis wrote: >Firewire does not scale and has a limited reach. Doug, I take exception to the first part of your statement, both because I believe that IEEE 1394 does scale and for another reason (see below). As for limited reach, I agree that point-to-point connections measured in the hundreds of meters are not adequate for a MAN / WAN. But I also don't think that consideration is relevant to iSCSI (from the protocol perspective), whose purpose is to permit a diversity of media solutions appropriate to each user's needs. >Firewire also places scatter/gather on the drive together with direct access. I think you're confusing SBP-2 (which does locate the scatter / gather chore in the disk drive) with IEEE 1394 (which is agnostic about such matters). SBP-2 cannot be "native" iSCSI any more than can FCP! It's my understanding that the charter of iSCSI is to be transport neutral. That is, iSCSI is supposed to run over anything---copper, fibre, tautly stretched string---that supports Internet protocol. Is this belief in error? If that assumption is correct, then when it comes time to compare and contrast the cost and efficiency of different media, IEEE 1394 ought to be in the matrix along with Ethernet, FC, UTP-5, etc., etc. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Mon Aug 14 18:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08530; 14 Aug 2000 13:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA27600 for ips-outgoing; Mon, 14 Aug 2000 12:06:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA27594 for ; Mon, 14 Aug 2000 12:06:07 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA38364 for ; Mon, 14 Aug 2000 18:05:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id SAA22938 for ; Mon, 14 Aug 2000 18:05:24 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693B.0058600B ; Mon, 14 Aug 2000 18:05:17 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 14 Aug 2000 19:03:15 +0300 Subject: RE: DRAFT Pittsburgh ips Minutes Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk This is an accurate description of what was said. One additional and relevant element in the discussion was the proposal made my Brian Carpenter to mandate implementations to "understand" any form of encryption and authentication but indicate that support or lack of support when asked. That might be appropriate for simple controllers aimed at closed environments and deferring authentication/encryption to a "proxy" (a firewall) for all relevant flows. I assume that we will also want to specifically state which security elements are mandatory for an initiator and which for a target. Julo Black_David@emc.com on 12/08/2000 03:38:16 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: DRAFT Pittsburgh ips Minutes An important clarification has been pointed out in the DRAFT minutes. The DRAFT minutes say: Consensus call -------------- There was a consensus call eventually. Should encryption be mandatory? The consensus in the room is that encryption of the stream should not be mandatory. A lot of people were concerned about the overhead it would imply on cheap devices. The consensus call question was actually "should implementation of encryption be mandatory"? Since the consensus was "no", implementation of encryption will not be required by the specification. Note that implementation of authentication and related data integrity measures (e.g., that make it impossible to insert data into or hijack an authenticated session) will be mandatory, per the AD (i.e., if they are not mandatory in the spec that the WG produces, the spec will be returned to the WG with instructions to make it so). --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Aug 14 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08942; 14 Aug 2000 15:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA00351 for ips-outgoing; Mon, 14 Aug 2000 13:22:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA00343 for ; Mon, 14 Aug 2000 13:22:38 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA03266; Mon, 14 Aug 2000 10:17:56 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA10144; Mon, 14 Aug 2000 10:15:23 -0700 From: "Douglas Otis" To: "Peter Johansson" , "IP Storage" Subject: RE: Towards Consensus on TCP Connections Date: Mon, 14 Aug 2000 10:24:47 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-reply-to: <4.3.2.7.2.20000812091559.00bcdcd0@PacBell.net> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, Firewire has many strikes against it for broad acceptance. It uses an imbalanced pair to identify the PHY. Licensed PHY from Apple at a cost per port with mode use restricted. Limited range of the copper solution does not satisfy most commercial environments. No aggregation as Firewire is a half-duplex daisy chain. Although possible to use as a typical SCSI target with IP, it is not for storage. With that said, indeed, implementing any IP standard on any medium would be encouraged. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Peter Johansson > Sent: Saturday, August 12, 2000 9:29 AM > To: IP Storage > Subject: RE: Towards Consensus on TCP Connections > > > At 11:14 AM 8/11/00, Douglas Otis wrote: > > >Firewire does not scale and has a limited reach. > > Doug, I take exception to the first part of your statement, both > because I > believe that IEEE 1394 does scale and for another reason (see below). As > for limited reach, I agree that point-to-point connections > measured in the > hundreds of meters are not adequate for a MAN / WAN. But I also > don't think > that consideration is relevant to iSCSI (from the protocol perspective), > whose purpose is to permit a diversity of media solutions appropriate to > each user's needs. > > >Firewire also places scatter/gather on the drive together with > direct access. > > I think you're confusing SBP-2 (which does locate the scatter / gather > chore in the disk drive) with IEEE 1394 (which is agnostic about such > matters). SBP-2 cannot be "native" iSCSI any more than can FCP! > > It's my understanding that the charter of iSCSI is to be > transport neutral. > That is, iSCSI is supposed to run over anything---copper, fibre, tautly > stretched string---that supports Internet protocol. Is this > belief in error? > > If that assumption is correct, then when it comes time to compare and > contrast the cost and efficiency of different media, IEEE 1394 > ought to be > in the matrix along with Ethernet, FC, UTP-5, etc., etc. > > > Regards, > > Peter Johansson > > Congruent Software, Inc. > 98 Colorado Avenue > Berkeley, CA 94707 > > (510) 527-3926 > (510) 527-3856 FAX > > PJohansson@ACM.org > From owner-ips@ECE.cmu.edu Mon Aug 14 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09182; 14 Aug 2000 16:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA03858 for ips-outgoing; Mon, 14 Aug 2000 14:51:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA03853 for ; Mon, 14 Aug 2000 14:51:00 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0FZA00AA6P3Z0R@mta5.snfc21.pbi.net> for IPS@ECE.CMU.edu; Mon, 14 Aug 2000 11:38:24 -0700 (PDT) Date: Mon, 14 Aug 2000 11:36:27 -0700 From: Peter Johansson Subject: RE: Towards Consensus on TCP Connections In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000814111853.00bd7e50@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed References: <4.3.2.7.2.20000812091559.00bcdcd0@PacBell.net> Sender: owner-ips@ece.cmu.edu Precedence: bulk At 10:24 AM 8/14/00, Douglas Otis wrote: >Firewire has many strikes against it for broad acceptance. Not the least of which is the surprising persistence of outdated and inaccurate information. >It uses animbalanced pair to identify the PHY. I'm not sure I understand your assertion---nor what you perceive as a problem. >Licensed PHY from Apple at a cost per port with mode use restricted. Not true. Various owners of intellectual property pertinent to IEEE 1394 formed what is informally known as the "patent pool"; the 1394 Licensing Authority makes licenses available on reasonable terms. Not only are the charges NOT per port, as you say, they are per INTEGRATED SYSTEM (not even component) and at a flat rate per system. >Limited range of the copper solution does not satisfy most commercial >environments. Partially correct, but applicable only to the DS-encoding over copper specified by IEEE 1394-1995. The diversity of media specified by P1394b (glass fibre, plastic fibre, UTP-5) provide point-to-point lengths in the hundreds of meters. Since this is the same geographical range proposed by interconnects such as Infiniband or gigabit Ethernet, it seems suited to some commercial environments. >No aggregation as Firewire is a half-duplex daisy chain. Please explain more; I do not understand your point. What is being aggregated? Or failing to be aggregated? Give me a concrete example. >Although possible to use as a typical SCSI target with IP, it is not for >storage. IEEE 1394 is well-suited to storage, as evidenced by the performance characteristics of SBP-2 disks. As previously mentioned, though, neither SBP-2 nor other "native" SCSI transports (such as FCP) are compatible with the group's direction with iSCSI. But that has nothing to do with the underlying utility of IEEE 1394 of Fibre Channel as an IP transport which is, in turn, used by iSCSI. >... implementing any IP standard on any medium would be encouraged. Agreed. See RFC 2734 for the details. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Mon Aug 14 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09190; 14 Aug 2000 16:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA00334 for ips-outgoing; Mon, 14 Aug 2000 13:22:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA00320 for ; Mon, 14 Aug 2000 13:22:22 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA03262; Mon, 14 Aug 2000 10:17:39 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA10138; Mon, 14 Aug 2000 10:15:07 -0700 From: "Douglas Otis" To: , Subject: RE: Towards Consensus on TCP Connections Date: Mon, 14 Aug 2000 10:24:29 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-reply-to: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, Keep the interface suitable for the end device. A controller in the middle does not justify a unique interface for that scenario. In trying to optimize "Bandwidth utilization", you are binding thousands of logical units together into the same session. This is an added burden within the standard, if used at the device. In doing so, session tags must be kept unique across all logical units. Reset all drives to recover from even a single tag error? If you wish to cache information at the gateway, it should still use the interface suitable for the end device. At least this isolates error handling. By creating a simple FC tunnel, the controller remains bolted into the back-plane where it is known to work effectively. Error handling is understood with a stateless translation and you have not created a standard that will set back efforts at getting a solution without myopic compromises. There WILL be errors not seen by this "In the Middle Controller" translating Fibre-Channel into iSCSI as FC does not have the same transport. States within this ITMC adds to complexity and error migration when you intertwine logical units. ITMC is a poor starting point and only makes for a bad interface as seen by this specification. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Saturday, August 12, 2000 12:24 AM > To: ips@ece.cmu.edu > Subject: RE: Towards Consensus on TCP Connections > > > > > Well - I understand the argument about a connection/LU and I did even > implement it (as many others - see Paul's reply). You can aggregate at the > TCP level but you have to aggregate somewhere > to use effectively the bandwidth. Eliminating the CU is not a concern of > this group and not a suggestion to be accepted lightly by the community > (who will do caching, storage virtualization and management etc.) and I am > confident most of the participants on this list do not want to discuss > this subject. FC over IP as replacement for iSCSI is a poor suggestion - > FC is a networking protocol and IP too. FCP over IP is a gateway solution. > > Julo > > "Douglas Otis" on 12/08/2000 02:47:37 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: Towards Consensus on TCP Connections > > > > > Julo, > > An architecture that scales does not need to provide the entire bandwidth > from a single device. If these devices are each handling 25 Mbytes per > second, then 2 of them exceed Fibre-Channel. Next week you could use 4, > and > the week after that 8. The advantage of using a network without any > practical architectural restriction with intelligence at the client. On > the > other hand, you want to design a single mid-point gateway to handle the > entire bandwidth. To what end? You argue it can not process a > TCP session > per device. You will not be tracking mid-stream errors as this is more > costly. You add to the burden of tracking the state of the individual > device with this mid-stream state machine requiring additional sorting due > to this merged protocol without taking advantage of TCP to aid this > process. > For your type of solution, a simple hardware based tunnel would be better. > Do not include handling of the encapsulated protocol and at least it > becomes > understandable and more likely to manage the task. In other > words, make it > Fibre-Channel over IP and you have a chance with your architecture. > > The point to my statements was to indicate the device is able to handle an > IP interface today using Fast Ethernet as the rate from the drive is > relatively low. Not to warm the cockles of marketing pointing to latency, > but at least they sell more drives overcoming this problem using scale. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Friday, August 11, 2000 2:17 PM > > To: ips@ece.cmu.edu > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Doug, > > > > I am not sure that I agree with your architecture statements but I like > to > > play with numbers (as most of the fellows engineers on this > list probably > > do). What would be in your opinion > > reasonable requirements for command and data transfer rates for the next > > 3-7 years? > > > > I would like to decouple that discussion from architecture - data > > rates can > > scale even in a shared > > architecture as mainframe channels have shown for years. > > > > I would rather like to understand if we can meet the data rates with > > reasonable latency. > > > > Julo > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > Please respond to "Douglas Otis" > > > > To: "Stephen Byan" , ips@ece.cmu.edu > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data on the outside > > cylinders. Improvement of the mechanics comes at a high price > > with respect > > to power and cost. The cost/volume trend takes us to a single > disk which > > increases access time as read channel data rate increases. By offering > > scaled throughput using more drives where each drive's interface > bandwidth > > is restricted with respect to read channel data rates provides a system > > with > > uniform and superior performance. The advantage of such an approach is > > found with respect to smaller random traffic. With more devices, > > redundancy > > is easily achieved and parallel access offers a means of performance > > improvement by spreading activity over more devices. The > switch provides > > bandwidth aggregation and is not found in the individual device. > > > > An 8ms access + latency figure in the high cost drives restricts > > the number > > of 'independent' operations that average 64k byte to 100 per > second or 52 > > Mbit per second. Such an architecture of 'restricted' drives > would scale > > whereas the solicitated burst approach does not. An > independent nexus at > > the LUN is the only design that offers required scaling and > configuration > > flexibility. Keeping up with the read channel is a wasted effort. In > > time, > > 1 Gbit Ethernet will be the practical solution about the time drives are > 1 > > inche in size. Several Fast Ethernet disks combined at a 1 > Gbit Ethernet > > client makes sense in cost, performance, capacity, reliability, and > > scalability at this point in time. The protocol overhead should be > > addressed. There are substantial improvements to be made to allow this > > innovation using standard adapters. > > > > The power cost to use copper 1 Gbit is high. Firewire does not > scale and > > has a limited reach. Firewire also places scatter/gather on the drive > > together with direct access. Doing such over a WAN will impose > > significant > > changes. Serial ATA is nothing more than IDE through a SERDES. > The read > > channel data rate is like a drug, just say no. It is hard not to buy > > enough > > dram to allow a proper buffer these days. Serial ATA removes all > buffers. > > Intel is just usurping any remaining electronics at the cost of > > sensitivity > > to a near by cell phone. Fewer drives with less electronics. What a > good > > idea? > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Stephen Byan > > > Sent: Friday, August 11, 2000 7:07 AM > > > To: 'ips@ece.cmu.edu' > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > In general, I agree. iSCSI can succeed in the high and > midrange storage > > > market without link aggregation for a single LUN. These markets can > > afford > > > 10 Gb/s links. > > > > > > As a disk device level interface, iSCSI will not succeed unless > > > it offers at > > > least 2 Gb/s by around 2002, at very low cost for the link. Note that > > even > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA > > speed > > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > > http://serialata.org/F9pp.pdf. > > > > > > One can argue the technical merits, but from a marketing > > > viewpoint, the disk > > > industry (both suppliers and customers) has long held the view that > > > interface speeds need to match the media data rate. iSCSI can try > > > to make an > > > argument that slower speeds are technically adequate, but this > > > will increase > > > the barriers to establishing iSCSI as a device interface. > > > > > > > If iSCSI works at ALL in a cost effective way that can be > implemented > > > > in a disk, there'll be wild dancing in the streets and > you'll all (or > > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > > > The easier you can make it for the implementors, the more likely it > > > > will succeed. > > > > > > Disk drive companies have implemented much more complex > interfaces than > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > > > connections don't look very hard to implement. They just look like a > > wart. > > > But I think a necessary one. > > > > > > Regards, > > > -Steve > > > > > > Steve Byan > > > > > > Design Engineer > > > MS 1-3/E23 > > > 333 South Street > > > Shrewsbury, MA 01545 > > > (508)770-3414 > > > fax: (508)770-2604 > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Tue Aug 15 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13790; 15 Aug 2000 9:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA09132 for ips-outgoing; Tue, 15 Aug 2000 08:19:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA09124 for ; Tue, 15 Aug 2000 08:19:49 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id OAA310460 for ; Tue, 15 Aug 2000 14:19:16 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id OAA50632 for ; Tue, 15 Aug 2000 14:19:12 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693C.0043ABD6 ; Tue, 15 Aug 2000 14:19:09 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 15 Aug 2000 15:17:17 +0300 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, You can use the standard at the end user device, if you so wish. If you choose a "transprent" controller - i.e. one not holding any state or ordering - and view every LU as and independent unit - you ignore the numbering (that you are free to do anyway for the commands) and ship the commands to the device for execution. Initiator tags are unique per initiator and it should not be a problem to keep the unique per session. Target task tags are a different issue. We decided to remove the LUN field in both the incoming RTT and the the outgoing data to avoid the need to test for inconsistencies. But as this is the only thing that implies unique target task tags we might as well "reinstate" the LUN (the field is still unused) in the DATA packet (in the RTT it is implied by the initiator task tag). Relaxing the requirement of uniqueness for initiator tag is possible too (you can have the unique per LU) but that might complicate unecessarily the initiator. If this (minor) change does not anoy target controller implementors and nobody sees something else wrong with it we might introduce it in the next version. As for your FC claims - I still can't follow your argument. Are you talking about FC (the network protocol) or FCP-2 (SCSI-over-FC)? Julo "Douglas Otis" on 14/08/2000 20:24:29 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Towards Consensus on TCP Connections Julo, Keep the interface suitable for the end device. A controller in the middle does not justify a unique interface for that scenario. In trying to optimize "Bandwidth utilization", you are binding thousands of logical units together into the same session. This is an added burden within the standard, if used at the device. In doing so, session tags must be kept unique across all logical units. Reset all drives to recover from even a single tag error? If you wish to cache information at the gateway, it should still use the interface suitable for the end device. At least this isolates error handling. By creating a simple FC tunnel, the controller remains bolted into the back-plane where it is known to work effectively. Error handling is understood with a stateless translation and you have not created a standard that will set back efforts at getting a solution without myopic compromises. There WILL be errors not seen by this "In the Middle Controller" translating Fibre-Channel into iSCSI as FC does not have the same transport. States within this ITMC adds to complexity and error migration when you intertwine logical units. ITMC is a poor starting point and only makes for a bad interface as seen by this specification. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Saturday, August 12, 2000 12:24 AM > To: ips@ece.cmu.edu > Subject: RE: Towards Consensus on TCP Connections > > > > > Well - I understand the argument about a connection/LU and I did even > implement it (as many others - see Paul's reply). You can aggregate at the > TCP level but you have to aggregate somewhere > to use effectively the bandwidth. Eliminating the CU is not a concern of > this group and not a suggestion to be accepted lightly by the community > (who will do caching, storage virtualization and management etc.) and I am > confident most of the participants on this list do not want to discuss > this subject. FC over IP as replacement for iSCSI is a poor suggestion - > FC is a networking protocol and IP too. FCP over IP is a gateway solution. > > Julo > > "Douglas Otis" on 12/08/2000 02:47:37 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: Towards Consensus on TCP Connections > > > > > Julo, > > An architecture that scales does not need to provide the entire bandwidth > from a single device. If these devices are each handling 25 Mbytes per > second, then 2 of them exceed Fibre-Channel. Next week you could use 4, > and > the week after that 8. The advantage of using a network without any > practical architectural restriction with intelligence at the client. On > the > other hand, you want to design a single mid-point gateway to handle the > entire bandwidth. To what end? You argue it can not process a > TCP session > per device. You will not be tracking mid-stream errors as this is more > costly. You add to the burden of tracking the state of the individual > device with this mid-stream state machine requiring additional sorting due > to this merged protocol without taking advantage of TCP to aid this > process. > For your type of solution, a simple hardware based tunnel would be better. > Do not include handling of the encapsulated protocol and at least it > becomes > understandable and more likely to manage the task. In other > words, make it > Fibre-Channel over IP and you have a chance with your architecture. > > The point to my statements was to indicate the device is able to handle an > IP interface today using Fast Ethernet as the rate from the drive is > relatively low. Not to warm the cockles of marketing pointing to latency, > but at least they sell more drives overcoming this problem using scale. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Friday, August 11, 2000 2:17 PM > > To: ips@ece.cmu.edu > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Doug, > > > > I am not sure that I agree with your architecture statements but I like > to > > play with numbers (as most of the fellows engineers on this > list probably > > do). What would be in your opinion > > reasonable requirements for command and data transfer rates for the next > > 3-7 years? > > > > I would like to decouple that discussion from architecture - data > > rates can > > scale even in a shared > > architecture as mainframe channels have shown for years. > > > > I would rather like to understand if we can meet the data rates with > > reasonable latency. > > > > Julo > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > Please respond to "Douglas Otis" > > > > To: "Stephen Byan" , ips@ece.cmu.edu > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data on the outside > > cylinders. Improvement of the mechanics comes at a high price > > with respect > > to power and cost. The cost/volume trend takes us to a single > disk which > > increases access time as read channel data rate increases. By offering > > scaled throughput using more drives where each drive's interface > bandwidth > > is restricted with respect to read channel data rates provides a system > > with > > uniform and superior performance. The advantage of such an approach is > > found with respect to smaller random traffic. With more devices, > > redundancy > > is easily achieved and parallel access offers a means of performance > > improvement by spreading activity over more devices. The > switch provides > > bandwidth aggregation and is not found in the individual device. > > > > An 8ms access + latency figure in the high cost drives restricts > > the number > > of 'independent' operations that average 64k byte to 100 per > second or 52 > > Mbit per second. Such an architecture of 'restricted' drives > would scale > > whereas the solicitated burst approach does not. An > independent nexus at > > the LUN is the only design that offers required scaling and > configuration > > flexibility. Keeping up with the read channel is a wasted effort. In > > time, > > 1 Gbit Ethernet will be the practical solution about the time drives are > 1 > > inche in size. Several Fast Ethernet disks combined at a 1 > Gbit Ethernet > > client makes sense in cost, performance, capacity, reliability, and > > scalability at this point in time. The protocol overhead should be > > addressed. There are substantial improvements to be made to allow this > > innovation using standard adapters. > > > > The power cost to use copper 1 Gbit is high. Firewire does not > scale and > > has a limited reach. Firewire also places scatter/gather on the drive > > together with direct access. Doing such over a WAN will impose > > significant > > changes. Serial ATA is nothing more than IDE through a SERDES. > The read > > channel data rate is like a drug, just say no. It is hard not to buy > > enough > > dram to allow a proper buffer these days. Serial ATA removes all > buffers. > > Intel is just usurping any remaining electronics at the cost of > > sensitivity > > to a near by cell phone. Fewer drives with less electronics. What a > good > > idea? > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Stephen Byan > > > Sent: Friday, August 11, 2000 7:07 AM > > > To: 'ips@ece.cmu.edu' > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > The gating factor for whether iSCSI succeeds is not going to be 200 > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > In general, I agree. iSCSI can succeed in the high and > midrange storage > > > market without link aggregation for a single LUN. These markets can > > afford > > > 10 Gb/s links. > > > > > > As a disk device level interface, iSCSI will not succeed unless > > > it offers at > > > least 2 Gb/s by around 2002, at very low cost for the link. Note that > > even > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA > > speed > > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > > http://serialata.org/F9pp.pdf. > > > > > > One can argue the technical merits, but from a marketing > > > viewpoint, the disk > > > industry (both suppliers and customers) has long held the view that > > > interface speeds need to match the media data rate. iSCSI can try > > > to make an > > > argument that slower speeds are technically adequate, but this > > > will increase > > > the barriers to establishing iSCSI as a device interface. > > > > > > > If iSCSI works at ALL in a cost effective way that can be > implemented > > > > in a disk, there'll be wild dancing in the streets and > you'll all (or > > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > > > The easier you can make it for the implementors, the more likely it > > > > will succeed. > > > > > > Disk drive companies have implemented much more complex > interfaces than > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > > > connections don't look very hard to implement. They just look like a > > wart. > > > But I think a necessary one. > > > > > > Regards, > > > -Steve > > > > > > Steve Byan > > > > > > Design Engineer > > > MS 1-3/E23 > > > 333 South Street > > > Shrewsbury, MA 01545 > > > (508)770-3414 > > > fax: (508)770-2604 > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Tue Aug 15 15:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14214; 15 Aug 2000 11:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA13036 for ips-outgoing; Tue, 15 Aug 2000 10:12:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA13025 for ; Tue, 15 Aug 2000 10:12:13 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id C5A3BB06; Tue, 15 Aug 2000 07:12:12 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA01600; Tue, 15 Aug 2000 07:11:22 -0700 (PDT) Message-Id: <4.2.2.20000815064454.00b2fc70@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 15 Aug 2000 06:46:25 -0700 To: Peter Johansson , IP Storage From: Michael Krause Subject: RE: Towards Consensus on TCP Connections In-Reply-To: <4.3.2.7.2.20000814111853.00bd7e50@PacBell.net> References: <4.3.2.7.2.20000812091559.00bcdcd0@PacBell.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:36 AM 8/14/00 -0700, Peter Johansson wrote: >At 10:24 AM 8/14/00, Douglas Otis wrote: > >Partially correct, but applicable only to the DS-encoding over copper >specified by IEEE 1394-1995. The diversity of media specified by P1394b >(glass fibre, plastic fibre, UTP-5) provide point-to-point lengths in the >hundreds of meters. Since this is the same geographical range proposed by >interconnects such as Infiniband or gigabit Ethernet, it seems suited to >some commercial environments. InfiniBand is 300 meters or less. 10 GbE will drive to 40 Km. Mike From owner-ips@ECE.cmu.edu Tue Aug 15 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14797; 15 Aug 2000 13:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA18929 for ips-outgoing; Tue, 15 Aug 2000 12:22:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA18923 for ; Tue, 15 Aug 2000 12:22:27 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA03505; Tue, 15 Aug 2000 09:17:33 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA13494; Tue, 15 Aug 2000 09:15:01 -0700 From: "Douglas Otis" To: , Subject: RE: Towards Consensus on TCP Connections Date: Tue, 15 Aug 2000 09:24:34 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, For details on proposed Fibre-Channel over IP which does not impose additional catastrophic error states as does iSCSI, please see: http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt "While the FC over IP specification is independent of the link level transport protocol, it assumes a high bandwidth, high reliability, low loss link level transport such as Gigabit Ethernet, SONET, ATM, or DWDM. This specification treats all classes of FC frames the same -- as datagrams." ipfc "Fibre Channel over IP's main advantage is it needs no modifications to the storage subsystem or the server operating system. SCSI over TCP/IP, on the other hand, requires filter drivers so the server operating system can do IP-network-to-SAN emulation (for which I humbly propose the acronym INSANE)." Wayne Rickard Network World, 06/12/00 wayne@gadzoox.com To see who is doing Fibre-Channel encapsulation see: http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_061300b.htm At keeping the same architecture as the present Fibre-Channel configuration, you have not improved reliability. By making compromises that combine states, you are reducing reliability. It would appear, in efforts to reduce overhead in TCP transport, you have added dangerous overhead at the SCSI emulation layer. This is taking advantage of TCP in hardware? Allow a session per LUN, (remove the LUN field) then there is no danger of inducing an error across multiple devices when sorting millions of tags for a LUN value. Care about reliability. Doug > Doug, > > You can use the standard at the end user device, if you so wish. > If you choose a "transprent" controller - i.e. one not holding > any state or > ordering - and > view every LU as and independent unit - you ignore the numbering (that you > are free to do anyway for the commands) and ship the commands to > the device > for execution. > > Initiator tags are unique per initiator and it should not be a problem to > keep the unique > per session. > > Target task tags are a different issue. We decided to remove the LUN field > in both the incoming RTT and the the outgoing data to avoid the need to > test for inconsistencies. > But as this is the only thing that implies unique target task > tags we might > as well "reinstate" the LUN (the field is still unused) in the DATA packet > (in the RTT it is implied by the initiator > task tag). > > Relaxing the requirement of uniqueness for initiator tag is possible too > (you can have the unique per LU) but that might complicate > unecessarily the > initiator. > > If this (minor) change does not anoy target controller implementors and > nobody sees something else wrong with it we might introduce it in the next > version. > > As for your FC claims - I still can't follow your argument. Are > you talking > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > Julo > > > "Douglas Otis" on 14/08/2000 20:24:29 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: Towards Consensus on TCP Connections > > > > > Julo, > > Keep the interface suitable for the end device. A controller in > the middle > does not justify a unique interface for that scenario. In trying to > optimize "Bandwidth utilization", you are binding thousands of logical > units > together into the same session. This is an added burden within the > standard, if used at the device. In doing so, session tags must be kept > unique across all logical units. Reset all drives to recover from even a > single tag error? If you wish to cache information at the gateway, it > should still use the interface suitable for the end device. At least this > isolates error handling. > > By creating a simple FC tunnel, the controller remains bolted into the > back-plane where it is known to work effectively. Error handling is > understood with a stateless translation and you have not created > a standard > that will set back efforts at getting a solution without myopic > compromises. > There WILL be errors not seen by this "In the Middle Controller" > translating > Fibre-Channel into iSCSI as FC does not have the same transport. States > within this ITMC adds to complexity and error migration when you > intertwine > logical units. ITMC is a poor starting point and only makes for a bad > interface as seen by this specification. > > Doug > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Saturday, August 12, 2000 12:24 AM > > To: ips@ece.cmu.edu > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Well - I understand the argument about a connection/LU and I did even > > implement it (as many others - see Paul's reply). You can aggregate at > the > > TCP level but you have to aggregate somewhere > > to use effectively the bandwidth. Eliminating the CU is not a concern of > > this group and not a suggestion to be accepted lightly by the community > > (who will do caching, storage virtualization and management etc.) and I > am > > confident most of the participants on this list do not want to discuss > > this subject. FC over IP as replacement for iSCSI is a poor > suggestion - > > FC is a networking protocol and IP too. FCP over IP is a gateway > solution. > > > > Julo > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Julo, > > > > An architecture that scales does not need to provide the entire > bandwidth > > from a single device. If these devices are each handling 25 Mbytes per > > second, then 2 of them exceed Fibre-Channel. Next week you could use 4, > > and > > the week after that 8. The advantage of using a network without any > > practical architectural restriction with intelligence at the client. On > > the > > other hand, you want to design a single mid-point gateway to handle the > > entire bandwidth. To what end? You argue it can not process a > > TCP session > > per device. You will not be tracking mid-stream errors as this is more > > costly. You add to the burden of tracking the state of the individual > > device with this mid-stream state machine requiring additional sorting > due > > to this merged protocol without taking advantage of TCP to aid this > > process. > > For your type of solution, a simple hardware based tunnel would be > better. > > Do not include handling of the encapsulated protocol and at least it > > becomes > > understandable and more likely to manage the task. In other > > words, make it > > Fibre-Channel over IP and you have a chance with your architecture. > > > > The point to my statements was to indicate the device is able to handle > an > > IP interface today using Fast Ethernet as the rate from the drive is > > relatively low. Not to warm the cockles of marketing pointing to > latency, > > but at least they sell more drives overcoming this problem using scale. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Friday, August 11, 2000 2:17 PM > > > To: ips@ece.cmu.edu > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Doug, > > > > > > I am not sure that I agree with your architecture statements > but I like > > to > > > play with numbers (as most of the fellows engineers on this > > list probably > > > do). What would be in your opinion > > > reasonable requirements for command and data transfer rates for the > next > > > 3-7 years? > > > > > > I would like to decouple that discussion from architecture - data > > > rates can > > > scale even in a shared > > > architecture as mainframe channels have shown for years. > > > > > > I would rather like to understand if we can meet the data rates with > > > reasonable latency. > > > > > > Julo > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > Please respond to "Douglas Otis" > > > > > > To: "Stephen Byan" , ips@ece.cmu.edu > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data on the outside > > > cylinders. Improvement of the mechanics comes at a high price > > > with respect > > > to power and cost. The cost/volume trend takes us to a single > > disk which > > > increases access time as read channel data rate increases. > By offering > > > scaled throughput using more drives where each drive's interface > > bandwidth > > > is restricted with respect to read channel data rates > provides a system > > > with > > > uniform and superior performance. The advantage of such an > approach is > > > found with respect to smaller random traffic. With more devices, > > > redundancy > > > is easily achieved and parallel access offers a means of performance > > > improvement by spreading activity over more devices. The > > switch provides > > > bandwidth aggregation and is not found in the individual device. > > > > > > An 8ms access + latency figure in the high cost drives restricts > > > the number > > > of 'independent' operations that average 64k byte to 100 per > > second or 52 > > > Mbit per second. Such an architecture of 'restricted' drives > > would scale > > > whereas the solicitated burst approach does not. An > > independent nexus at > > > the LUN is the only design that offers required scaling and > > configuration > > > flexibility. Keeping up with the read channel is a wasted effort. In > > > time, > > > 1 Gbit Ethernet will be the practical solution about the time drives > are > > 1 > > > inche in size. Several Fast Ethernet disks combined at a 1 > > Gbit Ethernet > > > client makes sense in cost, performance, capacity, reliability, and > > > scalability at this point in time. The protocol overhead should be > > > addressed. There are substantial improvements to be made to > allow this > > > innovation using standard adapters. > > > > > > The power cost to use copper 1 Gbit is high. Firewire does not > > scale and > > > has a limited reach. Firewire also places scatter/gather on the drive > > > together with direct access. Doing such over a WAN will impose > > > significant > > > changes. Serial ATA is nothing more than IDE through a SERDES. > > The read > > > channel data rate is like a drug, just say no. It is hard not to buy > > > enough > > > dram to allow a proper buffer these days. Serial ATA removes all > > buffers. > > > Intel is just usurping any remaining electronics at the cost of > > > sensitivity > > > to a near by cell phone. Fewer drives with less electronics. What a > > good > > > idea? > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > Stephen Byan > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > To: 'ips@ece.cmu.edu' > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > to be 200 > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > midrange storage > > > > market without link aggregation for a single LUN. These markets can > > > afford > > > > 10 Gb/s links. > > > > > > > > As a disk device level interface, iSCSI will not succeed unless > > > > it offers at > > > > least 2 Gb/s by around 2002, at very low cost for the link. > Note that > > > even > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA > > > speed > > > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > > > http://serialata.org/F9pp.pdf. > > > > > > > > One can argue the technical merits, but from a marketing > > > > viewpoint, the disk > > > > industry (both suppliers and customers) has long held the view that > > > > interface speeds need to match the media data rate. iSCSI can try > > > > to make an > > > > argument that slower speeds are technically adequate, but this > > > > will increase > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > implemented > > > > > in a disk, there'll be wild dancing in the streets and > > you'll all (or > > > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > > > > > The easier you can make it for the implementors, the more > likely it > > > > > will succeed. > > > > > > > > Disk drive companies have implemented much more complex > > interfaces than > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > > > > connections don't look very hard to implement. They just look like a > > > wart. > > > > But I think a necessary one. > > > > > > > > Regards, > > > > -Steve > > > > > > > > Steve Byan > > > > > > > > Design Engineer > > > > MS 1-3/E23 > > > > 333 South Street > > > > Shrewsbury, MA 01545 > > > > (508)770-3414 > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Tue Aug 15 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15069; 15 Aug 2000 15:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA22646 for ips-outgoing; Tue, 15 Aug 2000 13:50:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA22641 for ; Tue, 15 Aug 2000 13:50:26 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0FZC00A39GZOUN@mta5.snfc21.pbi.net> for IPS@ECE.CMU.edu; Tue, 15 Aug 2000 10:38:12 -0700 (PDT) Date: Tue, 15 Aug 2000 10:38:46 -0700 From: Peter Johansson Subject: RE: Towards Consensus on TCP Connections In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000815102924.00ba2890@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed References: Sender: owner-ips@ece.cmu.edu Precedence: bulk At 09:24 AM 8/15/00, Douglas Otis wrote: >For details on proposed Fibre-Channel over IP ... Doug, just sort me out on this. Are you suggesting SCSI over (FCP-2 over (FC over (IP over FC))) to reach the storage devices? Incidentally, by "SCSI" I mean the SCSI command set. Or did I get the nesting parentheses wrong? If I do misunderstand, would you please set me straight. I though iSCSI was focused on SCSI over (SAM-compliant transport over (IP over [insert your favorite medium here])). I count one less nesting level---which usually goes hand-in-hand with less encapsulation overhead. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Wed Aug 16 00:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16498; 15 Aug 2000 20:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA06098 for ips-outgoing; Tue, 15 Aug 2000 19:38:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA06091 for ; Tue, 15 Aug 2000 19:38:09 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id QAA13767 for ; Tue, 15 Aug 2000 16:38:07 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.187.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id QAA15124 for ; Tue, 15 Aug 2000 16:34:20 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id QAA14953; Tue, 15 Aug 2000 16:34:21 -0700 (PDT) Message-Id: <200008152334.QAA14953@ha10nwk.EBay.Sun.COM> Date: Tue, 15 Aug 2000 16:34:20 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Towards Consensus on TCP Connections To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: sRhEnGTrlNFpGBxajil0RQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > iSCSI intertwines LUN states to save TCP overhead. In doing so, one tag > error sorting millions of tags within the mid-stream point of aggregation > (the iSCSI design) necessitates a catastrophic system reset. With FC over > IP, the end point handles errors in accordance to the FC protocol such as > FCP-2 with simple frame aligned objects. I count an additional complex > state machine in the middle of the communication path for iSCSI in addition > to new adapters with complex (INSANE) state machines at the client. A > switch or a router would be a far safer choice for aggregation. I think you greatly over estimate the complexity, risk, and recovery issues. What is being described is a specific case of common multiplex/demultiplex scheme that is well understood and widely deployed. The complexity of doing it at the iSCSI session layer is no greater than doing it at the TCP transport layer. We have years of experience building reliable TCP port mux/demux software. Furthermore the same technique is inherent in the Layer-2/3 switches that today can process 250M+ packets/sec. I don't see why doing it for IPS would be any more difficult. I am no FC/FCP expert but I am willing to bet such a scheme exists in existing fabrics. Although I have argued in the past for a session per LUN, complexity of doing the mux/demux at either the session (iSCSI) or transport (TCP) layer is not a factor in either design. -David From owner-ips@ECE.cmu.edu Wed Aug 16 00:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16496; 15 Aug 2000 20:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA05220 for ips-outgoing; Tue, 15 Aug 2000 19:05:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA05215 for ; Tue, 15 Aug 2000 19:05:37 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA03595; Tue, 15 Aug 2000 16:00:41 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA14498; Tue, 15 Aug 2000 15:58:04 -0700 From: "Douglas Otis" To: "Peter Johansson" , "IP Storage" Subject: RE: Towards Consensus on TCP Connections Date: Tue, 15 Aug 2000 16:07:41 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-reply-to: <4.3.2.7.2.20000815102924.00ba2890@PacBell.net> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, iSCSI intertwines LUN states to save TCP overhead. In doing so, one tag error sorting millions of tags within the mid-stream point of aggregation (the iSCSI design) necessitates a catastrophic system reset. With FC over IP, the end point handles errors in accordance to the FC protocol such as FCP-2 with simple frame aligned objects. I count an additional complex state machine in the middle of the communication path for iSCSI in addition to new adapters with complex (INSANE) state machines at the client. A switch or a router would be a far safer choice for aggregation. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Peter Johansson > Sent: Tuesday, August 15, 2000 10:39 AM > To: IP Storage > Subject: RE: Towards Consensus on TCP Connections > > > At 09:24 AM 8/15/00, Douglas Otis wrote: > > >For details on proposed Fibre-Channel over IP ... > > Doug, just sort me out on this. Are you suggesting SCSI over (FCP-2 over > (FC over (IP over FC))) to reach the storage devices? Incidentally, by > "SCSI" I mean the SCSI command set. Or did I get the nesting > parentheses wrong? > > If I do misunderstand, would you please set me straight. I though > iSCSI was > focused on SCSI over (SAM-compliant transport over (IP over [insert your > favorite medium here])). I count one less nesting level---which usually > goes hand-in-hand with less encapsulation overhead. > > > Regards, > > Peter Johansson > > Congruent Software, Inc. > 98 Colorado Avenue > Berkeley, CA 94707 > > (510) 527-3926 > (510) 527-3856 FAX > > PJohansson@ACM.org > From owner-ips@ECE.cmu.edu Wed Aug 16 06:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18089; 16 Aug 2000 2:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA15228 for ips-outgoing; Wed, 16 Aug 2000 01:08:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA15221 for ; Wed, 16 Aug 2000 01:08:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA16420 for ; Wed, 16 Aug 2000 07:07:49 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id HAA15306 for ; Wed, 16 Aug 2000 07:07:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693D.001C2D14 ; Wed, 16 Aug 2000 07:07:45 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 16 Aug 2000 08:05:49 +0300 Subject: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, Please cool down and keep the discussion technical. I personally do not enjoy bringing press quotes to a technical discussion. And so much more when they are inaccurate. Encapsulating FC ovr IP is not SCSI. Your basic assumption is that FCP will be encapsulated on IP. FCP is tightly tied to FC (as TCP is to IP) and has already a complex (and not yet closed) recovery mechanism based on the assumption it runs over FC. Moreover - like SCSI - FCP has the notion of a target distinct from the notion of a LUN. It could be an acceptable solution to extend the range of a FC island but even for this it has to go some way in solving the congestion and security issues. IMHO even for this application a FCP-iSCSI gateway could provide a better solution as the mapping is straightforward. Julo "Douglas Otis" on 15/08/2000 19:24:34 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Towards Consensus on TCP Connections Julo, For details on proposed Fibre-Channel over IP which does not impose additional catastrophic error states as does iSCSI, please see: http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt "While the FC over IP specification is independent of the link level transport protocol, it assumes a high bandwidth, high reliability, low loss link level transport such as Gigabit Ethernet, SONET, ATM, or DWDM. This specification treats all classes of FC frames the same -- as datagrams." ipfc "Fibre Channel over IP's main advantage is it needs no modifications to the storage subsystem or the server operating system. SCSI over TCP/IP, on the other hand, requires filter drivers so the server operating system can do IP-network-to-SAN emulation (for which I humbly propose the acronym INSANE)." Wayne Rickard Network World, 06/12/00 wayne@gadzoox.com To see who is doing Fibre-Channel encapsulation see: http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_061300b.htm At keeping the same architecture as the present Fibre-Channel configuration, you have not improved reliability. By making compromises that combine states, you are reducing reliability. It would appear, in efforts to reduce overhead in TCP transport, you have added dangerous overhead at the SCSI emulation layer. This is taking advantage of TCP in hardware? Allow a session per LUN, (remove the LUN field) then there is no danger of inducing an error across multiple devices when sorting millions of tags for a LUN value. Care about reliability. Doug > Doug, > > You can use the standard at the end user device, if you so wish. > If you choose a "transprent" controller - i.e. one not holding > any state or > ordering - and > view every LU as and independent unit - you ignore the numbering (that you > are free to do anyway for the commands) and ship the commands to > the device > for execution. > > Initiator tags are unique per initiator and it should not be a problem to > keep the unique > per session. > > Target task tags are a different issue. We decided to remove the LUN field > in both the incoming RTT and the the outgoing data to avoid the need to > test for inconsistencies. > But as this is the only thing that implies unique target task > tags we might > as well "reinstate" the LUN (the field is still unused) in the DATA packet > (in the RTT it is implied by the initiator > task tag). > > Relaxing the requirement of uniqueness for initiator tag is possible too > (you can have the unique per LU) but that might complicate > unecessarily the > initiator. > > If this (minor) change does not anoy target controller implementors and > nobody sees something else wrong with it we might introduce it in the next > version. > > As for your FC claims - I still can't follow your argument. Are > you talking > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > Julo > > > "Douglas Otis" on 14/08/2000 20:24:29 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: Towards Consensus on TCP Connections > > > > > Julo, > > Keep the interface suitable for the end device. A controller in > the middle > does not justify a unique interface for that scenario. In trying to > optimize "Bandwidth utilization", you are binding thousands of logical > units > together into the same session. This is an added burden within the > standard, if used at the device. In doing so, session tags must be kept > unique across all logical units. Reset all drives to recover from even a > single tag error? If you wish to cache information at the gateway, it > should still use the interface suitable for the end device. At least this > isolates error handling. > > By creating a simple FC tunnel, the controller remains bolted into the > back-plane where it is known to work effectively. Error handling is > understood with a stateless translation and you have not created > a standard > that will set back efforts at getting a solution without myopic > compromises. > There WILL be errors not seen by this "In the Middle Controller" > translating > Fibre-Channel into iSCSI as FC does not have the same transport. States > within this ITMC adds to complexity and error migration when you > intertwine > logical units. ITMC is a poor starting point and only makes for a bad > interface as seen by this specification. > > Doug > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Saturday, August 12, 2000 12:24 AM > > To: ips@ece.cmu.edu > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Well - I understand the argument about a connection/LU and I did even > > implement it (as many others - see Paul's reply). You can aggregate at > the > > TCP level but you have to aggregate somewhere > > to use effectively the bandwidth. Eliminating the CU is not a concern of > > this group and not a suggestion to be accepted lightly by the community > > (who will do caching, storage virtualization and management etc.) and I > am > > confident most of the participants on this list do not want to discuss > > this subject. FC over IP as replacement for iSCSI is a poor > suggestion - > > FC is a networking protocol and IP too. FCP over IP is a gateway > solution. > > > > Julo > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Julo, > > > > An architecture that scales does not need to provide the entire > bandwidth > > from a single device. If these devices are each handling 25 Mbytes per > > second, then 2 of them exceed Fibre-Channel. Next week you could use 4, > > and > > the week after that 8. The advantage of using a network without any > > practical architectural restriction with intelligence at the client. On > > the > > other hand, you want to design a single mid-point gateway to handle the > > entire bandwidth. To what end? You argue it can not process a > > TCP session > > per device. You will not be tracking mid-stream errors as this is more > > costly. You add to the burden of tracking the state of the individual > > device with this mid-stream state machine requiring additional sorting > due > > to this merged protocol without taking advantage of TCP to aid this > > process. > > For your type of solution, a simple hardware based tunnel would be > better. > > Do not include handling of the encapsulated protocol and at least it > > becomes > > understandable and more likely to manage the task. In other > > words, make it > > Fibre-Channel over IP and you have a chance with your architecture. > > > > The point to my statements was to indicate the device is able to handle > an > > IP interface today using Fast Ethernet as the rate from the drive is > > relatively low. Not to warm the cockles of marketing pointing to > latency, > > but at least they sell more drives overcoming this problem using scale. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Friday, August 11, 2000 2:17 PM > > > To: ips@ece.cmu.edu > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Doug, > > > > > > I am not sure that I agree with your architecture statements > but I like > > to > > > play with numbers (as most of the fellows engineers on this > > list probably > > > do). What would be in your opinion > > > reasonable requirements for command and data transfer rates for the > next > > > 3-7 years? > > > > > > I would like to decouple that discussion from architecture - data > > > rates can > > > scale even in a shared > > > architecture as mainframe channels have shown for years. > > > > > > I would rather like to understand if we can meet the data rates with > > > reasonable latency. > > > > > > Julo > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > Please respond to "Douglas Otis" > > > > > > To: "Stephen Byan" , ips@ece.cmu.edu > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data on the outside > > > cylinders. Improvement of the mechanics comes at a high price > > > with respect > > > to power and cost. The cost/volume trend takes us to a single > > disk which > > > increases access time as read channel data rate increases. > By offering > > > scaled throughput using more drives where each drive's interface > > bandwidth > > > is restricted with respect to read channel data rates > provides a system > > > with > > > uniform and superior performance. The advantage of such an > approach is > > > found with respect to smaller random traffic. With more devices, > > > redundancy > > > is easily achieved and parallel access offers a means of performance > > > improvement by spreading activity over more devices. The > > switch provides > > > bandwidth aggregation and is not found in the individual device. > > > > > > An 8ms access + latency figure in the high cost drives restricts > > > the number > > > of 'independent' operations that average 64k byte to 100 per > > second or 52 > > > Mbit per second. Such an architecture of 'restricted' drives > > would scale > > > whereas the solicitated burst approach does not. An > > independent nexus at > > > the LUN is the only design that offers required scaling and > > configuration > > > flexibility. Keeping up with the read channel is a wasted effort. In > > > time, > > > 1 Gbit Ethernet will be the practical solution about the time drives > are > > 1 > > > inche in size. Several Fast Ethernet disks combined at a 1 > > Gbit Ethernet > > > client makes sense in cost, performance, capacity, reliability, and > > > scalability at this point in time. The protocol overhead should be > > > addressed. There are substantial improvements to be made to > allow this > > > innovation using standard adapters. > > > > > > The power cost to use copper 1 Gbit is high. Firewire does not > > scale and > > > has a limited reach. Firewire also places scatter/gather on the drive > > > together with direct access. Doing such over a WAN will impose > > > significant > > > changes. Serial ATA is nothing more than IDE through a SERDES. > > The read > > > channel data rate is like a drug, just say no. It is hard not to buy > > > enough > > > dram to allow a proper buffer these days. Serial ATA removes all > > buffers. > > > Intel is just usurping any remaining electronics at the cost of > > > sensitivity > > > to a near by cell phone. Fewer drives with less electronics. What a > > good > > > idea? > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > Stephen Byan > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > To: 'ips@ece.cmu.edu' > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > to be 200 > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > midrange storage > > > > market without link aggregation for a single LUN. These markets can > > > afford > > > > 10 Gb/s links. > > > > > > > > As a disk device level interface, iSCSI will not succeed unless > > > > it offers at > > > > least 2 Gb/s by around 2002, at very low cost for the link. > Note that > > > even > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial ATA > > > speed > > > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > > > http://serialata.org/F9pp.pdf. > > > > > > > > One can argue the technical merits, but from a marketing > > > > viewpoint, the disk > > > > industry (both suppliers and customers) has long held the view that > > > > interface speeds need to match the media data rate. iSCSI can try > > > > to make an > > > > argument that slower speeds are technically adequate, but this > > > > will increase > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > implemented > > > > > in a disk, there'll be wild dancing in the streets and > > you'll all (or > > > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > > > > > The easier you can make it for the implementors, the more > likely it > > > > > will succeed. > > > > > > > > Disk drive companies have implemented much more complex > > interfaces than > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple TCP > > > > connections don't look very hard to implement. They just look like a > > > wart. > > > > But I think a necessary one. > > > > > > > > Regards, > > > > -Steve > > > > > > > > Steve Byan > > > > > > > > Design Engineer > > > > MS 1-3/E23 > > > > 333 South Street > > > > Shrewsbury, MA 01545 > > > > (508)770-3414 > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Wed Aug 16 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20847; 16 Aug 2000 12:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA09477 for ips-outgoing; Wed, 16 Aug 2000 11:02:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA09468 for ; Wed, 16 Aug 2000 11:02:06 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id HAA03671; Wed, 16 Aug 2000 07:57:02 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id HAA17069; Wed, 16 Aug 2000 07:54:28 -0700 From: "Douglas Otis" To: , Subject: RE: Towards Consensus on TCP Connections Date: Wed, 16 Aug 2000 08:04:10 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-reply-to: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Tuesday, August 15, 2000 10:06 PM > To: ips@ece.cmu.edu > Subject: RE: Towards Consensus on TCP Connections > > Doug, > > Please cool down and keep the discussion technical. I > personally do not enjoy bringing press quotes to a technical > discussion. And so much more when they are inaccurate. You indicated confusion as to the comparison being made. I tried to enlighten you as to the alternative. I did it with humor and with technical details. I see nothing inaccurate nor have you indicated errors. > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > that FCP will be encapsulated on IP. FCP is tightly tied to FC > (as TCP is to IP) and has already a complex (and not yet closed) > recovery mechanism based on the assumption it runs over FC. An IP datagram is nothing more than a vehicle for FC encapsulation and indeed includes only the defragmentation process. There is no need to manage the defrag process or to be concerned about possible failure at the IP level. It is transparent and self healing as FCP-2 on FC handles these errors and would be no less reliable in doing so. There are few uses for FC beyond SCSI so your point escapes me especially when you are advocating a protocol wholly unsupported. Are you suggesting FC over IP will not work? > Moreover - like SCSI - FCP has the notion of a target distinct > from the notion of a LUN. It could be an acceptable solution to > extend the range of a FC island but even for this it has to go > some way in solving the congestion and security issues. There are already provisions for security for FC over IP. In many applications, bandwidth is controlled by a dedicated channel which solves both issues. There are many means for flow control where TCP is but one. Reliability should be the differentiating factor. > IMHO even for this application a FCP-iSCSI gateway could provide a better > solution as the mapping is straightforward. > > Julo This provides few techincal details as to how you base this opinion. If I had the choice of purchasing a blade for a switch or new system adapters with high overhead and new SAN controllers with limited connectivity where I could also expect massive faults, I would not see that as straightforward. But then again, maybe I would. Doug > "Douglas Otis" on 15/08/2000 19:24:34 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: Towards Consensus on TCP Connections > > Julo, > > For details on proposed Fibre-Channel over IP which does not impose > additional catastrophic error states as does iSCSI, please see: > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > "While the FC over IP specification is independent of the link level > transport protocol, it assumes a high bandwidth, high > reliability, low loss > link level transport such as Gigabit Ethernet, SONET, ATM, or DWDM. This > specification treats all classes of FC frames the same -- as datagrams." > ipfc > > "Fibre Channel over IP's main advantage is it needs no > modifications to the > storage subsystem or the server operating system. SCSI over TCP/IP, on the > other hand, requires filter drivers so the server operating system can do > IP-network-to-SAN emulation (for which I humbly propose the acronym > INSANE)." Wayne Rickard > Network World, 06/12/00 wayne@gadzoox.com > > To see who is doing Fibre-Channel encapsulation see: > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_061300b.htm > > At keeping the same architecture as the present Fibre-Channel > configuration, > you have not improved reliability. By making compromises that combine > states, you are reducing reliability. It would appear, in efforts to > reduce > overhead in TCP transport, you have added dangerous overhead at the SCSI > emulation layer. This is taking advantage of TCP in hardware? Allow a > session per LUN, (remove the LUN field) then there is no danger > of inducing > an error across multiple devices when sorting millions of tags for a LUN > value. Care about reliability. > > Doug > > > Doug, > > > > You can use the standard at the end user device, if you so wish. > > If you choose a "transprent" controller - i.e. one not holding > > any state or > > ordering - and > > view every LU as and independent unit - you ignore the numbering (that > you > > are free to do anyway for the commands) and ship the commands to > > the device > > for execution. > > > > Initiator tags are unique per initiator and it should not be a > problem to > > keep the unique > > per session. > > > > Target task tags are a different issue. We decided to remove the LUN > field > > in both the incoming RTT and the the outgoing data to avoid the need to > > test for inconsistencies. > > But as this is the only thing that implies unique target task > > tags we might > > as well "reinstate" the LUN (the field is still unused) in the DATA > packet > > (in the RTT it is implied by the initiator > > task tag). > > > > Relaxing the requirement of uniqueness for initiator tag is possible too > > (you can have the unique per LU) but that might complicate > > unecessarily the > > initiator. > > > > If this (minor) change does not anoy target controller implementors and > > nobody sees something else wrong with it we might introduce it in the > next > > version. > > > > As for your FC claims - I still can't follow your argument. Are > > you talking > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > Julo > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > Julo, > > > > Keep the interface suitable for the end device. A controller in > > the middle > > does not justify a unique interface for that scenario. In trying to > > optimize "Bandwidth utilization", you are binding thousands of logical > > units > > together into the same session. This is an added burden within the > > standard, if used at the device. In doing so, session tags must be kept > > unique across all logical units. Reset all drives to recover > from even a > > single tag error? If you wish to cache information at the gateway, it > > should still use the interface suitable for the end device. At least > this > > isolates error handling. > > > > By creating a simple FC tunnel, the controller remains bolted into the > > back-plane where it is known to work effectively. Error handling is > > understood with a stateless translation and you have not created > > a standard > > that will set back efforts at getting a solution without myopic > > compromises. > > There WILL be errors not seen by this "In the Middle Controller" > > translating > > Fibre-Channel into iSCSI as FC does not have the same transport. States > > within this ITMC adds to complexity and error migration when you > > intertwine > > logical units. ITMC is a poor starting point and only makes for a bad > > interface as seen by this specification. > > > > Doug > > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Saturday, August 12, 2000 12:24 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU and I did even > > > implement it (as many others - see Paul's reply). You can aggregate at > > the > > > TCP level but you have to aggregate somewhere > > > to use effectively the bandwidth. Eliminating the CU is not a concern > of > > > this group and not a suggestion to be accepted lightly by the > community > > > (who will do caching, storage virtualization and management > etc.) and I > > am > > > confident most of the participants on this list do not want to discuss > > > this subject. FC over IP as replacement for iSCSI is a poor > > suggestion - > > > FC is a networking protocol and IP too. FCP over IP is a gateway > > solution. > > > > > > Julo > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Julo, > > > > > > An architecture that scales does not need to provide the entire > > bandwidth > > > from a single device. If these devices are each handling 25 > Mbytes per > > > second, then 2 of them exceed Fibre-Channel. Next week you could use > 4, > > > and > > > the week after that 8. The advantage of using a network without any > > > practical architectural restriction with intelligence at the client. > On > > > the > > > other hand, you want to design a single mid-point gateway to > handle the > > > entire bandwidth. To what end? You argue it can not process a > > > TCP session > > > per device. You will not be tracking mid-stream errors as > this is more > > > costly. You add to the burden of tracking the state of the individual > > > device with this mid-stream state machine requiring additional sorting > > due > > > to this merged protocol without taking advantage of TCP to aid this > > > process. > > > For your type of solution, a simple hardware based tunnel would be > > better. > > > Do not include handling of the encapsulated protocol and at least it > > > becomes > > > understandable and more likely to manage the task. In other > > > words, make it > > > Fibre-Channel over IP and you have a chance with your architecture. > > > > > > The point to my statements was to indicate the device is able > to handle > > an > > > IP interface today using Fast Ethernet as the rate from the drive is > > > relatively low. Not to warm the cockles of marketing pointing to > > latency, > > > but at least they sell more drives overcoming this problem > using scale. > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > julian_satran@il.ibm.com > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > I am not sure that I agree with your architecture statements > > but I like > > > to > > > > play with numbers (as most of the fellows engineers on this > > > list probably > > > > do). What would be in your opinion > > > > reasonable requirements for command and data transfer rates for the > > next > > > > 3-7 years? > > > > > > > > I would like to decouple that discussion from architecture - data > > > > rates can > > > > scale even in a shared > > > > architecture as mainframe channels have shown for years. > > > > > > > > I would rather like to understand if we can meet the data rates with > > > > reasonable latency. > > > > > > > > Julo > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: "Stephen Byan" , ips@ece.cmu.edu > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data on the outside > > > > cylinders. Improvement of the mechanics comes at a high price > > > > with respect > > > > to power and cost. The cost/volume trend takes us to a single > > > disk which > > > > increases access time as read channel data rate increases. > > By offering > > > > scaled throughput using more drives where each drive's interface > > > bandwidth > > > > is restricted with respect to read channel data rates > > provides a system > > > > with > > > > uniform and superior performance. The advantage of such an > > approach is > > > > found with respect to smaller random traffic. With more devices, > > > > redundancy > > > > is easily achieved and parallel access offers a means of performance > > > > improvement by spreading activity over more devices. The > > > switch provides > > > > bandwidth aggregation and is not found in the individual device. > > > > > > > > An 8ms access + latency figure in the high cost drives restricts > > > > the number > > > > of 'independent' operations that average 64k byte to 100 per > > > second or 52 > > > > Mbit per second. Such an architecture of 'restricted' drives > > > would scale > > > > whereas the solicitated burst approach does not. An > > > independent nexus at > > > > the LUN is the only design that offers required scaling and > > > configuration > > > > flexibility. Keeping up with the read channel is a wasted effort. > In > > > > time, > > > > 1 Gbit Ethernet will be the practical solution about the time drives > > are > > > 1 > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > Gbit Ethernet > > > > client makes sense in cost, performance, capacity, reliability, and > > > > scalability at this point in time. The protocol overhead should be > > > > addressed. There are substantial improvements to be made to > > allow this > > > > innovation using standard adapters. > > > > > > > > The power cost to use copper 1 Gbit is high. Firewire does not > > > scale and > > > > has a limited reach. Firewire also places scatter/gather on the > drive > > > > together with direct access. Doing such over a WAN will impose > > > > significant > > > > changes. Serial ATA is nothing more than IDE through a SERDES. > > > The read > > > > channel data rate is like a drug, just say no. It is hard > not to buy > > > > enough > > > > dram to allow a proper buffer these days. Serial ATA removes all > > > buffers. > > > > Intel is just usurping any remaining electronics at the cost of > > > > sensitivity > > > > to a near by cell phone. Fewer drives with less > electronics. What a > > > good > > > > idea? > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > Stephen Byan > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > To: 'ips@ece.cmu.edu' > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > > to be 200 > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > midrange storage > > > > > market without link aggregation for a single LUN. These > markets can > > > > afford > > > > > 10 Gb/s links. > > > > > > > > > > As a disk device level interface, iSCSI will not succeed unless > > > > > it offers at > > > > > least 2 Gb/s by around 2002, at very low cost for the link. > > Note that > > > > even > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look at the Serial > ATA > > > > speed > > > > > roadmap on slide 16 of Intel's Serial ATA presentation at WinHEC: > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > viewpoint, the disk > > > > > industry (both suppliers and customers) has long held the > view that > > > > > interface speeds need to match the media data rate. iSCSI can try > > > > > to make an > > > > > argument that slower speeds are technically adequate, but this > > > > > will increase > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > > implemented > > > > > > in a disk, there'll be wild dancing in the streets and > > > you'll all (or > > > > > > maybe your companies will) be rich beyond the dreams of avarice. > > > > > > > > > > > > The easier you can make it for the implementors, the more > > likely it > > > > > > will succeed. > > > > > > > > > > Disk drive companies have implemented much more complex > > > interfaces than > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. And multiple > TCP > > > > > connections don't look very hard to implement. They just look like > a > > > > wart. > > > > > But I think a necessary one. > > > > > > > > > > Regards, > > > > > -Steve > > > > > > > > > > Steve Byan > > > > > > > > > > Design Engineer > > > > > MS 1-3/E23 > > > > > 333 South Street > > > > > Shrewsbury, MA 01545 > > > > > (508)770-3414 > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Wed Aug 16 20:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22100; 16 Aug 2000 16:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA18744 for ips-outgoing; Wed, 16 Aug 2000 14:42:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from dirty.research.bell-labs.com (dirty.research.bell-labs.com [204.178.16.6]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id OAA18740 for ; Wed, 16 Aug 2000 14:42:04 -0400 (EDT) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by dirty; Wed Aug 16 14:41:30 EDT 2000 Received: from aura.research.bell-labs.com ([135.104.46.10]) by grubby; Wed Aug 16 14:41:28 EDT 2000 Received: (from jkf@localhost) by aura.research.bell-labs.com (8.9.1/8.9.1) id OAA15622 for ips@ece.cmu.edu; Wed, 16 Aug 2000 14:41:28 -0400 (EDT) Date: Wed, 16 Aug 2000 14:41:28 -0400 (EDT) From: Jeff Fellin Message-Id: <200008161841.OAA15622@aura.research.bell-labs.com> To: ips@ece.cmu.edu Subject: Re: Towards Consensus on TCP connections Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, After reading all the discussion and debate on this issue, I decided to research if the SCSI standards (T10/X3) have done with developing other transport protocols, and how those protocols dealt with connectivity issues between the endpoints. For the most part the other transport protocols SCSI over Scheduled Transport (SST), which is a working draft and the SCSI-3 Generic Packetized Prococol (SCSI-GPP), which is a proposed technical report have a concept of a Logical or Virtual connection. This Logical or Virtual connection has at least one connection through the network, but may have more up to endpoint configuration parameters. So, it appears the other groups have not come to a consensus as to the approach. Most of the drafts are available on the T10's web site: www.t10.org Except for the definition of the Scheduled transfer protocol, which is on T11's web site: www.t11.org The most interesting thing I came up with is the SCSI-GPP definition is written to allow connections over IP networks. Annexes E and F describe the mapping and processing capabilities. If this is a work in progress should we take advantage of there previous start and work with T10 to have a better definition of a protocol for storage over IP? What I understand from a quick read of the SST and SCSI-GPP protocol definitions is they have dealt with the issues of security, error recovery, connection management, transfer of SCSI bus control signals (bus reset, abort tasks. If we don't use the SCSI-GPP than I propose changing the one LUN per TCP connection group to be a SCSI target id that multiplexes all LUN's of the target id onto the connection group. I believe this will decrease the amount of connections and provide data channels for large transfers. Sincerely, Jeff Fellin MH 2A-352 (908) 582-7673 fellin@lucent.com From owner-ips@ECE.cmu.edu Wed Aug 16 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22744; 16 Aug 2000 18:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA24596 for ips-outgoing; Wed, 16 Aug 2000 17:11:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA24589 for ; Wed, 16 Aug 2000 17:10:55 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e7GLAsw12473 for ; Wed, 16 Aug 2000 16:10:54 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA28399; Wed, 16 Aug 00 16:08:45 -0500 Message-Id: <10008162108.AA28399@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Towards Consensus on TCP Connections In-Reply-To: Message from "Douglas Otis" of "Tue, 15 Aug 2000 09:24:34 PDT." Date: Wed, 16 Aug 2000 16:10:14 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > For details on proposed Fibre-Channel over IP which does not impose > additional catastrophic error states as does iSCSI, please see: > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt FCIP does not provide congestion avoidance, so it does not appear to satisfy the requirements defined in the iSCSI Requirements document. Steph From owner-ips@ECE.cmu.edu Wed Aug 16 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22749; 16 Aug 2000 18:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA24783 for ips-outgoing; Wed, 16 Aug 2000 17:15:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA24779 for ; Wed, 16 Aug 2000 17:15:39 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA03711; Wed, 16 Aug 2000 14:10:31 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA17957; Wed, 16 Aug 2000 14:07:55 -0700 From: "Douglas Otis" To: "Jeff Fellin" , Subject: RE: Towards Consensus on TCP connections Date: Wed, 16 Aug 2000 14:17:42 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-reply-to: <200008161841.OAA15622@aura.research.bell-labs.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jeff Fellin > Sent: Wednesday, August 16, 2000 11:41 AM > To: ips@ece.cmu.edu > Subject: Re: Towards Consensus on TCP connections > > > > Folks, > After reading all the discussion and debate on this issue, I decided to > research if the SCSI standards (T10/X3) have done with developing other > transport protocols, and how those protocols dealt with connectivity > issues between the endpoints. For the most part the other transport > protocols SCSI over Scheduled Transport (SST), which is a working draft > and the SCSI-3 Generic Packetized Prococol (SCSI-GPP), which is a proposed > technical report have a concept of a Logical or Virtual connection. This > Logical or Virtual connection has at least one connection through the > network, but may have more up to endpoint configuration parameters. So, > it appears the other groups have not come to a consensus as to > the approach. > > Most of the drafts are available on the T10's web site: > www.t10.org > > Except for the definition of the Scheduled transfer protocol, which is on > T11's web site: > www.t11.org > > The most interesting thing I came up with is the SCSI-GPP definition is > written to allow connections over IP networks. Annexes E and F describe > the mapping and processing capabilities. If this is a work in progress > should we take advantage of there previous start and work with T10 to > have a better definition of a protocol for storage over IP? > > What I understand from a quick read of the SST and SCSI-GPP protocol > definitions is they have dealt with the issues of security, error > recovery, > connection management, transfer of SCSI bus control signals (bus reset, > abort tasks. > > If we don't use the SCSI-GPP than I propose changing the one LUN per > TCP connection group to be a SCSI target id that multiplexes all LUN's > of the target id onto the connection group. I believe this will decrease > the amount of connections and provide data channels for large transfers. see: ftp://ftp.t10.org/t10/drafts/gpp/gpp-r09.pdf Jeff, There is a considerable difference between the SCSI-GPP and iSCSI. With GPP there are only 256 possible LUNs per target (among other notable features). Under iSCSI, there are 18,447,000,000,000,000,000 possible LUNs per target and this 64-bit LUN value is implied in some cases. Doug From owner-ips@ECE.cmu.edu Wed Aug 16 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22838; 16 Aug 2000 18:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA26073 for ips-outgoing; Wed, 16 Aug 2000 17:56:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (truffle.Stanford.EDU [171.64.73.147]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA26069 for ; Wed, 16 Aug 2000 17:56:54 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id OAA01574; Wed, 16 Aug 2000 14:56:25 -0700 To: Subject: FC/IP vs. iSCSI References: cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 16 Aug 2000 14:56:25 -0700 In-Reply-To: "Douglas Otis"'s message of "Wed, 16 Aug 2000 08:04:10 -0700" Message-ID: Lines: 23 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk FC/IP is NOT an alternative to iSCSI for most applications. Unfortunately, making a protocol that works well in complex networks is not as simple as just putting stuff into IP packets. IP packet headers are not magical pixie dust that suddenly make higher-layer protocol issues go away. The FibreChannel stack today has the following deficiencies which do not disappear when tunneling over IP: - FCP has no congestion control - FCP deals poorly with packet loss - Target naming is done with either 24-bit port IDs or 64-bit WWNs, neither of which scale to Internet size - No secure login Of course, you could address all the deficiencies by fixing FCP. But by the time you do that, I maintain you will most likely end up with something in the same order of complexity as iSCSI/TCP/IP. -Costa From owner-ips@ECE.cmu.edu Wed Aug 16 23:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22934; 16 Aug 2000 19:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA27009 for ips-outgoing; Wed, 16 Aug 2000 18:29:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA27004 for ; Wed, 16 Aug 2000 18:29:13 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0FZE001TNORKTP@mta6.snfc21.pbi.net> for IPS@ECE.CMU.edu; Wed, 16 Aug 2000 15:21:21 -0700 (PDT) Date: Wed, 16 Aug 2000 15:21:22 -0700 From: Peter Johansson Subject: Re: Towards Consensus on TCP connections In-reply-to: <200008161841.OAA15622@aura.research.bell-labs.com> X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000816151844.00b889f0@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 02:41 PM 8/16/00, Jeff Fellin wrote: >... SCSI over Scheduled Transport (SST), which is a working draft and the >SCSI-3 Generic Packetized Prococol (SCSI-GPP), which is a proposed >technical report ... Jeff, if you've limited yourself to what's on the T10 web site, you've missed the boat. Approved standards, such as FCP and SBP-2 are not on that site because ANSI sells them! My information about GPP may be stale, but I believe that work on it was shelved. I'll count on any T10 listener on this reflector who knows better to correct me if I'm mistaken. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Wed Aug 16 23:40:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23008; 16 Aug 2000 19:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA27840 for ips-outgoing; Wed, 16 Aug 2000 19:00:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA27834 for ; Wed, 16 Aug 2000 19:00:29 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id PAA03739; Wed, 16 Aug 2000 15:55:21 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA18225; Wed, 16 Aug 2000 15:52:49 -0700 From: "Douglas Otis" To: "Stephen Bailey" , Subject: RE: Towards Consensus on TCP Connections Date: Wed, 16 Aug 2000 16:02:37 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: <10008162108.AA28399@candide.cs.uchicago.edu> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Bailey > Sent: Wednesday, August 16, 2000 2:10 PM > To: ips@ece.cmu.edu > Subject: Re: Towards Consensus on TCP Connections > > For details on proposed Fibre-Channel over IP which does not impose > > additional catastrophic error states as does iSCSI, please see: > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > FCIP does not provide congestion avoidance, so it does not appear to > satisfy the requirements defined in the iSCSI Requirements document. > > Steph Place FC over IP/SCTP and it adds additional security, congestions control and address space via the tunnel addressing. You missed the point in reducing reliability with compromises lead by a desire to place a controller mid-stream. The protocol should be appropriate to the end device. An intervening controller should be transparent, if present. Errors are then contained to the single device. Why compromise reliability? Doug From owner-ips@ECE.cmu.edu Thu Aug 17 00:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23210; 16 Aug 2000 20:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28107 for ips-outgoing; Wed, 16 Aug 2000 19:12:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraab.compuserve.com (ds-img-rel-2.compuserve.com [149.174.206.155]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28100 for ; Wed, 16 Aug 2000 19:12:45 -0400 (EDT) Received: (from mailgate@localhost) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id TAA27916 for ips@ece.cmu.edu; Wed, 16 Aug 2000 19:12:15 -0400 (EDT) Received: from compuserve.com (dal-qbu-zol-vty2.as.wcom.net [216.192.244.2]) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id TAA27910; Wed, 16 Aug 2000 19:12:11 -0400 (EDT) Message-ID: <399B2064.49EAF26A@compuserve.com> Date: Wed, 16 Aug 2000 18:14:44 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector CC: Jeff Fellin Subject: Re: Towards Consensus on TCP connections References: <200008161841.OAA15622@aura.research.bell-labs.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Jeff, You missed a few: > SPI-4 ftp://ftp.t10.org/t10/drafts/spi4/spi4r00.pdf > FCP-2 ftp://ftp.t10.org/t10/drafts/fcp2/fcp2r04.pdf > SVP ftp://ftp.t10.org/t10/drafts/svp/svp-r01.pdf > SBP-2 ftp://ftp.t10.org/t10/drafts/sbp2/sbp2r04.pdf Every one of these is newer and represents more current thinking on packetized SCSI protocols than GPP. I'd be the first to admit that SPI-4 is easy to miss as a key SCSI packetized protocol document, sorting your way through its 328 pages is no small feat. Still, clause 14 (pdf pages 170-183) defines a packetized protocol for the parallel SCSI bus and the packet formats on pdf pages 175-183 are an important part of the current packetized work. It is also very important to note that SPI-4, FCP-2, and SVP are active projects that share a goal of common packet formats to simplify implementation of bridges between the parallel bus, fibre channel, and VI. While I don't wish to dogmatically cling to this goal for IP SCSI, it definitely is worthy of consideration. A particular issue in this regard is maintaining the LUN field in each packet. Thanks. Ralph Weber SAM-2 & SPC-2 Technical Editor Jeff Fellin wrote: > Folks, > After reading all the discussion and debate on this issue, I decided to > research if the SCSI standards (T10/X3) have done with developing other > transport protocols, and how those protocols dealt with connectivity > issues between the endpoints. For the most part the other transport > protocols SCSI over Scheduled Transport (SST), which is a working draft > and the SCSI-3 Generic Packetized Prococol (SCSI-GPP), which is a proposed > technical report have a concept of a Logical or Virtual connection. This > Logical or Virtual connection has at least one connection through the > network, but may have more up to endpoint configuration parameters. So, > it appears the other groups have not come to a consensus as to the approach. > > Most of the drafts are available on the T10's web site: > www.t10.org > > Except for the definition of the Scheduled transfer protocol, which is on > T11's web site: > www.t11.org > > The most interesting thing I came up with is the SCSI-GPP definition is > written to allow connections over IP networks. Annexes E and F describe > the mapping and processing capabilities. If this is a work in progress > should we take advantage of there previous start and work with T10 to > have a better definition of a protocol for storage over IP? > > What I understand from a quick read of the SST and SCSI-GPP protocol > definitions is they have dealt with the issues of security, error recovery, > connection management, transfer of SCSI bus control signals (bus reset, > abort tasks. > > If we don't use the SCSI-GPP than I propose changing the one LUN per > TCP connection group to be a SCSI target id that multiplexes all LUN's > of the target id onto the connection group. I believe this will decrease > the amount of connections and provide data channels for large transfers. > > Sincerely, > Jeff Fellin > MH 2A-352 > (908) 582-7673 > fellin@lucent.com From owner-ips@ECE.cmu.edu Thu Aug 17 00:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23223; 16 Aug 2000 20:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28157 for ips-outgoing; Wed, 16 Aug 2000 19:15:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28152 for ; Wed, 16 Aug 2000 19:15:38 -0400 (EDT) Received: from beppo.feral.com (beppo [192.67.166.79]) by feral.com (8.9.3/8.9.3) with ESMTP id QAA03118; Wed, 16 Aug 2000 16:15:36 -0700 Date: Wed, 16 Aug 2000 16:15:36 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: csapuntz@cisco.com cc: ips@ece.cmu.edu Subject: Re: FC/IP vs. iSCSI In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk I'm certainly inclined to agree. The sole realistic purpose of requirements I've seen to date FC/IP is to utilize an existing FC environment so that management and data go over the same wire. If you're carrying SCSI over TCP/IP now, it's somewhat absured to try and carry FC/IP over the same transport. It's as if you want to tunnel TCP/IP over TCP/IP, which is a cute trick, but a trick for all that. If you have an existing FC optical environment, it could probably run GigEthernet just as well- and it then becomes a question as to whether iSCSI serves the systems and applications as well or better than FC-SCSI or FC/IP over the same physical media. -matt On 16 Aug 2000 csapuntz@cisco.com wrote: > > FC/IP is NOT an alternative to iSCSI for most applications. > > Unfortunately, making a protocol that works well in complex networks > is not as simple as just putting stuff into IP packets. IP packet > headers are not magical pixie dust that suddenly make higher-layer > protocol issues go away. > > The FibreChannel stack today has the following deficiencies which do > not disappear when tunneling over IP: > - FCP has no congestion control > - FCP deals poorly with packet loss > - Target naming is done with either 24-bit port IDs or 64-bit WWNs, > neither of which scale to Internet size > - No secure login > > Of course, you could address all the deficiencies by fixing FCP. But > by the time you do that, I maintain you will most likely end up with > something in the same order of complexity as iSCSI/TCP/IP. > > -Costa > > From owner-ips@ECE.cmu.edu Thu Aug 17 00:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23234; 16 Aug 2000 20:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28501 for ips-outgoing; Wed, 16 Aug 2000 19:29:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28493 for ; Wed, 16 Aug 2000 19:29:39 -0400 (EDT) Received: from beppo.feral.com (beppo [192.67.166.79]) by feral.com (8.9.3/8.9.3) with ESMTP id QAA03161; Wed, 16 Aug 2000 16:29:31 -0700 Date: Wed, 16 Aug 2000 16:29:31 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: Peter Johansson cc: IP Storage Subject: Re: Towards Consensus on TCP connections In-Reply-To: <4.3.2.7.2.20000816151844.00b889f0@PacBell.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter- t10's site doesn't have the official standards, but I strongly doubt that there is substantive work done between the drafts that are there and those final official specs that ANSI/GED sell. On Wed, 16 Aug 2000, Peter Johansson wrote: > At 02:41 PM 8/16/00, Jeff Fellin wrote: > > >... SCSI over Scheduled Transport (SST), which is a working draft and the > >SCSI-3 Generic Packetized Prococol (SCSI-GPP), which is a proposed > >technical report ... > > Jeff, if you've limited yourself to what's on the T10 web site, you've > missed the boat. > > Approved standards, such as FCP and SBP-2 are not on that site because ANSI > sells them! > > My information about GPP may be stale, but I believe that work on it was > shelved. I'll count on any T10 listener on this reflector who knows better > to correct me if I'm mistaken. > > > > > Regards, > > Peter Johansson > > Congruent Software, Inc. > 98 Colorado Avenue > Berkeley, CA 94707 > > (510) 527-3926 > (510) 527-3856 FAX > > PJohansson@ACM.org > From owner-ips@ECE.cmu.edu Thu Aug 17 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23910; 16 Aug 2000 23:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA02329 for ips-outgoing; Wed, 16 Aug 2000 21:41:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA02320 for ; Wed, 16 Aug 2000 21:41:25 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id SAA09673; Wed, 16 Aug 2000 18:41:17 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Wed, 16 Aug 2000 18:41:15 -0700 Message-ID: From: Jim McGrath To: "'Douglas Otis'" , Jeff Fellin , ips@ece.cmu.edu Subject: RE: Towards Consensus on TCP connections Date: Wed, 16 Aug 2000 18:41:15 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk It is perfectly SAM compliant to restrict the size of the LUN space if you wanted for a given transport (as an obvious example, parallel SCSI has only a handful of LUNs per target). I'm not saying it is the right thing to do, only pointing out that the 64 bit LUN in SAM can be restricted (and is today as a matter of course). Jim -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Wednesday, August 16, 2000 2:18 PM To: Jeff Fellin; ips@ece.cmu.edu Subject: RE: Towards Consensus on TCP connections > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jeff Fellin > Sent: Wednesday, August 16, 2000 11:41 AM > To: ips@ece.cmu.edu > Subject: Re: Towards Consensus on TCP connections > > > > Folks, > After reading all the discussion and debate on this issue, I decided to > research if the SCSI standards (T10/X3) have done with developing other > transport protocols, and how those protocols dealt with connectivity > issues between the endpoints. For the most part the other transport > protocols SCSI over Scheduled Transport (SST), which is a working draft > and the SCSI-3 Generic Packetized Prococol (SCSI-GPP), which is a proposed > technical report have a concept of a Logical or Virtual connection. This > Logical or Virtual connection has at least one connection through the > network, but may have more up to endpoint configuration parameters. So, > it appears the other groups have not come to a consensus as to > the approach. > > Most of the drafts are available on the T10's web site: > www.t10.org > > Except for the definition of the Scheduled transfer protocol, which is on > T11's web site: > www.t11.org > > The most interesting thing I came up with is the SCSI-GPP definition is > written to allow connections over IP networks. Annexes E and F describe > the mapping and processing capabilities. If this is a work in progress > should we take advantage of there previous start and work with T10 to > have a better definition of a protocol for storage over IP? > > What I understand from a quick read of the SST and SCSI-GPP protocol > definitions is they have dealt with the issues of security, error > recovery, > connection management, transfer of SCSI bus control signals (bus reset, > abort tasks. > > If we don't use the SCSI-GPP than I propose changing the one LUN per > TCP connection group to be a SCSI target id that multiplexes all LUN's > of the target id onto the connection group. I believe this will decrease > the amount of connections and provide data channels for large transfers. see: ftp://ftp.t10.org/t10/drafts/gpp/gpp-r09.pdf Jeff, There is a considerable difference between the SCSI-GPP and iSCSI. With GPP there are only 256 possible LUNs per target (among other notable features). Under iSCSI, there are 18,447,000,000,000,000,000 possible LUNs per target and this 64-bit LUN value is implied in some cases. Doug From owner-ips@ECE.cmu.edu Thu Aug 17 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24641; 17 Aug 2000 1:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA06190 for ips-outgoing; Thu, 17 Aug 2000 00:31:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from auemlsrv.firewall.lucent.com (auemail1.lucent.com [192.11.223.161]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA06179 for ; Thu, 17 Aug 2000 00:31:43 -0400 (EDT) Received: from auemlsrv.firewall.lucent.com (localhost [127.0.0.1]) by auemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id AAA17369 for ; Thu, 17 Aug 2000 00:31:07 -0400 (EDT) Received: from il0015exch001h.wins.lucent.com (h135-1-23-83.lucent.com [135.1.23.83]) by auemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id AAA17115 for ; Thu, 17 Aug 2000 00:30:31 -0400 (EDT) Received: by il0015exch001h.ih.lucent.com with Internet Mail Service (5.5.2650.21) id ; Wed, 16 Aug 2000 23:30:31 -0500 Message-ID: <80B684C5E29FD211AA8000A0C9CDD91904D0984D@IL0015EXCH005U> From: "Rodriguez, Elizabeth G (Elizabeth)" To: "'mjacob@feral.com'" , csapuntz@cisco.com Cc: ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Date: Wed, 16 Aug 2000 23:30:30 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk All, The purpose of FC over IP is to extend FC networks with minimal impact to existing devices and existing installations. It is not intended to be an iSCSI equivalent/counterpart, e.g. I would not expect someone to put FC over IP into an end device. It is really intended to be something put into a switch, not at an endpoint. If you have an existing FC environment, the question is not whether you could run GigE as well (you could, and probably would), but do you want to replace all the existing FC environment (e.g. all the existing FC controllers, disks, tapes, RAID, JBOD, etc), with new equipment, (equipment that is iSCSI compliant), or do you want to add a single device at the edge of each of your FC fabrics, and enable the fabrics to overcome distance limitations utilizing a network that is already in place for other LAN/WAN traffic. Over the next few days, I will post more information about FC over IP to this reflector. This will include some of the items that will most likely become requirements for the FC over IP draft. Included will be some requirements addressing Costa's comments below. Elizabeth Rodriguez Lucent Technologies -----Original Message----- From: Matthew Jacob [mailto:mjacob@feral.com] Sent: Wednesday, August 16, 2000 6:16 PM To: csapuntz@cisco.com Cc: ips@ece.cmu.edu Subject: Re: FC/IP vs. iSCSI I'm certainly inclined to agree. The sole realistic purpose of requirements I've seen to date FC/IP is to utilize an existing FC environment so that management and data go over the same wire. If you're carrying SCSI over TCP/IP now, it's somewhat absured to try and carry FC/IP over the same transport. It's as if you want to tunnel TCP/IP over TCP/IP, which is a cute trick, but a trick for all that. If you have an existing FC optical environment, it could probably run GigEthernet just as well- and it then becomes a question as to whether iSCSI serves the systems and applications as well or better than FC-SCSI or FC/IP over the same physical media. -matt On 16 Aug 2000 csapuntz@cisco.com wrote: > > FC/IP is NOT an alternative to iSCSI for most applications. > > Unfortunately, making a protocol that works well in complex networks > is not as simple as just putting stuff into IP packets. IP packet > headers are not magical pixie dust that suddenly make higher-layer > protocol issues go away. > > The FibreChannel stack today has the following deficiencies which do > not disappear when tunneling over IP: > - FCP has no congestion control > - FCP deals poorly with packet loss > - Target naming is done with either 24-bit port IDs or 64-bit WWNs, > neither of which scale to Internet size > - No secure login > > Of course, you could address all the deficiencies by fixing FCP. But > by the time you do that, I maintain you will most likely end up with > something in the same order of complexity as iSCSI/TCP/IP. > > -Costa > > From owner-ips@ECE.cmu.edu Thu Aug 17 06:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24987; 17 Aug 2000 2:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA07242 for ips-outgoing; Thu, 17 Aug 2000 01:21:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA07237 for ; Thu, 17 Aug 2000 01:20:57 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA69424 for ; Thu, 17 Aug 2000 07:20:17 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id HAA28412 for ; Thu, 17 Aug 2000 07:20:17 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693E.001D4F13 ; Thu, 17 Aug 2000 07:20:07 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 17 Aug 2000 08:18:10 +0300 Subject: Re: Towards Consensus on TCP connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jeff Fellin wrote... ------------------ If we don't use the SCSI-GPP than I propose changing the one LUN per TCP connection group to be a SCSI target id that multiplexes all LUN's of the target id onto the connection group. I believe this will decrease the amount of connections and provide data channels for large transfers. _______________ Thanks Jeff, We went through the same documents and reached the same conclusion. Errors are handled better that in the basic parallel SCSI and it has a neat CRC mechanism at predefined points in the stream. We hope to achieve the same result with a digest+CRC mechanism. The sentence quoted above from your note ... I am not sure that I understand it. The LUN in iSCSI is carried in the packet and the target-id is mostly implied except for third party commands (and the name sent with the login). What did you have in mind? Regards, Julo From owner-ips@ECE.cmu.edu Thu Aug 17 12:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26632; 17 Aug 2000 8:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA22658 for ips-outgoing; Thu, 17 Aug 2000 07:15:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA22653 for ; Thu, 17 Aug 2000 07:15:14 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id HAA02106 for ips@ece.cmu.edu; Thu, 17 Aug 2000 07:14:55 -0400 (EDT) Date: Thu, 17 Aug 2000 07:14:55 -0400 (EDT) From: Scott Bradner Message-Id: <200008171114.HAA02106@newdev.harvard.edu> To: ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Sender: owner-ips@ece.cmu.edu Precedence: bulk Elizabeth sez: > e.g. I would not > expect someone to put FC over IP into an end device. It is really intended > to be something put into a switch, not at an endpoint. I would expect vendors to do anything that they think will sell and I expect that some vendors would think that connecting a FC endpoint to a FC switch over IP would sell Scott From owner-ips@ECE.cmu.edu Thu Aug 17 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27476; 17 Aug 2000 11:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA27945 for ips-outgoing; Thu, 17 Aug 2000 10:24:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imps0014.us.dg.com (imps0014.us.dg.com [128.221.44.14]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA27938 for ; Thu, 17 Aug 2000 10:24:55 -0400 (EDT) Received: by imps0014.us.dg.com with Internet Mail Service (5.5.2650.21) id ; Thu, 17 Aug 2000 10:24:01 -0400 Message-ID: <0D4CDC691CA3D211BF0B00104B2ABA7206F2A0C5@imps0044.us.dg.com> From: "Ericson, George" To: "'csapuntz@cisco.com'" , ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Date: Thu, 17 Aug 2000 10:24:01 -0400 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, Not arguing for or against iSCSI. However, I don't agree with two of your points on FC. FC ports are identified by both a 64 bit port-WWN and a 64 bit node-WWN. Together, they form a 128 bit WWN that should scale to the internet. (Note, that these addresses do not provide any routing hints. But that's a different kettle of fish.) Secure Login is not an FCP layer problem. My opinion is that the trust relationship must be established end-to-end. For SCSI, this means between a SCSI Application Client and a SCSI Logical Unit. There is ongoing work in this area within T10 that should be relevant to the iSCSI effort. Note that a trust relationship can also be established at the FC-2 level. However, the usefulness is more limited at that level. Regards, George Ericson EMC Corporation < http://www.emc.com/> CLARiiON Advanced Technology 4 Coslin Drive, MS C44, Southboro, MA 01772 Office: (508) 480-7349; Mobile: (508) 498-8461; Fax: (508) 480-7913 -----Original Message----- From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] Sent: Wednesday, August 16, 2000 5:56 PM To: ips@ece.cmu.edu Cc: csapuntz@cisco.com Subject: FC/IP vs. iSCSI FC/IP is NOT an alternative to iSCSI for most applications. Unfortunately, making a protocol that works well in complex networks is not as simple as just putting stuff into IP packets. IP packet headers are not magical pixie dust that suddenly make higher-layer protocol issues go away. The FibreChannel stack today has the following deficiencies which do not disappear when tunneling over IP: - FCP has no congestion control - FCP deals poorly with packet loss - Target naming is done with either 24-bit port IDs or 64-bit WWNs, neither of which scale to Internet size - No secure login Of course, you could address all the deficiencies by fixing FCP. But by the time you do that, I maintain you will most likely end up with something in the same order of complexity as iSCSI/TCP/IP. -Costa From owner-ips@ECE.cmu.edu Thu Aug 17 16:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27502; 17 Aug 2000 12:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA29517 for ips-outgoing; Thu, 17 Aug 2000 11:02:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA29510 for ; Thu, 17 Aug 2000 11:02:47 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e7HF2lw27884 for ; Thu, 17 Aug 2000 10:02:47 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA29124; Thu, 17 Aug 00 10:00:37 -0500 Message-Id: <10008171500.AA29124@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Towards Consensus on TCP connections In-Reply-To: Message from Ralph Weber of "Wed, 16 Aug 2000 18:14:44 CDT." <399B2064.49EAF26A@compuserve.com> Date: Thu, 17 Aug 2000 10:02:06 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > It is also very important to note that SPI-4, FCP-2, and SVP are > active projects that share a goal of common packet formats to > simplify implementation of bridges between the parallel bus, > fibre channel, and VI. SST also has this goal. The addressing, and command and status formats are as similar as we could make them to FCP. It was quite handy to do this. On one hand, given the mismatch in other aspects than information unit format between iSCSI and FCP (specifically, the congestion avoidance requirements), cribbing IU format from FCP[-2] is not going to go a huge distance towards making iSCSI easier to implement. On the other hand, making everything possible about iSCSI `same as before', WILL still help substantially to anybody approaching the technology (device implementer, driver writer, etc.). For example, in SST, it was useful to be able to start with an FCP driver source and just adapt that to fit the ST layer interface. Again, not a huge deal, but it does help make it seem like a familiar task, rather than a bizarre one. Steph From owner-ips@ECE.cmu.edu Thu Aug 17 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27817; 17 Aug 2000 13:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA02204 for ips-outgoing; Thu, 17 Aug 2000 12:08:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA02200 for ; Thu, 17 Aug 2000 12:08:13 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA03881; Thu, 17 Aug 2000 09:02:53 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA20671; Thu, 17 Aug 2000 09:00:16 -0700 From: "Douglas Otis" To: "Rodriguez, Elizabeth G \(Elizabeth\)" , , Cc: Subject: RE: FC/IP vs. iSCSI Date: Thu, 17 Aug 2000 09:10:12 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <80B684C5E29FD211AA8000A0C9CDD91904D0984D@IL0015EXCH005U> Sender: owner-ips@ece.cmu.edu Precedence: bulk Elizabeth, You should investigate the SCTP specification. http://www.ietf.org/internet-drafts/draft-ietf-sigtran-sctp-13.txt It can be used to add congestion and packet loss control for FC over IP applications with less than dedicated networks. It directly handles fragmentation, adds additional authentication and spoofing security, improved error checking, and does not require sequential delivery. SCTP still allows frame alignment and would need only a minimal amount of management (LDAP as example). As a quick extension to the existing specification, it then places FC over IP within the iSCSI charter. For applications where aggregation takes place by a hierarchy of FC controllers and switches, FC over IP and FC over SCTP/IP makes sense. iSCSI as proposed does not for that environment. Once a native IP device becomes available it should make use of existing networking conventions for reliability and scalability. iSCSI should not be designed to touch the top of the FC pyramid, but instead the bottom. Lock, cache, and array servers should remain close to the client or network latency becomes a significant problem. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Rodriguez, Elizabeth G (Elizabeth) > Sent: Wednesday, August 16, 2000 9:31 PM > To: 'mjacob@feral.com'; csapuntz@cisco.com > Cc: ips@ece.cmu.edu > Subject: RE: FC/IP vs. iSCSI > > > All, > > The purpose of FC over IP is to extend FC networks with minimal impact to > existing devices and existing installations. > It is not intended to be an iSCSI equivalent/counterpart, e.g. I would not > expect someone to put FC over IP into an end device. It is > really intended > to be something put into a switch, not at an endpoint. > > If you have an existing FC environment, the question is not whether you > could run GigE as well (you could, and probably would), but do you want to > replace all the existing FC environment (e.g. all the existing FC > controllers, disks, tapes, RAID, JBOD, etc), with new equipment, > (equipment > that is iSCSI compliant), or do you want to add a single device > at the edge > of each of your FC fabrics, and enable the fabrics to overcome distance > limitations utilizing a network that is already in place for other LAN/WAN > traffic. > > Over the next few days, I will post more information about FC over IP to > this reflector. > This will include some of the items that will most likely become > requirements for the FC over IP draft. > Included will be some requirements addressing Costa's comments below. > > Elizabeth Rodriguez > Lucent Technologies > -----Original Message----- > From: Matthew Jacob [mailto:mjacob@feral.com] > Sent: Wednesday, August 16, 2000 6:16 PM > To: csapuntz@cisco.com > Cc: ips@ece.cmu.edu > Subject: Re: FC/IP vs. iSCSI > > > > I'm certainly inclined to agree. The sole realistic purpose of > requirements > I've seen to date FC/IP is to utilize an existing FC environment so that > management and data go over the same wire. If you're carrying SCSI over > TCP/IP > now, it's somewhat absured to try and carry FC/IP over the same transport. > It's as if you want to tunnel TCP/IP over TCP/IP, which is a cute > trick, but > a > trick for all that. > > If you have an existing FC optical environment, it could probably run > GigEthernet just as well- and it then becomes a question as to > whether iSCSI > serves the systems and applications as well or better than > FC-SCSI or FC/IP > over the same physical media. > > -matt > > > On 16 Aug 2000 csapuntz@cisco.com wrote: > > > > > FC/IP is NOT an alternative to iSCSI for most applications. > > > > Unfortunately, making a protocol that works well in complex networks > > is not as simple as just putting stuff into IP packets. IP packet > > headers are not magical pixie dust that suddenly make higher-layer > > protocol issues go away. > > > > The FibreChannel stack today has the following deficiencies which do > > not disappear when tunneling over IP: > > - FCP has no congestion control > > - FCP deals poorly with packet loss > > - Target naming is done with either 24-bit port IDs or 64-bit WWNs, > > neither of which scale to Internet size > > - No secure login > > > > Of course, you could address all the deficiencies by fixing FCP. But > > by the time you do that, I maintain you will most likely end up with > > something in the same order of complexity as iSCSI/TCP/IP. > > > > -Costa > > > > > From owner-ips@ECE.cmu.edu Thu Aug 17 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28056; 17 Aug 2000 14:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA04184 for ips-outgoing; Thu, 17 Aug 2000 13:00:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ihemlsrv.firewall.lucent.com (ihemail1.lucent.com [192.11.222.161]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA04179 for ; Thu, 17 Aug 2000 13:00:54 -0400 (EDT) Received: from ihemlsrv.firewall.lucent.com (localhost [127.0.0.1]) by ihemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id NAA16082 for ; Thu, 17 Aug 2000 13:00:53 -0400 (EDT) Received: from il0015exch001h.wins.lucent.com (h135-1-23-83.lucent.com [135.1.23.83]) by ihemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id NAA15998 for ; Thu, 17 Aug 2000 13:00:53 -0400 (EDT) Received: by il0015exch001h.ih.lucent.com with Internet Mail Service (5.5.2650.21) id ; Thu, 17 Aug 2000 12:00:52 -0500 Message-ID: <80B684C5E29FD211AA8000A0C9CDD91904D09857@IL0015EXCH005U> From: "Rodriguez, Elizabeth G (Elizabeth)" To: "'Scott Bradner'" , ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Date: Thu, 17 Aug 2000 12:00:43 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Scott, I will not argue that point. But my point in that email is not that it could not be put integrated onto an end device, but that it is not the intent. The focus is to bridge existing deployments over IP and address limitations in the physical media. Elizabeth -----Original Message----- From: Scott Bradner [mailto:sob@harvard.edu] Sent: Thursday, August 17, 2000 6:15 AM To: ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Elizabeth sez: > e.g. I would not > expect someone to put FC over IP into an end device. It is really intended > to be something put into a switch, not at an endpoint. I would expect vendors to do anything that they think will sell and I expect that some vendors would think that connecting a FC endpoint to a FC switch over IP would sell Scott From owner-ips@ECE.cmu.edu Thu Aug 17 19:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28297; 17 Aug 2000 15:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA07006 for ips-outgoing; Thu, 17 Aug 2000 14:07:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA07001 for ; Thu, 17 Aug 2000 14:07:22 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id UAA28826 for ; Thu, 17 Aug 2000 20:06:47 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id UAA20268 for ; Thu, 17 Aug 2000 20:06:47 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693E.00637E6D ; Thu, 17 Aug 2000 20:06:44 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 17 Aug 2000 21:02:49 +0300 Subject: Re: Towards Consensus on TCP connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk That is a very good point. We too avoided touching structures we didn't have too (this is how we ended-up with huge LUNs). Bridging will be simple for "proper networks" but might involve more logic for mapping to SCSI-2 or very simple SPI. And I would appreciate any more insights into bridging. We assume that drivers for widely available OSs will be readily available in source. Julo Stephen Bailey on 17/08/2000 18:02:06 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Towards Consensus on TCP connections > It is also very important to note that SPI-4, FCP-2, and SVP are > active projects that share a goal of common packet formats to > simplify implementation of bridges between the parallel bus, > fibre channel, and VI. SST also has this goal. The addressing, and command and status formats are as similar as we could make them to FCP. It was quite handy to do this. On one hand, given the mismatch in other aspects than information unit format between iSCSI and FCP (specifically, the congestion avoidance requirements), cribbing IU format from FCP[-2] is not going to go a huge distance towards making iSCSI easier to implement. On the other hand, making everything possible about iSCSI `same as before', WILL still help substantially to anybody approaching the technology (device implementer, driver writer, etc.). For example, in SST, it was useful to be able to start with an FCP driver source and just adapt that to fit the ST layer interface. Again, not a huge deal, but it does help make it seem like a familiar task, rather than a bizarre one. Steph From owner-ips@ECE.cmu.edu Thu Aug 17 19:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28310; 17 Aug 2000 15:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06430 for ips-outgoing; Thu, 17 Aug 2000 13:53:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from manganese.network-alchemy.com (NAT-10-net.Network-Alchemy.COM [199.46.17.205]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06425 for ; Thu, 17 Aug 2000 13:53:27 -0400 (EDT) Received: from manganese.network-alchemy.com (localhost [127.0.0.1]) by manganese.network-alchemy.com (8.9.3/8.8.8) with ESMTP id KAA80262; Thu, 17 Aug 2000 10:54:17 -0700 (PDT) (envelope-from rdv@manganese.network-alchemy.com) Message-Id: <200008171754.KAA80262@manganese.network-alchemy.com> X-Mailer: exmh version 2.0.2 2/24/98 To: mjacob@feral.com Subject: Re: Towards Consensus on TCP connections In-reply-to: mjacob's message of Wed, 16 Aug 2000 16:29:31 -0700. Reply-To: rdv@Network-Alchemy.COM Cc: IP Storage Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 17 Aug 2000 10:54:17 -0700 From: Rod Van Meter Sender: owner-ips@ece.cmu.edu Precedence: bulk Sigh, I've been way too busy to contribute significantly here, and there's nowhere I'd rather be :-(. We had MANY of these same arguments at Quantum in 1998 and 1999; they're endlessly fascinating but can really only be resolved by implementation and comparison. I've known about GPP for years, but my memory fades. I was told by a member of T10 at one point that it was really put together and driven by someone in particular, and when his application failed to take off, was more or less abandoned. I don't recall who, or what the application was. SCSI over ST and ST over IP offer a clear path today, but I don't think it's adequate. I'm glad to see the full iSCSI work going on. I'm in favor of keeping FC over IP out of the equation; I'm opposed to it, anyway, but I think this group should continue to concentrate on iSCSI as currently understood. It's a little late (I think we're past the "TCP is too expensive" argument for the moment), but I happened across this URL this morning in the references of Wright & Stevens, v.2. http://www.kohala.com/start/vanj.93sep07.txt TCP processing in 30 instructions, according to Van Jacobson. I hand-counted the Linux 2.2 stack, and came up with about 200 instructions, not counting checksum, but including the interrupt handler, ethernet processing, IP, and TCP, I believe. I'll see if I can dig up my notes and post them here. I'm in favor, btw, of multiple TCP streams. Anyway, I'm hoping to contribute some to this discussion from this point forward. Sorry I haven't been involved much to date. --Rod From owner-ips@ECE.cmu.edu Thu Aug 17 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28490; 17 Aug 2000 15:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA08338 for ips-outgoing; Thu, 17 Aug 2000 14:41:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ihemlsrv.firewall.lucent.com (ihemail1.lucent.com [192.11.222.161]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA08323 for ; Thu, 17 Aug 2000 14:41:28 -0400 (EDT) Received: from ihemlsrv.firewall.lucent.com (localhost [127.0.0.1]) by ihemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id OAA25919 for ; Thu, 17 Aug 2000 14:41:27 -0400 (EDT) Received: from il0015exch001h.wins.lucent.com (h135-1-23-83.lucent.com [135.1.23.83]) by ihemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id OAA25908 for ; Thu, 17 Aug 2000 14:41:27 -0400 (EDT) Received: by il0015exch001h.ih.lucent.com with Internet Mail Service (5.5.2650.21) id ; Thu, 17 Aug 2000 13:41:27 -0500 Message-ID: <80B684C5E29FD211AA8000A0C9CDD91904D09858@IL0015EXCH005U> From: "Rodriguez, Elizabeth G (Elizabeth)" To: "'Douglas Otis'" , mjacob@feral.com, csapuntz@cisco.com Cc: ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Date: Thu, 17 Aug 2000 13:41:26 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Doug, Thanks for your comments/input. SCTP has been mentioned as a possible transport to use for congestion management. It sounds as if it has a lot to offer. I am planning to look into SCTP as a possible approach. Thanks for the reference/information. In my presentation to the IPS, I do not recall if I mentioned that we did not have any plans for congestion management or not. It has been made clear (by the ADs) that this will be a requirement of FC over IP if it is to be approved by the IETF. So, any recommendations/input into what FC over IP should use for congestion management are all welcome. I would like to see more input on recommendations for FC over IP. As I mentioned previously, I will be sending an email out shortly on more about FC over IP, and hopefully start some dialog on requirements for that specification and recommendations. Thanks, Elizabeth Rodriguez Lucent Technologies -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Thursday, August 17, 2000 11:10 AM To: Rodriguez, Elizabeth G (Elizabeth); mjacob@feral.com; csapuntz@cisco.com Cc: ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Elizabeth, You should investigate the SCTP specification. http://www.ietf.org/internet-drafts/draft-ietf-sigtran-sctp-13.txt It can be used to add congestion and packet loss control for FC over IP applications with less than dedicated networks. It directly handles fragmentation, adds additional authentication and spoofing security, improved error checking, and does not require sequential delivery. SCTP still allows frame alignment and would need only a minimal amount of management (LDAP as example). As a quick extension to the existing specification, it then places FC over IP within the iSCSI charter. For applications where aggregation takes place by a hierarchy of FC controllers and switches, FC over IP and FC over SCTP/IP makes sense. iSCSI as proposed does not for that environment. Once a native IP device becomes available it should make use of existing networking conventions for reliability and scalability. iSCSI should not be designed to touch the top of the FC pyramid, but instead the bottom. Lock, cache, and array servers should remain close to the client or network latency becomes a significant problem. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Rodriguez, Elizabeth G (Elizabeth) > Sent: Wednesday, August 16, 2000 9:31 PM > To: 'mjacob@feral.com'; csapuntz@cisco.com > Cc: ips@ece.cmu.edu > Subject: RE: FC/IP vs. iSCSI > > > All, > > The purpose of FC over IP is to extend FC networks with minimal impact to > existing devices and existing installations. > It is not intended to be an iSCSI equivalent/counterpart, e.g. I would not > expect someone to put FC over IP into an end device. It is > really intended > to be something put into a switch, not at an endpoint. > > If you have an existing FC environment, the question is not whether you > could run GigE as well (you could, and probably would), but do you want to > replace all the existing FC environment (e.g. all the existing FC > controllers, disks, tapes, RAID, JBOD, etc), with new equipment, > (equipment > that is iSCSI compliant), or do you want to add a single device > at the edge > of each of your FC fabrics, and enable the fabrics to overcome distance > limitations utilizing a network that is already in place for other LAN/WAN > traffic. > > Over the next few days, I will post more information about FC over IP to > this reflector. > This will include some of the items that will most likely become > requirements for the FC over IP draft. > Included will be some requirements addressing Costa's comments below. > > Elizabeth Rodriguez > Lucent Technologies > -----Original Message----- > From: Matthew Jacob [mailto:mjacob@feral.com] > Sent: Wednesday, August 16, 2000 6:16 PM > To: csapuntz@cisco.com > Cc: ips@ece.cmu.edu > Subject: Re: FC/IP vs. iSCSI > > > > I'm certainly inclined to agree. The sole realistic purpose of > requirements > I've seen to date FC/IP is to utilize an existing FC environment so that > management and data go over the same wire. If you're carrying SCSI over > TCP/IP > now, it's somewhat absured to try and carry FC/IP over the same transport. > It's as if you want to tunnel TCP/IP over TCP/IP, which is a cute > trick, but > a > trick for all that. > > If you have an existing FC optical environment, it could probably run > GigEthernet just as well- and it then becomes a question as to > whether iSCSI > serves the systems and applications as well or better than > FC-SCSI or FC/IP > over the same physical media. > > -matt > > > On 16 Aug 2000 csapuntz@cisco.com wrote: > > > > > FC/IP is NOT an alternative to iSCSI for most applications. > > > > Unfortunately, making a protocol that works well in complex networks > > is not as simple as just putting stuff into IP packets. IP packet > > headers are not magical pixie dust that suddenly make higher-layer > > protocol issues go away. > > > > The FibreChannel stack today has the following deficiencies which do > > not disappear when tunneling over IP: > > - FCP has no congestion control > > - FCP deals poorly with packet loss > > - Target naming is done with either 24-bit port IDs or 64-bit WWNs, > > neither of which scale to Internet size > > - No secure login > > > > Of course, you could address all the deficiencies by fixing FCP. But > > by the time you do that, I maintain you will most likely end up with > > something in the same order of complexity as iSCSI/TCP/IP. > > > > -Costa > > > > > From owner-ips@ECE.cmu.edu Thu Aug 17 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28651; 17 Aug 2000 16:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA10254 for ips-outgoing; Thu, 17 Aug 2000 15:28:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-34.cisco.com [204.69.198.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA10249 for ; Thu, 17 Aug 2000 15:28:26 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id MAA02658; Thu, 17 Aug 2000 12:27:59 -0700 To: ips@ece.cmu.edu Subject: Re: FC/IP vs. iSCSI References: <0D4CDC691CA3D211BF0B00104B2ABA7206F2A0C5@imps0044.us.dg.com> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 17 Aug 2000 12:27:58 -0700 In-Reply-To: "Ericson, George"'s message of "Thu, 17 Aug 2000 10:24:01 -0400" Message-ID: Lines: 24 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi George, > FC ports are identified by both a 64 bit port-WWN and a 64 bit node-WWN. > Together, they form a 128 bit WWN that should scale to the internet. While a 64-bit flat namespace can certainly uniquely identify all hosts on the Internet, the name cannot usefully be translated to a route or address. The flat namespace leads to a central authority that translates all 64-bit ids to routes. I do not believe such a central authority which maintains a flat namespace would scale from a management perspective (imagine having to register storage LUN with some central authority!) And it seems to me that building a system that can handle mapping 1 billion LUN WWNs to IP addresses would be a non-trivial engineering problem. > (Note, that these addresses do not provide any routing hints. But that's a > different kettle of fish.) Actually, that's a big problem with using WWNs exclusively. -Costa From owner-ips@ECE.cmu.edu Thu Aug 17 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28664; 17 Aug 2000 16:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA10743 for ips-outgoing; Thu, 17 Aug 2000 15:39:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-34.cisco.com [204.69.198.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA10734 for ; Thu, 17 Aug 2000 15:38:56 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id MAA02687; Thu, 17 Aug 2000 12:38:29 -0700 To: ips@ece.cmu.edu Subject: Re: FC/IP vs. iSCSI References: <80B684C5E29FD211AA8000A0C9CDD91904D0984D@IL0015EXCH005U> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 17 Aug 2000 12:38:29 -0700 In-Reply-To: "Rodriguez, Elizabeth G's message of "Wed, 16 Aug 2000 23:30:30 -0500" Message-ID: Lines: 25 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk "Rodriguez, Elizabeth G (Elizabeth)" writes: > All, > > The purpose of FC over IP is to extend FC networks with minimal impact to > existing devices and existing installations. > It is not intended to be an iSCSI equivalent/counterpart, e.g. I would not > expect someone to put FC over IP into an end device. It is really intended > to be something put into a switch, not at an endpoint. Elizabeth, You have never represented FC/IP as anything but a tunneling/bridging solution. I believe tunneling is an application where FC/IP has interesting story to tell. My e-mail was meant to respond to those who would propose FC/IP as an alternative to iSCSI for other applications, e.g. connecting storage to the Internet. I apologize if it came across as an assault on your good work in this area. -Costa From owner-ips@ECE.cmu.edu Thu Aug 17 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29424; 17 Aug 2000 19:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA16613 for ips-outgoing; Thu, 17 Aug 2000 18:16:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA16608 for ; Thu, 17 Aug 2000 18:16:28 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id PAA03957; Thu, 17 Aug 2000 15:11:09 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA21611; Thu, 17 Aug 2000 15:08:38 -0700 From: "Douglas Otis" To: , Subject: RE: FC/IP vs. iSCSI Date: Thu, 17 Aug 2000 15:18:37 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, For FC over IP, the implementers must make it compliant for use on the Internet in obtaining IETF approval. As the IPS charter would encompass such an effort, there is no reason for excluding FC over SCTP/IP. In reality, the iSCSI proposal was to a be a modification for a Fibre-Channel controller. A complete and unaltered encapsulation is the most straightforward approach and likely with the lowest overhead and risk. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > csapuntz@cisco.com > Sent: Thursday, August 17, 2000 12:38 PM > To: ips@ece.cmu.edu > Cc: csapuntz@cisco.com > Subject: Re: FC/IP vs. iSCSI > > > > "Rodriguez, Elizabeth G (Elizabeth)" writes: > > > All, > > > > The purpose of FC over IP is to extend FC networks with minimal > impact to > > existing devices and existing installations. > > It is not intended to be an iSCSI equivalent/counterpart, e.g. > I would not > > expect someone to put FC over IP into an end device. It is > really intended > > to be something put into a switch, not at an endpoint. > > Elizabeth, > > You have never represented FC/IP as anything but a tunneling/bridging > solution. I believe tunneling is an application where FC/IP has > interesting story to tell. > > My e-mail was meant to respond to those who would propose FC/IP as > an alternative to iSCSI for other applications, e.g. connecting > storage to the Internet. > > I apologize if it came across as an assault on your good work in this > area. > > -Costa > From owner-ips@ECE.cmu.edu Fri Aug 18 00:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29616; 17 Aug 2000 20:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA18101 for ips-outgoing; Thu, 17 Aug 2000 19:12:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from proxy1.it.stsj.seagate.com (ns1.seagate.com [204.160.183.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA25436 for ; Thu, 17 Aug 2000 09:09:02 -0400 (EDT) From: Chris.Malakapalli@seagate.com Received: (from smap) by proxy1.it.stsj.seagate.com id NAA17294 for ; Thu, 17 Aug 2000 13:08:31 GMT Received: from auth0.it.stsj.seagate.com(192.168.109.16) by proxy1.it.stsj.seagate.com via smap (V5.0) id xma017063; Thu, 17 Aug 00 13:07:37 GMT Received: from sv-gw1.stsv.seagate.com (sv-gw1.stsv.seagate.com [134.204.14.95]) by auth0.it.stsj.seagate.com with ESMTP id NAA11678 for ; Thu, 17 Aug 2000 13:07:37 GMT Subject: RE: FC/IP vs. iSCSI To: ips@ece.cmu.edu Date: Thu, 17 Aug 2000 08:07:31 -0500 Message-ID: X-MIMETrack: Serialize by Router on SV-GW1/Seagate Internet(Release 5.0.2a (Intl)|23 November 1999) at 08/17/2000 06:07:37 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk > Over the next few days, I will post more information about FC over IP to > this reflector. I would like to suggest that perhaps a separate reflector be created for FC over IP related discussions. I think combining the discussions in one reflector would create unnecessary confusion and burden for everyone to sort out which belongs where. Please don't tell me that reflector and workgroup have a one-to-one relationship. From owner-ips@ECE.cmu.edu Fri Aug 18 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30095; 17 Aug 2000 22:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA20322 for ips-outgoing; Thu, 17 Aug 2000 20:41:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel2.hp.com (cosrel2.hp.com [156.153.255.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA20317 for ; Thu, 17 Aug 2000 20:41:33 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by cosrel2.hp.com (Postfix) with ESMTP id C70836E3; Thu, 17 Aug 2000 18:41:22 -0600 (MDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id RAA24364; Thu, 17 Aug 2000 17:41:31 -0700 (PDT) X-OpenMail-Hops: 1 Date: Thu, 17 Aug 2000 17:41:14 -0700 Message-Id: Subject: RE: RE: Towards Consensus on TCP Connections MIME-Version: 1.0 To: dotis@sanlight.net, ips@ece.cmu.edu, julian_satran@il.ibm.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, Tunneling FC through IP to connect FC islands together is a good application and has its place, and an iSCSI <-> FCP gateway will be an important app to connect the two worlds together. However neither of these is a replacement for using TCP/IP as a native transport for SCSI commands. FC networks appear to me to be limited scale networks providing a very reliable link (channel) using b-b credits to avoid dropping packets. Therefore any protocol using FC network makes those assumptions. However this limits the scale, and also other factors are driving the use of IP based networks. When FC goes over an IP based networks, then either the assumptions (of a reliable channel) are no longer true, or a convergence layer has to be built to provide the same level of service (using TCP and another sub-layer on top perhaps). Even in FC, you have SCSI states etc and FC states (fairly involved protocol and does not have all the capabilities as IP based networks provide - or at not yet at the same level of interoperability and robustness). You have made a comment multiple times about the compexlity of SCSI over TCP and complexity of states and demuxing etc. I somehow don't get it (when I compare it to the fact that systems have been managing loads of apps running on top of TCP and that switches and NICs are implementing TCP in silicon). Perhaps you can give some details of the state and demux complexity and error recovery issues and we can explore them to see if there is something we are missing or perhaps some of us can help with explaining why they may not be issues. SCSI over TCP is probably more complex than FCP/FC. However, it should provide more capabilities and a converged network. It should also be simpler than encapsulating FC over IP (TCP). Also there is far more expertise in this area and it is using existing components except the iSCSI encapsulation which is fairly simple. I don't think there is any unknowns in this area. So let us explore and complexity issues in more detail. Somesh > -----Original Message----- > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > Sent: Wednesday, August 16, 2000 8:04 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: FW: RE: Towards Consensus on TCP Connections > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Tuesday, August 15, 2000 10:06 PM > > To: ips@ece.cmu.edu > > Subject: RE: Towards Consensus on TCP Connections > > > > Doug, > > > > Please cool down and keep the discussion technical. I > > personally do not enjoy bringing press quotes to a technical > > discussion. And so much more when they are inaccurate. > > You indicated confusion as to the comparison being made. I tried to > enlighten you as to the alternative. I did it with humor and > with technical > details. I see nothing inaccurate nor have you indicated errors. > > > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > > that FCP will be encapsulated on IP. FCP is tightly tied to FC > > (as TCP is to IP) and has already a complex (and not yet closed) > > recovery mechanism based on the assumption it runs over FC. > > An IP datagram is nothing more than a vehicle for FC encapsulation and > indeed includes only the defragmentation process. There is no need to > manage the defrag process or to be concerned about possible > failure at the > IP level. It is transparent and self healing as FCP-2 on FC > handles these > errors and would be no less reliable in doing so. There are > few uses for FC > beyond SCSI so your point escapes me especially when you are > advocating a > protocol wholly unsupported. Are you suggesting FC over IP > will not work? > > > Moreover - like SCSI - FCP has the notion of a target distinct > > from the notion of a LUN. It could be an acceptable solution to > > extend the range of a FC island but even for this it has to go > > some way in solving the congestion and security issues. > > There are already provisions for security for FC over IP. In many > applications, bandwidth is controlled by a dedicated channel > which solves > both issues. There are many means for flow control where TCP > is but one. > Reliability should be the differentiating factor. > > > IMHO even for this application a FCP-iSCSI gateway could > provide a better > > solution as the mapping is straightforward. > > > > Julo > > This provides few techincal details as to how you base this > opinion. If I > had the choice of purchasing a blade for a switch or new > system adapters > with high overhead and new SAN controllers with limited > connectivity where I > could also expect massive faults, I would not see that as > straightforward. > But then again, maybe I would. > > Doug > > > > "Douglas Otis" on 15/08/2000 19:24:34 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Towards Consensus on TCP Connections > > > > Julo, > > > > For details on proposed Fibre-Channel over IP which does not impose > > additional catastrophic error states as does iSCSI, please see: > > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > > > "While the FC over IP specification is independent of the link level > > transport protocol, it assumes a high bandwidth, high > > reliability, low loss > > link level transport such as Gigabit Ethernet, SONET, ATM, > or DWDM. This > > specification treats all classes of FC frames the same -- > as datagrams." > > ipfc > > > > "Fibre Channel over IP's main advantage is it needs no > > modifications to the > > storage subsystem or the server operating system. SCSI over > TCP/IP, on the > > other hand, requires filter drivers so the server operating > system can do > > IP-network-to-SAN emulation (for which I humbly propose the acronym > > INSANE)." Wayne Rickard > > Network World, 06/12/00 wayne@gadzoox.com > > > > To see who is doing Fibre-Channel encapsulation see: > > > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_0 > 61300b.htm > > > > At keeping the same architecture as the present Fibre-Channel > > configuration, > > you have not improved reliability. By making compromises > that combine > > states, you are reducing reliability. It would appear, in > efforts to > > reduce > > overhead in TCP transport, you have added dangerous > overhead at the SCSI > > emulation layer. This is taking advantage of TCP in > hardware? Allow a > > session per LUN, (remove the LUN field) then there is no danger > > of inducing > > an error across multiple devices when sorting millions of > tags for a LUN > > value. Care about reliability. > > > > Doug > > > > > Doug, > > > > > > You can use the standard at the end user device, if you so wish. > > > If you choose a "transprent" controller - i.e. one not holding > > > any state or > > > ordering - and > > > view every LU as and independent unit - you ignore the > numbering (that > > you > > > are free to do anyway for the commands) and ship the commands to > > > the device > > > for execution. > > > > > > Initiator tags are unique per initiator and it should not be a > > problem to > > > keep the unique > > > per session. > > > > > > Target task tags are a different issue. We decided to > remove the LUN > > field > > > in both the incoming RTT and the the outgoing data to > avoid the need to > > > test for inconsistencies. > > > But as this is the only thing that implies unique target task > > > tags we might > > > as well "reinstate" the LUN (the field is still unused) > in the DATA > > packet > > > (in the RTT it is implied by the initiator > > > task tag). > > > > > > Relaxing the requirement of uniqueness for initiator tag > is possible too > > > (you can have the unique per LU) but that might complicate > > > unecessarily the > > > initiator. > > > > > > If this (minor) change does not anoy target controller > implementors and > > > nobody sees something else wrong with it we might > introduce it in the > > next > > > version. > > > > > > As for your FC claims - I still can't follow your argument. Are > > > you talking > > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > > > Julo > > > > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Julo, > > > > > > Keep the interface suitable for the end device. A controller in > > > the middle > > > does not justify a unique interface for that scenario. > In trying to > > > optimize "Bandwidth utilization", you are binding > thousands of logical > > > units > > > together into the same session. This is an added burden > within the > > > standard, if used at the device. In doing so, session > tags must be kept > > > unique across all logical units. Reset all drives to recover > > from even a > > > single tag error? If you wish to cache information at > the gateway, it > > > should still use the interface suitable for the end > device. At least > > this > > > isolates error handling. > > > > > > By creating a simple FC tunnel, the controller remains > bolted into the > > > back-plane where it is known to work effectively. Error > handling is > > > understood with a stateless translation and you have not created > > > a standard > > > that will set back efforts at getting a solution without myopic > > > compromises. > > > There WILL be errors not seen by this "In the Middle Controller" > > > translating > > > Fibre-Channel into iSCSI as FC does not have the same > transport. States > > > within this ITMC adds to complexity and error migration when you > > > intertwine > > > logical units. ITMC is a poor starting point and only > makes for a bad > > > interface as seen by this specification. > > > > > > Doug > > > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Saturday, August 12, 2000 12:24 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU > and I did even > > > > implement it (as many others - see Paul's reply). You > can aggregate at > > > the > > > > TCP level but you have to aggregate somewhere > > > > to use effectively the bandwidth. Eliminating the CU is > not a concern > > of > > > > this group and not a suggestion to be accepted lightly by the > > community > > > > (who will do caching, storage virtualization and management > > etc.) and I > > > am > > > > confident most of the participants on this list do not > want to discuss > > > > this subject. FC over IP as replacement for iSCSI is a poor > > > suggestion - > > > > FC is a networking protocol and IP too. FCP over IP is a gateway > > > solution. > > > > > > > > Julo > > > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > An architecture that scales does not need to provide the entire > > > bandwidth > > > > from a single device. If these devices are each handling 25 > > Mbytes per > > > > second, then 2 of them exceed Fibre-Channel. Next week > you could use > > 4, > > > > and > > > > the week after that 8. The advantage of using a > network without any > > > > practical architectural restriction with intelligence > at the client. > > On > > > > the > > > > other hand, you want to design a single mid-point gateway to > > handle the > > > > entire bandwidth. To what end? You argue it can not process a > > > > TCP session > > > > per device. You will not be tracking mid-stream errors as > > this is more > > > > costly. You add to the burden of tracking the state of > the individual > > > > device with this mid-stream state machine requiring > additional sorting > > > due > > > > to this merged protocol without taking advantage of TCP > to aid this > > > > process. > > > > For your type of solution, a simple hardware based > tunnel would be > > > better. > > > > Do not include handling of the encapsulated protocol > and at least it > > > > becomes > > > > understandable and more likely to manage the task. In other > > > > words, make it > > > > Fibre-Channel over IP and you have a chance with your > architecture. > > > > > > > > The point to my statements was to indicate the device is able > > to handle > > > an > > > > IP interface today using Fast Ethernet as the rate from > the drive is > > > > relatively low. Not to warm the cockles of marketing > pointing to > > > latency, > > > > but at least they sell more drives overcoming this problem > > using scale. > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > > > I am not sure that I agree with your architecture statements > > > but I like > > > > to > > > > > play with numbers (as most of the fellows engineers on this > > > > list probably > > > > > do). What would be in your opinion > > > > > reasonable requirements for command and data transfer > rates for the > > > next > > > > > 3-7 years? > > > > > > > > > > I would like to decouple that discussion from > architecture - data > > > > > rates can > > > > > scale even in a shared > > > > > architecture as mainframe channels have shown for years. > > > > > > > > > > I would rather like to understand if we can meet the > data rates with > > > > > reasonable latency. > > > > > > > > > > Julo > > > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > To: "Stephen Byan" , > ips@ece.cmu.edu > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data > on the outside > > > > > cylinders. Improvement of the mechanics comes at a high price > > > > > with respect > > > > > to power and cost. The cost/volume trend takes us to a single > > > > disk which > > > > > increases access time as read channel data rate increases. > > > By offering > > > > > scaled throughput using more drives where each > drive's interface > > > > bandwidth > > > > > is restricted with respect to read channel data rates > > > provides a system > > > > > with > > > > > uniform and superior performance. The advantage of such an > > > approach is > > > > > found with respect to smaller random traffic. With > more devices, > > > > > redundancy > > > > > is easily achieved and parallel access offers a means > of performance > > > > > improvement by spreading activity over more devices. The > > > > switch provides > > > > > bandwidth aggregation and is not found in the > individual device. > > > > > > > > > > An 8ms access + latency figure in the high cost > drives restricts > > > > > the number > > > > > of 'independent' operations that average 64k byte to 100 per > > > > second or 52 > > > > > Mbit per second. Such an architecture of 'restricted' drives > > > > would scale > > > > > whereas the solicitated burst approach does not. An > > > > independent nexus at > > > > > the LUN is the only design that offers required scaling and > > > > configuration > > > > > flexibility. Keeping up with the read channel is a > wasted effort. > > In > > > > > time, > > > > > 1 Gbit Ethernet will be the practical solution about > the time drives > > > are > > > > 1 > > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > > Gbit Ethernet > > > > > client makes sense in cost, performance, capacity, > reliability, and > > > > > scalability at this point in time. The protocol > overhead should be > > > > > addressed. There are substantial improvements to be made to > > > allow this > > > > > innovation using standard adapters. > > > > > > > > > > The power cost to use copper 1 Gbit is high. > Firewire does not > > > > scale and > > > > > has a limited reach. Firewire also places > scatter/gather on the > > drive > > > > > together with direct access. Doing such over a WAN > will impose > > > > > significant > > > > > changes. Serial ATA is nothing more than IDE through > a SERDES. > > > > The read > > > > > channel data rate is like a drug, just say no. It is hard > > not to buy > > > > > enough > > > > > dram to allow a proper buffer these days. Serial ATA > removes all > > > > buffers. > > > > > Intel is just usurping any remaining electronics at > the cost of > > > > > sensitivity > > > > > to a near by cell phone. Fewer drives with less > > electronics. What a > > > > good > > > > > idea? > > > > > > > > > > Doug > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Stephen Byan > > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > > To: 'ips@ece.cmu.edu' > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > > > to be 200 > > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > > midrange storage > > > > > > market without link aggregation for a single LUN. These > > markets can > > > > > afford > > > > > > 10 Gb/s links. > > > > > > > > > > > > As a disk device level interface, iSCSI will not > succeed unless > > > > > > it offers at > > > > > > least 2 Gb/s by around 2002, at very low cost for the link. > > > Note that > > > > > even > > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look > at the Serial > > ATA > > > > > speed > > > > > > roadmap on slide 16 of Intel's Serial ATA > presentation at WinHEC: > > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > > viewpoint, the disk > > > > > > industry (both suppliers and customers) has long held the > > view that > > > > > > interface speeds need to match the media data rate. > iSCSI can try > > > > > > to make an > > > > > > argument that slower speeds are technically > adequate, but this > > > > > > will increase > > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > > > implemented > > > > > > > in a disk, there'll be wild dancing in the streets and > > > > you'll all (or > > > > > > > maybe your companies will) be rich beyond the > dreams of avarice. > > > > > > > > > > > > > > The easier you can make it for the implementors, the more > > > likely it > > > > > > > will succeed. > > > > > > > > > > > > Disk drive companies have implemented much more complex > > > > interfaces than > > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. > And multiple > > TCP > > > > > > connections don't look very hard to implement. They > just look like > > a > > > > > wart. > > > > > > But I think a necessary one. > > > > > > > > > > > > Regards, > > > > > > -Steve > > > > > > > > > > > > Steve Byan > > > > > > > > > > > > Design Engineer > > > > > > MS 1-3/E23 > > > > > > 333 South Street > > > > > > Shrewsbury, MA 01545 > > > > > > (508)770-3414 > > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Fri Aug 18 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30430; 17 Aug 2000 23:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA22861 for ips-outgoing; Thu, 17 Aug 2000 22:26:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA22857 for ; Thu, 17 Aug 2000 22:26:22 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id TAA04005; Thu, 17 Aug 2000 19:20:57 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id TAA22213; Thu, 17 Aug 2000 19:18:26 -0700 From: "Douglas Otis" To: , Subject: RE: RE: Towards Consensus on TCP Connections Date: Thu, 17 Aug 2000 19:28:24 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, iSCSI<->FCP is simple if iSCSI is implemented as FC over SCTP/IP. Notice that I did not indicate TCP as a choice. The translation between iSCSI becomes defined by years of work already in place and together with a frame aligned message, it is easily converted to a hardware state machine for implementation. This alignment simplifies and improves performance for both software and hardware implementations as well. The differences between encapsulation and the present iSCSI are too few to get worked up about or to take a Not Invented Here attitude toward. I can see intelligent caches doing IU requests to accelerate write performance as one improvement eliminating even this slight justification. The iSCSI spec as written is seriously flawed as I have pointed out to Julo. Implementation will reveal many of more such flaws. If your remotest goal is to convert to FCP (Type 8), then encapsulation is the safest and most effective means. Unless you are willing to get closer to the device and completely forego a Fibre-Channel conversion, then encapsulation is the best choice. Encapsulation is also going to be the first implemented in real products. Pinch your nose, Fibre-Channel encapsulation is iSCSI. Encapsulation is the standard for Fibre-Channel compatibility. I can see no reason for two such standards. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > somesh_gupta@hp.com > Sent: Thursday, August 17, 2000 5:41 PM > To: dotis@sanlight.net; ips@ece.cmu.edu; julian_satran@il.ibm.com > Subject: RE: RE: Towards Consensus on TCP Connections > > > Doug, > > Tunneling FC through IP to connect FC islands together > is a good application and has its place, and an iSCSI <-> FCP > gateway will be an important app to connect the two worlds > together. However neither of these is a replacement for > using TCP/IP as a native transport for SCSI commands. > > FC networks appear to me to be limited scale networks providing > a very reliable link (channel) using b-b credits to avoid > dropping packets. Therefore any protocol using FC network makes > those assumptions. However this limits the scale, and also other > factors are driving the use of IP based networks. When FC goes over > an IP based networks, then either the assumptions (of a reliable > channel) are no longer true, or a convergence > layer has to be built to provide the same level of service (using > TCP and another sub-layer on top perhaps). > > Even in FC, you have SCSI states etc and FC states > (fairly involved protocol and does not have all the capabilities > as IP based networks provide - or at not yet at the same level > of interoperability and robustness). > > You have made a comment multiple times about the compexlity of > SCSI over TCP and complexity of states and demuxing etc. I somehow > don't get it (when I compare it to the fact that systems have been > managing loads of apps running on top of TCP and that switches and > NICs are implementing TCP in silicon). Perhaps you can give some > details of the state and demux complexity and error recovery > issues and we can explore them to see if there is something we are > missing or perhaps some of us can help with explaining why they > may not be issues. > > SCSI over TCP is probably more complex than FCP/FC. However, it should > provide more capabilities and a converged network. It should > also be simpler than encapsulating FC over IP (TCP). Also there > is far more expertise in this area and it is using existing > components except the iSCSI encapsulation which is fairly simple. I > don't think there is any unknowns in this area. So let us explore > and complexity issues in more detail. > > Somesh > > > -----Original Message----- > > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > > Sent: Wednesday, August 16, 2000 8:04 AM > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: FW: RE: Towards Consensus on TCP Connections > > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Tuesday, August 15, 2000 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Doug, > > > > > > Please cool down and keep the discussion technical. I > > > personally do not enjoy bringing press quotes to a technical > > > discussion. And so much more when they are inaccurate. > > > > You indicated confusion as to the comparison being made. I tried to > > enlighten you as to the alternative. I did it with humor and > > with technical > > details. I see nothing inaccurate nor have you indicated errors. > > > > > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > > > that FCP will be encapsulated on IP. FCP is tightly tied to FC > > > (as TCP is to IP) and has already a complex (and not yet closed) > > > recovery mechanism based on the assumption it runs over FC. > > > > An IP datagram is nothing more than a vehicle for FC encapsulation and > > indeed includes only the defragmentation process. There is no need to > > manage the defrag process or to be concerned about possible > > failure at the > > IP level. It is transparent and self healing as FCP-2 on FC > > handles these > > errors and would be no less reliable in doing so. There are > > few uses for FC > > beyond SCSI so your point escapes me especially when you are > > advocating a > > protocol wholly unsupported. Are you suggesting FC over IP > > will not work? > > > > > Moreover - like SCSI - FCP has the notion of a target distinct > > > from the notion of a LUN. It could be an acceptable solution to > > > extend the range of a FC island but even for this it has to go > > > some way in solving the congestion and security issues. > > > > There are already provisions for security for FC over IP. In many > > applications, bandwidth is controlled by a dedicated channel > > which solves > > both issues. There are many means for flow control where TCP > > is but one. > > Reliability should be the differentiating factor. > > > > > IMHO even for this application a FCP-iSCSI gateway could > > provide a better > > > solution as the mapping is straightforward. > > > > > > Julo > > > > This provides few techincal details as to how you base this > > opinion. If I > > had the choice of purchasing a blade for a switch or new > > system adapters > > with high overhead and new SAN controllers with limited > > connectivity where I > > could also expect massive faults, I would not see that as > > straightforward. > > But then again, maybe I would. > > > > Doug > > > > > > > "Douglas Otis" on 15/08/2000 19:24:34 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Julo, > > > > > > For details on proposed Fibre-Channel over IP which does not impose > > > additional catastrophic error states as does iSCSI, please see: > > > > > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > > > > > "While the FC over IP specification is independent of the link level > > > transport protocol, it assumes a high bandwidth, high > > > reliability, low loss > > > link level transport such as Gigabit Ethernet, SONET, ATM, > > or DWDM. This > > > specification treats all classes of FC frames the same -- > > as datagrams." > > > ipfc > > > > > > "Fibre Channel over IP's main advantage is it needs no > > > modifications to the > > > storage subsystem or the server operating system. SCSI over > > TCP/IP, on the > > > other hand, requires filter drivers so the server operating > > system can do > > > IP-network-to-SAN emulation (for which I humbly propose the acronym > > > INSANE)." Wayne Rickard > > > Network World, 06/12/00 wayne@gadzoox.com > > > > > > To see who is doing Fibre-Channel encapsulation see: > > > > > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_0 > > 61300b.htm > > > > > > At keeping the same architecture as the present Fibre-Channel > > > configuration, > > > you have not improved reliability. By making compromises > > that combine > > > states, you are reducing reliability. It would appear, in > > efforts to > > > reduce > > > overhead in TCP transport, you have added dangerous > > overhead at the SCSI > > > emulation layer. This is taking advantage of TCP in > > hardware? Allow a > > > session per LUN, (remove the LUN field) then there is no danger > > > of inducing > > > an error across multiple devices when sorting millions of > > tags for a LUN > > > value. Care about reliability. > > > > > > Doug > > > > > > > Doug, > > > > > > > > You can use the standard at the end user device, if you so wish. > > > > If you choose a "transprent" controller - i.e. one not holding > > > > any state or > > > > ordering - and > > > > view every LU as and independent unit - you ignore the > > numbering (that > > > you > > > > are free to do anyway for the commands) and ship the commands to > > > > the device > > > > for execution. > > > > > > > > Initiator tags are unique per initiator and it should not be a > > > problem to > > > > keep the unique > > > > per session. > > > > > > > > Target task tags are a different issue. We decided to > > remove the LUN > > > field > > > > in both the incoming RTT and the the outgoing data to > > avoid the need to > > > > test for inconsistencies. > > > > But as this is the only thing that implies unique target task > > > > tags we might > > > > as well "reinstate" the LUN (the field is still unused) > > in the DATA > > > packet > > > > (in the RTT it is implied by the initiator > > > > task tag). > > > > > > > > Relaxing the requirement of uniqueness for initiator tag > > is possible too > > > > (you can have the unique per LU) but that might complicate > > > > unecessarily the > > > > initiator. > > > > > > > > If this (minor) change does not anoy target controller > > implementors and > > > > nobody sees something else wrong with it we might > > introduce it in the > > > next > > > > version. > > > > > > > > As for your FC claims - I still can't follow your argument. Are > > > > you talking > > > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > Keep the interface suitable for the end device. A controller in > > > > the middle > > > > does not justify a unique interface for that scenario. > > In trying to > > > > optimize "Bandwidth utilization", you are binding > > thousands of logical > > > > units > > > > together into the same session. This is an added burden > > within the > > > > standard, if used at the device. In doing so, session > > tags must be kept > > > > unique across all logical units. Reset all drives to recover > > > from even a > > > > single tag error? If you wish to cache information at > > the gateway, it > > > > should still use the interface suitable for the end > > device. At least > > > this > > > > isolates error handling. > > > > > > > > By creating a simple FC tunnel, the controller remains > > bolted into the > > > > back-plane where it is known to work effectively. Error > > handling is > > > > understood with a stateless translation and you have not created > > > > a standard > > > > that will set back efforts at getting a solution without myopic > > > > compromises. > > > > There WILL be errors not seen by this "In the Middle Controller" > > > > translating > > > > Fibre-Channel into iSCSI as FC does not have the same > > transport. States > > > > within this ITMC adds to complexity and error migration when you > > > > intertwine > > > > logical units. ITMC is a poor starting point and only > > makes for a bad > > > > interface as seen by this specification. > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Saturday, August 12, 2000 12:24 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU > > and I did even > > > > > implement it (as many others - see Paul's reply). You > > can aggregate at > > > > the > > > > > TCP level but you have to aggregate somewhere > > > > > to use effectively the bandwidth. Eliminating the CU is > > not a concern > > > of > > > > > this group and not a suggestion to be accepted lightly by the > > > community > > > > > (who will do caching, storage virtualization and management > > > etc.) and I > > > > am > > > > > confident most of the participants on this list do not > > want to discuss > > > > > this subject. FC over IP as replacement for iSCSI is a poor > > > > suggestion - > > > > > FC is a networking protocol and IP too. FCP over IP is a gateway > > > > solution. > > > > > > > > > > Julo > > > > > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > > cc: > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > > > An architecture that scales does not need to provide the entire > > > > bandwidth > > > > > from a single device. If these devices are each handling 25 > > > Mbytes per > > > > > second, then 2 of them exceed Fibre-Channel. Next week > > you could use > > > 4, > > > > > and > > > > > the week after that 8. The advantage of using a > > network without any > > > > > practical architectural restriction with intelligence > > at the client. > > > On > > > > > the > > > > > other hand, you want to design a single mid-point gateway to > > > handle the > > > > > entire bandwidth. To what end? You argue it can not process a > > > > > TCP session > > > > > per device. You will not be tracking mid-stream errors as > > > this is more > > > > > costly. You add to the burden of tracking the state of > > the individual > > > > > device with this mid-stream state machine requiring > > additional sorting > > > > due > > > > > to this merged protocol without taking advantage of TCP > > to aid this > > > > > process. > > > > > For your type of solution, a simple hardware based > > tunnel would be > > > > better. > > > > > Do not include handling of the encapsulated protocol > > and at least it > > > > > becomes > > > > > understandable and more likely to manage the task. In other > > > > > words, make it > > > > > Fibre-Channel over IP and you have a chance with your > > architecture. > > > > > > > > > > The point to my statements was to indicate the device is able > > > to handle > > > > an > > > > > IP interface today using Fast Ethernet as the rate from > > the drive is > > > > > relatively low. Not to warm the cockles of marketing > > pointing to > > > > latency, > > > > > but at least they sell more drives overcoming this problem > > > using scale. > > > > > > > > > > Doug > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > julian_satran@il.ibm.com > > > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > > > > > I am not sure that I agree with your architecture statements > > > > but I like > > > > > to > > > > > > play with numbers (as most of the fellows engineers on this > > > > > list probably > > > > > > do). What would be in your opinion > > > > > > reasonable requirements for command and data transfer > > rates for the > > > > next > > > > > > 3-7 years? > > > > > > > > > > > > I would like to decouple that discussion from > > architecture - data > > > > > > rates can > > > > > > scale even in a shared > > > > > > architecture as mainframe channels have shown for years. > > > > > > > > > > > > I would rather like to understand if we can meet the > > data rates with > > > > > > reasonable latency. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > > > To: "Stephen Byan" , > > ips@ece.cmu.edu > > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data > > on the outside > > > > > > cylinders. Improvement of the mechanics comes at a high price > > > > > > with respect > > > > > > to power and cost. The cost/volume trend takes us to a single > > > > > disk which > > > > > > increases access time as read channel data rate increases. > > > > By offering > > > > > > scaled throughput using more drives where each > > drive's interface > > > > > bandwidth > > > > > > is restricted with respect to read channel data rates > > > > provides a system > > > > > > with > > > > > > uniform and superior performance. The advantage of such an > > > > approach is > > > > > > found with respect to smaller random traffic. With > > more devices, > > > > > > redundancy > > > > > > is easily achieved and parallel access offers a means > > of performance > > > > > > improvement by spreading activity over more devices. The > > > > > switch provides > > > > > > bandwidth aggregation and is not found in the > > individual device. > > > > > > > > > > > > An 8ms access + latency figure in the high cost > > drives restricts > > > > > > the number > > > > > > of 'independent' operations that average 64k byte to 100 per > > > > > second or 52 > > > > > > Mbit per second. Such an architecture of 'restricted' drives > > > > > would scale > > > > > > whereas the solicitated burst approach does not. An > > > > > independent nexus at > > > > > > the LUN is the only design that offers required scaling and > > > > > configuration > > > > > > flexibility. Keeping up with the read channel is a > > wasted effort. > > > In > > > > > > time, > > > > > > 1 Gbit Ethernet will be the practical solution about > > the time drives > > > > are > > > > > 1 > > > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > > > Gbit Ethernet > > > > > > client makes sense in cost, performance, capacity, > > reliability, and > > > > > > scalability at this point in time. The protocol > > overhead should be > > > > > > addressed. There are substantial improvements to be made to > > > > allow this > > > > > > innovation using standard adapters. > > > > > > > > > > > > The power cost to use copper 1 Gbit is high. > > Firewire does not > > > > > scale and > > > > > > has a limited reach. Firewire also places > > scatter/gather on the > > > drive > > > > > > together with direct access. Doing such over a WAN > > will impose > > > > > > significant > > > > > > changes. Serial ATA is nothing more than IDE through > > a SERDES. > > > > > The read > > > > > > channel data rate is like a drug, just say no. It is hard > > > not to buy > > > > > > enough > > > > > > dram to allow a proper buffer these days. Serial ATA > > removes all > > > > > buffers. > > > > > > Intel is just usurping any remaining electronics at > > the cost of > > > > > > sensitivity > > > > > > to a near by cell phone. Fewer drives with less > > > electronics. What a > > > > > good > > > > > > idea? > > > > > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > Of > > > > > > > Stephen Byan > > > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > > > To: 'ips@ece.cmu.edu' > > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > > > > to be 200 > > > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > > > midrange storage > > > > > > > market without link aggregation for a single LUN. These > > > markets can > > > > > > afford > > > > > > > 10 Gb/s links. > > > > > > > > > > > > > > As a disk device level interface, iSCSI will not > > succeed unless > > > > > > > it offers at > > > > > > > least 2 Gb/s by around 2002, at very low cost for the link. > > > > Note that > > > > > > even > > > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look > > at the Serial > > > ATA > > > > > > speed > > > > > > > roadmap on slide 16 of Intel's Serial ATA > > presentation at WinHEC: > > > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > > > viewpoint, the disk > > > > > > > industry (both suppliers and customers) has long held the > > > view that > > > > > > > interface speeds need to match the media data rate. > > iSCSI can try > > > > > > > to make an > > > > > > > argument that slower speeds are technically > > adequate, but this > > > > > > > will increase > > > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > > > > implemented > > > > > > > > in a disk, there'll be wild dancing in the streets and > > > > > you'll all (or > > > > > > > > maybe your companies will) be rich beyond the > > dreams of avarice. > > > > > > > > > > > > > > > > The easier you can make it for the implementors, the more > > > > likely it > > > > > > > > will succeed. > > > > > > > > > > > > > > Disk drive companies have implemented much more complex > > > > > interfaces than > > > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. > > And multiple > > > TCP > > > > > > > connections don't look very hard to implement. They > > just look like > > > a > > > > > > wart. > > > > > > > But I think a necessary one. > > > > > > > > > > > > > > Regards, > > > > > > > -Steve > > > > > > > > > > > > > > Steve Byan > > > > > > > > > > > > > > Design Engineer > > > > > > > MS 1-3/E23 > > > > > > > 333 South Street > > > > > > > Shrewsbury, MA 01545 > > > > > > > (508)770-3414 > > > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Fri Aug 18 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31017; 18 Aug 2000 1:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA25256 for ips-outgoing; Fri, 18 Aug 2000 00:06:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA25244 for ; Fri, 18 Aug 2000 00:06:42 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id VAA00444; Thu, 17 Aug 2000 21:06:36 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Thu, 17 Aug 2000 21:06:34 -0700 Message-ID: From: Jim McGrath To: "'somesh_gupta@hp.com'" , dotis@sanlight.net, ips@ece.cmu.edu, julian_satran@il.ibm.com Subject: RE: RE: Towards Consensus on TCP Connections Date: Thu, 17 Aug 2000 21:06:33 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Observation spanning many emails: I think part of the problem here is that people have some different experiences. For instance, you mentioned silicon based TCP/IP implementations, and yet I personally (and I think a number of other storage focused people) are not familiar with them and their characteristics (e.g. availability, cost, etc...). Storage vendors have had to automate a lot of protocol in silicon for years, and this had had a pronounced influence on the evolution of that protocol. For instance, in both SCSI and Fibre Channel standards activities you have a lot of discussion as to whether something can be easily and inexpensively implemented in silicon - this is often a consideration that weighs heavily in the decision making. This is due to the combination of lost cost and high performance focus of the industry. Indeed, this focus on "simple" protocols (relying of good links, etc...) is I think a reason why some folks in the networking community have doubts concerning issues like congestion control for storage protocols or anything based on them. The networking industry is, as seem from a storage vendors perspective, almost the exact opposite. Its history has been dominated by software implementations of (comparatively) slow transports on (comparatively) expensive machines (normally computers of some sort). I think you are seeing some doubts being expressed about the speed and cost of these approaches, and whether they can be transplanted to the storage world (which is the focus for iSCSI). Personally, I'd like to hear most on cheap, fast, TCP/IP implementations. Afterall, if it really was cheap and fast to send FC out one end on top of TCP/IP and get it back on the other end, then I for one would certainly support that approach given its other benefits. Jim -----Original Message----- From: somesh_gupta@hp.com [mailto:somesh_gupta@hp.com] Sent: Thursday, August 17, 2000 5:41 PM To: dotis@sanlight.net; ips@ece.cmu.edu; julian_satran@il.ibm.com Subject: RE: RE: Towards Consensus on TCP Connections Doug, Tunneling FC through IP to connect FC islands together is a good application and has its place, and an iSCSI <-> FCP gateway will be an important app to connect the two worlds together. However neither of these is a replacement for using TCP/IP as a native transport for SCSI commands. FC networks appear to me to be limited scale networks providing a very reliable link (channel) using b-b credits to avoid dropping packets. Therefore any protocol using FC network makes those assumptions. However this limits the scale, and also other factors are driving the use of IP based networks. When FC goes over an IP based networks, then either the assumptions (of a reliable channel) are no longer true, or a convergence layer has to be built to provide the same level of service (using TCP and another sub-layer on top perhaps). Even in FC, you have SCSI states etc and FC states (fairly involved protocol and does not have all the capabilities as IP based networks provide - or at not yet at the same level of interoperability and robustness). You have made a comment multiple times about the compexlity of SCSI over TCP and complexity of states and demuxing etc. I somehow don't get it (when I compare it to the fact that systems have been managing loads of apps running on top of TCP and that switches and NICs are implementing TCP in silicon). Perhaps you can give some details of the state and demux complexity and error recovery issues and we can explore them to see if there is something we are missing or perhaps some of us can help with explaining why they may not be issues. SCSI over TCP is probably more complex than FCP/FC. However, it should provide more capabilities and a converged network. It should also be simpler than encapsulating FC over IP (TCP). Also there is far more expertise in this area and it is using existing components except the iSCSI encapsulation which is fairly simple. I don't think there is any unknowns in this area. So let us explore and complexity issues in more detail. Somesh > -----Original Message----- > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > Sent: Wednesday, August 16, 2000 8:04 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: FW: RE: Towards Consensus on TCP Connections > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Tuesday, August 15, 2000 10:06 PM > > To: ips@ece.cmu.edu > > Subject: RE: Towards Consensus on TCP Connections > > > > Doug, > > > > Please cool down and keep the discussion technical. I > > personally do not enjoy bringing press quotes to a technical > > discussion. And so much more when they are inaccurate. > > You indicated confusion as to the comparison being made. I tried to > enlighten you as to the alternative. I did it with humor and > with technical > details. I see nothing inaccurate nor have you indicated errors. > > > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > > that FCP will be encapsulated on IP. FCP is tightly tied to FC > > (as TCP is to IP) and has already a complex (and not yet closed) > > recovery mechanism based on the assumption it runs over FC. > > An IP datagram is nothing more than a vehicle for FC encapsulation and > indeed includes only the defragmentation process. There is no need to > manage the defrag process or to be concerned about possible > failure at the > IP level. It is transparent and self healing as FCP-2 on FC > handles these > errors and would be no less reliable in doing so. There are > few uses for FC > beyond SCSI so your point escapes me especially when you are > advocating a > protocol wholly unsupported. Are you suggesting FC over IP > will not work? > > > Moreover - like SCSI - FCP has the notion of a target distinct > > from the notion of a LUN. It could be an acceptable solution to > > extend the range of a FC island but even for this it has to go > > some way in solving the congestion and security issues. > > There are already provisions for security for FC over IP. In many > applications, bandwidth is controlled by a dedicated channel > which solves > both issues. There are many means for flow control where TCP > is but one. > Reliability should be the differentiating factor. > > > IMHO even for this application a FCP-iSCSI gateway could > provide a better > > solution as the mapping is straightforward. > > > > Julo > > This provides few techincal details as to how you base this > opinion. If I > had the choice of purchasing a blade for a switch or new > system adapters > with high overhead and new SAN controllers with limited > connectivity where I > could also expect massive faults, I would not see that as > straightforward. > But then again, maybe I would. > > Doug > > > > "Douglas Otis" on 15/08/2000 19:24:34 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Towards Consensus on TCP Connections > > > > Julo, > > > > For details on proposed Fibre-Channel over IP which does not impose > > additional catastrophic error states as does iSCSI, please see: > > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > > > "While the FC over IP specification is independent of the link level > > transport protocol, it assumes a high bandwidth, high > > reliability, low loss > > link level transport such as Gigabit Ethernet, SONET, ATM, > or DWDM. This > > specification treats all classes of FC frames the same -- > as datagrams." > > ipfc > > > > "Fibre Channel over IP's main advantage is it needs no > > modifications to the > > storage subsystem or the server operating system. SCSI over > TCP/IP, on the > > other hand, requires filter drivers so the server operating > system can do > > IP-network-to-SAN emulation (for which I humbly propose the acronym > > INSANE)." Wayne Rickard > > Network World, 06/12/00 wayne@gadzoox.com > > > > To see who is doing Fibre-Channel encapsulation see: > > > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_0 > 61300b.htm > > > > At keeping the same architecture as the present Fibre-Channel > > configuration, > > you have not improved reliability. By making compromises > that combine > > states, you are reducing reliability. It would appear, in > efforts to > > reduce > > overhead in TCP transport, you have added dangerous > overhead at the SCSI > > emulation layer. This is taking advantage of TCP in > hardware? Allow a > > session per LUN, (remove the LUN field) then there is no danger > > of inducing > > an error across multiple devices when sorting millions of > tags for a LUN > > value. Care about reliability. > > > > Doug > > > > > Doug, > > > > > > You can use the standard at the end user device, if you so wish. > > > If you choose a "transprent" controller - i.e. one not holding > > > any state or > > > ordering - and > > > view every LU as and independent unit - you ignore the > numbering (that > > you > > > are free to do anyway for the commands) and ship the commands to > > > the device > > > for execution. > > > > > > Initiator tags are unique per initiator and it should not be a > > problem to > > > keep the unique > > > per session. > > > > > > Target task tags are a different issue. We decided to > remove the LUN > > field > > > in both the incoming RTT and the the outgoing data to > avoid the need to > > > test for inconsistencies. > > > But as this is the only thing that implies unique target task > > > tags we might > > > as well "reinstate" the LUN (the field is still unused) > in the DATA > > packet > > > (in the RTT it is implied by the initiator > > > task tag). > > > > > > Relaxing the requirement of uniqueness for initiator tag > is possible too > > > (you can have the unique per LU) but that might complicate > > > unecessarily the > > > initiator. > > > > > > If this (minor) change does not anoy target controller > implementors and > > > nobody sees something else wrong with it we might > introduce it in the > > next > > > version. > > > > > > As for your FC claims - I still can't follow your argument. Are > > > you talking > > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > > > Julo > > > > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Julo, > > > > > > Keep the interface suitable for the end device. A controller in > > > the middle > > > does not justify a unique interface for that scenario. > In trying to > > > optimize "Bandwidth utilization", you are binding > thousands of logical > > > units > > > together into the same session. This is an added burden > within the > > > standard, if used at the device. In doing so, session > tags must be kept > > > unique across all logical units. Reset all drives to recover > > from even a > > > single tag error? If you wish to cache information at > the gateway, it > > > should still use the interface suitable for the end > device. At least > > this > > > isolates error handling. > > > > > > By creating a simple FC tunnel, the controller remains > bolted into the > > > back-plane where it is known to work effectively. Error > handling is > > > understood with a stateless translation and you have not created > > > a standard > > > that will set back efforts at getting a solution without myopic > > > compromises. > > > There WILL be errors not seen by this "In the Middle Controller" > > > translating > > > Fibre-Channel into iSCSI as FC does not have the same > transport. States > > > within this ITMC adds to complexity and error migration when you > > > intertwine > > > logical units. ITMC is a poor starting point and only > makes for a bad > > > interface as seen by this specification. > > > > > > Doug > > > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Saturday, August 12, 2000 12:24 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU > and I did even > > > > implement it (as many others - see Paul's reply). You > can aggregate at > > > the > > > > TCP level but you have to aggregate somewhere > > > > to use effectively the bandwidth. Eliminating the CU is > not a concern > > of > > > > this group and not a suggestion to be accepted lightly by the > > community > > > > (who will do caching, storage virtualization and management > > etc.) and I > > > am > > > > confident most of the participants on this list do not > want to discuss > > > > this subject. FC over IP as replacement for iSCSI is a poor > > > suggestion - > > > > FC is a networking protocol and IP too. FCP over IP is a gateway > > > solution. > > > > > > > > Julo > > > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > An architecture that scales does not need to provide the entire > > > bandwidth > > > > from a single device. If these devices are each handling 25 > > Mbytes per > > > > second, then 2 of them exceed Fibre-Channel. Next week > you could use > > 4, > > > > and > > > > the week after that 8. The advantage of using a > network without any > > > > practical architectural restriction with intelligence > at the client. > > On > > > > the > > > > other hand, you want to design a single mid-point gateway to > > handle the > > > > entire bandwidth. To what end? You argue it can not process a > > > > TCP session > > > > per device. You will not be tracking mid-stream errors as > > this is more > > > > costly. You add to the burden of tracking the state of > the individual > > > > device with this mid-stream state machine requiring > additional sorting > > > due > > > > to this merged protocol without taking advantage of TCP > to aid this > > > > process. > > > > For your type of solution, a simple hardware based > tunnel would be > > > better. > > > > Do not include handling of the encapsulated protocol > and at least it > > > > becomes > > > > understandable and more likely to manage the task. In other > > > > words, make it > > > > Fibre-Channel over IP and you have a chance with your > architecture. > > > > > > > > The point to my statements was to indicate the device is able > > to handle > > > an > > > > IP interface today using Fast Ethernet as the rate from > the drive is > > > > relatively low. Not to warm the cockles of marketing > pointing to > > > latency, > > > > but at least they sell more drives overcoming this problem > > using scale. > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > > > I am not sure that I agree with your architecture statements > > > but I like > > > > to > > > > > play with numbers (as most of the fellows engineers on this > > > > list probably > > > > > do). What would be in your opinion > > > > > reasonable requirements for command and data transfer > rates for the > > > next > > > > > 3-7 years? > > > > > > > > > > I would like to decouple that discussion from > architecture - data > > > > > rates can > > > > > scale even in a shared > > > > > architecture as mainframe channels have shown for years. > > > > > > > > > > I would rather like to understand if we can meet the > data rates with > > > > > reasonable latency. > > > > > > > > > > Julo > > > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > To: "Stephen Byan" , > ips@ece.cmu.edu > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data > on the outside > > > > > cylinders. Improvement of the mechanics comes at a high price > > > > > with respect > > > > > to power and cost. The cost/volume trend takes us to a single > > > > disk which > > > > > increases access time as read channel data rate increases. > > > By offering > > > > > scaled throughput using more drives where each > drive's interface > > > > bandwidth > > > > > is restricted with respect to read channel data rates > > > provides a system > > > > > with > > > > > uniform and superior performance. The advantage of such an > > > approach is > > > > > found with respect to smaller random traffic. With > more devices, > > > > > redundancy > > > > > is easily achieved and parallel access offers a means > of performance > > > > > improvement by spreading activity over more devices. The > > > > switch provides > > > > > bandwidth aggregation and is not found in the > individual device. > > > > > > > > > > An 8ms access + latency figure in the high cost > drives restricts > > > > > the number > > > > > of 'independent' operations that average 64k byte to 100 per > > > > second or 52 > > > > > Mbit per second. Such an architecture of 'restricted' drives > > > > would scale > > > > > whereas the solicitated burst approach does not. An > > > > independent nexus at > > > > > the LUN is the only design that offers required scaling and > > > > configuration > > > > > flexibility. Keeping up with the read channel is a > wasted effort. > > In > > > > > time, > > > > > 1 Gbit Ethernet will be the practical solution about > the time drives > > > are > > > > 1 > > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > > Gbit Ethernet > > > > > client makes sense in cost, performance, capacity, > reliability, and > > > > > scalability at this point in time. The protocol > overhead should be > > > > > addressed. There are substantial improvements to be made to > > > allow this > > > > > innovation using standard adapters. > > > > > > > > > > The power cost to use copper 1 Gbit is high. > Firewire does not > > > > scale and > > > > > has a limited reach. Firewire also places > scatter/gather on the > > drive > > > > > together with direct access. Doing such over a WAN > will impose > > > > > significant > > > > > changes. Serial ATA is nothing more than IDE through > a SERDES. > > > > The read > > > > > channel data rate is like a drug, just say no. It is hard > > not to buy > > > > > enough > > > > > dram to allow a proper buffer these days. Serial ATA > removes all > > > > buffers. > > > > > Intel is just usurping any remaining electronics at > the cost of > > > > > sensitivity > > > > > to a near by cell phone. Fewer drives with less > > electronics. What a > > > > good > > > > > idea? > > > > > > > > > > Doug > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Stephen Byan > > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > > To: 'ips@ece.cmu.edu' > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > > > to be 200 > > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > > midrange storage > > > > > > market without link aggregation for a single LUN. These > > markets can > > > > > afford > > > > > > 10 Gb/s links. > > > > > > > > > > > > As a disk device level interface, iSCSI will not > succeed unless > > > > > > it offers at > > > > > > least 2 Gb/s by around 2002, at very low cost for the link. > > > Note that > > > > > even > > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look > at the Serial > > ATA > > > > > speed > > > > > > roadmap on slide 16 of Intel's Serial ATA > presentation at WinHEC: > > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > > viewpoint, the disk > > > > > > industry (both suppliers and customers) has long held the > > view that > > > > > > interface speeds need to match the media data rate. > iSCSI can try > > > > > > to make an > > > > > > argument that slower speeds are technically > adequate, but this > > > > > > will increase > > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > > > implemented > > > > > > > in a disk, there'll be wild dancing in the streets and > > > > you'll all (or > > > > > > > maybe your companies will) be rich beyond the > dreams of avarice. > > > > > > > > > > > > > > The easier you can make it for the implementors, the more > > > likely it > > > > > > > will succeed. > > > > > > > > > > > > Disk drive companies have implemented much more complex > > > > interfaces than > > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. > And multiple > > TCP > > > > > > connections don't look very hard to implement. They > just look like > > a > > > > > wart. > > > > > > But I think a necessary one. > > > > > > > > > > > > Regards, > > > > > > -Steve > > > > > > > > > > > > Steve Byan > > > > > > > > > > > > Design Engineer > > > > > > MS 1-3/E23 > > > > > > 333 South Street > > > > > > Shrewsbury, MA 01545 > > > > > > (508)770-3414 > > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Fri Aug 18 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31852; 18 Aug 2000 4:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA29359 for ips-outgoing; Fri, 18 Aug 2000 03:27:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA29353 for ; Fri, 18 Aug 2000 03:27:03 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA22960 for ; Fri, 18 Aug 2000 09:26:30 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id JAA24248 for ; Fri, 18 Aug 2000 09:26:30 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693F.0028E001 ; Fri, 18 Aug 2000 09:26:27 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 18 Aug 2000 10:26:19 +0300 Subject: iSCSI single control channel Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a repost of my proposal (from the end of June) for a single control channel with multiple data channels, as per Julian's request that I repost this memo. It simplifies the current iSCSI specification proposal in that it does away with the reference numbers and simplifies recovery. - Kalman ------------------------------------------------------------------------------------------------------------------- Proposal to support single Control Channel with multiple Data Channels in the iSCSI protocol. by Kalman Meth 27 June 2000 In our discussions on the iSCSI protocol, we came to the conclusion that we needed to send data over multiple channels in order to make best use of the available network resources. We also were inclined to have all of the channels acting in a symmetric manner so as to simplify the protocol by not having to deal differently with some channels. This allows vendors to introduce uniform iSCSI NICs for all of the network connections that will be exploited by iSCSI. We decided on allowing commands to be sent over any of the multiple connections, with the command's data and status being sent in the same channel that was used to issue the command. The use of multiple channels to pass commands introduced a complication of servicing the commands on the receiving end in the original order that the commands were issued. We had a further complication when one of the connections failed; how do we determine which command got lost on a broken connection, and what actions are required to recover from the failed connection. The solution we found to these problems (introducing a Command Reference Number and placing the commands back in order on the receiver's end) introduced flow control problems, such as maintaining a window on commands to ensure that we don't overrun the reference count, and that we don't block up all of the channels just because one channel failed and its lost command causes us to fill up the command queue on the target (while we wait for the lost command to arrive). I would like us to go back and consider a variation of the model we originally proposed with one Command Channel and multiple Data Channels. Some ideas that came up during our discussions are included below and also apply to the symmetric model. Session establishment: as in existing draft. Naming: as in existing draft with adjustments from design discussions. security: as decided in design discussions. (0) none (1) challenge/response (2) IPSec or SSL Normal case: An iSCSI session between and initiator and a target consists of a number of TCP connections. Each TCP connection between initiator and target requires an iSCSI login. The first established connection of a session between initiator and target (numbered 0) is the Control Connection (also called Control Channel). Subsequent connections between the same initiator and target can be added to an existing session upon request of the initiator during login. These connections are numbered 1,2,3, etc, and are called Data Connections (also called Data Channels). An initiator may establish several sessions with the same target, each session having its own Control Channel and its own set of Data Channels. All SCSI commands and task management messages will go over the Control Connection. Order is maintained within a single session by virtue of all commands going through the same TCP connection. The iSCSI packets for RTT and Data may go over any of the channels. iSCSI Login must be performed on each of the connections. iSCSI Ping may be performed over any of the connections. It is recommended that large data transfers be performed on the Data Channels (rather than the Control Channel) so as to ensure that the Control Channel is always free. It is permissible, however, to establish a single connection and perform all iSCSI operations on that single channel. On a READ or WRITE command, the initiator specifies on which channel it expects to perform the data transfer. This gives the initiator and target a chance to set up buffers for DMA ahead of time. Once a data transfer for a particular SCSI command begins on a particular Data Channel, all subsequent data that is transferred for the same SCSI command is to be transferred over the same Data Channel. On RTT, the target confirms on which channel it is expecting the data transfer. An RTT request will be sent over the same channel as the expected data transfer (as was specified by the initiator). If the target decides (for whatever reason) that it wants to receive the data transfer on another channel, it sends the RTT over the Control Channel with an indication as to which Data Channel it wants to use. It is understood that this may entail a performance cost on the initiator's side to now move the data transfer to another Data Channel (which may be another NIC, thus requiring DMA to be set up all over again). A target will usually change the connection for a data transfer only in case of some problem it has with the originally specified connection (unresponsive connection, or couldn't handle large amount of data on specified connection, etc). Commands may be sent with immediate data (in the Control Channel) if the immediate data is small (say less than 8K), thereby avoiding the need to later match up the data with the corresponding command. A bit in the iSCSI command header indicates that there is immediate data. An initiator may also send unsolicited data (no RTT) over the Data Channels, in case the initiator and target have agreed (during login on the Control Channel) to not use RTT. The initiator and target may renegotiate the use (or non-use) of RTT between commands, using an iSCSI Text command. The initiator sends the request to the target and does not send any other commands to the target until the target has responded. The change in using RTT will take affect with the command following the response of the target. The status of a READ command is sent with the last data packet, thus allowing hardware implementations to perform a single interrupt when the entire data transfer has completed. Similarly, a flag in a data packet sent from initiator to target indicates the last data buffer in an unsolicited WRITE operation. If the initiator sends unsolicited data for a WRITE operation (i.e. without an RTT) over one of the Data Channels, it is possible that the data will arrive before the command arrived on the Control Channel. It is also possible that the target will not have enough buffers to receive the unsolicited data. The target has the option of placing the unsolicited data in reserve buffers or of completely discarding the data. If the target discards the data, the target will later issue an RTT to instruct the initiator to resend the data. Multiple iSCSI NICs: One argument to support the symmetric model was to allow having identical iSCSI NICs to handle all iSCSI connections. In the symmetric model, since all channels look alike, all of the (identical) NICs can be fully utilized. We argue that even in the model with one Control connection and many Data Connections that we can still utilize the NICs to their maximum. The main operations to be implemented by iSCSI NICs will be to send data packets and RTTs. Data Channels can be spread across these iSCSI NICs. The less frequent iSCSI operations (and especially recovery) can be performed in software in a device driver. Note also that a Control Channel and a Data Channel can go over the same wire (NIC) even if they are different TCP connections. In order to handle additional iSCSI operations in hardware, vendors can introduce fancier NICs that also handle some other iSCSI operations. A target may use one NIC to handle the Control channel from one initiator, and another NIC to handle the Control channel from another initiator. Thus, even if all NICs can handle the entire iSCSI set, they can still be utilized to the maximum by using each NIC for the Control Channel of a different session. Similarly, if an initiator has devices on several targets, it can use each NIC to handle the Control Connection of a different session. An initiator can also open multiple sessions with the same tartet using a different NIC for the Control Channel of the different sessions. Recovery: An initiator must hold on to data it has sent via a WRITE operation until it has received the status for the corresponding command. Even if the initiator sends immediate data (in the Control Channel) or unsolicited data (in one of the Data Channels), the target may discard the data in case it didn't have the resources to handle the data at that instant. The target may then request that the data be resent with an RTT. A target need not keep a copy of the data buffers it has sent, if such data can be regenerated from the storage device. However, the target must keep around the status information until it has been acknowledged by the initiator. The initiator sends Status Ack info (a new iSCSI message type) over the Control Channel. If strict ordering between commands is needed (such as reading and writing of the same device) then the application must perform the proper synchronization by not issuing the second command until it has received the status of the first command (as in linked commands). If it seems that a connection has stopped functioning, then either the initiator or the target may issue an iSCSI Ping command to determine if the connection is still alive. (A bit in the Ping header determines which side initiated the ping operation.) If the Ping operation times out, then it may be assumed that the connection is not functioning properly. When a Ping operation fails, the connection should immediately be closed. Note: It is not required to support iSCSI level recovery. It is sufficient for the initiator to report failure for the commands that did not complete and let the upper layer protocol handle the recovery. In this case, all channels of the session should be closed, all data structures should be cleaned up, and a new session may be established between the initiator and target. There is an advanced recovery mechanism that MAY be implemented by the initiator and target, as described below. Data that was sent over a failed Data Connection will have to be resent over another Data Connection. On a WRITE operation, the target will eventually issue an RTT over the Control Connection to inform the initiator as to which other Data Connection to use. (Is it OK to wait for the target to figure out that the connection is down? Can the initiator somehow bring this to the target's attention?) On a READ operation, the initiator will indicate to the target which data it wants resent from the failed data transfer. This is done using an RTT (sent from the initiator to the target over the Control Channel) to resend the data from some preivous READ operation. (This is the only time an RTT is sent from the initiator to a target.) (Since the status of that READ operation did not arrive at the initiator and it was never acknowledged, the target will have kept the relevant information about the corresponding command.) If the Control Channel stops working properly (agin, determined by a time out on an iSCSI Ping operation) then the initiator must know which commands made it to the target and were not lost, which commands were completed whose status got lost, and which commands never made it to the target. Upon setting up a new session (by establishing a Control Channel), the initiator may specify whether it is in fact starting a new session or taking over an existing session. When taking over an existing session, the initiator must specify the identifiers of the session to be taken over. The target then stops transmitting on the old Control Channel, and transfers all of the old session resources to the new Control Channel. The target returns to the initiator the Initiator Task Tag of the last command it received on the old Control Channel. For each command that was sent to the target before the specified Initiator Task Tag, the initiator queries (new iSCSI Query command) about the status of that command. (The information about the status of those commands will not have been discarded since the target never received an Ack about them from the initiator.) Some of the commands may have had incomplete data transfers (use special iSCSI status code), and the target and initiator will re-issue RTTs to recover the data from those commands. Once the initiator has received (and acknowledged) the status of all pending commands, the initiator sends an iSCSI Sync message to the target to inform it that they are back in sync, and that all commands before the specified Initiator Task Tag have been satisfactorily accounted for. From owner-ips@ECE.cmu.edu Fri Aug 18 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31836; 18 Aug 2000 4:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA28842 for ips-outgoing; Fri, 18 Aug 2000 03:04:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA28838 for ; Fri, 18 Aug 2000 03:04:37 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA23750 for ; Fri, 18 Aug 2000 09:03:56 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id JAA11100 for ; Fri, 18 Aug 2000 09:03:56 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693F.0026CECD ; Fri, 18 Aug 2000 09:03:53 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 18 Aug 2000 10:03:49 +0300 Subject: RE: FC/IP vs. iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug wrote: "In reality, the iSCSI proposal was to a be a modification for a Fibre-Channel controller. A complete and unaltered encapsulation is the most straightforward approach and likely with the lowest overhead and risk." This is not correct. The original iSCSI proposal was meant (at least by some of the authors) to provide a SAM-2 compliant native transport for SCSI over (specifically) TCP. The ultimate goal is to have a single network infrastructure for regular internet traffic and for storage traffic; i.e. the ultimate goal is to not need a separate Fibre-Channel infrastructure. (Sorry Fibre-Channel fans ;-).) The same management tools and off-the-shelf components can then be used for both ordinary internet infrastructure and for remote storage infrastructure. Now the IP Storage WG charter is broader than what iSCSI provides. There is no need for iSCSI to provide a universal solution to all IP storage problems. Let's develop the iSCSI protocol to do well what it was originally designed for: native SCSI over TCP. We have to keep in mind, however, that in order for iSCSI to be adopted, we will probably have to provide some support for existing storage SAN infrastructure (i.e. Fibre Channel) bridging. There is no need, however, to make that the primary focus of the iSCSI protocol. - Kalman From owner-ips@ECE.cmu.edu Fri Aug 18 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31841; 18 Aug 2000 4:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA28668 for ips-outgoing; Fri, 18 Aug 2000 02:55:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA28663 for ; Fri, 18 Aug 2000 02:55:53 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA126192 for ; Fri, 18 Aug 2000 08:55:21 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id IAA35584 for ; Fri, 18 Aug 2000 08:55:21 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693F.002604F6 ; Fri, 18 Aug 2000 08:55:16 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 18 Aug 2000 09:53:24 +0300 Subject: RE: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, Your are probably right. FC and FCP are designed mainly based on hardware considerations. And when the network started being to complex to handle it in simple state machine the protocol became complex and the solutions expensive. The IP stacks (mainly TCP) have a software history but on any of the modern adapters you will see the "main path" functions implemented in hardware - and this is true for switches routers etc. The basic TCP state machine is very simple (most implementation require a very low instruction count) and the major burden - copies - can be avoided by cleverly designing the upper layer protocol. Any "simple encapsulation" protocol will inherit the complexity of both the IP stack - as it can't rely anymore on the link layer credit functions provided by the FC hardware - and of FC - as FCP is designed on a transport that does not handle loss (FCP is both a transport and an application protocol). Nevertheless for tunneling an FC encapsulation might be a good solution as the cost/complexity ADDED is not high (the islands are FCP) and they can be interconnected by TCP - provided that a good addressing/routing mechanism can be put in place. It can be done fast and will satify a need. It is not an end-to-end IP solution. This led us to build a mapping specific for TCP (as others have built mapping specific to Serial Bus, ST etc.). Julo Jim McGrath on 18/08/2000 07:06:33 Please respond to Jim McGrath To: "'somesh_gupta@hp.com'" , dotis@sanlight.net, ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: RE: Towards Consensus on TCP Connections Observation spanning many emails: I think part of the problem here is that people have some different experiences. For instance, you mentioned silicon based TCP/IP implementations, and yet I personally (and I think a number of other storage focused people) are not familiar with them and their characteristics (e.g. availability, cost, etc...). Storage vendors have had to automate a lot of protocol in silicon for years, and this had had a pronounced influence on the evolution of that protocol. For instance, in both SCSI and Fibre Channel standards activities you have a lot of discussion as to whether something can be easily and inexpensively implemented in silicon - this is often a consideration that weighs heavily in the decision making. This is due to the combination of lost cost and high performance focus of the industry. Indeed, this focus on "simple" protocols (relying of good links, etc...) is I think a reason why some folks in the networking community have doubts concerning issues like congestion control for storage protocols or anything based on them. The networking industry is, as seem from a storage vendors perspective, almost the exact opposite. Its history has been dominated by software implementations of (comparatively) slow transports on (comparatively) expensive machines (normally computers of some sort). I think you are seeing some doubts being expressed about the speed and cost of these approaches, and whether they can be transplanted to the storage world (which is the focus for iSCSI). Personally, I'd like to hear most on cheap, fast, TCP/IP implementations. Afterall, if it really was cheap and fast to send FC out one end on top of TCP/IP and get it back on the other end, then I for one would certainly support that approach given its other benefits. Jim -----Original Message----- From: somesh_gupta@hp.com [mailto:somesh_gupta@hp.com] Sent: Thursday, August 17, 2000 5:41 PM To: dotis@sanlight.net; ips@ece.cmu.edu; julian_satran@il.ibm.com Subject: RE: RE: Towards Consensus on TCP Connections Doug, Tunneling FC through IP to connect FC islands together is a good application and has its place, and an iSCSI <-> FCP gateway will be an important app to connect the two worlds together. However neither of these is a replacement for using TCP/IP as a native transport for SCSI commands. FC networks appear to me to be limited scale networks providing a very reliable link (channel) using b-b credits to avoid dropping packets. Therefore any protocol using FC network makes those assumptions. However this limits the scale, and also other factors are driving the use of IP based networks. When FC goes over an IP based networks, then either the assumptions (of a reliable channel) are no longer true, or a convergence layer has to be built to provide the same level of service (using TCP and another sub-layer on top perhaps). Even in FC, you have SCSI states etc and FC states (fairly involved protocol and does not have all the capabilities as IP based networks provide - or at not yet at the same level of interoperability and robustness). You have made a comment multiple times about the compexlity of SCSI over TCP and complexity of states and demuxing etc. I somehow don't get it (when I compare it to the fact that systems have been managing loads of apps running on top of TCP and that switches and NICs are implementing TCP in silicon). Perhaps you can give some details of the state and demux complexity and error recovery issues and we can explore them to see if there is something we are missing or perhaps some of us can help with explaining why they may not be issues. SCSI over TCP is probably more complex than FCP/FC. However, it should provide more capabilities and a converged network. It should also be simpler than encapsulating FC over IP (TCP). Also there is far more expertise in this area and it is using existing components except the iSCSI encapsulation which is fairly simple. I don't think there is any unknowns in this area. So let us explore and complexity issues in more detail. Somesh > -----Original Message----- > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > Sent: Wednesday, August 16, 2000 8:04 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: FW: RE: Towards Consensus on TCP Connections > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Tuesday, August 15, 2000 10:06 PM > > To: ips@ece.cmu.edu > > Subject: RE: Towards Consensus on TCP Connections > > > > Doug, > > > > Please cool down and keep the discussion technical. I > > personally do not enjoy bringing press quotes to a technical > > discussion. And so much more when they are inaccurate. > > You indicated confusion as to the comparison being made. I tried to > enlighten you as to the alternative. I did it with humor and > with technical > details. I see nothing inaccurate nor have you indicated errors. > > > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > > that FCP will be encapsulated on IP. FCP is tightly tied to FC > > (as TCP is to IP) and has already a complex (and not yet closed) > > recovery mechanism based on the assumption it runs over FC. > > An IP datagram is nothing more than a vehicle for FC encapsulation and > indeed includes only the defragmentation process. There is no need to > manage the defrag process or to be concerned about possible > failure at the > IP level. It is transparent and self healing as FCP-2 on FC > handles these > errors and would be no less reliable in doing so. There are > few uses for FC > beyond SCSI so your point escapes me especially when you are > advocating a > protocol wholly unsupported. Are you suggesting FC over IP > will not work? > > > Moreover - like SCSI - FCP has the notion of a target distinct > > from the notion of a LUN. It could be an acceptable solution to > > extend the range of a FC island but even for this it has to go > > some way in solving the congestion and security issues. > > There are already provisions for security for FC over IP. In many > applications, bandwidth is controlled by a dedicated channel > which solves > both issues. There are many means for flow control where TCP > is but one. > Reliability should be the differentiating factor. > > > IMHO even for this application a FCP-iSCSI gateway could > provide a better > > solution as the mapping is straightforward. > > > > Julo > > This provides few techincal details as to how you base this > opinion. If I > had the choice of purchasing a blade for a switch or new > system adapters > with high overhead and new SAN controllers with limited > connectivity where I > could also expect massive faults, I would not see that as > straightforward. > But then again, maybe I would. > > Doug > > > > "Douglas Otis" on 15/08/2000 19:24:34 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Towards Consensus on TCP Connections > > > > Julo, > > > > For details on proposed Fibre-Channel over IP which does not impose > > additional catastrophic error states as does iSCSI, please see: > > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > > > "While the FC over IP specification is independent of the link level > > transport protocol, it assumes a high bandwidth, high > > reliability, low loss > > link level transport such as Gigabit Ethernet, SONET, ATM, > or DWDM. This > > specification treats all classes of FC frames the same -- > as datagrams." > > ipfc > > > > "Fibre Channel over IP's main advantage is it needs no > > modifications to the > > storage subsystem or the server operating system. SCSI over > TCP/IP, on the > > other hand, requires filter drivers so the server operating > system can do > > IP-network-to-SAN emulation (for which I humbly propose the acronym > > INSANE)." Wayne Rickard > > Network World, 06/12/00 wayne@gadzoox.com > > > > To see who is doing Fibre-Channel encapsulation see: > > > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_0 > 61300b.htm > > > > At keeping the same architecture as the present Fibre-Channel > > configuration, > > you have not improved reliability. By making compromises > that combine > > states, you are reducing reliability. It would appear, in > efforts to > > reduce > > overhead in TCP transport, you have added dangerous > overhead at the SCSI > > emulation layer. This is taking advantage of TCP in > hardware? Allow a > > session per LUN, (remove the LUN field) then there is no danger > > of inducing > > an error across multiple devices when sorting millions of > tags for a LUN > > value. Care about reliability. > > > > Doug > > > > > Doug, > > > > > > You can use the standard at the end user device, if you so wish. > > > If you choose a "transprent" controller - i.e. one not holding > > > any state or > > > ordering - and > > > view every LU as and independent unit - you ignore the > numbering (that > > you > > > are free to do anyway for the commands) and ship the commands to > > > the device > > > for execution. > > > > > > Initiator tags are unique per initiator and it should not be a > > problem to > > > keep the unique > > > per session. > > > > > > Target task tags are a different issue. We decided to > remove the LUN > > field > > > in both the incoming RTT and the the outgoing data to > avoid the need to > > > test for inconsistencies. > > > But as this is the only thing that implies unique target task > > > tags we might > > > as well "reinstate" the LUN (the field is still unused) > in the DATA > > packet > > > (in the RTT it is implied by the initiator > > > task tag). > > > > > > Relaxing the requirement of uniqueness for initiator tag > is possible too > > > (you can have the unique per LU) but that might complicate > > > unecessarily the > > > initiator. > > > > > > If this (minor) change does not anoy target controller > implementors and > > > nobody sees something else wrong with it we might > introduce it in the > > next > > > version. > > > > > > As for your FC claims - I still can't follow your argument. Are > > > you talking > > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > > > Julo > > > > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > Julo, > > > > > > Keep the interface suitable for the end device. A controller in > > > the middle > > > does not justify a unique interface for that scenario. > In trying to > > > optimize "Bandwidth utilization", you are binding > thousands of logical > > > units > > > together into the same session. This is an added burden > within the > > > standard, if used at the device. In doing so, session > tags must be kept > > > unique across all logical units. Reset all drives to recover > > from even a > > > single tag error? If you wish to cache information at > the gateway, it > > > should still use the interface suitable for the end > device. At least > > this > > > isolates error handling. > > > > > > By creating a simple FC tunnel, the controller remains > bolted into the > > > back-plane where it is known to work effectively. Error > handling is > > > understood with a stateless translation and you have not created > > > a standard > > > that will set back efforts at getting a solution without myopic > > > compromises. > > > There WILL be errors not seen by this "In the Middle Controller" > > > translating > > > Fibre-Channel into iSCSI as FC does not have the same > transport. States > > > within this ITMC adds to complexity and error migration when you > > > intertwine > > > logical units. ITMC is a poor starting point and only > makes for a bad > > > interface as seen by this specification. > > > > > > Doug > > > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Saturday, August 12, 2000 12:24 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU > and I did even > > > > implement it (as many others - see Paul's reply). You > can aggregate at > > > the > > > > TCP level but you have to aggregate somewhere > > > > to use effectively the bandwidth. Eliminating the CU is > not a concern > > of > > > > this group and not a suggestion to be accepted lightly by the > > community > > > > (who will do caching, storage virtualization and management > > etc.) and I > > > am > > > > confident most of the participants on this list do not > want to discuss > > > > this subject. FC over IP as replacement for iSCSI is a poor > > > suggestion - > > > > FC is a networking protocol and IP too. FCP over IP is a gateway > > > solution. > > > > > > > > Julo > > > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > An architecture that scales does not need to provide the entire > > > bandwidth > > > > from a single device. If these devices are each handling 25 > > Mbytes per > > > > second, then 2 of them exceed Fibre-Channel. Next week > you could use > > 4, > > > > and > > > > the week after that 8. The advantage of using a > network without any > > > > practical architectural restriction with intelligence > at the client. > > On > > > > the > > > > other hand, you want to design a single mid-point gateway to > > handle the > > > > entire bandwidth. To what end? You argue it can not process a > > > > TCP session > > > > per device. You will not be tracking mid-stream errors as > > this is more > > > > costly. You add to the burden of tracking the state of > the individual > > > > device with this mid-stream state machine requiring > additional sorting > > > due > > > > to this merged protocol without taking advantage of TCP > to aid this > > > > process. > > > > For your type of solution, a simple hardware based > tunnel would be > > > better. > > > > Do not include handling of the encapsulated protocol > and at least it > > > > becomes > > > > understandable and more likely to manage the task. In other > > > > words, make it > > > > Fibre-Channel over IP and you have a chance with your > architecture. > > > > > > > > The point to my statements was to indicate the device is able > > to handle > > > an > > > > IP interface today using Fast Ethernet as the rate from > the drive is > > > > relatively low. Not to warm the cockles of marketing > pointing to > > > latency, > > > > but at least they sell more drives overcoming this problem > > using scale. > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > > > I am not sure that I agree with your architecture statements > > > but I like > > > > to > > > > > play with numbers (as most of the fellows engineers on this > > > > list probably > > > > > do). What would be in your opinion > > > > > reasonable requirements for command and data transfer > rates for the > > > next > > > > > 3-7 years? > > > > > > > > > > I would like to decouple that discussion from > architecture - data > > > > > rates can > > > > > scale even in a shared > > > > > architecture as mainframe channels have shown for years. > > > > > > > > > > I would rather like to understand if we can meet the > data rates with > > > > > reasonable latency. > > > > > > > > > > Julo > > > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > To: "Stephen Byan" , > ips@ece.cmu.edu > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data > on the outside > > > > > cylinders. Improvement of the mechanics comes at a high price > > > > > with respect > > > > > to power and cost. The cost/volume trend takes us to a single > > > > disk which > > > > > increases access time as read channel data rate increases. > > > By offering > > > > > scaled throughput using more drives where each > drive's interface > > > > bandwidth > > > > > is restricted with respect to read channel data rates > > > provides a system > > > > > with > > > > > uniform and superior performance. The advantage of such an > > > approach is > > > > > found with respect to smaller random traffic. With > more devices, > > > > > redundancy > > > > > is easily achieved and parallel access offers a means > of performance > > > > > improvement by spreading activity over more devices. The > > > > switch provides > > > > > bandwidth aggregation and is not found in the > individual device. > > > > > > > > > > An 8ms access + latency figure in the high cost > drives restricts > > > > > the number > > > > > of 'independent' operations that average 64k byte to 100 per > > > > second or 52 > > > > > Mbit per second. Such an architecture of 'restricted' drives > > > > would scale > > > > > whereas the solicitated burst approach does not. An > > > > independent nexus at > > > > > the LUN is the only design that offers required scaling and > > > > configuration > > > > > flexibility. Keeping up with the read channel is a > wasted effort. > > In > > > > > time, > > > > > 1 Gbit Ethernet will be the practical solution about > the time drives > > > are > > > > 1 > > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > > Gbit Ethernet > > > > > client makes sense in cost, performance, capacity, > reliability, and > > > > > scalability at this point in time. The protocol > overhead should be > > > > > addressed. There are substantial improvements to be made to > > > allow this > > > > > innovation using standard adapters. > > > > > > > > > > The power cost to use copper 1 Gbit is high. > Firewire does not > > > > scale and > > > > > has a limited reach. Firewire also places > scatter/gather on the > > drive > > > > > together with direct access. Doing such over a WAN > will impose > > > > > significant > > > > > changes. Serial ATA is nothing more than IDE through > a SERDES. > > > > The read > > > > > channel data rate is like a drug, just say no. It is hard > > not to buy > > > > > enough > > > > > dram to allow a proper buffer these days. Serial ATA > removes all > > > > buffers. > > > > > Intel is just usurping any remaining electronics at > the cost of > > > > > sensitivity > > > > > to a near by cell phone. Fewer drives with less > > electronics. What a > > > > good > > > > > idea? > > > > > > > > > > Doug > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > Stephen Byan > > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > > To: 'ips@ece.cmu.edu' > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > > > to be 200 > > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > > midrange storage > > > > > > market without link aggregation for a single LUN. These > > markets can > > > > > afford > > > > > > 10 Gb/s links. > > > > > > > > > > > > As a disk device level interface, iSCSI will not > succeed unless > > > > > > it offers at > > > > > > least 2 Gb/s by around 2002, at very low cost for the link. > > > Note that > > > > > even > > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look > at the Serial > > ATA > > > > > speed > > > > > > roadmap on slide 16 of Intel's Serial ATA > presentation at WinHEC: > > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > > viewpoint, the disk > > > > > > industry (both suppliers and customers) has long held the > > view that > > > > > > interface speeds need to match the media data rate. > iSCSI can try > > > > > > to make an > > > > > > argument that slower speeds are technically > adequate, but this > > > > > > will increase > > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > > > implemented > > > > > > > in a disk, there'll be wild dancing in the streets and > > > > you'll all (or > > > > > > > maybe your companies will) be rich beyond the > dreams of avarice. > > > > > > > > > > > > > > The easier you can make it for the implementors, the more > > > likely it > > > > > > > will succeed. > > > > > > > > > > > > Disk drive companies have implemented much more complex > > > > interfaces than > > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. > And multiple > > TCP > > > > > > connections don't look very hard to implement. They > just look like > > a > > > > > wart. > > > > > > But I think a necessary one. > > > > > > > > > > > > Regards, > > > > > > -Steve > > > > > > > > > > > > Steve Byan > > > > > > > > > > > > Design Engineer > > > > > > MS 1-3/E23 > > > > > > 333 South Street > > > > > > Shrewsbury, MA 01545 > > > > > > (508)770-3414 > > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Fri Aug 18 09:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32232; 18 Aug 2000 5:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA00097 for ips-outgoing; Fri, 18 Aug 2000 04:05:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00092 for ; Fri, 18 Aug 2000 04:05:51 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id EAA19336 for ; Fri, 18 Aug 2000 04:00:02 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.92) with ESMTP id CAA50552 for ; Fri, 18 Aug 2000 02:05:49 -0600 Importance: Normal Subject: RE: FC/IP vs. iSCSI & Towards Consensus on TCP Connections To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 18 Aug 2000 01:03:07 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/18/2000 02:05:48 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with Kalman, and I think he stated it well. Further, we had -- I thought -- a Schedule for focus items, on which we were to concentrate. I think we are straying into distance corners, and it is not clear that this is useful. If it were possible to either separate the FC on IP stuff, or perhaps delay this discussion I think it will be very useful (Note I did not say ignore). I think we need to focus on the iSCSI protocols and refine it as we think it should be, and try to meet the direction and schedule that David Black laid out. I understand that some folks want to have many TCP connections to a storage entity, including at least one TCP/IP connection per LUNs. I think it has been agreed by Julian that iSCSI will support that, with some extensions which he suggested. Even though most folks will not implement their Storage Controllers in that manor, I think that issue is perhaps settled. Julian could perhaps restate the change that would make the TCP per LUN folks (if more then one) happy. This may permit us to really move off of point "A" (specified by David Black as "Should iSCSI require a TCP connection per LUN?"). If that is so, then I also submit that we have not been focusing enough time on point "B" ("Should iSCSI have a session abstraction that binds multiple TCP connections into one iSCSI connection? "). I would like to hear from folks that believe that iSCSI does not support any number of conversations per session. I say this because I thought iSCSI could support one or more TCP/IP connections per session. Therefore, the various implementers can decide on what they will support -- one, or more then one, compatibly with any other implementations since one connection per session is the default. I say default since the target can refuse or accept more then one connection per session. So I would like to understand if this is acceptable -- especially since as we have seen that working group have both positions represented (one & more then one). I think that it might be correct that the current spec satisfies the need enough to say that point "B" has been addressed. . . John L. Hufferd meth@il.ibm.com@ece.cmu.edu on 08/18/2000 12:03:49 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: FC/IP vs. iSCSI Doug wrote: "In reality, the iSCSI proposal was to a be a modification for a Fibre-Channel controller. A complete and unaltered encapsulation is the most straightforward approach and likely with the lowest overhead and risk." This is not correct. The original iSCSI proposal was meant (at least by some of the authors) to provide a SAM-2 compliant native transport for SCSI over (specifically) TCP. The ultimate goal is to have a single network infrastructure for regular internet traffic and for storage traffic; i.e. the ultimate goal is to not need a separate Fibre-Channel infrastructure. (Sorry Fibre-Channel fans ;-).) The same management tools and off-the-shelf components can then be used for both ordinary internet infrastructure and for remote storage infrastructure. Now the IP Storage WG charter is broader than what iSCSI provides. There is no need for iSCSI to provide a universal solution to all IP storage problems. Let's develop the iSCSI protocol to do well what it was originally designed for: native SCSI over TCP. We have to keep in mind, however, that in order for iSCSI to be adopted, we will probably have to provide some support for existing storage SAN infrastructure (i.e. Fibre Channel) bridging. There is no need, however, to make that the primary focus of the iSCSI protocol. - Kalman From owner-ips@ECE.cmu.edu Fri Aug 18 11:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32685; 18 Aug 2000 7:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA10305 for ips-outgoing; Fri, 18 Aug 2000 05:43:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA10296 for ; Fri, 18 Aug 2000 05:43:42 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA03556 for ; Fri, 18 Aug 2000 11:43:10 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id LAA36200 for ; Fri, 18 Aug 2000 11:43:10 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693F.00356349 ; Fri, 18 Aug 2000 11:43:08 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 18 Aug 2000 12:14:24 +0300 Subject: RE: FC/IP vs. iSCSI & Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk John, For the record! We will have a LUN field in the packets that currently rely on the target-tag to be unique for delivery (at least DATA-OUT). This way an implementor building state machines/LUN will not have to issue unique tags and can deffer the whole state machine handling to a "LU related" (drive?) unit with the controller acting only as a hub. Julo "John Hufferd/San Jose/IBM" on 18/08/2000 11:03:07 Please respond to "John Hufferd/San Jose/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: FC/IP vs. iSCSI & Towards Consensus on TCP Connections I agree with Kalman, and I think he stated it well. Further, we had -- I thought -- a Schedule for focus items, on which we were to concentrate. I think we are straying into distance corners, and it is not clear that this is useful. If it were possible to either separate the FC on IP stuff, or perhaps delay this discussion I think it will be very useful (Note I did not say ignore). I think we need to focus on the iSCSI protocols and refine it as we think it should be, and try to meet the direction and schedule that David Black laid out. I understand that some folks want to have many TCP connections to a storage entity, including at least one TCP/IP connection per LUNs. I think it has been agreed by Julian that iSCSI will support that, with some extensions which he suggested. Even though most folks will not implement their Storage Controllers in that manor, I think that issue is perhaps settled. Julian could perhaps restate the change that would make the TCP per LUN folks (if more then one) happy. This may permit us to really move off of point "A" (specified by David Black as "Should iSCSI require a TCP connection per LUN?"). If that is so, then I also submit that we have not been focusing enough time on point "B" ("Should iSCSI have a session abstraction that binds multiple TCP connections into one iSCSI connection? "). I would like to hear from folks that believe that iSCSI does not support any number of conversations per session. I say this because I thought iSCSI could support one or more TCP/IP connections per session. Therefore, the various implementers can decide on what they will support -- one, or more then one, compatibly with any other implementations since one connection per session is the default. I say default since the target can refuse or accept more then one connection per session. So I would like to understand if this is acceptable -- especially since as we have seen that working group have both positions represented (one & more then one). I think that it might be correct that the current spec satisfies the need enough to say that point "B" has been addressed. . . John L. Hufferd meth@il.ibm.com@ece.cmu.edu on 08/18/2000 12:03:49 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: FC/IP vs. iSCSI Doug wrote: "In reality, the iSCSI proposal was to a be a modification for a Fibre-Channel controller. A complete and unaltered encapsulation is the most straightforward approach and likely with the lowest overhead and risk." This is not correct. The original iSCSI proposal was meant (at least by some of the authors) to provide a SAM-2 compliant native transport for SCSI over (specifically) TCP. The ultimate goal is to have a single network infrastructure for regular internet traffic and for storage traffic; i.e. the ultimate goal is to not need a separate Fibre-Channel infrastructure. (Sorry Fibre-Channel fans ;-).) The same management tools and off-the-shelf components can then be used for both ordinary internet infrastructure and for remote storage infrastructure. Now the IP Storage WG charter is broader than what iSCSI provides. There is no need for iSCSI to provide a universal solution to all IP storage problems. Let's develop the iSCSI protocol to do well what it was originally designed for: native SCSI over TCP. We have to keep in mind, however, that in order for iSCSI to be adopted, we will probably have to provide some support for existing storage SAN infrastructure (i.e. Fibre Channel) bridging. There is no need, however, to make that the primary focus of the iSCSI protocol. - Kalman From owner-ips@ECE.cmu.edu Fri Aug 18 13:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00434; 18 Aug 2000 9:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA12715 for ips-outgoing; Fri, 18 Aug 2000 07:55:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ztxmail04.ztx.compaq.com (ztxmail04.ztx.compaq.com [161.114.1.208]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00958 for ; Fri, 18 Aug 2000 04:50:27 -0400 (EDT) Received: by ztxmail04.ztx.compaq.com (Postfix, from userid 12345) id 2169819FF; Fri, 18 Aug 2000 03:49:57 -0500 (CDT) Received: from excmun-gh02.eur.compaq.com (excmun-gh02.dem.cpqcorp.net [16.41.92.161]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 797F71AB7 for ; Fri, 18 Aug 2000 03:49:56 -0500 (CDT) Received: by excmun-gh02.dem.cpqcorp.net with Internet Mail Service (5.5.2650.21) id ; Fri, 18 Aug 2000 10:49:55 +0200 Message-ID: <1FF17ADDAC64D0119A6E0000F830C9EA0291037C@aeoexc1.aeo.cpqcorp.net> From: "Havard, Philippe" To: ips@ece.cmu.edu Subject: unsubscribe Date: Fri, 18 Aug 2000 10:49:49 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: vendredi 18 aout 2000 10:03 To: ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI & Towards Consensus on TCP Connections I agree with Kalman, and I think he stated it well. Further, we had -- I thought -- a Schedule for focus items, on which we were to concentrate. I think we are straying into distance corners, and it is not clear that this is useful. If it were possible to either separate the FC on IP stuff, or perhaps delay this discussion I think it will be very useful (Note I did not say ignore). I think we need to focus on the iSCSI protocols and refine it as we think it should be, and try to meet the direction and schedule that David Black laid out. I understand that some folks want to have many TCP connections to a storage entity, including at least one TCP/IP connection per LUNs. I think it has been agreed by Julian that iSCSI will support that, with some extensions which he suggested. Even though most folks will not implement their Storage Controllers in that manor, I think that issue is perhaps settled. Julian could perhaps restate the change that would make the TCP per LUN folks (if more then one) happy. This may permit us to really move off of point "A" (specified by David Black as "Should iSCSI require a TCP connection per LUN?"). If that is so, then I also submit that we have not been focusing enough time on point "B" ("Should iSCSI have a session abstraction that binds multiple TCP connections into one iSCSI connection? "). I would like to hear from folks that believe that iSCSI does not support any number of conversations per session. I say this because I thought iSCSI could support one or more TCP/IP connections per session. Therefore, the various implementers can decide on what they will support -- one, or more then one, compatibly with any other implementations since one connection per session is the default. I say default since the target can refuse or accept more then one connection per session. So I would like to understand if this is acceptable -- especially since as we have seen that working group have both positions represented (one & more then one). I think that it might be correct that the current spec satisfies the need enough to say that point "B" has been addressed. . . John L. Hufferd meth@il.ibm.com@ece.cmu.edu on 08/18/2000 12:03:49 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: FC/IP vs. iSCSI Doug wrote: "In reality, the iSCSI proposal was to a be a modification for a Fibre-Channel controller. A complete and unaltered encapsulation is the most straightforward approach and likely with the lowest overhead and risk." This is not correct. The original iSCSI proposal was meant (at least by some of the authors) to provide a SAM-2 compliant native transport for SCSI over (specifically) TCP. The ultimate goal is to have a single network infrastructure for regular internet traffic and for storage traffic; i.e. the ultimate goal is to not need a separate Fibre-Channel infrastructure. (Sorry Fibre-Channel fans ;-).) The same management tools and off-the-shelf components can then be used for both ordinary internet infrastructure and for remote storage infrastructure. Now the IP Storage WG charter is broader than what iSCSI provides. There is no need for iSCSI to provide a universal solution to all IP storage problems. Let's develop the iSCSI protocol to do well what it was originally designed for: native SCSI over TCP. We have to keep in mind, however, that in order for iSCSI to be adopted, we will probably have to provide some support for existing storage SAN infrastructure (i.e. Fibre Channel) bridging. There is no need, however, to make that the primary focus of the iSCSI protocol. - Kalman From owner-ips@ECE.cmu.edu Fri Aug 18 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01104; 18 Aug 2000 11:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA16955 for ips-outgoing; Fri, 18 Aug 2000 10:23:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hoemlsrv.firewall.lucent.com (hoemail1.lucent.com [192.11.226.161]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA16951 for ; Fri, 18 Aug 2000 10:23:51 -0400 (EDT) Received: from hoemlsrv.firewall.lucent.com (localhost [127.0.0.1]) by hoemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id KAA19809 for ; Fri, 18 Aug 2000 10:23:51 -0400 (EDT) Received: from il0015exch001h.wins.lucent.com (h135-1-23-83.lucent.com [135.1.23.83]) by hoemlsrv.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id KAA19790 for ; Fri, 18 Aug 2000 10:23:50 -0400 (EDT) Received: by il0015exch001h.ih.lucent.com with Internet Mail Service (5.5.2650.21) id ; Fri, 18 Aug 2000 09:23:50 -0500 Message-ID: <80B684C5E29FD211AA8000A0C9CDD91904D09863@IL0015EXCH005U> From: "Rodriguez, Elizabeth G (Elizabeth)" To: "'ips@ece.cmu.edu'" , "'fc@network.com'" Subject: FCoverIP: draft & presentations available for download Date: Fri, 18 Aug 2000 09:23:48 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk All, The current FC over IP draft, as well as the presentation to the IPS and the presentation on Thursday afternoon (unofficial at IPFC) are now available at: http://www.opticalareanetworks.com/IETF48.zip The size of the file is 132KB. I apologize for the fact that the documents are not available at this time for separate download. Thanks, Elizabeth Rodriguez Lucent Technologies From owner-ips@ECE.cmu.edu Fri Aug 18 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01277; 18 Aug 2000 12:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA19001 for ips-outgoing; Fri, 18 Aug 2000 11:20:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA18997 for ; Fri, 18 Aug 2000 11:20:11 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id LAA06810 for ips@ece.cmu.edu; Fri, 18 Aug 2000 11:20:11 -0400 (EDT) Date: Fri, 18 Aug 2000 11:20:11 -0400 (EDT) From: Scott Bradner Message-Id: <200008181520.LAA06810@newdev.harvard.edu> To: ips@ece.cmu.edu Subject: RE: RE: Towards Consensus on TCP Connections Sender: owner-ips@ece.cmu.edu Precedence: bulk Re SCTP you might also look at draft-xie-stewart-usctp-00.txt which describes an extension to SCTP that came up in discussion during the meeting that could be quite helpful here Scott From owner-ips@ECE.cmu.edu Fri Aug 18 16:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01299; 18 Aug 2000 12:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA18725 for ips-outgoing; Fri, 18 Aug 2000 11:12:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA18706 for ; Fri, 18 Aug 2000 11:12:39 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id IAA04107; Fri, 18 Aug 2000 08:07:12 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id IAA24022; Fri, 18 Aug 2000 08:04:41 -0700 From: "Douglas Otis" To: , Subject: RE: RE: Towards Consensus on TCP Connections Date: Fri, 18 Aug 2000 08:14:44 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, In your iSCSI presentation, you mentioned the need for small fixes to TCP or perhaps the adoption of extensions to TCP to cure burdens created by this byte stream protocol. You indicated your desire and expectation of handling out of sequence packets without waiting for lost packets. You indicated your desire to restart a lost session and would work for modifications to meet those requirements. Your spec adds unique authentication requirements. You anticipate adoption of TCP extensions to aid in locating objects within the stream. Even so, as headers will be split across separate frames, the burden for handling these headers split at any byte location is high. Your dreams have been answered and your every desire has been met- Keeping object headers contained within the frame, keeping all objects aligned at 32-bit boundaries, allowing standard authentication, allowing independent streams to share a common flow control, allowing session restart, improving error detection, preventing blind spoofing or denial of service attacks. I would say the SIGTRAN group has done an outstanding job. Do not ignore it. I can not see how TCP can be justified for block traffic in light of SCTP or why you would continue to advocate its use. With SCTP, a FCP frame can continue to function over the Internet and be encapsulated as a chunk per the SCTP standard. Packet loss is eliminated, out of order processing is allowed, and congestion avoidance is achieved. It is simple and elegant. FCP structures are proximal to those you created. The direct use of these structures ensure no additional state-machine with memory beyond the frame is required to marry different structures. If you goal was to create a SAM compliant interface for new network centric hardware, then why accommodate 64-bit LUNs? Your intentions were and are to ensure FCP<->iSCSI otherwise it is also difficult to justify. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Thursday, August 17, 2000 11:53 PM > To: ips@ece.cmu.edu > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > Jim, > > Your are probably right. FC and FCP are designed mainly based on hardware > considerations. > And when the network started being to complex to handle it in simple state > machine > the protocol became complex and the solutions expensive. The IP stacks > (mainly TCP) > have a software history but on any of the modern adapters you will see > the "main path" functions implemented in hardware - and this is true for > switches > routers etc. > The basic TCP state machine is very simple (most implementation require a > very low > instruction count) and the major burden - copies - can be avoided by > cleverly designing > the upper layer protocol. > > Any "simple encapsulation" protocol will inherit the complexity > of both the > IP stack - as it can't > rely anymore on the link layer credit functions provided by the > FC hardware > - and of FC - as FCP is designed on a transport that does not handle loss > (FCP is both a transport and an application protocol). > > Nevertheless for tunneling an FC encapsulation might be a good solution as > the cost/complexity ADDED is not high (the islands are FCP) and > they can be > interconnected > by TCP - provided that a good addressing/routing mechanism can be put in > place. It can be done fast and will satify a need. It is not an end-to-end > IP solution. > > This led us to build a mapping specific for TCP (as others have built > mapping specific to > Serial Bus, ST etc.). > > Julo > > > Jim McGrath on 18/08/2000 07:06:33 > > Please respond to Jim McGrath > > To: "'somesh_gupta@hp.com'" , dotis@sanlight.net, > ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > > Observation spanning many emails: > > I think part of the problem here is that people have some different > experiences. For instance, you mentioned silicon based TCP/IP > implementations, and yet I personally (and I think a number of other > storage > focused people) are not familiar with them and their characteristics (e.g. > availability, cost, etc...). > > Storage vendors have had to automate a lot of protocol in silicon for > years, > and this had had a pronounced influence on the evolution of that protocol. > For instance, in both SCSI and Fibre Channel standards activities you have > a > lot of discussion as to whether something can be easily and inexpensively > implemented in silicon - this is often a consideration that weighs heavily > in the decision making. This is due to the combination of lost cost and > high performance focus of the industry. Indeed, this focus on "simple" > protocols (relying of good links, etc...) is I think a reason why some > folks > in the networking community have doubts concerning issues like congestion > control for storage protocols or anything based on them. > > The networking industry is, as seem from a storage vendors perspective, > almost the exact opposite. Its history has been dominated by software > implementations of (comparatively) slow transports on (comparatively) > expensive machines (normally computers of some sort). I think you are > seeing some doubts being expressed about the speed and cost of these > approaches, and whether they can be transplanted to the storage world > (which > is the focus for iSCSI). > > Personally, I'd like to hear most on cheap, fast, TCP/IP implementations. > Afterall, if it really was cheap and fast to send FC out one end on top of > TCP/IP and get it back on the other end, then I for one would certainly > support that approach given its other benefits. > > Jim > > > -----Original Message----- > From: somesh_gupta@hp.com [mailto:somesh_gupta@hp.com] > Sent: Thursday, August 17, 2000 5:41 PM > To: dotis@sanlight.net; ips@ece.cmu.edu; julian_satran@il.ibm.com > Subject: RE: RE: Towards Consensus on TCP Connections > > > Doug, > > Tunneling FC through IP to connect FC islands together > is a good application and has its place, and an iSCSI <-> FCP > gateway will be an important app to connect the two worlds > together. However neither of these is a replacement for > using TCP/IP as a native transport for SCSI commands. > > FC networks appear to me to be limited scale networks providing > a very reliable link (channel) using b-b credits to avoid > dropping packets. Therefore any protocol using FC network makes > those assumptions. However this limits the scale, and also other > factors are driving the use of IP based networks. When FC goes over > an IP based networks, then either the assumptions (of a reliable > channel) are no longer true, or a convergence > layer has to be built to provide the same level of service (using > TCP and another sub-layer on top perhaps). > > Even in FC, you have SCSI states etc and FC states > (fairly involved protocol and does not have all the capabilities > as IP based networks provide - or at not yet at the same level > of interoperability and robustness). > > You have made a comment multiple times about the compexlity of > SCSI over TCP and complexity of states and demuxing etc. I somehow > don't get it (when I compare it to the fact that systems have been > managing loads of apps running on top of TCP and that switches and > NICs are implementing TCP in silicon). Perhaps you can give some > details of the state and demux complexity and error recovery > issues and we can explore them to see if there is something we are > missing or perhaps some of us can help with explaining why they > may not be issues. > > SCSI over TCP is probably more complex than FCP/FC. However, it should > provide more capabilities and a converged network. It should > also be simpler than encapsulating FC over IP (TCP). Also there > is far more expertise in this area and it is using existing > components except the iSCSI encapsulation which is fairly simple. I > don't think there is any unknowns in this area. So let us explore > and complexity issues in more detail. > > Somesh > > > -----Original Message----- > > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > > Sent: Wednesday, August 16, 2000 8:04 AM > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: FW: RE: Towards Consensus on TCP Connections > > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Tuesday, August 15, 2000 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Doug, > > > > > > Please cool down and keep the discussion technical. I > > > personally do not enjoy bringing press quotes to a technical > > > discussion. And so much more when they are inaccurate. > > > > You indicated confusion as to the comparison being made. I tried to > > enlighten you as to the alternative. I did it with humor and > > with technical > > details. I see nothing inaccurate nor have you indicated errors. > > > > > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > > > that FCP will be encapsulated on IP. FCP is tightly tied to FC > > > (as TCP is to IP) and has already a complex (and not yet closed) > > > recovery mechanism based on the assumption it runs over FC. > > > > An IP datagram is nothing more than a vehicle for FC encapsulation and > > indeed includes only the defragmentation process. There is no need to > > manage the defrag process or to be concerned about possible > > failure at the > > IP level. It is transparent and self healing as FCP-2 on FC > > handles these > > errors and would be no less reliable in doing so. There are > > few uses for FC > > beyond SCSI so your point escapes me especially when you are > > advocating a > > protocol wholly unsupported. Are you suggesting FC over IP > > will not work? > > > > > Moreover - like SCSI - FCP has the notion of a target distinct > > > from the notion of a LUN. It could be an acceptable solution to > > > extend the range of a FC island but even for this it has to go > > > some way in solving the congestion and security issues. > > > > There are already provisions for security for FC over IP. In many > > applications, bandwidth is controlled by a dedicated channel > > which solves > > both issues. There are many means for flow control where TCP > > is but one. > > Reliability should be the differentiating factor. > > > > > IMHO even for this application a FCP-iSCSI gateway could > > provide a better > > > solution as the mapping is straightforward. > > > > > > Julo > > > > This provides few techincal details as to how you base this > > opinion. If I > > had the choice of purchasing a blade for a switch or new > > system adapters > > with high overhead and new SAN controllers with limited > > connectivity where I > > could also expect massive faults, I would not see that as > > straightforward. > > But then again, maybe I would. > > > > Doug > > > > > > > "Douglas Otis" on 15/08/2000 19:24:34 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Julo, > > > > > > For details on proposed Fibre-Channel over IP which does not impose > > > additional catastrophic error states as does iSCSI, please see: > > > > > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > > > > > "While the FC over IP specification is independent of the link level > > > transport protocol, it assumes a high bandwidth, high > > > reliability, low loss > > > link level transport such as Gigabit Ethernet, SONET, ATM, > > or DWDM. This > > > specification treats all classes of FC frames the same -- > > as datagrams." > > > ipfc > > > > > > "Fibre Channel over IP's main advantage is it needs no > > > modifications to the > > > storage subsystem or the server operating system. SCSI over > > TCP/IP, on the > > > other hand, requires filter drivers so the server operating > > system can do > > > IP-network-to-SAN emulation (for which I humbly propose the acronym > > > INSANE)." Wayne Rickard > > > Network World, 06/12/00 wayne@gadzoox.com > > > > > > To see who is doing Fibre-Channel encapsulation see: > > > > > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_0 > > 61300b.htm > > > > > > At keeping the same architecture as the present Fibre-Channel > > > configuration, > > > you have not improved reliability. By making compromises > > that combine > > > states, you are reducing reliability. It would appear, in > > efforts to > > > reduce > > > overhead in TCP transport, you have added dangerous > > overhead at the SCSI > > > emulation layer. This is taking advantage of TCP in > > hardware? Allow a > > > session per LUN, (remove the LUN field) then there is no danger > > > of inducing > > > an error across multiple devices when sorting millions of > > tags for a LUN > > > value. Care about reliability. > > > > > > Doug > > > > > > > Doug, > > > > > > > > You can use the standard at the end user device, if you so wish. > > > > If you choose a "transprent" controller - i.e. one not holding > > > > any state or > > > > ordering - and > > > > view every LU as and independent unit - you ignore the > > numbering (that > > > you > > > > are free to do anyway for the commands) and ship the commands to > > > > the device > > > > for execution. > > > > > > > > Initiator tags are unique per initiator and it should not be a > > > problem to > > > > keep the unique > > > > per session. > > > > > > > > Target task tags are a different issue. We decided to > > remove the LUN > > > field > > > > in both the incoming RTT and the the outgoing data to > > avoid the need to > > > > test for inconsistencies. > > > > But as this is the only thing that implies unique target task > > > > tags we might > > > > as well "reinstate" the LUN (the field is still unused) > > in the DATA > > > packet > > > > (in the RTT it is implied by the initiator > > > > task tag). > > > > > > > > Relaxing the requirement of uniqueness for initiator tag > > is possible too > > > > (you can have the unique per LU) but that might complicate > > > > unecessarily the > > > > initiator. > > > > > > > > If this (minor) change does not anoy target controller > > implementors and > > > > nobody sees something else wrong with it we might > > introduce it in the > > > next > > > > version. > > > > > > > > As for your FC claims - I still can't follow your argument. Are > > > > you talking > > > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > Keep the interface suitable for the end device. A controller in > > > > the middle > > > > does not justify a unique interface for that scenario. > > In trying to > > > > optimize "Bandwidth utilization", you are binding > > thousands of logical > > > > units > > > > together into the same session. This is an added burden > > within the > > > > standard, if used at the device. In doing so, session > > tags must be kept > > > > unique across all logical units. Reset all drives to recover > > > from even a > > > > single tag error? If you wish to cache information at > > the gateway, it > > > > should still use the interface suitable for the end > > device. At least > > > this > > > > isolates error handling. > > > > > > > > By creating a simple FC tunnel, the controller remains > > bolted into the > > > > back-plane where it is known to work effectively. Error > > handling is > > > > understood with a stateless translation and you have not created > > > > a standard > > > > that will set back efforts at getting a solution without myopic > > > > compromises. > > > > There WILL be errors not seen by this "In the Middle Controller" > > > > translating > > > > Fibre-Channel into iSCSI as FC does not have the same > > transport. States > > > > within this ITMC adds to complexity and error migration when you > > > > intertwine > > > > logical units. ITMC is a poor starting point and only > > makes for a bad > > > > interface as seen by this specification. > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Saturday, August 12, 2000 12:24 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU > > and I did even > > > > > implement it (as many others - see Paul's reply). You > > can aggregate at > > > > the > > > > > TCP level but you have to aggregate somewhere > > > > > to use effectively the bandwidth. Eliminating the CU is > > not a concern > > > of > > > > > this group and not a suggestion to be accepted lightly by the > > > community > > > > > (who will do caching, storage virtualization and management > > > etc.) and I > > > > am > > > > > confident most of the participants on this list do not > > want to discuss > > > > > this subject. FC over IP as replacement for iSCSI is a poor > > > > suggestion - > > > > > FC is a networking protocol and IP too. FCP over IP is a gateway > > > > solution. > > > > > > > > > > Julo > > > > > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > > cc: > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > > > An architecture that scales does not need to provide the entire > > > > bandwidth > > > > > from a single device. If these devices are each handling 25 > > > Mbytes per > > > > > second, then 2 of them exceed Fibre-Channel. Next week > > you could use > > > 4, > > > > > and > > > > > the week after that 8. The advantage of using a > > network without any > > > > > practical architectural restriction with intelligence > > at the client. > > > On > > > > > the > > > > > other hand, you want to design a single mid-point gateway to > > > handle the > > > > > entire bandwidth. To what end? You argue it can not process a > > > > > TCP session > > > > > per device. You will not be tracking mid-stream errors as > > > this is more > > > > > costly. You add to the burden of tracking the state of > > the individual > > > > > device with this mid-stream state machine requiring > > additional sorting > > > > due > > > > > to this merged protocol without taking advantage of TCP > > to aid this > > > > > process. > > > > > For your type of solution, a simple hardware based > > tunnel would be > > > > better. > > > > > Do not include handling of the encapsulated protocol > > and at least it > > > > > becomes > > > > > understandable and more likely to manage the task. In other > > > > > words, make it > > > > > Fibre-Channel over IP and you have a chance with your > > architecture. > > > > > > > > > > The point to my statements was to indicate the device is able > > > to handle > > > > an > > > > > IP interface today using Fast Ethernet as the rate from > > the drive is > > > > > relatively low. Not to warm the cockles of marketing > > pointing to > > > > latency, > > > > > but at least they sell more drives overcoming this problem > > > using scale. > > > > > > > > > > Doug > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > julian_satran@il.ibm.com > > > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > > > > > I am not sure that I agree with your architecture statements > > > > but I like > > > > > to > > > > > > play with numbers (as most of the fellows engineers on this > > > > > list probably > > > > > > do). What would be in your opinion > > > > > > reasonable requirements for command and data transfer > > rates for the > > > > next > > > > > > 3-7 years? > > > > > > > > > > > > I would like to decouple that discussion from > > architecture - data > > > > > > rates can > > > > > > scale even in a shared > > > > > > architecture as mainframe channels have shown for years. > > > > > > > > > > > > I would rather like to understand if we can meet the > > data rates with > > > > > > reasonable latency. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > > > To: "Stephen Byan" , > > ips@ece.cmu.edu > > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data > > on the outside > > > > > > cylinders. Improvement of the mechanics comes at a high price > > > > > > with respect > > > > > > to power and cost. The cost/volume trend takes us to a single > > > > > disk which > > > > > > increases access time as read channel data rate increases. > > > > By offering > > > > > > scaled throughput using more drives where each > > drive's interface > > > > > bandwidth > > > > > > is restricted with respect to read channel data rates > > > > provides a system > > > > > > with > > > > > > uniform and superior performance. The advantage of such an > > > > approach is > > > > > > found with respect to smaller random traffic. With > > more devices, > > > > > > redundancy > > > > > > is easily achieved and parallel access offers a means > > of performance > > > > > > improvement by spreading activity over more devices. The > > > > > switch provides > > > > > > bandwidth aggregation and is not found in the > > individual device. > > > > > > > > > > > > An 8ms access + latency figure in the high cost > > drives restricts > > > > > > the number > > > > > > of 'independent' operations that average 64k byte to 100 per > > > > > second or 52 > > > > > > Mbit per second. Such an architecture of 'restricted' drives > > > > > would scale > > > > > > whereas the solicitated burst approach does not. An > > > > > independent nexus at > > > > > > the LUN is the only design that offers required scaling and > > > > > configuration > > > > > > flexibility. Keeping up with the read channel is a > > wasted effort. > > > In > > > > > > time, > > > > > > 1 Gbit Ethernet will be the practical solution about > > the time drives > > > > are > > > > > 1 > > > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > > > Gbit Ethernet > > > > > > client makes sense in cost, performance, capacity, > > reliability, and > > > > > > scalability at this point in time. The protocol > > overhead should be > > > > > > addressed. There are substantial improvements to be made to > > > > allow this > > > > > > innovation using standard adapters. > > > > > > > > > > > > The power cost to use copper 1 Gbit is high. > > Firewire does not > > > > > scale and > > > > > > has a limited reach. Firewire also places > > scatter/gather on the > > > drive > > > > > > together with direct access. Doing such over a WAN > > will impose > > > > > > significant > > > > > > changes. Serial ATA is nothing more than IDE through > > a SERDES. > > > > > The read > > > > > > channel data rate is like a drug, just say no. It is hard > > > not to buy > > > > > > enough > > > > > > dram to allow a proper buffer these days. Serial ATA > > removes all > > > > > buffers. > > > > > > Intel is just usurping any remaining electronics at > > the cost of > > > > > > sensitivity > > > > > > to a near by cell phone. Fewer drives with less > > > electronics. What a > > > > > good > > > > > > idea? > > > > > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > Of > > > > > > > Stephen Byan > > > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > > > To: 'ips@ece.cmu.edu' > > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > > > > to be 200 > > > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > > > midrange storage > > > > > > > market without link aggregation for a single LUN. These > > > markets can > > > > > > afford > > > > > > > 10 Gb/s links. > > > > > > > > > > > > > > As a disk device level interface, iSCSI will not > > succeed unless > > > > > > > it offers at > > > > > > > least 2 Gb/s by around 2002, at very low cost for the link. > > > > Note that > > > > > > even > > > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look > > at the Serial > > > ATA > > > > > > speed > > > > > > > roadmap on slide 16 of Intel's Serial ATA > > presentation at WinHEC: > > > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > > > viewpoint, the disk > > > > > > > industry (both suppliers and customers) has long held the > > > view that > > > > > > > interface speeds need to match the media data rate. > > iSCSI can try > > > > > > > to make an > > > > > > > argument that slower speeds are technically > > adequate, but this > > > > > > > will increase > > > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > > > > implemented > > > > > > > > in a disk, there'll be wild dancing in the streets and > > > > > you'll all (or > > > > > > > > maybe your companies will) be rich beyond the > > dreams of avarice. > > > > > > > > > > > > > > > > The easier you can make it for the implementors, the more > > > > likely it > > > > > > > > will succeed. > > > > > > > > > > > > > > Disk drive companies have implemented much more complex > > > > > interfaces than > > > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. > > And multiple > > > TCP > > > > > > > connections don't look very hard to implement. They > > just look like > > > a > > > > > > wart. > > > > > > > But I think a necessary one. > > > > > > > > > > > > > > Regards, > > > > > > > -Steve > > > > > > > > > > > > > > Steve Byan > > > > > > > > > > > > > > Design Engineer > > > > > > > MS 1-3/E23 > > > > > > > 333 South Street > > > > > > > Shrewsbury, MA 01545 > > > > > > > (508)770-3414 > > > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Fri Aug 18 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01551; 18 Aug 2000 13:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA21372 for ips-outgoing; Fri, 18 Aug 2000 12:18:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA21368 for ; Fri, 18 Aug 2000 12:18:09 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA15602 for ; Fri, 18 Aug 2000 18:17:38 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id SAA45218 for ; Fri, 18 Aug 2000 18:17:38 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125693F.00597DDE ; Fri, 18 Aug 2000 18:17:29 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 18 Aug 2000 19:15:35 +0300 Subject: RE: RE: Towards Consensus on TCP Connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk We are closely looking at SCTP (for a long time). We will get bak to the list with a summary of findings. It looked appealing at start but then we found that so many of it's design goals where different for ours (it was designed as a SS7 like support for IP) that they both go over what we need (out of order processing is not an issue for storage - but placing data in memory as it arrives is) or do less (like dropping connections/sequences) when things go wrong. But as I said I am still looking at it. All your other statements are a inaccurate approximation of what I said. Authentication is not unique but it is only adapted. We are not requiring extensions to TCP (although we could benefit from some for high end-adapters in high-error-rate environments). Alignment with boundaries is not mandated by iSCSI. The only need for out of order processing (your name) is having data placed in host/target memory and not piling up in NIC cards - were they might consume large amounts of memory if some packets are delayed (and this we share with any TCP user at high speed). You keep throwing FCP at us as the universal panacea - and no matter how hard I try to explain to you that a large group of people have already looked at it and found this a bad solution (and not simply saying it but giving you the reasons behind the decision). As I (and probably many others) have a hard time coping with your "flow of consciousness" - let alone answering you - I would personally appreciate if you could put together a draft of how you think this protocol should look like in a coherent form and publish it on this list. Julo "Douglas Otis" on 18/08/2000 18:14:44 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: RE: Towards Consensus on TCP Connections Julo, In your iSCSI presentation, you mentioned the need for small fixes to TCP or perhaps the adoption of extensions to TCP to cure burdens created by this byte stream protocol. You indicated your desire and expectation of handling out of sequence packets without waiting for lost packets. You indicated your desire to restart a lost session and would work for modifications to meet those requirements. Your spec adds unique authentication requirements. You anticipate adoption of TCP extensions to aid in locating objects within the stream. Even so, as headers will be split across separate frames, the burden for handling these headers split at any byte location is high. Your dreams have been answered and your every desire has been met- Keeping object headers contained within the frame, keeping all objects aligned at 32-bit boundaries, allowing standard authentication, allowing independent streams to share a common flow control, allowing session restart, improving error detection, preventing blind spoofing or denial of service attacks. I would say the SIGTRAN group has done an outstanding job. Do not ignore it. I can not see how TCP can be justified for block traffic in light of SCTP or why you would continue to advocate its use. With SCTP, a FCP frame can continue to function over the Internet and be encapsulated as a chunk per the SCTP standard. Packet loss is eliminated, out of order processing is allowed, and congestion avoidance is achieved. It is simple and elegant. FCP structures are proximal to those you created. The direct use of these structures ensure no additional state-machine with memory beyond the frame is required to marry different structures. If you goal was to create a SAM compliant interface for new network centric hardware, then why accommodate 64-bit LUNs? Your intentions were and are to ensure FCP<->iSCSI otherwise it is also difficult to justify. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Thursday, August 17, 2000 11:53 PM > To: ips@ece.cmu.edu > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > Jim, > > Your are probably right. FC and FCP are designed mainly based on hardware > considerations. > And when the network started being to complex to handle it in simple state > machine > the protocol became complex and the solutions expensive. The IP stacks > (mainly TCP) > have a software history but on any of the modern adapters you will see > the "main path" functions implemented in hardware - and this is true for > switches > routers etc. > The basic TCP state machine is very simple (most implementation require a > very low > instruction count) and the major burden - copies - can be avoided by > cleverly designing > the upper layer protocol. > > Any "simple encapsulation" protocol will inherit the complexity > of both the > IP stack - as it can't > rely anymore on the link layer credit functions provided by the > FC hardware > - and of FC - as FCP is designed on a transport that does not handle loss > (FCP is both a transport and an application protocol). > > Nevertheless for tunneling an FC encapsulation might be a good solution as > the cost/complexity ADDED is not high (the islands are FCP) and > they can be > interconnected > by TCP - provided that a good addressing/routing mechanism can be put in > place. It can be done fast and will satify a need. It is not an end-to-end > IP solution. > > This led us to build a mapping specific for TCP (as others have built > mapping specific to > Serial Bus, ST etc.). > > Julo > > > Jim McGrath on 18/08/2000 07:06:33 > > Please respond to Jim McGrath > > To: "'somesh_gupta@hp.com'" , dotis@sanlight.net, > ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > > Observation spanning many emails: > > I think part of the problem here is that people have some different > experiences. For instance, you mentioned silicon based TCP/IP > implementations, and yet I personally (and I think a number of other > storage > focused people) are not familiar with them and their characteristics (e.g. > availability, cost, etc...). > > Storage vendors have had to automate a lot of protocol in silicon for > years, > and this had had a pronounced influence on the evolution of that protocol. > For instance, in both SCSI and Fibre Channel standards activities you have > a > lot of discussion as to whether something can be easily and inexpensively > implemented in silicon - this is often a consideration that weighs heavily > in the decision making. This is due to the combination of lost cost and > high performance focus of the industry. Indeed, this focus on "simple" > protocols (relying of good links, etc...) is I think a reason why some > folks > in the networking community have doubts concerning issues like congestion > control for storage protocols or anything based on them. > > The networking industry is, as seem from a storage vendors perspective, > almost the exact opposite. Its history has been dominated by software > implementations of (comparatively) slow transports on (comparatively) > expensive machines (normally computers of some sort). I think you are > seeing some doubts being expressed about the speed and cost of these > approaches, and whether they can be transplanted to the storage world > (which > is the focus for iSCSI). > > Personally, I'd like to hear most on cheap, fast, TCP/IP implementations. > Afterall, if it really was cheap and fast to send FC out one end on top of > TCP/IP and get it back on the other end, then I for one would certainly > support that approach given its other benefits. > > Jim > > > -----Original Message----- > From: somesh_gupta@hp.com [mailto:somesh_gupta@hp.com] > Sent: Thursday, August 17, 2000 5:41 PM > To: dotis@sanlight.net; ips@ece.cmu.edu; julian_satran@il.ibm.com > Subject: RE: RE: Towards Consensus on TCP Connections > > > Doug, > > Tunneling FC through IP to connect FC islands together > is a good application and has its place, and an iSCSI <-> FCP > gateway will be an important app to connect the two worlds > together. However neither of these is a replacement for > using TCP/IP as a native transport for SCSI commands. > > FC networks appear to me to be limited scale networks providing > a very reliable link (channel) using b-b credits to avoid > dropping packets. Therefore any protocol using FC network makes > those assumptions. However this limits the scale, and also other > factors are driving the use of IP based networks. When FC goes over > an IP based networks, then either the assumptions (of a reliable > channel) are no longer true, or a convergence > layer has to be built to provide the same level of service (using > TCP and another sub-layer on top perhaps). > > Even in FC, you have SCSI states etc and FC states > (fairly involved protocol and does not have all the capabilities > as IP based networks provide - or at not yet at the same level > of interoperability and robustness). > > You have made a comment multiple times about the compexlity of > SCSI over TCP and complexity of states and demuxing etc. I somehow > don't get it (when I compare it to the fact that systems have been > managing loads of apps running on top of TCP and that switches and > NICs are implementing TCP in silicon). Perhaps you can give some > details of the state and demux complexity and error recovery > issues and we can explore them to see if there is something we are > missing or perhaps some of us can help with explaining why they > may not be issues. > > SCSI over TCP is probably more complex than FCP/FC. However, it should > provide more capabilities and a converged network. It should > also be simpler than encapsulating FC over IP (TCP). Also there > is far more expertise in this area and it is using existing > components except the iSCSI encapsulation which is fairly simple. I > don't think there is any unknowns in this area. So let us explore > and complexity issues in more detail. > > Somesh > > > -----Original Message----- > > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > > Sent: Wednesday, August 16, 2000 8:04 AM > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: FW: RE: Towards Consensus on TCP Connections > > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Tuesday, August 15, 2000 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Doug, > > > > > > Please cool down and keep the discussion technical. I > > > personally do not enjoy bringing press quotes to a technical > > > discussion. And so much more when they are inaccurate. > > > > You indicated confusion as to the comparison being made. I tried to > > enlighten you as to the alternative. I did it with humor and > > with technical > > details. I see nothing inaccurate nor have you indicated errors. > > > > > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > > > that FCP will be encapsulated on IP. FCP is tightly tied to FC > > > (as TCP is to IP) and has already a complex (and not yet closed) > > > recovery mechanism based on the assumption it runs over FC. > > > > An IP datagram is nothing more than a vehicle for FC encapsulation and > > indeed includes only the defragmentation process. There is no need to > > manage the defrag process or to be concerned about possible > > failure at the > > IP level. It is transparent and self healing as FCP-2 on FC > > handles these > > errors and would be no less reliable in doing so. There are > > few uses for FC > > beyond SCSI so your point escapes me especially when you are > > advocating a > > protocol wholly unsupported. Are you suggesting FC over IP > > will not work? > > > > > Moreover - like SCSI - FCP has the notion of a target distinct > > > from the notion of a LUN. It could be an acceptable solution to > > > extend the range of a FC island but even for this it has to go > > > some way in solving the congestion and security issues. > > > > There are already provisions for security for FC over IP. In many > > applications, bandwidth is controlled by a dedicated channel > > which solves > > both issues. There are many means for flow control where TCP > > is but one. > > Reliability should be the differentiating factor. > > > > > IMHO even for this application a FCP-iSCSI gateway could > > provide a better > > > solution as the mapping is straightforward. > > > > > > Julo > > > > This provides few techincal details as to how you base this > > opinion. If I > > had the choice of purchasing a blade for a switch or new > > system adapters > > with high overhead and new SAN controllers with limited > > connectivity where I > > could also expect massive faults, I would not see that as > > straightforward. > > But then again, maybe I would. > > > > Doug > > > > > > > "Douglas Otis" on 15/08/2000 19:24:34 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Julo, > > > > > > For details on proposed Fibre-Channel over IP which does not impose > > > additional catastrophic error states as does iSCSI, please see: > > > > > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > > > > > "While the FC over IP specification is independent of the link level > > > transport protocol, it assumes a high bandwidth, high > > > reliability, low loss > > > link level transport such as Gigabit Ethernet, SONET, ATM, > > or DWDM. This > > > specification treats all classes of FC frames the same -- > > as datagrams." > > > ipfc > > > > > > "Fibre Channel over IP's main advantage is it needs no > > > modifications to the > > > storage subsystem or the server operating system. SCSI over > > TCP/IP, on the > > > other hand, requires filter drivers so the server operating > > system can do > > > IP-network-to-SAN emulation (for which I humbly propose the acronym > > > INSANE)." Wayne Rickard > > > Network World, 06/12/00 wayne@gadzoox.com > > > > > > To see who is doing Fibre-Channel encapsulation see: > > > > > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_0 > > 61300b.htm > > > > > > At keeping the same architecture as the present Fibre-Channel > > > configuration, > > > you have not improved reliability. By making compromises > > that combine > > > states, you are reducing reliability. It would appear, in > > efforts to > > > reduce > > > overhead in TCP transport, you have added dangerous > > overhead at the SCSI > > > emulation layer. This is taking advantage of TCP in > > hardware? Allow a > > > session per LUN, (remove the LUN field) then there is no danger > > > of inducing > > > an error across multiple devices when sorting millions of > > tags for a LUN > > > value. Care about reliability. > > > > > > Doug > > > > > > > Doug, > > > > > > > > You can use the standard at the end user device, if you so wish. > > > > If you choose a "transprent" controller - i.e. one not holding > > > > any state or > > > > ordering - and > > > > view every LU as and independent unit - you ignore the > > numbering (that > > > you > > > > are free to do anyway for the commands) and ship the commands to > > > > the device > > > > for execution. > > > > > > > > Initiator tags are unique per initiator and it should not be a > > > problem to > > > > keep the unique > > > > per session. > > > > > > > > Target task tags are a different issue. We decided to > > remove the LUN > > > field > > > > in both the incoming RTT and the the outgoing data to > > avoid the need to > > > > test for inconsistencies. > > > > But as this is the only thing that implies unique target task > > > > tags we might > > > > as well "reinstate" the LUN (the field is still unused) > > in the DATA > > > packet > > > > (in the RTT it is implied by the initiator > > > > task tag). > > > > > > > > Relaxing the requirement of uniqueness for initiator tag > > is possible too > > > > (you can have the unique per LU) but that might complicate > > > > unecessarily the > > > > initiator. > > > > > > > > If this (minor) change does not anoy target controller > > implementors and > > > > nobody sees something else wrong with it we might > > introduce it in the > > > next > > > > version. > > > > > > > > As for your FC claims - I still can't follow your argument. Are > > > > you talking > > > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > Keep the interface suitable for the end device. A controller in > > > > the middle > > > > does not justify a unique interface for that scenario. > > In trying to > > > > optimize "Bandwidth utilization", you are binding > > thousands of logical > > > > units > > > > together into the same session. This is an added burden > > within the > > > > standard, if used at the device. In doing so, session > > tags must be kept > > > > unique across all logical units. Reset all drives to recover > > > from even a > > > > single tag error? If you wish to cache information at > > the gateway, it > > > > should still use the interface suitable for the end > > device. At least > > > this > > > > isolates error handling. > > > > > > > > By creating a simple FC tunnel, the controller remains > > bolted into the > > > > back-plane where it is known to work effectively. Error > > handling is > > > > understood with a stateless translation and you have not created > > > > a standard > > > > that will set back efforts at getting a solution without myopic > > > > compromises. > > > > There WILL be errors not seen by this "In the Middle Controller" > > > > translating > > > > Fibre-Channel into iSCSI as FC does not have the same > > transport. States > > > > within this ITMC adds to complexity and error migration when you > > > > intertwine > > > > logical units. ITMC is a poor starting point and only > > makes for a bad > > > > interface as seen by this specification. > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Saturday, August 12, 2000 12:24 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU > > and I did even > > > > > implement it (as many others - see Paul's reply). You > > can aggregate at > > > > the > > > > > TCP level but you have to aggregate somewhere > > > > > to use effectively the bandwidth. Eliminating the CU is > > not a concern > > > of > > > > > this group and not a suggestion to be accepted lightly by the > > > community > > > > > (who will do caching, storage virtualization and management > > > etc.) and I > > > > am > > > > > confident most of the participants on this list do not > > want to discuss > > > > > this subject. FC over IP as replacement for iSCSI is a poor > > > > suggestion - > > > > > FC is a networking protocol and IP too. FCP over IP is a gateway > > > > solution. > > > > > > > > > > Julo > > > > > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > > cc: > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > > > An architecture that scales does not need to provide the entire > > > > bandwidth > > > > > from a single device. If these devices are each handling 25 > > > Mbytes per > > > > > second, then 2 of them exceed Fibre-Channel. Next week > > you could use > > > 4, > > > > > and > > > > > the week after that 8. The advantage of using a > > network without any > > > > > practical architectural restriction with intelligence > > at the client. > > > On > > > > > the > > > > > other hand, you want to design a single mid-point gateway to > > > handle the > > > > > entire bandwidth. To what end? You argue it can not process a > > > > > TCP session > > > > > per device. You will not be tracking mid-stream errors as > > > this is more > > > > > costly. You add to the burden of tracking the state of > > the individual > > > > > device with this mid-stream state machine requiring > > additional sorting > > > > due > > > > > to this merged protocol without taking advantage of TCP > > to aid this > > > > > process. > > > > > For your type of solution, a simple hardware based > > tunnel would be > > > > better. > > > > > Do not include handling of the encapsulated protocol > > and at least it > > > > > becomes > > > > > understandable and more likely to manage the task. In other > > > > > words, make it > > > > > Fibre-Channel over IP and you have a chance with your > > architecture. > > > > > > > > > > The point to my statements was to indicate the device is able > > > to handle > > > > an > > > > > IP interface today using Fast Ethernet as the rate from > > the drive is > > > > > relatively low. Not to warm the cockles of marketing > > pointing to > > > > latency, > > > > > but at least they sell more drives overcoming this problem > > > using scale. > > > > > > > > > > Doug > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > julian_satran@il.ibm.com > > > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > > > > > I am not sure that I agree with your architecture statements > > > > but I like > > > > > to > > > > > > play with numbers (as most of the fellows engineers on this > > > > > list probably > > > > > > do). What would be in your opinion > > > > > > reasonable requirements for command and data transfer > > rates for the > > > > next > > > > > > 3-7 years? > > > > > > > > > > > > I would like to decouple that discussion from > > architecture - data > > > > > > rates can > > > > > > scale even in a shared > > > > > > architecture as mainframe channels have shown for years. > > > > > > > > > > > > I would rather like to understand if we can meet the > > data rates with > > > > > > reasonable latency. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > > > To: "Stephen Byan" , > > ips@ece.cmu.edu > > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data > > on the outside > > > > > > cylinders. Improvement of the mechanics comes at a high price > > > > > > with respect > > > > > > to power and cost. The cost/volume trend takes us to a single > > > > > disk which > > > > > > increases access time as read channel data rate increases. > > > > By offering > > > > > > scaled throughput using more drives where each > > drive's interface > > > > > bandwidth > > > > > > is restricted with respect to read channel data rates > > > > provides a system > > > > > > with > > > > > > uniform and superior performance. The advantage of such an > > > > approach is > > > > > > found with respect to smaller random traffic. With > > more devices, > > > > > > redundancy > > > > > > is easily achieved and parallel access offers a means > > of performance > > > > > > improvement by spreading activity over more devices. The > > > > > switch provides > > > > > > bandwidth aggregation and is not found in the > > individual device. > > > > > > > > > > > > An 8ms access + latency figure in the high cost > > drives restricts > > > > > > the number > > > > > > of 'independent' operations that average 64k byte to 100 per > > > > > second or 52 > > > > > > Mbit per second. Such an architecture of 'restricted' drives > > > > > would scale > > > > > > whereas the solicitated burst approach does not. An > > > > > independent nexus at > > > > > > the LUN is the only design that offers required scaling and > > > > > configuration > > > > > > flexibility. Keeping up with the read channel is a > > wasted effort. > > > In > > > > > > time, > > > > > > 1 Gbit Ethernet will be the practical solution about > > the time drives > > > > are > > > > > 1 > > > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > > > Gbit Ethernet > > > > > > client makes sense in cost, performance, capacity, > > reliability, and > > > > > > scalability at this point in time. The protocol > > overhead should be > > > > > > addressed. There are substantial improvements to be made to > > > > allow this > > > > > > innovation using standard adapters. > > > > > > > > > > > > The power cost to use copper 1 Gbit is high. > > Firewire does not > > > > > scale and > > > > > > has a limited reach. Firewire also places > > scatter/gather on the > > > drive > > > > > > together with direct access. Doing such over a WAN > > will impose > > > > > > significant > > > > > > changes. Serial ATA is nothing more than IDE through > > a SERDES. > > > > > The read > > > > > > channel data rate is like a drug, just say no. It is hard > > > not to buy > > > > > > enough > > > > > > dram to allow a proper buffer these days. Serial ATA > > removes all > > > > > buffers. > > > > > > Intel is just usurping any remaining electronics at > > the cost of > > > > > > sensitivity > > > > > > to a near by cell phone. Fewer drives with less > > > electronics. What a > > > > > good > > > > > > idea? > > > > > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > Of > > > > > > > Stephen Byan > > > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > > > To: 'ips@ece.cmu.edu' > > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > > > > to be 200 > > > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > > > midrange storage > > > > > > > market without link aggregation for a single LUN. These > > > markets can > > > > > > afford > > > > > > > 10 Gb/s links. > > > > > > > > > > > > > > As a disk device level interface, iSCSI will not > > succeed unless > > > > > > > it offers at > > > > > > > least 2 Gb/s by around 2002, at very low cost for the link. > > > > Note that > > > > > > even > > > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look > > at the Serial > > > ATA > > > > > > speed > > > > > > > roadmap on slide 16 of Intel's Serial ATA > > presentation at WinHEC: > > > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > > > viewpoint, the disk > > > > > > > industry (both suppliers and customers) has long held the > > > view that > > > > > > > interface speeds need to match the media data rate. > > iSCSI can try > > > > > > > to make an > > > > > > > argument that slower speeds are technically > > adequate, but this > > > > > > > will increase > > > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > > > > implemented > > > > > > > > in a disk, there'll be wild dancing in the streets and > > > > > you'll all (or > > > > > > > > maybe your companies will) be rich beyond the > > dreams of avarice. > > > > > > > > > > > > > > > > The easier you can make it for the implementors, the more > > > > likely it > > > > > > > > will succeed. > > > > > > > > > > > > > > Disk drive companies have implemented much more complex > > > > > interfaces than > > > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. > > And multiple > > > TCP > > > > > > > connections don't look very hard to implement. They > > just look like > > > a > > > > > > wart. > > > > > > > But I think a necessary one. > > > > > > > > > > > > > > Regards, > > > > > > > -Steve > > > > > > > > > > > > > > Steve Byan > > > > > > > > > > > > > > Design Engineer > > > > > > > MS 1-3/E23 > > > > > > > 333 South Street > > > > > > > Shrewsbury, MA 01545 > > > > > > > (508)770-3414 > > > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Fri Aug 18 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01553; 18 Aug 2000 13:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA22177 for ips-outgoing; Fri, 18 Aug 2000 12:40:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (adsl-63-192-219-28.dsl.snfc21.pacbell.net [63.192.219.28]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA22173 for ; Fri, 18 Aug 2000 12:40:12 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id JAA01215; Fri, 18 Aug 2000 09:39:34 -0700 To: Subject: SCTP [WAS: Re: Towards Consensus on TCP connections] References: Date: 18 Aug 2000 09:39:33 -0700 In-Reply-To: "Douglas Otis"'s message of "Fri, 18 Aug 2000 08:14:44 -0700" Message-ID: Lines: 8 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk SCTP is not stable currently and there is very little experience with it. This does not seem like the best route for a fairly conservative field like storage. A later organization, once SCTP is mature, can once more reconsider it. -Costa From owner-ips@ECE.cmu.edu Fri Aug 18 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02492; 18 Aug 2000 17:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA01549 for ips-outgoing; Fri, 18 Aug 2000 16:17:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA01545 for ; Fri, 18 Aug 2000 16:17:50 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA04201; Fri, 18 Aug 2000 13:12:22 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id NAA24804; Fri, 18 Aug 2000 13:09:51 -0700 From: "Douglas Otis" To: , Subject: RE: SCTP [WAS: Re: Towards Consensus on TCP connections] Date: Fri, 18 Aug 2000 13:20:00 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, I understand the desire for a conservative approach. SCTP is based on the mature TCP/IP algorithms so SCTP features added are squarely aimed at providing an object rather than a byte stream, a feature SCSI needs and yet ignored due to these slight structural changes. Too new? The iSCSI layer took the opposite approach. Rather than adopting existing serial structures in prominent use in high-end SAN facilities, these structures were discarded in toto. This will ensure tools and drivers will require extensive modification, in addition to existing equipment made incompatible. Too old? Can you explain this conservative/non-conservative choice a little more fully and provide a bit of detail or history on the harm caused by adopting SCTP and FCP structures? It would not be impossible or impractical for FC over IP/SCTP to be a solution for next generation products. It would not be difficult to make such a specification or to implement. Tools, drivers, and equipment investments remain and system developers have fewer exceptions in what is already difficult. FC has management structures that could implement the bridge flow adjustments as well as other tricks in buffer management. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > csapuntz@cisco.com > Sent: Friday, August 18, 2000 9:40 AM > To: ips@ece.cmu.edu > Subject: SCTP [WAS: Re: Towards Consensus on TCP connections] > > > > SCTP is not stable currently and there is very little experience with > it. This does not seem like the best route for a fairly conservative > field like storage. A later organization, once SCTP is mature, can > once more reconsider it. > > -Costa > From owner-ips@ECE.cmu.edu Fri Aug 18 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02481; 18 Aug 2000 17:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA00199 for ips-outgoing; Fri, 18 Aug 2000 15:41:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtprch1.nortel.com (smtprch1.nortelnetworks.com [192.135.215.14]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA00191 for ; Fri, 18 Aug 2000 15:41:35 -0400 (EDT) Received: from zsc4c002.corpwest.baynetworks.com by smtprch1.nortel.com; Fri, 18 Aug 2000 13:14:42 -0500 Received: from zsc4c006.corpwest.baynetworks.com ([134.177.2.153]) by zsc4c002.corpwest.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id RDYRRAJC; Fri, 18 Aug 2000 11:13:47 -0700 Received: from long-pc.corpwest.baynetworks.com. (long-pc.corpwest.baynetworks.com [134.177.35.89]) by zsc4c006.corpwest.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id RA28VY98; Fri, 18 Aug 2000 11:13:46 -0700 Message-Id: <4.2.2.20000818111017.00beaae0@zsc4c006.corpwest.baynetworks.com> X-Sender: long@zsc4c006.corpwest.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 18 Aug 2000 11:14:19 -0700 To: csapuntz@cisco.com, ips From: "Lyndon Ong" Subject: Re: SCTP [WAS: Re: Towards Consensus on TCP connections] In-Reply-To: References: <"Douglas Otis"'s message of "Fri, 18 Aug 2000 08:14:44 -0700"> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk It's a bit unfair to call SCTP "not stable", it has reached Proposed and has already had one successful bakeoff (another one in October). It is fair to say there has been little experience with it outside the lab. Motorola had extensive experience with an internal version in their lab. You might check with your new colleague, Randy Stewart :-) Cheers, L. Ong At 09:39 AM 8/18/2000 -0700, csapuntz@cisco.com wrote: >SCTP is not stable currently and there is very little experience with >it. This does not seem like the best route for a fairly conservative >field like storage. A later organization, once SCTP is mature, can >once more reconsider it. > >-Costa From owner-ips@ECE.cmu.edu Fri Aug 18 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02640; 18 Aug 2000 17:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA02807 for ips-outgoing; Fri, 18 Aug 2000 16:50:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA02803 for ; Fri, 18 Aug 2000 16:50:24 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 18 Aug 2000 16:49:53 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100EC3@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: Connection Consensus Progress Date: Fri, 18 Aug 2000 16:49:52 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > I understand that some folks want to have many TCP connections to a storage > entity, including at least one TCP/IP connection per LUNs. I think it has > been agreed by Julian that iSCSI will support that, with some extensions > which he suggested. Even though most folks will not implement their > Storage Controllers in that manor, I think that issue is perhaps settled. > Julian could perhaps restate the change that would make the TCP per LUN > folks (if more then one) happy. This may permit us to really move off of > point "A" (specified by David Black as "Should iSCSI require a TCP > connection per LUN?"). If that is so, then I also submit that we have not > been focusing enough time on point "B" ("Should iSCSI have a session > abstraction that binds multiple TCP connections into one iSCSI connection?"). If I weren't on vacation and dealing with email sporadically, I'd have sent email along the lines of what John Hufferd wrote above. I think that the discussion of (A) has lead to the rough consensus that iSCSI will not REQUIRE a TCP connection per LUN. It is still possible to build systems that use a connection per LUN (e.g., a collection of targets, each of which only implements LUN 0, although initiators will be REQUIRED to support multiple LUNs/target). The consensus is a bit rough, but I think it is consensus. If anyone disagrees (i.e., thinks that things are too rough to be called consensus), please send me email directly. The session discussion (B) has been widening the scope of possibilities. We need to start narrowing it. In looking over the emails on this topic, there are a bunch of intertwined issues. The original issue (B) was: (B) Should iSCSI have a session abstraction that binds multiple TCP connections into one iSCSI connection? For a "yes" answer to (B) we need a clear grasp of the requirements that motivate multiple connections (i.e., what problems does they address). So far, I think I've seen: R1) Parallel transfers to/from and failover support for tape devices. In contrast to disks, multiple SCSI connections to the same tape do not work (e.g., blocks can be written in the wrong order). R2) Obtaining parallelism for a single SCSI command across multiple transport connections using different physical links. R3) Obtaining parallelism for a single SCSI command across multiple transport connections using the same physical links. R4) Optimize failure handling, so that a single TCP connection loss doesn't immediately translate into a SCSI error visible to higher level (time-consuming) recovery logic. R1) and R2) are beyond the capabilities of existing SCSI- based systems (note that a parallel bus is a single link). R3) and R4) are related to the use of TCP as a transport. R3) needs more explanation, as TCP is known to be able to saturate Gigabit Ethernet, given enough data to transfer. Is the argument for R3) that for the transfer sizes likely to be seen in iSCSI, TCP spends enough of its time in slow start and the like that multiple TCP connections gain performance? I should also note that the error handling procedures for multiple connections will need to be specified in complete detail -- I expect to see state machine descriptions in the final version of the spec if multiple connections are supported. Beyond this are the issues of what sort of multiple connection model(s) to adopt if we decide to go that route. I've seen at least three models proposed: - All connections are equivalent, but all traffic for a single SCSI command uses one connection. - Single control connection, but data traffic can be striped across multiple connections. - LUNs are assigned to specific connections. The third model is to some extent orthogonal to the first two, and leads to another approach to building systems with a single LUN/connection. Can I ask people to focus on the requirements issues for (B)? For those in favor of multiple connections, please indicate which of the requirements (R1-R4) are important, or what additional requirements I've left out. Those against should check that none of R1-R4 are important enough to be requirements. I'm looking not only for Yes/No consensus on (B), but also the corresponding discussion of R1-R4, which I would expect to see reflected in a future version of the iSCSI requirements draft. Consensus of this sort is hard to take on the mailing list, so I'd like to see more discussion focussed on the requirements before I figure out how to go about establishing consensus. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Aug 18 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02644; 18 Aug 2000 17:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA03411 for ips-outgoing; Fri, 18 Aug 2000 17:06:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA03407 for ; Fri, 18 Aug 2000 17:06:12 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 18 Aug 2000 17:05:41 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100EC4@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: Mailing list Date: Fri, 18 Aug 2000 17:05:35 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I would like to suggest that perhaps a separate reflector be created for FC > over IP related discussions. I think combining the discussions in one > reflector would create unnecessary confusion and burden for everyone to > sort out which belongs where. That's not likely to happen because there are significant areas of overlap between iSCSI and FC over IP (which is why they're in a common WG). In the past few days, both the discussions of SCTP and the contrasting design motivations between iSCSI and FC over IP have been relevant to both efforts (and contribute to results that the WG is expected to produce), and there's more to come. I should note that the people working on FC over IP have been informed that congestion control is required. A useful convention is to flag the subject of a message that's about only one of the protocol efforts. iSCSI: is a good flag, and for FC over IP, I'm informed that FCoverIP: is preferable to FC/IP:. Leave the flag out of a message that's germane to both, to avoid mail filters of the form: if (flag-for-protocol-I-want-to-ignore) delete message; And of course, I just sent an iSCSI message without the flag. Hopefully, I won't provide too many more examples of what not to do :-(. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Aug 18 22:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02672; 18 Aug 2000 17:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA03816 for ips-outgoing; Fri, 18 Aug 2000 17:18:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA03811 for ; Fri, 18 Aug 2000 17:18:17 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id OAA03317; Fri, 18 Aug 2000 14:18:13 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Fri, 18 Aug 2000 14:18:12 -0700 Message-ID: From: Jim McGrath To: "'meth@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Date: Fri, 18 Aug 2000 14:18:11 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Actually, I don't really think there was an original intent on FC or TCP (I remember early conversations on whether TCP or other transport protocols should be used), just a "lets get a SCSI (block level) protocol working on networks." The T10 (SCSI) committee had a similar problem years ago with people pushing for advancements on parallel SCSI, fibre channel, SSA (remember that?), and 1394 (Firewire) lower level protocols. The result was an agreement that all were worthy, all could be called SCSI (since there is a lot people have invested in the name), but that each require different technical solutions at some point. These different solutions were often done by different bodies, and SAM was created to make sure that the word "SCSI" meant something at the end. I'd advocate something similar here. I see advocates for a full, unrestricted Internet access model; for a bridge model (like FC to FC); for an Intranet, or machine room only model; and perhaps others. Encapsulated FC may be right for one, TCP for another, FC over TCP for a third, etc... Rather than argue over which one is the "real" iSCSI, I'd suggest spending some time to cleanly identify these different potential application spaces. At the end one technical solution may work for multiple spaces, but perhaps there are good reasons for different solutions. Even there it would be nice if some leverage could be obtained. The ultimate point of commonality is SCSI and SAM. Jim PS even this thought may be too restrictive at some point - we earlier discussed ATA (another block level protocol) over networks - given that there are a lot more ATA drives than SCSI ones (relative shipment rates are 6:1), that may be of interest at some point. But I'd focus on the SCSI space first :-) -----Original Message----- From: meth@il.ibm.com [mailto:meth@il.ibm.com] Sent: Friday, August 18, 2000 12:04 AM To: ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI Doug wrote: "In reality, the iSCSI proposal was to a be a modification for a Fibre-Channel controller. A complete and unaltered encapsulation is the most straightforward approach and likely with the lowest overhead and risk." This is not correct. The original iSCSI proposal was meant (at least by some of the authors) to provide a SAM-2 compliant native transport for SCSI over (specifically) TCP. The ultimate goal is to have a single network infrastructure for regular internet traffic and for storage traffic; i.e. the ultimate goal is to not need a separate Fibre-Channel infrastructure. (Sorry Fibre-Channel fans ;-).) The same management tools and off-the-shelf components can then be used for both ordinary internet infrastructure and for remote storage infrastructure. Now the IP Storage WG charter is broader than what iSCSI provides. There is no need for iSCSI to provide a universal solution to all IP storage problems. Let's develop the iSCSI protocol to do well what it was originally designed for: native SCSI over TCP. We have to keep in mind, however, that in order for iSCSI to be adopted, we will probably have to provide some support for existing storage SAN infrastructure (i.e. Fibre Channel) bridging. There is no need, however, to make that the primary focus of the iSCSI protocol. - Kalman From owner-ips@ECE.cmu.edu Fri Aug 18 23:00:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02939; 18 Aug 2000 18:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA04476 for ips-outgoing; Fri, 18 Aug 2000 17:40:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-34.cisco.com [204.69.198.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA04468 for ; Fri, 18 Aug 2000 17:40:15 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id OAA02010; Fri, 18 Aug 2000 14:39:35 -0700 To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: Connection Consensus Progress References: <0F31E5C394DAD311B60C00E029101A0704100EC3@corpmx9.isus.emc.com> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 18 Aug 2000 14:39:35 -0700 In-Reply-To: Black_David@emc.com's message of "Fri, 18 Aug 2000 16:49:52 -0400" Message-ID: Lines: 45 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi David, > I've seen: > R1) Parallel transfers to/from and failover support for > tape devices. In contrast to disks, multiple SCSI > connections to the same tape do not work (e.g., > blocks can be written in the wrong order). > R2) Obtaining parallelism for a single SCSI command > across multiple transport connections using > different physical links. > R3) Obtaining parallelism for a single SCSI command > across multiple transport connections using the > same physical links. > R4) Optimize failure handling, so that a single TCP > connection loss doesn't immediately translate > into a SCSI error visible to higher level > (time-consuming) recovery logic. > R1) and R2) are beyond the capabilities of existing SCSI- > based systems (note that a parallel bus is a single link). > R3) and R4) are related to the use of TCP as a transport. That's a good summary. However, I'm a bit confused by what you mean by single SCSI command. iSCSI today doesn't obtain parallelism for a single SCSI command. A single SCSI command is executed over a single transport connection. Also, you don't need a session concept to get parallelism per se. You need it to get parallelism + ordered task queueing. > R3) needs more explanation, as TCP is known to be able > to saturate Gigabit Ethernet, given enough data to > transfer. Is the argument for R3) that for the > transfer sizes likely to be seen in iSCSI, TCP > spends enough of its time in slow start and the > like that multiple TCP connections gain performance? This is not the strongest argument, but: At 10 gigabits, one processor may not be able to saturate the link. Using multiple transport connections allows you to more readily use multiple processors. Note that these multiple processors can come in the form of either an SMP or multiple NICs. -Costa From owner-ips@ECE.cmu.edu Fri Aug 18 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03189; 18 Aug 2000 19:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA06600 for ips-outgoing; Fri, 18 Aug 2000 18:58:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ztxmail04.ztx.compaq.com (ztxmail04.ztx.compaq.com [161.114.1.208]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00958 for ; Fri, 18 Aug 2000 04:50:27 -0400 (EDT) Received: by ztxmail04.ztx.compaq.com (Postfix, from userid 12345) id 2169819FF; Fri, 18 Aug 2000 03:49:57 -0500 (CDT) Received: from excmun-gh02.eur.compaq.com (excmun-gh02.dem.cpqcorp.net [16.41.92.161]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 797F71AB7 for ; Fri, 18 Aug 2000 03:49:56 -0500 (CDT) Received: by excmun-gh02.dem.cpqcorp.net with Internet Mail Service (5.5.2650.21) id ; Fri, 18 Aug 2000 10:49:55 +0200 Message-ID: <1FF17ADDAC64D0119A6E0000F830C9EA0291037C@aeoexc1.aeo.cpqcorp.net> From: "Havard, Philippe" To: ips@ece.cmu.edu Subject: unsubscribe Date: Fri, 18 Aug 2000 10:49:49 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: vendredi 18 aout 2000 10:03 To: ips@ece.cmu.edu Subject: RE: FC/IP vs. iSCSI & Towards Consensus on TCP Connections I agree with Kalman, and I think he stated it well. Further, we had -- I thought -- a Schedule for focus items, on which we were to concentrate. I think we are straying into distance corners, and it is not clear that this is useful. If it were possible to either separate the FC on IP stuff, or perhaps delay this discussion I think it will be very useful (Note I did not say ignore). I think we need to focus on the iSCSI protocols and refine it as we think it should be, and try to meet the direction and schedule that David Black laid out. I understand that some folks want to have many TCP connections to a storage entity, including at least one TCP/IP connection per LUNs. I think it has been agreed by Julian that iSCSI will support that, with some extensions which he suggested. Even though most folks will not implement their Storage Controllers in that manor, I think that issue is perhaps settled. Julian could perhaps restate the change that would make the TCP per LUN folks (if more then one) happy. This may permit us to really move off of point "A" (specified by David Black as "Should iSCSI require a TCP connection per LUN?"). If that is so, then I also submit that we have not been focusing enough time on point "B" ("Should iSCSI have a session abstraction that binds multiple TCP connections into one iSCSI connection? "). I would like to hear from folks that believe that iSCSI does not support any number of conversations per session. I say this because I thought iSCSI could support one or more TCP/IP connections per session. Therefore, the various implementers can decide on what they will support -- one, or more then one, compatibly with any other implementations since one connection per session is the default. I say default since the target can refuse or accept more then one connection per session. So I would like to understand if this is acceptable -- especially since as we have seen that working group have both positions represented (one & more then one). I think that it might be correct that the current spec satisfies the need enough to say that point "B" has been addressed. . . John L. Hufferd meth@il.ibm.com@ece.cmu.edu on 08/18/2000 12:03:49 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: FC/IP vs. iSCSI Doug wrote: "In reality, the iSCSI proposal was to a be a modification for a Fibre-Channel controller. A complete and unaltered encapsulation is the most straightforward approach and likely with the lowest overhead and risk." This is not correct. The original iSCSI proposal was meant (at least by some of the authors) to provide a SAM-2 compliant native transport for SCSI over (specifically) TCP. The ultimate goal is to have a single network infrastructure for regular internet traffic and for storage traffic; i.e. the ultimate goal is to not need a separate Fibre-Channel infrastructure. (Sorry Fibre-Channel fans ;-).) The same management tools and off-the-shelf components can then be used for both ordinary internet infrastructure and for remote storage infrastructure. Now the IP Storage WG charter is broader than what iSCSI provides. There is no need for iSCSI to provide a universal solution to all IP storage problems. Let's develop the iSCSI protocol to do well what it was originally designed for: native SCSI over TCP. We have to keep in mind, however, that in order for iSCSI to be adopted, we will probably have to provide some support for existing storage SAN infrastructure (i.e. Fibre Channel) bridging. There is no need, however, to make that the primary focus of the iSCSI protocol. - Kalman From owner-ips@ECE.cmu.edu Fri Aug 18 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03134; 18 Aug 2000 19:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA06309 for ips-outgoing; Fri, 18 Aug 2000 18:46:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from router.spinnakernet.com (client-141-151-129-90.bellatlantic.net [141.151.129.90]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA18431 for ; Fri, 18 Aug 2000 11:05:45 -0400 (EDT) Received: from kazar-pc (kazar-pc.spinnakernet.com [10.1.1.110]) by router.spinnakernet.com (8.9.3/8.8.7) with ESMTP id LAA08342 for ; Fri, 18 Aug 2000 11:05:44 -0400 Message-Id: <4.2.2.20000818104954.00aa71e0@pop.icubed.com> X-Sender: kazar@pop.icubed.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 18 Aug 2000 11:13:53 -0400 To: ips@ece.cmu.edu From: Mike Kazar Subject: Re: iscsi single control channel Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At a high level, I like this proposal a lot. The feature that makes me most uncomfortable about iSCSI as it exists is this: iSCSI uses multiple TCP connections, partly to get good performance, but then requires an *additional* windowing protocol layered on top of multiple TCP connections, sharing state between those connections, to reimpose the desired ordering constraints. This means that even if you throw hardware at the problem of running TCP well at high speeds, you still need independent hardware to handle iSCSI sequence number processing at high(er) speeds. This iSCSI hardware will be more expensive, since it will only sell to a smaller market than a general TCP accelerator. Kalman's proposal has the benefit that requests can be removed from each connection and processed independently, without any further protocol-imposed synchronization constraints. Mike Kazar (kazar@spinnakernet.com) From owner-ips@ECE.cmu.edu Fri Aug 18 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03136; 18 Aug 2000 19:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA06339 for ips-outgoing; Fri, 18 Aug 2000 18:47:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from router.spinnakernet.com (client-141-151-129-90.bellatlantic.net [141.151.129.90]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA20286 for ; Fri, 18 Aug 2000 11:52:18 -0400 (EDT) Received: from kazar-pc (kazar-pc.spinnakernet.com [10.1.1.110]) by router.spinnakernet.com (8.9.3/8.8.7) with ESMTP id LAA10336 for ; Fri, 18 Aug 2000 11:52:17 -0400 Message-Id: <4.2.2.20000818111834.00aa4ed0@pop.icubed.com> X-Sender: kazar@pop.icubed.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 18 Aug 2000 12:00:27 -0400 To: ips@ece.cmu.edu From: Mike Kazar Subject: iSCSI failover and reconfiguration Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk In the iSCSI protocol document, a SCSI target is named by a URL, with the assumption being that DNS would map the domain name to a set of IP addresses. This works fine when the set of servers that might be supporting the SCSI target is fixed for all time. In the NFS V4 specification, the protocol allows the migration of a file system from one server to another, by allowing for an error code that directs the initiator to a new location for that file system. I believe that this would be a useful feature to have in iSCSI as well. With the addition of a distinctive error that indicates responsibility for a particular target has been moved to a different URL, migration of a SCSI target between IP hosts could be achieved in iSCSI, just as migration of file system can be performed in NFS V4. Do other readers think this would be a good thing to support in iSCSI? Mike (kazar@spinnakernet.com) From owner-ips@ECE.cmu.edu Sat Aug 19 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03202; 18 Aug 2000 19:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA06400 for ips-outgoing; Fri, 18 Aug 2000 18:49:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA28875 for ; Fri, 18 Aug 2000 15:02:43 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA04191; Fri, 18 Aug 2000 11:57:07 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA24626; Fri, 18 Aug 2000 11:54:36 -0700 From: "Douglas Otis" To: , Subject: RE: RE: Towards Consensus on TCP Connections Date: Fri, 18 Aug 2000 12:04:41 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, I would be interested in your detailed list of SCTP short-comings. As SCTP enjoys the flow-control mechanism of TCP, it is not something as unstable as you indicate. With respect to FCP over SCTP/IP, flow control of FCP is indeed different. A buffer of adequate depth would likely accommodate much of these differences. Most would attest the impact high latency would have on FCP throughput. For devices that use FCP over SCTP/IP as the native flow control, your objections would bear even less merit. There is also management controls within FCP that could be enjoined by the SCTP translating agent to further assist in these differences. The advantage would be a standard that would work well either as a simple bridge or as a point of entry. Other that saying FCP is different, can you indicate why FCP structures are impractical to use if encapsulated? You have been through this effort and I have not benefited by this thought process. I must say, looking at the iSCSI and FCP structures, I don't see any substantial problems. As far as stable products are concerned, unless error handling and structures are used from FCP, there will not be any tools or stable systems for some time. I think such a loss deserves some justification. I know John wishes you to say on schedule for your product roll-out, and my endless questions and challenges may seem petty. Judging from your resent change, it has not been without benefit. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Friday, August 18, 2000 9:16 AM > To: ips@ece.cmu.edu > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > We are closely looking at SCTP (for a long time). We will get bak to the > list > with a summary of findings. It looked appealing at start but > then we found > that so many > of it's design goals where different for ours (it was designed as a SS7 > like support for IP) > that they both go over what we need (out of order processing is not an > issue for storage > - but placing data in memory as it arrives is) or do less (like dropping > connections/sequences) when things go wrong. But as I said I am still > looking at it. > > All your other statements are a inaccurate approximation of what I said. > > Authentication is not unique but it is only adapted. > > We are not requiring extensions to TCP (although we could benefit > from some > for high end-adapters in high-error-rate environments). > > Alignment with boundaries is not mandated by iSCSI. > > The only need for out of order processing (your name) is having > data placed in host/target memory and not piling up in NIC > cards - were they might consume large amounts of memory if some > packets are delayed (and this we share with any TCP user at high speed). > > You keep throwing FCP at us as the universal panacea - and no matter how > hard I try to explain to you that a large group of people have already > looked at it > and found this a bad solution (and not simply saying it but giving you the > reasons > behind the decision). > > As I (and probably many others) have a hard time coping with your "flow of > consciousness" - let alone answering you - I would personally appreciate > if you could put together a draft of how you think this protocol should > look like > in a coherent form and publish it on this list. > > Julo > > > "Douglas Otis" on 18/08/2000 18:14:44 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > Julo, > > In your iSCSI presentation, you mentioned the need for small fixes to TCP > or > perhaps the adoption of extensions to TCP to cure burdens created by this > byte stream protocol. You indicated your desire and expectation of > handling > out of sequence packets without waiting for lost packets. You indicated > your desire to restart a lost session and would work for modifications to > meet those requirements. Your spec adds unique authentication > requirements. > You anticipate adoption of TCP extensions to aid in locating > objects within > the stream. Even so, as headers will be split across separate > frames, the > burden for handling these headers split at any byte location is high. > > Your dreams have been answered and your every desire has been > met- Keeping > object headers contained within the frame, keeping all objects aligned at > 32-bit boundaries, allowing standard authentication, allowing independent > streams to share a common flow control, allowing session restart, > improving > error detection, preventing blind spoofing or denial of service > attacks. I > would say the SIGTRAN group has done an outstanding job. Do not > ignore it. > I can not see how TCP can be justified for block traffic in light of SCTP > or > why you would continue to advocate its use. > > With SCTP, a FCP frame can continue to function over the Internet and be > encapsulated as a chunk per the SCTP standard. Packet loss is eliminated, > out of order processing is allowed, and congestion avoidance is achieved. > It is simple and elegant. FCP structures are proximal to those > you created. > The direct use of these structures ensure no additional state-machine with > memory beyond the frame is required to marry different structures. If you > goal was to create a SAM compliant interface for new network centric > hardware, then why accommodate 64-bit LUNs? Your intentions were and are > to > ensure FCP<->iSCSI otherwise it is also difficult to justify. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Thursday, August 17, 2000 11:53 PM > > To: ips@ece.cmu.edu > > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > > > > > > Jim, > > > > Your are probably right. FC and FCP are designed mainly based > on hardware > > considerations. > > And when the network started being to complex to handle it in simple > state > > machine > > the protocol became complex and the solutions expensive. The IP stacks > > (mainly TCP) > > have a software history but on any of the modern adapters you will see > > the "main path" functions implemented in hardware - and this > is true for > > switches > > routers etc. > > The basic TCP state machine is very simple (most implementation > require a > > very low > > instruction count) and the major burden - copies - can be avoided by > > cleverly designing > > the upper layer protocol. > > > > Any "simple encapsulation" protocol will inherit the complexity > > of both the > > IP stack - as it can't > > rely anymore on the link layer credit functions provided by the > > FC hardware > > - and of FC - as FCP is designed on a transport that does not handle > loss > > (FCP is both a transport and an application protocol). > > > > Nevertheless for tunneling an FC encapsulation might be a good solution > as > > the cost/complexity ADDED is not high (the islands are FCP) and > > they can be > > interconnected > > by TCP - provided that a good addressing/routing mechanism can be put in > > place. It can be done fast and will satify a need. It is not an > end-to-end > > IP solution. > > > > This led us to build a mapping specific for TCP (as others have built > > mapping specific to > > Serial Bus, ST etc.). > > > > Julo > > > > > > Jim McGrath on 18/08/2000 07:06:33 > > > > Please respond to Jim McGrath > > > > To: "'somesh_gupta@hp.com'" , dotis@sanlight.net, > > ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL > > cc: > > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > Observation spanning many emails: > > > > I think part of the problem here is that people have some different > > experiences. For instance, you mentioned silicon based TCP/IP > > implementations, and yet I personally (and I think a number of other > > storage > > focused people) are not familiar with them and their characteristics > (e.g. > > availability, cost, etc...). > > > > Storage vendors have had to automate a lot of protocol in silicon for > > years, > > and this had had a pronounced influence on the evolution of that > protocol. > > For instance, in both SCSI and Fibre Channel standards activities you > have > > a > > lot of discussion as to whether something can be easily and > inexpensively > > implemented in silicon - this is often a consideration that weighs > heavily > > in the decision making. This is due to the combination of lost cost and > > high performance focus of the industry. Indeed, this focus on "simple" > > protocols (relying of good links, etc...) is I think a reason why some > > folks > > in the networking community have doubts concerning issues like > congestion > > control for storage protocols or anything based on them. > > > > The networking industry is, as seem from a storage vendors perspective, > > almost the exact opposite. Its history has been dominated by software > > implementations of (comparatively) slow transports on (comparatively) > > expensive machines (normally computers of some sort). I think you are > > seeing some doubts being expressed about the speed and cost of these > > approaches, and whether they can be transplanted to the storage world > > (which > > is the focus for iSCSI). > > > > Personally, I'd like to hear most on cheap, fast, TCP/IP > implementations. > > Afterall, if it really was cheap and fast to send FC out one end on top > of > > TCP/IP and get it back on the other end, then I for one would certainly > > support that approach given its other benefits. > > > > Jim > > > > > > -----Original Message----- > > From: somesh_gupta@hp.com [mailto:somesh_gupta@hp.com] > > Sent: Thursday, August 17, 2000 5:41 PM > > To: dotis@sanlight.net; ips@ece.cmu.edu; julian_satran@il.ibm.com > > Subject: RE: RE: Towards Consensus on TCP Connections > > > > > > Doug, > > > > Tunneling FC through IP to connect FC islands together > > is a good application and has its place, and an iSCSI <-> FCP > > gateway will be an important app to connect the two worlds > > together. However neither of these is a replacement for > > using TCP/IP as a native transport for SCSI commands. > > > > FC networks appear to me to be limited scale networks providing > > a very reliable link (channel) using b-b credits to avoid > > dropping packets. Therefore any protocol using FC network makes > > those assumptions. However this limits the scale, and also other > > factors are driving the use of IP based networks. When FC goes over > > an IP based networks, then either the assumptions (of a reliable > > channel) are no longer true, or a convergence > > layer has to be built to provide the same level of service (using > > TCP and another sub-layer on top perhaps). > > > > Even in FC, you have SCSI states etc and FC states > > (fairly involved protocol and does not have all the capabilities > > as IP based networks provide - or at not yet at the same level > > of interoperability and robustness). > > > > You have made a comment multiple times about the compexlity of > > SCSI over TCP and complexity of states and demuxing etc. I somehow > > don't get it (when I compare it to the fact that systems have been > > managing loads of apps running on top of TCP and that switches and > > NICs are implementing TCP in silicon). Perhaps you can give some > > details of the state and demux complexity and error recovery > > issues and we can explore them to see if there is something we are > > missing or perhaps some of us can help with explaining why they > > may not be issues. > > > > SCSI over TCP is probably more complex than FCP/FC. However, it should > > provide more capabilities and a converged network. It should > > also be simpler than encapsulating FC over IP (TCP). Also there > > is far more expertise in this area and it is using existing > > components except the iSCSI encapsulation which is fairly simple. I > > don't think there is any unknowns in this area. So let us explore > > and complexity issues in more detail. > > > > Somesh > > > > > -----Original Message----- > > > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > > > Sent: Wednesday, August 16, 2000 8:04 AM > > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > > Subject: FW: RE: Towards Consensus on TCP Connections > > > > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Tuesday, August 15, 2000 10:06 PM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > Doug, > > > > > > > > Please cool down and keep the discussion technical. I > > > > personally do not enjoy bringing press quotes to a technical > > > > discussion. And so much more when they are inaccurate. > > > > > > You indicated confusion as to the comparison being made. I tried to > > > enlighten you as to the alternative. I did it with humor and > > > with technical > > > details. I see nothing inaccurate nor have you indicated errors. > > > > > > > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > > > > that FCP will be encapsulated on IP. FCP is tightly tied to FC > > > > (as TCP is to IP) and has already a complex (and not yet closed) > > > > recovery mechanism based on the assumption it runs over FC. > > > > > > An IP datagram is nothing more than a vehicle for FC encapsulation and > > > indeed includes only the defragmentation process. There is no need to > > > manage the defrag process or to be concerned about possible > > > failure at the > > > IP level. It is transparent and self healing as FCP-2 on FC > > > handles these > > > errors and would be no less reliable in doing so. There are > > > few uses for FC > > > beyond SCSI so your point escapes me especially when you are > > > advocating a > > > protocol wholly unsupported. Are you suggesting FC over IP > > > will not work? > > > > > > > Moreover - like SCSI - FCP has the notion of a target distinct > > > > from the notion of a LUN. It could be an acceptable solution to > > > > extend the range of a FC island but even for this it has to go > > > > some way in solving the congestion and security issues. > > > > > > There are already provisions for security for FC over IP. In many > > > applications, bandwidth is controlled by a dedicated channel > > > which solves > > > both issues. There are many means for flow control where TCP > > > is but one. > > > Reliability should be the differentiating factor. > > > > > > > IMHO even for this application a FCP-iSCSI gateway could > > > provide a better > > > > solution as the mapping is straightforward. > > > > > > > > Julo > > > > > > This provides few techincal details as to how you base this > > > opinion. If I > > > had the choice of purchasing a blade for a switch or new > > > system adapters > > > with high overhead and new SAN controllers with limited > > > connectivity where I > > > could also expect massive faults, I would not see that as > > > straightforward. > > > But then again, maybe I would. > > > > > > Doug > > > > > > > > > > "Douglas Otis" on 15/08/2000 19:24:34 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > Julo, > > > > > > > > For details on proposed Fibre-Channel over IP which does not impose > > > > additional catastrophic error states as does iSCSI, please see: > > > > > > > > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > > > > > > > "While the FC over IP specification is independent of the link level > > > > transport protocol, it assumes a high bandwidth, high > > > > reliability, low loss > > > > link level transport such as Gigabit Ethernet, SONET, ATM, > > > or DWDM. This > > > > specification treats all classes of FC frames the same -- > > > as datagrams." > > > > ipfc > > > > > > > > "Fibre Channel over IP's main advantage is it needs no > > > > modifications to the > > > > storage subsystem or the server operating system. SCSI over > > > TCP/IP, on the > > > > other hand, requires filter drivers so the server operating > > > system can do > > > > IP-network-to-SAN emulation (for which I humbly propose the acronym > > > > INSANE)." Wayne Rickard > > > > Network World, 06/12/00 wayne@gadzoox.com > > > > > > > > To see who is doing Fibre-Channel encapsulation see: > > > > > > > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_0 > > > 61300b.htm > > > > > > > > At keeping the same architecture as the present Fibre-Channel > > > > configuration, > > > > you have not improved reliability. By making compromises > > > that combine > > > > states, you are reducing reliability. It would appear, in > > > efforts to > > > > reduce > > > > overhead in TCP transport, you have added dangerous > > > overhead at the SCSI > > > > emulation layer. This is taking advantage of TCP in > > > hardware? Allow a > > > > session per LUN, (remove the LUN field) then there is no danger > > > > of inducing > > > > an error across multiple devices when sorting millions of > > > tags for a LUN > > > > value. Care about reliability. > > > > > > > > Doug > > > > > > > > > Doug, > > > > > > > > > > You can use the standard at the end user device, if you so wish. > > > > > If you choose a "transprent" controller - i.e. one not holding > > > > > any state or > > > > > ordering - and > > > > > view every LU as and independent unit - you ignore the > > > numbering (that > > > > you > > > > > are free to do anyway for the commands) and ship the commands to > > > > > the device > > > > > for execution. > > > > > > > > > > Initiator tags are unique per initiator and it should not be a > > > > problem to > > > > > keep the unique > > > > > per session. > > > > > > > > > > Target task tags are a different issue. We decided to > > > remove the LUN > > > > field > > > > > in both the incoming RTT and the the outgoing data to > > > avoid the need to > > > > > test for inconsistencies. > > > > > But as this is the only thing that implies unique target task > > > > > tags we might > > > > > as well "reinstate" the LUN (the field is still unused) > > > in the DATA > > > > packet > > > > > (in the RTT it is implied by the initiator > > > > > task tag). > > > > > > > > > > Relaxing the requirement of uniqueness for initiator tag > > > is possible too > > > > > (you can have the unique per LU) but that might complicate > > > > > unecessarily the > > > > > initiator. > > > > > > > > > > If this (minor) change does not anoy target controller > > > implementors and > > > > > nobody sees something else wrong with it we might > > > introduce it in the > > > > next > > > > > version. > > > > > > > > > > As for your FC claims - I still can't follow your argument. Are > > > > > you talking > > > > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > > > > > > > Julo > > > > > > > > > > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > > cc: > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > > > Keep the interface suitable for the end device. A controller in > > > > > the middle > > > > > does not justify a unique interface for that scenario. > > > In trying to > > > > > optimize "Bandwidth utilization", you are binding > > > thousands of logical > > > > > units > > > > > together into the same session. This is an added burden > > > within the > > > > > standard, if used at the device. In doing so, session > > > tags must be kept > > > > > unique across all logical units. Reset all drives to recover > > > > from even a > > > > > single tag error? If you wish to cache information at > > > the gateway, it > > > > > should still use the interface suitable for the end > > > device. At least > > > > this > > > > > isolates error handling. > > > > > > > > > > By creating a simple FC tunnel, the controller remains > > > bolted into the > > > > > back-plane where it is known to work effectively. Error > > > handling is > > > > > understood with a stateless translation and you have not created > > > > > a standard > > > > > that will set back efforts at getting a solution without myopic > > > > > compromises. > > > > > There WILL be errors not seen by this "In the Middle Controller" > > > > > translating > > > > > Fibre-Channel into iSCSI as FC does not have the same > > > transport. States > > > > > within this ITMC adds to complexity and error migration when you > > > > > intertwine > > > > > logical units. ITMC is a poor starting point and only > > > makes for a bad > > > > > interface as seen by this specification. > > > > > > > > > > Doug > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > > > julian_satran@il.ibm.com > > > > > > Sent: Saturday, August 12, 2000 12:24 AM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU > > > and I did even > > > > > > implement it (as many others - see Paul's reply). You > > > can aggregate at > > > > > the > > > > > > TCP level but you have to aggregate somewhere > > > > > > to use effectively the bandwidth. Eliminating the CU is > > > not a concern > > > > of > > > > > > this group and not a suggestion to be accepted lightly by the > > > > community > > > > > > (who will do caching, storage virtualization and management > > > > etc.) and I > > > > > am > > > > > > confident most of the participants on this list do not > > > want to discuss > > > > > > this subject. FC over IP as replacement for iSCSI is a poor > > > > > suggestion - > > > > > > FC is a networking protocol and IP too. FCP over IP is a gateway > > > > > solution. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > > > cc: > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > > > > > An architecture that scales does not need to provide the entire > > > > > bandwidth > > > > > > from a single device. If these devices are each handling 25 > > > > Mbytes per > > > > > > second, then 2 of them exceed Fibre-Channel. Next week > > > you could use > > > > 4, > > > > > > and > > > > > > the week after that 8. The advantage of using a > > > network without any > > > > > > practical architectural restriction with intelligence > > > at the client. > > > > On > > > > > > the > > > > > > other hand, you want to design a single mid-point gateway to > > > > handle the > > > > > > entire bandwidth. To what end? You argue it can not process a > > > > > > TCP session > > > > > > per device. You will not be tracking mid-stream errors as > > > > this is more > > > > > > costly. You add to the burden of tracking the state of > > > the individual > > > > > > device with this mid-stream state machine requiring > > > additional sorting > > > > > due > > > > > > to this merged protocol without taking advantage of TCP > > > to aid this > > > > > > process. > > > > > > For your type of solution, a simple hardware based > > > tunnel would be > > > > > better. > > > > > > Do not include handling of the encapsulated protocol > > > and at least it > > > > > > becomes > > > > > > understandable and more likely to manage the task. In other > > > > > > words, make it > > > > > > Fibre-Channel over IP and you have a chance with your > > > architecture. > > > > > > > > > > > > The point to my statements was to indicate the device is able > > > > to handle > > > > > an > > > > > > IP interface today using Fast Ethernet as the rate from > > > the drive is > > > > > > relatively low. Not to warm the cockles of marketing > > > pointing to > > > > > latency, > > > > > > but at least they sell more drives overcoming this problem > > > > using scale. > > > > > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > Of > > > > > > > julian_satran@il.ibm.com > > > > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > > > > To: ips@ece.cmu.edu > > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > > > > > > > I am not sure that I agree with your architecture statements > > > > > but I like > > > > > > to > > > > > > > play with numbers (as most of the fellows engineers on this > > > > > > list probably > > > > > > > do). What would be in your opinion > > > > > > > reasonable requirements for command and data transfer > > > rates for the > > > > > next > > > > > > > 3-7 years? > > > > > > > > > > > > > > I would like to decouple that discussion from > > > architecture - data > > > > > > > rates can > > > > > > > scale even in a shared > > > > > > > architecture as mainframe channels have shown for years. > > > > > > > > > > > > > > I would rather like to understand if we can meet the > > > data rates with > > > > > > > reasonable latency. > > > > > > > > > > > > > > Julo > > > > > > > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > > > > > To: "Stephen Byan" , > > > ips@ece.cmu.edu > > > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data > > > on the outside > > > > > > > cylinders. Improvement of the mechanics comes at a high price > > > > > > > with respect > > > > > > > to power and cost. The cost/volume trend takes us to a single > > > > > > disk which > > > > > > > increases access time as read channel data rate increases. > > > > > By offering > > > > > > > scaled throughput using more drives where each > > > drive's interface > > > > > > bandwidth > > > > > > > is restricted with respect to read channel data rates > > > > > provides a system > > > > > > > with > > > > > > > uniform and superior performance. The advantage of such an > > > > > approach is > > > > > > > found with respect to smaller random traffic. With > > > more devices, > > > > > > > redundancy > > > > > > > is easily achieved and parallel access offers a means > > > of performance > > > > > > > improvement by spreading activity over more devices. The > > > > > > switch provides > > > > > > > bandwidth aggregation and is not found in the > > > individual device. > > > > > > > > > > > > > > An 8ms access + latency figure in the high cost > > > drives restricts > > > > > > > the number > > > > > > > of 'independent' operations that average 64k byte to 100 per > > > > > > second or 52 > > > > > > > Mbit per second. Such an architecture of 'restricted' drives > > > > > > would scale > > > > > > > whereas the solicitated burst approach does not. An > > > > > > independent nexus at > > > > > > > the LUN is the only design that offers required scaling and > > > > > > configuration > > > > > > > flexibility. Keeping up with the read channel is a > > > wasted effort. > > > > In > > > > > > > time, > > > > > > > 1 Gbit Ethernet will be the practical solution about > > > the time drives > > > > > are > > > > > > 1 > > > > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > > > > Gbit Ethernet > > > > > > > client makes sense in cost, performance, capacity, > > > reliability, and > > > > > > > scalability at this point in time. The protocol > > > overhead should be > > > > > > > addressed. There are substantial improvements to be made to > > > > > allow this > > > > > > > innovation using standard adapters. > > > > > > > > > > > > > > The power cost to use copper 1 Gbit is high. > > > Firewire does not > > > > > > scale and > > > > > > > has a limited reach. Firewire also places > > > scatter/gather on the > > > > drive > > > > > > > together with direct access. Doing such over a WAN > > > will impose > > > > > > > significant > > > > > > > changes. Serial ATA is nothing more than IDE through > > > a SERDES. > > > > > > The read > > > > > > > channel data rate is like a drug, just say no. It is hard > > > > not to buy > > > > > > > enough > > > > > > > dram to allow a proper buffer these days. Serial ATA > > > removes all > > > > > > buffers. > > > > > > > Intel is just usurping any remaining electronics at > > > the cost of > > > > > > > sensitivity > > > > > > > to a near by cell phone. Fewer drives with less > > > > electronics. What a > > > > > > good > > > > > > > idea? > > > > > > > > > > > > > > Doug > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: owner-ips@ece.cmu.edu > > > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > > Of > > > > > > > > Stephen Byan > > > > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > > > > To: 'ips@ece.cmu.edu' > > > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > > > > > > > The gating factor for whether iSCSI succeeds is not going > > > > > to be 200 > > > > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > > > > midrange storage > > > > > > > > market without link aggregation for a single LUN. These > > > > markets can > > > > > > > afford > > > > > > > > 10 Gb/s links. > > > > > > > > > > > > > > > > As a disk device level interface, iSCSI will not > > > succeed unless > > > > > > > > it offers at > > > > > > > > least 2 Gb/s by around 2002, at very low cost for the link. > > > > > Note that > > > > > > > even > > > > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look > > > at the Serial > > > > ATA > > > > > > > speed > > > > > > > > roadmap on slide 16 of Intel's Serial ATA > > > presentation at WinHEC: > > > > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > > > > viewpoint, the disk > > > > > > > > industry (both suppliers and customers) has long held the > > > > view that > > > > > > > > interface speeds need to match the media data rate. > > > iSCSI can try > > > > > > > > to make an > > > > > > > > argument that slower speeds are technically > > > adequate, but this > > > > > > > > will increase > > > > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > > > > > > > If iSCSI works at ALL in a cost effective way that can be > > > > > > implemented > > > > > > > > > in a disk, there'll be wild dancing in the streets and > > > > > > you'll all (or > > > > > > > > > maybe your companies will) be rich beyond the > > > dreams of avarice. > > > > > > > > > > > > > > > > > > The easier you can make it for the implementors, the more > > > > > likely it > > > > > > > > > will succeed. > > > > > > > > > > > > > > > > Disk drive companies have implemented much more complex > > > > > > interfaces than > > > > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. > > > And multiple > > > > TCP > > > > > > > > connections don't look very hard to implement. They > > > just look like > > > > a > > > > > > > wart. > > > > > > > > But I think a necessary one. > > > > > > > > > > > > > > > > Regards, > > > > > > > > -Steve > > > > > > > > > > > > > > > > Steve Byan > > > > > > > > > > > > > > > > Design Engineer > > > > > > > > MS 1-3/E23 > > > > > > > > 333 South Street > > > > > > > > Shrewsbury, MA 01545 > > > > > > > > (508)770-3414 > > > > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Sat Aug 19 01:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03464; 18 Aug 2000 20:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA07751 for ips-outgoing; Fri, 18 Aug 2000 19:50:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA07745 for ; Fri, 18 Aug 2000 19:50:21 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA04256 for ; Fri, 18 Aug 2000 16:44:50 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA25331 for ; Fri, 18 Aug 2000 16:42:14 -0700 From: "Douglas Otis" To: "Ips" Subject: FCoverSCTP/IP specification Date: Fri, 18 Aug 2000 16:52:24 -0700 Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0000_01C00934.AF2EC080" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0000_01C00934.AF2EC080 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Lifting from the FCoverIP specification, FCoverSCTP/IP could look even simpler. As example: ------=_NextPart_000_0000_01C00934.AF2EC080 Content-Type: text/plain; name="FCoverSCTP-IP.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="FCoverSCTP-IP.txt" FC over SCTP/IP Copyright (C) The Internet Society (1999). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Required Definitions: The Payload Protocol Identifier is TBD. All Payload Data types will be sent Unordered. Stream 0 is reserved as a control stream with TBD messages. Each FC port is designated a stream starting from 1. Each FC Frame is contained within a Payload Data chunk. Authentication Management using LDAP- TBD. FC Frame Format: The exact size of each frame varies depending on the size of the variable fields. The size of the variable field ranges from 0 to 2112-bytes as shown in the FC Frame Format in Fig. 1 resulting in the minimum size FC Frame of 36 bytes and the maximum size FC frame of 2148 bytes. +------+--------+-----------+----//-------+------+------+ | SOF |Frame |Optional | Frame | CRC | EOF | | (4B) |Header |Header | Payload | (4B) | (4B) | | |(24B) |<----------------------->| | | | | | Data Field = (0-2112B) | | | +------+--------+-----------+----//-------+------+------+ Fig. 1 FC Frame Format SOF and EOF Delimiters: On a FC link, SOF and EOF are called Ordered Sets and are sent as special out-of-band words constructed from the 10-bit comma character (K28.5) followed by 3 additional 10-bit data characters. On a non-Fibre Channel link the Start of Frame (SOF) and End of Frame (EOF) delimiters are both byte-encoded and 4-bytes long. On a FC link the SOF delimiter serves to identify the beginning of a frame and prepares the receiver for frame reception. The SOF contains information about the frame's Class of Service, position within a sequence, and in some cases, connection status. The EOF delimiter identifies the end of the frame and the final frame of a sequence. In addition, it serves to force the running disparity to negative. The EOF is used to end the connection in connection-oriented classes of service. It is therefore important to preserve the information conveyed by the delimiters across the IP-based network, so that the receiving FCIP device can correctly construct the FC frame in its original SOF and EOF format before forwarding it to its ultimate FC destination on the FC link. Start of Frame (SOF) and End of Frame (EOF) byte- encodings are defined in Annex A. Although, the SOF and EOF codes are 32-bits, the format makes use of a single-byte to represent each FC Ordered Set. Frame Header: The Frame Header is 24-bytes long and has several fields that are associated with the identification and control of the payload. Current FC Standards allow up to 3 Optional Header fields [4]: - Network_Header (16-bytes) - Association_Header (32-bytes) - Device_Header (up to 64-bytes). Frame Payload: The FC Frame Payload is transparent to the FCIP device. An FC application level payload is called an Information Unit at the FC-4 Level. This is mapped into the Frame Payload of the FC Frame. A large Information Unit is segmented using a structure consisting of FC Sequences. Typically, a Sequence consists of more than one FC frame. FCIP does not maintain any state information regarding the relationship of frames within a FC Sequence. CRC: The CRC is 4-bytes long and uses the same 32-bit polynomial used in FDDI and is specified in ANSI X3.139 Fiber Distributed Data Interface. When FC frames are encapsulated into IP packets, the CRC is untouched. APPENDIX A: Fibre Channel EOF and SOF Encodings A.1 Ordered Sets On a FC link, Ordered Sets (OS) are sent as special out-of-band words constructed of the 10-bit comma character (K28.5) followed by 3 additional 10-bit data characters. The Ordered Sets defined by FC include the Frame Delimiter, Start of Frame (SOF) and End of Frame (EOF), and other Primitive Signals. When FC frames are encapsulated in an IP packet, the Byte-encoded frame format is used. The Byte-encoded frame format uses 32-bit OS Code Words to represent valid FC frame delimiter. This format uses a single-byte OS Code to represent each FC Ordered Set. FC Over IP makes use of the OS Codes defined in Annex A of [7] for the frame delimiters. SOF and EOF codes defined in the figures (see below) in this Annex are inserted into the FC frame. Primitive Signals and Primitive Sequences are stripped at the FCIP boundary. The frame delimiters are identified by their position. An encapsulated Byte-encoded frame must use the corresponding 32-bit OS Code Word as the first and last words in the encapsulated PDU. FC frame delimiters shall be encoded in the format shown in Table below. Table 1. Frame Delimiter Format +---+----------------+----------------+----------------+-------------- + |Wrd| <31:24> | <23:16> | <15:08> | <07:00> | +---+----------------+----------------+----------------+-------------- + |0 | OS Code | Reserved | +---+----------------+----------------+----------------+-------------- + A.2 Encoded FC Frame Delimiters The SOF OS-codes are a single byte encoding of the SOF Ordered Set. The first word in an encapsulated Byte-encoded FC frame shall map the SOF Ordered Set to the corresponding 32-bit OS Code Word. The EOF OS-codes are a single byte encoding of the EOF Ordered Set. The last word in an encapsulated Byte-encoded FC frame shall map the EOF Ordered Set to the corresponding 32-bit OS Code Word. +-----------------+----------------+ | OS-Code | Delimiter Name | | (hex) | | +-----------------+----------------+ | 0x28 | SOFf | +-----------------+----------------+ | 0x3F | SOFc1 | +-----------------+----------------+ | 0x2F | SOFi1 | +-----------------+----------------+ | 0x37 | SOFn1 | +-----------------+----------------+ | 0x3D | SOFc2 | +-----------------+----------------+ | 0x2D | SOFi2 | +-----------------+----------------+ | 0x35 | SOFn2 | +-----------------+----------------+ | 0x3E | SOFc3 | +-----------------+----------------+ | 0x2E | SOFi3 | +-----------------+----------------+ | 0x36 | SOFn3 | +-----------------+----------------+ | 0x39 | SOFc4 | +-----------------+----------------+ | 0x29 | SOFi4 | +-----------------+----------------+ | 0x31 | SOFn4 | +-----------------+----------------+ | 0x38 | SOFcf | +-----------------+----------------+ | 0x30 | SOFnf | +-----------------+----------------+ | 0x41 | EOFn | +-----------------+----------------+ | 0x42 | EOFt | +-----------------+----------------+ | 0x46 | EOFdt | +-----------------+----------------+ | 0x44 | EOFrt | +-----------------+----------------+ | 0x49 | EOFni | +-----------------+----------------+ | 0x4E | EOFdti | +-----------------+----------------+ | 0x4F | EOFrti | +-----------------+----------------+ | 0x50 | EOFa | +-----------------+----------------+ [END] ------=_NextPart_000_0000_01C00934.AF2EC080-- From owner-ips@ECE.cmu.edu Sat Aug 19 01:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03459; 18 Aug 2000 20:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA08379 for ips-outgoing; Fri, 18 Aug 2000 20:15:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA08375 for ; Fri, 18 Aug 2000 20:15:29 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id CAA15576 for ; Sat, 19 Aug 2000 02:14:58 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.92) with SMTP id CAA37018 for ; Sat, 19 Aug 2000 02:14:58 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256940.00015BE5 ; Sat, 19 Aug 2000 02:14:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 19 Aug 2000 02:33:08 +0300 Subject: Re: Connection Consensus Progress Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I agree with Costa that this is very neat summary. However sessions are there for 2 purposes - parallelism and fail-over. Ordering is an by-product of several control connections. And data parallelism was still in iSCSI-00 (not that far back). All other arguments hold. 2 questions are still very much on my mind: J1- is data parallelism beyond the capability of a single connection (i.e. for a single transfer) a requirement (as was suggested on the list) J2 - is the symmetric connection handling making the design simpler or more complex I have no definite answer any of the two. iSCSI-00 assumed J1 - yes. iSCSI-01 assumes J1-no or the pplication will do "command striping" (not always feasible at the device). Julo csapuntz@cisco.com on 19/08/2000 00:39:35 Please respond to csapuntz@cisco.com To: Black_David@emc.com cc: ips@ece.cmu.edu, csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: Re: Connection Consensus Progress Hi David, > I've seen: > R1) Parallel transfers to/from and failover support for > tape devices. In contrast to disks, multiple SCSI > connections to the same tape do not work (e.g., > blocks can be written in the wrong order). > R2) Obtaining parallelism for a single SCSI command > across multiple transport connections using > different physical links. > R3) Obtaining parallelism for a single SCSI command > across multiple transport connections using the > same physical links. > R4) Optimize failure handling, so that a single TCP > connection loss doesn't immediately translate > into a SCSI error visible to higher level > (time-consuming) recovery logic. > R1) and R2) are beyond the capabilities of existing SCSI- > based systems (note that a parallel bus is a single link). > R3) and R4) are related to the use of TCP as a transport. That's a good summary. However, I'm a bit confused by what you mean by single SCSI command. iSCSI today doesn't obtain parallelism for a single SCSI command. A single SCSI command is executed over a single transport connection. Also, you don't need a session concept to get parallelism per se. You need it to get parallelism + ordered task queueing. > R3) needs more explanation, as TCP is known to be able > to saturate Gigabit Ethernet, given enough data to > transfer. Is the argument for R3) that for the > transfer sizes likely to be seen in iSCSI, TCP > spends enough of its time in slow start and the > like that multiple TCP connections gain performance? This is not the strongest argument, but: At 10 gigabits, one processor may not be able to saturate the link. Using multiple transport connections allows you to more readily use multiple processors. Note that these multiple processors can come in the form of either an SMP or multiple NICs. -Costa From owner-ips@ECE.cmu.edu Sat Aug 19 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03477; 18 Aug 2000 21:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA08584 for ips-outgoing; Fri, 18 Aug 2000 20:25:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA08578 for ; Fri, 18 Aug 2000 20:25:01 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id RAA00976 for ; Fri, 18 Aug 2000 17:25:00 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Fri, 18 Aug 2000 17:24:59 -0700 Message-ID: From: Jim McGrath To: ips@ece.cmu.edu Subject: TCP/IP over SCSI in silicon Date: Fri, 18 Aug 2000 17:24:58 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Can anyone direct me to a shipping product (silicon or box, preferably inexpensive) that has TCP/IP implemented in silicon (by that I mean the high speed main path - exceptions can always be handled in software). I'm interested in buying one to verify (for myself) the point that TCP/IP per se need not be a performance issue for storage. I'm also interested in unreleased products, although being unreleased they don't constitute as strong a proof positive of the proposition. Jim McGrath Director New Business Development Quantum Corporation From owner-ips@ECE.cmu.edu Sat Aug 19 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03867; 18 Aug 2000 22:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA09880 for ips-outgoing; Fri, 18 Aug 2000 21:23:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail03-oak.pilot.net (mail-oak-3.pilot.net [198.232.147.18]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA09872 for ; Fri, 18 Aug 2000 21:23:54 -0400 (EDT) Received: from msd-gw.hitachi.com (root@msd.hitachi.com [206.189.5.161]) by mail03-oak.pilot.net with ESMTP id SAA09865; Fri, 18 Aug 2000 18:22:51 -0700 (PDT) Received: from hicam-msd.hitachi.com (thunder [137.168.16.3]) by msd-gw.hitachi.com with ESMTP id SAA10668; Fri, 18 Aug 2000 18:18:00 -0700 (PDT) Received: from SANJAYS-DPC.hicam-msd.hitachi.com (sanjays-dpc [192.48.127.192]) by hicam-msd.hitachi.com with ESMTP id SAA06054; Fri, 18 Aug 2000 18:18:00 -0700 (PDT) Message-Id: <4.3.0.20000818182904.00b824d0@thunder> X-Sender: sanjays@thunder X-Mailer: QUALCOMM Windows Eudora Version 4.3 Date: Fri, 18 Aug 2000 18:29:48 -0700 To: Jim McGrath , ips@ece.cmu.edu From: Sanjay Sreenath Subject: Re: TCP/IP over SCSI in silicon In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk http://www.interprophet.com http://www.alacritech.com/home.html At 05:24 PM 8/18/00 -0700, Jim McGrath wrote: >Can anyone direct me to a shipping product (silicon or box, preferably >inexpensive) that has TCP/IP implemented in silicon (by that I mean the high >speed main path - exceptions can always be handled in software). I'm >interested in buying one to verify (for myself) the point that TCP/IP per se >need not be a performance issue for storage. > >I'm also interested in unreleased products, although being unreleased they >don't constitute as strong a proof positive of the proposition. > >Jim McGrath >Director >New Business Development >Quantum Corporation From owner-ips@ECE.cmu.edu Sat Aug 19 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03878; 18 Aug 2000 22:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA09743 for ips-outgoing; Fri, 18 Aug 2000 21:17:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA09738 for ; Fri, 18 Aug 2000 21:17:40 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id SAA12310; Fri, 18 Aug 2000 18:17:39 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.144.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id SAA24492; Fri, 18 Aug 2000 18:17:38 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id SAA28362; Fri, 18 Aug 2000 18:17:38 -0700 (PDT) Message-Id: <200008190117.SAA28362@ha10nwk.EBay.Sun.COM> Date: Fri, 18 Aug 2000 18:17:38 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: TCP/IP over SCSI in silicon To: ips@ece.cmu.edu, Jim.McGrath@quantum.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: te7YNmZYM1J/L+c61D3glQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > Can anyone direct me to a shipping product (silicon or box, preferably > inexpensive) that has TCP/IP implemented in silicon (by that I mean the high > speed main path - exceptions can always be handled in software). I'm > interested in buying one to verify (for myself) the point that TCP/IP per se > need not be a performance issue for storage. > > I'm also interested in unreleased products, although being unreleased they > don't constitute as strong a proof positive of the proposition. Try: http://www.alacritech.com/home.html Looks to be priced at $399 - $599 for 2 or 4 ports. Never used one so I can't comment on them. -David From owner-ips@ECE.cmu.edu Sat Aug 19 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05344; 19 Aug 2000 3:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA16283 for ips-outgoing; Sat, 19 Aug 2000 02:30:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (man-97-106.Reshall.Berkeley.EDU [169.229.97.106]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA16278 for ; Sat, 19 Aug 2000 02:30:15 -0400 (EDT) From: csapuntz@cisco.com Received: (from csapuntz@localhost) by csapuntz-u1.cisco.com (8.9.3/8.9.3/SuSE Linux 8.9.3-0.1) id XAA01206; Fri, 18 Aug 2000 23:29:32 -0700 To: ips@ece.cmu.edu Subject: Re: iSCSI failover and reconfiguration References: <4.2.2.20000818111834.00aa4ed0@pop.icubed.com> cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 18 Aug 2000 23:29:32 -0700 In-Reply-To: Mike Kazar's message of "Fri, 18 Aug 2000 12:00:27 -0400" Message-ID: Lines: 53 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Here's a little proposal I wrote up a while back but never posted: When the initiator attempts to login to a given Target, it gets a response: Redirect: domain.name/modifier The initiator would attempt to connect to the new domain name and modifier. The initiator would follow at most 7 redirects. The redirected name is presented to the machine you connect to in the Target: field of the login message. A similar response is Resolve Resolve: domain.name/modifier Again, the initator should try to connect to the new machine. In this case, the original name should be presented to the target. This mechanism allows you to build primitive name resolution into iSCSI. -Costa -Costa Mike Kazar writes: > In the iSCSI protocol document, a SCSI target is named by a URL, with the > assumption being that DNS would map the domain name to a set of IP > addresses. This works fine when the set of servers that might be > supporting the SCSI target is fixed for all time. > > In the NFS V4 specification, the protocol allows the migration of a file > system from one server to another, by allowing for an error code that > directs the initiator to a new location for that file system. > > I believe that this would be a useful feature to have in iSCSI as > well. With the addition of a distinctive error that indicates > responsibility for a particular target has been moved to a different URL, > migration of a SCSI target between IP hosts could be achieved in iSCSI, > just as migration of file system can be performed in NFS V4. > > Do other readers think this would be a good thing to support in iSCSI? > > Mike > (kazar@spinnakernet.com) From owner-ips@ECE.cmu.edu Sat Aug 19 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08675; 19 Aug 2000 16:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA07209 for ips-outgoing; Sat, 19 Aug 2000 15:27:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA07203 for ; Sat, 19 Aug 2000 15:27:38 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA04379; Sat, 19 Aug 2000 12:21:58 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA28041; Sat, 19 Aug 2000 12:19:24 -0700 From: "Douglas Otis" To: , Subject: RE: Connection Consensus Progress Date: Sat, 19 Aug 2000 12:29:44 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100EC3@corpmx9.isus.emc.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, As I am not convinced you are asking the correct questions, let me pose some of mine. Question E(sctp) Should use of TCP be changed to SCTP allowing for- . Headers contained within one frame. . Objects aligned at 32-bit boundaries. . Out of sequence frame processing. . Standard authentication. . Independent streams under common control. . Session restart (no multi-session binding required for reliability). . Improved error detection. . Prevention of blind spoofing and denial of service attacks. . Standard Heartbeat and multi-homing. (optional improvement for reliability) Add to this list or reorder by importance. Getting TCP done is half the battle with an object stream like SCSI. SCTP allows simple state-machines to handle this entire nightmare of obtaining performance. (The deciding factor in the end.) Perhaps they should rename SCTP to TOP for Transport Object Protocol or perhaps GTSSB Greatest Thing Since Sliced Bread. Question A(sctp) revised Should LUN resolve to stream? Question B(sctp) SCTP streams, session restarts, and switches obsolete Question B. Question C(sctp) Should systems connecting to Fibre-Channel retain FCP structures? Question D(sctp) If using FCP structures, should FC domain resolve to a stream? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > Sent: Friday, August 18, 2000 1:50 PM > To: ips@ece.cmu.edu > Subject: Connection Consensus Progress > > > > I understand that some folks want to have many TCP connections to a > storage > > entity, including at least one TCP/IP connection per LUNs. I > think it has > > been agreed by Julian that iSCSI will support that, with some extensions > > which he suggested. Even though most folks will not implement their > > Storage Controllers in that manor, I think that issue is > perhaps settled. > > Julian could perhaps restate the change that would make the TCP per LUN > > folks (if more then one) happy. This may permit us to really move off of > > point "A" (specified by David Black as "Should iSCSI require a TCP > > connection per LUN?"). If that is so, then I also submit that > we have not > > been focusing enough time on point "B" ("Should iSCSI have a session > > abstraction that binds multiple TCP connections into one iSCSI > connection?"). > > If I weren't on vacation and dealing with email sporadically, > I'd have sent email along the lines of what John Hufferd > wrote above. I think that the discussion of (A) has lead > to the rough consensus that iSCSI will not REQUIRE a TCP > connection per LUN. It is still possible to build systems > that use a connection per LUN (e.g., a collection of targets, > each of which only implements LUN 0, although initiators > will be REQUIRED to support multiple LUNs/target). The > consensus is a bit rough, but I think it is consensus. If > anyone disagrees (i.e., thinks that things are too rough > to be called consensus), please send me email directly. > > The session discussion (B) has been widening the scope > of possibilities. We need to start narrowing it. In > looking over the emails on this topic, there are a bunch > of intertwined issues. The original issue (B) was: > > (B) Should iSCSI have a session abstraction that > binds multiple TCP connections into one > iSCSI connection? > > For a "yes" answer to (B) we need a clear grasp of the > requirements that motivate multiple connections (i.e., > what problems does they address). So far, I think > I've seen: > R1) Parallel transfers to/from and failover support for > tape devices. In contrast to disks, multiple SCSI > connections to the same tape do not work (e.g., > blocks can be written in the wrong order). > R2) Obtaining parallelism for a single SCSI command > across multiple transport connections using > different physical links. > R3) Obtaining parallelism for a single SCSI command > across multiple transport connections using the > same physical links. > R4) Optimize failure handling, so that a single TCP > connection loss doesn't immediately translate > into a SCSI error visible to higher level > (time-consuming) recovery logic. > R1) and R2) are beyond the capabilities of existing SCSI- > based systems (note that a parallel bus is a single link). > R3) and R4) are related to the use of TCP as a transport. > > R3) needs more explanation, as TCP is known to be able > to saturate Gigabit Ethernet, given enough data to > transfer. Is the argument for R3) that for the > transfer sizes likely to be seen in iSCSI, TCP > spends enough of its time in slow start and the > like that multiple TCP connections gain performance? > > I should also note that the error handling procedures > for multiple connections will need to be specified > in complete detail -- I expect to see state machine > descriptions in the final version of the spec if > multiple connections are supported. > > Beyond this are the issues of what sort of multiple > connection model(s) to adopt if we decide to go that > route. I've seen at least three models proposed: > - All connections are equivalent, but all traffic > for a single SCSI command uses one connection. > - Single control connection, but data traffic can > be striped across multiple connections. > - LUNs are assigned to specific connections. > The third model is to some extent orthogonal to the > first two, and leads to another approach to building > systems with a single LUN/connection. > > Can I ask people to focus on the requirements issues > for (B)? For those in favor of multiple connections, > please indicate which of the requirements (R1-R4) > are important, or what additional requirements I've > left out. Those against should check that none of > R1-R4 are important enough to be requirements. > > I'm looking not only for Yes/No consensus on (B), > but also the corresponding discussion of R1-R4, which > I would expect to see reflected in a future version of > the iSCSI requirements draft. Consensus of this sort > is hard to take on the mailing list, so I'd like to > see more discussion focussed on the requirements > before I figure out how to go about establishing > consensus. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Mon Aug 21 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20245; 21 Aug 2000 12:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA20342 for ips-outgoing; Mon, 21 Aug 2000 10:41:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA20336 for ; Mon, 21 Aug 2000 10:41:24 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id HAA21178 for ; Mon, 21 Aug 2000 07:41:08 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Mon, 21 Aug 2000 07:41:05 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF02EEA07A@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Connection Consensus Progress Date: Mon, 21 Aug 2000 07:41:04 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com [mailto:Black_David@emc.com] wrote: > For a "yes" answer to (B) we need a clear grasp of the > requirements that motivate multiple connections (i.e., > what problems does they address). So far, I think > I've seen: > R1) Parallel transfers to/from and failover support for > tape devices. In contrast to disks, multiple SCSI > connections to the same tape do not work (e.g., > blocks can be written in the wrong order). > R2) Obtaining parallelism for a single SCSI command > across multiple transport connections using > different physical links. > R3) Obtaining parallelism for a single SCSI command > across multiple transport connections using the > same physical links. > R4) Optimize failure handling, so that a single TCP > connection loss doesn't immediately translate > into a SCSI error visible to higher level > (time-consuming) recovery logic. I think there is an additional proposed requirement, one which is satisfied by the current iSCSI draft: R5) Obtaining parallelism between multiple SCSI commands across multiple transport connections using different physical links. If I can't get R2, I'll settle for R5. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Tue Aug 22 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22182; 21 Aug 2000 19:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA11526 for ips-outgoing; Mon, 21 Aug 2000 18:46:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA11522 for ; Mon, 21 Aug 2000 18:46:49 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 8E4494E21 for ; Mon, 21 Aug 2000 15:46:48 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA10244; Mon, 21 Aug 2000 15:46:44 -0700 (PDT) Message-ID: <39A1B2D6.E9CC14DE@hp.com> Date: Mon, 21 Aug 2000 15:53:10 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Command Reference Number in the iSCSI draft Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello, A question about the CmdRN. How does the CmdRM must handled in the data PDUs in the case the initiator can't generate it? The initiator can't generate it for example in the case of several TCP connexions in a same session, each one using a different adapter. Which value to put in this field in this case, and how to say to the target that it is meaningless? Regards, Pierre From owner-ips@ECE.cmu.edu Tue Aug 22 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22187; 21 Aug 2000 19:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA11429 for ips-outgoing; Mon, 21 Aug 2000 18:42:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from falcon.vixel.com (mail.vixel.com [207.115.190.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA11424 for ; Mon, 21 Aug 2000 18:42:47 -0400 (EDT) Received: from vixel.com ([192.168.1.89]) by falcon.vixel.com (Netscape Messaging Server 4.15) with ESMTP id FZNZ3900.7BM; Mon, 21 Aug 2000 15:42:45 -0700 Message-ID: <39A1B06D.C99478C2@vixel.com> Date: Mon, 21 Aug 2000 15:42:53 -0700 From: "Marjorie Krueger" X-Mailer: Mozilla 4.61 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis CC: Ips Subject: Re: FCoverSCTP/IP specification References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Where have you interjected SCTP, other than the title? Seems like somethings missing from this proposal... Marjorie Krueger Vixel Corp. Douglas Otis wrote: > Lifting from the FCoverIP specification, FCoverSCTP/IP could look even > simpler. > As example: > > -------------------------------------------------------------------------------- > Name: FCoverSCTP-IP.txt > FCoverSCTP-IP.txt Type: Plain Text (text/plain) > Encoding: 7bit From owner-ips@ECE.cmu.edu Tue Aug 22 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22442; 21 Aug 2000 20:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA14380 for ips-outgoing; Mon, 21 Aug 2000 20:12:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA14374 for ; Mon, 21 Aug 2000 20:12:38 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id RAA04721 for ; Mon, 21 Aug 2000 17:06:34 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id RAA03050 for ; Mon, 21 Aug 2000 17:04:04 -0700 From: "Douglas Otis" To: "Ips" Subject: FCoverIP (FC/SCTP/IP) Date: Mon, 21 Aug 2000 17:14:49 -0700 Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0000_01C00B93.4FB90F00" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0000_01C00B93.4FB90F00 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Here is an updated version so less is missing... ------=_NextPart_000_0000_01C00B93.4FB90F00 Content-Type: text/plain; name="draft-ietf-ips-fc-sctp-ip-00.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="draft-ietf-ips-fc-sctp-ip-00.txt" IPS Working Group Douglas Otis Networks INTERNET-DRAFT SANlight (Expires February 21, 2001) FC over SCTP/IP (FC/SCTP/IP) Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC 2026 [1]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as Reference material or to cite them other than as "work in progress". The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/lid-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Copyright (C) The Internet Society (2000). All Rights Reserved. Normative References: RFC 826, Ethernet Address Resolution Protocol(ARP) RFC 792, Internet Control Message Protocol (ICMP) RFC 2132, DHCP Options and BOOTP Vendor Extensions (DHCP-BOOTP) RFC 2589, Lightweight Directory Access Protocol (LDAPv3) RFC 1157, Simple Network Management Protocol (SNMP) RFC 1907, Management Information Base for SNMP (SNMPv2-MIB) RFC "wip", Stream Control Transmission Protocol (SCTP) RFC "wip", LDAP structures for FC/SCTP/IP (LDAP/FC) Definitions: The Payload Protocol Identifier is TBD. All Payload Data types will be sent Unordered. Stream 0 is reserved as a control stream with TBD messages. TBD... Each FC port is designated a stream starting from 1. Each FC Frame is contained within a Payload Data chunk. Authentication Management is facilitated with LDAP/FC. Devices implementing FC/SCTP/IP will also implement ARP, ICMP and optionally DHCP-BOOTP, and SNMP. FC Frame Format: The exact size of each frame varies depending on the size of the variable fields. The size of the variable field ranges from 0 to 2112-bytes as shown in the FC Frame Format in Fig. 1 resulting in the minimum size FC Frame of 36 bytes and the maximum size FC frame of 2148 bytes. +------+--------+-----------+----//-------+------+------+ | SOF |Frame |Optional | Frame | CRC | EOF | | (4B) |Header |Header | Payload | (4B) | (4B) | | |(24B) |<----------------------->| | | | | | Data Field = (0-2112B) | | | +------+--------+-----------+----//-------+------+------+ Fig. 1 FC Frame Format SOF and EOF Delimiters: On a FC link, SOF and EOF are called Ordered Sets and are sent as special out-of-band words constructed from the 10-bit comma character (K28.5) followed by 3 additional 10-bit data characters. On a non-Fibre Channel link the Start of Frame (SOF) and End of Frame (EOF) delimiters are both byte-encoded and 4-bytes long. On a FC link the SOF delimiter serves to identify the beginning of a frame and prepares the receiver for frame reception. The SOF contains information about the frame's Class of Service, position within a sequence, and in some cases, connection status. The EOF delimiter identifies the end of the frame and the final frame of a sequence. In addition, it serves to force the running disparity to negative. The EOF is used to end the connection in connection-oriented classes of service. It is therefore important to preserve the information conveyed by the delimiters across the IP-based network, so that the receiving FCIP device can correctly construct the FC frame in its original SOF and EOF format before forwarding it to its ultimate FC destination on the FC link. Start of Frame (SOF) and End of Frame (EOF) byte- encodings are defined in Annex A. Although, the SOF and EOF codes are 32-bits, the format makes use of a single-byte to represent each FC Ordered Set. Frame Header: The Frame Header is 24-bytes long and has several fields that are associated with the identification and control of the payload. Current FC Standards allow up to 3 Optional Header fields [4]: - Network_Header (16-bytes) - Association_Header (32-bytes) - Device_Header (up to 64-bytes). Frame Payload: The FC Frame Payload is transparent to the FCIP device. An FC application level payload is called an Information Unit at the FC-4 Level. This is mapped into the Frame Payload of the FC Frame. A large Information Unit is segmented using a structure consisting of FC Sequences. Typically, a Sequence consists of more than one FC frame. FCIP does not maintain any state information regarding the relationship of frames within a FC Sequence. CRC: The CRC is 4-bytes long and uses the same 32-bit polynomial used in FDDI and is specified in ANSI X3.139 Fiber Distributed Data Interface. When FC frames are encapsulated into IP packets, the CRC is untouched. APPENDIX A: Fibre Channel EOF and SOF Encodings A.1 Ordered Sets On a FC link, Ordered Sets (OS) are sent as special out-of-band words constructed of the 10-bit comma character (K28.5) followed by 3 additional 10-bit data characters. The Ordered Sets defined by FC include the Frame Delimiter, Start of Frame (SOF) and End of Frame (EOF), and other Primitive Signals. When FC frames are encapsulated in an IP packet, the Byte-encoded frame format is used. The Byte-encoded frame format uses 32-bit OS Code Words to represent valid FC frame delimiter. This format uses a single-byte OS Code to represent each FC Ordered Set. FC Over IP makes use of the OS Codes defined in Annex A of [7] for the frame delimiters. SOF and EOF codes defined in the figures (see below) in this Annex are inserted into the FC frame. Primitive Signals and Primitive Sequences are stripped at the FCIP boundary. The frame delimiters are identified by their position. An encapsulated Byte-encoded frame must use the corresponding 32-bit OS Code Word as the first and last words in the encapsulated PDU. FC frame delimiters shall be encoded in the format shown in Table below. Table 1. Frame Delimiter Format +---+----------------+----------------+----------------+-------------- + |Wrd| <31:24> | <23:16> | <15:08> | <07:00> | +---+----------------+----------------+----------------+-------------- + |0 | OS Code | Reserved | +---+----------------+----------------+----------------+-------------- + A.2 Encoded FC Frame Delimiters The SOF OS-codes are a single byte encoding of the SOF Ordered Set. The first word in an encapsulated Byte-encoded FC frame shall map the SOF Ordered Set to the corresponding 32-bit OS Code Word. The EOF OS-codes are a single byte encoding of the EOF Ordered Set. The last word in an encapsulated Byte-encoded FC frame shall map the EOF Ordered Set to the corresponding 32-bit OS Code Word. +-----------------+----------------+ | OS-Code | Delimiter Name | | (hex) | | +-----------------+----------------+ | 0x28 | SOFf | +-----------------+----------------+ | 0x3F | SOFc1 | +-----------------+----------------+ | 0x2F | SOFi1 | +-----------------+----------------+ | 0x37 | SOFn1 | +-----------------+----------------+ | 0x3D | SOFc2 | +-----------------+----------------+ | 0x2D | SOFi2 | +-----------------+----------------+ | 0x35 | SOFn2 | +-----------------+----------------+ | 0x3E | SOFc3 | +-----------------+----------------+ | 0x2E | SOFi3 | +-----------------+----------------+ | 0x36 | SOFn3 | +-----------------+----------------+ | 0x39 | SOFc4 | +-----------------+----------------+ | 0x29 | SOFi4 | +-----------------+----------------+ | 0x31 | SOFn4 | +-----------------+----------------+ | 0x38 | SOFcf | +-----------------+----------------+ | 0x30 | SOFnf | +-----------------+----------------+ | 0x41 | EOFn | +-----------------+----------------+ | 0x42 | EOFt | +-----------------+----------------+ | 0x46 | EOFdt | +-----------------+----------------+ | 0x44 | EOFrt | +-----------------+----------------+ | 0x49 | EOFni | +-----------------+----------------+ | 0x4E | EOFdti | +-----------------+----------------+ | 0x4F | EOFrti | +-----------------+----------------+ | 0x50 | EOFa | +-----------------+----------------+ Full Copyright Statement Copyright (C) The Internet Society (2000). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. [END] ------=_NextPart_000_0000_01C00B93.4FB90F00-- From owner-ips@ECE.cmu.edu Tue Aug 22 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22366; 21 Aug 2000 20:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA12847 for ips-outgoing; Mon, 21 Aug 2000 19:25:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA12826 for ; Mon, 21 Aug 2000 19:25:39 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA04703; Mon, 21 Aug 2000 16:19:28 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA02920; Mon, 21 Aug 2000 16:16:58 -0700 From: "Douglas Otis" To: "Marjorie Krueger" Cc: "Ips" Subject: RE: FCoverSCTP/IP specification Date: Mon, 21 Aug 2000 16:27:42 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <39A1B06D.C99478C2@vixel.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Marjorie Krueger, As SCTP can establish a connection as server using a single Well-Known-Port, the next level of configuration can happen within stream 0 as a convention. (A nice feature for embedded adapters as there is less OS interaction.) Don't embed url stuff into these configuration structures but instead leave that to LDAP and then IT managers have tools to configure the equipment. This keeps real-time configuration quick using binary structures and you can fix configuration security issues without rewriting this spec as it would be a different spec describing the LDAP structures and how they are used. Setting up FC domains and attributes on streams would be the configuration process based on structures found within LDAP as well as defining SCTP authentication. Looking at SCTP should help with most details, as it is fairly complete. With your help I will continue this process as a group effort. Doug > -----Original Message----- > From: Marjorie Krueger [mailto:Marjorie.Krueger@Vixel.com] > Sent: Monday, August 21, 2000 3:43 PM > To: Douglas Otis > Cc: Ips > Subject: Re: FCoverSCTP/IP specification > > > Where have you interjected SCTP, other than the title? Seems > like somethings missing > from this proposal... > > Marjorie Krueger > Vixel Corp. > > Douglas Otis wrote: > > > Lifting from the FCoverIP specification, FCoverSCTP/IP could look even > > simpler. > > As example: > > > > > ------------------------------------------------------------------ > -------------- > > Name: FCoverSCTP-IP.txt > > FCoverSCTP-IP.txt Type: Plain Text (text/plain) > > Encoding: 7bit > From owner-ips@ECE.cmu.edu Tue Aug 22 01:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22461; 21 Aug 2000 20:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA14110 for ips-outgoing; Mon, 21 Aug 2000 20:04:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA14105 for ; Mon, 21 Aug 2000 20:04:50 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA11725 for ; Mon, 21 Aug 2000 17:04:49 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.152.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id RAA06906 for ; Mon, 21 Aug 2000 17:04:48 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id RAA10012; Mon, 21 Aug 2000 17:04:48 -0700 (PDT) Message-Id: <200008220004.RAA10012@ha10nwk.EBay.Sun.COM> Date: Mon, 21 Aug 2000 17:04:48 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: FCoverSCTP/IP specification To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: N9XhBCyRF4INOq30Adz47A== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk In reading the discussion of FC over SCTP there has been a lot of high level posturing but a distinct lack of details. Assuming we all have and understand both the FC specs and SCTP specs, there is a whole lot of details beyond just stating "run FC over SCTP". There are lots of interesting questions that need to be addressed, simple examples of which might be: 1) What level of FC is running over SCTP? FC-0 or FC-4 or something in between? Or just FCP? Or FCP-2? 2) Of the various FC service classes, which are supported and which are not? 3) Of the various Mode pages defined in SCSI and FCP, how are the values interpreted and what should the recommended values be? 4) If third part transfers are supported what is the addressing used? 5) How is login and authentication done? etc etc etc. [I really don't want to hear answers to these particular questions in response to this e-mail, they are but 5 of many dozens I can think of.] What I would like to see is the proponents of FCoverSCTP/IP take the time to write up a detailed draft that has enough information that someone could make an attempt to prototype an implementation. Without such a draft there is no purpose debating the theoretical aspects of an undefined protocol. The other IPS protocols, SEP, iSCSI and the FC tunneling encapsulation, have such detailed drafts that I can have a concrete discussion about. I believe FCoverSCTP/IP may have merit, but I can't know until I see details. So until I see a real draft with real details I can discuss, I am going to ignore any further discussion and strongly encourage others to do the same. -David From owner-ips@ECE.cmu.edu Tue Aug 22 02:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22900; 21 Aug 2000 22:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA16610 for ips-outgoing; Mon, 21 Aug 2000 21:30:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA16600 for ; Mon, 21 Aug 2000 21:30:32 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id SAA04748; Mon, 21 Aug 2000 18:24:21 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id SAA03255; Mon, 21 Aug 2000 18:21:45 -0700 From: "Douglas Otis" To: "David Robinson" , Subject: RE: FCoverSCTP/IP specification Date: Mon, 21 Aug 2000 18:32:30 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <200008220004.RAA10012@ha10nwk.EBay.Sun.COM> Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Any FC traffic could be encapsulated. You may wish to make a separate informative RFC as to how to use such an encapsulation to exchange IP over FC, FCP-x, etc. This could eventually include buffering and bandwidth management tricks, but with this separate RFC. Again, I would expect that to be other documents and likely ones created by T11 themselves. http://www.t11.org I think the major trick is to avoid crossing into T11 areas as their work is ongoing and would be a duplication of efforts to attempt to define some kind of subset. This spec should relate to the mundane interconnects, cross-links and the like. (Networking.) Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > David Robinson > Sent: Monday, August 21, 2000 5:05 PM > To: ips@ece.cmu.edu > Subject: RE: FCoverSCTP/IP specification > > > In reading the discussion of FC over SCTP there has been a lot of > high level posturing but a distinct lack of details. > > Assuming we all have and understand both the FC specs and SCTP specs, > there is a whole lot of details beyond just stating "run FC over SCTP". > > There are lots of interesting questions that need to be addressed, simple > examples of which might be: > > 1) What level of FC is running over SCTP? FC-0 or FC-4 or something > in between? Or just FCP? Or FCP-2? > > 2) Of the various FC service classes, which are supported and > which are not? > > 3) Of the various Mode pages defined in SCSI and FCP, how are the values > interpreted and what should the recommended values be? > > 4) If third part transfers are supported what is the addressing used? > > 5) How is login and authentication done? > > etc etc etc. > > [I really don't want to hear answers to these particular questions > in response to this e-mail, they are but 5 of many dozens I can think of.] > > What I would like to see is the proponents of FCoverSCTP/IP take the > time to write up a detailed draft that has enough information that > someone could make an attempt to prototype an implementation. Without > such a draft there is no purpose debating the theoretical aspects of an > undefined protocol. The other IPS protocols, SEP, iSCSI and the FC > tunneling encapsulation, have such detailed drafts that I can have a > concrete discussion about. I believe FCoverSCTP/IP may have merit, but > I can't know until I see details. > > So until I see a real draft with real details I can discuss, I am going > to ignore any further discussion and strongly encourage others to > do the same. > > -David > > From owner-ips@ECE.cmu.edu Tue Aug 22 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23134; 21 Aug 2000 23:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA17642 for ips-outgoing; Mon, 21 Aug 2000 22:10:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA17638 for ; Mon, 21 Aug 2000 22:10:02 -0400 (EDT) Received: from sunmail1.Sun.COM ([129.145.1.2]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id TAA07142; Mon, 21 Aug 2000 19:10:00 -0700 (PDT) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.84.31]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id TAA05033; Mon, 21 Aug 2000 19:10:00 -0700 (PDT) Received: from eng.sun.com (dsl196-127.Eng.Sun.COM [129.146.196.127]) by jurassic.eng.sun.com (8.11.0+Sun/8.11.0) with ESMTP id e7M29vK861053; Mon, 21 Aug 2000 19:09:58 -0700 (PDT) Message-ID: <39A1E0BD.A5894AEC@eng.sun.com> Date: Mon, 21 Aug 2000 21:09:01 -0500 From: Spencer Shepler Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.7 [en] (X11; U; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis CC: Ips Subject: Re: FCoverIP (FC/SCTP/IP) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > > Here is an updated version so less is missing... > > ... > Name: draft-ietf-ips-fc-sctp-ip-00.txt > draft-ietf-ips-fc-sctp-ip-00.txt Type: Plain Text (text/plain) > Encoding: 7bit Douglas Otis, As a document editor for the NFSv4 WG I would like to make a couple of friendly observations/suggestions. First, your draft is not at this point a WG document and therefore is mis-named. It is a personal draft and therefore should be named 'draft-otis-fc-sctp-ip-00.txt' or something like that. Second, if you have not already reviewed the "Guidelines to Authors of Internet-Drafts" (http://www.ietf.org/ietf/1id-guidelines.txt), I suggest that you do. Third, it is considered poor form to forward a draft directly to the WG, especially before it has been published on the IETF site. It is okay to announce the availability of a draft or after the draft name has been established to announce that you have submitted an updated version. Personal experience has shown the IETF draft publication process to be reasonably responsive. Give it a chance. Spencer From owner-ips@ECE.cmu.edu Tue Aug 22 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23339; 22 Aug 2000 0:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA19101 for ips-outgoing; Mon, 21 Aug 2000 23:08:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA19091 for ; Mon, 21 Aug 2000 23:08:24 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by atlrel2.hp.com (Postfix) with ESMTP id 90825516A; Mon, 21 Aug 2000 23:08:23 -0400 (EDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id UAA27995; Mon, 21 Aug 2000 20:08:22 -0700 (PDT) X-OpenMail-Hops: 1 Date: Mon, 21 Aug 2000 20:08:15 -0700 Message-Id: Subject: RE: RE: RE: Towards Consensus on TCP Connections MIME-Version: 1.0 To: somesh_gupta@hp.com, dotis@sanlight.net, ips@ece.cmu.edu, Jim.McGrath@quantum.com, julian_satran@il.ibm.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, Some people have already pointed out vendors with silicon based TCP/IP implementations. There are others who are in various stages of development, but they have to speak up for themselves. The capability of silicon implementations for networking has been increasing dramatically of late (and being very richly rewarded). Reasonably inexpensive switches implement very complex state machines for flow classification & prioritization, layer 3 forwarding, layer 2 forwarding etc etc. Though not the same as managing the TCP connection state, it demonstrates rapidly increasing capability. Since TCP is the transport protocol for all of web traffic, it is becoming as important a protocol as ip at the network layer and ethernet at the link level. I don't think anyone believes that iSCSI will succeed if it consumes a lot more CPU resources or the adapters cost a lot more. The competition is FC and it has to do at least as well. Regarding alternate transports, once you decide to operate in the internet, you have to handle retransmission/packet loss and congestion control. (BTW, where can I find a specification for SCTP). Other protocols may have framing advantages over TCP (which can be solved), but TCP is a protocol that is proven to work in the internet from years of experience. It is better to committ TCP to hardware than to committ some new protocol to h/w and find that it causes instability under some scenarios. Also, it would be a mistake to let an intermediate objective of FCP <-> iSCSI bridging drive the end goal. As others have pointed out, bridging will - to paraphrase from the switch book by Rich Siefter - give you the insersection of the capabilities while giving superset of problems. To summarize, 1. operating in the internet requires congestion control and recovery. 2. TCP is the most (only?) proven protocol for congestion control and recovery in the internet with strong momentum and investment 3. Putting FC on top will require an additional convergence layers. 4. The set of inter-operable management tools for IP networks & systems is very comprehensive. 5. Security and authentication tools (if used) are much stronger. I don't even know what it is in fibre-channel (my ignornance) other than "verifying" www-name. 6. Initial implementations of iSCSI may be bridges or native arrays. But as the silicon implementation becomes cheap and gigabit eithernet becomes "very cheap (already at low hundreds per port), you will see that in disk drives. (was that a summary?) Somesh > -----Original Message----- > From: Jim.McGrath@quantum.com [mailto:Jim.McGrath@quantum.com] > Sent: Thursday, August 17, 2000 9:07 PM > To: somesh_gupta@hp.com; dotis@sanlight.net; ips@ece.cmu.edu; > julian_satran@il.ibm.com > Cc: Jim.McGrath@quantum.com > Subject: FW: RE: RE: Towards Consensus on TCP Connections > > > > Observation spanning many emails: > > I think part of the problem here is that people have some different > experiences. For instance, you mentioned silicon based TCP/IP > implementations, and yet I personally (and I think a number > of other storage > focused people) are not familiar with them and their > characteristics (e.g. > availability, cost, etc...). > > Storage vendors have had to automate a lot of protocol in > silicon for years, > and this had had a pronounced influence on the evolution of > that protocol. > For instance, in both SCSI and Fibre Channel standards > activities you have a > lot of discussion as to whether something can be easily and > inexpensively > implemented in silicon - this is often a consideration that > weighs heavily > in the decision making. This is due to the combination of > lost cost and > high performance focus of the industry. Indeed, this focus > on "simple" > protocols (relying of good links, etc...) is I think a reason > why some folks > in the networking community have doubts concerning issues > like congestion > control for storage protocols or anything based on them. > > The networking industry is, as seem from a storage vendors > perspective, > almost the exact opposite. Its history has been dominated by software > implementations of (comparatively) slow transports on (comparatively) > expensive machines (normally computers of some sort). I think you are > seeing some doubts being expressed about the speed and cost of these > approaches, and whether they can be transplanted to the > storage world (which > is the focus for iSCSI). > > Personally, I'd like to hear most on cheap, fast, TCP/IP > implementations. > Afterall, if it really was cheap and fast to send FC out one > end on top of > TCP/IP and get it back on the other end, then I for one would > certainly > support that approach given its other benefits. > > Jim > > > -----Original Message----- > From: somesh_gupta@hp.com [mailto:somesh_gupta@hp.com] > Sent: Thursday, August 17, 2000 5:41 PM > To: dotis@sanlight.net; ips@ece.cmu.edu; julian_satran@il.ibm.com > Subject: RE: RE: Towards Consensus on TCP Connections > > > Doug, > > Tunneling FC through IP to connect FC islands together > is a good application and has its place, and an iSCSI <-> FCP > gateway will be an important app to connect the two worlds > together. However neither of these is a replacement for > using TCP/IP as a native transport for SCSI commands. > > FC networks appear to me to be limited scale networks providing > a very reliable link (channel) using b-b credits to avoid > dropping packets. Therefore any protocol using FC network makes > those assumptions. However this limits the scale, and also other > factors are driving the use of IP based networks. When FC goes over > an IP based networks, then either the assumptions (of a reliable > channel) are no longer true, or a convergence > layer has to be built to provide the same level of service (using > TCP and another sub-layer on top perhaps). > > Even in FC, you have SCSI states etc and FC states > (fairly involved protocol and does not have all the capabilities > as IP based networks provide - or at not yet at the same level > of interoperability and robustness). > > You have made a comment multiple times about the compexlity of > SCSI over TCP and complexity of states and demuxing etc. I somehow > don't get it (when I compare it to the fact that systems have been > managing loads of apps running on top of TCP and that switches and > NICs are implementing TCP in silicon). Perhaps you can give some > details of the state and demux complexity and error recovery > issues and we can explore them to see if there is something we are > missing or perhaps some of us can help with explaining why they > may not be issues. > > SCSI over TCP is probably more complex than FCP/FC. However, it should > provide more capabilities and a converged network. It should > also be simpler than encapsulating FC over IP (TCP). Also there > is far more expertise in this area and it is using existing > components except the iSCSI encapsulation which is fairly simple. I > don't think there is any unknowns in this area. So let us explore > and complexity issues in more detail. > > Somesh > > > -----Original Message----- > > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > > Sent: Wednesday, August 16, 2000 8:04 AM > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: FW: RE: Towards Consensus on TCP Connections > > > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Tuesday, August 15, 2000 10:06 PM > > > To: ips@ece.cmu.edu > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Doug, > > > > > > Please cool down and keep the discussion technical. I > > > personally do not enjoy bringing press quotes to a technical > > > discussion. And so much more when they are inaccurate. > > > > You indicated confusion as to the comparison being made. I tried to > > enlighten you as to the alternative. I did it with humor and > > with technical > > details. I see nothing inaccurate nor have you indicated errors. > > > > > Encapsulating FC ovr IP is not SCSI. Your basic assumption is > > > that FCP will be encapsulated on IP. FCP is tightly tied to FC > > > (as TCP is to IP) and has already a complex (and not yet closed) > > > recovery mechanism based on the assumption it runs over FC. > > > > An IP datagram is nothing more than a vehicle for FC > encapsulation and > > indeed includes only the defragmentation process. There is > no need to > > manage the defrag process or to be concerned about possible > > failure at the > > IP level. It is transparent and self healing as FCP-2 on FC > > handles these > > errors and would be no less reliable in doing so. There are > > few uses for FC > > beyond SCSI so your point escapes me especially when you are > > advocating a > > protocol wholly unsupported. Are you suggesting FC over IP > > will not work? > > > > > Moreover - like SCSI - FCP has the notion of a target distinct > > > from the notion of a LUN. It could be an acceptable solution to > > > extend the range of a FC island but even for this it has to go > > > some way in solving the congestion and security issues. > > > > There are already provisions for security for FC over IP. In many > > applications, bandwidth is controlled by a dedicated channel > > which solves > > both issues. There are many means for flow control where TCP > > is but one. > > Reliability should be the differentiating factor. > > > > > IMHO even for this application a FCP-iSCSI gateway could > > provide a better > > > solution as the mapping is straightforward. > > > > > > Julo > > > > This provides few techincal details as to how you base this > > opinion. If I > > had the choice of purchasing a blade for a switch or new > > system adapters > > with high overhead and new SAN controllers with limited > > connectivity where I > > could also expect massive faults, I would not see that as > > straightforward. > > But then again, maybe I would. > > > > Doug > > > > > > > "Douglas Otis" on 15/08/2000 19:24:34 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > Julo, > > > > > > For details on proposed Fibre-Channel over IP which does > not impose > > > additional catastrophic error states as does iSCSI, please see: > > > > > > > > > http://search.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt > > > > > > "While the FC over IP specification is independent of the > link level > > > transport protocol, it assumes a high bandwidth, high > > > reliability, low loss > > > link level transport such as Gigabit Ethernet, SONET, ATM, > > or DWDM. This > > > specification treats all classes of FC frames the same -- > > as datagrams." > > > ipfc > > > > > > "Fibre Channel over IP's main advantage is it needs no > > > modifications to the > > > storage subsystem or the server operating system. SCSI over > > TCP/IP, on the > > > other hand, requires filter drivers so the server operating > > system can do > > > IP-network-to-SAN emulation (for which I humbly propose > the acronym > > > INSANE)." Wayne Rickard > > > Network World, 06/12/00 wayne@gadzoox.com > > > > > > To see who is doing Fibre-Channel encapsulation see: > > > > > http://www.cisco.com/warp/public/146/pressroom/2000/jun00/sp_0 > > 61300b.htm > > > > > > At keeping the same architecture as the present Fibre-Channel > > > configuration, > > > you have not improved reliability. By making compromises > > that combine > > > states, you are reducing reliability. It would appear, in > > efforts to > > > reduce > > > overhead in TCP transport, you have added dangerous > > overhead at the SCSI > > > emulation layer. This is taking advantage of TCP in > > hardware? Allow a > > > session per LUN, (remove the LUN field) then there is no danger > > > of inducing > > > an error across multiple devices when sorting millions of > > tags for a LUN > > > value. Care about reliability. > > > > > > Doug > > > > > > > Doug, > > > > > > > > You can use the standard at the end user device, if you so wish. > > > > If you choose a "transprent" controller - i.e. one not holding > > > > any state or > > > > ordering - and > > > > view every LU as and independent unit - you ignore the > > numbering (that > > > you > > > > are free to do anyway for the commands) and ship the commands to > > > > the device > > > > for execution. > > > > > > > > Initiator tags are unique per initiator and it should not be a > > > problem to > > > > keep the unique > > > > per session. > > > > > > > > Target task tags are a different issue. We decided to > > remove the LUN > > > field > > > > in both the incoming RTT and the the outgoing data to > > avoid the need to > > > > test for inconsistencies. > > > > But as this is the only thing that implies unique target task > > > > tags we might > > > > as well "reinstate" the LUN (the field is still unused) > > in the DATA > > > packet > > > > (in the RTT it is implied by the initiator > > > > task tag). > > > > > > > > Relaxing the requirement of uniqueness for initiator tag > > is possible too > > > > (you can have the unique per LU) but that might complicate > > > > unecessarily the > > > > initiator. > > > > > > > > If this (minor) change does not anoy target controller > > implementors and > > > > nobody sees something else wrong with it we might > > introduce it in the > > > next > > > > version. > > > > > > > > As for your FC claims - I still can't follow your argument. Are > > > > you talking > > > > about FC (the network protocol) or FCP-2 (SCSI-over-FC)? > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 14/08/2000 20:24:29 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > Keep the interface suitable for the end device. A controller in > > > > the middle > > > > does not justify a unique interface for that scenario. > > In trying to > > > > optimize "Bandwidth utilization", you are binding > > thousands of logical > > > > units > > > > together into the same session. This is an added burden > > within the > > > > standard, if used at the device. In doing so, session > > tags must be kept > > > > unique across all logical units. Reset all drives to recover > > > from even a > > > > single tag error? If you wish to cache information at > > the gateway, it > > > > should still use the interface suitable for the end > > device. At least > > > this > > > > isolates error handling. > > > > > > > > By creating a simple FC tunnel, the controller remains > > bolted into the > > > > back-plane where it is known to work effectively. Error > > handling is > > > > understood with a stateless translation and you have not created > > > > a standard > > > > that will set back efforts at getting a solution without myopic > > > > compromises. > > > > There WILL be errors not seen by this "In the Middle Controller" > > > > translating > > > > Fibre-Channel into iSCSI as FC does not have the same > > transport. States > > > > within this ITMC adds to complexity and error migration when you > > > > intertwine > > > > logical units. ITMC is a poor starting point and only > > makes for a bad > > > > interface as seen by this specification. > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Saturday, August 12, 2000 12:24 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Well - I understand the argument about a connection/LU > > and I did even > > > > > implement it (as many others - see Paul's reply). You > > can aggregate at > > > > the > > > > > TCP level but you have to aggregate somewhere > > > > > to use effectively the bandwidth. Eliminating the CU is > > not a concern > > > of > > > > > this group and not a suggestion to be accepted lightly by the > > > community > > > > > (who will do caching, storage virtualization and management > > > etc.) and I > > > > am > > > > > confident most of the participants on this list do not > > want to discuss > > > > > this subject. FC over IP as replacement for iSCSI is a poor > > > > suggestion - > > > > > FC is a networking protocol and IP too. FCP over IP > is a gateway > > > > solution. > > > > > > > > > > Julo > > > > > > > > > > "Douglas Otis" on 12/08/2000 02:47:37 > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > > cc: > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > > > An architecture that scales does not need to provide > the entire > > > > bandwidth > > > > > from a single device. If these devices are each handling 25 > > > Mbytes per > > > > > second, then 2 of them exceed Fibre-Channel. Next week > > you could use > > > 4, > > > > > and > > > > > the week after that 8. The advantage of using a > > network without any > > > > > practical architectural restriction with intelligence > > at the client. > > > On > > > > > the > > > > > other hand, you want to design a single mid-point gateway to > > > handle the > > > > > entire bandwidth. To what end? You argue it can not > process a > > > > > TCP session > > > > > per device. You will not be tracking mid-stream errors as > > > this is more > > > > > costly. You add to the burden of tracking the state of > > the individual > > > > > device with this mid-stream state machine requiring > > additional sorting > > > > due > > > > > to this merged protocol without taking advantage of TCP > > to aid this > > > > > process. > > > > > For your type of solution, a simple hardware based > > tunnel would be > > > > better. > > > > > Do not include handling of the encapsulated protocol > > and at least it > > > > > becomes > > > > > understandable and more likely to manage the task. In other > > > > > words, make it > > > > > Fibre-Channel over IP and you have a chance with your > > architecture. > > > > > > > > > > The point to my statements was to indicate the device is able > > > to handle > > > > an > > > > > IP interface today using Fast Ethernet as the rate from > > the drive is > > > > > relatively low. Not to warm the cockles of marketing > > pointing to > > > > latency, > > > > > but at least they sell more drives overcoming this problem > > > using scale. > > > > > > > > > > Doug > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > julian_satran@il.ibm.com > > > > > > Sent: Friday, August 11, 2000 2:17 PM > > > > > > To: ips@ece.cmu.edu > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > > > > > I am not sure that I agree with your architecture statements > > > > but I like > > > > > to > > > > > > play with numbers (as most of the fellows engineers on this > > > > > list probably > > > > > > do). What would be in your opinion > > > > > > reasonable requirements for command and data transfer > > rates for the > > > > next > > > > > > 3-7 years? > > > > > > > > > > > > I would like to decouple that discussion from > > architecture - data > > > > > > rates can > > > > > > scale even in a shared > > > > > > architecture as mainframe channels have shown for years. > > > > > > > > > > > > I would rather like to understand if we can meet the > > data rates with > > > > > > reasonable latency. > > > > > > > > > > > > Julo > > > > > > > > > > > > "Douglas Otis" on 11/08/2000 21:14:13 > > > > > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > > > > > To: "Stephen Byan" , > > ips@ece.cmu.edu > > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Today's drives can deliver 320 Mbits/second of data > > on the outside > > > > > > cylinders. Improvement of the mechanics comes at a > high price > > > > > > with respect > > > > > > to power and cost. The cost/volume trend takes us > to a single > > > > > disk which > > > > > > increases access time as read channel data rate increases. > > > > By offering > > > > > > scaled throughput using more drives where each > > drive's interface > > > > > bandwidth > > > > > > is restricted with respect to read channel data rates > > > > provides a system > > > > > > with > > > > > > uniform and superior performance. The advantage of such an > > > > approach is > > > > > > found with respect to smaller random traffic. With > > more devices, > > > > > > redundancy > > > > > > is easily achieved and parallel access offers a means > > of performance > > > > > > improvement by spreading activity over more devices. The > > > > > switch provides > > > > > > bandwidth aggregation and is not found in the > > individual device. > > > > > > > > > > > > An 8ms access + latency figure in the high cost > > drives restricts > > > > > > the number > > > > > > of 'independent' operations that average 64k byte to 100 per > > > > > second or 52 > > > > > > Mbit per second. Such an architecture of > 'restricted' drives > > > > > would scale > > > > > > whereas the solicitated burst approach does not. An > > > > > independent nexus at > > > > > > the LUN is the only design that offers required scaling and > > > > > configuration > > > > > > flexibility. Keeping up with the read channel is a > > wasted effort. > > > In > > > > > > time, > > > > > > 1 Gbit Ethernet will be the practical solution about > > the time drives > > > > are > > > > > 1 > > > > > > inche in size. Several Fast Ethernet disks combined at a 1 > > > > > Gbit Ethernet > > > > > > client makes sense in cost, performance, capacity, > > reliability, and > > > > > > scalability at this point in time. The protocol > > overhead should be > > > > > > addressed. There are substantial improvements to be made to > > > > allow this > > > > > > innovation using standard adapters. > > > > > > > > > > > > The power cost to use copper 1 Gbit is high. > > Firewire does not > > > > > scale and > > > > > > has a limited reach. Firewire also places > > scatter/gather on the > > > drive > > > > > > together with direct access. Doing such over a WAN > > will impose > > > > > > significant > > > > > > changes. Serial ATA is nothing more than IDE through > > a SERDES. > > > > > The read > > > > > > channel data rate is like a drug, just say no. It is hard > > > not to buy > > > > > > enough > > > > > > dram to allow a proper buffer these days. Serial ATA > > removes all > > > > > buffers. > > > > > > Intel is just usurping any remaining electronics at > > the cost of > > > > > > sensitivity > > > > > > to a near by cell phone. Fewer drives with less > > > electronics. What a > > > > > good > > > > > > idea? > > > > > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > > Of > > > > > > > Stephen Byan > > > > > > > Sent: Friday, August 11, 2000 7:07 AM > > > > > > > To: 'ips@ece.cmu.edu' > > > > > > > Subject: RE: Towards Consensus on TCP Connections > > > > > > > > > > > > > > > > > > > > > Stephen Bailey [mailto:steph@cs.uchicago.edu] wrote: > > > > > > > > > > > > > > > The gating factor for whether iSCSI succeeds is > not going > > > > to be 200 > > > > > > > > MB/s instead of 100 MB/s out of a single LUN. > > > > > > > > > > > > > > In general, I agree. iSCSI can succeed in the high and > > > > > midrange storage > > > > > > > market without link aggregation for a single LUN. These > > > markets can > > > > > > afford > > > > > > > 10 Gb/s links. > > > > > > > > > > > > > > As a disk device level interface, iSCSI will not > > succeed unless > > > > > > > it offers at > > > > > > > least 2 Gb/s by around 2002, at very low cost for > the link. > > > > Note that > > > > > > even > > > > > > > Serial ATA starts at 1.5 Gb/s in 2001. Take a look > > at the Serial > > > ATA > > > > > > speed > > > > > > > roadmap on slide 16 of Intel's Serial ATA > > presentation at WinHEC: > > > > > > > http://serialata.org/F9pp.pdf. > > > > > > > > > > > > > > One can argue the technical merits, but from a marketing > > > > > > > viewpoint, the disk > > > > > > > industry (both suppliers and customers) has long held the > > > view that > > > > > > > interface speeds need to match the media data rate. > > iSCSI can try > > > > > > > to make an > > > > > > > argument that slower speeds are technically > > adequate, but this > > > > > > > will increase > > > > > > > the barriers to establishing iSCSI as a device interface. > > > > > > > > > > > > > > > If iSCSI works at ALL in a cost effective way > that can be > > > > > implemented > > > > > > > > in a disk, there'll be wild dancing in the streets and > > > > > you'll all (or > > > > > > > > maybe your companies will) be rich beyond the > > dreams of avarice. > > > > > > > > > > > > > > > > The easier you can make it for the > implementors, the more > > > > likely it > > > > > > > > will succeed. > > > > > > > > > > > > > > Disk drive companies have implemented much more complex > > > > > interfaces than > > > > > > > iSCSI and TCP - e.g. fibre channel arbitrated loop. > > And multiple > > > TCP > > > > > > > connections don't look very hard to implement. They > > just look like > > > a > > > > > > wart. > > > > > > > But I think a necessary one. > > > > > > > > > > > > > > Regards, > > > > > > > -Steve > > > > > > > > > > > > > > Steve Byan > > > > > > > > > > > > > > Design Engineer > > > > > > > MS 1-3/E23 > > > > > > > 333 South Street > > > > > > > Shrewsbury, MA 01545 > > > > > > > (508)770-3414 > > > > > > > fax: (508)770-2604 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Tue Aug 22 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23344; 22 Aug 2000 0:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA19371 for ips-outgoing; Mon, 21 Aug 2000 23:20:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA19366 for ; Mon, 21 Aug 2000 23:20:30 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id XAA05205; Mon, 21 Aug 2000 23:20:04 -0400 (EDT) Date: Mon, 21 Aug 2000 23:20:04 -0400 (EDT) From: Scott Bradner Message-Id: <200008220320.XAA05205@newdev.harvard.edu> To: somesh_gupta@hp.com Subject: RE: RE: RE: Towards Consensus on TCP Connections Cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Precedence: bulk > BTW, where can I find a specification for SCTP the approved version of SCTP is the Internet Draft draft-ietf-sigtran-sctp-13.txt - which you can get off the IETF web page (www.ietf.org) Scott From owner-ips@ECE.cmu.edu Tue Aug 22 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24651; 22 Aug 2000 4:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA24669 for ips-outgoing; Tue, 22 Aug 2000 03:02:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA24665 for ; Tue, 22 Aug 2000 03:02:31 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA170496 for ; Tue, 22 Aug 2000 09:01:59 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA24050 for ; Tue, 22 Aug 2000 09:01:58 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256943.00269F7E ; Tue, 22 Aug 2000 09:01:51 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 22 Aug 2000 09:59:56 +0300 Subject: RE: FCoverSCTP/IP specification Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, You are right. And, as was already pointed out, we could hardly consider the one page outline as a proposal (or could we?). Julo David Robinson on 22/08/2000 03:04:48 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: FCoverSCTP/IP specification In reading the discussion of FC over SCTP there has been a lot of high level posturing but a distinct lack of details. Assuming we all have and understand both the FC specs and SCTP specs, there is a whole lot of details beyond just stating "run FC over SCTP". There are lots of interesting questions that need to be addressed, simple examples of which might be: 1) What level of FC is running over SCTP? FC-0 or FC-4 or something in between? Or just FCP? Or FCP-2? 2) Of the various FC service classes, which are supported and which are not? 3) Of the various Mode pages defined in SCSI and FCP, how are the values interpreted and what should the recommended values be? 4) If third part transfers are supported what is the addressing used? 5) How is login and authentication done? etc etc etc. [I really don't want to hear answers to these particular questions in response to this e-mail, they are but 5 of many dozens I can think of.] What I would like to see is the proponents of FCoverSCTP/IP take the time to write up a detailed draft that has enough information that someone could make an attempt to prototype an implementation. Without such a draft there is no purpose debating the theoretical aspects of an undefined protocol. The other IPS protocols, SEP, iSCSI and the FC tunneling encapsulation, have such detailed drafts that I can have a concrete discussion about. I believe FCoverSCTP/IP may have merit, but I can't know until I see details. So until I see a real draft with real details I can discuss, I am going to ignore any further discussion and strongly encourage others to do the same. -David From owner-ips@ECE.cmu.edu Tue Aug 22 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24643; 22 Aug 2000 4:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA24582 for ips-outgoing; Tue, 22 Aug 2000 02:59:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA24578 for ; Tue, 22 Aug 2000 02:59:34 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA24696 for ; Tue, 22 Aug 2000 08:59:02 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA57732 for ; Tue, 22 Aug 2000 08:59:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256943.002658A3 ; Tue, 22 Aug 2000 08:58:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 22 Aug 2000 09:56:52 +0300 Subject: Re: Command Reference Number in the iSCSI draft Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, CmdRN is supposed to be maintained across the adapters (a central counter) and either handed by the adapter driver to the adapter or by a read-modify-store by the adapters. For outgoing data blocks there is no need to increment CmdRN as those can be always replayed from host memory either all or through RTT. If the initiator sends without RTT and wants to recover by sending from where it stopped it has to keep track of data sent by himself and rely on the target to request missing pieces. I however wonder if in would not be wise to give-up on this optimization and have the initiator resend all data at recovery. Regards, Julo Pierre Labat on 22/08/2000 01:53:10 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Command Reference Number in the iSCSI draft Hello, A question about the CmdRN. How does the CmdRM must handled in the data PDUs in the case the initiator can't generate it? The initiator can't generate it for example in the case of several TCP connexions in a same session, each one using a different adapter. Which value to put in this field in this case, and how to say to the target that it is meaningless? Regards, Pierre From owner-ips@ECE.cmu.edu Tue Aug 22 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27262; 22 Aug 2000 14:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA23322 for ips-outgoing; Tue, 22 Aug 2000 12:54:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA23316 for ; Tue, 22 Aug 2000 12:54:14 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id JAA14973 for ; Tue, 22 Aug 2000 09:54:10 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.142.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id JAA11331 for ; Tue, 22 Aug 2000 09:54:10 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id JAA28846; Tue, 22 Aug 2000 09:54:09 -0700 (PDT) Message-ID: <39A2AF02.CDBA1032@ebay.sun.com> Date: Tue, 22 Aug 2000 09:49:07 -0700 From: David Robinson X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: FCoverSCTP/IP specification References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > Any FC traffic could be encapsulated. You may wish to make a separate > informative RFC as to how to use such an encapsulation to exchange IP over > FC, FCP-x, etc. This could eventually include buffering and bandwidth > management tricks, but with this separate RFC. Again, I would expect that > to be other documents and likely ones created by T11 themselves. > http://www.t11.org > > I think the major trick is to avoid crossing into T11 areas as their work is > ongoing and would be a duplication of efforts to attempt to define some kind > of subset. This spec should relate to the mundane interconnects, > cross-links and the like. (Networking.) This is just an assertion of the principle anything can be engineered which I doubt anyone would disagree with. The devil is in the details and we have no details. By analogy running IP over FC is just a matter of encapuslating of a few RFCs over existing T11 docs, if it was that easy then why has the IPoverFC WG been working for over a year, and in fact a WG at all? >From what you seem to be asserting this WG has no purpose as the solution is trivially obvious. Most of us disagree. -David P.S. Now I will do as I promised and ignore any further discussion without a draft. From owner-ips@ECE.cmu.edu Tue Aug 22 22:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28113; 22 Aug 2000 17:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA02890 for ips-outgoing; Tue, 22 Aug 2000 16:29:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA02885 for ; Tue, 22 Aug 2000 16:29:47 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Tue, 22 Aug 2000 16:29:10 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100EDC@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: Final Pittsburgh Minutes Date: Tue, 22 Aug 2000 16:29:00 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk IP Storage BOF minutes Summer IETF 2000 Pittsburgh, PA Wednesday, August 2, 2000 --------------------------- Co-chairs: Steve Bellovin (smb@research.att.com) and David Black (black_david@emc.com) Area directors: Allison Mankin (mankin@isi.edu) and Scott Bradner (sob@harvard.edu) Attendance: ~100 people NOTE on consensus: There are several places in the minutes in which "consensus in the room" is noted. These items can be re-opened on the mailing list (preferably by those who were not at the meeting), especially if there are important technical aspects that are not reflected in the minutes. In the absence of list discussion, the WG co-chairs will assume that the "consensus in the room" is the WG consensus. Agenda bashing - David Black, co-chair -------------- There were no comments on the agenda. Area director - Scott Bradner, Transport AD ------------- Intellectual property rights: If there is intellectual property in anything discussed in the WG, the person discussing it is responsible for saying that there is intellectual property involved. If what someone says is going to be constrained by intellectual property rights, they shouldn't say it, especially if they're not prepared to describe the resulting restrictions. A lot of work on iSCSI has been done in a very focused and very dedicated design team. From now on, this is a WG effort. Most of the work should be done on the WG's mailing list in order to: - minimize interim meetings - minimize phone calls - discuss issues on a common mailing list accessible to all Charter status and bashing - David Black, co-chair -------------------------- Concerns were raised about emphasis of TCP as the protocol. The charter should not and currently does not limit itself to TCP. Instead, it states a requirement for congestion control. Congestion control without TCP is possible, the ECM working group and SCTP are examples. Scott Bradner reminded the group that the charter comes from the IESG and IAB. This group is only suggesting a charter to the IESG/IAB. It was agreed that all specific references to Gigabit Ethernet should be removed from the charter (editor's note: there weren't any). The question was raised about why FC/IP may be worked on by ips. The answers were: similar application domains, same people interested in both technologies, maybe even similar network traffic patterns. The question was raised of how a charter and a requirements document differ. Scott Bradner answered that a charter is commands from the IESG to the working group, i.e., it's the general structure of what the working group is going to do and what it won't do. A requirements document lays out requirements for the area of investigations. It is expected to be longer, it is expected to go into detail, doesn't prescribe a specific solution, and is expected to be published as an RFC. iSCSI requirements - Randy Haagens, Hewlett-Packard ------------------ iSCSI is a native SCSI transport on IP, not FCP on top of IP. TCP may be deficient for iSCSI use in a couple of areas. The TCP reassembly buffer is inefficient at higher speeds. The 16-bit checksum seems insufficient. These problems need to be addressed. A concern was raised that using multiple TCP connections adds a lot of complexity to the protocol that is not present in other IETF protocols, and that this is premature optimization. Randy responded that storage bandwidth needs (esp. at high end) will always outrun speeds on a single link, thus the need to use multiple links through the fabric (which multiple TCP connection enables). There was also a brief discussion of whether to use a TCP connection per LUN - the primary reason not to is scalability concerns for large storage controllers with lots of LUNs. Suggestions were made by a number of people that the requirements document should focus on the required functionality, not the required implementation. The requirements presentation spent time talking about the SMU concept from the SCSI Architecture Manual-2. Ralph Weber, secretary of T10, and technical editor of SAM-2, pointed out that SAM-2 is changing significantly in that area. T10 is also working on lifting the 64-bit limit on the size of target names. The need for multiple connections to a single LUN was questioned, since a LUN will not be able to use more than one network link (a gigabit link). Several people pointed out that this is not true, since many LUNs are backed by RAM, which can source data at full link rates. An issue was raised about whether gigabit performance should be required. It was agreed that the requirements should allow a gigabit or more, but not require it, and it was noted that the charter explicitly mentions small devices. ------------------ Question for consensus: All we have available is the 16-bit checksum. looking for direction on whether 16-bit checksum is adequate, or should be requirements that there will be strong checksum? Consensus in the room: Need to investigate checksums stronger than 16-bits. SCTP was noted as having a 32-bit checksum. iSCSI spec - Julian Satran, IBM Research - Haifa ---------- General overview and summary of iSCSI. See the document for details. The issue of using multiple connections came up again. After some discussion, it was agreed to take this issue to the list for discussion and resolution. An issue was raised about whether iSCSI's error recovery on dropped TCP connections is necessary, as some other have no such error recovery scheme. FCP (the Fibre Channel encapsulation of SCSI) has discovered the hard way that it needs error recovery for dropped frames, which is now part of the FCP-2 effort. On the other hand, a parallel SCSI bus makes no attempt to recover from breaking the physical connection. It was pointed out that since iSCSI uses TCP, it has fewer cases to deal with than FCP, since it must only deal with connection drops, not packet loss. More discussion on the list is needed on error recovery, including scenarios that may require the target to drop data that TCP has successfully delivered (e.g., where the TCP connection breaks before the ACK or response is delivered and hence the initiator retries the operation via some other connection). There was some confusion on SCSI's ordering requirements. SCSI requires commands to be delivered to the target in the same order as the initiator issued them. However, the order in which the target executes the commands is also controlled by the queuing attributes on the device and command. There is an inconsistent use of words like should, may, and can in the document. For example, the ping response appears to be optional as currently worded. This is wrong and needs to be corrected. In general, the authors of the document should review RFC-2119 for definitions of terms like MUST, SHOULD, and MAY and use them appropriately to clarify the requirements on implementations. Security Discussion - led by Steve Bellovin, co-chair ------------------- The discussion started by asking about the need for encryption: It was generally stated by the audience that authentication and integrity seem most important. Speed is also a requirement, as initial link speeds of a gigabit are envisioned. Steve Bellovin noted that encryption generally runs faster than authentication algorithms and that there were encryption algorithms doing a gigabit in hardware already. It was noted that it is difficult to turn on IPSec in the middle of a connection. Consensus call -------------- There was a consensus call eventually. Should encryption be mandatory to implement? The consensus in the room was that encryption should not be mandatory to implement. A lot of people were concerned about the overhead that implementation would imply on cheap devices. Security negotiation is required so that a device that doesn't implement encryption can say so in an understandable fashion. There was some discussion that secure negotiation was highly desirable, to prevent a man-in-the-middle from causing the security negotiation to result in no security. It was noted that most application of iSCSI expect a secure interconnect. Also, secure interconnects are least likely to be available at the low end, where people have the least ability to pay for a secure network. ------- Steve Bellovin asked the group about authentication. "Whom are you authenticating? Is it bi-directional or uni-directional? How many parties does the SCSI device have to authenticate. How do you administer this?" The AD noted that the WG did not have the option of not making the implementation of authentication non-mandatory. It was suggested that the host OS be authenticated and that the authentication should be bi-directional. Web servers could be used for administration. It was pointed out that the web server doesn't scale to large installations and that LDAP could be used to configure a group of machines. It was also pointed out that disks may not have room for a large number of keys and ACLs, which may necessitate the use of an authentication server. At the other end of the spectrum, it was observed that storage arrays are in many cases more powerful than the servers they connect to. It was decided that it is feasible have good random numbers from disks. Steve Bellovin asked if we can do public key calculations at login time? The general consensus in the room is that even most disks have enough CPU power to do this at login time. A hard disk has about as much CPU as a cell phone, which is capable of doing public key calculations. The issue of providing customer support for badly configured security mechanism came up. How do you deal with a user that has lost his keys? Add back-door passwords? There were no good answers in the meeting. Consensus call -------------- Should public keys be mandatory for login authentication? (About half of the people voted for and half against) Scott Bradner suggested that this issue be remitted to the mailing list. Steve Bellovin pointed out that it's much harder to do something for security for UDP than TCP. TCP is easier because there's a steady, in-order context. For UDP, one have to do something like IPSec. Other parting words: There's no confidentiality without integrity checking. A separate session key is probably needed for each TCP connection. For some choices of crypto algorithms (e.g. RC4), it is important to make sure that you NEVER reuse the same key, as key reuse enables some relatively easy attacks on the crypto. ---------------- There was no iSCSI MIB discussion due to insufficient time. FC-over-IP - Elizabeth Rodriguez, Lucent ------------------ It has not been decided where FC-over-IP will be worked on in the IETF. The ipfc WG's time slot will be used to discuss FC-over-IP, but it is not an official work item of the ipfc WG. FC-over-IP is a gateway-to-gateway protocol. Multiple gateways can talk amongst themselves. This was described as complementary to iSCSI. See the draft for current technical details. Scott Bradner asked for input over about where the FC-over-IP work should be done. Send Scott mail privately (sob@harvard.edu) or on the ips mailing list. There was some discussion of a 10km limit in FibreChannel. Standard Fibre Channel has a 10km limit on a single link based on emitter and detector characteristics. The actual protocol timeouts are several seconds which should go allow the protocol to go much further. One important timeout is R_A_TOV (resource allocation time-out value), maximum packet lifetime in a Fibre Channel fabric, which is in the seconds. Default time-out values in fibre-channel fabrics are between 1-10 seconds. Should be good for fairly long hauls. SEP - Paul von Stamwitz, Adaptec ---------------- SEP work is progressing. The one LUN per connection limitation has been removed and the packet format is being changed to resemble packetized SCSI. iSCSI/VI - Constantine Sapuntzakis, Cisco --------------------- This was a quick presentation to set up a bar/dinner BOF for further discussion. Note that VI is currently not an official work item for the ips WG. Protocol stack overhead is too high for IP-based protocols. To reduce the number of copies/page flips, the NIC must place data in the correct place in memory the first time. One way for the NIC to do this is to parse the upper-layer protocol. Another way is to add a generic RDMA protocol which works with multiple upper-layer protocols. VI is one good way of doing an RDMA abstraction. By running iSCSI over VI, we can leverage this generic infrastructure. From owner-ips@ECE.cmu.edu Wed Aug 23 13:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32233; 23 Aug 2000 9:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA07365 for ips-outgoing; Wed, 23 Aug 2000 08:03:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pop3.icubed.com (pop3.icubed.com [208.0.145.4]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA07098 for ; Wed, 23 Aug 2000 07:51:18 -0400 (EDT) Received: from kazar-ppp.fore.com (du200p91.icubed.com [204.215.200.91]) by pop3.icubed.com (8.10.2/8.10.2) with SMTP id e7NBqMw18762; Wed, 23 Aug 2000 07:52:22 -0400 (EDT) Message-Id: <3.0.6.32.20000823073511.007ab100@pop.icubed.com> X-Sender: kazar@pop.icubed.com X-Mailer: QUALCOMM Windows Eudora Light Version 3.0.6 (32) Date: Wed, 23 Aug 2000 07:35:11 -0400 To: ips@ece.cmu.edu, csapuntz@cisco.com From: Mike Kazar Subject: Re: iSCSI failover and reconfiguration Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk The problem with making this part of the login operation is that this type of redirection needs to happen at other times aside from login. In general, what makes this type of data movement most useful is if it can happen without regard to whether the data is being accessed. Most generally, a client might have a set of N commands outstanding when a data move occurs, and of these N commands, K of them would succeed, and N-K of them would return some standard error indicating that further requests would be handled elsewhere. Perhaps the same redirect and resolve strings could be used, but returned by a specific text command instead of during the login process, so that the redirection could happen at any time. Thus, a client, after having received this distinguished "Data moved" error code, would send a text command with "SendRedirectResolve:yes" and the server would then send back a text response of the forms "Redirect: domain/" or "Resolve:domain/". Mike (kazar@spinnakernet.com) > > >Here's a little proposal I wrote up a while back but never posted: > >When the initiator attempts to login to a given Target, >it gets a response: > >Redirect: domain.name/modifier > >The initiator would attempt to connect to the new domain name and >modifier. The initiator would follow at most 7 redirects. > >The redirected name is presented to the machine you connect to in >the Target: field of the login message. > >A similar response is Resolve > >Resolve: domain.name/modifier > >Again, the initator should try to connect to the new machine. >In this case, the original name should be presented to the target. > >This mechanism allows you to build primitive name resolution into iSCSI. > >-Costa > > > From owner-ips@ECE.cmu.edu Wed Aug 23 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00409; 23 Aug 2000 12:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA15066 for ips-outgoing; Wed, 23 Aug 2000 11:22:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from proxy1.it.stsj.seagate.com (ns1.seagate.com [204.160.183.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA15025 for ; Wed, 23 Aug 2000 11:21:18 -0400 (EDT) From: Ayman.M.Ghanem@seagate.com Received: (from smap) by proxy1.it.stsj.seagate.com id PAA28411 for ; Wed, 23 Aug 2000 15:20:47 GMT Received: from auth0.it.stsj.seagate.com(192.168.109.16) by proxy1.it.stsj.seagate.com via smap (V5.0) id xma027804; Wed, 23 Aug 00 15:19:39 GMT Received: from sv-gw1.stsv.seagate.com (sv-gw1.stsv.seagate.com [134.204.14.95]) by auth0.it.stsj.seagate.com with ESMTP id PAA21106 for ; Wed, 23 Aug 2000 15:19:39 GMT Subject: iSCSI: Question about CID To: ips@ece.cmu.edu Date: Wed, 23 Aug 2000 10:19:31 -0500 Message-ID: X-MIMETrack: Serialize by Router on SV-GW1/Seagate Internet(Release 5.0.2a (Intl)|23 November 1999) at 08/23/2000 08:19:39 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I have a question about the CID field sent by the initiator in a login message. When the target sends back login accepted, how does the initiator know which login request was accepted (in case it wants to establish multiple connections per session with each connection requiring a login)?. The login response does not return the CID to the initiator. -Ayman From owner-ips@ECE.cmu.edu Wed Aug 23 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00842; 23 Aug 2000 14:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA20729 for ips-outgoing; Wed, 23 Aug 2000 13:28:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ipperformance.com (cooper.gilhooley.com [207.8.92.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA20719 for ; Wed, 23 Aug 2000 13:28:32 -0400 (EDT) From: rbg@ipperformance.com Received: from rbg.gilhooley.com (rbg [10.1.0.66]) by ipperformance.com (8.9.3/8.9.3) with ESMTP id MAA46607; Wed, 23 Aug 2000 12:28:30 -0500 (CDT) (envelope-from rbg@ipperformance.com) Received: from localhost (localhost [127.0.0.1]) by rbg.gilhooley.com (8.9.3/8.9.3) with ESMTP id MAA23773; Wed, 23 Aug 2000 12:27:51 -0500 (CDT) (envelope-from rbg@ipperformance.com) Date: Wed, 23 Aug 2000 12:27:51 -0500 (CDT) Message-Id: <20000823.122751.48516241.rbg@ipperformance.com> To: Ayman.M.Ghanem@seagate.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Question about CID In-Reply-To: References: X-Mailer: Mew version 1.95b43 on Emacs 20.4 / Mule 4.0 (HANANOEN) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I'm passing it back at the same offset (8) in the login reply. Robert.. > On Wed, 23 Aug 2000 10:19:31 -0500 you said: > I have a question about the CID field sent by the initiator in a login > message. > When the target sends back login accepted, how does the initiator know > which > login request was accepted (in case it wants to establish multiple > connections > per session with each connection requiring a login)?. The login response > does > not return the CID to the initiator. > > -Ayman > _____________________________________________________________ Robert Gordon rbg@ipperf.com IP Performance, Inc Austin, Texas. From owner-ips@ECE.cmu.edu Wed Aug 23 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01112; 23 Aug 2000 15:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA23110 for ips-outgoing; Wed, 23 Aug 2000 14:24:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.5]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA23104 for ; Wed, 23 Aug 2000 14:24:41 -0400 (EDT) Received: by apollo.pirus.com with Internet Mail Service (5.5.2650.21) id ; Wed, 23 Aug 2000 14:35:32 -0400 Message-ID: <618471D08ABDD31188B6009027E500933844CC@apollo.pirus.com> From: "Henderson, Steve" To: "'rbg@ipperformance.com'" , Ayman.M.Ghanem@seagate.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Question about CID Date: Wed, 23 Aug 2000 14:35:30 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Since the login response is returned on the same connection as the login command came in on, the initiator knows that its the login on that connection that is being accepted ( or rejected ). Passing the CID back in the login reply at offset 8 is not what the RFC defines ( although it doesn't seem an unreasonable idea). Since that place in the login response is reserved, a very picky initiator implementation might object that the field contained a non zero value. Steve -----Original Message----- From: rbg@ipperformance.com [mailto:rbg@ipperformance.com] Sent: Wednesday, August 23, 2000 1:28 PM To: Ayman.M.Ghanem@seagate.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Question about CID I'm passing it back at the same offset (8) in the login reply. Robert.. > On Wed, 23 Aug 2000 10:19:31 -0500 you said: > I have a question about the CID field sent by the initiator in a login > message. > When the target sends back login accepted, how does the initiator know > which > login request was accepted (in case it wants to establish multiple > connections > per session with each connection requiring a login)?. The login response > does > not return the CID to the initiator. > > -Ayman > _____________________________________________________________ Robert Gordon rbg@ipperf.com IP Performance, Inc Austin, Texas. From owner-ips@ECE.cmu.edu Wed Aug 23 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01107; 23 Aug 2000 15:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA24708 for ips-outgoing; Wed, 23 Aug 2000 14:56:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ipperformance.com (cooper.gilhooley.com [207.8.92.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA24703 for ; Wed, 23 Aug 2000 14:56:37 -0400 (EDT) From: rbg@ipperformance.com Received: from rbg.gilhooley.com (rbg [10.1.0.66]) by ipperformance.com (8.9.3/8.9.3) with ESMTP id NAA46803 for ; Wed, 23 Aug 2000 13:56:35 -0500 (CDT) (envelope-from rbg@ipperformance.com) Received: from localhost (localhost [127.0.0.1]) by rbg.gilhooley.com (8.9.3/8.9.3) with ESMTP id NAA23986 for ; Wed, 23 Aug 2000 13:55:56 -0500 (CDT) (envelope-from rbg@ipperformance.com) Date: Wed, 23 Aug 2000 13:55:56 -0500 (CDT) Message-Id: <20000823.135556.88492817.rbg@ipperformance.com> To: ips@ece.cmu.edu Subject: Re: iSCSI: Question about CID In-Reply-To: <618471D08ABDD31188B6009027E500933844CC@apollo.pirus.com> References: <618471D08ABDD31188B6009027E500933844CC@apollo.pirus.com> X-Mailer: Mew version 1.95b43 on Emacs 20.4 / Mule 4.0 (HANANOEN) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I guess I missed that in the July 10 draft. I had assumed that the reply might come back on one of the other connections, but on reflection that was rather silly of me :-) Robert.. > On Wed, 23 Aug 2000 14:35:30 -0400 you said: > Since the login response is returned on the same connection as the login > command came in on, the initiator knows that its the login on that > connection that is being accepted ( or rejected ). > > Passing the CID back in the login reply at offset 8 is not what the RFC > defines ( although it doesn't seem an unreasonable idea). Since that place > in the login response is reserved, a very picky initiator implementation > might object that the field contained a non zero value. > > Steve > > > -----Original Message----- > From: rbg@ipperformance.com [mailto:rbg@ipperformance.com] > Sent: Wednesday, August 23, 2000 1:28 PM > To: Ayman.M.Ghanem@seagate.com > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI: Question about CID > > > > I'm passing it back at the same offset (8) in the login reply. > > Robert.. > > > On Wed, 23 Aug 2000 10:19:31 -0500 you said: > > I have a question about the CID field sent by the initiator in a login > > message. > > When the target sends back login accepted, how does the initiator know > > which > > login request was accepted (in case it wants to establish multiple > > connections > > per session with each connection requiring a login)?. The login response > > does > > not return the CID to the initiator. > > > > -Ayman > > > _____________________________________________________________ > Robert Gordon rbg@ipperf.com > IP Performance, Inc > Austin, Texas. _____________________________________________________________ Robert Gordon rbg@ipperf.com IP Performance, Inc Austin, Texas. From owner-ips@ECE.cmu.edu Wed Aug 23 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01881; 23 Aug 2000 18:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA01231 for ips-outgoing; Wed, 23 Aug 2000 17:17:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from prodweb.crossroads.com ([63.237.99.242]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id RAA01226 for ; Wed, 23 Aug 2000 17:17:16 -0400 (EDT) Received: by www.crossroads.com with Internet Mail Service (5.5.2650.21) id ; Wed, 23 Aug 2000 16:22:39 -0700 Message-ID: <670C86291BFCD311AFE500508B94D4B6630BF4@hqmail1.crossroads.com> From: Robert Reynolds To: "'ips@ece.cmu.edu'" Subject: Latest IScsi draft Date: Wed, 23 Aug 2000 14:19:23 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Where can I get a copy of the latest ISCSI draft? <><><><><><><><><><><><><><><> Robert Reynolds Crossroads Systems Inc. (512) 928-7222 (512)928-7401 (fax) robertr@crossroads.com From owner-ips@ECE.cmu.edu Thu Aug 24 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02484; 23 Aug 2000 21:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA06718 for ips-outgoing; Wed, 23 Aug 2000 19:54:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA06714 for ; Wed, 23 Aug 2000 19:54:46 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Wed, 23 Aug 2000 19:54:07 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100EF5@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI: Session Consensus/Plan Date: Wed, 23 Aug 2000 19:54:07 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Since traffic on this has died down, we may be close to consensus. Let me try to state what I believe the consensus is and propose some means for making further progress ... There were four possible requirements for a session abstraction: R1) Tape parallelism and failover R2) Stripe data: one command, multiple links R3) Stripe data: one command, same links R4) Recover from TCP failure without SCSI failure Steve Byan proposed: R5) Stripe commands and data: multiple links, ok to restrict each command and associated data to a single link I had not listed that, because if Tape (R1) is factored out then load balancing and failover across the links involved is achievable for disk above the SCSI level via multiple SCSI connections. In contrast, use of multiple links to tape must be below the SCSI level (i.e., in the transport). The original four requirements appear to fall into three categories: 1) I have seen no objections to R1) [Tape parallelism and failover]. Therefore I believe rough consensus exists for a session abstraction that bundles multiple transport connections into a single iSCSI connection, but that support for multiple connections/session is OPTIONAL. It would be ok to delete failover from this consensus for the reason discussed in the next paragraph. 2) R2) [Stripe data: one command, multiple physical links] and R4) [Recover from TCP failure without SCSI failure] require a detailed design to evaluate whether they should be requirements. In particular, R4) has been the focal point of the "keep it simple" objections to the session concept, and the complexity of achieving R2) depends on the details of the session design (what data and/or commands are allowed on which connections when). 3) Costa has characterized R3) [Stripe data: one command, same physical links] and generalizations involving multiple TCP connections over the same link as weak, and I've seen no disagreement. In the absence of strong support for R3), I believe rough consensus exists to reject it as a requirement. The final session abstraction may be able to do this, but will not be REQUIRED to do so. In summary: R1) would be the major reason to require sessions, R2) and R4) are potential goals for the selection/design of the session abstraction, and R3) is not a requirement. Anyone who disagrees should comment on the list. It would be fine to re-raise the complexity objection to sessions when the complete design including error handling is done, but re-raising it at this juncture is probably not productive. Based on the above consensus to have a session abstraction, the next issue is to determine what that should be. I think we're going to need two parallel efforts on this, because in addition to the previous possible session models ...: - All connections are equivalent, but all traffic for a single SCSI command uses one connection. - Single control connection, but data traffic can be striped across multiple connections. - LUNs are assigned to specific connections. ... a new one has been proposed: - Use SCTP instead of TCP and employ SCTP's support for multiple paths to the same destination. SCTP's multi-path support appears to be primarily designed for failover, but there are hooks that appear to make it usable by load balancing logic that resides above SCTP. We need further discussion of SCTP, as in the discussion to date I have seen neither a comprehensive design for encapsulation of either FC or SCSI in SCTP, nor have I seen any showstopper to the use of SCTP. Comments are hereby solicited ... Determination/design of the session abstraction for iSCSI over TCP should go on in parallel with the SCTP discussion, as that should continue to expose particular requirements of SCSI encapsulation, such as how transparent recovery from errors needs to work. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Aug 24 03:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03121; 23 Aug 2000 23:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA10347 for ips-outgoing; Wed, 23 Aug 2000 22:08:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA10338 for ; Wed, 23 Aug 2000 22:08:30 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id WAA57924; Wed, 23 Aug 2000 22:06:31 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id UAA153346; Wed, 23 Aug 2000 20:08:29 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: Session Consensus/Plan -- SCTP To: Black_David@emc.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 23 Aug 2000 19:03:59 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/23/2000 08:08:28 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk The objection that I have to SCTP has to do with getting HW support, and normal customer acceptance. I have the following points: 1. The NIC industry is rapidly moving to supporting TCP/IP offload and acceleration of OS native TCP/IP Stacks. This is being done by major vendors making major investments in ASICs etc. they are NOT building SCTP Hardware. They have business cases that support TCP/IP in hardware, and we wanted to exploit this by having iSCSI ride on this wave. 2. SCTP is a non proven technology from a robust and load situation. It may or may not be great, but we can not afford to find out about a "oops" situation in load, stability, scaling, etc. as we try to deploy it in vast numbers. 3. We will need to get the Switch and Router folks to also support SCTP and to do it in Hardware so that the speed and throughput can be maintained. I do not see that happening, at least not right away, when we need to get volumes moving in order to validate the IP storage SAN concept. 4. I do not think that we can afford to put together the marketing effort that will be needed to convince customers that SCTP is a great thing, at the same time that we are trying to convince them that an IP SAN is the right answer. Having Two new concept sales points is going to divide the focus and confuse the issues. It would be much better to focus on the concept of an IP SAN that supports the well known and understood TCP/IP, to which the customer feels comfortable, and for which they have various management tools. 5. Because of point 1-3 above, I think we will have a hard time getting the customers to accept SCTP, since there will not be the HW and therefore the performance we need, when we need it. . . . John L. Hufferd From owner-ips@ECE.cmu.edu Thu Aug 24 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06800; 24 Aug 2000 12:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA10188 for ips-outgoing; Thu, 24 Aug 2000 12:05:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.5]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA10182 for ; Thu, 24 Aug 2000 12:04:56 -0400 (EDT) Received: by apollo.pirus.com with Internet Mail Service (5.5.2650.21) id ; Thu, 24 Aug 2000 12:15:58 -0400 Message-ID: <618471D08ABDD31188B6009027E500932715C1@apollo.pirus.com> From: "Ferrari, Stephen" To: "'ips@ece.cmu.edu'" Subject: Data in SCSI Response or SCSI Data Date: Thu, 24 Aug 2000 12:15:56 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk For successful SCSI commands where the target sends data to the host, there appear to be several combinations of data and response that are allowed: * Data sent in a SCSI Data packet and status sent in a SCSI Response packet * Data and status sent in a SCSI Data packet, using the (S) bit * Data and status sent in a SCSI Response packet Is the target free to choose any of these, at its option, requiring the initiator to handle all of them? Is there a requirement that some commands (say, READ) use the first or second method, and other commands (say, INQUIRY) use the third? Is the data in a successful SCSI Response packet exactly equivalent to data in a SCSI Data packet, with an assumed offset of zero? I couldn't find the answer in the July 10th draft. If someone could point me at it, I'd appreciate it. Thanks, Stephen Ferrari Pirus Networks From owner-ips@ECE.cmu.edu Thu Aug 24 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06802; 24 Aug 2000 12:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA09838 for ips-outgoing; Thu, 24 Aug 2000 11:55:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA09827 for ; Thu, 24 Aug 2000 11:55:24 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Thu, 24 Aug 2000 11:54:49 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100EF6@corpmx9.isus.emc.com> To: hufferd@us.ibm.com Cc: ips@ece.cmu.edu Subject: iSCSI: SCTP Switch and Router support Date: Thu, 24 Aug 2000 11:54:46 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk John, > 3. We will need to get the Switch and Router folks to also support SCTP and > to do it in Hardware so that the speed and throughput can be maintained. I > do not see that happening, at least not right away, when we need to get > volumes moving in order to validate the IP storage SAN concept. I don't understand this. Most switches and routers have no knowledge of whether TCP, UDP, or even IPsec encapsulated (and hence opaque) traffic is flowing through them. What's special about SCTP -- is this about layer 4 and higher switches/routers? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Aug 24 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06819; 24 Aug 2000 13:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA10527 for ips-outgoing; Thu, 24 Aug 2000 12:13:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA10510 for ; Thu, 24 Aug 2000 12:12:53 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA05293; Thu, 24 Aug 2000 09:06:07 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA12207; Thu, 24 Aug 2000 09:03:38 -0700 From: "Douglas Otis" To: "John Hufferd/San Jose/IBM" Cc: "Ips" , "Elizabeth G Rodriguez \(Elizabeth\)" Subject: RE: iSCSI: Session Consensus/Plan -- SCTP (FCoverIP) Date: Thu, 24 Aug 2000 09:14:51 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk John, The journey of a thousand miles begins with the first step. http://www.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-00.txt In conjunction with http://www.ietf.org/internet-drafts/draft-ietf-sigtran-sctp-13.txt My apologies for not following proper procedures previously. Problems to overcome implementating a new protocol for routers and switches that look beyond IP layers should be of the same order of difficulty as the IP datagram protocol initially proposed for FCoverIP. At least the expensive fragmentation is handled within the SCTP protocol to ensure a normal router does not see a burden. The remaining work is simply to configure the channels. (Flipping streams on buried protocols, splitting and merging streams to match technologies at opposite ends etc.) I would hope to limit this to simple structures with a common header to establish revision level negotiations. Any ideas? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > John Hufferd/San Jose/IBM > Sent: Wednesday, August 23, 2000 7:04 PM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI: Session Consensus/Plan -- SCTP > > > The objection that I have to SCTP has to do with getting HW support, and > normal customer acceptance. I have the following points: > > 1. The NIC industry is rapidly moving to supporting TCP/IP offload and > acceleration of OS native TCP/IP Stacks. This is being done by major > vendors making major investments in ASICs etc. they are NOT building SCTP > Hardware. They have business cases that support TCP/IP in > hardware, and we > wanted to exploit this by having iSCSI ride on this wave. > > 2. SCTP is a non proven technology from a robust and load situation. It > may or may not be great, but we can not afford to find out about a "oops" > situation in load, stability, scaling, etc. as we try to deploy it in vast > numbers. > > 3. We will need to get the Switch and Router folks to also > support SCTP and > to do it in Hardware so that the speed and throughput can be > maintained. I > do not see that happening, at least not right away, when we need to get > volumes moving in order to validate the IP storage SAN concept. > > 4. I do not think that we can afford to put together the marketing effort > that will be needed to convince customers that SCTP is a great thing, at > the same time that we are trying to convince them that an IP SAN is the > right answer. Having Two new concept sales points is going to divide the > focus and confuse the issues. It would be much better to focus on the > concept of an IP SAN that supports the well known and understood > TCP/IP, to > which the customer feels comfortable, and for which they have various > management tools. > > 5. Because of point 1-3 above, I think we will have a hard time > getting the > customers to accept SCTP, since there will not be the HW and > therefore the > performance we need, when we need it. > > . > . > . > John L. Hufferd > From owner-ips@ECE.cmu.edu Thu Aug 24 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07048; 24 Aug 2000 13:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA11654 for ips-outgoing; Thu, 24 Aug 2000 12:44:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA11647 for ; Thu, 24 Aug 2000 12:44:45 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id MAA15654; Thu, 24 Aug 2000 12:42:46 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id KAA68554; Thu, 24 Aug 2000 10:44:43 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: SCTP Switch and Router support To: Black_David@emc.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Thu, 24 Aug 2000 09:19:01 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/24/2000 10:44:43 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David, A lot of what you say is true, however, the word "Most" is the key. There are a number of Switches, NATs, Proxies, and Load Balancers that actually care about TCP/IP. I do not believe they will handle the SCTP in a transparent manor. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 Black_David@emc.com on 08/24/2000 08:54:46 AM To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu Subject: iSCSI: SCTP Switch and Router support John, > 3. We will need to get the Switch and Router folks to also support SCTP and > to do it in Hardware so that the speed and throughput can be maintained. I > do not see that happening, at least not right away, when we need to get > volumes moving in order to validate the IP storage SAN concept. I don't understand this. Most switches and routers have no knowledge of whether TCP, UDP, or even IPsec encapsulated (and hence opaque) traffic is flowing through them. What's special about SCTP -- is this about layer 4 and higher switches/routers? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Aug 24 18:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07291; 24 Aug 2000 14:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA13904 for ips-outgoing; Thu, 24 Aug 2000 13:41:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tornado.jni.com ([204.118.127.253]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA13890 for ; Thu, 24 Aug 2000 13:41:29 -0400 (EDT) Received: by TORNADO with Internet Mail Service (5.5.2650.21) id ; Thu, 24 Aug 2000 10:38:33 -0700 Message-ID: <59D68F695E95D31193040090279AA94FAD1D4F@TORNADO> From: Swami Viswanathan To: ips@ece.cmu.edu Subject: RE: iSCSI: SCTP Switch and Router support Date: Thu, 24 Aug 2000 10:38:32 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with John on this one.... The intermediate switches/routers do need a valid (known) protocol field in the IP header. If they don't like it, its quite possible that they may drop the packet.. This also has got to do with filtering of unknown protocols... Also if the protocol is TCP/UDP, to support QOS and flow, most of the intermediate switches will even look at the TCP/UDP ports - (Layer 4 classification)... My 2 cents >> -----Original Message----- >> From: Black_David@emc.com [mailto:Black_David@emc.com] >> Sent: Thursday, August 24, 2000 8:55 AM >> To: hufferd@us.ibm.com >> Cc: ips@ece.cmu.edu >> Subject: iSCSI: SCTP Switch and Router support >> >> >> John, >> >> > 3. We will need to get the Switch and Router folks to also >> support SCTP >> and >> > to do it in Hardware so that the speed and throughput can >> be maintained. >> I >> > do not see that happening, at least not right away, when >> we need to get >> > volumes moving in order to validate the IP storage SAN concept. >> >> I don't understand this. Most switches and routers >> have no knowledge of whether TCP, UDP, or even IPsec >> encapsulated (and hence opaque) traffic is flowing >> through them. What's special about SCTP -- is this >> about layer 4 and higher switches/routers? >> >> --David >> >> --------------------------------------------------- >> David L. Black, Senior Technologist >> EMC Corporation, 42 South St., Hopkinton, MA 01748 >> +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 >> black_david@emc.com Cellular: +1 (978) 394-7754 >> --------------------------------------------------- >> From owner-ips@ECE.cmu.edu Thu Aug 24 18:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07296; 24 Aug 2000 14:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA13116 for ips-outgoing; Thu, 24 Aug 2000 13:22:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA13109 for ; Thu, 24 Aug 2000 13:22:36 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by atlrel2.hp.com (Postfix) with ESMTP id 32CAE2550; Thu, 24 Aug 2000 13:22:35 -0400 (EDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id KAA14274; Thu, 24 Aug 2000 10:22:34 -0700 (PDT) X-OpenMail-Hops: 1 Date: Thu, 24 Aug 2000 10:22:23 -0700 Message-Id: Subject: RE: Re: iSCSI: SCTP Switch and Router support MIME-Version: 1.0 To: Black_David@emc.com, hufferd@us.ibm.com Cc: ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree. Look at switches by a lot of vendors (like Alteon, Foundry etc.). In general, the trend towards being aware of TCP connection flows (keeping actual connection states) is increasing with some switches capable of maintaining state information for upto a million connections. In addition to things mentioned by John, it is also going to be used (or already being used) for traffic classification (VLAN and priority). It is an important component of traffic engineering. (on a comment by Doug, most (all??) TCP implementations today avoid fragmentation also (by using path MTU discovery) since fragmentation is so expensive for routers or receivers. Somesh > -----Original Message----- > From: hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] > Sent: Thursday, August 24, 2000 9:19 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: FW: Re: iSCSI: SCTP Switch and Router support > > > David, > A lot of what you say is true, however, the word "Most" is > the key. There > are a number of Switches, NATs, Proxies, and Load Balancers > that actually > care about TCP/IP. I do not believe they will handle the SCTP in a > transparent manor. > > . > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/SSD San Jose Ca > (408) 256-0403, Tie: 276-0403 > Internet address: hufferd@us.ibm.com > Notes address: John Hufferd/San Jose/IBM @ IBMUS > VM address: hufferd at IBMUSM54 > > > Black_David@emc.com on 08/24/2000 08:54:46 AM > > To: John Hufferd/San Jose/IBM@IBMUS > cc: ips@ece.cmu.edu > Subject: iSCSI: SCTP Switch and Router support > > > > John, > > > 3. We will need to get the Switch and Router folks to also > support SCTP > and > > to do it in Hardware so that the speed and throughput can > be maintained. > I > > do not see that happening, at least not right away, when we > need to get > > volumes moving in order to validate the IP storage SAN concept. > > I don't understand this. Most switches and routers > have no knowledge of whether TCP, UDP, or even IPsec > encapsulated (and hence opaque) traffic is flowing > through them. What's special about SCTP -- is this > about layer 4 and higher switches/routers? > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > > > From owner-ips@ECE.cmu.edu Thu Aug 24 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07381; 24 Aug 2000 15:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA15057 for ips-outgoing; Thu, 24 Aug 2000 14:12:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (truffle.Stanford.EDU [171.64.73.147]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA15052 for ; Thu, 24 Aug 2000 14:12:33 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id C2A448E6A; Thu, 24 Aug 2000 11:11:25 -0700 (PDT) To: "Ferrari, Stephen" Cc: "'ips@ece.cmu.edu'" Subject: Re: Data in SCSI Response or SCSI Data References: <618471D08ABDD31188B6009027E500932715C1@apollo.pirus.com> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 24 Aug 2000 11:11:25 -0700 In-Reply-To: "Ferrari, Stephen"'s message of "Thu, 24 Aug 2000 12:15:56 -0400" Message-ID: Lines: 18 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk "Ferrari, Stephen" writes: > For successful SCSI commands where the target sends data to the host, there > appear to be several combinations of data and response that are allowed: > > * Data sent in a SCSI Data packet and status sent in a SCSI Response > packet The approach above is preferred. > * Data and status sent in a SCSI Data packet, using the (S) bit > * Data and status sent in a SCSI Response packet I think the plan is to remove these last two approaches in the next edition of the spec. -Costa From owner-ips@ECE.cmu.edu Thu Aug 24 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07714; 24 Aug 2000 16:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA17479 for ips-outgoing; Thu, 24 Aug 2000 15:09:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA17473 for ; Thu, 24 Aug 2000 15:09:13 -0400 (EDT) Received: from zeppo.feral.com (IDENT:mjacob@zeppo [192.67.166.71]) by feral.com (8.9.3/8.9.3) with ESMTP id MAA09174; Thu, 24 Aug 2000 12:09:10 -0700 Date: Thu, 24 Aug 2000 12:05:47 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: csapuntz@cisco.com cc: "'ips@ece.cmu.edu'" Subject: Re: Data in SCSI Response or SCSI Data In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > > * Data and status sent in a SCSI Response packet > > I think the plan is to remove these last two approaches in the next > edition of the spec. Why? There are a lot of 'trivial' commands that could be done with one packet. -matt From owner-ips@ECE.cmu.edu Thu Aug 24 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07984; 24 Aug 2000 17:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA22658 for ips-outgoing; Thu, 24 Aug 2000 16:51:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch11.nsmg.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA21328 for ; Thu, 24 Aug 2000 16:18:20 -0400 (EDT) Received: by lmoxch11.nsmg.veritas.com with Internet Mail Service (5.5.2448.0) id ; Thu, 24 Aug 2000 16:19:03 -0400 Message-ID: From: John Matze To: "'ips@ece.cmu.edu'" Subject: RE: Data in SCSI Response or SCSI Data Date: Thu, 24 Aug 2000 16:19:02 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree there is no reason to eliminate this functionality. Small commands could be optimized with one packet. John -----Original Message----- From: Matthew Jacob [mailto:mjacob@feral.com] Sent: Thursday, August 24, 2000 12:06 PM To: csapuntz@cisco.com Cc: 'ips@ece.cmu.edu' Subject: Re: Data in SCSI Response or SCSI Data > > * Data and status sent in a SCSI Response packet > > I think the plan is to remove these last two approaches in the next > edition of the spec. Why? There are a lot of 'trivial' commands that could be done with one packet. -matt From owner-ips@ECE.cmu.edu Thu Aug 24 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07979; 24 Aug 2000 17:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA21811 for ips-outgoing; Thu, 24 Aug 2000 16:31:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA21804 for ; Thu, 24 Aug 2000 16:31:05 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id WAA229224 for ; Thu, 24 Aug 2000 22:30:34 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id WAA34724 for ; Thu, 24 Aug 2000 22:30:33 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256945.0070A6B5 ; Thu, 24 Aug 2000 22:30:27 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 24 Aug 2000 23:28:31 +0300 Subject: Re: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Correct. Julo Matthew Jacob on 24/08/2000 22:05:47 Please respond to mjacob@feral.com To: csapuntz@cisco.com cc: "'ips@ece.cmu.edu'" (bcc: Julian Satran/Haifa/IBM) Subject: Re: Data in SCSI Response or SCSI Data > > * Data and status sent in a SCSI Response packet > > I think the plan is to remove these last two approaches in the next > edition of the spec. Why? There are a lot of 'trivial' commands that could be done with one packet. -matt From owner-ips@ECE.cmu.edu Thu Aug 24 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07974; 24 Aug 2000 17:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA21176 for ips-outgoing; Thu, 24 Aug 2000 16:12:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA21168 for ; Thu, 24 Aug 2000 16:12:46 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA05367; Thu, 24 Aug 2000 13:05:58 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id NAA12827; Thu, 24 Aug 2000 13:03:29 -0700 From: "Douglas Otis" To: , , Cc: Subject: RE: Re: iSCSI: SCTP Switch and Router support (FCoverIP) Date: Thu, 24 Aug 2000 13:14:44 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, "Problems to overcome implementing a new protocol for routers and switches that look beyond IP layers should be of the same order of difficulty as the IP datagram protocol initially proposed for *FCoverIP*. At least the expensive fragmentation is handled within the SCTP protocol to ensure a normal router does not see a burden." The comparison was between the original FCoverIP proposal and SCTP. Comparing to TCP is a different conversation. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > somesh_gupta@hp.com > Sent: Thursday, August 24, 2000 10:22 AM > To: Black_David@emc.com; hufferd@us.ibm.com > Cc: ips@ece.cmu.edu > Subject: RE: Re: iSCSI: SCTP Switch and Router support > > > I agree. Look at switches by a lot of vendors (like Alteon, > Foundry etc.). In general, the trend towards being aware of > TCP connection flows (keeping actual connection states) is > increasing with some switches capable of maintaining state > information for upto a million connections. > > In addition to things mentioned by John, it is also going to > be used (or already being used) for traffic classification > (VLAN and priority). It is an important component of traffic > engineering. > > (on a comment by Doug, most (all??) TCP implementations today > avoid fragmentation also (by using path MTU discovery) since > fragmentation is so expensive for routers or receivers. > > Somesh From owner-ips@ECE.cmu.edu Thu Aug 24 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07986; 24 Aug 2000 17:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA22086 for ips-outgoing; Thu, 24 Aug 2000 16:38:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA22079 for ; Thu, 24 Aug 2000 16:37:57 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id WAA30820 for ; Thu, 24 Aug 2000 22:37:25 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id WAA22870 for ; Thu, 24 Aug 2000 22:37:25 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256945.007147AE ; Thu, 24 Aug 2000 22:37:19 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 24 Aug 2000 23:35:25 +0300 Subject: Re: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk It is assumed that short operations - let's say a 2-4k read will be handled is one block including status enabling an initiator to get them in maximum 2 read operations. Data and status will appear in the response. Julo "Ferrari, Stephen" on 24/08/2000 19:15:56 Please respond to "Ferrari, Stephen" To: "'ips@ece.cmu.edu'" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Data in SCSI Response or SCSI Data For successful SCSI commands where the target sends data to the host, there appear to be several combinations of data and response that are allowed: * Data sent in a SCSI Data packet and status sent in a SCSI Response packet * Data and status sent in a SCSI Data packet, using the (S) bit * Data and status sent in a SCSI Response packet Is the target free to choose any of these, at its option, requiring the initiator to handle all of them? Is there a requirement that some commands (say, READ) use the first or second method, and other commands (say, INQUIRY) use the third? Is the data in a successful SCSI Response packet exactly equivalent to data in a SCSI Data packet, with an assumed offset of zero? I couldn't find the answer in the July 10th draft. If someone could point me at it, I'd appreciate it. Thanks, Stephen Ferrari Pirus Networks From owner-ips@ECE.cmu.edu Thu Aug 24 23:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08499; 24 Aug 2000 19:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA25540 for ips-outgoing; Thu, 24 Aug 2000 18:07:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from swan.prod.itd.earthlink.net (swan.prod.itd.earthlink.net [207.217.120.123]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA25523 for ; Thu, 24 Aug 2000 18:07:39 -0400 (EDT) Received: from TOPDOG (1Cust243.tnt1.hitchcock.tx.da.uu.net [63.24.92.243]) by swan.prod.itd.earthlink.net (8.9.3-EL_1_3/8.9.3) with SMTP id PAA06186; Thu, 24 Aug 2000 15:07:35 -0700 (PDT) From: "Bill Casey" To: Cc: Subject: RE: Data in SCSI Response or SCSI Data Date: Thu, 24 Aug 2000 17:08:05 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > > * Data sent in a SCSI Data packet and status sent in a > SCSI Response packet > > The approach above is preferred. Why? This increases overhead. Our customers asked us to change our data functions to optionally/automatically append Status & Cmd Cmplt after the Data Phase. Bill Casey == SCSI Adapters & VirtualSCSI Target Mode Libs == Advanced Storage Concepts, Inc. (409) 744-2129 2720 Terminal Drive info@advstor.com Galveston, TX 77554 USA www.virtualscsi.com From owner-ips@ECE.cmu.edu Thu Aug 24 23:40:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08504; 24 Aug 2000 19:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA26573 for ips-outgoing; Thu, 24 Aug 2000 18:39:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (truffle.Stanford.EDU [171.64.73.147]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA26561 for ; Thu, 24 Aug 2000 18:39:43 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id DCBF18E85; Thu, 24 Aug 2000 15:38:24 -0700 (PDT) To: "'ips@ece.cmu.edu'" Subject: Re: Data in SCSI Response or SCSI Data References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 24 Aug 2000 15:38:24 -0700 In-Reply-To: Matthew Jacob's message of "Thu, 24 Aug 2000 12:05:47 -0700 (PDT)" Message-ID: Lines: 24 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk If I remember the discussion correctly, I remember some folks who were implementing hardware did not like having 3 different mechanisms for doing the same thing, as this required more logic and verification. >From a network performance standpoint, there is minimal difference between sending two iSCSI messages back-to-back and one larger iSCSI message. In fact, both iSCSI messages might even share the same packet. -Costa Matthew Jacob writes: > > > * Data and status sent in a SCSI Response packet > > > > I think the plan is to remove these last two approaches in the next > > edition of the spec. > > Why? There are a lot of 'trivial' commands that could be done with one packet. > > -matt From owner-ips@ECE.cmu.edu Fri Aug 25 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08932; 24 Aug 2000 21:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA00195 for ips-outgoing; Thu, 24 Aug 2000 20:08:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA00191 for ; Thu, 24 Aug 2000 20:07:57 -0400 (EDT) Received: from beppo.feral.com (beppo [192.67.166.79]) by feral.com (8.9.3/8.9.3) with ESMTP id RAA10447; Thu, 24 Aug 2000 17:07:54 -0700 Date: Thu, 24 Aug 2000 17:07:55 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: csapuntz@cisco.com cc: "'ips@ece.cmu.edu'" Subject: Re: Data in SCSI Response or SCSI Data In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > > If I remember the discussion correctly, I remember some folks who > were implementing hardware did not like having 3 different > mechanisms for doing the same thing, as this required more logic > and verification. > > From a network performance standpoint, there is minimal difference > between sending two iSCSI messages back-to-back and one larger > iSCSI message. > > In fact, both iSCSI messages might even share the same packet. That's a fair point. In other contexts (FC in this case), I've really liked being able (in target mode) to send back data && status in packet- it means I don't have to manage multiple resources on the send side, and in the case of sending back canned data for 'trivial' commands (like Mode data), I don't even have to do a context or thread switch. What this might like like in an IPS implementation is not clear to me, but I believe I do see the point about the h/w implementation issues. It's not clear to me that you can state as a general rule that two packets back-to-back is minimally different- not from a performance point of view, but from a likelihood of lossage and retransmit. If a high proportion of your trivial (or not so trivial) read commands come back in one frame, that's a lot better than managing lost frames, no? -matt From owner-ips@ECE.cmu.edu Fri Aug 25 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08949; 24 Aug 2000 21:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA00342 for ips-outgoing; Thu, 24 Aug 2000 20:13:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA00336 for ; Thu, 24 Aug 2000 20:12:59 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id RGJL07PT; Thu, 24 Aug 2000 17:11:06 -0700 From: "Y P Cheng" To: , Subject: RE: Data in SCSI Response or SCSI Data Date: Thu, 24 Aug 2000 17:11:16 -0700 Message-ID: <003801c00e28$fcc4a3c0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk I am new in this discussion. Yes, for those of us design an adapter transmitting the IP packets, we prefer one way instead three different ways. In fact, we prefer status packet going to the TCP receive buffer and the data packet going to the data channel for supporting RDMA. This allows us the minimum parsing of a packet. Y.P. Cheng, Connectcom Solutions -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of csapuntz@cisco.com Sent: Thursday, August 24, 2000 3:38 PM To: 'ips@ece.cmu.edu' Cc: csapuntz@cisco.com Subject: Re: Data in SCSI Response or SCSI Data If I remember the discussion correctly, I remember some folks who were implementing hardware did not like having 3 different mechanisms for doing the same thing, as this required more logic and verification. >From a network performance standpoint, there is minimal difference between sending two iSCSI messages back-to-back and one larger iSCSI message. In fact, both iSCSI messages might even share the same packet. -Costa Matthew Jacob writes: > > > * Data and status sent in a SCSI Response packet > > > > I think the plan is to remove these last two approaches in the next > > edition of the spec. > > Why? There are a lot of 'trivial' commands that could be done with one packet. > > -matt From owner-ips@ECE.cmu.edu Fri Aug 25 12:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12085; 25 Aug 2000 7:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA25506 for ips-outgoing; Fri, 25 Aug 2000 07:00:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA25501 for ; Fri, 25 Aug 2000 07:00:28 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA227612 for ; Fri, 25 Aug 2000 12:59:56 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA39670 for ; Fri, 25 Aug 2000 12:59:57 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256946.003C66FC ; Fri, 25 Aug 2000 12:59:45 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 25 Aug 2000 13:57:43 +0300 Subject: Re: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Any software implemementation will have 2 reads/iSCSI message. As we asume a large number of software implementations being used as initiators having the double the number of reads for small I/O is not wise. Hardware will easily manage the different read modes. Julo csapuntz@cisco.com on 25/08/2000 01:38:24 Please respond to csapuntz@cisco.com To: "'ips@ece.cmu.edu'" cc: csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: Re: Data in SCSI Response or SCSI Data If I remember the discussion correctly, I remember some folks who were implementing hardware did not like having 3 different mechanisms for doing the same thing, as this required more logic and verification. >From a network performance standpoint, there is minimal difference between sending two iSCSI messages back-to-back and one larger iSCSI message. In fact, both iSCSI messages might even share the same packet. -Costa Matthew Jacob writes: > > > * Data and status sent in a SCSI Response packet > > > > I think the plan is to remove these last two approaches in the next > > edition of the spec. > > Why? There are a lot of 'trivial' commands that could be done with one packet. > > -matt From owner-ips@ECE.cmu.edu Fri Aug 25 12:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12090; 25 Aug 2000 7:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA25875 for ips-outgoing; Fri, 25 Aug 2000 07:19:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch11.nsmg.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA12401 for ; Fri, 25 Aug 2000 02:28:34 -0400 (EDT) Received: by lmoxch11.nsmg.veritas.com with Internet Mail Service (5.5.2448.0) id ; Fri, 25 Aug 2000 02:29:17 -0400 Message-ID: From: John Matze To: "'ips@ece.cmu.edu'" Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 25 Aug 2000 02:29:04 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Again I have to agree. Not everyone writing this protocol is developing a hardware device. Every time I (or any hardware vendor) sends one less IP packet down the wire is one less chance of it getting lost. -----Original Message----- From: Matthew Jacob [mailto:mjacob@feral.com] Sent: Thursday, August 24, 2000 5:08 PM To: csapuntz@cisco.com Cc: 'ips@ece.cmu.edu' Subject: Re: Data in SCSI Response or SCSI Data > > If I remember the discussion correctly, I remember some folks who > were implementing hardware did not like having 3 different > mechanisms for doing the same thing, as this required more logic > and verification. > > From a network performance standpoint, there is minimal difference > between sending two iSCSI messages back-to-back and one larger > iSCSI message. > > In fact, both iSCSI messages might even share the same packet. That's a fair point. In other contexts (FC in this case), I've really liked being able (in target mode) to send back data && status in packet- it means I don't have to manage multiple resources on the send side, and in the case of sending back canned data for 'trivial' commands (like Mode data), I don't even have to do a context or thread switch. What this might like like in an IPS implementation is not clear to me, but I believe I do see the point about the h/w implementation issues. It's not clear to me that you can state as a general rule that two packets back-to-back is minimally different- not from a performance point of view, but from a likelihood of lossage and retransmit. If a high proportion of your trivial (or not so trivial) read commands come back in one frame, that's a lot better than managing lost frames, no? -matt From owner-ips@ECE.cmu.edu Fri Aug 25 14:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12658; 25 Aug 2000 10:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA28080 for ips-outgoing; Fri, 25 Aug 2000 08:56:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp2.mail.yahoo.com (smtp2.mail.yahoo.com [128.11.68.32]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id IAA28075 for ; Fri, 25 Aug 2000 08:56:00 -0400 (EDT) Received: from unknown (HELO wazoo) (216.132.202.111) by smtp2.mail.yahoo.com with SMTP; 25 Aug 2000 12:55:59 -0000 X-Apparently-From: From: "Thomas Crowe" To: , , Cc: Subject: RE: Re: iSCSI: SCTP Switch and Router support Date: Fri, 25 Aug 2000 09:01:35 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Switches are aware of protocol from a layer 4 perspective to enhance performance, not make switching decisions. The two company's that you mention (Alteon and Foundry) both make layer 4 switches that are designed to make switching decisions on TCP sessions, one of those devices would not be used in the transport of iSCSI packets, they are used instead for the loadbalancing of webservers and the like. Foundry also make pure layer 2 switches as well as layer 2/3 routing switches. AccessLists would be affected but that alone should not be a driving factor in not supporting SCTP. MTU path discovery is NOT a function of TCP, that is done with ICMP. Just to clarify, I am not supporting / not supporting SCTP I simply wanted to clarify some of the network issues. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > somesh_gupta@hp.com > Sent: Thursday, August 24, 2000 1:22 PM > To: Black_David@emc.com; hufferd@us.ibm.com > Cc: ips@ece.cmu.edu > Subject: RE: Re: iSCSI: SCTP Switch and Router support > > > I agree. Look at switches by a lot of vendors (like Alteon, > Foundry etc.). In general, the trend towards being aware of > TCP connection flows (keeping actual connection states) is > increasing with some switches capable of maintaining state > information for upto a million connections. > > In addition to things mentioned by John, it is also going to > be used (or already being used) for traffic classification > (VLAN and priority). It is an important component of traffic > engineering. > > (on a comment by Doug, most (all??) TCP implementations today > avoid fragmentation also (by using path MTU discovery) since > fragmentation is so expensive for routers or receivers. > > Somesh > > > -----Original Message----- > > From: hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] > > Sent: Thursday, August 24, 2000 9:19 AM > > To: Black_David@emc.com > > Cc: ips@ece.cmu.edu > > Subject: FW: Re: iSCSI: SCTP Switch and Router support > > > > > > David, > > A lot of what you say is true, however, the word "Most" is > > the key. There > > are a number of Switches, NATs, Proxies, and Load Balancers > > that actually > > care about TCP/IP. I do not believe they will handle the SCTP in a > > transparent manor. > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSD San Jose Ca > > (408) 256-0403, Tie: 276-0403 > > Internet address: hufferd@us.ibm.com > > Notes address: John Hufferd/San Jose/IBM @ IBMUS > > VM address: hufferd at IBMUSM54 > > > > > > Black_David@emc.com on 08/24/2000 08:54:46 AM > > > > To: John Hufferd/San Jose/IBM@IBMUS > > cc: ips@ece.cmu.edu > > Subject: iSCSI: SCTP Switch and Router support > > > > > > > > John, > > > > > 3. We will need to get the Switch and Router folks to also > > support SCTP > > and > > > to do it in Hardware so that the speed and throughput can > > be maintained. > > I > > > do not see that happening, at least not right away, when we > > need to get > > > volumes moving in order to validate the IP storage SAN concept. > > > > I don't understand this. Most switches and routers > > have no knowledge of whether TCP, UDP, or even IPsec > > encapsulated (and hence opaque) traffic is flowing > > through them. What's special about SCTP -- is this > > about layer 4 and higher switches/routers? > > > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > > black_david@emc.com Cellular: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > > __________________________________________________ Do You Yahoo!? Talk to your friends online with Yahoo! Messenger. http://im.yahoo.com From owner-ips@ECE.cmu.edu Fri Aug 25 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13510; 25 Aug 2000 13:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07660 for ips-outgoing; Fri, 25 Aug 2000 12:16:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07654 for ; Fri, 25 Aug 2000 12:16:12 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e7PGGBw18767 for ; Fri, 25 Aug 2000 11:16:11 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA07283; Fri, 25 Aug 00 11:13:56 -0500 Message-Id: <10008251613.AA07283@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data In-Reply-To: Message from John Matze of "Fri, 25 Aug 2000 02:29:04 EDT." Date: Fri, 25 Aug 2000 11:14:48 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Again I have to agree. Not everyone writing this protocol is developing a > hardware device. Every time I (or any hardware vendor) sends one less IP > packet down the wire is one less chance of it getting lost. I do not think concessions should be made to software implementations at the expense of hardware implementations. Our experience is nobody cares about a software network storage implementation if there are not also hardware implementations. The value proposition completely dries up. With available hardware implementations software implementations become interesting. I agree with Y.P. Cheng that implementing hardware for three ways to find status IS a going to be a substantial irritation. More importantly, the outbound path of a hardware implementation (which will include most targets) is very likely to chose to implement the single easiest way for it, and use it in all cases. In other words, if data in one packet with RDMA information and status in a second packet without is the natural way (seems like it), that's what you're going to get coming back to the initiator. Your software `fast path' will never get exercised. Even host `software' implementations are going to be able to support various levels of hardware acceleration. In the case of using a hardware accelerated adapter, the host software will not see the data packet, only the status. Steph From owner-ips@ECE.cmu.edu Fri Aug 25 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13535; 25 Aug 2000 13:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07647 for ips-outgoing; Fri, 25 Aug 2000 12:15:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp2.mail.yahoo.com (smtp2.mail.yahoo.com [128.11.68.32]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id MAA07632 for ; Fri, 25 Aug 2000 12:15:42 -0400 (EDT) Received: from unknown (HELO wazoo) (216.132.202.111) by smtp2.mail.yahoo.com with SMTP; 25 Aug 2000 16:15:37 -0000 X-Apparently-From: From: "Thomas Crowe" To: "John Hufferd/San Jose/IBM" Cc: , , Subject: RE: Re: iSCSI: SCTP Switch and Router support Date: Fri, 25 Aug 2000 12:21:11 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk I am not suggesting that that iSCSI could not be load balanced, however there would be significant challenges to doing so. Firstly most hardware based loadbalancers are based totally on the TCP session, how would this fit into the iSCSI model? In the usual configuration you would have the devices providing the service *behind* the loadbalancers and the loadbalancer would route the traffic based on defined rules. This would have the requirement that n devices contain the exact same information in the exact same location. |----| |_____|---| | A |______|--|___|--|_____| |---| |----| |--| |--| | D B C |_____|---| | |---| E A: Client requesting the data B: Virtual address that the client's use to access the service C: Physical LoadBalancer (i.e. Foundry ServerIron, Alteon, Cisco LocalDirector) D, E: devices providing iSCSI services Just to clarify my position again :-) At this time I can not make an educated stand either for or against using SCTP or TCP. I wanted to point out what I perceived as possibly a misunderstanding of the switch / router support for SCTP. Layer 2 / 3 devices should have no problem switching or routing SCTP the issue comes in at Layer 4, for functions such as ACL's, LoadBalancing, Firewall's etc.... Thomas Crowe Senior Systems Engineer CTSinc.net Atlanta, GA > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Friday, August 25, 2000 11:32 AM > To: Thomas Crowe > Cc: somesh_gupta@hp.com; Black_David@emc.com; ips@ece.cmu.edu > Subject: RE: Re: iSCSI: SCTP Switch and Router support > > > > I know of at least one vendor that wanted to do iSCSI load balancing > across Multiple Physical Targets, that needed to be looked at as one > Virtual Target. Are you suggesting that the Load Balancing Switches could > not be used by iSCSI even if they used TCP/IP? > > . > . > . > John L. Hufferd > > > "Thomas Crowe" on 08/25/2000 06:01:35 AM > > To: , , John Hufferd/San > Jose/IBM@IBMUS > cc: > Subject: RE: Re: iSCSI: SCTP Switch and Router support > > > > Switches are aware of protocol from a layer 4 perspective to enhance > performance, not make switching decisions. The two company's that you > mention (Alteon and Foundry) both make layer 4 switches that are designed > to > make switching decisions on TCP sessions, one of those devices > would not be > used in the transport of iSCSI packets, they are used instead for the > loadbalancing of webservers and the like. Foundry also make pure layer 2 > switches as well as layer 2/3 routing switches. AccessLists would be > affected but that alone should not be a driving factor in not supporting > SCTP. > > MTU path discovery is NOT a function of TCP, that is done with ICMP. > > Just to clarify, I am not supporting / not supporting SCTP I simply wanted > to clarify some of the network issues. > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > somesh_gupta@hp.com > > Sent: Thursday, August 24, 2000 1:22 PM > > To: Black_David@emc.com; hufferd@us.ibm.com > > Cc: ips@ece.cmu.edu > > Subject: RE: Re: iSCSI: SCTP Switch and Router support > > > > > > I agree. Look at switches by a lot of vendors (like Alteon, > > Foundry etc.). In general, the trend towards being aware of > > TCP connection flows (keeping actual connection states) is > > increasing with some switches capable of maintaining state > > information for upto a million connections. > > > > In addition to things mentioned by John, it is also going to > > be used (or already being used) for traffic classification > > (VLAN and priority). It is an important component of traffic > > engineering. > > > > (on a comment by Doug, most (all??) TCP implementations today > > avoid fragmentation also (by using path MTU discovery) since > > fragmentation is so expensive for routers or receivers. > > > > Somesh > > > > > -----Original Message----- > > > From: hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] > > > Sent: Thursday, August 24, 2000 9:19 AM > > > To: Black_David@emc.com > > > Cc: ips@ece.cmu.edu > > > Subject: FW: Re: iSCSI: SCTP Switch and Router support > > > > > > > > > David, > > > A lot of what you say is true, however, the word "Most" is > > > the key. There > > > are a number of Switches, NATs, Proxies, and Load Balancers > > > that actually > > > care about TCP/IP. I do not believe they will handle the SCTP in a > > > transparent manor. > > > > > > . > > > . > > > . > > > John L. Hufferd > > > Senior Technical Staff Member (STSM) > > > IBM/SSD San Jose Ca > > > (408) 256-0403, Tie: 276-0403 > > > Internet address: hufferd@us.ibm.com > > > Notes address: John Hufferd/San Jose/IBM @ IBMUS > > > VM address: hufferd at IBMUSM54 > > > > > > > > > Black_David@emc.com on 08/24/2000 08:54:46 AM > > > > > > To: John Hufferd/San Jose/IBM@IBMUS > > > cc: ips@ece.cmu.edu > > > Subject: iSCSI: SCTP Switch and Router support > > > > > > > > > > > > John, > > > > > > > 3. We will need to get the Switch and Router folks to also > > > support SCTP > > > and > > > > to do it in Hardware so that the speed and throughput can > > > be maintained. > > > I > > > > do not see that happening, at least not right away, when we > > > need to get > > > > volumes moving in order to validate the IP storage SAN concept. > > > > > > I don't understand this. Most switches and routers > > > have no knowledge of whether TCP, UDP, or even IPsec > > > encapsulated (and hence opaque) traffic is flowing > > > through them. What's special about SCTP -- is this > > > about layer 4 and higher switches/routers? > > > > > > --David > > > > > > --------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > > > black_david@emc.com Cellular: +1 (978) 394-7754 > > > --------------------------------------------------- > > > > > > > > > > > > __________________________________________________ > Do You Yahoo!? > Talk to your friends online with Yahoo! Messenger. > http://im.yahoo.com > > > __________________________________________________ Do You Yahoo!? Talk to your friends online with Yahoo! Messenger. http://im.yahoo.com From owner-ips@ECE.cmu.edu Fri Aug 25 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13521; 25 Aug 2000 13:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07883 for ips-outgoing; Fri, 25 Aug 2000 12:21:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from smtp2.mail.yahoo.com (smtp2.mail.yahoo.com [128.11.68.32]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id MAA07863 for ; Fri, 25 Aug 2000 12:21:24 -0400 (EDT) Received: from unknown (HELO wazoo) (216.132.202.111) by smtp2.mail.yahoo.com with SMTP; 25 Aug 2000 16:21:19 -0000 X-Apparently-From: From: "Thomas Crowe" To: "Robert Snively" , "'John Matze'" , Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 25 Aug 2000 12:26:55 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree that, from an end node perspective, TCP guarantees delivery, however that is different than not losing packets. A TCP packet can indeed get *lost*, TCP works around this by utilizing re-transmission, which results in out-of-order reception on the client side. I have seen some discussion on out of order packets relating to iSCSI, however I have not digested the iSCSI proposed standard enough to tell if this is a problem of not. Thomas Crowe Senior Systems Engineer CTSinc.net Atlanta, GA > > While networks lose packets, TCP/IP connections never do, so you > should not see this as a problem at the iSCSI layer. If you can > lose packets at the iSCSI layer, then having to guess at whether the > recovery will involve two kinds of information units or one kind of > information unit seems like an unnecessary complexity. > __________________________________________________ Do You Yahoo!? Talk to your friends online with Yahoo! Messenger. http://im.yahoo.com From owner-ips@ECE.cmu.edu Fri Aug 25 18:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13640; 25 Aug 2000 14:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09549 for ips-outgoing; Fri, 25 Aug 2000 13:04:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA09539 for ; Fri, 25 Aug 2000 13:03:50 -0400 (EDT) Received: from zeppo.feral.com (IDENT:mjacob@zeppo [192.67.166.71]) by feral.com (8.9.3/8.9.3) with ESMTP id KAA13337; Fri, 25 Aug 2000 10:03:47 -0700 Date: Fri, 25 Aug 2000 10:00:21 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: Robert Snively cc: "'John Matze'" , "'ips@ece.cmu.edu'" Subject: RE: Data in SCSI Response or SCSI Data In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > While networks lose packets, TCP/IP connections never do, so you > should not see this as a problem at the iSCSI layer. If you can > lose packets at the iSCSI layer, then having to guess at whether the > recovery will involve two kinds of information units or one kind of > information unit seems like an unnecessary complexity. > > This should be kept simple, providing one kind of information in > a packet, thus making it simpler to implement hardware that can > route the information directly to the appropriate memory space. > > If the overhead of preparing and sending two packets instead of one > is burdensome, then we have that problem to solve as well. If there is any kind of non-zero percentage packet lossage in a IP SAN, I would expect, on average, better throughput and fewer flow control and recovery disruptions if the one packet lost isn't a frag. But- hey, I'd say that this ain't my first area of expertise so I'd be glad if the stronger TCP/IP network types whap me upside the head if I'm wrong. The preparing and sending of two packets is not 'burdensome'. But the case of sending one packet instead of two can be 'advantageous' for reaons I stated elsewhere. -matt From owner-ips@ECE.cmu.edu Fri Aug 25 18:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13699; 25 Aug 2000 14:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA08838 for ips-outgoing; Fri, 25 Aug 2000 12:48:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA08830 for ; Fri, 25 Aug 2000 12:48:13 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA187424 for ; Fri, 25 Aug 2000 18:47:42 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA16488 for ; Fri, 25 Aug 2000 18:47:37 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256946.005C3D99 ; Fri, 25 Aug 2000 18:47:30 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 25 Aug 2000 19:35:07 +0300 Subject: Re: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Is there anything preventing your hypothetical hardware implementor to send always good status within the last block of data? Does the hardware implementor make more decisions when finding status allone in a response block or within the header of the last block? Software based initiators will be, IMHO, a very important part of the market - to have them go through two calls is not wise. On the other hand I am not religious about it. However - sending "good status" with the last piece of data has been part of the the I/O scene for a long time and I did attempt to keep the status quo. Julo Stephen Bailey on 25/08/2000 19:14:48 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Data in SCSI Response or SCSI Data > Again I have to agree. Not everyone writing this protocol is developing a > hardware device. Every time I (or any hardware vendor) sends one less IP > packet down the wire is one less chance of it getting lost. I do not think concessions should be made to software implementations at the expense of hardware implementations. Our experience is nobody cares about a software network storage implementation if there are not also hardware implementations. The value proposition completely dries up. With available hardware implementations software implementations become interesting. I agree with Y.P. Cheng that implementing hardware for three ways to find status IS a going to be a substantial irritation. More importantly, the outbound path of a hardware implementation (which will include most targets) is very likely to chose to implement the single easiest way for it, and use it in all cases. In other words, if data in one packet with RDMA information and status in a second packet without is the natural way (seems like it), that's what you're going to get coming back to the initiator. Your software `fast path' will never get exercised. Even host `software' implementations are going to be able to support various levels of hardware acceleration. In the case of using a hardware accelerated adapter, the host software will not see the data packet, only the status. Steph From owner-ips@ECE.cmu.edu Fri Aug 25 18:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13701; 25 Aug 2000 14:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09800 for ips-outgoing; Fri, 25 Aug 2000 13:11:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA09789 for ; Fri, 25 Aug 2000 13:11:40 -0400 (EDT) Received: from zeppo.feral.com (IDENT:mjacob@zeppo [192.67.166.71]) by feral.com (8.9.3/8.9.3) with ESMTP id KAA13365; Fri, 25 Aug 2000 10:11:32 -0700 Date: Fri, 25 Aug 2000 10:08:05 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: Y P Cheng cc: csapuntz@cisco.com, ips@ece.cmu.edu Subject: RE: Data in SCSI Response or SCSI Data In-Reply-To: <000201c00ea7$644bf240$65c809c0@yp_portable.advansys.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > Matt, I see your point in software implementation. May be this is a good > time to introduce the concept of "Good Status" flag bit. In fact, 99.99% > of iSCSI reads should be completed without error. For the normal > completions, all we need to is a flag bit in the data packet message > header indicating "Everything is OK. This SCSI read is now completed > normally." If this flag bit is not set, there will be a status packet > with sense data following. In the hardware implementation, after > transferring data packet into the buffer, if this flag bit is set, we > interrupt the microcode which post the Request Block (RB) completion > interrupt to the device driver. If the bit is not set, the hardware does > nothing after data transfer. The status packet arrived later will be > passed to the microcode that, in turn, generates a proper RB completion > interrupt to pass the error information to the device driver. Sounds good to me. If it makes the h/w easier- great. Although full status isn't that much extra, I could see that h/w for it would be harder. I'd like to point out that the software state machine in target mode for this kind of stuff isn't entirely trivial either. If you have a layered system (as you probably would) you still might have to set "Send Status with Data" state at the upper end and check completion later to see whether it was actually sent. But the performance differential (in the non-IPS case) makes it worth the effort. -matt From owner-ips@ECE.cmu.edu Fri Aug 25 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13807; 25 Aug 2000 14:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA04627 for ips-outgoing; Fri, 25 Aug 2000 11:37:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA04548 for ; Fri, 25 Aug 2000 11:37:21 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA31716; Fri, 25 Aug 2000 11:35:09 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id JAA148164; Fri, 25 Aug 2000 09:37:05 -0600 Importance: Normal Subject: RE: Re: iSCSI: SCTP Switch and Router support To: "Thomas Crowe" Cc: , , X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 25 Aug 2000 08:31:55 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/25/2000 09:37:05 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I know of at least one vendor that wanted to do iSCSI load balancing across Multiple Physical Targets, that needed to be looked at as one Virtual Target. Are you suggesting that the Load Balancing Switches could not be used by iSCSI even if they used TCP/IP? . . . John L. Hufferd "Thomas Crowe" on 08/25/2000 06:01:35 AM To: , , John Hufferd/San Jose/IBM@IBMUS cc: Subject: RE: Re: iSCSI: SCTP Switch and Router support Switches are aware of protocol from a layer 4 perspective to enhance performance, not make switching decisions. The two company's that you mention (Alteon and Foundry) both make layer 4 switches that are designed to make switching decisions on TCP sessions, one of those devices would not be used in the transport of iSCSI packets, they are used instead for the loadbalancing of webservers and the like. Foundry also make pure layer 2 switches as well as layer 2/3 routing switches. AccessLists would be affected but that alone should not be a driving factor in not supporting SCTP. MTU path discovery is NOT a function of TCP, that is done with ICMP. Just to clarify, I am not supporting / not supporting SCTP I simply wanted to clarify some of the network issues. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > somesh_gupta@hp.com > Sent: Thursday, August 24, 2000 1:22 PM > To: Black_David@emc.com; hufferd@us.ibm.com > Cc: ips@ece.cmu.edu > Subject: RE: Re: iSCSI: SCTP Switch and Router support > > > I agree. Look at switches by a lot of vendors (like Alteon, > Foundry etc.). In general, the trend towards being aware of > TCP connection flows (keeping actual connection states) is > increasing with some switches capable of maintaining state > information for upto a million connections. > > In addition to things mentioned by John, it is also going to > be used (or already being used) for traffic classification > (VLAN and priority). It is an important component of traffic > engineering. > > (on a comment by Doug, most (all??) TCP implementations today > avoid fragmentation also (by using path MTU discovery) since > fragmentation is so expensive for routers or receivers. > > Somesh > > > -----Original Message----- > > From: hufferd@us.ibm.com [mailto:hufferd@us.ibm.com] > > Sent: Thursday, August 24, 2000 9:19 AM > > To: Black_David@emc.com > > Cc: ips@ece.cmu.edu > > Subject: FW: Re: iSCSI: SCTP Switch and Router support > > > > > > David, > > A lot of what you say is true, however, the word "Most" is > > the key. There > > are a number of Switches, NATs, Proxies, and Load Balancers > > that actually > > care about TCP/IP. I do not believe they will handle the SCTP in a > > transparent manor. > > > > . > > . > > . > > John L. Hufferd > > Senior Technical Staff Member (STSM) > > IBM/SSD San Jose Ca > > (408) 256-0403, Tie: 276-0403 > > Internet address: hufferd@us.ibm.com > > Notes address: John Hufferd/San Jose/IBM @ IBMUS > > VM address: hufferd at IBMUSM54 > > > > > > Black_David@emc.com on 08/24/2000 08:54:46 AM > > > > To: John Hufferd/San Jose/IBM@IBMUS > > cc: ips@ece.cmu.edu > > Subject: iSCSI: SCTP Switch and Router support > > > > > > > > John, > > > > > 3. We will need to get the Switch and Router folks to also > > support SCTP > > and > > > to do it in Hardware so that the speed and throughput can > > be maintained. > > I > > > do not see that happening, at least not right away, when we > > need to get > > > volumes moving in order to validate the IP storage SAN concept. > > > > I don't understand this. Most switches and routers > > have no knowledge of whether TCP, UDP, or even IPsec > > encapsulated (and hence opaque) traffic is flowing > > through them. What's special about SCTP -- is this > > about layer 4 and higher switches/routers? > > > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > > black_david@emc.com Cellular: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > > __________________________________________________ Do You Yahoo!? Talk to your friends online with Yahoo! Messenger. http://im.yahoo.com From owner-ips@ECE.cmu.edu Fri Aug 25 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13808; 25 Aug 2000 14:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03037 for ips-outgoing; Fri, 25 Aug 2000 11:13:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03024 for ; Fri, 25 Aug 2000 11:13:27 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e7PFDQw17047 for ; Fri, 25 Aug 2000 10:13:26 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA07246; Fri, 25 Aug 00 10:11:11 -0500 Message-Id: <10008251511.AA07246@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Connection Consensus Progress In-Reply-To: Message from Black_David@emc.com of "Fri, 18 Aug 2000 16:49:52 EDT." <0F31E5C394DAD311B60C00E029101A0704100EC3@corpmx9.isus.emc.com> Date: Fri, 25 Aug 2000 10:12:01 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry this is a little late, I haven't had a chance to send email in couple days. > (B) Should iSCSI have a session abstraction that > binds multiple TCP connections into one > iSCSI connection? You already know this, but I'd say no. > R1) Parallel transfers to/from and failover support for > tape devices. In contrast to disks, multiple SCSI > connections to the same tape do not work (e.g., > blocks can be written in the wrong order). I'd like to hear from a tape guru who believes that this a) is important b) will work. My limited experience in tape is that neither is the case. The tape drivers I have dug into use only a single SCSI command at a time and rely on read-ahead and write-behind buffering in the device to keep the performance up. Assuming that this is the case the performance portion of R1) is subsumed by R2) (parallelism for a single SCSI data transfer across multiple links), and the failover support is equivalent to R4). Plus: > R1) and R2) are beyond the capabilities of existing SCSI- > based systems (note that a parallel bus is a single link). iSCSI is hard enough as it is, I don't see the point of making it harder just to provide a capability which has not yet proven wide applicability. > R2) Obtaining parallelism for a single SCSI command > across multiple transport connections using > different physical links. As I have mentioned before, I believe that physical link speeds will increase at a more than adequate rate, so even if this feature is designed in, it will not be widely used. We have already seen a huge acceleration in the rate at which faster links are coming, and iSCSI (+ hardware TCP or equivalent) will only increase that rate. I also think multiple adapter/connections per session will be incapable of delivering better performance in common circumstances. One reason is that in order to get good throughput on a link, you need to ensure that the operation is large enough to a) mask fixed processing latencies b) ensure sufficient outstanding credit on each link to mask the latency of returning additional credit. If you are using N links, your minimum optimal SCSI operation may be up to N times as large. The N times as large case will only occur if there is ONLY a single SCSI op outstanding at a time (the tape case), because none of the network latencies will be masked by previous and subsequent operations. If, in the typical case, there are multiple outstanding operations, the minimum optimal SCSI operation will not be N times as large, but it will still need to be larger than the single link case because of whatever critical path overhead comes from processing N times as many credit flows. My experience with current FC targets and various OS initiators is that the size of single SCSI operations from a typical file system is already on small side for a single short gigabit link. The typical operation size is usually somewhat immutable for a particular OS. It's usually wedded to fundamental memory management design decisions. We've been on the wrong side of the `if only the OS would give me bigger operations, we could really kick ass' enough times that it seems like a fools game to hope for that. OS initiators ARE capable of generating lots of concurrent transfer demand, but it's usually with more outstanding commands rather than fewer, larger ones. See R5) below. iSCSI is intended to work on networks with larger latencies (i.e. bigger) than the current batch of storage technologies, so the link latency effects will become even more pronounced than is commonly expected now. We have seen substantial overall performance degradation on FC running @ 40 km [contrary to the Pittsburgh meeting minutes, Finisar makes FC transceivers that go 40+km, and maybe other companies do too], even with a large pool of link credits, because of inadequate transfer demand to mask the link latency. Finally, the `iSCSI is hard enough without tackling additional capabilities of unproven merit' argument applies to this too. > R3) Obtaining parallelism for a single SCSI command > across multiple transport connections using the > same physical links. > R3) needs more explanation, as TCP is known to be able > to saturate Gigabit Ethernet, given enough data to > transfer. Is the argument for R3) that for the > transfer sizes likely to be seen in iSCSI, TCP > spends enough of its time in slow start and the > like that multiple TCP connections gain performance? My hunch is that doing this is horribly poor network citizenship. If there is a way to get more performance out of a single end to end connection, it's the transport's (TCP's) responsibility to get it. Running multiple connections to end run TCP's congestion avoidance algorithms has the potential to either slow everybody down or make the network unstable (which will certainly slow everybody down too). For that reason, I would suggest that iSCSI should categorically prohibit this behavior. If you want to live by the sword (operate well on a general network), you have to die by the sword (put up with the inefficiencies required to keep the network healthy). > R4) Optimize failure handling, so that a single TCP > connection loss doesn't immediately translate > into a SCSI error visible to higher level > (time-consuming) recovery logic. This seems like a straw-man for several reasons. First, this requirement suggests that the SCSI layer is not well adapted to handle errors. A major part of any SCSI layer is all about error handling. However, SCSI layers usually assume that the low level driver will make allowances for handling media-specific conditions. The big problem with non-fatal FC conditions causing fatal SCSI errors was inadequate FC layer engineering. Early FC drivers badly abused the hospitality of the upper SCSI layers. For example, an event like a LIP (or any other link level event) typically had some finite duration and was directly detectable by the driver, so stupid drivers would detect the link failure and immediately return the SCSI operation with a retriable error code. The retry operation would come back to the FC driver which would then observe that the link was still down and fail the operation retriably again. This would burn through the retry count instantly and result in a hard error. More subtle was when a LIP caused other nodes to LIP themselves, at some substantial interval later, often to work around implementation bugs (can you say Tachyon?). This would lead to many link up/down transitions in a short period of time. This is not a hard problem to solve, but many early driver writers did not contemplate how horrible it was going to be out there on the loop. One very large company even went so far as to say that FC-AL could never be implemented reliably and the only solution was to make sure all their FC was fabric just because they got surprised by the LIP storms. A connection drop in iSCSI is essentially a `media' event, and an iSCSI driver should not immediately fail subsequent operations to the addressed target without attempting to reestablish the connection first. We make this same assumption in SST. In fact, SST goes so far as to specify that blowing away a connection by either end is a perfectly acceptable and expected error recovery strategy in the case of some infrequent non-nominal conditions. Second, I do not believe multiple connections will work effectively to handle errors which can not be handled with appropriate connection failure recovery strategies. There are actually two cases. The first is a single interface with multiple connections (which I already suggested should be outlawed in response to R3). In this case, when one connection fails, so will the other. The second is multiple interfaces, each with a single connection. In this case, the broken connection must be discovered before any form of recovery can occur for the transfers on it. Having multiple open connections does not reduce the length of the critical path for recovery, so supporting multiple connections per iSCSI session can not satisfy this requirement. > R5) Obtaining parallelism between multiple SCSI commands > across multiple transport connections using > different physical links. I do not see that this offers anything which can not be achieved with multiple iSCSI sessions using different physical links. The only thing it seems to give potentially is link aggregation in the case where all commands are sent to the target using ordered queue instead of simple queue. I've never seen this happen. Has anybody else? Disk drivers use simple queue when they don't care, and some form of synchronous behavior (unqueued, or just sending one command at a time) when they care about order. If the commands are simple queue, it doesn't matter whether they're sent in a single session or multiple sessions. The tape case is discussed under R1). > Those against should check that none of R1-R4 are important enough > to be requirements. I have also argued in some cases that multiple connections per iSCSI session would not be capable of effectively satisfying the requirements. Don't get me wrong, I'm not arguing that link aggregation is a bad thing. It would be great if somehow (magically) it just worked. It would be a nice selling point for iSCSI whether or not it is actually widely used. I AM arguing that any straightforward proposal is unlikely to deliver on the promise for reasons which are beyond the control of the iSCSI standard. And, more complexity in the standard will slow down its deployment. Steph From owner-ips@ECE.cmu.edu Fri Aug 25 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13805; 25 Aug 2000 14:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03749 for ips-outgoing; Fri, 25 Aug 2000 11:27:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA02820 for ; Fri, 25 Aug 2000 11:09:08 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Fri, 25 Aug 2000 08:09:54 -0700 Message-ID: From: Robert Snively To: "'John Matze'" , "'ips@ece.cmu.edu'" Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 25 Aug 2000 08:07:41 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk While networks lose packets, TCP/IP connections never do, so you should not see this as a problem at the iSCSI layer. If you can lose packets at the iSCSI layer, then having to guess at whether the recovery will involve two kinds of information units or one kind of information unit seems like an unnecessary complexity. This should be kept simple, providing one kind of information in a packet, thus making it simpler to implement hardware that can route the information directly to the appropriate memory space. If the overhead of preparing and sending two packets instead of one is burdensome, then we have that problem to solve as well. Bob Snively Brocade Communications Phone 408 487 8135 1901 Guadalupe Parkway San Jose, CA 95131 Email rsnively@brocade.com > -----Original Message----- > From: John Matze [mailto:john.matze@veritas.com] > Sent: Thursday, August 24, 2000 11:29 PM > To: 'ips@ece.cmu.edu' > Subject: RE: Data in SCSI Response or SCSI Data > > > Again I have to agree. Not everyone writing this protocol > is developing a > hardware device. Every time I (or any hardware vendor) > sends one less IP > packet down the wire is one less chance of it getting lost. > > > -----Original Message----- > From: Matthew Jacob [mailto:mjacob@feral.com] > Sent: Thursday, August 24, 2000 5:08 PM > To: csapuntz@cisco.com > Cc: 'ips@ece.cmu.edu' > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > If I remember the discussion correctly, I remember some folks who > > were implementing hardware did not like having 3 different > > mechanisms for doing the same thing, as this required more logic > > and verification. > > > > From a network performance standpoint, there is minimal difference > > between sending two iSCSI messages back-to-back and one larger > > iSCSI message. > > > > In fact, both iSCSI messages might even share the same packet. > > That's a fair point. In other contexts (FC in this case), > I've really liked > being able (in target mode) to send back data && status in > packet- it means > I > don't have to manage multiple resources on the send side, > and in the case of > sending back canned data for 'trivial' commands (like Mode > data), I don't > even > have to do a context or thread switch. > > What this might like like in an IPS implementation is not > clear to me, but I > believe I do see the point about the h/w implementation issues. > > It's not clear to me that you can state as a general rule > that two packets > back-to-back is minimally different- not from a performance > point of view, > but > from a likelihood of lossage and retransmit. If a high > proportion of your > trivial (or not so trivial) read commands come back in one > frame, that's a > lot > better than managing lost frames, no? > > -matt > > From owner-ips@ECE.cmu.edu Fri Aug 25 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13800; 25 Aug 2000 14:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03198 for ips-outgoing; Fri, 25 Aug 2000 11:16:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03185 for ; Fri, 25 Aug 2000 11:16:29 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id RGJL075W; Fri, 25 Aug 2000 08:14:39 -0700 From: "Y P Cheng" To: , Cc: Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 25 Aug 2000 08:16:07 -0700 Message-ID: <000201c00ea7$644bf240$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, I see your point in software implementation. May be this is a good time to introduce the concept of "Good Status" flag bit. In fact, 99.99% of iSCSI reads should be completed without error. For the normal completions, all we need to is a flag bit in the data packet message header indicating "Everything is OK. This SCSI read is now completed normally." If this flag bit is not set, there will be a status packet with sense data following. In the hardware implementation, after transferring data packet into the buffer, if this flag bit is set, we interrupt the microcode which post the Request Block (RB) completion interrupt to the device driver. If the bit is not set, the hardware does nothing after data transfer. The status packet arrived later will be passed to the microcode that, in turn, generates a proper RB completion interrupt to pass the error information to the device driver. Y.P. Cheng, CTO, ConnectCom Solutions Corp. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Matthew Jacob Sent: Thursday, August 24, 2000 5:08 PM To: csapuntz@cisco.com Cc: 'ips@ece.cmu.edu' Subject: Re: Data in SCSI Response or SCSI Data That's a fair point. In other contexts (FC in this case), I've really liked being able (in target mode) to send back data && status in packet- it means I don't have to manage multiple resources on the send side, and in the case of sending back canned data for 'trivial' commands (like Mode data), I don't even have to do a context or thread switch. What this might like like in an IPS implementation is not clear to me, but I believe I do see the point about the h/w implementation issues. It's not clear to me that you can state as a general rule that two packets back-to-back is minimally different- not from a performance point of view, but from a likelihood of lossage and retransmit. If a high proportion of your trivial (or not so trivial) read commands come back in one frame, that's a lot better than managing lost frames, no? -matt From owner-ips@ECE.cmu.edu Fri Aug 25 19:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14005; 25 Aug 2000 15:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA12077 for ips-outgoing; Fri, 25 Aug 2000 14:03:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from prodweb.crossroads.com ([63.237.99.242]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id OAA12058 for ; Fri, 25 Aug 2000 14:03:23 -0400 (EDT) Received: by www.crossroads.com with Internet Mail Service (5.5.2650.21) id ; Fri, 25 Aug 2000 13:08:50 -0700 Message-ID: <670C86291BFCD311AFE500508B94D4B6630BFD@hqmail1.crossroads.com> From: Robert Reynolds To: "'ips@ece.cmu.edu'" Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 25 Aug 2000 11:05:33 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Matthew Jacob [mailto:mjacob@feral.com] > > > From: Y P Cheng > > > > Matt, I see your point in software implementation. May be > > this is a good time to introduce the concept of "Good > > Status" flag bit. In fact, 99.99% of iSCSI reads should > > be completed without error. For the normal completions, > > all we need to is a flag bit in the data packet message > > header indicating "Everything is OK. This SCSI read is now completed > > normally." If this flag bit is not set, there will be a > > status packet with sense data following. In the hardware > > implementation, after transferring data packet into the > > buffer, if this flag bit is set, we interrupt the microcode > > which post the Request Block (RB) completion > > interrupt to the device driver. If the bit is not set, the > > hardware does nothing after data transfer. The status packet > > arrived later will be passed to the microcode that, in turn, > > generates a proper RB completion interrupt to pass the error > > information to the device driver. > > Sounds good to me. If it makes the h/w easier- great. > Although full status isn't that much extra, I could see that > h/w for it would be harder. > Although it's true that 99.99% of reads will be successful it is not always true that the status can be implemented with one bit. If you are doing Tape Reads using variable block records then you also need to ensure that the underrun data length is included as part of the status. Another thought on why it would be a good thing to combine status and data is for congestion control. Either the sender has to wait for more data in the window before sending the status or it is sending a lot of small TCP packets containing the status which is probably not what we would want to see on the network. By combining the status with the last data frame you can avoid this potential problem. Bob <><><><><><><><><><><><><><><> Robert Reynolds Crossroads Systems Inc. (512) 928-7222 (512)928-7401 (fax) robertr@crossroads.com From owner-ips@ECE.cmu.edu Fri Aug 25 20:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14191; 25 Aug 2000 16:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14086 for ips-outgoing; Fri, 25 Aug 2000 14:52:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14082 for ; Fri, 25 Aug 2000 14:52:40 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA04515 for ; Fri, 25 Aug 2000 11:52:37 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.144.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA15080 for ; Fri, 25 Aug 2000 11:52:36 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA12354; Fri, 25 Aug 2000 11:52:37 -0700 (PDT) Message-Id: <200008251852.LAA12354@ha10nwk.EBay.Sun.COM> Date: Fri, 25 Aug 2000 11:52:37 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Connection Consensus Progress To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: TbTx3ENiI3mJbdfhGvOIdA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Just for the record, I will restate that I prefer to have a single TCP connection per initiator/target session. The complexity of error handling greatly outweighs the perceived performance implications. Also any performance problems are not fundemental limitations of TCP but issues in existing implementations, complicating a protocol to cover bad/lazy implementations is a bad idea. iSCSI should not try to end-run congestion control, creating and managing multiple connections may cause unexpected interactions with congestion control algorithms and other QOS mechanisms. I don't know this to be true but we must be certain we are not causing problems before we go down this path. -David From owner-ips@ECE.cmu.edu Fri Aug 25 20:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14265; 25 Aug 2000 16:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA14985 for ips-outgoing; Fri, 25 Aug 2000 15:13:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA14981 for ; Fri, 25 Aug 2000 15:13:40 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e7PJDdw24193 for ; Fri, 25 Aug 2000 14:13:39 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA07399; Fri, 25 Aug 00 14:11:25 -0500 Message-Id: <10008251911.AA07399@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data In-Reply-To: Message from julian_satran@il.ibm.com of "Fri, 25 Aug 2000 19:35:07 +0300." Date: Fri, 25 Aug 2000 14:12:09 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Is there anything preventing your hypothetical hardware implementor > to send always good status within the last block of data? It depends upon the RDMA mechanism you are using. I admit that I have not studied them in detail, other than ST of course, which is essentially an transport protocol based upon a particular RDMA mechanism. I will try to do so soon to determine if I'm all wet. However, assuming that the RDMA mechanism operates on a per-datagram basis, I can only imagine that you will not be able to append `general delivery' data to the end of an RDMA datagram. In this case, you will need a separate datagram to ensure that status is delivered through a separate path from the data. So, while I think concatenating status and data in its general form is not a good idea, a good-status fast path (like the success bit) is definitely the right way to think about it. Nonsuccess SCSI status is so rare that any compromise you can make in the nonsuccess path to make the success path go faster is worth it. Steph From owner-ips@ECE.cmu.edu Fri Aug 25 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14527; 25 Aug 2000 17:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA19392 for ips-outgoing; Fri, 25 Aug 2000 16:54:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA19382 for ; Fri, 25 Aug 2000 16:54:37 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 93D828E87; Fri, 25 Aug 2000 13:53:19 -0700 (PDT) To: ips@ece.cmu.edu Subject: Some suggested requirements on the transport layer Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 25 Aug 2000 13:53:18 -0700 Message-ID: Lines: 26 Sender: owner-ips@ece.cmu.edu Precedence: bulk To move the requirements discussion along, I would like to suggest the following requirements and non-requirements for the transport protocol on which iSCSI rests. REQUIREMENTS _ Ability to segment application-layer messages across multiple packets _ Reliable delivery of application-layer messages _ Congestion control _ Fairly shares the network with other TCP streams _ Latency comparable to TCP (in a comparable implementation) _ Throughput comparable to TCP (in a comparable implementation) _ Avoids IP fragmentation NON-REQUIREMENTS but perhaps desirable: _ RDMA abstraction _ Unreliable or not-totally reliable delivery modes _ In-order delivery of application-layer message _ Fail-over across IP addresses _ Minimal reassembly buffers _ Load-balancing across IP addresses -Costa From owner-ips@ECE.cmu.edu Fri Aug 25 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14529; 25 Aug 2000 17:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA18019 for ips-outgoing; Fri, 25 Aug 2000 16:23:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA18013 for ; Fri, 25 Aug 2000 16:23:25 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id RGJL08GM; Fri, 25 Aug 2000 13:21:35 -0700 From: "Y P Cheng" To: "Robert Reynolds" , Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 25 Aug 2000 13:23:00 -0700 Message-ID: <002001c00ed2$4355ee80$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: <670C86291BFCD311AFE500508B94D4B6630BFD@hqmail1.crossroads.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: Robert Reynolds > > Although it's true that 99.99% of reads will be successful it is > not always true that the status can be implemented with one bit. > If you are doing Tape Reads using variable block records then > you also need to ensure that the underrun data length is included > as part of the status. No, this is not a problem. In reading, the SCSI initiator knows how many bytes it has received and will report the underflow condition correctly. The SCSI target needs not report the transfer size. On a write operation, the SCSI target always return a status message in which it stores the residue count, the amount of data not written. > Another thought on why it would be a good thing to combine status > and data is for congestion control. Either the sender has to wait > for more data in the window before sending the status or it is sending > a lot of small TCP packets containing the status which is probably > not what we would want to see on the network. By combining the status > with the last data frame you can avoid this potential problem. Yes. However, this is how the hardware handles the bundled packets. 1) the whole packet is in the receive FIFO. 2148 bytes for the FC and 1500 for Ethernet. 2) the media header is decoded to get the FC or Ethernet media control information. 3) the IP header is decoded from which we learn if there is a iSCSI, FTP, or VI header. 4) from the iSCSI header we direct the command and status to the microcode and move the data to a pre-defined receive buffer (This is where TCP/RDMA will be handled.) Note, at step 4, the hardware parses the packet. Normally, the hardware does one thing well repeatedly. If there is a status message tagged along the data message, the hardware must first perform DMA data transfer. Then, it repeats step 4 for the status message. It is quite doable if we got a programmable sequencer in the hardware. However, what happen if we got a DMA transfer error, how do we report to the microcode that besides the DMA transfer error, there is still a status message left in the receive FIFO? It gets a little more complicated. From owner-ips@ECE.cmu.edu Fri Aug 25 21:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14549; 25 Aug 2000 17:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA18541 for ips-outgoing; Fri, 25 Aug 2000 16:35:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch11.nsmg.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA18535 for ; Fri, 25 Aug 2000 16:35:26 -0400 (EDT) Received: by lmoxch11.nsmg.veritas.com with Internet Mail Service (5.5.2448.0) id ; Fri, 25 Aug 2000 16:36:10 -0400 Message-ID: From: Roger Cummings To: "'Stephen Bailey'" Cc: "'ips@ece.cmu.edu'" Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 25 Aug 2000 16:36:10 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, I think that you're right about all RDMA mechanisms not being able to append "general delivery" data to the end of a transfer. Years ago, before many of the current FC controller chips were designed, I seem to remember a discussion in T11 where a bunch of hardware guys demonstrated that this would add significant complexity even if it was possible. I think that they also showed that there were pathological cases where "no errors" would be reported where in fact an error would later be detected. I don't guarantee that the same pathological cases exist in iSCSI, of course, and I don't know if the implementers would say the same now given experience with current hardware. What I've wondered since then is if its possible to add a flag to a data transfer that says in effect "all of the data transfers previous to this one completed with no error and no good status was transferred". Once an error is detected, this flag is reset, and then ALL status (even good status) is transferred for subsequent commands until there is a specific reset action. Yes, I know that's another state machine, but it is a fairly simple one, and it ought to be fairly efficient if errors are as rare as everybody thinks. I don't think this has to be done on a per Initiator basis, either. Regards, Roger Cummings VERITAS Software -----Original Message----- From: Stephen Bailey [mailto:steph@cs.uchicago.edu] Sent: Friday, August 25, 2000 3:12 PM To: ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data > Is there anything preventing your hypothetical hardware implementor > to send always good status within the last block of data? It depends upon the RDMA mechanism you are using. I admit that I have not studied them in detail, other than ST of course, which is essentially an transport protocol based upon a particular RDMA mechanism. I will try to do so soon to determine if I'm all wet. However, assuming that the RDMA mechanism operates on a per-datagram basis, I can only imagine that you will not be able to append `general delivery' data to the end of an RDMA datagram. In this case, you will need a separate datagram to ensure that status is delivered through a separate path from the data. So, while I think concatenating status and data in its general form is not a good idea, a good-status fast path (like the success bit) is definitely the right way to think about it. Nonsuccess SCSI status is so rare that any compromise you can make in the nonsuccess path to make the success path go faster is worth it. Steph From owner-ips@ECE.cmu.edu Fri Aug 25 21:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14544; 25 Aug 2000 17:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA16846 for ips-outgoing; Fri, 25 Aug 2000 15:57:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA16837 for ; Fri, 25 Aug 2000 15:57:11 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA66314; Fri, 25 Aug 2000 15:55:12 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id NAA68476; Fri, 25 Aug 2000 13:57:09 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: Connection Consensus Progress To: David Robinson Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 25 Aug 2000 12:53:47 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/25/2000 01:57:08 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I agree with you up to a point. I know of customers that always need multiple physical paths to the Storage Controller. Regardless of how fast the link is, they need a faster link, and these hosts need to be able to spread the load across several different HBAs. (Some are on one PCI bus, and some on another, etc.) When this happens, as it does today, with Fibre Channel, we are required, as are a number of other vendors, to come up with a multi HBA balancer. We call our Fibre Channel version "DPO" (Dynamic Path Optimizer), EMC has another version (I do not know what they call theirs). This Code sits as a "Wedge" Driver above the FC Device Drivers and balances the work across the different FC HBAs. I think this same thing will be required in the iSCSI situation. Note:I think, the FC versions only work with IBM or EMC's etc. Controllers. (SUN probably has a similar one also.) If possible I would like to avoid adding a vendor specific wedge driver, or better yet avoid any kind of an additional wedge driver with the iSCSI, by building the same capabilities into the Transport layer. Having said that, I would like to see the multiple Connection per Session (as it applies to multiple Ethernet Ports) kept in the spec. with the understanding that one connection per session is a proper subset. If we did that, the simpler and early implementations can get to market quickly, and we can use things like DPO for a while. I would just like vendors that could write the multiple connection iSCSI Device Drivers to actually do it, and then load balance between the HBAs without depending on external Wedge Device Drivers. . . . John L. Hufferd David Robinson @ece.cmu.edu on 08/25/2000 11:52:37 AM Please respond to David Robinson Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Connection Consensus Progress Just for the record, I will restate that I prefer to have a single TCP connection per initiator/target session. The complexity of error handling greatly outweighs the perceived performance implications. Also any performance problems are not fundemental limitations of TCP but issues in existing implementations, complicating a protocol to cover bad/lazy implementations is a bad idea. iSCSI should not try to end-run congestion control, creating and managing multiple connections may cause unexpected interactions with congestion control algorithms and other QOS mechanisms. I don't know this to be true but we must be certain we are not causing problems before we go down this path. -David From owner-ips@ECE.cmu.edu Fri Aug 25 21:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14616; 25 Aug 2000 17:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA19550 for ips-outgoing; Fri, 25 Aug 2000 16:59:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA19540 for ; Fri, 25 Aug 2000 16:58:54 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id DA7F98E87; Fri, 25 Aug 2000 13:57:36 -0700 (PDT) To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 25 Aug 2000 13:57:36 -0700 In-Reply-To: julian_satran@il.ibm.com's message of "Fri, 25 Aug 2000 13:57:43 +0300" Message-ID: Lines: 16 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, What kind of reads are you talking about? I assume you are talking about reads to the socket READ API? -Costa julian_satran@il.ibm.com writes: > Any software implemementation will have 2 reads/iSCSI message. > As we asume a large number of software implementations being used as > initiators having the double the number of reads for small I/O is not wise. > Hardware will easily manage the different read modes. > > Julo From owner-ips@ECE.cmu.edu Fri Aug 25 21:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14628; 25 Aug 2000 17:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA18423 for ips-outgoing; Fri, 25 Aug 2000 16:32:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA18407 for ; Fri, 25 Aug 2000 16:32:24 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id NAA11119 for ; Fri, 25 Aug 2000 13:32:24 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.188.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id NAA21815 for ; Fri, 25 Aug 2000 13:32:23 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id NAA06166; Fri, 25 Aug 2000 13:32:23 -0700 (PDT) Message-Id: <200008252032.NAA06166@ha10nwk.EBay.Sun.COM> Date: Fri, 25 Aug 2000 13:32:23 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Connection Consensus Progress To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 0X7N2ldTr02wlRr8pD3+Pg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > I agree with you up to a point. I know of customers that always need > multiple physical paths to the Storage Controller. Regardless of how fast > the link is, they need a faster link, and these hosts need to be able to > spread the load across several different HBAs. (Some are on one PCI bus, > and some on another, etc.) When this happens, as it does today, with Fibre > Channel, we are required, as are a number of other vendors, to come up with > a multi HBA balancer. We call our Fibre Channel version "DPO" (Dynamic > Path Optimizer), EMC has another version (I do not know what they call > theirs). This Code sits as a "Wedge" Driver above the FC Device Drivers > and balances the work across the different FC HBAs. I think this same > thing will be required in the iSCSI situation. Note:I think, the FC > versions only work with IBM or EMC's etc. Controllers. (SUN probably has a > similar one also.) I understand this scenerio, it is often used as a high availability feature. The key question is if this should be handled above at the SCSI layer as it is most often done now, or in the iSCSI transport. While I like the goal to unify this into one architecture, I have serious doubts that we have the understanding of the requirements and needs necessary to get it right. Thus we will ultimately end up in the same situation we are in today with a SCSI layer solution. In addition, if the promise of a hardware iSCSI NIC/HBA is acheived, allowing multiple paths using different NIC/HBAs will still require "wedge" software. -David From owner-ips@ECE.cmu.edu Fri Aug 25 21:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14633; 25 Aug 2000 17:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA19724 for ips-outgoing; Fri, 25 Aug 2000 17:03:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA19719 for ; Fri, 25 Aug 2000 17:02:59 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id RGJL08H6; Fri, 25 Aug 2000 14:01:23 -0700 From: "Y P Cheng" To: "Stephen Bailey" , Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 25 Aug 2000 14:02:47 -0700 Message-ID: <002601c00ed7$d254bc60$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: <10008251911.AA07399@candide.cs.uchicago.edu> Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: Stephen Bailey > So, while I think concatenating status and data in its general form is > not a good idea, a good-status fast path (like the success bit) is > definitely the right way to think about it. Nonsuccess SCSI status is > so rare that any compromise you can make in the nonsuccess path to > make the success path go faster is worth it. To support your argument, let me describe how the hardware works in setting the good status flag bit: In hardware for each outgoing datagram, there is an exchange table entry describing the payload size, the total data amount (packet size), and the information to build a media header plus a "device header" which typically is protocol dependent. The microcode builds the Upper Level Protocol (ULP) header. Therefore, setting a "Good Status" flag bit in a iSCSI header is a function of the microcode of a NIC adapter. When a packet is broken down into multiple frames (or messages) the hardware is smart not to repeat the ULP header. Therefore, for a successful read, the microcode builds the iSCSI header with a good status flag bit set and activate the hardware to send the data out. The hardware puts the iSCSI header in the first frame of the data packet. The microcode reports back to the NIC device driver that everything is done. For an unsuccessful read, the microcode first transmits whatever data it has. Then, it builds a status message with sense data. Finally, it activates the hardware to ship it out -- with proper media and IP headers -- and reports to the NIC device driver. > From: David Robinson > iSCSI should not try to end-run congestion control, creating and managing > multiple connections may cause unexpected interactions with congestion > control algorithms and other QOS mechanisms. I don't know this to > be true but we must be certain we are not causing problems before > we go down this path. Yes, iSCSI should only deal with SCSI command, data, and status encapsulation in IP packets. Let TCP or other ULP's deal with reliable delivery, multi-path, load balance, security, transmission errors, and recovery. From owner-ips@ECE.cmu.edu Fri Aug 25 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14904; 25 Aug 2000 18:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA19973 for ips-outgoing; Fri, 25 Aug 2000 17:08:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA19966 for ; Fri, 25 Aug 2000 17:08:43 -0400 (EDT) Received: from zeppo.feral.com (IDENT:mjacob@zeppo [192.67.166.71]) by feral.com (8.9.3/8.9.3) with ESMTP id OAA14282; Fri, 25 Aug 2000 14:08:36 -0700 Date: Fri, 25 Aug 2000 14:05:09 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: Robert Reynolds cc: "'ips@ece.cmu.edu'" Subject: RE: Data in SCSI Response or SCSI Data In-Reply-To: <670C86291BFCD311AFE500508B94D4B6630BFD@hqmail1.crossroads.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > > Although it's true that 99.99% of reads will be successful it is > not always true that the status can be implemented with one bit. > If you are doing Tape Reads using variable block records then > you also need to ensure that the underrun data length is included > as part of the status. That's a Check Condition, ergo, a non-zero Status, and a single bit can inform that another frame is coming. -matt From owner-ips@ECE.cmu.edu Fri Aug 25 23:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15080; 25 Aug 2000 19:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA21802 for ips-outgoing; Fri, 25 Aug 2000 18:02:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA21796 for ; Fri, 25 Aug 2000 18:02:00 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 25 Aug 2000 18:01:29 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F0A@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI: "Wedge" drivers Date: Fri, 25 Aug 2000 18:01:28 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk I guess I need to take my co-chair hat off and say a few things about EMC's and other "Wedge" drivers. Wedge drivers are motivated by fault tolerance and fail-over in addition to load balancing/spreading. EMC actually has two products, PowerPath for Symmetrix (does all three), and ATF for Clariion (fail-over only). Other products include HDS's SafePath, Veritas's DMP and HP's PV Links. Apologies to those I've left out. John's goal of eliminating of wedge drivers via iSCSI sessions may not be achievable in practice due to fault tolerance and fail-over concerns. The basic fault tolerance/failover concern arises from the requirement that failure of one interface processor in the disk array should not disable all access to the host's storage. Hence the host to storage connectivity usually encompasses more than one interface processor, facing array designers with the choice of using multiple SCSI connections and keeping the SCSI state in each interface processor vs. sharing the SCSI state across multiple interface processors, and making sure everything works right when one of them fails in an arbitrary fashion (ouch). The former is considerably easier to implement, and requires a wedge driver on the host side. The corresponding issue of how a host deals with possible failure of an iSCSI HBA has already been noted, and I agree that the most likely approach is a wedge driver. Since John works for IBM, I should note that the phrase "interface processor(s)" isn't directly applicable to the IBM Shark array, but nonetheless, this issue is still present -- for fault tolerance, storage access has to be spread across both RS/6000 systems running AIX in a Shark, and sharing SCSI state across AIX instances doesn't sound like an easy thing to do. The bottom line is that failure and fault tolerance concerns make it unlikely (IMHO) that an iSCSI session concept will lead to the extinction of wedge drivers. There have been some questions about difficulty of development of wedge drivers. EMC's experience is that fault tolerance and fail-over require much more effort than load balancing for a couple of reasons. The first is that more complex systems tend to have more complex ways of failing than of working correctly :-(. The second is that correct behavior of a wedge driver depends on the correct behavior of HBA drivers in error cases, and HBA vendors in general do not exhaustively test correct behavior of error cases before releasing drivers -- we find driver bugs on a regular basis. Given the intention to implement much/all of iSCSI in hardware, I suspect that this situation will continue relatively unchanged. I'm not sure what the latter implies for difficulty of development of error handling code for multi-connection iSCSI sessions when a single vendor is responsible for both the hardware and the driver. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Aug 25 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15157; 25 Aug 2000 19:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23448 for ips-outgoing; Fri, 25 Aug 2000 18:54:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23442 for ; Fri, 25 Aug 2000 18:54:18 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 25 Aug 2000 18:53:36 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F0E@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: FW: ipfc discussion of draft-otis-fc-sctp-ip-00.txt Date: Fri, 25 Aug 2000 18:53:34 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk For those not on the ipfc list, here is the discussion of draft-otis-fc-sctp-ip.txt on the use of SCTP and draft-xie-stewart-usctp-00.txt on a proposed unreliable delivery mechanism for SCTP. Anyone replying to this message who quotes or forwards its entire contents will be summarily shot :-). --David ----------------------------------------------------- From: Randall R. Stewart [randall@stewart.chicago.il.us] Sent: Thursday, August 24, 2000 9:47 AM To: ipfc@standards.gadzoox.com Subject: Re:draft-otis-fc-sctp-ip-00.txt Greetings: (I apologize for the earlier subscribe mis-send... I should know better than to do anything before a second cup of coffee in the A.M. :->) I have just finished reading both the above mentioned draft and the draft - draft-ietf-ipfc-fcoverip-02.txt I think that Douglas did a nice job of summarizing some of the benefits that FC over SCTP/IP can gain in this draft i.e.: - Headers contained within one frame. - Objects aligned at 32-bit boundaries. - Out of sequence frame processing. - Standard authentication. - Independent streams under common control. - Session restart. - Improved error detection. - Prevention of blind spoofing and denial of service attacks. - Standard Heartbeat and multi-homing. (optional) One other things not mentioned was: - Congestion control Now after reviewing the ipfc-fcoverip draft particularly the section that specifies why TCP should not be used, there may be objections to the Otis draft... One of problems mentioned in ipfc-fcoverip is the retransmission by TCP being in conflict to retransmissions that would occur at the FC level. I would like to call your attention to a draft that I think will solve this problem... http://search.ietf.org/internet-drafts/draft-xie-stewart-usctp-00.txt This draft defines a new extension that we have proposed for SCTP that allows a particular stream to be specified has "no retransmission". This way you can mix both completly reliable and unreliable transmissions over the same association AND still maintain correct congestion control policy's in the network. We still are debating the addition of a couple of things to this draft: - Should we have the ability to turn off checksums on selected data portions traveling on a stream. This way one could disable the alder-32 checksum on a unreliable stream packet but not the header, control, or any reliable data. - How do you handle larger than the P-MTU sized messages? Currently the draft dis-allows this since when sending to an unreliable stream you can never re-assemble the larger packet if a piece of one is lost. We may want to think about adding the ability to still fragment and hook in a mechanism to the reliable re-assembly queues to discard in-complete reassembly's where some datagrams are dropped due to the unreliable transmission. This would be desireable to keep from having to do a double P-MTU discover (one at the SCTP layer and the other at the IPFC layer). We would appreciate any comments on the draft and I would be more than glad to present either here on the mailing list or in San Diego a short overview of both SCTP or SCTP-U if this is helpful... Thanks for your time and I would welcome any comments or questions.. R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From: Wayland Jeong [wayland@troikanetworks.com] Sent: Thursday, August 24, 2000 12:42 PM To: 'Randall R. Stewart'; ipfc@standards.gadzoox.com Subject: RE: draft-otis-fc-sctp-ip-00.txt [ stuff deleted ] > Now after reviewing the ipfc-fcoverip draft particularly the > section that specifies why TCP should not be used, there may > be objections to the Otis draft... > > One of problems mentioned in ipfc-fcoverip is the retransmission > by TCP being in conflict to retransmissions that would occur > at the FC level. I would like to call your attention to a > draft that I think will solve this problem... > > http://search.ietf.org/internet-drafts/draft-xie-stewart-usctp-00.txt > > This draft defines a new extension that we have proposed for SCTP > that allows a particular stream to be specified has "no retransmission". > This way you can mix both completly reliable and unreliable transmissions > over the same association AND still maintain correct congestion control > policy's in the network. We still are debating the addition of > a couple of things to this draft: > The problem with this is that you don't want to rely on an I/O level retry to solve congestion problems in the network. If frames are being dropped due to congestion issues in the network (i.e. WRED) then having SCSI suffer a timeout and retry an entire I/O will help exacerbate congestion issues and bring performance to its knees. Depending on the traffic type, an I/O level retry may result in a 64K block retransmission after a timeout which is on the order of seconds. Unless you can absolutely rely on diffserv and/or RSVP or some other QoS mechanism to guarantee reliable delivery, the overall SCSI performance will suffer. I guess you can solve the problem with a well-tuned private network or a bandwidth matched WAN link, but this can be expensive and takes away from the business case for this technology. I think that any protocol which cannot tolerate lost data must have a reliable transport underneath it in order to achieve performance over gigabit networks. I'm not all that familiar with SCTP, but I thought that one of the features that would be useful is the capability of using SACK and message numbers to implement selective retransmission. It seems like, for storage, if we use SCTP as the transport, we want retransmission. -Wayland From: Randall R. Stewart [randall@stewart.chicago.il.us] Sent: Thursday, August 24, 2000 1:17 PM To: Wayland Jeong Cc: ipfc@standards.gadzoox.com Subject: Re: draft-otis-fc-sctp-ip-00.txt Wayland Jeong wrote: > > [ stuff deleted ] > > > Now after reviewing the ipfc-fcoverip draft particularly the > > section that specifies why TCP should not be used, there may > > be objections to the Otis draft... > > > > One of problems mentioned in ipfc-fcoverip is the retransmission > > by TCP being in conflict to retransmissions that would occur > > at the FC level. I would like to call your attention to a > > draft that I think will solve this problem... > > > > http://search.ietf.org/internet-drafts/draft-xie-stewart-usctp-00.txt > > > > This draft defines a new extension that we have proposed for SCTP > > that allows a particular stream to be specified has "no retransmission". > > This way you can mix both completly reliable and unreliable transmissions > > over the same association AND still maintain correct congestion control > > policy's in the network. We still are debating the addition of > > a couple of things to this draft: > > > > The problem with this is that you don't want to rely on an I/O level retry > to solve congestion problems in the network. If frames are being dropped due > to congestion issues in the network (i.e. WRED) then having SCSI suffer > a timeout and retry an entire I/O will help exacerbate congestion issues > and bring performance to its knees. Depending on the traffic type, an I/O > level retry may result in a 64K block retransmission after a timeout which > is on the order of seconds. Unless you can absolutely rely on diffserv > and/or RSVP or some other QoS mechanism to guarantee reliable delivery, > the overall SCSI performance will suffer. > The point of the u-sctp draft is that you DO NOT retransmit the data. You let the upper layer do it (if it so desires). Instead you drop your cwnd (as normal) and move forward the cumulative TSN point. There is NO retransmission on a u-sctp stream. Now we also did not specify in this draft if the upper layer should be informed of data not making it to the peer.. this may be a good thing to add ... > I guess you can solve the problem with a well-tuned private network or a > bandwidth matched WAN link, but this can be expensive and takes away > from the business case for this technology. > > I think that any protocol which cannot tolerate lost data must have a reliable > transport underneath it in order to achieve performance over gigabit networks. > I'm not all that familiar with SCTP, but I thought that one of the features > that would be useful is the capability of using SACK and message numbers > to implement selective retransmission. It seems like, for storage, if we use > SCTP as the transport, we want retransmission. Hmm. thats an option as well.. but if you look at: http://www.ietf.org/internet-drafts/draft-ietf-ipfc-fcoverip-02.txt it specifically states that you DO NOT want retransmission... In either case a SCTP endpoint that supported u-sctp and sctp would be able to have retransmission turned on OR turned off, on any particular stream... and it would be able to have one association doing both at the same time :-> You basically can have it any way you want it :-) R > > -Wayland -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From: Wayland Jeong [wayland@troikanetworks.com] Sent: Thursday, August 24, 2000 10:37 PM To: 'Marjorie Krueger'; Randall R. Stewart Cc: Wayland Jeong; ipfc@standards.gadzoox.com Subject: RE: draft-otis-fc-sctp-ip-00.txt > It seems like people are missing the intended application of FCoverIP - namely, a > switching device, NOT an endpoint. Switching devices are not aware of the > protocol running over FC and must treat every FC packet similarly. SCSI is only > one application that runs over FC. An FCoverIP device can't selectively decide to > retransmit a packet without implementing some pretty complicated policies, and > looking into the headers of each packet, which would degrade performance so much > that it's questionable whether this would be a feasable application at all. This > has been discussed somewhat on the IPS reflector. > Layer 4 through layer 7 LAN switches exist today (Foundry, Alteon, Extreme, etc.) that inspect deep into each packet and make wire-speed decisions about how to service the data. There are routers and firewalls which terminate TCP and thus run entire stacks in the box. I'm not saying that it is trivial to design a port that can keep-up with the wire and run a session protocol in hardware, but it is not out of the question. If it was, iSCSI would be doomed as a high-performance gigabit solution (and that is an entirely different argument). I guess, what I'm trying to say is that someone should clearly state the requirements for FCoverIP. If the requirements are that it simply run on a private tuned network which can virtually guarantee a loss-less medium, then I would argue that we don't need congestion control and hence a retry mechanism. I would also argue that FCoverIP is a niche, proprietary solution which probably does not belong in IETF since it is not intended for the internet. If the requirement is that FCoverIP must co-exist with other LAN/WAN traffic, then I would argue that at a minimum we need a congestion management scheme since the networks depend on it. If we have a need for congestion management that implies that things are getting dropped (unless QoS ensures that does not happen.). And if things are getting dropped, then retries at the SCSI I/O level could severely limit performance. That was the convoluted reason for my comments about needing to have a retry mechanism. I believe that you need a reliable medium for transporting any SCSI-based protocol. > Marjorie Krueger > Vixel Corporation > -Wayland From: Randall R. Stewart [randall@stewart.chicago.il.us] Sent: Friday, August 25, 2000 4:45 AM To: Wayland Jeong Cc: 'Marjorie Krueger'; ipfc@standards.gadzoox.com Subject: Re: draft-otis-fc-sctp-ip-00.txt Wayland: My sole reason for pointing out the unreliable/non-retransmit option of SCTP is because of a working group document. In the document it is very very specific and states that the WG does not like TCP because of the retransmit storms that can occur if TCP and the FC layer both retransmit. I simply offered a solution, and since SCTP could offer both non-retransmit and retransmit it takes this argument off the table... Other comments in-line below: Wayland Jeong wrote: > > > It seems like people are missing the intended application of FCoverIP - namely, a > > switching device, NOT an endpoint. Switching devices are not aware of the > > protocol running over FC and must treat every FC packet similarly. SCSI is only > > one application that runs over FC. An FCoverIP device can't selectively decide to > > retransmit a packet without implementing some pretty complicated policies, and > > looking into the headers of each packet, which would degrade performance so much > > that it's questionable whether this would be a feasable application at all. This > > has been discussed somewhat on the IPS reflector. > > > Layer 4 through layer 7 LAN switches exist today (Foundry, Alteon, Extreme, etc.) that > inspect deep into each packet and make wire-speed decisions about how to service the > data. There are routers and firewalls which terminate TCP and thus run entire stacks in > the box. I'm not saying that it is trivial to design a port that can keep-up with the wire and > run a session protocol in hardware, but it is not out of the question. If it was, iSCSI would > be doomed as a high-performance gigabit solution (and that is an entirely > different argument). > > I guess, what I'm trying to say is that someone should clearly state the requirements for > FCoverIP. If the requirements are that it simply run on a private tuned network which can > virtually guarantee a loss-less medium, then I would argue that we don't need congestion > control and hence a retry mechanism. I would also argue that FCoverIP is a niche, proprietary > solution which probably does not belong in IETF since it is not intended for the internet. > If you ARE on a private net, even if Congestion Control is in place it SHOULD never be engaged. Here a SCTP with non-retransmit will help you. You never have the danger of a retransmit storm, and the fact that your network IS so reliable means you never get the CC algorithms working against you . Also if someone makes a mistake, and puts a private configuration across the big I-Internet it provides a protection.. > If the requirement is that FCoverIP must co-exist with other LAN/WAN traffic, then I would > argue that at a minimum we need a congestion management scheme since the networks > depend on it. If we have a need for congestion management that implies that things are > getting dropped (unless QoS ensures that does not happen.). And if things are getting > dropped, then retries at the SCSI I/O level could severely limit performance. And here, no matter what the performance level, one could turn on the reliable side transport of SCTP since it IS on a WAN and the retransmit on the SCTP level would save you from hitting SCSI I/O retransmits.. > > That was the convoluted reason for my comments about needing to have a retry mechanism. > I believe that you need a reliable medium for transporting any SCSI-based protocol. > > > Marjorie Krueger > > Vixel Corporation > > > > -Wayland R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Fri Aug 25 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15176; 25 Aug 2000 19:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA22996 for ips-outgoing; Fri, 25 Aug 2000 18:40:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from feral.com (IDENT:root@feral.com [192.67.166.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA22990 for ; Fri, 25 Aug 2000 18:40:10 -0400 (EDT) Received: from zeppo.feral.com (IDENT:mjacob@zeppo [192.67.166.71]) by feral.com (8.9.3/8.9.3) with ESMTP id PAA14676; Fri, 25 Aug 2000 15:40:07 -0700 Date: Fri, 25 Aug 2000 15:36:40 -0700 (PDT) From: Matthew Jacob Reply-To: mjacob@feral.com To: Robert Reynolds cc: "'ips@ece.cmu.edu'" Subject: RE: Data in SCSI Response or SCSI Data In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk > > Although it's true that 99.99% of reads will be successful it is > > not always true that the status can be implemented with one bit. > > If you are doing Tape Reads using variable block records then > > you also need to ensure that the underrun data length is included > > as part of the status. > > That's a Check Condition, ergo, a non-zero Status, and a single bit can inform > that another frame is coming. Oops- sorry- was thinking of the ILI case (if you're in fixed block mode) From owner-ips@ECE.cmu.edu Fri Aug 25 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15181; 25 Aug 2000 19:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23095 for ips-outgoing; Fri, 25 Aug 2000 18:42:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23088 for ; Fri, 25 Aug 2000 18:42:34 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2448.0) id ; Fri, 25 Aug 2000 18:41:59 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F0D@corpmx9.isus.emc.com> To: ips@ece.cmu.edu, ipfc@standards.gadzoox.com Subject: RE: draft-otis-fc-sctp-ip-00.txt Date: Fri, 25 Aug 2000 18:41:58 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk A couple of comments: - This discussion should be moved onto the ips mailing list (ips@ece.cmu.edu) as further IETF work on FC-over-IP will occur there. Despite it's name, the fcoverip draft has not been an official work item of the ipfc WG. The use of the ipfc list rather than ips is completely understandable, as the draft was misnamed, nonetheless, further discussion should be on ips. - Wayland wrote: > I guess, what I'm trying to say is that someone should clearly state the requirements for > FCoverIP. If the requirements are that it simply run on a private tuned network which can > virtually guarantee a loss-less medium, then I would argue that we don't need congestion > control and hence a retry mechanism. I would also argue that FCoverIP is a niche, proprietary > solution which probably does not belong in IETF since it is not intended for the internet. Not only is that a good argument, it is in fact basically the position of the IETF, as conveyed by the Area Directors to the authors of the FC-over-IP draft - congestion control is REQUIRED if the work is to advance in the IETF because otherwise the protocol would be unsuitable for the internet. Whether to have a retry mechanism is an engineering decision that can be made as the work progresses (an implementation that didn't have to buffer for retransmission might consume less memory than one that did), but congestion control is REQUIRED, period. Thanks, --David (co-chair of the ips WG-to-be) p.s. For those on ips and not ipfc, I'm going to forward the ipfc discussion in a separate message. --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sat Aug 26 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19304; 26 Aug 2000 11:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA20026 for ips-outgoing; Sat, 26 Aug 2000 09:36:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA20016 for ; Sat, 26 Aug 2000 09:36:14 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id PAA103992 for ; Sat, 26 Aug 2000 15:35:39 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id PAA12016 for ; Sat, 26 Aug 2000 15:35:40 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256947.004AAB03 ; Sat, 26 Aug 2000 15:35:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 26 Aug 2000 13:07:25 +0300 Subject: RE: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The status message is in the iSCSI header (not after the data). If the status is bad and/or includes sense data it should be sent separately. Julo "Y P Cheng" on 25/08/2000 23:23:00 Please respond to "Y P Cheng" To: "Robert Reynolds" , ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Data in SCSI Response or SCSI Data > From: Robert Reynolds > > Although it's true that 99.99% of reads will be successful it is > not always true that the status can be implemented with one bit. > If you are doing Tape Reads using variable block records then > you also need to ensure that the underrun data length is included > as part of the status. No, this is not a problem. In reading, the SCSI initiator knows how many bytes it has received and will report the underflow condition correctly. The SCSI target needs not report the transfer size. On a write operation, the SCSI target always return a status message in which it stores the residue count, the amount of data not written. > Another thought on why it would be a good thing to combine status > and data is for congestion control. Either the sender has to wait > for more data in the window before sending the status or it is sending > a lot of small TCP packets containing the status which is probably > not what we would want to see on the network. By combining the status > with the last data frame you can avoid this potential problem. Yes. However, this is how the hardware handles the bundled packets. 1) the whole packet is in the receive FIFO. 2148 bytes for the FC and 1500 for Ethernet. 2) the media header is decoded to get the FC or Ethernet media control information. 3) the IP header is decoded from which we learn if there is a iSCSI, FTP, or VI header. 4) from the iSCSI header we direct the command and status to the microcode and move the data to a pre-defined receive buffer (This is where TCP/RDMA will be handled.) Note, at step 4, the hardware parses the packet. Normally, the hardware does one thing well repeatedly. If there is a status message tagged along the data message, the hardware must first perform DMA data transfer. Then, it repeats step 4 for the status message. It is quite doable if we got a programmable sequencer in the hardware. However, what happen if we got a DMA transfer error, how do we report to the microcode that besides the DMA transfer error, there is still a status message left in the receive FIFO? It gets a little more complicated. From owner-ips@ECE.cmu.edu Sat Aug 26 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19306; 26 Aug 2000 11:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA20024 for ips-outgoing; Sat, 26 Aug 2000 09:36:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA20008 for ; Sat, 26 Aug 2000 09:36:11 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id PAA103990 for ; Sat, 26 Aug 2000 15:35:39 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id PAA12014 for ; Sat, 26 Aug 2000 15:35:39 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256947.004AAB3C ; Sat, 26 Aug 2000 15:35:35 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 26 Aug 2000 13:19:57 +0300 Subject: Re: Connection Consensus Progress Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I understand and share your concerns about how good we understand the requirements for recovery and balancing. But as I stated repeatedly we can't wait for somebody else to solve our problem and the field requirement is there as witnessed by the products that attempt to solve it in a proprietary fashion (and BTW a TCP connection failure could also be repaired simply by TCP but TCP does not do it). However if they are there the both sets have to be solved at SCSI level - since several links if not handled properly increase the failure probability. However your last point about multiple HBAs is lost on me. We attempted to make iSCSI work with several HBAs and went to some length to keep the requirements to the HBA hardware as if the HBAs act independently (counters can be shared). Is there something we missed? Julo David Robinson on 25/08/2000 23:32:23 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Connection Consensus Progress > I agree with you up to a point. I know of customers that always need > multiple physical paths to the Storage Controller. Regardless of how fast > the link is, they need a faster link, and these hosts need to be able to > spread the load across several different HBAs. (Some are on one PCI bus, > and some on another, etc.) When this happens, as it does today, with Fibre > Channel, we are required, as are a number of other vendors, to come up with > a multi HBA balancer. We call our Fibre Channel version "DPO" (Dynamic > Path Optimizer), EMC has another version (I do not know what they call > theirs). This Code sits as a "Wedge" Driver above the FC Device Drivers > and balances the work across the different FC HBAs. I think this same > thing will be required in the iSCSI situation. Note:I think, the FC > versions only work with IBM or EMC's etc. Controllers. (SUN probably has a > similar one also.) I understand this scenerio, it is often used as a high availability feature. The key question is if this should be handled above at the SCSI layer as it is most often done now, or in the iSCSI transport. While I like the goal to unify this into one architecture, I have serious doubts that we have the understanding of the requirements and needs necessary to get it right. Thus we will ultimately end up in the same situation we are in today with a SCSI layer solution. In addition, if the promise of a hardware iSCSI NIC/HBA is acheived, allowing multiple paths using different NIC/HBAs will still require "wedge" software. -David From owner-ips@ECE.cmu.edu Sat Aug 26 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19368; 26 Aug 2000 11:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA20020 for ips-outgoing; Sat, 26 Aug 2000 09:36:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA20007 for ; Sat, 26 Aug 2000 09:36:10 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA16654 for ; Sat, 26 Aug 2000 15:35:38 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id PAA39630 for ; Sat, 26 Aug 2000 15:35:39 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256947.004AA88E ; Sat, 26 Aug 2000 15:35:28 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 26 Aug 2000 12:28:20 +0300 Subject: Re: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, I assume that your hardware knows when it is going to send the last SCSI datagram. In this case it can insert the status in the datagram header (not at the end) - (and BTW that is how the current draft assumes that things will be done). If you have to send sense-data (bad status) then you will send a separate datagram. Julo Stephen Bailey on 25/08/2000 22:12:09 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Data in SCSI Response or SCSI Data > Is there anything preventing your hypothetical hardware implementor > to send always good status within the last block of data? It depends upon the RDMA mechanism you are using. I admit that I have not studied them in detail, other than ST of course, which is essentially an transport protocol based upon a particular RDMA mechanism. I will try to do so soon to determine if I'm all wet. However, assuming that the RDMA mechanism operates on a per-datagram basis, I can only imagine that you will not be able to append `general delivery' data to the end of an RDMA datagram. In this case, you will need a separate datagram to ensure that status is delivered through a separate path from the data. So, while I think concatenating status and data in its general form is not a good idea, a good-status fast path (like the success bit) is definitely the right way to think about it. Nonsuccess SCSI status is so rare that any compromise you can make in the nonsuccess path to make the success path go faster is worth it. Steph From owner-ips@ECE.cmu.edu Sat Aug 26 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21049; 26 Aug 2000 18:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA29129 for ips-outgoing; Sat, 26 Aug 2000 16:53:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA29122 for ; Sat, 26 Aug 2000 16:53:16 -0400 (EDT) Received: from purol.East.Sun.COM ([129.148.9.11]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id NAA12666 for ; Sat, 26 Aug 2000 13:53:14 -0700 (PDT) Received: from grace (grace [129.148.174.38]) by purol.East.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with SMTP id QAA08576 for ; Sat, 26 Aug 2000 16:53:13 -0400 (EDT) Message-Id: <200008262053.QAA08576@purol.East.Sun.COM> Date: Sat, 26 Aug 2000 16:53:13 -0400 (EDT) From: Christopher Stein - Network Storage Reply-To: Christopher Stein - Network Storage Subject: Re: Connection Consensus Progress To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 7IPRTQHdRizyc5DDxp82ZQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.3.5 SunOS 5.7 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, If one of the reasons for supporting multiple TCP connections is to allow load balancing across HBAs for fault tolerance, this reason will be lost as the TCP processing moves down into the HBA. My understanding was that TCP on a chip technology is desired for high-performance iSCSI. If this is the case, then multiple connections for tolerance of HBA faults is a misfired bullet. -Chris >>X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f >From: julian_satran@il.ibm.com >X-Lotus-FromDomain: IBMIL@IBMDE >To: ips@ece.cmu.edu >Subject: Re: Connection Consensus Progress >Mime-Version: 1.0 >Content-Disposition: inline > > > >David, > >I understand and share your concerns about how good we understand the >requirements for recovery and balancing. > >But as I stated repeatedly we can't wait for somebody else to solve our >problem and the >field requirement is there as witnessed by the products that attempt to >solve it in a >proprietary fashion (and BTW a TCP connection failure could also be >repaired simply by >TCP but TCP does not do it). > >However if they are there the both sets have to be solved at SCSI level - >since several links >if not handled properly increase the failure probability. > >However your last point about multiple HBAs is lost on me. >We attempted to make iSCSI work with several HBAs and went to some length >to keep >the requirements to the HBA hardware as if the HBAs act independently >(counters can be >shared). Is there something we missed? > >Julo > >David Robinson on 25/08/2000 23:32:23 > >Please respond to David Robinson > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Connection Consensus Progress > > > > >> I agree with you up to a point. I know of customers that always need >> multiple physical paths to the Storage Controller. Regardless of how >fast >> the link is, they need a faster link, and these hosts need to be able to >> spread the load across several different HBAs. (Some are on one PCI bus, >> and some on another, etc.) When this happens, as it does today, with >Fibre >> Channel, we are required, as are a number of other vendors, to come up >with >> a multi HBA balancer. We call our Fibre Channel version "DPO" (Dynamic >> Path Optimizer), EMC has another version (I do not know what they call >> theirs). This Code sits as a "Wedge" Driver above the FC Device Drivers >> and balances the work across the different FC HBAs. I think this same >> thing will be required in the iSCSI situation. Note:I think, the FC >> versions only work with IBM or EMC's etc. Controllers. (SUN probably has >a >> similar one also.) > >I understand this scenerio, it is often used as a high availability >feature. >The key question is if this should be handled above at the SCSI layer >as it is most often done now, or in the iSCSI transport. While I like >the goal to unify this into one architecture, I have serious doubts that >we have the understanding of the requirements and needs necessary >to get it right. Thus we will ultimately end up in the same situation >we are in today with a SCSI layer solution. In addition, if the promise >of a hardware iSCSI NIC/HBA is acheived, allowing multiple paths using >different NIC/HBAs will still require "wedge" software. > > -David > > > > > From owner-ips@ECE.cmu.edu Sun Aug 27 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21653; 26 Aug 2000 20:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA00603 for ips-outgoing; Sat, 26 Aug 2000 17:59:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA00599 for ; Sat, 26 Aug 2000 17:59:02 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id RAA16520; Sat, 26 Aug 2000 17:47:04 -0400 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id PAA141578; Sat, 26 Aug 2000 15:58:59 -0600 Importance: Normal Subject: Re: iSCSI: "Wedge" drivers To: Black_David@emc.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.2a (Intl) 23 November 1999 Message-ID: From: "Prasenjit Sarkar/Almaden/IBM" Date: Sat, 26 Aug 2000 14:58:58 -0700 X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Build V504_06062000 |June 6, 2000) at 08/26/2000 02:58:59 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I essentially got the same feedback from a Fibre Channel HBA vendor who also indicated that they would be very unlikely to get into the "wedge driver" business for the reasons stated in the mail below. (Please correct me if I am wrong). I shall reserve my arguments about complexity till the proponents of the multiple-connections per session concept come up with their error handling mechanisms. Sincerely, Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose Black_David@emc.com@ece.cmu.edu on 08/25/2000 03:01:28 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: iSCSI: "Wedge" drivers I guess I need to take my co-chair hat off and say a few things about EMC's and other "Wedge" drivers. Wedge drivers are motivated by fault tolerance and fail-over in addition to load balancing/spreading. EMC actually has two products, PowerPath for Symmetrix (does all three), and ATF for Clariion (fail-over only). Other products include HDS's SafePath, Veritas's DMP and HP's PV Links. Apologies to those I've left out. John's goal of eliminating of wedge drivers via iSCSI sessions may not be achievable in practice due to fault tolerance and fail-over concerns. The basic fault tolerance/failover concern arises from the requirement that failure of one interface processor in the disk array should not disable all access to the host's storage. Hence the host to storage connectivity usually encompasses more than one interface processor, facing array designers with the choice of using multiple SCSI connections and keeping the SCSI state in each interface processor vs. sharing the SCSI state across multiple interface processors, and making sure everything works right when one of them fails in an arbitrary fashion (ouch). The former is considerably easier to implement, and requires a wedge driver on the host side. The corresponding issue of how a host deals with possible failure of an iSCSI HBA has already been noted, and I agree that the most likely approach is a wedge driver. Since John works for IBM, I should note that the phrase "interface processor(s)" isn't directly applicable to the IBM Shark array, but nonetheless, this issue is still present -- for fault tolerance, storage access has to be spread across both RS/6000 systems running AIX in a Shark, and sharing SCSI state across AIX instances doesn't sound like an easy thing to do. The bottom line is that failure and fault tolerance concerns make it unlikely (IMHO) that an iSCSI session concept will lead to the extinction of wedge drivers. There have been some questions about difficulty of development of wedge drivers. EMC's experience is that fault tolerance and fail-over require much more effort than load balancing for a couple of reasons. The first is that more complex systems tend to have more complex ways of failing than of working correctly :-(. The second is that correct behavior of a wedge driver depends on the correct behavior of HBA drivers in error cases, and HBA vendors in general do not exhaustively test correct behavior of error cases before releasing drivers -- we find driver bugs on a regular basis. Given the intention to implement much/all of iSCSI in hardware, I suspect that this situation will continue relatively unchanged. I'm not sure what the latter implies for difficulty of development of error handling code for multi-connection iSCSI sessions when a single vendor is responsible for both the hardware and the driver. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sun Aug 27 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21826; 26 Aug 2000 21:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA03587 for ips-outgoing; Sat, 26 Aug 2000 20:20:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA03581 for ; Sat, 26 Aug 2000 20:19:52 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id UAA13594; Sat, 26 Aug 2000 20:13:47 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id SAA41120; Sat, 26 Aug 2000 18:19:38 -0600 Importance: Normal Subject: Re: Connection Consensus Progress To: Christopher Stein - Network Storage Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 26 Aug 2000 17:18:56 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/26/2000 06:19:38 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Chris, Even when the TCP/IP processing moves into the NIC, the ability to manage multiple NICs will be in an iSCSI Device Driver. Therefore it is possible to perform iSCSI alternate path balancing and error recovery. If Wedge Drivers will be able to do it, and they can, then it can also be done in the iSCSI device driver. The only important question is -- do we want to prevent the iSCSI Device Drivers from being able to load balance and being able to perform alternate path recovery. If we want to PREVENT the iSCSI Device Driver from doing this, then we should NOT have Multiple connections per Session. If we would like the iSCSI Device Drivers (sometime) to do the load balancing and alternate path recovery, then we should leave the Session definition as it is. The current default is a single connection per session, so that will always permit the vendors to use their own Wedge Device Drivers, therefore, we do not need to eliminate the possibility of a smarter iSCSI Device Driver, sometime in the future. . . . John L. Hufferd Christopher Stein - Network Storage @ece.cmu.edu on 08/26/2000 01:53:13 PM Please respond to Christopher Stein - Network Storage Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Connection Consensus Progress Julian, If one of the reasons for supporting multiple TCP connections is to allow load balancing across HBAs for fault tolerance, this reason will be lost as the TCP processing moves down into the HBA. My understanding was that TCP on a chip technology is desired for high-performance iSCSI. If this is the case, then multiple connections for tolerance of HBA faults is a misfired bullet. -Chris >>X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f >From: julian_satran@il.ibm.com >X-Lotus-FromDomain: IBMIL@IBMDE >To: ips@ece.cmu.edu >Subject: Re: Connection Consensus Progress >Mime-Version: 1.0 >Content-Disposition: inline > > > >David, > >I understand and share your concerns about how good we understand the >requirements for recovery and balancing. > >But as I stated repeatedly we can't wait for somebody else to solve our >problem and the >field requirement is there as witnessed by the products that attempt to >solve it in a >proprietary fashion (and BTW a TCP connection failure could also be >repaired simply by >TCP but TCP does not do it). > >However if they are there the both sets have to be solved at SCSI level - >since several links >if not handled properly increase the failure probability. > >However your last point about multiple HBAs is lost on me. >We attempted to make iSCSI work with several HBAs and went to some length >to keep >the requirements to the HBA hardware as if the HBAs act independently >(counters can be >shared). Is there something we missed? > >Julo > >David Robinson on 25/08/2000 23:32:23 > >Please respond to David Robinson > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Connection Consensus Progress > > > > >> I agree with you up to a point. I know of customers that always need >> multiple physical paths to the Storage Controller. Regardless of how >fast >> the link is, they need a faster link, and these hosts need to be able to >> spread the load across several different HBAs. (Some are on one PCI bus, >> and some on another, etc.) When this happens, as it does today, with >Fibre >> Channel, we are required, as are a number of other vendors, to come up >with >> a multi HBA balancer. We call our Fibre Channel version "DPO" (Dynamic >> Path Optimizer), EMC has another version (I do not know what they call >> theirs). This Code sits as a "Wedge" Driver above the FC Device Drivers >> and balances the work across the different FC HBAs. I think this same >> thing will be required in the iSCSI situation. Note:I think, the FC >> versions only work with IBM or EMC's etc. Controllers. (SUN probably has >a >> similar one also.) > >I understand this scenerio, it is often used as a high availability >feature. >The key question is if this should be handled above at the SCSI layer >as it is most often done now, or in the iSCSI transport. While I like >the goal to unify this into one architecture, I have serious doubts that >we have the understanding of the requirements and needs necessary >to get it right. Thus we will ultimately end up in the same situation >we are in today with a SCSI layer solution. In addition, if the promise >of a hardware iSCSI NIC/HBA is acheived, allowing multiple paths using >different NIC/HBAs will still require "wedge" software. > > -David > > > > > From owner-ips@ECE.cmu.edu Sun Aug 27 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21824; 26 Aug 2000 21:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA03144 for ips-outgoing; Sat, 26 Aug 2000 19:57:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA03139 for ; Sat, 26 Aug 2000 19:57:09 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id TAA70284; Sat, 26 Aug 2000 19:45:11 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id RAA25014; Sat, 26 Aug 2000 17:57:07 -0600 Importance: Normal Subject: Re: iSCSI: "Wedge" drivers To: Black_David@emc.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 26 Aug 2000 16:55:13 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/26/2000 05:57:06 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David, good write up. Let me hypothesize some configurations and approaches, then ask you some questions and then let you pick them apart with your knowledge of your hardware. And everyone else, see if it all makes iSCSI sense. Suppose that a host has only one adapter (NIC), and the failure occurs at the storage controller in what you call an interface processor (lets suppose it is an iSCSI capable interface processor). Lets further suppose that the single NIC had two TCP/IP connections, one TCP/IP connection to one interface processor and another TCP/IP connection to another interface processor. First we would want information available (perhaps in a LDAP "Naming" Directory of some type), such that the iSCSI driver would be able to tell that both TCP/IP connections could drive commands to the same LUs. If that information was there, iSCSI should be able to re-drive commands even during a link failure (which in this case was caused by the failure of an interface controller). I believe this could be made to work with IBM equipment and EMC's, but do you believe that it will work? The question of course, is whether this is a valid Session. That is, the session is generally defined by the same initiator (with the same ISID) connecting with the same target. In this case it is probable that the target has two IP addresses, but since the target will fill in the TSID that forms the Session ID (SSID), whether of not the Initiator gets the information from the LDAP "Naming Directory", the Target can force the relationship to happen, by returning the same TSID across all the interface processors. So that, by definition, should make this a single session with multiple connections. Next suppose (using the same host and NIC) it was possible to put a TCP/IP load balancing switch ahead of the interface processors. Since this switch exports a single TCP/IP address the Initiator might not fully understand the connection to the target, however, since the Target sets the TSID, the Initiator should see this as a valid multiple connection per session. This requires the Interface Processors (as above) to return the same TSID. If that is done, do you think that, just retrying commands on the alternate TCP/IP connection -- within the session -- would work with EMC equipment (assuming that the load balancer drove the command to the surviving interface processor)? With the above examples, I am just wondering about you opinion whether a generic iSCSI Device Driver would do an adequate job of error retry, or whether there are so many things that need to be done, that only your (in this case EMC written) Wedge Driver could do the recovery job, and the types of things that it did would be incompatible with other vendors hardware. My general opinion is that it could work with IBM Shark. If by any chance you think that the iSCSI Device Driver could handle the failures and retry adequately on the above single NIC adapters, then it should be possible to have a single session with multiple TCP/IP connections, that are each on different Host NICs, and have iSCSI perform recover just as well (maybe better). On the other hand if you believe that iSCSI would not be able to handle the Multi connections per Session on a single HBA, is there anything better iSCSI could do in recovery if it had multiple TCP/IP connections, each on a different NICs, but part of a Single Session? I think both approaches could work on IBM Shark (at least for the basic recovery). Now of course the devil is in the details and I am sure that I can hold debates with people envolved and they might take the opposit view. If on the other hand, you do NOT think iSCSI can be made to adequately address your interface processor recovery needs, by performing rather generic recovery techniques, then I would agree with you that a separate Wedge Driver would be required. Further, if that is the case, then paths -- which the Wedge sees -- need to be known by the Wedge and it must be able to map them to alternate physical interface processors. That would mean that only a single connection per Session would be useful, along with a single session per NIC. But this would also probably mean that vendor specific information would need to be put into some kind of a data base that the wedge driver could use, or a lot of administrative work would be needed. That is, the LDAP "Name Server" data base, will need to not only have a way for iSCSI device drivers to access the information, but we will have to generalize the interface so that applications such as vendor specific Wedge Drivers can set and access the information. (On the other hand the Wedge could collate the LU views it gets from each path to determine which ones are alternates, the way it does today.) However, since I think it is possible to actually do correct (fundamental) error recovery, I would not like to remove the multiple connections per session stuff. It could be that it never gets used, but I doubt that. In any case, since a single connection per session is the "default", I think we can leave the Multiple connections per session, and let vendors exploit the possibilities of the Multiple Connections/Session, not only for failure recovery but also for additional performance (via Multiple NICs) without having vendor specific Wedge Device Drivers. Having said all that, it still maybe the case that the generic iSCSI Device Driver can only do fundamental alternate path recovery, and that more advanced functions are still needed in Vendor specific Wedge Drivers. So the question is, just because, the iSCSI Device Drivers can not do all things, do we prevent it from doing any of the failure recovery etc. by eliminating the Multiple connections per Session? . . . John L. Hufferd Black_David@emc.com@ece.cmu.edu on 08/25/2000 03:01:28 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: iSCSI: "Wedge" drivers I guess I need to take my co-chair hat off and say a few things about EMC's and other "Wedge" drivers. Wedge drivers are motivated by fault tolerance and fail-over in addition to load balancing/spreading. EMC actually has two products, PowerPath for Symmetrix (does all three), and ATF for Clariion (fail-over only). Other products include HDS's SafePath, Veritas's DMP and HP's PV Links. Apologies to those I've left out. John's goal of eliminating of wedge drivers via iSCSI sessions may not be achievable in practice due to fault tolerance and fail-over concerns. The basic fault tolerance/failover concern arises from the requirement that failure of one interface processor in the disk array should not disable all access to the host's storage. Hence the host to storage connectivity usually encompasses more than one interface processor, facing array designers with the choice of using multiple SCSI connections and keeping the SCSI state in each interface processor vs. sharing the SCSI state across multiple interface processors, and making sure everything works right when one of them fails in an arbitrary fashion (ouch). The former is considerably easier to implement, and requires a wedge driver on the host side. The corresponding issue of how a host deals with possible failure of an iSCSI HBA has already been noted, and I agree that the most likely approach is a wedge driver. Since John works for IBM, I should note that the phrase "interface processor(s)" isn't directly applicable to the IBM Shark array, but nonetheless, this issue is still present -- for fault tolerance, storage access has to be spread across both RS/6000 systems running AIX in a Shark, and sharing SCSI state across AIX instances doesn't sound like an easy thing to do. The bottom line is that failure and fault tolerance concerns make it unlikely (IMHO) that an iSCSI session concept will lead to the extinction of wedge drivers. There have been some questions about difficulty of development of wedge drivers. EMC's experience is that fault tolerance and fail-over require much more effort than load balancing for a couple of reasons. The first is that more complex systems tend to have more complex ways of failing than of working correctly :-(. The second is that correct behavior of a wedge driver depends on the correct behavior of HBA drivers in error cases, and HBA vendors in general do not exhaustively test correct behavior of error cases before releasing drivers -- we find driver bugs on a regular basis. Given the intention to implement much/all of iSCSI in hardware, I suspect that this situation will continue relatively unchanged. I'm not sure what the latter implies for difficulty of development of error handling code for multi-connection iSCSI sessions when a single vendor is responsible for both the hardware and the driver. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sun Aug 27 06:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23357; 27 Aug 2000 2:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA09109 for ips-outgoing; Sun, 27 Aug 2000 01:35:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA09103 for ; Sun, 27 Aug 2000 01:35:42 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA30734 for ; Sun, 27 Aug 2000 07:35:11 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA19824 for ; Sun, 27 Aug 2000 07:35:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256948.001EAC7F ; Sun, 27 Aug 2000 07:35:02 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 27 Aug 2000 08:33:09 +0300 Subject: Re: Connection Consensus Progress Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Chris, Could you elaborate? Julo Christopher Stein - Network Storage on 26/08/2000 23:53:13 Please respond to Christopher Stein - Network Storage To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Connection Consensus Progress Julian, If one of the reasons for supporting multiple TCP connections is to allow load balancing across HBAs for fault tolerance, this reason will be lost as the TCP processing moves down into the HBA. My understanding was that TCP on a chip technology is desired for high-performance iSCSI. If this is the case, then multiple connections for tolerance of HBA faults is a misfired bullet. -Chris >>X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f >From: julian_satran@il.ibm.com >X-Lotus-FromDomain: IBMIL@IBMDE >To: ips@ece.cmu.edu >Subject: Re: Connection Consensus Progress >Mime-Version: 1.0 >Content-Disposition: inline > > > >David, > >I understand and share your concerns about how good we understand the >requirements for recovery and balancing. > >But as I stated repeatedly we can't wait for somebody else to solve our >problem and the >field requirement is there as witnessed by the products that attempt to >solve it in a >proprietary fashion (and BTW a TCP connection failure could also be >repaired simply by >TCP but TCP does not do it). > >However if they are there the both sets have to be solved at SCSI level - >since several links >if not handled properly increase the failure probability. > >However your last point about multiple HBAs is lost on me. >We attempted to make iSCSI work with several HBAs and went to some length >to keep >the requirements to the HBA hardware as if the HBAs act independently >(counters can be >shared). Is there something we missed? > >Julo > >David Robinson on 25/08/2000 23:32:23 > >Please respond to David Robinson > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Connection Consensus Progress > > > > >> I agree with you up to a point. I know of customers that always need >> multiple physical paths to the Storage Controller. Regardless of how >fast >> the link is, they need a faster link, and these hosts need to be able to >> spread the load across several different HBAs. (Some are on one PCI bus, >> and some on another, etc.) When this happens, as it does today, with >Fibre >> Channel, we are required, as are a number of other vendors, to come up >with >> a multi HBA balancer. We call our Fibre Channel version "DPO" (Dynamic >> Path Optimizer), EMC has another version (I do not know what they call >> theirs). This Code sits as a "Wedge" Driver above the FC Device Drivers >> and balances the work across the different FC HBAs. I think this same >> thing will be required in the iSCSI situation. Note:I think, the FC >> versions only work with IBM or EMC's etc. Controllers. (SUN probably has >a >> similar one also.) > >I understand this scenerio, it is often used as a high availability >feature. >The key question is if this should be handled above at the SCSI layer >as it is most often done now, or in the iSCSI transport. While I like >the goal to unify this into one architecture, I have serious doubts that >we have the understanding of the requirements and needs necessary >to get it right. Thus we will ultimately end up in the same situation >we are in today with a SCSI layer solution. In addition, if the promise >of a hardware iSCSI NIC/HBA is acheived, allowing multiple paths using >different NIC/HBAs will still require "wedge" software. > > -David > > > > > From owner-ips@ECE.cmu.edu Sun Aug 27 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26311; 27 Aug 2000 13:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA29076 for ips-outgoing; Sun, 27 Aug 2000 12:00:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA29067 for ; Sun, 27 Aug 2000 12:00:41 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA33924 for ; Sun, 27 Aug 2000 18:00:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA24738 for ; Sun, 27 Aug 2000 18:00:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256948.0057E1D3 ; Sun, 27 Aug 2000 17:59:54 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 27 Aug 2000 18:58:00 +0300 Subject: bridging issues Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, Lately some iSCSI issues related to bridging to/from FC networks where brought to my attention. I thought that it would be wise to share them (and my first rough attempt to solve them) with you for two reasons: that's why we have a Working Group (don't we?) there might be more of this kind and this is a good time to think about them and bring the with or without a solution to the WG FC and iSCSI are SCSI mappings over a network that share a "network" view of the transport infrastructure but have widely different views of the network with regard to distances, reliability, flow control etc.. FCP assumes a locally controlled environment using an unreliable delivery version of FC and doing recovery (a very rare event) at FCP level. FC addressing model uses locally generated dynamic addresses as part of the "naming" and its tagging. An iSCSI initiator should be able to operate a FCP target through a very simple and stateless (or almost stateless) gateway. A designer attempting to do this (and I know many of you already did or are in the midst of doing that) will find it difficult to do achieve a stateless design (but not impossible). The FCP recovery model from packet loses (commands, data and status) is based on sequencing (if lost packets are within sequences) or timeouts if those are at sequence boundaries. In the FC world those are very rare events and in an iSCSI+FCP hybrid could be handled by a statefull gateway (expensive) or by a stateless gateway and a more radical initiator recovery - based on TCP connection close and reopen. The later solution requires iSCSI to enforce a set of timeouts (an initiator requirement only) and provide an API for timeout setting and connection abort to an upper layer. Enforcing a set of timeouts would have the good side-effect of making iSCSI also tolerant to aberrant device behavior - and that is unfortunately bound to happen more often as the complexity of the devices will increase. Please observe that the set of timeouts I am referring to are not a "protocol" element (they do not appear on the wire). Specifying the expected behavior of an iSCSI initiator will simplify the task of building bridges to FCP. We propose to do so by adding a "host requirements" chapter. Another bridging issue is the initiator tag-translation mechanism. iSCSI is requiring the initiator issue a unique tag for each task but does not require any structure for the tag. This way an implementer may choose the most efficient structure for his implementation (e.g., use the address of it's control structure representing the task as a tag). FCP on the other hand demands some structure for the tag. A stateless bridge would want to pass the tag unchanged from initiator to target or translate it with a stateless rule. If we restrict the design space to gateways that present a "homogenous" view of the device conglomerate behind them (e.g., all devices are FCP) and we may pass this information to the initiator - then the initiator can use the slightly more restrictive FCP tagging mechanism. Julo From owner-ips@ECE.cmu.edu Mon Aug 28 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00452; 28 Aug 2000 13:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA04027 for ips-outgoing; Mon, 28 Aug 2000 12:20:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA04007 for ; Mon, 28 Aug 2000 12:20:26 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA09426 for ; Mon, 28 Aug 2000 18:19:55 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA26562 for ; Mon, 28 Aug 2000 18:19:54 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256949.0059B016 ; Mon, 28 Aug 2000 18:19:37 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 28 Aug 2000 19:17:37 +0300 Subject: RE: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, I would appreciate if instead of a general and fuzzy referency you would quote the document page and paragraph. Julo "Douglas Otis" on 28/08/2000 19:00:56 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Data in SCSI Response or SCSI Data Julo, Is it compliant to assume knowledge of the TCP frame by the iSCSI application? In many places within iSCSI discussions, references are made of packets, frames, and datagrams as an element within iSCSI. If creating frame alignment is your intent, rather than corrupting TCP APIs, perhaps you should consider SCTP. Otherwise, such discussions are little more than a wink and a nod at creating a frame aligned TCP. How do you advise a change to the TCP API to resolve frame alignment as you have just suggested? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Saturday, August 26, 2000 2:28 AM > To: ips@ece.cmu.edu > Subject: Re: Data in SCSI Response or SCSI Data > > > > > Steph, > > I assume that your hardware knows when it is going to send the last SCSI > datagram. > In this case it can insert the status in the datagram header (not at the > end) - > (and BTW that is how the current draft assumes that things will be done). > > If you have to send sense-data (bad status) then you will send a separate > datagram. > > Julo > > Stephen Bailey on 25/08/2000 22:12:09 > > Please respond to Stephen Bailey > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > Is there anything preventing your hypothetical hardware implementor > > to send always good status within the last block of data? > > It depends upon the RDMA mechanism you are using. I admit that I have > not studied them in detail, other than ST of course, which is > essentially an transport protocol based upon a particular RDMA > mechanism. I will try to do so soon to determine if I'm all wet. > > However, assuming that the RDMA mechanism operates on a per-datagram > basis, I can only imagine that you will not be able to append > `general delivery' data to the end of an RDMA datagram. In this case, > you will need a separate datagram to ensure that status is delivered > through a separate path from the data. > > So, while I think concatenating status and data in its general form is > not a good idea, a good-status fast path (like the success bit) is > definitely the right way to think about it. Nonsuccess SCSI status is > so rare that any compromise you can make in the nonsuccess path to > make the success path go faster is worth it. > > Steph > > > From owner-ips@ECE.cmu.edu Mon Aug 28 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00454; 28 Aug 2000 13:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA02448 for ips-outgoing; Mon, 28 Aug 2000 11:51:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA02425 for ; Mon, 28 Aug 2000 11:51:21 -0400 (EDT) Received: from purol.East.Sun.COM ([129.148.9.11]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id IAA22275; Mon, 28 Aug 2000 08:51:18 -0700 (PDT) Received: from grace (grace [129.148.174.38]) by purol.East.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with SMTP id LAA02971; Mon, 28 Aug 2000 11:51:16 -0400 (EDT) Message-Id: <200008281551.LAA02971@purol.East.Sun.COM> Date: Mon, 28 Aug 2000 11:51:16 -0400 (EDT) From: Christopher Stein - Network Storage Reply-To: Christopher Stein - Network Storage Subject: Re: Connection Consensus Progress To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: x1j99HvHT7GcEWCTfP+IRQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.3.5 SunOS 5.7 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, John Hufferd interpreted my point correctly and set me straight on "wedge" drivers (thanks, John) -- a software component in the iSCSI driver that load balances across multiple HBA TCPs. Of course, it may be a challenge to get anyone but a second-rate engineer to build something as unbeautiful as a "wedge" driver, fating it to be a single-point potentially more buggy than the HBAs themselves. As an added bonus, the fault profile would be software, rather than fail-stop hardware (which top firmware engineering in the HBA could present to the iSCSI driver). -Chris > >Chris, > >Could you elaborate? > >Julo > >Christopher Stein - Network Storage on >26/08/2000 23:53:13 > >Please respond to Christopher Stein - Network Storage > > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Connection Consensus Progress > > > > > >Julian, > >If one of the reasons for supporting multiple TCP connections is >to allow load balancing across HBAs for fault tolerance, this >reason will be lost as the TCP processing moves down into the HBA. > >My understanding was that TCP on a chip technology is desired for >high-performance iSCSI. If this is the case, then multiple >connections for tolerance of HBA faults is a misfired bullet. > >-Chris > >>>X-Authentication-Warning: ece.cmu.edu: majordom set sender to >owner-ips@ece.cmu.edu using -f >>From: julian_satran@il.ibm.com >>X-Lotus-FromDomain: IBMIL@IBMDE >>To: ips@ece.cmu.edu >>Subject: Re: Connection Consensus Progress >>Mime-Version: 1.0 >>Content-Disposition: inline >> >> >> >>David, >> >>I understand and share your concerns about how good we understand the >>requirements for recovery and balancing. >> >>But as I stated repeatedly we can't wait for somebody else to solve our >>problem and the >>field requirement is there as witnessed by the products that attempt to >>solve it in a >>proprietary fashion (and BTW a TCP connection failure could also be >>repaired simply by >>TCP but TCP does not do it). >> >>However if they are there the both sets have to be solved at SCSI level - >>since several links >>if not handled properly increase the failure probability. >> >>However your last point about multiple HBAs is lost on me. >>We attempted to make iSCSI work with several HBAs and went to some length >>to keep >>the requirements to the HBA hardware as if the HBAs act independently >>(counters can be >>shared). Is there something we missed? >> >>Julo >> >>David Robinson on 25/08/2000 23:32:23 >> >>Please respond to David Robinson >> >>To: ips@ece.cmu.edu >>cc: (bcc: Julian Satran/Haifa/IBM) >>Subject: Re: Connection Consensus Progress >> >> >> >> >>> I agree with you up to a point. I know of customers that always need >>> multiple physical paths to the Storage Controller. Regardless of how >>fast >>> the link is, they need a faster link, and these hosts need to be able to >>> spread the load across several different HBAs. (Some are on one PCI >bus, >>> and some on another, etc.) When this happens, as it does today, with >>Fibre >>> Channel, we are required, as are a number of other vendors, to come up >>with >>> a multi HBA balancer. We call our Fibre Channel version "DPO" (Dynamic >>> Path Optimizer), EMC has another version (I do not know what they call >>> theirs). This Code sits as a "Wedge" Driver above the FC Device Drivers >>> and balances the work across the different FC HBAs. I think this same >>> thing will be required in the iSCSI situation. Note:I think, the FC >>> versions only work with IBM or EMC's etc. Controllers. (SUN probably >has >>a >>> similar one also.) >> >>I understand this scenerio, it is often used as a high availability >>feature. >>The key question is if this should be handled above at the SCSI layer >>as it is most often done now, or in the iSCSI transport. While I like >>the goal to unify this into one architecture, I have serious doubts that >>we have the understanding of the requirements and needs necessary >>to get it right. Thus we will ultimately end up in the same situation >>we are in today with a SCSI layer solution. In addition, if the promise >>of a hardware iSCSI NIC/HBA is acheived, allowing multiple paths using >>different NIC/HBAs will still require "wedge" software. >> >> -David >> >> >> >> >> > > > > > --- Christopher A. Stein (cstein@east.sun.com) Network Storage, Sun Microsystems, Inc. BUR02-213 1 Network Dr. Burlington, MA 01803-0902 781-442-2343 From owner-ips@ECE.cmu.edu Mon Aug 28 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00469; 28 Aug 2000 13:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA04168 for ips-outgoing; Mon, 28 Aug 2000 12:25:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA04155 for ; Mon, 28 Aug 2000 12:24:40 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 28 Aug 2000 12:24:09 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F1A@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: iSCSI: "Wedge" drivers Date: Mon, 28 Aug 2000 12:24:06 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Still with my co-chair hat off, let me try to summarize, rather than quoting and responding: If one assumes that an iSCSI session spans multiple interface processors on the storage side, then those interface processors have to share SCSI state (e.g., ranges of data for each I/O on which Ready to Transfer has been vs. needs to be issued) in some fashion so that the other processor has it when failure causes it to become involved. Use of a load balancing switch does not change this situation. Julian (IIRC) has observed that the corresponding state sharing is done for mainframe storage (e.g., I/O completion can come back on a different link than the one used for initiation), but it is not done by any SCSI implementation I'm aware of. If nothing else, that's proof by example that any box (EMC, IBM, etc.) that supports mainframe storage is in principle capable of spreading a SCSI connection across more than one interface processor. This is additional implementation complexity and effort, as state sharing across failure domains is not easy to get right, and this is a new form of sharing that SCSI implementers have little experience with. A similar issue occurs on the host side. If the system is equipped with NICs so that iSCSI above TCP/IP is entirely in software, then spreading a SCSI connection across multiple NICs should work fine. OTOH, if the system has HBAs that implement much of iSCSI in hardware in addition to TCP/IP, then not sharing iSCSI state across HBAs is again easier to build (and entails a wedge driver). I'm trying not to take a position for or against iSCSI sessions (e.g., I can see a strong rationale for using them with tape, where wedge drivers don't work and the fault tolerance requirements aren't as stringent). The point I wanted to make is that engineering and implementation concerns make it highly unlikely that iSCSI sessions would remove the need for wedge drivers for disk. My goal is to contribute to a clear understanding of the technical rationale behind sessions, for the requirements document if nothing else. Several minor issues ... John described a model in which an initiator discovers that multiple TCP/IP connections are part of the same session. That seems awkward by comparison to having each participant tell the other what the alternate transport addresses are (or how to find them) as the latter avoids having to splice a new connection into one that's already actively doing I/O. SCTP sets up multiple addresses in this fashion, FWIW. EMC error retry does not always require an EMC-written wedge driver. For example, Symmetrix works with the Veritas DMP wedge driver. There's no free lunch on configuration -- since wedge drivers are not going to go away, iSCSI sessions become something configured "in addition to" wedge driver configuration rather than "instead of". --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Aug 28 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00477; 28 Aug 2000 13:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA03699 for ips-outgoing; Mon, 28 Aug 2000 12:15:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA03691 for ; Mon, 28 Aug 2000 12:15:09 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA171330 for ; Mon, 28 Aug 2000 18:14:37 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA34740 for ; Mon, 28 Aug 2000 18:14:37 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256949.005939C2 ; Mon, 28 Aug 2000 18:14:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 28 Aug 2000 19:12:36 +0300 Subject: Re: Connection Consensus Progress Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Chris, Thanks for the clarification. Unlike proprietary wedge drivers the iSCSI recovery mechanism is meant to be generic. It might get help from HBAs (like detecting a failing connection) but it can be implemented entirely above TCP (at the iSCSI level). Here is a comment I sent about wedge drivers: -------------------------- I assume that FC folks where so overwhelmed by how complex the FCP ended up being that they did not even contemplate doing multiple connections. And the FCP complexity is caused by stressing too much the hardware implementation in the first place and then placing functions with the hardware implementation in mind as the #1 concern. However an older stuff - the ESCON had it all there - multiple connections built in a channel group. The only thing we are trying with the connection group is bringing back some sanity and devising only basic recovery mechanisms. And I am proud that we did it with so little complexity (Kalman's proposal could simplify it further). With so many proprietary things - if we leave them as they are - we might end having a multitude of incompatible implementations because of unexpected failure modes for which the wedge drivers will break. ------------------------ Julo Christopher Stein - Network Storage on 28/08/2000 18:51:16 Please respond to Christopher Stein - Network Storage To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Connection Consensus Progress Hi, John Hufferd interpreted my point correctly and set me straight on "wedge" drivers (thanks, John) -- a software component in the iSCSI driver that load balances across multiple HBA TCPs. Of course, it may be a challenge to get anyone but a second-rate engineer to build something as unbeautiful as a "wedge" driver, fating it to be a single-point potentially more buggy than the HBAs themselves. As an added bonus, the fault profile would be software, rather than fail-stop hardware (which top firmware engineering in the HBA could present to the iSCSI driver). -Chris > >Chris, > >Could you elaborate? > >Julo > >Christopher Stein - Network Storage on >26/08/2000 23:53:13 > >Please respond to Christopher Stein - Network Storage > > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Connection Consensus Progress > > > > > >Julian, > >If one of the reasons for supporting multiple TCP connections is >to allow load balancing across HBAs for fault tolerance, this >reason will be lost as the TCP processing moves down into the HBA. > >My understanding was that TCP on a chip technology is desired for >high-performance iSCSI. If this is the case, then multiple >connections for tolerance of HBA faults is a misfired bullet. > >-Chris > >>>X-Authentication-Warning: ece.cmu.edu: majordom set sender to >owner-ips@ece.cmu.edu using -f >>From: julian_satran@il.ibm.com >>X-Lotus-FromDomain: IBMIL@IBMDE >>To: ips@ece.cmu.edu >>Subject: Re: Connection Consensus Progress >>Mime-Version: 1.0 >>Content-Disposition: inline >> >> >> >>David, >> >>I understand and share your concerns about how good we understand the >>requirements for recovery and balancing. >> >>But as I stated repeatedly we can't wait for somebody else to solve our >>problem and the >>field requirement is there as witnessed by the products that attempt to >>solve it in a >>proprietary fashion (and BTW a TCP connection failure could also be >>repaired simply by >>TCP but TCP does not do it). >> >>However if they are there the both sets have to be solved at SCSI level - >>since several links >>if not handled properly increase the failure probability. >> >>However your last point about multiple HBAs is lost on me. >>We attempted to make iSCSI work with several HBAs and went to some length >>to keep >>the requirements to the HBA hardware as if the HBAs act independently >>(counters can be >>shared). Is there something we missed? >> >>Julo >> >>David Robinson on 25/08/2000 23:32:23 >> >>Please respond to David Robinson >> >>To: ips@ece.cmu.edu >>cc: (bcc: Julian Satran/Haifa/IBM) >>Subject: Re: Connection Consensus Progress >> >> >> >> >>> I agree with you up to a point. I know of customers that always need >>> multiple physical paths to the Storage Controller. Regardless of how >>fast >>> the link is, they need a faster link, and these hosts need to be able to >>> spread the load across several different HBAs. (Some are on one PCI >bus, >>> and some on another, etc.) When this happens, as it does today, with >>Fibre >>> Channel, we are required, as are a number of other vendors, to come up >>with >>> a multi HBA balancer. We call our Fibre Channel version "DPO" (Dynamic >>> Path Optimizer), EMC has another version (I do not know what they call >>> theirs). This Code sits as a "Wedge" Driver above the FC Device Drivers >>> and balances the work across the different FC HBAs. I think this same >>> thing will be required in the iSCSI situation. Note:I think, the FC >>> versions only work with IBM or EMC's etc. Controllers. (SUN probably >has >>a >>> similar one also.) >> >>I understand this scenerio, it is often used as a high availability >>feature. >>The key question is if this should be handled above at the SCSI layer >>as it is most often done now, or in the iSCSI transport. While I like >>the goal to unify this into one architecture, I have serious doubts that >>we have the understanding of the requirements and needs necessary >>to get it right. Thus we will ultimately end up in the same situation >>we are in today with a SCSI layer solution. In addition, if the promise >>of a hardware iSCSI NIC/HBA is acheived, allowing multiple paths using >>different NIC/HBAs will still require "wedge" software. >> >> -David >> >> >> >> >> > > > > > --- Christopher A. Stein (cstein@east.sun.com) Network Storage, Sun Microsystems, Inc. BUR02-213 1 Network Dr. Burlington, MA 01803-0902 781-442-2343 From owner-ips@ECE.cmu.edu Mon Aug 28 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00475; 28 Aug 2000 13:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA02779 for ips-outgoing; Mon, 28 Aug 2000 11:59:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA02751 for ; Mon, 28 Aug 2000 11:58:56 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id IAA05964; Mon, 28 Aug 2000 08:51:24 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id IAA25523; Mon, 28 Aug 2000 08:48:57 -0700 From: "Douglas Otis" To: , Subject: RE: Data in SCSI Response or SCSI Data Date: Mon, 28 Aug 2000 09:00:56 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, Is it compliant to assume knowledge of the TCP frame by the iSCSI application? In many places within iSCSI discussions, references are made of packets, frames, and datagrams as an element within iSCSI. If creating frame alignment is your intent, rather than corrupting TCP APIs, perhaps you should consider SCTP. Otherwise, such discussions are little more than a wink and a nod at creating a frame aligned TCP. How do you advise a change to the TCP API to resolve frame alignment as you have just suggested? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Saturday, August 26, 2000 2:28 AM > To: ips@ece.cmu.edu > Subject: Re: Data in SCSI Response or SCSI Data > > > > > Steph, > > I assume that your hardware knows when it is going to send the last SCSI > datagram. > In this case it can insert the status in the datagram header (not at the > end) - > (and BTW that is how the current draft assumes that things will be done). > > If you have to send sense-data (bad status) then you will send a separate > datagram. > > Julo > > Stephen Bailey on 25/08/2000 22:12:09 > > Please respond to Stephen Bailey > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > Is there anything preventing your hypothetical hardware implementor > > to send always good status within the last block of data? > > It depends upon the RDMA mechanism you are using. I admit that I have > not studied them in detail, other than ST of course, which is > essentially an transport protocol based upon a particular RDMA > mechanism. I will try to do so soon to determine if I'm all wet. > > However, assuming that the RDMA mechanism operates on a per-datagram > basis, I can only imagine that you will not be able to append > `general delivery' data to the end of an RDMA datagram. In this case, > you will need a separate datagram to ensure that status is delivered > through a separate path from the data. > > So, while I think concatenating status and data in its general form is > not a good idea, a good-status fast path (like the success bit) is > definitely the right way to think about it. Nonsuccess SCSI status is > so rare that any compromise you can make in the nonsuccess path to > make the success path go faster is worth it. > > Steph > > > From owner-ips@ECE.cmu.edu Mon Aug 28 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01066; 28 Aug 2000 15:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09715 for ips-outgoing; Mon, 28 Aug 2000 14:22:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA09698 for ; Mon, 28 Aug 2000 14:22:23 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA06013; Mon, 28 Aug 2000 11:15:03 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA25907; Mon, 28 Aug 2000 11:12:36 -0700 From: "Douglas Otis" To: , Subject: RE: Data in SCSI Response or SCSI Data Date: Mon, 28 Aug 2000 11:24:36 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, RFC 793 (TCP pg 12) "A sending TCP is allowed to collect data from the sending user and to send that data in segments at its own convenience, until the push function is signaled, then it must send all unsent data. When a receiving TCP sees the PUSH flag, it must not wait for more data from the sending TCP before passing the data to the receiving process. There is no necessary relationship between push functions and segment boundaries. The data in any particular segment may be the result of a single SEND call, in whole or part, or of multiple SEND calls. The purpose of push function and the PUSH flag is to push data through from the sending user to the receiving user. It does not provide a record service. There is a coupling between the push function and the use of buffers of data that cross the TCP/user interface. Each time a PUSH flag is associated with data placed into the receiving user's buffer, the buffer is returned to the user for processing even if the buffer is not filled. If data arrives that fills the user's buffer before a PUSH is seen, the data is passed to the user in buffer size units." It would appear you are assuming a relationship between PUSH flags and segment boundaries. In addition, as iSCSI is the confluence of independent traffic, at what point would you set the PUSH flag? You suggest rather than a push function, there is a last datagram signal according to your comments below. Even with a PUSH flag, adding data to the buffer before completion of push function does not provide guaranteed segment alignment nor is there any signaling for completion nor is there any intent at providing the alignment function you indicate. You suggest a 'last datagram' signal provides a simple means of positioning information at the beginning of the TCP Datagram. What signal? In keeping within TCP, how do you implement your comments below which you state is the basis of iSCSI. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Monday, August 28, 2000 9:18 AM > To: ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > > > Doug, > > I would appreciate if instead of a general and fuzzy referency you would > quote > the document page and paragraph. > > Julo > > "Douglas Otis" on 28/08/2000 19:00:56 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: Data in SCSI Response or SCSI Data > > > > > Julo, > > Is it compliant to assume knowledge of the TCP frame by the iSCSI > application? In many places within iSCSI discussions, references are made > of packets, frames, and datagrams as an element within iSCSI. If creating > frame alignment is your intent, rather than corrupting TCP APIs, perhaps > you > should consider SCTP. Otherwise, such discussions are little more than a > wink and a nod at creating a frame aligned TCP. How do you > advise a change > to the TCP API to resolve frame alignment as you have just suggested? > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Saturday, August 26, 2000 2:28 AM > > To: ips@ece.cmu.edu > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > Steph, > > > > I assume that your hardware knows when it is going to send the last SCSI > > datagram. > > In this case it can insert the status in the datagram header (not at the > > end) - > > (and BTW that is how the current draft assumes that things will > be done). > > > > If you have to send sense-data (bad status) then you will send > a separate > > datagram. > > > > Julo > > > > Stephen Bailey on 25/08/2000 22:12:09 > > > > Please respond to Stephen Bailey > > > > To: ips@ece.cmu.edu > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > Is there anything preventing your hypothetical hardware implementor > > > to send always good status within the last block of data? > > > > It depends upon the RDMA mechanism you are using. I admit that I have > > not studied them in detail, other than ST of course, which is > > essentially an transport protocol based upon a particular RDMA > > mechanism. I will try to do so soon to determine if I'm all wet. > > > > However, assuming that the RDMA mechanism operates on a per-datagram > > basis, I can only imagine that you will not be able to append > > `general delivery' data to the end of an RDMA datagram. In this case, > > you will need a separate datagram to ensure that status is delivered > > through a separate path from the data. > > > > So, while I think concatenating status and data in its general form is > > not a good idea, a good-status fast path (like the success bit) is > > definitely the right way to think about it. Nonsuccess SCSI status is > > so rare that any compromise you can make in the nonsuccess path to > > make the success path go faster is worth it. > > > > Steph From owner-ips@ECE.cmu.edu Mon Aug 28 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01061; 28 Aug 2000 15:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA10221 for ips-outgoing; Mon, 28 Aug 2000 14:33:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA10192 for ; Mon, 28 Aug 2000 14:33:23 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id NAA04180; Mon, 28 Aug 2000 13:33:37 -0500 Message-ID: <39AAB080.76C8B314@stewart.chicago.il.us> Date: Mon, 28 Aug 2000 13:33:36 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: ips@ece.cmu.edu Subject: Re: iSCSI: "Wedge" drivers References: <0F31E5C394DAD311B60C00E029101A0704100F1A@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com wrote: > John described a model in which an initiator discovers > that multiple TCP/IP connections are part of the same > session. That seems awkward by comparison to having > each participant tell the other what the alternate > transport addresses are (or how to find them) as the > latter avoids having to splice a new connection into > one that's already actively doing I/O. SCTP sets up > multiple addresses in this fashion, FWIW. > David: This is exactly what SCTP does in setting up an association. The peer endpoints exchange there address lists when establishing the association. So if you have 3 NIC cards each with seperate IP addresses, your endpoint will pass the 3 addresses as valid (presuming you bound all of the addresses). SCTP will automatically (on retransmission due to fast retransmit algorithm or T3-Timer expiration) retransmit to one of the alternate addresses. Presuming you are not using the unreliable extension that will NOT do retransmissions :). I think (from my limited understanding of your discussion so far) that SCTP will solve the failure scenario in this case. Points it does not help with are: 1) Load balancing across NIC's. This was taken out of the specification long ago and is now left up to the upper layer application... so if you wanted it the SCSI layer above would need to be overriding the primary destination address (The primary is set at startup and can be changed or overridden). All the addresses should be available by API query of the endpoint as well. 2) Implementations in hardware. I don't see this happening for quite some time since SCTP is so new it will be some time before someone sticks it in a ASIC... IMHO.. of course I have been wrong a lot of times :0 One other point that is worth noting... SCTP is designed to be a bit more configurable than TCP. Most of the parameters are designed to be tweaked. I think on a private Intra-Net you can get a lot more control of some of the failure thresholds and parameters than you will in most common TCP implemenations that are out there today. This could be of some benefit to the IPS working group... R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Mon Aug 28 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01071; 28 Aug 2000 15:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09306 for ips-outgoing; Mon, 28 Aug 2000 14:13:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA09289 for ; Mon, 28 Aug 2000 14:13:09 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA28556 for ; Mon, 28 Aug 2000 11:13:08 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.143.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA29448 for ; Mon, 28 Aug 2000 11:13:07 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA26716; Mon, 28 Aug 2000 11:13:07 -0700 (PDT) Message-Id: <200008281813.LAA26716@ha10nwk.EBay.Sun.COM> Date: Mon, 28 Aug 2000 11:13:08 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: iSCSI: "Wedge" drivers To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: xQDyFfaO8QCppXUUh2rJfQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk There has been discussion in this thread about what "could" be done if we had multiple links per session. Not being a Fiber Channel guru, is it true that FC does not have such a link aggregation mechanism? If not, why not and is this perceived as a problem in the FC community? -David From owner-ips@ECE.cmu.edu Mon Aug 28 23:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02065; 28 Aug 2000 19:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA19041 for ips-outgoing; Mon, 28 Aug 2000 17:48:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA19036 for ; Mon, 28 Aug 2000 17:48:29 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id QAA04566; Mon, 28 Aug 2000 16:47:58 -0500 Message-ID: <39AADE0E.52A3FF6F@stewart.chicago.il.us> Date: Mon, 28 Aug 2000 16:47:58 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis CC: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas: Your notes below are along the sender side path i.e. the pusher :) But it is even worse on the receiver side. Unless the receiver puts record markings in and observes them very carefully what you get at each read is really quite dependant on the kernel buffering and what as arrives since you last did a read... Generally stuff I have seen and done in the past either is sending "fixed" records of exactly the same number of bytes so it can do a blocking read to get X bytes ... OR the old tried and true: A) read the int size followed by B) read until you fill the buffer you are expecting based on the int size you got... El.Yucko if you are doing any sort of message sending/receiving... which I presume iSCSI does... If it does not have clear delineation via fixed records OR a size indicator in a common header that is always read this will be a problem :0 R Douglas Otis wrote: > > Julo, > > RFC 793 (TCP pg 12) > "A sending TCP is allowed to collect data from the sending user and to > send that data in segments at its own convenience, until the push > function is signaled, then it must send all unsent data. When a > receiving TCP sees the PUSH flag, it must not wait for more data from > the sending TCP before passing the data to the receiving process. > > There is no necessary relationship between push functions and segment > boundaries. The data in any particular segment may be the result of a > single SEND call, in whole or part, or of multiple SEND calls. > > The purpose of push function and the PUSH flag is to push data through > from the sending user to the receiving user. It does not provide a > record service. > > There is a coupling between the push function and the use of buffers > of data that cross the TCP/user interface. Each time a PUSH flag is > associated with data placed into the receiving user's buffer, the > buffer is returned to the user for processing even if the buffer is > not filled. If data arrives that fills the user's buffer before a > PUSH is seen, the data is passed to the user in buffer size units." > > It would appear you are assuming a relationship between PUSH flags and > segment boundaries. In addition, as iSCSI is the confluence of independent > traffic, at what point would you set the PUSH flag? You suggest rather than > a push function, there is a last datagram signal according to your comments > below. Even with a PUSH flag, adding data to the buffer before completion > of push function does not provide guaranteed segment alignment nor is there > any signaling for completion nor is there any intent at providing the > alignment function you indicate. You suggest a 'last datagram' signal > provides a simple means of positioning information at the beginning of the > TCP Datagram. What signal? In keeping within TCP, how do you implement > your comments below which you state is the basis of iSCSI. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Monday, August 28, 2000 9:18 AM > > To: ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > Doug, > > > > I would appreciate if instead of a general and fuzzy referency you would > > quote > > the document page and paragraph. > > > > Julo > > > > "Douglas Otis" on 28/08/2000 19:00:56 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > Julo, > > > > Is it compliant to assume knowledge of the TCP frame by the iSCSI > > application? In many places within iSCSI discussions, references are made > > of packets, frames, and datagrams as an element within iSCSI. If creating > > frame alignment is your intent, rather than corrupting TCP APIs, perhaps > > you > > should consider SCTP. Otherwise, such discussions are little more than a > > wink and a nod at creating a frame aligned TCP. How do you > > advise a change > > to the TCP API to resolve frame alignment as you have just suggested? > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Saturday, August 26, 2000 2:28 AM > > > To: ips@ece.cmu.edu > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > Steph, > > > > > > I assume that your hardware knows when it is going to send the last SCSI > > > datagram. > > > In this case it can insert the status in the datagram header (not at the > > > end) - > > > (and BTW that is how the current draft assumes that things will > > be done). > > > > > > If you have to send sense-data (bad status) then you will send > > a separate > > > datagram. > > > > > > Julo > > > > > > Stephen Bailey on 25/08/2000 22:12:09 > > > > > > Please respond to Stephen Bailey > > > > > > To: ips@ece.cmu.edu > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > Is there anything preventing your hypothetical hardware implementor > > > > to send always good status within the last block of data? > > > > > > It depends upon the RDMA mechanism you are using. I admit that I have > > > not studied them in detail, other than ST of course, which is > > > essentially an transport protocol based upon a particular RDMA > > > mechanism. I will try to do so soon to determine if I'm all wet. > > > > > > However, assuming that the RDMA mechanism operates on a per-datagram > > > basis, I can only imagine that you will not be able to append > > > `general delivery' data to the end of an RDMA datagram. In this case, > > > you will need a separate datagram to ensure that status is delivered > > > through a separate path from the data. > > > > > > So, while I think concatenating status and data in its general form is > > > not a good idea, a good-status fast path (like the success bit) is > > > definitely the right way to think about it. Nonsuccess SCSI status is > > > so rare that any compromise you can make in the nonsuccess path to > > > make the success path go faster is worth it. > > > > > > Steph -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Tue Aug 29 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02591; 28 Aug 2000 21:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA23868 for ips-outgoing; Mon, 28 Aug 2000 20:12:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ihemail2.firewall.lucent.com (ihemail2.lucent.com [192.11.222.163]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA23863 for ; Mon, 28 Aug 2000 20:12:24 -0400 (EDT) Received: from ihemail2.firewall.lucent.com (localhost [127.0.0.1]) by ihemail2.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id UAA27849 for ; Mon, 28 Aug 2000 20:12:23 -0400 (EDT) Received: from il0015exch001h.wins.lucent.com (h135-1-23-83.lucent.com [135.1.23.83]) by ihemail2.firewall.lucent.com (Pro-8.9.3/8.9.3) with ESMTP id UAA27833 for ; Mon, 28 Aug 2000 20:12:23 -0400 (EDT) Received: by il0015exch001h.ih.lucent.com with Internet Mail Service (5.5.2650.21) id ; Mon, 28 Aug 2000 19:12:23 -0500 Message-ID: <80B684C5E29FD211AA8000A0C9CDD91904D098BE@il0015exch005u.ih.lucent.com> From: "Rodriguez, Elizabeth G (Elizabeth)" To: "'ipfc@standards.gadzoox.com'" , "'ips@ece.cmu.edu'" Cc: "'marjorie.krueger@vixel.com'" , "'rajb@lightsand.com'" , "'muralir@lightsand.com'" Subject: FC over IP: Requirements Date: Mon, 28 Aug 2000 19:12:18 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hello all, The authors of the FC over IP draft are still in the process of trying to reformulate the requirements for the FC over IP draft. In addition, (according to what I have been told), the decision for where the FC over IP work item is to be hosted is still being discussed. It may be placed in the IPS working group, or the decision may be made to place it elsewhere. In order to try to minimize confusion, the other authors and I have been talking offline, while monitoring these lists. That said, we list below some of what we (the authors of the FC over IP draft) have been talking about wrt FC over IP requirements: The basic premise of the FC over IP draft remains unchanged -- Have a high speed IP based interconnect which will bridge FC traffic between two (or more) SAN islands, such that only the edge devices are aware of the fact that tunneling of FC frames over IP is being performed. Included in the initial draft was the following assumption: The backbone would be at least as reliable and at least as fast as the FC fabric itself. With such a restriction, it was expected that congestion control would not be necessary. At the Pittsburgh meeting, it was mandated that FC over IP would include congestion management of some form. In addition, a few people have indicated that they expect that this would be run over a lower speed backbone than the fabric itself. So, we are investigating options, including looking at the various transport mechanisms, as well as other non-transport related mechanisms to perform congestion management. As mentioned before, FC over IP is intended (i.e. being written with the assumption that) this protocol will be implemented in a switch device. There will be no prohibitions against implementing in a different type of device, such as an edge device, it will simply not be optimized for implementation in an edge device. Keeping that a switch topology is the target, the addition of transport support/mechanism to an otherwise level 2 type switch seems to the authors to be expensive, especially if retries as specified in many of the transports is required. Expensive in terms of additional overhead to add the transport to every FC frame as well as additional hardware requirements. Another issues that needs to be considered w.r.t. use of a transport is a transport level retry mechanism. Some transports do not provide a means by which retries can be turned off, plus are dynamic in their timeout values (e.g. when retries are invoked). If retries are required to be handled in the transport due to transport design (e.g. we use a transport such as TCP, where retries cannot be turned off), we feel it may conflict with upper level exception handling (as mentioned in the 02 draft) as well as create a cost (financial as well as engineering wise) prohibitive requirement for buffering of packets at the FC over IP SWITCH while waiting for acknowledgements. This is especially when RTT can be on the order of seconds (e.g. approaches RA_TOV). It is for this very reason we are trying to consider the various options available to us. We are considering ECM, ECN, SCTP, TCP as well as other mechanisms. We are monitoring the iSCSI threads regarding Transport for their applicability to this draft as well. While we cannot guarantee that the FC over IP draft will follow the transport method iSCSI selects, it will be considered in the analysis for FC over IP. There has never been any intention for this draft to be constrained to a private network, though that is a possible/plausible environment. There was the assumption of a well engineered network though. This requirement is being omitted, due to both feedback from attendees of the ipfc presentation as well as direction from the ADs. Summary of Requirements for FC over IP (from the authors' perspective) First Draft: (1) FC and IP devices can be used in conjunction with an 'FC over IP' compliant edge device UNMODIFIED to bridge traffic between FC islands over an IP based backbone. (2) FC over IP compliant devices will support some form of congestion management. (3) FC over IP compliant devices will support authentication of the FC over IP devices before the tunnel shall be established. Other possible requirements, which will most likely be deferred to a follow on draft: (4) Dynamic discovery of other FC over IP edge devices. (5) Data Encryption support. Additional requirements are also being sought from the Fibre Channel community. A new effort has recently been chartered in the T11 organization, FC-BB2 (Fibre Channel BackBone 2). This group will be used to solicit further requirements for FC over IP. FC-BB2 will be a companion specification to FC over IP and will leverage work done in FC over IP. It will not redefine/conflict with FC over IP, but instead will identify and define amendments to certain FC documents/values to allow greater distance and performance to be attained over an FC over IP backbone. FC devices will not be required to be FC-BB2 compliant in order to work with FC over IP (this would violate the basic premise of this work), but should a customer choose to upgrade their FC network to comply with FC-BB2, they could expect better performance and longer distances to be attained. Specifically, media specific issues, speed of light vs. timeout issues, BB credit issues, among others will be addressed in this specification. Notes: (1) Indicates that FC devices CAN be used unmodified in conjunction with FC over IP. FC-BB2 will address FC specific issues, and will identify and define amendments to certain FC documents/values to allow for greater distance/performance to be attained. (2) Retries need to be addressed. It needs to be determined if retries support at the transport level, should a transport be included, need to be required, optional or prohibited. The authors believe much of this decision is dependent on how retries are defined by the selected transport. If it can be insured that the selected transport retry mechanism will not conflict with other retry mechanisms, then it may be valuable to support. On the other hand, if requirement of retry support makes the FC over IP device an unattractive solution, due to prohibitive costs associated with implementing retry, then it should not be supported. The authors plan to come back with recommendations as well as an initial FC over IP draft that addresses the new requirements. Please direct any feedback or input to the authors (egrodriguez@lucent.com, muralir@lightsand.com, rajb@lightsand.com, and marjorie.krueger@vixel.com) as well as to the ips(ips@ece.cmu.edu) and ipfc(ipfc@standards.gadzoox.com) mailing lists. Thanks, Elizabeth G. Rodriguez Marjorie Krueger Lucent Technologies Vixel Corporation Murali Rajagopal Raj Bhagwat LightSand Communications LightSand Communications -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Friday, August 25, 2000 5:42 PM To: ips@ece.cmu.edu; ipfc@standards.gadzoox.com Subject: RE: draft-otis-fc-sctp-ip-00.txt A couple of comments: - This discussion should be moved onto the ips mailing list (ips@ece.cmu.edu) as further IETF work on FC-over-IP will occur there. Despite it's name, the fcoverip draft has not been an official work item of the ipfc WG. The use of the ipfc list rather than ips is completely understandable, as the draft was misnamed, nonetheless, further discussion should be on ips. - Wayland wrote: > I guess, what I'm trying to say is that someone should clearly state the requirements for > FCoverIP. If the requirements are that it simply run on a private tuned network which can > virtually guarantee a loss-less medium, then I would argue that we don't need congestion > control and hence a retry mechanism. I would also argue that FCoverIP is a niche, proprietary > solution which probably does not belong in IETF since it is not intended for the internet. Not only is that a good argument, it is in fact basically the position of the IETF, as conveyed by the Area Directors to the authors of the FC-over-IP draft - congestion control is REQUIRED if the work is to advance in the IETF because otherwise the protocol would be unsuitable for the internet. Whether to have a retry mechanism is an engineering decision that can be made as the work progresses (an implementation that didn't have to buffer for retransmission might consume less memory than one that did), but congestion control is REQUIRED, period. Thanks, --David (co-chair of the ips WG-to-be) p.s. For those on ips and not ipfc, I'm going to forward the ipfc discussion in a separate message. --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Aug 29 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02666; 28 Aug 2000 21:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA24679 for ips-outgoing; Mon, 28 Aug 2000 20:42:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA24674 for ; Mon, 28 Aug 2000 20:42:41 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA02544; Mon, 28 Aug 2000 17:42:26 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.144.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id RAA11086; Mon, 28 Aug 2000 17:42:25 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id RAA19306; Mon, 28 Aug 2000 17:42:25 -0700 (PDT) Message-Id: <200008290042.RAA19306@ha10nwk.EBay.Sun.COM> Date: Mon, 28 Aug 2000 17:42:26 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: FC over IP: Requirements To: ipfc@standards.gadzoox.com, ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: xU2xDPyfPmVWO0z7XR5IHA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Elizabeth, It is good to see you making progress on the FC over IP requirements. As there is now another draft that is FC over IP (with SCTP as the transport) published, is it possible that we can merge the two efforts? I would not like to see as a result one protocol to tunnel between two FC clouds and another to direct attach FC to IP. Possible considerations are if there are mechanisms to adapt one or the other draft to allow a merging? Are the transport level retry mechanisms you discuss also going to be fundemental problems with the SCTP proposal as well? If you do use TCP as a transport will your solutions to congestion control etc be sufficient to remove the need for SCTP? I would not like to see two groups trying to solve the same problem independantly. -David From owner-ips@ECE.cmu.edu Tue Aug 29 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02668; 28 Aug 2000 21:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA24436 for ips-outgoing; Mon, 28 Aug 2000 20:34:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA24430 for ; Mon, 28 Aug 2000 20:34:19 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id UAA60016; Mon, 28 Aug 2000 20:22:19 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id SAA43824; Mon, 28 Aug 2000 18:34:17 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: "Wedge" drivers To: Black_David@emc.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 28 Aug 2000 17:30:55 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/28/2000 06:34:16 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David, again we mostly agree. I do not know if the iSCSI initiator finding out about the other "equal" targets is awkward or not (I expect this is in eye of beholder) however, having explicit exchange of information perhaps at Login time, seem to be quite acceptable. What I feel we are talking about is NOT whether ALL wedge drivers will go away -- they probably WILL NOT, because of various value added functions we all think is important to have in our controllers -- but instead we are talking about whether we think there is a layering type approach, which can perform some generic load balancing across NICs and perhaps some generic alternate path retry, and where that layering is within the iSCSI driver. One contrary argument to the above, is that since we need Vendor Specific Wedge Drivers anyway, we might as well use them for load balancing and alternate path recovery. Let me argue against that for a moment. As standards occur in this industry, there has been a consistent march to make what was once a vendor specific feature into a generic feature to all/most vendors. So I believe that it is right within the normal standards march for there to be constant changes to the Wedge Drivers as new things are brought in, and old things become standards and located into some "normalized" point in the common structure. With this is mind, it seems to me, that we can move to a position (perhaps over time) where the iSCSI Device Driver is able to perform the load balancing, and alternate path recovery, yet leave other functions for a higher level Wedge Driver to perform. This tends to keep the upper layers, simpler and easier to write and maintain. If the above concept seems reasonable, then we need to keep the Multiple Conversations per Session. Again let me state my belief that the implementations that will ship out, originally, will be Single Conversations per Session, especially since it is easier and the default. This will mean that some of the current Wedge Device Drivers will remain unchanged, and therefore, alternate path recovery with load balancing will be done, (or not) at this layer. However, over time, I think this will change -- as the various vendors have time to implement the Multiple Conversations per Session. This will permit, in some cases, the elimination of Wedge Device Drivers, but probably just the simplification of the Wedge Device Drivers. . . . John L. Hufferd Black_David@emc.com@ece.cmu.edu on 08/28/2000 09:24:06 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: iSCSI: "Wedge" drivers Still with my co-chair hat off, let me try to summarize, rather than quoting and responding: If one assumes that an iSCSI session spans multiple interface processors on the storage side, then those interface processors have to share SCSI state (e.g., ranges of data for each I/O on which Ready to Transfer has been vs. needs to be issued) in some fashion so that the other processor has it when failure causes it to become involved. Use of a load balancing switch does not change this situation. Julian (IIRC) has observed that the corresponding state sharing is done for mainframe storage (e.g., I/O completion can come back on a different link than the one used for initiation), but it is not done by any SCSI implementation I'm aware of. If nothing else, that's proof by example that any box (EMC, IBM, etc.) that supports mainframe storage is in principle capable of spreading a SCSI connection across more than one interface processor. This is additional implementation complexity and effort, as state sharing across failure domains is not easy to get right, and this is a new form of sharing that SCSI implementers have little experience with. A similar issue occurs on the host side. If the system is equipped with NICs so that iSCSI above TCP/IP is entirely in software, then spreading a SCSI connection across multiple NICs should work fine. OTOH, if the system has HBAs that implement much of iSCSI in hardware in addition to TCP/IP, then not sharing iSCSI state across HBAs is again easier to build (and entails a wedge driver). I'm trying not to take a position for or against iSCSI sessions (e.g., I can see a strong rationale for using them with tape, where wedge drivers don't work and the fault tolerance requirements aren't as stringent). The point I wanted to make is that engineering and implementation concerns make it highly unlikely that iSCSI sessions would remove the need for wedge drivers for disk. My goal is to contribute to a clear understanding of the technical rationale behind sessions, for the requirements document if nothing else. Several minor issues ... John described a model in which an initiator discovers that multiple TCP/IP connections are part of the same session. That seems awkward by comparison to having each participant tell the other what the alternate transport addresses are (or how to find them) as the latter avoids having to splice a new connection into one that's already actively doing I/O. SCTP sets up multiple addresses in this fashion, FWIW. EMC error retry does not always require an EMC-written wedge driver. For example, Symmetrix works with the Veritas DMP wedge driver. There's no free lunch on configuration -- since wedge drivers are not going to go away, iSCSI sessions become something configured "in addition to" wedge driver configuration rather than "instead of". --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Aug 29 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02768; 28 Aug 2000 22:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA25456 for ips-outgoing; Mon, 28 Aug 2000 21:14:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraab.compuserve.com (ds-img-rel-2.compuserve.com [149.174.206.155]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA25448 for ; Mon, 28 Aug 2000 21:14:29 -0400 (EDT) Received: (from mailgate@localhost) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id VAA18243 for ips@ece.cmu.edu; Mon, 28 Aug 2000 21:13:54 -0400 (EDT) Received: from compuserve.com (dal-qbu-zor-vty33.as.wcom.net [216.192.249.33]) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id VAA18204 for ; Mon, 28 Aug 2000 21:13:52 -0400 (EDT) Message-ID: <39AB0958.4061389A@compuserve.com> Date: Mon, 28 Aug 2000 19:52:40 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: "Wedge" drivers References: <200008281813.LAA26716@ha10nwk.EBay.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I am not a Fibre Channel guru either, but I've listened to those who are talking about something called 'hunt groups' that have the properties of "multiple links per session." Ralph Weber David Robinson wrote: > > There has been discussion in this thread about what "could" be done > if we had multiple links per session. Not being a Fiber Channel guru, > is it true that FC does not have such a link aggregation mechanism? > If not, why not and is this perceived as a problem in the FC > community? > > -David From owner-ips@ECE.cmu.edu Tue Aug 29 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02852; 28 Aug 2000 22:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA25815 for ips-outgoing; Mon, 28 Aug 2000 21:33:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA25808 for ; Mon, 28 Aug 2000 21:33:08 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 28 Aug 2000 21:32:30 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F25@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: iSCSI: "Wedge" drivers Date: Mon, 28 Aug 2000 21:32:29 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Hunt groups are unimplemented for the most part. Common Fibre Channel implementations do not currently support link aggregation to endpoints. The usual solution to multiple links between servers and storage is a "wedge" driver using multiple SCSI over Fibre Channel connections. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Ralph Weber [SMTP:ralphoweber@compuserve.com] > Sent: Monday, August 28, 2000 8:53 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: "Wedge" drivers > > I am not a Fibre Channel guru either, but I've listened to those > who are talking about something called 'hunt groups' that have > the properties of "multiple links per session." > > Ralph Weber > > David Robinson wrote: > > > > > There has been discussion in this thread about what "could" be done > > if we had multiple links per session. Not being a Fiber Channel guru, > > is it true that FC does not have such a link aggregation mechanism? > > If not, why not and is this perceived as a problem in the FC > > community? > > > > -David > From owner-ips@ECE.cmu.edu Tue Aug 29 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03308; 29 Aug 2000 0:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA27539 for ips-outgoing; Mon, 28 Aug 2000 23:00:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraab.compuserve.com (ds-img-rel-2.compuserve.com [149.174.206.155]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA27535 for ; Mon, 28 Aug 2000 23:00:50 -0400 (EDT) Received: (from mailgate@localhost) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id XAA12169 for ips@ece.cmu.edu; Mon, 28 Aug 2000 23:00:20 -0400 (EDT) Received: from compuserve.com (dal-qbu-zox-vty40.as.wcom.net [216.192.251.40]) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id XAA12161 for ; Mon, 28 Aug 2000 23:00:08 -0400 (EDT) Message-ID: <39AB27D0.9B5EF65F@compuserve.com> Date: Mon, 28 Aug 2000 22:02:40 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: iSCSI Autosense Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Note: Clause numbers reference draft-satran-iscsi-01.txt. There is nothing in the SCSI Architecture that requires the A Flags bit (byte 2 bit 7) described in 3.2 and 3.2.1. Specifically, the SCSI Architecture does not require a transport to have the ability to disable Autosense. Contingent Allegiance (the target state that follows a CHECK CONDITION in the absence of Autosense) is not a required SCSI behavior. In fact, Contingent Allegiance is an out dated SCSI behavior that continues to be documented in SAM for backwards compatibility. Contingent Allegiance is wholly superseded by the Autosense capability. Autosense is dramatically simpler to implement provided the SCSI transport protocol allows for delivery of the sense data in the same packet as the CHECK CONDITION status (as is the case in iSCSI). IMHO iSCSI would be well served by following the precedent set by FCP. Remove the A bit and state explicitly that all iSCSI devices are required to support Autosense and that all instances of a CHECK CONDITION status shall be accompanied by Autosense data. Note: These changes will also affect 3.3.7, 5.3 and possibly other clauses of the internet draft. Thanks. Ralph... From owner-ips@ECE.cmu.edu Tue Aug 29 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04170; 29 Aug 2000 3:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA00629 for ips-outgoing; Tue, 29 Aug 2000 02:01:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA00625 for ; Tue, 29 Aug 2000 02:01:28 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA43994 for ; Tue, 29 Aug 2000 08:00:54 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA33774 for ; Tue, 29 Aug 2000 08:00:54 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.0021075F ; Tue, 29 Aug 2000 08:00:45 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 08:58:47 +0300 Subject: Re: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Randall, Your comments are valuable - but I don't exactly see how they relate to iSCSI as drafted (although I see how they relate to Doug's comments). Julo "Randall R. Stewart" on 29/08/2000 00:47:58 Please respond to "Randall R. Stewart" To: Douglas Otis cc: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data Douglas: Your notes below are along the sender side path i.e. the pusher :) But it is even worse on the receiver side. Unless the receiver puts record markings in and observes them very carefully what you get at each read is really quite dependant on the kernel buffering and what as arrives since you last did a read... Generally stuff I have seen and done in the past either is sending "fixed" records of exactly the same number of bytes so it can do a blocking read to get X bytes ... OR the old tried and true: A) read the int size followed by B) read until you fill the buffer you are expecting based on the int size you got... El.Yucko if you are doing any sort of message sending/receiving... which I presume iSCSI does... If it does not have clear delineation via fixed records OR a size indicator in a common header that is always read this will be a problem :0 R Douglas Otis wrote: > > Julo, > > RFC 793 (TCP pg 12) > "A sending TCP is allowed to collect data from the sending user and to > send that data in segments at its own convenience, until the push > function is signaled, then it must send all unsent data. When a > receiving TCP sees the PUSH flag, it must not wait for more data from > the sending TCP before passing the data to the receiving process. > > There is no necessary relationship between push functions and segment > boundaries. The data in any particular segment may be the result of a > single SEND call, in whole or part, or of multiple SEND calls. > > The purpose of push function and the PUSH flag is to push data through > from the sending user to the receiving user. It does not provide a > record service. > > There is a coupling between the push function and the use of buffers > of data that cross the TCP/user interface. Each time a PUSH flag is > associated with data placed into the receiving user's buffer, the > buffer is returned to the user for processing even if the buffer is > not filled. If data arrives that fills the user's buffer before a > PUSH is seen, the data is passed to the user in buffer size units." > > It would appear you are assuming a relationship between PUSH flags and > segment boundaries. In addition, as iSCSI is the confluence of independent > traffic, at what point would you set the PUSH flag? You suggest rather than > a push function, there is a last datagram signal according to your comments > below. Even with a PUSH flag, adding data to the buffer before completion > of push function does not provide guaranteed segment alignment nor is there > any signaling for completion nor is there any intent at providing the > alignment function you indicate. You suggest a 'last datagram' signal > provides a simple means of positioning information at the beginning of the > TCP Datagram. What signal? In keeping within TCP, how do you implement > your comments below which you state is the basis of iSCSI. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Monday, August 28, 2000 9:18 AM > > To: ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > Doug, > > > > I would appreciate if instead of a general and fuzzy referency you would > > quote > > the document page and paragraph. > > > > Julo > > > > "Douglas Otis" on 28/08/2000 19:00:56 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > Julo, > > > > Is it compliant to assume knowledge of the TCP frame by the iSCSI > > application? In many places within iSCSI discussions, references are made > > of packets, frames, and datagrams as an element within iSCSI. If creating > > frame alignment is your intent, rather than corrupting TCP APIs, perhaps > > you > > should consider SCTP. Otherwise, such discussions are little more than a > > wink and a nod at creating a frame aligned TCP. How do you > > advise a change > > to the TCP API to resolve frame alignment as you have just suggested? > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Saturday, August 26, 2000 2:28 AM > > > To: ips@ece.cmu.edu > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > Steph, > > > > > > I assume that your hardware knows when it is going to send the last SCSI > > > datagram. > > > In this case it can insert the status in the datagram header (not at the > > > end) - > > > (and BTW that is how the current draft assumes that things will > > be done). > > > > > > If you have to send sense-data (bad status) then you will send > > a separate > > > datagram. > > > > > > Julo > > > > > > Stephen Bailey on 25/08/2000 22:12:09 > > > > > > Please respond to Stephen Bailey > > > > > > To: ips@ece.cmu.edu > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > Is there anything preventing your hypothetical hardware implementor > > > > to send always good status within the last block of data? > > > > > > It depends upon the RDMA mechanism you are using. I admit that I have > > > not studied them in detail, other than ST of course, which is > > > essentially an transport protocol based upon a particular RDMA > > > mechanism. I will try to do so soon to determine if I'm all wet. > > > > > > However, assuming that the RDMA mechanism operates on a per-datagram > > > basis, I can only imagine that you will not be able to append > > > `general delivery' data to the end of an RDMA datagram. In this case, > > > you will need a separate datagram to ensure that status is delivered > > > through a separate path from the data. > > > > > > So, while I think concatenating status and data in its general form is > > > not a good idea, a good-status fast path (like the success bit) is > > > definitely the right way to think about it. Nonsuccess SCSI status is > > > so rare that any compromise you can make in the nonsuccess path to > > > make the success path go faster is worth it. > > > > > > Steph -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Tue Aug 29 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04157; 29 Aug 2000 3:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA00377 for ips-outgoing; Tue, 29 Aug 2000 01:45:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA00373 for ; Tue, 29 Aug 2000 01:45:15 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA67916 for ; Tue, 29 Aug 2000 07:44:43 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA34890 for ; Tue, 29 Aug 2000 07:44:43 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.001F8EB8 ; Tue, 29 Aug 2000 07:44:41 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 08:42:38 +0300 Subject: RE: iSCSI: "Wedge" drivers Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I am aware of the complexity related to recovery and load balancing (as I had to design it in several systems). There are several points to be made: - the view used by all the I/O software above channel in the mainframe world is extremely simple (no wedge driver) as all they see is one connection - recovery is also simple as the commands are restarted (no need to track data transfers) - load balancing is also simple - it is based on a naive greedy scheme - take the first available channel with some randomness inserted at search start I attempted this approach in the 00 draft (Adelaide) and we changed it to the current draft (with symmetric connections) as most of my colleagues felt that asymmetric connections will be hard to understand and implement properly. While many of you keep telling me that wedge drivers won't go away nobody gave one good example to support that assertion. I think that as SCSI moves to a level of complexity that goes beyond what was available on mainframes (with the standardization of 3rd party commands and the object model) it would be a mistake not to attempt to simplify SCSIs view of the transport and remove recovery and transport load balancing from it. Julo Black_David@emc.com on 28/08/2000 19:24:06 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: iSCSI: "Wedge" drivers Still with my co-chair hat off, let me try to summarize, rather than quoting and responding: If one assumes that an iSCSI session spans multiple interface processors on the storage side, then those interface processors have to share SCSI state (e.g., ranges of data for each I/O on which Ready to Transfer has been vs. needs to be issued) in some fashion so that the other processor has it when failure causes it to become involved. Use of a load balancing switch does not change this situation. Julian (IIRC) has observed that the corresponding state sharing is done for mainframe storage (e.g., I/O completion can come back on a different link than the one used for initiation), but it is not done by any SCSI implementation I'm aware of. If nothing else, that's proof by example that any box (EMC, IBM, etc.) that supports mainframe storage is in principle capable of spreading a SCSI connection across more than one interface processor. This is additional implementation complexity and effort, as state sharing across failure domains is not easy to get right, and this is a new form of sharing that SCSI implementers have little experience with. A similar issue occurs on the host side. If the system is equipped with NICs so that iSCSI above TCP/IP is entirely in software, then spreading a SCSI connection across multiple NICs should work fine. OTOH, if the system has HBAs that implement much of iSCSI in hardware in addition to TCP/IP, then not sharing iSCSI state across HBAs is again easier to build (and entails a wedge driver). I'm trying not to take a position for or against iSCSI sessions (e.g., I can see a strong rationale for using them with tape, where wedge drivers don't work and the fault tolerance requirements aren't as stringent). The point I wanted to make is that engineering and implementation concerns make it highly unlikely that iSCSI sessions would remove the need for wedge drivers for disk. My goal is to contribute to a clear understanding of the technical rationale behind sessions, for the requirements document if nothing else. Several minor issues ... John described a model in which an initiator discovers that multiple TCP/IP connections are part of the same session. That seems awkward by comparison to having each participant tell the other what the alternate transport addresses are (or how to find them) as the latter avoids having to splice a new connection into one that's already actively doing I/O. SCTP sets up multiple addresses in this fashion, FWIW. EMC error retry does not always require an EMC-written wedge driver. For example, Symmetrix works with the Veritas DMP wedge driver. There's no free lunch on configuration -- since wedge drivers are not going to go away, iSCSI sessions become something configured "in addition to" wedge driver configuration rather than "instead of". --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Aug 29 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04159; 29 Aug 2000 3:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA00515 for ips-outgoing; Tue, 29 Aug 2000 01:55:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA00508 for ; Tue, 29 Aug 2000 01:55:42 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA40544 for ; Tue, 29 Aug 2000 07:55:10 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA22976 for ; Tue, 29 Aug 2000 07:55:10 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.00208210 ; Tue, 29 Aug 2000 07:55:04 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 08:53:03 +0300 Subject: RE: iSCSI: "Wedge" drivers Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I assume that high end users have this problem and that is why we have the wedge drivers. Julo David Robinson on 28/08/2000 21:13:08 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: iSCSI: "Wedge" drivers There has been discussion in this thread about what "could" be done if we had multiple links per session. Not being a Fiber Channel guru, is it true that FC does not have such a link aggregation mechanism? If not, why not and is this perceived as a problem in the FC community? -David From owner-ips@ECE.cmu.edu Tue Aug 29 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04301; 29 Aug 2000 3:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA01130 for ips-outgoing; Tue, 29 Aug 2000 02:36:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA01126 for ; Tue, 29 Aug 2000 02:36:01 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA27506 for ; Tue, 29 Aug 2000 08:35:30 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA39238 for ; Tue, 29 Aug 2000 08:35:30 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.002434ED ; Tue, 29 Aug 2000 08:35:28 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 09:33:29 +0300 Subject: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, With all the heated debate about multiple vs. single connection a request I made a while ago got no significant reply (neither for nor against). The request was to consider a proposal made by Kalman Meth to reconsider the asymmetric model with the addition of a path selection made by the initiator. This proposal allows removing the command counters - as commands use a single TCP connection. The single connection can also be a shared data+control connection. In case of multiple connection the data path to be used is selected and maintained until the command ends. Thanks, Julo From owner-ips@ECE.cmu.edu Tue Aug 29 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04559; 29 Aug 2000 4:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA01721 for ips-outgoing; Tue, 29 Aug 2000 03:19:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA01717 for ; Tue, 29 Aug 2000 03:19:20 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id AAA29536 for ; Tue, 29 Aug 2000 00:19:19 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.154.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id AAA25868 for ; Tue, 29 Aug 2000 00:19:18 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id AAA17470; Tue, 29 Aug 2000 00:19:18 -0700 (PDT) Message-ID: <39AB63F3.42D3D94D@ebay.sun.com> Date: Tue, 29 Aug 2000 00:19:15 -0700 From: David Robinson X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk While I am not yet convinced of the actual need for multiple connections, I agree with Kalman that using a single command connection will greatly simplify the protocol and error recovery. I would prefer it over the current propsal. While this will address the perceived performance problem that will be addressed by multiple data connections, I don't see how it will help availability. Commands are only being sent on one connection so if it fails there is no redundant path and we are back to higher level software managing connections. Exactly what the other thread is arguing against. -David julian_satran@il.ibm.com wrote: > > Dear colleagues, > > With all the heated debate about multiple vs. single connection a request I > made > a while ago got no significant reply (neither for nor against). > > The request was to consider a proposal made by Kalman Meth to reconsider > the asymmetric model with the addition of a path selection made by the > initiator. > > This proposal allows removing the command counters - as commands use a > single TCP connection. The single connection can also be a shared > data+control connection. > > In case of multiple connection the data path to be used is selected and > maintained until > the command ends. > > Thanks, > Julo From owner-ips@ECE.cmu.edu Tue Aug 29 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04569; 29 Aug 2000 4:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA01909 for ips-outgoing; Tue, 29 Aug 2000 03:30:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA01902 for ; Tue, 29 Aug 2000 03:30:16 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id JAA204064 for ; Tue, 29 Aug 2000 09:29:44 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA54818 for ; Tue, 29 Aug 2000 09:29:44 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.00292C26 ; Tue, 29 Aug 2000 09:29:42 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 10:27:42 +0300 Subject: Re: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The command connection role can be taken over by any other connection or a new connection. All commands that have not have their status received will be reissued. Commad numbering is not needed anymore but status numbering is needed for recovery. This proposal is similar to what we had in the 00 draft with the added restriction that data and status will travel over a single connection and this connection will be designated. Julo David Robinson on 29/08/2000 10:19:15 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple connections While I am not yet convinced of the actual need for multiple connections, I agree with Kalman that using a single command connection will greatly simplify the protocol and error recovery. I would prefer it over the current propsal. While this will address the perceived performance problem that will be addressed by multiple data connections, I don't see how it will help availability. Commands are only being sent on one connection so if it fails there is no redundant path and we are back to higher level software managing connections. Exactly what the other thread is arguing against. -David julian_satran@il.ibm.com wrote: > > Dear colleagues, > > With all the heated debate about multiple vs. single connection a request I > made > a while ago got no significant reply (neither for nor against). > > The request was to consider a proposal made by Kalman Meth to reconsider > the asymmetric model with the addition of a path selection made by the > initiator. > > This proposal allows removing the command counters - as commands use a > single TCP connection. The single connection can also be a shared > data+control connection. > > In case of multiple connection the data path to be used is selected and > maintained until > the command ends. > > Thanks, > Julo From owner-ips@ECE.cmu.edu Tue Aug 29 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04626; 29 Aug 2000 4:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA02142 for ips-outgoing; Tue, 29 Aug 2000 03:45:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA02138 for ; Tue, 29 Aug 2000 03:45:33 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA27336 for ; Tue, 29 Aug 2000 09:45:00 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA33938 for ; Tue, 29 Aug 2000 09:44:59 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.002A9175 ; Tue, 29 Aug 2000 09:44:57 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 10:42:57 +0300 Subject: products Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk You will find this interesting: http://www.searchstorage.com/searchStorage_Original_Content_Item/0,1779,331144,00.html Julo From owner-ips@ECE.cmu.edu Tue Aug 29 14:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06081; 29 Aug 2000 10:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA15262 for ips-outgoing; Tue, 29 Aug 2000 08:50:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA15248 for ; Tue, 29 Aug 2000 08:50:43 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id HAA02189; Tue, 29 Aug 2000 07:48:45 -0500 Message-ID: <39ABB12D.F76CBA77@stewart.chicago.il.us> Date: Tue, 29 Aug 2000 07:48:45 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: David Robinson CC: ipfc@standards.gadzoox.com, ips@ece.cmu.edu Subject: Re: FC over IP: Requirements References: <200008290042.RAA19306@ha10nwk.EBay.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David: Comments below... David Robinson wrote: > > Elizabeth, > It is good to see you making progress on the FC over IP requirements. > As there is now another draft that is FC over IP (with SCTP as > the transport) published, is it possible that we can merge the > two efforts? I would not like to see as a result one protocol > to tunnel between two FC clouds and another to direct attach > FC to IP. > > Possible considerations are if there are mechanisms to adapt > one or the other draft to allow a merging? Are the transport level > retry mechanisms you discuss also going to be fundemental problems > with the SCTP proposal as well? If you do use TCP as a transport > will your solutions to congestion control etc be sufficient to > remove the need for SCTP? I would not like to see two groups > trying to solve the same problem independantly. SCTP and TCP have essentially the same Congestion Control algorithms. The differences between them can be summarized as follows: - Message oriented on word boundary(SCTP) vs Byte stream oriented(TCP) - Multi-homing support in SCTP and not in TCP, retry's go to alternates + - Unordered delivery service available in SCTP - Ability to order messages within different context's (in SCTP) to escape the "head-of-line" blocking issue. - Extension draft allows for a non-retransmit option in SCTP (so you can escape retransmissions if desired). - Acknowledgement in SCTP is always done via SACK, SACK is an option in some TCP implementations. - Interface designed to have more of the parameters tuneable for private networks. I think the bottom line is if you go with TCP you will not need to put in SCTP and visa-versa. Of course you can also have an option for either one.. but if you do use TCP (or both) you need to specify: o How under TCP you do message boundaries, Push's et.al. and parsing on the receiver side, since a Push does not assure the receiver of getting data in any particular unit size. o How you intend to support failover between multiple NIC cards (you get this for free with SCTP), if you even want this. Note that during a failure one of the nice things with SCTP is the first time-out retransmission (if you are retransmitting data) moves the retransmission to an alternate NIC if possible. This means on a broken NIC, recovery is 1 retransmit timer away versus going through N retransmissions before failure OR having to run application level timers to send FEC type packets to a second NIC and hope you are not wasting bandwidth on the network.. o If you have any head of line blocking issues you will need to out-line how you will escape these in TCP i.e. the use of multiple connections (and management of these) to provide some parrallel ordering. o You will have to live with ordered messages since there is no way to completely escape ordering in TCP. This may be of no issue to you... o You may need to pick implemenations that allow some flexibility in setting up TCP parameters. Most don't allow this so it may limit some of your choices. Now some of these issues I list may not even bother you.. thats ok.. but they are considerations that you need to take into account before you make any decision... R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Tue Aug 29 14:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06162; 29 Aug 2000 10:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA16315 for ips-outgoing; Tue, 29 Aug 2000 09:26:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from marlborough.cnchost.com (marlborough.concentric.net [207.155.248.14]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA16311 for ; Tue, 29 Aug 2000 09:26:09 -0400 (EDT) Received: from Trebia5 (45.mht.dialup.G4.NET [216.177.2.45]) by marlborough.cnchost.com id JAA09655; Tue, 29 Aug 2000 09:26:06 -0400 (EDT) [ConcentricHost SMTP Relay 1.8] From: "James Smart" To: "ips" Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 09:30:40 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 In-Reply-To: <39AB27D0.9B5EF65F@compuserve.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk As a long time implementer of SCSI and SCSI driver stacks, I too recommend that Autosense be mandatory. As SCSI becomes more packetized with more packets in flight - this feature becomes more and more necessary. It also begs for more support of ACA. -- james -------------------- James Smart Trebia Networks, Inc Ph: 978-318-9547 35 Forest Ridge Rd Cell: 603-674-3687 Concord, MA 01742 james.smart@trebia.com > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ralph Weber > Sent: Monday, August 28, 2000 11:03 PM > To: IPS Reflector > Subject: iSCSI Autosense > > > Note: Clause numbers reference draft-satran-iscsi-01.txt. > > There is nothing in the SCSI Architecture that requires the A > Flags bit (byte 2 bit 7) described in 3.2 and 3.2.1. Specifically, > the SCSI Architecture does not require a transport to have the > ability to disable Autosense. > > Contingent Allegiance (the target state that follows a CHECK > CONDITION in the absence of Autosense) is not a required SCSI > behavior. In fact, Contingent Allegiance is an out dated > SCSI behavior that continues to be documented in SAM for > backwards compatibility. Contingent Allegiance is wholly > superseded by the Autosense capability. > > Autosense is dramatically simpler to implement provided the > SCSI transport protocol allows for delivery of the sense data > in the same packet as the CHECK CONDITION status (as is the > case in iSCSI). > > IMHO iSCSI would be well served by following the precedent > set by FCP. Remove the A bit and state explicitly that all > iSCSI devices are required to support Autosense and that > all instances of a CHECK CONDITION status shall be accompanied > by Autosense data. > > Note: These changes will also affect 3.3.7, 5.3 and possibly > other clauses of the internet draft. > > Thanks. > > Ralph... > > > > From owner-ips@ECE.cmu.edu Tue Aug 29 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06213; 29 Aug 2000 10:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA16364 for ips-outgoing; Tue, 29 Aug 2000 09:27:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA16356 for ; Tue, 29 Aug 2000 09:27:29 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 29 Aug 2000 09:26:53 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F28@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: "Wedge" drivers Date: Tue, 29 Aug 2000 09:26:49 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > While many of you keep telling me that wedge drivers won't go away nobody > gave one good example to support that assertion. With my co-chair hat off, here are four, two from the mail exchange: (1) Sharing SCSI state across array interface processors involves some work. Use of multiple SCSI connections and wedge drivers for fault tolerance and load balancing will persist in systems that choose not to follow that implementation path. (2) If iSCSI is implemented in hardware, so that a system has multiple iSCSI HBAs rather than multiple NICs with a common iSCSI software stack, multiple SCSI connections and wedge drivers are again the path of least resistance to fault tolerance and load balancing. And two new ones: (3) Naive greedy load balancing is not optimal. Significant improvements are possible based on an understanding of how the storage behaves. PowerPath contains Symmetrix-specific logic that does this, and I don't think that's appropriate for standardization. (4) Some arrays need additional support for fail-over. Clariion uses an active fail-over architecture in which the array must be instructed to fail-over, and the fail-over occurs across separate SCSI connections. I don't think that the Clariion-specific logic that does this is appropriate for standardization. As I wrote earlier, I have no problem with specifying a session abstraction for iSCSI, but it won't eliminate wedge drivers. With my co-chair hat on, the current consensus is to specify a session abstraction and review the result. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Aug 29 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06227; 29 Aug 2000 10:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA16718 for ips-outgoing; Tue, 29 Aug 2000 09:38:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA16713 for ; Tue, 29 Aug 2000 09:38:33 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 29 Aug 2000 09:38:02 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F2A@corpmx9.isus.emc.com> To: dotis@sanlight.net, julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: Data in SCSI Response or SCSI Data Date: Tue, 29 Aug 2000 09:37:56 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I should point out that TCP's URGENT functionality may be of some use here -- it won't provide the level of support for message boundaries found in SCTP, but might still be quite useful. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Douglas Otis [SMTP:dotis@sanlight.net] > Sent: Monday, August 28, 2000 2:25 PM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > Julo, > > RFC 793 (TCP pg 12) > "A sending TCP is allowed to collect data from the sending user and to > send that data in segments at its own convenience, until the push > function is signaled, then it must send all unsent data. When a > receiving TCP sees the PUSH flag, it must not wait for more data from > the sending TCP before passing the data to the receiving process. > > There is no necessary relationship between push functions and segment > boundaries. The data in any particular segment may be the result of a > single SEND call, in whole or part, or of multiple SEND calls. > > The purpose of push function and the PUSH flag is to push data through > from the sending user to the receiving user. It does not provide a > record service. > > There is a coupling between the push function and the use of buffers > of data that cross the TCP/user interface. Each time a PUSH flag is > associated with data placed into the receiving user's buffer, the > buffer is returned to the user for processing even if the buffer is > not filled. If data arrives that fills the user's buffer before a > PUSH is seen, the data is passed to the user in buffer size units." > > It would appear you are assuming a relationship between PUSH flags and > segment boundaries. In addition, as iSCSI is the confluence of > independent > traffic, at what point would you set the PUSH flag? You suggest rather > than > a push function, there is a last datagram signal according to your > comments > below. Even with a PUSH flag, adding data to the buffer before completion > of push function does not provide guaranteed segment alignment nor is > there > any signaling for completion nor is there any intent at providing the > alignment function you indicate. You suggest a 'last datagram' signal > provides a simple means of positioning information at the beginning of the > TCP Datagram. What signal? In keeping within TCP, how do you implement > your comments below which you state is the basis of iSCSI. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Monday, August 28, 2000 9:18 AM > > To: ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > Doug, > > > > I would appreciate if instead of a general and fuzzy referency you would > > quote > > the document page and paragraph. > > > > Julo > > > > "Douglas Otis" on 28/08/2000 19:00:56 > > > > Please respond to "Douglas Otis" > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > cc: > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > Julo, > > > > Is it compliant to assume knowledge of the TCP frame by the iSCSI > > application? In many places within iSCSI discussions, references are > made > > of packets, frames, and datagrams as an element within iSCSI. If > creating > > frame alignment is your intent, rather than corrupting TCP APIs, perhaps > > you > > should consider SCTP. Otherwise, such discussions are little more than > a > > wink and a nod at creating a frame aligned TCP. How do you > > advise a change > > to the TCP API to resolve frame alignment as you have just suggested? > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Saturday, August 26, 2000 2:28 AM > > > To: ips@ece.cmu.edu > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > Steph, > > > > > > I assume that your hardware knows when it is going to send the last > SCSI > > > datagram. > > > In this case it can insert the status in the datagram header (not at > the > > > end) - > > > (and BTW that is how the current draft assumes that things will > > be done). > > > > > > If you have to send sense-data (bad status) then you will send > > a separate > > > datagram. > > > > > > Julo > > > > > > Stephen Bailey on 25/08/2000 22:12:09 > > > > > > Please respond to Stephen Bailey > > > > > > To: ips@ece.cmu.edu > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > Is there anything preventing your hypothetical hardware implementor > > > > to send always good status within the last block of data? > > > > > > It depends upon the RDMA mechanism you are using. I admit that I have > > > not studied them in detail, other than ST of course, which is > > > essentially an transport protocol based upon a particular RDMA > > > mechanism. I will try to do so soon to determine if I'm all wet. > > > > > > However, assuming that the RDMA mechanism operates on a per-datagram > > > basis, I can only imagine that you will not be able to append > > > `general delivery' data to the end of an RDMA datagram. In this case, > > > you will need a separate datagram to ensure that status is delivered > > > through a separate path from the data. > > > > > > So, while I think concatenating status and data in its general form is > > > not a good idea, a good-status fast path (like the success bit) is > > > definitely the right way to think about it. Nonsuccess SCSI status is > > > so rare that any compromise you can make in the nonsuccess path to > > > make the success path go faster is worth it. > > > > > > Steph From owner-ips@ECE.cmu.edu Tue Aug 29 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06418; 29 Aug 2000 11:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA17771 for ips-outgoing; Tue, 29 Aug 2000 10:05:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA17767 for ; Tue, 29 Aug 2000 10:04:58 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id BDCED49A5 for ; Tue, 29 Aug 2000 07:04:57 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA28462 for ; Tue, 29 Aug 2000 07:04:31 -0700 (PDT) Message-Id: <4.2.2.20000829060616.00c12d50@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 29 Aug 2000 06:14:41 -0700 To: ips@ece.cmu.edu From: Michael Krause Subject: RE: iSCSI: "Wedge" drivers In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 05:30 PM 8/28/00 -0700, John Hufferd/San Jose/IBM wrote: >structure. With this is mind, it seems to me, that we can move to a >position (perhaps over time) where the iSCSI Device Driver is able to >perform the load balancing, and alternate path recovery, yet leave other >functions for a higher level Wedge Driver to perform. This tends to keep >the upper layers, simpler and easier to write and maintain. This is exactly the point of the previous discussion on QoS, dealing with fabric events (hot-plug / removal), path selection, etc. Keep these problems outside of the upper layers but define the interfaces and basic policies within the iSCSI layer and standardize that. This simplifies quite a bit of the implementation since one can choose what operations they support and to what extent while allowing one to transparently upgrade their offerings while maintaining complete backward compatibility and interoperability. >If the above concept seems reasonable, then we need to keep the Multiple >Conversations per Session. Agreed. >Again let me state my belief that the implementations that will ship out, >originally, will be Single Conversations per Session, especially since it >is easier and the default. This will mean that some of the current Wedge >Device Drivers will remain unchanged, and therefore, alternate path >recovery with load balancing will be done, (or not) at this layer. >However, over time, I think this will change -- as the various vendors have >time to implement the Multiple Conversations per Session. This will >permit, in some cases, the elimination of Wedge Device Drivers, but >probably just the simplification of the Wedge Device Drivers. Wedge drivers may provide more value-add capabilities in the future especially for determining the QoS policies on dynamically changing workloads, cost models, and fabrics with changing routes or capabilities (these are also functions that could be done outside of the wedge driver in a standard way but leverage the standard interfaces to the iSCSI component). A layered approach keeps iSCSI simple and allows vendors and customers to put together the functional and complexity level that best meets their market requirements and more easily enhance / modify using a phased delivery approach. Mike From owner-ips@ECE.cmu.edu Tue Aug 29 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06433; 29 Aug 2000 11:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA18987 for ips-outgoing; Tue, 29 Aug 2000 10:33:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA18981 for ; Tue, 29 Aug 2000 10:33:44 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA30030 for ; Tue, 29 Aug 2000 16:33:12 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA36796 for ; Tue, 29 Aug 2000 16:33:13 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.004FF054 ; Tue, 29 Aug 2000 16:33:08 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 17:31:09 +0300 Subject: what's in a name Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Yaron Klein just pointed out to me tha Virtual Private SAN and VPS are TM of Pathlight (www.pathlight.com) and so are SAN Gateway and SAN router. Costa earlier pointed out that iSCSI is the name of a cable for iMAC (but not a trademark as far as I can tell). Can somebody tell us how to protect at least iSCSI from TM claims? Can the Internet Society do it (before it is too late)? Julo From owner-ips@ECE.cmu.edu Tue Aug 29 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06507; 29 Aug 2000 11:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA19303 for ips-outgoing; Tue, 29 Aug 2000 10:43:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA19295 for ; Tue, 29 Aug 2000 10:43:16 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id KAA17420; Tue, 29 Aug 2000 10:42:47 -0400 (EDT) Date: Tue, 29 Aug 2000 10:42:47 -0400 (EDT) From: Scott Bradner Message-Id: <200008291442.KAA17420@newdev.harvard.edu> To: ips@ece.cmu.edu, julian_satran@il.ibm.com Subject: Re: what's in a name Sender: owner-ips@ece.cmu.edu Precedence: bulk > Can the Internet Society do it (before it is too late)? the ISOC has not been trademarking IETF protocol names - there are too many, its hard to tell when a name has become solid enough to even think of doing so and the cost mounts up quickly Scott From owner-ips@ECE.cmu.edu Tue Aug 29 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06525; 29 Aug 2000 11:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA19845 for ips-outgoing; Tue, 29 Aug 2000 10:56:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA19833 for ; Tue, 29 Aug 2000 10:56:38 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA115456 for ; Tue, 29 Aug 2000 16:56:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA28690 for ; Tue, 29 Aug 2000 16:56:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.00520706 ; Tue, 29 Aug 2000 16:55:57 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 17:53:58 +0300 Subject: RE: iSCSI: "Wedge" drivers Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I understand 3 and I somehow expected it. If I would have to reimplement this under iSCSI I would choose a plug-in in iSCSI to do it (a policy module) and iSCSI would clearly support such a construct (and make it also future-proof). I have some trouble with 1 and 2. In whatever design you choose at the array level you have to have some sharing as some commands are targeted to a LU and all it's queues (i.e. separation per initiator is never complete if you take SAM seriously). However some design might choose to share only for those commands - and talking to you with you chair hat off - I don't think that you are with one of those (:- I am no sure I understand 4 either - if the array does no "sense" the fail over. Now talking to you with your hat on - we are ready to get to the drafting - but I feel that we are still left with the symmetric/asymmetric dilemma and I would appreciate input: The asymmetric solution is simpler for those using a single connection. Is there any serious drawback? Dear Colleagues - Please give us feedback. We can choose only one solution? Which is the right one? Julo Black_David@emc.com on 29/08/2000 16:26:49 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: "Wedge" drivers > While many of you keep telling me that wedge drivers won't go away nobody > gave one good example to support that assertion. With my co-chair hat off, here are four, two from the mail exchange: (1) Sharing SCSI state across array interface processors involves some work. Use of multiple SCSI connections and wedge drivers for fault tolerance and load balancing will persist in systems that choose not to follow that implementation path. (2) If iSCSI is implemented in hardware, so that a system has multiple iSCSI HBAs rather than multiple NICs with a common iSCSI software stack, multiple SCSI connections and wedge drivers are again the path of least resistance to fault tolerance and load balancing. And two new ones: (3) Naive greedy load balancing is not optimal. Significant improvements are possible based on an understanding of how the storage behaves. PowerPath contains Symmetrix-specific logic that does this, and I don't think that's appropriate for standardization. (4) Some arrays need additional support for fail-over. Clariion uses an active fail-over architecture in which the array must be instructed to fail-over, and the fail-over occurs across separate SCSI connections. I don't think that the Clariion-specific logic that does this is appropriate for standardization. As I wrote earlier, I have no problem with specifying a session abstraction for iSCSI, but it won't eliminate wedge drivers. With my co-chair hat on, the current consensus is to specify a session abstraction and review the result. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Aug 29 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06520; 29 Aug 2000 11:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA19973 for ips-outgoing; Tue, 29 Aug 2000 10:59:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA19965 for ; Tue, 29 Aug 2000 10:59:35 -0400 (EDT) Received: from lightsand.com (cx418298-b.orng1.occa.home.com [24.1.179.117]) by lightsand.com (8.9.3+Sun/8.9.1) with ESMTP id HAA28906; Tue, 29 Aug 2000 07:58:27 -0700 (PDT) Message-ID: <39ABD121.8DFE0A95@lightsand.com> Date: Tue, 29 Aug 2000 08:05:05 -0700 From: Murali Rajagopal Organization: @lightsand.com X-Mailer: Mozilla 4.5 [en]C-AtHome0405 (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: David Robinson CC: ipfc@standards.gadzoox.com, ips@ece.cmu.edu Subject: Re: FC over IP: Requirements References: <200008290042.RAA19306@ha10nwk.EBay.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David: The authors of FC Over IP draft see at least 3 distinct options. Briefly, this is what we think: 1. Make effective use of the Congestion Indicator (CE) bit from a modern ECN capable Router but without the use of any transport. This indicator will then be used as a feedback signal to the FC-IP device and will have a bearing on the FC BB credit control between this device and the upstream FC Switch. 2. Investigate the possible use of a subset of full TCP functionality that would suit the requirements of this application. 3. Similarly, investigate the posible use of a subset of SCTP. As you can tell, these are mutually exclusive and would be difficult to combine them. Regards, Murali Rajagopal muralir@lightSand.com David Robinson wrote: > > Elizabeth, > It is good to see you making progress on the FC over IP requirements. > As there is now another draft that is FC over IP (with SCTP as > the transport) published, is it possible that we can merge the > two efforts? I would not like to see as a result one protocol > to tunnel between two FC clouds and another to direct attach > FC to IP. > > Possible considerations are if there are mechanisms to adapt > one or the other draft to allow a merging? Are the transport level > retry mechanisms you discuss also going to be fundemental problems > with the SCTP proposal as well? If you do use TCP as a transport > will your solutions to congestion control etc be sufficient to > remove the need for SCTP? I would not like to see two groups > trying to solve the same problem independantly. > > -David > From owner-ips@ECE.cmu.edu Tue Aug 29 17:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07021; 29 Aug 2000 13:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA23616 for ips-outgoing; Tue, 29 Aug 2000 12:28:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA23610 for ; Tue, 29 Aug 2000 12:28:38 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA20278 for ; Tue, 29 Aug 2000 18:28:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA35028 for ; Tue, 29 Aug 2000 18:28:04 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694A.005A73BA ; Tue, 29 Aug 2000 18:27:58 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 29 Aug 2000 19:25:59 +0300 Subject: Re: what's in a name Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Scott, Isn't there anything in the rules for giving TM names that forbids use of public names? Can somebody ask a TM for TCP? Julo Scott Bradner on 29/08/2000 17:42:47 Please respond to Scott Bradner To: ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: what's in a name > Can the Internet Society do it (before it is too late)? the ISOC has not been trademarking IETF protocol names - there are too many, its hard to tell when a name has become solid enough to even think of doing so and the cost mounts up quickly Scott From owner-ips@ECE.cmu.edu Tue Aug 29 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07035; 29 Aug 2000 13:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA23763 for ips-outgoing; Tue, 29 Aug 2000 12:32:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA23754 for ; Tue, 29 Aug 2000 12:32:45 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id JAA20271 for ; Tue, 29 Aug 2000 09:32:44 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Tue, 29 Aug 2000 09:32:41 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD185@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: multiple connections Date: Tue, 29 Aug 2000 09:32:39 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] wrote: > The request was to consider a proposal made by Kalman Meth to > reconsider > the asymmetric model with the addition of a path selection made by the > initiator. I don't see the current proposal in the iSCSI draft to be burdensome to implement. I see no reason to change. The asymmetric model feels like more of a kludge, but I have a hard time constructing a technical argument against it, so I wouldn't (couldn't :-) object if it was adopted. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Tue Aug 29 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07060; 29 Aug 2000 13:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA23317 for ips-outgoing; Tue, 29 Aug 2000 12:19:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA23313 for ; Tue, 29 Aug 2000 12:19:30 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id LAA02988; Tue, 29 Aug 2000 11:19:27 -0500 Message-ID: <39ABE28E.BFF0AE11@stewart.chicago.il.us> Date: Tue, 29 Aug 2000 11:19:26 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Murali Rajagopal CC: David Robinson , ipfc@standards.gadzoox.com, ips@ece.cmu.edu Subject: Re: FC over IP: Requirements References: <200008290042.RAA19306@ha10nwk.EBay.Sun.COM> <39ABD121.8DFE0A95@lightsand.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Some thoughts and notes: Murali Rajagopal wrote: > > David: > > The authors of FC Over IP draft see at least 3 distinct options. > Briefly, this is what we think: > > 1. Make effective use of the Congestion Indicator (CE) bit from a modern > ECN capable Router but > without the use of any transport. This indicator will then be used as a > feedback signal to the > FC-IP device and will have a bearing on the FC BB credit control between > this device and the upstream > FC Switch. > To do ECN you need more than just the two bits provided in the IP header. You also need a method to communicate to your peer that you have seen the "congestion notification" at a given point in the data stream. This notification must be propagated back to the sender until the sender itself reduces its sending rate and sends forward an acknowledgement that it has done so. The RFC (RFC2481) can be very helpful if you have not read it. It is short and there are some tricks to implementing it... these make it most suitable for implemenation in a transport protocol. I think this is what you would be building to a minor degree if you choose this option. How well deployed and available this is, (in the network) is another question I can't answer. I do like useing ECN though, this is why the SCTP reference implementation comes ECN enabled... You may also need to answer the question of what do you do if ECN is not supported by the routers? Do you kill the iSCSI connection? How do you proceed... if you need to move forward then you must also add back in the TCP friendly congestion control... > 2. Investigate the possible use of a subset of full TCP functionality > that would suit the requirements > of this application. Hmm, you could gut TCP leaving in the congestion control and attempting to put message boundaries in. Look into getting rid of the head-of-line blocking issue (if you need to get around this). This is how SCTP got started and the end result is SCTP... You might possibly not need all the features in SCTP but the more you dig into a transport protocol the more you will find it involves more than you think :) > > 3. Similarly, investigate the posible use of a subset of SCTP. SCTP is fairly sub-settable. You do not have to understand how to do bundling and some of the other features. You do have to understand how to decode a bundle of chunks.. just not encode it. If one wants you can build an implemenation that eliminates the stream concept, by simply only asking for one stream and setting a limit of only one stream on all outbound association setups. I would think you may possibly want the unreliable transport extension that we are currently working on.. but this does seem a item for debate... in any event, it is always available if the WG decides it needs it(the unreliable extension)... In general I think if you dig in a bit you will find that SCTP is A) both extensible and B) can be kept to a minimum subset to some extent. I don't have a complete picture of all of the iSCSI requirments but I think IMHO you will find a closer fit in SCTP then you will in any of the other options... of course you knew I would say that :) R > > As you can tell, these are mutually exclusive and would be difficult to > combine them. > > Regards, > > Murali Rajagopal > > muralir@lightSand.com > > David Robinson wrote: > > > > Elizabeth, > > It is good to see you making progress on the FC over IP requirements. > > As there is now another draft that is FC over IP (with SCTP as > > the transport) published, is it possible that we can merge the > > two efforts? I would not like to see as a result one protocol > > to tunnel between two FC clouds and another to direct attach > > FC to IP. > > > > Possible considerations are if there are mechanisms to adapt > > one or the other draft to allow a merging? Are the transport level > > retry mechanisms you discuss also going to be fundemental problems > > with the SCTP proposal as well? If you do use TCP as a transport > > will your solutions to congestion control etc be sufficient to > > remove the need for SCTP? I would not like to see two groups > > trying to solve the same problem independantly. > > > > -David > > -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Tue Aug 29 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07230; 29 Aug 2000 14:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA25242 for ips-outgoing; Tue, 29 Aug 2000 13:17:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel2.hp.com (cosrel2.hp.com [156.153.255.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA25233 for ; Tue, 29 Aug 2000 13:17:09 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by cosrel2.hp.com (Postfix) with ESMTP id 5DD49445; Tue, 29 Aug 2000 11:16:57 -0600 (MDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id KAA19277; Tue, 29 Aug 2000 10:17:07 -0700 (PDT) X-OpenMail-Hops: 1 Date: Tue, 29 Aug 2000 10:17:02 -0700 Message-Id: Subject: RE: iSCSI Autosense MIME-Version: 1.0 To: ENDL_TX@computer.org, ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk If we are looking for consensus on this issue, I agree with Ralph. Better to clean up now than implement something that is waste of energy. Somesh > -----Original Message----- > From: ralphoweber@compuserve.com [mailto:ralphoweber@compuserve.com] > Sent: Monday, August 28, 2000 8:03 PM > To: ips@ece.cmu.edu > Subject: FW: iSCSI Autosense > > > Note: Clause numbers reference draft-satran-iscsi-01.txt. > > There is nothing in the SCSI Architecture that requires the A > Flags bit (byte 2 bit 7) described in 3.2 and 3.2.1. Specifically, > the SCSI Architecture does not require a transport to have the > ability to disable Autosense. > > Contingent Allegiance (the target state that follows a CHECK > CONDITION in the absence of Autosense) is not a required SCSI > behavior. In fact, Contingent Allegiance is an out dated > SCSI behavior that continues to be documented in SAM for > backwards compatibility. Contingent Allegiance is wholly > superseded by the Autosense capability. > > Autosense is dramatically simpler to implement provided the > SCSI transport protocol allows for delivery of the sense data > in the same packet as the CHECK CONDITION status (as is the > case in iSCSI). > > IMHO iSCSI would be well served by following the precedent > set by FCP. Remove the A bit and state explicitly that all > iSCSI devices are required to support Autosense and that > all instances of a CHECK CONDITION status shall be accompanied > by Autosense data. > > Note: These changes will also affect 3.3.7, 5.3 and possibly > other clauses of the internet draft. > > Thanks. > > Ralph... > > > From owner-ips@ECE.cmu.edu Tue Aug 29 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07215; 29 Aug 2000 14:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA25525 for ips-outgoing; Tue, 29 Aug 2000 13:25:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA25520 for ; Tue, 29 Aug 2000 13:25:51 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA06287; Tue, 29 Aug 2000 10:18:14 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA29310; Tue, 29 Aug 2000 10:15:48 -0700 From: "Douglas Otis" To: , , Subject: RE: Data in SCSI Response or SCSI Data Date: Tue, 29 Aug 2000 10:27:57 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100F2A@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Clearly, the URGENT flag does even less at defining a point within the TCP stream. RFC 793 (TCP pg 12 & 13 continued.) "TCP also provides a means to communicate to the receiver of data that at some point further along in the data stream than the receiver is currently reading there is urgent data. TCP does not attempt to define what the user specifically does upon being notified of pending urgent data, but the general notion is that the receiving process will take action to process the urgent data quickly." Care should be taken not to make assumptions as to the behavior of TCP regarding frame alignment. These assumptions are being made by Julian in describing iSCSI placement of information. His being evasive does not help with assessments under such erroneous expectations. In describing iSCSI as being able to place information in specific datagrams, Julian modifies TCP. "Steph, I assume that your hardware knows when it is going to send the last SCSI datagram. In this case it can insert the status in the datagram header (not at the end) - (and BTW that is how the current draft assumes that things will be done). If you have to send sense-data (bad status) then you will send a separate datagram." Julo Aug 26, 2000 RE: Data in SCSI Response or SCSI Data Julian should clarify where in the iSCSI draft these assumptions he indicates exist. No where within the iSCSI document is datagram defined or used; should it be if this behavior is required? Perhaps there is no expectation of adhering to TCP specifications in the implementation of iSCSI and leaving out this information from iSCSI is an attempt to hide these TCP modifications. Doug > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, August 29, 2000 6:38 AM > To: dotis@sanlight.net; julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > I should point out that TCP's URGENT functionality > may be of some use here -- it won't provide the > level of support for message boundaries found > in SCTP, but might still be quite useful. --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > > > > -----Original Message----- > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > Sent: Monday, August 28, 2000 2:25 PM > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > Julo, > > > > RFC 793 (TCP pg 12) > > "A sending TCP is allowed to collect data from the sending user and to > > send that data in segments at its own convenience, until the push > > function is signaled, then it must send all unsent data. When a > > receiving TCP sees the PUSH flag, it must not wait for more data from > > the sending TCP before passing the data to the receiving process. > > > > There is no necessary relationship between push functions and segment > > boundaries. The data in any particular segment may be the result of a > > single SEND call, in whole or part, or of multiple SEND calls. > > > > The purpose of push function and the PUSH flag is to push data through > > from the sending user to the receiving user. It does not provide a > > record service. > > > > There is a coupling between the push function and the use of buffers > > of data that cross the TCP/user interface. Each time a PUSH flag is > > associated with data placed into the receiving user's buffer, the > > buffer is returned to the user for processing even if the buffer is > > not filled. If data arrives that fills the user's buffer before a > > PUSH is seen, the data is passed to the user in buffer size units." > > > > It would appear you are assuming a relationship between PUSH flags and > > segment boundaries. In addition, as iSCSI is the confluence of > > independent > > traffic, at what point would you set the PUSH flag? You suggest rather > > than > > a push function, there is a last datagram signal according to your > > comments > > below. Even with a PUSH flag, adding data to the buffer before > completion > > of push function does not provide guaranteed segment alignment nor is > > there > > any signaling for completion nor is there any intent at providing the > > alignment function you indicate. You suggest a 'last datagram' signal > > provides a simple means of positioning information at the > beginning of the > > TCP Datagram. What signal? In keeping within TCP, how do you implement > > your comments below which you state is the basis of iSCSI. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Monday, August 28, 2000 9:18 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > Doug, > > > > > > I would appreciate if instead of a general and fuzzy > referency you would > > > quote > > > the document page and paragraph. > > > > > > Julo > > > > > > "Douglas Otis" on 28/08/2000 19:00:56 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > Julo, > > > > > > Is it compliant to assume knowledge of the TCP frame by the iSCSI > > > application? In many places within iSCSI discussions, references are > > made > > > of packets, frames, and datagrams as an element within iSCSI. If > > creating > > > frame alignment is your intent, rather than corrupting TCP > APIs, perhaps > > > you > > > should consider SCTP. Otherwise, such discussions are little > more than > > a > > > wink and a nod at creating a frame aligned TCP. How do you > > > advise a change > > > to the TCP API to resolve frame alignment as you have just suggested? > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Saturday, August 26, 2000 2:28 AM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > Steph, > > > > > > > > I assume that your hardware knows when it is going to send the last > > SCSI > > > > datagram. > > > > In this case it can insert the status in the datagram header (not at > > the > > > > end) - > > > > (and BTW that is how the current draft assumes that things will > > > be done). > > > > > > > > If you have to send sense-data (bad status) then you will send > > > a separate > > > > datagram. > > > > > > > > Julo > > > > > > > > Stephen Bailey on 25/08/2000 22:12:09 > > > > > > > > Please respond to Stephen Bailey > > > > > > > > To: ips@ece.cmu.edu > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > > Is there anything preventing your hypothetical hardware > implementor > > > > > to send always good status within the last block of data? > > > > > > > > It depends upon the RDMA mechanism you are using. I admit > that I have > > > > not studied them in detail, other than ST of course, which is > > > > essentially an transport protocol based upon a particular RDMA > > > > mechanism. I will try to do so soon to determine if I'm all wet. > > > > > > > > However, assuming that the RDMA mechanism operates on a per-datagram > > > > basis, I can only imagine that you will not be able to append > > > > `general delivery' data to the end of an RDMA datagram. In > this case, > > > > you will need a separate datagram to ensure that status is delivered > > > > through a separate path from the data. > > > > > > > > So, while I think concatenating status and data in its > general form is > > > > not a good idea, a good-status fast path (like the success bit) is > > > > definitely the right way to think about it. Nonsuccess > SCSI status is > > > > so rare that any compromise you can make in the nonsuccess path to > > > > make the success path go faster is worth it. > > > > > > > > Steph > From owner-ips@ECE.cmu.edu Tue Aug 29 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07232; 29 Aug 2000 14:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA26019 for ips-outgoing; Tue, 29 Aug 2000 13:40:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from raidnet.raidpower.com (smtp.raidpower.com [38.242.39.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA26006 for ; Tue, 29 Aug 2000 13:40:43 -0400 (EDT) Received: by RAIDNET with Internet Mail Service (5.5.2650.21) id ; Tue, 29 Aug 2000 13:33:24 -0400 Message-ID: <6FC8F601FBDED111A1A800A0C92D895C0125DB2C@RAIDNET> From: "Pistor, Ernie" To: ips@ece.cmu.edu Subject: RE: iSCSI: "Wedge" drivers Date: Tue, 29 Aug 2000 13:33:24 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo (and others)- I believe David's points 1 and 4 are directly related; the Clariion array (and others that I know of) is active-active in that both controllers will accept SCSI commands, but active-passive in that each LU is explicitly owned by one contrller or the other. Commands for a LU to the non-owning controller are rejected, and AFAIK a controller has no knowledge of commands that are active (through the other controller) to LUs it does not own. Ownership of a LU can be changed to the other controller, but the method is vendor-specific and requires a wedge driver, as it must be initiated by the host system. Note also that it may be desirable to change LU ownership for reasons other than controller failure (i.e. host-side HBA failure, where you want to move a LU to the other controller so you can still access its data), so that, even if the array did support internal failover (I think Clariion does as a not-so-widely-used option) a wedge driver would be desirable. Enough vendor-specific babble... the intent was to help clarify David's points through a bit of elaboration. As for multiple connections, I cast my vote for the asymmetric approach as long as it still permits us to support one connection per session (as in Kalman Meth's proposal). As you pointed out in an earlier message, a single control connection inherently takes care of command ordering without the reference number; command processing is simplified. As far as (control) connection recovery is concerned, an alternate approach (to the one specified in Kalman Meth's proposal) that does not require that a new connection be opened immediately might be to designate the lowest-numbered non-failed connection as the control connection. There would need to be some sort of message passed to inform the other end of the new control connection by the side that first detects the broken connection, followed by the rest of Kalman's recovery scheme (Query/Sync) or some variant. The advantage to this approach is that, in the multiple connections case, it could prevent the target from cleaning up its session state upon control connection breakage; I don't believe Kalman's proposal specified the target's actions if it discovers a dead control connection, and in the spirit of the 00 draft my assumption would be cleanup (otherwise the target is left with a "zombie" session if the initiator does not log in again). This isn't an explicit proposal, but it's something that might be worth thinking about... Ernie Pistor -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Tuesday, August 29, 2000 10:54 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: "Wedge" drivers David, I understand 3 and I somehow expected it. If I would have to reimplement this under iSCSI I would choose a plug-in in iSCSI to do it (a policy module) and iSCSI would clearly support such a construct (and make it also future-proof). I have some trouble with 1 and 2. In whatever design you choose at the array level you have to have some sharing as some commands are targeted to a LU and all it's queues (i.e. separation per initiator is never complete if you take SAM seriously). However some design might choose to share only for those commands - and talking to you with you chair hat off - I don't think that you are with one of those (:- I am no sure I understand 4 either - if the array does no "sense" the fail over. Now talking to you with your hat on - we are ready to get to the drafting - but I feel that we are still left with the symmetric/asymmetric dilemma and I would appreciate input: The asymmetric solution is simpler for those using a single connection. Is there any serious drawback? Dear Colleagues - Please give us feedback. We can choose only one solution? Which is the right one? Julo Black_David@emc.com on 29/08/2000 16:26:49 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: "Wedge" drivers > While many of you keep telling me that wedge drivers won't go away nobody > gave one good example to support that assertion. With my co-chair hat off, here are four, two from the mail exchange: (1) Sharing SCSI state across array interface processors involves some work. Use of multiple SCSI connections and wedge drivers for fault tolerance and load balancing will persist in systems that choose not to follow that implementation path. (2) If iSCSI is implemented in hardware, so that a system has multiple iSCSI HBAs rather than multiple NICs with a common iSCSI software stack, multiple SCSI connections and wedge drivers are again the path of least resistance to fault tolerance and load balancing. And two new ones: (3) Naive greedy load balancing is not optimal. Significant improvements are possible based on an understanding of how the storage behaves. PowerPath contains Symmetrix-specific logic that does this, and I don't think that's appropriate for standardization. (4) Some arrays need additional support for fail-over. Clariion uses an active fail-over architecture in which the array must be instructed to fail-over, and the fail-over occurs across separate SCSI connections. I don't think that the Clariion-specific logic that does this is appropriate for standardization. As I wrote earlier, I have no problem with specifying a session abstraction for iSCSI, but it won't eliminate wedge drivers. With my co-chair hat on, the current consensus is to specify a session abstraction and review the result. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Aug 29 20:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07648; 29 Aug 2000 16:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA28176 for ips-outgoing; Tue, 29 Aug 2000 14:42:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA28172 for ; Tue, 29 Aug 2000 14:42:31 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA06334; Tue, 29 Aug 2000 11:34:55 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA29531; Tue, 29 Aug 2000 11:32:29 -0700 From: "Douglas Otis" To: , , Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 11:44:39 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, If adapters on a device basis provide access, can one assume such equipment will support ACA? If not, is it practical for the adapter to recognize this limitation and make required translations? This could be problematic in less aggressive technologies such as tape, loaders, and the like. As it has an impact on error handling, mistakes in this area could be costly. It would be nice to assume all devices and initiators support ACA and Autosense but a network transport does not by necessity exclude devices that do not. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > somesh_gupta@hp.com > Sent: Tuesday, August 29, 2000 10:17 AM > To: ENDL_TX@computer.org; ips@ece.cmu.edu > Subject: RE: iSCSI Autosense > > > If we are looking for consensus on this issue, I agree with Ralph. > Better to clean up now than implement something that is waste of > energy. > > Somesh > > > -----Original Message----- > > From: ralphoweber@compuserve.com [mailto:ralphoweber@compuserve.com] > > Sent: Monday, August 28, 2000 8:03 PM > > To: ips@ece.cmu.edu > > Subject: FW: iSCSI Autosense > > > > > > Note: Clause numbers reference draft-satran-iscsi-01.txt. > > > > There is nothing in the SCSI Architecture that requires the A > > Flags bit (byte 2 bit 7) described in 3.2 and 3.2.1. Specifically, > > the SCSI Architecture does not require a transport to have the > > ability to disable Autosense. > > > > Contingent Allegiance (the target state that follows a CHECK > > CONDITION in the absence of Autosense) is not a required SCSI > > behavior. In fact, Contingent Allegiance is an out dated > > SCSI behavior that continues to be documented in SAM for > > backwards compatibility. Contingent Allegiance is wholly > > superseded by the Autosense capability. > > > > Autosense is dramatically simpler to implement provided the > > SCSI transport protocol allows for delivery of the sense data > > in the same packet as the CHECK CONDITION status (as is the > > case in iSCSI). > > > > IMHO iSCSI would be well served by following the precedent > > set by FCP. Remove the A bit and state explicitly that all > > iSCSI devices are required to support Autosense and that > > all instances of a CHECK CONDITION status shall be accompanied > > by Autosense data. > > > > Note: These changes will also affect 3.3.7, 5.3 and possibly > > other clauses of the internet draft. > > > > Thanks. > > > > Ralph... > > > > > > > From owner-ips@ECE.cmu.edu Tue Aug 29 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07788; 29 Aug 2000 16:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA29732 for ips-outgoing; Tue, 29 Aug 2000 15:21:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA29724 for ; Tue, 29 Aug 2000 15:21:29 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 29 Aug 2000 15:20:50 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F32@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 15:20:24 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > If adapters on a device basis provide access, can one assume such equipment > will support ACA? If not, is it practical for the adapter to recognize this > limitation and make required translations? This could be problematic in > less aggressive technologies such as tape, loaders, and the like. I believe ACA should have been Autosense in the first question above. A target adapter (i.e., iSCSI target to SCSI initiator) can definitely make the translation - in response to a CHECK CONDITION from a device that does not support Autosense, it fetches the sense data, clears the condition, and returns the data as specified by Autosense. The requirements document already contains a statement that older SCSI devices may require translations of this ilk. This is much better than expecting the iSCSI initiator to come in and clean things up, since the device can't do anything else until this is done, and the iSCSI initiator may be some distance away. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Aug 29 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07978; 29 Aug 2000 17:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA01910 for ips-outgoing; Tue, 29 Aug 2000 16:12:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spectrum2.osicom.com (spectrum2.osicom.com [131.143.8.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA00692 for ; Tue, 29 Aug 2000 15:43:59 -0400 (EDT) Received: from aj.osicom.com (azure2.aj.osicom.com [131.143.44.35]) by spectrum2.osicom.com (8.9.2/8.9.2) with SMTP id PAA10779; Tue, 29 Aug 2000 15:46:40 -0400 (EDT) Received: from entradanetworks.com by aj.osicom.com (SMI-8.6/SMI-SVR4(azure2)) id PAA27092; Tue, 29 Aug 2000 15:45:00 -0400 Message-ID: <39AC135C.84883E26@entradanetworks.com> Date: Tue, 29 Aug 2000 15:47:40 -0400 From: "Robert B. Harmon" X-Mailer: Mozilla 4.74 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis CC: Black_David@emc.com, julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I have had to fix/debug TCP once, no 3 three times, soo.. TCP has 2 parts to urgent mechanism. and URG flag in header to indicate that urgent field is "significant" The URGENT pointer itself is an offset from the indicated sequence number, (the sequence number of "this" segment"). I have heard that there is generally very little use of URGENT. A better place to read about this is the TCP man page, showing its use in a sockets program. TCP provides an urgent data mechanism, which may be invoked using the out-of-band provisions of send(3N). The caller may mark one byte as "urgent" with the MSG_OOB flag to send(3N). This sets an "urgent pointer" pointing to this byte in the TCP stream. The receiver on the other side of the stream is notified of the urgent data by a SIGURG sig- nal. The SIOCATMARK ioctl() request returns a value indi- cating whether the stream is at the urgent mark. Because the system never returns data across the urgent mark in a single read(2) call, it is possible to advance to the urgent data in a simple loop which reads data, testing the socket with the SIOCATMARK ioctl() request, until it reaches the mark. The TCP Push bit cannot be used. Most if not all TCP implementations always set the push bit. Douglas Otis wrote: > > David, > > Clearly, the URGENT flag does even less at defining a point within the TCP > stream. > > RFC 793 (TCP pg 12 & 13 continued.) > > "TCP also provides a means to communicate to the receiver of data that > at some point further along in the data stream than the receiver is > currently reading there is urgent data. TCP does not attempt to > define what the user specifically does upon being notified of pending > urgent data, but the general notion is that the receiving process will > take action to process the urgent data quickly." > > Care should be taken not to make assumptions as to the behavior of TCP > regarding frame alignment. These assumptions are being made by Julian in > describing iSCSI placement of information. His being evasive does not help > with assessments under such erroneous expectations. In describing iSCSI as > being able to place information in specific datagrams, Julian modifies TCP. > > "Steph, > > I assume that your hardware knows when it is going to send the > last SCSI datagram. In this case it can insert the status in > the datagram header (not at the end) - (and BTW that is how > the current draft assumes that things will be done). > > If you have to send sense-data (bad status) then you will send > a separate datagram." > > Julo Aug 26, 2000 RE: Data in SCSI Response or SCSI Data > > Julian should clarify where in the iSCSI draft these assumptions he > indicates exist. No where within the iSCSI document is datagram defined or > used; should it be if this behavior is required? Perhaps there is no > expectation of adhering to TCP specifications in the implementation of iSCSI > and leaving out this information from iSCSI is an attempt to hide these TCP > modifications. > > Doug > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Tuesday, August 29, 2000 6:38 AM > > To: dotis@sanlight.net; julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > I should point out that TCP's URGENT functionality > > may be of some use here -- it won't provide the > > level of support for message boundaries found > > in SCTP, but might still be quite useful. --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > > black_david@emc.com Cellular: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > > > -----Original Message----- > > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > > Sent: Monday, August 28, 2000 2:25 PM > > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > Julo, > > > > > > RFC 793 (TCP pg 12) > > > "A sending TCP is allowed to collect data from the sending user and to > > > send that data in segments at its own convenience, until the push > > > function is signaled, then it must send all unsent data. When a > > > receiving TCP sees the PUSH flag, it must not wait for more data from > > > the sending TCP before passing the data to the receiving process. > > > > > > There is no necessary relationship between push functions and segment > > > boundaries. The data in any particular segment may be the result of a > > > single SEND call, in whole or part, or of multiple SEND calls. > > > > > > The purpose of push function and the PUSH flag is to push data through > > > from the sending user to the receiving user. It does not provide a > > > record service. > > > > > > There is a coupling between the push function and the use of buffers > > > of data that cross the TCP/user interface. Each time a PUSH flag is > > > associated with data placed into the receiving user's buffer, the > > > buffer is returned to the user for processing even if the buffer is > > > not filled. If data arrives that fills the user's buffer before a > > > PUSH is seen, the data is passed to the user in buffer size units." > > > > > > It would appear you are assuming a relationship between PUSH flags and > > > segment boundaries. In addition, as iSCSI is the confluence of > > > independent > > > traffic, at what point would you set the PUSH flag? You suggest rather > > > than > > > a push function, there is a last datagram signal according to your > > > comments > > > below. Even with a PUSH flag, adding data to the buffer before > > completion > > > of push function does not provide guaranteed segment alignment nor is > > > there > > > any signaling for completion nor is there any intent at providing the > > > alignment function you indicate. You suggest a 'last datagram' signal > > > provides a simple means of positioning information at the > > beginning of the > > > TCP Datagram. What signal? In keeping within TCP, how do you implement > > > your comments below which you state is the basis of iSCSI. > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Monday, August 28, 2000 9:18 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > I would appreciate if instead of a general and fuzzy > > referency you would > > > > quote > > > > the document page and paragraph. > > > > > > > > Julo > > > > > > > > "Douglas Otis" on 28/08/2000 19:00:56 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > Is it compliant to assume knowledge of the TCP frame by the iSCSI > > > > application? In many places within iSCSI discussions, references are > > > made > > > > of packets, frames, and datagrams as an element within iSCSI. If > > > creating > > > > frame alignment is your intent, rather than corrupting TCP > > APIs, perhaps > > > > you > > > > should consider SCTP. Otherwise, such discussions are little > > more than > > > a > > > > wink and a nod at creating a frame aligned TCP. How do you > > > > advise a change > > > > to the TCP API to resolve frame alignment as you have just suggested? > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Saturday, August 26, 2000 2:28 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > > > > > > Steph, > > > > > > > > > > I assume that your hardware knows when it is going to send the last > > > SCSI > > > > > datagram. > > > > > In this case it can insert the status in the datagram header (not at > > > the > > > > > end) - > > > > > (and BTW that is how the current draft assumes that things will > > > > be done). > > > > > > > > > > If you have to send sense-data (bad status) then you will send > > > > a separate > > > > > datagram. > > > > > > > > > > Julo > > > > > > > > > > Stephen Bailey on 25/08/2000 22:12:09 > > > > > > > > > > Please respond to Stephen Bailey > > > > > > > > > > To: ips@ece.cmu.edu > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > > > > > > > Is there anything preventing your hypothetical hardware > > implementor > > > > > > to send always good status within the last block of data? > > > > > > > > > > It depends upon the RDMA mechanism you are using. I admit > > that I have > > > > > not studied them in detail, other than ST of course, which is > > > > > essentially an transport protocol based upon a particular RDMA > > > > > mechanism. I will try to do so soon to determine if I'm all wet. > > > > > > > > > > However, assuming that the RDMA mechanism operates on a per-datagram > > > > > basis, I can only imagine that you will not be able to append > > > > > `general delivery' data to the end of an RDMA datagram. In > > this case, > > > > > you will need a separate datagram to ensure that status is delivered > > > > > through a separate path from the data. > > > > > > > > > > So, while I think concatenating status and data in its > > general form is > > > > > not a good idea, a good-status fast path (like the success bit) is > > > > > definitely the right way to think about it. Nonsuccess > > SCSI status is > > > > > so rare that any compromise you can make in the nonsuccess path to > > > > > make the success path go faster is worth it. > > > > > > > > > > Steph > > From owner-ips@ECE.cmu.edu Tue Aug 29 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08170; 29 Aug 2000 18:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA03735 for ips-outgoing; Tue, 29 Aug 2000 16:55:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from falcon.vixel.com (mail.vixel.com [207.115.190.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA03728 for ; Tue, 29 Aug 2000 16:55:02 -0400 (EDT) Received: from vixel.com ([192.168.1.89]) by falcon.vixel.com (Netscape Messaging Server 4.15) with ESMTP id G02NFK00.RTI; Tue, 29 Aug 2000 13:54:56 -0700 Message-ID: <39AC231E.10AD6D91@vixel.com> Date: Tue, 29 Aug 2000 13:54:54 -0700 From: "Marjorie Krueger" X-Mailer: Mozilla 4.61 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: "Randall R. Stewart" CC: ipfc@standards.gadzoox.com, ips@ece.cmu.edu Subject: Re: FC over IP: Requirements References: <200008290042.RAA19306@ha10nwk.EBay.Sun.COM> <39ABD121.8DFE0A95@lightsand.com> <39ABE28E.BFF0AE11@stewart.chicago.il.us> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "Randall R. Stewart" wrote: ..snip.. > I would think you may possibly want the unreliable transport extension > that we are currently working on.. but this does seem a item for > debate... in any event, it is always available if the WG decides it > needs it(the unreliable extension)... What does the SCTP unreliable transport option offer that UDP doesn't? -Marjorie From owner-ips@ECE.cmu.edu Tue Aug 29 23:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08418; 29 Aug 2000 18:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA05182 for ips-outgoing; Tue, 29 Aug 2000 17:36:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA05178 for ; Tue, 29 Aug 2000 17:36:00 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA06380; Tue, 29 Aug 2000 14:28:21 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA29977; Tue, 29 Aug 2000 14:25:52 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 14:38:04 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100F32@corpmx9.isus.emc.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, If you were to add Autosense emulation at the remote adapter, then should it be done with ACA emulation as well if tags are supported? (FCP drives require Autosense.) The intent of Autosense and ACA is to ensure recognition of an error condition before continuing operation. SAM provides for conditional enabling of Autosense if allowed, whereas iSCSI provides for the conditional disabling of Autosense making Autosense a required feature. This would also seem to imply ACA would also be a required feature. An effort to achieve for a device which does not implement ACA. By requiring Autosense on all devices, does this create a difficult situation? Keeping the SAM view of Autosense allows a simple means of handling these cases. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > Sent: Tuesday, August 29, 2000 12:20 PM > To: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense > > > > If adapters on a device basis provide access, can one assume such > equipment > > will support ACA? If not, is it practical for the adapter to recognize > this > > limitation and make required translations? This could be problematic in > > less aggressive technologies such as tape, loaders, and the like. > > I believe ACA should have been Autosense in the first question above. > > A target adapter (i.e., iSCSI target to SCSI initiator) can > definitely make > the translation - in response to a CHECK CONDITION from a device that > does not support Autosense, it fetches the sense data, clears the > condition, > and returns the data as specified by Autosense. The requirements > document already contains a statement that older SCSI devices > may require translations of this ilk. This is much better than > expecting the iSCSI initiator to come in and clean things up, > since the device can't do anything else until this is done, and the > iSCSI initiator may be some distance away. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Tue Aug 29 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08434; 29 Aug 2000 19:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA06859 for ips-outgoing; Tue, 29 Aug 2000 18:24:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA06855 for ; Tue, 29 Aug 2000 18:23:58 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id RAA01718; Tue, 29 Aug 2000 17:24:17 -0500 Message-ID: <39AC3811.ABAC273F@stewart.chicago.il.us> Date: Tue, 29 Aug 2000 17:24:17 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Marjorie Krueger CC: ipfc@standards.gadzoox.com, ips@ece.cmu.edu Subject: Re: FC over IP: Requirements References: <200008290042.RAA19306@ha10nwk.EBay.Sun.COM> <39ABD121.8DFE0A95@lightsand.com> <39ABE28E.BFF0AE11@stewart.chicago.il.us> <39AC231E.10AD6D91@vixel.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Marjorie Krueger wrote: > > "Randall R. Stewart" wrote: > ..snip.. > > > I would think you may possibly want the unreliable transport extension > > that we are currently working on.. but this does seem a item for > > debate... in any event, it is always available if the WG decides it > > needs it(the unreliable extension)... > > What does the SCTP unreliable transport option offer that UDP doesn't? > > -Marjorie Something that is stated that you are required to have.. congestion control AND as a added benefit multi-homing. R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Tue Aug 29 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08443; 29 Aug 2000 19:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA06998 for ips-outgoing; Tue, 29 Aug 2000 18:29:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA06993 for ; Tue, 29 Aug 2000 18:29:32 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id PAA06401; Tue, 29 Aug 2000 15:21:59 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA30123; Tue, 29 Aug 2000 15:19:32 -0700 From: "Douglas Otis" To: "Marjorie Krueger" , "Randall R. Stewart" Cc: , Subject: RE: FC over IP: Requirements Date: Tue, 29 Aug 2000 15:31:45 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <39AC231E.10AD6D91@vixel.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Marjorie, Congestion avoidance and protocol level fragmentation to name but a few of the many features required for robust use on IP would still be offered by this SCTP extension. Although possible after man years of work on new UDP or raw datagram protocols, SCTP provides a uniform means to obtain common features desired in object (messaging) protocols of which FC is but one. The removal of retransmit limits dispersion in time of such messages but places the burden on a less effective retry mechanism unaware of the packet loss until extensive timeouts. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Marjorie Krueger > Sent: Tuesday, August 29, 2000 1:55 PM > To: Randall R. Stewart > Cc: ipfc@standards.gadzoox.com; ips@ece.cmu.edu > Subject: Re: FC over IP: Requirements > > > "Randall R. Stewart" wrote: > ..snip.. > > > I would think you may possibly want the unreliable transport extension > > that we are currently working on.. but this does seem a item for > > debate... in any event, it is always available if the WG decides it > > needs it(the unreliable extension)... > > What does the SCTP unreliable transport option offer that UDP doesn't? > > -Marjorie > From owner-ips@ECE.cmu.edu Tue Aug 29 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08488; 29 Aug 2000 19:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA06433 for ips-outgoing; Tue, 29 Aug 2000 18:11:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA06426 for ; Tue, 29 Aug 2000 18:11:51 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 29 Aug 2000 18:11:19 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F33@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 18:11:19 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > If you were to add Autosense emulation at the remote adapter, then should it > be done with ACA emulation as well if tags are supported? (FCP drives > require Autosense.) I believe the proposal is to require Autosense for iSCSI and not support ACA. Autosense can be emulated on top of ACA in the remote adapter as outlined in previous mail - this has no effect on what has to be specified for iSCSI. FCP drives do not pose a problem as Autosense would be used on both sides of the remote adapter. > SAM provides for conditional enabling of Autosense if allowed, whereas iSCSI > provides for the conditional disabling of Autosense making Autosense a > required feature. This would also seem to imply ACA would also be a > required feature. That is not the case - please reread Ralph Weber's email on this topic. If FCP can forbid ACA, so can iSCSI. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Aug 30 01:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09020; 29 Aug 2000 21:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA12376 for ips-outgoing; Tue, 29 Aug 2000 19:57:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA12369 for ; Tue, 29 Aug 2000 19:57:25 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA06436; Tue, 29 Aug 2000 16:49:35 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA30359; Tue, 29 Aug 2000 16:47:09 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 16:59:22 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100F33@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I think you are confusing ACA (Auto Contingent Allegiance) with CA (Contingent Allegiance). My concern was implementing ACA on a device that only implemented CA and then forced to implement Autosense. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > Sent: Tuesday, August 29, 2000 3:11 PM > To: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense > > > > If you were to add Autosense emulation at the remote adapter, > then should > it > > be done with ACA emulation as well if tags are supported? (FCP drives > > require Autosense.) > > I believe the proposal is to require Autosense for iSCSI and not support > ACA. > Autosense can be emulated on top of ACA in the remote adapter as outlined > in previous mail - this has no effect on what has to be specified > for iSCSI. > FCP drives do not pose a problem as Autosense would be used on both > sides of the remote adapter. > > > SAM provides for conditional enabling of Autosense if allowed, whereas > iSCSI > > provides for the conditional disabling of Autosense making Autosense a > > required feature. This would also seem to imply ACA would also be a > > required feature. > > That is not the case - please reread Ralph Weber's email on this > topic. If > FCP can forbid ACA, so can iSCSI. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Wed Aug 30 01:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09018; 29 Aug 2000 21:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA12178 for ips-outgoing; Tue, 29 Aug 2000 19:50:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA12173 for ; Tue, 29 Aug 2000 19:50:06 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA06428 for ; Tue, 29 Aug 2000 16:42:19 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA30335 for ; Tue, 29 Aug 2000 16:39:51 -0700 From: "Douglas Otis" To: "Ips" Subject: (FCoverIP) SCTP extension. Date: Tue, 29 Aug 2000 16:52:03 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk As a possible extension to SCTP to deal with the limitation of FC, perhaps a trinary setting of the retransmission could be made: Unlimited, 1, or 0. By allowing at least one retry, most lost frames would be recovered within a constrained window of time. The danger with FC is an underlying transport overlap of recovery mechanisms. One other consideration is the size of the buffers at each end of SCTP as this would also skew a time window but would be required to prevent loss at this point as well. BTW, what is the latest T11 DMP specification? Doug From owner-ips@ECE.cmu.edu Wed Aug 30 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09117; 29 Aug 2000 21:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA13789 for ips-outgoing; Tue, 29 Aug 2000 20:59:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA13783 for ; Tue, 29 Aug 2000 20:59:45 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G02008ARYMO9R@mta5.snfc21.pbi.net> for IPS@ECE.CMU.edu; Tue, 29 Aug 2000 17:56:48 -0700 (PDT) Date: Tue, 29 Aug 2000 17:56:25 -0700 From: Peter Johansson Subject: RE: iSCSI Autosense In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000829175240.00c00930@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed References: Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:44 AM 8/29/00, Douglas Otis wrote: >It would be nice to assume all devices and initiators support ACA and >Autosense but a network transport does not by necessity exclude devices >that do not. Doug, would you clarify for me please? Do you mean to say: a) A network transport MUST not exclude certain devices; b) A network transport SHOULD not exclude certain devices; or c) Something else. I think it is perfectly reasonable for iSCSI to profile the SAM features required of devices that wish to use iSCSI---and eminently reasonable to require such devices to support autosense! Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Wed Aug 30 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09214; 29 Aug 2000 21:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA13445 for ips-outgoing; Tue, 29 Aug 2000 20:45:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA13440 for ; Tue, 29 Aug 2000 20:45:33 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 29 Aug 2000 20:45:00 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F38@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 20:44:58 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I think you are confusing ACA (Auto Contingent Allegiance) with CA > (Contingent Allegiance). My concern was implementing ACA on a device that > only implemented CA and then forced to implement Autosense. It does look that way, oops :-). Pulling foot out of mouth and trying again ... The proposal is to require Autosense to match FCP, which would then require ACA. If the SCSI device on the far side of the remote adapter only implements CA, then there's additional work to implement ACA in the remote adapter. As noted earlier, the requirements draft already anticipates that conversion like this will be needed for SCSI-2 devices, and I read Ralph's mail as encouragement from T10 to mandate Autosense. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Aug 30 03:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09476; 29 Aug 2000 23:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA14668 for ips-outgoing; Tue, 29 Aug 2000 21:39:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA14664 for ; Tue, 29 Aug 2000 21:38:57 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id VAA33230 for ; Tue, 29 Aug 2000 21:36:46 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id TAA167764; Tue, 29 Aug 2000 19:38:38 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: symmetric/asymmetric & "Wedge" drivers To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 29 Aug 2000 18:37:58 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/29/2000 07:38:38 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, As I understand the Asymmetric proposal, it is an extension to the Single Connection per Session (SC/S). The extension to SC/S permits data to be sent on other TCP/IP connections (which may or may not be on other NICs/HBAs). This means, I think, that if the Asymmetric proposal is accepted, that the whole discussion we had over whether or not we need the Wedge Drivers to perform "Alternate Path" retry, is moot. Wedge Drivers, I believe, will be needed to do that function if Asymmetric is chosen, since we have no construct, other then Session, to identify "Alternate Retry Paths". And in this case, the Session only identifies other Data Channels, not other Alternate Command Paths, leaving no way to know what the Alternate Retry Path might be. Is my analyses correct? . . . John L. Hufferd julian_satran@il.ibm.com@ece.cmu.edu on 08/29/2000 07:53:58 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: iSCSI: "Wedge" drivers David, I understand 3 and I somehow expected it. If I would have to reimplement this under iSCSI I would choose a plug-in in iSCSI to do it (a policy module) and iSCSI would clearly support such a construct (and make it also future-proof). I have some trouble with 1 and 2. In whatever design you choose at the array level you have to have some sharing as some commands are targeted to a LU and all it's queues (i.e. separation per initiator is never complete if you take SAM seriously). However some design might choose to share only for those commands - and talking to you with you chair hat off - I don't think that you are with one of those (:- I am no sure I understand 4 either - if the array does no "sense" the fail over. Now talking to you with your hat on - we are ready to get to the drafting - but I feel that we are still left with the symmetric/asymmetric dilemma and I would appreciate input: The asymmetric solution is simpler for those using a single connection. Is there any serious drawback? Dear Colleagues - Please give us feedback. We can choose only one solution? Which is the right one? Julo Black_David@emc.com on 29/08/2000 16:26:49 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: "Wedge" drivers > While many of you keep telling me that wedge drivers won't go away nobody > gave one good example to support that assertion. With my co-chair hat off, here are four, two from the mail exchange: (1) Sharing SCSI state across array interface processors involves some work. Use of multiple SCSI connections and wedge drivers for fault tolerance and load balancing will persist in systems that choose not to follow that implementation path. (2) If iSCSI is implemented in hardware, so that a system has multiple iSCSI HBAs rather than multiple NICs with a common iSCSI software stack, multiple SCSI connections and wedge drivers are again the path of least resistance to fault tolerance and load balancing. And two new ones: (3) Naive greedy load balancing is not optimal. Significant improvements are possible based on an understanding of how the storage behaves. PowerPath contains Symmetrix-specific logic that does this, and I don't think that's appropriate for standardization. (4) Some arrays need additional support for fail-over. Clariion uses an active fail-over architecture in which the array must be instructed to fail-over, and the fail-over occurs across separate SCSI connections. I don't think that the Clariion-specific logic that does this is appropriate for standardization. As I wrote earlier, I have no problem with specifying a session abstraction for iSCSI, but it won't eliminate wedge drivers. With my co-chair hat on, the current consensus is to specify a session abstraction and review the result. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Aug 30 03:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09481; 29 Aug 2000 23:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA15176 for ips-outgoing; Tue, 29 Aug 2000 21:59:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA15169 for ; Tue, 29 Aug 2000 21:59:08 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id RGJL009R; Tue, 29 Aug 2000 18:56:54 -0700 From: "Y P Cheng" To: , Subject: RE: iSCSI: "Wedge" drivers Date: Tue, 29 Aug 2000 18:58:45 -0700 Message-ID: <004301c01225$d49e2dc0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo Wrote: >I understand 3 and I somehow expected it. If I would have to >reimplement this under iSCSI I would choose a plug-in in >iSCSI to do it (a policy module) and iSCSI would clearly >support such a construct (and make it also future-proof). >I have some trouble with 1 and 2. In whatever design you >choose at the array level you have to have some sharing as >some commands are targeted to a LU and all it's queues >(i.e. separation per initiator is never complete if you >take SAM seriously). However some design might choose to >share only for those commands - and talking to you >with you chair hat off - I don't think that you are with >one of those (:- I am no sure I understand 4 either - if >the array does no "sense" the fail over. I have been reading the "Wedge Drivers" and their "load balance and fail over recovery" with great interest. I am not familiar with the mainframe I/O design. However, I worked on SCSI and fibre channel host bus adapters (HBA's) for long time. I would like apply those discussions on the adapters that I know of. The understanding of the HBA's would greatly simplify the discussions of wedge driver and its load balance and recovery. The SCSI and fibre channel HBA's today executes a SCSI command atomically without any intervention from the device driver. When there are multiple HBAs on one server, they don't share command execution states. Each HBA executes their own SCSI commands atomically. The HBA and its driver perform discovery of SCSI or FCP devices at power up. It numbers the devices sequentially. A table is created to map the device number to a SCSI ID or a fibre channel 24-bit address. When there are multiple HBA's, it is the responsibility of the Upper Level Drivers to direct the SCSI commands through their appropriate HBA's. The HBA itself can care less. For a iSCSI HBA, there won't be any discovery at powerup. Instead, the HBA will monitor the ARP and FARP requests and responses to create a IP-to-FC-address translation table. (This discussion does not exclude an Ethernet NIC card.) The HBA relies on the FCP/SCTP driver to create endpoints and associates for connection oriented exchanges. The iSCSI driver running above the FCP/SCTP driver and below the SCSI-class driver will be responsible for SCSI-over-IP messages. In fact the iSCSI HBA is no different from an NIC with the exception of the FCP/RDMA support. When there are multiple iSCSI HBA's, The FCP/SCTP driver will be responsible to determine if an IP device can be reached from a different HBA. (Whether an IP device can be reached by another IP address is beyond the scope of this discussion.) We may choose to include a subset of the SCTP function in the iSCSI driver if we don't wait for the SCTP implementation. In the discussion below we refer the iSCSI driver with a subset of the SCTP function. Knowing there is an alternative HBA to reach the same IP device, it is certainly the discretion of the iSCSI driver to sent IP-SCSI messages to the HBA of its choice. A response must come back to the same HBA because it is difficult for two HBA's to share execution states. For example, if we set up one HBA to do the RDMA function, it is not nice to have the data coming back to the other HBA. Both fail over and load balance can be done in iSCSI driver relatively easily because it is fully aware of the availability of alternative HBA's reaching the same IP device. In theory, we could break up the iSCSI driver by moving the SCSI-to-IP-packet conversion function to the HBA and by keeping all the remaining functions. If we do so, the SCSI command will be executed atomically inside an HBA. From owner-ips@ECE.cmu.edu Wed Aug 30 04:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09977; 30 Aug 2000 0:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16632 for ips-outgoing; Tue, 29 Aug 2000 23:05:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA16627 for ; Tue, 29 Aug 2000 23:05:08 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id UAA03091; Tue, 29 Aug 2000 20:05:04 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Tue, 29 Aug 2000 20:05:04 -0700 Message-ID: From: Jim McGrath To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: what's in a name Date: Tue, 29 Aug 2000 20:05:02 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk You file iSCSI as a Service mark (SM). That essentially reserves the name so no one else can trademark it, but allows other to use it (if you TM the name then you have to defend it if others use it or you lose the TM - the Xerox problem). I recollect that we did this with Ultra SCSI and it worked ok. Jim -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Tuesday, August 29, 2000 7:31 AM To: ips@ece.cmu.edu Subject: what's in a name Yaron Klein just pointed out to me tha Virtual Private SAN and VPS are TM of Pathlight (www.pathlight.com) and so are SAN Gateway and SAN router. Costa earlier pointed out that iSCSI is the name of a cable for iMAC (but not a trademark as far as I can tell). Can somebody tell us how to protect at least iSCSI from TM claims? Can the Internet Society do it (before it is too late)? Julo From owner-ips@ECE.cmu.edu Wed Aug 30 04:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09975; 30 Aug 2000 0:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA16771 for ips-outgoing; Tue, 29 Aug 2000 23:11:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sphmraaa.compuserve.com (hs-img-rel-1.compuserve.com [149.174.177.156]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA16765 for ; Tue, 29 Aug 2000 23:11:08 -0400 (EDT) Received: (from mailgate@localhost) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id XAA15200 for ips@ece.cmu.edu; Tue, 29 Aug 2000 23:10:38 -0400 (EDT) Received: from compuserve.com (dal-qbu-zop-vty41.as.wcom.net [216.192.247.41]) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id XAA15162 for ; Tue, 29 Aug 2000 23:10:25 -0400 (EDT) Message-ID: <39AC7BB5.6A32330F@compuserve.com> Date: Tue, 29 Aug 2000 22:12:53 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI Autosense References: <39AB27D0.9B5EF65F@compuserve.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk This is going to be fun, trying to respond to all the information and misinformation that's been bandied about today. David Black's note is correct. The method for bridging a CA-only (Contingent Allegiance only) device to an Autosense representation of that device is well known and trivial. In addition to being documented in the requirements, the method is documented in the first paragraph of 5.3 in draft-satran-iscsi-01.txt. I have two minor complaints about the 5.3 wording. 1) The method will work all the time, so the suggestion that it 'may' work is incorrect. 2) I would prefer that the REQUEST SENSE command be explicitly mentioned not alluded to. As regards CA, let me restate my position more emphatically. CA is a historical artifact that dates to the single byte of status model employed by the parallel SCSI bus. To the best of my knowledge it is extremely difficult to implement the CA model in a packetized transfer model because the CA condition is cleared by the next command to arrive at the target regardless of what kind of command that is. Since most packetized protocols allow several commands to be in flight concurrently between initiator and target, the most probably case in a packetized protocol is that an in flight command will clear a CA condition long before the initiator finds out about it and has a chance to fire off the needed REQUEST SENSE in to get the sense data. The bottom line here is that Autosense works for packetized protocols and CA does not. Since it is trivial to translate a packetized Autosense front end to the CA behavior of an older backend SCSI device, the right thing for iSCSI to do is to make Autosense mandatory and not to provide any way whatsoever to disable it. The fact that SAM is written with a preference for CA is (like CA itself) a historical artifact. So long as the wording continues to be correct, it is unlikely that T10 will change the wording (if it ain't broke don't fix it). However, the ancient leanings of SAM should not under any circumstances be used as a justification for iSCSI support for a CA-like interface. As noted above, CA doesn't work for packetized protocols like iSCSI. ACA (Auto Contingent Allegiance) is absolutely positively NOT the super-duper, all fixed up for a packetized world brother of CA. ACA is orthogonal to Autosense and actually has no place in this discussion. IN ALL CASES, ACA is optional. Simply return 0 in the NormACA bit (bit 5 byte 3) of the Standard INQUIRY Data and you don't support ACA and you are expected to refuse to process any CDB with the NACA bit set to 1. ACA is more akin to the SCSI-2 ECA (Extended Contingent Allegiance) than it is to either CA or Autosense. The assumption in ACA and ECA is that the initiator needs to send several commands to the target in order to cleanup whatever error occurred. The most frequently cited excuse for this is the cleanup needed when a tape writes past the EOT (End Of Tape) reflective strip. Thanks. Ralph Weber ENDL Texas From owner-ips@ECE.cmu.edu Wed Aug 30 04:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09936; 30 Aug 2000 0:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA17643 for ips-outgoing; Tue, 29 Aug 2000 23:48:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA17631 for ; Tue, 29 Aug 2000 23:47:48 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id UAA06488; Tue, 29 Aug 2000 20:40:12 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id UAA30921; Tue, 29 Aug 2000 20:37:46 -0700 From: "Douglas Otis" To: "Peter Johansson" , "IP Storage" Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 20:50:00 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <4.3.2.7.2.20000829175240.00c00930@PacBell.net> Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Peter Johansson > Sent: Tuesday, August 29, 2000 5:56 PM > To: IP Storage > Subject: RE: iSCSI Autosense > > > At 11:44 AM 8/29/00, Douglas Otis wrote: > > >It would be nice to assume all devices and initiators support ACA and > >Autosense but a network transport does not by necessity exclude devices > >that do not. > > Doug, would you clarify for me please? > > Do you mean to say: > > a) A network transport MUST not exclude certain devices; > b) A network transport SHOULD not exclude certain devices; or > c) Something else. SAM provides a means of enabling Autosense that can be refused. iSCSI instead defaults with Autosense. Autosense is problematic on devices that only support CA. It would be nice if everything looked the same, but is it safe to require a kludge emulating ACA in this case when simply disabling Autosense provides a cleaner solution? Yes, I know FC requires Autosense, but is iSCSI limited to just newer devices? Doug > I think it is perfectly reasonable for iSCSI to profile the SAM features > required of devices that wish to use iSCSI---and eminently reasonable to > require such devices to support autosense! > > Regards, > > Peter Johansson > > Congruent Software, Inc. > 98 Colorado Avenue > Berkeley, CA 94707 > > (510) 527-3926 > (510) 527-3856 FAX > > PJohansson@ACM.org > From owner-ips@ECE.cmu.edu Wed Aug 30 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10001; 30 Aug 2000 0:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA17880 for ips-outgoing; Tue, 29 Aug 2000 23:58:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA17876 for ; Tue, 29 Aug 2000 23:58:54 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G03008J56YTPT@mta6.snfc21.pbi.net> for IPS@ece.cmu.edu; Tue, 29 Aug 2000 20:56:54 -0700 (PDT) Date: Tue, 29 Aug 2000 20:56:50 -0700 From: Peter Johansson Subject: RE: iSCSI Autosense In-reply-to: X-Sender: Celeborn@PacBell.net To: Douglas Otis Cc: IP Storage Message-id: <4.3.2.7.2.20000829205139.00bb4e30@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed References: <4.3.2.7.2.20000829175240.00c00930@PacBell.net> Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:50 PM 8/29/00, Douglas Otis wrote: >It would be nice if everything looked the same, but is it safe to require >a kludge emulating ACA in this case when simply disabling >Autosense provides a cleaner solution? In my opinion, mandating autosense in an environment such as iSCSI (...or FCP ... or SBP-2) provides the cleanest solution. >Yes, I know FC requires Autosense, but is iSCSI limited to just newer devices? The greater "you" (designers of iSCSI) are free to make of it whatever seems advisable. There is nothing forbidden in a SAM-compliant transport, whether FCP, iSCSI or SBP-2, from requiring autosense capabilities in the endpoints. As you can tell, I think it would be a good design choice to limit iSCSI to these devices. I also think you would not find it limiting, but that's a different thread. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Wed Aug 30 06:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10309; 30 Aug 2000 1:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA18788 for ips-outgoing; Wed, 30 Aug 2000 00:40:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA18772 for ; Wed, 30 Aug 2000 00:40:41 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e7U4eeo13517 for ; Tue, 29 Aug 2000 23:40:40 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA11502; Tue, 29 Aug 00 23:38:23 -0500 Message-Id: <10008300438.AA11502@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense In-Reply-To: Message from Ralph Weber of "Mon, 28 Aug 2000 22:02:40 CDT." <39AB27D0.9B5EF65F@compuserve.com> Date: Tue, 29 Aug 2000 23:39:03 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > IMHO iSCSI would be well served by following the precedent > set by FCP. Remove the A bit and state explicitly that all > iSCSI devices are required to support Autosense and that > all instances of a CHECK CONDITION status shall be accompanied > by Autosense data. What he said! SST did this too. Steph From owner-ips@ECE.cmu.edu Wed Aug 30 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10693; 30 Aug 2000 2:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA20244 for ips-outgoing; Wed, 30 Aug 2000 02:05:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA20240 for ; Wed, 30 Aug 2000 02:05:01 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id BAA62002; Wed, 30 Aug 2000 01:52:58 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id AAA93122; Wed, 30 Aug 2000 00:04:58 -0600 Importance: Normal Subject: RE: iSCSI: "Wedge" drivers To: "Y P Cheng" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 29 Aug 2000 22:26:10 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 08/30/2000 12:04:57 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Y.P. I thought the first part of your note was useful, but then you went to something about an FCP/SCTP and creating an IP-to-FC-address from a translation table, etc. At that point, you diverted from the iSCSI draft and went where I do not believe many of us were headed. The current draft assumes we are using TCP/IP not SCTP and has no additional layer of FCP. . . . John L. Hufferd "Y P Cheng" @ece.cmu.edu on 08/29/2000 06:58:45 PM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, cc: Subject: RE: iSCSI: "Wedge" drivers Julo Wrote: >I understand 3 and I somehow expected it. If I would have to >reimplement this under iSCSI I would choose a plug-in in >iSCSI to do it (a policy module) and iSCSI would clearly >support such a construct (and make it also future-proof). >I have some trouble with 1 and 2. In whatever design you >choose at the array level you have to have some sharing as >some commands are targeted to a LU and all it's queues >(i.e. separation per initiator is never complete if you >take SAM seriously). However some design might choose to >share only for those commands - and talking to you >with you chair hat off - I don't think that you are with >one of those (:- I am no sure I understand 4 either - if >the array does no "sense" the fail over. I have been reading the "Wedge Drivers" and their "load balance and fail over recovery" with great interest. I am not familiar with the mainframe I/O design. However, I worked on SCSI and fibre channel host bus adapters (HBA's) for long time. I would like apply those discussions on the adapters that I know of. The understanding of the HBA's would greatly simplify the discussions of wedge driver and its load balance and recovery. The SCSI and fibre channel HBA's today executes a SCSI command atomically without any intervention from the device driver. When there are multiple HBAs on one server, they don't share command execution states. Each HBA executes their own SCSI commands atomically. The HBA and its driver perform discovery of SCSI or FCP devices at power up. It numbers the devices sequentially. A table is created to map the device number to a SCSI ID or a fibre channel 24-bit address. When there are multiple HBA's, it is the responsibility of the Upper Level Drivers to direct the SCSI commands through their appropriate HBA's. The HBA itself can care less. For a iSCSI HBA, there won't be any discovery at powerup. Instead, the HBA will monitor the ARP and FARP requests and responses to create a IP-to-FC-address translation table. (This discussion does not exclude an Ethernet NIC card.) The HBA relies on the FCP/SCTP driver to create endpoints and associates for connection oriented exchanges. The iSCSI driver running above the FCP/SCTP driver and below the SCSI-class driver will be responsible for SCSI-over-IP messages. In fact the iSCSI HBA is no different from an NIC with the exception of the FCP/RDMA support. When there are multiple iSCSI HBA's, The FCP/SCTP driver will be responsible to determine if an IP device can be reached from a different HBA. (Whether an IP device can be reached by another IP address is beyond the scope of this discussion.) We may choose to include a subset of the SCTP function in the iSCSI driver if we don't wait for the SCTP implementation. In the discussion below we refer the iSCSI driver with a subset of the SCTP function. Knowing there is an alternative HBA to reach the same IP device, it is certainly the discretion of the iSCSI driver to sent IP-SCSI messages to the HBA of its choice. A response must come back to the same HBA because it is difficult for two HBA's to share execution states. For example, if we set up one HBA to do the RDMA function, it is not nice to have the data coming back to the other HBA. Both fail over and load balance can be done in iSCSI driver relatively easily because it is fully aware of the availability of alternative HBA's reaching the same IP device. In theory, we could break up the iSCSI driver by moving the SCSI-to-IP-packet conversion function to the HBA and by keeping all the remaining functions. If we do so, the SCSI command will be executed atomically inside an HBA. From owner-ips@ECE.cmu.edu Wed Aug 30 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10698; 30 Aug 2000 2:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA19889 for ips-outgoing; Wed, 30 Aug 2000 01:45:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA19884 for ; Wed, 30 Aug 2000 01:45:15 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id WAA06524; Tue, 29 Aug 2000 22:37:31 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id WAA31220; Tue, 29 Aug 2000 22:35:05 -0700 From: "Douglas Otis" To: , "IPS Reflector" Subject: RE: iSCSI Autosense Date: Tue, 29 Aug 2000 22:47:21 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <39AC7BB5.6A32330F@compuserve.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ralph Weber > Sent: Tuesday, August 29, 2000 8:13 PM > To: IPS Reflector > Subject: Re: iSCSI Autosense > > > This is going to be fun, trying to respond to all the information > and misinformation that's been bandied about today. > > David Black's note is correct. The method for bridging a CA-only > (Contingent Allegiance only) device to an Autosense representation > of that device is well known and trivial. In addition to being > documented in the requirements, the method is documented in the > first paragraph of 5.3 in draft-satran-iscsi-01.txt. > > I have two minor complaints about the 5.3 wording. > > 1) The method will work all the time, so the suggestion > that it 'may' work is incorrect. > 2) I would prefer that the REQUEST SENSE command be explicitly > mentioned not alluded to. > > As regards CA, let me restate my position more emphatically. > CA is a historical artifact that dates to the single byte of > status model employed by the parallel SCSI bus. > > To the best of my knowledge it is extremely difficult to > implement the CA model in a packetized transfer model because > the CA condition is cleared by the next command to arrive > at the target regardless of what kind of command that is. Not true. Only for the nexus is this true. > Since most packetized protocols allow several commands to be > in flight concurrently between initiator and target, the most > probably case in a packetized protocol is that an in flight > command will clear a CA condition long before the initiator > finds out about it and has a chance to fire off the needed > REQUEST SENSE in to get the sense data. These in flight commands are a different nexus such that Sense data is not lost. If the device can not maintain separate Sense information BUSY is returned. > The bottom line here is that Autosense works for packetized > protocols and CA does not. Since it is trivial to translate > a packetized Autosense front end to the CA behavior of an > older backend SCSI device, the right thing for iSCSI to do > is to make Autosense mandatory and not to provide any way > whatsoever to disable it. iSCSI does provide a means to disable it, but not to refuse it. > The fact that SAM is written with a preference for CA is > (like CA itself) a historical artifact. So long as the > wording continues to be correct, it is unlikely that T10 > will change the wording (if it ain't broke don't fix it). > > However, the ancient leanings of SAM should not under any > circumstances be used as a justification for iSCSI support > for a CA-like interface. As noted above, CA doesn't work > for packetized protocols like iSCSI. A mistaken conclusion. > ACA (Auto Contingent Allegiance) is absolutely positively > NOT the super-duper, all fixed up for a packetized world > brother of CA. ACA is orthogonal to Autosense and actually > has no place in this discussion. Odd, but error control and reporting does not seem orthogonal. > IN ALL CASES, ACA is optional. Simply return 0 in the > NormACA bit (bit 5 byte 3) of the Standard INQUIRY Data > and you don't support ACA and you are expected to refuse > to process any CDB with the NACA bit set to 1. For a means of get things to stop, some mechanism should exist. This becomes difficult by suggesting refusal of Autosense is a bad choice. ACA may not exist for this purpose if you allow older devices. I suggest that there be a means to refuse Autosense. If the driver can not support this option, it can refuse the device. Doug > ACA is more akin to the SCSI-2 ECA (Extended Contingent > Allegiance) than it is to either CA or Autosense. The > assumption in ACA and ECA is that the initiator needs to > send several commands to the target in order to cleanup > whatever error occurred. The most frequently cited > excuse for this is the cleanup needed when a tape writes > past the EOT (End Of Tape) reflective strip. > > Thanks. > > Ralph Weber > ENDL Texas > > > From owner-ips@ECE.cmu.edu Wed Aug 30 07:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10869; 30 Aug 2000 3:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA20646 for ips-outgoing; Wed, 30 Aug 2000 02:34:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA20641 for ; Wed, 30 Aug 2000 02:33:59 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA29544 for ; Wed, 30 Aug 2000 08:33:18 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA32988 for ; Wed, 30 Aug 2000 08:33:17 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694B.00240020 ; Wed, 30 Aug 2000 08:33:13 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 30 Aug 2000 09:31:13 +0300 Subject: RE: iSCSI Autosense Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, I have on objection on making Autosense mandatory and removing the bit from iSCSI. My issue (if it is an issue) is SCSI related. How are asynchronous events that are not error related handled without Contingent Allegiance - and what effect will it have on iSCSI. The case that was raised a while ago (by Costa I think) was that of a volume change - with the message reporting it "crossing" a write on the wire or in the OS. With CA the state synchronization can be easily enforced (command are rejected until a sense is received). Without it - even with autosense - state synchronization in the OS becomes tricky. Perhaps somebody on the list active also in T10 can give us a hint of how this type of sequence will be handled without CA. I understand that CA can be kept even with autosense and we can remove the bit but what if T10 decides to keep both? Regards, Julo somesh_gupta@hp.com on 29/08/2000 20:17:02 Please respond to somesh_gupta@hp.com To: ENDL_TX@computer.org, ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: iSCSI Autosense If we are looking for consensus on this issue, I agree with Ralph. Better to clean up now than implement something that is waste of energy. Somesh > -----Original Message----- > From: ralphoweber@compuserve.com [mailto:ralphoweber@compuserve.com] > Sent: Monday, August 28, 2000 8:03 PM > To: ips@ece.cmu.edu > Subject: FW: iSCSI Autosense > > > Note: Clause numbers reference draft-satran-iscsi-01.txt. > > There is nothing in the SCSI Architecture that requires the A > Flags bit (byte 2 bit 7) described in 3.2 and 3.2.1. Specifically, > the SCSI Architecture does not require a transport to have the > ability to disable Autosense. > > Contingent Allegiance (the target state that follows a CHECK > CONDITION in the absence of Autosense) is not a required SCSI > behavior. In fact, Contingent Allegiance is an out dated > SCSI behavior that continues to be documented in SAM for > backwards compatibility. Contingent Allegiance is wholly > superseded by the Autosense capability. > > Autosense is dramatically simpler to implement provided the > SCSI transport protocol allows for delivery of the sense data > in the same packet as the CHECK CONDITION status (as is the > case in iSCSI). > > IMHO iSCSI would be well served by following the precedent > set by FCP. Remove the A bit and state explicitly that all > iSCSI devices are required to support Autosense and that > all instances of a CHECK CONDITION status shall be accompanied > by Autosense data. > > Note: These changes will also affect 3.3.7, 5.3 and possibly > other clauses of the internet draft. > > Thanks. > > Ralph... > > > From owner-ips@ECE.cmu.edu Wed Aug 30 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11790; 30 Aug 2000 6:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA23118 for ips-outgoing; Wed, 30 Aug 2000 05:35:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA23112 for ; Wed, 30 Aug 2000 05:34:59 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA279052 for ; Wed, 30 Aug 2000 11:34:25 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id LAA51648 for ; Wed, 30 Aug 2000 11:34:25 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694B.0034959C ; Wed, 30 Aug 2000 11:34:21 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 30 Aug 2000 12:32:19 +0300 Subject: RE: iSCSI: "Wedge" drivers Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Earnie, Thanks - I see your point. There are a multitude of controllers that virtualize a set of controllers into a "super-unit" and could have the same issues. But the wedge driver is in this case left only with the clustering issues and will not have to do anything with transport? And thanks for your thoughts on the asymmetric/symmetric issue. Julo "Pistor, Ernie" on 29/08/2000 20:33:24 Please respond to "Pistor, Ernie" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: iSCSI: "Wedge" drivers Julo (and others)- I believe David's points 1 and 4 are directly related; the Clariion array (and others that I know of) is active-active in that both controllers will accept SCSI commands, but active-passive in that each LU is explicitly owned by one contrller or the other. Commands for a LU to the non-owning controller are rejected, and AFAIK a controller has no knowledge of commands that are active (through the other controller) to LUs it does not own. Ownership of a LU can be changed to the other controller, but the method is vendor-specific and requires a wedge driver, as it must be initiated by the host system. Note also that it may be desirable to change LU ownership for reasons other than controller failure (i.e. host-side HBA failure, where you want to move a LU to the other controller so you can still access its data), so that, even if the array did support internal failover (I think Clariion does as a not-so-widely-used option) a wedge driver would be desirable. Enough vendor-specific babble... the intent was to help clarify David's points through a bit of elaboration. As for multiple connections, I cast my vote for the asymmetric approach as long as it still permits us to support one connection per session (as in Kalman Meth's proposal). As you pointed out in an earlier message, a single control connection inherently takes care of command ordering without the reference number; command processing is simplified. As far as (control) connection recovery is concerned, an alternate approach (to the one specified in Kalman Meth's proposal) that does not require that a new connection be opened immediately might be to designate the lowest-numbered non-failed connection as the control connection. There would need to be some sort of message passed to inform the other end of the new control connection by the side that first detects the broken connection, followed by the rest of Kalman's recovery scheme (Query/Sync) or some variant. The advantage to this approach is that, in the multiple connections case, it could prevent the target from cleaning up its session state upon control connection breakage; I don't believe Kalman's proposal specified the target's actions if it discovers a dead control connection, and in the spirit of the 00 draft my assumption would be cleanup (otherwise the target is left with a "zombie" session if the initiator does not log in again). This isn't an explicit proposal, but it's something that might be worth thinking about... Ernie Pistor -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Tuesday, August 29, 2000 10:54 AM To: ips@ece.cmu.edu Subject: RE: iSCSI: "Wedge" drivers David, I understand 3 and I somehow expected it. If I would have to reimplement this under iSCSI I would choose a plug-in in iSCSI to do it (a policy module) and iSCSI would clearly support such a construct (and make it also future-proof). I have some trouble with 1 and 2. In whatever design you choose at the array level you have to have some sharing as some commands are targeted to a LU and all it's queues (i.e. separation per initiator is never complete if you take SAM seriously). However some design might choose to share only for those commands - and talking to you with you chair hat off - I don't think that you are with one of those (:- I am no sure I understand 4 either - if the array does no "sense" the fail over. Now talking to you with your hat on - we are ready to get to the drafting - but I feel that we are still left with the symmetric/asymmetric dilemma and I would appreciate input: The asymmetric solution is simpler for those using a single connection. Is there any serious drawback? Dear Colleagues - Please give us feedback. We can choose only one solution? Which is the right one? Julo Black_David@emc.com on 29/08/2000 16:26:49 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: "Wedge" drivers > While many of you keep telling me that wedge drivers won't go away nobody > gave one good example to support that assertion. With my co-chair hat off, here are four, two from the mail exchange: (1) Sharing SCSI state across array interface processors involves some work. Use of multiple SCSI connections and wedge drivers for fault tolerance and load balancing will persist in systems that choose not to follow that implementation path. (2) If iSCSI is implemented in hardware, so that a system has multiple iSCSI HBAs rather than multiple NICs with a common iSCSI software stack, multiple SCSI connections and wedge drivers are again the path of least resistance to fault tolerance and load balancing. And two new ones: (3) Naive greedy load balancing is not optimal. Significant improvements are possible based on an understanding of how the storage behaves. PowerPath contains Symmetrix-specific logic that does this, and I don't think that's appropriate for standardization. (4) Some arrays need additional support for fail-over. Clariion uses an active fail-over architecture in which the array must be instructed to fail-over, and the fail-over occurs across separate SCSI connections. I don't think that the Clariion-specific logic that does this is appropriate for standardization. As I wrote earlier, I have no problem with specifying a session abstraction for iSCSI, but it won't eliminate wedge drivers. With my co-chair hat on, the current consensus is to specify a session abstraction and review the result. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Aug 30 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11792; 30 Aug 2000 6:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA22989 for ips-outgoing; Wed, 30 Aug 2000 05:26:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA22985 for ; Wed, 30 Aug 2000 05:26:18 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA323070 for ; Wed, 30 Aug 2000 11:25:45 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id LAA34940 for ; Wed, 30 Aug 2000 11:25:44 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694B.0033CA54 ; Wed, 30 Aug 2000 11:25:41 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 30 Aug 2000 12:23:39 +0300 Subject: RE: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The datagrams I am referring in the note to Steph are iSCSI PDUs. No relation whatsoever to TCP PDUs or IP PDUs. Julo "Douglas Otis" on 29/08/2000 20:27:57 Please respond to "Douglas Otis" To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Data in SCSI Response or SCSI Data David, Clearly, the URGENT flag does even less at defining a point within the TCP stream. RFC 793 (TCP pg 12 & 13 continued.) "TCP also provides a means to communicate to the receiver of data that at some point further along in the data stream than the receiver is currently reading there is urgent data. TCP does not attempt to define what the user specifically does upon being notified of pending urgent data, but the general notion is that the receiving process will take action to process the urgent data quickly." Care should be taken not to make assumptions as to the behavior of TCP regarding frame alignment. These assumptions are being made by Julian in describing iSCSI placement of information. His being evasive does not help with assessments under such erroneous expectations. In describing iSCSI as being able to place information in specific datagrams, Julian modifies TCP. "Steph, I assume that your hardware knows when it is going to send the last SCSI datagram. In this case it can insert the status in the datagram header (not at the end) - (and BTW that is how the current draft assumes that things will be done). If you have to send sense-data (bad status) then you will send a separate datagram." Julo Aug 26, 2000 RE: Data in SCSI Response or SCSI Data Julian should clarify where in the iSCSI draft these assumptions he indicates exist. No where within the iSCSI document is datagram defined or used; should it be if this behavior is required? Perhaps there is no expectation of adhering to TCP specifications in the implementation of iSCSI and leaving out this information from iSCSI is an attempt to hide these TCP modifications. Doug > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, August 29, 2000 6:38 AM > To: dotis@sanlight.net; julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > I should point out that TCP's URGENT functionality > may be of some use here -- it won't provide the > level of support for message boundaries found > in SCTP, but might still be quite useful. --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > > > > -----Original Message----- > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > Sent: Monday, August 28, 2000 2:25 PM > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > Julo, > > > > RFC 793 (TCP pg 12) > > "A sending TCP is allowed to collect data from the sending user and to > > send that data in segments at its own convenience, until the push > > function is signaled, then it must send all unsent data. When a > > receiving TCP sees the PUSH flag, it must not wait for more data from > > the sending TCP before passing the data to the receiving process. > > > > There is no necessary relationship between push functions and segment > > boundaries. The data in any particular segment may be the result of a > > single SEND call, in whole or part, or of multiple SEND calls. > > > > The purpose of push function and the PUSH flag is to push data through > > from the sending user to the receiving user. It does not provide a > > record service. > > > > There is a coupling between the push function and the use of buffers > > of data that cross the TCP/user interface. Each time a PUSH flag is > > associated with data placed into the receiving user's buffer, the > > buffer is returned to the user for processing even if the buffer is > > not filled. If data arrives that fills the user's buffer before a > > PUSH is seen, the data is passed to the user in buffer size units." > > > > It would appear you are assuming a relationship between PUSH flags and > > segment boundaries. In addition, as iSCSI is the confluence of > > independent > > traffic, at what point would you set the PUSH flag? You suggest rather > > than > > a push function, there is a last datagram signal according to your > > comments > > below. Even with a PUSH flag, adding data to the buffer before > completion > > of push function does not provide guaranteed segment alignment nor is > > there > > any signaling for completion nor is there any intent at providing the > > alignment function you indicate. You suggest a 'last datagram' signal > > provides a simple means of positioning information at the > beginning of the > > TCP Datagram. What signal? In keeping within TCP, how do you implement > > your comments below which you state is the basis of iSCSI. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > julian_satran@il.ibm.com > > > Sent: Monday, August 28, 2000 9:18 AM > > > To: ips@ece.cmu.edu > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > Doug, > > > > > > I would appreciate if instead of a general and fuzzy > referency you would > > > quote > > > the document page and paragraph. > > > > > > Julo > > > > > > "Douglas Otis" on 28/08/2000 19:00:56 > > > > > > Please respond to "Douglas Otis" > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > cc: > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > Julo, > > > > > > Is it compliant to assume knowledge of the TCP frame by the iSCSI > > > application? In many places within iSCSI discussions, references are > > made > > > of packets, frames, and datagrams as an element within iSCSI. If > > creating > > > frame alignment is your intent, rather than corrupting TCP > APIs, perhaps > > > you > > > should consider SCTP. Otherwise, such discussions are little > more than > > a > > > wink and a nod at creating a frame aligned TCP. How do you > > > advise a change > > > to the TCP API to resolve frame alignment as you have just suggested? > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > julian_satran@il.ibm.com > > > > Sent: Saturday, August 26, 2000 2:28 AM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > Steph, > > > > > > > > I assume that your hardware knows when it is going to send the last > > SCSI > > > > datagram. > > > > In this case it can insert the status in the datagram header (not at > > the > > > > end) - > > > > (and BTW that is how the current draft assumes that things will > > > be done). > > > > > > > > If you have to send sense-data (bad status) then you will send > > > a separate > > > > datagram. > > > > > > > > Julo > > > > > > > > Stephen Bailey on 25/08/2000 22:12:09 > > > > > > > > Please respond to Stephen Bailey > > > > > > > > To: ips@ece.cmu.edu > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > > Is there anything preventing your hypothetical hardware > implementor > > > > > to send always good status within the last block of data? > > > > > > > > It depends upon the RDMA mechanism you are using. I admit > that I have > > > > not studied them in detail, other than ST of course, which is > > > > essentially an transport protocol based upon a particular RDMA > > > > mechanism. I will try to do so soon to determine if I'm all wet. > > > > > > > > However, assuming that the RDMA mechanism operates on a per-datagram > > > > basis, I can only imagine that you will not be able to append > > > > `general delivery' data to the end of an RDMA datagram. In > this case, > > > > you will need a separate datagram to ensure that status is delivered > > > > through a separate path from the data. > > > > > > > > So, while I think concatenating status and data in its > general form is > > > > not a good idea, a good-status fast path (like the success bit) is > > > > definitely the right way to think about it. Nonsuccess > SCSI status is > > > > so rare that any compromise you can make in the nonsuccess path to > > > > make the success path go faster is worth it. > > > > > > > > Steph > From owner-ips@ECE.cmu.edu Wed Aug 30 12:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12132; 30 Aug 2000 7:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA02574 for ips-outgoing; Wed, 30 Aug 2000 06:55:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA02568 for ; Wed, 30 Aug 2000 06:55:50 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id FAA03227; Wed, 30 Aug 2000 05:56:09 -0500 Message-ID: <39ACE848.63DCBE3A@stewart.chicago.il.us> Date: Wed, 30 Aug 2000 05:56:08 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis CC: Ips Subject: Re: (FCoverIP) SCTP extension. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > > As a possible extension to SCTP to deal with the limitation of FC, perhaps a > trinary setting of the retransmission could be made: Unlimited, 1, or 0. > By allowing at least one retry, most lost frames would be recovered within a > constrained window of time. The danger with FC is an underlying transport > overlap of recovery mechanisms. One other consideration is the size of the > buffers at each end of SCTP as this would also skew a time window but would > be required to prevent loss at this point as well. > > BTW, what is the latest T11 DMP specification? > > Doug Doug, I think I like this idea. We (the authors of the u-sctp extension) are currently debating the addtion of the bit to turn off the adler32... I will add this to the debate too :-) It is not really a wire change so I don't think there is any problem... R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Wed Aug 30 12:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12360; 30 Aug 2000 8:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA03708 for ips-outgoing; Wed, 30 Aug 2000 08:01:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA03696 for ; Wed, 30 Aug 2000 08:01:43 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id OAA275104; Wed, 30 Aug 2000 14:01:10 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id OAA39468; Wed, 30 Aug 2000 14:01:05 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694B.00420131 ; Wed, 30 Aug 2000 14:00:56 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Jim McGrath , ips@ece.cmu.edu Message-ID: Date: Wed, 30 Aug 2000 14:58:54 +0300 Subject: RE: what's in a name Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks Jim. I assume several of our companies will want to do this. Julo From owner-ips@ECE.cmu.edu Wed Aug 30 12:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12374; 30 Aug 2000 8:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA03405 for ips-outgoing; Wed, 30 Aug 2000 07:47:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA03401 for ; Wed, 30 Aug 2000 07:47:39 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA323042 for ; Wed, 30 Aug 2000 13:47:07 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id NAA44940 for ; Wed, 30 Aug 2000 13:47:07 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694B.0040B7AA ; Wed, 30 Aug 2000 13:46:53 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 30 Aug 2000 14:24:52 +0300 Subject: RE: what's in a name Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I asked around - and since we have here one person that has a PhD in CS and a Law Degree - and he claims that as long as we use the any sequence of words not in the context of an advertisement or other product reference (e.g., within the standard we are not required to say anything about the TM - and that applies also to the Virtual Private SAN as it is a sequence of words and has a technical meaning). If the organization that owns the TM doesn't like it - they may choose to go to court - and there is a good chance (if the word sequence has a specific meaning and is understood in the technical circles without regard to the TM and prior to the TM) that they will loose the TM. In any case let's not loose too much sleep on this! Julo "Don Woelz" on 29/08/2000 23:52:28 Please respond to "Don Woelz" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: what's in a name > Isn't there anything in the rules for giving TM names that forbids use of > public names? Can somebody ask a TM for TCP? Julian, If you are refering to their TM on iSCSI, if they had prior use and a TM before you started using it, then they own it. Don Donald D. Woelz Tel: 262-644-2505 V. P. of Marketing Mobile: 414-732-7705 GENROCO, Inc. Toll Free: 800-243-6762 255 Info Hwy Fax: 262-644-6667 Slinger, WI 53086 From owner-ips@ECE.cmu.edu Wed Aug 30 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12792; 30 Aug 2000 10:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA07510 for ips-outgoing; Wed, 30 Aug 2000 09:16:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraab.compuserve.com (ds-img-rel-2.compuserve.com [149.174.206.155]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA07501 for ; Wed, 30 Aug 2000 09:16:38 -0400 (EDT) Received: (from mailgate@localhost) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id JAA18992 for ips@ece.cmu.edu; Wed, 30 Aug 2000 09:16:08 -0400 (EDT) Received: from compuserve.com (dal-qbu-zoy-vty24.as.wcom.net [216.192.252.24]) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id JAA18925 for ; Wed, 30 Aug 2000 09:16:00 -0400 (EDT) Message-ID: <39AD09AF.7A5B3A63@compuserve.com> Date: Wed, 30 Aug 2000 08:18:39 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, I hope Somesh will forgive me for butting in on his discussion topic. } I have on objection on making Autosense mandatory and } removing the bit from iSCSI. } } My issue (if it is an issue) is SCSI related. } } How are asynchronous events that are not error related } handled without Contingent Allegiance - and what effect } will it have on iSCSI. } } The case that was raised a while ago (by Costa I think) } was that of a volume change - with the message reporting } it "crossing" a write on the wire or in the OS. } } With CA the state synchronization can be easily enforced } (command are rejected until a sense is received). Please understand that this description is not how CA works. With CA, commands are rejected only until another command is received from the same initiator that got the CHECK CONDITION status. That command, whatever it is, gets processed and the CA gets cleared and the sense data gets cleared too. Let's suppose that an iSCSI initiator and target a transacting SCSI business over the world wide internet. There is only one initiator (and one target and one LUN) in this example. The initiator's commands are getting bounced from Boise to Bangkok to the target in Moscow and vice versa. So, there's a noticeable latency in the iSCSI link and the initiator (who's doing this gigantic database update, for reasons that I cannot easily justify) is trying to make up for the latency by pumping commands in to the network as fast as it can. In this situation there are dozens of commands in flight from the initiator to the target at any given instant. At the instant the CA condition is set in the target by the unit attention condition, the initiator has say 9 commands that have been sent but have not yet reached the target. The first of these will clear the CA and return to the initiator with a CHECK CONDITION status. The remaining 8 will be processed as if nothing happened. CA works only on the parallel SCSI bus. Once upon a time, the protocol for the parallel SCSI bus was called the 'interlocked' protocol because the initiator and target march through the transfers necessary to move data in lock step. CA takes advantage the interlocked parallel SCSI bus. Proper functioning of CA requires that the initiator know that it is getting a CHECK CONDITION status before it has any chance to send another command. CA cannot work when the initiator and target are not transferring commands, data and status in lock step. } Without it [CA] - even with autosense - state synchronization } in the OS becomes tricky. My personal opinion is that state synchronization issues should be eliminated by requiring the target to fully process conditions before reporting them. Most unit attention events can be resolved at the target before they are reported to the initiator. For the rarely occurring rest, I'd recommend requiring the target to break its link (connection, whatever, I have trouble with IETF terms) with the initiator. That will force the initiator to login again and thus verify that the changed operating conditions are valid. The 'volume change' case mentioned above clearly falls in the latter category, since the connection that existed before the volume change might not be allowed to form after the change. } Perhaps somebody on the list active also in T10 can give us } a hint of how this type of sequence will be handled without CA. The other possibility is ACA (see the discussion with Douglas Otis). However, I feel that ACA is a very large hammer to be wielding at otherwise small gnats. } I understand that CA can be kept even with autosense and we } can remove the bit but what if T10 decides to keep both? T10 decides to keep both in the SCSI Architecture because T10 still has the interlocked protocol to support on the parallel SCSI bus. When T10 invents a non-interlocked or packetized protocol for a network-like transport (e.g., FCP), T10 dumps CA overboard. Thanks. Ralph Weber ENDL Texas From owner-ips@ECE.cmu.edu Wed Aug 30 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12808; 30 Aug 2000 10:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA06971 for ips-outgoing; Wed, 30 Aug 2000 09:01:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA06967 for ; Wed, 30 Aug 2000 09:01:49 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id IAA03678; Wed, 30 Aug 2000 08:01:55 -0500 Message-ID: <39AD05C1.F4C6D1F8@stewart.chicago.il.us> Date: Wed, 30 Aug 2000 08:01:53 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: John Hufferd/San Jose/IBM CC: Y P Cheng , ips@ece.cmu.edu Subject: Re: iSCSI: "Wedge" drivers References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk John: I don't think that it has been decided that you are using TCP or SCTP as of yet. This issue is still up for debate and I have not yet sensed a consenses (IMHO) as to which way to go... TCP does provide reliable delivery and one connection. It will give you the congestion control you need but it does have issues such as message boundaries and multi-homing support. I have not heard and I guess I have not formally asked... is the head of line blocking issue at all relevant. It sounds like you CAN fire off several commands in parallel... If you do this, MUST they be delivered in order, or do you want them delivered and processed in any order? Do you want to have the option of NOT retransmitting datagrams or Douglas's retransmit at most 1 time... I think this will be added to the u-sctp draft.. it is a neat feature... Now as to availablility of SCTP.. v9 is now available (you can get a ref-impl and I know 10 vendors working on supporting SCTP .. at least 10). SCTP V13 (the rfc version) will be supported in the ref-impl next week.. inbetween email, this is what I am working on... almost done I just need to see what I have broken with a bit of testing :) So SCTP is out there and is available.. it is not as predominant as TCP nor do I know of it in silicon.. but on the other hand it adds a LOT of features that the iSCSI can use... R John Hufferd/San Jose/IBM wrote: > > Y.P. > I thought the first part of your note was useful, but then you went to > something about an FCP/SCTP and creating an IP-to-FC-address from a > translation table, etc. At that point, you diverted from the iSCSI draft > and went where I do not believe many of us were headed. The current draft > assumes we are using TCP/IP not SCTP and has no additional layer of FCP. > > . > . > . > John L. Hufferd > > "Y P Cheng" @ece.cmu.edu on 08/29/2000 06:58:45 PM > > Sent by: owner-ips@ece.cmu.edu > > To: Julian Satran/Haifa/IBM@IBMIL, > cc: > Subject: RE: iSCSI: "Wedge" drivers > > Julo Wrote: > >I understand 3 and I somehow expected it. If I would have to > >reimplement this under iSCSI I would choose a plug-in in > >iSCSI to do it (a policy module) and iSCSI would clearly > >support such a construct (and make it also future-proof). > >I have some trouble with 1 and 2. In whatever design you > >choose at the array level you have to have some sharing as > >some commands are targeted to a LU and all it's queues > >(i.e. separation per initiator is never complete if you > >take SAM seriously). However some design might choose to > >share only for those commands - and talking to you > >with you chair hat off - I don't think that you are with > >one of those (:- I am no sure I understand 4 either - if > >the array does no "sense" the fail over. > > I have been reading the "Wedge Drivers" and their "load balance and fail > over recovery" with great interest. I am not familiar with the mainframe > I/O design. However, I worked on SCSI and fibre channel host bus adapters > (HBA's) for long time. I would like apply those discussions on the > adapters > that I know of. The understanding of the HBA's would greatly simplify the > discussions of wedge driver and its load balance and recovery. > > The SCSI and fibre channel HBA's today executes a SCSI command atomically > without any intervention from the device driver. When there are multiple > HBAs on one server, they don't share command execution states. Each HBA > executes their own SCSI commands atomically. The HBA and its driver > perform > discovery of SCSI or FCP devices at power up. It numbers the devices > sequentially. A table is created to map the device number to a SCSI ID or > a > fibre channel 24-bit address. When there are multiple HBA's, it is the > responsibility of the Upper Level Drivers to direct the SCSI commands > through their appropriate HBA's. The HBA itself can care less. > > For a iSCSI HBA, there won't be any discovery at powerup. Instead, the HBA > will monitor the ARP and FARP requests and responses to create a > IP-to-FC-address translation table. (This discussion does not exclude an > Ethernet NIC card.) The HBA relies on the FCP/SCTP driver to create > endpoints and associates for connection oriented exchanges. The iSCSI > driver > running above the FCP/SCTP driver and below the SCSI-class driver will be > responsible for SCSI-over-IP messages. In fact the iSCSI HBA is no > different from an NIC with the exception of the FCP/RDMA support. When > there > are multiple iSCSI HBA's, The FCP/SCTP driver will be responsible to > determine if an IP device can be reached from a different HBA. (Whether an > IP device can be reached by another IP address is beyond the scope of this > discussion.) We may choose to include a subset of the SCTP function in the > iSCSI driver if we don't wait for the SCTP implementation. In the > discussion below we refer the iSCSI driver with a subset of the SCTP > function. > > Knowing there is an alternative HBA to reach the same IP device, it is > certainly the discretion of the iSCSI driver to sent IP-SCSI messages to > the > HBA of its choice. A response must come back to the same HBA because it is > difficult for two HBA's to share execution states. For example, if we set > up one HBA to do the RDMA function, it is not nice to have the data coming > back to the other HBA. Both fail over and load balance can be done in iSCSI > driver relatively easily because it is fully aware of the availability of > alternative HBA's reaching the same IP device. In theory, we could break > up > the iSCSI driver by moving the SCSI-to-IP-packet conversion function to the > HBA and by keeping all the remaining functions. If we do so, the SCSI > command will be executed atomically inside an HBA. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Wed Aug 30 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12794; 30 Aug 2000 10:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA07509 for ips-outgoing; Wed, 30 Aug 2000 09:16:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraab.compuserve.com (ds-img-rel-2.compuserve.com [149.174.206.155]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA07500 for ; Wed, 30 Aug 2000 09:16:38 -0400 (EDT) Received: (from mailgate@localhost) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id JAA18978 for ips@ece.cmu.edu; Wed, 30 Aug 2000 09:16:07 -0400 (EDT) Received: from compuserve.com (dal-qbu-zoy-vty24.as.wcom.net [216.192.252.24]) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id JAA18900 for ; Wed, 30 Aug 2000 09:15:55 -0400 (EDT) Message-ID: <39AD0203.4EFA4947@compuserve.com> Date: Wed, 30 Aug 2000 07:45:55 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI Autosense References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas, I am very pleased to see closure on the issue of whether or not it is possible to build a reliable bridge between a packetized protocol and a parallel SCSI device that supports only CA (Contingent Allegiance). Agreement on that point was clearly a critical step between previous discussions and making Autosense the modus operandi of iSCSI. } > To the best of my knowledge it is extremely difficult to } > implement the CA model in a packetized transfer model because } > the CA condition is cleared by the next command to arrive } > at the target regardless of what kind of command that is. } Not true. Only for the nexus is this true. Yes, I am assuming that only a single I_T_L nexus is involved. Unless there are some iSCSI constraints that I don't know about (see below), that seems like a reasonable assumption. } > Since most packetized protocols allow several commands to be } > in flight concurrently between initiator and target, the most } > probably case in a packetized protocol is that an in flight } > command will clear a CA condition long before the initiator } > finds out about it and has a chance to fire off the needed } > REQUEST SENSE in to get the sense data. } These in flight commands are a different nexus such that Sense } data is not lost. If the device can not maintain separate Sense } information BUSY is returned. Are you suggesting that iSCSI prohibits more than one command from being in flight from a single initiator to a single target/LUN? I have not read anything in the iSCSI draft that would suggest this. I find it difficult to believe that there is such a limitation in iSCSI as it would be a horrific performance bottleneck. } > The bottom line here is that Autosense works for packetized } > protocols and CA does not. Since it is trivial to translate } > a packetized Autosense front end to the CA behavior of an } > older backend SCSI device, the right thing for iSCSI to do } > is to make Autosense mandatory and not to provide any way } > whatsoever to disable it. } iSCSI does provide a means to disable it, but not to refuse it. What is the point of disabling Autosense if the only result is to put iSCSI in a dysfunctional mode of operation? } > The fact that SAM is written with a preference for CA is } > (like CA itself) a historical artifact. So long as the } > wording continues to be correct, it is unlikely that T10 } > will change the wording (if it ain't broke don't fix it). } > } > However, the ancient leanings of SAM should not under any } > circumstances be used as a justification for iSCSI support } > for a CA-like interface. As noted above, CA doesn't work } > for packetized protocols like iSCSI. } A mistaken conclusion. I have no response to such a slander on my knowledge of SCSI that lacks even the decency to include an explanation. } > ACA (Auto Contingent Allegiance) is absolutely positively } > NOT the super-duper, all fixed up for a packetized world } > brother of CA. ACA is orthogonal to Autosense and actually } > has no place in this discussion. } Odd, but error control and reporting does not seem orthogonal. But they are. Error reporting means getting a description of the error from the place where the error occurred to the place that needs to know that the error occurred. Error control is an embellishment that obliges the place where the error occurred to cooperate with some other entity in resolving the error. Error reporting is a necessary basic feature. Error control adds additional complexity based on the assumption that the system designers cannot adequately define error processing so that the point of error detection takes all necessary recovery steps before reporting the error. } > IN ALL CASES, ACA is optional. Simply return 0 in the } > NormACA bit (bit 5 byte 3) of the Standard INQUIRY Data } > and you don't support ACA and you are expected to refuse } > to process any CDB with the NACA bit set to 1. } For a means of get things to stop, some mechanism should exist. } This becomes difficult by suggesting refusal of Autosense is a } bad choice. ACA may not exist for this purpose if you allow } older devices. I suggest that there be a means to refuse } Autosense. If the driver can not support this option, it can } refuse the device. Absolutely correct, "a means for get[ting] things to stop" must be provided for in the standard. For a packetized protocol, the tools are: 1) no stop - Autosense 2) stop - ACA Like bridging Autosense for a CA-only device, bridging ACA for a non ACA device is possible although not as trivial as Autosense. Stated simplistically, bridging software monitors the NACA bit, CHECK CONDITION status returns and performs the functions that an ACA capable device would. Complexity arises only when the NACA bit is not a constant value, so to keep this short I'll consider only the case where NACA always equals 1. Note, the case where NACA always equals 0 is not interesting because that means ACA is never used. Assuming that NACA always equals 1, the bridging software takes the following steps upon receipt of a CHECK CONDITION status from its non-ACA endpoint device: 1) Establish a state on the I_T_L Nexus that causes all newly received commands to be immediately returned with an ACA ACTIVE status unless the ACA Task Attribute is specified in the SCSI Command packet (see 3.2.2 in draft-satran- iscsi-01.txt). 2) Send a REQUEST SENSE command to the endpoint device. 3) Package any previous command status with the sense data received as a result of the REQUEST SENSE command in a SCSI Response packet (see 3.3 in draft-satran-iscsi-01.txt) and send the packet to the initiator. 4) Continue processing incoming SCSI Command packets as described above until one arrives with NACA equal to 0 (this is where things get dicey) or until a SCSI Task Management Command packet is received with the Clear ACA function specified (see 3.7 in draft-satran-iscsi-01.txt). N.B. Steps 1 and 4 are the 'error control' capability discussed above. Steps 2 and 3 are the 'error reporting' capability and they are incidentally the same steps the bridge would be required to perform if ACA were not in use. The two are orthogonal. Thanks. Ralph Weber ENDL Texas From owner-ips@ECE.cmu.edu Wed Aug 30 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13172; 30 Aug 2000 11:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA11024 for ips-outgoing; Wed, 30 Aug 2000 10:38:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from renown.cnchost.com (renown.concentric.net [207.155.248.7]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA11016 for ; Wed, 30 Aug 2000 10:38:18 -0400 (EDT) Received: from Trebia5 (ts002d47.box-ma.concentric.net [206.173.12.107]) by renown.cnchost.com id KAA09664; Wed, 30 Aug 2000 10:38:15 -0400 (EDT) [ConcentricHost SMTP Relay 1.10] From: "James Smart" To: "ips" Subject: Target Reset handling Date: Wed, 30 Aug 2000 10:42:48 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk In reading the iscsi-01 draft, I was bothered by several things in the handling of Target Reset. a) The lack of at least a basic ACCept on the Target Reset. If the target can send an async event, why not at least notify reception of the function ? Given connections with lots of outstanding traffic, I'd see this as a more graceful reset procedure. It allows any outstanding i/o that may be completing while the TR is in transit (or queued for processing on the target) to do so, possibly lightening the load of i/o that has to error to complete. This would potentially quicken the recovery time post reset. I would expect this to be more important as the "network" gets larger and longer. Note: FCP does support this behavior. b) Why not require async events to all initiators ? The biggest headache with Target Reset is how long it takes for the other initiators to recognize the device has been reset. The 1st new i/o will get a Unit Attention CA, but this status is typically seen only by the SCSI class driver (e.g. disk/tape/etc). Unless instructed by the class driver, the port level driver (e.g. scsi/fc/iscsi hba) will have to timeout the i/o's (if timing was requested) to recover their context. If the class driver does try to tell the port driver, it typically will do so in a crude fashion - issuing abort requests on the i/o's it knows about. Perhaps, if the TCP connections are gracefully shutdown between the initiator and target, the initiator will be to abort the i/o on the connections quickly (in this case, it looks like a pseudo async event). However, if there is no handshaking on the connections, my limited experience with TCP says it takes a long time for the connection to error out and reset. And during this process, we'll be sending i/o abort requests down the terminated-on-one-end connection. All this would make the recovery time on these other intiators very large. Note: this point assumes that if async events are required - they are ack'd. c) Is there something inherent that requires the TCP connections to be terminated ? The TCP connections look very similar to (but not the same as) FCP Process logins between the initiator and target. In FCP, the reset did not necessarily disrupt the port or process logins. It only had to affect the FPC/SCSI task manager. (note: a device was free to really reset, thus indeed tearing down the logins - with the FC port machine handling it as an error) What is the background that required the TCP sessions to be broken ? Obviously, if they are not broken, it affects the answers to (a) and (b) above. d) Given the history of long error recovery times in multi-initiator environments in both parallel scsi and fibre channel on BDR's/Target Reset's, any speed up in this area would be advantageous. -- James -------------------- James Smart Trebia Networks, Inc Ph: 978-318-9547 35 Forest Ridge Rd Cell: 603-674-3687 Concord, MA 01742 james.smart@trebia.com From owner-ips@ECE.cmu.edu Wed Aug 30 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13423; 30 Aug 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA12932 for ips-outgoing; Wed, 30 Aug 2000 11:24:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA12926 for ; Wed, 30 Aug 2000 11:24:26 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA14262 for ; Wed, 30 Aug 2000 17:23:49 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA21342 for ; Wed, 30 Aug 2000 17:23:49 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694B.00549189 ; Wed, 30 Aug 2000 17:23:42 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 30 Aug 2000 18:21:34 +0300 Subject: Re: iSCSI Autosense Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, How are volume change events supposed to be treated with CA, ACA and autosense (or for that matter any Async. Event that intends to convey to an initiator state information that the initiator should be aware of before sending commands - i.e. how is state sync. achieved)? I was under the impression that this is one of vestigial uses of CA. It is obvious that autosense is fine but until I can't be convinced that it is completely orthogonal to CA use (or to some other mechanism that can assure state sync) I would be reluctant to force it. If state sync. is provided by some other means that I will be happy to oblige and assume autosense is always there. Thanks, Julo Ralph Weber on 30/08/2000 06:12:53 Please respond to ENDL_TX@computer.org To: IPS Reflector cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI Autosense This is going to be fun, trying to respond to all the information and misinformation that's been bandied about today. David Black's note is correct. The method for bridging a CA-only (Contingent Allegiance only) device to an Autosense representation of that device is well known and trivial. In addition to being documented in the requirements, the method is documented in the first paragraph of 5.3 in draft-satran-iscsi-01.txt. I have two minor complaints about the 5.3 wording. 1) The method will work all the time, so the suggestion that it 'may' work is incorrect. 2) I would prefer that the REQUEST SENSE command be explicitly mentioned not alluded to. As regards CA, let me restate my position more emphatically. CA is a historical artifact that dates to the single byte of status model employed by the parallel SCSI bus. To the best of my knowledge it is extremely difficult to implement the CA model in a packetized transfer model because the CA condition is cleared by the next command to arrive at the target regardless of what kind of command that is. Since most packetized protocols allow several commands to be in flight concurrently between initiator and target, the most probably case in a packetized protocol is that an in flight command will clear a CA condition long before the initiator finds out about it and has a chance to fire off the needed REQUEST SENSE in to get the sense data. The bottom line here is that Autosense works for packetized protocols and CA does not. Since it is trivial to translate a packetized Autosense front end to the CA behavior of an older backend SCSI device, the right thing for iSCSI to do is to make Autosense mandatory and not to provide any way whatsoever to disable it. The fact that SAM is written with a preference for CA is (like CA itself) a historical artifact. So long as the wording continues to be correct, it is unlikely that T10 will change the wording (if it ain't broke don't fix it). However, the ancient leanings of SAM should not under any circumstances be used as a justification for iSCSI support for a CA-like interface. As noted above, CA doesn't work for packetized protocols like iSCSI. ACA (Auto Contingent Allegiance) is absolutely positively NOT the super-duper, all fixed up for a packetized world brother of CA. ACA is orthogonal to Autosense and actually has no place in this discussion. IN ALL CASES, ACA is optional. Simply return 0 in the NormACA bit (bit 5 byte 3) of the Standard INQUIRY Data and you don't support ACA and you are expected to refuse to process any CDB with the NACA bit set to 1. ACA is more akin to the SCSI-2 ECA (Extended Contingent Allegiance) than it is to either CA or Autosense. The assumption in ACA and ECA is that the initiator needs to send several commands to the target in order to cleanup whatever error occurred. The most frequently cited excuse for this is the cleanup needed when a tape writes past the EOT (End Of Tape) reflective strip. Thanks. Ralph Weber ENDL Texas From owner-ips@ECE.cmu.edu Wed Aug 30 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13877; 30 Aug 2000 14:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA19357 for ips-outgoing; Wed, 30 Aug 2000 13:57:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from www.indstorage.com (windjo.ppp.frii.com [216.17.133.220] (may be forged)) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA19343 for ; Wed, 30 Aug 2000 13:57:44 -0400 (EDT) Received: from aegis ([192.168.1.12]) by www.indstorage.com (8.9.3/8.9.2) with SMTP id LAA31281; Wed, 30 Aug 2000 11:50:51 -0600 From: "Mark Bradley" To: "Jim McGrath" , , Subject: RE: what's in a name Date: Wed, 30 Aug 2000 11:51:22 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk This is incorrect information. There is no difference between SM and TM except that one applies to a service and the other to a product. -- markb > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jim McGrath > Sent: Tuesday, August 29, 2000 9:05 PM > To: 'julian_satran@il.ibm.com'; ips@ece.cmu.edu > Subject: RE: what's in a name > > > > You file iSCSI as a Service mark (SM). That essentially reserves the name > so no one else can trademark it, but allows other to use it (if you TM the > name then you have to defend it if others use it or you lose the TM - the > Xerox problem). > > I recollect that we did this with Ultra SCSI and it worked ok. > > Jim > > > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, August 29, 2000 7:31 AM > To: ips@ece.cmu.edu > Subject: what's in a name > > > > > Yaron Klein just pointed out to me tha Virtual Private SAN and VPS are TM > of > Pathlight (www.pathlight.com) and so are SAN Gateway and SAN router. > > Costa earlier pointed out that iSCSI is the name of a cable for iMAC (but > not a trademark > as far as I can tell). > > Can somebody tell us how to protect at least iSCSI from TM claims? > > Can the Internet Society do it (before it is too late)? > > Julo > From owner-ips@ECE.cmu.edu Wed Aug 30 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13886; 30 Aug 2000 14:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA17162 for ips-outgoing; Wed, 30 Aug 2000 13:02:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraaa.compuserve.com (ds-img-rel-1.compuserve.com [149.174.206.140]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA17157 for ; Wed, 30 Aug 2000 13:02:04 -0400 (EDT) Received: (from mailgate@localhost) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id NAA06741 for ips@ece.cmu.edu; Wed, 30 Aug 2000 13:01:33 -0400 (EDT) Received: from compuserve.com (dal-qbu-zoy-vty12.as.wcom.net [216.192.252.12]) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id NAA06701 for ; Wed, 30 Aug 2000 13:01:26 -0400 (EDT) Message-ID: <39AD3EAC.77CDFAA6@compuserve.com> Date: Wed, 30 Aug 2000 12:04:44 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips Subject: Re: Target Reset handling References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk James Smart wrote: } Given connections with lots of outstanding traffic, I'd see this as a more } graceful reset procedure. It allows any outstanding i/o that may be } completing while the TR is in transit (or queued for processing on the } target) to do so, possibly lightening the load of i/o that has to error to } complete. This would potentially quicken the recovery time post reset. I } would expect this to be more important as the "network" gets larger and } longer. } Note: FCP does support this behavior. The concept of a target queuing I/O for processing across a target reset boggles my mind. I'm having a really hard time believing that FCP goes that far. } b) Why not require async events to all initiators ? } The biggest headache with Target Reset is how long it takes for the other } initiators to recognize the device has been reset. The 1st new i/o will get } a Unit Attention CA, but this status is typically seen only by the SCSI } class driver (e.g. disk/tape/etc). Unless instructed by the class driver, } the port level driver (e.g. scsi/fc/iscsi hba) will have to timeout the } i/o's (if timing was requested) to recover their context. If the class } driver does try to tell the port driver, it typically will do so in a crude } fashion - issuing abort requests on the i/o's it knows about. Checkout ftp://ftp.t10.org/t10/document.00/00-229r1.pdf. It's a proposal in progress and has not been approved yet. But, the issue raised above is not limited to iSCSI. } d) Given the history of long error recovery times in multi-initiator } environments in both parallel scsi and fibre channel on BDR's/Target } Reset's, any speed up in this area would be advantageous. The real problem is using Target Resets for purposes other than clearing an obviously stuck target. T10 keeps trying to provided the right tools for the right uses (e.g., Persistent Reservations) but at least one major software vendor sees them as inconveniences instead of assistants. Thanks. Ralph... From owner-ips@ECE.cmu.edu Wed Aug 30 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14210; 30 Aug 2000 15:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA21350 for ips-outgoing; Wed, 30 Aug 2000 14:43:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraac.compuserve.com (ds-img-rel-3.compuserve.com [149.174.206.154]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA21346 for ; Wed, 30 Aug 2000 14:43:28 -0400 (EDT) Received: (from mailgate@localhost) by spdmraac.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id OAA20259 for ips@ece.cmu.edu; Wed, 30 Aug 2000 14:42:53 -0400 (EDT) Received: from compuserve.com (dal-qbu-zok-vty53.as.wcom.net [216.192.243.53]) by spdmraac.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id OAA20233 for ; Wed, 30 Aug 2000 14:42:48 -0400 (EDT) Message-ID: <39AD5633.956F4B3C@compuserve.com> Date: Wed, 30 Aug 2000 13:45:07 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 CC: ips@ece.cmu.edu Subject: Re: iSCSI Autosense References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, } How are volume change events supposed to be treated with CA, } ACA and autosense... There are two ways to report a Unit Attention condition: 1) Asynchronous Event Reporting 2) Use of a Unit Attention CHECK CONDITION Asynchronous Event Reporting is optional for both targets and initiators, and must be enabled by the initiator via the Control mode page before it can be used. I have assumed that 3.6 in draft-satran-iscsi-01.txt is somehow related to AER, and have yet to work through the details. The use of a Unit Attention CHECK CONDITION follows all the same rules as any other CHECK CONDITION with respect to CA, Autosense, and ACA. When this mechanism is used (which is far and away the most common case), there are no functional differences between a Unit Attention CHECK CONDITION and a Hardware Error CHECK CONDITION. I'll provide detailed answers to detailed questions, but the above information seems like about all that can be said in response to the question raised. } (or for that matter any Async. Event that intends } to convey to an initiator state information that the initiator } should be aware of before sending commands - i.e. how is state } sync. achieved)? I believe that the vast majority of asynchronous events do not require the initiator to be aware of them before other commands are sent. That is, I believe that 'volume change' is the exception not the rule and somebody is going to have to produce a substantial list of exceptions before I'm likely to change my opinion. If indeed 'volume change' is the exception, then I have no problems with according it exceptional handling, as in forcing logouts (or whatever is equivalent) for all initiators as a response. } I was under the impression that this is one of vestigial uses } of CA. I believe that the impression is incorrect and I have explained at great length why CA offers not even a fig leaf's worth of protection. I regret that I cannot argue against this incorrect impression any more forcefully than I already have. } It is obvious that autosense is fine but until I can't(sic) be } convinced that it is completely orthogonal to CA use (or to some } other mechanism that can assure state sync) I would be reluctant } to force it. CA is NOT orthogonal to Autosense. They are two mechanisms for achieving the same end, which is to ensure that the sense data for a CHECK CONDITION reaches the initiator. However, CA is based on assumptions that apply ONLY to the interlocked parallel SCSI bus. Therefore, the only mechanism that is available to a packetized protocol is Autosense. Combining CA with a packetized protocol does not improve reliability or state synchronization between initiator and target. Exactly the opposite is true, CA depends for is proper operation on a level of state synchronization between initiator and target that is available ONLY on the interlocked parallel SCSI bus. Combining CA with a packetized protocol presents a fictitious image of synchronization, that's all it does. } If state sync. is provided by some other means that I will be happy } to oblige and assume autosense is always there. ACA provides state synchronization between initiators and targets and I have not contested support for ACA in iSCSI. In fact, as you can see from my response this morning to Douglas Otis, I have reviewed the ACA support provisions in the iSCSI draft and verified that they are appropriate. I submit that ACA is the 'other means' that you have requested. Thanks. Ralph Weber ENDL Texas From owner-ips@ECE.cmu.edu Wed Aug 30 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14205; 30 Aug 2000 15:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA22106 for ips-outgoing; Wed, 30 Aug 2000 15:00:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA22102 for ; Wed, 30 Aug 2000 15:00:02 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA06658; Wed, 30 Aug 2000 11:52:13 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA00691; Wed, 30 Aug 2000 11:49:47 -0700 From: "Douglas Otis" To: , Subject: RE: Data in SCSI Response or SCSI Data Date: Wed, 30 Aug 2000 12:02:08 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, My apology for the misunderstanding. PDU is defined as Protocol Data Unit within the iSCSI spec. Using datagram, packet or frame is not a good choice of terminology if the meaning is PDU as it implies a far different meaning with respect to IP. We were speaking past each other with these unfortunate terms. Much as I think RTT is not a good mnemonic with respect to a standard intended to run on IP protocols. I guess I should first quiz the terms. Although I think I made my concerns clear, it is obviously a confusion based on terminology. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Wednesday, August 30, 2000 2:24 AM > To: ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > > > The datagrams I am referring in the note to Steph are iSCSI PDUs. No > relation whatsoever to TCP PDUs or IP PDUs. > > Julo > > "Douglas Otis" on 29/08/2000 20:27:57 > > Please respond to "Douglas Otis" > > To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: Data in SCSI Response or SCSI Data > > > > > David, > > Clearly, the URGENT flag does even less at defining a point within the TCP > stream. > > RFC 793 (TCP pg 12 & 13 continued.) > > "TCP also provides a means to communicate to the receiver of data that > at some point further along in the data stream than the receiver is > currently reading there is urgent data. TCP does not attempt to > define what the user specifically does upon being notified of pending > urgent data, but the general notion is that the receiving process will > take action to process the urgent data quickly." > > Care should be taken not to make assumptions as to the behavior of TCP > regarding frame alignment. These assumptions are being made by Julian in > describing iSCSI placement of information. His being evasive > does not help > with assessments under such erroneous expectations. In > describing iSCSI as > being able to place information in specific datagrams, Julian > modifies TCP. > > "Steph, > > I assume that your hardware knows when it is going to send the > last SCSI datagram. In this case it can insert the status in > the datagram header (not at the end) - (and BTW that is how > the current draft assumes that things will be done). > > If you have to send sense-data (bad status) then you will send > a separate datagram." > > Julo Aug 26, 2000 RE: Data in SCSI Response or SCSI Data > > Julian should clarify where in the iSCSI draft these assumptions he > indicates exist. No where within the iSCSI document is datagram > defined or > used; should it be if this behavior is required? Perhaps there is no > expectation of adhering to TCP specifications in the implementation of > iSCSI > and leaving out this information from iSCSI is an attempt to hide > these TCP > modifications. > > Doug > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Tuesday, August 29, 2000 6:38 AM > > To: dotis@sanlight.net; julian_satran@il.ibm.com; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > I should point out that TCP's URGENT functionality > > may be of some use here -- it won't provide the > > level of support for message boundaries found > > in SCTP, but might still be quite useful. --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > > black_david@emc.com Cellular: +1 (978) 394-7754 > > --------------------------------------------------- > > > > > > > -----Original Message----- > > > From: Douglas Otis [SMTP:dotis@sanlight.net] > > > Sent: Monday, August 28, 2000 2:25 PM > > > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > Julo, > > > > > > RFC 793 (TCP pg 12) > > > "A sending TCP is allowed to collect data from the sending user and > to > > > send that data in segments at its own convenience, until the push > > > function is signaled, then it must send all unsent data. When a > > > receiving TCP sees the PUSH flag, it must not wait for more > data from > > > the sending TCP before passing the data to the receiving process. > > > > > > There is no necessary relationship between push functions > and segment > > > boundaries. The data in any particular segment may be the result of > a > > > single SEND call, in whole or part, or of multiple SEND calls. > > > > > > The purpose of push function and the PUSH flag is to push data > through > > > from the sending user to the receiving user. It does not provide a > > > record service. > > > > > > There is a coupling between the push function and the use of buffers > > > of data that cross the TCP/user interface. Each time a PUSH flag is > > > associated with data placed into the receiving user's buffer, the > > > buffer is returned to the user for processing even if the buffer is > > > not filled. If data arrives that fills the user's buffer before a > > > PUSH is seen, the data is passed to the user in buffer size units." > > > > > > It would appear you are assuming a relationship between PUSH flags and > > > segment boundaries. In addition, as iSCSI is the confluence of > > > independent > > > traffic, at what point would you set the PUSH flag? You > suggest rather > > > than > > > a push function, there is a last datagram signal according to your > > > comments > > > below. Even with a PUSH flag, adding data to the buffer before > > completion > > > of push function does not provide guaranteed segment alignment nor is > > > there > > > any signaling for completion nor is there any intent at providing the > > > alignment function you indicate. You suggest a 'last datagram' signal > > > provides a simple means of positioning information at the > > beginning of the > > > TCP Datagram. What signal? In keeping within TCP, how do you > implement > > > your comments below which you state is the basis of iSCSI. > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > julian_satran@il.ibm.com > > > > Sent: Monday, August 28, 2000 9:18 AM > > > > To: ips@ece.cmu.edu > > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > Doug, > > > > > > > > I would appreciate if instead of a general and fuzzy > > referency you would > > > > quote > > > > the document page and paragraph. > > > > > > > > Julo > > > > > > > > "Douglas Otis" on 28/08/2000 19:00:56 > > > > > > > > Please respond to "Douglas Otis" > > > > > > > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > > > > cc: > > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > Julo, > > > > > > > > Is it compliant to assume knowledge of the TCP frame by the iSCSI > > > > application? In many places within iSCSI discussions, > references are > > > made > > > > of packets, frames, and datagrams as an element within iSCSI. If > > > creating > > > > frame alignment is your intent, rather than corrupting TCP > > APIs, perhaps > > > > you > > > > should consider SCTP. Otherwise, such discussions are little > > more than > > > a > > > > wink and a nod at creating a frame aligned TCP. How do you > > > > advise a change > > > > to the TCP API to resolve frame alignment as you have just > suggested? > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > > julian_satran@il.ibm.com > > > > > Sent: Saturday, August 26, 2000 2:28 AM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > > > > > > Steph, > > > > > > > > > > I assume that your hardware knows when it is going to > send the last > > > SCSI > > > > > datagram. > > > > > In this case it can insert the status in the datagram header (not > at > > > the > > > > > end) - > > > > > (and BTW that is how the current draft assumes that things will > > > > be done). > > > > > > > > > > If you have to send sense-data (bad status) then you will send > > > > a separate > > > > > datagram. > > > > > > > > > > Julo > > > > > > > > > > Stephen Bailey on 25/08/2000 22:12:09 > > > > > > > > > > Please respond to Stephen Bailey > > > > > > > > > > To: ips@ece.cmu.edu > > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > > > > > > > > > > > Is there anything preventing your hypothetical hardware > > implementor > > > > > > to send always good status within the last block of data? > > > > > > > > > > It depends upon the RDMA mechanism you are using. I admit > > that I have > > > > > not studied them in detail, other than ST of course, which is > > > > > essentially an transport protocol based upon a particular RDMA > > > > > mechanism. I will try to do so soon to determine if I'm all wet. > > > > > > > > > > However, assuming that the RDMA mechanism operates on a > per-datagram > > > > > basis, I can only imagine that you will not be able to append > > > > > `general delivery' data to the end of an RDMA datagram. In > > this case, > > > > > you will need a separate datagram to ensure that status is > delivered > > > > > through a separate path from the data. > > > > > > > > > > So, while I think concatenating status and data in its > > general form is > > > > > not a good idea, a good-status fast path (like the success bit) is > > > > > definitely the right way to think about it. Nonsuccess > > SCSI status is > > > > > so rare that any compromise you can make in the nonsuccess path to > > > > > make the success path go faster is worth it. > > > > > > > > > > Steph > > > > > > From owner-ips@ECE.cmu.edu Wed Aug 30 22:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14966; 30 Aug 2000 18:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA28543 for ips-outgoing; Wed, 30 Aug 2000 17:20:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com ([171.64.79.35]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA28511 for ; Wed, 30 Aug 2000 17:20:12 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 055568E9B; Wed, 30 Aug 2000 14:17:30 -0700 (PDT) To: IPS Reflector Subject: Re: iSCSI Autosense References: <39AD0203.4EFA4947@compuserve.com> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 30 Aug 2000 14:17:30 -0700 In-Reply-To: Ralph Weber's message of "Wed, 30 Aug 2000 07:45:55 -0500" Message-ID: Lines: 19 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Why would an initiator not want to use autosense? I believe the only reason is that the initiator doesn't want Contingent Allegiance to be cleared automatically. Why would an initiator not want Contingent Allegiance to be cleared automatically? 1) Initiator wants to issue a command other than REQUEST SENSE while the target is in Contingent Allegiance. 2) Initiators rely on the freezing of the target as a means of synchronization. I don't know of any implementation that relies on 1 or 2. Does anybody? Thanks, -Costa From owner-ips@ECE.cmu.edu Wed Aug 30 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15235; 30 Aug 2000 19:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA01864 for ips-outgoing; Wed, 30 Aug 2000 18:29:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA01860 for ; Wed, 30 Aug 2000 18:29:04 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id PAA06697; Wed, 30 Aug 2000 15:21:02 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA01198; Wed, 30 Aug 2000 15:18:27 -0700 From: "Douglas Otis" To: Cc: Subject: RE: iSCSI Autosense Date: Wed, 30 Aug 2000 15:30:50 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <39AD5633.956F4B3C@compuserve.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, Perhaps I should say Mr. SCSI as I would not wish to slander obvious knowledge even if I may disagree. I agree ACA provides desired interlocks and Autosense is also highly desired. I was not concerned about disk drives as these products are easily found supporting these standards and represent no change to existing software. Although your emulation description approximates ACA with CA devices, it is not as simple as not doing it at all in cases where it is not needed. For the odd device that does run one command per nexus and where such use is not a horrific bottleneck and the removal of Autosense leaves the operation of the device unchanged, why not refuse Autosense? Loaders, tape and every other odd widget you can imagine may fall into that CA category. Mucking with ACA emulation seems wrong in these cases where this fig leaf is enough. By creating an Autosense refusal, at least those such as yourself wishing to have a pure environment can enforce such desires. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ralph Weber > Sent: Wednesday, August 30, 2000 11:45 AM > Cc: ips@ece.cmu.edu > Subject: Re: iSCSI Autosense > > > Julo, > > } How are volume change events supposed to be treated with CA, > } ACA and autosense... > > There are two ways to report a Unit Attention condition: > > 1) Asynchronous Event Reporting > 2) Use of a Unit Attention CHECK CONDITION > > Asynchronous Event Reporting is optional for both targets and > initiators, and must be enabled by the initiator via the Control > mode page before it can be used. I have assumed that 3.6 in > draft-satran-iscsi-01.txt is somehow related to AER, and have > yet to work through the details. > > The use of a Unit Attention CHECK CONDITION follows all the > same rules as any other CHECK CONDITION with respect to CA, > Autosense, and ACA. When this mechanism is used (which is > far and away the most common case), there are no functional > differences between a Unit Attention CHECK CONDITION and a > Hardware Error CHECK CONDITION. > > I'll provide detailed answers to detailed questions, but the > above information seems like about all that can be said in > response to the question raised. > > } (or for that matter any Async. Event that intends > } to convey to an initiator state information that the initiator > } should be aware of before sending commands - i.e. how is state > } sync. achieved)? > > I believe that the vast majority of asynchronous events do > not require the initiator to be aware of them before other > commands are sent. That is, I believe that 'volume change' > is the exception not the rule and somebody is going to have > to produce a substantial list of exceptions before I'm likely > to change my opinion. > > If indeed 'volume change' is the exception, then I have no > problems with according it exceptional handling, as in > forcing logouts (or whatever is equivalent) for all initiators > as a response. > > } I was under the impression that this is one of vestigial uses > } of CA. > > I believe that the impression is incorrect and I have explained > at great length why CA offers not even a fig leaf's worth of > protection. I regret that I cannot argue against this incorrect > impression any more forcefully than I already have. > > } It is obvious that autosense is fine but until I can't(sic) be > } convinced that it is completely orthogonal to CA use (or to some > } other mechanism that can assure state sync) I would be reluctant > } to force it. > > CA is NOT orthogonal to Autosense. They are two mechanisms for > achieving the same end, which is to ensure that the sense data > for a CHECK CONDITION reaches the initiator. However, CA is > based on assumptions that apply ONLY to the interlocked parallel > SCSI bus. Therefore, the only mechanism that is available to > a packetized protocol is Autosense. > > Combining CA with a packetized protocol does not improve reliability > or state synchronization between initiator and target. Exactly the > opposite is true, CA depends for is proper operation on a level of > state synchronization between initiator and target that is available > ONLY on the interlocked parallel SCSI bus. Combining CA with a > packetized protocol presents a fictitious image of synchronization, > that's all it does. > > } If state sync. is provided by some other means that I will be happy > } to oblige and assume autosense is always there. > > ACA provides state synchronization between initiators and targets > and I have not contested support for ACA in iSCSI. In fact, as > you can see from my response this morning to Douglas Otis, I have > reviewed the ACA support provisions in the iSCSI draft and verified > that they are appropriate. > > I submit that ACA is the 'other means' that you have requested. > > Thanks. > > Ralph Weber > ENDL Texas > > > From owner-ips@ECE.cmu.edu Thu Aug 31 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15474; 30 Aug 2000 20:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA03459 for ips-outgoing; Wed, 30 Aug 2000 19:14:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA03453 for ; Wed, 30 Aug 2000 19:14:50 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id QAA21012; Wed, 30 Aug 2000 16:14:41 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Wed, 30 Aug 2000 16:14:37 -0700 Message-ID: From: Jim McGrath To: "'Mark Bradley'" , Jim McGrath , julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: what's in a name Date: Wed, 30 Aug 2000 16:14:38 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Actually, I misspoke. We used a Service Mark precisely because it was feared that someone else had a trademark on the term (Ultra SCSI). Essentially using SM allows you a way to avoid any infringement on a TM of the same name (the distinction between good and service often being castable in the eye of the beholder). Jim -----Original Message----- From: Mark Bradley [mailto:markb@indstorage.com] Sent: Wednesday, August 30, 2000 10:51 AM To: Jim McGrath; julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: RE: what's in a name This is incorrect information. There is no difference between SM and TM except that one applies to a service and the other to a product. -- markb > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jim McGrath > Sent: Tuesday, August 29, 2000 9:05 PM > To: 'julian_satran@il.ibm.com'; ips@ece.cmu.edu > Subject: RE: what's in a name > > > > You file iSCSI as a Service mark (SM). That essentially reserves the name > so no one else can trademark it, but allows other to use it (if you TM the > name then you have to defend it if others use it or you lose the TM - the > Xerox problem). > > I recollect that we did this with Ultra SCSI and it worked ok. > > Jim > > > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, August 29, 2000 7:31 AM > To: ips@ece.cmu.edu > Subject: what's in a name > > > > > Yaron Klein just pointed out to me tha Virtual Private SAN and VPS are TM > of > Pathlight (www.pathlight.com) and so are SAN Gateway and SAN router. > > Costa earlier pointed out that iSCSI is the name of a cable for iMAC (but > not a trademark > as far as I can tell). > > Can somebody tell us how to protect at least iSCSI from TM claims? > > Can the Internet Society do it (before it is too late)? > > Julo > From owner-ips@ECE.cmu.edu Thu Aug 31 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15410; 30 Aug 2000 20:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA03548 for ips-outgoing; Wed, 30 Aug 2000 19:17:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraab.compuserve.com (ds-img-rel-2.compuserve.com [149.174.206.155]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA03542 for ; Wed, 30 Aug 2000 19:17:32 -0400 (EDT) Received: (from mailgate@localhost) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id TAA05887 for ips@ece.cmu.edu; Wed, 30 Aug 2000 19:17:02 -0400 (EDT) Received: from compuserve.com (dal-qbu-zow-vty8.as.wcom.net [216.192.250.8]) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id TAA05868 for ; Wed, 30 Aug 2000 19:16:52 -0400 (EDT) Message-ID: <39AD9617.8122E3EF@compuserve.com> Date: Wed, 30 Aug 2000 18:17:43 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas, So, let's see if I understand what you are proposing: 1) Keep the A bit 2) Restrict usage of A=1 to those cases where the only Task Attribute value supported is 0 (Untagged) 3) In all other cases the receipt of a SCSI Command packet with A=1 would be cause for a protocol error That being the case, there are a couple of editorial fixes needed at the end of 3.2.1 in draft-satran-iscsi-01.txt as well: a) Remove the reference to SAM-2 in the description of the A bit. There is nothing in SAM-2 that will be even remotely helpful to the reader. b) Change the last sentence of the clause as follows to clarify the obligations of the initiator: "If autosense is turned off, the initiator must explicitly request transfer of the sense data by sending a REQUEST SENSE command as the first command delivered to the target after a command has completed with a CHECK CONDITION status." Good grief. For a group that prides itself on limiting the options in their standards, you all sure do cling to them with Visegrip(TM) tenacity. Thanks. Ralph Weber ENDL Texas Douglas Otis wrote: > > Ralph, > > Perhaps I should say Mr. SCSI as I would not wish to slander obvious > knowledge even if I may disagree. I agree ACA provides desired interlocks > and Autosense is also highly desired. I was not concerned about disk drives > as these products are easily found supporting these standards and represent > no change to existing software. Although your emulation description > approximates ACA with CA devices, it is not as simple as not doing it at all > in cases where it is not needed. For the odd device that does run one > command per nexus and where such use is not a horrific bottleneck and the > removal of Autosense leaves the operation of the device unchanged, why not > refuse Autosense? Loaders, tape and every other odd widget you can imagine > may fall into that CA category. Mucking with ACA emulation seems wrong in > these cases where this fig leaf is enough. By creating an Autosense > refusal, at least those such as yourself wishing to have a pure environment > can enforce such desires. > > Doug From owner-ips@ECE.cmu.edu Thu Aug 31 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16152; 30 Aug 2000 22:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA06974 for ips-outgoing; Wed, 30 Aug 2000 21:38:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel2.hp.com (cosrel2.hp.com [156.153.255.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA06970 for ; Wed, 30 Aug 2000 21:38:08 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by cosrel2.hp.com (Postfix) with ESMTP id 27083904; Wed, 30 Aug 2000 19:37:47 -0600 (MDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id SAA23293; Wed, 30 Aug 2000 18:37:57 -0700 (PDT) X-OpenMail-Hops: 1 Date: Wed, 30 Aug 2000 18:37:50 -0700 Message-Id: Subject: RE: RE: iSCSI Autosense MIME-Version: 1.0 To: dotis@sanlight.net, ENDL_TX@computer.org Cc: ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I really don't know where keeping the A bit helps anyone. All that is saying is that the sense data is returned with the command status. First of all, we should have an enumeration of modern (is it reasonable to go back only 3 yrs?) devices that do not support autosense. This would give this some realistic foundation. If we cannot list a reasonable number (?) of devices that don't support autosense, then I think it is a meaningless discussion. In the future, it seems unlikely that devices will not support autosense. Then there is the other aspect 1. A device that does not support autosense is connected through a bridge. In this case, as mentioned before, the bridge can hide the lack of autosense support by the device. 2. There is no bridge and the device supports a native iSCSI interface. In this case the it is probably a new rev of the device and probably a small incremental effort to support autosense. If any case, the software/firmware driver (of course there has to be one managing the iSCSI chip/adpater whatever) can provide the necessary functionality. It seems that if the lack of autosense support has been very uncommon, and is rapidly dying, it is better to shift the burden of "providing autosense" to the oddball cases rather than spread the malaise. Anyhow, without a list of devices that do not support autosense, my practical engineer side is not going to be convinced that it makes any sense to keep the A bit. Somesh > -----Original Message----- > From: dotis@sanlight.net [mailto:dotis@sanlight.net] > Sent: Wednesday, August 30, 2000 3:31 PM > To: ENDL_TX@computer.org > Cc: dotis@sanlight.net; ips@ece.cmu.edu > Subject: FW: RE: iSCSI Autosense > > > Ralph, > > Perhaps I should say Mr. SCSI as I would not wish to slander obvious > knowledge even if I may disagree. I agree ACA provides > desired interlocks > and Autosense is also highly desired. I was not concerned > about disk drives > as these products are easily found supporting these standards > and represent > no change to existing software. Although your emulation description > approximates ACA with CA devices, it is not as simple as not > doing it at all > in cases where it is not needed. For the odd device that does run one > command per nexus and where such use is not a horrific > bottleneck and the > removal of Autosense leaves the operation of the device > unchanged, why not > refuse Autosense? Loaders, tape and every other odd widget > you can imagine > may fall into that CA category. Mucking with ACA emulation > seems wrong in > these cases where this fig leaf is enough. By creating an Autosense > refusal, at least those such as yourself wishing to have a > pure environment > can enforce such desires. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Ralph Weber > > Sent: Wednesday, August 30, 2000 11:45 AM > > Cc: ips@ece.cmu.edu > > Subject: Re: iSCSI Autosense > > > > > > Julo, > > > > } How are volume change events supposed to be treated with CA, > > } ACA and autosense... > > > > There are two ways to report a Unit Attention condition: > > > > 1) Asynchronous Event Reporting > > 2) Use of a Unit Attention CHECK CONDITION > > > > Asynchronous Event Reporting is optional for both targets and > > initiators, and must be enabled by the initiator via the Control > > mode page before it can be used. I have assumed that 3.6 in > > draft-satran-iscsi-01.txt is somehow related to AER, and have > > yet to work through the details. > > > > The use of a Unit Attention CHECK CONDITION follows all the > > same rules as any other CHECK CONDITION with respect to CA, > > Autosense, and ACA. When this mechanism is used (which is > > far and away the most common case), there are no functional > > differences between a Unit Attention CHECK CONDITION and a > > Hardware Error CHECK CONDITION. > > > > I'll provide detailed answers to detailed questions, but the > > above information seems like about all that can be said in > > response to the question raised. > > > > } (or for that matter any Async. Event that intends > > } to convey to an initiator state information that the initiator > > } should be aware of before sending commands - i.e. how is state > > } sync. achieved)? > > > > I believe that the vast majority of asynchronous events do > > not require the initiator to be aware of them before other > > commands are sent. That is, I believe that 'volume change' > > is the exception not the rule and somebody is going to have > > to produce a substantial list of exceptions before I'm likely > > to change my opinion. > > > > If indeed 'volume change' is the exception, then I have no > > problems with according it exceptional handling, as in > > forcing logouts (or whatever is equivalent) for all initiators > > as a response. > > > > } I was under the impression that this is one of vestigial uses > > } of CA. > > > > I believe that the impression is incorrect and I have explained > > at great length why CA offers not even a fig leaf's worth of > > protection. I regret that I cannot argue against this incorrect > > impression any more forcefully than I already have. > > > > } It is obvious that autosense is fine but until I can't(sic) be > > } convinced that it is completely orthogonal to CA use (or to some > > } other mechanism that can assure state sync) I would be reluctant > > } to force it. > > > > CA is NOT orthogonal to Autosense. They are two mechanisms for > > achieving the same end, which is to ensure that the sense data > > for a CHECK CONDITION reaches the initiator. However, CA is > > based on assumptions that apply ONLY to the interlocked parallel > > SCSI bus. Therefore, the only mechanism that is available to > > a packetized protocol is Autosense. > > > > Combining CA with a packetized protocol does not improve reliability > > or state synchronization between initiator and target. Exactly the > > opposite is true, CA depends for is proper operation on a level of > > state synchronization between initiator and target that is available > > ONLY on the interlocked parallel SCSI bus. Combining CA with a > > packetized protocol presents a fictitious image of synchronization, > > that's all it does. > > > > } If state sync. is provided by some other means that I > will be happy > > } to oblige and assume autosense is always there. > > > > ACA provides state synchronization between initiators and targets > > and I have not contested support for ACA in iSCSI. In fact, as > > you can see from my response this morning to Douglas Otis, I have > > reviewed the ACA support provisions in the iSCSI draft and verified > > that they are appropriate. > > > > I submit that ACA is the 'other means' that you have requested. > > > > Thanks. > > > > Ralph Weber > > ENDL Texas > > > > > > > From owner-ips@ECE.cmu.edu Thu Aug 31 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16141; 30 Aug 2000 22:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA07150 for ips-outgoing; Wed, 30 Aug 2000 21:44:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA07145 for ; Wed, 30 Aug 2000 21:44:17 -0400 (EDT) Received: from cx926635a (cx926635-a.msnv1.occa.home.com [24.10.148.23]) by lightsand.com (8.9.3+Sun/8.9.1) with ESMTP id SAA15350; Wed, 30 Aug 2000 18:43:07 -0700 (PDT) Message-ID: <005301c012ed$9193b720$17940a18@msnv1.occa.home.com> Reply-To: "Raj Bhagwat" From: "Raj Bhagwat" To: "Douglas Otis" Cc: References: Subject: Re: Latest DMP specifications? Date: Wed, 30 Aug 2000 18:48:31 -0700 Organization: LightSand Communications MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, DMP is a backbone routing protocol for FC switching. It is part of the ANSI-T11 FC-SW2 specification. The latest draft (version 4.5) is available at .. ftp://209.26.30.221/t11/pub/fc/sw-2/00-144v2.pdf Regards, Raj Bhagwat LightSand Communications ----- Original Message ----- From: Douglas Otis To: fc Sent: Wednesday, August 30, 2000 3:32 PM Subject: Latest DMP specifications? > * > * From the fc reflector, posted by: > * "Douglas Otis" > * > Is there a working draft for DMP beyond rev 0? > > Doug > From owner-ips@ECE.cmu.edu Thu Aug 31 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16339; 30 Aug 2000 23:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA08392 for ips-outgoing; Wed, 30 Aug 2000 22:39:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from proxy1.it.stsj.seagate.com (ns1.seagate.com [204.160.183.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA03012 for ; Wed, 30 Aug 2000 18:58:46 -0400 (EDT) From: Paul.A.Suhler@seagate.com Received: (from smap) by proxy1.it.stsj.seagate.com id WAA05040 for ; Wed, 30 Aug 2000 22:58:09 GMT Received: from auth0.it.stsj.seagate.com(192.168.109.16) by proxy1.it.stsj.seagate.com via smap (V5.0) id xma004810; Wed, 30 Aug 00 22:57:04 GMT Received: from sv-gw1.stsv.seagate.com (sv-gw1.stsv.seagate.com [134.204.14.95]) by auth0.it.stsj.seagate.com with ESMTP id WAA09604 for ; Wed, 30 Aug 2000 22:57:04 GMT Subject: RE: iSCSI Autosense To: ips@ece.cmu.edu Date: Wed, 30 Aug 2000 15:56:51 -0700 Message-ID: X-MIMETrack: Serialize by Router on SV-GW1/Seagate Internet(Release 5.0.2a (Intl)|23 November 1999) at 08/30/2000 03:57:04 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I would look at the need a bit differently. How many native iSCSI devices will actually be built without autosense? I'd expect that any devices incapable of autosense will be legacy parallel SCSI target which can be hidden behind a bridge, as happened with Fibre Channel. That can handle the functionality. Doug, you know your market better than I do. Are there people who want to slap on an Ethernet/iSCSI front end without revising their task manager? Cheers, Paul Suhler Seagate Removable Storage Solutions "Douglas Otis" @ece.cmu.edu on 08/30/2000 15:30:50 Sent by: owner-ips@ece.cmu.edu To: cc: Subject: RE: iSCSI Autosense Ralph, Perhaps I should say Mr. SCSI as I would not wish to slander obvious knowledge even if I may disagree. I agree ACA provides desired interlocks and Autosense is also highly desired. I was not concerned about disk drives as these products are easily found supporting these standards and represent no change to existing software. Although your emulation description approximates ACA with CA devices, it is not as simple as not doing it at all in cases where it is not needed. For the odd device that does run one command per nexus and where such use is not a horrific bottleneck and the removal of Autosense leaves the operation of the device unchanged, why not refuse Autosense? Loaders, tape and every other odd widget you can imagine may fall into that CA category. Mucking with ACA emulation seems wrong in these cases where this fig leaf is enough. By creating an Autosense refusal, at least those such as yourself wishing to have a pure environment can enforce such desires. Doug From owner-ips@ECE.cmu.edu Thu Aug 31 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16421; 30 Aug 2000 23:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA08211 for ips-outgoing; Wed, 30 Aug 2000 22:32:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA08205 for ; Wed, 30 Aug 2000 22:32:07 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id TAA06790; Wed, 30 Aug 2000 19:24:11 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id TAA01829; Wed, 30 Aug 2000 19:21:38 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI Autosense Date: Wed, 30 Aug 2000 19:34:03 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <39AD9617.8122E3EF@compuserve.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, Few wish to mess with odd devices that are talking SCSI in some unknown fashion. To add ACA emulation at an adapter and then fiddle code in the original application to send now expected ACA commands seems counter productive with these low reward tasks should there be a prior reliance on CA. SAM simply suggests that Sense is not returned as an indication of no Autosense. An explicit refusal could signal the presents of some legacy hardware best left untouched and forgotten. It should mean less work by leaving it to the application. This is not a matter of determining which devices support Autosense but rather which devices support ACA or may rely on CA. The adapter should be able to work out if refusal is desired. Perhaps a response to login could provide this feedback. An alternative is to not send Sense without notification and leave everything as is with this understanding. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ralph Weber > Sent: Wednesday, August 30, 2000 4:18 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI Autosense > > > Douglas, > > So, let's see if I understand what you are proposing: > > 1) Keep the A bit > 2) Restrict usage of A=1 to those cases where the only > Task Attribute value supported is 0 (Untagged) > 3) In all other cases the receipt of a SCSI Command packet > with A=1 would be cause for a protocol error > > That being the case, there are a couple of editorial fixes > needed at the end of 3.2.1 in draft-satran-iscsi-01.txt as well: > > a) Remove the reference to SAM-2 in the description of the > A bit. There is nothing in SAM-2 that will be even > remotely helpful to the reader. > b) Change the last sentence of the clause as follows to > clarify the obligations of the initiator: > > "If autosense is turned off, the initiator must explicitly request > transfer of the sense data by sending a REQUEST SENSE command as > the first command delivered to the target after a command has > completed with a CHECK CONDITION status." > > Good grief. For a group that prides itself on limiting the > options in their > standards, you all sure do cling to them with Visegrip(TM) tenacity. > > Thanks. > > Ralph Weber > ENDL Texas > > Douglas Otis wrote: > > > > > Ralph, > > > > Perhaps I should say Mr. SCSI as I would not wish to slander obvious > > knowledge even if I may disagree. I agree ACA provides desired > interlocks > > and Autosense is also highly desired. I was not concerned about > disk drives > > as these products are easily found supporting these standards > and represent > > no change to existing software. Although your emulation description > > approximates ACA with CA devices, it is not as simple as not > doing it at all > > in cases where it is not needed. For the odd device that does run one > > command per nexus and where such use is not a horrific > bottleneck and the > > removal of Autosense leaves the operation of the device > unchanged, why not > > refuse Autosense? Loaders, tape and every other odd widget you > can imagine > > may fall into that CA category. Mucking with ACA emulation > seems wrong in > > these cases where this fig leaf is enough. By creating an Autosense > > refusal, at least those such as yourself wishing to have a pure > environment > > can enforce such desires. > > > > Doug > From owner-ips@ECE.cmu.edu Thu Aug 31 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16337; 30 Aug 2000 23:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA08229 for ips-outgoing; Wed, 30 Aug 2000 22:33:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ipperformance.com (cooper.gilhooley.com [207.8.92.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA10971 for ; Wed, 30 Aug 2000 10:37:09 -0400 (EDT) From: rbg@ipperf.com Received: from rbg.gilhooley.com (rbg [10.1.0.66]) by ipperformance.com (8.9.3/8.9.3) with ESMTP id JAA90939 for ; Wed, 30 Aug 2000 09:36:57 -0500 (CDT) (envelope-from rbg@ipperf.com) Received: from localhost (localhost [127.0.0.1]) by rbg.gilhooley.com (8.9.3/8.9.3) with ESMTP id JAA83053 for ; Wed, 30 Aug 2000 09:35:48 -0500 (CDT) (envelope-from rbg@ipperf.com) Date: Wed, 30 Aug 2000 09:35:47 -0500 (CDT) Message-Id: <20000830.093547.22032990.rbg@ipperformance.com> To: ips@ece.cmu.edu Subject: Re: (FCoverIP) SCTP extension. In-Reply-To: References: X-Mailer: Mew version 1.95b43 on Emacs 20.4 / Mule 4.0 (HANANOEN) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Is there a SCTP mail list ? > On Tue, 29 Aug 2000 16:52:03 -0700 you said: > As a possible extension to SCTP to deal with the limitation of FC, perhaps a > trinary setting of the retransmission could be made: Unlimited, 1, or 0. > By allowing at least one retry, most lost frames would be recovered within a > constrained window of time. The danger with FC is an underlying transport > overlap of recovery mechanisms. One other consideration is the size of the > buffers at each end of SCTP as this would also skew a time window but would > be required to prevent loss at this point as well. > > BTW, what is the latest T11 DMP specification? > > Doug > _____________________________________________________________ Robert Gordon rbg@ipperf.com IP Performance, Inc Austin, Texas. From owner-ips@ECE.cmu.edu Thu Aug 31 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16373; 30 Aug 2000 23:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA08241 for ips-outgoing; Wed, 30 Aug 2000 22:34:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA12231 for ; Wed, 30 Aug 2000 11:07:51 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Wed, 30 Aug 2000 08:08:46 -0700 Message-ID: From: Robert Snively To: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Date: Wed, 30 Aug 2000 08:06:24 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph Weber is right in every respect and correctly reflects the SCSI architecture to you. It is that architecture that iSCSI is trying to retain unchanged so that the wide variety of SCSI-based applications and operating systems can be fully utilized by iSCSI devices. Bob From owner-ips@ECE.cmu.edu Thu Aug 31 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16973; 31 Aug 2000 1:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA10816 for ips-outgoing; Thu, 31 Aug 2000 00:43:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA10810 for ; Thu, 31 Aug 2000 00:43:38 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G0500HL73N1QF@mta6.snfc21.pbi.net> for IPS@ECE.CMU.edu; Wed, 30 Aug 2000 21:40:14 -0700 (PDT) Date: Wed, 30 Aug 2000 21:38:43 -0700 From: Peter Johansson Subject: RE: iSCSI Autosense In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000830213115.00bb6740@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed References: <39AD9617.8122E3EF@compuserve.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:34 PM 8/30/00, Douglas Otis wrote: >Few wish to mess with odd devices that are talking SCSI in some unknown >fashion. To add ACA emulation at an adapter and then fiddle code in the >original application to send now expected ACA commands seems counter >productive with these low reward tasks should there be a prior reliance on CA. Before spending much time or energy worrying about these hypothetical devices, what are the design goals of iSCSI? Is one of its important goals support for crusty devices which may or may not exist? Are we trying to enable support, over IP, for exceptions to the rule? Or is iSCSI focused on a different goal, that of efficient, high-performance connection to mass storage devices (both disk and tape)? It's not as if iSCSI will be exclusively tailored to mass storage, but one has to ask "Why is iSCSI a hot topic today?" For the sake of some unidentified "odd devices"? Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Thu Aug 31 07:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17375; 31 Aug 2000 3:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA11982 for ips-outgoing; Thu, 31 Aug 2000 01:54:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA11978 for ; Thu, 31 Aug 2000 01:54:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id HAA64512 for ; Thu, 31 Aug 2000 07:53:49 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA40726 for ; Thu, 31 Aug 2000 07:53:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694C.0020634C ; Thu, 31 Aug 2000 07:53:45 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 31 Aug 2000 08:51:40 +0300 Subject: autosense - CA & ACA Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, I think that we can remove the Autosense bit and request every iSCSI target to implement autosense. Thanks Ralph for the clarifications. It looks like whatever is needed to sync. state is offered by ACA. Julo From owner-ips@ECE.cmu.edu Thu Aug 31 11:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18754; 31 Aug 2000 7:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA24420 for ips-outgoing; Thu, 31 Aug 2000 06:35:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sphmraaa.compuserve.com (hs-img-rel-1.compuserve.com [149.174.177.156]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA24413 for ; Thu, 31 Aug 2000 06:35:27 -0400 (EDT) Received: (from mailgate@localhost) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id GAA02635 for ips@ece.cmu.edu; Thu, 31 Aug 2000 06:34:57 -0400 (EDT) Received: from compuserve.com (hil-qbu-ppk-vty16.as.wcom.net [206.175.99.16]) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id GAA02607 for ; Thu, 31 Aug 2000 06:34:51 -0400 (EDT) Message-ID: <39AE3575.61B79B33@compuserve.com> Date: Thu, 31 Aug 2000 05:37:41 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas, Since your response contains nothing that disputes the changes proposed for draft-satran-iscsi-01.txt, I assume that you agree with the changes. You are to be commended for your consistency. Your Visegrip(TM) on the option remains without a hint of diminution. Thanks. Ralph Weber ENDL Texas Douglas Otis wrote: > > Ralph, > > Few wish to mess with odd devices that are talking SCSI in some unknown > fashion. To add ACA emulation at an adapter and then fiddle code in the > original application to send now expected ACA commands seems counter > productive with these low reward tasks should there be a prior reliance on > CA. SAM simply suggests that Sense is not returned as an indication of no > Autosense. An explicit refusal could signal the presents of some legacy > hardware best left untouched and forgotten. It should mean less work by > leaving it to the application. This is not a matter of determining which > devices support Autosense but rather which devices support ACA or may rely > on CA. The adapter should be able to work out if refusal is desired. > Perhaps a response to login could provide this feedback. An alternative is > to not send Sense without notification and leave everything as is with this > understanding. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Ralph Weber > > Sent: Wednesday, August 30, 2000 4:18 PM > > To: ips@ece.cmu.edu > > Subject: Re: iSCSI Autosense > > > > > > Douglas, > > > > So, let's see if I understand what you are proposing: > > > > 1) Keep the A bit > > 2) Restrict usage of A=1 to those cases where the only > > Task Attribute value supported is 0 (Untagged) > > 3) In all other cases the receipt of a SCSI Command packet > > with A=1 would be cause for a protocol error > > > > That being the case, there are a couple of editorial fixes > > needed at the end of 3.2.1 in draft-satran-iscsi-01.txt as well: > > > > a) Remove the reference to SAM-2 in the description of the > > A bit. There is nothing in SAM-2 that will be even > > remotely helpful to the reader. > > b) Change the last sentence of the clause as follows to > > clarify the obligations of the initiator: > > > > "If autosense is turned off, the initiator must explicitly request > > transfer of the sense data by sending a REQUEST SENSE command as > > the first command delivered to the target after a command has > > completed with a CHECK CONDITION status." > > > > Good grief. For a group that prides itself on limiting the > > options in their > > standards, you all sure do cling to them with Visegrip(TM) tenacity. > > > > Thanks. > > > > Ralph Weber > > ENDL Texas > > > > Douglas Otis wrote: > > > > > > > > Ralph, > > > > > > Perhaps I should say Mr. SCSI as I would not wish to slander obvious > > > knowledge even if I may disagree. I agree ACA provides desired > > interlocks > > > and Autosense is also highly desired. I was not concerned about > > disk drives > > > as these products are easily found supporting these standards > > and represent > > > no change to existing software. Although your emulation description > > > approximates ACA with CA devices, it is not as simple as not > > doing it at all > > > in cases where it is not needed. For the odd device that does run one > > > command per nexus and where such use is not a horrific > > bottleneck and the > > > removal of Autosense leaves the operation of the device > > unchanged, why not > > > refuse Autosense? Loaders, tape and every other odd widget you > > can imagine > > > may fall into that CA category. Mucking with ACA emulation > > seems wrong in > > > these cases where this fig leaf is enough. By creating an Autosense > > > refusal, at least those such as yourself wishing to have a pure > > environment > > > can enforce such desires. > > > > > > Doug > > From owner-ips@ECE.cmu.edu Thu Aug 31 12:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18879; 31 Aug 2000 8:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA25125 for ips-outgoing; Thu, 31 Aug 2000 07:23:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA25121 for ; Thu, 31 Aug 2000 07:23:50 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA01085; Thu, 31 Aug 2000 06:24:15 -0500 Message-ID: <39AE405F.35331452@stewart.chicago.il.us> Date: Thu, 31 Aug 2000 06:24:15 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: rbg@ipperf.com CC: ips@ece.cmu.edu Subject: Re: (FCoverIP) SCTP extension. References: <20000830.093547.22032990.rbg@ipperformance.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk There is not per se a SCTP mailing list. SCTP is owned by the sigtran working group. The specification itself is in the RFC editors queue. There are several extensions for SCTP being worked upon ... i.e. u-sctp, and others. If you are interested, I would suggest the mailing lists sigtran and a companion sigtran-test and rspool all of which are on the nortelnetworks standards site. R rbg@ipperf.com wrote: > > Is there a SCTP mail list ? > > > On Tue, 29 Aug 2000 16:52:03 -0700 you said: > > As a possible extension to SCTP to deal with the limitation of FC, perhaps a > > trinary setting of the retransmission could be made: Unlimited, 1, or 0. > > By allowing at least one retry, most lost frames would be recovered within a > > constrained window of time. The danger with FC is an underlying transport > > overlap of recovery mechanisms. One other consideration is the size of the > > buffers at each end of SCTP as this would also skew a time window but would > > be required to prevent loss at this point as well. > > > > BTW, what is the latest T11 DMP specification? > > > > Doug > > > _____________________________________________________________ > Robert Gordon rbg@ipperf.com > IP Performance, Inc > Austin, Texas. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 From owner-ips@ECE.cmu.edu Thu Aug 31 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19574; 31 Aug 2000 11:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA28949 for ips-outgoing; Thu, 31 Aug 2000 09:46:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA28944 for ; Thu, 31 Aug 2000 09:46:18 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 08D375892 for ; Thu, 31 Aug 2000 09:46:18 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id GAA26504 for ; Thu, 31 Aug 2000 06:46:16 -0700 (PDT) Received: from agilent.com ([15.26.30.187]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA44D3; Thu, 31 Aug 2000 06:46:11 -0700 Message-ID: <39AE61C3.E464F148@agilent.com> Date: Thu, 31 Aug 2000 06:46:43 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: draft-csapuntz-tcpmsgbnd-00.txt References: <200008311049.GAA03197@ietf.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I was toying around with an idea similar to this. My idea is, have the first byte in the iSCSI header be "reserved". Whenever an iSCSI message (n) is sent, this first byte of the "next" message (n+1) would also be sent, marked urgent. The result is that the TCP segment(s) for the message (n) will have the urgent pointer pointing to the reserved byte of the next message (n+1), indicating the beginning of the next message. Or each iSCSI message could be sent with the "last" byte of the message marked urgent, having the same result. The urgent pointer would then point to the last byte of each iSCSI message, thus indicating where in the byte stream the next iSCSI message header begins. Comments? -Matt Wakeley Agilent Technologies Internet-Drafts@ietf.org wrote: > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > Title : TCP Message Boundary Option > Author(s) : C. Sapuntzakis > Filename : draft-csapuntz-tcpmsgbnd-00.txt > Pages : 4 > Date : 30-Aug-00 > > TCP does not have a mechanism for specifying message boundaries in > a stream. This I-D describes a new TCP option and a new way of > using the TCP urgent field to specify message boundaries in the > stream. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-csapuntz-tcpmsgbnd-00.txt From owner-ips@ECE.cmu.edu Thu Aug 31 16:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19817; 31 Aug 2000 12:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA01248 for ips-outgoing; Thu, 31 Aug 2000 10:47:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA01233 for ; Thu, 31 Aug 2000 10:47:22 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id HAA06898; Thu, 31 Aug 2000 07:39:30 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id HAA03580; Thu, 31 Aug 2000 07:36:58 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI Autosense Date: Thu, 31 Aug 2000 07:49:29 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Paul, The answer is in how broadly you see Ethernet SCSI extending. If in the business of making incredibly big machines, then rejecting Autosense is a nuisance. Keeping SCSI applications unchanged is important when deployed by companies promoting adapters. We've all been in the trench created by a new paradigm. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Paul.A.Suhler@seagate.com > Sent: Wednesday, August 30, 2000 3:57 PM > To: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense > > > > I would look at the need a bit differently. > > How many native iSCSI devices will actually be built without autosense? > I'd expect that any devices incapable of autosense will be legacy parallel > SCSI target which can be hidden behind a bridge, as happened with Fibre > Channel. That can handle the functionality. > > Doug, you know your market better than I do. Are there people who want to > slap on an Ethernet/iSCSI front end without revising their task manager? > > Cheers, > > Paul Suhler > > Seagate Removable Storage Solutions > > > > > > "Douglas Otis" @ece.cmu.edu on 08/30/2000 15:30:50 > > Sent by: owner-ips@ece.cmu.edu > > > To: > cc: > > Subject: RE: iSCSI Autosense > > > Ralph, > > Perhaps I should say Mr. SCSI as I would not wish to slander obvious > knowledge even if I may disagree. I agree ACA provides desired interlocks > and Autosense is also highly desired. I was not concerned about > disk drives > as these products are easily found supporting these standards and > represent > no change to existing software. Although your emulation description > approximates ACA with CA devices, it is not as simple as not doing it at > all > in cases where it is not needed. For the odd device that does run one > command per nexus and where such use is not a horrific bottleneck and the > removal of Autosense leaves the operation of the device unchanged, why not > refuse Autosense? Loaders, tape and every other odd widget you > can imagine > may fall into that CA category. Mucking with ACA emulation seems wrong in > these cases where this fig leaf is enough. By creating an Autosense > refusal, at least those such as yourself wishing to have a pure > environment > can enforce such desires. > > Doug > > From owner-ips@ECE.cmu.edu Thu Aug 31 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20060; 31 Aug 2000 12:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03043 for ips-outgoing; Thu, 31 Aug 2000 11:31:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from boston3.g4.net (boston3.G4.NET [216.177.0.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA28055 for ; Thu, 31 Aug 2000 09:22:24 -0400 (EDT) Received: from Trebia5 (ppp73.qcy.dialup.G4.net [140.186.53.73]) by boston3.g4.net (8.9.3/8.9.3) with SMTP id JAA52478 for ; Thu, 31 Aug 2000 09:25:28 -0400 (EDT) (envelope-from jssmart@nhinternet.com) From: "James Smart" To: "ips" Subject: RE: Target Reset handling Date: Thu, 31 Aug 2000 09:26:39 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal In-Reply-To: <39AD3EAC.77CDFAA6@compuserve.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, Either I grossly missed led you, or you misread me... > } Given connections with lots of outstanding traffic, I'd see > this as a more > } graceful reset procedure. It allows any outstanding i/o that may be > } completing while the TR is in transit (or queued for processing on the > } target) to do so, possibly lightening the load of i/o that has > to error to > } complete. This would potentially quicken the recovery time post reset. I > } would expect this to be more important as the "network" gets larger and > } longer. > > } Note: FCP does support this behavior. > > The concept of a target queuing I/O for processing across a target reset > boggles my mind. I'm having a really hard time believing that FCP goes > that far. I wasn't describing commands being queued across a reset - but rather, the additional (very small) time on the target side when it may have received the task mgmt command packet, but hasn't processed it yet (it's behind other commands received previously, or waiting for a deferred processing thread to deal with it). Basically more queuing/transmit time during which other i/o can be completing before the reset takes effect. My comment on FCP was that it allows the target to ack (send a response) to the Target Reset - also useful if it does not support the target reset task management function. > Checkout ftp://ftp.t10.org/t10/documaent.00/00-229r1.pdf. It's a proposal > in progress and has not been approved yet. But, the issue raised above > is not limited to iSCSI. Agreed. I really like the proposal. However, it's yet another optional item enabled by mode page. Allowing the iscsi target to respond the target reset function allows some additional flexibility. > > > } d) Given the history of long error recovery times in multi-initiator > } environments in both parallel scsi and fibre channel on BDR's/Target > } Reset's, any speed up in this area would be advantageous. > > The real problem is using Target Resets for purposes other than clearing > an obviously stuck target. T10 keeps trying to provided the right tools > for the right uses (e.g., Persistent Reservations) but at least one major > software vendor sees them as inconveniences instead of assistants. Agreed. Emulation of that old parallel scsi BDR via Target Reset and the other implications on reservations, etc has made it difficult for the OS drivers to leave it behind. -- James From owner-ips@ECE.cmu.edu Thu Aug 31 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20070; 31 Aug 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03127 for ips-outgoing; Thu, 31 Aug 2000 11:32:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from scallop.baynetworks.com (ns5.baynetworks.com [194.133.90.101]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03116 for ; Thu, 31 Aug 2000 11:32:23 -0400 (EDT) Received: from mailhost.BayNetworks.COM (h8754.s84f5.BayNetworks.COM [132.245.135.84]) by scallop.baynetworks.com (8.9.1/8.9.1) with ESMTP id RAA07230 for ; Thu, 31 Aug 2000 17:37:17 +0200 (MET DST) Received: from pobox.engeast.BayNetworks.COM (pobox.engeast.baynetworks.com [192.32.61.6]) by mailhost.BayNetworks.COM (8.9.1/8.8.8) with ESMTP id LAA05131 for ; Thu, 31 Aug 2000 11:31:46 -0400 (EDT) Received: from ftravost (dhcp223-237 [192.32.223.237]) by pobox.engeast.BayNetworks.COM (SMI-8.6/BNET-97/04/24-S) with SMTP id LAA17122; Thu, 31 Aug 2000 11:31:46 -0400 for Message-Id: <3.0.32.20000831112245.00e0d984@pobox.corpeast.baynetworks.com> X-Sender: ftravost@pobox.corpeast.baynetworks.com X-Mailer: Windows Eudora Pro Version 3.0 (32) Date: Thu, 31 Aug 2000 11:22:45 -0400 To: ips@ece.cmu.edu From: Franco Travostino Subject: Re: multiple connections Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk A bit of Related Work ... On the forever-young 1 vs. N TCP connections subject, you may be interested in the results shown in the '97 "An Application-Level Solution to TCP's Satellite Inefficiencies" paper (available at http://jarok.cs.ohiou.edu/projects/satellite/) and on Sally Floyd's comment hereafter. The intersection with satellite links is quite accidental, and the arguments/results apply above and beyond satellite links. -franco >>From majordom@ISI.EDU Fri Dec 6 23:05:58 1996 >To: fred@cisco.com (Fred Baker) >Cc: end2end-tf@isi.edu >Subject: Re: Related paper/re:satellites >Date: Fri, 06 Dec 1996 23:05:47 PST >From: Sally Floyd >Sender: owner-end2end-tf@ISI.EDU >Precedence: bulk >Content-Length: 2309 >X-Lines: 44 > >Fred - > >>You might be interested in reviewing this paper, which is what I'm >>discussing with Karen Hansen, Dan Shell, and the folks from Comsat. It >>relates to some TCP/Satellite work being done at NASA Lewis Research >>Center. > >Basically (from a quick read), the paper, on "An Application-Level >Solution to TCP's Satellite Inefficiencies", recommends breaking a >single TCP connection into multiple connections at the >application level, to increase throughput on satellite circuits. >It is not too surprising that this increases the TCP throughput, but it >is still not a good idea. For a single TCP connection, a single packet >drop results in the throughput for that connection being cut by half, >and then increased by roughly one packet per RTT. For a TCP connection >that has instead been separated into N different TCP connections, a >single packet drop results in one of the N connections, receiving >1/N-th of the total bandwidth, having its throughput cut in half. Thus >the bandwidth of the aggregate connection has its bandwidth reduced to >(1 - 1/(2N))-th of its former bandwidth - that is, the larger the value >for N, the smaller the aggregate bandwidth is cut. And then, because >each TCP connection continues to increase its congestion window by one >packet per RTT, for those TCP connections that have not yet reached the >receiver's advertised window, the aggregate TCP connections together >increase their bandwidth by up to N packets per RTT. > >Summarizing, splitting a TCP connection into N separate connections >simply increases the aggressiveness of the TCP congestion control. >Meaning that this TCP is now more likely to "steal" bandwidth from >other TCP connections in times of congestion. And increasing the >aggressiveness of the TCP congestion control too far (by choosing N too >large) is counterproductive even for the aggregate TCP connection, as >the paper shows. > >I would suggest that this is exactly the kind of development for which >[RED is needed]. > >- Sally > > > From owner-ips@ECE.cmu.edu Thu Aug 31 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20065; 31 Aug 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03808 for ips-outgoing; Thu, 31 Aug 2000 11:48:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from c014.sfo.cp.net (c014-h003.c014.sfo.cp.net [209.228.12.67]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id LAA03802 for ; Thu, 31 Aug 2000 11:48:02 -0400 (EDT) Received: (cpmta 19594 invoked from network); 31 Aug 2000 08:47:31 -0700 Received: from unknown (HELO gis.net) (216.41.37.127) by smtp.flashcom.net (209.228.12.67) with SMTP; 31 Aug 2000 08:47:31 -0700 X-Sent: 31 Aug 2000 15:47:31 GMT Message-ID: <39AE7DD0.BE45048F@gis.net> Date: Thu, 31 Aug 2000 11:46:24 -0400 From: Bill Main Reply-To: bill.main@bigfoot.com X-Mailer: Mozilla 4.5 [en]C-CCK-MCD compaq (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley CC: IPS Reflector Subject: Re: draft-csapuntz-tcpmsgbnd-00.txt References: <200008311049.GAA03197@ietf.org> <39AE61C3.E464F148@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk As a general rule, I would hope that we can keep all the "control" bits in the headers (versus "last" byte or trailers) to make it simpler for the Silicon to process it. -Bill Main Matt Wakeley wrote: > I was toying around with an idea similar to this. > > My idea is, have the first byte in the iSCSI header be "reserved". > Whenever an iSCSI message (n) is sent, this first byte of the "next" message (n+1) > would also be sent, marked urgent. The result is that the TCP segment(s) for the > message (n) will have the urgent pointer pointing to the reserved byte of the next > message (n+1), indicating the beginning of the next message. > > Or each iSCSI message could be sent with the "last" byte of the message marked > urgent, having the same result. The urgent pointer would then point to the last > byte of each iSCSI message, thus indicating where in the byte stream the next iSCSI > message header begins. > > Comments? > > -Matt Wakeley > Agilent Technologies > > Internet-Drafts@ietf.org wrote: > > > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > > > Title : TCP Message Boundary Option > > Author(s) : C. Sapuntzakis > > Filename : draft-csapuntz-tcpmsgbnd-00.txt > > Pages : 4 > > Date : 30-Aug-00 > > > > TCP does not have a mechanism for specifying message boundaries in > > a stream. This I-D describes a new TCP option and a new way of > > using the TCP urgent field to specify message boundaries in the > > stream. > > > > A URL for this Internet-Draft is: > > http://www.ietf.org/internet-drafts/draft-csapuntz-tcpmsgbnd-00.txt From owner-ips@ECE.cmu.edu Thu Aug 31 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20232; 31 Aug 2000 13:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA05174 for ips-outgoing; Thu, 31 Aug 2000 12:20:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA05166 for ; Thu, 31 Aug 2000 12:20:47 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA263062; Thu, 31 Aug 2000 18:20:16 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA22184; Thu, 31 Aug 2000 18:20:15 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694C.0059BC93 ; Thu, 31 Aug 2000 18:20:09 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu cc: Black_David@emc.com Message-ID: Date: Thu, 31 Aug 2000 19:18:06 +0300 Subject: Re: iSCSI Autosense Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, It lookslike we have consensus - but the chairmen have the call. Julo Ralph Weber on 31/08/2000 13:44:20 Please respond to ENDL_TX@computer.org To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: iSCSI Autosense Julo, Thanks for listening. As for the patience, I only hope you all will accord me the same when I go of the deep end over some IP point that is obvious to you by magic to me. Again, thanks. Ralph... P.S. Do we need a consensus call on this, or is your agreement to change the draft sufficient? I'm new to these IETF discussions. julian_satran@il.ibm.com wrote: > > > Thanks for your patience. Julo From owner-ips@ECE.cmu.edu Thu Aug 31 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20475; 31 Aug 2000 14:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06783 for ips-outgoing; Thu, 31 Aug 2000 13:06:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06776 for ; Thu, 31 Aug 2000 13:06:34 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id TAA166278 for ; Thu, 31 Aug 2000 19:06:03 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id TAA37344 for ; Thu, 31 Aug 2000 19:06:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694C.005DEB68 ; Thu, 31 Aug 2000 19:05:51 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 31 Aug 2000 20:03:47 +0300 Subject: RE: Target Reset handling Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk James, The intent of the reset is radical - and assumes that something is bad at the target. Initiator May (or should?) be advised by AE on this happening. The TCP sessions are closed to eliminate any zombies (how gracefully it is up to the implementer - but he should at least make sure that the AE gets to the initiators if he decides to signal it (although this might not be very useful). And what you describe at 2 might happen but then some resources are overcomitted and that is an administrative issue. I have the feeling that I am not understanding exactly the scenario you are concerned with. Are there initiators waiting to be connected before the reset that can't get connected? Are the host sockets over committed? In this case you might want to have the iSCSI drivers have some pre-allocated ranges of sockets. Regards, Julo "James Smart" on 31/08/2000 16:40:19 Please respond to "James Smart" To: Julian Satran/Haifa/IBM@IBMIL cc: "Stephanie Smart" Subject: RE: Target Reset handling Julian, I don't understand your response.... I guess I'd like to understand the history of why the TCP sessions should be closed. I'm going to assume (forgive my lack of TCP knowledge) that the TCP sessions will be gracefully shut down, which will involve a handshake between the initiator and target. Anyway - my reason for asking this is for two fold: a) a significant time delay may occur when tearing down and rebuilding the tcp sessions. All the resetting of the sessions may be unnecessary. Is the groups simply advocating the need to reset these sessions as part of reseting the device ? And must this actually affect all sessions (or only those to this initiator) ? b) What happens if the initiator is not able to reinstantiate the TCP sessions ? I expect that the number of sessions supported per target will be limited. If there are enough initiators contending for the device - or if the resources backing the sessions get moved to other sessions (from other intiators), the initiator may be denied access and all heck may break loose in the applications on the initiator. -- James PS: I see your reply was not via the ips reflect - if you deem appropriate, you may want to forward this back out to the reflector. ============== I though that iSCSI has now and adequate mechanism that includes connection close. Julo > > In reading the iscsi-01 draft, I was bothered by several things in the > handling of Target Reset. > > a) The lack of at least a basic ACCept on the Target Reset. If the target > can send an async event, why not at least notify reception of the function > ? > > Given connections with lots of outstanding traffic, I'd see this as a more > graceful reset procedure. It allows any outstanding i/o that may be > completing while the TR is in transit (or queued for processing on the > target) to do so, possibly lightening the load of i/o that has to error to > complete. This would potentially quicken the recovery time post reset. I > would expect this to be more important as the "network" gets larger and > longer. > > Note: FCP does support this behavior. > > b) Why not require async events to all initiators ? > > The biggest headache with Target Reset is how long it takes for the other > initiators to recognize the device has been reset. The 1st new > i/o will get > a Unit Attention CA, but this status is typically seen only by the SCSI > class driver (e.g. disk/tape/etc). Unless instructed by the class driver, > the port level driver (e.g. scsi/fc/iscsi hba) will have to timeout the > i/o's (if timing was requested) to recover their context. If the class > driver does try to tell the port driver, it typically will do so > in a crude > fashion - issuing abort requests on the i/o's it knows about. > > Perhaps, if the TCP connections are gracefully shutdown between the > initiator and target, the initiator will be to abort the i/o on the > connections quickly (in this case, it looks like a pseudo async event). > However, if there is no handshaking on the connections, my limited > experience with TCP says it takes a long time for the connection to error > out and reset. And during this process, we'll be sending i/o > abort requests > down the terminated-on-one-end connection. All this would make > the recovery > time on these other intiators very large. > > Note: this point assumes that if async events are required - they are > ack'd. > > c) Is there something inherent that requires the TCP connections to be > terminated ? > > The TCP connections look very similar to (but not the same as) FCP Process > logins between the initiator and target. In FCP, the reset did not > necessarily disrupt the port or process logins. It only had to affect the > FPC/SCSI task manager. (note: a device was free to really reset, thus > indeed > tearing down the logins - with the FC port machine handling it as > an error) > > What is the background that required the TCP sessions to be broken ? > > Obviously, if they are not broken, it affects the answers to (a) and (b) > above. > > d) Given the history of long error recovery times in multi-initiator > environments in both parallel scsi and fibre channel on BDR's/Target > Reset's, any speed up in this area would be advantageous. > > -- James > From owner-ips@ECE.cmu.edu Thu Aug 31 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20604; 31 Aug 2000 14:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA08696 for ips-outgoing; Thu, 31 Aug 2000 13:51:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA08643 for ; Thu, 31 Aug 2000 13:51:02 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id DD5524D1C for ; Thu, 31 Aug 2000 13:50:57 -0400 (EDT) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id KAA09248 for ips@ece.cmu.edu; Thu, 31 Aug 2000 10:51:30 -0700 (PDT) Message-Id: <200008311751.KAA09248@core.rose.hp.com> Subject: Re: Target Reset handling To: ips@ece.cmu.edu Date: Thu, 31 Aug 2000 10:51:30 PDT In-Reply-To: ; from "James Smart" at Aug 30, 100 10:42 am Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk James, I am in complete agreement with you on all these concerns as they parallel mine as noted in an earlier posting (http://ips.pdl.cs.cmu.edu/mail/msg00348.html), the relevant portion of which is reproduced below - o Section 3.7.1 on page 23. For the Target Reset task management function, the target is not expected to provide a response - and this is concerning to me. - how would the initiator confirm the successful completion of the target reset? - not having a response effectively makes the target reset an operation iSCSI hardware cannot assist. Having a response would make it no different from the rest of the iSCSI transactions and the hardware can gracefully deal with it. - SAM-2 (section 6.6, page 63) specifies what a target should do "Before returning a FUNCTION COMPLETE response". This seems to me as an implicit requirement that a response be returned on a Target Reset task management function. - I am also unclear as to why the sessions are allowed to be terminated. In the FC world, as far as I can recall, the process login sessions remain intact after a target reset. If it is required that the sessions and the associated TCP connections be cleared in iSCSI, it is helpful to mandate (as opposed to leaving it up to the implementation) that an Async event shall be reported to all the initiators currently logged in. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >In reading the iscsi-01 draft, I was bothered by several things in the >handling of Target Reset. > >a) The lack of at least a basic ACCept on the Target Reset. If the target >can send an async event, why not at least notify reception of the function ? > >Given connections with lots of outstanding traffic, I'd see this as a more >graceful reset procedure. It allows any outstanding i/o that may be >completing while the TR is in transit (or queued for processing on the >target) to do so, possibly lightening the load of i/o that has to error to >complete. This would potentially quicken the recovery time post reset. I >would expect this to be more important as the "network" gets larger and >longer. > >Note: FCP does support this behavior. > >b) Why not require async events to all initiators ? > >The biggest headache with Target Reset is how long it takes for the other >initiators to recognize the device has been reset. The 1st new i/o will get >a Unit Attention CA, but this status is typically seen only by the SCSI >class driver (e.g. disk/tape/etc). Unless instructed by the class driver, >the port level driver (e.g. scsi/fc/iscsi hba) will have to timeout the >i/o's (if timing was requested) to recover their context. If the class >driver does try to tell the port driver, it typically will do so in a crude >fashion - issuing abort requests on the i/o's it knows about. > >Perhaps, if the TCP connections are gracefully shutdown between the >initiator and target, the initiator will be to abort the i/o on the >connections quickly (in this case, it looks like a pseudo async event). >However, if there is no handshaking on the connections, my limited >experience with TCP says it takes a long time for the connection to error >out and reset. And during this process, we'll be sending i/o abort requests >down the terminated-on-one-end connection. All this would make the recovery >time on these other intiators very large. > >Note: this point assumes that if async events are required - they are ack'd. > >c) Is there something inherent that requires the TCP connections to be >terminated ? > >The TCP connections look very similar to (but not the same as) FCP Process >logins between the initiator and target. In FCP, the reset did not >necessarily disrupt the port or process logins. It only had to affect the >FPC/SCSI task manager. (note: a device was free to really reset, thus indeed >tearing down the logins - with the FC port machine handling it as an error) > >What is the background that required the TCP sessions to be broken ? > >Obviously, if they are not broken, it affects the answers to (a) and (b) >above. > >d) Given the history of long error recovery times in multi-initiator >environments in both parallel scsi and fibre channel on BDR's/Target >Reset's, any speed up in this area would be advantageous. > >-- James > > > >-------------------- >James Smart >Trebia Networks, Inc Ph: 978-318-9547 >35 Forest Ridge Rd Cell: 603-674-3687 >Concord, MA 01742 james.smart@trebia.com > > From owner-ips@ECE.cmu.edu Thu Aug 31 19:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20686; 31 Aug 2000 15:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09909 for ips-outgoing; Thu, 31 Aug 2000 14:18:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA09901 for ; Thu, 31 Aug 2000 14:18:16 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Thu, 31 Aug 2000 11:17:13 -0700 Message-ID: From: Charles Monia To: Douglas Otis , Paul.A.Suhler@seagate.com, ips@ece.cmu.edu Subject: RE: iSCSI Autosense Date: Thu, 31 Aug 2000 11:17:13 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Thursday, August 31, 2000 7:49 AM > To: Paul.A.Suhler@seagate.com; ips@ece.cmu.edu > Subject: RE: iSCSI Autosense > > > Paul, > > The answer is in how broadly you see Ethernet SCSI extending. > If in the > business of making incredibly big machines, then rejecting > Autosense is a > nuisance. Keeping SCSI applications unchanged is important > when deployed by > companies promoting adapters. We've all been in the trench > created by a new > paradigm. > > Doug > A coda to this thread: While there may be significant new issues to be dealt with in the trenches, this is not one of them. Since autosense-capable protocols have been deployed for some time, implementations have already found ways to coexist with parallel SCSI in this regard. Charles Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Fri Sep 01 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24661; 1 Sep 2000 4:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA04057 for ips-outgoing; Fri, 1 Sep 2000 03:05:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA04051 for ; Fri, 1 Sep 2000 03:05:21 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id JAA63150 for ; Fri, 1 Sep 2000 09:04:49 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA36204 for ; Fri, 1 Sep 2000 09:04:49 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694D.0026E309 ; Fri, 1 Sep 2000 09:04:44 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 1 Sep 2000 10:02:40 +0300 Subject: Re: Target Reset handling Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, We assumed that a reset for a controller including such a complex piece of code/hardware as protocol stack is never complete without resetting it too. making AE signalling mandatory is not big help as they might get lost if done before the connections break. But if the group feels we should make it mandatory we will. For any simple minded (minimalist design) a protocol function that does what a reset button would do should be enough. For complex controllers you can choose to: - record the reset results in NV store and report them during the next session or even make reading this log in a sense mandatory through an ACA - report the event through a vendor specific text response to the next login - use management interfaces (SNMP) I personally am inclined to think that the "in-band" rest should be left as simple as possible (but not simpler!) and out-of-band communication should be used to check the process. Regards, Julo "Mallikarjun C." on 31/08/2000 20:51:30 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Target Reset handling James, I am in complete agreement with you on all these concerns as they parallel mine as noted in an earlier posting ( http://ips.pdl.cs.cmu.edu/mail/msg00348.html), the relevant portion of which is reproduced below - o Section 3.7.1 on page 23. For the Target Reset task management function, the target is not expected to provide a response - and this is concerning to me. - how would the initiator confirm the successful completion of the target reset? - not having a response effectively makes the target reset an operation iSCSI hardware cannot assist. Having a response would make it no different from the rest of the iSCSI transactions and the hardware can gracefully deal with it. - SAM-2 (section 6.6, page 63) specifies what a target should do "Before returning a FUNCTION COMPLETE response". This seems to me as an implicit requirement that a response be returned on a Target Reset task management function. - I am also unclear as to why the sessions are allowed to be terminated. In the FC world, as far as I can recall, the process login sessions remain intact after a target reset. If it is required that the sessions and the associated TCP connections be cleared in iSCSI, it is helpful to mandate (as opposed to leaving it up to the implementation) that an Async event shall be reported to all the initiators currently logged in. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >In reading the iscsi-01 draft, I was bothered by several things in the >handling of Target Reset. > >a) The lack of at least a basic ACCept on the Target Reset. If the target >can send an async event, why not at least notify reception of the function ? > >Given connections with lots of outstanding traffic, I'd see this as a more >graceful reset procedure. It allows any outstanding i/o that may be >completing while the TR is in transit (or queued for processing on the >target) to do so, possibly lightening the load of i/o that has to error to >complete. This would potentially quicken the recovery time post reset. I >would expect this to be more important as the "network" gets larger and >longer. > >Note: FCP does support this behavior. > >b) Why not require async events to all initiators ? > >The biggest headache with Target Reset is how long it takes for the other >initiators to recognize the device has been reset. The 1st new i/o will get >a Unit Attention CA, but this status is typically seen only by the SCSI >class driver (e.g. disk/tape/etc). Unless instructed by the class driver, >the port level driver (e.g. scsi/fc/iscsi hba) will have to timeout the >i/o's (if timing was requested) to recover their context. If the class >driver does try to tell the port driver, it typically will do so in a crude >fashion - issuing abort requests on the i/o's it knows about. > >Perhaps, if the TCP connections are gracefully shutdown between the >initiator and target, the initiator will be to abort the i/o on the >connections quickly (in this case, it looks like a pseudo async event). >However, if there is no handshaking on the connections, my limited >experience with TCP says it takes a long time for the connection to error >out and reset. And during this process, we'll be sending i/o abort requests >down the terminated-on-one-end connection. All this would make the recovery >time on these other intiators very large. > >Note: this point assumes that if async events are required - they are ack'd. > >c) Is there something inherent that requires the TCP connections to be >terminated ? > >The TCP connections look very similar to (but not the same as) FCP Process >logins between the initiator and target. In FCP, the reset did not >necessarily disrupt the port or process logins. It only had to affect the >FPC/SCSI task manager. (note: a device was free to really reset, thus indeed >tearing down the logins - with the FC port machine handling it as an error) > >What is the background that required the TCP sessions to be broken ? > >Obviously, if they are not broken, it affects the answers to (a) and (b) >above. > >d) Given the history of long error recovery times in multi-initiator >environments in both parallel scsi and fibre channel on BDR's/Target >Reset's, any speed up in this area would be advantageous. > >-- James > > > >-------------------- >James Smart >Trebia Networks, Inc Ph: 978-318-9547 >35 Forest Ridge Rd Cell: 603-674-3687 >Concord, MA 01742 james.smart@trebia.com > > From owner-ips@ECE.cmu.edu Fri Sep 01 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26530; 1 Sep 2000 11:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA20994 for ips-outgoing; Fri, 1 Sep 2000 09:38:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA20988 for ; Fri, 1 Sep 2000 09:38:12 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id JAA15105; Fri, 1 Sep 2000 09:38:11 -0400 Message-Id: <200009011338.JAA15105@yogi.ece.cmu.edu> To: ips@ece.cmu.edu subject: security model Date: Fri, 01 Sep 2000 09:38:11 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk From: Banu Ozden & Mike Reiter We suggest the attached extensions to the security model proposed in the iSCSI draft (Section 6). The existing iSCSI security model covers "communication security" between an initiator and a target. It does not address "data security". Data security provides protection against possible attacks to the data stored at the target. These include threats like unauthorized disclosure of data at the target to administrators or other clients of the target, and unauthorized modification of data at the target. Our main motivation is to enhance the security model for storage outsourcing environments where the Storage Service Provider (SSP) personnel is not necessarily trusted or where the sharing of target between different customers of the SSP raises a security concern. We are working on a security architecture for storage outsourcing. We would like to know whether there is interest on including data security considerations to iSCSI in addition to communication security. Banu Ozden & Mike Reiter Bell Labs 600 Mountain Ave. Murray Hill, NJ 07974 http://www.bell-labs.com/who/ozden http://www.bell-labs.com/who/reiter _______________________________________________________________ Threat Model T1. Disclosure of message contents to an eavesdropper intercepting communication between an initiator and a target. T2. An attacker masquerading as the initiator to a target or the target to an initiator. This includes an attacker manipulating communication between an initiator and a target, e.g., to introduce false messages, modify passing messages, or delete messages. T3. Disclosure of data to personnel maintaining the target or to other customers of the target. T4. The modification of data by the target or other customers of the target. Security Model 1. No Security (same as described in the iSCSI draft) This mode does not authenticate nor does it encrypt data. This mode should be used in environments where there is minimal security risk and little chance for configuration errors. 2. Entity Authentication (referred to as End-to-End Authentication in the iSCSI draft) The initiator's and/or target's identity is authenticated. Once the client is authenticated, all messages are sent and received in the clear. This mode should only be used when there is minimal risk to man-in-the-middle attacks, eavesdropping, message insertion, deletion, and modification. 3. Message Integrity (new) This mode protects against T2 types of threats. It provides communication integrity. 4. Message Integrity Combined with Encryption (referred to as Encryption in the iSCSI draft) This mode protects against threats T1 and T2. Thus, it provides communication integrity and communication privacy. It protects against man-in-the-middle attacks, eavesdropping, message insertion, deletion, and modification. 5. Data Privacy (new) This mode protects against T3 types of threats. The initiator encrypts/decrypts data. The target stores encrypted data. 6. Data Privacy with Data Integrity This mode protects against threats T3 and T4. 7. Some combinations of the above security options For example, data privacy with message authentication (5 & 3) protects against threats T1, T2 and T3. From owner-ips@ECE.cmu.edu Fri Sep 01 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26912; 1 Sep 2000 12:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA25968 for ips-outgoing; Fri, 1 Sep 2000 11:29:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA25961 for ; Fri, 1 Sep 2000 11:29:28 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA49148; Fri, 1 Sep 2000 11:27:33 -0400 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id JAA25916; Fri, 1 Sep 2000 09:29:25 -0600 Importance: Normal Subject: Re: security model To: Dave Nagle Cc: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Fri, 1 Sep 2000 08:29:24 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Build V504_06062000 |June 6, 2000) at 09/01/2000 08:29:25 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Dave, You suggest the following: 5. Data Privacy (new) This mode protects against T3 types of threats. The initiator encrypts/decrypts data. The target stores encrypted data. This sort of idea has come up before in a number of other (non-iSCSI) contexts. However, I've always felt that this was not an issue for either the transport (iSCSI) or for the SCSI layer itself. If an initiator wants to protect its data in this way (from unauthorized use at the storage device), he needs only to encrypt it at the source. That is LONG before it ever gets to the SCSI or iSCSI/FCP/SPI/SST.... layer. The point is that none of these layers need participate in this process; the target (and any of its protocol stack layers) need not participate as well. So no specification in these layers is required; e.g., it need not be included in the iSCSI spec (though a NOTE commenting on this point might be useful). The application layer above all this SCSI stuff can do it autonomously. However, you run into interesting design problems if the data needs to be shared amongst different hosts or different applications. But that exists so long as the data is stored in encrypted form at all, regardless of which layer did the encrypting. The more important issue, I think, is unauthorized access to the data while it is stored. If the wrong guy can get to the data, whether it's encrypted or not, that guy can DESTROY the data. This is the more fundamental threat as it attacks the heart of a storage device's nature. I think the login authentication should handle that role. Jim Hafner From owner-ips@ECE.cmu.edu Fri Sep 01 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26968; 1 Sep 2000 12:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA27039 for ips-outgoing; Fri, 1 Sep 2000 11:52:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA27032 for ; Fri, 1 Sep 2000 11:52:37 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA13330 for ; Fri, 1 Sep 2000 17:52:05 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA31894 for ; Fri, 1 Sep 2000 17:52:05 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694D.005728E0 ; Fri, 1 Sep 2000 17:52:00 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 1 Sep 2000 18:49:58 +0300 Subject: Re: security model Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The subject is certainly interesting. The security chapters (login and appendix) are getting a fresh a look. I assume that this is all WG will want to do for the first version of iSCSI. Integration in Storage Service Provider architecture is certainly important and and will continue to be handled after the first release. Is this work related to Ocean Store? Julo Dave Nagle on 01/09/2000 16:38:11 Please respond to Dave Nagle To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: security model From: Banu Ozden & Mike Reiter We suggest the attached extensions to the security model proposed in the iSCSI draft (Section 6). The existing iSCSI security model covers "communication security" between an initiator and a target. It does not address "data security". Data security provides protection against possible attacks to the data stored at the target. These include threats like unauthorized disclosure of data at the target to administrators or other clients of the target, and unauthorized modification of data at the target. Our main motivation is to enhance the security model for storage outsourcing environments where the Storage Service Provider (SSP) personnel is not necessarily trusted or where the sharing of target between different customers of the SSP raises a security concern. We are working on a security architecture for storage outsourcing. We would like to know whether there is interest on including data security considerations to iSCSI in addition to communication security. Banu Ozden & Mike Reiter Bell Labs 600 Mountain Ave. Murray Hill, NJ 07974 http://www.bell-labs.com/who/ozden http://www.bell-labs.com/who/reiter _______________________________________________________________ Threat Model T1. Disclosure of message contents to an eavesdropper intercepting communication between an initiator and a target. T2. An attacker masquerading as the initiator to a target or the target to an initiator. This includes an attacker manipulating communication between an initiator and a target, e.g., to introduce false messages, modify passing messages, or delete messages. T3. Disclosure of data to personnel maintaining the target or to other customers of the target. T4. The modification of data by the target or other customers of the target. Security Model 1. No Security (same as described in the iSCSI draft) This mode does not authenticate nor does it encrypt data. This mode should be used in environments where there is minimal security risk and little chance for configuration errors. 2. Entity Authentication (referred to as End-to-End Authentication in the iSCSI draft) The initiator's and/or target's identity is authenticated. Once the client is authenticated, all messages are sent and received in the clear. This mode should only be used when there is minimal risk to man-in-the-middle attacks, eavesdropping, message insertion, deletion, and modification. 3. Message Integrity (new) This mode protects against T2 types of threats. It provides communication integrity. 4. Message Integrity Combined with Encryption (referred to as Encryption in the iSCSI draft) This mode protects against threats T1 and T2. Thus, it provides communication integrity and communication privacy. It protects against man-in-the-middle attacks, eavesdropping, message insertion, deletion, and modification. 5. Data Privacy (new) This mode protects against T3 types of threats. The initiator encrypts/decrypts data. The target stores encrypted data. 6. Data Privacy with Data Integrity This mode protects against threats T3 and T4. 7. Some combinations of the above security options For example, data privacy with message authentication (5 & 3) protects against threats T1, T2 and T3. From owner-ips@ECE.cmu.edu Fri Sep 01 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27173; 1 Sep 2000 13:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA28295 for ips-outgoing; Fri, 1 Sep 2000 12:23:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA28288 for ; Fri, 1 Sep 2000 12:22:57 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA07129; Fri, 1 Sep 2000 09:14:50 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA07268; Fri, 1 Sep 2000 09:12:23 -0700 From: "Douglas Otis" To: "Douglas Otis" , , Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 1 Sep 2000 09:25:07 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, After painfully understanding your status here not there conversation and being ensured Autosense is now a fact of life, rather than placing status within a READ PDU, perhaps you should consider simply revising the definition of the flags and use a common DATA PDU. Change the Status bit to indicate Error Detected and perhaps add a bit to signal End of Exchange. >From End of Exchange you could deduce Good Status. I see little benefit in status presented twice or dummy status sent with partial exchanges. It only seems to make room for conflicts. (The two values not agreeing, I, on the other hand, will never agree.) At what point would the O, and U flags be valid? Would this be a result of an error? If so, perhap these flags could be left to response PDU. Doug From owner-ips@ECE.cmu.edu Fri Sep 01 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27557; 1 Sep 2000 15:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA01037 for ips-outgoing; Fri, 1 Sep 2000 13:31:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA01024 for ; Fri, 1 Sep 2000 13:31:07 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id D82FA5D06 for ; Fri, 1 Sep 2000 13:31:05 -0400 (EDT) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id KAA28178 for ips@ece.cmu.edu; Fri, 1 Sep 2000 10:31:39 -0700 (PDT) Message-Id: <200009011731.KAA28178@core.rose.hp.com> Subject: Re: Target Reset handling To: ips@ece.cmu.edu Date: Fri, 01 Sep 2000 10:31:39 PDT In-Reply-To: ; from "julian_satran@il.ibm.com" at Sep 1, 100 10:02 am Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I agree with you that the interactions should be left as simple as possible, but I strongly believe that a response to a target reset is highly desirable and very useful. Besides, my reading of SAM-2 (as quoted below) indicates that a response is required. While SAM-2 leaves the definition of the target reset process to the specific SCSI protocol and interconnect, I would argue that in the case of iSCSI, resetting the entire TCP/IP stack and tearing up the TCP connections should not be necessary. It would seem that the SCSI operational portion (task manager, device server(s), and the tasks themselves) should be affected by a SCSI target reset. As I recall, this is the approach FC target implementations took leaving the process login sessions intact. Having the TCP connections operational would allow us to - - reliably deliver the AEN to all the iSCSI-logged-in initiators. - deliver a response to target reset task mgmt request There's a fair amount of SCSI/LVM software that relies on a reliable target reset confirmation, and an out-of-band confirmation of the same would not go well with it. Regards. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >Mallikarjun, > >We assumed that a reset for a controller including such a complex piece >of code/hardware as protocol stack is never complete without resetting it >too. >making AE signalling mandatory is not big help as they might get lost >if done before the connections break. But if the group feels we should make >it mandatory we will. > >For any simple minded (minimalist design) a protocol function >that does what a reset button would do should be enough. > >For complex controllers you can choose to: > >- record the reset results in NV store and report them during the next >session >or even make reading this log in a sense mandatory through an ACA > >- report the event through a vendor specific text response to the next >login > >- use management interfaces (SNMP) > >I personally am inclined to think that the "in-band" rest should be left >as simple as possible (but not simpler!) and out-of-band communication >should >be used to check the process. > >Regards, >Julo > > >"Mallikarjun C." on 31/08/2000 20:51:30 > >Please respond to cbm@rose.hp.com > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Target Reset handling > > > > >James, > >I am in complete agreement with you on all these concerns as they parallel >mine as noted in an earlier posting ( >http://ips.pdl.cs.cmu.edu/mail/msg00348.html), >the relevant portion of which is reproduced below - > >o Section 3.7.1 on page 23. For the Target Reset task management > function, the target is not expected to provide a response - and > this is concerning to me. > - how would the initiator confirm the successful completion of > the target reset? > > - not having a response effectively makes the target reset > an operation iSCSI hardware cannot assist. Having a response > would make it no different from the rest of the iSCSI transactions > and the hardware can gracefully deal with it. > > - SAM-2 (section 6.6, page 63) specifies what a target should > do "Before returning a FUNCTION COMPLETE response". This seems > to me as an implicit requirement that a response be returned > on a Target Reset task management function. > > - I am also unclear as to why the sessions are allowed to be > terminated. In the FC world, as far as I can recall, the > process login sessions remain intact after a target reset. > If it is required that the sessions and the associated TCP > connections be cleared in iSCSI, it is helpful to mandate (as > opposed to leaving it up to the implementation) that an Async > event shall be reported to all the initiators currently logged in. > >-- >Mallikarjun >M/S 5601 >Networked Storage Architecture >HP Storage Organization >Hewlett-Packard, Roseville. >cbm@rose.hp.com > > > > >>In reading the iscsi-01 draft, I was bothered by several things in the >>handling of Target Reset. >> >>a) The lack of at least a basic ACCept on the Target Reset. If the target >>can send an async event, why not at least notify reception of the function >? >> >>Given connections with lots of outstanding traffic, I'd see this as a more >>graceful reset procedure. It allows any outstanding i/o that may be >>completing while the TR is in transit (or queued for processing on the >>target) to do so, possibly lightening the load of i/o that has to error to >>complete. This would potentially quicken the recovery time post reset. I >>would expect this to be more important as the "network" gets larger and >>longer. >> >>Note: FCP does support this behavior. >> >>b) Why not require async events to all initiators ? >> >>The biggest headache with Target Reset is how long it takes for the other >>initiators to recognize the device has been reset. The 1st new i/o will >get >>a Unit Attention CA, but this status is typically seen only by the SCSI >>class driver (e.g. disk/tape/etc). Unless instructed by the class driver, >>the port level driver (e.g. scsi/fc/iscsi hba) will have to timeout the >>i/o's (if timing was requested) to recover their context. If the class >>driver does try to tell the port driver, it typically will do so in a >crude >>fashion - issuing abort requests on the i/o's it knows about. >> >>Perhaps, if the TCP connections are gracefully shutdown between the >>initiator and target, the initiator will be to abort the i/o on the >>connections quickly (in this case, it looks like a pseudo async event). >>However, if there is no handshaking on the connections, my limited >>experience with TCP says it takes a long time for the connection to error >>out and reset. And during this process, we'll be sending i/o abort >requests >>down the terminated-on-one-end connection. All this would make the >recovery >>time on these other intiators very large. >> >>Note: this point assumes that if async events are required - they are >ack'd. >> >>c) Is there something inherent that requires the TCP connections to be >>terminated ? >> >>The TCP connections look very similar to (but not the same as) FCP Process >>logins between the initiator and target. In FCP, the reset did not >>necessarily disrupt the port or process logins. It only had to affect the >>FPC/SCSI task manager. (note: a device was free to really reset, thus >indeed >>tearing down the logins - with the FC port machine handling it as an >error) >> >>What is the background that required the TCP sessions to be broken ? >> >>Obviously, if they are not broken, it affects the answers to (a) and (b) >>above. >> >>d) Given the history of long error recovery times in multi-initiator >>environments in both parallel scsi and fibre channel on BDR's/Target >>Reset's, any speed up in this area would be advantageous. >> >>-- James >> >> >> >>-------------------- >>James Smart >>Trebia Networks, Inc Ph: 978-318-9547 >>35 Forest Ridge Rd Cell: 603-674-3687 >>Concord, MA 01742 james.smart@trebia.com >> >> > > > > > > From owner-ips@ECE.cmu.edu Fri Sep 01 20:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27919; 1 Sep 2000 16:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA04329 for ips-outgoing; Fri, 1 Sep 2000 14:47:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA04276 for ; Fri, 1 Sep 2000 14:46:37 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 1 Sep 2000 11:45:41 -0700 Message-ID: From: Charles Monia To: "Ips (E-mail)" Subject: FW: Target Reset handling Date: Fri, 1 Sep 2000 11:45:40 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Abject apologies if this is a duplicate. Since I never received a copy from the ips mailing list or spotted the message in the ips archive, I figured this may have been missent or lost in transit. -----Original Message----- From: Charles Monia Sent: Thursday, August 31, 2000 12:18 PM To: Julian Satran (E-mail) Cc: Ips (E-mail) Subject: RE: Target Reset handling A word to the wise: There are a number of features in the SCSI Architecture that are not widely supported by existing implementations. Asynchronous event notification and auto-contingent allegiance are among them. For that reason, relying on SCSI Asynchronous Event Notification, as mentioned below, along with the auto-contingent-allegiance mechanism for device synchronization discussed in another thread should be viewed with caution. As a design goal, at the level of the storage device abstraction, every effort should be made to remain within the mainstream of features supported by deployed SCSI implementations. Charles > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, August 31, 2000 10:04 AM > To: ips@ece.cmu.edu > Subject: RE: Target Reset handling > > > > > James, > > The intent of the reset is radical - and assumes that something is bad > at the target. Initiator May (or should?) be advised by AE on this > happening. > The TCP sessions are closed to eliminate any zombies (how > gracefully it is > up to the > implementer - but he should at least make sure that the AE gets to the > initiators > if he decides to signal it (although this might not be very useful). > > And what you describe at 2 might happen but then some resources are > overcomitted > and that is an administrative issue. > > I have the feeling that I am not understanding exactly the > scenario you are > concerned > with. Are there initiators waiting to be connected before > the reset that > can't get > connected? Are the host sockets over committed? In this > case you might > want to have > the iSCSI drivers have some pre-allocated ranges of sockets. > > Regards, > Julo > Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Sat Sep 02 07:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30870; 2 Sep 2000 3:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA22447 for ips-outgoing; Sat, 2 Sep 2000 01:59:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA22443 for ; Sat, 2 Sep 2000 01:59:33 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA268500 for ; Sat, 2 Sep 2000 07:58:56 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA27958 for ; Sat, 2 Sep 2000 07:58:56 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694E.0020DA85 ; Sat, 2 Sep 2000 07:58:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 2 Sep 2000 08:56:51 +0300 Subject: Re: Target Reset handling Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikajun, Assuming that we leave the connections on - how would you suggest we reset the protocol stack and the connections? Is adding a new iSCSI reset better? Should the target imply a reset when all connections drop and a new session get's established or a long time passes? What happens to the security context across a reset? I have the feeling that answers to those questions will complicate the reset handling far beyond what the technical community will find acceptable. Julo "Mallikarjun C." on 01/09/2000 20:31:39 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Target Reset handling Julian, I agree with you that the interactions should be left as simple as possible, but I strongly believe that a response to a target reset is highly desirable and very useful. Besides, my reading of SAM-2 (as quoted below) indicates that a response is required. While SAM-2 leaves the definition of the target reset process to the specific SCSI protocol and interconnect, I would argue that in the case of iSCSI, resetting the entire TCP/IP stack and tearing up the TCP connections should not be necessary. It would seem that the SCSI operational portion (task manager, device server(s), and the tasks themselves) should be affected by a SCSI target reset. As I recall, this is the approach FC target implementations took leaving the process login sessions intact. Having the TCP connections operational would allow us to - - reliably deliver the AEN to all the iSCSI-logged-in initiators. - deliver a response to target reset task mgmt request There's a fair amount of SCSI/LVM software that relies on a reliable target reset confirmation, and an out-of-band confirmation of the same would not go well with it. Regards. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >Mallikarjun, > >We assumed that a reset for a controller including such a complex piece >of code/hardware as protocol stack is never complete without resetting it >too. >making AE signalling mandatory is not big help as they might get lost >if done before the connections break. But if the group feels we should make >it mandatory we will. > >For any simple minded (minimalist design) a protocol function >that does what a reset button would do should be enough. > >For complex controllers you can choose to: > >- record the reset results in NV store and report them during the next >session >or even make reading this log in a sense mandatory through an ACA > >- report the event through a vendor specific text response to the next >login > >- use management interfaces (SNMP) > >I personally am inclined to think that the "in-band" rest should be left >as simple as possible (but not simpler!) and out-of-band communication >should >be used to check the process. > >Regards, >Julo > > >"Mallikarjun C." on 31/08/2000 20:51:30 > >Please respond to cbm@rose.hp.com > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Target Reset handling > > > > >James, > >I am in complete agreement with you on all these concerns as they parallel >mine as noted in an earlier posting ( >http://ips.pdl.cs.cmu.edu/mail/msg00348.html), >the relevant portion of which is reproduced below - > >o Section 3.7.1 on page 23. For the Target Reset task management > function, the target is not expected to provide a response - and > this is concerning to me. > - how would the initiator confirm the successful completion of > the target reset? > > - not having a response effectively makes the target reset > an operation iSCSI hardware cannot assist. Having a response > would make it no different from the rest of the iSCSI transactions > and the hardware can gracefully deal with it. > > - SAM-2 (section 6.6, page 63) specifies what a target should > do "Before returning a FUNCTION COMPLETE response". This seems > to me as an implicit requirement that a response be returned > on a Target Reset task management function. > > - I am also unclear as to why the sessions are allowed to be > terminated. In the FC world, as far as I can recall, the > process login sessions remain intact after a target reset. > If it is required that the sessions and the associated TCP > connections be cleared in iSCSI, it is helpful to mandate (as > opposed to leaving it up to the implementation) that an Async > event shall be reported to all the initiators currently logged in. > >-- >Mallikarjun >M/S 5601 >Networked Storage Architecture >HP Storage Organization >Hewlett-Packard, Roseville. >cbm@rose.hp.com > > > > >>In reading the iscsi-01 draft, I was bothered by several things in the >>handling of Target Reset. >> >>a) The lack of at least a basic ACCept on the Target Reset. If the target >>can send an async event, why not at least notify reception of the function >? >> >>Given connections with lots of outstanding traffic, I'd see this as a more >>graceful reset procedure. It allows any outstanding i/o that may be >>completing while the TR is in transit (or queued for processing on the >>target) to do so, possibly lightening the load of i/o that has to error to >>complete. This would potentially quicken the recovery time post reset. I >>would expect this to be more important as the "network" gets larger and >>longer. >> >>Note: FCP does support this behavior. >> >>b) Why not require async events to all initiators ? >> >>The biggest headache with Target Reset is how long it takes for the other >>initiators to recognize the device has been reset. The 1st new i/o will >get >>a Unit Attention CA, but this status is typically seen only by the SCSI >>class driver (e.g. disk/tape/etc). Unless instructed by the class driver, >>the port level driver (e.g. scsi/fc/iscsi hba) will have to timeout the >>i/o's (if timing was requested) to recover their context. If the class >>driver does try to tell the port driver, it typically will do so in a >crude >>fashion - issuing abort requests on the i/o's it knows about. >> >>Perhaps, if the TCP connections are gracefully shutdown between the >>initiator and target, the initiator will be to abort the i/o on the >>connections quickly (in this case, it looks like a pseudo async event). >>However, if there is no handshaking on the connections, my limited >>experience with TCP says it takes a long time for the connection to error >>out and reset. And during this process, we'll be sending i/o abort >requests >>down the terminated-on-one-end connection. All this would make the >recovery >>time on these other intiators very large. >> >>Note: this point assumes that if async events are required - they are >ack'd. >> >>c) Is there something inherent that requires the TCP connections to be >>terminated ? >> >>The TCP connections look very similar to (but not the same as) FCP Process >>logins between the initiator and target. In FCP, the reset did not >>necessarily disrupt the port or process logins. It only had to affect the >>FPC/SCSI task manager. (note: a device was free to really reset, thus >indeed >>tearing down the logins - with the FC port machine handling it as an >error) >> >>What is the background that required the TCP sessions to be broken ? >> >>Obviously, if they are not broken, it affects the answers to (a) and (b) >>above. >> >>d) Given the history of long error recovery times in multi-initiator >>environments in both parallel scsi and fibre channel on BDR's/Target >>Reset's, any speed up in this area would be advantageous. >> >>-- James >> >> >> >>-------------------- >>James Smart >>Trebia Networks, Inc Ph: 978-318-9547 >>35 Forest Ridge Rd Cell: 603-674-3687 >>Concord, MA 01742 james.smart@trebia.com >> >> > > > > > > From owner-ips@ECE.cmu.edu Sun Sep 03 04:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03765; 3 Sep 2000 0:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA26377 for ips-outgoing; Sat, 2 Sep 2000 23:08:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA26367 for ; Sat, 2 Sep 2000 23:07:52 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id 842002D9 for ; Sat, 2 Sep 2000 23:07:51 -0400 (EDT) Received: from rose.hp.com (sindhu.rose.hp.com [15.9.210.155]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id UAA11296 for ; Sat, 2 Sep 2000 20:08:20 -0700 (PDT) Message-ID: <39B1C05B.CCEC7086@rose.hp.com> Date: Sat, 02 Sep 2000 20:07:07 -0700 From: "Mallikarjun C." Reply-To: cbm@rose.hp.com Organization: Hewlett-Packard Company X-Mailer: Mozilla 4.61 [en] (X11; U; HP-UX B.10.20 9000/778) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Target Reset handling References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Let me try again, I was arguing that the protocol stack (I assume you mean the state information in various layers by this) and the TCP connections be *not* reset - since I do not see a need for the SCSI transport mechanism to be reset/cleared in the context of "SCSI target reset". I would again request your attention on the FC precedent, and the software expectations of a confirmed target reset. Logging in after an arbitrary "long time" and assuming the target reset to be complete is simply unreliable. I do not see a need for a special "iSCSI reset". I believe that the SCSI target reset task management request is completely adequate to address our requirements. All I am proposing is a change in its current definition in view of the various reasons I had already stated, and the benefits of making this change (not the least of which is SAM-2 compliance). Addition of a new reset mechanism would also defeat our common goal to keep a fairly lean protocol. The question of security context needs to addressed regardless of the SCSI transport behavior on a target reset. If security context is being designed as part of the transient operating environment (as opposed to say, creating/modifying mode pages), then my first guess would be that it has to be reset as well to initial values, on a target reset. I am not sure what you implied, but are you suggesting that the security context cannot be reset with TCP connections living across a SCSI target reset? Or, did I totally miss something? Regards. Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com julian_satran@il.ibm.com wrote: > > Mallikajun, > > Assuming that we leave the connections on - how would you suggest we reset > the protocol stack and the connections? > > Is adding a new iSCSI reset better? > > Should the target imply a reset when all connections drop and a new session > get's established or a long time passes? > > What happens to the security context across a reset? > > I have the feeling that answers to those questions will complicate the > reset handling > far beyond what the technical community will find acceptable. > > Julo > > "Mallikarjun C." on 01/09/2000 20:31:39 > > Please respond to cbm@rose.hp.com > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: Target Reset handling > > Julian, > > I agree with you that the interactions should be left as simple > as possible, but I strongly believe that a response to a target > reset is highly desirable and very useful. Besides, my reading > of SAM-2 (as quoted below) indicates that a response is required. > > While SAM-2 leaves the definition of the target reset process to > the specific SCSI protocol and interconnect, I would argue that in > the case of iSCSI, resetting the entire TCP/IP stack and tearing > up the TCP connections should not be necessary. It would seem that > the SCSI operational portion (task manager, device server(s), and > the tasks themselves) should be affected by a SCSI target reset. > As I recall, this is the approach FC target implementations took > leaving the process login sessions intact. Having the TCP connections > operational would allow us to - > - reliably deliver the AEN to all the iSCSI-logged-in initiators. > - deliver a response to target reset task mgmt request > > There's a fair amount of SCSI/LVM software that relies on a reliable > target reset confirmation, and an out-of-band confirmation of the > same would not go well with it. > > Regards. > -- > Mallikarjun > M/S 5601 > Networked Storage Architecture > HP Storage Organization > Hewlett-Packard, Roseville. > cbm@rose.hp.com > > >Mallikarjun, > > > >We assumed that a reset for a controller including such a complex piece > >of code/hardware as protocol stack is never complete without resetting it > >too. > >making AE signalling mandatory is not big help as they might get lost > >if done before the connections break. But if the group feels we should > make > >it mandatory we will. > > > >For any simple minded (minimalist design) a protocol function > >that does what a reset button would do should be enough. > > > >For complex controllers you can choose to: > > > >- record the reset results in NV store and report them during the next > >session > >or even make reading this log in a sense mandatory through an ACA > > > >- report the event through a vendor specific text response to the next > >login > > > >- use management interfaces (SNMP) > > > >I personally am inclined to think that the "in-band" rest should be left > >as simple as possible (but not simpler!) and out-of-band communication > >should > >be used to check the process. > > > >Regards, > >Julo > > From owner-ips@ECE.cmu.edu Sun Sep 03 10:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05598; 3 Sep 2000 6:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA02183 for ips-outgoing; Sun, 3 Sep 2000 04:52:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from radmail.rad.co.il (radmail.rad.co.il [207.232.32.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA02179 for ; Sun, 3 Sep 2000 04:52:46 -0400 (EDT) Received: from sanrad.com ([192.114.26.219]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-54115U1200L1200S0V35) with ESMTP id il; Sun, 3 Sep 2000 11:55:36 +0300 Message-ID: <39B22D75.1BBADD3B@sanrad.com> Date: Sun, 03 Sep 2000 11:52:37 +0100 From: Yaron Klein Organization: SANRAD X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: Dave Nagle CC: ips@ece.cmu.edu Subject: Re: security model References: <200009011338.JAA15105@yogi.ece.cmu.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk We are now working on the on integrating the security model in the iSCSI draft. Some remarks considering your model: The data privacy (number 5) should be solved in a higher level, i.e., the block should be encrypted or digested by some hash before it is sent to the target. This issue is not in the scope of the transfer protocol. Number 2 should be expanded to initiator and target masquerading. If the enemy masquerades as the target, he can get all our secrets... Furthermore, the security considerations are affected by the performance consideration, i.e., the computation power and the transfer rate. Thus, in high-speed network, compromise should be made (it is not feasible or very expensive to totally secure the data in 1G). The security model should adjust itself to the nature of the iSCSI features. It should enable partial security (only command, only headers, digestion of data etc.) We are now in an intensive work to integrate the security issue in the next draft. The login chapter will enable to establish a flexible and negotiable security/authentication model for the iSCSI session. Any ideas, comments and solution regarding security, let me know, Reagrds, Yaron Dave Nagle wrote: > From: Banu Ozden & Mike Reiter > > We suggest the attached extensions to the security model proposed in > the iSCSI draft (Section 6). > > The existing iSCSI security model covers "communication security" > between an initiator and a target. It does not address "data > security". Data security provides protection against possible attacks > to the data stored at the target. These include threats like > unauthorized disclosure of data at the target to administrators or > other clients of the target, and unauthorized modification of data at > the target. > > Our main motivation is to enhance the security model for storage > outsourcing environments where the Storage Service Provider (SSP) > personnel is not necessarily trusted or where the sharing of target > between different customers of the SSP raises a security concern. > > We are working on a security architecture for storage outsourcing. > We would like to know whether there is interest on including data > security considerations to iSCSI in addition to communication > security. > > Banu Ozden & Mike Reiter > > Bell Labs > 600 Mountain Ave. > Murray Hill, NJ 07974 > http://www.bell-labs.com/who/ozden > http://www.bell-labs.com/who/reiter > > _______________________________________________________________ > Threat Model > > T1. Disclosure of message contents to an eavesdropper intercepting > communication between an initiator and a target. > > T2. An attacker masquerading as the initiator to a target or the target > to an initiator. This includes an attacker manipulating communication > between an initiator and a target, e.g., to introduce false messages, > modify passing messages, or delete messages. > > T3. Disclosure of data to personnel maintaining the target or to other > customers of the target. > > T4. The modification of data by the target or other customers of the > target. > > Security Model > > 1. No Security (same as described in the iSCSI draft) > > This mode does not authenticate nor does it encrypt data. This mode > should be used in environments where there is minimal security risk > and little chance for configuration errors. > > 2. Entity Authentication (referred to as End-to-End Authentication in the > iSCSI draft) > > The initiator's and/or target's identity is authenticated. > Once the client is authenticated, all messages are > sent and received in the clear. This mode should only be used when > there is minimal risk to man-in-the-middle attacks, > eavesdropping, message insertion, deletion, and modification. > > 3. Message Integrity (new) > > This mode protects against T2 types of threats. It provides > communication integrity. > > 4. Message Integrity Combined with Encryption (referred to as Encryption in > the iSCSI draft) > > This mode protects against threats T1 and T2. Thus, it provides > communication integrity and communication privacy. It protects against > man-in-the-middle attacks, eavesdropping, message insertion, deletion, > and modification. > > 5. Data Privacy (new) > > This mode protects against T3 types of threats. The initiator > encrypts/decrypts data. The target stores encrypted data. > > 6. Data Privacy with Data Integrity > > This mode protects against threats T3 and T4. > > 7. Some combinations of the above security options > > For example, data privacy with message authentication (5 & 3) > protects against threats T1, T2 and T3. From owner-ips@ECE.cmu.edu Sun Sep 03 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07205; 3 Sep 2000 12:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA15808 for ips-outgoing; Sun, 3 Sep 2000 11:23:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA15745 for ; Sun, 3 Sep 2000 11:20:50 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Sun, 3 Sep 2000 11:20:17 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F4F@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Cc: Black_David@emc.com Subject: iSCSI Autosense Consensus, Connection next steps Date: Sun, 3 Sep 2000 11:20:15 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > It looks like we have consensus - but the chairmen have the call. Indeed we do, and I apologize for the delay, due to an inability to get connected. First, on Autosense, I believe that rough consensus exists for iSCSI to require Autosense. I have seen only one objection; If anyone other than Doug Otis disagrees with this, please say so on the list. Second, on connections, I haven't seen enough discussion to call consensus, but I am going to try to narrow the option space and structure the discussion. Four models for sessions have been proposed: (1) Symmetric - all connections usable for command and data. (2) Asymmetric - single command connection, others are data. (3) Split - assign LUN sets to specific connections or pools of connections. (4) SCTP - use SCTP's support for multiple connections. I have only seen one message suggesting/supporting (3) Split, so I believe rough consensus exists to not pursue that model any further. If anyone other than Paul von Stamwitz disagrees, and believes that the Split model should be pursued, please say so on the list with technical rationale. I haven't seen enough discussion on use of SCTP instead of TCP to call consensus. I am concerned that this list is not going to produce a consensus on the issue phrased in that fashion (i.e., pick exactly one). The one point on which there does seem to be consensus is that SCTP is a considerably younger protocol that TCP, and hence the likely timelines to availability of hardware acceleration seem to favor TCP. OTOH, this sort of future prediction can easily miss the mark. I can see two possible ways to make progress in this area: - An off-line design team to do an intensive evaluation of SCTP vs. TCP for iSCSI. - Recognize the merits of SCTP as well as TCP, plan for both with the anticipation that TCP will be used first. The latter makes more sense to me, because it appears to be more amenable to consensus, avoids the investment of effort in the design team, and avoids having the prospect of abandoning TCP hanging over the heads of those engaged in TCP-specific work. The practical implication of proceeding in this fashion is that SCTP-friendliness becomes an additional criteria to use in evaluating iSCSI design proposals. Proceeding in this fashion is only a proposal at this juncture -- please comment on whether this is the right way to proceed (to me or on the list). The specification of sessions for iSCSI over TCP needs to proceed, so under the assumption that either the "plan for both" path will be pursued, or that we can't wait for the design team's conclusions, and hence need to work on TCP in case it is selected, I want to summarize the tradeoffs between the Symmetric and Asymmetric session models, in the aim of simulating more discussions so that we can get to consensus. [X] The Symmetric model imposes additional work on implementations that do not use multiple connections because they will have to implement the command numbering. The Asymmetric model does not do this. [Y] The Asymmetric model makes dealing with a failed control connection more complicated than the symmetric model because agreement between the two sides is required to establish a new control connection (or use an existing data connection for control). I have seen the discussion of [X] favoring Asymmetric for systems that will not support multiple TCP connections per iSCSI session. I also note (as an example of applying the SCTP-friendliness from above) that [X] favoring Asymmetric also applies to SCTP, as there seems to be no point in using multiple SCTP sessions in a single iSCSI session. We need to give the folks working on revising the iSCSI draft some direction on this issue in fairly short order, so comments are hereby solicited ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sun Sep 03 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07434; 3 Sep 2000 13:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA16517 for ips-outgoing; Sun, 3 Sep 2000 12:08:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA16511 for ; Sun, 3 Sep 2000 12:08:43 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA38306 for ; Sun, 3 Sep 2000 18:08:12 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA16764 for ; Sun, 3 Sep 2000 18:08:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125694F.0058A10A ; Sun, 3 Sep 2000 18:08:03 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 3 Sep 2000 19:06:00 +0300 Subject: Re: iSCSI Autosense Consensus, Connection next steps Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David - an excellent and timely summary. Indeed we need more feedback on the symmetric/asymmetric item and some consideration to the fact that - if we go for the asymmetric - we might need a longer time as the rewrite is more extensive (although not that complex). The selection of a control channel adds some complexity - but not too much - as we have already agreed to replace the implicit logout with an explicit logout (done on a different connection). Another item in which I feel that we need some more feedback is the reset issue. Mallikarjun has a good point - but I have a hard time finding a decent (simple) solution. And with some vacation and holidays coming we are about to run out of time. Julo Black_David@emc.com on 03/09/2000 18:20:15 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Black_David@emc.com Subject: iSCSI Autosense Consensus, Connection next steps > It looks like we have consensus - but the chairmen have the call. Indeed we do, and I apologize for the delay, due to an inability to get connected. First, on Autosense, I believe that rough consensus exists for iSCSI to require Autosense. I have seen only one objection; If anyone other than Doug Otis disagrees with this, please say so on the list. Second, on connections, I haven't seen enough discussion to call consensus, but I am going to try to narrow the option space and structure the discussion. Four models for sessions have been proposed: (1) Symmetric - all connections usable for command and data. (2) Asymmetric - single command connection, others are data. (3) Split - assign LUN sets to specific connections or pools of connections. (4) SCTP - use SCTP's support for multiple connections. I have only seen one message suggesting/supporting (3) Split, so I believe rough consensus exists to not pursue that model any further. If anyone other than Paul von Stamwitz disagrees, and believes that the Split model should be pursued, please say so on the list with technical rationale. I haven't seen enough discussion on use of SCTP instead of TCP to call consensus. I am concerned that this list is not going to produce a consensus on the issue phrased in that fashion (i.e., pick exactly one). The one point on which there does seem to be consensus is that SCTP is a considerably younger protocol that TCP, and hence the likely timelines to availability of hardware acceleration seem to favor TCP. OTOH, this sort of future prediction can easily miss the mark. I can see two possible ways to make progress in this area: - An off-line design team to do an intensive evaluation of SCTP vs. TCP for iSCSI. - Recognize the merits of SCTP as well as TCP, plan for both with the anticipation that TCP will be used first. The latter makes more sense to me, because it appears to be more amenable to consensus, avoids the investment of effort in the design team, and avoids having the prospect of abandoning TCP hanging over the heads of those engaged in TCP-specific work. The practical implication of proceeding in this fashion is that SCTP-friendliness becomes an additional criteria to use in evaluating iSCSI design proposals. Proceeding in this fashion is only a proposal at this juncture -- please comment on whether this is the right way to proceed (to me or on the list). The specification of sessions for iSCSI over TCP needs to proceed, so under the assumption that either the "plan for both" path will be pursued, or that we can't wait for the design team's conclusions, and hence need to work on TCP in case it is selected, I want to summarize the tradeoffs between the Symmetric and Asymmetric session models, in the aim of simulating more discussions so that we can get to consensus. [X] The Symmetric model imposes additional work on implementations that do not use multiple connections because they will have to implement the command numbering. The Asymmetric model does not do this. [Y] The Asymmetric model makes dealing with a failed control connection more complicated than the symmetric model because agreement between the two sides is required to establish a new control connection (or use an existing data connection for control). I have seen the discussion of [X] favoring Asymmetric for systems that will not support multiple TCP connections per iSCSI session. I also note (as an example of applying the SCTP-friendliness from above) that [X] favoring Asymmetric also applies to SCTP, as there seems to be no point in using multiple SCTP sessions in a single iSCSI session. We need to give the folks working on revising the iSCSI draft some direction on this issue in fairly short order, so comments are hereby solicited ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Sep 04 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10932; 4 Sep 2000 2:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA00724 for ips-outgoing; Mon, 4 Sep 2000 01:11:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA00713 for ; Mon, 4 Sep 2000 01:11:37 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA09434 for ; Mon, 4 Sep 2000 07:11:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA42992 for ; Mon, 4 Sep 2000 07:11:00 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256950.001C7741 ; Mon, 4 Sep 2000 07:10:55 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 4 Sep 2000 08:08:52 +0300 Subject: Re: Target Reset handling Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, How about having two different function: target warm-reset (connections stay) and target cold-rest (connections get also reset)? Julo "Mallikarjun C." on 03/09/2000 06:07:07 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Target Reset handling Julian, Let me try again, I was arguing that the protocol stack (I assume you mean the state information in various layers by this) and the TCP connections be *not* reset - since I do not see a need for the SCSI transport mechanism to be reset/cleared in the context of "SCSI target reset". I would again request your attention on the FC precedent, and the software expectations of a confirmed target reset. Logging in after an arbitrary "long time" and assuming the target reset to be complete is simply unreliable. I do not see a need for a special "iSCSI reset". I believe that the SCSI target reset task management request is completely adequate to address our requirements. All I am proposing is a change in its current definition in view of the various reasons I had already stated, and the benefits of making this change (not the least of which is SAM-2 compliance). Addition of a new reset mechanism would also defeat our common goal to keep a fairly lean protocol. The question of security context needs to addressed regardless of the SCSI transport behavior on a target reset. If security context is being designed as part of the transient operating environment (as opposed to say, creating/modifying mode pages), then my first guess would be that it has to be reset as well to initial values, on a target reset. I am not sure what you implied, but are you suggesting that the security context cannot be reset with TCP connections living across a SCSI target reset? Or, did I totally miss something? Regards. Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com julian_satran@il.ibm.com wrote: > > Mallikajun, > > Assuming that we leave the connections on - how would you suggest we reset > the protocol stack and the connections? > > Is adding a new iSCSI reset better? > > Should the target imply a reset when all connections drop and a new session > get's established or a long time passes? > > What happens to the security context across a reset? > > I have the feeling that answers to those questions will complicate the > reset handling > far beyond what the technical community will find acceptable. > > Julo > > "Mallikarjun C." on 01/09/2000 20:31:39 > > Please respond to cbm@rose.hp.com > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: Target Reset handling > > Julian, > > I agree with you that the interactions should be left as simple > as possible, but I strongly believe that a response to a target > reset is highly desirable and very useful. Besides, my reading > of SAM-2 (as quoted below) indicates that a response is required. > > While SAM-2 leaves the definition of the target reset process to > the specific SCSI protocol and interconnect, I would argue that in > the case of iSCSI, resetting the entire TCP/IP stack and tearing > up the TCP connections should not be necessary. It would seem that > the SCSI operational portion (task manager, device server(s), and > the tasks themselves) should be affected by a SCSI target reset. > As I recall, this is the approach FC target implementations took > leaving the process login sessions intact. Having the TCP connections > operational would allow us to - > - reliably deliver the AEN to all the iSCSI-logged-in initiators. > - deliver a response to target reset task mgmt request > > There's a fair amount of SCSI/LVM software that relies on a reliable > target reset confirmation, and an out-of-band confirmation of the > same would not go well with it. > > Regards. > -- > Mallikarjun > M/S 5601 > Networked Storage Architecture > HP Storage Organization > Hewlett-Packard, Roseville. > cbm@rose.hp.com > > >Mallikarjun, > > > >We assumed that a reset for a controller including such a complex piece > >of code/hardware as protocol stack is never complete without resetting it > >too. > >making AE signalling mandatory is not big help as they might get lost > >if done before the connections break. But if the group feels we should > make > >it mandatory we will. > > > >For any simple minded (minimalist design) a protocol function > >that does what a reset button would do should be enough. > > > >For complex controllers you can choose to: > > > >- record the reset results in NV store and report them during the next > >session > >or even make reading this log in a sense mandatory through an ACA > > > >- report the event through a vendor specific text response to the next > >login > > > >- use management interfaces (SNMP) > > > >I personally am inclined to think that the "in-band" rest should be left > >as simple as possible (but not simpler!) and out-of-band communication > >should > >be used to check the process. > > > >Regards, > >Julo > > From owner-ips@ECE.cmu.edu Mon Sep 04 09:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11938; 4 Sep 2000 5:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA03437 for ips-outgoing; Mon, 4 Sep 2000 04:21:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA03432 for ; Mon, 4 Sep 2000 04:21:26 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id DAA70612; Mon, 4 Sep 2000 03:15:05 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id CAA96446; Mon, 4 Sep 2000 02:21:24 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI Autosense Consensus, Connection next steps To: Black_David@emc.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 4 Sep 2000 01:20:22 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/04/2000 02:21:24 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I have seen some potential value in SCTP, however, it is far too young to become a compelling proposal for Storage which will excite the industry and the customers. Also, I do not want SCTP to get in the way of the TCP/IP version of iSCSI which has, so far, shown to have compelling interest to both the industry and customers. I know of numerous companies that are creating TCP/IP offload NICs etc. for use in a lot of situations including iSCSI, and many that are creating iSCSI-TCP/IP Device Drivers to use with their NICs. I also know of a number of companies that have created iSCSI Device Drivers that use OS platform standard TCP/IP stacks. Neither of these (HW or SW) will be able to exploit iSCSI on a SCTP base very quickly. Not only will a SCTP version require the HW vendors to change their HW NICs to support SCTP (which they will be reluctant to do until they see a proven business case for it, which is very hard to produce), but it will also require the software only implementations to wait until the OS platform supports SCTP (who knows when that will be, for any given OS). Of course the software vendors of SCTP/iSCSI could become their own implementer and owner of an SCTP stack, in the various OSs, however since many of these vendors were going to give their software iSCSI Device Drivers away, this does not seem like good business. All this tells me that an SCTP version of iSCSI will cause the industry to push out the iSCSI for at least another year or two. Therefore, I think we need to focus on the TCP/IP version of iSCSI. Upon completion of that, if you feel there is sufficient value in a SCTP version, then I can support IETF-ips creating that definition also. . . . John L. Hufferd From owner-ips@ECE.cmu.edu Mon Sep 04 12:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12616; 4 Sep 2000 8:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA13997 for ips-outgoing; Mon, 4 Sep 2000 07:06:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sphmraaa.compuserve.com (hs-img-rel-1.compuserve.com [149.174.177.156]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA13993 for ; Mon, 4 Sep 2000 07:06:49 -0400 (EDT) Received: (from mailgate@localhost) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id HAA15156 for ips@ece.cmu.edu; Mon, 4 Sep 2000 07:06:19 -0400 (EDT) Received: from compuserve.com (hil-qbu-ppu-vty8.as.wcom.net [209.154.55.8]) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id HAA15142 for ; Mon, 4 Sep 2000 07:06:08 -0400 (EDT) Message-ID: <39B382E7.EE1DBB4E@compuserve.com> Date: Mon, 04 Sep 2000 06:09:27 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Target Reset handling References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, > How about having two different function: > > target warm-reset (connections stay) and target cold-rest (connections get > also reset)? This sounds like Bus Device Reset (connections stay) and Target Reset (connections disappear) to me. Or, if there's only one LUN per target, then it's like Logical Unit Reset and Target Reset. Thanks. Ralph... From owner-ips@ECE.cmu.edu Tue Sep 05 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20540; 5 Sep 2000 14:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA14398 for ips-outgoing; Tue, 5 Sep 2000 13:05:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA14363 for ; Tue, 5 Sep 2000 13:05:38 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel1.hp.com (Postfix) with ESMTP id A56CAC80 for ; Tue, 5 Sep 2000 10:05:21 -0700 (PDT) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id KAA17708 for ips@ece.cmu.edu; Tue, 5 Sep 2000 10:03:56 -0700 (PDT) Message-Id: <200009051703.KAA17708@core.rose.hp.com> Subject: Re: Target Reset handling To: ips@ece.cmu.edu Date: Tue, 05 Sep 2000 10:03:56 PDT In-Reply-To: ; from "julian_satran@il.ibm.com" at Sep 4, 100 8:08 am Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, Thanks for the suggestion. I am in broad agreement with this kind of definition, if the rest of the folks accept it. I would however suggest a single Target Reset task management function (in accordance with SAM-2), and provide the choice of a "cold-reset" with a "C" bit in the Task Management Command PDU. Assuming that we do this, let me try to summarize our tentative agreement: o Target Reset task management requires a response, unless the the C-bit is set in which case the sessions would be terminated as well and no response can be expected. o A Unit Attention AER shall be reported to all currently logged-in initiators, in accordance with the provisions contained in clauses 7.9 and 8.3.6 of SPC-2 (basically, to be reported only if agreed in advance between the initiator and the LU). Regards. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >Mallikarjun, > >How about having two different function: > >target warm-reset (connections stay) and target cold-rest (connections get >also reset)? > >Julo > >"Mallikarjun C." on 03/09/2000 06:07:07 > >Please respond to cbm@rose.hp.com > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Target Reset handling > > > > >Julian, > >Let me try again, I was arguing that the protocol stack (I assume you >mean the state information in various layers by this) and the TCP >connections be *not* reset - since I do not see a need for the SCSI >transport mechanism to be reset/cleared in the context of "SCSI target >reset". I would again request your attention on the FC precedent, and >the software expectations of a confirmed target reset. Logging in after >an arbitrary "long time" and assuming the target reset to be complete is >simply unreliable. > >I do not see a need for a special "iSCSI reset". I believe that the >SCSI target reset task management request is completely adequate to >address our requirements. All I am proposing is a change in its current >definition in view of the various reasons I had already stated, and the >benefits of making this change (not the least of which is SAM-2 >compliance). Addition of a new reset mechanism would also defeat our >common goal to keep a fairly lean protocol. > >The question of security context needs to addressed regardless of the >SCSI transport behavior on a target reset. If security context is being >designed as part of the transient operating environment (as opposed to >say, creating/modifying mode pages), then my first guess would be that >it has to be reset as well to initial values, on a target reset. I am >not sure what you implied, but are you suggesting that the security >context cannot be reset with TCP connections living across a SCSI target >reset? Or, did I totally miss something? > >Regards. > >Mallikarjun >M/S 5601 >Networked Storage Architecture >HP Storage Organization >Hewlett-Packard, Roseville. >cbm@rose.hp.com > From owner-ips@ECE.cmu.edu Tue Sep 05 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20545; 5 Sep 2000 14:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA11967 for ips-outgoing; Tue, 5 Sep 2000 12:07:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA11918 for ; Tue, 5 Sep 2000 12:06:38 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Tue, 5 Sep 2000 09:07:40 -0700 Message-ID: From: Robert Snively To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: Target Reset handling Date: Tue, 5 Sep 2000 09:05:06 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk I would hope this is not a major issue. The Target Reset provided according to the SCSI SAM-2 definitions is a SCSI protocol function, which does specified stuff to the SCSI protocol stack and nothing else. Since security in the SCSI stack consists of persistent reservation and access control, those properties are reset or not as defined in SAM-2 and SPC-2. All the management of the transport protocol (TCP/IP) should remain unchanged from present TCP/IP management, including resetting of the protocol stack. Maintaining security of the connections should similarly remain unchanged for the TCP/IP transport protocol. Target Reset, a SCSI artifact, should have no effect on the connections or sessions, a TCP/IP artifact. As part of the protocol, iSCSI (like FCP) will have to define whether or not certain TCP/IP functions effect any SCSI functions. As one example, if there are no connections or sessions established to an iSCSI peripheral, an iSCSI command to such a peripheral has no meaning. An attempt by a peripheral to complete a command when there are no connections established should probably start a resource recovery timeout, resulting in the peripheral throwing the command away, and perhaps even becoming a non-device. The general approach FCP took was to treat a successful login and process login for the "SCSI" type of device as both the creation of the "SCSI-ness" of a SCSI device and the establishment of wires to the SCSI device. The problem for both FC and Ethernet attached devices is that the transport mechanisms support multiple protocols and until the sessions and connections have been established, neither side knows whether it will be working with iSCSI, http, or NFS. As a result, clearing all connections and sessions to a device should have an effect on the device, clearing it to the "neutral" state and making it able to form new connections of whatever types are supported. So, using those principles, maybe it is not as complex or difficult as Julian suggests. Bob > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Friday, September 01, 2000 10:57 PM > To: ips@ece.cmu.edu > Subject: Re: Target Reset handling > > > > > Mallikajun, > > Assuming that we leave the connections on - how would you > suggest we reset > the protocol stack and the connections? > > Is adding a new iSCSI reset better? > > Should the target imply a reset when all connections drop > and a new session > get's established or a long time passes? > > What happens to the security context across a reset? > > I have the feeling that answers to those questions will > complicate the > reset handling > far beyond what the technical community will find acceptable. > > Julo > > "Mallikarjun C." on 01/09/2000 20:31:39 > > Please respond to cbm@rose.hp.com > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: Target Reset handling > > > > > Julian, > > I agree with you that the interactions should be left as simple > as possible, but I strongly believe that a response to a target > reset is highly desirable and very useful. Besides, my reading > of SAM-2 (as quoted below) indicates that a response is required. > > While SAM-2 leaves the definition of the target reset process to > the specific SCSI protocol and interconnect, I would argue that in > the case of iSCSI, resetting the entire TCP/IP stack and tearing > up the TCP connections should not be necessary. It would seem that > the SCSI operational portion (task manager, device server(s), and > the tasks themselves) should be affected by a SCSI target reset. > As I recall, this is the approach FC target implementations took > leaving the process login sessions intact. Having the TCP > connections > operational would allow us to - > - reliably deliver the AEN to all the iSCSI-logged-in > initiators. > - deliver a response to target reset task mgmt request > > There's a fair amount of SCSI/LVM software that relies on a reliable > target reset confirmation, and an out-of-band confirmation of the > same would not go well with it. > > Regards. > -- > Mallikarjun > M/S 5601 > Networked Storage Architecture > HP Storage Organization > Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > >Mallikarjun, > > > >We assumed that a reset for a controller including such a > complex piece > >of code/hardware as protocol stack is never complete > without resetting it > >too. > >making AE signalling mandatory is not big help as they > might get lost > >if done before the connections break. But if the group > feels we should > make > >it mandatory we will. > > > >For any simple minded (minimalist design) a protocol function > >that does what a reset button would do should be enough. > > > >For complex controllers you can choose to: > > > >- record the reset results in NV store and report them > during the next > >session > >or even make reading this log in a sense mandatory through an ACA > > > >- report the event through a vendor specific text response > to the next > >login > > > >- use management interfaces (SNMP) > > > >I personally am inclined to think that the "in-band" rest > should be left > >as simple as possible (but not simpler!) and out-of-band > communication > >should > >be used to check the process. > > > >Regards, > >Julo > > > > > >"Mallikarjun C." on 31/08/2000 20:51:30 > > > >Please respond to cbm@rose.hp.com > > > >To: ips@ece.cmu.edu > >cc: (bcc: Julian Satran/Haifa/IBM) > >Subject: Re: Target Reset handling > > > > > > > > > >James, > > > >I am in complete agreement with you on all these concerns > as they parallel > >mine as noted in an earlier posting ( > >http://ips.pdl.cs.cmu.edu/mail/msg00348.html), > >the relevant portion of which is reproduced below - > > > >o Section 3.7.1 on page 23. For the Target Reset task management > > function, the target is not expected to provide a response - and > > this is concerning to me. > > - how would the initiator confirm the successful > completion of > > the target reset? > > > > - not having a response effectively makes the target reset > > an operation iSCSI hardware cannot assist. Having > a response > > would make it no different from the rest of the iSCSI > transactions > > and the hardware can gracefully deal with it. > > > > - SAM-2 (section 6.6, page 63) specifies what a target should > > do "Before returning a FUNCTION COMPLETE > response". This seems > > to me as an implicit requirement that a response > be returned > > on a Target Reset task management function. > > > > - I am also unclear as to why the sessions are allowed to be > > terminated. In the FC world, as far as I can recall, the > > process login sessions remain intact after a target reset. > > If it is required that the sessions and the associated TCP > > connections be cleared in iSCSI, it is helpful to > mandate (as > > opposed to leaving it up to the implementation) > that an Async > > event shall be reported to all the initiators > currently logged > in. > > > >-- > >Mallikarjun > >M/S 5601 > >Networked Storage Architecture > >HP Storage Organization > >Hewlett-Packard, Roseville. > >cbm@rose.hp.com > > > > > > > > > >>In reading the iscsi-01 draft, I was bothered by several > things in the > >>handling of Target Reset. > >> > >>a) The lack of at least a basic ACCept on the Target > Reset. If the target > >>can send an async event, why not at least notify reception of the > function > >? > >> > >>Given connections with lots of outstanding traffic, I'd > see this as a > more > >>graceful reset procedure. It allows any outstanding i/o that may be > >>completing while the TR is in transit (or queued for > processing on the > >>target) to do so, possibly lightening the load of i/o that > has to error > to > >>complete. This would potentially quicken the recovery time > post reset. I > >>would expect this to be more important as the "network" > gets larger and > >>longer. > >> > >>Note: FCP does support this behavior. > >> > >>b) Why not require async events to all initiators ? > >> > >>The biggest headache with Target Reset is how long it > takes for the other > >>initiators to recognize the device has been reset. The 1st > new i/o will > >get > >>a Unit Attention CA, but this status is typically seen > only by the SCSI > >>class driver (e.g. disk/tape/etc). Unless instructed by > the class driver, > >>the port level driver (e.g. scsi/fc/iscsi hba) will have > to timeout the > >>i/o's (if timing was requested) to recover their context. > If the class > >>driver does try to tell the port driver, it typically will > do so in a > >crude > >>fashion - issuing abort requests on the i/o's it knows about. > >> > >>Perhaps, if the TCP connections are gracefully shutdown between the > >>initiator and target, the initiator will be to abort the i/o on the > >>connections quickly (in this case, it looks like a pseudo > async event). > >>However, if there is no handshaking on the connections, my limited > >>experience with TCP says it takes a long time for the > connection to error > >>out and reset. And during this process, we'll be sending i/o abort > >requests > >>down the terminated-on-one-end connection. All this would make the > >recovery > >>time on these other intiators very large. > >> > >>Note: this point assumes that if async events are required > - they are > >ack'd. > >> > >>c) Is there something inherent that requires the TCP > connections to be > >>terminated ? > >> > >>The TCP connections look very similar to (but not the same as) FCP > Process > >>logins between the initiator and target. In FCP, the reset did not > >>necessarily disrupt the port or process logins. It only > had to affect the > >>FPC/SCSI task manager. (note: a device was free to really > reset, thus > >indeed > >>tearing down the logins - with the FC port machine > handling it as an > >error) > >> > >>What is the background that required the TCP sessions to > be broken ? > >> > >>Obviously, if they are not broken, it affects the answers > to (a) and (b) > >>above. > >> > >>d) Given the history of long error recovery times in > multi-initiator > >>environments in both parallel scsi and fibre channel on > BDR's/Target > >>Reset's, any speed up in this area would be advantageous. > >> > >>-- James > >> > >> > >> > >>-------------------- > >>James Smart > >>Trebia Networks, Inc Ph: 978-318-9547 > >>35 Forest Ridge Rd Cell: 603-674-3687 > >>Concord, MA 01742 james.smart@trebia.com > >> > >> > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Tue Sep 05 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20547; 5 Sep 2000 14:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA15045 for ips-outgoing; Tue, 5 Sep 2000 13:17:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA15034 for ; Tue, 5 Sep 2000 13:17:05 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id TAA292664 for ; Tue, 5 Sep 2000 19:16:34 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id TAA57674 for ; Tue, 5 Sep 2000 19:16:33 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256951.005EE3C3 ; Tue, 5 Sep 2000 19:16:26 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 5 Sep 2000 20:14:17 +0300 Subject: Re: Target Reset handling Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mallikarjun, Is the C bit hidden in the CDB? We as a rule avoided examining the CDB (iSCSI never has to). If we can make it external then we are in wild agreement. Regards, Julo "Mallikarjun C." on 05/09/2000 20:03:56 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Target Reset handling Julian, Thanks for the suggestion. I am in broad agreement with this kind of definition, if the rest of the folks accept it. I would however suggest a single Target Reset task management function (in accordance with SAM-2), and provide the choice of a "cold-reset" with a "C" bit in the Task Management Command PDU. Assuming that we do this, let me try to summarize our tentative agreement: o Target Reset task management requires a response, unless the the C-bit is set in which case the sessions would be terminated as well and no response can be expected. o A Unit Attention AER shall be reported to all currently logged-in initiators, in accordance with the provisions contained in clauses 7.9 and 8.3.6 of SPC-2 (basically, to be reported only if agreed in advance between the initiator and the LU). Regards. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >Mallikarjun, > >How about having two different function: > >target warm-reset (connections stay) and target cold-rest (connections get >also reset)? > >Julo > >"Mallikarjun C." on 03/09/2000 06:07:07 > >Please respond to cbm@rose.hp.com > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Target Reset handling > > > > >Julian, > >Let me try again, I was arguing that the protocol stack (I assume you >mean the state information in various layers by this) and the TCP >connections be *not* reset - since I do not see a need for the SCSI >transport mechanism to be reset/cleared in the context of "SCSI target >reset". I would again request your attention on the FC precedent, and >the software expectations of a confirmed target reset. Logging in after >an arbitrary "long time" and assuming the target reset to be complete is >simply unreliable. > >I do not see a need for a special "iSCSI reset". I believe that the >SCSI target reset task management request is completely adequate to >address our requirements. All I am proposing is a change in its current >definition in view of the various reasons I had already stated, and the >benefits of making this change (not the least of which is SAM-2 >compliance). Addition of a new reset mechanism would also defeat our >common goal to keep a fairly lean protocol. > >The question of security context needs to addressed regardless of the >SCSI transport behavior on a target reset. If security context is being >designed as part of the transient operating environment (as opposed to >say, creating/modifying mode pages), then my first guess would be that >it has to be reset as well to initial values, on a target reset. I am >not sure what you implied, but are you suggesting that the security >context cannot be reset with TCP connections living across a SCSI target >reset? Or, did I totally miss something? > >Regards. > >Mallikarjun >M/S 5601 >Networked Storage Architecture >HP Storage Organization >Hewlett-Packard, Roseville. >cbm@rose.hp.com > From owner-ips@ECE.cmu.edu Tue Sep 05 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20562; 5 Sep 2000 14:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA15075 for ips-outgoing; Tue, 5 Sep 2000 13:17:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA15065 for ; Tue, 5 Sep 2000 13:17:29 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA07499; Tue, 5 Sep 2000 10:08:39 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA20403; Tue, 5 Sep 2000 10:06:11 -0700 From: "Douglas Otis" To: , Subject: RE: Target Reset handling Date: Tue, 5 Sep 2000 10:19:41 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <39B382E7.EE1DBB4E@compuserve.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, 1) I agree. This would allow a transport monitor the decision if there was a problem requiring a communication reset. 2) Separate then reset function further with an PDU Op-Code for requesting a communication reset versus a management reset and leave connections on task management resets. If a reset messages does not clean up state with the transport protocol functioning, a transport protocol abort would not improve this likelihood? There is status within the transport protocol to determine transport state. 3) Recommend the protocol Keep-Alive setting to ensure a dead connection is detected in a timely manner. If there is no connection, there is no conflict as how to reset. 4) Should a reset message not kill the connection as suggested, then reply with an Asynchronous Event indicating this request to all affected initiators. 5) Again, although a Autosense is mandated, (Makes sense.) lack of sense data within the Response PDU not seen as an error. 6) Do not send status with the optional PDU Op-Code 0x45 Read Data when it will only be used with Good Status and much of the time, not used at all. Two places to look for status at different times makes for more problems and overhead. 7) Adding an Error and End of Exchange flags to the PDU 0x05 improves reliability of the transfer state machine. As a result, a Client Detected Error event, not covered by the Task Management command PDU opcode 0x02, is added. 8) Change the RTT Ready To Transfer to RFE Request For Exchange to avoid confusing mnemonics. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ralph Weber > Sent: Monday, September 04, 2000 4:09 AM > To: ips@ece.cmu.edu > Subject: Re: Target Reset handling > > > Julo, > > > > How about having two different function: > > > > target warm-reset (connections stay) and target cold-rest > (connections get > > also reset)? > > This sounds like Bus Device Reset (connections stay) and Target Reset > (connections disappear) to me. Or, if there's only one LUN per target, > then it's like Logical Unit Reset and Target Reset. > > Thanks. > > Ralph... > > From owner-ips@ECE.cmu.edu Tue Sep 05 20:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21091; 5 Sep 2000 16:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA19290 for ips-outgoing; Tue, 5 Sep 2000 14:44:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA19282 for ; Tue, 5 Sep 2000 14:44:26 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id E3B8A1F88 for ; Tue, 5 Sep 2000 14:44:23 -0400 (EDT) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id LAA07848 for ips@ece.cmu.edu; Tue, 5 Sep 2000 11:44:58 -0700 (PDT) Message-Id: <200009051844.LAA07848@core.rose.hp.com> Subject: Re: Target Reset handling To: ips@ece.cmu.edu Date: Tue, 05 Sep 2000 11:44:57 PDT In-Reply-To: ; from "julian_satran@il.ibm.com" at Sep 5, 100 8:14 pm Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I was thinking that the C-bit would go into bytes 2/3 (currently reserved) of the SCSI Task Management Command PDU (as defined in section 3.7 of draft-satran-iscsi-01.txt) - which does not have a CDB. So, I guess it's part of the iSCSI protocol headers, as you were suggesting. Thanks. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >Mallikarjun, > >Is the C bit hidden in the CDB? We as a rule avoided examining the CDB >(iSCSI never has to). If we can make it external then we are in wild >agreement. > >Regards, >Julo > >"Mallikarjun C." on 05/09/2000 20:03:56 > >Please respond to cbm@rose.hp.com > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Target Reset handling > > > > >Julian, > >Thanks for the suggestion. > >I am in broad agreement with this kind of definition, if the >rest of the folks accept it. I would however suggest a single >Target Reset task management function (in accordance with SAM-2), >and provide the choice of a "cold-reset" with a "C" bit in the >Task Management Command PDU. Assuming that we do this, let me >try to summarize our tentative agreement: > >o Target Reset task management requires a response, unless the > the C-bit is set in which case the sessions would be terminated > as well and no response can be expected. > >o A Unit Attention AER shall be reported to all currently logged-in > initiators, in accordance with the provisions contained in clauses > 7.9 and 8.3.6 of SPC-2 (basically, to be reported only if agreed > in advance between the initiator and the LU). > >Regards. >-- >Mallikarjun >M/S 5601 >Networked Storage Architecture >HP Storage Organization >Hewlett-Packard, Roseville. >cbm@rose.hp.com > > > >>Mallikarjun, >> >>How about having two different function: >> >>target warm-reset (connections stay) and target cold-rest (connections get >>also reset)? >> >>Julo >> From owner-ips@ECE.cmu.edu Tue Sep 05 20:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21074; 5 Sep 2000 16:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA17846 for ips-outgoing; Tue, 5 Sep 2000 14:13:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA17830 for ; Tue, 5 Sep 2000 14:13:23 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel1.hp.com (Postfix) with ESMTP id E9E4D1D32 for ; Tue, 5 Sep 2000 14:13:21 -0400 (EDT) Received: (from cbm@localhost) by core.rose.hp.com (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id LAA01812 for ips@ece.cmu.edu; Tue, 5 Sep 2000 11:13:56 -0700 (PDT) Message-Id: <200009051813.LAA01812@core.rose.hp.com> Subject: Re: Target Reset handling (fwd) To: ips@ece.cmu.edu Date: Tue, 05 Sep 2000 11:13:55 PDT Reply-To: cbm@rose.hp.com From: "Mallikarjun C." X-Mailer: Elm [revision: 212.4] Sender: owner-ips@ece.cmu.edu Precedence: bulk Apologies if this is a second copy, I had not seen my copy of this posting or on the web archive even after more than an hour.... -- Mallikarjun Forwarded message: Date: Tue, 05 Sep 2000 10:03:56 PDT To: ips@ece.cmu.edu Subject: Re: Target Reset handling In-Reply-To: ; from "julian_satran@il.ibm.com" at Sep 4, 100 8:08 am Status: RO Julian, Thanks for the suggestion. I am in broad agreement with this kind of definition, if the rest of the folks accept it. I would however suggest a single Target Reset task management function (in accordance with SAM-2), and provide the choice of a "cold-reset" with a "C" bit in the Task Management Command PDU. Assuming that we do this, let me try to summarize our tentative agreement: o Target Reset task management requires a response, unless the the C-bit is set in which case the sessions would be terminated as well and no response can be expected. o A Unit Attention AER shall be reported to all currently logged-in initiators, in accordance with the provisions contained in clauses 7.9 and 8.3.6 of SPC-2 (basically, to be reported only if agreed in advance between the initiator and the LU). Regards. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >Mallikarjun, > >How about having two different function: > >target warm-reset (connections stay) and target cold-rest (connections get >also reset)? > >Julo > >"Mallikarjun C." on 03/09/2000 06:07:07 > >Please respond to cbm@rose.hp.com > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Target Reset handling > > > > >Julian, > >Let me try again, I was arguing that the protocol stack (I assume you >mean the state information in various layers by this) and the TCP >connections be *not* reset - since I do not see a need for the SCSI >transport mechanism to be reset/cleared in the context of "SCSI target >reset". I would again request your attention on the FC precedent, and >the software expectations of a confirmed target reset. Logging in after >an arbitrary "long time" and assuming the target reset to be complete is >simply unreliable. > >I do not see a need for a special "iSCSI reset". I believe that the >SCSI target reset task management request is completely adequate to >address our requirements. All I am proposing is a change in its current >definition in view of the various reasons I had already stated, and the >benefits of making this change (not the least of which is SAM-2 >compliance). Addition of a new reset mechanism would also defeat our >common goal to keep a fairly lean protocol. > >The question of security context needs to addressed regardless of the >SCSI transport behavior on a target reset. If security context is being >designed as part of the transient operating environment (as opposed to >say, creating/modifying mode pages), then my first guess would be that >it has to be reset as well to initial values, on a target reset. I am >not sure what you implied, but are you suggesting that the security >context cannot be reset with TCP connections living across a SCSI target >reset? Or, did I totally miss something? > >Regards. > >Mallikarjun >M/S 5601 >Networked Storage Architecture >HP Storage Organization >Hewlett-Packard, Roseville. >cbm@rose.hp.com > From owner-ips@ECE.cmu.edu Tue Sep 05 23:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21872; 5 Sep 2000 19:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA27713 for ips-outgoing; Tue, 5 Sep 2000 17:46:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA27705 for ; Tue, 5 Sep 2000 17:46:33 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e85LkWx01194 for ; Tue, 5 Sep 2000 16:46:32 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA04226; Tue, 5 Sep 00 16:44:47 -0500 Message-Id: <10009052144.AA04226@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps In-Reply-To: Message from Black_David@emc.com of "Sun, 03 Sep 2000 11:20:15 EDT." <0F31E5C394DAD311B60C00E029101A0704100F4F@corpmx9.isus.emc.com> Date: Tue, 05 Sep 2000 16:44:21 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Second, on connections, I haven't seen enough discussion to call > consensus, but I am going to try to narrow the option space and > structure the discussion. Four models for sessions have been > proposed: > > (1) Symmetric - all connections usable for command and data. > (2) Asymmetric - single command connection, others are data. > (3) Split - assign LUN sets to specific connections or pools of > connections. > (4) SCTP - use SCTP's support for multiple connections. For the purposes of shaping the consensus, here's my stance: 1) SCTP should only be pursued if TCP does not admit a viable solution to the iSCSI requirements. Given that there are many iSCSI community participants expressing the belief that iSCSI on TCP IS possible, I believe iSCSI on SCTP proponents are just going to be stuck holding an `I told you so' card. This gates my second statement: 2) multiple connections per session should only be supported if the underlying transport (e.g. SCTP) layer supports it. Obviously, TCP does not presently support this abstraction, so assuming SCTP gets killed for now, I am against multiple connections per session. In general, I am dubious that connection bundling above the transport layer, but below some more application-informed layer (e.g. a wedge driver) will work acceptably. However, if the transport layer provides it, then, by definition, it must work (ha, ha), or at least it's not iSCSI's place to say that it won't. Steph From owner-ips@ECE.cmu.edu Tue Sep 05 23:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21938; 5 Sep 2000 19:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA28137 for ips-outgoing; Tue, 5 Sep 2000 17:55:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA28127 for ; Tue, 5 Sep 2000 17:55:15 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Tue, 5 Sep 2000 14:55:06 -0700 Message-ID: From: Joshua Tseng To: Black_David@emc.com, julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI Autosense Consensus, Connection next steps Date: Tue, 5 Sep 2000 14:54:56 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, If I may give my input on the TCP connection model discussion, I'd like to advocate the asymmetric model for the following reasons. Using multiple TCP connections, each TCP connection will likely have different performance characteristics. If this is so, then some commands may reach the target out of order from other commands sent down different TCP connections. Commands sent down the faster connection will then need to be buffered until commands sent down slower connections arrive. This will lead to the following problems that I can think of: 1) There will be increased memory requirements for the iSCSI target, which must buffer commands and related data and reorder them before passing them on to the SCSI controller. 2) If the initiator cannot sense the performance characteristics of each TCP connection, then interactions between initiator and target will be dependent on the slowest TCP connection. For example, if there are five TCP connections and one of them is extraordinarily slow, and the initiator merely round-robins the commands down each of the five connections, all interactions between initiator and target will be only as fast as the slowest connection. In addition, it has been stated that detecting and recovering from a failed asymmetric control connection is more complicated and difficult than detecting and recovering from a failed symmetric data/command TCP connection. I do not understand why this is so. In both cases, iSCSI initiator will need to abort the lost command AND those following the lost command, and resend all commands in a new or different TCP connection. In the symmetric case, iSCSI commands received by a target following a lost command will not be acknowledged, since ExpStatRN is a cummulative windowing mechanism. Josh Tseng -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Sunday, September 03, 2000 8:20 AM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Cc: Black_David@emc.com Subject: iSCSI Autosense Consensus, Connection next steps > It looks like we have consensus - but the chairmen have the call. Indeed we do, and I apologize for the delay, due to an inability to get connected. First, on Autosense, I believe that rough consensus exists for iSCSI to require Autosense. I have seen only one objection; If anyone other than Doug Otis disagrees with this, please say so on the list. Second, on connections, I haven't seen enough discussion to call consensus, but I am going to try to narrow the option space and structure the discussion. Four models for sessions have been proposed: (1) Symmetric - all connections usable for command and data. (2) Asymmetric - single command connection, others are data. (3) Split - assign LUN sets to specific connections or pools of connections. (4) SCTP - use SCTP's support for multiple connections. I have only seen one message suggesting/supporting (3) Split, so I believe rough consensus exists to not pursue that model any further. If anyone other than Paul von Stamwitz disagrees, and believes that the Split model should be pursued, please say so on the list with technical rationale. I haven't seen enough discussion on use of SCTP instead of TCP to call consensus. I am concerned that this list is not going to produce a consensus on the issue phrased in that fashion (i.e., pick exactly one). The one point on which there does seem to be consensus is that SCTP is a considerably younger protocol that TCP, and hence the likely timelines to availability of hardware acceleration seem to favor TCP. OTOH, this sort of future prediction can easily miss the mark. I can see two possible ways to make progress in this area: - An off-line design team to do an intensive evaluation of SCTP vs. TCP for iSCSI. - Recognize the merits of SCTP as well as TCP, plan for both with the anticipation that TCP will be used first. The latter makes more sense to me, because it appears to be more amenable to consensus, avoids the investment of effort in the design team, and avoids having the prospect of abandoning TCP hanging over the heads of those engaged in TCP-specific work. The practical implication of proceeding in this fashion is that SCTP-friendliness becomes an additional criteria to use in evaluating iSCSI design proposals. Proceeding in this fashion is only a proposal at this juncture -- please comment on whether this is the right way to proceed (to me or on the list). The specification of sessions for iSCSI over TCP needs to proceed, so under the assumption that either the "plan for both" path will be pursued, or that we can't wait for the design team's conclusions, and hence need to work on TCP in case it is selected, I want to summarize the tradeoffs between the Symmetric and Asymmetric session models, in the aim of simulating more discussions so that we can get to consensus. [X] The Symmetric model imposes additional work on implementations that do not use multiple connections because they will have to implement the command numbering. The Asymmetric model does not do this. [Y] The Asymmetric model makes dealing with a failed control connection more complicated than the symmetric model because agreement between the two sides is required to establish a new control connection (or use an existing data connection for control). I have seen the discussion of [X] favoring Asymmetric for systems that will not support multiple TCP connections per iSCSI session. I also note (as an example of applying the SCTP-friendliness from above) that [X] favoring Asymmetric also applies to SCTP, as there seems to be no point in using multiple SCTP sessions in a single iSCSI session. We need to give the folks working on revising the iSCSI draft some direction on this issue in fairly short order, so comments are hereby solicited ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Sep 05 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22050; 5 Sep 2000 19:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA29633 for ips-outgoing; Tue, 5 Sep 2000 18:29:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA29526 for ; Tue, 5 Sep 2000 18:27:10 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id 3D6DC42D for ; Tue, 5 Sep 2000 15:27:08 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id PAA08941 for ; Tue, 5 Sep 2000 15:27:06 -0700 (PDT) Received: from rtl.rose.agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA26A3 for ; Tue, 5 Sep 2000 15:27:05 -0700 Message-ID: <39B5735C.2C0DA776@rtl.rose.agilent.com> Date: Tue, 05 Sep 2000 15:27:40 -0700 From: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: exactly the same protocol ?? References: <0F31E5C394DAD311B60C00E029101A0704100EAB@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David, FC-SW and FC-AL are completely different beasts. FC-SW "specifies tools and algorithms for interconnection and initialization of Fibre Channel switches to create a multi-switch Fibre Channel Fabric." whereas FC-AL defines a protocol of communication between two or more nodes on a FC loop. Are you referring to the difference between FC-AL and FC-PH (also known as N_Port or old port mode)? (I fail to understand why EMC would be connecting storage boxes up as a fabric switch element.) If that is the case, this is the definition of how the data is transferred across the wire, not the definition of the protocol of the data itself. It's like comparing ethernet to token ring. These are two different wire protocols, but they can (and do) carry the same "data protocol" (TCP/IP, etc). -Matt Wakeley Agilent Technologies Black_David@emc.com wrote: > With my WG co-chair hat off, it seems that Robert > Snively and I have different notions of what > "exactly the same protocol" means. It would have > been nice if FC-AL and FC-SW were exactly the > same protocol, as it would have saved EMC and a > number of other companies a considerable amount > of software/firmware development time and effort. > > FC-AL and FC-SW are different protocols at the FC-2 > level that do not interoperate - a device that only > speaks FC-AL cannot communicate with a device that > only speaks FC-SW and vice-versa. FCP (which > encapsulates SCSI for Fibre Channel) is an FC-4 level > upper layer protocol that runs over both FC-AL and > FC-SW. I don't believe that there are any disk > drives that speak FC-SW. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 06 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22126; 5 Sep 2000 19:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA00378 for ips-outgoing; Tue, 5 Sep 2000 18:49:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA00374 for ; Tue, 5 Sep 2000 18:49:21 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e85MnLx02337 for ; Tue, 5 Sep 2000 17:49:21 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA04286; Tue, 5 Sep 00 17:47:36 -0500 Message-Id: <10009052247.AA04286@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Target Reset handling In-Reply-To: Message from julian_satran@il.ibm.com of "Tue, 05 Sep 2000 20:14:17 +0300." Date: Tue, 05 Sep 2000 17:47:16 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Is the C bit hidden in the CDB? We as a rule avoided examining the CDB > (iSCSI never has to). If we can make it external then we are in wild > agreement. Clearly all these resets are forms of SCSI Task management, and unlike FCP (and SST, in its imitation of FCP), iSCSI doesn't use the same PDU for task management and command delivery. Therefore, there's no CDB involved in the reset process at all. The big hammer (reset the transport as well as the SCSI target) and little hammer (just reset the SCSI target, or SCSI LUN) resets are just different task management functions. This seems to call for different task management Function code, not a modifier bit. Historically, having a big hammer reset is critical to making early configurations work. Realistically, the first iSCSI implementations are going to range from broke to horribly broke. ||SCSI and FCP had the same problem (is it over yet?). Both ||SCSI and FCP further benefitted from a side-band reset signal (the reset LIPs in FCP on FC-AL). This was important in FCP because frequently targets (never those fair-haired, golden child, initiators, of course) would become stuck to the point where no credit would flow back to the initiator with which to send an in-band reset request. I've heard similar stories from old line ||SCSI implementors. The side-band reset is usually detected by the interface hardware, which will pull a real hardware reset for the target. This capability is gone (without replacement, as far as I know) in FCP on fabric, but by that time, the target logic had been pretty well shaken out under FC-AL. Implementation rules like `ya gotta empty your queues come hell or high water' had mostly been successfully implemented by that point. The big hammer reset mechanisms were the difference between success and failure in our early FCP deployments, and I believe that they can still make the difference between a robust implementation and a frail one. I can't imagine a side-band reset mechanism for iSCSI (give it some thought), but in lieu of that, a big hammer in-band reset should be there. On the subject of HOW transport connections are closed on a Target Reset, note that FCP and iSCSI (and SST) necessarily diverge. In FCP, when a PDU is received outside of a connection, the end-point is supposed to initiate the disconnect (logout) protocol. That way, when an end-point loses its mind (or is simply power cycled), and resets, the far endpoint gets immediate feedback that this has happened. TCP and ST define that anything received outside a valid connection simply gets chucked (this is necessary for antialiasing to work). On a large network it can be a long time before a connection is closed as a result of a simple timeout (as in, Web site found, waiting for reply...). Therefore, SST (and iSCSI, as far as I understand the current draft) requires that an end-point always attempt the connection close message protocol (with appropriate timeouts and retries), before declaring the connection closed. In the `common' case, this gives zippy recovery of connection resources and notification to the far endpoint. In the worst case (e.g. target's transmitter engine is stuck or power cycle), it all works out eventually. You can't do any better than that. The ACK for a big hammer reset is the closing of the transport connection. Again, this can come as a result of a successful close handshake or a timeout. The current iSCSI draft seems to indicate that Target Reset is transport and SCSI layer (big hammer) reset, and LUN reset is only a SCSI layer reset. That works for me. On the other hand, if you want to add an addition Target Reset (w/o transport reset) task management function, knock yourself out. Steph From owner-ips@ECE.cmu.edu Wed Sep 06 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22142; 5 Sep 2000 19:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA29634 for ips-outgoing; Tue, 5 Sep 2000 18:29:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA28909 for ; Tue, 5 Sep 2000 18:12:06 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Tue, 5 Sep 2000 15:13:09 -0700 Message-ID: From: Robert Snively To: ips@ece.cmu.edu Subject: RE: Target Reset handling Date: Tue, 5 Sep 2000 15:10:36 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk I do not believe that the added complexity of a cold and hot reset is appropriate. SCSI-2 had such a concept, and it was ultimately determined that there was no benefit to it, and the complexity was immense. I strongly feel that the SCSI Target Reset should do exactly no more and no less than the corresponding function does in all other SCSI devices. If a session/connection reset is required, it should be performed by session/connection control mechanisms already defined by TCP/IP. Its effect on the SCSI target should be defined, but may or may not be the same as the SCSI Target Reset. FCP-2 provides some examples of the kind of things that you may choose to reset in table 4. No "C" bit should be used. The session/connection reset should not be in the CDB or in the task management functions. Bob > -----Original Message----- > From: Mallikarjun C. [mailto:cbm@rose.hp.com] > Sent: Tuesday, September 05, 2000 11:14 AM > To: ips@ece.cmu.edu > Subject: Re: Target Reset handling (fwd) > > > Apologies if this is a second copy, I had not seen my copy of > this posting or on the web archive even after more than an hour.... > -- > Mallikarjun > > > Forwarded message: > > Date: Tue, 05 Sep 2000 10:03:56 PDT > To: ips@ece.cmu.edu > Subject: Re: Target Reset handling > In-Reply-To: ; > from "julian_satran@il.ibm.com" at Sep 4, 100 8:08 am > Status: RO > > Julian, > > Thanks for the suggestion. > > I am in broad agreement with this kind of definition, if the > rest of the folks accept it. I would however suggest a single > Target Reset task management function (in accordance with SAM-2), > and provide the choice of a "cold-reset" with a "C" bit in the > Task Management Command PDU. Assuming that we do this, let me > try to summarize our tentative agreement: > > o Target Reset task management requires a response, unless the > the C-bit is set in which case the sessions would be terminated > as well and no response can be expected. > > o A Unit Attention AER shall be reported to all currently logged-in > initiators, in accordance with the provisions contained in clauses > 7.9 and 8.3.6 of SPC-2 (basically, to be reported only if agreed > in advance between the initiator and the LU). > > Regards. > -- > Mallikarjun > M/S 5601 > Networked Storage Architecture > HP Storage Organization > Hewlett-Packard, Roseville. > cbm@rose.hp.com > > > > >Mallikarjun, > > > >How about having two different function: > > > >target warm-reset (connections stay) and target cold-rest > (connections get > >also reset)? > > > >Julo > > > >"Mallikarjun C." on 03/09/2000 06:07:07 > > > >Please respond to cbm@rose.hp.com > > > >To: ips@ece.cmu.edu > >cc: (bcc: Julian Satran/Haifa/IBM) > >Subject: Re: Target Reset handling > > > > > > > > > >Julian, > > > >Let me try again, I was arguing that the protocol stack (I > assume you > >mean the state information in various layers by this) and the TCP > >connections be *not* reset - since I do not see a need for the SCSI > >transport mechanism to be reset/cleared in the context of > "SCSI target > >reset". I would again request your attention on the FC > precedent, and > >the software expectations of a confirmed target reset. > Logging in after > >an arbitrary "long time" and assuming the target reset to > be complete is > >simply unreliable. > > > >I do not see a need for a special "iSCSI reset". I believe that the > >SCSI target reset task management request is completely adequate to > >address our requirements. All I am proposing is a change > in its current > >definition in view of the various reasons I had already > stated, and the > >benefits of making this change (not the least of which is SAM-2 > >compliance). Addition of a new reset mechanism would also > defeat our > >common goal to keep a fairly lean protocol. > > > >The question of security context needs to addressed > regardless of the > >SCSI transport behavior on a target reset. If security > context is being > >designed as part of the transient operating environment (as > opposed to > >say, creating/modifying mode pages), then my first guess > would be that > >it has to be reset as well to initial values, on a target > reset. I am > >not sure what you implied, but are you suggesting that the security > >context cannot be reset with TCP connections living across > a SCSI target > >reset? Or, did I totally miss something? > > > >Regards. > > > >Mallikarjun > >M/S 5601 > >Networked Storage Architecture > >HP Storage Organization > >Hewlett-Packard, Roseville. > >cbm@rose.hp.com > > > > > From owner-ips@ECE.cmu.edu Wed Sep 06 00:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22216; 5 Sep 2000 20:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA01719 for ips-outgoing; Tue, 5 Sep 2000 19:28:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA01714 for ; Tue, 5 Sep 2000 19:28:52 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA07626; Tue, 5 Sep 2000 16:19:59 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA21365; Tue, 5 Sep 2000 16:17:36 -0700 From: "Douglas Otis" To: , , Subject: RE: iSCSI Autosense Consensus, Connection next steps Date: Tue, 5 Sep 2000 16:31:08 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100F4F@corpmx9.isus.emc.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > Sent: Sunday, September 03, 2000 8:20 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Cc: Black_David@emc.com > Subject: iSCSI Autosense Consensus, Connection next steps > > > > It looks like we have consensus - but the chairmen have the call. > > Indeed we do, and I apologize for the delay, due to an inability to > get connected. > > First, on Autosense, I believe that rough consensus exists for > iSCSI to require Autosense. I have seen only one objection; > If anyone other than Doug Otis disagrees with this, please > say so on the list. I am in favor of removing the A bit to control Autosense. I do not disagree with respect to that change. The only reason for having an Autosense option is to support legacy applications. As things like tape drives, loaders, etc, use untagged CA environments without suffering limitations even in a packet mode, to preserve the application (as you would not want to bit fiddle the CDBs within the client side drivers) allowing lack of automatic sense *data* at the adapter may provide a solution. It would not violate SAM-2. Although you certainly can enforce interface compliance, you have less ability to enforce application compliance sitting above a SCSI API that does not make decisions enabling ACA. Microsoft assumes the SCSI interface is SCSI-II as example. As network based SCSI is likely to introduce multiple initiators into the picture normally occupied by a single adapter, CA still provides untagged protection for this sloth equipment. The legacy application would not have knowledge of this limitation and so the option was on the wrong side of the interface. There is no need for any option if a back door is desired and no sense data does not cause an error within the client side driver. (You can always insist such is an error, but if it increases customer satisfaction it may be hard to enforce.) I suspect there are many archive applications that will be excluded from a pure Autosense environment as it relates to a network version of SCSI. Archive equipment tends to have a longer life cycle than disk drives and as such remains behind the curve with respect to conformance to the latest versions optimized for disk drives that have a life cycle in months. Doug From owner-ips@ECE.cmu.edu Wed Sep 06 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22241; 5 Sep 2000 20:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA01521 for ips-outgoing; Tue, 5 Sep 2000 19:22:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA01510 for ; Tue, 5 Sep 2000 19:22:44 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id TAA63154; Tue, 5 Sep 2000 19:10:31 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id RAA106172; Tue, 5 Sep 2000 17:22:39 -0600 Importance: Normal Subject: Re: iSCSI Autosense Consensus, Connection next steps To: Stephen Bailey Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 5 Sep 2000 16:17:20 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/05/2000 05:22:38 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, I certainly agree with your first point. However, we need to focus a little more on the second point. You talked about Transport layer and Wedge layers. I guess I do not know exactly what to call iSCSI if given these two options, other then to consider iSCSI part of the SCSI Transport (layer). It is a device specific driver, which in this case deals with iSCSI Devices. I do not think we should call iSCSI a Wedge Driver. In fact if you do not have iSCSI supporting the balancing of commands and data across the multiple connections (and perhaps some generic Alternate Path Retry), you will have to have a Wedge Driver Do it. In was David Black's point when he said that Wedge Drivers are needed anyway to handle a number of vendor specific functions. I think that David is correct the only point that I was making was that if iSCSI was able to do the load balancing and some Generic Alternate Path recovery, that the Vendor Specific Wedge Drivers will be much simpler. Having said all that, I think the point you wanted to make was: if the command load balancing functions can not be done in the xxxx/IP transport then no one should do it. That is clearly NOT an option, since the various Wedge Drivers do that today and will do that in the future. So the remaining point is: considering only the iSCSI & xxxx/IP layers, do we want load balancing to be done only in the xxxx/IP layer or can it be done in the iSCSI layer. The iSCSI Device Driver might be easier to create, if the xxxx/IP layer did the Balancing. But, I think a number of vendors of iSCSI will be willing to do this balancing if the xxxx/IP does not do it. As long as it is not required for every one to build multiple connections per session (remember: the default in Synchronous is a single connection per Session) then Synchronous permits those that want to build a balancing and generic Alternate Path Retry function, to do so. . . . John L. Hufferd Stephen Bailey @ece.cmu.edu on 09/05/2000 02:44:21 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI Autosense Consensus, Connection next steps > Second, on connections, I haven't seen enough discussion to call > consensus, but I am going to try to narrow the option space and > structure the discussion. Four models for sessions have been > proposed: > > (1) Symmetric - all connections usable for command and data. > (2) Asymmetric - single command connection, others are data. > (3) Split - assign LUN sets to specific connections or pools of > connections. > (4) SCTP - use SCTP's support for multiple connections. For the purposes of shaping the consensus, here's my stance: 1) SCTP should only be pursued if TCP does not admit a viable solution to the iSCSI requirements. Given that there are many iSCSI community participants expressing the belief that iSCSI on TCP IS possible, I believe iSCSI on SCTP proponents are just going to be stuck holding an `I told you so' card. This gates my second statement: 2) multiple connections per session should only be supported if the underlying transport (e.g. SCTP) layer supports it. Obviously, TCP does not presently support this abstraction, so assuming SCTP gets killed for now, I am against multiple connections per session. In general, I am dubious that connection bundling above the transport layer, but below some more application-informed layer (e.g. a wedge driver) will work acceptably. However, if the transport layer provides it, then, by definition, it must work (ha, ha), or at least it's not iSCSI's place to say that it won't. Steph From owner-ips@ECE.cmu.edu Wed Sep 06 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22236; 5 Sep 2000 20:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA02537 for ips-outgoing; Tue, 5 Sep 2000 19:53:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA02529 for ; Tue, 5 Sep 2000 19:53:44 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA66846; Tue, 5 Sep 2000 19:47:47 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id RAA81290; Tue, 5 Sep 2000 17:53:42 -0600 Importance: Normal Subject: RE: iSCSI Autosense Consensus, Connection next steps To: Joshua Tseng Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 5 Sep 2000 16:52:39 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/05/2000 05:53:42 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, I think some of your points are correct. But we also need to examine a few statements. First, the round robin approach of balancing, if only done in the manor you specified, would of course cause every thing to slow to the speed of the slowest link. But this is a well known problem, and general managed by a modified round robin so that it continues to other connections if one of the connections is still busy (several different way of determining this). In fact this would be a problem in the current wedge drivers today used with Fibre Channels if only a blind round robin approach was used. Your point about the target perhaps needing additional memory, is probably correct. This is also solvable on the initiator side (by a number of techniques) however, the target can not be sure that the initiator will have the right techniques, so it must assume that it needs additional memory. This seems like a correct point. If this is a worry, however, the Target can just reject additional conversations per Session. The counter I would make to my last point, however, would be if Asynchronous was used, I could still could handle multiple data ports and at least get that kind of balancing. My question then is: does Data Balancing on multiple connections cause a similar need for additional memory as does the synchronous connections with Multiple Commands on different connections. On the other hand, think that your point about link recovery not being (significantly) more difficult with Async then Sync is essentially correct. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/05/2000 02:54:56 PM Sent by: owner-ips@ece.cmu.edu To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI Autosense Consensus, Connection next steps David, If I may give my input on the TCP connection model discussion, I'd like to advocate the asymmetric model for the following reasons. Using multiple TCP connections, each TCP connection will likely have different performance characteristics. If this is so, then some commands may reach the target out of order from other commands sent down different TCP connections. Commands sent down the faster connection will then need to be buffered until commands sent down slower connections arrive. This will lead to the following problems that I can think of: 1) There will be increased memory requirements for the iSCSI target, which must buffer commands and related data and reorder them before passing them on to the SCSI controller. 2) If the initiator cannot sense the performance characteristics of each TCP connection, then interactions between initiator and target will be dependent on the slowest TCP connection. For example, if there are five TCP connections and one of them is extraordinarily slow, and the initiator merely round-robins the commands down each of the five connections, all interactions between initiator and target will be only as fast as the slowest connection. In addition, it has been stated that detecting and recovering from a failed asymmetric control connection is more complicated and difficult than detecting and recovering from a failed symmetric data/command TCP connection. I do not understand why this is so. In both cases, iSCSI initiator will need to abort the lost command AND those following the lost command, and resend all commands in a new or different TCP connection. In the symmetric case, iSCSI commands received by a target following a lost command will not be acknowledged, since ExpStatRN is a cummulative windowing mechanism. Josh Tseng -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Sunday, September 03, 2000 8:20 AM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Cc: Black_David@emc.com Subject: iSCSI Autosense Consensus, Connection next steps > It looks like we have consensus - but the chairmen have the call. Indeed we do, and I apologize for the delay, due to an inability to get connected. First, on Autosense, I believe that rough consensus exists for iSCSI to require Autosense. I have seen only one objection; If anyone other than Doug Otis disagrees with this, please say so on the list. Second, on connections, I haven't seen enough discussion to call consensus, but I am going to try to narrow the option space and structure the discussion. Four models for sessions have been proposed: (1) Symmetric - all connections usable for command and data. (2) Asymmetric - single command connection, others are data. (3) Split - assign LUN sets to specific connections or pools of connections. (4) SCTP - use SCTP's support for multiple connections. I have only seen one message suggesting/supporting (3) Split, so I believe rough consensus exists to not pursue that model any further. If anyone other than Paul von Stamwitz disagrees, and believes that the Split model should be pursued, please say so on the list with technical rationale. I haven't seen enough discussion on use of SCTP instead of TCP to call consensus. I am concerned that this list is not going to produce a consensus on the issue phrased in that fashion (i.e., pick exactly one). The one point on which there does seem to be consensus is that SCTP is a considerably younger protocol that TCP, and hence the likely timelines to availability of hardware acceleration seem to favor TCP. OTOH, this sort of future prediction can easily miss the mark. I can see two possible ways to make progress in this area: - An off-line design team to do an intensive evaluation of SCTP vs. TCP for iSCSI. - Recognize the merits of SCTP as well as TCP, plan for both with the anticipation that TCP will be used first. The latter makes more sense to me, because it appears to be more amenable to consensus, avoids the investment of effort in the design team, and avoids having the prospect of abandoning TCP hanging over the heads of those engaged in TCP-specific work. The practical implication of proceeding in this fashion is that SCTP-friendliness becomes an additional criteria to use in evaluating iSCSI design proposals. Proceeding in this fashion is only a proposal at this juncture -- please comment on whether this is the right way to proceed (to me or on the list). The specification of sessions for iSCSI over TCP needs to proceed, so under the assumption that either the "plan for both" path will be pursued, or that we can't wait for the design team's conclusions, and hence need to work on TCP in case it is selected, I want to summarize the tradeoffs between the Symmetric and Asymmetric session models, in the aim of simulating more discussions so that we can get to consensus. [X] The Symmetric model imposes additional work on implementations that do not use multiple connections because they will have to implement the command numbering. The Asymmetric model does not do this. [Y] The Asymmetric model makes dealing with a failed control connection more complicated than the symmetric model because agreement between the two sides is required to establish a new control connection (or use an existing data connection for control). I have seen the discussion of [X] favoring Asymmetric for systems that will not support multiple TCP connections per iSCSI session. I also note (as an example of applying the SCTP-friendliness from above) that [X] favoring Asymmetric also applies to SCTP, as there seems to be no point in using multiple SCTP sessions in a single iSCSI session. We need to give the folks working on revising the iSCSI draft some direction on this issue in fairly short order, so comments are hereby solicited ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 06 03:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23097; 5 Sep 2000 23:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA06539 for ips-outgoing; Tue, 5 Sep 2000 21:53:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA06527 for ; Tue, 5 Sep 2000 21:53:03 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Tue, 5 Sep 2000 18:52:55 -0700 Message-ID: From: Joshua Tseng To: John Hufferd/San Jose/IBM , Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Consensus, Connection next steps Date: Tue, 5 Sep 2000 18:52:54 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, >My question then is: does Data Balancing on multiple >connections cause a similar need for additional memory as does the >synchronous connections with Multiple Commands on different connections. In the symmetric model, the target must be prepared to buffer all of the commands and associated unsolicited data PDU's that may be sent between ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that or discard commands. But in order to realize the performance benefits of command windowing, the target must expand the window in order to let the initiator to "let her rip!" and put multiple commands in flight. It seems to me that the existing symmetric model adds complexity which would only be of benefit if the target has lots of memory. So my question is this: The command windowing mechanism seems to require lots of resources (including memory) and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to implement, but does all this really buy very much? In the end, symmetric implementations may end up discarding commands anyway, as in the asymmetric case, unless they want to install globs of memory. WRT the load balancing algorithm implemented by iSCSI, anything other than round robin will require more complexity to sense which connection is over-utilized or poorly performing. Once again, what does the added complexity buy us? I'm not an development engineer, but my guess is getting iSCSI to talk to TCP to get this info and use it isn't cheap. In asymmetric, commands are sent in-order down a relatively less-utilized command TCP connection. A slow TCP connection only affects unfortunate exchanges whose data PDU's happen to use that connection, and not all sessions between initiator and target. Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Tuesday, September 05, 2000 4:53 PM To: Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Consensus, Connection next steps Joshua, I think some of your points are correct. But we also need to examine a few statements. First, the round robin approach of balancing, if only done in the manor you specified, would of course cause every thing to slow to the speed of the slowest link. But this is a well known problem, and general managed by a modified round robin so that it continues to other connections if one of the connections is still busy (several different way of determining this). In fact this would be a problem in the current wedge drivers today used with Fibre Channels if only a blind round robin approach was used. Your point about the target perhaps needing additional memory, is probably correct. This is also solvable on the initiator side (by a number of techniques) however, the target can not be sure that the initiator will have the right techniques, so it must assume that it needs additional memory. This seems like a correct point. If this is a worry, however, the Target can just reject additional conversations per Session. The counter I would make to my last point, however, would be if Asynchronous was used, I could still could handle multiple data ports and at least get that kind of balancing. My question then is: does Data Balancing on multiple connections cause a similar need for additional memory as does the synchronous connections with Multiple Commands on different connections. On the other hand, think that your point about link recovery not being (significantly) more difficult with Async then Sync is essentially correct. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/05/2000 02:54:56 PM Sent by: owner-ips@ece.cmu.edu To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI Autosense Consensus, Connection next steps David, If I may give my input on the TCP connection model discussion, I'd like to advocate the asymmetric model for the following reasons. Using multiple TCP connections, each TCP connection will likely have different performance characteristics. If this is so, then some commands may reach the target out of order from other commands sent down different TCP connections. Commands sent down the faster connection will then need to be buffered until commands sent down slower connections arrive. This will lead to the following problems that I can think of: 1) There will be increased memory requirements for the iSCSI target, which must buffer commands and related data and reorder them before passing them on to the SCSI controller. 2) If the initiator cannot sense the performance characteristics of each TCP connection, then interactions between initiator and target will be dependent on the slowest TCP connection. For example, if there are five TCP connections and one of them is extraordinarily slow, and the initiator merely round-robins the commands down each of the five connections, all interactions between initiator and target will be only as fast as the slowest connection. In addition, it has been stated that detecting and recovering from a failed asymmetric control connection is more complicated and difficult than detecting and recovering from a failed symmetric data/command TCP connection. I do not understand why this is so. In both cases, iSCSI initiator will need to abort the lost command AND those following the lost command, and resend all commands in a new or different TCP connection. In the symmetric case, iSCSI commands received by a target following a lost command will not be acknowledged, since ExpStatRN is a cummulative windowing mechanism. Josh Tseng -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Sunday, September 03, 2000 8:20 AM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Cc: Black_David@emc.com Subject: iSCSI Autosense Consensus, Connection next steps > It looks like we have consensus - but the chairmen have the call. Indeed we do, and I apologize for the delay, due to an inability to get connected. First, on Autosense, I believe that rough consensus exists for iSCSI to require Autosense. I have seen only one objection; If anyone other than Doug Otis disagrees with this, please say so on the list. Second, on connections, I haven't seen enough discussion to call consensus, but I am going to try to narrow the option space and structure the discussion. Four models for sessions have been proposed: (1) Symmetric - all connections usable for command and data. (2) Asymmetric - single command connection, others are data. (3) Split - assign LUN sets to specific connections or pools of connections. (4) SCTP - use SCTP's support for multiple connections. I have only seen one message suggesting/supporting (3) Split, so I believe rough consensus exists to not pursue that model any further. If anyone other than Paul von Stamwitz disagrees, and believes that the Split model should be pursued, please say so on the list with technical rationale. I haven't seen enough discussion on use of SCTP instead of TCP to call consensus. I am concerned that this list is not going to produce a consensus on the issue phrased in that fashion (i.e., pick exactly one). The one point on which there does seem to be consensus is that SCTP is a considerably younger protocol that TCP, and hence the likely timelines to availability of hardware acceleration seem to favor TCP. OTOH, this sort of future prediction can easily miss the mark. I can see two possible ways to make progress in this area: - An off-line design team to do an intensive evaluation of SCTP vs. TCP for iSCSI. - Recognize the merits of SCTP as well as TCP, plan for both with the anticipation that TCP will be used first. The latter makes more sense to me, because it appears to be more amenable to consensus, avoids the investment of effort in the design team, and avoids having the prospect of abandoning TCP hanging over the heads of those engaged in TCP-specific work. The practical implication of proceeding in this fashion is that SCTP-friendliness becomes an additional criteria to use in evaluating iSCSI design proposals. Proceeding in this fashion is only a proposal at this juncture -- please comment on whether this is the right way to proceed (to me or on the list). The specification of sessions for iSCSI over TCP needs to proceed, so under the assumption that either the "plan for both" path will be pursued, or that we can't wait for the design team's conclusions, and hence need to work on TCP in case it is selected, I want to summarize the tradeoffs between the Symmetric and Asymmetric session models, in the aim of simulating more discussions so that we can get to consensus. [X] The Symmetric model imposes additional work on implementations that do not use multiple connections because they will have to implement the command numbering. The Asymmetric model does not do this. [Y] The Asymmetric model makes dealing with a failed control connection more complicated than the symmetric model because agreement between the two sides is required to establish a new control connection (or use an existing data connection for control). I have seen the discussion of [X] favoring Asymmetric for systems that will not support multiple TCP connections per iSCSI session. I also note (as an example of applying the SCTP-friendliness from above) that [X] favoring Asymmetric also applies to SCTP, as there seems to be no point in using multiple SCTP sessions in a single iSCSI session. We need to give the folks working on revising the iSCSI draft some direction on this issue in fairly short order, so comments are hereby solicited ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 06 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23614; 6 Sep 2000 1:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA09133 for ips-outgoing; Tue, 5 Sep 2000 23:26:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA09126 for ; Tue, 5 Sep 2000 23:25:58 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id B16A48E39; Tue, 5 Sep 2000 20:24:00 -0700 (PDT) To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 05 Sep 2000 20:24:00 -0700 In-Reply-To: Joshua Tseng's message of "Tue, 5 Sep 2000 14:54:56 -0700" Message-ID: Lines: 54 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Asymmetric connections may be the way to go. Let's look at characteristics of control vs. data: * iSCSI control - low bandwidth (handled on a CPU) - should generally be processed in FIFO order - complex state machines (probably software implemented) - flow control nice - doesn't need to end up in any special buffers (can be processed in-place) * iSCSI data/ready to transmit (rtt) - potentially high bandwidth - with appropriate headers, segments can be processed entirely out-of-order - simple data transfer state machine - no flow control on data needed, just congestion control - rtt's need to be flow-controlled, but that can be done with a simple credit mechanism - data needs to end up in special buffers (e.g. buffer cache) The processing requirements of the data and control seem quite asymmetric. Mixing them on one TCP channel makes separating out the data and control more difficult. Conversely, by putting control and data on separate TCP connections, you can use the flow label (IP addresses, TCP ports) to determine whether to route this along the control or data path in your machine. So, I think there is value in the proposal that we use 1 control connection and up to n separate dedicated data/RTT connections. The symmetric, connection allegiance case was developed when we were smoking some really good stuff in Haifa. It really addresses a corner case: minimizing the communication needs between NICs in systems that will stripe requests across NICs. However, somewhere along the line, we forgot to optimize for the single NIC case. ------------------ Also using a separate TCP connections may allow us to migrate one day to an architecture where a different node in the IP network returns the actual data from a SCSI READ! Client <---- iSCSI Control ----> Storage Controller <---- TBD ----> Disk array ^ ^ +-------------------------------iSCSI Data protocol -----------------+ -Costa From owner-ips@ECE.cmu.edu Wed Sep 06 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23682; 6 Sep 2000 1:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA09511 for ips-outgoing; Tue, 5 Sep 2000 23:39:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA09504 for ; Tue, 5 Sep 2000 23:39:44 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id XAA95916 for ; Tue, 5 Sep 2000 23:27:34 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id VAA31688 for ; Tue, 5 Sep 2000 21:39:43 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI Autosense Consensus, Connection next steps To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 5 Sep 2000 20:34:18 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/05/2000 09:39:42 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I think Joshua has made some interesting points, do the rest of you have some supportive, or contrary opinions about what he has said? . . . John L. Hufferd Joshua Tseng on 09/05/2000 06:52:54 PM To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng cc: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Consensus, Connection next steps John, >My question then is: does Data Balancing on multiple >connections cause a similar need for additional memory as does the >synchronous connections with Multiple Commands on different connections. In the symmetric model, the target must be prepared to buffer all of the commands and associated unsolicited data PDU's that may be sent between ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that or discard commands. But in order to realize the performance benefits of command windowing, the target must expand the window in order to let the initiator to "let her rip!" and put multiple commands in flight. It seems to me that the existing symmetric model adds complexity which would only be of benefit if the target has lots of memory. So my question is this: The command windowing mechanism seems to require lots of resources (including memory) and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to implement, but does all this really buy very much? In the end, symmetric implementations may end up discarding commands anyway, as in the asymmetric case, unless they want to install globs of memory. WRT the load balancing algorithm implemented by iSCSI, anything other than round robin will require more complexity to sense which connection is over-utilized or poorly performing. Once again, what does the added complexity buy us? I'm not an development engineer, but my guess is getting iSCSI to talk to TCP to get this info and use it isn't cheap. In asymmetric, commands are sent in-order down a relatively less-utilized command TCP connection. A slow TCP connection only affects unfortunate exchanges whose data PDU's happen to use that connection, and not all sessions between initiator and target. Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Tuesday, September 05, 2000 4:53 PM To: Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Consensus, Connection next steps Joshua, I think some of your points are correct. But we also need to examine a few statements. First, the round robin approach of balancing, if only done in the manor you specified, would of course cause every thing to slow to the speed of the slowest link. But this is a well known problem, and general managed by a modified round robin so that it continues to other connections if one of the connections is still busy (several different way of determining this). In fact this would be a problem in the current wedge drivers today used with Fibre Channels if only a blind round robin approach was used. Your point about the target perhaps needing additional memory, is probably correct. This is also solvable on the initiator side (by a number of techniques) however, the target can not be sure that the initiator will have the right techniques, so it must assume that it needs additional memory. This seems like a correct point. If this is a worry, however, the Target can just reject additional conversations per Session. The counter I would make to my last point, however, would be if Asynchronous was used, I could still could handle multiple data ports and at least get that kind of balancing. My question then is: does Data Balancing on multiple connections cause a similar need for additional memory as does the synchronous connections with Multiple Commands on different connections. On the other hand, think that your point about link recovery not being (significantly) more difficult with Async then Sync is essentially correct. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/05/2000 02:54:56 PM Sent by: owner-ips@ece.cmu.edu To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI Autosense Consensus, Connection next steps David, If I may give my input on the TCP connection model discussion, I'd like to advocate the asymmetric model for the following reasons. Using multiple TCP connections, each TCP connection will likely have different performance characteristics. If this is so, then some commands may reach the target out of order from other commands sent down different TCP connections. Commands sent down the faster connection will then need to be buffered until commands sent down slower connections arrive. This will lead to the following problems that I can think of: 1) There will be increased memory requirements for the iSCSI target, which must buffer commands and related data and reorder them before passing them on to the SCSI controller. 2) If the initiator cannot sense the performance characteristics of each TCP connection, then interactions between initiator and target will be dependent on the slowest TCP connection. For example, if there are five TCP connections and one of them is extraordinarily slow, and the initiator merely round-robins the commands down each of the five connections, all interactions between initiator and target will be only as fast as the slowest connection. In addition, it has been stated that detecting and recovering from a failed asymmetric control connection is more complicated and difficult than detecting and recovering from a failed symmetric data/command TCP connection. I do not understand why this is so. In both cases, iSCSI initiator will need to abort the lost command AND those following the lost command, and resend all commands in a new or different TCP connection. In the symmetric case, iSCSI commands received by a target following a lost command will not be acknowledged, since ExpStatRN is a cummulative windowing mechanism. Josh Tseng -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Sunday, September 03, 2000 8:20 AM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Cc: Black_David@emc.com Subject: iSCSI Autosense Consensus, Connection next steps > It looks like we have consensus - but the chairmen have the call. Indeed we do, and I apologize for the delay, due to an inability to get connected. First, on Autosense, I believe that rough consensus exists for iSCSI to require Autosense. I have seen only one objection; If anyone other than Doug Otis disagrees with this, please say so on the list. Second, on connections, I haven't seen enough discussion to call consensus, but I am going to try to narrow the option space and structure the discussion. Four models for sessions have been proposed: (1) Symmetric - all connections usable for command and data. (2) Asymmetric - single command connection, others are data. (3) Split - assign LUN sets to specific connections or pools of connections. (4) SCTP - use SCTP's support for multiple connections. I have only seen one message suggesting/supporting (3) Split, so I believe rough consensus exists to not pursue that model any further. If anyone other than Paul von Stamwitz disagrees, and believes that the Split model should be pursued, please say so on the list with technical rationale. I haven't seen enough discussion on use of SCTP instead of TCP to call consensus. I am concerned that this list is not going to produce a consensus on the issue phrased in that fashion (i.e., pick exactly one). The one point on which there does seem to be consensus is that SCTP is a considerably younger protocol that TCP, and hence the likely timelines to availability of hardware acceleration seem to favor TCP. OTOH, this sort of future prediction can easily miss the mark. I can see two possible ways to make progress in this area: - An off-line design team to do an intensive evaluation of SCTP vs. TCP for iSCSI. - Recognize the merits of SCTP as well as TCP, plan for both with the anticipation that TCP will be used first. The latter makes more sense to me, because it appears to be more amenable to consensus, avoids the investment of effort in the design team, and avoids having the prospect of abandoning TCP hanging over the heads of those engaged in TCP-specific work. The practical implication of proceeding in this fashion is that SCTP-friendliness becomes an additional criteria to use in evaluating iSCSI design proposals. Proceeding in this fashion is only a proposal at this juncture -- please comment on whether this is the right way to proceed (to me or on the list). The specification of sessions for iSCSI over TCP needs to proceed, so under the assumption that either the "plan for both" path will be pursued, or that we can't wait for the design team's conclusions, and hence need to work on TCP in case it is selected, I want to summarize the tradeoffs between the Symmetric and Asymmetric session models, in the aim of simulating more discussions so that we can get to consensus. [X] The Symmetric model imposes additional work on implementations that do not use multiple connections because they will have to implement the command numbering. The Asymmetric model does not do this. [Y] The Asymmetric model makes dealing with a failed control connection more complicated than the symmetric model because agreement between the two sides is required to establish a new control connection (or use an existing data connection for control). I have seen the discussion of [X] favoring Asymmetric for systems that will not support multiple TCP connections per iSCSI session. I also note (as an example of applying the SCTP-friendliness from above) that [X] favoring Asymmetric also applies to SCTP, as there seems to be no point in using multiple SCTP sessions in a single iSCSI session. We need to give the folks working on revising the iSCSI draft some direction on this issue in fairly short order, so comments are hereby solicited ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 06 06:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23793; 6 Sep 2000 1:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA11357 for ips-outgoing; Wed, 6 Sep 2000 00:46:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA11351 for ; Wed, 6 Sep 2000 00:46:18 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA44894 for ; Wed, 6 Sep 2000 00:40:20 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id WAA122632 for ; Tue, 5 Sep 2000 22:46:16 -0600 Importance: Normal Subject: Re: iSCSI Autosense Consensus, Connection next steps To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 5 Sep 2000 21:44:29 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/05/2000 10:46:16 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk OK, you Sync and Async Fans, let me put up an Async strawman and I would like to read your constructive comments (please leave out the comments that say I am a flaming idiot for even suggesting this strawman). Situation: Client (Host) system has more then one NIC that supports at least TCP/IP offload, and perhaps some parts of iSCSI. Approach: Project to the upper layers -- "n" ports where "n" is the number of NICs Establish "n" sessions For each session, place one Command Stream TCP/IP connection on a NIC No session's Command Stream is on a the same NIC as any other session For each session, place one Data Stream TCP/IP connection on each of the other NICs Configuration Summary: Each NIC supports one session's Command connection, and "n-1" Data connections from other sessions. Results: The upper layers (SCSI drivers or Wedge drivers) will use the "n" ports as they currently do today. If the wedge drivers perform load balancing today, they can do the same thing tomorrow with this configuration. Data can flow back on any of the NICs, any NIC that did not have the Command Flow for a specific session. Does this Configuration and Approach make since? ---------------------------------------------------------------------------- Variant 1: Project only one upper layer port, and perform a type of round robin scheduling between the sessions (and thus the NICs). Does this Configuration and Approach make since? ---------------------------------------------------------------------------- Variant 2: Project only one upper layer port, and perform a type of modified round robin scheduling that forces all the commands for any specific LU to any Session that has a current I/O, for that LU, in some way outstanding. If the LU has no other I/O outstanding, then place the I/O on any Session. Does this Configuration and Approach make since? --------------------------------------------------------------------------- . . . John L. Hufferd From owner-ips@ECE.cmu.edu Wed Sep 06 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24155; 6 Sep 2000 3:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA13107 for ips-outgoing; Wed, 6 Sep 2000 02:05:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA13103 for ; Wed, 6 Sep 2000 02:05:32 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA295978 for ; Wed, 6 Sep 2000 08:05:01 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA21304 for ; Wed, 6 Sep 2000 08:05:00 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256952.002169B3 ; Wed, 6 Sep 2000 08:04:57 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 6 Sep 2000 08:47:50 +0300 Subject: Re: Target Reset handling Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk OK. Sorry for the confusion. We are on agreement. Julo "Mallikarjun C." on 05/09/2000 21:44:57 Please respond to cbm@rose.hp.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Target Reset handling Julian, I was thinking that the C-bit would go into bytes 2/3 (currently reserved) of the SCSI Task Management Command PDU (as defined in section 3.7 of draft-satran-iscsi-01.txt) - which does not have a CDB. So, I guess it's part of the iSCSI protocol headers, as you were suggesting. Thanks. -- Mallikarjun M/S 5601 Networked Storage Architecture HP Storage Organization Hewlett-Packard, Roseville. cbm@rose.hp.com >Mallikarjun, > >Is the C bit hidden in the CDB? We as a rule avoided examining the CDB >(iSCSI never has to). If we can make it external then we are in wild >agreement. > >Regards, >Julo > >"Mallikarjun C." on 05/09/2000 20:03:56 > >Please respond to cbm@rose.hp.com > >To: ips@ece.cmu.edu >cc: (bcc: Julian Satran/Haifa/IBM) >Subject: Re: Target Reset handling > > > > >Julian, > >Thanks for the suggestion. > >I am in broad agreement with this kind of definition, if the >rest of the folks accept it. I would however suggest a single >Target Reset task management function (in accordance with SAM-2), >and provide the choice of a "cold-reset" with a "C" bit in the >Task Management Command PDU. Assuming that we do this, let me >try to summarize our tentative agreement: > >o Target Reset task management requires a response, unless the > the C-bit is set in which case the sessions would be terminated > as well and no response can be expected. > >o A Unit Attention AER shall be reported to all currently logged-in > initiators, in accordance with the provisions contained in clauses > 7.9 and 8.3.6 of SPC-2 (basically, to be reported only if agreed > in advance between the initiator and the LU). > >Regards. >-- >Mallikarjun >M/S 5601 >Networked Storage Architecture >HP Storage Organization >Hewlett-Packard, Roseville. >cbm@rose.hp.com > > > >>Mallikarjun, >> >>How about having two different function: >> >>target warm-reset (connections stay) and target cold-rest (connections get >>also reset)? >> >>Julo >> From owner-ips@ECE.cmu.edu Wed Sep 06 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24168; 6 Sep 2000 3:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA13816 for ips-outgoing; Wed, 6 Sep 2000 02:39:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA13811 for ; Wed, 6 Sep 2000 02:38:59 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 2F03761B for ; Wed, 6 Sep 2000 02:38:58 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id XAA18761 for ; Tue, 5 Sep 2000 23:38:56 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA3147 for ; Tue, 5 Sep 2000 23:38:53 -0700 Message-ID: <39B5E671.B191FF9A@agilent.com> Date: Tue, 05 Sep 2000 23:38:41 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Ok, maybe I'm missing something, but as I read the latest (July 10) draft, I see this data/status thing working as follows. I'm going to use psuedo code, so that (perhaps) everyone might understand... (Note that this whole discussion is only applicable to a target read operation) /* as a target... */ if ((this is the last iSCSI data PDU to send to fullfill the SCSI command) and (this SCSI STATUS is GOOD (except for residual counts)) then { set the "S" bit in the iSCSI DATA PDU header /* setting the "S" bit explicitely defines "end of exchange" */ update the "iSCSI Status" field in the iSCSI DATA PDU header if (there is a residual count) then { set the "O" or "U" bits and update the "Residual Count" field of the iSCSI DATA PDU header } send the last iSCSI DATA PDU (completing the SCSI command) } else { clear the "S" bit in the iSCSI DATA PDU header send the iSCSI DATA PDU if (the last iSCSI DATA PDU was sent) { create the iSCSI RESPONSE PDU send it } } Now, what exactly is the big issue that's required all this discussion? -Matt Wakeley Agilent Technologies Douglas Otis wrote: > Julo, > > After painfully understanding your status here not there conversation and > being ensured Autosense is now a fact of life, rather than placing status > within a READ PDU, perhaps you should consider simply revising the > definition of the flags and use a common DATA PDU. Change the Status bit to > indicate Error Detected and perhaps add a bit to signal End of Exchange. > From End of Exchange you could deduce Good Status. I see little benefit in > status presented twice or dummy status sent with partial exchanges. It only > seems to make room for conflicts. (The two values not agreeing, I, on the > other hand, will never agree.) At what point would the O, and U flags be > valid? Would this be a result of an error? If so, perhap these flags could > be left to response PDU. > > Doug From owner-ips@ECE.cmu.edu Wed Sep 06 09:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24917; 6 Sep 2000 5:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA15956 for ips-outgoing; Wed, 6 Sep 2000 04:17:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA15951 for ; Wed, 6 Sep 2000 04:17:26 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id ED1CD7A5 for ; Wed, 6 Sep 2000 01:17:25 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id BAA19853 for ; Wed, 6 Sep 2000 01:17:25 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA330E for ; Wed, 6 Sep 2000 01:17:21 -0700 Message-ID: <39B5FD85.A033376B@agilent.com> Date: Wed, 06 Sep 2000 01:17:09 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > Dear colleagues, > > With all the heated debate about multiple vs. single connection a request I > made a while ago got no significant reply (neither for nor against). > > The request was to consider a proposal made by Kalman Meth to reconsider > the asymmetric model with the addition of a path selection made by the > initiator. One downside I see to this proposal is that for a host with multiple iSCSI nics, it will have to issue two commands (in this order): one to the "data path nic" to describe where the data is to be read from/written to, and one to the "control path nic" to send the command. This would be communication to two cards instead of one. Also, note that the communication to the data path nic MUST complete before the control path nic sends the command, requiring some kind of sync mechanism. > This proposal allows removing the command counters - as commands use a > single TCP connection. The single connection can also be a shared > data+control connection. For the case of the command connection failure and fail-over to a new connection, I don't see how you can get away from the command counters. When a fail over occurs, you will need some way of finding out what commands made it to the target and which didn't. The easiest way to do this is with command numbering. > > > In case of multiple connection the data path to be used is selected and > maintained until the command ends. > > Thanks, > Julo -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Wed Sep 06 09:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24919; 6 Sep 2000 5:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA15332 for ips-outgoing; Wed, 6 Sep 2000 03:56:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA15327 for ; Wed, 6 Sep 2000 03:56:29 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 81E83408 for ; Wed, 6 Sep 2000 03:56:28 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id AAA19637 for ; Wed, 6 Sep 2000 00:56:26 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA32C4 for ; Wed, 6 Sep 2000 00:56:23 -0700 Message-ID: <39B5F89B.7B38549A@agilent.com> Date: Wed, 06 Sep 2000 00:56:11 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: bridging issues References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > Another bridging issue is the initiator tag-translation mechanism. > iSCSI is requiring the initiator issue a unique tag for each task but does > not require any structure for the tag. This way an implementer may choose > the most efficient structure for his implementation (e.g., > use the address of it's control structure representing the task as a tag). > > FCP on the other hand demands some structure for the tag. FCP does not demand any structure for the tag. The only difference between FCP and iSCSI is the FCP tag is 16 bits (the "OX_ID") and the iSCSI tag is 32 bits. FCP does not demand how those 16 bits are used. > > > A stateless bridge would want to pass the tag unchanged from initiator to > target or translate it with a stateless rule. well, it would be a problem to try and translate a 32 bit space (iscsi) into a 16 bit space (FCP). > > > If we restrict the design space to gateways that present a "homogenous" > view of the device conglomerate behind them (e.g., all devices are FCP) > and we may pass this information to the initiator - then the initiator can > use the slightly more restrictive FCP tagging mechanism. > > Julo -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Wed Sep 06 10:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25137; 6 Sep 2000 6:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA17016 for ips-outgoing; Wed, 6 Sep 2000 05:05:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA17008 for ; Wed, 6 Sep 2000 05:05:51 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id 6A353DC7 for ; Wed, 6 Sep 2000 02:05:50 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id CAA26898 for ; Wed, 6 Sep 2000 02:05:49 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA33F9 for ; Wed, 6 Sep 2000 02:05:43 -0700 Message-ID: <39B608DC.212728AF@agilent.com> Date: Wed, 06 Sep 2000 02:05:32 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Target Reset handling References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk One could make the argument that a "big hammer" reset is desireable, especially in new implementations that might hang - the initiator could attempt to "power cycle" the box if it hangs. The problem is that at this reset is at a really high level (the iSCSI/TCP/IP stack must be working in the first place to even receive and process the command). FC loop defined a "link primitive" that could be used the reset the entire device (the LIP_RESET) that required only the lowest level link primitive state machine to be running to process it - and yank on a global reset line. If iSCSI/TCP/IP is in a state that it can receive and decode the "big hammer" reset iSCSI PDU, I don't think it would be in a state that it needs to be reset. -Matt Wakeley Agilent Technologies Robert Snively wrote: > I do not believe that the added complexity of a cold and hot > reset is appropriate. SCSI-2 had such a concept, and it was ultimately > determined that there was no benefit to it, and the complexity was > immense. > > I strongly feel that the SCSI Target Reset should do exactly > no more and no less than the corresponding function does in all > other SCSI devices. > > If a session/connection reset is required, it should be performed > by session/connection control mechanisms already defined by TCP/IP. > Its effect on the SCSI target should be defined, but may or may > not be the same as the SCSI Target Reset. FCP-2 provides some examples > of the kind of things that you may choose to reset in table 4. > > No "C" bit should be used. > > The session/connection reset should not be in the CDB or in the > task management functions. > > Bob > > > -----Original Message----- > > From: Mallikarjun C. [mailto:cbm@rose.hp.com] > > Sent: Tuesday, September 05, 2000 11:14 AM > > To: ips@ece.cmu.edu > > Subject: Re: Target Reset handling (fwd) > > > > > > Apologies if this is a second copy, I had not seen my copy of > > this posting or on the web archive even after more than an hour.... > > -- > > Mallikarjun > > > > > > Forwarded message: > > > > Date: Tue, 05 Sep 2000 10:03:56 PDT > > To: ips@ece.cmu.edu > > Subject: Re: Target Reset handling > > In-Reply-To: ; > > from "julian_satran@il.ibm.com" at Sep 4, 100 8:08 am > > Status: RO > > > > Julian, > > > > Thanks for the suggestion. > > > > I am in broad agreement with this kind of definition, if the > > rest of the folks accept it. I would however suggest a single > > Target Reset task management function (in accordance with SAM-2), > > and provide the choice of a "cold-reset" with a "C" bit in the > > Task Management Command PDU. Assuming that we do this, let me > > try to summarize our tentative agreement: > > > > o Target Reset task management requires a response, unless the > > the C-bit is set in which case the sessions would be terminated > > as well and no response can be expected. > > > > o A Unit Attention AER shall be reported to all currently logged-in > > initiators, in accordance with the provisions contained in clauses > > 7.9 and 8.3.6 of SPC-2 (basically, to be reported only if agreed > > in advance between the initiator and the LU). > > > > Regards. > > -- > > Mallikarjun > > M/S 5601 > > Networked Storage Architecture > > HP Storage Organization > > Hewlett-Packard, Roseville. > > cbm@rose.hp.com > > > > > > > > >Mallikarjun, > > > > > >How about having two different function: > > > > > >target warm-reset (connections stay) and target cold-rest > > (connections get > > >also reset)? > > > > > >Julo > > > > > >"Mallikarjun C." on 03/09/2000 06:07:07 > > > > > >Please respond to cbm@rose.hp.com > > > > > >To: ips@ece.cmu.edu > > >cc: (bcc: Julian Satran/Haifa/IBM) > > >Subject: Re: Target Reset handling > > > > > > > > > > > > > > >Julian, > > > > > >Let me try again, I was arguing that the protocol stack (I > > assume you > > >mean the state information in various layers by this) and the TCP > > >connections be *not* reset - since I do not see a need for the SCSI > > >transport mechanism to be reset/cleared in the context of > > "SCSI target > > >reset". I would again request your attention on the FC > > precedent, and > > >the software expectations of a confirmed target reset. > > Logging in after > > >an arbitrary "long time" and assuming the target reset to > > be complete is > > >simply unreliable. > > > > > >I do not see a need for a special "iSCSI reset". I believe that the > > >SCSI target reset task management request is completely adequate to > > >address our requirements. All I am proposing is a change > > in its current > > >definition in view of the various reasons I had already > > stated, and the > > >benefits of making this change (not the least of which is SAM-2 > > >compliance). Addition of a new reset mechanism would also > > defeat our > > >common goal to keep a fairly lean protocol. > > > > > >The question of security context needs to addressed > > regardless of the > > >SCSI transport behavior on a target reset. If security > > context is being > > >designed as part of the transient operating environment (as > > opposed to > > >say, creating/modifying mode pages), then my first guess > > would be that > > >it has to be reset as well to initial values, on a target > > reset. I am > > >not sure what you implied, but are you suggesting that the security > > >context cannot be reset with TCP connections living across > > a SCSI target > > >reset? Or, did I totally miss something? > > > > > >Regards. > > > > > >Mallikarjun > > >M/S 5601 > > >Networked Storage Architecture > > >HP Storage Organization > > >Hewlett-Packard, Roseville. > > >cbm@rose.hp.com > > > > > > > > > From owner-ips@ECE.cmu.edu Wed Sep 06 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26265; 6 Sep 2000 10:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA01929 for ips-outgoing; Wed, 6 Sep 2000 09:36:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA01925 for ; Wed, 6 Sep 2000 09:36:15 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id IAA02994; Wed, 6 Sep 2000 08:36:38 -0500 Message-ID: <39B64866.961ABDCD@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 08:36:38 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: csapuntz@cisco.com CC: ips@ece.cmu.edu, Allison Mankin , Scott Bradner Subject: Re: iSCSI Autosense Consensus, Connection next steps References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Ok, explain something to me... csapuntz@cisco.com wrote: > > Asymmetric connections may be the way to go. > > Let's look at characteristics of control vs. data: > > * iSCSI control > - low bandwidth (handled on a CPU) > - should generally be processed in FIFO order > - complex state machines (probably software implemented) > - flow control nice > - doesn't need to end up in any special buffers > (can be processed in-place) > > * iSCSI data/ready to transmit (rtt) > - potentially high bandwidth > - with appropriate headers, segments can be > processed entirely out-of-order > - simple data transfer state machine > - no flow control on data needed, just congestion control > - rtt's need to be flow-controlled, but > that can be done with a simple credit mechanism > - data needs to end up in special buffers (e.g. buffer cache) > I understand how you will obtain all of the above with two TCP connecions. I can even support this, considering the 1 for control is "low bandwidth" (note: I can't support N data connections.. only 1). But my question is to the Data connection. How do you get "no flow control on data needed, just congestion control" on a TCP connection? > The processing requirements of the data and control seem quite > asymmetric. Mixing them on one TCP channel makes separating out the > data and control more difficult. > > Conversely, by putting control and data on separate TCP connections, > you can use the flow label (IP addresses, TCP ports) to determine > whether to route this along the control or data path in your machine. > > So, I think there is value in the proposal that we use 1 control > connection and up to n separate dedicated data/RTT connections. No I can NOT agree with N seperate dedicated data/RTT connections. I think you should go look at Sally's email that was posted here earlier... here it is... ************************************************************************** Franco Travostino wrote: > > A bit of Related Work ... On the forever-young 1 vs. N TCP connections > subject, you may be interested in the results shown in the '97 "An > Application-Level Solution to TCP's Satellite Inefficiencies" paper > (available at http://jarok.cs.ohiou.edu/projects/satellite/) and on Sally > Floyd's comment hereafter. The intersection with satellite links is quite > accidental, and the arguments/results apply above and beyond satellite links. > > -franco > > >From majordom@ISI.EDU Fri Dec 6 23:05:58 1996 > >To: fred@cisco.com (Fred Baker) > >Cc: end2end-tf@isi.edu > >Subject: Re: Related paper/re:satellites > >Date: Fri, 06 Dec 1996 23:05:47 PST > >From: Sally Floyd > >Sender: owner-end2end-tf@ISI.EDU > >Precedence: bulk > >Content-Length: 2309 > >X-Lines: 44 > > > >Fred - > > > >>You might be interested in reviewing this paper, which is what I'm > >>discussing with Karen Hansen, Dan Shell, and the folks from Comsat. It > >>relates to some TCP/Satellite work being done at NASA Lewis Research > >>Center. > > > >Basically (from a quick read), the paper, on "An Application-Level > >Solution to TCP's Satellite Inefficiencies", recommends breaking a > >single TCP connection into multiple connections at the > >application level, to increase throughput on satellite circuits. > >It is not too surprising that this increases the TCP throughput, but it > >is still not a good idea. For a single TCP connection, a single packet > >drop results in the throughput for that connection being cut by half, > >and then increased by roughly one packet per RTT. For a TCP connection > >that has instead been separated into N different TCP connections, a > >single packet drop results in one of the N connections, receiving > >1/N-th of the total bandwidth, having its throughput cut in half. Thus > >the bandwidth of the aggregate connection has its bandwidth reduced to > >(1 - 1/(2N))-th of its former bandwidth - that is, the larger the value > >for N, the smaller the aggregate bandwidth is cut. And then, because > >each TCP connection continues to increase its congestion window by one > >packet per RTT, for those TCP connections that have not yet reached the > >receiver's advertised window, the aggregate TCP connections together > >increase their bandwidth by up to N packets per RTT. > > > >Summarizing, splitting a TCP connection into N separate connections > >simply increases the aggressiveness of the TCP congestion control. > >Meaning that this TCP is now more likely to "steal" bandwidth from > >other TCP connections in times of congestion. And increasing the > >aggressiveness of the TCP congestion control too far (by choosing N too > >large) is counterproductive even for the aggregate TCP connection, as > >the paper shows. > > > >I would suggest that this is exactly the kind of development for which > >[RED is needed]. > > > >- Sally > > > > > > ****************************************************************** In my mind it will be a hard sell to the IESG to ask for multiple connections for high-bandwidth data? Scott/Allision any comments on this? > > The symmetric, connection allegiance case was developed when we were > smoking some really good stuff in Haifa. It really addresses a > corner case: minimizing the communication needs between NICs in systems > that will stripe requests across NICs. > > However, somewhere along the line, we forgot to optimize for the > single NIC case. > > ------------------ > > Also using a separate TCP connections may allow us to migrate one > day to an architecture where a different node in the IP network > returns the actual data from a SCSI READ! > > Client <---- iSCSI Control ----> Storage Controller <---- TBD ----> Disk array > ^ ^ > +-------------------------------iSCSI Data protocol -----------------+ > > -Costa -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 06 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26267; 6 Sep 2000 10:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA01409 for ips-outgoing; Wed, 6 Sep 2000 09:24:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA01402 for ; Wed, 6 Sep 2000 09:24:40 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id IAA02944; Wed, 6 Sep 2000 08:23:49 -0500 Message-ID: <39B64565.2BAB4A4F@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 08:23:49 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: <0F31E5C394DAD311B60C00E029101A0704100F4F@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David: Nice summary... please see my thoughts to the transport issues below... Black_David@emc.com wrote: > > > It looks like we have consensus - but the chairmen have the call. > > Indeed we do, and I apologize for the delay, due to an inability to > get connected. > > First, on Autosense, I believe that rough consensus exists for > iSCSI to require Autosense. I have seen only one objection; > If anyone other than Doug Otis disagrees with this, please > say so on the list. > > Second, on connections, I haven't seen enough discussion to call > consensus, but I am going to try to narrow the option space and > structure the discussion. Four models for sessions have been > proposed: > > (1) Symmetric - all connections usable for command and data. > (2) Asymmetric - single command connection, others are data. > (3) Split - assign LUN sets to specific connections or pools of > connections. > (4) SCTP - use SCTP's support for multiple connections. > > I have only seen one message suggesting/supporting (3) Split, so > I believe rough consensus exists to not pursue that model > any further. If anyone other than Paul von Stamwitz disagrees, > and believes that the Split model should be pursued, please > say so on the list with technical rationale. > > I haven't seen enough discussion on use of SCTP instead of > TCP to call consensus. I am concerned that this list is not > going to produce a consensus on the issue phrased in that > fashion (i.e., pick exactly one). The one point on which there > does seem to be consensus is that SCTP is a considerably > younger protocol that TCP, and hence the likely timelines > to availability of hardware acceleration seem to favor TCP. > OTOH, this sort of future prediction can easily miss the mark. > > I can see two possible ways to make progress in this area: > - An off-line design team to do an intensive evaluation of SCTP > vs. TCP for iSCSI. > - Recognize the merits of SCTP as well as TCP, plan for both > with the anticipation that TCP will be used first. > The latter makes more sense to me, because it appears to > be more amenable to consensus, avoids the investment > of effort in the design team, and avoids having the prospect > of abandoning TCP hanging over the heads of those engaged > in TCP-specific work. The practical implication of proceeding > in this fashion is that SCTP-friendliness becomes an additional > criteria to use in evaluating iSCSI design proposals. Proceeding > in this fashion is only a proposal at this juncture -- please > comment on whether this is the right way to proceed (to me > or on the list). > > The specification of sessions for iSCSI over TCP needs to > proceed, so under the assumption that either the "plan for > both" path will be pursued, or that we can't wait for the > design team's conclusions, and hence need to work on TCP > in case it is selected, I want to summarize the tradeoffs > between the Symmetric and Asymmetric session models, > in the aim of simulating more discussions so that we can > get to consensus. > > [X] The Symmetric model imposes additional work on implementations > that do not use multiple connections because they will > have to implement the command numbering. The > Asymmetric model does not do this. > [Y] The Asymmetric model makes dealing with a failed control > connection more complicated than the symmetric model > because agreement between the two sides is required > to establish a new control connection (or use an existing > data connection for control). > > I have seen the discussion of [X] favoring Asymmetric for systems > that will not support multiple TCP connections per iSCSI session. > I also note (as an example of applying the SCTP-friendliness > from above) that [X] favoring Asymmetric also applies to SCTP, as > there seems to be no point in using multiple SCTP sessions in a > single iSCSI session. > I don't really think that it matters in the SCTP sense if you are using X and Y... In fact in Y it is easier to establish a new association/connection (if you were using seperate SCTP associations which I would strongly recommend AGAINST). This is because SCTP supports both a TCPish flavored model and a UDPish model.. In the UDPish model you just send to a peer (that you do NOT have a association with) and the SCTP stack will automatically create the association. There is not need to keep connection state etc. Now on the multiple connections issue. I really strongly advise AGAINST this. I could understand 1 control connection and 1 data connection, but if you do N data connection you are breaking the congestion control algorithms. Earlier a pointer to some comments from Sally Floyd were posted.. please go have a look at these (I did) and I totally agree with Sally. Having N connections violates the CC of the network and is a very very BAD idea. I do understand why you would want to have your control (with TCP) in a seperate connection. As long as this is a low bandwidth type of connection it is probably ok. You would not need to do this with SCTP though, since the multiple streams allow you to have 1 stream dedicated to control and N streams dedicated to data. And (optionally) you can even have a no or one retransmit stream for data (as well as full reliablility). I will post more comments to this thread, I do see it is long, I have been updating the reference implementation of SCTP and have been disconnected from email a bit...:0 > We need to give the folks working on revising the iSCSI draft some > direction on this issue in fairly short order, so comments are hereby > solicited ... > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 06 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26287; 6 Sep 2000 10:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA01621 for ips-outgoing; Wed, 6 Sep 2000 09:29:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA01616 for ; Wed, 6 Sep 2000 09:28:57 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id IAA02974; Wed, 6 Sep 2000 08:29:21 -0500 Message-ID: <39B646B1.816538D6@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 08:29:21 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Stephen Bailey CC: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: <10009052144.AA04226@candide.cs.uchicago.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen Bailey wrote: > > > Second, on connections, I haven't seen enough discussion to call > > consensus, but I am going to try to narrow the option space and > > structure the discussion. Four models for sessions have been > > proposed: > > > > (1) Symmetric - all connections usable for command and data. > > (2) Asymmetric - single command connection, others are data. > > (3) Split - assign LUN sets to specific connections or pools of > > connections. > > (4) SCTP - use SCTP's support for multiple connections. > > For the purposes of shaping the consensus, here's my stance: > > 1) SCTP should only be pursued if TCP does not admit a viable > solution to the iSCSI requirements. > > Given that there are many iSCSI community participants expressing the > belief that iSCSI on TCP IS possible, I believe iSCSI on SCTP > proponents are just going to be stuck holding an `I told you so' > card. > I would rather have it in the spec and a option and then not have to say "I told you so" but instead say .. lets go try it with SCTP since it is allowed... > This gates my second statement: > > 2) multiple connections per session should only be supported if the > underlying transport (e.g. SCTP) layer supports it. > > Obviously, TCP does not presently support this abstraction, so > assuming SCTP gets killed for now, I am against multiple connections > per session. I strongly concur here. We can NOT have multiple TCP connections and be able to stay TCP friendly to the rest of the internet... R > > In general, I am dubious that connection bundling above the transport > layer, but below some more application-informed layer (e.g. a wedge > driver) will work acceptably. However, if the transport layer > provides it, then, by definition, it must work (ha, ha), or at least > it's not iSCSI's place to say that it won't. > > Steph -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 06 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26282; 6 Sep 2000 10:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA02100 for ips-outgoing; Wed, 6 Sep 2000 09:40:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA02095 for ; Wed, 6 Sep 2000 09:40:11 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id IAA03014; Wed, 6 Sep 2000 08:40:34 -0500 Message-ID: <39B64952.FA2C859A@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 08:40:34 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: John Hufferd/San Jose/IBM CC: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk John: Here are some of mine...(interspersed with Joshua's email) John Hufferd/San Jose/IBM wrote: > > I think Joshua has made some interesting points, do the rest of you have > some supportive, or contrary opinions about what he has said? > . > . > . > John L. Hufferd > > Joshua Tseng on 09/05/2000 06:52:54 PM > > To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng > > cc: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > John, > > >My question then is: does Data Balancing on multiple > >connections cause a similar need for additional memory as does the > >synchronous connections with Multiple Commands on different connections. > > In the symmetric model, the target must be prepared to buffer all of the > commands and associated unsolicited data PDU's that may be sent between > ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that or > discard > commands. But in order to realize the performance benefits of command > windowing, the target must expand the window in order to let the initiator > to "let her rip!" and put multiple commands in flight. It seems to me that > the existing symmetric model adds complexity which would only be of benefit > if the target has lots of memory. So my question is this: The command > windowing mechanism seems to require lots of resources (including memory) > and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to > implement, but does all this really buy very much? In the end, symmetric > implementations may end up discarding commands anyway, as in the asymmetric > case, unless they want to install globs of memory. > > WRT the load balancing algorithm implemented by iSCSI, anything other than > round robin will require more complexity to sense which connection is > over-utilized or poorly performing. Once again, what does the added > complexity buy us? I'm not an development engineer, but my guess is > getting > iSCSI to talk to TCP to get this info and use it isn't cheap. In > asymmetric, commands are sent in-order down a relatively less-utilized > command TCP connection. A slow TCP connection only affects unfortunate > exchanges whose data PDU's happen to use that connection, and not all > sessions between initiator and target. > It will be highly stack dependant on WHAT TCP information is available (if ANY) and how you access it. In many cases the ONLY way you can get at the types of information this would require is by reading the /dev/memory and gloming through the kernel.. el.yucko.. and even this may not be able to get all the information you need. R > Josh > > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Tuesday, September 05, 2000 4:53 PM > To: Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > Joshua, > I think some of your points are correct. But we also need to examine a few > statements. > > First, the round robin approach of balancing, if only done in the manor you > specified, would of course cause every thing to slow to the speed of the > slowest link. But this is a well known problem, and general managed by a > modified round robin so that it continues to other connections if one of > the connections is still busy (several different way of determining this). > In fact this would be a problem in the current wedge drivers today used > with Fibre Channels if only a blind round robin approach was used. > > Your point about the target perhaps needing additional memory, is probably > correct. This is also solvable on the initiator side (by a number of > techniques) however, the target can not be sure that the initiator will > have the right techniques, so it must assume that it needs additional > memory. This seems like a correct point. If this is a worry, however, the > Target can just reject additional conversations per Session. The counter I > would make to my last point, however, would be if Asynchronous was used, I > could still could handle multiple data ports and at least get that kind of > balancing. My question then is: does Data Balancing on multiple > connections cause a similar need for additional memory as does the > synchronous connections with Multiple Commands on different connections. > > On the other hand, think that your point about link recovery not being > (significantly) more difficult with Async then Sync is essentially correct. > > . > . > . > John L. Hufferd > > Joshua Tseng @ece.cmu.edu on 09/05/2000 02:54:56 > PM > > Sent by: owner-ips@ece.cmu.edu > > To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > David, > > If I may give my input on the TCP connection model discussion, I'd like > to advocate the asymmetric model for the following reasons. Using > multiple TCP connections, each TCP connection will likely have > different performance characteristics. If this is so, then some commands > may reach the target out of order from other commands sent down different > TCP connections. Commands sent down the faster connection will then need > to be buffered until commands sent down slower connections arrive. This > will lead to the following problems that I can think of: > > 1) There will be increased memory requirements for the iSCSI target, > which must buffer commands and related data and reorder them before passing > them on to the SCSI controller. > > 2) If the initiator cannot sense the performance characteristics of > each TCP connection, then interactions between initiator and target will > be dependent on the slowest TCP connection. For example, if there are > five TCP connections and one of them is extraordinarily slow, and the > initiator merely round-robins the commands down each of the five > connections, > all interactions between initiator and target will be only as fast as the > slowest connection. > > In addition, it has been stated that detecting and recovering from a failed > asymmetric control connection is more complicated and difficult than > detecting and recovering from a failed symmetric data/command TCP > connection. > I do not understand why this is so. In both cases, iSCSI initiator will > need > to abort the lost command AND those following the lost command, and resend > all > commands in a new or different TCP connection. In the symmetric case, > iSCSI > commands received by a target following a lost command will not be > acknowledged, > since ExpStatRN is a cummulative windowing mechanism. > > Josh Tseng > > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Sunday, September 03, 2000 8:20 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Cc: Black_David@emc.com > Subject: iSCSI Autosense Consensus, Connection next steps > > > It looks like we have consensus - but the chairmen have the call. > > Indeed we do, and I apologize for the delay, due to an inability to > get connected. > > First, on Autosense, I believe that rough consensus exists for > iSCSI to require Autosense. I have seen only one objection; > If anyone other than Doug Otis disagrees with this, please > say so on the list. > > Second, on connections, I haven't seen enough discussion to call > consensus, but I am going to try to narrow the option space and > structure the discussion. Four models for sessions have been > proposed: > > (1) Symmetric - all connections usable for command and data. > (2) Asymmetric - single command connection, others are data. > (3) Split - assign LUN sets to specific connections or pools of > connections. > (4) SCTP - use SCTP's support for multiple connections. > > I have only seen one message suggesting/supporting (3) Split, so > I believe rough consensus exists to not pursue that model > any further. If anyone other than Paul von Stamwitz disagrees, > and believes that the Split model should be pursued, please > say so on the list with technical rationale. > > I haven't seen enough discussion on use of SCTP instead of > TCP to call consensus. I am concerned that this list is not > going to produce a consensus on the issue phrased in that > fashion (i.e., pick exactly one). The one point on which there > does seem to be consensus is that SCTP is a considerably > younger protocol that TCP, and hence the likely timelines > to availability of hardware acceleration seem to favor TCP. > OTOH, this sort of future prediction can easily miss the mark. > > I can see two possible ways to make progress in this area: > - An off-line design team to do an intensive evaluation of SCTP > vs. TCP for iSCSI. > - Recognize the merits of SCTP as well as TCP, plan for both > with the anticipation that TCP will be used first. > The latter makes more sense to me, because it appears to > be more amenable to consensus, avoids the investment > of effort in the design team, and avoids having the prospect > of abandoning TCP hanging over the heads of those engaged > in TCP-specific work. The practical implication of proceeding > in this fashion is that SCTP-friendliness becomes an additional > criteria to use in evaluating iSCSI design proposals. Proceeding > in this fashion is only a proposal at this juncture -- please > comment on whether this is the right way to proceed (to me > or on the list). > > The specification of sessions for iSCSI over TCP needs to > proceed, so under the assumption that either the "plan for > both" path will be pursued, or that we can't wait for the > design team's conclusions, and hence need to work on TCP > in case it is selected, I want to summarize the tradeoffs > between the Symmetric and Asymmetric session models, > in the aim of simulating more discussions so that we can > get to consensus. > > [X] The Symmetric model imposes additional work on implementations > that do not use multiple connections because they will > have to implement the command numbering. The > Asymmetric model does not do this. > [Y] The Asymmetric model makes dealing with a failed control > connection more complicated than the symmetric model > because agreement between the two sides is required > to establish a new control connection (or use an existing > data connection for control). > > I have seen the discussion of [X] favoring Asymmetric for systems > that will not support multiple TCP connections per iSCSI session. > I also note (as an example of applying the SCTP-friendliness > from above) that [X] favoring Asymmetric also applies to SCTP, as > there seems to be no point in using multiple SCTP sessions in a > single iSCSI session. > > We need to give the folks working on revising the iSCSI draft some > direction on this issue in fairly short order, so comments are hereby > solicited ... > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 06 15:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26477; 6 Sep 2000 11:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA03153 for ips-outgoing; Wed, 6 Sep 2000 10:03:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from boston3.g4.net (boston3.G4.NET [216.177.0.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA01790 for ; Wed, 6 Sep 2000 09:32:40 -0400 (EDT) Received: from Trebiajws (ppp66.qcy.dialup.G4.net [140.186.53.66]) by boston3.g4.net (8.9.3/8.9.3) with SMTP id JAA84082 for ; Wed, 6 Sep 2000 09:36:43 -0400 (EDT) (envelope-from jssmart@nhinternet.com) From: "James Smart" To: "ips" Subject: Command Queue Depth (was asymmetric/Symmetric) Date: Wed, 6 Sep 2000 09:37:08 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Josh, Wanted to double-check on the scsi-isms implied by the statements below, and post a new question on managing command buffering ... >>My question then is: does Data Balancing on multiple >>connections cause a similar need for additional memory as does the >>synchronous connections with Multiple Commands on different connections. > >In the symmetric model, the target must be prepared to buffer all of the >commands and associated unsolicited data PDU's that may be sent between >ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that or discard >commands. But in order to realize the performance benefits of command >windowing, the target must expand the window in order to let the initiator >to "let her rip!" and put multiple commands in flight. It seems to me that >the existing symmetric model adds complexity which would only be of benefit >if the target has lots of memory. So my question is this: The command >windowing mechanism seems to require lots of resources (including memory) >and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to >implement, but does all this really buy very much? In the end, symmetric >implementations may end up discarding commands anyway, as in the asymmetric >case, unless they want to install globs of memory. First - I assume that "discard commands" actually means completing the command with a SCSI status of BUSY or QUEUE FULL - *NOT* simply dropping the command on the floor. As per previous statements on the reflector, OS's and existing drivers expect older-style SCSI behaviors - dropping of commands from lack of resources expect QUEUE FULL or BUSY indicators to invoke backoff algorithms. OS's typically don't deal well with i/o timeouts for error recovery. Second, are the buffering expectations above in line with existing SCSI/FC devices ? Most scsi devices today, which support command queuing, do indeed support buffering up to N commands (N being the maximum queue depth and is shared across all initiators). Simple disk drives typically support queue depths of 16, 32 or 64 (all to a single lun), while large controllers accept hundreds (shared across multiple luns). Lastly, this topic touches on one of the more sensitive areas in the use of SCSI over packetized interfaces - that of managing the command queue. In parallel SCSI, it was dealt with more efficiently, as the QUEUE FULL or BUSY status was received immediately on the interlocked bus before an additional commands could be sent. However, in FC and iSCSI, the commands will be in the pipe, and there will (are) storms of commands bombarding the target by multiple initiators. As mentioned, unless status was sent back to invoke the backoff algorithms, things continued unabated. I've seen FCP targets fully consumed just trying to hold off these storms. It gets worse as the initiators can be different OS's, with different backoff policies (some do onesy/twosy based on command completion and noting depth levels to manage how much they keep outstanding; while others simply delay for a time, then let the gates wide open to whatever maximum they believe they can send (e.g. a storm)). Given the varied and uncoordinated rates at which the initiators can beat on the command queue resource, there are times where things melt down both on the target and on the initiator backoff/retry algorithms such that little real work gets achieved. I know this last issue is not generic to iSCSI and may be classified as a configuration error. However - to the group - has iSCSI considered any mechanism to report the command queue depth (buffering) available to an initiator ? (my guess is that the delta between ExpCmdRN and MaxCmdRN via the login response implies this). Further - any thoughts on how the resources could then be multiplexed across multiple initiators (thus the ExpCmdRN/MaxCmdRN can't really apply any more). -- James -------------------- James Smart james.smart@nhinternet.com From owner-ips@ECE.cmu.edu Wed Sep 06 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26557; 6 Sep 2000 11:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06041 for ips-outgoing; Wed, 6 Sep 2000 11:08:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06032 for ; Wed, 6 Sep 2000 11:08:12 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e86F8Bx14880 for ; Wed, 6 Sep 2000 10:08:11 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA04945; Wed, 6 Sep 00 10:06:25 -0500 Message-Id: <10009061506.AA04945@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps In-Reply-To: Message from "John Hufferd/San Jose/IBM" of "Tue, 05 Sep 2000 16:17:20 PDT." Date: Wed, 06 Sep 2000 10:06:04 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > You talked about Transport layer and Wedge layers. I guess I do not know > exactly what to call iSCSI if given these two options, other then to > consider iSCSI part of the SCSI Transport (layer). It is a device specific > driver, which in this case deals with iSCSI Devices. I guess we have a terminological gap here. Here's how I understand the terms. Generally, I believe the term `transport layer' is used to describe the uppermost layer in a stack that carries data, without any particular thought to what the data is. For example, ST, TCP, UDP, FC-2. Above the transport layer are layers that know what the particular data being carried is e.g. FTP, NFS, HTTP. Clearly this `knowing' is relative---an HTTP layer doesn't know everything about the data it's carrying, but it knows more than nothing. These more specialized protocols are called upper layer protocols (ULPs). The iSCSI protocol is one of these ULPs. The layers above iSCSI are the parts of a SCSI stack. Typically this includes a mux/demux layer, and, above that, a set of SCSI peripheral drivers (disk, tape, cluster communication, etc..). SCSI wedge drivers are in this SCSI stack, hooked somehow (on top, bottom or possibly in the guts), to the mux/demux layer. > I do not think we should call iSCSI a Wedge Driver. Agreed. My mistake if I suggested that that's how I was thinking of it. iSCSI is a protocol. An iSCSI driver is both a SCSI port driver and a network protocol driver, certainly not a wedge driver. > Having said all that, I think the point you wanted to make was: if the > command load balancing functions can not be done in the xxxx/IP transport > then no one should do it. No. The point I am trying to make is that if CONNECTION-level load balancing (or connection bundling) services are not available from the transport layer (TCP, ST, STCP, etc.), iSCSI should not take this responsibility for itself. I did not say no one should do it. The clear implication if iSCSI does not directly support transport connection bundling, is that a layer above iSCSI will do it. That is what wedge drivers do presently. When we started bringing up FC controllers, existing ||SCSI multipathing code worked as-is. Nobody in the legacy (%^) network SCSI community (FCP) seems to feel a compelling need to add transport layer connection bundling. We certainly didn't see any need to reinvent that wheel early on, and I still don't see it. > I think that David is correct the only point that I was making was that if > iSCSI was able to do the load balancing and some Generic Alternate Path > recovery, that the Vendor Specific Wedge Drivers will be much simpler. Or, my guess is that the support just won't be used. After all, these same wedge drivers already support all types of different SCSI interfaces (||, FCP, SST etc.) concurrently in a more or less interface independent way. Why gunk up a piece of mission-critical code with additional paths that don't offer any user-visible improvement? If we could get a credible wedge driver person to come say `absolutely, connection bundling below the SCSI stack is what we need!', I'll roll over. As it is, I think we're trying to build this feature primarily on speculation. My second objection is that I think it's an architectural error to start filling iSCSI with what are essentially transport-layer enhancements. iSCSI says what data to move, and it is the transport layer's requirement to move it as fast and as reliably as possible by any means available. If you don't dig what the current transport has to offer, use another one. The advantage a ULP gets by not trying to `enhance' the transport-layer, is that it is relatively immune to modifications to the underlying transport. None of the arguments made for iSCSI connection bundling are unique to storage. If connection bundling is so compelling, TCP or some relative will develop its own connection bundling mechanism, and then iSCSI can take advantage of it more or less transparently. Steph From owner-ips@ECE.cmu.edu Wed Sep 06 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26555; 6 Sep 2000 11:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA04659 for ips-outgoing; Wed, 6 Sep 2000 10:39:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.spinnakernet.com (client-141-151-129-90.bellatlantic.net [141.151.129.90]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA04577 for ; Wed, 6 Sep 2000 10:37:47 -0400 (EDT) Received: from kazar-pc (kazar-pc.spinnakernet.com [10.1.1.110]) by mail.spinnakernet.com (8.9.3/8.8.7) with ESMTP id KAA26583; Wed, 6 Sep 2000 10:37:41 -0400 Message-Id: <4.2.2.20000906081014.00ac2450@pop.icubed.com> X-Sender: kazar@pop.icubed.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 06 Sep 2000 10:46:39 -0400 To: ips@ece.cmu.edu From: Mike Kazar Subject: a vote for asymmetric connections in a session Cc: kazar@spinnakernet.com Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, Here are my reasons for preferring the asymmetric connection model to the symmetric connection model, in decreasing order of importance. 1. Implementing a sliding window protocol for seqRn processing is likely to be as hard, or harder, as implementing TCP. Getting something that works will be easy, but getting something that works well, even when various iSCSI TCP connections are running at different rates is likely to be an "interesting" research problem. Yet connections running at different rates is likely to occur in real life quite frequently, sometimes due to different connections seeing different packet drop events in a somewhat congested network, sometimes due to different connections taking different paths (the whole *goal* of multiple connections in a session, after all). Furthermore, the sliding window state machine for a given session's seqRn processing is running at N times the event rate of an individual TCP connection's state machine (N is the # of connections in a session), making it even more challenging to implement. 2. It is easier to implement target-specific load sharing mechanisms if the target gets to choose, perhaps with input from the host, the connection on which a particular transfer should be performed. In the symmetric proposal, the host chooses the connection to use completely on its own. Mike (kazar@spinnakernet.com) From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26734; 6 Sep 2000 12:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA07209 for ips-outgoing; Wed, 6 Sep 2000 11:30:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA07199 for ; Wed, 6 Sep 2000 11:30:17 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id LAA00745 for ips@ece.cmu.edu; Wed, 6 Sep 2000 11:30:17 -0400 (EDT) Date: Wed, 6 Sep 2000 11:30:17 -0400 (EDT) From: Scott Bradner Message-Id: <200009061530.LAA00745@newdev.harvard.edu> To: ips@ece.cmu.edu Subject: multiple sessions Sender: owner-ips@ece.cmu.edu Precedence: bulk people might want to take a look at http://www.ietf.org/internet-drafts/draft-ietf-ecm-cm-01.txt - it offers a way to deal with multiple streams between two points Scott From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26742; 6 Sep 2000 12:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA07839 for ips-outgoing; Wed, 6 Sep 2000 11:43:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA07829 for ; Wed, 6 Sep 2000 11:43:51 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id 2049C883 for ; Wed, 6 Sep 2000 08:43:50 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id IAA29596 for ; Wed, 6 Sep 2000 08:43:49 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA3892 for ; Wed, 6 Sep 2000 08:43:45 -0700 Message-ID: <39B66625.905D3377@agilent.com> Date: Wed, 06 Sep 2000 08:43:34 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > I am strongly in favor of the asymmetric model but I would like to hear all > possible arguments against it. > > Julo Ok, how about this: The biggest isssue with the "asymmetric" model is that it is NOT "asymmetric" when there is only 1 TCP connection. When there is only one connection, it is the "symmetric" model - I have to do both commands and data on the same connection. Then, when there is more than one TCP connection, I switch over to a different behavior. Does the "asymmetric" model mandate more than 1 connection? (maybe it should) The whole purpose of having multiple TCP connections was to enable some kind of "data stripping". If there are only two links, and one is "low bandwidth", then effectively only one is being used. I suppose that if there are "n" physical links between an initiator/target, that "n+1" TCP connections could be established, with the extra 1 running on one of the connections to transport the commands. -Matt From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26738; 6 Sep 2000 12:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA07432 for ips-outgoing; Wed, 6 Sep 2000 11:34:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA07427 for ; Wed, 6 Sep 2000 11:34:36 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id ECE694C0 for ; Wed, 6 Sep 2000 08:34:33 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id IAA29420 for ; Wed, 6 Sep 2000 08:34:32 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA381C; Wed, 6 Sep 2000 08:34:25 -0700 Message-ID: <39B663F4.15648FC@agilent.com> Date: Wed, 06 Sep 2000 08:34:13 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Joshua Tseng , ips@ece.cmu.edu Subject: Re: multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua Tseng wrote: > >One downside I see to this proposal is that for a host with multiple iSCSI > >nics, it will have to issue two commands (in this order): one to the "data > path > >nic" to describe where the data is to be read from/written to, and one to > the > >"control path nic" to send the command. This would be communication to two > >cards instead of one. Also, note that the communication to the data path > nic > >MUST complete before the control path nic sends the command, requiring some > >kind of sync mechanism. > > I don't understand why you think two commands are necessary. A single > commanddown the command/control connection will work. This is my view of how > an > async model would work: I'm sorry for the confusion. When I said "two commands" I was referring to whatever register read/write operations the host CPU would have to perform to the NIC hardware itself to set up the intended SCSI I/O. Not a command that would be delivered to the target. So, in my example, if the command is sent across NIC "A" and the data across NIC "B", then the host will have to communicate to NIC "B" the scatter/gather list information, as well as what to do with the SCSI status, then it will have to communicate to NIC "A" to send the iSCSI command. That's just that much more control overhead that needs to be performed across the host's I/O busses. -Matt From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26771; 6 Sep 2000 12:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06349 for ips-outgoing; Wed, 6 Sep 2000 11:14:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06341 for ; Wed, 6 Sep 2000 11:13:57 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id 35E24E8C for ; Wed, 6 Sep 2000 08:13:55 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id IAA29049 for ; Wed, 6 Sep 2000 08:13:54 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA3780 for ; Wed, 6 Sep 2000 08:13:47 -0700 Message-ID: <39B65F1F.2BA8110B@agilent.com> Date: Wed, 06 Sep 2000 08:13:35 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: <4.2.2.20000906081014.00ac2450@pop.icubed.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike Kazar wrote: > Folks, > > Here are my reasons for preferring the asymmetric connection model to the > symmetric connection model, in decreasing order of importance. > > 1. Implementing a sliding window protocol for seqRn processing is likely > to be as hard, or harder, as implementing TCP. Getting something that > works will be easy, but getting something that works well, even when > various iSCSI TCP connections are running at different rates is likely to > be an "interesting" research problem. Yet connections running at different > rates is likely to occur in real life quite frequently, sometimes due to > different connections seeing different packet drop events in a somewhat > congested network, sometimes due to different connections taking different > paths (the whole *goal* of multiple connections in a session, after all). Well, I disagree. Implementing sliding windows is not that hard. If sliding windows were the toughest thing to implement in iSCSI, this would be a cake walk. > > > Furthermore, the sliding window state machine for a given session's seqRn > processing is running at N times the event rate of an individual TCP > connection's state machine (N is the # of connections in a session), making > it even more challenging to implement. I disagree again. The TCP sliding window SM is running at once per TCP segment. If just iSCSI command messages where crammed down the connection, that means the session's seqRn processing will be done once every 48 bytes. Every time an iSCSI message is received, there is a lot of processing invoked. Adding to the list a simple sliding window algorithm is not a challenge. > > > 2. It is easier to implement target-specific load sharing mechanisms if > the target gets to choose, perhaps with input from the host, the connection > on which a particular transfer should be performed. In the symmetric > proposal, the host chooses the connection to use completely on its own. And it's good that the host chooses the connection. It is the one that has set up DMA structures on a particular connection to get the data in/out with minimal CPU intervention. > > > Mike > (kazar@spinnakernet.com) In my opinion, these are not "good" reasons to vote for one or the other. Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26766; 6 Sep 2000 12:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06424 for ips-outgoing; Wed, 6 Sep 2000 11:15:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06407 for ; Wed, 6 Sep 2000 11:15:04 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id RAA25718 for ; Wed, 6 Sep 2000 17:14:27 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA40916 for ; Wed, 6 Sep 2000 17:14:27 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256952.0053AB10 ; Wed, 6 Sep 2000 17:13:52 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 6 Sep 2000 16:35:56 +0300 Subject: Re: iSCSI Autosense Consensus, Connection next steps Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk One minor point as clarification - on both a multiple connection symmetric and asymmetric schemes the minimum will be 1 connection. The difference vs. draft-00 is that the "lead" connection will be able to carry both control and data while the rest will carry only data (or take-over the lead). Julo From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26783; 6 Sep 2000 12:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06408 for ips-outgoing; Wed, 6 Sep 2000 11:15:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06402 for ; Wed, 6 Sep 2000 11:14:59 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id RAA267464 for ; Wed, 6 Sep 2000 17:14:26 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA40910 for ; Wed, 6 Sep 2000 17:14:26 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256952.0053AC20 ; Wed, 6 Sep 2000 17:13:55 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 6 Sep 2000 17:24:18 +0300 Subject: RE: iSCSI Autosense Consensus, Connection next steps Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, I agree that asymmetric is simpler and I personally would prefer it - but playing devil's advocate (!) - the simple backpresure mechanism and TCP windows and send buffers proportional to the pipe rate will get you the skew down to acceptable levels. The total buffer set needed is still proportional to the total peak rate. And you will need similar mechanisms to regulate an asymmetric flow too (a bit simpler though!). I am strongly in favor of the asymmetric model but I would like to hear all possible arguments against it. Julo Joshua Tseng on 06/09/2000 04:52:54 Please respond to Joshua Tseng To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: iSCSI Autosense Consensus, Connection next steps John, >My question then is: does Data Balancing on multiple >connections cause a similar need for additional memory as does the >synchronous connections with Multiple Commands on different connections. In the symmetric model, the target must be prepared to buffer all of the commands and associated unsolicited data PDU's that may be sent between ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that or discard commands. But in order to realize the performance benefits of command windowing, the target must expand the window in order to let the initiator to "let her rip!" and put multiple commands in flight. It seems to me that the existing symmetric model adds complexity which would only be of benefit if the target has lots of memory. So my question is this: The command windowing mechanism seems to require lots of resources (including memory) and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to implement, but does all this really buy very much? In the end, symmetric implementations may end up discarding commands anyway, as in the asymmetric case, unless they want to install globs of memory. WRT the load balancing algorithm implemented by iSCSI, anything other than round robin will require more complexity to sense which connection is over-utilized or poorly performing. Once again, what does the added complexity buy us? I'm not an development engineer, but my guess is getting iSCSI to talk to TCP to get this info and use it isn't cheap. In asymmetric, commands are sent in-order down a relatively less-utilized command TCP connection. A slow TCP connection only affects unfortunate exchanges whose data PDU's happen to use that connection, and not all sessions between initiator and target. Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Tuesday, September 05, 2000 4:53 PM To: Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: iSCSI Autosense Consensus, Connection next steps Joshua, I think some of your points are correct. But we also need to examine a few statements. First, the round robin approach of balancing, if only done in the manor you specified, would of course cause every thing to slow to the speed of the slowest link. But this is a well known problem, and general managed by a modified round robin so that it continues to other connections if one of the connections is still busy (several different way of determining this). In fact this would be a problem in the current wedge drivers today used with Fibre Channels if only a blind round robin approach was used. Your point about the target perhaps needing additional memory, is probably correct. This is also solvable on the initiator side (by a number of techniques) however, the target can not be sure that the initiator will have the right techniques, so it must assume that it needs additional memory. This seems like a correct point. If this is a worry, however, the Target can just reject additional conversations per Session. The counter I would make to my last point, however, would be if Asynchronous was used, I could still could handle multiple data ports and at least get that kind of balancing. My question then is: does Data Balancing on multiple connections cause a similar need for additional memory as does the synchronous connections with Multiple Commands on different connections. On the other hand, think that your point about link recovery not being (significantly) more difficult with Async then Sync is essentially correct. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/05/2000 02:54:56 PM Sent by: owner-ips@ece.cmu.edu To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI Autosense Consensus, Connection next steps David, If I may give my input on the TCP connection model discussion, I'd like to advocate the asymmetric model for the following reasons. Using multiple TCP connections, each TCP connection will likely have different performance characteristics. If this is so, then some commands may reach the target out of order from other commands sent down different TCP connections. Commands sent down the faster connection will then need to be buffered until commands sent down slower connections arrive. This will lead to the following problems that I can think of: 1) There will be increased memory requirements for the iSCSI target, which must buffer commands and related data and reorder them before passing them on to the SCSI controller. 2) If the initiator cannot sense the performance characteristics of each TCP connection, then interactions between initiator and target will be dependent on the slowest TCP connection. For example, if there are five TCP connections and one of them is extraordinarily slow, and the initiator merely round-robins the commands down each of the five connections, all interactions between initiator and target will be only as fast as the slowest connection. In addition, it has been stated that detecting and recovering from a failed asymmetric control connection is more complicated and difficult than detecting and recovering from a failed symmetric data/command TCP connection. I do not understand why this is so. In both cases, iSCSI initiator will need to abort the lost command AND those following the lost command, and resend all commands in a new or different TCP connection. In the symmetric case, iSCSI commands received by a target following a lost command will not be acknowledged, since ExpStatRN is a cummulative windowing mechanism. Josh Tseng -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Sunday, September 03, 2000 8:20 AM To: julian_satran@il.ibm.com; ips@ece.cmu.edu Cc: Black_David@emc.com Subject: iSCSI Autosense Consensus, Connection next steps > It looks like we have consensus - but the chairmen have the call. Indeed we do, and I apologize for the delay, due to an inability to get connected. First, on Autosense, I believe that rough consensus exists for iSCSI to require Autosense. I have seen only one objection; If anyone other than Doug Otis disagrees with this, please say so on the list. Second, on connections, I haven't seen enough discussion to call consensus, but I am going to try to narrow the option space and structure the discussion. Four models for sessions have been proposed: (1) Symmetric - all connections usable for command and data. (2) Asymmetric - single command connection, others are data. (3) Split - assign LUN sets to specific connections or pools of connections. (4) SCTP - use SCTP's support for multiple connections. I have only seen one message suggesting/supporting (3) Split, so I believe rough consensus exists to not pursue that model any further. If anyone other than Paul von Stamwitz disagrees, and believes that the Split model should be pursued, please say so on the list with technical rationale. I haven't seen enough discussion on use of SCTP instead of TCP to call consensus. I am concerned that this list is not going to produce a consensus on the issue phrased in that fashion (i.e., pick exactly one). The one point on which there does seem to be consensus is that SCTP is a considerably younger protocol that TCP, and hence the likely timelines to availability of hardware acceleration seem to favor TCP. OTOH, this sort of future prediction can easily miss the mark. I can see two possible ways to make progress in this area: - An off-line design team to do an intensive evaluation of SCTP vs. TCP for iSCSI. - Recognize the merits of SCTP as well as TCP, plan for both with the anticipation that TCP will be used first. The latter makes more sense to me, because it appears to be more amenable to consensus, avoids the investment of effort in the design team, and avoids having the prospect of abandoning TCP hanging over the heads of those engaged in TCP-specific work. The practical implication of proceeding in this fashion is that SCTP-friendliness becomes an additional criteria to use in evaluating iSCSI design proposals. Proceeding in this fashion is only a proposal at this juncture -- please comment on whether this is the right way to proceed (to me or on the list). The specification of sessions for iSCSI over TCP needs to proceed, so under the assumption that either the "plan for both" path will be pursued, or that we can't wait for the design team's conclusions, and hence need to work on TCP in case it is selected, I want to summarize the tradeoffs between the Symmetric and Asymmetric session models, in the aim of simulating more discussions so that we can get to consensus. [X] The Symmetric model imposes additional work on implementations that do not use multiple connections because they will have to implement the command numbering. The Asymmetric model does not do this. [Y] The Asymmetric model makes dealing with a failed control connection more complicated than the symmetric model because agreement between the two sides is required to establish a new control connection (or use an existing data connection for control). I have seen the discussion of [X] favoring Asymmetric for systems that will not support multiple TCP connections per iSCSI session. I also note (as an example of applying the SCTP-friendliness from above) that [X] favoring Asymmetric also applies to SCTP, as there seems to be no point in using multiple SCTP sessions in a single iSCSI session. We need to give the folks working on revising the iSCSI draft some direction on this issue in fairly short order, so comments are hereby solicited ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 black_david@emc.com Cellular: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26740; 6 Sep 2000 12:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA07374 for ips-outgoing; Wed, 6 Sep 2000 11:33:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA07364 for ; Wed, 6 Sep 2000 11:33:25 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e86FXOx15686 for ; Wed, 6 Sep 2000 10:33:25 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA04977; Wed, 6 Sep 00 10:31:39 -0500 Message-Id: <10009061531.AA04977@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Target Reset handling In-Reply-To: Message from "Matt Wakeley" of "Wed, 06 Sep 2000 02:05:32 PDT." <39B608DC.212728AF@agilent.com> Date: Wed, 06 Sep 2000 10:31:17 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > The problem is that at this reset is at a really high level (the > iSCSI/TCP/IP stack must be working in the first place to even > receive and process the command). Agreed. > If iSCSI/TCP/IP is in a state that it can receive and decode the > "big hammer" reset iSCSI PDU, I don't think it would be in a state > that it needs to be reset. I don't agree with this. I've seen plenty of failure scenarios with FCP targets where they were swallowing packets, but still in the weeds. The FCP target reset task management function cured them in this case. I can't say that what percentage of the total failure modes were in this category. There were certainly also many where the directed LIP (or even power cycle!) was the only way out. The initiator should be allowed to hit anything within reach of a hammer to provide the maximum set of recovery/work-around options. Providing this capability will encourage target vendors to make the hammer work as robustly and forcefully as possible, to avoid the unpleasant alternative of having to follow their products into the field on short notice to fix them while THE CUSTOMER IS WATCHING. Steph From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26761; 6 Sep 2000 12:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06898 for ips-outgoing; Wed, 6 Sep 2000 11:24:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06886 for ; Wed, 6 Sep 2000 11:24:12 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id LAA00697 for ips@ece.cmu.edu; Wed, 6 Sep 2000 11:23:48 -0400 (EDT) Date: Wed, 6 Sep 2000 11:23:48 -0400 (EDT) From: Scott Bradner Message-Id: <200009061523.LAA00697@newdev.harvard.edu> To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session Sender: owner-ips@ece.cmu.edu Precedence: bulk > Implementing sliding windows is not that hard note that the issue is not "just" sliding windows - ips also has to deal with congestion TCP-friendly way - that can get quite complicated Scott From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26773; 6 Sep 2000 12:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06871 for ips-outgoing; Wed, 6 Sep 2000 11:23:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06832 for ; Wed, 6 Sep 2000 11:23:19 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Wed, 6 Sep 2000 08:23:21 -0700 Message-ID: From: Joshua Tseng To: Matt Wakeley , ips@ece.cmu.edu Subject: RE: multiple connections Date: Wed, 6 Sep 2000 08:23:20 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk >One downside I see to this proposal is that for a host with multiple iSCSI >nics, it will have to issue two commands (in this order): one to the "data path >nic" to describe where the data is to be read from/written to, and one to the >"control path nic" to send the command. This would be communication to two >cards instead of one. Also, note that the communication to the data path nic >MUST complete before the control path nic sends the command, requiring some >kind of sync mechanism. I don't understand why you think two commands are necessary. A single command down the command/control connection will work. This is my view of how an async model would work: Data and command/control connections would be pre-established. In the model with one command/control and multiple data connections, the command PDU could go first down the control connection, and have a "data connection handle" field which would specify which data connection will be used. In a write command, unsolicited data may be part of the command PDU, which would be sent down the control connection. The target would respond with a RTT using the data connection specified by the original command, and all subsequent data PDU's would be sent down the specified data connection. In a read command, the target would respond by sending back data on the specified data connection. >For the case of the command connection failure and fail-over to a new >connection, I don't see how you can get away from the command counters. When a >fail over occurs, you will need some way of finding out what commands made it >to the target and which didn't. The easiest way to do this is with command >numbering. Since commands must be delivered sequentially by iSCSI, the lost command is the last command for which there was no reply or acknowledgement. I think the issue is how to detect a lost command, and I'm not sure if command counters will help here either. To detect a lost command, you either need a timeout, or implement something similar to FCP's REC link service command. Neither of these options is a flow control mechanism, and therefore does not solve a major cause of the lost commands--the initiator overrunning the target with too many commands. Another possibility is the command windowing mechanism in the existing iSCSI specification. But then how wide shall the target open the window to allow the initiator to send multiple commands? And at what threshold shall the difference between StatRN and ExpStatRN before the initiator determines that the command has been lost? An alternative proposal is for the target to communicate to the initiator the amount of buffer he has available at iSCSI login. This is a static parameter dependent on the amount of memory available in the target. It is up to the initiator to decide how many commands to put in flight simultaneously. I believe the initiator (not the target, as currently specified by the existing iSCSI spec) is in the best position to decide how many commands to put in flight, because the initiator knows what commands it has coming, and more importantly the size of the command PDU's, including any unsolicited write data, which will affect the buffer usage at the target. Additionally, the initiator knows what recovery mechanisms (if any) it has at its disposal in the event of a lost command, and can decide whether to be aggressive or conservative based on its specific implementation. Comments would be appreciated on this proposed model. Josh -----Original Message----- From: Matt Wakeley [mailto:matt_wakeley@agilent.com] Sent: Wednesday, September 06, 2000 1:17 AM To: ips@ece.cmu.edu Subject: Re: multiple connections julian_satran@il.ibm.com wrote: > Dear colleagues, > > With all the heated debate about multiple vs. single connection a request I > made a while ago got no significant reply (neither for nor against). > > The request was to consider a proposal made by Kalman Meth to reconsider > the asymmetric model with the addition of a path selection made by the > initiator. One downside I see to this proposal is that for a host with multiple iSCSI nics, it will have to issue two commands (in this order): one to the "data path nic" to describe where the data is to be read from/written to, and one to the "control path nic" to send the command. This would be communication to two cards instead of one. Also, note that the communication to the data path nic MUST complete before the control path nic sends the command, requiring some kind of sync mechanism. > This proposal allows removing the command counters - as commands use a > single TCP connection. The single connection can also be a shared > data+control connection. For the case of the command connection failure and fail-over to a new connection, I don't see how you can get away from the command counters. When a fail over occurs, you will need some way of finding out what commands made it to the target and which didn't. The easiest way to do this is with command numbering. > > > In case of multiple connection the data path to be used is selected and > maintained until the command ends. > > Thanks, > Julo -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Wed Sep 06 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26736; 6 Sep 2000 12:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA07760 for ips-outgoing; Wed, 6 Sep 2000 11:42:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA07755 for ; Wed, 6 Sep 2000 11:42:05 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Wed, 6 Sep 2000 08:42:07 -0700 Message-ID: From: Charles Monia To: "Julian Satran (E-mail)" Cc: "Ips (E-mail)" , Charles Monia Subject: RE: bridging issues -- Converging FCP-2 and iSCSI control struct ures Date: Wed, 6 Sep 2000 08:42:06 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julio: In comparing FCP SCSI control structures with iSCSI, there are a lot of fields that are functionally identical. Therefore, as a step towards bridging between iSCSI and FCP, as you mentioned, I went through the exercise of rearranging the transport-independent fields within the iSCSI control structures to reflect the equivalent FCP-2 structures. (See ftp://ftp.t10.org/t10/drafts/fcp2/fcp2r04.pdf, for a copy of FCP-2.) The gory details are below, including a field-by-field mapping between corresponding iSCSI and FCP parameters. The conversion seems fairly straightforward and preserves iSCSI semantics. Functionally, the main differences are: a) Since FCP combines the task management and SCSI command blocks into a single control structure and response, I've done the same thing with the corresponding control blocks as mapped to iSCSI. i.e.. In the reformatted version, the iSCSI task management command and return status are merged with the corresponding iSCSI command and status blocks. b) All iSCSI-specific fields have been collected into special iSCSI headers. I have not gone through the exercise of defining a fixed length iSCSI preamble that fits every case however. c) Since command ordering is handled by iSCSI and TCP/IP, the Command Reference Number in the FCP command block is an iSCSI reserved field. The one change in the FCP command template was the need to provide an encoding for the ABORT TASK management function. (FCP does this by aborting the FC-2 Exchange). An issue for further study is how best to map the mechanism for Asynchronous Event Notification between FCP and iSCSI. By the way, in comparing the two kinds of command control blocks, I noticed that iSCSI does not seem to allocate space for CDBs longer than 16 bytes (unless that's what the "command-data" field was to be used for). Anyhow, assuming the command-data in iSCSI is not intended for this purpose, I defined a separate optional extension for iSCSI to contain this information as described in the iSCSI spec. Issues for further study are a) How best to map the mechanism for Asynchronous Event Notification between FCP and iSCSI. b) An efficient mapping between the iSCSI data transfer model and Fibre channel. 1. iSCSI command structure (iSCSI section 3.2) The iSCSI Command data structure is modified to include the FCP_CMND Payload (see table 22 of the Rev 4 FCP-2 draft standard). iSCSI information is contained in a transport-specific header as shown below, followed by the FCP_CMD template, and terminated with optional command-specific data. +------------------+ | iSCSI Header | +------------------+ | SCSI Command | | Block (modeled | | on FCP-2) | +------------------+ | Additional Data | | (Optional) | +------------------+ 1.1 Contents of iSCSI header: The header up to ExpStatRn is as defined in the iSCSI spec. OpCode Length CmdRN ExpStatRn Initiator Tag Length of "Additional Data" The "Length of Additional Data" is a 32-bit field, new to iSCSI, containing the size of the space reserved for additional, command-dependent data. 1.2 iSCSI mapping of FCP-2 command template: FCP_LUN -- 8-byte Logical Unit Number ISCSI Definition: Same as FCP FCP_CRN -- 1-byte command reference number: iSCSI definition: reserved field Task Attribute -- 3-bit field encoding Head of Queue, Simple, etc. iSCSI definition -- Same as FCP Task Management Flags -- 8-bit field containing bit-significant task management flags. Bit 7 -- Obsolete Bit 6 -- Clear ACA Bit 5 -- Target Reset Bit 3 -- Reserved Bit 2 -- Clear Task Set Bit 1 -- Abort Task Set Bit 0 -- Reserved iSCSI Definition -- Bit 7 defined as "Abort Task", remaining bits same as FCP-2. Alternative iSCSI definition: Convert the task management flags to a 4-bit encoded field. [As described in what follows, the above field allows the reformatted command block to perform the function of the iSCSI task management block defined in iSCSI, section 3.7.] RDDATA, WRDATA -- One-bit fields indicating direction of data transfer iSCSI definition -- Same as FCP-2. Additional FCP Length -- Amount of space allocated for a CDB whose length exceeds 16 bytes. iSCSI definition -- Same as FCP-2. Additional FCP_CDB -- Contains that portion of the CDB which exceeds 16-bytes. iSCSI definition -- Same as FCP-2 (Note: this field does not seem to be defined in the current iSCSI proposal). FCP_DL -- Count of the greatest number of bytes expected to be transferred to or from the application client data buffer by the SCSI CDB. iSCSI definition -- Same as FCP-2. Other FCP-2 reserved fields are mapped to corresponding reserved fields in iSCSI. 1.3 iSCSI features not reflected in the FCP-2 data structure: "Additional data": Command-specific data appended to SCSI command. Suggest modifying iSCSI to append this information to the command block and reflect its extent through the iSCSI header shown above. 2. SCSI Task Management command (iSCSI section 3.7) Deleted. The function of this data structure in iSCSI is subsumed by the revised iSCSI command block shown above. 3. SCSI Response (iSCSI section 3.3) and SCSI Task Management response (section 3.8) The iSCSI data structures for responding to command and task management requests are combined to reflect the merger of the iSCSI command and task management blocks described above. +------------------+ | iSCSI Header | +------------------+ | SCSI Response | | Block modeled | | on FCP-2 | +------------------+ | Additional Data | | (Optional) | +------------------+ 3.1 Contents of iSCSI header: The following are as defined above. OpCode Length CmdRN ExpStatRn Initiator Tag Length of "Additional Data" (Must be zero). 3.2 iSCSI Mapping of FCP-2 response template. FCP_RSP, bytes 0 through 9 -- Reserved iSCSI mapping -- Same as FCP FCP_RSP, byte 10 -- Flags, contents: FCP_RSP_LEN_VALID -- 1 = FCP_RSP_INFO field contains valid data iSCSI mapping -- Equivalent to FCP mapping. FCP_SNS_LEN_VALID -- 1 = FCP_SNS_INFO field contains valid data. The length of the returned sense data is contained in FCP_SNS_LEN. iSCSI mapping -- Equivalent to FCP mapping. FCP_RESID_OVER, FCP_RESID_UNDER -- Residual underflow or overflow count. iSCSI mapping -- Equivalent to FCP mapping. FCP_CONF_REQ -- 1 = Initiator must confirm receipt of status information unit. iSCSI Mapping -- Reserved. FCP_RESID -- if FCP_RESID_UNDER or FCP_RESID_OVER is set, contains count of residual data iSCSI Mapping -- Same as FCP. FCP_SNS_LEN -- if FCP_SNS_LEN_VALID, contains length of sense data. iSCSI Mapping -- Same as FCP. FCP_RSP_LEN -- If FCP_RSP_LEN_VALID, contains length of FCP_RESPONSE field. (FCP-2 limits FCP_RSP_LEN to 4 or 8). iSCSI Mapping -- Same as FCP FCP_RESP_INFO -- If FCP_RSP_LEN_VALIS = 1, this is a 4- or 8-byte field containing the iSCSI mapping -- See 3.2.1 below. FCP_SNS_INFO -- If FCP_SNS_LEN_VALID =1, contains FCP_SNS_LEN bytes of sense data. iSCSI mapping -- Same as FCP-2 3.2.1. iSCSI mapping of FCP_RSP_INFO field FCP_RSP_INFO is a 4- or 8-byte field containing the following: Bytes 0 through 2: Reserved iSCSI mapping: Same as FCP. Byte 3: RSP_CODE Code values (hex): 00 -- No failure or task management function complete 04 -- Task management function not supported 05 -- Task management function falied All other values reserved. The above code values map directly to iSCSI. Bytes 4 through 7: Reserved. In the RSP_CODE field, iSCSI equivalents for the following FCP-specific codes may be useful: Code values (hex): 01 -- FCP_DATA length different than burst length. 02 -- FCP_CMND fields invalid 03 -- FCP_DATA RO mismatch with FCP_XFER_RDY DATA_RO Codes 01 and 03 indicate a mismatch between an FCP_XFR_RDY parameter and the corresponding parameter in the actual data transfer information unit. 4. iSCSI RTT and FCP_XFR_RDY Information Unit +------------------+ | iSCSI RTT | | Header | +------------------+ | iSCSI RTT | | modeled on | | FCP_XFR_RDY | +------------------+ In iSCSI, The use of RTT is negotiated between the target and initiator. 4.1 RTT Header contents The following are as defined in the iSCSI spec. OpCode Length CmdRN ExpStatRn Initiator Tag Target Tag 4.2 iSCSI Mapping from FCP_XFR_RDY to RTT DATA_RO -- Offset of the first byte of data relative to the start of the initiator's buffer iSCSI Mapping -- Buffer offset as defined in iSCSI, section 3.9 BURST_LEN -- On a transfer from the FCP initiator to the FCP target, exact number of bytes to be transferred. iSCSI Mapping -- Desired data transfer length as defined in iSCSI, section 3.9 > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Sunday, August 27, 2000 8:58 AM > To: ips@ece.cmu.edu > Subject: bridging issues > > > > > Dear colleagues, > > Lately some iSCSI issues related to bridging to/from FC networks where > brought to my > attention. I thought that it would be wise to share them > (and my first > rough attempt to solve them) with you for two reasons: > > that's why we have a Working Group (don't we?) > there might be more of this kind and this is a good time > to think about > them and bring the with or without a solution to the WG > > FC and iSCSI are SCSI mappings over a network that share a > "network" view > of the > transport infrastructure but have widely different views of > the network > with regard > to distances, reliability, flow control etc.. > > FCP assumes a locally controlled environment using an > unreliable delivery > version of FC > and doing recovery (a very rare event) at FCP level. > > FC addressing model uses locally generated dynamic addresses > as part of the > "naming" > and its tagging. > > An iSCSI initiator should be able to operate a FCP target > through a very > simple and > stateless (or almost stateless) gateway. > > A designer attempting to do this (and I know many of you > already did or are > in the midst > of doing that) will find it difficult to do achieve a > stateless design (but > not impossible). > > The FCP recovery model from packet loses (commands, data and > status) is > based on sequencing (if lost packets are within sequences) or > timeouts if > those are at sequence > boundaries. In the FC world those are very rare events and > in an iSCSI+FCP > hybrid > could be handled by a statefull gateway (expensive) or by a stateless > gateway and > a more radical initiator recovery - based on TCP connection close and > reopen. > The later solution requires iSCSI to enforce a set of timeouts (an > initiator requirement only) > and provide an API for timeout setting and connection abort > to an upper > layer. > Enforcing a set of timeouts would have the good side-effect > of making iSCSI > also > tolerant to aberrant device behavior - and that is > unfortunately bound to > happen more > often as the complexity of the devices will increase. > Please observe that the set of timeouts I am referring to are not a > "protocol" element > (they do not appear on the wire). Specifying the expected > behavior of an > iSCSI > initiator will simplify the task of building bridges to FCP. > We propose to do so by adding a "host requirements" chapter. > > Another bridging issue is the initiator tag-translation mechanism. > iSCSI is requiring the initiator issue a unique tag for each > task but does > not require any structure for the tag. This way an > implementer may choose > the most efficient structure for his implementation (e.g., > use the address of it's control structure representing the > task as a tag). > > FCP on the other hand demands some structure for the tag. > > A stateless bridge would want to pass the tag unchanged from > initiator to > target or translate it with a stateless rule. > > If we restrict the design space to gateways that present a > "homogenous" > view of the device conglomerate behind them (e.g., all > devices are FCP) > and we may pass this information to the initiator - then the > initiator can > use the slightly more restrictive FCP tagging mechanism. > > Julo > > From owner-ips@ECE.cmu.edu Wed Sep 06 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27032; 6 Sep 2000 13:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA09787 for ips-outgoing; Wed, 6 Sep 2000 12:24:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA09783 for ; Wed, 6 Sep 2000 12:24:30 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Wed, 6 Sep 2000 09:24:32 -0700 Message-ID: From: Joshua Tseng To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI Autosense Consensus, Connection next steps Date: Wed, 6 Sep 2000 09:24:31 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, >- the simple backpresure mechanism and TCP windows and send buffers >proportional to the pipe rate will get you the skew down to acceptable >levels. If this is the case, does the command windowing mechanism really buy us very much? >The total buffer set needed is still proportional >to the total peak rate. And you will need similar mechanisms to regulate an >asymmetric flow too >(a bit simpler though!). I agree. Buffer requirements are proportional to aggregate throughput. Josh From owner-ips@ECE.cmu.edu Wed Sep 06 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27030; 6 Sep 2000 13:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA10166 for ips-outgoing; Wed, 6 Sep 2000 12:32:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA10160 for ; Wed, 6 Sep 2000 12:32:38 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id LAA03917; Wed, 6 Sep 2000 11:33:01 -0500 Message-ID: <39B671BD.5817E03@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 11:33:01 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Stephen Bailey CC: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: <10009061506.AA04945@candide.cs.uchicago.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen Bailey wrote: > > None of the arguments made for iSCSI connection bundling are unique to > storage. If connection bundling is so compelling, TCP or some > relative will develop its own connection bundling mechanism, and then > iSCSI can take advantage of it more or less transparently. > Stephen: About 1 year ago we had this "transport bundling" aka load sharing IN SCTP. At the request of the IESG it was removed because (if I remember right) it causes all sorts of problems with Congestion Control and is still deemed a research topic by the IETF (belonging to the IRTF).It is aong the same lines (to a small degree) as to why you don't want multiple TCP connections. SCTP now only offers reliablility gained by spanning multiple NIC cards on one connection/association but no load balancing. Load balancing was looked at as being done (if ever) from an upper layer, and in fact there are hooks in the protocol for this... A ULP can: 1) A change the primary address (aka where default sends go to).. change this every time before sending and you have load balancing 2) Query to get the list of addresses the peer has, useful so you know what to change the primary to or override and the ULP does not need to maintain some list of peer addresses. 3) Do a directed send that overrides the primary address to one of the alternates. But the actual implemenation of the load balancing is above the transport layer... :0 R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 06 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26971; 6 Sep 2000 13:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA08860 for ips-outgoing; Wed, 6 Sep 2000 12:05:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA08096 for ; Wed, 6 Sep 2000 11:48:11 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Wed, 6 Sep 2000 08:49:14 -0700 Message-ID: From: Robert Snively To: ips@ece.cmu.edu Subject: Multiple Connections. How would they work? Date: Wed, 6 Sep 2000 08:46:40 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk What am I missing? The reason I felt that iSCSI might possibly work was the text from 2.2.4 of draft-satran-iscsi-01.txt. " For SCSI commands that require data and/or parameter transfer, the (optional) data and the status for a command must be sent over the same TCP connection that was used to deliver the SCSI command (connection allegiance). Thus if an initiator issues a READ command, the target must send the requested data followed by the status to the initiator over the same TCP connection that was used to deliver the SCSI command." >>From this, I extrapolated that multiple connections and sessions were mechanisms to manage multiple paths, each of which would carry all the traffic related to a single command, probably with no ordering among them. I had expected TCP/IP and iSCSI state required for execution of an iSCSI operation to be contained within a single NIC. How does this map into the multiple connection structures that are being discussed? How do the multiple connection structures map into the low cost highly efficient hardware assisted NIC implementations that presently typify SCSI implementations? From owner-ips@ECE.cmu.edu Wed Sep 06 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27047; 6 Sep 2000 13:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA08970 for ips-outgoing; Wed, 6 Sep 2000 12:07:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA08965 for ; Wed, 6 Sep 2000 12:07:06 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Wed, 6 Sep 2000 09:07:06 -0700 Message-ID: From: Joshua Tseng To: James Smart , ips Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Wed, 6 Sep 2000 09:07:06 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk James, I agree with others that there may be an issue with the command windowing mechanism in the existing iSCSI spec. It is like "TCP in reverse", in that the target determines the size of the window, and not the initiator as in TCP. Rather, I believe that everything that this windowing mechanism is attempting to achieve can be more easily obtained by having the target communicate its buffer size to the initiator at iSCSI login. It should be the role of the initiator to determine how many commands to put in flight simultaneously, given this input on available buffer size from the target. As far as multiple initiators, could this not be resolved by the target refusing additional logins beyond the number of initiators it can safely support? Not being a storage expert, this is my best guess/suggestion at how to do it. Josh -----Original Message----- From: James Smart [mailto:jssmart@nhinternet.com] Sent: Wednesday, September 06, 2000 6:37 AM To: ips Subject: Command Queue Depth (was asymmetric/Symmetric) Josh, Wanted to double-check on the scsi-isms implied by the statements below, and post a new question on managing command buffering ... >>My question then is: does Data Balancing on multiple >>connections cause a similar need for additional memory as does the >>synchronous connections with Multiple Commands on different connections. > >In the symmetric model, the target must be prepared to buffer all of the >commands and associated unsolicited data PDU's that may be sent between >ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that or discard >commands. But in order to realize the performance benefits of command >windowing, the target must expand the window in order to let the initiator >to "let her rip!" and put multiple commands in flight. It seems to me that >the existing symmetric model adds complexity which would only be of benefit >if the target has lots of memory. So my question is this: The command >windowing mechanism seems to require lots of resources (including memory) >and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to >implement, but does all this really buy very much? In the end, symmetric >implementations may end up discarding commands anyway, as in the asymmetric >case, unless they want to install globs of memory. First - I assume that "discard commands" actually means completing the command with a SCSI status of BUSY or QUEUE FULL - *NOT* simply dropping the command on the floor. As per previous statements on the reflector, OS's and existing drivers expect older-style SCSI behaviors - dropping of commands from lack of resources expect QUEUE FULL or BUSY indicators to invoke backoff algorithms. OS's typically don't deal well with i/o timeouts for error recovery. Second, are the buffering expectations above in line with existing SCSI/FC devices ? Most scsi devices today, which support command queuing, do indeed support buffering up to N commands (N being the maximum queue depth and is shared across all initiators). Simple disk drives typically support queue depths of 16, 32 or 64 (all to a single lun), while large controllers accept hundreds (shared across multiple luns). Lastly, this topic touches on one of the more sensitive areas in the use of SCSI over packetized interfaces - that of managing the command queue. In parallel SCSI, it was dealt with more efficiently, as the QUEUE FULL or BUSY status was received immediately on the interlocked bus before an additional commands could be sent. However, in FC and iSCSI, the commands will be in the pipe, and there will (are) storms of commands bombarding the target by multiple initiators. As mentioned, unless status was sent back to invoke the backoff algorithms, things continued unabated. I've seen FCP targets fully consumed just trying to hold off these storms. It gets worse as the initiators can be different OS's, with different backoff policies (some do onesy/twosy based on command completion and noting depth levels to manage how much they keep outstanding; while others simply delay for a time, then let the gates wide open to whatever maximum they believe they can send (e.g. a storm)). Given the varied and uncoordinated rates at which the initiators can beat on the command queue resource, there are times where things melt down both on the target and on the initiator backoff/retry algorithms such that little real work gets achieved. I know this last issue is not generic to iSCSI and may be classified as a configuration error. However - to the group - has iSCSI considered any mechanism to report the command queue depth (buffering) available to an initiator ? (my guess is that the delta between ExpCmdRN and MaxCmdRN via the login response implies this). Further - any thoughts on how the resources could then be multiplexed across multiple initiators (thus the ExpCmdRN/MaxCmdRN can't really apply any more). -- James -------------------- James Smart james.smart@nhinternet.com From owner-ips@ECE.cmu.edu Wed Sep 06 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27061; 6 Sep 2000 13:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA10566 for ips-outgoing; Wed, 6 Sep 2000 12:42:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA10557 for ; Wed, 6 Sep 2000 12:42:36 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA234246 for ; Wed, 6 Sep 2000 18:42:04 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA90442 for ; Wed, 6 Sep 2000 18:42:04 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256952.005BB7D8 ; Wed, 6 Sep 2000 18:41:48 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 6 Sep 2000 19:37:50 +0300 Subject: Re: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Command numbering won't be needed anymore - every command with an outstanding status is restarted. As for the 2 commands - synch can be achieved by setting data first and then sending commands. Julo "Matt Wakeley" on 06/09/2000 11:17:09 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple connections julian_satran@il.ibm.com wrote: > Dear colleagues, > > With all the heated debate about multiple vs. single connection a request I > made a while ago got no significant reply (neither for nor against). > > The request was to consider a proposal made by Kalman Meth to reconsider > the asymmetric model with the addition of a path selection made by the > initiator. One downside I see to this proposal is that for a host with multiple iSCSI nics, it will have to issue two commands (in this order): one to the "data path nic" to describe where the data is to be read from/written to, and one to the "control path nic" to send the command. This would be communication to two cards instead of one. Also, note that the communication to the data path nic MUST complete before the control path nic sends the command, requiring some kind of sync mechanism. > This proposal allows removing the command counters - as commands use a > single TCP connection. The single connection can also be a shared > data+control connection. For the case of the command connection failure and fail-over to a new connection, I don't see how you can get away from the command counters. When a fail over occurs, you will need some way of finding out what commands made it to the target and which didn't. The easiest way to do this is with command numbering. > > > In case of multiple connection the data path to be used is selected and > maintained until the command ends. > > Thanks, > Julo -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Wed Sep 06 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27152; 6 Sep 2000 13:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA10757 for ips-outgoing; Wed, 6 Sep 2000 12:47:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from rly-ip01.mx.aol.com (rly-ip01.mx.aol.com [205.188.156.49]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA10753 for ; Wed, 6 Sep 2000 12:47:17 -0400 (EDT) Received: from tot-we.proxy.aol.com (tot-we.proxy.aol.com [205.188.195.1]) by rly-ip01.mx.aol.com (8.8.8/8.8.8/AOL-5.0.0) with ESMTP id MAA15733 for ; Wed, 6 Sep 2000 12:46:37 -0400 (EDT) Received: from VAIO.PacBell.net (AC8113F0.ipt.aol.com [172.129.19.240]) by tot-we.proxy.aol.com (8.10.0/8.10.0) with ESMTP id e86GkVc06092 for ; Wed, 6 Sep 2000 12:46:32 -0400 (EDT) Message-Id: <4.3.2.7.2.20000906175329.00b76c40@PacBell.net> X-Sender: Celeborn@PacBell.net X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Wed, 06 Sep 2000 17:56:49 +0200 To: IP Storage From: Peter Johansson Subject: RE: iSCSI Autosense Consensus, Connection next steps In-Reply-To: References: <0F31E5C394DAD311B60C00E029101A0704100F4F@corpmx9.isus.emc.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:31 PM 9/5/00, Douglas Otis wrote: >Microsoft assumes the SCSI interface is SCSI-II as example. I don't know what this has to do with the discussion about autosense. Microsoft implements SBP-2 in the more recent Windows OS; SBP-2, if used to transport SCSI commands and status, requires autosense; inductively, there is no conflict between this sensible requirement of a packetized protocol (SBP-2) and a belief that Microsoft assumes SCSI-2 behavior. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Wed Sep 06 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27581; 6 Sep 2000 14:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA13604 for ips-outgoing; Wed, 6 Sep 2000 13:51:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA13597 for ; Wed, 6 Sep 2000 13:51:51 -0400 (EDT) Received: from mail.nuspeed.com (mail.nuspeed.com [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 218D9837F3; Wed, 6 Sep 2000 12:51:45 -0500 (CDT) Received: from nuspeed.com (michigan.eng.nuspeed.com [172.18.48.12]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S12WCY2H; Wed, 6 Sep 2000 12:59:39 -0500 Message-ID: <39B68431.1E957C5B@nuspeed.com> Date: Wed, 06 Sep 2000 12:51:45 -0500 From: Dave Thompson Organization: NuSpeed X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en MIME-Version: 1.0 To: "Satran, Julian" , ips@ece.cmu.edu Subject: iSCSI version info Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, A version negotiation scheme was suggested by Mark Bakke's 29 Jun 2000 e-mail. > Logins for iSCSI draft 03 might benefit from the following additions: > > 1. Version number text fields should be placed in both login requests > and responses. the request could either specify the version of the > protocol the initiator wishes to run, or perhaps a list of versions > that the initiator is capable of running. > > Version: 1.0 > > -OR- > > Version: 1.0 1.1 1.2 > > The target would then choose the highest version that it supported > in common with those requested, and return it in the response: > > Version: 1.1 Is there any plans to support a mechanism like this in the next iSCSI draft? In addition, perhaps a one byte field in the login command and response headers (byte 3?) could be added which would contain the major protocol version number. This could then be used to determine the proper format of the header in cases where major header layout changes occur (e.g. the last draft's size change from 40 to 48 bytes). -- Dave Thompson NuSpeed, Inc. dave.thompson@nuspeed.com From owner-ips@ECE.cmu.edu Wed Sep 06 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27579; 6 Sep 2000 14:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14123 for ips-outgoing; Wed, 6 Sep 2000 14:02:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.spinnakernet.com (client-141-151-129-90.bellatlantic.net [141.151.129.90]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14096 for ; Wed, 6 Sep 2000 14:01:42 -0400 (EDT) Received: from kazar-pc (kazar-pc.spinnakernet.com [10.1.1.110]) by mail.spinnakernet.com (8.9.3/8.8.7) with ESMTP id OAA06684; Wed, 6 Sep 2000 14:01:30 -0400 Message-Id: <4.2.2.20000906133134.00abc4b0@router.spinnakernet.com> X-Sender: kazar@router.spinnakernet.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 06 Sep 2000 14:10:28 -0400 To: Matt Wakeley , ips@ece.cmu.edu From: Mike Kazar Subject: Re: a vote for asymmetric connections in a session In-Reply-To: <39B65F1F.2BA8110B@agilent.com> References: <4.2.2.20000906081014.00ac2450@pop.icubed.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:13 AM 9/6/00 -0700, Matt Wakeley wrote: >Mike Kazar wrote: > > > Folks, > > > > Here are my reasons for preferring the asymmetric connection model to the > > symmetric connection model, in decreasing order of importance. > > > > 1. Implementing a sliding window protocol for seqRn processing is likely > > to be as hard, or harder, as implementing TCP. Getting something that > > works will be easy, but getting something that works well, even when > > various iSCSI TCP connections are running at different rates is likely to > > be an "interesting" research problem. Yet connections running at different > > rates is likely to occur in real life quite frequently, sometimes due to > > different connections seeing different packet drop events in a somewhat > > congested network, sometimes due to different connections taking different > > paths (the whole *goal* of multiple connections in a session, after all). > >Well, I disagree. Implementing sliding windows is not that hard. If sliding >windows were the toughest thing to implement in iSCSI, this would be a cake >walk. My point isn't that sliding window functionality is hard to implement; it isn't. But getting good performance in the event of network congestion is the last 10% of the problem that takes 90% of the work. Remember, you'll be dealing with N different connections, all potentially running at different speeds, at differing times, and you have to choose a window size that makes sense based upon the dynamic behavior of all of these TCP connections. I think it might be illuminating to try to figure out some algorithms to keep the throughput high even when one of the connections drops a packet and cuts its TCP window size down significantly. All the other connections are still pumping in data at full speed, most of which is now stuck behind data from the slow connection. Do you just write off half your performance until the slow connection picks up speed again? Does the sender just stop using the slow connection for "a while"? Do you crank up the iSCSI window size so you can buffer data from the fast connections until the slow connection increases its window again? And what if instead of running slowly because of a dropped packet, the slow connection is slow because it has been routed over a slower physical link? At that point, all you can do (assuming you can figure out this is what's happening) is have the sender send less data through the slow link. This whole problem sounds non-trivial to me. > > > > > > Furthermore, the sliding window state machine for a given session's seqRn > > processing is running at N times the event rate of an individual TCP > > connection's state machine (N is the # of connections in a session), making > > it even more challenging to implement. > >I disagree again. The TCP sliding window SM is running at once per TCP >segment. If just iSCSI command messages where crammed down the connection, >that means the session's seqRn processing will be done once every 48 bytes. >Every time an iSCSI message is received, there is a lot of processing invoked. >Adding to the list a simple sliding window algorithm is not a challenge. I think we just disagree on terms here. If you have N connections in a particular session, you have N independent TCP state machines that do not communicate with each other, *each one* getting an event every 48*N bytes of session traffic. There is only one iSCSI session instance state machine, and it is seeing a new event for its seqRn processing state machine every 48 bytes of session traffic. That was my point. > > > > > > 2. It is easier to implement target-specific load sharing mechanisms if > > the target gets to choose, perhaps with input from the host, the connection > > on which a particular transfer should be performed. In the symmetric > > proposal, the host chooses the connection to use completely on its own. > >And it's good that the host chooses the connection. It is the one that >has set >up DMA structures on a particular connection to get the data in/out with >minimal CPU intervention. I wouldn't expect it to be significantly harder for the host to set up a DMA transfer for a connection whose final selection was done by the target than one the host selects completely on its own. Mike (kazar@spinnakernet.com) > > > > > > Mike > > (kazar@spinnakernet.com) > >In my opinion, these are not "good" reasons to vote for one or the other. > >Matt Wakeley >Agilent Technologies From owner-ips@ECE.cmu.edu Wed Sep 06 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27564; 6 Sep 2000 14:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA15179 for ips-outgoing; Wed, 6 Sep 2000 14:23:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from www.indstorage.com (windjo.ppp.frii.com [216.17.133.220] (may be forged)) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14614 for ; Wed, 6 Sep 2000 14:12:42 -0400 (EDT) Received: from aegis ([192.168.1.12]) by www.indstorage.com (8.9.3/8.9.2) with SMTP id MAA04846; Wed, 6 Sep 2000 12:03:45 -0600 From: "Mark Bradley" To: "Joshua Tseng" , "John Hufferd/San Jose/IBM" Cc: Subject: RE: iSCSI Autosense Consensus, Connection next steps Date: Wed, 6 Sep 2000 12:03:12 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk This is an excellent hit on the BIG issue, from my perspective. Also, it seems that much of this discussion is from the computer side, not the target side, where resources are likely to be quite scarce (or even unavailable). -- markb > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Joshua Tseng > Sent: Tuesday, September 05, 2000 7:53 PM > To: John Hufferd/San Jose/IBM; Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > > John, > > >My question then is: does Data Balancing on multiple > >connections cause a similar need for additional memory as does the > >synchronous connections with Multiple Commands on different connections. > > In the symmetric model, the target must be prepared to buffer all of the > commands and associated unsolicited data PDU's that may be sent between > ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that > or discard > commands. But in order to realize the performance benefits of command > windowing, the target must expand the window in order to let the initiator > to "let her rip!" and put multiple commands in flight. It seems > to me that > the existing symmetric model adds complexity which would only be > of benefit > if the target has lots of memory. So my question is this: The command > windowing mechanism seems to require lots of resources (including memory) > and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to > implement, but does all this really buy very much? In the end, symmetric > implementations may end up discarding commands anyway, as in the > asymmetric > case, unless they want to install globs of memory. > > WRT the load balancing algorithm implemented by iSCSI, anything other than > round robin will require more complexity to sense which connection is > over-utilized or poorly performing. Once again, what does the added > complexity buy us? I'm not an development engineer, but my guess > is getting > iSCSI to talk to TCP to get this info and use it isn't cheap. In > asymmetric, commands are sent in-order down a relatively less-utilized > command TCP connection. A slow TCP connection only affects unfortunate > exchanges whose data PDU's happen to use that connection, and not all > sessions between initiator and target. > > Josh > > > > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Tuesday, September 05, 2000 4:53 PM > To: Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > > > Joshua, > I think some of your points are correct. But we also need to > examine a few > statements. > > First, the round robin approach of balancing, if only done in the > manor you > specified, would of course cause every thing to slow to the speed of the > slowest link. But this is a well known problem, and general managed by a > modified round robin so that it continues to other connections if one of > the connections is still busy (several different way of determining this). > In fact this would be a problem in the current wedge drivers today used > with Fibre Channels if only a blind round robin approach was used. > > Your point about the target perhaps needing additional memory, is probably > correct. This is also solvable on the initiator side (by a number of > techniques) however, the target can not be sure that the initiator will > have the right techniques, so it must assume that it needs additional > memory. This seems like a correct point. If this is a worry, > however, the > Target can just reject additional conversations per Session. The > counter I > would make to my last point, however, would be if Asynchronous was used, I > could still could handle multiple data ports and at least get that kind of > balancing. My question then is: does Data Balancing on multiple > connections cause a similar need for additional memory as does the > synchronous connections with Multiple Commands on different connections. > > On the other hand, think that your point about link recovery not being > (significantly) more difficult with Async then Sync is > essentially correct. > > . > . > . > John L. Hufferd > > > > Joshua Tseng @ece.cmu.edu on 09/05/2000 02:54:56 > PM > > Sent by: owner-ips@ece.cmu.edu > > > To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > > > David, > > If I may give my input on the TCP connection model discussion, I'd like > to advocate the asymmetric model for the following reasons. Using > multiple TCP connections, each TCP connection will likely have > different performance characteristics. If this is so, then some commands > may reach the target out of order from other commands sent down different > TCP connections. Commands sent down the faster connection will then need > to be buffered until commands sent down slower connections arrive. This > will lead to the following problems that I can think of: > > 1) There will be increased memory requirements for the iSCSI target, > which must buffer commands and related data and reorder them > before passing > them on to the SCSI controller. > > 2) If the initiator cannot sense the performance characteristics of > each TCP connection, then interactions between initiator and target will > be dependent on the slowest TCP connection. For example, if there are > five TCP connections and one of them is extraordinarily slow, and the > initiator merely round-robins the commands down each of the five > connections, > all interactions between initiator and target will be only as fast as the > slowest connection. > > In addition, it has been stated that detecting and recovering > from a failed > asymmetric control connection is more complicated and difficult than > detecting and recovering from a failed symmetric data/command TCP > connection. > I do not understand why this is so. In both cases, iSCSI initiator will > need > to abort the lost command AND those following the lost command, and resend > all > commands in a new or different TCP connection. In the symmetric case, > iSCSI > commands received by a target following a lost command will not be > acknowledged, > since ExpStatRN is a cummulative windowing mechanism. > > Josh Tseng > > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Sunday, September 03, 2000 8:20 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Cc: Black_David@emc.com > Subject: iSCSI Autosense Consensus, Connection next steps > > > > It looks like we have consensus - but the chairmen have the call. > > Indeed we do, and I apologize for the delay, due to an inability to > get connected. > > First, on Autosense, I believe that rough consensus exists for > iSCSI to require Autosense. I have seen only one objection; > If anyone other than Doug Otis disagrees with this, please > say so on the list. > > Second, on connections, I haven't seen enough discussion to call > consensus, but I am going to try to narrow the option space and > structure the discussion. Four models for sessions have been > proposed: > > (1) Symmetric - all connections usable for command and data. > (2) Asymmetric - single command connection, others are data. > (3) Split - assign LUN sets to specific connections or pools of > connections. > (4) SCTP - use SCTP's support for multiple connections. > > I have only seen one message suggesting/supporting (3) Split, so > I believe rough consensus exists to not pursue that model > any further. If anyone other than Paul von Stamwitz disagrees, > and believes that the Split model should be pursued, please > say so on the list with technical rationale. > > I haven't seen enough discussion on use of SCTP instead of > TCP to call consensus. I am concerned that this list is not > going to produce a consensus on the issue phrased in that > fashion (i.e., pick exactly one). The one point on which there > does seem to be consensus is that SCTP is a considerably > younger protocol that TCP, and hence the likely timelines > to availability of hardware acceleration seem to favor TCP. > OTOH, this sort of future prediction can easily miss the mark. > > I can see two possible ways to make progress in this area: > - An off-line design team to do an intensive evaluation of SCTP > vs. TCP for iSCSI. > - Recognize the merits of SCTP as well as TCP, plan for both > with the anticipation that TCP will be used first. > The latter makes more sense to me, because it appears to > be more amenable to consensus, avoids the investment > of effort in the design team, and avoids having the prospect > of abandoning TCP hanging over the heads of those engaged > in TCP-specific work. The practical implication of proceeding > in this fashion is that SCTP-friendliness becomes an additional > criteria to use in evaluating iSCSI design proposals. Proceeding > in this fashion is only a proposal at this juncture -- please > comment on whether this is the right way to proceed (to me > or on the list). > > The specification of sessions for iSCSI over TCP needs to > proceed, so under the assumption that either the "plan for > both" path will be pursued, or that we can't wait for the > design team's conclusions, and hence need to work on TCP > in case it is selected, I want to summarize the tradeoffs > between the Symmetric and Asymmetric session models, > in the aim of simulating more discussions so that we can > get to consensus. > > [X] The Symmetric model imposes additional work on implementations > that do not use multiple connections because they will > have to implement the command numbering. The > Asymmetric model does not do this. > [Y] The Asymmetric model makes dealing with a failed control > connection more complicated than the symmetric model > because agreement between the two sides is required > to establish a new control connection (or use an existing > data connection for control). > > I have seen the discussion of [X] favoring Asymmetric for systems > that will not support multiple TCP connections per iSCSI session. > I also note (as an example of applying the SCTP-friendliness > from above) that [X] favoring Asymmetric also applies to SCTP, as > there seems to be no point in using multiple SCTP sessions in a > single iSCSI session. > > We need to give the folks working on revising the iSCSI draft some > direction on this issue in fairly short order, so comments are hereby > solicited ... > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > > > > From owner-ips@ECE.cmu.edu Wed Sep 06 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27566; 6 Sep 2000 14:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA13221 for ips-outgoing; Wed, 6 Sep 2000 13:42:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA13209 for ; Wed, 6 Sep 2000 13:42:35 -0400 (EDT) Received: from p7020-img-nt.cisco.com (fred-hm-dhcp3.cisco.com [171.69.128.118]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id KAA26200; Wed, 6 Sep 2000 10:41:53 -0700 (PDT) Message-Id: <4.3.2.7.2.20000906103002.03450ae0@flipper.cisco.com> X-Sender: fred@flipper.cisco.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Wed, 06 Sep 2000 10:32:23 -0700 To: "Randall R. Stewart" From: Fred Baker Subject: Re: iSCSI Autosense Consensus, Connection next steps Cc: csapuntz@cisco.com, ips@ece.cmu.edu, Allison Mankin , Scott Bradner In-Reply-To: <39B64866.961ABDCD@stewart.chicago.il.us> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:36 AM 9/6/00 -0500, Randall R. Stewart wrote: >But my question is to the Data connection. How do you get >"no flow control on data needed, just congestion control" on >a TCP connection? well, you can't. What you can do is use large windows and make the window as large as the buffer you are receiving the file into, so that effectively there is no flow control after the first couple of round trips. you will want to use SACK, and you will want to follow RFC 2581's comments about the initial transmission - send two or three segments adding up to ~4K bytes. From owner-ips@ECE.cmu.edu Wed Sep 06 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27568; 6 Sep 2000 14:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14484 for ips-outgoing; Wed, 6 Sep 2000 14:10:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from www.indstorage.com (windjo.ppp.frii.com [216.17.133.220] (may be forged)) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14426 for ; Wed, 6 Sep 2000 14:09:29 -0400 (EDT) Received: from aegis ([192.168.1.12]) by www.indstorage.com (8.9.3/8.9.2) with SMTP id MAA04846; Wed, 6 Sep 2000 12:03:45 -0600 From: "Mark Bradley" To: "Joshua Tseng" , "John Hufferd/San Jose/IBM" Cc: Subject: RE: iSCSI Autosense Consensus, Connection next steps Date: Wed, 6 Sep 2000 12:03:12 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk This is an excellent hit on the BIG issue, from my perspective. Also, it seems that much of this discussion is from the computer side, not the target side, where resources are likely to be quite scarce (or even unavailable). -- markb > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Joshua Tseng > Sent: Tuesday, September 05, 2000 7:53 PM > To: John Hufferd/San Jose/IBM; Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > > John, > > >My question then is: does Data Balancing on multiple > >connections cause a similar need for additional memory as does the > >synchronous connections with Multiple Commands on different connections. > > In the symmetric model, the target must be prepared to buffer all of the > commands and associated unsolicited data PDU's that may be sent between > ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that > or discard > commands. But in order to realize the performance benefits of command > windowing, the target must expand the window in order to let the initiator > to "let her rip!" and put multiple commands in flight. It seems > to me that > the existing symmetric model adds complexity which would only be > of benefit > if the target has lots of memory. So my question is this: The command > windowing mechanism seems to require lots of resources (including memory) > and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to > implement, but does all this really buy very much? In the end, symmetric > implementations may end up discarding commands anyway, as in the > asymmetric > case, unless they want to install globs of memory. > > WRT the load balancing algorithm implemented by iSCSI, anything other than > round robin will require more complexity to sense which connection is > over-utilized or poorly performing. Once again, what does the added > complexity buy us? I'm not an development engineer, but my guess > is getting > iSCSI to talk to TCP to get this info and use it isn't cheap. In > asymmetric, commands are sent in-order down a relatively less-utilized > command TCP connection. A slow TCP connection only affects unfortunate > exchanges whose data PDU's happen to use that connection, and not all > sessions between initiator and target. > > Josh > > > > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Tuesday, September 05, 2000 4:53 PM > To: Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > > > Joshua, > I think some of your points are correct. But we also need to > examine a few > statements. > > First, the round robin approach of balancing, if only done in the > manor you > specified, would of course cause every thing to slow to the speed of the > slowest link. But this is a well known problem, and general managed by a > modified round robin so that it continues to other connections if one of > the connections is still busy (several different way of determining this). > In fact this would be a problem in the current wedge drivers today used > with Fibre Channels if only a blind round robin approach was used. > > Your point about the target perhaps needing additional memory, is probably > correct. This is also solvable on the initiator side (by a number of > techniques) however, the target can not be sure that the initiator will > have the right techniques, so it must assume that it needs additional > memory. This seems like a correct point. If this is a worry, > however, the > Target can just reject additional conversations per Session. The > counter I > would make to my last point, however, would be if Asynchronous was used, I > could still could handle multiple data ports and at least get that kind of > balancing. My question then is: does Data Balancing on multiple > connections cause a similar need for additional memory as does the > synchronous connections with Multiple Commands on different connections. > > On the other hand, think that your point about link recovery not being > (significantly) more difficult with Async then Sync is > essentially correct. > > . > . > . > John L. Hufferd > > > > Joshua Tseng @ece.cmu.edu on 09/05/2000 02:54:56 > PM > > Sent by: owner-ips@ece.cmu.edu > > > To: Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: iSCSI Autosense Consensus, Connection next steps > > > > David, > > If I may give my input on the TCP connection model discussion, I'd like > to advocate the asymmetric model for the following reasons. Using > multiple TCP connections, each TCP connection will likely have > different performance characteristics. If this is so, then some commands > may reach the target out of order from other commands sent down different > TCP connections. Commands sent down the faster connection will then need > to be buffered until commands sent down slower connections arrive. This > will lead to the following problems that I can think of: > > 1) There will be increased memory requirements for the iSCSI target, > which must buffer commands and related data and reorder them > before passing > them on to the SCSI controller. > > 2) If the initiator cannot sense the performance characteristics of > each TCP connection, then interactions between initiator and target will > be dependent on the slowest TCP connection. For example, if there are > five TCP connections and one of them is extraordinarily slow, and the > initiator merely round-robins the commands down each of the five > connections, > all interactions between initiator and target will be only as fast as the > slowest connection. > > In addition, it has been stated that detecting and recovering > from a failed > asymmetric control connection is more complicated and difficult than > detecting and recovering from a failed symmetric data/command TCP > connection. > I do not understand why this is so. In both cases, iSCSI initiator will > need > to abort the lost command AND those following the lost command, and resend > all > commands in a new or different TCP connection. In the symmetric case, > iSCSI > commands received by a target following a lost command will not be > acknowledged, > since ExpStatRN is a cummulative windowing mechanism. > > Josh Tseng > > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Sunday, September 03, 2000 8:20 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Cc: Black_David@emc.com > Subject: iSCSI Autosense Consensus, Connection next steps > > > > It looks like we have consensus - but the chairmen have the call. > > Indeed we do, and I apologize for the delay, due to an inability to > get connected. > > First, on Autosense, I believe that rough consensus exists for > iSCSI to require Autosense. I have seen only one objection; > If anyone other than Doug Otis disagrees with this, please > say so on the list. > > Second, on connections, I haven't seen enough discussion to call > consensus, but I am going to try to narrow the option space and > structure the discussion. Four models for sessions have been > proposed: > > (1) Symmetric - all connections usable for command and data. > (2) Asymmetric - single command connection, others are data. > (3) Split - assign LUN sets to specific connections or pools of > connections. > (4) SCTP - use SCTP's support for multiple connections. > > I have only seen one message suggesting/supporting (3) Split, so > I believe rough consensus exists to not pursue that model > any further. If anyone other than Paul von Stamwitz disagrees, > and believes that the Split model should be pursued, please > say so on the list with technical rationale. > > I haven't seen enough discussion on use of SCTP instead of > TCP to call consensus. I am concerned that this list is not > going to produce a consensus on the issue phrased in that > fashion (i.e., pick exactly one). The one point on which there > does seem to be consensus is that SCTP is a considerably > younger protocol that TCP, and hence the likely timelines > to availability of hardware acceleration seem to favor TCP. > OTOH, this sort of future prediction can easily miss the mark. > > I can see two possible ways to make progress in this area: > - An off-line design team to do an intensive evaluation of SCTP > vs. TCP for iSCSI. > - Recognize the merits of SCTP as well as TCP, plan for both > with the anticipation that TCP will be used first. > The latter makes more sense to me, because it appears to > be more amenable to consensus, avoids the investment > of effort in the design team, and avoids having the prospect > of abandoning TCP hanging over the heads of those engaged > in TCP-specific work. The practical implication of proceeding > in this fashion is that SCTP-friendliness becomes an additional > criteria to use in evaluating iSCSI design proposals. Proceeding > in this fashion is only a proposal at this juncture -- please > comment on whether this is the right way to proceed (to me > or on the list). > > The specification of sessions for iSCSI over TCP needs to > proceed, so under the assumption that either the "plan for > both" path will be pursued, or that we can't wait for the > design team's conclusions, and hence need to work on TCP > in case it is selected, I want to summarize the tradeoffs > between the Symmetric and Asymmetric session models, > in the aim of simulating more discussions so that we can > get to consensus. > > [X] The Symmetric model imposes additional work on implementations > that do not use multiple connections because they will > have to implement the command numbering. The > Asymmetric model does not do this. > [Y] The Asymmetric model makes dealing with a failed control > connection more complicated than the symmetric model > because agreement between the two sides is required > to establish a new control connection (or use an existing > data connection for control). > > I have seen the discussion of [X] favoring Asymmetric for systems > that will not support multiple TCP connections per iSCSI session. > I also note (as an example of applying the SCTP-friendliness > from above) that [X] favoring Asymmetric also applies to SCTP, as > there seems to be no point in using multiple SCTP sessions in a > single iSCSI session. > > We need to give the folks working on revising the iSCSI draft some > direction on this issue in fairly short order, so comments are hereby > solicited ... > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909 > black_david@emc.com Cellular: +1 (978) 394-7754 > --------------------------------------------------- > > > > From owner-ips@ECE.cmu.edu Wed Sep 06 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27590; 6 Sep 2000 14:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14977 for ips-outgoing; Wed, 6 Sep 2000 14:19:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14964 for ; Wed, 6 Sep 2000 14:18:59 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Wed, 6 Sep 2000 11:18:53 -0700 Message-ID: From: Joshua Tseng To: Yaron Klein , Dave Nagle Cc: ips@ece.cmu.edu Subject: RE: security model Date: Wed, 6 Sep 2000 11:18:51 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Yaron, My suggestion WRT security is to take a good look at IPSec. IPSec has several advantages I can think of over SSL or TLS. 1) Standardized key distribution methodologies already exist (ISAKMP/IKE) and have been implemented for use with IPSec. 2) IPSec has the advantage that it is flexible, and security policies can be granularly applied to each iSCSI conversation, even when multiple conversations are going over the same TCP connection. iSCSI can thus decide to encrypt only data PDU's, or only certain conversations, etc... The precise granularity is implementation-specific. With TLS, everything sent through a TCP socket will use the same security policy. If multiple iSCSI conversations are taking place over the same TCP connection, then all will be uniformly handled by the same security policy, whether it's 3DES or no encryption at all. If this is desired, then that's fine, but I believe there will be cases where the user wants to only authenticate Command PDU's, while encrypting & authenticating data PDU's, for example. The following is an excerpt from section 4.4.1 of RFC 2041 (Security Architecture for IP). In this case, "application" refers to iSCSI. "In host systems, applications MAY be allowed to select what security processing is to be applied to the traffic they generate and consume. (Means of signalling such requests to the IPsec implementation are outside the scope of this standard.) However, the system administrator MUST be able to specify whether or not a user or application can override (default) system policies. Note that application specified policies may satisfy system requirements, so that the system may not need to do additional IPsec processing beyond that needed to meet an application's requirements. The form of the management interface is not specified by this document and may differ for hosts vs. security gateways, and within hosts the interface may differ for socket-based vs. BITS implementations. However, this document does specify a standard set of SPD elements that all IPsec implementations MUST support." -----Original Message----- From: Yaron Klein [mailto:klein@sanrad.com] Sent: Sunday, September 03, 2000 3:53 AM To: Dave Nagle Cc: ips@ece.cmu.edu Subject: Re: security model We are now working on the on integrating the security model in the iSCSI draft. Some remarks considering your model: The data privacy (number 5) should be solved in a higher level, i.e., the block should be encrypted or digested by some hash before it is sent to the target. This issue is not in the scope of the transfer protocol. Number 2 should be expanded to initiator and target masquerading. If the enemy masquerades as the target, he can get all our secrets... Furthermore, the security considerations are affected by the performance consideration, i.e., the computation power and the transfer rate. Thus, in high-speed network, compromise should be made (it is not feasible or very expensive to totally secure the data in 1G). The security model should adjust itself to the nature of the iSCSI features. It should enable partial security (only command, only headers, digestion of data etc.) We are now in an intensive work to integrate the security issue in the next draft. The login chapter will enable to establish a flexible and negotiable security/authentication model for the iSCSI session. Any ideas, comments and solution regarding security, let me know, Reagrds, Yaron Dave Nagle wrote: > From: Banu Ozden & Mike Reiter > > We suggest the attached extensions to the security model proposed in > the iSCSI draft (Section 6). > > The existing iSCSI security model covers "communication security" > between an initiator and a target. It does not address "data > security". Data security provides protection against possible attacks > to the data stored at the target. These include threats like > unauthorized disclosure of data at the target to administrators or > other clients of the target, and unauthorized modification of data at > the target. > > Our main motivation is to enhance the security model for storage > outsourcing environments where the Storage Service Provider (SSP) > personnel is not necessarily trusted or where the sharing of target > between different customers of the SSP raises a security concern. > > We are working on a security architecture for storage outsourcing. > We would like to know whether there is interest on including data > security considerations to iSCSI in addition to communication > security. > > Banu Ozden & Mike Reiter > > Bell Labs > 600 Mountain Ave. > Murray Hill, NJ 07974 > http://www.bell-labs.com/who/ozden > http://www.bell-labs.com/who/reiter > > _______________________________________________________________ > Threat Model > > T1. Disclosure of message contents to an eavesdropper intercepting > communication between an initiator and a target. > > T2. An attacker masquerading as the initiator to a target or the target > to an initiator. This includes an attacker manipulating communication > between an initiator and a target, e.g., to introduce false messages, > modify passing messages, or delete messages. > > T3. Disclosure of data to personnel maintaining the target or to other > customers of the target. > > T4. The modification of data by the target or other customers of the > target. > > Security Model > > 1. No Security (same as described in the iSCSI draft) > > This mode does not authenticate nor does it encrypt data. This mode > should be used in environments where there is minimal security risk > and little chance for configuration errors. > > 2. Entity Authentication (referred to as End-to-End Authentication in the > iSCSI draft) > > The initiator's and/or target's identity is authenticated. > Once the client is authenticated, all messages are > sent and received in the clear. This mode should only be used when > there is minimal risk to man-in-the-middle attacks, > eavesdropping, message insertion, deletion, and modification. > > 3. Message Integrity (new) > > This mode protects against T2 types of threats. It provides > communication integrity. > > 4. Message Integrity Combined with Encryption (referred to as Encryption in > the iSCSI draft) > > This mode protects against threats T1 and T2. Thus, it provides > communication integrity and communication privacy. It protects against > man-in-the-middle attacks, eavesdropping, message insertion, deletion, > and modification. > > 5. Data Privacy (new) > > This mode protects against T3 types of threats. The initiator > encrypts/decrypts data. The target stores encrypted data. > > 6. Data Privacy with Data Integrity > > This mode protects against threats T3 and T4. > > 7. Some combinations of the above security options > > For example, data privacy with message authentication (5 & 3) > protects against threats T1, T2 and T3. From owner-ips@ECE.cmu.edu Wed Sep 06 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27588; 6 Sep 2000 14:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA15323 for ips-outgoing; Wed, 6 Sep 2000 14:25:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA15317 for ; Wed, 6 Sep 2000 14:25:53 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA37104; Wed, 6 Sep 2000 14:24:03 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id MAA91494; Wed, 6 Sep 2000 12:25:52 -0600 Importance: Normal Subject: Re: iSCSI Autosense Consensus, Connection next steps To: "Randall R. Stewart" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 6 Sep 2000 11:25:23 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/06/2000 12:25:52 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I think there is some serious miss reading of the paper, or perhaps miss understanding on what we are trying to do with iSCSI. The paper talks about breaking a TCP/IP connection into multiple paths (connections). There is no iSCSI proposal to do this. In all cases, the iSCSI proposals talk about the data for a command returning on ONE connection (which connection it is maybe variable). To say that we should not use multiple NICs for iSCSI is perhaps not reasonable. . . . John L. Hufferd "Randall R. Stewart" @ece.cmu.edu on 09/06/2000 06:36:38 AM Sent by: owner-ips@ece.cmu.edu To: csapuntz@cisco.com cc: ips@ece.cmu.edu, Allison Mankin , Scott Bradner Subject: Re: iSCSI Autosense Consensus, Connection next steps Ok, explain something to me... csapuntz@cisco.com wrote: > > Asymmetric connections may be the way to go. > > Let's look at characteristics of control vs. data: > > * iSCSI control > - low bandwidth (handled on a CPU) > - should generally be processed in FIFO order > - complex state machines (probably software implemented) > - flow control nice > - doesn't need to end up in any special buffers > (can be processed in-place) > > * iSCSI data/ready to transmit (rtt) > - potentially high bandwidth > - with appropriate headers, segments can be > processed entirely out-of-order > - simple data transfer state machine > - no flow control on data needed, just congestion control > - rtt's need to be flow-controlled, but > that can be done with a simple credit mechanism > - data needs to end up in special buffers (e.g. buffer cache) > I understand how you will obtain all of the above with two TCP connecions. I can even support this, considering the 1 for control is "low bandwidth" (note: I can't support N data connections.. only 1). But my question is to the Data connection. How do you get "no flow control on data needed, just congestion control" on a TCP connection? > The processing requirements of the data and control seem quite > asymmetric. Mixing them on one TCP channel makes separating out the > data and control more difficult. > > Conversely, by putting control and data on separate TCP connections, > you can use the flow label (IP addresses, TCP ports) to determine > whether to route this along the control or data path in your machine. > > So, I think there is value in the proposal that we use 1 control > connection and up to n separate dedicated data/RTT connections. No I can NOT agree with N seperate dedicated data/RTT connections. I think you should go look at Sally's email that was posted here earlier... here it is... ************************************************************************** Franco Travostino wrote: > > A bit of Related Work ... On the forever-young 1 vs. N TCP connections > subject, you may be interested in the results shown in the '97 "An > Application-Level Solution to TCP's Satellite Inefficiencies" paper > (available at http://jarok.cs.ohiou.edu/projects/satellite/) and on Sally > Floyd's comment hereafter. The intersection with satellite links is quite > accidental, and the arguments/results apply above and beyond satellite links. > > -franco > > >From majordom@ISI.EDU Fri Dec 6 23:05:58 1996 > >To: fred@cisco.com (Fred Baker) > >Cc: end2end-tf@isi.edu > >Subject: Re: Related paper/re:satellites > >Date: Fri, 06 Dec 1996 23:05:47 PST > >From: Sally Floyd > >Sender: owner-end2end-tf@ISI.EDU > >Precedence: bulk > >Content-Length: 2309 > >X-Lines: 44 > > > >Fred - > > > >>You might be interested in reviewing this paper, which is what I'm > >>discussing with Karen Hansen, Dan Shell, and the folks from Comsat. It > >>relates to some TCP/Satellite work being done at NASA Lewis Research > >>Center. > > > >Basically (from a quick read), the paper, on "An Application-Level > >Solution to TCP's Satellite Inefficiencies", recommends breaking a > >single TCP connection into multiple connections at the > >application level, to increase throughput on satellite circuits. > >It is not too surprising that this increases the TCP throughput, but it > >is still not a good idea. For a single TCP connection, a single packet > >drop results in the throughput for that connection being cut by half, > >and then increased by roughly one packet per RTT. For a TCP connection > >that has instead been separated into N different TCP connections, a > >single packet drop results in one of the N connections, receiving > >1/N-th of the total bandwidth, having its throughput cut in half. Thus > >the bandwidth of the aggregate connection has its bandwidth reduced to > >(1 - 1/(2N))-th of its former bandwidth - that is, the larger the value > >for N, the smaller the aggregate bandwidth is cut. And then, because > >each TCP connection continues to increase its congestion window by one > >packet per RTT, for those TCP connections that have not yet reached the > >receiver's advertised window, the aggregate TCP connections together > >increase their bandwidth by up to N packets per RTT. > > > >Summarizing, splitting a TCP connection into N separate connections > >simply increases the aggressiveness of the TCP congestion control. > >Meaning that this TCP is now more likely to "steal" bandwidth from > >other TCP connections in times of congestion. And increasing the > >aggressiveness of the TCP congestion control too far (by choosing N too > >large) is counterproductive even for the aggregate TCP connection, as > >the paper shows. > > > >I would suggest that this is exactly the kind of development for which > >[RED is needed]. > > > >- Sally > > > > > > ****************************************************************** In my mind it will be a hard sell to the IESG to ask for multiple connections for high-bandwidth data? Scott/Allision any comments on this? > > The symmetric, connection allegiance case was developed when we were > smoking some really good stuff in Haifa. It really addresses a > corner case: minimizing the communication needs between NICs in systems > that will stripe requests across NICs. > > However, somewhere along the line, we forgot to optimize for the > single NIC case. > > ------------------ > > Also using a separate TCP connections may allow us to migrate one > day to an architecture where a different node in the IP network > returns the actual data from a SCSI READ! > > Client <---- iSCSI Control ----> Storage Controller <---- TBD ----> Disk array > ^ ^ > +-------------------------------iSCSI Data protocol -----------------+ > > -Costa -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 06 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27554; 6 Sep 2000 14:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA13902 for ips-outgoing; Wed, 6 Sep 2000 13:58:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (adsl-63-192-219-28.dsl.snfc21.pacbell.net [63.192.219.28]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA13898 for ; Wed, 6 Sep 2000 13:58:17 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 94D708EAE; Wed, 6 Sep 2000 10:56:30 -0700 (PDT) To: Matt Wakeley Cc: ips@ece.cmu.edu Subject: Re: multiple connections References: <39B5FD85.A033376B@agilent.com> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 06 Sep 2000 10:56:30 -0700 In-Reply-To: "Matt Wakeley"'s message of "Wed, 06 Sep 2000 01:17:09 -0700" Message-ID: Lines: 20 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk > > This proposal allows removing the command counters - as commands use a > > single TCP connection. The single connection can also be a shared > > data+control connection. > > For the case of the command connection failure and fail-over to a new > connection, I don't see how you can get away from the command counters. When a > fail over occurs, you will need some way of finding out what commands made it > to the target and which didn't. The easiest way to do this is with command > numbering. > I believe the initiator task tag could be used for recovery. You still need StatRN's though, unless you want to ACK each status. With only one control connection, maybe our motivation for doing recovery decreases. -Costa From owner-ips@ECE.cmu.edu Wed Sep 06 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27894; 6 Sep 2000 15:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA16159 for ips-outgoing; Wed, 6 Sep 2000 14:42:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub1.almaden.ibm.com (mailhub1.almaden.ibm.com [198.4.83.44]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA16143 for ; Wed, 6 Sep 2000 14:42:17 -0400 (EDT) Received: from catalina.almaden.ibm.com (catalina.almaden.ibm.com [9.1.24.58]) by mailhub1.almaden.ibm.com (8.8.8/8.8.8) with ESMTP id LAA51586 for ; Wed, 6 Sep 2000 11:37:36 -0700 Received: (from dfsmith@localhost) by catalina.almaden.ibm.com (AIX4.3/UCB 8.8.8/8.7) id LAA25098 for ips@ece.cmu.edu; Wed, 6 Sep 2000 11:41:46 -0700 From: Daniel Smith Message-Id: <200009061841.LAA25098@catalina.almaden.ibm.com> Subject: Re: iSCSI Autosense Consensus, Connection next steps To: ips@ece.cmu.edu Date: Wed, 6 Sep 2000 11:41:46 -0700 (PDT) In-Reply-To: <39B646B1.816538D6@stewart.chicago.il.us> from "Randall R. Stewart" at Sep 06, 2000 08:29:21 AM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Randall R. Stewart wrote: > > I strongly concur here. We can NOT have multiple TCP connections and > be able to stay TCP friendly to the rest of the internet... FTP (2+ TCP streams), HTTP (5 TCP streams).... -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 Home: +1(408)227-5786 From owner-ips@ECE.cmu.edu Wed Sep 06 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27819; 6 Sep 2000 15:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA16394 for ips-outgoing; Wed, 6 Sep 2000 14:46:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA16380 for ; Wed, 6 Sep 2000 14:46:17 -0400 (EDT) Received: from p7020-img-nt.cisco.com (fred-hm-dhcp3.cisco.com [171.69.128.118]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id LAA19669 for ; Wed, 6 Sep 2000 11:46:07 -0700 (PDT) Message-Id: <4.3.2.7.2.20000906113913.03956580@flipper.cisco.com> X-Sender: fred@flipper.cisco.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Wed, 06 Sep 2000 11:42:00 -0700 To: ips@ece.cmu.edu From: Fred Baker Subject: Re: a vote for asymmetric connections in a session In-Reply-To: <4.2.2.20000906133134.00abc4b0@router.spinnakernet.com> References: <39B65F1F.2BA8110B@agilent.com> <4.2.2.20000906081014.00ac2450@pop.icubed.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 02:10 PM 9/6/00 -0400, Mike Kazar wrote: >I think it might be illuminating to try to figure out some algorithms to >keep the throughput high even when one of the connections drops a packet >and cuts its TCP window size down significantly. dumb question. Has anyone looked into the ECN proposal? It seems like the issue here is the impacts of loss in the TCP state machine, and those can be mitigated by managing the queues actively. You could assert that you want ECN to be implemented in any iSCSI path, which should have the effect of managing queue depths without incurring loss. http://www.isi.edu/in-notes/rfc2481.txt From owner-ips@ECE.cmu.edu Wed Sep 06 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27907; 6 Sep 2000 15:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA17100 for ips-outgoing; Wed, 6 Sep 2000 15:03:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA17095 for ; Wed, 6 Sep 2000 15:03:04 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id OAA35032; Wed, 6 Sep 2000 14:50:53 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id NAA110076; Wed, 6 Sep 2000 13:03:03 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI Autosense Consensus, Connection next steps To: Stephen Bailey Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 6 Sep 2000 11:57:34 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/06/2000 01:03:02 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, I think we are now communicating, and you have made some good points. Not sure I fully agree, but the points of discussion and thought are, I think, clearer. . . . John L. Hufferd Stephen Bailey @ece.cmu.edu on 09/06/2000 08:06:04 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI Autosense Consensus, Connection next steps > You talked about Transport layer and Wedge layers. I guess I do not know > exactly what to call iSCSI if given these two options, other then to > consider iSCSI part of the SCSI Transport (layer). It is a device specific > driver, which in this case deals with iSCSI Devices. I guess we have a terminological gap here. Here's how I understand the terms. Generally, I believe the term `transport layer' is used to describe the uppermost layer in a stack that carries data, without any particular thought to what the data is. For example, ST, TCP, UDP, FC-2. Above the transport layer are layers that know what the particular data being carried is e.g. FTP, NFS, HTTP. Clearly this `knowing' is relative---an HTTP layer doesn't know everything about the data it's carrying, but it knows more than nothing. These more specialized protocols are called upper layer protocols (ULPs). The iSCSI protocol is one of these ULPs. The layers above iSCSI are the parts of a SCSI stack. Typically this includes a mux/demux layer, and, above that, a set of SCSI peripheral drivers (disk, tape, cluster communication, etc..). SCSI wedge drivers are in this SCSI stack, hooked somehow (on top, bottom or possibly in the guts), to the mux/demux layer. > I do not think we should call iSCSI a Wedge Driver. Agreed. My mistake if I suggested that that's how I was thinking of it. iSCSI is a protocol. An iSCSI driver is both a SCSI port driver and a network protocol driver, certainly not a wedge driver. > Having said all that, I think the point you wanted to make was: if the > command load balancing functions can not be done in the xxxx/IP transport > then no one should do it. No. The point I am trying to make is that if CONNECTION-level load balancing (or connection bundling) services are not available from the transport layer (TCP, ST, STCP, etc.), iSCSI should not take this responsibility for itself. I did not say no one should do it. The clear implication if iSCSI does not directly support transport connection bundling, is that a layer above iSCSI will do it. That is what wedge drivers do presently. When we started bringing up FC controllers, existing ||SCSI multipathing code worked as-is. Nobody in the legacy (%^) network SCSI community (FCP) seems to feel a compelling need to add transport layer connection bundling. We certainly didn't see any need to reinvent that wheel early on, and I still don't see it. > I think that David is correct the only point that I was making was that if > iSCSI was able to do the load balancing and some Generic Alternate Path > recovery, that the Vendor Specific Wedge Drivers will be much simpler. Or, my guess is that the support just won't be used. After all, these same wedge drivers already support all types of different SCSI interfaces (||, FCP, SST etc.) concurrently in a more or less interface independent way. Why gunk up a piece of mission-critical code with additional paths that don't offer any user-visible improvement? If we could get a credible wedge driver person to come say `absolutely, connection bundling below the SCSI stack is what we need!', I'll roll over. As it is, I think we're trying to build this feature primarily on speculation. My second objection is that I think it's an architectural error to start filling iSCSI with what are essentially transport-layer enhancements. iSCSI says what data to move, and it is the transport layer's requirement to move it as fast and as reliably as possible by any means available. If you don't dig what the current transport has to offer, use another one. The advantage a ULP gets by not trying to `enhance' the transport-layer, is that it is relatively immune to modifications to the underlying transport. None of the arguments made for iSCSI connection bundling are unique to storage. If connection bundling is so compelling, TCP or some relative will develop its own connection bundling mechanism, and then iSCSI can take advantage of it more or less transparently. Steph From owner-ips@ECE.cmu.edu Wed Sep 06 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28075; 6 Sep 2000 16:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA18389 for ips-outgoing; Wed, 6 Sep 2000 15:29:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA18379 for ; Wed, 6 Sep 2000 15:28:59 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA07984; Wed, 6 Sep 2000 12:19:57 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA24399; Wed, 6 Sep 2000 12:17:32 -0700 From: "Douglas Otis" To: "Matt Wakeley" , Subject: RE: Data in SCSI Response or SCSI Data Date: Wed, 6 Sep 2000 12:31:14 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <39B5E671.B191FF9A@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Tuesday, September 05, 2000 11:39 PM > To: ips@ece.cmu.edu > Subject: Re: Data in SCSI Response or SCSI Data > > > Ok, maybe I'm missing something, but as I read the latest (July > 10) draft, I see > this data/status thing working as follows. I'm going to use > psuedo code, so > that (perhaps) everyone might understand... > > (Note that this whole discussion is only applicable to a target > read operation) > > /* as a target... */ > if ((this is the last iSCSI data PDU to send to fullfill the SCSI > command) and > (this SCSI STATUS is GOOD (except for residual counts)) then When do you get GOOD status and residual counts on a read? What is causing the target to get the length wrong? Is the residual count to be updated during each Read PDU? Except for the MaxCmdRN, (add this to PDU 0x05 and make it valid during EOE) there is little benefit in using the PDU 0x45 structure. Nor can there be a residual condition that is not an error. I do not understand the benefit using this Read PDU. Mark the end of exchange to allow hole checking and clean-up. Both directions could benefit from end notification matched to a tag. In addition, at this level of transport, the initiator may detect errors that should be relayed in some manner at the transport level. Simply add a CHECK or ERROR flag and EOE End of Exchange flag to PDU 0x05 together with a MaxCmdRN if needed. Doug > { > set the "S" bit in the iSCSI DATA PDU header > /* setting the "S" bit explicitely defines "end of exchange" */ > update the "iSCSI Status" field in the iSCSI DATA PDU header > if (there is a residual count) then > { > set the "O" or "U" bits and update the > "Residual Count" field of the iSCSI DATA PDU header > } > send the last iSCSI DATA PDU (completing the SCSI command) > } > else > { > clear the "S" bit in the iSCSI DATA PDU header > send the iSCSI DATA PDU > if (the last iSCSI DATA PDU was sent) > { > create the iSCSI RESPONSE PDU > send it > } > } > > Now, what exactly is the big issue that's required all this discussion? > > -Matt Wakeley > Agilent Technologies > > > Douglas Otis wrote: > > > Julo, > > > > After painfully understanding your status here not there > conversation and > > being ensured Autosense is now a fact of life, rather than > placing status > > within a READ PDU, perhaps you should consider simply revising the > > definition of the flags and use a common DATA PDU. Change the > Status bit to > > indicate Error Detected and perhaps add a bit to signal End of Exchange. > > From End of Exchange you could deduce Good Status. I see > little benefit in > > status presented twice or dummy status sent with partial > exchanges. It only > > seems to make room for conflicts. (The two values not > agreeing, I, on the > > other hand, will never agree.) At what point would the O, and > U flags be > > valid? Would this be a result of an error? If so, perhap > these flags could > > be left to response PDU. > > > > Doug > From owner-ips@ECE.cmu.edu Wed Sep 06 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28086; 6 Sep 2000 16:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA19113 for ips-outgoing; Wed, 6 Sep 2000 15:44:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA19069 for ; Wed, 6 Sep 2000 15:42:59 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id OAA04700; Wed, 6 Sep 2000 14:43:13 -0500 Message-ID: <39B69E51.3154C496@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 14:43:13 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: John Hufferd/San Jose/IBM CC: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I would hope one would use multiple NIC's... multi-homing is a nice feature... What I wonder about is if you have the multiple connections up through the same path to work around TCP CC (as talked about in Sally's email). As long as you can restrict things so that only one connection per NIC is up between target and host.. I don't think I have a problem with that.. unless of course they are all on the same sub-net.. but of course this defeats the purpose of the redundancy.. unless of course the redundancy is there to get a fatter pipe by working around the CC algorithms and increasing your low level NIC card buffers.. Having support in the transport for the multi-homing is IMHO a better solution since that way you CAN fail over to an alternate on the first retransmission.... R John Hufferd/San Jose/IBM wrote: > > I think there is some serious miss reading of the paper, or perhaps miss > understanding on what we are trying to do with iSCSI. The paper talks > about breaking a TCP/IP connection into multiple paths (connections). > There is no iSCSI proposal to do this. In all cases, the iSCSI proposals > talk about the data for a command returning on ONE connection (which > connection it is maybe variable). > > To say that we should not use multiple NICs for iSCSI is perhaps not > reasonable. > > . > . > . > John L. Hufferd > > "Randall R. Stewart" @ece.cmu.edu on > 09/06/2000 06:36:38 AM > > Sent by: owner-ips@ece.cmu.edu > > To: csapuntz@cisco.com > cc: ips@ece.cmu.edu, Allison Mankin , Scott Bradner > > Subject: Re: iSCSI Autosense Consensus, Connection next steps > > Ok, > > explain something to me... > > csapuntz@cisco.com wrote: > > > > Asymmetric connections may be the way to go. > > > > Let's look at characteristics of control vs. data: > > > > * iSCSI control > > - low bandwidth (handled on a CPU) > > - should generally be processed in FIFO order > > - complex state machines (probably software implemented) > > - flow control nice > > - doesn't need to end up in any special buffers > > (can be processed in-place) > > > > * iSCSI data/ready to transmit (rtt) > > - potentially high bandwidth > > - with appropriate headers, segments can be > > processed entirely out-of-order > > - simple data transfer state machine > > - no flow control on data needed, just congestion control > > - rtt's need to be flow-controlled, but > > that can be done with a simple credit mechanism > > - data needs to end up in special buffers (e.g. buffer cache) > > > I understand how you will obtain all of the above with > two TCP connecions. I can even support this, considering > the 1 for control is "low bandwidth" (note: I can't support > N data connections.. only 1). > > But my question is to the Data connection. How do you get > "no flow control on data needed, just congestion control" on > a TCP connection? > > > The processing requirements of the data and control seem quite > > asymmetric. Mixing them on one TCP channel makes separating out the > > data and control more difficult. > > > > Conversely, by putting control and data on separate TCP connections, > > you can use the flow label (IP addresses, TCP ports) to determine > > whether to route this along the control or data path in your machine. > > > > So, I think there is value in the proposal that we use 1 control > > connection and up to n separate dedicated data/RTT connections. > > No I can NOT agree with N seperate dedicated data/RTT connections. I > think you should go look at Sally's email that was posted here > earlier... here it is... > ************************************************************************** > Franco Travostino wrote: > > > > A bit of Related Work ... On the forever-young 1 vs. N TCP connections > > subject, you may be interested in the results shown in the '97 "An > > Application-Level Solution to TCP's Satellite Inefficiencies" paper > > (available at http://jarok.cs.ohiou.edu/projects/satellite/) and on Sally > > Floyd's comment hereafter. The intersection with satellite links is quite > > accidental, and the arguments/results apply above and beyond satellite > links. > > > > -franco > > > > >From majordom@ISI.EDU Fri Dec 6 23:05:58 1996 > > >To: fred@cisco.com (Fred Baker) > > >Cc: end2end-tf@isi.edu > > >Subject: Re: Related paper/re:satellites > > >Date: Fri, 06 Dec 1996 23:05:47 PST > > >From: Sally Floyd > > >Sender: owner-end2end-tf@ISI.EDU > > >Precedence: bulk > > >Content-Length: 2309 > > >X-Lines: 44 > > > > > >Fred - > > > > > >>You might be interested in reviewing this paper, which is what I'm > > >>discussing with Karen Hansen, Dan Shell, and the folks from Comsat. It > > >>relates to some TCP/Satellite work being done at NASA Lewis Research > > >>Center. > > > > > >Basically (from a quick read), the paper, on "An Application-Level > > >Solution to TCP's Satellite Inefficiencies", recommends breaking a > > >single TCP connection into multiple connections at the > > >application level, to increase throughput on satellite circuits. > > >It is not too surprising that this increases the TCP throughput, but it > > >is still not a good idea. For a single TCP connection, a single packet > > >drop results in the throughput for that connection being cut by half, > > >and then increased by roughly one packet per RTT. For a TCP connection > > >that has instead been separated into N different TCP connections, a > > >single packet drop results in one of the N connections, receiving > > >1/N-th of the total bandwidth, having its throughput cut in half. Thus > > >the bandwidth of the aggregate connection has its bandwidth reduced to > > >(1 - 1/(2N))-th of its former bandwidth - that is, the larger the value > > >for N, the smaller the aggregate bandwidth is cut. And then, because > > >each TCP connection continues to increase its congestion window by one > > >packet per RTT, for those TCP connections that have not yet reached the > > >receiver's advertised window, the aggregate TCP connections together > > >increase their bandwidth by up to N packets per RTT. > > > > > >Summarizing, splitting a TCP connection into N separate connections > > >simply increases the aggressiveness of the TCP congestion control. > > >Meaning that this TCP is now more likely to "steal" bandwidth from > > >other TCP connections in times of congestion. And increasing the > > >aggressiveness of the TCP congestion control too far (by choosing N too > > >large) is counterproductive even for the aggregate TCP connection, as > > >the paper shows. > > > > > >I would suggest that this is exactly the kind of development for which > > >[RED is needed]. > > > > > >- Sally > > > > > > > > > > ****************************************************************** > In my mind it will be a hard sell to the IESG to ask for > multiple connections for high-bandwidth data? Scott/Allision any > comments on this? > > > > > The symmetric, connection allegiance case was developed when we were > > smoking some really good stuff in Haifa. It really addresses a > > corner case: minimizing the communication needs between NICs in systems > > that will stripe requests across NICs. > > > > However, somewhere along the line, we forgot to optimize for the > > single NIC case. > > > > ------------------ > > > > Also using a separate TCP connections may allow us to migrate one > > day to an architecture where a different node in the IP network > > returns the actual data from a SCSI READ! > > > > Client <---- iSCSI Control ----> Storage Controller <---- TBD ----> Disk > array > > ^ ^ > > +-------------------------------iSCSI Data protocol -----------------+ > > > > -Costa > > -- > Randall R. Stewart > randall@stewart.chicago.il.us or rrs@cisco.com > 815-342-5222 (cell) 815-477-2127 (work) -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 06 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28198; 6 Sep 2000 17:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA20437 for ips-outgoing; Wed, 6 Sep 2000 16:10:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA20432 for ; Wed, 6 Sep 2000 16:10:51 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA35558; Wed, 6 Sep 2000 16:04:53 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id OAA84008; Wed, 6 Sep 2000 14:10:49 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: Command Queue Depth (was asymmetric/Symmetric) To: Joshua Tseng Cc: ips X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 6 Sep 2000 13:08:08 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/06/2000 02:10:48 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, It is not reasonable for a Target to reject Logins based on Buffer Size. In many situations their may be many Host attempting to connect, and to have a "some times" you can connect and "some times" you can not. Is not a reasonable thing to do when dealing with Host's Storage Connections. Some Type of fair Buffer Management and allocation is required. Clearly there is a limit on the TCP/IP connections but generally these are high enough to not be an issue. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/06/2000 09:07:06 AM Sent by: owner-ips@ece.cmu.edu To: James Smart , ips cc: Subject: RE: Command Queue Depth (was asymmetric/Symmetric) James, I agree with others that there may be an issue with the command windowing mechanism in the existing iSCSI spec. It is like "TCP in reverse", in that the target determines the size of the window, and not the initiator as in TCP. Rather, I believe that everything that this windowing mechanism is attempting to achieve can be more easily obtained by having the target communicate its buffer size to the initiator at iSCSI login. It should be the role of the initiator to determine how many commands to put in flight simultaneously, given this input on available buffer size from the target. As far as multiple initiators, could this not be resolved by the target refusing additional logins beyond the number of initiators it can safely support? Not being a storage expert, this is my best guess/suggestion at how to do it. Josh -----Original Message----- From: James Smart [mailto:jssmart@nhinternet.com] Sent: Wednesday, September 06, 2000 6:37 AM To: ips Subject: Command Queue Depth (was asymmetric/Symmetric) Josh, Wanted to double-check on the scsi-isms implied by the statements below, and post a new question on managing command buffering ... >>My question then is: does Data Balancing on multiple >>connections cause a similar need for additional memory as does the >>synchronous connections with Multiple Commands on different connections. > >In the symmetric model, the target must be prepared to buffer all of the >commands and associated unsolicited data PDU's that may be sent between >ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that or discard >commands. But in order to realize the performance benefits of command >windowing, the target must expand the window in order to let the initiator >to "let her rip!" and put multiple commands in flight. It seems to me that >the existing symmetric model adds complexity which would only be of benefit >if the target has lots of memory. So my question is this: The command >windowing mechanism seems to require lots of resources (including memory) >and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to >implement, but does all this really buy very much? In the end, symmetric >implementations may end up discarding commands anyway, as in the asymmetric >case, unless they want to install globs of memory. First - I assume that "discard commands" actually means completing the command with a SCSI status of BUSY or QUEUE FULL - *NOT* simply dropping the command on the floor. As per previous statements on the reflector, OS's and existing drivers expect older-style SCSI behaviors - dropping of commands from lack of resources expect QUEUE FULL or BUSY indicators to invoke backoff algorithms. OS's typically don't deal well with i/o timeouts for error recovery. Second, are the buffering expectations above in line with existing SCSI/FC devices ? Most scsi devices today, which support command queuing, do indeed support buffering up to N commands (N being the maximum queue depth and is shared across all initiators). Simple disk drives typically support queue depths of 16, 32 or 64 (all to a single lun), while large controllers accept hundreds (shared across multiple luns). Lastly, this topic touches on one of the more sensitive areas in the use of SCSI over packetized interfaces - that of managing the command queue. In parallel SCSI, it was dealt with more efficiently, as the QUEUE FULL or BUSY status was received immediately on the interlocked bus before an additional commands could be sent. However, in FC and iSCSI, the commands will be in the pipe, and there will (are) storms of commands bombarding the target by multiple initiators. As mentioned, unless status was sent back to invoke the backoff algorithms, things continued unabated. I've seen FCP targets fully consumed just trying to hold off these storms. It gets worse as the initiators can be different OS's, with different backoff policies (some do onesy/twosy based on command completion and noting depth levels to manage how much they keep outstanding; while others simply delay for a time, then let the gates wide open to whatever maximum they believe they can send (e.g. a storm)). Given the varied and uncoordinated rates at which the initiators can beat on the command queue resource, there are times where things melt down both on the target and on the initiator backoff/retry algorithms such that little real work gets achieved. I know this last issue is not generic to iSCSI and may be classified as a configuration error. However - to the group - has iSCSI considered any mechanism to report the command queue depth (buffering) available to an initiator ? (my guess is that the delta between ExpCmdRN and MaxCmdRN via the login response implies this). Further - any thoughts on how the resources could then be multiplexed across multiple initiators (thus the ExpCmdRN/MaxCmdRN can't really apply any more). -- James -------------------- James Smart james.smart@nhinternet.com From owner-ips@ECE.cmu.edu Wed Sep 06 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28510; 6 Sep 2000 18:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA22389 for ips-outgoing; Wed, 6 Sep 2000 16:57:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA22379 for ; Wed, 6 Sep 2000 16:57:00 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id PAA04835; Wed, 6 Sep 2000 15:57:20 -0500 Message-ID: <39B6AFB0.C69512F5@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 15:57:20 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Daniel Smith CC: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: <200009061841.LAA25098@catalina.almaden.ibm.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Daniel Smith wrote: > > Randall R. Stewart wrote: > > > > I strongly concur here. We can NOT have multiple TCP connections and > > be able to stay TCP friendly to the rest of the internet... > > FTP (2+ TCP streams), HTTP (5 TCP streams).... > -- FTP has a command stream and then the data transfer stream. The command stream is low bandwidth.. while the data transfer is of course high bandwidth... HTTP --- They did not have any way around the head-of-line blocking issues... iSCSI does have an option that provides both the way around this and keeps the same CC algorithm AND can provide an un-ordered delivery service...something the draft currently leaves ???'s by... I would hope in the future that HTTP would move off of TCP and reduce back to 1 connection that uses the same CC via SCTP or ECN (which SCTP also provides for by the way) or even the new ECM work that Scott pointed out... R > IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 Home: +1(408)227-5786 -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 06 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28471; 6 Sep 2000 18:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA23709 for ips-outgoing; Wed, 6 Sep 2000 17:25:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (truffle.Stanford.EDU [171.64.73.147]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA23705 for ; Wed, 6 Sep 2000 17:25:16 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 2DF488EAC; Wed, 6 Sep 2000 11:58:16 -0700 (PDT) To: "Randall R. Stewart" Cc: ips@ece.cmu.edu Subject: Assymetric connections [WAS: Re: iSCSI Autosense Consensus, Connection next steps] References: <39B64866.961ABDCD@stewart.chicago.il.us> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 06 Sep 2000 11:58:16 -0700 In-Reply-To: "Randall R. Stewart"'s message of "Wed, 06 Sep 2000 08:36:38 -0500" Message-ID: Lines: 59 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Randall, > > * iSCSI data/ready to transmit (rtt) > > - potentially high bandwidth > > - with appropriate headers, segments can be > > processed entirely out-of-order > > - simple data transfer state machine > > - no flow control on data needed, just congestion control > > - rtt's need to be flow-controlled, but > > that can be done with a simple credit mechanism > > - data needs to end up in special buffers (e.g. buffer cache) > > > I understand how you will obtain all of the above with > two TCP connecions. I can even support this, considering > the 1 for control is "low bandwidth" (note: I can't support > N data connections.. only 1). Thanks for pointing this out. The wording was a bit loose. How does this sound? Multiple parallel data connections should only be used in the presence of an ECM-like congestion layer that does congestion control on macro-flows. > But my question is to the Data connection. How do you get > "no flow control on data needed, just congestion control" on > a TCP connection? You don't, but you can approximate it by opening the window to 1-2 gigabytes. A longer answer for why no flow control is needed. Excuse me if this is redundant: I was assuming that all data transfers on the control connection are requested by one of the parties. Since they are requested, there is a buffer waiting to receive the result of the data transfer. For example, let's say an initiator issues a SCSI READ of 64k bytes. It is fairly safe to assume that the initiator has a 64k byte buffer to receive the data. In the case of writes, the initiator does not send the data along with the SCSI WRITE (as is the case in NFS, CIFS, and HTTP PUT). Instead, the initiator waits for the target to request the data. This is done using the request to transmit (RTT) command. By the time a target issues a request to transmit, it presumably has sufficient buffer to receive the response. There is, of course, flow control on data transfers in this system, but it is done at the higher layers and need not be done at the transport. Congestion control, of course, still needs to be done at the transport layer. _Costa From owner-ips@ECE.cmu.edu Wed Sep 06 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28538; 6 Sep 2000 18:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA24438 for ips-outgoing; Wed, 6 Sep 2000 17:40:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA24433 for ; Wed, 6 Sep 2000 17:40:05 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id 1A3AC699 for ; Wed, 6 Sep 2000 14:39:58 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id OAA19161 for ; Wed, 6 Sep 2000 14:39:55 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA4176 for ; Wed, 6 Sep 2000 14:39:52 -0700 Message-ID: <39B6B9CC.6335DEBE@agilent.com> Date: Wed, 06 Sep 2000 14:40:28 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > Command numbering won't be needed anymore - every command with an > outstanding status is restarted. I suppose that could be done, as long as the "retry" bit was still there, so there was no chance of a command being re-issued while the status is in flight back to the initiator... > As for the 2 commands - synch can be achieved by setting data first and then > sending commands. My point was that if there are two NICs using some kind of "mailbox" interface to the system (like I2O), the host will have to put the data command describing the scatter/gather list in the data NIC's mailbox, and wait for some kind of notification that the NIC has pulled the message from the mailbox (can you say extra unnecessary interrupt?) before it can post the command to the control NIC's mailbox. If this synchronization was not done, there would be a small possibility that the command could be sent to the target and the target respond with the data before the data NIC has set itself up to receive and properly place the data. > > > Julo -Matt From owner-ips@ECE.cmu.edu Wed Sep 06 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28544; 6 Sep 2000 18:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA24137 for ips-outgoing; Wed, 6 Sep 2000 17:33:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA24127 for ; Wed, 6 Sep 2000 17:32:58 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id B7F926DE for ; Wed, 6 Sep 2000 17:32:53 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id OAA18962 for ; Wed, 6 Sep 2000 14:32:52 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA4143; Wed, 6 Sep 2000 14:32:48 -0700 Message-ID: <39B6B824.962C49DB@agilent.com> Date: Wed, 06 Sep 2000 14:33:24 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: csapuntz@cisco.com, ips@ece.cmu.edu Subject: Re: multiple connections References: <39B5FD85.A033376B@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk csapuntz@cisco.com wrote: > > > This proposal allows removing the command counters - as commands use a > > > single TCP connection. The single connection can also be a shared > > > data+control connection. > > > > For the case of the command connection failure and fail-over to a new > > connection, I don't see how you can get away from the command counters. When a > > fail over occurs, you will need some way of finding out what commands made it > > to the target and which didn't. The easiest way to do this is with command > > numbering. > > > > I believe the initiator task tag could be used for recovery. How is that? You'd have to query the target about each outstanding command. > > > You still need StatRN's though, unless you want to ACK each status. > > With only one control connection, maybe our motivation for doing recovery > decreases. I don't think this reduces the need for connection failure (be it control or data) recovery. > > > -Costa -Matt From owner-ips@ECE.cmu.edu Wed Sep 06 22:50:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28709; 6 Sep 2000 18:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA25245 for ips-outgoing; Wed, 6 Sep 2000 17:56:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA25239 for ; Wed, 6 Sep 2000 17:56:03 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id C03A080A for ; Wed, 6 Sep 2000 17:55:32 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id OAA19388 for ; Wed, 6 Sep 2000 14:55:31 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA41D8 for ; Wed, 6 Sep 2000 14:55:27 -0700 Message-ID: <39B6BD73.6180DB82@agilent.com> Date: Wed, 06 Sep 2000 14:56:03 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: Re: Command Queue Depth (was asymmetric/Symmetric) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua Tseng wrote: > James, > > I agree with others that there may be an issue with the > command windowing mechanism in the existing iSCSI spec. It is like > "TCP in reverse", in that the target determines the size of the window, and > not the initiator as in TCP. Rather, I believe that everything that this > windowing mechanism is attempting to achieve can be more easily obtained > by having the target communicate its buffer size to the initiator at > iSCSI login. It should be the role of the initiator to determine how > many commands to put in flight simultaneously, given this input on available > buffer size from the target. As more initiators connect to a target, it may need to scale back the amount of this buffering it has allocated to each previously logged in initiator (to prevent rejecting new logins). > > > As far as multiple initiators, could this not be resolved by the target > refusing additional logins beyond the number of initiators it can safely > support? Not being a storage expert, this is my best guess/suggestion > at how to do it. I believe John already answered this... -Matt From owner-ips@ECE.cmu.edu Thu Sep 07 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29004; 6 Sep 2000 19:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA27168 for ips-outgoing; Wed, 6 Sep 2000 18:47:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA27164 for ; Wed, 6 Sep 2000 18:47:05 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e86Ml5x29397 for ; Wed, 6 Sep 2000 17:47:05 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA05358; Wed, 6 Sep 00 17:45:19 -0500 Message-Id: <10009062245.AA05358@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data In-Reply-To: Message from "Douglas Otis" of "Wed, 06 Sep 2000 12:31:14 PDT." Date: Wed, 06 Sep 2000 17:44:57 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > When do you get GOOD status and residual counts on a read? What is causing > the target to get the length wrong? One example is an INQUIRY command. The inquiry data length is target-specific. Typically the CDB allocation length (and DL) are set to some arbitrary large value (0xff), and the target sends back everything it has. The transfer ends with success status. There can certainly be transfer residual and no SCSI error status. Steph From owner-ips@ECE.cmu.edu Thu Sep 07 00:50:06 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29429; 6 Sep 2000 20:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28355 for ips-outgoing; Wed, 6 Sep 2000 19:26:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28348 for ; Wed, 6 Sep 2000 19:26:08 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e86NQ7x29888 for ; Wed, 6 Sep 2000 18:26:07 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA05378; Wed, 6 Sep 00 18:24:22 -0500 Message-Id: <10009062324.AA05378@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps In-Reply-To: Message from "Randall R. Stewart" of "Wed, 06 Sep 2000 11:33:01 CDT." <39B671BD.5817E03@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 18:24:00 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > About 1 year ago we had this "transport bundling" aka load sharing IN > SCTP. At the request of the IESG it was removed because (if I remember > right) it causes all sorts of problems with Congestion Control and > is still deemed a research topic by the IETF (belonging to the > IRTF). Interesting. This speaks volumes. I think more than anything else this goes to the heart of the `it won't work' claim. I have claimed that connection bundling support should either: a) be in the transport (or a low network layer), because layers above the transport (ULPs) do not have effective control over network path. or b) be in the highest possible layer, because the low level components can not be engineered to provide the desired behavior without compromise, so ultimately the user must be exposed to, aware of, and control the nature of the compromises. > It is aong the same lines (to a small degree) as to why you don't > want multiple TCP connections. Very roughly. The specific problem is that NONE of the layers in an IP end-point have complete control over path selection. The IP layer has the best control over this, and it only controls the initial interface selection. There's no robust way to know that outside of the first (and possibly last) hop, you're actually using separate paths. If there is a shared path component, bundled connections through different interfaces will mess with congestion control in the same way as two connections between the same end-points. This doesn't mean that it's impossible to control path selection in ALL cases. It's just that you have to carefully control the network configuration in addition to the software behavior. This is an issue even in the fault tolerance application. You can not say ``all you need to do to get fully redundant paths is hook two NIC cards to each endpoint.'' If the intervening network is beyond your control, you have no idea, and no way to tell what multipathing you're actually getting. The question is whether there should be a feature in iSCSI which breaks if and when lower layer path selection logic changes. I am arguing that a clean separation of concerns between iSCSI and the lower layers on path selection will make iSCSI a more durable protocol. > But the actual implemenation of the load balancing is above the > transport layer... :0 This has to be because IETF doesn't have an acceptable solution, so they're punting. They say that connection bundling messes with congestion control, and then the proceed to hand the responsibility over to layers that are completely ignorant of congestion. Pushing this function up above the transport is not likely to meaningfully change the nature of the traffic flow at all. It will ensure that there are numerically fewer ULPs actually using it, but that's about it. A ULP solution still has the same potential failure modes as that same ULP using a transport level solution. Nonetheless, it certainly has been argued here that iSCSI is a ULP that needs this function. If people really think that the reality that the feature may or may not work on a configuration by configuration basis is OK, and an iSCSI-specific solution offers something important anything over the wedge driver solution, hey, rock on. Steph From owner-ips@ECE.cmu.edu Thu Sep 07 00:50:06 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29427; 6 Sep 2000 20:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28772 for ips-outgoing; Wed, 6 Sep 2000 19:38:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28767 for ; Wed, 6 Sep 2000 19:38:15 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e86NcEx00011 for ; Wed, 6 Sep 2000 18:38:14 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA05390; Wed, 6 Sep 00 18:36:29 -0500 Message-Id: <10009062336.AA05390@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI version info In-Reply-To: Message from Dave Thompson of "Wed, 06 Sep 2000 12:51:45 CDT." <39B68431.1E957C5B@nuspeed.com> Date: Wed, 06 Sep 2000 18:36:07 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > A version negotiation scheme was suggested by Mark Bakke's 29 Jun > 2000 e-mail. > > Is there any plans to support a mechanism like this in the next iSCSI draft? I can never figure out which side of this argument to be on. The basic version idea seems sound, but the problem is that for various reasons, the version number never ends up getting moved. For example, if I recall, FC-PH seemed to go through many version numbers long before it was ever widely implemented, and now it's stabilized at version 4.3. I have never known any implementations that did anything other than 4.3. In ST, we went back and forth numerous times. We ultimately decided that we were going to stipulate that a reved protocol was a new protocol, so no version negotiation would be necessary. The logic for ST went that hardware accelerated protocols are on thin ice if they think they can rev. You're almost always going to work around existing protocol bugs before you ever `fix' the protocol, so the standards committees better be sure of themselves before they forward. I think iSCSI is in this same position. FWIW Steph From owner-ips@ECE.cmu.edu Thu Sep 07 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29896; 6 Sep 2000 21:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA29815 for ips-outgoing; Wed, 6 Sep 2000 20:12:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA29806 for ; Wed, 6 Sep 2000 20:12:17 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by atlrel2.hp.com (Postfix) with ESMTP id 07A236C2 for ; Wed, 6 Sep 2000 20:12:16 -0400 (EDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id RAA14363; Wed, 6 Sep 2000 17:12:16 -0700 (PDT) X-OpenMail-Hops: 1 Date: Wed, 6 Sep 2000 17:12:05 -0700 Message-Id: Subject: RE: Re: multiple connections MIME-Version: 1.0 To: ips@ece.cmu.edu, matt_wakeley@agilent.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I think Matt has an important point here. Let us consider READs and WRITEs seperately. 1. READ - In this case the host software can post the appropriate read buffers to the connection on which the data is to be received and then post the read command to the connection on which the data is to be sent. By the time the target gets the command, and returns data, it is most likely that the NIC receiving the data will have accepted the read buffers. Although synchronization issues are minimized, there is a reasonable amount of performance impact. Two different "commands" (in the sense of host to adapter commands - not in the SCSI sense) have to be posted. Two different queues have to be written to (for sure if they are on different NICs) and two different NICs triggered to accept the "commands". Then there will be two different completion events that will be returned by the NICs to the host - one indicating the acceptance/"completion"/"return" of the command buffer (indicating command sent successfully), and the second when the data is all received (along with the SCSI command status). This may also require interrupt handling etc which could be minimized by some clever schemes. So the amount of work done by the host for the host/adapter interaction is close to doubled. There is probably not that much impact on the target in this case. 2. WRITES - This is the really bad one in my opinion. For me, avoiding RTTs in iSCSI would just by itself make iSCSI a superior "transport" for SCSI. So assuming RTTs are not being used, the host would (changing the posting order from READs), first post the write command to the connection on which the data is to be sent, and then post the write buffers to the connection which is to be used for sending data. One case is where for whatever reason, the target gets the data before it gets the command, and has no clue what to do with the data. Let us assume that the target does get the command before it gets the data. The target gets a command indicating that data being written to whatever lun and whatever location is going to arrive on some other connection. First the target has taken an extra event from the adapter. Then, when data arrives on another NIC (and the target gets another event), the target goes through the list of outstanding WRITE commands to match the command with the data and then go about the business of processing the data. So in this case the work has increased for the initiator as well as the target. Somesh > -----Original Message----- > From: matt_wakeley@agilent.com [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 1:17 AM > To: ips@ece.cmu.edu > Subject: FW: Re: multiple connections > > > julian_satran@il.ibm.com wrote: > > > Dear colleagues, > > > > With all the heated debate about multiple vs. single > connection a request I > > made a while ago got no significant reply (neither for nor against). > > > > The request was to consider a proposal made by Kalman Meth > to reconsider > > the asymmetric model with the addition of a path selection > made by the > > initiator. > > One downside I see to this proposal is that for a host with > multiple iSCSI > nics, it will have to issue two commands (in this order): one > to the "data path > nic" to describe where the data is to be read from/written > to, and one to the > "control path nic" to send the command. This would be > communication to two > cards instead of one. Also, note that the communication to > the data path nic > MUST complete before the control path nic sends the command, > requiring some > kind of sync mechanism. > > > > This proposal allows removing the command counters - as > commands use a > > single TCP connection. The single connection can also be a shared > > data+control connection. > > For the case of the command connection failure and fail-over to a new > connection, I don't see how you can get away from the command > counters. When a > fail over occurs, you will need some way of finding out what > commands made it > to the target and which didn't. The easiest way to do this > is with command > numbering. > > > > > > > In case of multiple connection the data path to be used is > selected and > > maintained until the command ends. > > > > Thanks, > > Julo > > -Matt Wakeley > Agilent Technologies > From owner-ips@ECE.cmu.edu Thu Sep 07 01:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29967; 6 Sep 2000 21:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA01216 for ips-outgoing; Wed, 6 Sep 2000 20:59:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA01212 for ; Wed, 6 Sep 2000 20:59:14 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id TAA05448; Wed, 6 Sep 2000 19:59:30 -0500 Message-ID: <39B6E872.F618FD2C@stewart.chicago.il.us> Date: Wed, 06 Sep 2000 19:59:30 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: csapuntz@cisco.com CC: ips@ece.cmu.edu Subject: Re: Assymetric connections [WAS: Re: iSCSI Autosense Consensus, Connection next steps] References: <39B64866.961ABDCD@stewart.chicago.il.us> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk csapuntz@cisco.com wrote: > > Hi Randall, > > > > * iSCSI data/ready to transmit (rtt) > > > - potentially high bandwidth > > > - with appropriate headers, segments can be > > > processed entirely out-of-order > > > - simple data transfer state machine > > > - no flow control on data needed, just congestion control > > > - rtt's need to be flow-controlled, but > > > that can be done with a simple credit mechanism > > > - data needs to end up in special buffers (e.g. buffer cache) > > > > > I understand how you will obtain all of the above with > > two TCP connecions. I can even support this, considering > > the 1 for control is "low bandwidth" (note: I can't support > > N data connections.. only 1). > > Thanks for pointing this out. The wording was a bit loose. > > How does this sound? > > Multiple parallel data connections should only be used in the > presence of an ECM-like congestion layer that does congestion > control on macro-flows. Much better :) but are you going to use the standard SHOULD/MUST/MUST NOT conventions? I did not see the standard pointers in the document and I also did not see any MUST/SHOULD's in here.. I think the above needs to not be a should but more a MUST... how about " If multiple parallel data connections are to be used, they MUST be used with an ECM-like congestion layer that does congestion control on macro-flows. " Also you may need to put some references in to ECM or describe it some where :) > > > But my question is to the Data connection. How do you get > > "no flow control on data needed, just congestion control" on > > a TCP connection? > > You don't, but you can approximate it by opening the window to 1-2 > gigabytes. > As long as the data buffers are there to support this it should be no problem. I have found in my playing that often times if the kernel underneath has some buffer set (say 64k in a solaris TLI UDP stack) a user application above this that tries to set a window above 64k is wasting its time i.e. you start getting data losses at the host, a very sad occurance when you transport it all the way across a large IP network :0 But I have to assume we are talking about not your ordinary TCP/IP stack here by the things that are being asked... or one that has some extra knobs to turn... I am still unsure how you will get unordered messages across though... -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 07 02:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30004; 6 Sep 2000 22:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA01979 for ips-outgoing; Wed, 6 Sep 2000 21:23:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA01974 for ; Wed, 6 Sep 2000 21:23:46 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id SAA08072; Wed, 6 Sep 2000 18:14:41 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id SAA25313; Wed, 6 Sep 2000 18:12:18 -0700 From: "Douglas Otis" To: "Stephen Bailey" , Subject: RE: Data in SCSI Response or SCSI Data Date: Wed, 6 Sep 2000 18:26:03 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <10009062245.AA05358@candide.cs.uchicago.edu> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, Although a logical use of a residual, I can not see where it is defined should the allocation length be greater than the returned length. Should the allocation be less than the response, this is a Check Condition and the residual is defined. It would seem a residual in a case with adequate allocation is not interesting or defined. Do you know where this mechanism you describe is defined? In normal use, without a Check Condition such information is not returned to the application. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Bailey > Sent: Wednesday, September 06, 2000 3:45 PM > To: ips@ece.cmu.edu > Subject: Re: Data in SCSI Response or SCSI Data > > > > When do you get GOOD status and residual counts on a read? > What is causing > > the target to get the length wrong? > > One example is an INQUIRY command. The inquiry data length is > target-specific. Typically the CDB allocation length (and DL) are set > to some arbitrary large value (0xff), and the target sends back > everything it has. The transfer ends with success status. > > There can certainly be transfer residual and no SCSI error status. > > Steph > From owner-ips@ECE.cmu.edu Thu Sep 07 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30667; 7 Sep 2000 0:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA05359 for ips-outgoing; Wed, 6 Sep 2000 23:07:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA05346 for ; Wed, 6 Sep 2000 23:06:50 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id UAA05218; Wed, 6 Sep 2000 20:06:44 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Wed, 6 Sep 2000 20:06:43 -0700 Message-ID: From: Jim McGrath To: "'Matt Wakeley'" , ips Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Wed, 6 Sep 2000 20:06:40 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk The issue of buffer space allocation for multiple initiators has a long and troubled history in SCSI. We have never been able to come up with a good answer. Fibre Channel tried to fix this with the notion of "login BB credit" - when you login you get a minimum number of credits you are always guaranteed when you start data transfers. The problem with this is that storage devices had no realistic ability to discriminate between initiators or to change the login BB credit. In addition, the expectation is that all possible initiators would get these credits on login. So storage devices vendors have played it safe and kept this number low (at 0 until recently, now around 2). For iSCSI the number of initial credits you need to "prime the pump" until normal data flow is established is probably large (given the latencies are higher than in Fibre Channel, especially FC-AL), and the number of potential initiators larger than in Fibre Channel, making this a whole lot worse for the storage device. As soon as we allow the devices to start adjusting these credits, then you have the protocol problem of making sure people know when their credits are adjusted and the policy problem of how, who, and when to adjust the credits. Changing everyone's credit when you add a new initiator can get into a notification nightmare, although it is "fair." Any policy brings up all sorts of nasty issues regarding fairness vs efficient use of the transmission media. Jim Note: the same problem has plagued other attempts to allocate device resources between multiple initiators, like command queue space. In general policies with respect to multiple initiators are not really standard in the SCSI world. -----Original Message----- From: Matt Wakeley [mailto:matt_wakeley@agilent.com] Sent: Wednesday, September 06, 2000 2:56 PM To: ips Subject: Re: Command Queue Depth (was asymmetric/Symmetric) Joshua Tseng wrote: > James, > > I agree with others that there may be an issue with the > command windowing mechanism in the existing iSCSI spec. It is like > "TCP in reverse", in that the target determines the size of the window, and > not the initiator as in TCP. Rather, I believe that everything that this > windowing mechanism is attempting to achieve can be more easily obtained > by having the target communicate its buffer size to the initiator at > iSCSI login. It should be the role of the initiator to determine how > many commands to put in flight simultaneously, given this input on available > buffer size from the target. As more initiators connect to a target, it may need to scale back the amount of this buffering it has allocated to each previously logged in initiator (to prevent rejecting new logins). > > > As far as multiple initiators, could this not be resolved by the target > refusing additional logins beyond the number of initiators it can safely > support? Not being a storage expert, this is my best guess/suggestion > at how to do it. I believe John already answered this... -Matt From owner-ips@ECE.cmu.edu Thu Sep 07 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30686; 7 Sep 2000 0:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA05618 for ips-outgoing; Wed, 6 Sep 2000 23:14:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA05610 for ; Wed, 6 Sep 2000 23:14:01 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id UAA05772; Wed, 6 Sep 2000 20:13:59 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Wed, 6 Sep 2000 20:13:58 -0700 Message-ID: From: Jim McGrath To: "'Stephen Bailey'" , ips@ece.cmu.edu Subject: RE: iSCSI Autosense Consensus, Connection next steps Date: Wed, 6 Sep 2000 20:13:55 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="ISO-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk In practice a number of interesting applications (like a machine room or limited LAN) can control (or easily know) their configurations, and so might be able to use a feature that was configuration dependent (SCSI, Fibre Channel, and Ethernet all have some configuration constraints today for certain things). I agree that this is not the case for the general network environment (lots of mixing of low layer protocols, fabrics not under the control of a single organization). Jim -----Original Message----- From: Stephen Bailey [mailto:steph@cs.uchicago.edu] Sent: Wednesday, September 06, 2000 4:24 PM To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps > About 1 year ago we had this "transport bundling" aka load sharing IN > SCTP. At the request of the IESG it was removed because (if I remember > right) it causes all sorts of problems with Congestion Control and > is still deemed a research topic by the IETF (belonging to the > IRTF). Interesting. This speaks volumes. I think more than anything else this goes to the heart of the `it won't work' claim. I have claimed that connection bundling support should either: a) be in the transport (or a low network layer), because layers above the transport (ULPs) do not have effective control over network path. or b) be in the highest possible layer, because the low level components can not be engineered to provide the desired behavior without compromise, so ultimately the user must be exposed to, aware of, and control the nature of the compromises. > It is aong the same lines (to a small degree) as to why you don't > want multiple TCP connections. Very roughly. The specific problem is that NONE of the layers in an IP end-point have complete control over path selection. The IP layer has the best control over this, and it only controls the initial interface selection. There's no robust way to know that outside of the first (and possibly last) hop, you're actually using separate paths. If there is a shared path component, bundled connections through different interfaces will mess with congestion control in the same way as two connections between the same end-points. This doesn't mean that it's impossible to control path selection in ALL cases. It's just that you have to carefully control the network configuration in addition to the software behavior. This is an issue even in the fault tolerance application. You can not say ``all you need to do to get fully redundant paths is hook two NIC cards to each endpoint.'' If the intervening network is beyond your control, you have no idea, and no way to tell what multipathing you're actually getting. The question is whether there should be a feature in iSCSI which breaks if and when lower layer path selection logic changes. I am arguing that a clean separation of concerns between iSCSI and the lower layers on path selection will make iSCSI a more durable protocol. > But the actual implemenation of the load balancing is above the > transport layer... :0 This has to be because IETF doesn't have an acceptable solution, so they're punting. They say that connection bundling messes with congestion control, and then the proceed to hand the responsibility over to layers that are completely ignorant of congestion. Pushing this function up above the transport is not likely to meaningfully change the nature of the traffic flow at all. It will ensure that there are numerically fewer ULPs actually using it, but that's about it. A ULP solution still has the same potential failure modes as that same ULP using a transport level solution. Nonetheless, it certainly has been argued here that iSCSI is a ULP that needs this function. If people really think that the reality that the feature may or may not work on a configuration by configuration basis is OK, and an iSCSI-specific solution offers something important anything over the wedge driver solution, hey, rock on. Steph From owner-ips@ECE.cmu.edu Thu Sep 07 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30691; 7 Sep 2000 0:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA05939 for ips-outgoing; Wed, 6 Sep 2000 23:23:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA05926 for ; Wed, 6 Sep 2000 23:23:37 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id 44B1F16C for ; Wed, 6 Sep 2000 20:23:36 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id UAA23684 for ; Wed, 6 Sep 2000 20:23:34 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA49CD for ; Wed, 6 Sep 2000 20:23:30 -0700 Message-ID: <39B70A27.CB216B11@agilent.com> Date: Wed, 06 Sep 2000 20:23:19 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips Subject: Re: Command Queue Depth (was asymmetric/Symmetric) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, I agree that FC has tried (unsuccessfully) to address this command queue allocation problem. However, the "login bb credit" mechanism in FC does not address the command queue depth issue at all. BB credit is used to receive commands and/or data, and the target has no clue in advance what is coming. BB credit is just there to ensure that there is a lowest layer buffer available to receive the FC frame (as opposed to dropping it on the floor if there is no "mac" buffer, like ethernet does). It does not mean the command queue has any room for the frame. At one time, there was a big push to have "data" credits and "command" credits to take care of this problem, but it couldn't be made to work and be "backwards compatible". > The issue of buffer space allocation for multiple initiators has a long and > troubled history in SCSI. We have never been able to come up with a good > answer. > > Fibre Channel tried to fix this with the notion of "login BB credit" - when > you login you get a minimum number of credits you are always guaranteed when > you start data transfers. The problem with this is that storage devices had > no realistic ability to discriminate between initiators or to change the > login BB credit. In addition, the expectation is that all possible > initiators would get these credits on login. So storage devices vendors > have played it safe and kept this number low (at 0 until recently, now > around 2). For iSCSI the number of initial credits you need to "prime the > pump" until normal data flow is established is probably large (given the > latencies are higher than in Fibre Channel, especially FC-AL), and the How is the latency low on FC-AL?? given that you need to arbitrate and win the loop, then receive bb credit, before you can send anything? -Matt > > number of potential initiators larger than in Fibre Channel, making this a > whole lot worse for the storage device. > > As soon as we allow the devices to start adjusting these credits, then you > have the protocol problem of making sure people know when their credits are > adjusted and the policy problem of how, who, and when to adjust the credits. > Changing everyone's credit when you add a new initiator can get into a > notification nightmare, although it is "fair." Any policy brings up all > sorts of nasty issues regarding fairness vs efficient use of the > transmission media. > > Jim > > Note: the same problem has plagued other attempts to allocate device > resources between multiple initiators, like command queue space. In general > policies with respect to multiple initiators are not really standard in the > SCSI world. > > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 2:56 PM > To: ips > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > Joshua Tseng wrote: > > > James, > > > > I agree with others that there may be an issue with the > > command windowing mechanism in the existing iSCSI spec. It is like > > "TCP in reverse", in that the target determines the size of the window, > and > > not the initiator as in TCP. Rather, I believe that everything that this > > windowing mechanism is attempting to achieve can be more easily obtained > > by having the target communicate its buffer size to the initiator at > > iSCSI login. It should be the role of the initiator to determine how > > many commands to put in flight simultaneously, given this input on > available > > buffer size from the target. > > As more initiators connect to a target, it may need to scale back the amount > of > this buffering it has allocated to each previously logged in initiator (to > prevent rejecting new logins). > > > > > > > As far as multiple initiators, could this not be resolved by the target > > refusing additional logins beyond the number of initiators it can safely > > support? Not being a storage expert, this is my best guess/suggestion > > at how to do it. > > I believe John already answered this... > > -Matt From owner-ips@ECE.cmu.edu Thu Sep 07 07:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31677; 7 Sep 2000 3:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA10645 for ips-outgoing; Thu, 7 Sep 2000 02:30:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA10638 for ; Thu, 7 Sep 2000 02:30:14 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id CAA46392; Thu, 7 Sep 2000 02:24:16 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id AAA130496; Thu, 7 Sep 2000 00:30:12 -0600 X-Priority: 1 (High) Importance: Normal Subject: Symmetric vs Asymmetric To: black_david@emc.com Cc: ips X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 6 Sep 2000 23:26:35 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/07/2000 12:30:12 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David, One of the reasons I was holding out for Symmetric over Asymmetric, was for the workload balancing etc. that could be done at the iSCSI layer. I am now willing to "roll-over" but I think I owe you all my reasons for deciding to support the Asymmetric approach: First, in the degenerate case, they both look the same. That is, they can both support a single Connection per Session. So the simplest implementation is still possible with the Asymmetric approach as it was with the Symmetric approach. This means that most current iSCSI implementations can continue to exist, unchanged. But the most important reason follows: Since a session is a unique construct that identifies an Initiator and a Target, I am assuming that does NOT mean each NICs has a unique initiator ID but that they can, and should, share a common Initiator ID. The Target ID, on the other hand, is either unique per Storage Controller, or not, and this will depend on the type of Storage Controller. This means that the Host iSCSI device driver will be able to present, to the layers above it, either a single port per storage controller, or multiple ports, depending on how the Storage Controller answers Logins on each of its IP connections. That means iSCSI will not be able to determine when it knows enough to perform alternate path retry, and for the same reason will not be able to know when or how to perform command work load balancing across the different sessions. This is because either all connections to a storage controller have either the same iSCSI session ID (SSID), or they are different -- but iSCSI will not, by itself, know which sessions can be substituted for others (for balancing or retry). The only place where the symmetric approach can do workload balancing and alternate path retry is where the storage controller makes it look like a single target ID even though it might have multiple IP addresses. Though this might work in some IBM storage controllers (like Shark) it will not work for all IBM storage controllers (like those of Mylex). This same or similar argument probably will apply to EMC's and other vendors various storage controllers. Therefore, having iSCSI do any type of command balancing or command automatic alternate path retry is problematical without outside influences (probably from upper layers) and this will tend to make the iSCSI layer very thick and complicated. It will also mix the layering concepts (in general not a good idea). Therefore, I now believe that the Symmetrical approach can not be a reasonable general/generic approach to Command work load balance or alternate path retry, and that function is best left to the Wedge Drivers. The Asymmetric approach can not perform command balancing or command automatic path retry either, but it does not try. It only tries to balance the data across multiple connections (and that seems much more reasonable). With the Asymmetric approach, data for any specific command only flows on one connection. So you can look at the data from various commands as independent flows on the different connections (NICs) so there should not be blocking issues, or parallel flow issues, and RDMA type direct to memory functions can occur. With multiple storage controllers, there may be many sessions and many data connections spread across all the various NICs in a Host. The net of all this is that the Asymmetric approach: Probably has good enough spread of commands and data across multiple NICs, even without Wedge Drivers. Permits the upper layer Wedge Drivers to be left mostly unchanged. Permits the simple case with a single connection per session, even with multiple NICs. Permits the most simple case of a single connection per session with only a single NIC. Since I have seen no one else that made arguments for the Symmetric implementation, you maybe able to call the agreement on the Asymmetric approach a consensus. . . . John L. Hufferd From owner-ips@ECE.cmu.edu Thu Sep 07 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31690; 7 Sep 2000 3:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA10282 for ips-outgoing; Thu, 7 Sep 2000 02:11:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA10277 for ; Thu, 7 Sep 2000 02:11:30 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA103630 for ; Thu, 7 Sep 2000 08:10:59 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA93120 for ; Thu, 7 Sep 2000 08:10:59 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256953.0021F546 ; Thu, 7 Sep 2000 08:10:54 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 7 Sep 2000 09:08:41 +0300 Subject: Re: Command Queue Depth (was asymmetric/Symmetric) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Lower bounds for buffering are determined only by device and network latency and the actual buffering efficiency can be judged against them. Actual buffering needs are highly dependent on techniques used. I've recently looked at the differences in buffering needs between symmetric and asymmetric connections and the symmetric will probably result in slightly higher buffering needs (but not enough to tip the balance in favor of asymmetric if there are other arguments against it). Most of the buffering requirements in the target will be related to out-of-order unsolicited data. Julo "James Smart" on 06/09/2000 16:37:08 Please respond to "James Smart" To: "ips" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Command Queue Depth (was asymmetric/Symmetric) Josh, Wanted to double-check on the scsi-isms implied by the statements below, and post a new question on managing command buffering ... >>My question then is: does Data Balancing on multiple >>connections cause a similar need for additional memory as does the >>synchronous connections with Multiple Commands on different connections. > >In the symmetric model, the target must be prepared to buffer all of the >commands and associated unsolicited data PDU's that may be sent between >ExpCmdRN and MaxCmdRN. This can be a lot of memory. Either that or discard >commands. But in order to realize the performance benefits of command >windowing, the target must expand the window in order to let the initiator >to "let her rip!" and put multiple commands in flight. It seems to me that >the existing symmetric model adds complexity which would only be of benefit >if the target has lots of memory. So my question is this: The command >windowing mechanism seems to require lots of resources (including memory) >and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to >implement, but does all this really buy very much? In the end, symmetric >implementations may end up discarding commands anyway, as in the asymmetric >case, unless they want to install globs of memory. First - I assume that "discard commands" actually means completing the command with a SCSI status of BUSY or QUEUE FULL - *NOT* simply dropping the command on the floor. As per previous statements on the reflector, OS's and existing drivers expect older-style SCSI behaviors - dropping of commands from lack of resources expect QUEUE FULL or BUSY indicators to invoke backoff algorithms. OS's typically don't deal well with i/o timeouts for error recovery. Second, are the buffering expectations above in line with existing SCSI/FC devices ? Most scsi devices today, which support command queuing, do indeed support buffering up to N commands (N being the maximum queue depth and is shared across all initiators). Simple disk drives typically support queue depths of 16, 32 or 64 (all to a single lun), while large controllers accept hundreds (shared across multiple luns). Lastly, this topic touches on one of the more sensitive areas in the use of SCSI over packetized interfaces - that of managing the command queue. In parallel SCSI, it was dealt with more efficiently, as the QUEUE FULL or BUSY status was received immediately on the interlocked bus before an additional commands could be sent. However, in FC and iSCSI, the commands will be in the pipe, and there will (are) storms of commands bombarding the target by multiple initiators. As mentioned, unless status was sent back to invoke the backoff algorithms, things continued unabated. I've seen FCP targets fully consumed just trying to hold off these storms. It gets worse as the initiators can be different OS's, with different backoff policies (some do onesy/twosy based on command completion and noting depth levels to manage how much they keep outstanding; while others simply delay for a time, then let the gates wide open to whatever maximum they believe they can send (e.g. a storm)). Given the varied and uncoordinated rates at which the initiators can beat on the command queue resource, there are times where things melt down both on the target and on the initiator backoff/retry algorithms such that little real work gets achieved. I know this last issue is not generic to iSCSI and may be classified as a configuration error. However - to the group - has iSCSI considered any mechanism to report the command queue depth (buffering) available to an initiator ? (my guess is that the delta between ExpCmdRN and MaxCmdRN via the login response implies this). Further - any thoughts on how the resources could then be multiplexed across multiple initiators (thus the ExpCmdRN/MaxCmdRN can't really apply any more). -- James -------------------- James Smart james.smart@nhinternet.com From owner-ips@ECE.cmu.edu Thu Sep 07 07:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31770; 7 Sep 2000 3:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA11495 for ips-outgoing; Thu, 7 Sep 2000 03:07:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA11490 for ; Thu, 7 Sep 2000 03:07:03 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 7 Sep 2000 03:06:31 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F5B@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: ips Charter news Date: Thu, 7 Sep 2000 03:06:29 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Since it's been over a month since Pittsburgh, I thought I'd provide an update on the ips draft charter status. The latest charter draft is now in the hands of the IESG/IAB - aside from a bunch of editorial cleanups, there have been two substantial changes since the version made available before the Pittsburgh meeting: - The charter now envisions ips adopting a Technical Coordinator structure along the lines used in the issll WG. Technical Coordinators would be initially selected by the co-chairs for iSCSI and FC-over-IP. Among the things that this helps with is organizing parallel sessions at meetings to deal with protocol-specific items. - The dates for completion of documents have been pushed back by approximately one IETF meeting. Many people thought the original dates were aggressive, nobody thought they were too lenient, hence the result. If we meet the original dates, so much the better. There may be more changes to come. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Sep 07 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31878; 7 Sep 2000 4:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA11270 for ips-outgoing; Thu, 7 Sep 2000 02:59:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA11265 for ; Thu, 7 Sep 2000 02:58:58 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 7 Sep 2000 02:58:27 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F5A@corpmx9.isus.emc.com> To: hufferd@us.ibm.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: Symmetric vs Asymmetric Date: Thu, 7 Sep 2000 02:58:25 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Since I have seen no one else that made arguments for the Symmetric > implementation, you maybe able to call the agreement on the Asymmetric > approach a consensus. While I appreciate the invitation, I'm going to hold off on doing that. The current state of the discussion doesn't feel like consensus (rough or otherwise) to me yet, and it seems to me that there have been people other than John advocating the Symmetric model. I do want to kill off a red herring that found its way into the discussion. Squeezing additional bandwidth out of a single network link (HTTP-style) has not been an important motivation for multiple connections. The aims of multiple connections have been to use parallelism in the network for failover and load balancing. A few weeks ago, there was rough consensus that use of multiple connections to squeeze additional bandwidth out of single network would not be a requirement on iSCSI, and I haven't seen anything that leads me to believe that this consensus has changed. Progress is being made -- carry on ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Sep 07 09:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32446; 7 Sep 2000 5:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA13515 for ips-outgoing; Thu, 7 Sep 2000 04:42:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from radmail.rad.co.il (radmail.rad.co.il [207.232.32.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA13506 for ; Thu, 7 Sep 2000 04:42:02 -0400 (EDT) Received: from sanrad.com ([192.114.26.219]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-54115U1200L1200S0V35) with ESMTP id il; Thu, 7 Sep 2000 11:44:56 +0300 Message-ID: <39B770FB.19325042@sanrad.com> Date: Thu, 07 Sep 2000 11:42:03 +0100 From: Yaron Klein Organization: SANRAD X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: Dave Thompson CC: "Satran, Julian" , ips@ece.cmu.edu Subject: Re: iSCSI version info References: <39B68431.1E957C5B@nuspeed.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Dave, The version number will be negotiated in the login phase. I have allocated a field in the login command and request for it. Julian, Mark and I are now working on the login and security chapters, hope to come with preliminary version soon. Regards, Yaron Klein SANRAD Dave Thompson wrote: > Julian, > > A version negotiation scheme was suggested by Mark Bakke's 29 Jun 2000 e-mail. > > > Logins for iSCSI draft 03 might benefit from the following additions: > > > > 1. Version number text fields should be placed in both login requests > > and responses. the request could either specify the version of the > > protocol the initiator wishes to run, or perhaps a list of versions > > that the initiator is capable of running. > > > > Version: 1.0 > > > > -OR- > > > > Version: 1.0 1.1 1.2 > > > > The target would then choose the highest version that it supported > > in common with those requested, and return it in the response: > > > > Version: 1.1 > > Is there any plans to support a mechanism like this in the next iSCSI draft? > > In addition, perhaps a one byte field in the login command and response headers > (byte 3?) could be added which would contain the major protocol version number. > This could then be used to determine the proper format of the header in cases > where major header layout changes occur (e.g. the last draft's size change from > 40 to 48 bytes). > > -- > Dave Thompson > NuSpeed, Inc. > dave.thompson@nuspeed.com From owner-ips@ECE.cmu.edu Thu Sep 07 09:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32513; 7 Sep 2000 5:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA13162 for ips-outgoing; Thu, 7 Sep 2000 04:25:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA13158 for ; Thu, 7 Sep 2000 04:25:23 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA220380 for ; Thu, 7 Sep 2000 10:24:51 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA87266 for ; Thu, 7 Sep 2000 10:24:46 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256953.002E34FD ; Thu, 7 Sep 2000 10:24:42 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 7 Sep 2000 11:22:18 +0300 Subject: Re: iSCSI Autosense Consensus, Connection next steps Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Randal, OK - the main reason we want to use several connections is link aggregation (i.e. different physical connections). And then even attempting to use more aggressively congestion control will not give you an edge on the network (not a significant one) as long as you are using the same IP address as I assume that RED is looking for this too (and if not it should!). An interesting point - in a former life doing a satellite transmission I decide against such a trick and used a FEC scheme to bring BER in-line with the wired links (in fact 1 order of magnitude better to compensate for RTT). Julo "Randall R. Stewart" on 06/09/2000 16:36:38 Please respond to "Randall R. Stewart" To: csapuntz@cisco.com cc: ips@ece.cmu.edu, Allison Mankin , Scott Bradner (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI Autosense Consensus, Connection next steps Ok, explain something to me... csapuntz@cisco.com wrote: > > Asymmetric connections may be the way to go. > > Let's look at characteristics of control vs. data: > > * iSCSI control > - low bandwidth (handled on a CPU) > - should generally be processed in FIFO order > - complex state machines (probably software implemented) > - flow control nice > - doesn't need to end up in any special buffers > (can be processed in-place) > > * iSCSI data/ready to transmit (rtt) > - potentially high bandwidth > - with appropriate headers, segments can be > processed entirely out-of-order > - simple data transfer state machine > - no flow control on data needed, just congestion control > - rtt's need to be flow-controlled, but > that can be done with a simple credit mechanism > - data needs to end up in special buffers (e.g. buffer cache) > I understand how you will obtain all of the above with two TCP connecions. I can even support this, considering the 1 for control is "low bandwidth" (note: I can't support N data connections.. only 1). But my question is to the Data connection. How do you get "no flow control on data needed, just congestion control" on a TCP connection? > The processing requirements of the data and control seem quite > asymmetric. Mixing them on one TCP channel makes separating out the > data and control more difficult. > > Conversely, by putting control and data on separate TCP connections, > you can use the flow label (IP addresses, TCP ports) to determine > whether to route this along the control or data path in your machine. > > So, I think there is value in the proposal that we use 1 control > connection and up to n separate dedicated data/RTT connections. No I can NOT agree with N seperate dedicated data/RTT connections. I think you should go look at Sally's email that was posted here earlier... here it is... ************************************************************************** Franco Travostino wrote: > > A bit of Related Work ... On the forever-young 1 vs. N TCP connections > subject, you may be interested in the results shown in the '97 "An > Application-Level Solution to TCP's Satellite Inefficiencies" paper > (available at http://jarok.cs.ohiou.edu/projects/satellite/) and on Sally > Floyd's comment hereafter. The intersection with satellite links is quite > accidental, and the arguments/results apply above and beyond satellite links. > > -franco > > >From majordom@ISI.EDU Fri Dec 6 23:05:58 1996 > >To: fred@cisco.com (Fred Baker) > >Cc: end2end-tf@isi.edu > >Subject: Re: Related paper/re:satellites > >Date: Fri, 06 Dec 1996 23:05:47 PST > >From: Sally Floyd > >Sender: owner-end2end-tf@ISI.EDU > >Precedence: bulk > >Content-Length: 2309 > >X-Lines: 44 > > > >Fred - > > > >>You might be interested in reviewing this paper, which is what I'm > >>discussing with Karen Hansen, Dan Shell, and the folks from Comsat. It > >>relates to some TCP/Satellite work being done at NASA Lewis Research > >>Center. > > > >Basically (from a quick read), the paper, on "An Application-Level > >Solution to TCP's Satellite Inefficiencies", recommends breaking a > >single TCP connection into multiple connections at the > >application level, to increase throughput on satellite circuits. > >It is not too surprising that this increases the TCP throughput, but it > >is still not a good idea. For a single TCP connection, a single packet > >drop results in the throughput for that connection being cut by half, > >and then increased by roughly one packet per RTT. For a TCP connection > >that has instead been separated into N different TCP connections, a > >single packet drop results in one of the N connections, receiving > >1/N-th of the total bandwidth, having its throughput cut in half. Thus > >the bandwidth of the aggregate connection has its bandwidth reduced to > >(1 - 1/(2N))-th of its former bandwidth - that is, the larger the value > >for N, the smaller the aggregate bandwidth is cut. And then, because > >each TCP connection continues to increase its congestion window by one > >packet per RTT, for those TCP connections that have not yet reached the > >receiver's advertised window, the aggregate TCP connections together > >increase their bandwidth by up to N packets per RTT. > > > >Summarizing, splitting a TCP connection into N separate connections > >simply increases the aggressiveness of the TCP congestion control. > >Meaning that this TCP is now more likely to "steal" bandwidth from > >other TCP connections in times of congestion. And increasing the > >aggressiveness of the TCP congestion control too far (by choosing N too > >large) is counterproductive even for the aggregate TCP connection, as > >the paper shows. > > > >I would suggest that this is exactly the kind of development for which > >[RED is needed]. > > > >- Sally > > > > > > ****************************************************************** In my mind it will be a hard sell to the IESG to ask for multiple connections for high-bandwidth data? Scott/Allision any comments on this? > > The symmetric, connection allegiance case was developed when we were > smoking some really good stuff in Haifa. It really addresses a > corner case: minimizing the communication needs between NICs in systems > that will stripe requests across NICs. > > However, somewhere along the line, we forgot to optimize for the > single NIC case. > > ------------------ > > Also using a separate TCP connections may allow us to migrate one > day to an architecture where a different node in the IP network > returns the actual data from a SCSI READ! > > Client <---- iSCSI Control ----> Storage Controller <---- TBD ----> Disk array > ^ ^ > +-------------------------------iSCSI Data protocol -----------------+ > > -Costa -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 07 09:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32559; 7 Sep 2000 5:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA14045 for ips-outgoing; Thu, 7 Sep 2000 05:07:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA14041 for ; Thu, 7 Sep 2000 05:07:37 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA14444 for ; Thu, 7 Sep 2000 11:07:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id LAA95240 for ; Thu, 7 Sep 2000 11:07:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256953.0032133A ; Thu, 7 Sep 2000 11:06:57 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 7 Sep 2000 12:04:42 +0300 Subject: RE: Symmetric vs Asymmetric Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The pro-multiple connection argument was made to mainly to enable several connections. I made the argument that it may benefit a single connection too but I specified that the main reason I thought it is valuable is for parallelism in the endpoints - e.g., using SMPs. I was not arguing for a more aggressive recovery in case of congestion. And this is mainly due to poor implementation choices in current stacks - some of them bound to be fixed. Julo Black_David@emc.com on 07/09/2000 09:58:25 Please respond to Black_David@emc.com To: John Hufferd/San Jose/IBM@IBMUS, Black_David@emc.com cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: Symmetric vs Asymmetric > Since I have seen no one else that made arguments for the Symmetric > implementation, you maybe able to call the agreement on the Asymmetric > approach a consensus. While I appreciate the invitation, I'm going to hold off on doing that. The current state of the discussion doesn't feel like consensus (rough or otherwise) to me yet, and it seems to me that there have been people other than John advocating the Symmetric model. I do want to kill off a red herring that found its way into the discussion. Squeezing additional bandwidth out of a single network link (HTTP-style) has not been an important motivation for multiple connections. The aims of multiple connections have been to use parallelism in the network for failover and load balancing. A few weeks ago, there was rough consensus that use of multiple connections to squeeze additional bandwidth out of single network would not be a requirement on iSCSI, and I haven't seen anything that leads me to believe that this consensus has changed. Progress is being made -- carry on ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Sep 07 11:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00281; 7 Sep 2000 7:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA24289 for ips-outgoing; Thu, 7 Sep 2000 06:34:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from radmail.rad.co.il (radmail.rad.co.il [207.232.32.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA24282 for ; Thu, 7 Sep 2000 06:34:02 -0400 (EDT) Received: from sanrad.com ([192.114.26.219]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-54115U1200L1200S0V35) with ESMTP id il; Thu, 7 Sep 2000 13:36:41 +0300 Message-ID: <39B78B2F.C1CB925@sanrad.com> Date: Thu, 07 Sep 2000 13:33:51 +0100 From: Yaron Klein Organization: SANRAD X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: Joshua Tseng CC: Dave Nagle , ips@ece.cmu.edu Subject: Re: security model References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Joshua, We are now investigating the IPSEC and how to integrate it in the security mechanism. Our considerations are: The security scheme should be as flexible as possible. One of the options in the negotiation will be IPSEC. However, additional options will be available. The security is per session, not per TCP connection (established in the login phase). Regarding IPSEC policies, I m checking the possibility of defining a policy for iSCSI. It may be something proprietary if the standardization procedure will take to long. However, I believe that there is no escape from defining an iSCSI security. Due to the need for data integrity and the infeasibility of it in Gbit Ethernet, we must implement some digestion mechanism to insure it. This feature cannot be implemented with IPSEC. Hope to get the security and login chapter soon, so we can all discuss it. Any ideas are more than welcomed. Yaron Klein SANRAD Joshua Tseng wrote: > Yaron, > > My suggestion WRT security is to take a good look at IPSec. > IPSec has several advantages I can think of over SSL or TLS. > > 1) Standardized key distribution methodologies already exist > (ISAKMP/IKE) and have been implemented for use with IPSec. > > 2) IPSec has the advantage that it is flexible, and security > policies can be granularly applied to each iSCSI conversation, > even when multiple conversations are going over the same TCP > connection. iSCSI can thus decide to encrypt only data PDU's, > or only certain conversations, etc... The precise granularity > is implementation-specific. > > With TLS, everything sent through a TCP socket will use the > same security policy. If multiple iSCSI conversations are > taking place over the same TCP connection, then all will be > uniformly handled by the same security policy, whether it's > 3DES or no encryption at all. If this is desired, then that's > fine, but I believe there will be cases where the user wants > to only authenticate Command PDU's, while encrypting & > authenticating data PDU's, for example. > > The following is an excerpt from section 4.4.1 of RFC 2041 > (Security Architecture for IP). In this case, "application" > refers to iSCSI. > > "In host systems, applications MAY be allowed to select what security > processing is to be applied to the traffic they generate and consume. > (Means of signalling such requests to the IPsec implementation are > outside the scope of this standard.) However, the system > administrator MUST be able to specify whether or not a user or > application can override (default) system policies. Note that > application specified policies may satisfy system requirements, so > that the system may not need to do additional IPsec processing beyond > that needed to meet an application's requirements. The form of the > management interface is not specified by this document and may differ > for hosts vs. security gateways, and within hosts the interface may > differ for socket-based vs. BITS implementations. However, this > document does specify a standard set of SPD elements that all IPsec > implementations MUST support." > > -----Original Message----- > From: Yaron Klein [mailto:klein@sanrad.com] > Sent: Sunday, September 03, 2000 3:53 AM > To: Dave Nagle > Cc: ips@ece.cmu.edu > Subject: Re: security model > > We are now working on the on integrating the security model in the iSCSI > draft. > Some remarks considering your model: > > The data privacy (number 5) should be solved in a higher level, i.e., the > block > should be encrypted or digested by some hash before it is sent to the > target. > This issue is not in the scope of the transfer protocol. > > Number 2 should be expanded to initiator and target masquerading. If the > enemy > masquerades as the target, he can get all our secrets... > > Furthermore, the security considerations are affected by the performance > consideration, i.e., the computation power and the transfer rate. Thus, in > high-speed network, compromise should be made (it is not feasible or very > expensive to totally secure the data in 1G). The security model should > adjust > itself to the nature of the iSCSI features. It should enable partial > security > (only command, only headers, digestion of data etc.) > > We are now in an intensive work to integrate the security issue in the next > draft. The login chapter will enable to establish a flexible and negotiable > security/authentication model for the iSCSI session. > > Any ideas, comments and solution regarding security, let me know, > > Reagrds, > > Yaron > > Dave Nagle wrote: > > > From: Banu Ozden & Mike Reiter > > > > We suggest the attached extensions to the security model proposed in > > the iSCSI draft (Section 6). > > > > The existing iSCSI security model covers "communication security" > > between an initiator and a target. It does not address "data > > security". Data security provides protection against possible attacks > > to the data stored at the target. These include threats like > > unauthorized disclosure of data at the target to administrators or > > other clients of the target, and unauthorized modification of data at > > the target. > > > > Our main motivation is to enhance the security model for storage > > outsourcing environments where the Storage Service Provider (SSP) > > personnel is not necessarily trusted or where the sharing of target > > between different customers of the SSP raises a security concern. > > > > We are working on a security architecture for storage outsourcing. > > We would like to know whether there is interest on including data > > security considerations to iSCSI in addition to communication > > security. > > > > Banu Ozden & Mike Reiter > > > > Bell Labs > > 600 Mountain Ave. > > Murray Hill, NJ 07974 > > http://www.bell-labs.com/who/ozden > > http://www.bell-labs.com/who/reiter > > > > _______________________________________________________________ > > Threat Model > > > > T1. Disclosure of message contents to an eavesdropper intercepting > > communication between an initiator and a target. > > > > T2. An attacker masquerading as the initiator to a target or the target > > to an initiator. This includes an attacker manipulating communication > > between an initiator and a target, e.g., to introduce false messages, > > modify passing messages, or delete messages. > > > > T3. Disclosure of data to personnel maintaining the target or to other > > customers of the target. > > > > T4. The modification of data by the target or other customers of the > > target. > > > > Security Model > > > > 1. No Security (same as described in the iSCSI draft) > > > > This mode does not authenticate nor does it encrypt data. This mode > > should be used in environments where there is minimal security risk > > and little chance for configuration errors. > > > > 2. Entity Authentication (referred to as End-to-End Authentication in the > > iSCSI draft) > > > > The initiator's and/or target's identity is authenticated. > > Once the client is authenticated, all messages are > > sent and received in the clear. This mode should only be used when > > there is minimal risk to man-in-the-middle attacks, > > eavesdropping, message insertion, deletion, and modification. > > > > 3. Message Integrity (new) > > > > This mode protects against T2 types of threats. It provides > > communication integrity. > > > > 4. Message Integrity Combined with Encryption (referred to as Encryption > in > > the iSCSI draft) > > > > This mode protects against threats T1 and T2. Thus, it provides > > communication integrity and communication privacy. It protects > against > > man-in-the-middle attacks, eavesdropping, message insertion, > deletion, > > and modification. > > > > 5. Data Privacy (new) > > > > This mode protects against T3 types of threats. The initiator > > encrypts/decrypts data. The target stores encrypted data. > > > > 6. Data Privacy with Data Integrity > > > > This mode protects against threats T3 and T4. > > > > 7. Some combinations of the above security options > > > > For example, data privacy with message authentication (5 & 3) > > protects against threats T1, T2 and T3. From owner-ips@ECE.cmu.edu Thu Sep 07 12:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00570; 7 Sep 2000 8:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA25637 for ips-outgoing; Thu, 7 Sep 2000 07:46:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA25632 for ; Thu, 7 Sep 2000 07:46:44 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA06733; Thu, 7 Sep 2000 06:47:00 -0500 Message-ID: <39B78034.649B067A@stewart.chicago.il.us> Date: Thu, 07 Sep 2000 06:47:00 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Stephen Bailey CC: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps References: <10009062324.AA05378@candide.cs.uchicago.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen Bailey wrote: > > > About 1 year ago we had this "transport bundling" aka load sharing IN > > SCTP. At the request of the IESG it was removed because (if I remember > > right) it causes all sorts of problems with Congestion Control and > > is still deemed a research topic by the IETF (belonging to the > > IRTF). > > Interesting. This speaks volumes. > > I think more than anything else this goes to the heart of the `it > won't work' claim. I have claimed that connection bundling support > should either: > a) be in the transport (or a low network layer), because layers > above the transport (ULPs) do not have effective control over > network path. > or > b) be in the highest possible layer, because the low level > components can not be engineered to provide the desired behavior > without compromise, so ultimately the user must be exposed to, > aware of, and control the nature of the compromises. > > > It is aong the same lines (to a small degree) as to why you don't > > want multiple TCP connections. > > Very roughly. The specific problem is that NONE of the layers in an Yes... very very roughly :) > IP end-point have complete control over path selection. The IP layer > has the best control over this, and it only controls the initial > interface selection. There's no robust way to know that outside of > the first (and possibly last) hop, you're actually using separate > paths. If there is a shared path component, bundled connections > through different interfaces will mess with congestion control in the > same way as two connections between the same end-points. > > This doesn't mean that it's impossible to control path selection in > ALL cases. It's just that you have to carefully control the network > configuration in addition to the software behavior. > > This is an issue even in the fault tolerance application. You can not > say ``all you need to do to get fully redundant paths is hook two NIC > cards to each endpoint.'' If the intervening network is beyond your > control, you have no idea, and no way to tell what multipathing you're > actually getting. > Source AND destination based routing are still a research issue. This is what you must have to control the paths and it is NOT something we do in the IETF .. at least not yet :) So for now, if you really want to have a "fault tolerant" network, you must as you say control very carefully the setup of the routers and the NIC cards inbetween.. making sure that they are seperate and diverse.. This has been true in the telephone world for ages though... I can still remember an outage I was told about where they had two seperate power feeds to a phone company building.. but both transformers hung on the same pole... and of course a car hit that pole :-0 luckly batteries and generators were able to kick in :-) > The question is whether there should be a feature in iSCSI which > breaks if and when lower layer path selection logic changes. I am > arguing that a clean separation of concerns between iSCSI and the > lower layers on path selection will make iSCSI a more durable > protocol. I do like a clean seperation of layers for this. The concept is very nice.. > > > But the actual implemenation of the load balancing is above the > > transport layer... :0 > > This has to be because IETF doesn't have an acceptable solution, so > they're punting. They say that connection bundling messes with > congestion control, and then the proceed to hand the responsibility > over to layers that are completely ignorant of congestion. > > Pushing this function up above the transport is not likely to > meaningfully change the nature of the traffic flow at all. It will > ensure that there are numerically fewer ULPs actually using it, but > that's about it. A ULP solution still has the same potential failure > modes as that same ULP using a transport level solution. Ah, but if you think about it as to having to have been configured to make it work reasonably then allowing the ULP the hooks and knobs to do the load balancing is not so bad, tricky but not so bad... > > Nonetheless, it certainly has been argued here that iSCSI is a ULP > that needs this function. If people really think that the reality > that the feature may or may not work on a configuration by > configuration basis is OK, and an iSCSI-specific solution offers > something important anything over the wedge driver solution, hey, rock > on. > Hmm, I don't know about rock.. but with the right hooks I think it may be possible to make it work. In any event it should probably be made so that the load-balancing function is defaulted to off (if it is added) and that it SHOULD be turned on when you have done the careful network configuration thing :-) R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 07 12:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00575; 7 Sep 2000 8:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA25990 for ips-outgoing; Thu, 7 Sep 2000 08:03:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA25986 for ; Thu, 7 Sep 2000 08:03:09 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id HAA06771; Thu, 7 Sep 2000 07:03:24 -0500 Message-ID: <39B7840C.718DCEF8@stewart.chicago.il.us> Date: Thu, 07 Sep 2000 07:03:24 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: hufferd@us.ibm.com, ips@ece.cmu.edu Subject: Re: Symmetric vs Asymmetric References: <0F31E5C394DAD311B60C00E029101A0704100F5A@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com wrote: > > > Since I have seen no one else that made arguments for the Symmetric > > implementation, you maybe able to call the agreement on the Asymmetric > > approach a consensus. > > While I appreciate the invitation, I'm going to hold off on doing that. > The current state of the discussion doesn't feel like consensus (rough > or otherwise) to me yet, and it seems to me that there have been > people other than John advocating the Symmetric model. > > I do want to kill off a red herring that found its way into the discussion. > Squeezing additional bandwidth out of a single network link (HTTP-style) > has not been an important motivation for multiple connections. I guess I in part introduced this red herring... :0 sorry about that! But I am very glad to hear it is dead!! It kill one of my big concerns :) > The aims > of multiple connections have been to use parallelism in the network for > failover and load balancing. A few weeks ago, there was rough consensus You can easily get the failover from SCTP. It will automatically retry on alternate addresses if they exist... on the first retran. So SCTP could be used to get: A) CC B) Fail-over from one NIC to another on retransmissions C) Limited retransmission (i.e. only retransmit 0 or 1 times, for those custom configured networks where loss should not happen... doing 1 retran looks good here since it would go on an alternate NIC) D) Un-ordered delivery C) Multiple conversations seperated in the Streams without blocking between the streams. It can NOT help you with 1) Load balancing R > that use of multiple connections to squeeze additional bandwidth out of > single network would not be a requirement on iSCSI, and I haven't seen > anything that leads me to believe that this consensus has changed. > > Progress is being made -- carry on ... > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 07 14:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00998; 7 Sep 2000 10:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA28074 for ips-outgoing; Thu, 7 Sep 2000 09:13:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA28070 for ; Thu, 7 Sep 2000 09:13:31 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA16970 for ; Thu, 7 Sep 2000 15:12:55 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id PAA79482 for ; Thu, 7 Sep 2000 15:12:54 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256953.004894C6 ; Thu, 7 Sep 2000 15:12:46 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 7 Sep 2000 16:10:32 +0300 Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, Although the windowing mechanism in iSCSI-01 may seem to be there to solve a queueing issue it is mainly meant to limit the buffering space for commands that await "de-skewing". We assume that execution queue-lengths, policy etc. are beyond the scope of transport. As for SCSI queue length I assumed that the busy or queue full status followed by an Asynch Event message indicating readiness is the mechanism provided by SCSI to regulate the command flow. It is hard to imagine that give the variable life-time of SCSI commands and the opaque nature of the resources required to execute them that the transport has to help in this area. Julo Jim McGrath on 07/09/2000 06:06:40 Please respond to Jim McGrath To: "'Matt Wakeley'" , ips cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Command Queue Depth (was asymmetric/Symmetric) The issue of buffer space allocation for multiple initiators has a long and troubled history in SCSI. We have never been able to come up with a good answer. Fibre Channel tried to fix this with the notion of "login BB credit" - when you login you get a minimum number of credits you are always guaranteed when you start data transfers. The problem with this is that storage devices had no realistic ability to discriminate between initiators or to change the login BB credit. In addition, the expectation is that all possible initiators would get these credits on login. So storage devices vendors have played it safe and kept this number low (at 0 until recently, now around 2). For iSCSI the number of initial credits you need to "prime the pump" until normal data flow is established is probably large (given the latencies are higher than in Fibre Channel, especially FC-AL), and the number of potential initiators larger than in Fibre Channel, making this a whole lot worse for the storage device. As soon as we allow the devices to start adjusting these credits, then you have the protocol problem of making sure people know when their credits are adjusted and the policy problem of how, who, and when to adjust the credits. Changing everyone's credit when you add a new initiator can get into a notification nightmare, although it is "fair." Any policy brings up all sorts of nasty issues regarding fairness vs efficient use of the transmission media. Jim Note: the same problem has plagued other attempts to allocate device resources between multiple initiators, like command queue space. In general policies with respect to multiple initiators are not really standard in the SCSI world. -----Original Message----- From: Matt Wakeley [mailto:matt_wakeley@agilent.com] Sent: Wednesday, September 06, 2000 2:56 PM To: ips Subject: Re: Command Queue Depth (was asymmetric/Symmetric) Joshua Tseng wrote: > James, > > I agree with others that there may be an issue with the > command windowing mechanism in the existing iSCSI spec. It is like > "TCP in reverse", in that the target determines the size of the window, and > not the initiator as in TCP. Rather, I believe that everything that this > windowing mechanism is attempting to achieve can be more easily obtained > by having the target communicate its buffer size to the initiator at > iSCSI login. It should be the role of the initiator to determine how > many commands to put in flight simultaneously, given this input on available > buffer size from the target. As more initiators connect to a target, it may need to scale back the amount of this buffering it has allocated to each previously logged in initiator (to prevent rejecting new logins). > > > As far as multiple initiators, could this not be resolved by the target > refusing additional logins beyond the number of initiators it can safely > support? Not being a storage expert, this is my best guess/suggestion > at how to do it. I believe John already answered this... -Matt From owner-ips@ECE.cmu.edu Thu Sep 07 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02110; 7 Sep 2000 14:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09639 for ips-outgoing; Thu, 7 Sep 2000 13:22:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07665 for ; Thu, 7 Sep 2000 12:34:35 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Thu, 7 Sep 2000 09:35:40 -0700 Message-ID: From: Robert Snively To: somesh_gupta@hp.com, ips@ece.cmu.edu, matt_wakeley@agilent.com Subject: RE: Re: multiple connections Date: Thu, 7 Sep 2000 09:32:59 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: somesh_gupta@hp.com [mailto:somesh_gupta@hp.com] > Sent: Wednesday, September 06, 2000 5:12 PM > To: ips@ece.cmu.edu; matt_wakeley@agilent.com > Subject: RE: Re: multiple connections > snip > > 2. WRITES - This is the really bad one in my opinion. For > me, avoiding > RTTs in iSCSI would just by itself make iSCSI a superior "transport" > for SCSI. So assuming RTTs are not being used, the host > would (changing > the posting order from READs), first post the write command to the > connection on which the data is to be sent, and then post the write > buffers to the connection which is to be used for sending data. > One case is where for whatever reason, the target gets the > data before > it gets the command, and has no clue what to do with the data. > Let us assume that the target does get the command before it gets the > data. The target gets a command indicating that data being > written to > whatever lun and whatever location is going to arrive on some other > connection. First the target has taken an extra event from > the adapter. > Then, when data arrives on another NIC (and the target gets another > event), the target goes through the list of outstanding WRITE > commands to match the command with the data and then go about the > business of processing the data. > > So in this case the work has increased for the initiator as well as > the target. And to top it off, the target is active with a large number of initiators on behalf of a large number of logical units for a large number of queued commands, so there is absolutely no guarantee that any buffer exists for the data that was received, even if the command had been received first. Every SCSI command execution is managed by the target for this reason among many others. Thus RTT has been a part of every SCSI protocol for write operations. From owner-ips@ECE.cmu.edu Thu Sep 07 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02112; 7 Sep 2000 14:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09651 for ips-outgoing; Thu, 7 Sep 2000 13:23:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07669 for ; Thu, 7 Sep 2000 12:34:38 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Thu, 7 Sep 2000 09:35:43 -0700 Message-ID: From: Robert Snively To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Thu, 7 Sep 2000 09:33:03 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, September 07, 2000 6:11 AM > To: ips@ece.cmu.edu > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > snip > > As for SCSI queue length I assumed that the busy or queue full status > followed by an Asynch Event > message indicating readiness is the mechanism provided by > SCSI to regulate > the command flow. Actually, the approach was somewhat different, and is still only a recommendation and not a requirement. Every logical unit (not every target) was required to reserve at least one command queuing buffer for each logged-in initiator so that each logical unit could always accept at least one command. Then the queue was stuffed full until TASK SET FULL status was presented. At that time, as each initiator encountered TASK SET FULL, it would back off until a previously queued command was completed. At that time (serving the purpose of the asynchronous event notification hypothesized above), the queue could then be restuffed. This back-off provided maximum queuing for maximum throughput and allowed any particular initiator to access all available resources at the time they were needed. Given the bursty (self-similar statistical) nature of storage accesses, this appeared to be about the best we could do. If a system ran into continuing problems, additional resources (more devices, more initiators, more links) were the only reasonable solution to maintain the desired level of performance. The appropriate text from SAM-2 is provided here for your ASCII reference. Clause 5.2 Status TASK SET FULL. This status shall be implemented if the logical unit supports the creation of tagged tasks (see 4.9). This status shall not be implemented if the logical unit does not support the creation of tagged tasks. When the logical unit has at least one task in the task set for an initiator and a lack of task set resources prevents entering a newly received tagged task from that initiator in the task set, TASK SET FULL shall be returned. When the logical unit has no task in the task set for an initiator and a lack of task set resources prevents entering a newly received tagged task from that initiator in the task set, BUSY should be returned. When the logical unit has at least one task in the task set and a lack of task set resources prevents entering a newly received untagged task in the task, BUSY should be returned. The logical unit should allow at least one queued command for each supported initiator that has identified itself to the target by a protocol specific procedure or by the successful transmission of a command. From owner-ips@ECE.cmu.edu Thu Sep 07 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02859; 7 Sep 2000 17:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA18425 for ips-outgoing; Thu, 7 Sep 2000 16:05:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA18406 for ; Thu, 7 Sep 2000 16:05:41 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel1.hp.com (Postfix) with ESMTP id EA456143C; Thu, 7 Sep 2000 13:05:39 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id NAA12012; Thu, 7 Sep 2000 13:05:39 -0700 (PDT) X-OpenMail-Hops: 1 Date: Thu, 7 Sep 2000 10:53:26 -0700 Message-Id: Subject: RE: RE: Re: multiple connections MIME-Version: 1.0 To: somesh_gupta@hp.com, ips@ece.cmu.edu, rsnively@Brocade.COM, matt_wakeley@agilent.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: rsnively@Brocade.COM [mailto:rsnively@Brocade.COM] > Sent: Thursday, September 07, 2000 9:33 AM > To: somesh_gupta@hp.com; ips@ece.cmu.edu; matt_wakeley@agilent.com > Subject: FW: RE: Re: multiple connections > > > > > > -----Original Message----- > > From: somesh_gupta@hp.com [mailto:somesh_gupta@hp.com] > > Sent: Wednesday, September 06, 2000 5:12 PM > > To: ips@ece.cmu.edu; matt_wakeley@agilent.com > > Subject: RE: Re: multiple connections > > > snip > > > > 2. WRITES - This is the really bad one in my opinion. For > > me, avoiding > > RTTs in iSCSI would just by itself make iSCSI a superior > "transport" > > for SCSI. So assuming RTTs are not being used, the host > > would (changing > > the posting order from READs), first post the write command to the > > connection on which the data is to be sent, and then post the write > > buffers to the connection which is to be used for sending data. > > One case is where for whatever reason, the target gets the > > data before > > it gets the command, and has no clue what to do with the data. > > Let us assume that the target does get the command before > it gets the > > data. The target gets a command indicating that data being > > written to > > whatever lun and whatever location is going to arrive on some other > > connection. First the target has taken an extra event from > > the adapter. > > Then, when data arrives on another NIC (and the target gets another > > event), the target goes through the list of outstanding WRITE > > commands to match the command with the data and then go about the > > business of processing the data. > > > > So in this case the work has increased for the initiator as well as > > the target. > > > And to top it off, the target is active with a large number of > initiators on behalf of a large number of logical units for a large > number of queued commands, so there is absolutely no guarantee that > any buffer exists for the data that was received, even if the > command had been received first. > > Every SCSI command execution is managed by the target for > this reason among > many others. Thus RTT has been a part of every SCSI protocol > for write > operations. > TCP window size should provide a reasonable flow control mechanism, so an additional flow control should not be needed. There will be some statistical determination of the amount of memory needed in a large array vs the window size extended on each connection. I don't know if arrays like to keep commands in seperate memory from data, in which case a command queue depth may have to be communicated seperately (assuming most of window would typically be used for data) From owner-ips@ECE.cmu.edu Thu Sep 07 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02967; 7 Sep 2000 17:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA20556 for ips-outgoing; Thu, 7 Sep 2000 16:52:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA20545 for ; Thu, 7 Sep 2000 16:52:13 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Thu, 7 Sep 2000 13:52:23 -0700 Message-ID: From: Charles Monia To: "Julian Satran (E-mail)" Cc: "Ips (E-mail)" Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Thu, 7 Sep 2000 13:52:22 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julo: > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, September 07, 2000 6:11 AM > To: ips@ece.cmu.edu > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > Dear colleagues, > > Although the windowing mechanism in iSCSI-01 may seem to be > there to solve > a queueing issue > it is mainly meant to limit the buffering space for commands > that await > "de-skewing". > We assume that execution queue-lengths, policy etc. are > beyond the scope of > transport. > > As for SCSI queue length I assumed that the busy or queue full status > followed by an Asynch Event > message indicating readiness is the mechanism provided by > SCSI to regulate > the command flow. > > It is hard to imagine that give the variable life-time of > SCSI commands and > the > opaque nature of the resources required to execute them that > the transport > has > to help in this area. > While this issue has been discussed at some length in the past, as Jim McGrath stated, I believe the debate ought to be reopened (although we may end up reaching the same conclusion as before). As Ralph Weber pointed out, the SCSI model is to discard the command, return status, retrieve the next command in the transport pipeline and continue processing. The initiator is not notified when processing resumes. If there are many commands in flight, as there could be in an IP environment, and target resources free up in the meantime, the result is commands processed out of order. Historically, such a lapse in command ordering was not seen as an issue for the following reasons: a) Strict ordering was not required by the most commonly deployed device types (disks and tapes). Due to the nature of disk traffic, a simple retry mechanism was deemed sufficient to recover from these errors. Since legacy streaming devices, such as tapes, did not support command queuing, command ordering considerations were not a factor there either. b) Transport delays over storage interconnects were small, so not many commands were apt to be in flight. i.e.. The window for such errors was very small. c) The resource guarantees needed for a loss-avoidance mechanism in the target adversely effected device cost, especially at the high-volume, low-end of the market. Given the above considerations, there was little support within the storage community for measures addressing this issue. If we now believe that the iSCSI environment changes the rules, I believe the interconnect protocol can provide useful assists, such as: a) On a command overflow condition, have the iSCSI target flush the command pipeline by returning status and discarding all subsequently received commands until a host acknowledgement is received. b) Implement some sort of credit-based mechanism for overflow-avoidance. Comments? > Jim McGrath on 07/09/2000 06:06:40 > > Please respond to Jim McGrath > > To: "'Matt Wakeley'" , ips > > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > > > The issue of buffer space allocation for multiple initiators > has a long and > troubled history in SCSI. We have never been able to come up > with a good > answer. > > Fibre Channel tried to fix this with the notion of "login BB > credit" - when > you login you get a minimum number of credits you are always > guaranteed > when > you start data transfers. The problem with this is that > storage devices > had > no realistic ability to discriminate between initiators or to > change the > login BB credit. In addition, the expectation is that all possible > initiators would get these credits on login. So storage > devices vendors > have played it safe and kept this number low (at 0 until recently, now > around 2). For iSCSI the number of initial credits you need > to "prime the > pump" until normal data flow is established is probably large > (given the > latencies are higher than in Fibre Channel, especially FC-AL), and the > number of potential initiators larger than in Fibre Channel, > making this a > whole lot worse for the storage device. > > As soon as we allow the devices to start adjusting these > credits, then you > have the protocol problem of making sure people know when > their credits are > adjusted and the policy problem of how, who, and when to adjust the > credits. > Changing everyone's credit when you add a new initiator can get into a > notification nightmare, although it is "fair." Any policy > brings up all > sorts of nasty issues regarding fairness vs efficient use of the > transmission media. > > Jim > > Note: the same problem has plagued other attempts to allocate device > resources between multiple initiators, like command queue space. In > general > policies with respect to multiple initiators are not really > standard in the > SCSI world. > > > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 2:56 PM > To: ips > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > Joshua Tseng wrote: > > > James, > > > > I agree with others that there may be an issue with the > > command windowing mechanism in the existing iSCSI spec. It is like > > "TCP in reverse", in that the target determines the size of > the window, > and > > not the initiator as in TCP. Rather, I believe that > everything that this > > windowing mechanism is attempting to achieve can be more > easily obtained > > by having the target communicate its buffer size to the initiator at > > iSCSI login. It should be the role of the initiator to > determine how > > many commands to put in flight simultaneously, given this input on > available > > buffer size from the target. > > As more initiators connect to a target, it may need to scale back the > amount > of > this buffering it has allocated to each previously logged in > initiator (to > prevent rejecting new logins). > > > > > > > As far as multiple initiators, could this not be resolved > by the target > > refusing additional logins beyond the number of initiators > it can safely > > support? Not being a storage expert, this is my best > guess/suggestion > > at how to do it. > > I believe John already answered this... > > -Matt > > > Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Thu Sep 07 23:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03421; 7 Sep 2000 19:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23909 for ips-outgoing; Thu, 7 Sep 2000 18:18:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23903 for ; Thu, 7 Sep 2000 18:17:56 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e87MHtx24693 for ; Thu, 7 Sep 2000 17:17:55 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA06279; Thu, 7 Sep 00 17:16:09 -0500 Message-Id: <10009072216.AA06279@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps In-Reply-To: Message from Jim McGrath of "Wed, 06 Sep 2000 20:13:55 PDT." Date: Thu, 07 Sep 2000 17:15:45 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > In practice a number of interesting applications (like a machine room or > limited LAN) can control (or easily know) their configurations, and so might > be able to use a feature that was configuration dependent (SCSI, Fibre > Channel, and Ethernet all have some configuration constraints today for > certain things). Agreed. That's why I said: > This doesn't mean that it's impossible to control path selection in > ALL cases. It's just that you have to carefully control the network > configuration in addition to the software behavior. The question is, who (which layer entity) should be signed up to take the fall for it working or not. The use of wedge drivers usually already implies a careful level of configuration control. On the other hand I believe iSCSI should be a really generic component and not sign up for this type of thing. Steph (the horse seems dead) From owner-ips@ECE.cmu.edu Thu Sep 07 23:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03430; 7 Sep 2000 19:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA24848 for ips-outgoing; Thu, 7 Sep 2000 18:46:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA24833 for ; Thu, 7 Sep 2000 18:46:27 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id 9E1A4291 for ; Thu, 7 Sep 2000 15:46:26 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id PAA29092 for ; Thu, 7 Sep 2000 15:46:25 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA5A03 for ; Thu, 7 Sep 2000 15:46:22 -0700 Message-ID: <39B81ADD.D1978A70@agilent.com> Date: Thu, 07 Sep 2000 15:46:53 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: <200009061523.LAA00697@newdev.harvard.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk As Julian has stated in a different thread, the purpose of the "sliding windows" in iSCSI is not for congestion management. It is simply there to handle the case where if a connection goes down in a multiple connection session, it prevents the remaining connections from overwhelming the target with new commands that it can't process due to missing commands that where on the broken connection. Since all of this runs on top of TCP, and TCP performs congestion management, why must iSCSI perform congestion management on top of TCP? -Matt Wakeley Agilent Technologies Scott Bradner wrote: > > Implementing sliding windows is not that hard > > note that the issue is not "just" sliding windows - ips also has to deal > with congestion TCP-friendly way - that can get quite complicated > > Scott From owner-ips@ECE.cmu.edu Thu Sep 07 23:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03417; 7 Sep 2000 19:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA24183 for ips-outgoing; Thu, 7 Sep 2000 18:27:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA24179 for ; Thu, 7 Sep 2000 18:27:22 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e87MRMx24839 for ; Thu, 7 Sep 2000 17:27:22 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA06283; Thu, 7 Sep 00 17:25:36 -0500 Message-Id: <10009072225.AA06283@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI Autosense Consensus, Connection next steps In-Reply-To: Message from "Randall R. Stewart" of "Thu, 07 Sep 2000 06:47:00 CDT." <39B78034.649B067A@stewart.chicago.il.us> Date: Thu, 07 Sep 2000 17:25:12 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Ah, but if you think about it as to having to have been configured to > make it work reasonably then allowing the ULP the hooks and knobs to > do the load balancing is not so bad, tricky but not so bad... The definition of ULP is relative. With respect to the transport, the iSCSI driver, and the entire SCSI stack are the upper layer. In this case, a wedge driver is the ULP that should (and does) have the hooks and knobs for this. Steph From owner-ips@ECE.cmu.edu Thu Sep 07 23:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03419; 7 Sep 2000 19:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA24670 for ips-outgoing; Thu, 7 Sep 2000 18:41:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA24660 for ; Thu, 7 Sep 2000 18:40:56 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e87Meux25040 for ; Thu, 7 Sep 2000 17:40:56 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA06307; Thu, 7 Sep 00 17:39:10 -0500 Message-Id: <10009072239.AA06307@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Command Queue Depth (was asymmetric/Symmetric) In-Reply-To: Message from Charles Monia of "Thu, 07 Sep 2000 13:52:22 PDT." Date: Thu, 07 Sep 2000 17:38:46 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > b) Implement some sort of credit-based mechanism for overflow-avoidance. ST did this with a technique called `slot accounting'. Anything that could be unprocessed in work queue consumes a slot, and there are various ways to recover the credits. You could also turn slot accounting off. I didn't really groove to the idea, because historically, storage devices frequently operate in an oversubscribed mode (advertised queue depth is on a per LUN basis and doesn't account for number of initiators), and I though the slot accounting mechanism was too rigid to accomodate this accepted property. However, as you point out, the SCSI handling of heavy oversubscription may not be quite right, so perhaps a better solution in iSCSI is called for. Certainly the connectivity implied by iSCSI will only make the problem worse. Steph From owner-ips@ECE.cmu.edu Thu Sep 07 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03513; 7 Sep 2000 19:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA25546 for ips-outgoing; Thu, 7 Sep 2000 19:08:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA25541 for ; Thu, 7 Sep 2000 19:08:01 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id AF64C61B for ; Thu, 7 Sep 2000 19:07:59 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id QAA29627 for ; Thu, 7 Sep 2000 16:07:56 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA5A97 for ; Thu, 7 Sep 2000 16:07:53 -0700 Message-ID: <39B81FE7.7B017DF1@agilent.com> Date: Thu, 07 Sep 2000 16:08:23 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk somesh_gupta@hp.com wrote: > I think Matt has an important point here. Let us consider READs > and WRITEs seperately. > > 1. READ - In this case the host software can post the appropriate > read buffers to the connection on which the data is to be received > and then post the read command to the connection on which the data > is to be sent. By the time the target gets the command, and returns > data, it is most likely that the NIC receiving the data will have > accepted the read buffers. You say "most likely", but are you willing to implement and test code to handle the occasional times when the NIC receiving the data was delayed (perhaps it couldn't win access to the local system bus), and the data comes in with no place to put it? (if the data to be read was in array cache memory, it could come back to you very fast.) > > Although synchronization issues are minimized, there is a > reasonable amount of performance impact. Two different "commands" > (in the sense of host to adapter commands - not in the SCSI sense) > have to be posted. Two different queues have to be written to (for > sure if they are on different NICs) and two different NICs triggered > to accept the "commands". Then there will be two different completion > events that will be returned by the NICs to the host - one indicating > the acceptance/"completion"/"return" of the command buffer (indicating > command sent successfully), and the second when the data is all > received (along with the SCSI command status). This may also require > interrupt handling etc which could be minimized by some clever schemes. If you want to avoid the "occasional" race condition I described above, you will not be able to send the SCSI command until you are sure the NIC receiving the data is set up to receive it. I contend that for WRITES, the same applies. A command could be sent, and the target could respond with a RTT to the NIC to transmit the data before the NIC has retrieved state to know that the RTT is coming. > So the amount of work done by the host for the host/adapter interaction > is close to doubled. There is probably not that much impact on the > target in this case. I agree, probably not that much impact on the target. > 2. WRITES - This is the really bad one in my opinion. For me, avoiding > RTTs in iSCSI would just by itself make iSCSI a superior "transport" > for SCSI. So assuming RTTs are not being used, the host would (changing > the posting order from READs), first post the write command to the > connection on which the data is to be sent, and then post the write > buffers to the connection which is to be used for sending data. > One case is where for whatever reason, the target gets the data before > it gets the command, and has no clue what to do with the data. > Let us assume that the target does get the command before it gets the > data. The target gets a command indicating that data being written to > whatever lun and whatever location is going to arrive on some other > connection. First the target has taken an extra event from the adapter. > Then, when data arrives on another NIC (and the target gets another > event), the target goes through the list of outstanding WRITE > commands to match the command with the data and then go about the > business of processing the data. Well, I'm not convinced that targets/arrays will be able to implement a reliable write scheme that does not use RTT. But I'll step aside to let them raise any issues with data coming down a different path than the command. -Matt Wakeley Agilent Technologies > > > So in this case the work has increased for the initiator as well as > the target. > > Somesh From owner-ips@ECE.cmu.edu Thu Sep 07 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03511; 7 Sep 2000 19:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA25112 for ips-outgoing; Thu, 7 Sep 2000 18:55:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA25100 for ; Thu, 7 Sep 2000 18:55:23 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id EB93AD36 for ; Thu, 7 Sep 2000 18:55:20 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id PAA29303 for ; Thu, 7 Sep 2000 15:55:19 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA5A2D for ; Thu, 7 Sep 2000 15:55:17 -0700 Message-ID: <39B81CF3.9EF6D72C@agilent.com> Date: Thu, 07 Sep 2000 15:55:47 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk somesh_gupta@hp.com wrote: > > -----Original Message----- > > From: rsnively@Brocade.COM [mailto:rsnively@Brocade.COM] > > And to top it off, the target is active with a large number of > > initiators on behalf of a large number of logical units for a large > > number of queued commands, so there is absolutely no guarantee that > > any buffer exists for the data that was received, even if the > > command had been received first. > > > > Every SCSI command execution is managed by the target for > > this reason among > > many others. Thus RTT has been a part of every SCSI protocol > > for write > > operations. > > > > TCP window size should provide a reasonable flow control mechanism, > so an additional flow control should not be needed. There will be > some statistical determination of the amount of memory needed in > a large array vs the window size extended on each connection. > > I don't know if arrays like to keep commands in seperate memory > from data, in which case a command queue depth may have to be > communicated seperately (assuming most of window would typically > be used for data) Arrays very much do like to keep commands in separate memory from data. In fact, they like the data for a particular I/O to go to a particular place in memory. And they don't want that data arriving before they have this particular place set up and ready for it. That's why (most) arrays implement the RTT. It is not for flow control, it is for array memory management. (the array vendors can flame me for being way off base) -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Fri Sep 08 00:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03777; 7 Sep 2000 20:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA26359 for ips-outgoing; Thu, 7 Sep 2000 19:38:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA26355 for ; Thu, 7 Sep 2000 19:38:20 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA84656 for ; Thu, 7 Sep 2000 19:36:31 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id RAA57546 for ; Thu, 7 Sep 2000 17:38:19 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: a vote for asymmetric connections in a session To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Thu, 7 Sep 2000 16:33:40 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/07/2000 05:38:18 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk If, at the approprate time, David were to call consensus in favor of the Asymmetric case, I think this whole issue of the Sliding Window would go away (at least with regard to its original purpose). Do you folks agree with that statement? . . . John L. Hufferd "Matt Wakeley" @ece.cmu.edu on 09/07/2000 03:46:53 PM Please respond to Matt Wakeley Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: a vote for asymmetric connections in a session As Julian has stated in a different thread, the purpose of the "sliding windows" in iSCSI is not for congestion management. It is simply there to handle the case where if a connection goes down in a multiple connection session, it prevents the remaining connections from overwhelming the target with new commands that it can't process due to missing commands that where on the broken connection. Since all of this runs on top of TCP, and TCP performs congestion management, why must iSCSI perform congestion management on top of TCP? -Matt Wakeley Agilent Technologies Scott Bradner wrote: > > Implementing sliding windows is not that hard > > note that the issue is not "just" sliding windows - ips also has to deal > with congestion TCP-friendly way - that can get quite complicated > > Scott From owner-ips@ECE.cmu.edu Fri Sep 08 00:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03761; 7 Sep 2000 20:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA27183 for ips-outgoing; Thu, 7 Sep 2000 20:03:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA27167 for ; Thu, 7 Sep 2000 20:03:04 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Thu, 7 Sep 2000 17:03:18 -0700 Message-ID: From: Charles Monia To: Matt Wakeley , ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Thu, 7 Sep 2000 17:03:14 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi All: > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Thursday, September 07, 2000 3:47 PM > To: ips@ece.cmu.edu > Subject: Re: a vote for asymmetric connections in a session > > > As Julian has stated in a different thread, the purpose of > the "sliding > windows" in iSCSI is not for congestion management. It is > simply there to > handle the case where if a connection goes down in a multiple > connection > session, it prevents the remaining connections from > overwhelming the target > with new commands that it can't process due to missing > commands that where on > the broken connection. > > Since all of this runs on top of TCP, and TCP performs > congestion management, > why must iSCSI perform congestion management on top of TCP? > > -Matt Wakeley > Agilent Technologies > I'm not sure what is meant by "congestion." If we're talking about congestion in the TCP/IP transport, I'm in agreement. However, I thought we were referring to the sort of congestion that the application on top of iSCSI might see if it received more commands than it had room for. Unless I misunderstood your point, threfore, I think there might be an issue. The only way I can see flow control in the tranport layer being used to avoid dropping commands is if higher layer congestion results in back pressure to the iSCSI pipe. I believe that behavior is undesirable because it introduces "head-of-line" blocking, with the following consequences: a) It effectively shuts down the flow of commands to all logical units. b) It blocks the flow of task management commands (Abort Task, Clear task set, etc). Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Fri Sep 08 00:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03772; 7 Sep 2000 20:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA26883 for ips-outgoing; Thu, 7 Sep 2000 19:55:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA26874 for ; Thu, 7 Sep 2000 19:55:42 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id F3392711 for ; Thu, 7 Sep 2000 19:55:40 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id QAA00676 for ; Thu, 7 Sep 2000 16:55:36 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA5BE0; Thu, 7 Sep 2000 16:55:31 -0700 Message-ID: <39B82B11.3BB73B45@agilent.com> Date: Thu, 07 Sep 2000 16:56:01 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: John Hufferd/San Jose/IBM , ips Subject: Re: Symmetric vs Asymmetric References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk John Hufferd/San Jose/IBM wrote: > David, > One of the reasons I was holding out for Symmetric over Asymmetric, was for > the workload balancing etc. that could be done at the iSCSI layer. > > I am now willing to "roll-over" but I think I owe you all my reasons for > deciding to support the Asymmetric approach: > > First, in the degenerate case, they both look the same. That is, they can > both support a single Connection per Session. So the simplest > implementation is still possible with the Asymmetric approach as it was > with the Symmetric approach. This means that most current iSCSI > implementations can continue to exist, unchanged. The biggest issue with the "asymmetric" model is that it is NOT "asymmetric" when there is only 1 TCP connection. When there is only one TCP connection, it is the "symmetric" model - both commands and data on the same TCP connection. Then, when there is more than one TCP connection, the behavior is different. It's always easier to implement something that operates that same way all the time, than two different behaviors. I propose that the asymmetric model mandate at least two TCP connections (implies that at least one physical connection will have at least two TCP connections running on it) - one for commands, the other for data. This has other advantages, like commands not being flow controlled by large transfers of data. -Matt Wakeley Agilent Technologies > > > But the most important reason follows: > > Since a session is a unique construct that identifies an Initiator and a > Target, I am assuming that does NOT mean each NICs has a unique initiator > ID but that they can, and should, share a common Initiator ID. The Target > ID, on the other hand, is either unique per Storage Controller, or not, > and this will depend on the type of Storage Controller. This means that > the Host iSCSI device driver will be able to present, to the layers above > it, either a single port per storage controller, or multiple ports, > depending on how the Storage Controller answers Logins on each of its IP > connections. > > That means iSCSI will not be able to determine when it knows enough to > perform alternate path retry, and for the same reason will not be able to > know when or how to perform command work load balancing across the > different sessions. This is because either all connections to a storage > controller have either the same iSCSI session ID (SSID), or they are > different -- but iSCSI will not, by itself, know which sessions can be > substituted for others (for balancing or retry). > > The only place where the symmetric approach can do workload balancing and > alternate path retry is where the storage controller makes it look like a > single target ID even though it might have multiple IP addresses. Though > this might work in some IBM storage controllers (like Shark) it will not > work for all IBM storage controllers (like those of Mylex). This same or > similar argument probably will apply to EMC's and other vendors various > storage controllers. Therefore, having iSCSI do any type of command > balancing or command automatic alternate path retry is problematical > without outside influences (probably from upper layers) and this will tend > to make the iSCSI layer very thick and complicated. It will also mix the > layering concepts (in general not a good idea). Therefore, I now believe > that the Symmetrical approach can not be a reasonable general/generic > approach to Command work load balance or alternate path retry, and that > function is best left to the Wedge Drivers. > > The Asymmetric approach can not perform command balancing or command > automatic path retry either, but it does not try. It only tries to balance > the data across multiple connections (and that seems much more reasonable). > > With the Asymmetric approach, data for any specific command only flows on > one connection. So you can look at the data from various commands as > independent flows on the different connections (NICs) so there should not > be blocking issues, or parallel flow issues, and RDMA type direct to memory > functions can occur. > > With multiple storage controllers, there may be many sessions and many data > connections spread across all the various NICs in a Host. > > The net of all this is that the Asymmetric approach: > Probably has good enough spread of commands and data across multiple > NICs, even without Wedge Drivers. > Permits the upper layer Wedge Drivers to be left mostly unchanged. > Permits the simple case with a single connection per session, even with > multiple NICs. > Permits the most simple case of a single connection per session with > only a single NIC. > > Since I have seen no one else that made arguments for the Symmetric > implementation, you maybe able to call the agreement on the Asymmetric > approach a consensus. > . > . > . > John L. Hufferd From owner-ips@ECE.cmu.edu Fri Sep 08 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03790; 7 Sep 2000 20:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA26238 for ips-outgoing; Thu, 7 Sep 2000 19:33:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA26230 for ; Thu, 7 Sep 2000 19:33:08 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA78302 for ; Thu, 7 Sep 2000 19:27:10 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id RAA41090 for ; Thu, 7 Sep 2000 17:33:04 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: a vote for asymmetric connections in a session To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Thu, 7 Sep 2000 16:29:12 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/07/2000 05:33:04 PM MIME-Version: 1.0 Content-type: multipart/alternative; Boundary="0__=88256953008104458f9e8a93df938690918c8825695300810445"; X-Lotus-Encap=encap3 Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=88256953008104458f9e8a93df938690918c8825695300810445 Content-type: text/plain; charset=us-ascii Folks, Matt and Julian are correct. The sliding window was put in for a completely different reason then is currently being discussed. I suggest that we all look at it only as it was originally intended and then see if something else needs to be done, or not. TCP/IP probably handles all the other issues (well enough). . . . John L. Hufferd "Matt Wakeley" @ece.cmu.edu on 09/07/2000 03:46:53 PM Please respond to Matt Wakeley Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: a vote for asymmetric connections in a session As Julian has stated in a different thread, the purpose of the "sliding windows" in iSCSI is not for congestion management. It is simply there to handle the case where if a connection goes down in a multiple connection session, it prevents the remaining connections from overwhelming the target with new commands that it can't process due to missing commands that where on the broken connection. Since all of this runs on top of TCP, and TCP performs congestion management, why must iSCSI perform congestion management on top of TCP? -Matt Wakeley Agilent Technologies Scott Bradner wrote: > > Implementing sliding windows is not that hard > > note that the issue is not "just" sliding windows - ips also has to deal > with congestion TCP-friendly way - that can get quite complicated > > Scott --0__=88256953008104458f9e8a93df938690918c8825695300810445 Content-type: application/X-Lotus-Encap2; name="encap2.ond" Content-transfer-encoding: base64 GgAAAwAAEQABAApfgQBTaSWHAAAAvgAADgEAAPsHAAAAABVfgQBTaSWHCv8BAgAIAAAAZgEAoLMA CAAAAQAAugEA+iAAAP4gAAD8BwAgAEAAAkAAAEAAAAAAAgAAAAAAAAAAAAQAAAADAAAAAwAAAA0A AAABAAAAAAAAAAAAAAAAAwAAAAQAABNfgQBTaSWHAAAAAAAAAAAAAAAAAGABAAAAAAAAAAAAAAAB gAAAAAAAAAAACl+BAFNpJYcAAAAAAAAAAAAAAABFbmNhcHN1bGF0ZWQgTm90ZSAvbm90ZXMvQzg3 NTYxMDAuRFRGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAAAAAAGAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATX4EAU2klhwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAEAAQABAAAAAAAAAAAAAPogAACqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqoAvAAA AABoCgAAaAqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq qqqqqqqqqqqqqqqqqqqqqqqqqqoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACA AAAAAAAAAAAAAAAAAAAAAADgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADA//////////////////////////// //////////8AAAAA8AAAAAAAAAAAAAAAAAAAAAD///////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// /////////////////////////wYKAAAAAAYBAAAA3gAAAAAAAP////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// ////BwDmAAAAAQD//woAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAA 3voPAAAgACABAAAAAQAAAAQADAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAPu7u4AAAAAAAAAAAMAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAABAAAAAwAAAAA4AAAAAABAAAgAQAAQAEAu7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7uwEAAAAcAAoBAAAAAAAAAAAAAENOPUQwM05N MDk0L09VPTAzL09VPU0vTz1JQk27u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7sAAAD/ACAAAAAAAAAAAAEA8AAAAAAAAAAAAAAAAAAAAKEeAQAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAP8AIAAAAAAAAAAA AQDwAAAAAAAAAAAAAAAAAAAAoR4BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAA/wAgAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAA AAABAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAADAAD/ACC0HwAAAQABAAEAMAAAAAAAAAAAAAAAAAAAAAHuAR8QACAA3wABAAAA AADgHwEAAAAAAAAAAAAAAAAAAgAAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAoAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC4AAN8AAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAALu7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7uwEAAAgAAAEA MgACABwEAAAAAAAAAAAAAAAAAAAAAAAAAb4AAAYBAAAAugEA+iAAAABwAQAVX4EAU2klhwAACgAA BQAACgAOAAAFAAAYABAAAAUAACgAEgAABQAAOgAaAAAFAABUAAwAAAUAAGAABAAEAAAAZAAKAAAF AABuAAwAAAUAAHoACQAABQAAgwAEAAAFAACHAA8AAAUAAJYADwAABQAApQAQAAAFAAC1AA0AAAUA AMIAEwAABQAA1QAEAAAFAADZAAQAAAUAAN0ABwAABQAA5AAWAAAFAAD6AAgAAAUAAAIBCQAABQAA CwEKAAAFAAAVAQQAAQAAABkBFQAABQAALgEUAAAFAABCAREAAAMAAFMBFAAAAwAAZwENAAAFAAB0 AQwAAAUAAIABBwAABQAAhwESAAADAACZAQcAAAUAAKABCQAABQAAqQEGAAAFAACvAQsAAAUAALoB CgAABQAAxAEPAAAFAADTAQoAAAUAAN0BBwAABQAA5AEKAAAFAADuAQQAAAUAAPIBCgAABAAA/AEO AAAFAAAKAgoAAAUAABQCBQAAAwAAGQIMAAAFAAAlAhcAAAUAADwCCgABBQAARgIKAAEFAABTRUNV UkVNQUlMRGVsaXZlcnlSZXBvcnREZWxpdmVyeVByaW9yaXR5JEtlZXBQcml2YXRlQ2hvaWNlRXhw YW5kUGVyc29uYWxHcm91cHNDaG9pY2V0bXBTZW5kZXJUYWckUkVGTWFpbEZvcm1hdEZPTExPV1VQ REFURVNlbmRlclRhZ0Zvcm0kQ29uZmxpY3RBY3Rpb25Jbmhlcml0ZWRTZW5kVG9Jbmhlcml0ZWRS ZXBseVRvSW5oZXJpdGVkRnJvbUluaGVyaXRlZEZyb21Eb21haW5Mb2dvU2lnbkVuY3J5cHREZWZh dWx0TWFpbFNhdmVPcHRpb25zRm9ybU5hbWVTd2lwZUZyb21JbXBvcnRhbmNlQm9keUF0dGFjaG1l bnRzRGVsZXRlZE1zZ0V4cGFuZFBlcnNvbmFsR3JvdXBzVmFsaWRhdGlvblRlc3RpbmdBdHRhY2ht ZW50VmFsaWRhdGlvbkNsb3NpbmdGaWVsZHMkS2VlcFByaXZhdGVvbGRmcm9tdG1wQXR0YWNobWVu dFNpemVzU3ViamVjdHRtcEFjdGlvblNlbmRUb0JsaW5kQ29weVRvSU5ldENvcHlUb0lOZXRCbGlu ZENvcHlUbyRTdG9yYWdlVG8kTWFpbGVyJE1lc3NhZ2VJREZyb21Qb3N0ZWREYXRlJE1zZ1RyYWNr RmxhZ3NGcm9tRG9tYWluJEhvcHNPcmlnaW5hbEZyb20kRXhwb3J0SGVhZGVyc0NvbnZlcnRlZCRN SU1FVHJhY2skVXBkYXRlZEJ5u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7sCAru7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7BAAgBQAA +iAAAPQ4lYL8sBoLEF+BAFNpJYcBAAAAEF+BAFNpJYcAAAEAE1+BAFNpJYcyAAAAAAAAsAEA+ggA ABJfgQBTaSWHAAAMAAAAAAABAAwAAQAAAAIADAAIAAAAAwAMAAAAAAAEAAwAAAAAAAUADAABAAAA BgAMABIAAAAHAAwAAQAAAAgADAAOAAAACQAMAAEAAAAKAAwABQAAAAsADAABAAAADABMAA8AAAAN AEwAJwAAAA4ATAApAAAADwBMAAAAAAAQAAwAAAAAABEADAAAAAAAEgAMAAAAAAATAAwAAQAAABQA DAAEAAAAFQAMAB8AAAAWAAwABgAAABcACwD6CAAAGAAMAAAAAAAZAAwAAQAAABoADAAIAAAAGwAM AAgAAAAcAAwACQAAAB0ADAABAAAAHgAMACEAAAAfAAwACAAAACAADQAyAAAAIQAMAAAAAAAiAE0A DwAAACMATAAAAAAAJABNAAAAAAAlAEwAAAAAACYADAABAAAAJwAMAC4AAAAoAAwANQAAACkATQAw AAAAKgAMAAgAAAArAAwAAQAAACwADAAAAAAALQAMAAgAAAAuAAwAIQAAAC8ADAABAAAAMAAMAP4A AAAxAEwAQwAAAEIxIChIaWdoKU4BAMAF9IzEwu+s+kF9AFNpJYdCRmlsZVRvRm9sbG93VXBOUmVw bHkxaXBzQGVjZS5jbXUuZWR1TWF0dCBXYWtlbGV5IDxtYXR0X3dha2VsZXlAYWdpbGVudC5jb20+ Ik1hdHQgV2FrZWxleSIgPG1hdHRfd2FrZWxleUBhZ2lsZW50LmNvbT4xTWVtb0pvaG4gSHVmZmVy ZC9TYW4gSm9zZS9JQk1ASUJNVVNOb3JtYWwwAAAAAAAAAAAAAAAAAAAAAEJvZHksQm9keTBDTj1K b2huIEh1ZmZlcmQvT1U9U2FuIEpvc2UvTz1JQk0AAAAAAAAAAFJlOiBhIHZvdGUgZm9yIGFzeW1t ZXRyaWMgY29ubmVjdGlvbnMgaW4gYSBzZXNzaW9uaXBzQGVjZS5jbXUuZWR1LkxvdHVzIE5vdGVz IFJlbGVhc2UgNS4wLjMgKEludGwpIDIxIE1hcmNoIDIwMDA8T0Y5NjlBNkZDMS4wMTM5RDUzNy1P Tjg4MjU2OTUzLjAwN0ZCMkY5QExvY2FsRG9tYWluPiJKb2huIEh1ZmZlcmQvU2FuIEpvc2UvSUJN IiA8aHVmZmVyZEB1cy5pYm0uY29tPkQEgQBTaSWIMAAAAAAAADlAQ049Sm9obiBIdWZmZXJkL09V PVNhbiBKb3NlL089SUJNMQMAZgAsAGQAU2VyaWFsaXplIGJ5IFJvdXRlciBvbiBEMDNOTTA5NC8w My9NL0lCTShSZWxlYXNlIDUuMC4zIChJbnRsKXwyMSBNYXJjaCAyMDAwKSBhdCAwOS8wNy8yMDAw IDA1OjMzOjA0IFBNU2VyaWFsaXplIGNvbXBsZXRlIGF0IDA5LzA3LzIwMDAgMDU6MzM6MDQgUE1D RC1NSU1FIGJ5IFJvdXRlciBvbiBEMDNOTTA5NC8wMy9NL0lCTShSZWxlYXNlIDUuMC4zIChJbnRs KXwyMSBNYXJjaCAyMDAwKSBhdCAwOS8wNy8yMDAwIDA1OjMzOjA0IFBNAgAhABwAQ049Sm9obiBI dWZmZXJkL09VPVNhbiBKb3NlL089SUJNQ049RDAzTk0wOTQvT1U9MDMvT1U9TS9PPUlCTSMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCAOgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQDoAACMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAA6AAAjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDAOQAAIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMA gDkAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMDAEA5AAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjAwAAOQAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIwMAwDgAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIA4AAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBAOAAAIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAADgAACMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMA3AAAjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCA NwAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIwMAQDcAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMDAAA3AAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjAwDANgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgDYAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEA2AAAjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAANgAAIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwDUAACMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIA1 AAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjAwBANQAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIwMAADUAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMDAMA0AAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCANAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQDQAACMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAA0AAAjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDAMwAAIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgDMA ACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMDAEAzAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjAwAAMwAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIwMAwDIAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIAyAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBAMgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAADIAACMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMAxAAAjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCAMQAA IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IwMAQDEAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMDAAAxAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjAwDAMAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgDAAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAwAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAAMAAAIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwC8AACMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIAvAAAj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj AwBALwAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIwMAAC8AACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMDAMAuAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCALgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQC4AACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAAuAAAjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDALQAAIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgC0AACMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMD AEAtAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjAwAALQAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIwMAwCwAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMDAIAsAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBALAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAACwAACMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMArAAAjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCAKwAAIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMA QCsAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMDAAArAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjAwDAKgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIwMAgCoAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAqAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAAKgAAIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwCkAACMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIApAAAjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBA KQAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIwMAACkAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMDAMAoAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjAwCAKAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQCgAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAAoAAAjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDAJwAAIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgCcAACMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAn AAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjAwAAJwAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIwMAwCYAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMDAIAmAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBAJgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAACYAACMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMAlAAAjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCAJQAAIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQCUA ACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMDAAAlAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjAwDAJAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIwMAgCQAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAkAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAAJAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwCMAACMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIAjAAAjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBAIwAA IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IwMAACMAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMDAMAiAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjAwCAIgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQCIAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAAiAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDAIQAAIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgCEAACMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAhAAAj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj AwAAIQAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIwMAwCAAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMDAIAgAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBAIAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAACAAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMAfAAAjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCAHwAAIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQB8AACMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMD AAAfAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjAwDAHgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIwMAgB4AACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAeAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAAHgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwB0AACMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIAdAAAjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBAHQAAIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMA AB0AACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMDAMAcAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjAwCAHAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIwMAQBwAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAAcAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDAGwAAIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgBsAACMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAbAAAjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAA GwAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIwMAwBoAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMDAIAaAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjAwBAGgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAABoAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMAZAAAjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCAGQAAIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQBkAACMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAAZ AAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjAwDAGAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIwMAgBgAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMDAEAYAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAAGAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwBcAACMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIAXAAAjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBAFwAAIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAABcA ACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMDAMAWAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjAwCAFgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIwMAQBYAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAAWAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDAFQAAIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgBUAACMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAVAAAjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAAFQAA IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IwMAwBQAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMDAIAUAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjAwBAFAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAABQAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMATAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCAEwAAIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQBMAACMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAATAAAj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj AwDAEgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIwMAgBIAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMDAEASAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAAEgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwBEAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIARAAAjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBAEQAAIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAABEAACMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMD AMAQAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjAwCAEAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIwMAQBAAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMDAAAQAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDADwAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgA8AACMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAPAAAjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAADwAAIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMA wA4AACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMDAIAOAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjAwBADgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIwMAAA4AACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMANAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCADQAAIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQA0AACMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAANAAAjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDA DAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIwMAgAwAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMDAEAMAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjAwAADAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwAsAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIALAAAjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBACwAAIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAAAsAACMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMAK AAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjAwCACgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIwMAQAoAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMDAAAKAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDACQAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgAkAACMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEAJAAAjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAACQAAIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwAgA ACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMDAIAIAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjAwBACAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIwMAAAgAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMAHAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCABwAAIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQAcAACMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAAHAAAjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDABgAA IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IwMAgAYAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMDAEAGAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjAwAABgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwAUAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAIAFAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwBABQAAIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAAAUAACMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAMAEAAAj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj AwCABAAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIwMAQAQAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMDAAAEAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDAAwAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAgAMAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAEADAAAjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwAAAwAAIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAwAIAACMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMD AIACAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjAwBAAgAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIwMAAAIAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMDAMABAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwCAAQAAIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMAQAEAACMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMDAAABAAAjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjAwDAAAAAIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwMA gAAAACMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMDAEAAAAAjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjgv9SAAEAAAAAAAAAAACgBQAAoAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAQAAAACUBKAFCgCgBQoAAAAAAIMEAQCF/w4AAQAACkZvbGtzLIEC gwQBAIX/RAEBAAAKTWF0dCBhbmQgSnVsaWFuIGFyZSBjb3JyZWN0LiAgVGhlIHNsaWRpbmcgd2lu ZG93IHdhcyBwdXQgaW4gZm9yIGEgY29tcGxldGVseSBkaWZmZXJlbnQgcmVhc29uIHRoZW4gaXMg Y3VycmVudGx5IGJlaW5nIGRpc2N1c3NlZC4gIEkgc3VnZ2VzdCB0aGF0IHdlIGFsbCAgbG9vayBh dCBpdCBvbmx5IGFzIGl0IHdhcyBvcmlnaW5hbGx5IGludGVuZGVkIGFuZCB0aGVuIHNlZSBpZiBz b21ldGhpbmcgZWxzZSBuZWVkcyB0byBiZSBkb25lLCBvciBub3QuICBUQ1AvSVAgcHJvYmFibHkg aGFuZGxlcyBhbGwgdGhlIG90aGVyIGlzc3VlcyAod2VsbCBlbm91Z2gpLoECgwQBAIX/IQABAAAK AAAuAC4ALgBKb2huIEwuIEh1ZmZlcmQAAACBArccoAXBIakDBAAAAEgAAAAAAAAAAAAAAAAAAACD BAEAgQK6KmAMSAC5GhkDAQENCQEAAQD///8AKQAAAAAAAAAAAAAAAAAAAAAAAACDBAEAhf9XAAEB AAkiTWF0dCBXYWtlbGV5IiA8bWF0dF93YWtlbGV5QGFnaWxlbnQuY29tPkBlY2UuY211LmVkdSBv biAwOS8wNy8yMDAwIDAzOjQ2OjUzIFBNALkSAAAAAAAAAAAAAAAAAAAAAIECgv9SAAIAAAAAAAEA AADKCAAAoAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAQAAAACU BMoICgCgBQoAAABkAIMEAgDdBgAAhf+F/0IAAQALCVBsZWFzZSByZXNwb25kIHRvIE1hdHQgV2Fr ZWxleSA8bWF0dF93YWtlbGV5QGFnaWxlbnQuY29tPiDSCAEAgACAAN4GAACF/4ECgv9SAAMAAAAA AAEAAQDKCAAAoAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAQAA AACUBMoICgCgBQoAAABkAIMEAwDdBgAAhf+F/yYAAQALCVNlbnQgYnk6CW93bmVyLWlwc0BlY2Uu Y211LmVkddIIAQCAAIAA3gYAAIX/gQKDBAIA3QYAAIX/hf8MAAEACwlUbzoJ0ggBAIAAgADeBgAA hf+F/xcAAQAACWlwc0BlY2UuY211LmVkdQCBAoL/UgAEAAAAAAAAAAAAyggAAKAFAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABgEAAAAAlATKCAoAoAUKAAAAZACDBAQA 3QYAAIX/hf8NAAEACwljYzoJIADSCAEAgACAAN4GAACF/4ECgv9SAAUAAAAAAAAAAADKCAAAoAUA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAQAAAACUBMoICgCgBQoA AABkAIMEBQDdBgAAhf+F/xEAAQALCVN1YmplY3Q6CQDSCAEAgACAAN4GAACF/4X/OgABAAAJUmU6 IGEgdm90ZSBmb3IgYXN5bW1ldHJpYyBjb25uZWN0aW9ucyBpbiBhIHNlc3Npb26BAoMEAQCF/wgA AQAACoECgwQBAIX/CAABAAAJgQKDBAEAhf8IAAEAAAqBAoMEAQCF/5UBBAAACkFzIEp1bGlhbiBo YXMgc3RhdGVkIGluIGEgZGlmZmVyZW50IHRocmVhZCwgdGhlIHB1cnBvc2Ugb2YgdGhlICJzbGlk aW5nAHdpbmRvd3MiIGluIGlTQ1NJIGlzIG5vdCBmb3IgY29uZ2VzdGlvbiBtYW5hZ2VtZW50LiAg SXQgaXMgc2ltcGx5IHRoZXJlIHRvAGhhbmRsZSB0aGUgY2FzZSB3aGVyZSBpZiBhIGNvbm5lY3Rp b24gZ29lcyBkb3duIGluIGEgbXVsdGlwbGUgY29ubmVjdGlvbgBzZXNzaW9uLCBpdCBwcmV2ZW50 cyB0aGUgcmVtYWluaW5nIGNvbm5lY3Rpb25zIGZyb20gb3ZlcndoZWxtaW5nIHRoZSB0YXJnZXQA d2l0aCBuZXcgY29tbWFuZHMgdGhhdCBpdCBjYW4ndCBwcm9jZXNzIGR1ZSB0byBtaXNzaW5nIGNv bW1hbmRzIHRoYXQgd2hlcmUgb24AdGhlIGJyb2tlbiBjb25uZWN0aW9uLgAAgQKDBAEAhf+SAAQA AApTaW5jZSBhbGwgb2YgdGhpcyBydW5zIG9uIHRvcCBvZiBUQ1AsIGFuZCBUQ1AgcGVyZm9ybXMg Y29uZ2VzdGlvbiBtYW5hZ2VtZW50LAB3aHkgbXVzdCBpU0NTSSBwZXJmb3JtIGNvbmdlc3Rpb24g bWFuYWdlbWVudCBvbiB0b3Agb2YgVENQPwCBAoMEAQCF/ysABAAACi1NYXR0IFdha2VsZXkAQWdp bGVudCBUZWNobm9sb2dpZXMAAIECgwQBAIX/HQAEAAAKU2NvdHQgQnJhZG5lciB3cm90ZToAAIEC gwQBAIX/1AAEAAAKPiA+IEltcGxlbWVudGluZyBzbGlkaW5nIHdpbmRvd3MgaXMgbm90IHRoYXQg aGFyZAA+AD4gbm90ZSB0aGF0IHRoZSBpc3N1ZSBpcyBub3QgImp1c3QiIHNsaWRpbmcgd2luZG93 cyAtIGlwcyBhbHNvIGhhcyB0byBkZWFsAD4gd2l0aCBjb25nZXN0aW9uIFRDUC1mcmllbmRseSB3 YXkgLSB0aGF0IGNhbiBnZXQgcXVpdGUgY29tcGxpY2F0ZWQAPgA+IFNjb3R0gQKDBAEAhf8IAAEA AAmBAoMEAQCF/wgAAQAACoECgwQBAIX/CAABAAAKu7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7uwYKAAAAAPogAABAcAEA//////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////// ////////////////////////////////////////////////////////////////u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7 u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7u7s= --0__=88256953008104458f9e8a93df938690918c8825695300810445-- From owner-ips@ECE.cmu.edu Fri Sep 08 01:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04030; 7 Sep 2000 21:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA28104 for ips-outgoing; Thu, 7 Sep 2000 20:31:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA28093 for ; Thu, 7 Sep 2000 20:31:37 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA08188 for ; Thu, 7 Sep 2000 17:31:36 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.154.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id RAA15891 for ; Thu, 7 Sep 2000 17:31:35 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id RAA04993; Thu, 7 Sep 2000 17:31:34 -0700 (PDT) Message-Id: <200009080031.RAA04993@ha10nwk.EBay.Sun.COM> Date: Thu, 7 Sep 2000 17:31:34 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Symmetric vs Asymmetric To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: CKNeTvT4plWhI+YCN76MgQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > The biggest issue with the "asymmetric" model is that it is NOT "asymmetric" > when there is only 1 TCP connection. When there is only one TCP connection, it > is the "symmetric" model - both commands and data on the same TCP connection. > Then, when there is more than one TCP connection, the behavior is different. > It's always easier to implement something that operates that same way all the > time, than two different behaviors. > > I propose that the asymmetric model mandate at least two TCP connections > (implies that at least one physical connection will have at least two TCP > connections running on it) - one for commands, the other for data. This has > other advantages, like commands not being flow controlled by large transfers of > data. That's not a bug, that's a feature! Having the asymmetric degenerate into symmetric with just one connection is a good thing. For starters it is easier to implement initially or in cheap devices and doesn't have the baggage that a true symmetric design would require but is not needed with one connection. I would oppose mandating two connections minimum, if flow control is a problem then the defacto configuration will be two connections, but lets not require it. Personally I would still prefer one connection per LUN, but the proposed asymmetric model is a good compromise. You could still deploy an implementation with a connection per LUN and not have any significant unnecessary baggage. Not true of the symmetric model. -David From owner-ips@ECE.cmu.edu Fri Sep 08 02:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04245; 7 Sep 2000 22:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA29595 for ips-outgoing; Thu, 7 Sep 2000 21:25:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA29591 for ; Thu, 7 Sep 2000 21:25:25 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel1.hp.com (Postfix) with ESMTP id 870A9E49; Thu, 7 Sep 2000 18:25:23 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id SAA25805; Thu, 7 Sep 2000 18:25:22 -0700 (PDT) X-OpenMail-Hops: 1 Date: Thu, 7 Sep 2000 18:25:04 -0700 Message-Id: Subject: RE: Re: Symmetric vs Asymmetric MIME-Version: 1.0 To: hufferd@us.ibm.com, ips@ece.cmu.edu, matt_wakeley@agilent.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, In your response to my message, it seemed to me that you did not disagree with my concerns about performance impact (only amplified it I think) of the assymmetric approach. The impact will be there even when there are two connections on a single NIC - especially when the NIC might be a generic TCP accelerator kind of NIC (which will operate on a connection basis), and for any software based implementation. On the other hand having two connections minimizes head of line blocking for critical commands (but is still possible and might be most likely when the target is in a bad state anyhow) - so maybe it is not a good point. Having two connections mandated would cause significant performance penalty in many implementations and is best avoided. Somesh > -----Original Message----- > From: matt_wakeley@agilent.com [mailto:matt_wakeley@agilent.com] > Sent: Thursday, September 07, 2000 4:56 PM > To: matt_wakeley@agilent.com; hufferd@us.ibm.com; ips@ece.cmu.edu > Subject: FW: Re: Symmetric vs Asymmetric > > > John Hufferd/San Jose/IBM wrote: > > > David, > > One of the reasons I was holding out for Symmetric over > Asymmetric, was for > > the workload balancing etc. that could be done at the iSCSI layer. > > > > I am now willing to "roll-over" but I think I owe you all > my reasons for > > deciding to support the Asymmetric approach: > > > > First, in the degenerate case, they both look the same. > That is, they can > > both support a single Connection per Session. So the simplest > > implementation is still possible with the Asymmetric > approach as it was > > with the Symmetric approach. This means that most current iSCSI > > implementations can continue to exist, unchanged. > > The biggest issue with the "asymmetric" model is that it is > NOT "asymmetric" > when there is only 1 TCP connection. When there is only one > TCP connection, it > is the "symmetric" model - both commands and data on the same > TCP connection. > Then, when there is more than one TCP connection, the > behavior is different. > It's always easier to implement something that operates that > same way all the > time, than two different behaviors. > > I propose that the asymmetric model mandate at least two TCP > connections > (implies that at least one physical connection will have at > least two TCP > connections running on it) - one for commands, the other for > data. This has > other advantages, like commands not being flow controlled by > large transfers of > data. > > -Matt Wakeley > Agilent Technologies > > > > > > > But the most important reason follows: > > > > Since a session is a unique construct that identifies an > Initiator and a > > Target, I am assuming that does NOT mean each NICs has a > unique initiator > > ID but that they can, and should, share a common Initiator > ID. The Target > > ID, on the other hand, is either unique per Storage > Controller, or not, > > and this will depend on the type of Storage Controller. > This means that > > the Host iSCSI device driver will be able to present, to > the layers above > > it, either a single port per storage controller, or multiple ports, > > depending on how the Storage Controller answers Logins on > each of its IP > > connections. > > > > That means iSCSI will not be able to determine when it > knows enough to > > perform alternate path retry, and for the same reason will > not be able to > > know when or how to perform command work load balancing across the > > different sessions. This is because either all connections > to a storage > > controller have either the same iSCSI session ID (SSID), or they are > > different -- but iSCSI will not, by itself, know which > sessions can be > > substituted for others (for balancing or retry). > > > > The only place where the symmetric approach can do workload > balancing and > > alternate path retry is where the storage controller makes > it look like a > > single target ID even though it might have multiple IP > addresses. Though > > this might work in some IBM storage controllers (like > Shark) it will not > > work for all IBM storage controllers (like those of Mylex). > This same or > > similar argument probably will apply to EMC's and other > vendors various > > storage controllers. Therefore, having iSCSI do any type of command > > balancing or command automatic alternate path retry is problematical > > without outside influences (probably from upper layers) and > this will tend > > to make the iSCSI layer very thick and complicated. It > will also mix the > > layering concepts (in general not a good idea). Therefore, > I now believe > > that the Symmetrical approach can not be a reasonable > general/generic > > approach to Command work load balance or alternate path > retry, and that > > function is best left to the Wedge Drivers. > > > > The Asymmetric approach can not perform command balancing or command > > automatic path retry either, but it does not try. It only > tries to balance > > the data across multiple connections (and that seems much > more reasonable). > > > > With the Asymmetric approach, data for any specific command > only flows on > > one connection. So you can look at the data from various > commands as > > independent flows on the different connections (NICs) so > there should not > > be blocking issues, or parallel flow issues, and RDMA type > direct to memory > > functions can occur. > > > > With multiple storage controllers, there may be many > sessions and many data > > connections spread across all the various NICs in a Host. > > > > The net of all this is that the Asymmetric approach: > > Probably has good enough spread of commands and data > across multiple > > NICs, even without Wedge Drivers. > > Permits the upper layer Wedge Drivers to be left mostly > unchanged. > > Permits the simple case with a single connection per > session, even with > > multiple NICs. > > Permits the most simple case of a single connection per > session with > > only a single NIC. > > > > Since I have seen no one else that made arguments for the Symmetric > > implementation, you maybe able to call the agreement on the > Asymmetric > > approach a consensus. > > . > > . > > . > > John L. Hufferd > From owner-ips@ECE.cmu.edu Fri Sep 08 02:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04291; 7 Sep 2000 22:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA00086 for ips-outgoing; Thu, 7 Sep 2000 21:41:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA00081 for ; Thu, 7 Sep 2000 21:41:25 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Thu, 7 Sep 2000 18:41:42 -0700 Message-ID: From: Charles Monia To: John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Thu, 7 Sep 2000 18:41:35 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Thursday, September 07, 2000 4:29 PM > To: ips@ece.cmu.edu > Subject: Re: a vote for asymmetric connections in a session > > > Folks, > Matt and Julian are correct. The sliding window was put in for a > completely different reason then is currently being > discussed. I suggest > that we all look at it only as it was originally intended > and then see if > something else needs to be done, or not. TCP/IP probably > handles all the > other issues (well enough). > > > . > . > . > John L. Hufferd > Hi John: What other issues do uou mean? Cbarles From owner-ips@ECE.cmu.edu Fri Sep 08 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04320; 7 Sep 2000 22:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA00110 for ips-outgoing; Thu, 7 Sep 2000 21:42:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA00106 for ; Thu, 7 Sep 2000 21:42:04 -0400 (EDT) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA27407 for ; Thu, 7 Sep 2000 18:41:10 -0700 (PDT) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA03251 for ; Thu, 7 Sep 2000 18:38:13 -0700 (PDT) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Thu, 7 Sep 2000 18:41:10 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C05B31CDA@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: ips@ece.cmu.edu Subject: RE: Symmetric vs Asymmetric Date: Thu, 7 Sep 2000 18:41:00 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk In general I agree with the asymmetric model. David Robinson wrote: > > That's not a bug, that's a feature! Having the asymmetric degenerate > into symmetric with just one connection is a good thing. For starters > it is easier to implement initially or in cheap devices and doesn't > have the baggage that a true symmetric design would require > but is not needed with one connection. I would oppose mandating > two connections minimum, if flow control is a problem then the defacto > configuration will be two connections, but lets not require it. > I agree with David. I oppose a two connection minimum. > Personally I would still prefer one connection per LUN, but the > proposed asymmetric model is a good compromise. You could > still deploy an implementation with a connection per LUN and not > have any significant unnecessary baggage. Not true of the symmetric > model. > One connection per LUN alleviates commands being flow controlled by large transfers of data. Paul From owner-ips@ECE.cmu.edu Fri Sep 08 04:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05122; 8 Sep 2000 0:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA01501 for ips-outgoing; Thu, 7 Sep 2000 22:43:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA01496 for ; Thu, 7 Sep 2000 22:43:07 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id TAA20095; Thu, 7 Sep 2000 19:43:01 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Thu, 7 Sep 2000 19:43:00 -0700 Message-ID: From: Jim McGrath To: "'Matt Wakeley'" , ips Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Thu, 7 Sep 2000 19:42:59 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree that BB credit has nothing to do with commands per se, but illustrates the problem we have had with deciding on policies for the distributed allocation of device resources over multiple initiators. In my postscript I noted that the same problems have arisen on command queue (how many queue slots do you get), with similarly no satisfactory solution. On FC-AL, the latency to get an initial credit is typically measured in us for a couple of reasons. First, much of that logic has been automated in the interface hardware (indeed, the major source of delay is typically the elasticity buffer, which is not store and forward and so has a very low latency compared to many switches or routers). Second, the distances are very small (e.g. hundreds of meters), so both transmission time and the opportunity for intervening devices to increase latency is lower than in the general internet world. So generally 2 credits (of 2Kbyte frames) is enough to cover the latency and get you into streaming. If the latency of the system was measured in ms, then a transport on a Gibt wire would require more like 50 initial credits (or more) to cover the latency. Unless we are designing for a low latency environment for the exchange of credits (like those where FC-AL are used), then we probably need to allocate so much buffer space that it becomes difficult to promise initial credits to a lot of potential initiators. Jim PS general Fibre Channel (e.g. with switches and the like) is a bit different. -----Original Message----- From: Matt Wakeley [mailto:matt_wakeley@agilent.com] Sent: Wednesday, September 06, 2000 8:23 PM To: ips Subject: Re: Command Queue Depth (was asymmetric/Symmetric) Jim, I agree that FC has tried (unsuccessfully) to address this command queue allocation problem. However, the "login bb credit" mechanism in FC does not address the command queue depth issue at all. BB credit is used to receive commands and/or data, and the target has no clue in advance what is coming. BB credit is just there to ensure that there is a lowest layer buffer available to receive the FC frame (as opposed to dropping it on the floor if there is no "mac" buffer, like ethernet does). It does not mean the command queue has any room for the frame. At one time, there was a big push to have "data" credits and "command" credits to take care of this problem, but it couldn't be made to work and be "backwards compatible". > The issue of buffer space allocation for multiple initiators has a long and > troubled history in SCSI. We have never been able to come up with a good > answer. > > Fibre Channel tried to fix this with the notion of "login BB credit" - when > you login you get a minimum number of credits you are always guaranteed when > you start data transfers. The problem with this is that storage devices had > no realistic ability to discriminate between initiators or to change the > login BB credit. In addition, the expectation is that all possible > initiators would get these credits on login. So storage devices vendors > have played it safe and kept this number low (at 0 until recently, now > around 2). For iSCSI the number of initial credits you need to "prime the > pump" until normal data flow is established is probably large (given the > latencies are higher than in Fibre Channel, especially FC-AL), and the How is the latency low on FC-AL?? given that you need to arbitrate and win the loop, then receive bb credit, before you can send anything? -Matt > > number of potential initiators larger than in Fibre Channel, making this a > whole lot worse for the storage device. > > As soon as we allow the devices to start adjusting these credits, then you > have the protocol problem of making sure people know when their credits are > adjusted and the policy problem of how, who, and when to adjust the credits. > Changing everyone's credit when you add a new initiator can get into a > notification nightmare, although it is "fair." Any policy brings up all > sorts of nasty issues regarding fairness vs efficient use of the > transmission media. > > Jim > > Note: the same problem has plagued other attempts to allocate device > resources between multiple initiators, like command queue space. In general > policies with respect to multiple initiators are not really standard in the > SCSI world. > > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 2:56 PM > To: ips > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > Joshua Tseng wrote: > > > James, > > > > I agree with others that there may be an issue with the > > command windowing mechanism in the existing iSCSI spec. It is like > > "TCP in reverse", in that the target determines the size of the window, > and > > not the initiator as in TCP. Rather, I believe that everything that this > > windowing mechanism is attempting to achieve can be more easily obtained > > by having the target communicate its buffer size to the initiator at > > iSCSI login. It should be the role of the initiator to determine how > > many commands to put in flight simultaneously, given this input on > available > > buffer size from the target. > > As more initiators connect to a target, it may need to scale back the amount > of > this buffering it has allocated to each previously logged in initiator (to > prevent rejecting new logins). > > > > > > > As far as multiple initiators, could this not be resolved by the target > > refusing additional logins beyond the number of initiators it can safely > > support? Not being a storage expert, this is my best guess/suggestion > > at how to do it. > > I believe John already answered this... > > -Matt From owner-ips@ECE.cmu.edu Fri Sep 08 05:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05397; 8 Sep 2000 1:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA02872 for ips-outgoing; Thu, 7 Sep 2000 23:44:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA02866 for ; Thu, 7 Sep 2000 23:44:05 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id UAA26147; Thu, 7 Sep 2000 20:43:55 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.10) id ; Thu, 7 Sep 2000 20:43:54 -0700 Message-ID: From: Jim McGrath To: "'Charles Monia'" , "Julian Satran (E-mail)" Cc: "Ips (E-mail)" Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Thu, 7 Sep 2000 20:43:52 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.10) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk The path I'd recommend is to allow people to oversubscribe a target's resources, and then to do a graceful recovery when that gets you into trouble. Note that drive designers do this all of the time - we tend to optimize for common cases, and then worry about how to handle outlyers using other mechanisms. While a more complicated model, it gets you the best overall resource utilization. We use read on arrival, ECC on the Fly, Retrys, auto reallocation, all in attempts to handle the common path quickly and the rare path more slowly, where the differences are error rates. In this case I would allow initiators to send down data immediately - when that works (like when ECC on the Fly works) you get a benefit. If packets are dropped you can rely on existing mechanisms to recover, or you can put in a new, improved, and perhaps more friendly process. In either case I think the result would probably be better than a tight credit based model where a lot of delays would be introduced and a lot of (historically unresolved) allocation policy issues arise. Jim -----Original Message----- From: Charles Monia [mailto:cmonia@NishanSystems.com] Sent: Thursday, September 07, 2000 1:52 PM To: Julian Satran (E-mail) Cc: Ips (E-mail) Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Hi Julo: > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, September 07, 2000 6:11 AM > To: ips@ece.cmu.edu > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > Dear colleagues, > > Although the windowing mechanism in iSCSI-01 may seem to be > there to solve > a queueing issue > it is mainly meant to limit the buffering space for commands > that await > "de-skewing". > We assume that execution queue-lengths, policy etc. are > beyond the scope of > transport. > > As for SCSI queue length I assumed that the busy or queue full status > followed by an Asynch Event > message indicating readiness is the mechanism provided by > SCSI to regulate > the command flow. > > It is hard to imagine that give the variable life-time of > SCSI commands and > the > opaque nature of the resources required to execute them that > the transport > has > to help in this area. > While this issue has been discussed at some length in the past, as Jim McGrath stated, I believe the debate ought to be reopened (although we may end up reaching the same conclusion as before). As Ralph Weber pointed out, the SCSI model is to discard the command, return status, retrieve the next command in the transport pipeline and continue processing. The initiator is not notified when processing resumes. If there are many commands in flight, as there could be in an IP environment, and target resources free up in the meantime, the result is commands processed out of order. Historically, such a lapse in command ordering was not seen as an issue for the following reasons: a) Strict ordering was not required by the most commonly deployed device types (disks and tapes). Due to the nature of disk traffic, a simple retry mechanism was deemed sufficient to recover from these errors. Since legacy streaming devices, such as tapes, did not support command queuing, command ordering considerations were not a factor there either. b) Transport delays over storage interconnects were small, so not many commands were apt to be in flight. i.e.. The window for such errors was very small. c) The resource guarantees needed for a loss-avoidance mechanism in the target adversely effected device cost, especially at the high-volume, low-end of the market. Given the above considerations, there was little support within the storage community for measures addressing this issue. If we now believe that the iSCSI environment changes the rules, I believe the interconnect protocol can provide useful assists, such as: a) On a command overflow condition, have the iSCSI target flush the command pipeline by returning status and discarding all subsequently received commands until a host acknowledgement is received. b) Implement some sort of credit-based mechanism for overflow-avoidance. Comments? > Jim McGrath on 07/09/2000 06:06:40 > > Please respond to Jim McGrath > > To: "'Matt Wakeley'" , ips > > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > > > The issue of buffer space allocation for multiple initiators > has a long and > troubled history in SCSI. We have never been able to come up > with a good > answer. > > Fibre Channel tried to fix this with the notion of "login BB > credit" - when > you login you get a minimum number of credits you are always > guaranteed > when > you start data transfers. The problem with this is that > storage devices > had > no realistic ability to discriminate between initiators or to > change the > login BB credit. In addition, the expectation is that all possible > initiators would get these credits on login. So storage > devices vendors > have played it safe and kept this number low (at 0 until recently, now > around 2). For iSCSI the number of initial credits you need > to "prime the > pump" until normal data flow is established is probably large > (given the > latencies are higher than in Fibre Channel, especially FC-AL), and the > number of potential initiators larger than in Fibre Channel, > making this a > whole lot worse for the storage device. > > As soon as we allow the devices to start adjusting these > credits, then you > have the protocol problem of making sure people know when > their credits are > adjusted and the policy problem of how, who, and when to adjust the > credits. > Changing everyone's credit when you add a new initiator can get into a > notification nightmare, although it is "fair." Any policy > brings up all > sorts of nasty issues regarding fairness vs efficient use of the > transmission media. > > Jim > > Note: the same problem has plagued other attempts to allocate device > resources between multiple initiators, like command queue space. In > general > policies with respect to multiple initiators are not really > standard in the > SCSI world. > > > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 2:56 PM > To: ips > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > Joshua Tseng wrote: > > > James, > > > > I agree with others that there may be an issue with the > > command windowing mechanism in the existing iSCSI spec. It is like > > "TCP in reverse", in that the target determines the size of > the window, > and > > not the initiator as in TCP. Rather, I believe that > everything that this > > windowing mechanism is attempting to achieve can be more > easily obtained > > by having the target communicate its buffer size to the initiator at > > iSCSI login. It should be the role of the initiator to > determine how > > many commands to put in flight simultaneously, given this input on > available > > buffer size from the target. > > As more initiators connect to a target, it may need to scale back the > amount > of > this buffering it has allocated to each previously logged in > initiator (to > prevent rejecting new logins). > > > > > > > As far as multiple initiators, could this not be resolved > by the target > > refusing additional logins beyond the number of initiators > it can safely > > support? Not being a storage expert, this is my best > guess/suggestion > > at how to do it. > > I believe John already answered this... > > -Matt > > > Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Fri Sep 08 06:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05972; 8 Sep 2000 2:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA05567 for ips-outgoing; Fri, 8 Sep 2000 01:37:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA05563 for ; Fri, 8 Sep 2000 01:37:04 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 66416438 for ; Fri, 8 Sep 2000 01:37:03 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA03669 for ; Thu, 7 Sep 2000 22:37:01 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA603B for ; Thu, 7 Sep 2000 22:36:58 -0700 Message-ID: <39B87AEE.E2063161@agilent.com> Date: Thu, 07 Sep 2000 22:36:46 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Symmetric vs Asymmetric References: <5B0C98C89640D311A319002048406A1C05B31CDA@aimexc03.corp.adaptec.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "VonStamwitz, Paul" wrote: > In general I agree with the asymmetric model. > > David Robinson wrote: > > > > That's not a bug, that's a feature! Having the asymmetric degenerate > > into symmetric with just one connection is a good thing. For starters > > it is easier to implement initially or in cheap devices and doesn't > > have the baggage that a true symmetric design would require > > but is not needed with one connection. I would oppose mandating > > two connections minimum, if flow control is a problem then the defacto > > configuration will be two connections, but lets not require it. > > > > I agree with David. I oppose a two connection minimum. I disagree. Initial implementations and cheap hardware will have the iSCSI implementation in software. It is easy to do anything in software - open a thousand connections, perhaps hundreds of thousands, one per each LU, it doesn't care - it just uses up more memory. However, for the high performance implementations that will implement iSCSI in hardware, doing a function one way sometimes, and another other times, will require more hardware and more testing (by both the hardware vendors and the value add vendors) to test both functions. One connection per LU also has the huge issue of requiring lots of on-hardware resources and memory to maintain the hundreds or thousands of connections that will be required.... and you want a cost competitive solution? > > Personally I would still prefer one connection per LUN, but the > > proposed asymmetric model is a good compromise. You could > > still deploy an implementation with a connection per LUN and not > > have any significant unnecessary baggage. Not true of the symmetric > > model. > > > > One connection per LUN alleviates commands being flow controlled by large > transfers of data. Exactly. > > > Paul -Matt From owner-ips@ECE.cmu.edu Fri Sep 08 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05985; 8 Sep 2000 2:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA05701 for ips-outgoing; Fri, 8 Sep 2000 01:43:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA05696 for ; Fri, 8 Sep 2000 01:43:46 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id F01A5A8 for ; Fri, 8 Sep 2000 01:43:45 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA03698 for ; Thu, 7 Sep 2000 22:43:44 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA6059; Thu, 7 Sep 2000 22:43:39 -0700 Message-ID: <39B87C7A.9D655F3D@agilent.com> Date: Thu, 07 Sep 2000 22:43:22 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: somesh_gupta@hp.com, ips@ece.cmu.edu Cc: hufferd@us.ibm.com Subject: Re: Symmetric vs Asymmetric References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk somesh_gupta@hp.com wrote: > Matt, > > In your response to my message, it seemed to me that you did > not disagree with my concerns about performance impact (only > amplified it I think) of the assymmetric approach. That's right. > The impact > will be there even when there are two connections on a single > NIC - especially when the NIC might be a generic TCP accelerator > kind of NIC (which will operate on a connection basis), and > for any software based implementation. Yes, but software implementations will be slow anyway, so who cares? In the case where there is a iSCSI NIC implementing the iSCSI and TCP accelerators in hardware, the host will not know whether there is one or two TCP connections. It will simply post the "SCSI command" and the iSCSI NIC will take care of the rest. > On the other hand having two connections minimizes head of line > blocking for critical commands (but is still possible and might > be most likely when the target is in a bad state anyhow) - so > maybe it is not a good point. But the normal case will be that you will be able to pump the commands to the target without waiting for large data transfers to complete, allowing the target to get a jump start on the command. > Having two connections mandated would cause significant performance > penalty in many implementations and is best avoided. It will only impact the generic TCP (accelerated or not) implementation. It will improve the hardware based iSCSI implementation for the reasons of lower latency command transfer, less expensive hardware, and less vendor testing. -Matt Wakeley Agilent Technologies > > > Somesh > > > -----Original Message----- > > From: matt_wakeley@agilent.com [mailto:matt_wakeley@agilent.com] > > Sent: Thursday, September 07, 2000 4:56 PM > > To: matt_wakeley@agilent.com; hufferd@us.ibm.com; ips@ece.cmu.edu > > Subject: FW: Re: Symmetric vs Asymmetric > > > > > > John Hufferd/San Jose/IBM wrote: > > > > > David, > > > One of the reasons I was holding out for Symmetric over > > Asymmetric, was for > > > the workload balancing etc. that could be done at the iSCSI layer. > > > > > > I am now willing to "roll-over" but I think I owe you all > > my reasons for > > > deciding to support the Asymmetric approach: > > > > > > First, in the degenerate case, they both look the same. > > That is, they can > > > both support a single Connection per Session. So the simplest > > > implementation is still possible with the Asymmetric > > approach as it was > > > with the Symmetric approach. This means that most current iSCSI > > > implementations can continue to exist, unchanged. > > > > The biggest issue with the "asymmetric" model is that it is > > NOT "asymmetric" > > when there is only 1 TCP connection. When there is only one > > TCP connection, it > > is the "symmetric" model - both commands and data on the same > > TCP connection. > > Then, when there is more than one TCP connection, the > > behavior is different. > > It's always easier to implement something that operates that > > same way all the > > time, than two different behaviors. > > > > I propose that the asymmetric model mandate at least two TCP > > connections > > (implies that at least one physical connection will have at > > least two TCP > > connections running on it) - one for commands, the other for > > data. This has > > other advantages, like commands not being flow controlled by > > large transfers of > > data. > > > > -Matt Wakeley > > Agilent Technologies > > > > > > > > > > > But the most important reason follows: > > > > > > Since a session is a unique construct that identifies an > > Initiator and a > > > Target, I am assuming that does NOT mean each NICs has a > > unique initiator > > > ID but that they can, and should, share a common Initiator > > ID. The Target > > > ID, on the other hand, is either unique per Storage > > Controller, or not, > > > and this will depend on the type of Storage Controller. > > This means that > > > the Host iSCSI device driver will be able to present, to > > the layers above > > > it, either a single port per storage controller, or multiple ports, > > > depending on how the Storage Controller answers Logins on > > each of its IP > > > connections. > > > > > > That means iSCSI will not be able to determine when it > > knows enough to > > > perform alternate path retry, and for the same reason will > > not be able to > > > know when or how to perform command work load balancing across the > > > different sessions. This is because either all connections > > to a storage > > > controller have either the same iSCSI session ID (SSID), or they are > > > different -- but iSCSI will not, by itself, know which > > sessions can be > > > substituted for others (for balancing or retry). > > > > > > The only place where the symmetric approach can do workload > > balancing and > > > alternate path retry is where the storage controller makes > > it look like a > > > single target ID even though it might have multiple IP > > addresses. Though > > > this might work in some IBM storage controllers (like > > Shark) it will not > > > work for all IBM storage controllers (like those of Mylex). > > This same or > > > similar argument probably will apply to EMC's and other > > vendors various > > > storage controllers. Therefore, having iSCSI do any type of command > > > balancing or command automatic alternate path retry is problematical > > > without outside influences (probably from upper layers) and > > this will tend > > > to make the iSCSI layer very thick and complicated. It > > will also mix the > > > layering concepts (in general not a good idea). Therefore, > > I now believe > > > that the Symmetrical approach can not be a reasonable > > general/generic > > > approach to Command work load balance or alternate path > > retry, and that > > > function is best left to the Wedge Drivers. > > > > > > The Asymmetric approach can not perform command balancing or command > > > automatic path retry either, but it does not try. It only > > tries to balance > > > the data across multiple connections (and that seems much > > more reasonable). > > > > > > With the Asymmetric approach, data for any specific command > > only flows on > > > one connection. So you can look at the data from various > > commands as > > > independent flows on the different connections (NICs) so > > there should not > > > be blocking issues, or parallel flow issues, and RDMA type > > direct to memory > > > functions can occur. > > > > > > With multiple storage controllers, there may be many > > sessions and many data > > > connections spread across all the various NICs in a Host. > > > > > > The net of all this is that the Asymmetric approach: > > > Probably has good enough spread of commands and data > > across multiple > > > NICs, even without Wedge Drivers. > > > Permits the upper layer Wedge Drivers to be left mostly > > unchanged. > > > Permits the simple case with a single connection per > > session, even with > > > multiple NICs. > > > Permits the most simple case of a single connection per > > session with > > > only a single NIC. > > > > > > Since I have seen no one else that made arguments for the Symmetric > > > implementation, you maybe able to call the agreement on the > > Asymmetric > > > approach a consensus. > > > . > > > . > > > . > > > John L. Hufferd > > From owner-ips@ECE.cmu.edu Fri Sep 08 07:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05990; 8 Sep 2000 2:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA05321 for ips-outgoing; Fri, 8 Sep 2000 01:25:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA05313 for ; Fri, 8 Sep 2000 01:25:12 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id 49544AD9 for ; Thu, 7 Sep 2000 22:25:06 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA03604 for ; Thu, 7 Sep 2000 22:25:04 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA600F; Thu, 7 Sep 2000 22:24:59 -0700 Message-ID: <39B8781F.CAA9B067@agilent.com> Date: Thu, 07 Sep 2000 22:24:47 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Jim McGrath , ips Subject: Re: Command Queue Depth (was asymmetric/Symmetric) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim McGrath wrote: > I agree that BB credit has nothing to do with commands per se, but > illustrates the problem we have had with deciding on policies for the > distributed allocation of device resources over multiple initiators. In my > postscript I noted that the same problems have arisen on command queue (how > many queue slots do you get), with similarly no satisfactory solution. So it sounds to me like this distribution of (command) resources across initiators is a T10 SCSI issue, and should be solved there, not by each individual transport (yesterday FC - which didn't solve it, today iSCSI, tomorrow IB or whatever). You've lost me on the following two paragraphs... Ethernet doesn't require credits to send frames - it's a ship and pray model. I don't know what the latency through ethernet switches is, but I'd hope it wasn't in the ms range. Finally, a "usefull" FC-AL will have many devices on it (say 20 drives in a JBOD) and the latency of the elastic store really starts adding up. So I still contend that FC-AL is not a low latency medium. -Matt > On FC-AL, the latency to get an initial credit is typically measured in us > for a couple of reasons. First, much of that logic has been automated in > the interface hardware (indeed, the major source of delay is typically the > elasticity buffer, which is not store and forward and so has a very low > latency compared to many switches or routers). Second, the distances are > very small (e.g. hundreds of meters), so both transmission time and the > opportunity for intervening devices to increase latency is lower than in the > general internet world. > > So generally 2 credits (of 2Kbyte frames) is enough to cover the latency and > get you into streaming. If the latency of the system was measured in ms, > then a transport on a Gibt wire would require more like 50 initial credits > (or more) to cover the latency. Unless we are designing for a low latency > environment for the exchange of credits (like those where FC-AL are used), > then we probably need to allocate so much buffer space that it becomes > difficult to promise initial credits to a lot of potential initiators. > > Jim > > PS general Fibre Channel (e.g. with switches and the like) is a bit > different. > > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 8:23 PM > To: ips > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > Jim, > > I agree that FC has tried (unsuccessfully) to address this command queue > allocation problem. > > However, the "login bb credit" mechanism in FC does not address the command > queue depth issue at all. BB credit is used to receive commands and/or > data, > and the target has no clue in advance what is coming. BB credit is just > there > to ensure that there is a lowest layer buffer available to receive the FC > frame > (as opposed to dropping it on the floor if there is no "mac" buffer, like > ethernet does). It does not mean the command queue has any room for the > frame. > > At one time, there was a big push to have "data" credits and "command" > credits > to take care of this problem, but it couldn't be made to work and be > "backwards > compatible". > > > The issue of buffer space allocation for multiple initiators has a long > and > > troubled history in SCSI. We have never been able to come up with a good > > answer. > > > > Fibre Channel tried to fix this with the notion of "login BB credit" - > when > > you login you get a minimum number of credits you are always guaranteed > when > > you start data transfers. The problem with this is that storage devices > had > > no realistic ability to discriminate between initiators or to change the > > login BB credit. In addition, the expectation is that all possible > > initiators would get these credits on login. So storage devices vendors > > have played it safe and kept this number low (at 0 until recently, now > > around 2). For iSCSI the number of initial credits you need to "prime the > > pump" until normal data flow is established is probably large (given the > > latencies are higher than in Fibre Channel, especially FC-AL), and the > > How is the latency low on FC-AL?? given that you need to arbitrate and win > the > loop, then receive bb credit, before you can send anything? > > -Matt > > > > > number of potential initiators larger than in Fibre Channel, making this a > > whole lot worse for the storage device. > > > > As soon as we allow the devices to start adjusting these credits, then you > > have the protocol problem of making sure people know when their credits > are > > adjusted and the policy problem of how, who, and when to adjust the > credits. > > Changing everyone's credit when you add a new initiator can get into a > > notification nightmare, although it is "fair." Any policy brings up all > > sorts of nasty issues regarding fairness vs efficient use of the > > transmission media. > > > > Jim > > > > Note: the same problem has plagued other attempts to allocate device > > resources between multiple initiators, like command queue space. In > general > > policies with respect to multiple initiators are not really standard in > the > > SCSI world. > > > > -----Original Message----- > > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > > Sent: Wednesday, September 06, 2000 2:56 PM > > To: ips > > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > > Joshua Tseng wrote: > > > > > James, > > > > > > I agree with others that there may be an issue with the > > > command windowing mechanism in the existing iSCSI spec. It is like > > > "TCP in reverse", in that the target determines the size of the window, > > and > > > not the initiator as in TCP. Rather, I believe that everything that > this > > > windowing mechanism is attempting to achieve can be more easily obtained > > > by having the target communicate its buffer size to the initiator at > > > iSCSI login. It should be the role of the initiator to determine how > > > many commands to put in flight simultaneously, given this input on > > available > > > buffer size from the target. > > > > As more initiators connect to a target, it may need to scale back the > amount > > of > > this buffering it has allocated to each previously logged in initiator (to > > prevent rejecting new logins). > > > > > > > > > > > As far as multiple initiators, could this not be resolved by the target > > > refusing additional logins beyond the number of initiators it can safely > > > support? Not being a storage expert, this is my best guess/suggestion > > > at how to do it. > > > > I believe John already answered this... > > > > -Matt From owner-ips@ECE.cmu.edu Fri Sep 08 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06077; 8 Sep 2000 3:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA06332 for ips-outgoing; Fri, 8 Sep 2000 02:14:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA06328 for ; Fri, 8 Sep 2000 02:14:34 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Thu, 7 Sep 2000 23:14:51 -0700 Message-ID: From: Joshua Tseng To: Matt Wakeley , ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Thu, 7 Sep 2000 23:14:44 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk >As Julian has stated in a different thread, the purpose of the "sliding >windows" in iSCSI is not for congestion management. It is simply there to >handle the case where if a connection goes down in a multiple connection >session, it prevents the remaining connections from overwhelming the target >with new commands that it can't process due to missing commands that where on >the broken connection. Matt, In the event of a failed connection, the sliding window isn't going to help much since you have to first detect the failed connection. Once this is accomplished, the initiator will know which command was lost for sure, but will not know how many, if any, of the subsequent commands sent after the lost command, were also lost. I presume, the initiator will then retransmit all commands without knowing whether or not they were lost, rather than wait for the target to advance the acknowledgement counter. This is all awesome and impressive behavior, exactly what we want. The only problem is this exact same behavior can be had without the windowing mechanism. >Since all of this runs on top of TCP, and TCP performs congestion >management, >why must iSCSI perform congestion management on top of TCP? Please see my reply to John Hufferd. Josh From owner-ips@ECE.cmu.edu Fri Sep 08 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06075; 8 Sep 2000 3:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA06270 for ips-outgoing; Fri, 8 Sep 2000 02:11:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA06263 for ; Fri, 8 Sep 2000 02:11:13 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Thu, 7 Sep 2000 23:11:31 -0700 Message-ID: From: Joshua Tseng To: Yaron Klein , Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: security model Date: Thu, 7 Sep 2000 23:11:22 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Yaron, >However, I believe that there is no escape from defining an iSCSI security. Due >to the need for data integrity and the infeasibility of it in Gbit Ethernet, we >must implement some digestion mechanism to insure it. This feature cannot be >implemented with IPSEC. Can you explain what you mean by "This feature cannot be implemented with IPSEC."? Are you aware of the IP Authentication Header (AH) protocol defined by IPSec (RFC 2402)? Josh -----Original Message----- From: Yaron Klein [mailto:klein@sanrad.com] Sent: Thursday, September 07, 2000 5:34 AM To: Joshua Tseng Cc: Dave Nagle; ips@ece.cmu.edu Subject: Re: security model Hi Joshua, We are now investigating the IPSEC and how to integrate it in the security mechanism. Our considerations are: The security scheme should be as flexible as possible. One of the options in the negotiation will be IPSEC. However, additional options will be available. The security is per session, not per TCP connection (established in the login phase). Regarding IPSEC policies, I m checking the possibility of defining a policy for iSCSI. It may be something proprietary if the standardization procedure will take to long. However, I believe that there is no escape from defining an iSCSI security. Due to the need for data integrity and the infeasibility of it in Gbit Ethernet, we must implement some digestion mechanism to insure it. This feature cannot be implemented with IPSEC. Hope to get the security and login chapter soon, so we can all discuss it. Any ideas are more than welcomed. Yaron Klein SANRAD Joshua Tseng wrote: > Yaron, > > My suggestion WRT security is to take a good look at IPSec. > IPSec has several advantages I can think of over SSL or TLS. > > 1) Standardized key distribution methodologies already exist > (ISAKMP/IKE) and have been implemented for use with IPSec. > > 2) IPSec has the advantage that it is flexible, and security > policies can be granularly applied to each iSCSI conversation, > even when multiple conversations are going over the same TCP > connection. iSCSI can thus decide to encrypt only data PDU's, > or only certain conversations, etc... The precise granularity > is implementation-specific. > > With TLS, everything sent through a TCP socket will use the > same security policy. If multiple iSCSI conversations are > taking place over the same TCP connection, then all will be > uniformly handled by the same security policy, whether it's > 3DES or no encryption at all. If this is desired, then that's > fine, but I believe there will be cases where the user wants > to only authenticate Command PDU's, while encrypting & > authenticating data PDU's, for example. > > The following is an excerpt from section 4.4.1 of RFC 2041 > (Security Architecture for IP). In this case, "application" > refers to iSCSI. > > "In host systems, applications MAY be allowed to select what security > processing is to be applied to the traffic they generate and consume. > (Means of signalling such requests to the IPsec implementation are > outside the scope of this standard.) However, the system > administrator MUST be able to specify whether or not a user or > application can override (default) system policies. Note that > application specified policies may satisfy system requirements, so > that the system may not need to do additional IPsec processing beyond > that needed to meet an application's requirements. The form of the > management interface is not specified by this document and may differ > for hosts vs. security gateways, and within hosts the interface may > differ for socket-based vs. BITS implementations. However, this > document does specify a standard set of SPD elements that all IPsec > implementations MUST support." > > -----Original Message----- > From: Yaron Klein [mailto:klein@sanrad.com] > Sent: Sunday, September 03, 2000 3:53 AM > To: Dave Nagle > Cc: ips@ece.cmu.edu > Subject: Re: security model > > We are now working on the on integrating the security model in the iSCSI > draft. > Some remarks considering your model: > > The data privacy (number 5) should be solved in a higher level, i.e., the > block > should be encrypted or digested by some hash before it is sent to the > target. > This issue is not in the scope of the transfer protocol. > > Number 2 should be expanded to initiator and target masquerading. If the > enemy > masquerades as the target, he can get all our secrets... > > Furthermore, the security considerations are affected by the performance > consideration, i.e., the computation power and the transfer rate. Thus, in > high-speed network, compromise should be made (it is not feasible or very > expensive to totally secure the data in 1G). The security model should > adjust > itself to the nature of the iSCSI features. It should enable partial > security > (only command, only headers, digestion of data etc.) > > We are now in an intensive work to integrate the security issue in the next > draft. The login chapter will enable to establish a flexible and negotiable > security/authentication model for the iSCSI session. > > Any ideas, comments and solution regarding security, let me know, > > Reagrds, > > Yaron > > Dave Nagle wrote: > > > From: Banu Ozden & Mike Reiter > > > > We suggest the attached extensions to the security model proposed in > > the iSCSI draft (Section 6). > > > > The existing iSCSI security model covers "communication security" > > between an initiator and a target. It does not address "data > > security". Data security provides protection against possible attacks > > to the data stored at the target. These include threats like > > unauthorized disclosure of data at the target to administrators or > > other clients of the target, and unauthorized modification of data at > > the target. > > > > Our main motivation is to enhance the security model for storage > > outsourcing environments where the Storage Service Provider (SSP) > > personnel is not necessarily trusted or where the sharing of target > > between different customers of the SSP raises a security concern. > > > > We are working on a security architecture for storage outsourcing. > > We would like to know whether there is interest on including data > > security considerations to iSCSI in addition to communication > > security. > > > > Banu Ozden & Mike Reiter > > > > Bell Labs > > 600 Mountain Ave. > > Murray Hill, NJ 07974 > > http://www.bell-labs.com/who/ozden > > http://www.bell-labs.com/who/reiter > > > > _______________________________________________________________ > > Threat Model > > > > T1. Disclosure of message contents to an eavesdropper intercepting > > communication between an initiator and a target. > > > > T2. An attacker masquerading as the initiator to a target or the target > > to an initiator. This includes an attacker manipulating communication > > between an initiator and a target, e.g., to introduce false messages, > > modify passing messages, or delete messages. > > > > T3. Disclosure of data to personnel maintaining the target or to other > > customers of the target. > > > > T4. The modification of data by the target or other customers of the > > target. > > > > Security Model > > > > 1. No Security (same as described in the iSCSI draft) > > > > This mode does not authenticate nor does it encrypt data. This mode > > should be used in environments where there is minimal security risk > > and little chance for configuration errors. > > > > 2. Entity Authentication (referred to as End-to-End Authentication in the > > iSCSI draft) > > > > The initiator's and/or target's identity is authenticated. > > Once the client is authenticated, all messages are > > sent and received in the clear. This mode should only be used when > > there is minimal risk to man-in-the-middle attacks, > > eavesdropping, message insertion, deletion, and modification. > > > > 3. Message Integrity (new) > > > > This mode protects against T2 types of threats. It provides > > communication integrity. > > > > 4. Message Integrity Combined with Encryption (referred to as Encryption > in > > the iSCSI draft) > > > > This mode protects against threats T1 and T2. Thus, it provides > > communication integrity and communication privacy. It protects > against > > man-in-the-middle attacks, eavesdropping, message insertion, > deletion, > > and modification. > > > > 5. Data Privacy (new) > > > > This mode protects against T3 types of threats. The initiator > > encrypts/decrypts data. The target stores encrypted data. > > > > 6. Data Privacy with Data Integrity > > > > This mode protects against threats T3 and T4. > > > > 7. Some combinations of the above security options > > > > For example, data privacy with message authentication (5 & 3) > > protects against threats T1, T2 and T3. From owner-ips@ECE.cmu.edu Fri Sep 08 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06084; 8 Sep 2000 3:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA06309 for ips-outgoing; Fri, 8 Sep 2000 02:13:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA06304 for ; Fri, 8 Sep 2000 02:13:51 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Thu, 7 Sep 2000 23:14:11 -0700 Message-ID: From: Joshua Tseng To: John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Thu, 7 Sep 2000 23:14:08 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, TCP provides congestion management for the network. The command windowing mechanism is supposed to provide congestion management for the target's command queue, I presume. Another way to describe it is command flow control. Well, TCP will NOT fulfill this role unless iSCSI can actively tell TCP to halt ACK responses for TCP segments. Now, I don't think anybody is advocating this (right???). I do not know if a command flow control mechanism is necessary, but I do believe that the existing sliding window mechanism is a whole lot of work to do something that can be accomplished more easily by other means. Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Thursday, September 07, 2000 4:29 PM To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session Folks, Matt and Julian are correct. The sliding window was put in for a completely different reason then is currently being discussed. I suggest that we all look at it only as it was originally intended and then see if something else needs to be done, or not. TCP/IP probably handles all the other issues (well enough). . . . John L. Hufferd "Matt Wakeley" @ece.cmu.edu on 09/07/2000 03:46:53 PM Please respond to Matt Wakeley Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: a vote for asymmetric connections in a session As Julian has stated in a different thread, the purpose of the "sliding windows" in iSCSI is not for congestion management. It is simply there to handle the case where if a connection goes down in a multiple connection session, it prevents the remaining connections from overwhelming the target with new commands that it can't process due to missing commands that where on the broken connection. Since all of this runs on top of TCP, and TCP performs congestion management, why must iSCSI perform congestion management on top of TCP? -Matt Wakeley Agilent Technologies Scott Bradner wrote: > > Implementing sliding windows is not that hard > > note that the issue is not "just" sliding windows - ips also has to deal > with congestion TCP-friendly way - that can get quite complicated > > Scott From owner-ips@ECE.cmu.edu Fri Sep 08 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06485; 8 Sep 2000 4:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA07711 for ips-outgoing; Fri, 8 Sep 2000 03:26:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA07705 for ; Fri, 8 Sep 2000 03:25:59 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id DAA28130; Fri, 8 Sep 2000 03:24:10 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id BAA88328; Fri, 8 Sep 2000 01:25:57 -0600 Importance: Normal Subject: RE: a vote for asymmetric connections in a session To: Joshua Tseng , Charles Monia Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 8 Sep 2000 00:20:26 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/08/2000 01:25:57 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, Charles, I am not sure how to reply to you, but let me try. I think we are combining several items (perhaps layers) together in this discussion. Again, without going into all the details (and whether they were right or wrong) the reason for the Sliding Window was as Julian stated. Now without the need to recover from the problem of failures in another connection, the sliding window is not needed for what it was setup to do. I could be mistaken here but I think you, and some others have been addressing issues with the Storage Controller having the right amount of buffer space in the controllers for the commands, after they are delivered by iSCSI. I think this is a SCSI issue and not a iSCSI/TCP flow control problem. Originally we were trying to solve a set of problems that were caused by the Symmetric model which (for this discussion) was independent of memory problems caused by receiving the commands from iSCSI. We were concerned with memory problems that might be caused by either the subset of memory allocated to iSCSI, or the memory available on the NIC at the Target (perhaps also the Initiator). With respect to the Asymmetric approach, the issues change, and we are no longer trying to solve the problem of missing commands that occur because of a broken connection. Therefore, I think we can dump the sliding window and leave the flow control -- and recovery of lost commands, -- up to TCP. Yes, the SCSI layers on each end need to have their own buffer management under control, but I do not think this is a transport issue. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/07/2000 11:14:08 PM Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS, ips@ece.cmu.edu cc: Subject: RE: a vote for asymmetric connections in a session John, TCP provides congestion management for the network. The command windowing mechanism is supposed to provide congestion management for the target's command queue, I presume. Another way to describe it is command flow control. Well, TCP will NOT fulfill this role unless iSCSI can actively tell TCP to halt ACK responses for TCP segments. Now, I don't think anybody is advocating this (right???). I do not know if a command flow control mechanism is necessary, but I do believe that the existing sliding window mechanism is a whole lot of work to do something that can be accomplished more easily by other means. Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Thursday, September 07, 2000 4:29 PM To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session Folks, Matt and Julian are correct. The sliding window was put in for a completely different reason then is currently being discussed. I suggest that we all look at it only as it was originally intended and then see if something else needs to be done, or not. TCP/IP probably handles all the other issues (well enough). . . . John L. Hufferd "Matt Wakeley" @ece.cmu.edu on 09/07/2000 03:46:53 PM Please respond to Matt Wakeley Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: a vote for asymmetric connections in a session As Julian has stated in a different thread, the purpose of the "sliding windows" in iSCSI is not for congestion management. It is simply there to handle the case where if a connection goes down in a multiple connection session, it prevents the remaining connections from overwhelming the target with new commands that it can't process due to missing commands that where on the broken connection. Since all of this runs on top of TCP, and TCP performs congestion management, why must iSCSI perform congestion management on top of TCP? -Matt Wakeley Agilent Technologies Scott Bradner wrote: > > Implementing sliding windows is not that hard > > note that the issue is not "just" sliding windows - ips also has to deal > with congestion TCP-friendly way - that can get quite complicated > > Scott From owner-ips@ECE.cmu.edu Fri Sep 08 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07397; 8 Sep 2000 7:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA19496 for ips-outgoing; Fri, 8 Sep 2000 06:26:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA19477 for ; Fri, 8 Sep 2000 06:26:48 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA56924 for ; Fri, 8 Sep 2000 12:26:15 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA97066 for ; Fri, 8 Sep 2000 12:26:15 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256954.00395309 ; Fri, 8 Sep 2000 12:26:08 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 8 Sep 2000 13:20:05 +0300 Subject: Re: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Why would the motivation to do recovery decrease? With multiple links you have to do recovery/fail-over to keep the MTBF from nosediving. The motivation stay the same (and the question on where to do it also). It is not different than in the symmetric case and we keep track of closed action trough the status numbering scheme. Julo csapuntz@cisco.com on 06/09/2000 20:56:30 Please respond to csapuntz@cisco.com To: Matt Wakeley cc: ips@ece.cmu.edu, csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple connections > > This proposal allows removing the command counters - as commands use a > > single TCP connection. The single connection can also be a shared > > data+control connection. > > For the case of the command connection failure and fail-over to a new > connection, I don't see how you can get away from the command counters. When a > fail over occurs, you will need some way of finding out what commands made it > to the target and which didn't. The easiest way to do this is with command > numbering. > I believe the initiator task tag could be used for recovery. You still need StatRN's though, unless you want to ACK each status. With only one control connection, maybe our motivation for doing recovery decreases. -Costa From owner-ips@ECE.cmu.edu Fri Sep 08 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07385; 8 Sep 2000 7:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA19501 for ips-outgoing; Fri, 8 Sep 2000 06:27:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA19491 for ; Fri, 8 Sep 2000 06:26:54 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id MAA24804 for ; Fri, 8 Sep 2000 12:26:22 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA19118 for ; Fri, 8 Sep 2000 12:26:20 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256954.00395026 ; Fri, 8 Sep 2000 12:26:00 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 8 Sep 2000 12:51:49 +0300 Subject: Re: Multiple Connections. How would they work? Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The 00 draft (Adelaide) had the commands go down over one specific control stream while the data went on any available connection. Synchronizing them and providing for hardware implementation across several adapters was felt to be complex. However command ordering was trivial. The symmetric model in 01 requires ordering and this is why you have the command ordering. A proposal to get back to the asymmetric model but decide ahead of time (? when) what path data will be taking was forwarded by Kalman. We call this the asymmetric model (it can still evolve). I hope this helps. Regards, Julo Robert Snively on 06/09/2000 18:46:40 Please respond to Robert Snively To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Multiple Connections. How would they work? What am I missing? The reason I felt that iSCSI might possibly work was the text from 2.2.4 of draft-satran-iscsi-01.txt. " For SCSI commands that require data and/or parameter transfer, the (optional) data and the status for a command must be sent over the same TCP connection that was used to deliver the SCSI command (connection allegiance). Thus if an initiator issues a READ command, the target must send the requested data followed by the status to the initiator over the same TCP connection that was used to deliver the SCSI command." >>From this, I extrapolated that multiple connections and sessions were mechanisms to manage multiple paths, each of which would carry all the traffic related to a single command, probably with no ordering among them. I had expected TCP/IP and iSCSI state required for execution of an iSCSI operation to be contained within a single NIC. How does this map into the multiple connection structures that are being discussed? How do the multiple connection structures map into the low cost highly efficient hardware assisted NIC implementations that presently typify SCSI implementations? From owner-ips@ECE.cmu.edu Fri Sep 08 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07383; 8 Sep 2000 7:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA19494 for ips-outgoing; Fri, 8 Sep 2000 06:26:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA19478 for ; Fri, 8 Sep 2000 06:26:48 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA82662 for ; Fri, 8 Sep 2000 12:26:16 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA97068 for ; Fri, 8 Sep 2000 12:26:16 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256954.003952C6 ; Fri, 8 Sep 2000 12:26:07 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 8 Sep 2000 13:14:05 +0300 Subject: Re: iSCSI version info Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Next draft will have version support in the login phase. I though that there is a wide consensus on this but I did not manage to get it in 01. Regards, Julo Dave Thompson on 06/09/2000 20:51:45 Please respond to Dave Thompson To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: iSCSI version info Julian, A version negotiation scheme was suggested by Mark Bakke's 29 Jun 2000 e-mail. > Logins for iSCSI draft 03 might benefit from the following additions: > > 1. Version number text fields should be placed in both login requests > and responses. the request could either specify the version of the > protocol the initiator wishes to run, or perhaps a list of versions > that the initiator is capable of running. > > Version: 1.0 > > -OR- > > Version: 1.0 1.1 1.2 > > The target would then choose the highest version that it supported > in common with those requested, and return it in the response: > > Version: 1.1 Is there any plans to support a mechanism like this in the next iSCSI draft? In addition, perhaps a one byte field in the login command and response headers (byte 3?) could be added which would contain the major protocol version number. This could then be used to determine the proper format of the header in cases where major header layout changes occur (e.g. the last draft's size change from 40 to 48 bytes). -- Dave Thompson NuSpeed, Inc. dave.thompson@nuspeed.com From owner-ips@ECE.cmu.edu Fri Sep 08 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07392; 8 Sep 2000 7:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA19499 for ips-outgoing; Fri, 8 Sep 2000 06:27:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA19487 for ; Fri, 8 Sep 2000 06:26:52 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA313852 for ; Fri, 8 Sep 2000 12:26:20 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA19116 for ; Fri, 8 Sep 2000 12:26:05 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256954.00395072 ; Fri, 8 Sep 2000 12:26:01 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 8 Sep 2000 12:56:04 +0300 Subject: RE: bridging issues -- Converging FCP-2 and iSCSI control struct ures Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, Wow! - this is a detailed note asking for a detailed answer. I will have to wait a little (not too much tough). Thanks, Julo From owner-ips@ECE.cmu.edu Fri Sep 08 12:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07562; 8 Sep 2000 7:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA20383 for ips-outgoing; Fri, 8 Sep 2000 07:15:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA20378 for ; Fri, 8 Sep 2000 07:15:09 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id HAA04644; Fri, 8 Sep 2000 07:14:53 -0400 (EDT) Date: Fri, 8 Sep 2000 07:14:53 -0400 (EDT) From: Scott Bradner Message-Id: <200009081114.HAA04644@newdev.harvard.edu> To: matt_wakeley@agilent.com Subject: Re: Command Queue Depth (was asymmetric/Symmetric) Cc: ips@ece.cmu.edu Sender: owner-ips@ece.cmu.edu Precedence: bulk > I don't know what the > latency through ethernet switches is, but I'd hope it wasn't in the ms range. processing time is in the 10 to 50 microsecond range Scott From owner-ips@ECE.cmu.edu Fri Sep 08 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07817; 8 Sep 2000 8:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA21200 for ips-outgoing; Fri, 8 Sep 2000 07:57:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA21196 for ; Fri, 8 Sep 2000 07:57:40 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA01088; Fri, 8 Sep 2000 06:58:06 -0500 Message-ID: <39B8D44E.E47030EC@stewart.chicago.il.us> Date: Fri, 08 Sep 2000 06:58:06 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Charles Monia CC: Matt Wakeley , ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles Monia wrote: > I'm not sure what is meant by "congestion." If we're talking about > congestion in the TCP/IP transport, I'm in agreement. However, I thought we > were referring to the sort of congestion that the application on top of > iSCSI might see if it received more commands than it had room for. > > Unless I misunderstood your point, threfore, I think there might be an > issue. The only way I can see flow control in the tranport layer being used > to avoid dropping commands is if higher layer congestion results in back > pressure to the iSCSI pipe. I believe that behavior is undesirable because > it introduces "head-of-line" blocking, with the following consequences: > Any time you get ANY retransmissions in TCP you will create a "head-of-line" blocking scenario. This does not matter if it is from flow control as you state or network congestion where a packet is dropped by a router. You will have situations where the TCP is holding data waiting for the retransmissions of an earlier packet... this is one of the reasons that sigtran developed SCTP... > a) It effectively shuts down the flow of commands to all logical units. > > b) It blocks the flow of task management commands (Abort Task, Clear task > set, etc). > > > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 08 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08950; 8 Sep 2000 13:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA02756 for ips-outgoing; Fri, 8 Sep 2000 12:20:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA02751 for ; Fri, 8 Sep 2000 12:20:22 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA204220 for ; Fri, 8 Sep 2000 18:19:51 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA47860 for ; Fri, 8 Sep 2000 18:19:51 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256954.0059B331 ; Fri, 8 Sep 2000 18:19:45 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 8 Sep 2000 16:41:55 +0300 Subject: Re: a vote for asymmetric connections in a session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I assume that many of us are aware of ECN and the additional requirements to endpoint TCP stack. As iSCSI is destined for a wide variety of environments (very much like TCP is) I think that our position on it was that we can leave those requirements to TCP and require only an "effective TCP support" for the specific physical infrastructure. Imagine only that for reasons of end-to-end connectivity we will want to support iSCSI (over TCP) in an Inifiniband network - where ECN is neither supported or required. Would we want to require ECN? Julo Please respond to Fred Baker To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: a vote for asymmetric connections in a session At 02:10 PM 9/6/00 -0400, Mike Kazar wrote: >I think it might be illuminating to try to figure out some algorithms to >keep the throughput high even when one of the connections drops a packet >and cuts its TCP window size down significantly. dumb question. Has anyone looked into the ECN proposal? It seems like the issue here is the impacts of loss in the TCP state machine, and those can be mitigated by managing the queues actively. You could assert that you want ECN to be implemented in any iSCSI path, which should have the effect of managing queue depths without incurring loss. http://www.isi.edu/in-notes/rfc2481.txt From owner-ips@ECE.cmu.edu Fri Sep 08 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08943; 8 Sep 2000 13:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA02789 for ips-outgoing; Fri, 8 Sep 2000 12:20:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA02762 for ; Fri, 8 Sep 2000 12:20:36 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA08034 for ; Fri, 8 Sep 2000 18:20:03 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA81542 for ; Fri, 8 Sep 2000 18:20:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256954.0059B5BC ; Fri, 8 Sep 2000 18:19:52 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 8 Sep 2000 17:17:55 +0300 Subject: Re: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk My conceptual model was more like a chained list that directs the setup on all NICS and enforces sequence. Please note that several such chains can be "executed" in parallel as the sequence is important only whithin a list. Obviously the rgular caution applies as you have several "hands" on your NIC cards. And the chain excution means only setup. Julo "Matt Wakeley" on 07/09/2000 00:40:28 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: multiple connections julian_satran@il.ibm.com wrote: > Command numbering won't be needed anymore - every command with an > outstanding status is restarted. I suppose that could be done, as long as the "retry" bit was still there, so there was no chance of a command being re-issued while the status is in flight back to the initiator... > As for the 2 commands - synch can be achieved by setting data first and then > sending commands. My point was that if there are two NICs using some kind of "mailbox" interface to the system (like I2O), the host will have to put the data command describing the scatter/gather list in the data NIC's mailbox, and wait for some kind of notification that the NIC has pulled the message from the mailbox (can you say extra unnecessary interrupt?) before it can post the command to the control NIC's mailbox. If this synchronization was not done, there would be a small possibility that the command could be sent to the target and the target respond with the data before the data NIC has set itself up to receive and properly place the data. > > > Julo -Matt From owner-ips@ECE.cmu.edu Fri Sep 08 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08941; 8 Sep 2000 13:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA02772 for ips-outgoing; Fri, 8 Sep 2000 12:20:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA02758 for ; Fri, 8 Sep 2000 12:20:35 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA204236 for ; Fri, 8 Sep 2000 18:20:03 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA81540 for ; Fri, 8 Sep 2000 18:20:01 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256954.0059B5FA ; Fri, 8 Sep 2000 18:19:52 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 8 Sep 2000 17:27:31 +0300 Subject: Re: iSCSI version info Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, I think that there things in iSCSI for which speed is only marginally critical, are bound to be implemented in some form of software always and will evolve. Examples that keep coming back to us are authentication, key-exchange for security, some name mapping. For those it does no harm to have a version number specified at login. For the rest it will be painful if we keep revving numbers - so no version numbers in regular headers. Regards, Julo Stephen Bailey on 07/09/2000 02:36:07 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI version info > A version negotiation scheme was suggested by Mark Bakke's 29 Jun > 2000 e-mail. > > Is there any plans to support a mechanism like this in the next iSCSI draft? I can never figure out which side of this argument to be on. The basic version idea seems sound, but the problem is that for various reasons, the version number never ends up getting moved. For example, if I recall, FC-PH seemed to go through many version numbers long before it was ever widely implemented, and now it's stabilized at version 4.3. I have never known any implementations that did anything other than 4.3. In ST, we went back and forth numerous times. We ultimately decided that we were going to stipulate that a reved protocol was a new protocol, so no version negotiation would be necessary. The logic for ST went that hardware accelerated protocols are on thin ice if they think they can rev. You're almost always going to work around existing protocol bugs before you ever `fix' the protocol, so the standards committees better be sure of themselves before they forward. I think iSCSI is in this same position. FWIW Steph From owner-ips@ECE.cmu.edu Fri Sep 08 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08955; 8 Sep 2000 13:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA01220 for ips-outgoing; Fri, 8 Sep 2000 11:45:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA01215 for ; Fri, 8 Sep 2000 11:44:59 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id KAA79348; Fri, 8 Sep 2000 10:38:40 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id JAA46482; Fri, 8 Sep 2000 09:44:57 -0600 Importance: Normal Subject: Re: a vote for asymmetric connections in a session To: "Randall R. Stewart" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 8 Sep 2000 08:42:12 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/08/2000 09:44:57 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Randall, Your head-of-line blockage statement, is of course true, but you say it like it is a bad thing. I am not sure it is. In the Asymmetric configuration, there is a different TCP/IP connections from each Host to a Storage Controller to handle the Commands from that Host to the Storage Controller (or at least that Storage Controller's Port). Each Host has SCSI commands that are not related to the commands in other hosts flowing to the same Storage Controller (port) but on separate TCP/IP connections. They expect to have their commands handled in some "fair" allocation method, with respect to, the other Hosts. Nothing is blocking its "line" except its own Commands. So TCP/IP has delivered, or brought up to be delivered, each Hosts commands independently. Now it is up to the Target SCSI layer to chose which port/interface from which to bring additional commands into its buffers. I do not see this as an important problem, and is the way things have worked regardless of the storage connections, for as long as we have had storage connections. It was true on the SCSI buss, and it was even true on the Old IBM 360s (and continues to this day on the mainframes). There will always be back pressure and sooner or later it causes the Initiator to stop sending new packets, since they are not being acknowledged, this will either occur at the TCP/IP layer or at the SCSI layer. I am not sure that this is a bad thing, especially since we are not talking about dead locks. . . . John L. Hufferd "Randall R. Stewart" @ece.cmu.edu on 09/08/2000 04:58:06 AM Sent by: owner-ips@ece.cmu.edu To: Charles Monia cc: Matt Wakeley , ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session Charles Monia wrote: > I'm not sure what is meant by "congestion." If we're talking about > congestion in the TCP/IP transport, I'm in agreement. However, I thought we > were referring to the sort of congestion that the application on top of > iSCSI might see if it received more commands than it had room for. > > Unless I misunderstood your point, threfore, I think there might be an > issue. The only way I can see flow control in the tranport layer being used > to avoid dropping commands is if higher layer congestion results in back > pressure to the iSCSI pipe. I believe that behavior is undesirable because > it introduces "head-of-line" blocking, with the following consequences: > Any time you get ANY retransmissions in TCP you will create a "head-of-line" blocking scenario. This does not matter if it is from flow control as you state or network congestion where a packet is dropped by a router. You will have situations where the TCP is holding data waiting for the retransmissions of an earlier packet... this is one of the reasons that sigtran developed SCTP... > a) It effectively shuts down the flow of commands to all logical units. > > b) It blocks the flow of task management commands (Abort Task, Clear task > set, etc). > > > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 08 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09781; 8 Sep 2000 15:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA08964 for ips-outgoing; Fri, 8 Sep 2000 14:55:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from magic.adaptec.com (magic.adaptec.com [208.236.45.80]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA08957 for ; Fri, 8 Sep 2000 14:54:55 -0400 (EDT) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id LAA18145 for ; Fri, 8 Sep 2000 11:54:01 -0700 (PDT) Received: from aimexc03.corp.adaptec.com ([162.62.62.43]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id LAA09746 for ; Fri, 8 Sep 2000 11:51:02 -0700 (PDT) Received: by aimexc03.corp.adaptec.com with Internet Mail Service (5.5.2650.21) id ; Fri, 8 Sep 2000 11:54:00 -0700 Message-ID: <5B0C98C89640D311A319002048406A1C05B31CDF@aimexc03.corp.adaptec.com> From: "VonStamwitz, Paul" To: ips@ece.cmu.edu Subject: RE: Symmetric vs Asymmetric Date: Fri, 8 Sep 2000 11:53:59 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > "VonStamwitz, Paul" wrote: > > I agree with David. I oppose a two connection minimum. > [Matt Wakeley] > I disagree. Initial implementations and cheap hardware will > have the iSCSI > implementation in software. It is easy to do anything in > software - open a > thousand connections, perhaps hundreds of thousands, one per > each LU, it > doesn't care - it just uses up more memory. > > However, for the high performance implementations that will > implement iSCSI in > hardware, doing a function one way sometimes, and another > other times, will > require more hardware and more testing (by both the hardware > vendors and the > value add vendors) to test both functions. > If immediate data on writes is supported, won't you have to support both functions anyways? If the target (or initiator) chooses which connection to transfer data, why can't the command/status connection be one of the choices? > One connection per LU also has the huge issue of requiring > lots of on-hardware > resources and memory to maintain the hundreds or thousands of > connections that > will be required.... and you want a cost competitive solution? > I stated before that the resource issues preclude mandating a connection per LU. But if resources allow it, it's not a bad way to operate. -Paul From owner-ips@ECE.cmu.edu Fri Sep 08 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09766; 8 Sep 2000 15:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09191 for ips-outgoing; Fri, 8 Sep 2000 14:58:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA01746 for ; Fri, 8 Sep 2000 11:55:26 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Fri, 8 Sep 2000 08:56:32 -0700 Message-ID: From: Robert Snively To: "'Scott Bradner'" , matt_wakeley@agilent.com Cc: ips@ece.cmu.edu Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Fri, 8 Sep 2000 08:53:49 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Memory to memory time can be significant fractions of a second, depending on the stack implementation and connection path. Present disk drives routinely exceed 200 IO operations/second and a RAID box with less than 50 or so drives is considered small. The implication is that command queues will grow into the small numbers of thousands relatively routinely, if full random access throughput is to be maintained. Bob > -----Original Message----- > From: Scott Bradner [mailto:sob@harvard.edu] > Sent: Friday, September 08, 2000 4:15 AM > To: matt_wakeley@agilent.com > Cc: ips@ece.cmu.edu > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > > I don't know what the > > latency through ethernet switches is, but I'd hope it > wasn't in the ms range. > > processing time is in the 10 to 50 microsecond range > > > Scott > From owner-ips@ECE.cmu.edu Fri Sep 08 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09761; 8 Sep 2000 15:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA07874 for ips-outgoing; Fri, 8 Sep 2000 14:25:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA07859 for ; Fri, 8 Sep 2000 14:25:37 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by atlrel1.hp.com (Postfix) with ESMTP id D3F91355; Fri, 8 Sep 2000 14:25:32 -0400 (EDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id LAA27707; Fri, 8 Sep 2000 11:25:32 -0700 (PDT) X-OpenMail-Hops: 1 Date: Fri, 8 Sep 2000 11:25:21 -0700 Message-Id: Subject: RE: RE: Command Queue Depth (was asymmetric/Symmetric) MIME-Version: 1.0 To: cmonia@NishanSystems.com, Jim.McGrath@quantum.com, julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I think you and Charles have important points on this. First it is really important to reexamine the issue of communicating command queue depth given the way TCP/IP works, and also the fact that memory is much much cheaper than it used to be and is probably going to get cheaper still. If we have a way to indicate the command queue depth (like the sequence # mechanism in iSCSI), this lets the target provide an indication of the command queue depth to the initiator (the TCP window size provided by the target to the initiator will indicate how many bytes of data and commands can be pumped on this connection in that direction). If the device is a small device (perhaps like a drive), it will have very few connections, maybe one. In this case it has to make sure that the credit it extends is based on actual command buffer space (and the same goes for window size), (and perhaps processing rate and network latency?). If however, it is a large device with a large number of connections, it will have to do some over-subscription. And if there starts to be a backup of commands or data, the command queue depth indication and the TCP window size can start to be pinched. And of course, the worse case is when things overflow, and then have to be tossed. In this case, as someone else recommended, an indication that overflow has occurred should be sent to the initiator. Then all the commands get thrown away till an indication is received from the initiator that it is going to start resending the commands from the point at which the commands were thrown away. Again the command sequence #s come in handy. There is little need to optimize this last part through SACKs etc, hoping that the flow control provided will avoid this situation from happening too many times. Somesh > -----Original Message----- > From: Jim.McGrath@quantum.com [mailto:Jim.McGrath@quantum.com] > Sent: Thursday, September 07, 2000 8:44 PM > To: cmonia@NishanSystems.com; julian_satran@il.ibm.com > Cc: ips@ece.cmu.edu > Subject: FW: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > The path I'd recommend is to allow people to oversubscribe a target's > resources, and then to do a graceful recovery when that gets you into > trouble. > > Note that drive designers do this all of the time - we tend > to optimize for > common cases, and then worry about how to handle outlyers using other > mechanisms. While a more complicated model, it gets you the > best overall > resource utilization. We use read on arrival, ECC on the > Fly, Retrys, auto > reallocation, all in attempts to handle the common path > quickly and the rare > path more slowly, where the differences are error rates. > > In this case I would allow initiators to send down data > immediately - when > that works (like when ECC on the Fly works) you get a > benefit. If packets > are dropped you can rely on existing mechanisms to recover, > or you can put > in a new, improved, and perhaps more friendly process. In > either case I > think the result would probably be better than a tight credit > based model > where a lot of delays would be introduced and a lot of (historically > unresolved) allocation policy issues arise. > > Jim > > > > -----Original Message----- > From: Charles Monia [mailto:cmonia@NishanSystems.com] > Sent: Thursday, September 07, 2000 1:52 PM > To: Julian Satran (E-mail) > Cc: Ips (E-mail) > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > Hi Julo: > > > -----Original Message----- > > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > > Sent: Thursday, September 07, 2000 6:11 AM > > To: ips@ece.cmu.edu > > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > > > > > Dear colleagues, > > > > Although the windowing mechanism in iSCSI-01 may seem to be > > there to solve > > a queueing issue > > it is mainly meant to limit the buffering space for commands > > that await > > "de-skewing". > > We assume that execution queue-lengths, policy etc. are > > beyond the scope of > > transport. > > > > As for SCSI queue length I assumed that the busy or queue > full status > > followed by an Asynch Event > > message indicating readiness is the mechanism provided by > > SCSI to regulate > > the command flow. > > > > It is hard to imagine that give the variable life-time of > > SCSI commands and > > the > > opaque nature of the resources required to execute them that > > the transport > > has > > to help in this area. > > > > While this issue has been discussed at some length in the past, as Jim > McGrath stated, I believe the debate ought to be reopened > (although we may > end up reaching the same conclusion as before). > > As Ralph Weber pointed out, the SCSI model is to discard the > command, return > status, retrieve the next command in the transport pipeline > and continue > processing. The initiator is not notified when processing > resumes. If there > are many commands in flight, as there could be in an IP > environment, and > target resources free up in the meantime, the result is > commands processed > out of order. > > Historically, such a lapse in command ordering was not seen > as an issue for > the following reasons: > > a) Strict ordering was not required by the most commonly > deployed device > types (disks and tapes). Due to the nature of disk traffic, a > simple retry > mechanism was deemed sufficient to recover from these errors. > Since legacy > streaming devices, such as tapes, did not support command > queuing, command > ordering considerations were not a factor there either. > > b) Transport delays over storage interconnects were small, so not many > commands were apt to be in flight. i.e.. The window for such > errors was very > small. > > c) The resource guarantees needed for a loss-avoidance > mechanism in the > target adversely effected device cost, especially at the high-volume, > low-end of the market. > > Given the above considerations, there was little support > within the storage > community for measures addressing this issue. > > If we now believe that the iSCSI environment changes the > rules, I believe > the interconnect protocol can provide useful assists, such as: > > a) On a command overflow condition, have the iSCSI target > flush the command > pipeline by returning status and discarding all subsequently received > commands until a host acknowledgement is received. > > b) Implement some sort of credit-based mechanism for > overflow-avoidance. > > > Comments? > > > > > > Jim McGrath on 07/09/2000 06:06:40 > > > > Please respond to Jim McGrath > > > > To: "'Matt Wakeley'" , ips > > > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > > > > > > > > > The issue of buffer space allocation for multiple initiators > > has a long and > > troubled history in SCSI. We have never been able to come up > > with a good > > answer. > > > > Fibre Channel tried to fix this with the notion of "login BB > > credit" - when > > you login you get a minimum number of credits you are always > > guaranteed > > when > > you start data transfers. The problem with this is that > > storage devices > > had > > no realistic ability to discriminate between initiators or to > > change the > > login BB credit. In addition, the expectation is that all possible > > initiators would get these credits on login. So storage > > devices vendors > > have played it safe and kept this number low (at 0 until > recently, now > > around 2). For iSCSI the number of initial credits you need > > to "prime the > > pump" until normal data flow is established is probably large > > (given the > > latencies are higher than in Fibre Channel, especially > FC-AL), and the > > number of potential initiators larger than in Fibre Channel, > > making this a > > whole lot worse for the storage device. > > > > As soon as we allow the devices to start adjusting these > > credits, then you > > have the protocol problem of making sure people know when > > their credits are > > adjusted and the policy problem of how, who, and when to adjust the > > credits. > > Changing everyone's credit when you add a new initiator can > get into a > > notification nightmare, although it is "fair." Any policy > > brings up all > > sorts of nasty issues regarding fairness vs efficient use of the > > transmission media. > > > > Jim > > > > Note: the same problem has plagued other attempts to allocate device > > resources between multiple initiators, like command queue space. In > > general > > policies with respect to multiple initiators are not really > > standard in the > > SCSI world. > > > > > > -----Original Message----- > > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > > Sent: Wednesday, September 06, 2000 2:56 PM > > To: ips > > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > > > > Joshua Tseng wrote: > > > > > James, > > > > > > I agree with others that there may be an issue with the > > > command windowing mechanism in the existing iSCSI spec. > It is like > > > "TCP in reverse", in that the target determines the size of > > the window, > > and > > > not the initiator as in TCP. Rather, I believe that > > everything that this > > > windowing mechanism is attempting to achieve can be more > > easily obtained > > > by having the target communicate its buffer size to the > initiator at > > > iSCSI login. It should be the role of the initiator to > > determine how > > > many commands to put in flight simultaneously, given this input on > > available > > > buffer size from the target. > > > > As more initiators connect to a target, it may need to > scale back the > > amount > > of > > this buffering it has allocated to each previously logged in > > initiator (to > > prevent rejecting new logins). > > > > > > > > > > > As far as multiple initiators, could this not be resolved > > by the target > > > refusing additional logins beyond the number of initiators > > it can safely > > > support? Not being a storage expert, this is my best > > guess/suggestion > > > at how to do it. > > > > I believe John already answered this... > > > > -Matt > > > > > > > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 > > From owner-ips@ECE.cmu.edu Fri Sep 08 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09754; 8 Sep 2000 15:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09124 for ips-outgoing; Fri, 8 Sep 2000 14:57:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA29555 for ; Fri, 8 Sep 2000 11:08:34 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Fri, 8 Sep 2000 08:09:40 -0700 Message-ID: From: Robert Snively To: "'somesh_gupta@hp.com'" , ips@ece.cmu.edu, Robert Snively , matt_wakeley@agilent.com Subject: RE: RE: Re: multiple connections Date: Fri, 8 Sep 2000 08:07:02 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Something is missing here. A window based outbound flow control is useless to a RAID. The problem is that the data buffer structure is filled from two sides. Information is flowing outbound for write operations. Information is flowing into the data buffer in the inbound direction as a result of receiving and executing the small command packets. As a result, the available buffering for outbound operations may be zero at any time in a manner scheduled by the target, not the host. In addition, during the maintenance of a RAID device's internal redundancy, additional buffer space is used up that is unknown to both the read and write operations. The only ways I see to avoid a buffer resource deadlock are: 1) Send a transfer ready indication for each outbound transfer. (This is the universal SCSI solution). 2) Make the rather ridiculous and unenforceable rule that the sum of all the SCSI buffer areas for all initiators be less than the buffer space of each logical unit in a RAID. Note that communication with each initiator about the available target buffering may also not be sufficient, since any single initiator could use it all up in an instant. > > > > > > 2. WRITES - This is the really bad one in my opinion. For > > > me, avoiding > > > RTTs in iSCSI would just by itself make iSCSI a superior > > "transport" > > > for SCSI. So assuming RTTs are not being used, the host > > > would (changing > > > the posting order from READs), first post the write > command to the > > > connection on which the data is to be sent, and then > post the write > > > buffers to the connection which is to be used for sending data. > > > One case is where for whatever reason, the target gets the > > > data before > > > it gets the command, and has no clue what to do with the data. > > > Let us assume that the target does get the command before > > it gets the > > > data. The target gets a command indicating that data being > > > written to > > > whatever lun and whatever location is going to arrive > on some other > > > connection. First the target has taken an extra event from > > > the adapter. > > > Then, when data arrives on another NIC (and the target > gets another > > > event), the target goes through the list of outstanding WRITE > > > commands to match the command with the data and then go > about the > > > business of processing the data. > > > > > > So in this case the work has increased for the > initiator as well as > > > the target. > > > > > > And to top it off, the target is active with a large number of > > initiators on behalf of a large number of logical units for a large > > number of queued commands, so there is absolutely no guarantee that > > any buffer exists for the data that was received, even if the > > command had been received first. > > > > Every SCSI command execution is managed by the target for > > this reason among > > many others. Thus RTT has been a part of every SCSI protocol > > for write > > operations. > > > > TCP window size should provide a reasonable flow control mechanism, > so an additional flow control should not be needed. There will be > some statistical determination of the amount of memory needed in > a large array vs the window size extended on each connection. > > I don't know if arrays like to keep commands in seperate memory > from data, in which case a command queue depth may have to be > communicated seperately (assuming most of window would typically > be used for data) > > From owner-ips@ECE.cmu.edu Fri Sep 08 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09776; 8 Sep 2000 15:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA08686 for ips-outgoing; Fri, 8 Sep 2000 14:48:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA08680 for ; Fri, 8 Sep 2000 14:47:59 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA08663; Fri, 8 Sep 2000 11:38:34 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA31457; Fri, 8 Sep 2000 11:36:12 -0700 From: "Douglas Otis" To: "Robert Snively" , "Stephen Bailey" , Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 8 Sep 2000 11:50:16 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, Thanks, that helps an understanding of this PDU 0x45. If iSCSI is to copy FCP then it should send a response structure as FCP and not one or two and use it according to FCP. Undefined information of what some consider SCSI values cause variants in use. I have not seen tape drives use Good Sense residual but vary in how Check Sense residual is presented. If iSCSI is to copy FC, then why deviate on response symmetry? At least this provides an easier bridge. Iterative read operations will carry unused values to be examined and opens the door for further variants. Status presented prior to data must then ensure sequence without end confirmation. Doug > -----Original Message----- > From: Robert Snively [mailto:rsnively@Brocade.COM] > Sent: Friday, September 08, 2000 9:20 AM > To: 'Douglas Otis'; Stephen Bailey; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > In FCP, because FC is a multi-protocol environment that may have > drivers independent of the SCSI drivers, an actual buffer allocation > was transmitted independently of the SCSI command. Both overrun > and underrun indicators and a residual are returned. They are > protocol related, not SCSI related. > > There are cases where incorrect length indications are provided > in the SCSI model for both under-run and over-run cases, but they > are rare and typically associated with legacy tape drive programs. > > Bob > > > -----Original Message----- > > From: Douglas Otis [mailto:dotis@sanlight.net] > > Sent: Wednesday, September 06, 2000 6:26 PM > > To: Stephen Bailey; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > Steph, > > > > Although a logical use of a residual, I can not see where it > > is defined > > should the allocation length be greater than the returned > > length. Should > > the allocation be less than the response, this is a Check > > Condition and the > > residual is defined. It would seem a residual in a case > > with adequate > > allocation is not interesting or defined. Do you know where > > this mechanism > > you describe is defined? In normal use, without a Check > > Condition such > > information is not returned to the application. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Stephen Bailey > > > Sent: Wednesday, September 06, 2000 3:45 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > When do you get GOOD status and residual counts on a read? > > > What is causing > > > > the target to get the length wrong? > > > > > > One example is an INQUIRY command. The inquiry data length is > > > target-specific. Typically the CDB allocation length (and > > DL) are set > > > to some arbitrary large value (0xff), and the target sends back > > > everything it has. The transfer ends with success status. > > > > > > There can certainly be transfer residual and no SCSI error status. > > > > > > Steph > > > > > > > > From owner-ips@ECE.cmu.edu Fri Sep 08 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09746; 8 Sep 2000 15:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09208 for ips-outgoing; Fri, 8 Sep 2000 14:59:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA02862 for ; Fri, 8 Sep 2000 12:21:50 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Fri, 8 Sep 2000 09:22:56 -0700 Message-ID: From: Robert Snively To: "'Douglas Otis'" , Stephen Bailey , ips@ece.cmu.edu Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 8 Sep 2000 09:20:14 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk In FCP, because FC is a multi-protocol environment that may have drivers independent of the SCSI drivers, an actual buffer allocation was transmitted independently of the SCSI command. Both overrun and underrun indicators and a residual are returned. They are protocol related, not SCSI related. There are cases where incorrect length indications are provided in the SCSI model for both under-run and over-run cases, but they are rare and typically associated with legacy tape drive programs. Bob > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Wednesday, September 06, 2000 6:26 PM > To: Stephen Bailey; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > Steph, > > Although a logical use of a residual, I can not see where it > is defined > should the allocation length be greater than the returned > length. Should > the allocation be less than the response, this is a Check > Condition and the > residual is defined. It would seem a residual in a case > with adequate > allocation is not interesting or defined. Do you know where > this mechanism > you describe is defined? In normal use, without a Check > Condition such > information is not returned to the application. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Stephen Bailey > > Sent: Wednesday, September 06, 2000 3:45 PM > > To: ips@ece.cmu.edu > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > When do you get GOOD status and residual counts on a read? > > What is causing > > > the target to get the length wrong? > > > > One example is an INQUIRY command. The inquiry data length is > > target-specific. Typically the CDB allocation length (and > DL) are set > > to some arbitrary large value (0xff), and the target sends back > > everything it has. The transfer ends with success status. > > > > There can certainly be transfer residual and no SCSI error status. > > > > Steph > > > > From owner-ips@ECE.cmu.edu Fri Sep 08 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09751; 8 Sep 2000 15:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09175 for ips-outgoing; Fri, 8 Sep 2000 14:58:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA01393 for ; Fri, 8 Sep 2000 11:48:08 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Fri, 8 Sep 2000 08:49:14 -0700 Message-ID: From: Robert Snively To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: Multiple Connections. How would they work? Date: Fri, 8 Sep 2000 08:46:27 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk As I read it, the model in 01 was a "single" model, not a "symmetric" model. It did not place any limit on the number of "singles", but did not place any relationship values among them either. That makes ordering exactly the same as the present SCSI structure, which may have any number of initiator/target connections and paths, but treats each one of them as individual and, for most purposes, independent. That is a model I would favor, since I see no way to create interdependent paths without objectionable and expensive relationships being maintained among hardware NICs in either the hardware or software layers. These relationships have typically been handled very well and in highly optimized ways by wedge drivers and high availability file systems. Bob > The 00 draft (Adelaide) had the commands go down over one > specific control > stream while the data went on any available connection. > Synchronizing them and providing for hardware implementation > across several > adapters was felt to be complex. > > However command ordering was trivial. > > The symmetric model in 01 requires ordering and this is why > you have the > command ordering. > A proposal to get back to the asymmetric model but decide > ahead of time (? > when) what > path data will be taking was forwarded by Kalman. > > We call this the asymmetric model (it can still evolve). From owner-ips@ECE.cmu.edu Fri Sep 08 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10106; 8 Sep 2000 16:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA12458 for ips-outgoing; Fri, 8 Sep 2000 16:18:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA12445 for ; Fri, 8 Sep 2000 16:17:55 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 8 Sep 2000 13:18:22 -0700 Message-ID: From: Charles Monia To: "Randall R. Stewart" , Charles Monia Cc: Matt Wakeley , ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Fri, 8 Sep 2000 13:18:19 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Randal: > -----Original Message----- > From: Randall R. Stewart [mailto:randall@stewart.chicago.il.us] > Sent: Friday, September 08, 2000 4:58 AM > To: Charles Monia > Cc: Matt Wakeley; ips@ece.cmu.edu > Subject: Re: a vote for asymmetric connections in a session > > > Charles Monia wrote: > > > I'm not sure what is meant by "congestion." If we're talking about > > congestion in the TCP/IP transport, I'm in agreement. > However, I thought we > > were referring to the sort of congestion that the > application on top of > > iSCSI might see if it received more commands than it had room for. > > > > Unless I misunderstood your point, threfore, I think there > might be an > > issue. The only way I can see flow control in the tranport > layer being used > > to avoid dropping commands is if higher layer congestion > results in back > > pressure to the iSCSI pipe. I believe that behavior is > undesirable because > > it introduces "head-of-line" blocking, with the following > consequences: > > > > Any time you get ANY retransmissions in TCP you will create a > "head-of-line" > blocking scenario. This does not matter if it is from flow control as > you state or network congestion where a packet is dropped by a router. > You will have situations where the TCP is holding data waiting for the > retransmissions of an earlier packet... this is one of the reasons > that sigtran developed SCTP... > > Hi Randall: My concern was about coupling the SCSI command service layer with a transport layer of any kind such that congestion in the command service layer causes transport layer back pressure. From all the email traffic I've seen so far, I'm satisfied that this is not an issue. Charles Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Fri Sep 08 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10071; 8 Sep 2000 16:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA11251 for ips-outgoing; Fri, 8 Sep 2000 15:43:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA11247 for ; Fri, 8 Sep 2000 15:43:33 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id E88558E94; Fri, 8 Sep 2000 12:40:26 -0700 (PDT) To: ips@ece.cmu.edu Subject: Re: Command Queue Depth (was asymmetric/Symmetric) References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 08 Sep 2000 12:40:26 -0700 In-Reply-To: Jim McGrath's message of "Thu, 7 Sep 2000 20:43:52 -0700" Message-ID: Lines: 30 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim McGrath writes: > The path I'd recommend is to allow people to oversubscribe a target's > resources, and then to do a graceful recovery when that gets you into > trouble. > > Note that drive designers do this all of the time - we tend to optimize for > common cases, and then worry about how to handle outlyers using other > mechanisms. While a more complicated model, it gets you the best overall > resource utilization. We use read on arrival, ECC on the Fly, Retrys, auto > reallocation, all in attempts to handle the common path quickly and the rare > path more slowly, where the differences are error rates. > > In this case I would allow initiators to send down data immediately - when > that works (like when ECC on the Fly works) you get a benefit. If packets > are dropped you can rely on existing mechanisms to recover, or you can put > in a new, improved, and perhaps more friendly process. In either case I > think the result would probably be better than a tight credit based model > where a lot of delays would be introduced and a lot of (historically > unresolved) allocation policy issues arise. For oversubscribing command queues, is the TASK SET FULL error condition sufficient? For oversubscribing unsolicited data sent with WRITES, the current spec allows the target to drop such immediate data and issue RTTs. -Costa From owner-ips@ECE.cmu.edu Fri Sep 08 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10080; 8 Sep 2000 16:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA12263 for ips-outgoing; Fri, 8 Sep 2000 16:13:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA12250 for ; Fri, 8 Sep 2000 16:13:26 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 8 Sep 2000 13:13:52 -0700 Message-ID: From: Charles Monia To: John Hufferd/San Jose/IBM , Joshua Tseng , Charles Monia Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Fri, 8 Sep 2000 13:13:50 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Friday, September 08, 2000 12:20 AM > To: Joshua Tseng; Charles Monia > Cc: ips@ece.cmu.edu > Subject: RE: a vote for asymmetric connections in a session > > > > Joshua, Charles, > > With respect to the Asymmetric approach, the issues change, > and we are no > longer trying to solve the problem of missing commands that > occur because > of a broken connection. Therefore, I think we can dump the > sliding window > and leave the flow control -- and recovery of lost commands, > -- up to > TCP. Yes, the SCSI layers on each end need to have their own buffer > management under control, but I do not think this is a > transport issue. > John: Well put. While, strictly speaking, the issues I raised are not transport-related (and may not be perceived as problematic), they may be best addressed (if at all) with transport layer assists. Charles From owner-ips@ECE.cmu.edu Fri Sep 08 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10078; 8 Sep 2000 16:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA12195 for ips-outgoing; Fri, 8 Sep 2000 16:12:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA12190 for ; Fri, 8 Sep 2000 16:11:57 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id 46720430 for ; Fri, 8 Sep 2000 13:11:56 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id NAA24313 for ; Fri, 8 Sep 2000 13:11:54 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA77; Fri, 8 Sep 2000 13:11:50 -0700 Message-ID: <39B9481F.F7E0F09@agilent.com> Date: Fri, 08 Sep 2000 13:12:16 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: "VonStamwitz, Paul" , ips@ece.cmu.edu Subject: Re: Symmetric vs Asymmetric References: <5B0C98C89640D311A319002048406A1C05B31CDF@aimexc03.corp.adaptec.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "VonStamwitz, Paul" wrote: > > "VonStamwitz, Paul" wrote: > > > I agree with David. I oppose a two connection minimum. > > > [Matt Wakeley] > > I disagree. Initial implementations and cheap hardware will > > have the iSCSI > > implementation in software. It is easy to do anything in > > software - open a > > thousand connections, perhaps hundreds of thousands, one per > > each LU, it > > doesn't care - it just uses up more memory. > > > > However, for the high performance implementations that will > > implement iSCSI in > > hardware, doing a function one way sometimes, and another > > other times, will > > require more hardware and more testing (by both the hardware > > vendors and the > > value add vendors) to test both functions. > > > If immediate data on writes is supported, won't you have to support both > functions anyways? If the target (or initiator) chooses which connection to > transfer data, why can't the command/status connection be one of the > choices? We haven't discussed how immediate write data will work when commands all go down one tcp connection. If commands only go down one connection, you don't want them stuck behind (large) data transfers. > > One connection per LU also has the huge issue of requiring > > lots of on-hardware > > resources and memory to maintain the hundreds or thousands of > > connections that > > will be required.... and you want a cost competitive solution? > > > I stated before that the resource issues preclude mandating a connection per > LU. But if resources allow it, it's not a bad way to operate. The issue is that it's an OPTION. It is best (and more interoperable) when all devices operate the same way. Leaving this as a option would require both the targets and the initiators to figure out how the other guy wants to talk (connection per LU or connection per target) and implement, test, etc the appropriate hardware/software. For example, a host driver would have to be written so that it would sometimes talk to target connection based targets, and other times to LU connection based targets. > > > -Paul -Matt From owner-ips@ECE.cmu.edu Fri Sep 08 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10076; 8 Sep 2000 16:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA11857 for ips-outgoing; Fri, 8 Sep 2000 16:01:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA11845 for ; Fri, 8 Sep 2000 16:01:22 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 0E57683D for ; Fri, 8 Sep 2000 16:01:18 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id NAA24033 for ; Fri, 8 Sep 2000 13:01:15 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA74DA for ; Fri, 8 Sep 2000 13:01:11 -0700 Message-ID: <39B945A1.4761CA70@agilent.com> Date: Fri, 08 Sep 2000 13:01:37 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Data in SCSI Response or SCSI Data References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > Bob, > > Thanks, that helps an understanding of this PDU 0x45. If iSCSI is to copy > FCP I think this needs clarification. iSCSI is *NOT* copying FCP. iSCSI is attempting to transport SCSI over TCP/IP. It just so happens that the end result looks similar to what FCP did. > then it should send a response structure as FCP and not one or two and > use it according to FCP. Undefined information of what some consider SCSI > values cause variants in use. I have not seen tape drives use Good Sense > residual but vary in how Check Sense residual is presented. If iSCSI is to > copy FC, then why deviate on response symmetry? At least this provides an > easier bridge. Iterative read operations will carry unused values to be > examined and opens the door for further variants. Status presented prior to > data must then ensure sequence without end confirmation. Who said anything that status was presented before data? iSCSI allows the LAST iSCSI PDU that completes a SCSI command to also contain the GOOD SCSI status in the header, so that extra overhead to send the status PDU. Julian, if it will make this issue go away, maybe the residual fields should be removed from the data PDU. If there is residual, it will be sent in a status PDU. -Matt > > > Doug > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > Sent: Friday, September 08, 2000 9:20 AM > > To: 'Douglas Otis'; Stephen Bailey; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > In FCP, because FC is a multi-protocol environment that may have > > drivers independent of the SCSI drivers, an actual buffer allocation > > was transmitted independently of the SCSI command. Both overrun > > and underrun indicators and a residual are returned. They are > > protocol related, not SCSI related. > > > > There are cases where incorrect length indications are provided > > in the SCSI model for both under-run and over-run cases, but they > > are rare and typically associated with legacy tape drive programs. > > > > Bob > > > > > -----Original Message----- > > > From: Douglas Otis [mailto:dotis@sanlight.net] > > > Sent: Wednesday, September 06, 2000 6:26 PM > > > To: Stephen Bailey; ips@ece.cmu.edu > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > Steph, > > > > > > Although a logical use of a residual, I can not see where it > > > is defined > > > should the allocation length be greater than the returned > > > length. Should > > > the allocation be less than the response, this is a Check > > > Condition and the > > > residual is defined. It would seem a residual in a case > > > with adequate > > > allocation is not interesting or defined. Do you know where > > > this mechanism > > > you describe is defined? In normal use, without a Check > > > Condition such > > > information is not returned to the application. > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > Stephen Bailey > > > > Sent: Wednesday, September 06, 2000 3:45 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > When do you get GOOD status and residual counts on a read? > > > > What is causing > > > > > the target to get the length wrong? > > > > > > > > One example is an INQUIRY command. The inquiry data length is > > > > target-specific. Typically the CDB allocation length (and > > > DL) are set > > > > to some arbitrary large value (0xff), and the target sends back > > > > everything it has. The transfer ends with success status. > > > > > > > > There can certainly be transfer residual and no SCSI error status. > > > > > > > > Steph > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Fri Sep 08 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10095; 8 Sep 2000 16:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA13139 for ips-outgoing; Fri, 8 Sep 2000 16:34:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA13131 for ; Fri, 8 Sep 2000 16:34:46 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 8 Sep 2000 13:34:53 -0700 Message-ID: From: Charles Monia To: Matt Wakeley , Jim McGrath , ips Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Fri, 8 Sep 2000 13:34:51 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Thursday, September 07, 2000 10:25 PM > To: Jim McGrath; ips > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > Jim McGrath wrote: > > > I agree that BB credit has nothing to do with commands per se, but > > illustrates the problem we have had with deciding on > policies for the > > distributed allocation of device resources over multiple > initiators. In my > > postscript I noted that the same problems have arisen on > command queue (how > > many queue slots do you get), with similarly no > satisfactory solution. > > So it sounds to me like this distribution of (command) > resources across > initiators is a T10 SCSI issue, and should be solved there, > not by each > individual transport (yesterday FC - which didn't solve it, > today iSCSI, > tomorrow IB or whatever). > Hi Matt: One thing I got from Jim's note is that SCSI storage devices designs are carefully tuned. So, behaviors at the transport layer do make a difference -- especially to cost-driven products. In my opinion, the IPS community would be well advised not to ignore these concerns. Charles Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Fri Sep 08 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10089; 8 Sep 2000 16:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA11948 for ips-outgoing; Fri, 8 Sep 2000 16:04:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA11943 for ; Fri, 8 Sep 2000 16:03:56 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id AD5CA8E94; Fri, 8 Sep 2000 13:01:04 -0700 (PDT) To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 08 Sep 2000 13:01:04 -0700 In-Reply-To: Joshua Tseng's message of "Thu, 7 Sep 2000 23:14:44 -0700" Message-ID: Lines: 33 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk If I recall correctly, the windowing mechanism was put in to avoid a deadlock condition while using multiple control connections. I believe the logic went as follow: Assumption: You're not allowed to drop commands arriving in on any connection Because the TCP connections work at different rates, commands may not arrive in order. Let's say you have four connections and a command queue depth of 3. Commands #2,#3, and #4 arrive before command #1, filling the queue. However, the queue cannot be drained until command #1 arrives. So, deadlock. Solution: Allow commands to be dropped In the example above, we'd probably drop command #4 and replace it with command #1. Note that using a single TCP connection for control/data is not sufficient to solve the problem. Imagine the case that the queue is depth 3 and the host issues four WRITES commands in quick order. The device stops reading the TCP connection after three commands. Unfortunately, none of those three writes can complete until data is read from that connection! So, deadlock again. CIFS, NFS, and HTTP don't have this problem because they immediately send data after their WRITE commands. -Costa From owner-ips@ECE.cmu.edu Fri Sep 08 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10124; 8 Sep 2000 17:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA12663 for ips-outgoing; Fri, 8 Sep 2000 16:22:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel1.hp.com (cosrel1.hp.com [156.153.255.170]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA12657 for ; Fri, 8 Sep 2000 16:22:21 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by cosrel1.hp.com (Postfix) with ESMTP id A012793 for ; Fri, 8 Sep 2000 14:22:24 -0600 (MDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id NAA27743 for ips@ece.cmu.edu; Fri, 8 Sep 2000 13:22:15 -0700 (PDT) X-OpenMail-Hops: 1 Date: Fri, 8 Sep 2000 13:22:02 -0700 Message-Id: Subject: RE: RE: RE: Re: multiple connections MIME-Version: 1.0 To: ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I aplogise in advance for being confused about direction of flow implied in parts of the message, so please help clarify it so I can understand better. > -----Original Message----- > From: rsnively@Brocade.COM [mailto:rsnively@Brocade.COM] > Sent: Friday, September 08, 2000 8:07 AM > To: somesh_gupta@hp.com; ips@ece.cmu.edu; rsnively@Brocade.COM; > matt_wakeley@agilent.com > Subject: FW: RE: RE: Re: multiple connections > > > > Something is missing here. A window based outbound flow control is > useless to a RAID. The problem is that the data buffer > structure is filled > from two sides. Information is flowing outbound for write operations. -- Does outbound here means outbound from initiator to target (write seems to imply that)? This would then be controlled by the window size indicated by the target to the initiator. > Information is flowing into the data buffer in the inbound direction > as a result of receiving and executing the small command packets. -- Here is where I start to get confused. Does inbound imply from the target to the initiator (as opposed to outbound?). However you say "receiving and executing the small command packets" - seems to imply from initiator to target. > As a result, the available buffering for outbound operations may be > zero at any time in a manner scheduled by the target, not the host. > In addition, during the maintenance of a RAID device's internal > redundancy, additional buffer space is used up that is unknown to > both the read and write operations. > > The only ways I see to avoid a buffer resource deadlock are: > > 1) Send a transfer ready indication for each outbound > transfer. > (This is the universal SCSI solution). > > 2) Make the rather ridiculous and unenforceable rule that > the sum of all the SCSI buffer areas > for all initiators be less than the buffer space > of each logical unit in a RAID. > > Note that communication with each initiator about the available target > buffering may also not be sufficient, since any single initiator could > use it all up in an instant. -- I think (discussed in one of the other strings), an indication of command queue depth to manage buffers used up by commands, window based flow control, and recovery in the worst case when the buffers overflow (requires command numbering etc) should provide the protection. The performance improvements provided by sending write data without RTT (especially for small writes), and reasonable streaming provided by queue depth indications and windows should provide for very good performance. -- If might be worthwhile to have a analysis based on some "real" numbers to see how much memory would be needed and what that would cost etc, and the kind of window size/queue depths to be communicated - and the data rate they would sustain. If a couple of people are willing to work with me offline, I would be game. > > > > > > > > > 2. WRITES - This is the really bad one in my opinion. For > > > > me, avoiding > > > > RTTs in iSCSI would just by itself make iSCSI a superior > > > "transport" > > > > for SCSI. So assuming RTTs are not being used, the host > > > > would (changing > > > > the posting order from READs), first post the write > > command to the > > > > connection on which the data is to be sent, and then > > post the write > > > > buffers to the connection which is to be used for > sending data. > > > > One case is where for whatever reason, the target gets the > > > > data before > > > > it gets the command, and has no clue what to do with the data. > > > > Let us assume that the target does get the command before > > > it gets the > > > > data. The target gets a command indicating that data being > > > > written to > > > > whatever lun and whatever location is going to arrive > > on some other > > > > connection. First the target has taken an extra event from > > > > the adapter. > > > > Then, when data arrives on another NIC (and the target > > gets another > > > > event), the target goes through the list of outstanding WRITE > > > > commands to match the command with the data and then go > > about the > > > > business of processing the data. > > > > > > > > So in this case the work has increased for the > > initiator as well as > > > > the target. > > > > > > > > > And to top it off, the target is active with a large number of > > > initiators on behalf of a large number of logical units > for a large > > > number of queued commands, so there is absolutely no > guarantee that > > > any buffer exists for the data that was received, even if the > > > command had been received first. > > > > > > Every SCSI command execution is managed by the target for > > > this reason among > > > many others. Thus RTT has been a part of every SCSI protocol > > > for write > > > operations. > > > > > > > TCP window size should provide a reasonable flow control mechanism, > > so an additional flow control should not be needed. There will be > > some statistical determination of the amount of memory needed in > > a large array vs the window size extended on each connection. > > > > I don't know if arrays like to keep commands in seperate memory > > from data, in which case a command queue depth may have to be > > communicated seperately (assuming most of window would typically > > be used for data) > > > > > From owner-ips@ECE.cmu.edu Fri Sep 08 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10422; 8 Sep 2000 18:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA15123 for ips-outgoing; Fri, 8 Sep 2000 17:26:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA15119 for ; Fri, 8 Sep 2000 17:26:50 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA08748; Fri, 8 Sep 2000 14:17:25 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA31910; Fri, 8 Sep 2000 14:15:03 -0700 From: "Douglas Otis" To: "Matt Wakeley" , Subject: RE: Data in SCSI Response or SCSI Data Date: Fri, 8 Sep 2000 14:29:08 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <39B945A1.4761CA70@agilent.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Friday, September 08, 2000 1:02 PM > To: ips@ece.cmu.edu > Subject: Re: Data in SCSI Response or SCSI Data > > > Douglas Otis wrote: > > > Bob, > > > > Thanks, that helps an understanding of this PDU 0x45. If iSCSI > is to copy > > FCP > > I think this needs clarification. iSCSI is *NOT* copying FCP. iSCSI is > attempting to transport SCSI over TCP/IP. It just so happens that the end > result looks similar to what FCP did. Okay, similar architecture. > > > then it should send a response structure as FCP and not one or two and > > use it according to FCP. Undefined information of what some > consider SCSI > > values cause variants in use. I have not seen tape drives use > Good Sense > > residual but vary in how Check Sense residual is presented. If > iSCSI is to > > copy FC, then why deviate on response symmetry? At least this > provides an > > easier bridge. Iterative read operations will carry unused values to be > > examined and opens the door for further variants. Status > presented prior to > > data must then ensure sequence without end confirmation. > > Who said anything that status was presented before data? > iSCSI allows the LAST iSCSI PDU that completes a SCSI command to > also contain > the GOOD SCSI status in the header, so that extra overhead to > send the status > PDU. By placing status within the header of perhaps 4G of data payload with no defined end structure leaves nothing known to follow this transfer. By making Response a separate structure that physically follows the data payload there is an end confirmation. Should there be an error in sequence, data will be accepted followed by perhaps some unknown command responses until everything falls back into sync. A dubious savings having dummy status in each read exchange to save one possible structure at the end. Rather than processing one response structure, you must process two where one is out of sequence to the function. Doug > > Julian, if it will make this issue go away, maybe the residual > fields should be > removed from the data PDU. If there is residual, it will be sent > in a status > PDU. > > -Matt > > > > > > > Doug > > > > > -----Original Message----- > > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > > Sent: Friday, September 08, 2000 9:20 AM > > > To: 'Douglas Otis'; Stephen Bailey; ips@ece.cmu.edu > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > In FCP, because FC is a multi-protocol environment that may have > > > drivers independent of the SCSI drivers, an actual buffer allocation > > > was transmitted independently of the SCSI command. Both overrun > > > and underrun indicators and a residual are returned. They are > > > protocol related, not SCSI related. > > > > > > There are cases where incorrect length indications are provided > > > in the SCSI model for both under-run and over-run cases, but they > > > are rare and typically associated with legacy tape drive programs. > > > > > > Bob > > > > > > > -----Original Message----- > > > > From: Douglas Otis [mailto:dotis@sanlight.net] > > > > Sent: Wednesday, September 06, 2000 6:26 PM > > > > To: Stephen Bailey; ips@ece.cmu.edu > > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > > > > Steph, > > > > > > > > Although a logical use of a residual, I can not see where it > > > > is defined > > > > should the allocation length be greater than the returned > > > > length. Should > > > > the allocation be less than the response, this is a Check > > > > Condition and the > > > > residual is defined. It would seem a residual in a case > > > > with adequate > > > > allocation is not interesting or defined. Do you know where > > > > this mechanism > > > > you describe is defined? In normal use, without a Check > > > > Condition such > > > > information is not returned to the application. > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > > > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > > Stephen Bailey > > > > > Sent: Wednesday, September 06, 2000 3:45 PM > > > > > To: ips@ece.cmu.edu > > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > > > > When do you get GOOD status and residual counts on a read? > > > > > What is causing > > > > > > the target to get the length wrong? > > > > > > > > > > One example is an INQUIRY command. The inquiry data length is > > > > > target-specific. Typically the CDB allocation length (and > > > > DL) are set > > > > > to some arbitrary large value (0xff), and the target sends back > > > > > everything it has. The transfer ends with success status. > > > > > > > > > > There can certainly be transfer residual and no SCSI > error status. > > > > > > > > > > Steph > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Fri Sep 08 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10426; 8 Sep 2000 18:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA14588 for ips-outgoing; Fri, 8 Sep 2000 17:12:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA14583 for ; Fri, 8 Sep 2000 17:12:02 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA08744; Fri, 8 Sep 2000 14:02:37 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA31870; Fri, 8 Sep 2000 14:00:15 -0700 From: "Douglas Otis" To: "VonStamwitz, Paul" , Subject: RE: Symmetric vs Asymmetric Date: Fri, 8 Sep 2000 14:14:20 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <5B0C98C89640D311A319002048406A1C05B31CDF@aimexc03.corp.adaptec.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Paul, Although I agree a single connection per LUN is inviting, using TCP per LUN to the same device (controller) is not a fair player. Each TCP connection gets a share, but the controller share is large due to increased number of connections. Being a controller company, this may seem more than fair. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > VonStamwitz, Paul > Sent: Friday, September 08, 2000 11:54 AM > To: ips@ece.cmu.edu > Subject: RE: Symmetric vs Asymmetric > > > > "VonStamwitz, Paul" wrote: > > > I agree with David. I oppose a two connection minimum. > > > [Matt Wakeley] > > I disagree. Initial implementations and cheap hardware will > > have the iSCSI > > implementation in software. It is easy to do anything in > > software - open a > > thousand connections, perhaps hundreds of thousands, one per > > each LU, it > > doesn't care - it just uses up more memory. > > > > However, for the high performance implementations that will > > implement iSCSI in > > hardware, doing a function one way sometimes, and another > > other times, will > > require more hardware and more testing (by both the hardware > > vendors and the > > value add vendors) to test both functions. > > > If immediate data on writes is supported, won't you have to support both > functions anyways? If the target (or initiator) chooses which > connection to > transfer data, why can't the command/status connection be one of the > choices? > > > One connection per LU also has the huge issue of requiring > > lots of on-hardware > > resources and memory to maintain the hundreds or thousands of > > connections that > > will be required.... and you want a cost competitive solution? > > > I stated before that the resource issues preclude mandating a > connection per > LU. But if resources allow it, it's not a bad way to operate. > > -Paul > From owner-ips@ECE.cmu.edu Fri Sep 08 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10424; 8 Sep 2000 18:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA14195 for ips-outgoing; Fri, 8 Sep 2000 17:01:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA14186 for ; Fri, 8 Sep 2000 17:01:18 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 8 Sep 2000 14:01:44 -0700 Message-ID: From: Charles Monia To: csapuntz@cisco.com, ips@ece.cmu.edu Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Fri, 8 Sep 2000 14:01:40 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > Sent: Friday, September 08, 2000 12:40 PM > To: ips@ece.cmu.edu > Cc: csapuntz@cisco.com > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > > Jim McGrath writes: > > > The path I'd recommend is to allow people to oversubscribe > a target's > > resources, and then to do a graceful recovery when that > gets you into > > trouble. > > > > For oversubscribing command queues, is the TASK SET FULL > error condition > sufficient? > Hi Costa: IMHO: Depends on how often it occurs, of course, and whether or not the lapse in the ordering of SCSI commands is acceptable. Charles Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Fri Sep 08 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10435; 8 Sep 2000 18:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA15016 for ips-outgoing; Fri, 8 Sep 2000 17:24:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from storeage_nt.store-age.com ([199.203.178.211]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA15010 for ; Fri, 8 Sep 2000 17:24:29 -0400 (EDT) Received: from e4u5e0 (ras9-p28.hfa.netvision.net.il [62.0.190.28]) by storeage_nt.store-age.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id SKRFDH5N; Sat, 9 Sep 2000 00:22:19 +0200 Message-ID: <007f01c019e3$0edfea40$1cbe003e@e4u5e0> From: "Nelson Nahum" To: References: Subject: Enhancements for the iSCSI Date: Sat, 9 Sep 2000 00:20:54 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk I think that the iSCSI is a special opportunity to enhance the current SCSI protocol especially it networking capabilities. SCSI was designed with the concept of Initiator for the HBA and Target for the devices. The Initator sends commands to the Targets that are passive. But the networking world supposes every one talking with every one. The fact that iSCSI is implemented over TCP/IP can help to develop these enhacements. Modern SAN appliances need to "talk" with the storage devices but also need a way to update information in the HBAs in a asynchronous manner. My questions are the following: 1 - Is iSCSI designed in such a way that every Target can also be Initiator and viceversa or it supposes different charasterisitics (memory, CPU power etc.) between the Initiator and the Target? 2 - Is possible to add asynchronous events that the Target could send to the Initiators at any time? 3 - Another important enhancement could be the possibility to "broadcast" information to many iSCSI devices or HBAs. Nelson Nahum CTO StoreAge Networking Technologies From owner-ips@ECE.cmu.edu Sat Sep 09 00:20:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11048; 8 Sep 2000 20:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA17818 for ips-outgoing; Fri, 8 Sep 2000 18:58:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA17814 for ; Fri, 8 Sep 2000 18:58:37 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e88Mwax17659 for ; Fri, 8 Sep 2000 17:58:36 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA07261; Fri, 8 Sep 00 17:56:50 -0500 Message-Id: <10009082256.AA07261@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Command Queue Depth (was asymmetric/Symmetric) In-Reply-To: Message from Charles Monia of "Fri, 08 Sep 2000 13:34:51 PDT." Date: Fri, 08 Sep 2000 17:58:35 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > > So it sounds to me like this distribution of (command) > > resources across initiators is a T10 SCSI issue, and should be > > solved there, not by each individual transport (yesterday FC - > > which didn't solve it, today iSCSI, tomorrow IB or whatever). > [...] > One thing I got from Jim's note is that SCSI storage devices designs are > carefully tuned. So, behaviors at the transport layer do make a difference > -- especially to cost-driven products. In my opinion, the IPS community > would be well advised not to ignore these concerns. It seems like poor standards practice to work-around attached standards that are under active development. iSCSI should engineer to the level that it does not make the problem of handling oversubscription WORSE than on other LLPs (and can benefit from an ultimate ULP solution), but the real fix does belong in the SCSI layer, since it's responsible for the architecture of initiators and targets. It would be different if this problem were not even on the radar screen of T10, but the annecdotes about FC RAID boxes that can't do ANY useful work because they're too busy BUSYing incoming FC commands are already out there. If they aren't moving quickly enough, is the task of interested parties to drive the T10 process along. Render unto Caeser and all that. Steph [So which is Caeser?] From owner-ips@ECE.cmu.edu Sat Sep 09 00:30:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11053; 8 Sep 2000 20:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA18522 for ips-outgoing; Fri, 8 Sep 2000 19:24:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA18513 for ; Fri, 8 Sep 2000 19:24:17 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id SAA24964 for ; Fri, 8 Sep 2000 18:17:58 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id RAA103710 for ; Fri, 8 Sep 2000 17:24:16 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: a vote for asymmetric connections in a session To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 8 Sep 2000 16:22:17 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/08/2000 05:24:15 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I do not know how reasonable Costa's configuration is,in the real world, but he has brought up a point that, I think, might be important to address. If his example is reasonable, then it would suggest that we NEED to enforce the requirement of Two connections per Session in the Asymmetric case. What do you all think about this? . . . John L. Hufferd csapuntz@cisco.com@ece.cmu.edu on 09/08/2000 01:01:04 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: csapuntz@cisco.com Subject: Re: a vote for asymmetric connections in a session If I recall correctly, the windowing mechanism was put in to avoid a deadlock condition while using multiple control connections. I believe the logic went as follow: Assumption: You're not allowed to drop commands arriving in on any connection Because the TCP connections work at different rates, commands may not arrive in order. Let's say you have four connections and a command queue depth of 3. Commands #2,#3, and #4 arrive before command #1, filling the queue. However, the queue cannot be drained until command #1 arrives. So, deadlock. Solution: Allow commands to be dropped In the example above, we'd probably drop command #4 and replace it with command #1. Note that using a single TCP connection for control/data is not sufficient to solve the problem. Imagine the case that the queue is depth 3 and the host issues four WRITES commands in quick order. The device stops reading the TCP connection after three commands. Unfortunately, none of those three writes can complete until data is read from that connection! So, deadlock again. CIFS, NFS, and HTTP don't have this problem because they immediately send data after their WRITE commands. -Costa From owner-ips@ECE.cmu.edu Sat Sep 09 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12280; 9 Sep 2000 0:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA27221 for ips-outgoing; Fri, 8 Sep 2000 23:54:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA27211 for ; Fri, 8 Sep 2000 23:54:21 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 8 Sep 2000 20:53:50 -0700 Message-ID: From: Joshua Tseng To: John Hufferd/San Jose/IBM , Joshua Tseng , Charles Monia Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Fri, 8 Sep 2000 20:53:48 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi John, >I could be mistaken here but I think you, and some others have been >addressing issues with the Storage Controller having the right amount of >buffer space in the controllers for the commands, after they are delivered >by iSCSI. I think this is a SCSI issue and not a iSCSI/TCP flow control >problem. Yes, you are right that this is precisely the problem I'm addressing. Maybe I'm missing something here, but I still think there is an issue. Yes, as James described earlier SCSI has a mechanism for dealing with this, but is it sufficient for iSCSI/TCP? IP has distinctly different characteristics than parallel SCSI and FC. For one thing, latencies in the IP world are much higher. I'm assuming parallel SCSI latencies are next to nothing, while FC are in the <10us range. If iSCSI is going over the WAN, at least 50ms latencies are to be expected for coast-to-coast US. Over the Public Internet, we're talking about 80 to 250ms or even more (RT). Much of this latency is caused by the speed of light, the rest by the serialization process for WAN transport. This works out to be a huge amount of latency compared to FC. How much can happen in the 30ms before the SCSI QUEUE FULL and BUSY messages finally get back to the initiator. Are you sure leaving it to SCSI or TCP will be okay? >With respect to the Asymmetric approach, the issues change, and we are no >longer trying to solve the problem of missing commands that occur because >of a broken connection. Therefore, I think we can dump the sliding window >and leave the flow control -- and recovery of lost commands, -- up to >TCP. Yes, the SCSI layers on each end need to have their own buffer >management under control, but I do not think this is a transport issue. I agree dumping the sliding window is a good thing. But my question is should iSCSI replace it with something else. I believe the issue James brought up with multiple initiators needs to be considered. Or even with a single initiator, how do you flow control the oncoming commands from the initiator(s)? Is the SCSI mechanisms sufficient given the latencies that iSCSI must be designed to support? BTW, I don't think that TCP has any role in addressing this issue. If iSCSI and TCP operate independently at different layers, then TCP won't flow control iSCSI commands any more than it flow controls any other PDU's, because it can't tell the difference between them. Josh From owner-ips@ECE.cmu.edu Sat Sep 09 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12275; 9 Sep 2000 0:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA27222 for ips-outgoing; Fri, 8 Sep 2000 23:54:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA27216 for ; Fri, 8 Sep 2000 23:54:24 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 8 Sep 2000 20:53:50 -0700 Message-ID: From: Joshua Tseng To: John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Fri, 8 Sep 2000 20:53:41 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't get it. Maybe I'm dumb or something, but I thought the whole point of asymmetric was that commands wouldn't arrive out of order. Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Friday, September 08, 2000 4:22 PM To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session I do not know how reasonable Costa's configuration is,in the real world, but he has brought up a point that, I think, might be important to address. If his example is reasonable, then it would suggest that we NEED to enforce the requirement of Two connections per Session in the Asymmetric case. What do you all think about this? . . . John L. Hufferd csapuntz@cisco.com@ece.cmu.edu on 09/08/2000 01:01:04 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: csapuntz@cisco.com Subject: Re: a vote for asymmetric connections in a session If I recall correctly, the windowing mechanism was put in to avoid a deadlock condition while using multiple control connections. I believe the logic went as follow: Assumption: You're not allowed to drop commands arriving in on any connection Because the TCP connections work at different rates, commands may not arrive in order. Let's say you have four connections and a command queue depth of 3. Commands #2,#3, and #4 arrive before command #1, filling the queue. However, the queue cannot be drained until command #1 arrives. So, deadlock. Solution: Allow commands to be dropped In the example above, we'd probably drop command #4 and replace it with command #1. Note that using a single TCP connection for control/data is not sufficient to solve the problem. Imagine the case that the queue is depth 3 and the host issues four WRITES commands in quick order. The device stops reading the TCP connection after three commands. Unfortunately, none of those three writes can complete until data is read from that connection! So, deadlock again. CIFS, NFS, and HTTP don't have this problem because they immediately send data after their WRITE commands. -Costa From owner-ips@ECE.cmu.edu Sat Sep 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12718; 9 Sep 2000 2:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA29430 for ips-outgoing; Sat, 9 Sep 2000 01:36:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA29425 for ; Sat, 9 Sep 2000 01:36:48 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id 0C5A9223 for ; Sat, 9 Sep 2000 01:36:47 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA17187 for ; Fri, 8 Sep 2000 22:36:44 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAAAE2 for ; Fri, 8 Sep 2000 22:36:41 -0700 Message-ID: <39B9CC5E.17D6D89C@agilent.com> Date: Fri, 08 Sep 2000 22:36:30 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Enhancements for the iSCSI References: <007f01c019e3$0edfea40$1cbe003e@e4u5e0> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Nelson Nahum wrote: > I think that the iSCSI is a special opportunity to enhance the current SCSI > protocol especially it networking capabilities. > SCSI was designed with the concept of Initiator for the HBA and Target for > the devices. The Initator sends commands to the Targets that are passive. > But the networking world supposes every one talking with every one. The fact > that iSCSI is implemented over TCP/IP can help to develop these enhacements. > > Modern SAN appliances need to "talk" with the storage devices but also need > a way to update information in the HBAs in a asynchronous manner. > > My questions are the following: > > 1 - Is iSCSI designed in such a way that every Target can also be Initiator > and viceversa or it supposes different charasterisitics (memory, CPU power > etc.) between the Initiator and the Target? yep. > 2 - Is possible to add asynchronous events that the Target could send to the > Initiators at any time? That's in the spec too. I guess you haven't read it very closely. > 3 - Another important enhancement could be the possibility to "broadcast" > information to many iSCSI devices or HBAs. Why would you want to do that? > Nelson Nahum > CTO > StoreAge Networking Technologies Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Sat Sep 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12787; 9 Sep 2000 2:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA29530 for ips-outgoing; Sat, 9 Sep 2000 01:42:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA29525 for ; Sat, 9 Sep 2000 01:42:44 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 4E6F12B1 for ; Sat, 9 Sep 2000 01:42:43 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA17290 for ; Fri, 8 Sep 2000 22:42:40 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAAB01 for ; Fri, 8 Sep 2000 22:42:37 -0700 Message-ID: <39B9CDC2.399166F4@agilent.com> Date: Fri, 08 Sep 2000 22:42:26 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Data in SCSI Response or SCSI Data References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > > > then it should send a response structure as FCP and not one or two and > > > use it according to FCP. Undefined information of what some > > consider SCSI > > > values cause variants in use. I have not seen tape drives use > > Good Sense > > > residual but vary in how Check Sense residual is presented. If > > iSCSI is to > > > copy FC, then why deviate on response symmetry? At least this > > provides an > > > easier bridge. Iterative read operations will carry unused values to be > > > examined and opens the door for further variants. Status > > presented prior to > > > data must then ensure sequence without end confirmation. > > > > Who said anything that status was presented before data? > > iSCSI allows the LAST iSCSI PDU that completes a SCSI command to > > also contain > > the GOOD SCSI status in the header, so that extra overhead to > > send the status > > PDU. > > By placing status within the header of perhaps 4G of data payload with no > defined end structure leaves nothing known to follow this transfer. By > making Response a separate structure that physically follows the data > payload there is an end confirmation. Should there be an error in sequence, > data will be accepted followed by perhaps some unknown command responses > until everything falls back into sync. A dubious savings having dummy > status in each read exchange to save one possible structure at the end. > Rather than processing one response structure, you must process two where > one is out of sequence to the function. > > Doug Doug, this is TCP - a guaranteed byte stream. If the target successfully reads 4G of data off its media into cache memory, and transmits it as one iSCSI message, TCP will deliver the data/message. If TCP fails for some reason to deliver the message (and everything after it), then the initiator knows because the number of bytes read from TCP do not match the length of the iSCSI message (indicated in the header). Now, if the header also indicates good status (as read from the target media into target cache), what's the big deal that its in the header as opposed to a separate message after the data iSCSI message? -Matt From owner-ips@ECE.cmu.edu Sat Sep 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12723; 9 Sep 2000 2:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA29199 for ips-outgoing; Sat, 9 Sep 2000 01:24:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA29194 for ; Sat, 9 Sep 2000 01:24:55 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 78BAE86C for ; Sat, 9 Sep 2000 01:24:54 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA17114 for ; Fri, 8 Sep 2000 22:24:53 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAAACB for ; Fri, 8 Sep 2000 22:24:50 -0700 Message-ID: <39B9C997.54D39EE6@agilent.com> Date: Fri, 08 Sep 2000 22:24:39 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I think Costa's point is in the asymmetric single connection model: The target has room for 3 commands. The initiator sends 3 commands, followed by data, followed by a 4th command. The target could stop reading the TCP connection because the command queue is full, because it doesn't know that data is next on the TCP connection. It has to assume that a 4th command is coming, and it doesn't have room for it. Another case for a "command" tcp connection and a separate "data" connection (minimum of two TCP connections per session). -Matt Joshua Tseng wrote: > I don't get it. Maybe I'm dumb or something, but I thought the whole > point of asymmetric was that commands wouldn't arrive out of order. > > Josh > > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Friday, September 08, 2000 4:22 PM > To: ips@ece.cmu.edu > Subject: Re: a vote for asymmetric connections in a session > > I do not know how reasonable Costa's configuration is,in the real world, > but he has brought up a point that, I think, might be important to address. > If his example is reasonable, then it would suggest that we NEED to enforce > the requirement of Two connections per Session in the Asymmetric case. > What do you all think about this? > > . > . > . > John L. Hufferd > > csapuntz@cisco.com@ece.cmu.edu on 09/08/2000 01:01:04 PM > > Sent by: owner-ips@ece.cmu.edu > > To: ips@ece.cmu.edu > cc: csapuntz@cisco.com > Subject: Re: a vote for asymmetric connections in a session > > If I recall correctly, the windowing mechanism was put in to avoid > a deadlock condition while using multiple control connections. > I believe the logic went as follow: > > Assumption: You're not allowed to drop commands arriving in on any > connection > > Because the TCP connections work at different rates, commands > may not arrive in order. Let's say you have four connections and > a command queue depth of 3. Commands #2,#3, and #4 arrive before command > #1, > filling the queue. However, the queue cannot be drained > until command #1 arrives. So, deadlock. > > Solution: Allow commands to be dropped > > In the example above, we'd probably drop command #4 and replace it > with command #1. > > Note that using a single TCP connection for control/data is not > sufficient to solve the problem. Imagine the case that the queue is > depth 3 and the host issues four WRITES commands in quick order. > > The device stops reading the TCP connection after three commands. > Unfortunately, none of those three writes can complete until > data is read from that connection! So, deadlock again. > > CIFS, NFS, and HTTP don't have this problem because they immediately > send data after their WRITE commands. > > -Costa From owner-ips@ECE.cmu.edu Sat Sep 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12778; 9 Sep 2000 2:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA29697 for ips-outgoing; Sat, 9 Sep 2000 01:53:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA29692 for ; Sat, 9 Sep 2000 01:53:40 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id A095A1E7 for ; Fri, 8 Sep 2000 22:53:39 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA17443 for ; Fri, 8 Sep 2000 22:53:37 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAAB17 for ; Fri, 8 Sep 2000 22:53:33 -0700 Message-ID: <39B9D052.30B6B7B1@agilent.com> Date: Fri, 08 Sep 2000 22:53:22 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua Tseng wrote: > >As Julian has stated in a different thread, the purpose of the "sliding > >windows" in iSCSI is not for congestion management. It is simply there to > >handle the case where if a connection goes down in a multiple connection > >session, it prevents the remaining connections from overwhelming the target > >with new commands that it can't process due to missing commands that where > on > >the broken connection. > > Matt, > > In the event of a failed connection, the sliding window isn't going to help > much since you have to first detect the failed connection. Once this is > accomplished, the initiator will know which command was lost for sure, How is the initiator going to know what commands got to the target or not without the command reference numbers in the iSCSI document? > but will not know how many, if any, of the subsequent commands sent after the > lost command, were also lost. Well, all of them on the failed connection - if the connection failed, the following commands could not have been delivered. > I presume, the initiator will then > retransmit all commands without knowing whether or not they were lost, > rather than wait for the target to advance the acknowledgement counter. > > This is all awesome and impressive behavior, exactly what we want. The only > problem is this exact same behavior can be had without the windowing > mechanism. Could you describe how it can be had? > > > >Since all of this runs on top of TCP, and TCP performs congestion > >management, why must iSCSI perform congestion management on top of TCP? > > Please see my reply to John Hufferd. > > Josh -Matt From owner-ips@ECE.cmu.edu Sat Sep 09 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12776; 9 Sep 2000 2:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA29332 for ips-outgoing; Sat, 9 Sep 2000 01:32:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA29327 for ; Sat, 9 Sep 2000 01:31:59 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id 067E314A for ; Sat, 9 Sep 2000 01:31:59 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA17165 for ; Fri, 8 Sep 2000 22:31:57 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAAADB for ; Fri, 8 Sep 2000 22:31:55 -0700 Message-ID: <39B9CB40.545BC073@agilent.com> Date: Fri, 08 Sep 2000 22:31:44 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk If SCSI (T10, SAM-x?) can't solve the problem of flow controlling commands, then iSCSI can by using at least 2 TCP connections, one for commands and 1+ for data. The commands can back up, and be flow controlled by TCP and the data to satisfy the executing commands can continue. If you don't want to do this, then T10 (SAM) will have to address it, and it sould NOT be an issue of the transport (iSCSI). -Matt Joshua Tseng wrote: > Hi John, > > >I could be mistaken here but I think you, and some others have been > >addressing issues with the Storage Controller having the right amount of > >buffer space in the controllers for the commands, after they are delivered > >by iSCSI. I think this is a SCSI issue and not a iSCSI/TCP flow control > >problem. > > Yes, you are right that this is precisely the problem I'm addressing. Maybe > I'm missing something here, but I still think there is an issue. Yes, > as James described earlier SCSI has a mechanism for dealing with this, but > is it sufficient for iSCSI/TCP? IP has distinctly different characteristics > than parallel SCSI and FC. > > For one thing, latencies in the IP world are much higher. I'm assuming > parallel SCSI latencies are next to nothing, while FC are in the <10us range. > If > iSCSI is going over the WAN, at least 50ms latencies are to be expected for > coast-to-coast US. Over the Public Internet, we're talking about 80 to 250ms > or even more (RT). Much of this latency is caused by the speed of light, > the rest by the serialization process for WAN transport. This works out to be > > a huge amount of latency compared to FC. How much can happen in the 30ms > before the SCSI QUEUE FULL and BUSY messages finally get back to the > initiator. > Are you sure leaving it to SCSI or TCP will be okay? > > >With respect to the Asymmetric approach, the issues change, and we are no > >longer trying to solve the problem of missing commands that occur because > >of a broken connection. Therefore, I think we can dump the sliding window > >and leave the flow control -- and recovery of lost commands, -- up to > >TCP. Yes, the SCSI layers on each end need to have their own buffer > >management under control, but I do not think this is a transport issue. > > I agree dumping the sliding window is a good thing. But my question is should > > iSCSI replace it with something else. I believe the issue James brought up > with multiple initiators needs to be considered. Or even with a single > initiator, > how do you flow control the oncoming commands from the initiator(s)? Is the > SCSI mechanisms sufficient given the latencies that iSCSI must be designed > to support? > > BTW, I don't think that TCP has any role in addressing this issue. If iSCSI > and > TCP operate independently at different layers, then TCP won't flow control > iSCSI commands any more than it flow controls any other PDU's, because it > can't tell the difference between them. > > Josh From owner-ips@ECE.cmu.edu Sat Sep 09 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13185; 9 Sep 2000 4:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA00958 for ips-outgoing; Sat, 9 Sep 2000 03:06:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA00953 for ; Sat, 9 Sep 2000 03:06:48 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id DAA41614; Sat, 9 Sep 2000 03:00:49 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id BAA103088; Sat, 9 Sep 2000 01:06:47 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: a vote for asymmetric connections in a session To: Joshua Tseng Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 9 Sep 2000 00:05:18 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/09/2000 01:06:46 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, I do not understand whether this is a real issue or not. I know of a lot of Client Server applications that ship a lot of data on TCP/IP, and TCP/IP seems to be adequate. Now I understand there are some differences in Direct Storage Access, but it is more like the other applications then it is different. If we can address Costa's Issue or have at least two Connection per Asymmetric Session, I am not sure anything else is a real life problem. But I can be convinced, however, I would like to understand for each problem we come up with, why it is only a problem for iSCSI and not for the other real world applications, and why the SCSI layer can not handle the problem. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/08/2000 08:53:48 PM Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng , Charles Monia cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Hi John, >I could be mistaken here but I think you, and some others have been >addressing issues with the Storage Controller having the right amount of >buffer space in the controllers for the commands, after they are delivered >by iSCSI. I think this is a SCSI issue and not a iSCSI/TCP flow control >problem. Yes, you are right that this is precisely the problem I'm addressing. Maybe I'm missing something here, but I still think there is an issue. Yes, as James described earlier SCSI has a mechanism for dealing with this, but is it sufficient for iSCSI/TCP? IP has distinctly different characteristics than parallel SCSI and FC. For one thing, latencies in the IP world are much higher. I'm assuming parallel SCSI latencies are next to nothing, while FC are in the <10us range. If iSCSI is going over the WAN, at least 50ms latencies are to be expected for coast-to-coast US. Over the Public Internet, we're talking about 80 to 250ms or even more (RT). Much of this latency is caused by the speed of light, the rest by the serialization process for WAN transport. This works out to be a huge amount of latency compared to FC. How much can happen in the 30ms before the SCSI QUEUE FULL and BUSY messages finally get back to the initiator. Are you sure leaving it to SCSI or TCP will be okay? >With respect to the Asymmetric approach, the issues change, and we are no >longer trying to solve the problem of missing commands that occur because >of a broken connection. Therefore, I think we can dump the sliding window >and leave the flow control -- and recovery of lost commands, -- up to >TCP. Yes, the SCSI layers on each end need to have their own buffer >management under control, but I do not think this is a transport issue. I agree dumping the sliding window is a good thing. But my question is should iSCSI replace it with something else. I believe the issue James brought up with multiple initiators needs to be considered. Or even with a single initiator, how do you flow control the oncoming commands from the initiator(s)? Is the SCSI mechanisms sufficient given the latencies that iSCSI must be designed to support? BTW, I don't think that TCP has any role in addressing this issue. If iSCSI and TCP operate independently at different layers, then TCP won't flow control iSCSI commands any more than it flow controls any other PDU's, because it can't tell the difference between them. Josh From owner-ips@ECE.cmu.edu Sat Sep 09 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13190; 9 Sep 2000 4:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA00548 for ips-outgoing; Sat, 9 Sep 2000 02:41:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA00544 for ; Sat, 9 Sep 2000 02:41:08 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id CAA41698; Sat, 9 Sep 2000 02:35:09 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id AAA128662; Sat, 9 Sep 2000 00:41:07 -0600 Importance: Normal Subject: RE: a vote for asymmetric connections in a session To: Joshua Tseng Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 8 Sep 2000 23:36:43 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/09/2000 12:41:06 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk They are not out of order, you need to reread his note. . . . John L. Hufferd Joshua Tseng on 09/08/2000 08:53:41 PM To: John Hufferd/San Jose/IBM@IBMUS, ips@ece.cmu.edu cc: Subject: RE: a vote for asymmetric connections in a session I don't get it. Maybe I'm dumb or something, but I thought the whole point of asymmetric was that commands wouldn't arrive out of order. Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Friday, September 08, 2000 4:22 PM To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session I do not know how reasonable Costa's configuration is,in the real world, but he has brought up a point that, I think, might be important to address. If his example is reasonable, then it would suggest that we NEED to enforce the requirement of Two connections per Session in the Asymmetric case. What do you all think about this? . . . John L. Hufferd csapuntz@cisco.com@ece.cmu.edu on 09/08/2000 01:01:04 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: csapuntz@cisco.com Subject: Re: a vote for asymmetric connections in a session If I recall correctly, the windowing mechanism was put in to avoid a deadlock condition while using multiple control connections. I believe the logic went as follow: Assumption: You're not allowed to drop commands arriving in on any connection Because the TCP connections work at different rates, commands may not arrive in order. Let's say you have four connections and a command queue depth of 3. Commands #2,#3, and #4 arrive before command #1, filling the queue. However, the queue cannot be drained until command #1 arrives. So, deadlock. Solution: Allow commands to be dropped In the example above, we'd probably drop command #4 and replace it with command #1. Note that using a single TCP connection for control/data is not sufficient to solve the problem. Imagine the case that the queue is depth 3 and the host issues four WRITES commands in quick order. The device stops reading the TCP connection after three commands. Unfortunately, none of those three writes can complete until data is read from that connection! So, deadlock again. CIFS, NFS, and HTTP don't have this problem because they immediately send data after their WRITE commands. -Costa From owner-ips@ECE.cmu.edu Sat Sep 09 09:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13660; 9 Sep 2000 4:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA02221 for ips-outgoing; Sat, 9 Sep 2000 04:07:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA02121 for ; Sat, 9 Sep 2000 04:07:12 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA10166 for ; Sat, 9 Sep 2000 10:06:40 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA10612 for ; Sat, 9 Sep 2000 10:06:40 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.002C8C57 ; Sat, 9 Sep 2000 10:06:35 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 10:33:05 +0300 Subject: Re: a vote for asymmetric connections in a session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk A note of caution. The most serious dead-lock sitaution we are aware of steams from a mix of RTT (or should we call it R2T to accommodate Doug Ottis?) and unsolicited/immediate data. If channels are full with unsolicited data and the target requests something else - that something else will not get through. This dead-lock, as far as I can tell exists in all transports. A target should be able to detect it and iSCSI has provided for the target to be able to drop data and reclaim them later with R2T. To minimize the chances for it we are also not providing for switching between operating modes (R2T mandatory or not). Julo "John Hufferd/San Jose/IBM" on 08/09/2000 18:42:12 Please respond to "John Hufferd/San Jose/IBM" To: "Randall R. Stewart" cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: a vote for asymmetric connections in a session Randall, Your head-of-line blockage statement, is of course true, but you say it like it is a bad thing. I am not sure it is. In the Asymmetric configuration, there is a different TCP/IP connections from each Host to a Storage Controller to handle the Commands from that Host to the Storage Controller (or at least that Storage Controller's Port). Each Host has SCSI commands that are not related to the commands in other hosts flowing to the same Storage Controller (port) but on separate TCP/IP connections. They expect to have their commands handled in some "fair" allocation method, with respect to, the other Hosts. Nothing is blocking its "line" except its own Commands. So TCP/IP has delivered, or brought up to be delivered, each Hosts commands independently. Now it is up to the Target SCSI layer to chose which port/interface from which to bring additional commands into its buffers. I do not see this as an important problem, and is the way things have worked regardless of the storage connections, for as long as we have had storage connections. It was true on the SCSI buss, and it was even true on the Old IBM 360s (and continues to this day on the mainframes). There will always be back pressure and sooner or later it causes the Initiator to stop sending new packets, since they are not being acknowledged, this will either occur at the TCP/IP layer or at the SCSI layer. I am not sure that this is a bad thing, especially since we are not talking about dead locks. . . . John L. Hufferd "Randall R. Stewart" @ece.cmu.edu on 09/08/2000 04:58:06 AM Sent by: owner-ips@ece.cmu.edu To: Charles Monia cc: Matt Wakeley , ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session Charles Monia wrote: > I'm not sure what is meant by "congestion." If we're talking about > congestion in the TCP/IP transport, I'm in agreement. However, I thought we > were referring to the sort of congestion that the application on top of > iSCSI might see if it received more commands than it had room for. > > Unless I misunderstood your point, threfore, I think there might be an > issue. The only way I can see flow control in the tranport layer being used > to avoid dropping commands is if higher layer congestion results in back > pressure to the iSCSI pipe. I believe that behavior is undesirable because > it introduces "head-of-line" blocking, with the following consequences: > Any time you get ANY retransmissions in TCP you will create a "head-of-line" blocking scenario. This does not matter if it is from flow control as you state or network congestion where a packet is dropped by a router. You will have situations where the TCP is holding data waiting for the retransmissions of an earlier packet... this is one of the reasons that sigtran developed SCTP... > a) It effectively shuts down the flow of commands to all logical units. > > b) It blocks the flow of task management commands (Abort Task, Clear task > set, etc). > > > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 09 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13665; 9 Sep 2000 5:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA02229 for ips-outgoing; Sat, 9 Sep 2000 04:07:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA02123 for ; Sat, 9 Sep 2000 04:07:11 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA21914 for ; Sat, 9 Sep 2000 10:06:39 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA10610 for ; Sat, 9 Sep 2000 10:06:39 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.002C8C92 ; Sat, 9 Sep 2000 10:06:35 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 10:37:25 +0300 Subject: RE: Multiple Connections. How would they work? Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, Sorry but NO. 01 has a notion of a session - that is an organized collection and the numbering for commands is done across a session to reestablish order. Recovery is also on a session base - a failed connection can have it's commands reissued. Julo Robert Snively on 08/09/2000 18:46:27 Please respond to Robert Snively To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Multiple Connections. How would they work? As I read it, the model in 01 was a "single" model, not a "symmetric" model. It did not place any limit on the number of "singles", but did not place any relationship values among them either. That makes ordering exactly the same as the present SCSI structure, which may have any number of initiator/target connections and paths, but treats each one of them as individual and, for most purposes, independent. That is a model I would favor, since I see no way to create interdependent paths without objectionable and expensive relationships being maintained among hardware NICs in either the hardware or software layers. These relationships have typically been handled very well and in highly optimized ways by wedge drivers and high availability file systems. Bob > The 00 draft (Adelaide) had the commands go down over one > specific control > stream while the data went on any available connection. > Synchronizing them and providing for hardware implementation > across several > adapters was felt to be complex. > > However command ordering was trivial. > > The symmetric model in 01 requires ordering and this is why > you have the > command ordering. > A proposal to get back to the asymmetric model but decide > ahead of time (? > when) what > path data will be taking was forwarded by Kalman. > > We call this the asymmetric model (it can still evolve). From owner-ips@ECE.cmu.edu Sat Sep 09 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13675; 9 Sep 2000 5:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA02231 for ips-outgoing; Sat, 9 Sep 2000 04:07:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA02122 for ; Sat, 9 Sep 2000 04:07:11 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA11728 for ; Sat, 9 Sep 2000 10:06:40 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA10614 for ; Sat, 9 Sep 2000 10:06:40 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.002C8C11 ; Sat, 9 Sep 2000 10:06:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 09:35:04 +0300 Subject: Re: Symmetric vs Asymmetric Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, The main reason why we had 2 connections in draft-00 was that we did not like the TCP window being closed by a large data transfer preventing commands to go through (that is Matt's flow control argument). Julo David Robinson on 08/09/2000 03:31:34 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Symmetric vs Asymmetric > The biggest issue with the "asymmetric" model is that it is NOT "asymmetric" > when there is only 1 TCP connection. When there is only one TCP connection, it > is the "symmetric" model - both commands and data on the same TCP connection. > Then, when there is more than one TCP connection, the behavior is different. > It's always easier to implement something that operates that same way all the > time, than two different behaviors. > > I propose that the asymmetric model mandate at least two TCP connections > (implies that at least one physical connection will have at least two TCP > connections running on it) - one for commands, the other for data. This has > other advantages, like commands not being flow controlled by large transfers of > data. That's not a bug, that's a feature! Having the asymmetric degenerate into symmetric with just one connection is a good thing. For starters it is easier to implement initially or in cheap devices and doesn't have the baggage that a true symmetric design would require but is not needed with one connection. I would oppose mandating two connections minimum, if flow control is a problem then the defacto configuration will be two connections, but lets not require it. Personally I would still prefer one connection per LUN, but the proposed asymmetric model is a good compromise. You could still deploy an implementation with a connection per LUN and not have any significant unnecessary baggage. Not true of the symmetric model. -David From owner-ips@ECE.cmu.edu Sat Sep 09 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13679; 9 Sep 2000 5:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA02190 for ips-outgoing; Sat, 9 Sep 2000 04:07:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA02070 for ; Sat, 9 Sep 2000 04:07:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA178800 for ; Sat, 9 Sep 2000 10:06:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA38714 for ; Sat, 9 Sep 2000 10:06:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.002C872E ; Sat, 9 Sep 2000 10:06:21 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 08:31:36 +0300 Subject: RE: Re: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt & Somesh point to one of the main weaknesses of the asymmetric scheme. However the initiator adapters setup is generally synchronous and the overhead is not significant. For writes the TCP window will take care of the holding off the data that arrive before the command. The big question is - do we end-up better than with the symmetric scheme? Julo somesh_gupta@hp.com on 07/09/2000 03:12:05 Please respond to somesh_gupta@hp.com To: ips@ece.cmu.edu, matt_wakeley@agilent.com cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Re: multiple connections I think Matt has an important point here. Let us consider READs and WRITEs seperately. 1. READ - In this case the host software can post the appropriate read buffers to the connection on which the data is to be received and then post the read command to the connection on which the data is to be sent. By the time the target gets the command, and returns data, it is most likely that the NIC receiving the data will have accepted the read buffers. Although synchronization issues are minimized, there is a reasonable amount of performance impact. Two different "commands" (in the sense of host to adapter commands - not in the SCSI sense) have to be posted. Two different queues have to be written to (for sure if they are on different NICs) and two different NICs triggered to accept the "commands". Then there will be two different completion events that will be returned by the NICs to the host - one indicating the acceptance/"completion"/"return" of the command buffer (indicating command sent successfully), and the second when the data is all received (along with the SCSI command status). This may also require interrupt handling etc which could be minimized by some clever schemes. So the amount of work done by the host for the host/adapter interaction is close to doubled. There is probably not that much impact on the target in this case. 2. WRITES - This is the really bad one in my opinion. For me, avoiding RTTs in iSCSI would just by itself make iSCSI a superior "transport" for SCSI. So assuming RTTs are not being used, the host would (changing the posting order from READs), first post the write command to the connection on which the data is to be sent, and then post the write buffers to the connection which is to be used for sending data. One case is where for whatever reason, the target gets the data before it gets the command, and has no clue what to do with the data. Let us assume that the target does get the command before it gets the data. The target gets a command indicating that data being written to whatever lun and whatever location is going to arrive on some other connection. First the target has taken an extra event from the adapter. Then, when data arrives on another NIC (and the target gets another event), the target goes through the list of outstanding WRITE commands to match the command with the data and then go about the business of processing the data. So in this case the work has increased for the initiator as well as the target. Somesh > -----Original Message----- > From: matt_wakeley@agilent.com [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 1:17 AM > To: ips@ece.cmu.edu > Subject: FW: Re: multiple connections > > > julian_satran@il.ibm.com wrote: > > > Dear colleagues, > > > > With all the heated debate about multiple vs. single > connection a request I > > made a while ago got no significant reply (neither for nor against). > > > > The request was to consider a proposal made by Kalman Meth > to reconsider > > the asymmetric model with the addition of a path selection > made by the > > initiator. > > One downside I see to this proposal is that for a host with > multiple iSCSI > nics, it will have to issue two commands (in this order): one > to the "data path > nic" to describe where the data is to be read from/written > to, and one to the > "control path nic" to send the command. This would be > communication to two > cards instead of one. Also, note that the communication to > the data path nic > MUST complete before the control path nic sends the command, > requiring some > kind of sync mechanism. > > > > This proposal allows removing the command counters - as > commands use a > > single TCP connection. The single connection can also be a shared > > data+control connection. > > For the case of the command connection failure and fail-over to a new > connection, I don't see how you can get away from the command > counters. When a > fail over occurs, you will need some way of finding out what > commands made it > to the target and which didn't. The easiest way to do this > is with command > numbering. > > > > > > > In case of multiple connection the data path to be used is > selected and > > maintained until the command ends. > > > > Thanks, > > Julo > > -Matt Wakeley > Agilent Technologies > From owner-ips@ECE.cmu.edu Sat Sep 09 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13704; 9 Sep 2000 5:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA02177 for ips-outgoing; Sat, 9 Sep 2000 04:07:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA02069 for ; Sat, 9 Sep 2000 04:07:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA234082 for ; Sat, 9 Sep 2000 10:06:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA38712 for ; Sat, 9 Sep 2000 10:06:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.002C8972 ; Sat, 9 Sep 2000 10:06:27 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 09:08:21 +0300 Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, Command ordering was very much on our minds. With the symmetric model - this is why we have the command numbering scheme and the "ordering window". With the symmetric model - as commands travel on one TCP connection only TCP will guarantee that they arrive in order (TCP is a streaming protocol and guarantees ordering). Julo Charles Monia on 07/09/2000 23:52:22 Please respond to Charles Monia To: Julian Satran/Haifa/IBM@IBMIL cc: "Ips (E-mail)" Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Hi Julo: > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, September 07, 2000 6:11 AM > To: ips@ece.cmu.edu > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > Dear colleagues, > > Although the windowing mechanism in iSCSI-01 may seem to be > there to solve > a queueing issue > it is mainly meant to limit the buffering space for commands > that await > "de-skewing". > We assume that execution queue-lengths, policy etc. are > beyond the scope of > transport. > > As for SCSI queue length I assumed that the busy or queue full status > followed by an Asynch Event > message indicating readiness is the mechanism provided by > SCSI to regulate > the command flow. > > It is hard to imagine that give the variable life-time of > SCSI commands and > the > opaque nature of the resources required to execute them that > the transport > has > to help in this area. > While this issue has been discussed at some length in the past, as Jim McGrath stated, I believe the debate ought to be reopened (although we may end up reaching the same conclusion as before). As Ralph Weber pointed out, the SCSI model is to discard the command, return status, retrieve the next command in the transport pipeline and continue processing. The initiator is not notified when processing resumes. If there are many commands in flight, as there could be in an IP environment, and target resources free up in the meantime, the result is commands processed out of order. Historically, such a lapse in command ordering was not seen as an issue for the following reasons: a) Strict ordering was not required by the most commonly deployed device types (disks and tapes). Due to the nature of disk traffic, a simple retry mechanism was deemed sufficient to recover from these errors. Since legacy streaming devices, such as tapes, did not support command queuing, command ordering considerations were not a factor there either. b) Transport delays over storage interconnects were small, so not many commands were apt to be in flight. i.e.. The window for such errors was very small. c) The resource guarantees needed for a loss-avoidance mechanism in the target adversely effected device cost, especially at the high-volume, low-end of the market. Given the above considerations, there was little support within the storage community for measures addressing this issue. If we now believe that the iSCSI environment changes the rules, I believe the interconnect protocol can provide useful assists, such as: a) On a command overflow condition, have the iSCSI target flush the command pipeline by returning status and discarding all subsequently received commands until a host acknowledgement is received. b) Implement some sort of credit-based mechanism for overflow-avoidance. Comments? > Jim McGrath on 07/09/2000 06:06:40 > > Please respond to Jim McGrath > > To: "'Matt Wakeley'" , ips > > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Command Queue Depth (was asymmetric/Symmetric) > > > > > > The issue of buffer space allocation for multiple initiators > has a long and > troubled history in SCSI. We have never been able to come up > with a good > answer. > > Fibre Channel tried to fix this with the notion of "login BB > credit" - when > you login you get a minimum number of credits you are always > guaranteed > when > you start data transfers. The problem with this is that > storage devices > had > no realistic ability to discriminate between initiators or to > change the > login BB credit. In addition, the expectation is that all possible > initiators would get these credits on login. So storage > devices vendors > have played it safe and kept this number low (at 0 until recently, now > around 2). For iSCSI the number of initial credits you need > to "prime the > pump" until normal data flow is established is probably large > (given the > latencies are higher than in Fibre Channel, especially FC-AL), and the > number of potential initiators larger than in Fibre Channel, > making this a > whole lot worse for the storage device. > > As soon as we allow the devices to start adjusting these > credits, then you > have the protocol problem of making sure people know when > their credits are > adjusted and the policy problem of how, who, and when to adjust the > credits. > Changing everyone's credit when you add a new initiator can get into a > notification nightmare, although it is "fair." Any policy > brings up all > sorts of nasty issues regarding fairness vs efficient use of the > transmission media. > > Jim > > Note: the same problem has plagued other attempts to allocate device > resources between multiple initiators, like command queue space. In > general > policies with respect to multiple initiators are not really > standard in the > SCSI world. > > > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 2:56 PM > To: ips > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > Joshua Tseng wrote: > > > James, > > > > I agree with others that there may be an issue with the > > command windowing mechanism in the existing iSCSI spec. It is like > > "TCP in reverse", in that the target determines the size of > the window, > and > > not the initiator as in TCP. Rather, I believe that > everything that this > > windowing mechanism is attempting to achieve can be more > easily obtained > > by having the target communicate its buffer size to the initiator at > > iSCSI login. It should be the role of the initiator to > determine how > > many commands to put in flight simultaneously, given this input on > available > > buffer size from the target. > > As more initiators connect to a target, it may need to scale back the > amount > of > this buffering it has allocated to each previously logged in > initiator (to > prevent rejecting new logins). > > > > > > > As far as multiple initiators, could this not be resolved > by the target > > refusing additional logins beyond the number of initiators > it can safely > > support? Not being a storage expert, this is my best > guess/suggestion > > at how to do it. > > I believe John already answered this... > > -Matt > > > Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Sat Sep 09 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13677; 9 Sep 2000 5:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA02199 for ips-outgoing; Sat, 9 Sep 2000 04:07:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA02073 for ; Sat, 9 Sep 2000 04:07:02 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA101830 for ; Sat, 9 Sep 2000 10:06:31 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA18942 for ; Sat, 9 Sep 2000 10:06:31 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.002C89C1 ; Sat, 9 Sep 2000 10:06:28 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 09:27:22 +0300 Subject: Re: Symmetric vs Asymmetric Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree. It has the added advantage that the header structure for data can be streamlined and still keep with our constant header length - although a different constant for data. We are close to the draft-00 structure with the recovery from draft-01. Julo "Matt Wakeley" on 08/09/2000 02:56:01 Please respond to Matt Wakeley To: John Hufferd/San Jose/IBM@IBMUS, ips cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Symmetric vs Asymmetric John Hufferd/San Jose/IBM wrote: > David, > One of the reasons I was holding out for Symmetric over Asymmetric, was for > the workload balancing etc. that could be done at the iSCSI layer. > > I am now willing to "roll-over" but I think I owe you all my reasons for > deciding to support the Asymmetric approach: > > First, in the degenerate case, they both look the same. That is, they can > both support a single Connection per Session. So the simplest > implementation is still possible with the Asymmetric approach as it was > with the Symmetric approach. This means that most current iSCSI > implementations can continue to exist, unchanged. The biggest issue with the "asymmetric" model is that it is NOT "asymmetric" when there is only 1 TCP connection. When there is only one TCP connection, it is the "symmetric" model - both commands and data on the same TCP connection. Then, when there is more than one TCP connection, the behavior is different. It's always easier to implement something that operates that same way all the time, than two different behaviors. I propose that the asymmetric model mandate at least two TCP connections (implies that at least one physical connection will have at least two TCP connections running on it) - one for commands, the other for data. This has other advantages, like commands not being flow controlled by large transfers of data. -Matt Wakeley Agilent Technologies > > > But the most important reason follows: > > Since a session is a unique construct that identifies an Initiator and a > Target, I am assuming that does NOT mean each NICs has a unique initiator > ID but that they can, and should, share a common Initiator ID. The Target > ID, on the other hand, is either unique per Storage Controller, or not, > and this will depend on the type of Storage Controller. This means that > the Host iSCSI device driver will be able to present, to the layers above > it, either a single port per storage controller, or multiple ports, > depending on how the Storage Controller answers Logins on each of its IP > connections. > > That means iSCSI will not be able to determine when it knows enough to > perform alternate path retry, and for the same reason will not be able to > know when or how to perform command work load balancing across the > different sessions. This is because either all connections to a storage > controller have either the same iSCSI session ID (SSID), or they are > different -- but iSCSI will not, by itself, know which sessions can be > substituted for others (for balancing or retry). > > The only place where the symmetric approach can do workload balancing and > alternate path retry is where the storage controller makes it look like a > single target ID even though it might have multiple IP addresses. Though > this might work in some IBM storage controllers (like Shark) it will not > work for all IBM storage controllers (like those of Mylex). This same or > similar argument probably will apply to EMC's and other vendors various > storage controllers. Therefore, having iSCSI do any type of command > balancing or command automatic alternate path retry is problematical > without outside influences (probably from upper layers) and this will tend > to make the iSCSI layer very thick and complicated. It will also mix the > layering concepts (in general not a good idea). Therefore, I now believe > that the Symmetrical approach can not be a reasonable general/generic > approach to Command work load balance or alternate path retry, and that > function is best left to the Wedge Drivers. > > The Asymmetric approach can not perform command balancing or command > automatic path retry either, but it does not try. It only tries to balance > the data across multiple connections (and that seems much more reasonable). > > With the Asymmetric approach, data for any specific command only flows on > one connection. So you can look at the data from various commands as > independent flows on the different connections (NICs) so there should not > be blocking issues, or parallel flow issues, and RDMA type direct to memory > functions can occur. > > With multiple storage controllers, there may be many sessions and many data > connections spread across all the various NICs in a Host. > > The net of all this is that the Asymmetric approach: > Probably has good enough spread of commands and data across multiple > NICs, even without Wedge Drivers. > Permits the upper layer Wedge Drivers to be left mostly unchanged. > Permits the simple case with a single connection per session, even with > multiple NICs. > Permits the most simple case of a single connection per session with > only a single NIC. > > Since I have seen no one else that made arguments for the Symmetric > implementation, you maybe able to call the agreement on the Asymmetric > approach a consensus. > . > . > . > John L. Hufferd From owner-ips@ECE.cmu.edu Sat Sep 09 09:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13774; 9 Sep 2000 5:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA02247 for ips-outgoing; Sat, 9 Sep 2000 04:07:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA02117 for ; Sat, 9 Sep 2000 04:07:10 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA107028 for ; Sat, 9 Sep 2000 10:06:40 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA10608 for ; Sat, 9 Sep 2000 10:06:39 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.002C8CCB ; Sat, 9 Sep 2000 10:06:36 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 10:57:24 +0300 Subject: RE: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Overrun/Underrun situations are a "bridging patch" from the block count model used by the "classical SCSI" (perhaps matched by some buffering model in some transports) and the byte count model specified by SAM. As a command has a byte count associated to it and the CDB has a block count the resulting inconsistency (if any) is reported in overrun/underrun byte count delivered by the target. As for a non-zero residual being an error, as far as I can read it - it is today a target decision and except for some legacy devices most do not report it as an error ( a far cry from the 360 channel days where it was an error unless explicitely requesting it not to be by a bit in the command). Julo "Douglas Otis" on 08/09/2000 21:50:16 Please respond to "Douglas Otis" To: "Robert Snively" , "Stephen Bailey" , ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Data in SCSI Response or SCSI Data Bob, Thanks, that helps an understanding of this PDU 0x45. If iSCSI is to copy FCP then it should send a response structure as FCP and not one or two and use it according to FCP. Undefined information of what some consider SCSI values cause variants in use. I have not seen tape drives use Good Sense residual but vary in how Check Sense residual is presented. If iSCSI is to copy FC, then why deviate on response symmetry? At least this provides an easier bridge. Iterative read operations will carry unused values to be examined and opens the door for further variants. Status presented prior to data must then ensure sequence without end confirmation. Doug > -----Original Message----- > From: Robert Snively [mailto:rsnively@Brocade.COM] > Sent: Friday, September 08, 2000 9:20 AM > To: 'Douglas Otis'; Stephen Bailey; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > In FCP, because FC is a multi-protocol environment that may have > drivers independent of the SCSI drivers, an actual buffer allocation > was transmitted independently of the SCSI command. Both overrun > and underrun indicators and a residual are returned. They are > protocol related, not SCSI related. > > There are cases where incorrect length indications are provided > in the SCSI model for both under-run and over-run cases, but they > are rare and typically associated with legacy tape drive programs. > > Bob > > > -----Original Message----- > > From: Douglas Otis [mailto:dotis@sanlight.net] > > Sent: Wednesday, September 06, 2000 6:26 PM > > To: Stephen Bailey; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > Steph, > > > > Although a logical use of a residual, I can not see where it > > is defined > > should the allocation length be greater than the returned > > length. Should > > the allocation be less than the response, this is a Check > > Condition and the > > residual is defined. It would seem a residual in a case > > with adequate > > allocation is not interesting or defined. Do you know where > > this mechanism > > you describe is defined? In normal use, without a Check > > Condition such > > information is not returned to the application. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Stephen Bailey > > > Sent: Wednesday, September 06, 2000 3:45 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > When do you get GOOD status and residual counts on a read? > > > What is causing > > > > the target to get the length wrong? > > > > > > One example is an INQUIRY command. The inquiry data length is > > > target-specific. Typically the CDB allocation length (and > > DL) are set > > > to some arbitrary large value (0xff), and the target sends back > > > everything it has. The transfer ends with success status. > > > > > > There can certainly be transfer residual and no SCSI error status. > > > > > > Steph > > > > > > > > From owner-ips@ECE.cmu.edu Sat Sep 09 10:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14090; 9 Sep 2000 6:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA11763 for ips-outgoing; Sat, 9 Sep 2000 05:29:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA11759 for ; Sat, 9 Sep 2000 05:29:24 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id LAA38838 for ; Sat, 9 Sep 2000 11:28:53 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id LAA40540 for ; Sat, 9 Sep 2000 11:28:53 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.003413F9 ; Sat, 9 Sep 2000 11:28:49 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 12:26:35 +0300 Subject: Re: Data in SCSI Response or SCSI Data Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk If we colud gain something by removing the residuals I would agree. But if we stay within the length and having residuals is generaly not considered an error then why do it? Julo "Matt Wakeley" on 08/09/2000 23:01:37 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Data in SCSI Response or SCSI Data Douglas Otis wrote: > Bob, > > Thanks, that helps an understanding of this PDU 0x45. If iSCSI is to copy > FCP I think this needs clarification. iSCSI is *NOT* copying FCP. iSCSI is attempting to transport SCSI over TCP/IP. It just so happens that the end result looks similar to what FCP did. > then it should send a response structure as FCP and not one or two and > use it according to FCP. Undefined information of what some consider SCSI > values cause variants in use. I have not seen tape drives use Good Sense > residual but vary in how Check Sense residual is presented. If iSCSI is to > copy FC, then why deviate on response symmetry? At least this provides an > easier bridge. Iterative read operations will carry unused values to be > examined and opens the door for further variants. Status presented prior to > data must then ensure sequence without end confirmation. Who said anything that status was presented before data? iSCSI allows the LAST iSCSI PDU that completes a SCSI command to also contain the GOOD SCSI status in the header, so that extra overhead to send the status PDU. Julian, if it will make this issue go away, maybe the residual fields should be removed from the data PDU. If there is residual, it will be sent in a status PDU. -Matt > > > Doug > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > Sent: Friday, September 08, 2000 9:20 AM > > To: 'Douglas Otis'; Stephen Bailey; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > In FCP, because FC is a multi-protocol environment that may have > > drivers independent of the SCSI drivers, an actual buffer allocation > > was transmitted independently of the SCSI command. Both overrun > > and underrun indicators and a residual are returned. They are > > protocol related, not SCSI related. > > > > There are cases where incorrect length indications are provided > > in the SCSI model for both under-run and over-run cases, but they > > are rare and typically associated with legacy tape drive programs. > > > > Bob > > > > > -----Original Message----- > > > From: Douglas Otis [mailto:dotis@sanlight.net] > > > Sent: Wednesday, September 06, 2000 6:26 PM > > > To: Stephen Bailey; ips@ece.cmu.edu > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > Steph, > > > > > > Although a logical use of a residual, I can not see where it > > > is defined > > > should the allocation length be greater than the returned > > > length. Should > > > the allocation be less than the response, this is a Check > > > Condition and the > > > residual is defined. It would seem a residual in a case > > > with adequate > > > allocation is not interesting or defined. Do you know where > > > this mechanism > > > you describe is defined? In normal use, without a Check > > > Condition such > > > information is not returned to the application. > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > Stephen Bailey > > > > Sent: Wednesday, September 06, 2000 3:45 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > When do you get GOOD status and residual counts on a read? > > > > What is causing > > > > > the target to get the length wrong? > > > > > > > > One example is an INQUIRY command. The inquiry data length is > > > > target-specific. Typically the CDB allocation length (and > > > DL) are set > > > > to some arbitrary large value (0xff), and the target sends back > > > > everything it has. The transfer ends with success status. > > > > > > > > There can certainly be transfer residual and no SCSI error status. > > > > > > > > Steph > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Sat Sep 09 15:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15382; 9 Sep 2000 11:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA16643 for ips-outgoing; Sat, 9 Sep 2000 10:18:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA16635 for ; Sat, 9 Sep 2000 10:17:58 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA56868 for ; Sat, 9 Sep 2000 16:17:21 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA62630 for ; Sat, 9 Sep 2000 16:17:22 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256955.004E7D08 ; Sat, 9 Sep 2000 16:17:18 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 9 Sep 2000 17:14:44 +0300 Subject: RE: a vote for asymmetric connections in a session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk That is correct. The window was put to limit the size of the command set waiting for de-skewing. At thew same time - aware of the possible large amounts of unsolicited data accompanying them we added the "RTT should be always be supported" statement to allow a target to drop data and reclaim it later through RTT. Julo "John Hufferd/San Jose/IBM" on 09/09/2000 09:36:43 Please respond to "John Hufferd/San Jose/IBM" To: Joshua Tseng cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: a vote for asymmetric connections in a session They are not out of order, you need to reread his note. . . . John L. Hufferd Joshua Tseng on 09/08/2000 08:53:41 PM To: John Hufferd/San Jose/IBM@IBMUS, ips@ece.cmu.edu cc: Subject: RE: a vote for asymmetric connections in a session I don't get it. Maybe I'm dumb or something, but I thought the whole point of asymmetric was that commands wouldn't arrive out of order. Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Friday, September 08, 2000 4:22 PM To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session I do not know how reasonable Costa's configuration is,in the real world, but he has brought up a point that, I think, might be important to address. If his example is reasonable, then it would suggest that we NEED to enforce the requirement of Two connections per Session in the Asymmetric case. What do you all think about this? . . . John L. Hufferd csapuntz@cisco.com@ece.cmu.edu on 09/08/2000 01:01:04 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: csapuntz@cisco.com Subject: Re: a vote for asymmetric connections in a session If I recall correctly, the windowing mechanism was put in to avoid a deadlock condition while using multiple control connections. I believe the logic went as follow: Assumption: You're not allowed to drop commands arriving in on any connection Because the TCP connections work at different rates, commands may not arrive in order. Let's say you have four connections and a command queue depth of 3. Commands #2,#3, and #4 arrive before command #1, filling the queue. However, the queue cannot be drained until command #1 arrives. So, deadlock. Solution: Allow commands to be dropped In the example above, we'd probably drop command #4 and replace it with command #1. Note that using a single TCP connection for control/data is not sufficient to solve the problem. Imagine the case that the queue is depth 3 and the host issues four WRITES commands in quick order. The device stops reading the TCP connection after three commands. Unfortunately, none of those three writes can complete until data is read from that connection! So, deadlock again. CIFS, NFS, and HTTP don't have this problem because they immediately send data after their WRITE commands. -Costa From owner-ips@ECE.cmu.edu Sat Sep 09 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16236; 9 Sep 2000 14:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA20600 for ips-outgoing; Sat, 9 Sep 2000 13:22:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA20594 for ; Sat, 9 Sep 2000 13:22:12 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA79602 for ; Sat, 9 Sep 2000 13:16:13 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id LAA110878; Sat, 9 Sep 2000 11:22:09 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: a vote for asymmetric connections in a session To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 9 Sep 2000 10:21:06 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/09/2000 11:22:09 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I think I understood what you said in the context of the Symmetric model , but could you please take me through how this would occur in the Asymmetric when you have at least two connections? I had thought, perhaps incorrectly, that the immediate data went with the command, thereby leaving the other connections, in the Asymmetric case, free for solicited data? Or Perhaps I miss understood your statement. . . . John L. Hufferd julian_satran@il.ibm.com@ece.cmu.edu on 09/09/2000 12:33:05 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: a vote for asymmetric connections in a session A note of caution. The most serious dead-lock sitaution we are aware of steams from a mix of RTT (or should we call it R2T to accommodate Doug Ottis?) and unsolicited/immediate data. If channels are full with unsolicited data and the target requests something else - that something else will not get through. This dead-lock, as far as I can tell exists in all transports. A target should be able to detect it and iSCSI has provided for the target to be able to drop data and reclaim them later with R2T. To minimize the chances for it we are also not providing for switching between operating modes (R2T mandatory or not). Julo "John Hufferd/San Jose/IBM" on 08/09/2000 18:42:12 Please respond to "John Hufferd/San Jose/IBM" To: "Randall R. Stewart" cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: a vote for asymmetric connections in a session Randall, Your head-of-line blockage statement, is of course true, but you say it like it is a bad thing. I am not sure it is. In the Asymmetric configuration, there is a different TCP/IP connections from each Host to a Storage Controller to handle the Commands from that Host to the Storage Controller (or at least that Storage Controller's Port). Each Host has SCSI commands that are not related to the commands in other hosts flowing to the same Storage Controller (port) but on separate TCP/IP connections. They expect to have their commands handled in some "fair" allocation method, with respect to, the other Hosts. Nothing is blocking its "line" except its own Commands. So TCP/IP has delivered, or brought up to be delivered, each Hosts commands independently. Now it is up to the Target SCSI layer to chose which port/interface from which to bring additional commands into its buffers. I do not see this as an important problem, and is the way things have worked regardless of the storage connections, for as long as we have had storage connections. It was true on the SCSI buss, and it was even true on the Old IBM 360s (and continues to this day on the mainframes). There will always be back pressure and sooner or later it causes the Initiator to stop sending new packets, since they are not being acknowledged, this will either occur at the TCP/IP layer or at the SCSI layer. I am not sure that this is a bad thing, especially since we are not talking about dead locks. . . . John L. Hufferd "Randall R. Stewart" @ece.cmu.edu on 09/08/2000 04:58:06 AM Sent by: owner-ips@ece.cmu.edu To: Charles Monia cc: Matt Wakeley , ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session Charles Monia wrote: > I'm not sure what is meant by "congestion." If we're talking about > congestion in the TCP/IP transport, I'm in agreement. However, I thought we > were referring to the sort of congestion that the application on top of > iSCSI might see if it received more commands than it had room for. > > Unless I misunderstood your point, threfore, I think there might be an > issue. The only way I can see flow control in the tranport layer being used > to avoid dropping commands is if higher layer congestion results in back > pressure to the iSCSI pipe. I believe that behavior is undesirable because > it introduces "head-of-line" blocking, with the following consequences: > Any time you get ANY retransmissions in TCP you will create a "head-of-line" blocking scenario. This does not matter if it is from flow control as you state or network congestion where a packet is dropped by a router. You will have situations where the TCP is holding data waiting for the retransmissions of an earlier packet... this is one of the reasons that sigtran developed SCTP... > a) It effectively shuts down the flow of commands to all logical units. > > b) It blocks the flow of task management commands (Abort Task, Clear task > set, etc). > > > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Sep 10 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19273; 10 Sep 2000 1:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA03997 for ips-outgoing; Sun, 10 Sep 2000 01:02:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA03993 for ; Sun, 10 Sep 2000 01:02:49 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id FAA149964 for ; Sun, 10 Sep 2000 05:46:21 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id FAA31478 for ; Sun, 10 Sep 2000 05:46:22 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256956.0014B81A ; Sun, 10 Sep 2000 05:46:18 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 10 Sep 2000 06:44:07 +0300 Subject: Symmetric vs. Asymmetric and some random thoughts Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I would like to point out that the dead-lock situation described in several posts by Costa and myself is mostly a result of sending at leisure unsolicited data. With an asymmetric scheme with a single connection in which data and commands follow the same prescribed order the deadlock can be avoided - but then the closed window for new commands is more of a concern. The one big disadvantage of the asymmetric scheme is IMHO the fact that writes can't carry immediate data - and that is a performance hit for software based initiators that will have to handle 3 i/o operations per small write where the symmetric scheme required 2 i/os. Julo From owner-ips@ECE.cmu.edu Sun Sep 10 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19275; 10 Sep 2000 1:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA03651 for ips-outgoing; Sun, 10 Sep 2000 00:47:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA03646 for ; Sun, 10 Sep 2000 00:47:03 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Sat, 9 Sep 2000 21:46:08 -0700 Message-ID: From: Joshua Tseng To: John Hufferd/San Jose/IBM , Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Sat, 9 Sep 2000 21:46:07 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, "A lot of data" is relative. The fastest TCP implementations available today don't come close to what will be required for iSCSI, since they peak out at about 200Mbps. And the vast majority of applications are get much worse performance than that out of TCP--I would guess typically about 20Mbps max. iSCSI will use TCP like never before. I believe FCP has flow control/ command recovery mechanisms below SCSI. Unlike iSCSI, FCP has the benefit of operating in a low latency environment (<10us). To think iSCSI can do without it and still be able to function reliably is, well iffy at best from what I can see. Regads, Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Saturday, September 09, 2000 12:05 AM To: Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Joshua, I do not understand whether this is a real issue or not. I know of a lot of Client Server applications that ship a lot of data on TCP/IP, and TCP/IP seems to be adequate. Now I understand there are some differences in Direct Storage Access, but it is more like the other applications then it is different. If we can address Costa's Issue or have at least two Connection per Asymmetric Session, I am not sure anything else is a real life problem. But I can be convinced, however, I would like to understand for each problem we come up with, why it is only a problem for iSCSI and not for the other real world applications, and why the SCSI layer can not handle the problem. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/08/2000 08:53:48 PM Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng , Charles Monia cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Hi John, >I could be mistaken here but I think you, and some others have been >addressing issues with the Storage Controller having the right amount of >buffer space in the controllers for the commands, after they are delivered >by iSCSI. I think this is a SCSI issue and not a iSCSI/TCP flow control >problem. Yes, you are right that this is precisely the problem I'm addressing. Maybe I'm missing something here, but I still think there is an issue. Yes, as James described earlier SCSI has a mechanism for dealing with this, but is it sufficient for iSCSI/TCP? IP has distinctly different characteristics than parallel SCSI and FC. For one thing, latencies in the IP world are much higher. I'm assuming parallel SCSI latencies are next to nothing, while FC are in the <10us range. If iSCSI is going over the WAN, at least 50ms latencies are to be expected for coast-to-coast US. Over the Public Internet, we're talking about 80 to 250ms or even more (RT). Much of this latency is caused by the speed of light, the rest by the serialization process for WAN transport. This works out to be a huge amount of latency compared to FC. How much can happen in the 30ms before the SCSI QUEUE FULL and BUSY messages finally get back to the initiator. Are you sure leaving it to SCSI or TCP will be okay? >With respect to the Asymmetric approach, the issues change, and we are no >longer trying to solve the problem of missing commands that occur because >of a broken connection. Therefore, I think we can dump the sliding window >and leave the flow control -- and recovery of lost commands, -- up to >TCP. Yes, the SCSI layers on each end need to have their own buffer >management under control, but I do not think this is a transport issue. I agree dumping the sliding window is a good thing. But my question is should iSCSI replace it with something else. I believe the issue James brought up with multiple initiators needs to be considered. Or even with a single initiator, how do you flow control the oncoming commands from the initiator(s)? Is the SCSI mechanisms sufficient given the latencies that iSCSI must be designed to support? BTW, I don't think that TCP has any role in addressing this issue. If iSCSI and TCP operate independently at different layers, then TCP won't flow control iSCSI commands any more than it flow controls any other PDU's, because it can't tell the difference between them. Josh From owner-ips@ECE.cmu.edu Sun Sep 10 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19715; 10 Sep 2000 2:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA04839 for ips-outgoing; Sun, 10 Sep 2000 01:53:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA04834 for ; Sun, 10 Sep 2000 01:53:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA268396 for ; Sun, 10 Sep 2000 07:52:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA31956 for ; Sun, 10 Sep 2000 07:52:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256956.0020432C ; Sun, 10 Sep 2000 07:52:23 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 10 Sep 2000 07:56:19 +0300 Subject: Re: a vote for asymmetric connections in a session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk John, Unsolicited data come in 2 flavors - immediate (an optimization for small writes - e.g., data base writes - to reduce the number of i/o operations per SCSI I/O) that can go with the command and regular data blocks. Unsolicited data blocks will be sent over data connections in the asymmetric scheme (and immediate data should probably be removed in order to ease the control queue - control flow window association). Julo "John Hufferd/San Jose/IBM" on 09/09/2000 20:21:06 Please respond to "John Hufferd/San Jose/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session Julian, I think I understood what you said in the context of the Symmetric model , but could you please take me through how this would occur in the Asymmetric when you have at least two connections? I had thought, perhaps incorrectly, that the immediate data went with the command, thereby leaving the other connections, in the Asymmetric case, free for solicited data? Or Perhaps I miss understood your statement. . . . John L. Hufferd julian_satran@il.ibm.com@ece.cmu.edu on 09/09/2000 12:33:05 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: a vote for asymmetric connections in a session A note of caution. The most serious dead-lock sitaution we are aware of steams from a mix of RTT (or should we call it R2T to accommodate Doug Ottis?) and unsolicited/immediate data. If channels are full with unsolicited data and the target requests something else - that something else will not get through. This dead-lock, as far as I can tell exists in all transports. A target should be able to detect it and iSCSI has provided for the target to be able to drop data and reclaim them later with R2T. To minimize the chances for it we are also not providing for switching between operating modes (R2T mandatory or not). Julo "John Hufferd/San Jose/IBM" on 08/09/2000 18:42:12 Please respond to "John Hufferd/San Jose/IBM" To: "Randall R. Stewart" cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: a vote for asymmetric connections in a session Randall, Your head-of-line blockage statement, is of course true, but you say it like it is a bad thing. I am not sure it is. In the Asymmetric configuration, there is a different TCP/IP connections from each Host to a Storage Controller to handle the Commands from that Host to the Storage Controller (or at least that Storage Controller's Port). Each Host has SCSI commands that are not related to the commands in other hosts flowing to the same Storage Controller (port) but on separate TCP/IP connections. They expect to have their commands handled in some "fair" allocation method, with respect to, the other Hosts. Nothing is blocking its "line" except its own Commands. So TCP/IP has delivered, or brought up to be delivered, each Hosts commands independently. Now it is up to the Target SCSI layer to chose which port/interface from which to bring additional commands into its buffers. I do not see this as an important problem, and is the way things have worked regardless of the storage connections, for as long as we have had storage connections. It was true on the SCSI buss, and it was even true on the Old IBM 360s (and continues to this day on the mainframes). There will always be back pressure and sooner or later it causes the Initiator to stop sending new packets, since they are not being acknowledged, this will either occur at the TCP/IP layer or at the SCSI layer. I am not sure that this is a bad thing, especially since we are not talking about dead locks. . . . John L. Hufferd "Randall R. Stewart" @ece.cmu.edu on 09/08/2000 04:58:06 AM Sent by: owner-ips@ece.cmu.edu To: Charles Monia cc: Matt Wakeley , ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session Charles Monia wrote: > I'm not sure what is meant by "congestion." If we're talking about > congestion in the TCP/IP transport, I'm in agreement. However, I thought we > were referring to the sort of congestion that the application on top of > iSCSI might see if it received more commands than it had room for. > > Unless I misunderstood your point, threfore, I think there might be an > issue. The only way I can see flow control in the tranport layer being used > to avoid dropping commands is if higher layer congestion results in back > pressure to the iSCSI pipe. I believe that behavior is undesirable because > it introduces "head-of-line" blocking, with the following consequences: > Any time you get ANY retransmissions in TCP you will create a "head-of-line" blocking scenario. This does not matter if it is from flow control as you state or network congestion where a packet is dropped by a router. You will have situations where the TCP is holding data waiting for the retransmissions of an earlier packet... this is one of the reasons that sigtran developed SCTP... > a) It effectively shuts down the flow of commands to all logical units. > > b) It blocks the flow of task management commands (Abort Task, Clear task > set, etc). > > > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Sep 10 07:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19730; 10 Sep 2000 3:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA05350 for ips-outgoing; Sun, 10 Sep 2000 02:24:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA03661 for ; Sun, 10 Sep 2000 00:47:16 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Sat, 9 Sep 2000 21:46:53 -0700 Message-ID: From: Joshua Tseng To: Matt Wakeley , ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Sat, 9 Sep 2000 21:46:53 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Matt, >> In the event of a failed connection, the sliding window isn't going to help >> much since you have to first detect the failed connection. Once this is >> accomplished, the initiator will know which command was lost for sure, > >How is the initiator going to know what commands got to the target or not >without the command reference numbers in the iSCSI document? I believe the Task ID is sufficient. >> but will not know how many, if any, of the subsequent commands sent after the >> lost command, were also lost. > >Well, all of them on the failed connection - if the connection failed, the >following commands could not have been delivered. I believe we were talking about the symmetric model, where commands could be sent on more than one connection. The subsequent commands could have been sent on a different connection than the one that failed. >> I presume, the initiator will then >> retransmit all commands without knowing whether or not they were lost, >> rather than wait for the target to advance the acknowledgement counter. >> >> This is all awesome and impressive behavior, exactly what we want. The only >> problem is this exact same behavior can be had without the windowing >> mechanism. > >Could you describe how it can be had? Well, the initiator has the task ID of all outstanding commands when the connection failed. The initiator should also know what order they were transmitted. He could retransmit them again in the same order, with the retry bit set. I believe this was mentioned earlier. Regards, Josh From owner-ips@ECE.cmu.edu Sun Sep 10 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19805; 10 Sep 2000 3:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA05657 for ips-outgoing; Sun, 10 Sep 2000 02:44:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA05653 for ; Sun, 10 Sep 2000 02:44:28 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id CAA65984 for ; Sun, 10 Sep 2000 02:42:40 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8A6iRo46382 for ; Sun, 10 Sep 2000 00:44:27 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: a vote for asymmetric connections in a session To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 9 Sep 2000 23:42:19 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/10/2000 12:44:26 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, Many of us have seen speeds much faster then what you quoted, and with the new TCP/IP offload NICs, we will see performance equal to line speeds. . . . John L. Hufferd Joshua Tseng on 09/09/2000 09:46:07 PM To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session John, "A lot of data" is relative. The fastest TCP implementations available today don't come close to what will be required for iSCSI, since they peak out at about 200Mbps. And the vast majority of applications are get much worse performance than that out of TCP--I would guess typically about 20Mbps max. iSCSI will use TCP like never before. I believe FCP has flow control/ command recovery mechanisms below SCSI. Unlike iSCSI, FCP has the benefit of operating in a low latency environment (<10us). To think iSCSI can do without it and still be able to function reliably is, well iffy at best from what I can see. Regads, Josh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Saturday, September 09, 2000 12:05 AM To: Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Joshua, I do not understand whether this is a real issue or not. I know of a lot of Client Server applications that ship a lot of data on TCP/IP, and TCP/IP seems to be adequate. Now I understand there are some differences in Direct Storage Access, but it is more like the other applications then it is different. If we can address Costa's Issue or have at least two Connection per Asymmetric Session, I am not sure anything else is a real life problem. But I can be convinced, however, I would like to understand for each problem we come up with, why it is only a problem for iSCSI and not for the other real world applications, and why the SCSI layer can not handle the problem. . . . John L. Hufferd Joshua Tseng @ece.cmu.edu on 09/08/2000 08:53:48 PM Sent by: owner-ips@ece.cmu.edu To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng , Charles Monia cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Hi John, >I could be mistaken here but I think you, and some others have been >addressing issues with the Storage Controller having the right amount of >buffer space in the controllers for the commands, after they are delivered >by iSCSI. I think this is a SCSI issue and not a iSCSI/TCP flow control >problem. Yes, you are right that this is precisely the problem I'm addressing. Maybe I'm missing something here, but I still think there is an issue. Yes, as James described earlier SCSI has a mechanism for dealing with this, but is it sufficient for iSCSI/TCP? IP has distinctly different characteristics than parallel SCSI and FC. For one thing, latencies in the IP world are much higher. I'm assuming parallel SCSI latencies are next to nothing, while FC are in the <10us range. If iSCSI is going over the WAN, at least 50ms latencies are to be expected for coast-to-coast US. Over the Public Internet, we're talking about 80 to 250ms or even more (RT). Much of this latency is caused by the speed of light, the rest by the serialization process for WAN transport. This works out to be a huge amount of latency compared to FC. How much can happen in the 30ms before the SCSI QUEUE FULL and BUSY messages finally get back to the initiator. Are you sure leaving it to SCSI or TCP will be okay? >With respect to the Asymmetric approach, the issues change, and we are no >longer trying to solve the problem of missing commands that occur because >of a broken connection. Therefore, I think we can dump the sliding window >and leave the flow control -- and recovery of lost commands, -- up to >TCP. Yes, the SCSI layers on each end need to have their own buffer >management under control, but I do not think this is a transport issue. I agree dumping the sliding window is a good thing. But my question is should iSCSI replace it with something else. I believe the issue James brought up with multiple initiators needs to be considered. Or even with a single initiator, how do you flow control the oncoming commands from the initiator(s)? Is the SCSI mechanisms sufficient given the latencies that iSCSI must be designed to support? BTW, I don't think that TCP has any role in addressing this issue. If iSCSI and TCP operate independently at different layers, then TCP won't flow control iSCSI commands any more than it flow controls any other PDU's, because it can't tell the difference between them. Josh From owner-ips@ECE.cmu.edu Sun Sep 10 16:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22251; 10 Sep 2000 11:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA21586 for ips-outgoing; Sun, 10 Sep 2000 10:47:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA21581 for ; Sun, 10 Sep 2000 10:47:06 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA18378 for ; Sun, 10 Sep 2000 16:46:34 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA31798 for ; Sun, 10 Sep 2000 16:46:31 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256956.0051275D ; Sun, 10 Sep 2000 16:46:24 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 10 Sep 2000 17:44:11 +0300 Subject: RE: bridging issues -- Converging FCP-2 and iSCSI control struct ures Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, We (the iSCSI design team) had some guidelines that we followed (to the best of our knowledge) when deciding on formats. Here are some of those: constant header length no redundant information (as far as possible) to avoid the need for validity checks compact coding while keeping implementations efficient close resemblance to FCP to simplify bridging We also considered that requiring a bridge to do stateless transliteration is acceptable. Your proposal has many elements towards which we are neutral. We might be wiling to change some coding elements to better fit bridges as long as they don't affect iSCSI efficiency but we would like to keep the header length constant. There is some misunderstanding about the CDB length - the iSCSI is pretty liberal only that it use a tricky coding of the length field (not something we would necessarily want to keep in). The FCP task management encoding is strange - if any task management flag is set the CDB is unused; we might go for it if it where not for that the task abort command requires us to specify which task we are taking about and we don't want to expand every header. The net result is a header that exceeds our constant length. Responses - are simpler. There is an almost 1-to-1 match and the total length is within bounds. For R2T - we can rearange the fields. For Data - if we are moving to an asymmetric model with 2 connections we may want to have a streamlined header on the data stream(as in the 00-draft - no use for the current clutter). THE QUESTION of TAG MAPPING is still open. In summary I think that we can converge but we have some more work to do to on the commands and I think that before we settle the asymmetric/symmetric and related it will be hard to see all the details. But let us keep talking and agree to attempt a format convergence before the next version of the draft is due (before San Diego). Keep in mind that if we go for asymmetric we would like to get the header back to the original 40 bytes (?) if possible. Regards, Julo Charles Monia on 06/09/2000 18:42:06 Please respond to Charles Monia To: Julian Satran/Haifa/IBM@IBMIL cc: "Ips (E-mail)" , Charles Monia Subject: RE: bridging issues -- Converging FCP-2 and iSCSI control struct ures Hi Julio: In comparing FCP SCSI control structures with iSCSI, there are a lot of fields that are functionally identical. Therefore, as a step towards bridging between iSCSI and FCP, as you mentioned, I went through the exercise of rearranging the transport-independent fields within the iSCSI control structures to reflect the equivalent FCP-2 structures. (See ftp://ftp.t10.org/t10/drafts/fcp2/fcp2r04.pdf, for a copy of FCP-2.) The gory details are below, including a field-by-field mapping between corresponding iSCSI and FCP parameters. The conversion seems fairly straightforward and preserves iSCSI semantics. Functionally, the main differences are: a) Since FCP combines the task management and SCSI command blocks into a single control structure and response, I've done the same thing with the corresponding control blocks as mapped to iSCSI. i.e.. In the reformatted version, the iSCSI task management command and return status are merged with the corresponding iSCSI command and status blocks. b) All iSCSI-specific fields have been collected into special iSCSI headers. I have not gone through the exercise of defining a fixed length iSCSI preamble that fits every case however. c) Since command ordering is handled by iSCSI and TCP/IP, the Command Reference Number in the FCP command block is an iSCSI reserved field. The one change in the FCP command template was the need to provide an encoding for the ABORT TASK management function. (FCP does this by aborting the FC-2 Exchange). An issue for further study is how best to map the mechanism for Asynchronous Event Notification between FCP and iSCSI. By the way, in comparing the two kinds of command control blocks, I noticed that iSCSI does not seem to allocate space for CDBs longer than 16 bytes (unless that's what the "command-data" field was to be used for). Anyhow, assuming the command-data in iSCSI is not intended for this purpose, I defined a separate optional extension for iSCSI to contain this information as described in the iSCSI spec. Issues for further study are a) How best to map the mechanism for Asynchronous Event Notification between FCP and iSCSI. b) An efficient mapping between the iSCSI data transfer model and Fibre channel. 1. iSCSI command structure (iSCSI section 3.2) The iSCSI Command data structure is modified to include the FCP_CMND Payload (see table 22 of the Rev 4 FCP-2 draft standard). iSCSI information is contained in a transport-specific header as shown below, followed by the FCP_CMD template, and terminated with optional command-specific data. +------------------+ | iSCSI Header | +------------------+ | SCSI Command | | Block (modeled | | on FCP-2) | +------------------+ | Additional Data | | (Optional) | +------------------+ 1.1 Contents of iSCSI header: The header up to ExpStatRn is as defined in the iSCSI spec. OpCode Length CmdRN ExpStatRn Initiator Tag Length of "Additional Data" The "Length of Additional Data" is a 32-bit field, new to iSCSI, containing the size of the space reserved for additional, command-dependent data. 1.2 iSCSI mapping of FCP-2 command template: FCP_LUN -- 8-byte Logical Unit Number ISCSI Definition: Same as FCP FCP_CRN -- 1-byte command reference number: iSCSI definition: reserved field Task Attribute -- 3-bit field encoding Head of Queue, Simple, etc. iSCSI definition -- Same as FCP Task Management Flags -- 8-bit field containing bit-significant task management flags. Bit 7 -- Obsolete Bit 6 -- Clear ACA Bit 5 -- Target Reset Bit 3 -- Reserved Bit 2 -- Clear Task Set Bit 1 -- Abort Task Set Bit 0 -- Reserved iSCSI Definition -- Bit 7 defined as "Abort Task", remaining bits same as FCP-2. Alternative iSCSI definition: Convert the task management flags to a 4-bit encoded field. [As described in what follows, the above field allows the reformatted command block to perform the function of the iSCSI task management block defined in iSCSI, section 3.7.] RDDATA, WRDATA -- One-bit fields indicating direction of data transfer iSCSI definition -- Same as FCP-2. Additional FCP Length -- Amount of space allocated for a CDB whose length exceeds 16 bytes. iSCSI definition -- Same as FCP-2. Additional FCP_CDB -- Contains that portion of the CDB which exceeds 16-bytes. iSCSI definition -- Same as FCP-2 (Note: this field does not seem to be defined in the current iSCSI proposal). FCP_DL -- Count of the greatest number of bytes expected to be transferred to or from the application client data buffer by the SCSI CDB. iSCSI definition -- Same as FCP-2. Other FCP-2 reserved fields are mapped to corresponding reserved fields in iSCSI. 1.3 iSCSI features not reflected in the FCP-2 data structure: "Additional data": Command-specific data appended to SCSI command. Suggest modifying iSCSI to append this information to the command block and reflect its extent through the iSCSI header shown above. 2. SCSI Task Management command (iSCSI section 3.7) Deleted. The function of this data structure in iSCSI is subsumed by the revised iSCSI command block shown above. 3. SCSI Response (iSCSI section 3.3) and SCSI Task Management response (section 3.8) The iSCSI data structures for responding to command and task management requests are combined to reflect the merger of the iSCSI command and task management blocks described above. +------------------+ | iSCSI Header | +------------------+ | SCSI Response | | Block modeled | | on FCP-2 | +------------------+ | Additional Data | | (Optional) | +------------------+ 3.1 Contents of iSCSI header: The following are as defined above. OpCode Length CmdRN ExpStatRn Initiator Tag Length of "Additional Data" (Must be zero). 3.2 iSCSI Mapping of FCP-2 response template. FCP_RSP, bytes 0 through 9 -- Reserved iSCSI mapping -- Same as FCP FCP_RSP, byte 10 -- Flags, contents: FCP_RSP_LEN_VALID -- 1 = FCP_RSP_INFO field contains valid data iSCSI mapping -- Equivalent to FCP mapping. FCP_SNS_LEN_VALID -- 1 = FCP_SNS_INFO field contains valid data. The length of the returned sense data is contained in FCP_SNS_LEN. iSCSI mapping -- Equivalent to FCP mapping. FCP_RESID_OVER, FCP_RESID_UNDER -- Residual underflow or overflow count. iSCSI mapping -- Equivalent to FCP mapping. FCP_CONF_REQ -- 1 = Initiator must confirm receipt of status information unit. iSCSI Mapping -- Reserved. FCP_RESID -- if FCP_RESID_UNDER or FCP_RESID_OVER is set, contains count of residual data iSCSI Mapping -- Same as FCP. FCP_SNS_LEN -- if FCP_SNS_LEN_VALID, contains length of sense data. iSCSI Mapping -- Same as FCP. FCP_RSP_LEN -- If FCP_RSP_LEN_VALID, contains length of FCP_RESPONSE field. (FCP-2 limits FCP_RSP_LEN to 4 or 8). iSCSI Mapping -- Same as FCP FCP_RESP_INFO -- If FCP_RSP_LEN_VALIS = 1, this is a 4- or 8-byte field containing the iSCSI mapping -- See 3.2.1 below. FCP_SNS_INFO -- If FCP_SNS_LEN_VALID =1, contains FCP_SNS_LEN bytes of sense data. iSCSI mapping -- Same as FCP-2 3.2.1. iSCSI mapping of FCP_RSP_INFO field FCP_RSP_INFO is a 4- or 8-byte field containing the following: Bytes 0 through 2: Reserved iSCSI mapping: Same as FCP. Byte 3: RSP_CODE Code values (hex): 00 -- No failure or task management function complete 04 -- Task management function not supported 05 -- Task management function falied All other values reserved. The above code values map directly to iSCSI. Bytes 4 through 7: Reserved. In the RSP_CODE field, iSCSI equivalents for the following FCP-specific codes may be useful: Code values (hex): 01 -- FCP_DATA length different than burst length. 02 -- FCP_CMND fields invalid 03 -- FCP_DATA RO mismatch with FCP_XFER_RDY DATA_RO Codes 01 and 03 indicate a mismatch between an FCP_XFR_RDY parameter and the corresponding parameter in the actual data transfer information unit. 4. iSCSI RTT and FCP_XFR_RDY Information Unit +------------------+ | iSCSI RTT | | Header | +------------------+ | iSCSI RTT | | modeled on | | FCP_XFR_RDY | +------------------+ In iSCSI, The use of RTT is negotiated between the target and initiator. 4.1 RTT Header contents The following are as defined in the iSCSI spec. OpCode Length CmdRN ExpStatRn Initiator Tag Target Tag 4.2 iSCSI Mapping from FCP_XFR_RDY to RTT DATA_RO -- Offset of the first byte of data relative to the start of the initiator's buffer iSCSI Mapping -- Buffer offset as defined in iSCSI, section 3.9 BURST_LEN -- On a transfer from the FCP initiator to the FCP target, exact number of bytes to be transferred. iSCSI Mapping -- Desired data transfer length as defined in iSCSI, section 3.9 > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Sunday, August 27, 2000 8:58 AM > To: ips@ece.cmu.edu > Subject: bridging issues > > > > > Dear colleagues, > > Lately some iSCSI issues related to bridging to/from FC networks where > brought to my > attention. I thought that it would be wise to share them > (and my first > rough attempt to solve them) with you for two reasons: > > that's why we have a Working Group (don't we?) > there might be more of this kind and this is a good time > to think about > them and bring the with or without a solution to the WG > > FC and iSCSI are SCSI mappings over a network that share a > "network" view > of the > transport infrastructure but have widely different views of > the network > with regard > to distances, reliability, flow control etc.. > > FCP assumes a locally controlled environment using an > unreliable delivery > version of FC > and doing recovery (a very rare event) at FCP level. > > FC addressing model uses locally generated dynamic addresses > as part of the > "naming" > and its tagging. > > An iSCSI initiator should be able to operate a FCP target > through a very > simple and > stateless (or almost stateless) gateway. > > A designer attempting to do this (and I know many of you > already did or are > in the midst > of doing that) will find it difficult to do achieve a > stateless design (but > not impossible). > > The FCP recovery model from packet loses (commands, data and > status) is > based on sequencing (if lost packets are within sequences) or > timeouts if > those are at sequence > boundaries. In the FC world those are very rare events and > in an iSCSI+FCP > hybrid > could be handled by a statefull gateway (expensive) or by a stateless > gateway and > a more radical initiator recovery - based on TCP connection close and > reopen. > The later solution requires iSCSI to enforce a set of timeouts (an > initiator requirement only) > and provide an API for timeout setting and connection abort > to an upper > layer. > Enforcing a set of timeouts would have the good side-effect > of making iSCSI > also > tolerant to aberrant device behavior - and that is > unfortunately bound to > happen more > often as the complexity of the devices will increase. > Please observe that the set of timeouts I am referring to are not a > "protocol" element > (they do not appear on the wire). Specifying the expected > behavior of an > iSCSI > initiator will simplify the task of building bridges to FCP. > We propose to do so by adding a "host requirements" chapter. > > Another bridging issue is the initiator tag-translation mechanism. > iSCSI is requiring the initiator issue a unique tag for each > task but does > not require any structure for the tag. This way an > implementer may choose > the most efficient structure for his implementation (e.g., > use the address of it's control structure representing the > task as a tag). > > FCP on the other hand demands some structure for the tag. > > A stateless bridge would want to pass the tag unchanged from > initiator to > target or translate it with a stateless rule. > > If we restrict the design space to gateways that present a > "homogenous" > view of the device conglomerate behind them (e.g., all > devices are FCP) > and we may pass this information to the initiator - then the > initiator can > use the slightly more restrictive FCP tagging mechanism. > > Julo > > From owner-ips@ECE.cmu.edu Sun Sep 10 20:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23380; 10 Sep 2000 16:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA26741 for ips-outgoing; Sun, 10 Sep 2000 15:14:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA26736 for ; Sun, 10 Sep 2000 15:14:17 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id PAA00203 for ips@ece.cmu.edu; Sun, 10 Sep 2000 15:14:01 -0400 (EDT) Date: Sun, 10 Sep 2000 15:14:01 -0400 (EDT) From: Scott Bradner Message-Id: <200009101914.PAA00203@newdev.harvard.edu> To: ips@ece.cmu.edu Subject: TCP speed Sender: owner-ips@ece.cmu.edu Precedence: bulk the topic of TCP speeds came up - here is a note from Craig Partridge (someone who should know) on the topic. Scott >From craig@aland.bbn.com Sun Sep 10 15:04:54 2000 To: Scott Bradner Subject: Re: tcp speed In-reply-to: Your message of "Sun, 10 Sep 2000 09:15:25 EDT." <200009101315.JAA10411@newdev.harvard.edu> Date: Sun, 10 Sep 2000 13:26:59 -0400 From: Craig Partridge [Scott -- feel free to recirculate as appropriate.] This note is a brief summary of what is achievable in TCP performance. First off, let's talk about processing. Not including interrupt costs (more on those in a minute), it costs about 20 or 30 instructions to handle a TCP segment that is received in order, plus time for copying and checksumming. Copying and checksumming can be done at memory speed, and for large packets 20 or 30 instruction times is trivial, so effectively TCP can transfer data at memory rate. For more on how to achieve these rates, see Van Jacobson's work on TCP Header Prediction, my book on Gigabit Networking, and the performance notes in Rich Stevens' TCP-IP Illustrated. Three things, however, slow TCP down. The first problem is interrupts. Most fast CPUs these days really really don't like interrupts -- they may burn 1000s of instruction cycle to handle them. So having a network interface that minimizes interrupts (e.g., like the ones Jonathan Smith pioneered in the mid-90s) is critical or you'll spend all your time in the interrupt driver. The second issue is TCP window size. If you don't use TCP Large Windows, and actually set a large window size, you're limited to what is typically shipped with your system. In most systems the standard window size is about 16 to 32 KB, which limits you to about 250 Mb/s on a typical LAN. (Moral here: tune up your TCP). The third issue is TCP congestion control. TCP takes a while to get to send at full speed, as it is probing the network to determine available network capacity. To date, there's no agreed upon solution to allow TCP (or any other protocol for that matter) to send fast from the start. However, this time delay is only a problem over long distances -- you won't see problems of note on your LAN. In summary, in a local environment, with TCP big windows, you should go as fast as your processors let you. In a wide area environment, you'll bump against congestion control issues, but then again so will any other protocol. As a final comment, it is worth noting that some commercial TCPs were running at 1 Gbps nearly a decade ago. Craig From owner-ips@ECE.cmu.edu Sun Sep 10 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23830; 10 Sep 2000 18:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA29214 for ips-outgoing; Sun, 10 Sep 2000 16:48:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA29207 for ; Sun, 10 Sep 2000 16:48:11 -0400 (EDT) Received: from cs.duke.edu (user-152-16-65-113.adsl.duke.edu [152.16.65.113]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id QAA15555; Sun, 10 Sep 2000 16:48:04 -0400 (EDT) Message-ID: <39BBF43D.DD864F91@cs.duke.edu> Date: Sun, 10 Sep 2000 16:51:09 -0400 From: Jeff Chase Reply-To: chase@cs.duke.edu X-Mailer: Mozilla 4.74 [en] (X11; U; Linux 2.2.12 i386) X-Accept-Language: en MIME-Version: 1.0 To: Scott Bradner , craig@aland.bbn.com CC: ips@ece.cmu.edu Subject: Re: TCP speed References: <200009101914.PAA00203@newdev.harvard.edu> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------msD5A280F28099BA1FD478427A" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a cryptographically signed message in MIME format. --------------msD5A280F28099BA1FD478427A Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Scott Bradner wrote: > > the topic of TCP speeds came up - here is a note from Craig Partridge > (someone who should know) on the topic. > > Scott > > >From craig@aland.bbn.com Sun Sep 10 15:04:54 2000 > To: Scott Bradner > Subject: Re: tcp speed > In-reply-to: Your message of "Sun, 10 Sep 2000 09:15:25 EDT." > <200009101315.JAA10411@newdev.harvard.edu> > Date: Sun, 10 Sep 2000 13:26:59 -0400 > From: Craig Partridge > ... > protocol. As a final comment, it is worth noting that some commercial > TCPs were running at 1 Gbps nearly a decade ago. > > Craig I humbly add that our experience with TCP supports this last statement about achievable bandwidths. The poor-quality TCP stacks in circulation have left a widespread exaggerated bias that TCP is inherently slow. For a picture of a FreeBSD TCP stack running at 2Gb/s point-to-point, see: http://www.cs.duke.edu/ari/publications/end-system.ps http://www.cs.duke.edu/ari/publications/end-system.pdf "Your mileage may vary." Please, I don't want to start a thread on this topic here. This list gets quite enough traffic already, and the topic has been beaten to death elsewhere (e.g., end2end). E-mail any flames directly to me: I am always happy to receive them! Jeff --------------msD5A280F28099BA1FD478427A Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIIKOgYJKoZIhvcNAQcCoIIKKzCCCicCAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCC B8YwggSQMIID+aADAgECAhAS8L5nAuwrK5xBEbxnNfOkMA0GCSqGSIb3DQEBBAUAMIHMMRcw FQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29y azFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNvcnAuIEJ5 IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJbmRp dmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkMB4XDTAwMDgyNjAwMDAw MFoXDTAxMDgyNjIzNTk1OVowggEUMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UE CxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9y ZXBvc2l0b3J5L1JQQSBJbmNvcnAuIGJ5IFJlZi4sTElBQi5MVEQoYyk5ODEeMBwGA1UECxMV UGVyc29uYSBOb3QgVmFsaWRhdGVkMTMwMQYDVQQLEypEaWdpdGFsIElEIENsYXNzIDEgLSBO ZXRzY2FwZSBGdWxsIFNlcnZpY2UxGTAXBgNVBAMUEEplZmZyZXkgUy4gQ2hhc2UxIDAeBgkq hkiG9w0BCQEWEWNoYXNlQGNzLmR1a2UuZWR1MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB gQCn05BEbK+8ZnJCnIaGOrONtAIxMq35c6bAVd8IMXvHmfC4U4dUY46ILY8gyIoKhTgtzmRT +GQ9S/YIFkUYv382jNxi4db1S1a2TqETdyoLJY/S+zhSXGxeVM6cW/Q3SFNRNS9yljIw3ZLs CzEWSSsblOyh5wV6Sls3ycWQfLGTwwIDAQABo4IBJjCCASIwCQYDVR0TBAIwADBEBgNVHSAE PTA7MDkGC2CGSAGG+EUBBwEIMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LnZlcmlzaWdu LmNvbS9ycGEwEQYJYIZIAYb4QgEBBAQDAgeAMIGGBgpghkgBhvhFAQYDBHgWdmQ0NjUyYmQ2 M2YyMDQ3MDI5Mjk4NzYzYzlkMmYyNzUwNjljNzM1OWJlZDFiMDU5ZGE3NWJjNGJjOTcwMTc0 N2RhNWQzZjIxNDFiZWFkYjJiZDJlODkyMTRhYTZjZjFkMDExNDg5Y2ExYmE0NGZmZjNlYTQ1 MGMwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC52ZXJpc2lnbi5jb20vY2xhc3MxLmNy bDANBgkqhkiG9w0BAQQFAAOBgQB/xo7n7pAiJK1tq2tcm120fc+dfFTHFCeElmfpma0M4JIy qrqQ7SNAYx+zxON7hio1wyNixCKkeUXxVPEmxJt56xH89uo2sFDjANUEmVMQGVJ6ToWLIWft Ww2HTFy10QlMkAm5uaMG5OtPxKm7zavu6AOh7OFxGMqcgBPXeygB4jCCAy4wggKXoAMCAQIC EQDSdi6NFAw9fbKoJV2v7g11MA0GCSqGSIb3DQEBAgUAMF8xCzAJBgNVBAYTAlVTMRcwFQYD VQQKEw5WZXJpU2lnbiwgSW5jLjE3MDUGA1UECxMuQ2xhc3MgMSBQdWJsaWMgUHJpbWFyeSBD ZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw05ODA1MTIwMDAwMDBaFw0wODA1MTIyMzU5NTla MIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3Qg TmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNv cnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBD QSBJbmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkMIGfMA0GCSqG SIb3DQEBAQUAA4GNADCBiQKBgQC7WkSKBBa7Vf0DeootlE8VeDa4DUqyb5xUv7zodyqdufBo u5XZMUFweoFLuUgTVi3HCOGEQqvAopKrRFyqQvCCDgLpL/vCO7u+yScKXbawNkIztW5UiE+H Sr8Z2vkV6A+HthzjzMaajn9qJJLj/OBluqexfu/J2zdqyErICQbkmQIDAQABo3wwejARBglg hkgBhvhCAQEEBAMCAQYwRwYDVR0gBEAwPjA8BgtghkgBhvhFAQcBATAtMCsGCCsGAQUFBwIB Fh93d3cudmVyaXNpZ24uY29tL3JlcG9zaXRvcnkvUlBBMA8GA1UdEwQIMAYBAf8CAQAwCwYD VR0PBAQDAgEGMA0GCSqGSIb3DQEBAgUAA4GBAIi4Nzvd2pQ3AK2qn+GBAXEekmptL/bxndPK ZDjcG5gMB4ZbhRVqD7lJhaSV8Rd9Z7R/LSzdmkKewz60jqrlCwbe8lYq+jPHvhnXU0zDvcjj F7WkSUJj7MKmFw9dWBpJPJBcVaNlIAD9GCDlX4KmsaiSxVhqwY0DPOvDzQWikK5uMYICPDCC AjgCAQEwgeEwgcwxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2ln biBUcnVzdCBOZXR3b3JrMUYwRAYDVQQLEz13d3cudmVyaXNpZ24uY29tL3JlcG9zaXRvcnkv UlBBIEluY29ycC4gQnkgUmVmLixMSUFCLkxURChjKTk4MUgwRgYDVQQDEz9WZXJpU2lnbiBD bGFzcyAxIENBIEluZGl2aWR1YWwgU3Vic2NyaWJlci1QZXJzb25hIE5vdCBWYWxpZGF0ZWQC EBLwvmcC7CsrnEERvGc186QwCQYFKw4DAhoFAKCBsTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN AQcBMBwGCSqGSIb3DQEJBTEPFw0wMDA5MTAyMDUxMDlaMCMGCSqGSIb3DQEJBDEWBBSoKmX/ h43mumpHQx+nilXENvD8uDBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3 DQMCAgIAgDAHBgUrDgMCBzANBggqhkiG9w0DAgIBQDANBggqhkiG9w0DAgIBKDANBgkqhkiG 9w0BAQEFAASBgEVsBZvFwwmxjMyWEmWWMt3MJFRKLFykf20HMJiUnM+HRjCIhN8plcgxf8zJ fKHCKXLG8GySammi2cCwKry1oCKF0TIJ13FcPtDmIbzizsNF5+tFeuNmGWY0B+WIpOnGT07u PYT5vu+C1iKVfpvvNaTClO9WPRrf+/UasBOHojx2 --------------msD5A280F28099BA1FD478427A-- From owner-ips@ECE.cmu.edu Sun Sep 10 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23828; 10 Sep 2000 18:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA29134 for ips-outgoing; Sun, 10 Sep 2000 16:45:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from imo-r20.mx.aol.com (imo-r20.mx.aol.com [152.163.225.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA29120 for ; Sun, 10 Sep 2000 16:45:38 -0400 (EDT) From: AndInc@aol.com Received: from AndInc@aol.com by imo-r20.mx.aol.com (mail_out_v28.15.) id 5.5a.a827b56 (4243); Sun, 10 Sep 2000 16:44:35 -0400 (EDT) Message-ID: <5a.a827b56.26ed4cb2@aol.com> Date: Sun, 10 Sep 2000 16:44:34 EDT Subject: Re: TCP speed To: sob@harvard.edu CC: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: AOL 5.0 for Windows sub 114 Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Scott, Another couple of issues to consider: In an environment like a disk controller, the data has to take a few trips: (1) from MAC to Memory, (2) from Memory to Memory (to examine/strip headers), (3) from Memory to disk. Actually, (2) is worse than it looks. Since most CPUs use a separate CPU cache for the DRAM, coherency must be ensured. The effect of (1) (2) and (3) is at least 3xburden, and the side effect of (2) may be much worse than it looks. Also, one must consider that the trips taken in (1) and (3) are not just across the memory bus, but across the peripheral bus, which may be much lower speed. The way current disk controllers handle this problem is to pipeline the transfer, avoiding the memory trips and cache coherency problems altogether. Headers are recognized, stripped and passed to the disk by small hardware state machines in Fibre Channel and even ATA. Is anyone interested in doing a little research, ie, a development of a Verilog or VHDL model to show how this could be done for iSCSI? I'm interested in this area, and beyond that, it may be to everyone's advantage to consider the effects of the protocol on the ability to realize a hardware model that could sustain wire rates without a very high performance CPU in the path. I'd suggest using Verilog as the modeling language, and a commercial simulation tool like ModelSim to run the simulations. Drop me a note if you'd like to participate in an informal bit of group research. Mike Anderson From owner-ips@ECE.cmu.edu Mon Sep 11 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25659; 11 Sep 2000 0:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA02497 for ips-outgoing; Sun, 10 Sep 2000 19:14:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA02487 for ; Sun, 10 Sep 2000 19:14:09 -0400 (EDT) Received: from yp_portable (slip-32-102-64-142.ca.us.prserv.net [32.102.64.142]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLSZP; Sun, 10 Sep 2000 16:11:50 -0700 From: "Y P Cheng" To: "John Hufferd/San Jose/IBM" , , Cc: Subject: RE: a vote for asymmetric connections in a session Date: Sun, 10 Sep 2000 16:13:02 -0700 Message-ID: <000201c01b7c$ab4822e0$8e406620@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk (I apologize for this long response. However, I hope it worthies your reading.) John Wrote: >I think I understood what you said in the context >of the Symmetric model, but could you please take >me through how this would occur in the Asymmetric >when you have at least two connections? Juliano Wrote: >A note of caution. The most serious dead-lock sitaution >we are aware of steams from a mix of RTT (or should we >call it R2T to accommodate Doug Ottis?) and unsolicited >immediate data. If channels are full with unsolicited >data and the target requests something else - that something >else will not get through. This dead-lock, as far as I can >tell exists in all transports. A target should be able to >detect it and iSCSI has provided for the target >to be able to drop data and reclaim them later with R2T. An asymmetric connection, while doable, makes NO sense in today's technologies used in NIC adapters. Before people flaming me on such a statement, I will support my position with my understanding of today's NIC and its driver and iSCSI protocol. I don't know enough about mainframe NIC design. Therefore, in a different context, my position could be wrong. However, I welcome the teaching of mainframe NIC designs from someone. Beyond the position of asymmetric connection, I also take the position that failover is a function of a protocol and could be incorporated in iSCSI. Load balance is a function a NIC adapter and its driver. Detection of incomplete SCSI session due to traffic congestion or lack of resource is the responsibility of a SCSI initiator, not a SCSI target. Here are reasons for my positions: To understand my arguments, it requires one to understand the context of my analysis. Let's start with understanding of terms used herein. "Transport Connection" -- It is a unique pair of endpoints (IP address, port number), one sending and one receiving. A SCSI initiator may have many connections to send commands to different SCSI targets who in turn have many connections to different initiators to receive commands. The SOCKET system call returns a handle and data structure which stores an IP address and port number. The BIND and CONNECT system calls duplicates the socket structure and returns a unique port number. (Note, this is a software port number. Later, we will mention the hardware ports on a NIC adapter.) By duplicating the sockets and their handles, an initiator or target supports multiple connections. Often, people mistake the SCSI transport connection -- a socket -- to TCP connection. In fact, UDP is a better connection protocol for iSCSI as I should argue for later. "Server Client Protocol" -- A SCSI target is a server which enters a passive state listening to incoming SCSI commands. It does so by the SOCKET and BIND system calls which establish a receiving endpoint. A SCSI initiator is a client which enters an active state to send SCSI commands. It does so by the SOCKET and CONNECT system calls which establish a sending endpoint. The domain name to IP address conversion in the BIND and CONNECT are provided by an IP name server or the Address Resolution Protocol (ARP). "Peer to Peer" -- A SCSI target may become an initiator to start third party SCSI commands. Acting as an initiator, it does so by starting a new transport connection to another SCSI target. An iSCSI endpoint is either sending or receiving, but never both. Therefore, a SCSI storage device uses one connection to receive commands and another connection for send a third party copy commands. If the sending initiator can also act as a target, then, we have the appearance of peer-to-peer with two transport connections. Note, SCSI is never a full-duplex protocol. "Multi-path Connection" -- If a SCSI target can be reached by more than one IP addresses, the CONNECT system call on a SCSI initiator returns a list of addresses in the socket data structure. This list will be used for load balance and failover recovery. "SCSI Session" -- It is a stateless transaction between an initiator and a target. The session has a request and response relationship. The request is a SCSI command and the response is data-transfer-and-status. SCSI commands like mode select and sense and iSCSI messages like security and authentication create state information. But, they are uninteresting in this scope of this discussion. "iSCSI driver" -- A NIC adapter driver supports one or more NIC adapters who in turn support iSCSI protocols. For a legacy NIC adapter like old Ethernet cards, the driver much build the iSCSI messages. For a new NIC adapter like the fibre channel adapters or even gigabit Ethernet, the driver simply sends a SCSI request to the adapter which in turn builds the iSCSI messages. The new NIC adapter can accept a few hundreds or even thousands of SCSI requests. The iSCSI driver is a miniport driver -- in Windows/NT terminology -- running under the SCSI class driver which sends SCSI requests to SCSI devices. Needless to say, application software or file system code send requests to SCSI devices. "iSCSI NIC adapter" -- A NIC adapter with one or more functional interfaces and one or more ports connecting to the IP gateways executes the iSCSI requests and responses. It sends requests for a SCSI initiator and receives them for a SCSI target. Therefore, a NIC adapter is running in either initiator mode or target mode or in both. A multiple functional NIC adapter can accept FCP requests from one functional interface and iSCSI from another and even VI from a third interface. A dual-channel adapter will have two ports connecting to two different physical paths, say, one to intranet and another to internet. A NIC adapter has transmit- and receive-buffers for incoming and outgoing SCSI messages and data. When the receive buffer is full, incoming messages will be dropped and lost. Now, here is my argument for why asymmetric connections makes NO sense in the context of the NIC adapter technologies that I understand. For asymmetric connection, if the iSCSI driver is running on old legacy NIC adapters, it must send the SCSI command to one adapter and set up data transfer on another. While with great difficulty one may make these two adapters talking to each other to coordinate the command and data sequences, the newer NIC adapters execute hundreds or even thousands SCSI requests and responses "atomically." Therefore, there is no deadlock problem between processing commands and data in the context of either a SCSI initiator or a target. Furthermore, even with the NIC adapters built with the latest technology having two functional interfaces accepting command and data requests separately, there is nothing gained because the SCSI requests are executed atomically by the adapters. In the era of a NIC adapter that execute a whole iSCSI request in 25 microseconds, it does not make sense to have two NIC adapters to split the command and data processing with the coordination itself taking more much time. For the problem of lost SCSI messages due to traffic congestion, it must be detected by the sender who times out the responses, in this case, a SCSI initiator. The congestion problem can not be managed by BB credit used in FCP. For end-to-end connection to a switch or that of an arbitrated loop, one can use BB credit to manage the traffic. But, there is no way to manage that in an Ethernet connection because the collision avoidance protocol. In addition, the gateway can loss packets too. When an initiator is in New York and a target in Los Angeles, one can't afford a zero initial BB credit due to the long latency time. With a non-zero initial BB credit, hundreds of initiators around the world may send requests at the same time. Therefore, traffic jam and lost of packets must use smooth recovery in iSCSI. Only the initiator sends requests and target only returns responses. Therefore, it is very easy for initiator to detect the lost of messages by setting proper time out values. A target must accept at least one request from an initiator; it must manage its resource allocation with RTT. Once a time out on a SCSI request is detected by an initiator, the microcode on the NIC adapter is quite capable of sending the request again, even on another path for failover recovery -- if the adapter has a second port to reach the same target. If not, the NIC iSCSI driver can try another path. In resending the request, yes, the issue of duplicated requests must be managed. However, this is a well-understood problem when retry is allowed in a protocol. Notice, I never say the SCSI target will initiate a retry. If necessary, the target always sends a status message requesting the initiator to retry. Similar to retry, for load balancing the NIC adapter microcode and the iSCSI driver of an initiator is quite capable of selecting a different port or a NIC adapter to send a SCSI request as long as the adapter or the driver are made aware to the multiple paths in the socket data structure which was filled at time of making the connection. To keep the design simple, the target does not, should not, or must not take on the responsibility of load balance. On stripping the data transfers on multiple connections, I do believe we are using four 2.5 gigabit MAC chips to get the 10 gigabit fibre channel, Ethernet, and InfibiBand connections. In fact, the 12x option of InfiniBand stripes the data on 12 MAC chips to get three gigabyte per second data rate. Stripping data across multiple NIC adapters would be too difficult for the poor adapter designers to do. On using UDP instead of TCP for iSCSI, I am having trouble with the TCP because it is stream based. The READ and WRITE system calls are extremely inefficient for block oriented SCSI data transfer. On the other hand, the UDP datagram using and SEND and RECEIVE system calls is better suite for iSCSI. In fact, I believe NFS is built on UDP. The request and response relationship between SCSI initiator and target makes the connectionless UDP protocol possible. FCP is implemented using the class 3 fibre channel protocol which is designed for datagrams. Finally, comments on the resources used by the initiator and target. A SCSI initiator has self-regulating resource allocation. Where there is no resources to start new processes to initiate new SCSI requests, the SCSI requests cease. For each SCSI request, the required resources are pre-allocated waiting for responses from a target. A SCSI target receives requests from everyone on the net. While it must have room to accept new SCSI requests -- which can be done at login by specifying the queue depth -- it needs RTT (R2T) to control the buffer space for data transfers. This position has already been expressed by many storage controller people. I need not repeat the position here. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Mon Sep 11 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25901; 11 Sep 2000 1:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA08464 for ips-outgoing; Sun, 10 Sep 2000 23:38:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA08456 for ; Sun, 10 Sep 2000 23:38:38 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id UAA20277; Sun, 10 Sep 2000 20:38:33 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Sun, 10 Sep 2000 20:38:32 -0700 Message-ID: From: Jim McGrath To: "'Peter Johansson'" , IP Storage Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Sun, 10 Sep 2000 20:38:31 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, In SBP-2 the target initiates things like getting commands and data from the host (where they are stored). So the target does a number of things with respect to the initiator to discover if command/data exist for it and then get the command/data for processing. Then the initiator (at target request) has to send this information to the target. In the extreme the target would only has to accept out of the blue an alert message (just a bit of information) from each initiator, telling it that the initiator has something for it (the alternative is target polling of all the initiators, not attractive in this environment). The problem now is that you have to generate target to initiator traffic (with its own latency) before you even get to knowing the commands and data in the initiator space. Afterall, the desire to send out of the blue commands and data is there for two reasons: it allows the target to start working as fast as possible, it minimizes bus traffic by minimizing request/response transactions. You could say that a target initiated request for command/data information could occur just before the target became free (or at least target resources became free) from previous operations, allowing the target to always keep itself fed with tasks, but not overfed. That might work, although the timing is difficult due to the variance in the response times. But it would not address any bus utilization issues (and indeed, probably make them worse). It also adds some initial latency in lightly loaded systems since the simple initiator sending a command to a target is replaced by sending an alert and then having the target get a command. So yes, I think the SBP-2 style of target initiated actions might make some sense in heavily loaded systems, which a more traditional approach is appropriate for lightly loaded situations. The trick is having the distributed targets and initiators figuring out when to use each one (difficult) or focusing on one (as we have traditionally) and basically optimizing latency for light or heavy loads, but not both. Jim -----Original Message----- From: Peter Johansson [mailto:PJohansson@ACM.org] Sent: Sunday, September 10, 2000 4:38 AM To: IP Storage Subject: RE: Command Queue Depth (was asymmetric/Symmetric) At 08:06 PM 9/6/00, Jim McGrath wrote: >The issue of buffer space allocation for multiple initiators has a long >and troubled history in SCSI. We have never been able to come up with a >good answer. > >Note: the same problem has plagued other attempts to allocate device >resources between multiple initiators, like command queue space. In >general policies with respect to multiple initiators are not really >standard in the SCSI world. Jim's response is correct as far as it goes, but it ignores the solution adopted in SBP-2, which neatly finesses these issues by leaving the resources in the initiator. In fairness, it's likely that the appropriateness (or lack thereof) of an iSCSI solution similar to SBP-2 has been ignored in these discussions because SBP-2 has been less widely implemented than parallel SCSI or FCP. Is it possible that the SBP-2 approach is relevant to iSCSI? That is, put the initiator in the role of "memory server" and let the target manage when and how its own resources are used to effect the transfer of data. This might make sense if the model places more of the assumed resources (principally memory) in the initiator. Can this be effected efficiently? Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Mon Sep 11 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25890; 11 Sep 2000 1:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA09446 for ips-outgoing; Mon, 11 Sep 2000 00:17:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA09439 for ; Mon, 11 Sep 2000 00:17:13 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id VAA23668; Sun, 10 Sep 2000 21:16:57 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Sun, 10 Sep 2000 21:16:56 -0700 Message-ID: From: Jim McGrath To: "'Nelson Nahum'" , ips@ece.cmu.edu Subject: RE: Enhancements for the iSCSI Date: Sun, 10 Sep 2000 21:16:56 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Actually, in SCSI there are no such things as initiator devices and target devices. There are just devices, and they can play the role of a target or initiator with respect to a given command (nexus). Any device and be both a target and an initiator - just not for the same command. Third part copy is a good example. Another is any subsystem that operates as a target on one physical bus and an initiator on another physical bus (like a disk array controller). While the traditional separation into two distinct (normally parallel) buses make the device a target or initiator based on the bus, that quickly goes away when you start looking at other physical buses (like Fibre Channel), where the two buses may be one. A good example here is RAID functions distributed to disk drive controllers (making them both targets of the initial commands and initiators of multiple commands to other drives on the same bus). This became interesting with the advent of Fibre Channel. The reason why people think that a device must be one or the other is a result of market conditions - traditionally there has not been a big market for devices that perform both roles on the same bus. But that has nothing to do with SCSI, transports, and the like - but rather with the higher level functions these devices perform. So unless iSCSI changes what functions people use the devices for, putting in all the support for target/initiator capability in one device won't change the products people will make . Although, like SCSI, I would argue that the architecture should support that sort of dual role (note that other protocols do not - like ATA). Jim -----Original Message----- From: Nelson Nahum [mailto:nnahum@store-age.com] Sent: Friday, September 08, 2000 3:21 PM To: ips@ece.cmu.edu Subject: Enhancements for the iSCSI I think that the iSCSI is a special opportunity to enhance the current SCSI protocol especially it networking capabilities. SCSI was designed with the concept of Initiator for the HBA and Target for the devices. The Initator sends commands to the Targets that are passive. But the networking world supposes every one talking with every one. The fact that iSCSI is implemented over TCP/IP can help to develop these enhacements. Modern SAN appliances need to "talk" with the storage devices but also need a way to update information in the HBAs in a asynchronous manner. My questions are the following: 1 - Is iSCSI designed in such a way that every Target can also be Initiator and viceversa or it supposes different charasterisitics (memory, CPU power etc.) between the Initiator and the Target? 2 - Is possible to add asynchronous events that the Target could send to the Initiators at any time? 3 - Another important enhancement could be the possibility to "broadcast" information to many iSCSI devices or HBAs. Nelson Nahum CTO StoreAge Networking Technologies From owner-ips@ECE.cmu.edu Mon Sep 11 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25914; 11 Sep 2000 1:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA09855 for ips-outgoing; Mon, 11 Sep 2000 00:37:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA09848 for ; Mon, 11 Sep 2000 00:37:12 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id VAA25277; Sun, 10 Sep 2000 21:37:10 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Sun, 10 Sep 2000 21:37:08 -0700 Message-ID: From: Jim McGrath To: "'Matt Wakeley'" , Jim McGrath , ips Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Date: Sun, 10 Sep 2000 21:37:08 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, I agree that the T10 has to take a role in these issues - I was just pointing out that they are not new issues, T10 has discussed them before in the context of parallel SCSI and Fibre Channel, and has never come up with a very good solution. So I would advise caution on expecting iSCSI (whether the work is done here or in T10) to come up with a good solution anytime soon (basically this whole area is a "research project"). On the issue of latency, the key here is that different physical transports and (more importantly) the configurations they enable/use have a dramatic impact on the absolute amount of buffer space, credits, etc required for good performance. As an example, while you are right that the elasticity buffer in FC-AL is a major latency issue, it is not large in absolute terms (today I would expect < 1 us for the sum of all the latencies for all the elasticity buffers in the type of system you mentioned, a 20 drive loop). By contrast, store and forward switches introduce at least tens of us of latency per switch in the path - and that is if everything is done at hardware speeds. While not uncommon for layer-2 Ethernet switches, layer 3 and layer 4 switches are still usually much slower and/or more expensive due to the (historical) complexity of that degree of automation for those protocols, and thus the reliance on very fast (and expensive) CPUs. Of greater import is that FC-AL is so limited that it really is physically impossible to get too high a latency. By contrast, running TCP/IP on some mixture of physical plants, with potentially global transmission paths, run up latencies in the ms, if not seconds, range. While this is a great strength of the internet, it is a real problem in this context (100 ms latencies imply hundreds of MB of buffering for Gbit speeds - or losing a lot of performance if the protocol requires a lot of turnaround delays for things like getting buffer credits). As usual, confining the application space an make your job a lot easier. But if iSCSI is targeting the general, global, high speed, TCP/IP environment, then you have a lot of issues. Jim Personally, I'd rather restrict the application space some more to make the job easier. -----Original Message----- From: Matt Wakeley [mailto:matt_wakeley@agilent.com] Sent: Thursday, September 07, 2000 10:25 PM To: Jim McGrath; ips Subject: Re: Command Queue Depth (was asymmetric/Symmetric) Jim McGrath wrote: > I agree that BB credit has nothing to do with commands per se, but > illustrates the problem we have had with deciding on policies for the > distributed allocation of device resources over multiple initiators. In my > postscript I noted that the same problems have arisen on command queue (how > many queue slots do you get), with similarly no satisfactory solution. So it sounds to me like this distribution of (command) resources across initiators is a T10 SCSI issue, and should be solved there, not by each individual transport (yesterday FC - which didn't solve it, today iSCSI, tomorrow IB or whatever). You've lost me on the following two paragraphs... Ethernet doesn't require credits to send frames - it's a ship and pray model. I don't know what the latency through ethernet switches is, but I'd hope it wasn't in the ms range. Finally, a "usefull" FC-AL will have many devices on it (say 20 drives in a JBOD) and the latency of the elastic store really starts adding up. So I still contend that FC-AL is not a low latency medium. -Matt > On FC-AL, the latency to get an initial credit is typically measured in us > for a couple of reasons. First, much of that logic has been automated in > the interface hardware (indeed, the major source of delay is typically the > elasticity buffer, which is not store and forward and so has a very low > latency compared to many switches or routers). Second, the distances are > very small (e.g. hundreds of meters), so both transmission time and the > opportunity for intervening devices to increase latency is lower than in the > general internet world. > > So generally 2 credits (of 2Kbyte frames) is enough to cover the latency and > get you into streaming. If the latency of the system was measured in ms, > then a transport on a Gibt wire would require more like 50 initial credits > (or more) to cover the latency. Unless we are designing for a low latency > environment for the exchange of credits (like those where FC-AL are used), > then we probably need to allocate so much buffer space that it becomes > difficult to promise initial credits to a lot of potential initiators. > > Jim > > PS general Fibre Channel (e.g. with switches and the like) is a bit > different. > > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Wednesday, September 06, 2000 8:23 PM > To: ips > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > Jim, > > I agree that FC has tried (unsuccessfully) to address this command queue > allocation problem. > > However, the "login bb credit" mechanism in FC does not address the command > queue depth issue at all. BB credit is used to receive commands and/or > data, > and the target has no clue in advance what is coming. BB credit is just > there > to ensure that there is a lowest layer buffer available to receive the FC > frame > (as opposed to dropping it on the floor if there is no "mac" buffer, like > ethernet does). It does not mean the command queue has any room for the > frame. > > At one time, there was a big push to have "data" credits and "command" > credits > to take care of this problem, but it couldn't be made to work and be > "backwards > compatible". > > > The issue of buffer space allocation for multiple initiators has a long > and > > troubled history in SCSI. We have never been able to come up with a good > > answer. > > > > Fibre Channel tried to fix this with the notion of "login BB credit" - > when > > you login you get a minimum number of credits you are always guaranteed > when > > you start data transfers. The problem with this is that storage devices > had > > no realistic ability to discriminate between initiators or to change the > > login BB credit. In addition, the expectation is that all possible > > initiators would get these credits on login. So storage devices vendors > > have played it safe and kept this number low (at 0 until recently, now > > around 2). For iSCSI the number of initial credits you need to "prime the > > pump" until normal data flow is established is probably large (given the > > latencies are higher than in Fibre Channel, especially FC-AL), and the > > How is the latency low on FC-AL?? given that you need to arbitrate and win > the > loop, then receive bb credit, before you can send anything? > > -Matt > > > > > number of potential initiators larger than in Fibre Channel, making this a > > whole lot worse for the storage device. > > > > As soon as we allow the devices to start adjusting these credits, then you > > have the protocol problem of making sure people know when their credits > are > > adjusted and the policy problem of how, who, and when to adjust the > credits. > > Changing everyone's credit when you add a new initiator can get into a > > notification nightmare, although it is "fair." Any policy brings up all > > sorts of nasty issues regarding fairness vs efficient use of the > > transmission media. > > > > Jim > > > > Note: the same problem has plagued other attempts to allocate device > > resources between multiple initiators, like command queue space. In > general > > policies with respect to multiple initiators are not really standard in > the > > SCSI world. > > > > -----Original Message----- > > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > > Sent: Wednesday, September 06, 2000 2:56 PM > > To: ips > > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > > Joshua Tseng wrote: > > > > > James, > > > > > > I agree with others that there may be an issue with the > > > command windowing mechanism in the existing iSCSI spec. It is like > > > "TCP in reverse", in that the target determines the size of the window, > > and > > > not the initiator as in TCP. Rather, I believe that everything that > this > > > windowing mechanism is attempting to achieve can be more easily obtained > > > by having the target communicate its buffer size to the initiator at > > > iSCSI login. It should be the role of the initiator to determine how > > > many commands to put in flight simultaneously, given this input on > > available > > > buffer size from the target. > > > > As more initiators connect to a target, it may need to scale back the > amount > > of > > this buffering it has allocated to each previously logged in initiator (to > > prevent rejecting new logins). > > > > > > > > > > > As far as multiple initiators, could this not be resolved by the target > > > refusing additional logins beyond the number of initiators it can safely > > > support? Not being a storage expert, this is my best guess/suggestion > > > at how to do it. > > > > I believe John already answered this... > > > > -Matt From owner-ips@ECE.cmu.edu Mon Sep 11 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25948; 11 Sep 2000 1:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA04009 for ips-outgoing; Sun, 10 Sep 2000 20:29:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from rly-ip02.mx.aol.com (rly-ip02.mx.aol.com [152.163.225.160]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA03771 for ; Sun, 10 Sep 2000 20:17:04 -0400 (EDT) Received: from tot-te.proxy.aol.com (tot-te.proxy.aol.com [152.163.195.131]) by rly-ip02.mx.aol.com (8.8.8/8.8.8/AOL-5.0.0) with ESMTP id TAA21217 for ; Sun, 10 Sep 2000 19:55:32 -0400 (EDT) Received: from VAIO.PacBell.net (ACA0A142.ipt.aol.com [172.160.161.66]) by tot-te.proxy.aol.com (8.10.0/8.10.0) with ESMTP id e8ANtU802187 for ; Sun, 10 Sep 2000 19:55:30 -0400 (EDT) Message-Id: <4.3.2.7.2.20000910223027.00bc7b70@PacBell.net> X-Sender: Celeborn@PacBell.net (Unverified) X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Sun, 10 Sep 2000 13:38:07 +0200 To: IP Storage From: Peter Johansson Subject: RE: Command Queue Depth (was asymmetric/Symmetric) In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:06 PM 9/6/00, Jim McGrath wrote: >The issue of buffer space allocation for multiple initiators has a long >and troubled history in SCSI. We have never been able to come up with a >good answer. > >Note: the same problem has plagued other attempts to allocate device >resources between multiple initiators, like command queue space. In >general policies with respect to multiple initiators are not really >standard in the SCSI world. Jim's response is correct as far as it goes, but it ignores the solution adopted in SBP-2, which neatly finesses these issues by leaving the resources in the initiator. In fairness, it's likely that the appropriateness (or lack thereof) of an iSCSI solution similar to SBP-2 has been ignored in these discussions because SBP-2 has been less widely implemented than parallel SCSI or FCP. Is it possible that the SBP-2 approach is relevant to iSCSI? That is, put the initiator in the role of "memory server" and let the target manage when and how its own resources are used to effect the transfer of data. This might make sense if the model places more of the assumed resources (principally memory) in the initiator. Can this be effected efficiently? Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Mon Sep 11 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26454; 11 Sep 2000 3:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA11328 for ips-outgoing; Mon, 11 Sep 2000 01:52:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA11321 for ; Mon, 11 Sep 2000 01:52:14 -0400 (EDT) Received: from cx926635a (cx926635-a.msnv1.occa.home.com [24.10.148.23]) by lightsand.com (8.9.3+Sun/8.9.1) with ESMTP id WAA21288; Sun, 10 Sep 2000 22:51:08 -0700 (PDT) Message-ID: <002b01c01bb5$2228bea0$17940a18@msnv1.occa.home.com> Reply-To: "Raj Bhagwat" From: "Raj Bhagwat" To: "Vivek Kashyap" , Cc: References: <200009102217.PAA25348@eng4.sequent.com> Subject: Re: FCIP and wandering duplicates Date: Sun, 10 Sep 2000 22:57:12 -0700 Organization: LightSand Communications MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Vivek, In a properly implemented FC-only fabric, duplicates should not happen. A frame that enters a fabric should either be delivered within E_D_TOV (typically 2 sec) or be discarded. FC devices don't retransmit a frame before the E_D_TOV timer expires. You do bring up an interesting point when two SAN islands are bridged across an IP network. Once an FC frame enters the IP network, it is hard to say how the E_D_TOV is enforced by that part of the network. As authors of the FCoverIP draft, we have given some thought to the possibility of delivering duplicate frames when both the TCP stack at the FCIP gateway and the original FC device initiate recovery/retransmission procedure in an overlapping time period. This is one of the major reasons why we initially chose to make the FCIP gateway a stateless device with no buffering and no retransmission. The good news is that the {IP encapsulated} FC frame will be protected by the IP TTL once it enters the IP network. The bad news is that detecting the E_D_TOV timeout while the frame is inside the IP network may be difficult. Regards, Raj Bhagwat LightSand Communications ----- Original Message ----- From: Vivek Kashyap To: Sent: Sunday, September 10, 2000 3:17 PM Subject: FCIP and wandering duplicates > I read the FCIP document recently. It does not address the issue of > duplicates. > > Does fibre channel protect itself from duplicates ? I looked at the > header and it does not have a time to live field. Thus a duplicate > caught in a loop and then released at the 'right' time could be > accepted by the receiver. Even if the FC fabric path algorithms do > ensure loop free paths transient loops due to errors, switch > misconfigurations or failures are possible. Does FC handle such a case ? > > If it does not then FCIP will exacerbate the problem because the IP > network could have 'transient' loops. Such a 'wandering duplicate' > could be re-injected into the FC island and be accepted by a newer > incarnation of a FC inter-island exchange. Furthermore with no TTL one > can't really provide a timout between starting two conversations. > > > > Vivek > -- > Vivek Kashyap > IBM-NumaQ > viv@sequent.com > From owner-ips@ECE.cmu.edu Mon Sep 11 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26456; 11 Sep 2000 3:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA11979 for ips-outgoing; Mon, 11 Sep 2000 02:24:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA11975 for ; Mon, 11 Sep 2000 02:24:04 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA193242 for ; Mon, 11 Sep 2000 08:23:32 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA69270 for ; Mon, 11 Sep 2000 08:23:32 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256957.0023196D ; Mon, 11 Sep 2000 08:23:22 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 11 Sep 2000 07:56:25 +0300 Subject: Re: TCP speed Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Our experience is the same. TCP is FAST. The only remaining trouble is memory copy from TCP buffers to application buffers. Unless handled properly this may slow you down considerably. Julo Scott Bradner on 10/09/2000 22:14:01 Please respond to Scott Bradner To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: TCP speed the topic of TCP speeds came up - here is a note from Craig Partridge (someone who should know) on the topic. Scott >From craig@aland.bbn.com Sun Sep 10 15:04:54 2000 To: Scott Bradner Subject: Re: tcp speed In-reply-to: Your message of "Sun, 10 Sep 2000 09:15:25 EDT." <200009101315.JAA10411@newdev.harvard.edu> Date: Sun, 10 Sep 2000 13:26:59 -0400 From: Craig Partridge [Scott -- feel free to recirculate as appropriate.] This note is a brief summary of what is achievable in TCP performance. First off, let's talk about processing. Not including interrupt costs (more on those in a minute), it costs about 20 or 30 instructions to handle a TCP segment that is received in order, plus time for copying and checksumming. Copying and checksumming can be done at memory speed, and for large packets 20 or 30 instruction times is trivial, so effectively TCP can transfer data at memory rate. For more on how to achieve these rates, see Van Jacobson's work on TCP Header Prediction, my book on Gigabit Networking, and the performance notes in Rich Stevens' TCP-IP Illustrated. Three things, however, slow TCP down. The first problem is interrupts. Most fast CPUs these days really really don't like interrupts -- they may burn 1000s of instruction cycle to handle them. So having a network interface that minimizes interrupts (e.g., like the ones Jonathan Smith pioneered in the mid-90s) is critical or you'll spend all your time in the interrupt driver. The second issue is TCP window size. If you don't use TCP Large Windows, and actually set a large window size, you're limited to what is typically shipped with your system. In most systems the standard window size is about 16 to 32 KB, which limits you to about 250 Mb/s on a typical LAN. (Moral here: tune up your TCP). The third issue is TCP congestion control. TCP takes a while to get to send at full speed, as it is probing the network to determine available network capacity. To date, there's no agreed upon solution to allow TCP (or any other protocol for that matter) to send fast from the start. However, this time delay is only a problem over long distances -- you won't see problems of note on your LAN. In summary, in a local environment, with TCP big windows, you should go as fast as your processors let you. In a wide area environment, you'll bump against congestion control issues, but then again so will any other protocol. As a final comment, it is worth noting that some commercial TCPs were running at 1 Gbps nearly a decade ago. Craig From owner-ips@ECE.cmu.edu Mon Sep 11 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26641; 11 Sep 2000 3:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA12296 for ips-outgoing; Mon, 11 Sep 2000 02:42:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from storeage_nt.store-age.com ([199.203.178.211]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA12292 for ; Mon, 11 Sep 2000 02:42:20 -0400 (EDT) Received: by storeage_nt.store-age.com with Internet Mail Service (5.5.2650.21) id ; Mon, 11 Sep 2000 09:40:05 +0200 Message-ID: From: Nelson Nahum To: "'Matt Wakeley'" , IPS Reflector Subject: RE: Enhancements for the iSCSI Date: Mon, 11 Sep 2000 09:40:04 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ok, there are asynchronous events from the device to the iSCSI HBA, good!. The question is how these events are passed from the iSCSI HBA to the iSCSI driver and from the iSCSI driver to the application?. Applications like SAN File Systems and Virtualization would benefit from that. May be, the solution is to define into the standard how the iSCSI driver will interface with the applications in order to pass the information that is not strictly SCSI (IOCTLs etc.). Today, every FC HBAs has its own APIs, this difficult too much the applications that need to retrieve FC information from the HBA. Nelson Nahum StoreAge Networking Technologies -----Original Message----- From: Matt Wakeley [mailto:matt_wakeley@agilent.com] Sent: Saturday, September 09, 2000 6:37 AM To: IPS Reflector Subject: Re: Enhancements for the iSCSI Nelson Nahum wrote: > I think that the iSCSI is a special opportunity to enhance the current SCSI > protocol especially it networking capabilities. > SCSI was designed with the concept of Initiator for the HBA and Target for > the devices. The Initator sends commands to the Targets that are passive. > But the networking world supposes every one talking with every one. The fact > that iSCSI is implemented over TCP/IP can help to develop these enhacements. > > Modern SAN appliances need to "talk" with the storage devices but also need > a way to update information in the HBAs in a asynchronous manner. > > My questions are the following: > > 1 - Is iSCSI designed in such a way that every Target can also be Initiator > and viceversa or it supposes different charasterisitics (memory, CPU power > etc.) between the Initiator and the Target? yep. > 2 - Is possible to add asynchronous events that the Target could send to the > Initiators at any time? That's in the spec too. I guess you haven't read it very closely. > 3 - Another important enhancement could be the possibility to "broadcast" > information to many iSCSI devices or HBAs. Why would you want to do that? > Nelson Nahum > CTO > StoreAge Networking Technologies Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Mon Sep 11 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28744; 11 Sep 2000 10:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA00015 for ips-outgoing; Mon, 11 Sep 2000 09:45:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA00011 for ; Mon, 11 Sep 2000 09:45:04 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id PAA12606 for ; Mon, 11 Sep 2000 15:44:32 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id PAA51656 for ; Mon, 11 Sep 2000 15:44:31 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256957.004B7AE3 ; Mon, 11 Sep 2000 15:44:26 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 11 Sep 2000 16:42:03 +0300 Subject: RE: a vote for asymmetric connections in a session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear Mr. Cheng, I hove some trouble with your note as it contains many items some that where repeatedly discussed and already agreed upon, some with smaller or larger misunderstandings (like the duplex issue - the links are in fact used in duplex mode even for iSCSI as R2T and data can flow on the same links and outbound and inbound data can be used simultaneously with different commands, deadlocks are not caused by execution speed - or lack of, a NT miniport serves a port driver which in turns serves a class driver, UDP is not more efficient than TCP - although is has a better matching datagram model it lacks reliable delivery and congestion control, etc.). I will try to summarize your subject line position for my (and our list records) - and please correct if I am wrong: - you are against the asymmetric model as it requires more work to execute a SCSI command than the symmetric model. Regards, Julo "Y P Cheng" on 11/09/2000 02:13:02 Please respond to "Y P Cheng" To: John Hufferd/San Jose/IBM@IBMUS, Julian Satran/Haifa/IBM@IBMIL, black_david@emc.com cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session (I apologize for this long response. However, I hope it worthies your reading.) John Wrote: >I think I understood what you said in the context >of the Symmetric model, but could you please take >me through how this would occur in the Asymmetric >when you have at least two connections? Juliano Wrote: >A note of caution. The most serious dead-lock sitaution >we are aware of steams from a mix of RTT (or should we >call it R2T to accommodate Doug Ottis?) and unsolicited >immediate data. If channels are full with unsolicited >data and the target requests something else - that something >else will not get through. This dead-lock, as far as I can >tell exists in all transports. A target should be able to >detect it and iSCSI has provided for the target >to be able to drop data and reclaim them later with R2T. An asymmetric connection, while doable, makes NO sense in today's technologies used in NIC adapters. Before people flaming me on such a statement, I will support my position with my understanding of today's NIC and its driver and iSCSI protocol. I don't know enough about mainframe NIC design. Therefore, in a different context, my position could be wrong. However, I welcome the teaching of mainframe NIC designs from someone. Beyond the position of asymmetric connection, I also take the position that failover is a function of a protocol and could be incorporated in iSCSI. Load balance is a function a NIC adapter and its driver. Detection of incomplete SCSI session due to traffic congestion or lack of resource is the responsibility of a SCSI initiator, not a SCSI target. Here are reasons for my positions: To understand my arguments, it requires one to understand the context of my analysis. Let's start with understanding of terms used herein. "Transport Connection" -- It is a unique pair of endpoints (IP address, port number), one sending and one receiving. A SCSI initiator may have many connections to send commands to different SCSI targets who in turn have many connections to different initiators to receive commands. The SOCKET system call returns a handle and data structure which stores an IP address and port number. The BIND and CONNECT system calls duplicates the socket structure and returns a unique port number. (Note, this is a software port number. Later, we will mention the hardware ports on a NIC adapter.) By duplicating the sockets and their handles, an initiator or target supports multiple connections. Often, people mistake the SCSI transport connection -- a socket -- to TCP connection. In fact, UDP is a better connection protocol for iSCSI as I should argue for later. "Server Client Protocol" -- A SCSI target is a server which enters a passive state listening to incoming SCSI commands. It does so by the SOCKET and BIND system calls which establish a receiving endpoint. A SCSI initiator is a client which enters an active state to send SCSI commands. It does so by the SOCKET and CONNECT system calls which establish a sending endpoint. The domain name to IP address conversion in the BIND and CONNECT are provided by an IP name server or the Address Resolution Protocol (ARP). "Peer to Peer" -- A SCSI target may become an initiator to start third party SCSI commands. Acting as an initiator, it does so by starting a new transport connection to another SCSI target. An iSCSI endpoint is either sending or receiving, but never both. Therefore, a SCSI storage device uses one connection to receive commands and another connection for send a third party copy commands. If the sending initiator can also act as a target, then, we have the appearance of peer-to-peer with two transport connections. Note, SCSI is never a full-duplex protocol. "Multi-path Connection" -- If a SCSI target can be reached by more than one IP addresses, the CONNECT system call on a SCSI initiator returns a list of addresses in the socket data structure. This list will be used for load balance and failover recovery. "SCSI Session" -- It is a stateless transaction between an initiator and a target. The session has a request and response relationship. The request is a SCSI command and the response is data-transfer-and-status. SCSI commands like mode select and sense and iSCSI messages like security and authentication create state information. But, they are uninteresting in this scope of this discussion. "iSCSI driver" -- A NIC adapter driver supports one or more NIC adapters who in turn support iSCSI protocols. For a legacy NIC adapter like old Ethernet cards, the driver much build the iSCSI messages. For a new NIC adapter like the fibre channel adapters or even gigabit Ethernet, the driver simply sends a SCSI request to the adapter which in turn builds the iSCSI messages. The new NIC adapter can accept a few hundreds or even thousands of SCSI requests. The iSCSI driver is a miniport driver -- in Windows/NT terminology -- running under the SCSI class driver which sends SCSI requests to SCSI devices. Needless to say, application software or file system code send requests to SCSI devices. "iSCSI NIC adapter" -- A NIC adapter with one or more functional interfaces and one or more ports connecting to the IP gateways executes the iSCSI requests and responses. It sends requests for a SCSI initiator and receives them for a SCSI target. Therefore, a NIC adapter is running in either initiator mode or target mode or in both. A multiple functional NIC adapter can accept FCP requests from one functional interface and iSCSI from another and even VI from a third interface. A dual-channel adapter will have two ports connecting to two different physical paths, say, one to intranet and another to internet. A NIC adapter has transmit- and receive-buffers for incoming and outgoing SCSI messages and data. When the receive buffer is full, incoming messages will be dropped and lost. Now, here is my argument for why asymmetric connections makes NO sense in the context of the NIC adapter technologies that I understand. For asymmetric connection, if the iSCSI driver is running on old legacy NIC adapters, it must send the SCSI command to one adapter and set up data transfer on another. While with great difficulty one may make these two adapters talking to each other to coordinate the command and data sequences, the newer NIC adapters execute hundreds or even thousands SCSI requests and responses "atomically." Therefore, there is no deadlock problem between processing commands and data in the context of either a SCSI initiator or a target. Furthermore, even with the NIC adapters built with the latest technology having two functional interfaces accepting command and data requests separately, there is nothing gained because the SCSI requests are executed atomically by the adapters. In the era of a NIC adapter that execute a whole iSCSI request in 25 microseconds, it does not make sense to have two NIC adapters to split the command and data processing with the coordination itself taking more much time. For the problem of lost SCSI messages due to traffic congestion, it must be detected by the sender who times out the responses, in this case, a SCSI initiator. The congestion problem can not be managed by BB credit used in FCP. For end-to-end connection to a switch or that of an arbitrated loop, one can use BB credit to manage the traffic. But, there is no way to manage that in an Ethernet connection because the collision avoidance protocol. In addition, the gateway can loss packets too. When an initiator is in New York and a target in Los Angeles, one can't afford a zero initial BB credit due to the long latency time. With a non-zero initial BB credit, hundreds of initiators around the world may send requests at the same time. Therefore, traffic jam and lost of packets must use smooth recovery in iSCSI. Only the initiator sends requests and target only returns responses. Therefore, it is very easy for initiator to detect the lost of messages by setting proper time out values. A target must accept at least one request from an initiator; it must manage its resource allocation with RTT. Once a time out on a SCSI request is detected by an initiator, the microcode on the NIC adapter is quite capable of sending the request again, even on another path for failover recovery -- if the adapter has a second port to reach the same target. If not, the NIC iSCSI driver can try another path. In resending the request, yes, the issue of duplicated requests must be managed. However, this is a well-understood problem when retry is allowed in a protocol. Notice, I never say the SCSI target will initiate a retry. If necessary, the target always sends a status message requesting the initiator to retry. Similar to retry, for load balancing the NIC adapter microcode and the iSCSI driver of an initiator is quite capable of selecting a different port or a NIC adapter to send a SCSI request as long as the adapter or the driver are made aware to the multiple paths in the socket data structure which was filled at time of making the connection. To keep the design simple, the target does not, should not, or must not take on the responsibility of load balance. On stripping the data transfers on multiple connections, I do believe we are using four 2.5 gigabit MAC chips to get the 10 gigabit fibre channel, Ethernet, and InfibiBand connections. In fact, the 12x option of InfiniBand stripes the data on 12 MAC chips to get three gigabyte per second data rate. Stripping data across multiple NIC adapters would be too difficult for the poor adapter designers to do. On using UDP instead of TCP for iSCSI, I am having trouble with the TCP because it is stream based. The READ and WRITE system calls are extremely inefficient for block oriented SCSI data transfer. On the other hand, the UDP datagram using and SEND and RECEIVE system calls is better suite for iSCSI. In fact, I believe NFS is built on UDP. The request and response relationship between SCSI initiator and target makes the connectionless UDP protocol possible. FCP is implemented using the class 3 fibre channel protocol which is designed for datagrams. Finally, comments on the resources used by the initiator and target. A SCSI initiator has self-regulating resource allocation. Where there is no resources to start new processes to initiate new SCSI requests, the SCSI requests cease. For each SCSI request, the required resources are pre-allocated waiting for responses from a target. A SCSI target receives requests from everyone on the net. While it must have room to accept new SCSI requests -- which can be done at login by specifying the queue depth -- it needs RTT (R2T) to control the buffer space for data transfers. This position has already been expressed by many storage controller people. I need not repeat the position here. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Mon Sep 11 15:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28757; 11 Sep 2000 10:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA29082 for ips-outgoing; Mon, 11 Sep 2000 09:22:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA29076 for ; Mon, 11 Sep 2000 09:22:29 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 11 Sep 2000 09:21:49 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070485E6CF@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, Black_David@emc.com Cc: sob@harvard.edu, mankin@east.isi.edu, ips@ece.cmu.edu Subject: Technical Coordinators Date: Mon, 11 Sep 2000 09:21:46 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian asked a couple of questions about technical coordinators; I'm cc:ing the list because the answers are of general interest. > How is Technical Coordinator mechanism going to work? The coordinators have direct responsibility for making sure that the work required by their specific protocol gets done. This offloads the chairs in a manner in which each technical coordinator can focus primarily on the protocol for which s/he is responsible (as opposed to the chairs who have responsibilities spanning the protocols and things like the charter and framework document). Here are two examples of how TCs help: - In the short term, WG meetings can go into split parallel sessions for protocol-specific issues. TCs would be responsible for organizing (e.g., Agenda) and running the split sessions, avoiding a requirement for the chairs to be in multiple places at the same time :-). - In the intermediate term, we are likely to be dealing with multiple documents per protocol. Having the TCs responsible for tracking/encouraging/etc. the set of documents for each protocol distributes the workload (useful if there are a lot of documents involved). In practice, RFC 2418 on how IETF WGs operate applies to the per-protocol subgroups that result from this structure. The TCs would serve at the pleasure of the WG chairs in the same fashion that the WG chairs serve at the pleasure of the Area Directors. > Who are the current candidates? Send nominations/recommendations to Steve (smb@research.att.com) and myself (black_david@emc.com) directly - please do not use the list for this purpose. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Sep 11 15:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28762; 11 Sep 2000 10:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA00939 for ips-outgoing; Mon, 11 Sep 2000 10:04:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA00934 for ; Mon, 11 Sep 2000 10:04:18 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8BE4Ix17111 for ; Mon, 11 Sep 2000 09:04:18 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA01367; Mon, 11 Sep 00 09:02:36 -0500 Message-Id: <10009111402.AA01367@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session In-Reply-To: Message from Joshua Tseng of "Sat, 09 Sep 2000 21:46:07 PDT." Date: Mon, 11 Sep 2000 09:04:06 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Unlike iSCSI, FCP has the benefit of operating in a low latency > environment (<10us). Not all FCP has latency this low. I've seen disaster tolerant FC-AL configurations with 40 KM hops, which is about 140 us in transit time alone, plus the big elasticity buffers. It IS possible to bury an FCP box with just commands in flight from multiple initiators. This is a known problem for FCP too. Steph From owner-ips@ECE.cmu.edu Mon Sep 11 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28864; 11 Sep 2000 11:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA01096 for ips-outgoing; Mon, 11 Sep 2000 10:06:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA01091 for ; Mon, 11 Sep 2000 10:06:48 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id QAA42390 for ; Mon, 11 Sep 2000 16:06:17 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA16512 for ; Mon, 11 Sep 2000 16:06:17 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256957.004D79A7 ; Mon, 11 Sep 2000 16:06:14 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 11 Sep 2000 17:03:56 +0300 Subject: RE: Command Queue Depth (was asymmetric/Symmetric) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, I fact we did. The SBP solution is for the target to grab the commands from initiator memory. Obviously that doesn't work too good if you attempt to "hide latency" - so we attempted a scheme with command credits - to allow several commands to be in flight. But then we observed that with one control connection (that was before Adelaide) the TCP receiver window will give us this function for free. And as things are now we still consider that the TCP receive window (even for several connections) gives you the same overal effect (through back pressure). The trouble we are having is with the blocking conditions out of order delivery may cause and how to best avoid them. We have now decent mechanisms to avoid/handle deadlocks for both the asymmetric and the symmetric scheme - although they are definitely "cleaner" for the asymmetric scheme (you do things in the order mandated by the control channel). Julo Peter Johansson on 10/09/2000 14:38:07 Please respond to Peter Johansson To: IP Storage cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Command Queue Depth (was asymmetric/Symmetric) At 08:06 PM 9/6/00, Jim McGrath wrote: >The issue of buffer space allocation for multiple initiators has a long >and troubled history in SCSI. We have never been able to come up with a >good answer. > >Note: the same problem has plagued other attempts to allocate device >resources between multiple initiators, like command queue space. In >general policies with respect to multiple initiators are not really >standard in the SCSI world. Jim's response is correct as far as it goes, but it ignores the solution adopted in SBP-2, which neatly finesses these issues by leaving the resources in the initiator. In fairness, it's likely that the appropriateness (or lack thereof) of an iSCSI solution similar to SBP-2 has been ignored in these discussions because SBP-2 has been less widely implemented than parallel SCSI or FCP. Is it possible that the SBP-2 approach is relevant to iSCSI? That is, put the initiator in the role of "memory server" and let the target manage when and how its own resources are used to effect the transfer of data. This might make sense if the model places more of the assumed resources (principally memory) in the initiator. Can this be effected efficiently? Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Mon Sep 11 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28859; 11 Sep 2000 11:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA01029 for ips-outgoing; Mon, 11 Sep 2000 10:05:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA01004 for ; Mon, 11 Sep 2000 10:05:34 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8BE5Xx17131 for ; Mon, 11 Sep 2000 09:05:33 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA01371; Mon, 11 Sep 00 09:03:52 -0500 Message-Id: <10009111403.AA01371@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: bridging issues -- Converging FCP-2 and iSCSI control struct ures In-Reply-To: Message from julian_satran@il.ibm.com of "Sun, 10 Sep 2000 17:44:11 +0300." Date: Mon, 11 Sep 2000 09:05:22 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > But let us keep talking and agree to attempt a format convergence before > the next version of the draft is due (before San Diego). I also support Charles' proposal of more FCP like structures. Steph From owner-ips@ECE.cmu.edu Mon Sep 11 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28866; 11 Sep 2000 11:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA01343 for ips-outgoing; Mon, 11 Sep 2000 10:12:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ipperformance.com (cooper.gilhooley.com [207.8.92.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA01333 for ; Mon, 11 Sep 2000 10:12:08 -0400 (EDT) Received: from gil ([10.1.0.79]) by ipperformance.com (8.9.3/8.9.3) with SMTP id JAA47104; Mon, 11 Sep 2000 09:11:44 -0500 (CDT) (envelope-from greg@gilhooley.com) From: "Greg Joyce" To: , Subject: RE: TCP speed Date: Mon, 11 Sep 2000 09:13:47 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > Our experience is the same. TCP is FAST. > > The only remaining trouble is memory copy from TCP buffers to application > buffers. > Unless handled properly this may slow you down considerably. > > Julo I agree completely; TCP performs very well. With header prediction and hardware checksum, the mainline data path can be very lean. As with any software, the key to performance is often in the implementation. There is no place in the TCP RFC that specifies that the data has to be copied. I suggest that the data copies are just artifacts of implementation and the sockets API. In fact, given the proper API there is no reason that data can't be placed directly into user buffers. Greg -- Greg Joyce greg@ipperformance.com VP Product Development IP Performance 512-448-1879 From owner-ips@ECE.cmu.edu Mon Sep 11 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29356; 11 Sep 2000 12:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA04219 for ips-outgoing; Mon, 11 Sep 2000 11:05:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hqmail3.Crossroads.com ([63.237.99.226]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id LAA04212 for ; Mon, 11 Sep 2000 11:05:24 -0400 (EDT) Received: by hqmail3.Crossroads.com with Internet Mail Service (5.5.2650.21) id ; Mon, 11 Sep 2000 09:58:28 -0500 Message-ID: <670C86291BFCD311AFE500508B94D4B6630C59@hqmail1.crossroads.com> From: Robert Reynolds To: "'Stephen Bailey'" , ips@ece.cmu.edu Subject: RE: Data in SCSI Response or SCSI Data Date: Mon, 11 Sep 2000 10:06:51 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > Sent: Wednesday, September 06, 2000 5:45 PM > To: ips@ece.cmu.edu > Subject: Re: Data in SCSI Response or SCSI Data > > > > When do you get GOOD status and residual counts on a read? > What is causing > > the target to get the length wrong? > > One example is an INQUIRY command. The inquiry data length is > target-specific. Typically the CDB allocation length (and DL) are set > to some arbitrary large value (0xff), and the target sends back > everything it has. The transfer ends with success status. > > There can certainly be transfer residual and no SCSI error status. Another example is on variable block tape reads. The Initiator uses an arbitrary size to read the block and the target sends all the data in that block. In many cases the amount of data returned is less than the amount of data asked for but it is still a GOOD status. Bob From owner-ips@ECE.cmu.edu Mon Sep 11 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29388; 11 Sep 2000 12:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07653 for ips-outgoing; Mon, 11 Sep 2000 12:07:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07635 for ; Mon, 11 Sep 2000 12:07:21 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLT1S; Mon, 11 Sep 2000 09:05:09 -0700 From: "Y P Cheng" To: , Subject: RE: a vote for asymmetric connections in a session Date: Mon, 11 Sep 2000 09:06:20 -0700 Message-ID: <000701c01c0a$392bdb00$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo wrote: >I hove some trouble with your note as it contains many items >some that where repeatedly discussed and already agreed upon, >some with smaller or larger misunderstandings (like the duplex >issue - the links are in fact used in duplex mode ... Being a late comer into this discussion, I do apologize for repeating some subjects that were already been agreed before. However, I do have problems with the iSCSI over TCP -- see my reply to the TCP speed thread -- and the sliding window, alternate path retry, wedge driver, and load balance issues. I don't understand the deadlock issue when a iSCSI adapter can accept hundreds and even thousands SCSI requests and execute them atomically. Let me explain this. An iSCSI NIC adapter accepts a SCSI request and builds a table entry in its internal memory describing the SCSI session. It sends the request datagram to a target. When a response comes back in the form of datagrams, the adapter refers back to the table entry and quickly moves the datagrams to the application software. There is no possibility of deadlock and there are hundreds or thousands of the table entries in the adapter's local memory -- limited by the local memory size, of course. A target iSCSI NIC adapter waits for incoming requests after the application software has already been in "listening" mode. Like its initiator counterpart, referring to the table entries, the incoming datagrams can be moved to the application without delay. Of course, an NIC adapter can be both initiator and target. I am not familiar with the context of previous discussions which require TCP and sliding window to support iSCSI. With regard to the full-duplex transmission, while a target can piggy back the acknowledge in its datagrams, there is no SCSI command starting data transfers in both directions today. Although we could add such commands in the future, in general the need for acknowledge on transport with long latency on the net is bad. As I said, the SCSI status gives the ultimate acknowledge anyway. I don't know if we need to define iSCSI in such a manner that it would require a very complicated driver implementation to work with legacy Ethernet adapters. If the new information introduced creates confusion, I do apologize. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, September 11, 2000 6:42 AM To: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Dear Mr. Cheng, I hove some trouble with your note as it contains many items some that where repeatedly discussed and already agreed upon, some with smaller or larger misunderstandings (like the duplex issue - the links are in fact used in duplex mode even for iSCSI as R2T and data can flow on the same links and outbound and inbound data can be used simultaneously with different commands, deadlocks are not caused by execution speed - or lack of, a NT miniport serves a port driver which in turns serves a class driver, UDP is not more efficient than TCP - although is has a better matching datagram model it lacks reliable delivery and congestion control, etc.). I will try to summarize your subject line position for my (and our list records) - and please correct if I am wrong: - you are against the asymmetric model as it requires more work to execute a SCSI command than the symmetric model. Regards, Julo "Y P Cheng" on 11/09/2000 02:13:02 Please respond to "Y P Cheng" To: John Hufferd/San Jose/IBM@IBMUS, Julian Satran/Haifa/IBM@IBMIL, black_david@emc.com cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session From owner-ips@ECE.cmu.edu Mon Sep 11 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29361; 11 Sep 2000 12:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06464 for ips-outgoing; Mon, 11 Sep 2000 11:43:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.spinnakernet.com (client-141-151-129-90.bellatlantic.net [141.151.129.90]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06457 for ; Mon, 11 Sep 2000 11:43:11 -0400 (EDT) Received: from kazar-pc (kazar-pc.spinnakernet.com [10.1.1.110]) by mail.spinnakernet.com (8.9.3/8.8.7) with ESMTP id LAA31970; Mon, 11 Sep 2000 11:43:09 -0400 Message-Id: <4.2.2.20000911114945.00aa3380@router.spinnakernet.com> X-Sender: kazar@router.spinnakernet.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Mon, 11 Sep 2000 11:52:19 -0400 To: Matt Wakeley From: Mike Kazar Subject: Re: a vote for asymmetric connections in a session Cc: ips@ece.cmu.edu In-Reply-To: <39B9C997.54D39EE6@agilent.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't think that was Costa's point: he was explicitly talking about commands coming in out of order, and that the receiver was waiting for a command with an earlier sequence number. My understanding is that handling this type of resequencing is important in the symmetric model, where you have N pipes of commands arriving in pretty much random order, and limited capacity to buffer requests received out of order. In the assymetric model, since all commands are coming in on one channel, with only data using the other connections, this problem does not occur, and iSCSI sequence numbers are no longer needed (at least for this particular purpose). Mike (kazar@spinnakernet.com) At 10:24 PM 9/8/00 -0700, you wrote: >I think Costa's point is in the asymmetric single connection model: > >The target has room for 3 commands. >The initiator sends 3 commands, followed by data, followed by a 4th command. >The target could stop reading the TCP connection because the command queue is >full, because it doesn't know that data is next on the TCP connection. It has >to assume that a 4th command is coming, and it doesn't have room for it. > >Another case for a "command" tcp connection and a separate "data" connection >(minimum of two TCP connections per session). > >-Matt > >Joshua Tseng wrote: > > > I don't get it. Maybe I'm dumb or something, but I thought the whole > > point of asymmetric was that commands wouldn't arrive out of order. > > > > Josh > > > > -----Original Message----- > > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > > Sent: Friday, September 08, 2000 4:22 PM > > To: ips@ece.cmu.edu > > Subject: Re: a vote for asymmetric connections in a session > > > > I do not know how reasonable Costa's configuration is,in the real world, > > but he has brought up a point that, I think, might be important to address. > > If his example is reasonable, then it would suggest that we NEED to enforce > > the requirement of Two connections per Session in the Asymmetric case. > > What do you all think about this? > > > > . > > . > > . > > John L. Hufferd > > > > csapuntz@cisco.com@ece.cmu.edu on 09/08/2000 01:01:04 PM > > > > Sent by: owner-ips@ece.cmu.edu > > > > To: ips@ece.cmu.edu > > cc: csapuntz@cisco.com > > Subject: Re: a vote for asymmetric connections in a session > > > > If I recall correctly, the windowing mechanism was put in to avoid > > a deadlock condition while using multiple control connections. > > I believe the logic went as follow: > > > > Assumption: You're not allowed to drop commands arriving in on any > > connection > > > > Because the TCP connections work at different rates, commands > > may not arrive in order. Let's say you have four connections and > > a command queue depth of 3. Commands #2,#3, and #4 arrive before command > > #1, > > filling the queue. However, the queue cannot be drained > > until command #1 arrives. So, deadlock. > > > > Solution: Allow commands to be dropped > > > > In the example above, we'd probably drop command #4 and replace it > > with command #1. > > > > Note that using a single TCP connection for control/data is not > > sufficient to solve the problem. Imagine the case that the queue is > > depth 3 and the host issues four WRITES commands in quick order. > > > > The device stops reading the TCP connection after three commands. > > Unfortunately, none of those three writes can complete until > > data is read from that connection! So, deadlock again. > > > > CIFS, NFS, and HTTP don't have this problem because they immediately > > send data after their WRITE commands. > > > > -Costa From owner-ips@ECE.cmu.edu Mon Sep 11 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29394; 11 Sep 2000 12:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07526 for ips-outgoing; Mon, 11 Sep 2000 12:05:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07518 for ; Mon, 11 Sep 2000 12:05:11 -0400 (EDT) Received: from sunmail1.Sun.COM ([129.145.1.2]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id JAA23065; Mon, 11 Sep 2000 09:05:04 -0700 (PDT) Received: from jurassic.eng.sun.com (jurassic.Eng.Sun.COM [129.146.89.31]) by sunmail1.Sun.COM (8.9.1b+Sun/8.9.1/ENSMAIL,v1.6.1-sunmail1) with ESMTP id JAA15681; Mon, 11 Sep 2000 09:04:53 -0700 (PDT) Received: from eng.sun.com (dsl196-127.Eng.Sun.COM [129.146.196.127]) by jurassic.eng.sun.com (8.11.1.Beta0+Sun/8.11.1.Beta0) with ESMTP id e8BG4nJ731585; Mon, 11 Sep 2000 09:04:50 -0700 (PDT) Message-ID: <39BD025B.531402A5@eng.sun.com> Date: Mon, 11 Sep 2000 11:03:39 -0500 From: Spencer Shepler Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.7 [en] (X11; U; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: ips@ece.cmu.edu Subject: Re: TCP speed References: <000601c01c05$26275520$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P Cheng wrote: > > Julo wrote: > >Our experience is the same. TCP is FAST. > >The only remaining trouble is memory copy from TCP buffers > >to application buffers. Unless > >handled properly this may slow you down considerably. > > The issue is not the TCP memory to memory copy speed, it is the latency time > of receiving TCP acknowledges. Between two endpoints of New York and Los > Angeles, latency is in milliseconds if not in seconds. On a one-gigabit > network, for each millisecond there are 100K of data, or 66 1.5K datagrams > being transferred. In fibre channel, there is this EE-credit, End-to-End. > If the sending party has 10 EE credits, it can't send more than 10 > datagrams. EE-credit manages the TCP sliding window currently discussed in > iSCSI. After sending 10 datagrams, one must wait for acknowledges that may > take several hundred milliseconds to come. > > I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an > initiator is inherently acknowledged by its response from a target. > Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. And without additionly significant work, NFS over UDP does not have any form of congestion control and running NFS/UDP cross country should never be done. It bad news. :-) The NFSv4 specification requires the use of a transport that provides congestion control. NFSv4 does not mandate the use of TCP but does mandate the use of congestion control for comforming implementations. Spencer From owner-ips@ECE.cmu.edu Mon Sep 11 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29382; 11 Sep 2000 12:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA05828 for ips-outgoing; Mon, 11 Sep 2000 11:31:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA05823 for ; Mon, 11 Sep 2000 11:31:01 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLTDF; Mon, 11 Sep 2000 08:28:49 -0700 From: "Y P Cheng" To: , Subject: RE: TCP speed Date: Mon, 11 Sep 2000 08:30:00 -0700 Message-ID: <000601c01c05$26275520$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo wrote: >Our experience is the same. TCP is FAST. >The only remaining trouble is memory copy from TCP buffers >to application buffers. Unless >handled properly this may slow you down considerably. The issue is not the TCP memory to memory copy speed, it is the latency time of receiving TCP acknowledges. Between two endpoints of New York and Los Angeles, latency is in milliseconds if not in seconds. On a one-gigabit network, for each millisecond there are 100K of data, or 66 1.5K datagrams being transferred. In fibre channel, there is this EE-credit, End-to-End. If the sending party has 10 EE credits, it can't send more than 10 datagrams. EE-credit manages the TCP sliding window currently discussed in iSCSI. After sending 10 datagrams, one must wait for acknowledges that may take several hundred milliseconds to come. I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an initiator is inherently acknowledged by its response from a target. Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. The real question is how many datagrams can we pipe to a receiving node without causing overrun. This issue is being addressed by the number of receiving buffers inside the NIC adapter that moves data directly to the buffers of an application (known as remote DMA in VI for InfiniBand, or physical writes in 1394, or exchange handling in fibre channel.) It is a challenge for people who design the NIC adapter hardware which must keep up with the speed of connecting media at two gigabits per second today and 10 gigabits next year. The TCP memory to memory copy speed is irrelevant if we have a NIC adapter that can transfer data directly to buffers of application software. If we don't have such an adapter, it is an impossible task in trying to define a protocol regulating hundreds of SCSI target devices returning data to a single SCSI initiator at the same time. If we slow it down by allowing only one SCSI target to return data at a time, then, with the long latency time between two nodes, iSCSI does not stand a chance in the world of OC-192, gigabit Ethernet, Fibre Channel, InfiniBand, and even 1394B. From owner-ips@ECE.cmu.edu Mon Sep 11 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29691; 11 Sep 2000 13:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10558 for ips-outgoing; Mon, 11 Sep 2000 13:12:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10549 for ; Mon, 11 Sep 2000 13:11:53 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA09192; Mon, 11 Sep 2000 10:01:56 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA08876; Mon, 11 Sep 2000 09:59:35 -0700 From: "Douglas Otis" To: "Raj Bhagwat" , "Vivek Kashyap" , Cc: Subject: RE: FCIP and wandering duplicates Date: Mon, 11 Sep 2000 10:14:15 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <002b01c01bb5$2228bea0$17940a18@msnv1.occa.home.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Raj, Unless you are willing to wrap the FC frame within some protocol, duplicates can happen and this has nothing to do with TTL. TTL prevents circular paths from forever sending a packet within a loop. A modified SCTP protocol that will solve both of these problems, if restricted one time retry. A Metro-Area-Network should induce about 5ms RTT delay such that a 10ms skew becomes possible but is limited by the single retry. TTL buys you nothing with this E_D_TOV, Error_Detect Timeout. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Raj Bhagwat > Sent: Sunday, September 10, 2000 10:57 PM > To: Vivek Kashyap; ipfc@standards.gadzoox.com > Cc: ips@ece.cmu.edu > Subject: Re: FCIP and wandering duplicates > > > Vivek, > > In a properly implemented FC-only fabric, duplicates should not happen. A > frame that enters a fabric should either be delivered within E_D_TOV > (typically 2 sec) or be discarded. FC devices don't retransmit a frame > before the E_D_TOV timer expires. > > You do bring up an interesting point when two SAN islands are > bridged across > an IP network. Once an FC frame enters the IP network, it is hard > to say how > the E_D_TOV is enforced by that part of the network. As authors of the > FCoverIP draft, we have given some thought to the possibility of > delivering > duplicate frames when both the TCP stack at the FCIP gateway and the > original FC device initiate recovery/retransmission procedure in an > overlapping time period. This is one of the major reasons why we initially > chose to make the FCIP gateway a stateless device with no buffering and no > retransmission. > > The good news is that the {IP encapsulated} FC frame will be protected by > the IP TTL once it enters the IP network. The bad news is that > detecting the > E_D_TOV timeout while the frame is inside the IP network may be difficult. > > Regards, > > Raj Bhagwat > LightSand Communications > > > ----- Original Message ----- > From: Vivek Kashyap > To: > Sent: Sunday, September 10, 2000 3:17 PM > Subject: FCIP and wandering duplicates > > > > I read the FCIP document recently. It does not address the issue of > > duplicates. > > > > Does fibre channel protect itself from duplicates ? I looked at the > > header and it does not have a time to live field. Thus a duplicate > > caught in a loop and then released at the 'right' time could be > > accepted by the receiver. Even if the FC fabric path algorithms do > > ensure loop free paths transient loops due to errors, switch > > misconfigurations or failures are possible. Does FC handle such a case ? > > > > If it does not then FCIP will exacerbate the problem because the IP > > network could have 'transient' loops. Such a 'wandering duplicate' > > could be re-injected into the FC island and be accepted by a newer > > incarnation of a FC inter-island exchange. Furthermore with no TTL one > > can't really provide a timout between starting two conversations. > > > > > > > > Vivek > > -- > > Vivek Kashyap > > IBM-NumaQ > > viv@sequent.com > > > From owner-ips@ECE.cmu.edu Mon Sep 11 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29679; 11 Sep 2000 13:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10162 for ips-outgoing; Mon, 11 Sep 2000 13:03:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10157 for ; Mon, 11 Sep 2000 13:03:37 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id LAA20576 for ; Mon, 11 Sep 2000 11:57:15 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8BH3Wb148018 for ; Mon, 11 Sep 2000 11:03:32 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: Lets Move on To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 11 Sep 2000 10:01:36 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/11/2000 11:03:32 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk 1.This is not new information. 2.NFS uses TCP/IP esp on major networks. 3. We have discussed UDP before. 4. Lets move on and not repeat history for each new guy. . . . John L. Hufferd "Y P Cheng" @ece.cmu.edu on 09/11/2000 09:06:20 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, cc: Subject: RE: a vote for asymmetric connections in a session Julo wrote: >I hove some trouble with your note as it contains many items >some that where repeatedly discussed and already agreed upon, >some with smaller or larger misunderstandings (like the duplex >issue - the links are in fact used in duplex mode ... Being a late comer into this discussion, I do apologize for repeating some subjects that were already been agreed before. However, I do have problems with the iSCSI over TCP -- see my reply to the TCP speed thread -- and the sliding window, alternate path retry, wedge driver, and load balance issues. I don't understand the deadlock issue when a iSCSI adapter can accept hundreds and even thousands SCSI requests and execute them atomically. Let me explain this. An iSCSI NIC adapter accepts a SCSI request and builds a table entry in its internal memory describing the SCSI session. It sends the request datagram to a target. When a response comes back in the form of datagrams, the adapter refers back to the table entry and quickly moves the datagrams to the application software. There is no possibility of deadlock and there are hundreds or thousands of the table entries in the adapter's local memory -- limited by the local memory size, of course. A target iSCSI NIC adapter waits for incoming requests after the application software has already been in "listening" mode. Like its initiator counterpart, referring to the table entries, the incoming datagrams can be moved to the application without delay. Of course, an NIC adapter can be both initiator and target. I am not familiar with the context of previous discussions which require TCP and sliding window to support iSCSI. With regard to the full-duplex transmission, while a target can piggy back the acknowledge in its datagrams, there is no SCSI command starting data transfers in both directions today. Although we could add such commands in the future, in general the need for acknowledge on transport with long latency on the net is bad. As I said, the SCSI status gives the ultimate acknowledge anyway. I don't know if we need to define iSCSI in such a manner that it would require a very complicated driver implementation to work with legacy Ethernet adapters. If the new information introduced creates confusion, I do apologize. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of julian_satran@il.ibm.com Sent: Monday, September 11, 2000 6:42 AM To: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Dear Mr. Cheng, I hove some trouble with your note as it contains many items some that where repeatedly discussed and already agreed upon, some with smaller or larger misunderstandings (like the duplex issue - the links are in fact used in duplex mode even for iSCSI as R2T and data can flow on the same links and outbound and inbound data can be used simultaneously with different commands, deadlocks are not caused by execution speed - or lack of, a NT miniport serves a port driver which in turns serves a class driver, UDP is not more efficient than TCP - although is has a better matching datagram model it lacks reliable delivery and congestion control, etc.). I will try to summarize your subject line position for my (and our list records) - and please correct if I am wrong: - you are against the asymmetric model as it requires more work to execute a SCSI command than the symmetric model. Regards, Julo "Y P Cheng" on 11/09/2000 02:13:02 Please respond to "Y P Cheng" To: John Hufferd/San Jose/IBM@IBMUS, Julian Satran/Haifa/IBM@IBMIL, black_david@emc.com cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session From owner-ips@ECE.cmu.edu Mon Sep 11 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29670; 11 Sep 2000 13:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10584 for ips-outgoing; Mon, 11 Sep 2000 13:12:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from east.isi.edu (east.isi.edu [38.245.76.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10578 for ; Mon, 11 Sep 2000 13:12:26 -0400 (EDT) Received: from maia.east.isi.edu (maia.east.isi.edu [38.245.76.14]) by east.isi.edu (8.9.2/8.9.2) with SMTP id NAA17188; Mon, 11 Sep 2000 13:12:46 -0400 (EDT) Posted-Date: Mon, 11 Sep 2000 13:15:50 -0400 Message-Id: <10009111715.AA18716@maia.east.isi.edu> Received: from LOCALHOST.east.isi.edu by maia.east.isi.edu (4.1/4.0.3-6) id ; Mon, 11 Sep 00 13:15:50 EDT To: Black_David@emc.com Cc: julian_satran@il.ibm.com, sob@harvard.edu, mankin@east.isi.edu, ips@ece.cmu.edu Reply-To: mankin@east.isi.edu Subject: Re: Technical Coordinators In-Reply-To: Your message of Mon, 11 Sep 2000 09:21:46 -0400. <0F31E5C394DAD311B60C00E029101A070485E6CF@corpmx9.isus.emc.com> Date: Mon, 11 Sep 2000 13:15:50 -0400 From: Allison Mankin Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, David, > - In the short term, WG meetings can go into split parallel sessions > for protocol-specific issues. TCs would be responsible for > organizing (e.g., Agenda) and running the split sessions, > avoiding a requirement for the chairs to be in multiple places > at the same time :-). > In several informal chats we ADs had with the chairs and others about this structure, I heard that some welcomed the parallel structure, but others felt they would prefer to be able to attend both tracks. We can explore scheduling the tracks so that they aren't parallel (or aren't always so). And it's worth noting there should be plenary discussions of all the documents, certainly by the time they mature. Allison From owner-ips@ECE.cmu.edu Mon Sep 11 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29689; 11 Sep 2000 13:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10494 for ips-outgoing; Mon, 11 Sep 2000 13:10:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ipperformance.com (cooper.gilhooley.com [207.8.92.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10480 for ; Mon, 11 Sep 2000 13:10:50 -0400 (EDT) Received: from gil ([10.1.0.79]) by ipperformance.com (8.9.3/8.9.3) with SMTP id MAA63426; Mon, 11 Sep 2000 12:10:22 -0500 (CDT) (envelope-from greg@gilhooley.com) From: "Greg Joyce" To: "Y P Cheng" , Subject: RE: TCP speed Date: Mon, 11 Sep 2000 12:12:25 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: <000601c01c05$26275520$65c809c0@yp_portable.advansys.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: YP Cheng ... > > I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an > initiator is inherently acknowledged by its response from a target. > Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. You're not considering the other thing that TCP gives you: - congestion control - retransmission - sequencing - ... Once you build that on top of UDP, you'll have something very TCP-like without the benefit of years of tuning and refinement. I think you'd also find that the performance was comparable. In fact NFS is implemented on TCP as well and I believe that WAN performance is probably better over TCP than UDP. LAN performance should be similar. Greg -- Greg Joyce greg@ipperformance.com VP Product Development IP Performance 512-448-1879 From owner-ips@ECE.cmu.edu Mon Sep 11 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29684; 11 Sep 2000 13:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA09875 for ips-outgoing; Mon, 11 Sep 2000 12:57:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA09871 for ; Mon, 11 Sep 2000 12:57:50 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA09187; Mon, 11 Sep 2000 09:47:53 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA08838; Mon, 11 Sep 2000 09:45:32 -0700 From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: Data in SCSI Response or SCSI Data Date: Mon, 11 Sep 2000 10:00:11 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <39B9CDC2.399166F4@agilent.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Friday, September 08, 2000 10:42 PM > To: IPS Reflector > Subject: Re: Data in SCSI Response or SCSI Data > > > Douglas Otis wrote: > > > > > then it should send a response structure as FCP and not one > or two and > > > > use it according to FCP. Undefined information of what some > > > consider SCSI > > > > values cause variants in use. I have not seen tape drives use > > > Good Sense > > > > residual but vary in how Check Sense residual is presented. If > > > iSCSI is to > > > > copy FC, then why deviate on response symmetry? At least this > > > provides an > > > > easier bridge. Iterative read operations will carry unused > values to be > > > > examined and opens the door for further variants. Status > > > presented prior to > > > > data must then ensure sequence without end confirmation. > > > > > > Who said anything that status was presented before data? > > > iSCSI allows the LAST iSCSI PDU that completes a SCSI command to > > > also contain > > > the GOOD SCSI status in the header, so that extra overhead to > > > send the status > > > PDU. > > > > By placing status within the header of perhaps 4G of data > payload with no > > defined end structure leaves nothing known to follow this transfer. By > > making Response a separate structure that physically follows the data > > payload there is an end confirmation. Should there be an error > in sequence, > > data will be accepted followed by perhaps some unknown command responses > > until everything falls back into sync. A dubious savings having dummy > > status in each read exchange to save one possible structure at the end. > > Rather than processing one response structure, you must process > two where > > one is out of sequence to the function. > > > > Doug > > Doug, this is TCP - a guaranteed byte stream. If the target > successfully reads > 4G of data off its media into cache memory, and transmits it as one iSCSI > message, TCP will deliver the data/message. If TCP fails for > some reason to > deliver the message (and everything after it), then the initiator > knows because > the number of bytes read from TCP do not match the length of the > iSCSI message > (indicated in the header). Now, if the header also indicates > good status (as > read from the target media into target cache), what's the big > deal that its in > the header as opposed to a separate message after the data iSCSI message? > > -Matt Matt, With multi-path, session recovery, or buffer errors possible (unrelated to TCP) such that after the frame carrying PDU 0x45 status as a transfer is started, completion with many frames of an opaque data glob removes the ability to signal an error to prevent corrupted data from being accepted. It seems to be opening the door for a nasty problem- Data accepted without final confirmation. FCP at least provides this final opportunity with an ending Response PDU and it keeps the data transfer structure unencumbered with optional fields. Doug From owner-ips@ECE.cmu.edu Mon Sep 11 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29672; 11 Sep 2000 13:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA08797 for ips-outgoing; Mon, 11 Sep 2000 12:31:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA08792 for ; Mon, 11 Sep 2000 12:31:50 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLTGM; Mon, 11 Sep 2000 09:29:39 -0700 From: "Y P Cheng" To: "Spencer Shepler" Cc: Subject: RE: TCP speed Date: Mon, 11 Sep 2000 09:30:49 -0700 Message-ID: <000801c01c0d$a52cc0a0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <39BD025B.531402A5@eng.sun.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Spencer Wrote: >And without additionly significant work, NFS over UDP does not have >any form of congestion control and running NFS/UDP cross country should >never be done. It bad news. :-) >The NFSv4 specification requires the use of a transport that provides >congestion control. NFSv4 does not mandate the use of TCP but does >mandate the use of congestion control for comforming implementations. Yes, flow control is important. When hundreds of NFS clients on Internet write to a NFS server, it could be very bad. On SCSI the savor is RTT. While on the initiator side, the resource is pre-allocated before a SCSI request is generated, the target must manage its resource using RTT. The difference between TCP and UDP is the connection-oriented acknowledges. On long latency between two endpoints, waiting acknowledges can be very detrimental to performance. The ultimate solution to flow control in iSCSI with gigabit media is to move the datagrams to application software quickly. We have absolutely no control when the thousands of nodes should send datagrams to a server at gigabit speed. We can slow the flow down by using the concept of EE-credit. But, this would be a wrong solution when latency time is in milliseconds or even seconds. Y.P. Cheng, CTO, ConnectCom Solutions Corp. -----Original Message----- From: Spencer.Shepler@eng.sun.com [mailto:Spencer.Shepler@eng.sun.com]On Behalf Of Spencer Shepler Sent: Monday, September 11, 2000 9:04 AM To: Y P Cheng Cc: ips@ece.cmu.edu Subject: Re: TCP speed Y P Cheng wrote: > > Julo wrote: > >Our experience is the same. TCP is FAST. > >The only remaining trouble is memory copy from TCP buffers > >to application buffers. Unless > >handled properly this may slow you down considerably. > > The issue is not the TCP memory to memory copy speed, it is the latency time > of receiving TCP acknowledges. Between two endpoints of New York and Los > Angeles, latency is in milliseconds if not in seconds. On a one-gigabit > network, for each millisecond there are 100K of data, or 66 1.5K datagrams > being transferred. In fibre channel, there is this EE-credit, End-to-End. > If the sending party has 10 EE credits, it can't send more than 10 > datagrams. EE-credit manages the TCP sliding window currently discussed in > iSCSI. After sending 10 datagrams, one must wait for acknowledges that may > take several hundred milliseconds to come. > > I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an > initiator is inherently acknowledged by its response from a target. > Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. And without additionly significant work, NFS over UDP does not have any form of congestion control and running NFS/UDP cross country should never be done. It bad news. :-) The NFSv4 specification requires the use of a transport that provides congestion control. NFSv4 does not mandate the use of TCP but does mandate the use of congestion control for comforming implementations. Spencer From owner-ips@ECE.cmu.edu Mon Sep 11 19:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30157; 11 Sep 2000 15:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA15945 for ips-outgoing; Mon, 11 Sep 2000 14:51:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA15937 for ; Mon, 11 Sep 2000 14:51:07 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA09263; Mon, 11 Sep 2000 11:41:10 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA09177; Mon, 11 Sep 2000 11:38:49 -0700 From: "Douglas Otis" To: "Raj Bhagwat" , "Vivek Kashyap" , Cc: Subject: RE: FCIP and wandering duplicates Date: Mon, 11 Sep 2000 11:53:29 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <009301c01c1e$741e18e0$17940a18@msnv1.occa.home.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Raj, You can see duplications without the source sending twice. Reducing TTL value may prevent the packet from ever arriving. The number of times TTL is decremented depends on the path and not time in transit. Doug > -----Original Message----- > From: Raj Bhagwat [mailto:rajb@lightsand.com] > Sent: Monday, September 11, 2000 11:31 AM > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com > Cc: ips@ece.cmu.edu > Subject: Re: FCIP and wandering duplicates > > > Doug, > > Your point on TTL is well taken. It is there to prevent a packet from > getting stuck in a routing loop. Duplicates of unicast packets can happen > when the sender retransmits a packet and both the original and the > retransmitted packets reach the destination. If the sender does not > retransmit for an adequate amount of time, duplicates of unicast packets > should not happen. This is true even when there are routing loops or > topology changes in the network. > > As you have pointed out, there is no direct correlation between > E_D_TOV and > TTL. And that could be a problem. However, setting the TTL to a relatively > smaller value (larger than the # of hops between the source and > destination > FCIP gateways) can reduce the possibility of duplicates when the sender of > the FC frame waits for at least E_D_TOV before retransmitting the frame. > > I don't think SCTP at the FCIP gateway will solve the duplicate > problem even > with a single retry. There is no synchronization between the FC device and > the FCIP gateway for error recovery. So, the FC device can retransmit > independent of the SCTP at the FCIP gateway. One way to solve this problem > is to fine tune the timeout values on the FC side and at the SCTP level on > the FCIP gateway. > > Regards, > > Raj Bhagwat > LightSand Communications > > > ----- Original Message ----- > From: Douglas Otis > To: Raj Bhagwat ; Vivek Kashyap ; > > Cc: > Sent: Monday, September 11, 2000 10:14 AM > Subject: RE: FCIP and wandering duplicates > > > > Raj, > > > > Unless you are willing to wrap the FC frame within some protocol, > duplicates > > can happen and this has nothing to do with TTL. TTL prevents circular > paths > > from forever sending a packet within a loop. A modified SCTP protocol > that > > will solve both of these problems, if restricted one time retry. A > > Metro-Area-Network should induce about 5ms RTT delay such that > a 10ms skew > > becomes possible but is limited by the single retry. TTL buys > you nothing > > with this E_D_TOV, Error_Detect Timeout. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Raj Bhagwat > > > Sent: Sunday, September 10, 2000 10:57 PM > > > To: Vivek Kashyap; ipfc@standards.gadzoox.com > > > Cc: ips@ece.cmu.edu > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > Vivek, > > > > > > In a properly implemented FC-only fabric, duplicates should > not happen. > A > > > frame that enters a fabric should either be delivered within E_D_TOV > > > (typically 2 sec) or be discarded. FC devices don't retransmit a frame > > > before the E_D_TOV timer expires. > > > > > > You do bring up an interesting point when two SAN islands are > > > bridged across > > > an IP network. Once an FC frame enters the IP network, it is hard > > > to say how > > > the E_D_TOV is enforced by that part of the network. As authors of the > > > FCoverIP draft, we have given some thought to the possibility of > > > delivering > > > duplicate frames when both the TCP stack at the FCIP gateway and the > > > original FC device initiate recovery/retransmission procedure in an > > > overlapping time period. This is one of the major reasons why we > initially > > > chose to make the FCIP gateway a stateless device with no > buffering and > no > > > retransmission. > > > > > > The good news is that the {IP encapsulated} FC frame will be protected > by > > > the IP TTL once it enters the IP network. The bad news is that > > > detecting the > > > E_D_TOV timeout while the frame is inside the IP network may be > difficult. > > > > > > Regards, > > > > > > Raj Bhagwat > > > LightSand Communications > > > > > > > > > ----- Original Message ----- > > > From: Vivek Kashyap > > > To: > > > Sent: Sunday, September 10, 2000 3:17 PM > > > Subject: FCIP and wandering duplicates > > > > > > > > > > I read the FCIP document recently. It does not address the issue of > > > > duplicates. > > > > > > > > Does fibre channel protect itself from duplicates ? I looked at the > > > > header and it does not have a time to live field. Thus a duplicate > > > > caught in a loop and then released at the 'right' time could be > > > > accepted by the receiver. Even if the FC fabric path algorithms do > > > > ensure loop free paths transient loops due to errors, switch > > > > misconfigurations or failures are possible. Does FC handle > such a case > ? > > > > > > > > If it does not then FCIP will exacerbate the problem because the IP > > > > network could have 'transient' loops. Such a 'wandering duplicate' > > > > could be re-injected into the FC island and be accepted by a newer > > > > incarnation of a FC inter-island exchange. Furthermore with > no TTL one > > > > can't really provide a timout between starting two conversations. > > > > > > > > > > > > > > > > Vivek > > > > -- > > > > Vivek Kashyap > > > > IBM-NumaQ > > > > viv@sequent.com > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Sep 11 19:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30179; 11 Sep 2000 15:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA16093 for ips-outgoing; Mon, 11 Sep 2000 14:54:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA16084 for ; Mon, 11 Sep 2000 14:54:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id UAA86556 for ; Mon, 11 Sep 2000 20:53:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id UAA62210 for ; Mon, 11 Sep 2000 20:53:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256957.0067C434 ; Mon, 11 Sep 2000 20:53:24 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 11 Sep 2000 21:51:13 +0300 Subject: RE: TCP speed Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear Mr. Cheng, Did you ever consider that several commands can be in flight from an initiator to a taget? Julo "Y P Cheng" on 11/09/2000 18:30:00 Please respond to "Y P Cheng" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: TCP speed Julo wrote: >Our experience is the same. TCP is FAST. >The only remaining trouble is memory copy from TCP buffers >to application buffers. Unless >handled properly this may slow you down considerably. The issue is not the TCP memory to memory copy speed, it is the latency time of receiving TCP acknowledges. Between two endpoints of New York and Los Angeles, latency is in milliseconds if not in seconds. On a one-gigabit network, for each millisecond there are 100K of data, or 66 1.5K datagrams being transferred. In fibre channel, there is this EE-credit, End-to-End. If the sending party has 10 EE credits, it can't send more than 10 datagrams. EE-credit manages the TCP sliding window currently discussed in iSCSI. After sending 10 datagrams, one must wait for acknowledges that may take several hundred milliseconds to come. I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an initiator is inherently acknowledged by its response from a target. Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. The real question is how many datagrams can we pipe to a receiving node without causing overrun. This issue is being addressed by the number of receiving buffers inside the NIC adapter that moves data directly to the buffers of an application (known as remote DMA in VI for InfiniBand, or physical writes in 1394, or exchange handling in fibre channel.) It is a challenge for people who design the NIC adapter hardware which must keep up with the speed of connecting media at two gigabits per second today and 10 gigabits next year. The TCP memory to memory copy speed is irrelevant if we have a NIC adapter that can transfer data directly to buffers of application software. If we don't have such an adapter, it is an impossible task in trying to define a protocol regulating hundreds of SCSI target devices returning data to a single SCSI initiator at the same time. If we slow it down by allowing only one SCSI target to return data at a time, then, with the long latency time between two nodes, iSCSI does not stand a chance in the world of OC-192, gigabit Ethernet, Fibre Channel, InfiniBand, and even 1394B. From owner-ips@ECE.cmu.edu Mon Sep 11 19:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30105; 11 Sep 2000 15:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14505 for ips-outgoing; Mon, 11 Sep 2000 14:27:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14496 for ; Mon, 11 Sep 2000 14:27:32 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA11991; Mon, 11 Sep 2000 11:27:19 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.151.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA01444; Mon, 11 Sep 2000 11:27:18 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA04921; Mon, 11 Sep 2000 11:27:18 -0700 (PDT) Message-Id: <200009111827.LAA04921@ha10nwk.EBay.Sun.COM> Date: Mon, 11 Sep 2000 11:27:18 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: a vote for asymmetric connections in a session To: julian_satran@il.ibm.com, ips@ece.cmu.edu, ycheng@advansys.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: U2OXWfL4siefWcvhmmuECg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > I am not familiar with the context of previous discussions which require TCP > and sliding window to support iSCSI. With regard to the full-duplex > transmission, while a target can piggy back the acknowledge in its > datagrams, there is no SCSI command starting data transfers in both > directions today. Although we could add such commands in the future, in > general the need for acknowledge on transport with long latency on the net > is bad. As I said, the SCSI status gives the ultimate acknowledge anyway. > I don't know if we need to define iSCSI in such a manner that it would > require a very complicated driver implementation to work with legacy > Ethernet adapters. Having worked on NFS over UDP on high latency lossy links (old ARPAnet) the allure of saving a transport level ACK by leveraging the fact that the session layer is datagram-like is fatally flawed in practice. Works great on a LAN, but when faced with packet loss due to congestion the session layer is required to maintain adaptive timers, track round trip times, and base retransmissions primarily on timeouts. The dropped packet creates huge latency bubbles while waiting for timeouts, overly aggressive retransmissions further clog congestion, all of these problems are solveable but the result looks remarkably like TCP. In fact this is what we did do to NFS over UDP, implementing the various algorithms found in TCP, albeit poorly. Today, most NFS clients speak TCP by default with a minor penalty for local servers but a major win for distant servers. iSCSI should only consider a reliable connection oriented or reliable datagram transport protocol! Ultimately you can't get away from a transport level ACK without an overly complex session protocol. -David From owner-ips@ECE.cmu.edu Mon Sep 11 19:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30174; 11 Sep 2000 15:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA15472 for ips-outgoing; Mon, 11 Sep 2000 14:43:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA15450 for ; Mon, 11 Sep 2000 14:43:02 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA09251; Mon, 11 Sep 2000 11:32:57 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA09147; Mon, 11 Sep 2000 11:30:33 -0700 From: "Douglas Otis" To: "Robert Reynolds" , "'Stephen Bailey'" , Subject: RE: Data in SCSI Response or SCSI Data Date: Mon, 11 Sep 2000 11:45:13 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <670C86291BFCD311AFE500508B94D4B6630C59@hqmail1.crossroads.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, SAM-2 "As specified in clause 5, the application client may request autosense service for any SCSI command. If supported by the protocol and logical unit and requested by the application client, the device server shall only return sense data in this manner coincident with the completion of a command with a status of CHECK CONDITION." SSC "If the suppress incorrect length indicator (SILI) bit is one and the FIXED bit is zero, the device server shall: a) Report CHECK CONDITION status for an incorrect length condition only if the overlength condition exists and the BLOCK LENGTH field in the mode parameter block descriptor is nonzero (see SPC-2); or b) not report CHECK CONDITION status if the only error is the underlength condition, or if the only error is the overlength condition and the BLOCK LENGTH field of the mode parameters block descriptor is zero. NOTE 9 Since the residue information normally provided in the INFORMATION field of the sense data may not be available when the SILI bit is set, other methods for determining the actual block length should be used (e.g. including length information in the data block). If the SILI bit is one and the FIXED bit is one, the device server shall terminate the command with CHECK CONDITION status and the sense key shall be set to ILLEGAL REQUEST with an additional sense code and an additional sense code qualifier of INVALID FIELD IN CDB. If the SILI bit is zero and an incorrect length block is read, CHECK CONDITION status shall be returned and the ILI and VALID bits shall be set to one in the sense data with an additional sense code and an additional sense code qualifier of NO ADDITIONAL SENSE INFORMATION. Upon termination, the logical position shall be after the incorrect length block (end-of-partition side). If the FIXED bit is one, the INFORMATION field shall be set to the requested transfer length minus the actual number of blocks read (not including the incorrect length block). If the FIXED bit is zero, the INFORMATION field shall be set to the requested transfer length minus the actual block length. Logical units that do not support negative values shall set the INFORMATION field to zero if the overlength condition exists." >From these definitions, do not expect any defined information regarding residue without a Check Condition. The residue of iSCSI within the Read PDU 0x45 is not SCSI related nor does it represent the underlength condition you suggest. Such underlength data is normally lost and uninteresting if the SILI bit is set. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Robert Reynolds > Sent: Monday, September 11, 2000 8:07 AM > To: 'Stephen Bailey'; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > > -----Original Message----- > > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > > Sent: Wednesday, September 06, 2000 5:45 PM > > To: ips@ece.cmu.edu > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > When do you get GOOD status and residual counts on a read? > > What is causing > > > the target to get the length wrong? > > > > One example is an INQUIRY command. The inquiry data length is > > target-specific. Typically the CDB allocation length (and DL) are set > > to some arbitrary large value (0xff), and the target sends back > > everything it has. The transfer ends with success status. > > > > There can certainly be transfer residual and no SCSI error status. > > > Another example is on variable block tape reads. The Initiator > uses an arbitrary size to read the block and the target sends all > the data in that block. In many cases the amount of data returned > is less than the amount of data asked for but it is still a GOOD > status. > > Bob > From owner-ips@ECE.cmu.edu Mon Sep 11 19:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30100; 11 Sep 2000 15:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14789 for ips-outgoing; Mon, 11 Sep 2000 14:32:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14777 for ; Mon, 11 Sep 2000 14:32:11 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel1.hp.com (Postfix) with ESMTP id 95A4F1DF; Mon, 11 Sep 2000 11:31:58 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id LAA04380; Mon, 11 Sep 2000 11:31:58 -0700 (PDT) X-OpenMail-Hops: 1 Date: Mon, 11 Sep 2000 11:31:51 -0700 Message-Id: Subject: RE: RE: Re: multiple connections MIME-Version: 1.0 To: ips@ece.cmu.edu, julian_satran@il.ibm.com Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Friday, September 08, 2000 10:32 PM > To: ips@ece.cmu.edu > Cc: julian_satran@il.ibm.com > Subject: FW: RE: Re: multiple connections > > > > > Matt & Somesh point to one of the main weaknesses of the > asymmetric scheme. > However the initiator adapters setup is generally synchronous and the > overhead I would contend that the amount of overhead is fairly significant for anything other than a iSCSI accelerator NIC - and in this case the burden falls on the NIC. Doing two transactions is more expensive than one and you will hit the performance knee earlier. > is not significant. For writes the TCP window will take care > of the holding > off the data how does that happen when command and data are on different connections (and data for different commands is intermingled)? > that arrive before the command. The big question is - do we > end-up better > than > with the symmetric scheme? > > Julo > > somesh_gupta@hp.com on 07/09/2000 03:12:05 > > Please respond to somesh_gupta@hp.com > > To: ips@ece.cmu.edu, matt_wakeley@agilent.com > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Re: multiple connections > > > > > I think Matt has an important point here. Let us consider READs > and WRITEs seperately. > > 1. READ - In this case the host software can post the appropriate > read buffers to the connection on which the data is to be received > and then post the read command to the connection on which the data > is to be sent. By the time the target gets the command, and returns > data, it is most likely that the NIC receiving the data will have > accepted the read buffers. > > Although synchronization issues are minimized, there is a > reasonable amount of performance impact. Two different "commands" > (in the sense of host to adapter commands - not in the SCSI sense) > have to be posted. Two different queues have to be written to (for > sure if they are on different NICs) and two different NICs triggered > to accept the "commands". Then there will be two different completion > events that will be returned by the NICs to the host - one indicating > the acceptance/"completion"/"return" of the command buffer (indicating > command sent successfully), and the second when the data is all > received (along with the SCSI command status). This may also require > interrupt handling etc which could be minimized by some > clever schemes. > > So the amount of work done by the host for the host/adapter > interaction > is close to doubled. There is probably not that much impact on the > target in this case. > > 2. WRITES - This is the really bad one in my opinion. For me, avoiding > RTTs in iSCSI would just by itself make iSCSI a superior "transport" > for SCSI. So assuming RTTs are not being used, the host would > (changing > the posting order from READs), first post the write command to the > connection on which the data is to be sent, and then post the write > buffers to the connection which is to be used for sending data. > One case is where for whatever reason, the target gets the data before > it gets the command, and has no clue what to do with the data. > Let us assume that the target does get the command before it gets the > data. The target gets a command indicating that data being written to > whatever lun and whatever location is going to arrive on some other > connection. First the target has taken an extra event from > the adapter. > Then, when data arrives on another NIC (and the target gets another > event), the target goes through the list of outstanding WRITE > commands to match the command with the data and then go about the > business of processing the data. > > So in this case the work has increased for the initiator as well as > the target. > > Somesh > > > > -----Original Message----- > > From: matt_wakeley@agilent.com [mailto:matt_wakeley@agilent.com] > > Sent: Wednesday, September 06, 2000 1:17 AM > > To: ips@ece.cmu.edu > > Subject: FW: Re: multiple connections > > > > > > julian_satran@il.ibm.com wrote: > > > > > Dear colleagues, > > > > > > With all the heated debate about multiple vs. single > > connection a request I > > > made a while ago got no significant reply (neither for > nor against). > > > > > > The request was to consider a proposal made by Kalman Meth > > to reconsider > > > the asymmetric model with the addition of a path selection > > made by the > > > initiator. > > > > One downside I see to this proposal is that for a host with > > multiple iSCSI > > nics, it will have to issue two commands (in this order): one > > to the "data path > > nic" to describe where the data is to be read from/written > > to, and one to the > > "control path nic" to send the command. This would be > > communication to two > > cards instead of one. Also, note that the communication to > > the data path nic > > MUST complete before the control path nic sends the command, > > requiring some > > kind of sync mechanism. > > > > > > > This proposal allows removing the command counters - as > > commands use a > > > single TCP connection. The single connection can also be a shared > > > data+control connection. > > > > For the case of the command connection failure and > fail-over to a new > > connection, I don't see how you can get away from the command > > counters. When a > > fail over occurs, you will need some way of finding out what > > commands made it > > to the target and which didn't. The easiest way to do this > > is with command > > numbering. > > > > > > > > > > > In case of multiple connection the data path to be used is > > selected and > > > maintained until the command ends. > > > > > > Thanks, > > > Julo > > > > -Matt Wakeley > > Agilent Technologies > > > > > > From owner-ips@ECE.cmu.edu Mon Sep 11 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30192; 11 Sep 2000 15:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14436 for ips-outgoing; Mon, 11 Sep 2000 14:26:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14426 for ; Mon, 11 Sep 2000 14:26:38 -0400 (EDT) Received: from cx926635a (cx926635-a.msnv1.occa.home.com [24.10.148.23]) by lightsand.com (8.9.3+Sun/8.9.1) with ESMTP id LAA23159; Mon, 11 Sep 2000 11:25:02 -0700 (PDT) Message-ID: <009301c01c1e$741e18e0$17940a18@msnv1.occa.home.com> Reply-To: "Raj Bhagwat" From: "Raj Bhagwat" To: "Douglas Otis" , "Vivek Kashyap" , Cc: References: Subject: Re: FCIP and wandering duplicates Date: Mon, 11 Sep 2000 11:31:07 -0700 Organization: LightSand Communications MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, Your point on TTL is well taken. It is there to prevent a packet from getting stuck in a routing loop. Duplicates of unicast packets can happen when the sender retransmits a packet and both the original and the retransmitted packets reach the destination. If the sender does not retransmit for an adequate amount of time, duplicates of unicast packets should not happen. This is true even when there are routing loops or topology changes in the network. As you have pointed out, there is no direct correlation between E_D_TOV and TTL. And that could be a problem. However, setting the TTL to a relatively smaller value (larger than the # of hops between the source and destination FCIP gateways) can reduce the possibility of duplicates when the sender of the FC frame waits for at least E_D_TOV before retransmitting the frame. I don't think SCTP at the FCIP gateway will solve the duplicate problem even with a single retry. There is no synchronization between the FC device and the FCIP gateway for error recovery. So, the FC device can retransmit independent of the SCTP at the FCIP gateway. One way to solve this problem is to fine tune the timeout values on the FC side and at the SCTP level on the FCIP gateway. Regards, Raj Bhagwat LightSand Communications ----- Original Message ----- From: Douglas Otis To: Raj Bhagwat ; Vivek Kashyap ; Cc: Sent: Monday, September 11, 2000 10:14 AM Subject: RE: FCIP and wandering duplicates > Raj, > > Unless you are willing to wrap the FC frame within some protocol, duplicates > can happen and this has nothing to do with TTL. TTL prevents circular paths > from forever sending a packet within a loop. A modified SCTP protocol that > will solve both of these problems, if restricted one time retry. A > Metro-Area-Network should induce about 5ms RTT delay such that a 10ms skew > becomes possible but is limited by the single retry. TTL buys you nothing > with this E_D_TOV, Error_Detect Timeout. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Raj Bhagwat > > Sent: Sunday, September 10, 2000 10:57 PM > > To: Vivek Kashyap; ipfc@standards.gadzoox.com > > Cc: ips@ece.cmu.edu > > Subject: Re: FCIP and wandering duplicates > > > > > > Vivek, > > > > In a properly implemented FC-only fabric, duplicates should not happen. A > > frame that enters a fabric should either be delivered within E_D_TOV > > (typically 2 sec) or be discarded. FC devices don't retransmit a frame > > before the E_D_TOV timer expires. > > > > You do bring up an interesting point when two SAN islands are > > bridged across > > an IP network. Once an FC frame enters the IP network, it is hard > > to say how > > the E_D_TOV is enforced by that part of the network. As authors of the > > FCoverIP draft, we have given some thought to the possibility of > > delivering > > duplicate frames when both the TCP stack at the FCIP gateway and the > > original FC device initiate recovery/retransmission procedure in an > > overlapping time period. This is one of the major reasons why we initially > > chose to make the FCIP gateway a stateless device with no buffering and no > > retransmission. > > > > The good news is that the {IP encapsulated} FC frame will be protected by > > the IP TTL once it enters the IP network. The bad news is that > > detecting the > > E_D_TOV timeout while the frame is inside the IP network may be difficult. > > > > Regards, > > > > Raj Bhagwat > > LightSand Communications > > > > > > ----- Original Message ----- > > From: Vivek Kashyap > > To: > > Sent: Sunday, September 10, 2000 3:17 PM > > Subject: FCIP and wandering duplicates > > > > > > > I read the FCIP document recently. It does not address the issue of > > > duplicates. > > > > > > Does fibre channel protect itself from duplicates ? I looked at the > > > header and it does not have a time to live field. Thus a duplicate > > > caught in a loop and then released at the 'right' time could be > > > accepted by the receiver. Even if the FC fabric path algorithms do > > > ensure loop free paths transient loops due to errors, switch > > > misconfigurations or failures are possible. Does FC handle such a case ? > > > > > > If it does not then FCIP will exacerbate the problem because the IP > > > network could have 'transient' loops. Such a 'wandering duplicate' > > > could be re-injected into the FC island and be accepted by a newer > > > incarnation of a FC inter-island exchange. Furthermore with no TTL one > > > can't really provide a timout between starting two conversations. > > > > > > > > > > > > Vivek > > > -- > > > Vivek Kashyap > > > IBM-NumaQ > > > viv@sequent.com > > > > > > > From owner-ips@ECE.cmu.edu Mon Sep 11 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30297; 11 Sep 2000 15:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA17560 for ips-outgoing; Mon, 11 Sep 2000 15:21:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA17553 for ; Mon, 11 Sep 2000 15:21:21 -0400 (EDT) Received: from cx926635a (cx926635-a.msnv1.occa.home.com [24.10.148.23]) by lightsand.com (8.9.3+Sun/8.9.1) with ESMTP id MAA23380; Mon, 11 Sep 2000 12:19:44 -0700 (PDT) Message-ID: <00af01c01c26$18bdc9c0$17940a18@msnv1.occa.home.com> Reply-To: "Raj Bhagwat" From: "Raj Bhagwat" To: "Douglas Otis" , "Vivek Kashyap" , Cc: References: Subject: Re: FCIP and wandering duplicates Date: Mon, 11 Sep 2000 12:25:50 -0700 Organization: LightSand Communications MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, I understand how exactly TTL is used. That is why I stated that TTL should be set to a value larger than the number of hops between the source and destination. An IP end station may not know this and it may choose to set the TTL to the max value of 255. However, an FCIP gateway, with some intelligence, can perhaps know the # of hops to other FCIP gateways it talks to, especially in enterprise networks. It is not clear to me how there can be duplicate frames without the source sending twice. Could you please explain that? Thanks, Raj. ----- Original Message ----- From: Douglas Otis To: Raj Bhagwat ; Vivek Kashyap ; Cc: Sent: Monday, September 11, 2000 11:53 AM Subject: RE: FCIP and wandering duplicates > Raj, > > You can see duplications without the source sending twice. Reducing TTL > value may prevent the packet from ever arriving. The number of times TTL is > decremented depends on the path and not time in transit. > > Doug > > > -----Original Message----- > > From: Raj Bhagwat [mailto:rajb@lightsand.com] > > Sent: Monday, September 11, 2000 11:31 AM > > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com > > Cc: ips@ece.cmu.edu > > Subject: Re: FCIP and wandering duplicates > > > > > > Doug, > > > > Your point on TTL is well taken. It is there to prevent a packet from > > getting stuck in a routing loop. Duplicates of unicast packets can happen > > when the sender retransmits a packet and both the original and the > > retransmitted packets reach the destination. If the sender does not > > retransmit for an adequate amount of time, duplicates of unicast packets > > should not happen. This is true even when there are routing loops or > > topology changes in the network. > > > > As you have pointed out, there is no direct correlation between > > E_D_TOV and > > TTL. And that could be a problem. However, setting the TTL to a relatively > > smaller value (larger than the # of hops between the source and > > destination > > FCIP gateways) can reduce the possibility of duplicates when the sender of > > the FC frame waits for at least E_D_TOV before retransmitting the frame. > > > > I don't think SCTP at the FCIP gateway will solve the duplicate > > problem even > > with a single retry. There is no synchronization between the FC device and > > the FCIP gateway for error recovery. So, the FC device can retransmit > > independent of the SCTP at the FCIP gateway. One way to solve this problem > > is to fine tune the timeout values on the FC side and at the SCTP level on > > the FCIP gateway. > > > > Regards, > > > > Raj Bhagwat > > LightSand Communications > > > > > > ----- Original Message ----- > > From: Douglas Otis > > To: Raj Bhagwat ; Vivek Kashyap ; > > > > Cc: > > Sent: Monday, September 11, 2000 10:14 AM > > Subject: RE: FCIP and wandering duplicates > > > > > > > Raj, > > > > > > Unless you are willing to wrap the FC frame within some protocol, > > duplicates > > > can happen and this has nothing to do with TTL. TTL prevents circular > > paths > > > from forever sending a packet within a loop. A modified SCTP protocol > > that > > > will solve both of these problems, if restricted one time retry. A > > > Metro-Area-Network should induce about 5ms RTT delay such that > > a 10ms skew > > > becomes possible but is limited by the single retry. TTL buys > > you nothing > > > with this E_D_TOV, Error_Detect Timeout. > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > Raj Bhagwat > > > > Sent: Sunday, September 10, 2000 10:57 PM > > > > To: Vivek Kashyap; ipfc@standards.gadzoox.com > > > > Cc: ips@ece.cmu.edu > > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > > > > Vivek, > > > > > > > > In a properly implemented FC-only fabric, duplicates should > > not happen. > > A > > > > frame that enters a fabric should either be delivered within E_D_TOV > > > > (typically 2 sec) or be discarded. FC devices don't retransmit a frame > > > > before the E_D_TOV timer expires. > > > > > > > > You do bring up an interesting point when two SAN islands are > > > > bridged across > > > > an IP network. Once an FC frame enters the IP network, it is hard > > > > to say how > > > > the E_D_TOV is enforced by that part of the network. As authors of the > > > > FCoverIP draft, we have given some thought to the possibility of > > > > delivering > > > > duplicate frames when both the TCP stack at the FCIP gateway and the > > > > original FC device initiate recovery/retransmission procedure in an > > > > overlapping time period. This is one of the major reasons why we > > initially > > > > chose to make the FCIP gateway a stateless device with no > > buffering and > > no > > > > retransmission. > > > > > > > > The good news is that the {IP encapsulated} FC frame will be protected > > by > > > > the IP TTL once it enters the IP network. The bad news is that > > > > detecting the > > > > E_D_TOV timeout while the frame is inside the IP network may be > > difficult. > > > > > > > > Regards, > > > > > > > > Raj Bhagwat > > > > LightSand Communications > > > > > > > > > > > > ----- Original Message ----- > > > > From: Vivek Kashyap > > > > To: > > > > Sent: Sunday, September 10, 2000 3:17 PM > > > > Subject: FCIP and wandering duplicates > > > > > > > > > > > > > I read the FCIP document recently. It does not address the issue of > > > > > duplicates. > > > > > > > > > > Does fibre channel protect itself from duplicates ? I looked at the > > > > > header and it does not have a time to live field. Thus a duplicate > > > > > caught in a loop and then released at the 'right' time could be > > > > > accepted by the receiver. Even if the FC fabric path algorithms do > > > > > ensure loop free paths transient loops due to errors, switch > > > > > misconfigurations or failures are possible. Does FC handle > > such a case > > ? > > > > > > > > > > If it does not then FCIP will exacerbate the problem because the IP > > > > > network could have 'transient' loops. Such a 'wandering duplicate' > > > > > could be re-injected into the FC island and be accepted by a newer > > > > > incarnation of a FC inter-island exchange. Furthermore with > > no TTL one > > > > > can't really provide a timout between starting two conversations. > > > > > > > > > > > > > > > > > > > > Vivek > > > > > -- > > > > > Vivek Kashyap > > > > > IBM-NumaQ > > > > > viv@sequent.com > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Sep 11 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30285; 11 Sep 2000 15:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA16951 for ips-outgoing; Mon, 11 Sep 2000 15:09:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA16945 for ; Mon, 11 Sep 2000 15:09:11 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id VAA220908 for ; Mon, 11 Sep 2000 21:08:40 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id VAA93272 for ; Mon, 11 Sep 2000 21:08:39 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256957.0069265A ; Mon, 11 Sep 2000 21:08:30 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 11 Sep 2000 22:06:19 +0300 Subject: RE: RE: Re: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, It is hard to comment on the overhead - the terms will be hypothetical. But I will try to do it later. On the data hold - the scheme we are talking now you will accept data into targets (by the device manager) only from the commands that are in order - for which you know on what connection they are going to arrive. I assume that any initiator will not ship data in an order different than that in which it ships commands - if the mode is unsolicited. If you'll use a different queuing model at the target then data should be shipped accordingly - or you should use solicited mode. Regards, Julo somesh_gupta@hp.com on 11/09/2000 21:31:51 Please respond to somesh_gupta@hp.com To: ips@ece.cmu.edu, Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: RE: Re: multiple connections > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Friday, September 08, 2000 10:32 PM > To: ips@ece.cmu.edu > Cc: julian_satran@il.ibm.com > Subject: FW: RE: Re: multiple connections > > > > > Matt & Somesh point to one of the main weaknesses of the > asymmetric scheme. > However the initiator adapters setup is generally synchronous and the > overhead I would contend that the amount of overhead is fairly significant for anything other than a iSCSI accelerator NIC - and in this case the burden falls on the NIC. Doing two transactions is more expensive than one and you will hit the performance knee earlier. > is not significant. For writes the TCP window will take care > of the holding > off the data how does that happen when command and data are on different connections (and data for different commands is intermingled)? > that arrive before the command. The big question is - do we > end-up better > than > with the symmetric scheme? > > Julo > > somesh_gupta@hp.com on 07/09/2000 03:12:05 > > Please respond to somesh_gupta@hp.com > > To: ips@ece.cmu.edu, matt_wakeley@agilent.com > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Re: multiple connections > > > > > I think Matt has an important point here. Let us consider READs > and WRITEs seperately. > > 1. READ - In this case the host software can post the appropriate > read buffers to the connection on which the data is to be received > and then post the read command to the connection on which the data > is to be sent. By the time the target gets the command, and returns > data, it is most likely that the NIC receiving the data will have > accepted the read buffers. > > Although synchronization issues are minimized, there is a > reasonable amount of performance impact. Two different "commands" > (in the sense of host to adapter commands - not in the SCSI sense) > have to be posted. Two different queues have to be written to (for > sure if they are on different NICs) and two different NICs triggered > to accept the "commands". Then there will be two different completion > events that will be returned by the NICs to the host - one indicating > the acceptance/"completion"/"return" of the command buffer (indicating > command sent successfully), and the second when the data is all > received (along with the SCSI command status). This may also require > interrupt handling etc which could be minimized by some > clever schemes. > > So the amount of work done by the host for the host/adapter > interaction > is close to doubled. There is probably not that much impact on the > target in this case. > > 2. WRITES - This is the really bad one in my opinion. For me, avoiding > RTTs in iSCSI would just by itself make iSCSI a superior "transport" > for SCSI. So assuming RTTs are not being used, the host would > (changing > the posting order from READs), first post the write command to the > connection on which the data is to be sent, and then post the write > buffers to the connection which is to be used for sending data. > One case is where for whatever reason, the target gets the data before > it gets the command, and has no clue what to do with the data. > Let us assume that the target does get the command before it gets the > data. The target gets a command indicating that data being written to > whatever lun and whatever location is going to arrive on some other > connection. First the target has taken an extra event from > the adapter. > Then, when data arrives on another NIC (and the target gets another > event), the target goes through the list of outstanding WRITE > commands to match the command with the data and then go about the > business of processing the data. > > So in this case the work has increased for the initiator as well as > the target. > > Somesh > > > > -----Original Message----- > > From: matt_wakeley@agilent.com [mailto:matt_wakeley@agilent.com] > > Sent: Wednesday, September 06, 2000 1:17 AM > > To: ips@ece.cmu.edu > > Subject: FW: Re: multiple connections > > > > > > julian_satran@il.ibm.com wrote: > > > > > Dear colleagues, > > > > > > With all the heated debate about multiple vs. single > > connection a request I > > > made a while ago got no significant reply (neither for > nor against). > > > > > > The request was to consider a proposal made by Kalman Meth > > to reconsider > > > the asymmetric model with the addition of a path selection > > made by the > > > initiator. > > > > One downside I see to this proposal is that for a host with > > multiple iSCSI > > nics, it will have to issue two commands (in this order): one > > to the "data path > > nic" to describe where the data is to be read from/written > > to, and one to the > > "control path nic" to send the command. This would be > > communication to two > > cards instead of one. Also, note that the communication to > > the data path nic > > MUST complete before the control path nic sends the command, > > requiring some > > kind of sync mechanism. > > > > > > > This proposal allows removing the command counters - as > > commands use a > > > single TCP connection. The single connection can also be a shared > > > data+control connection. > > > > For the case of the command connection failure and > fail-over to a new > > connection, I don't see how you can get away from the command > > counters. When a > > fail over occurs, you will need some way of finding out what > > commands made it > > to the target and which didn't. The easiest way to do this > > is with command > > numbering. > > > > > > > > > > > In case of multiple connection the data path to be used is > > selected and > > > maintained until the command ends. > > > > > > Thanks, > > > Julo > > > > -Matt Wakeley > > Agilent Technologies > > > > > > From owner-ips@ECE.cmu.edu Mon Sep 11 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31000; 11 Sep 2000 18:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA23324 for ips-outgoing; Mon, 11 Sep 2000 17:22:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e23.nc.us.ibm.com (e23.nc.us.ibm.com [32.97.136.229]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA23320 for ; Mon, 11 Sep 2000 17:22:21 -0400 (EDT) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e23.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id QAA13850 for ; Mon, 11 Sep 2000 16:58:12 -0500 Received: from d04nms52.raleigh.ibm.com (d04nms51nms52.raleigh.ibm.com [9.67.228.55]) by southrelay02.raleigh.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id RAA40074 for ; Mon, 11 Sep 2000 17:22:20 -0400 Importance: Normal Subject: RE: FCIP and wandering duplicates To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John H Dowdy/Raleigh/IBM" Date: Mon, 11 Sep 2000 17:22:19 -0400 X-MIMETrack: Serialize by Router on D04NMS52/04/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/11/2000 05:22:19 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I might have missed the original issue here, but thought I'd toss in a couple of points: 1. TCP Retransmissions generate duplicate packets 2. Bridges sometimes create duplicates and send them to multiple interfaces Neither of these should be any concern to iSCSI. While they might consume resources, only one copy should ever make it thru the TCP stack to the application (iSCSI in this case). The TCP layer will recognize that duplicates are duplicates and will silently drop them - happens all the time. But, retransmissions from the application layer (FC, FCP) will be new data at the TCP level. Such retransmissions will have to be handled at the application layer. JD. From owner-ips@ECE.cmu.edu Mon Sep 11 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31002; 11 Sep 2000 18:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA22673 for ips-outgoing; Mon, 11 Sep 2000 17:07:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA22663 for ; Mon, 11 Sep 2000 17:06:56 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA09328; Mon, 11 Sep 2000 13:56:56 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id NAA09557; Mon, 11 Sep 2000 13:54:34 -0700 From: "Douglas Otis" To: "Raj Bhagwat" , "Vivek Kashyap" , Cc: Subject: RE: FCIP and wandering duplicates Date: Mon, 11 Sep 2000 14:09:14 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <00af01c01c26$18bdc9c0$17940a18@msnv1.occa.home.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Raj, I do not understand what tuning TTL does to benefit any situation. The path between locations is not fixed so such tuning may induce lost packets and is unrelated to time in transit. Depending on the domain, a packet may be repeated by equipment that does not necessarily decrement TTL. Although not a desired feature, duplication may happen if the packet transmit status is made uncertain by local signaling. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Raj Bhagwat > Sent: Monday, September 11, 2000 12:26 PM > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com > Cc: ips@ece.cmu.edu > Subject: Re: FCIP and wandering duplicates > > > Doug, > > I understand how exactly TTL is used. That is why I stated that TTL should > be set to a value larger than the number of hops between the source and > destination. An IP end station may not know this and it may choose to set > the TTL to the max value of 255. However, an FCIP gateway, with some > intelligence, can perhaps know the # of hops to other FCIP > gateways it talks > to, especially in enterprise networks. > > It is not clear to me how there can be duplicate frames without the source > sending twice. Could you please explain that? > > Thanks, > > Raj. > > ----- Original Message ----- > From: Douglas Otis > To: Raj Bhagwat ; Vivek Kashyap ; > > Cc: > Sent: Monday, September 11, 2000 11:53 AM > Subject: RE: FCIP and wandering duplicates > > > > Raj, > > > > You can see duplications without the source sending twice. Reducing TTL > > value may prevent the packet from ever arriving. The number of > times TTL > is > > decremented depends on the path and not time in transit. > > > > Doug > > > > > -----Original Message----- > > > From: Raj Bhagwat [mailto:rajb@lightsand.com] > > > Sent: Monday, September 11, 2000 11:31 AM > > > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com > > > Cc: ips@ece.cmu.edu > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > Doug, > > > > > > Your point on TTL is well taken. It is there to prevent a packet from > > > getting stuck in a routing loop. Duplicates of unicast packets can > happen > > > when the sender retransmits a packet and both the original and the > > > retransmitted packets reach the destination. If the sender does not > > > retransmit for an adequate amount of time, duplicates of > unicast packets > > > should not happen. This is true even when there are routing loops or > > > topology changes in the network. > > > > > > As you have pointed out, there is no direct correlation between > > > E_D_TOV and > > > TTL. And that could be a problem. However, setting the TTL to a > relatively > > > smaller value (larger than the # of hops between the source and > > > destination > > > FCIP gateways) can reduce the possibility of duplicates when > the sender > of > > > the FC frame waits for at least E_D_TOV before retransmitting > the frame. > > > > > > I don't think SCTP at the FCIP gateway will solve the duplicate > > > problem even > > > with a single retry. There is no synchronization between the FC device > and > > > the FCIP gateway for error recovery. So, the FC device can retransmit > > > independent of the SCTP at the FCIP gateway. One way to solve this > problem > > > is to fine tune the timeout values on the FC side and at the > SCTP level > on > > > the FCIP gateway. > > > > > > Regards, > > > > > > Raj Bhagwat > > > LightSand Communications > > > > > > > > > ----- Original Message ----- > > > From: Douglas Otis > > > To: Raj Bhagwat ; Vivek Kashyap ; > > > > > > Cc: > > > Sent: Monday, September 11, 2000 10:14 AM > > > Subject: RE: FCIP and wandering duplicates > > > > > > > > > > Raj, > > > > > > > > Unless you are willing to wrap the FC frame within some protocol, > > > duplicates > > > > can happen and this has nothing to do with TTL. TTL > prevents circular > > > paths > > > > from forever sending a packet within a loop. A modified > SCTP protocol > > > that > > > > will solve both of these problems, if restricted one time retry. A > > > > Metro-Area-Network should induce about 5ms RTT delay such that > > > a 10ms skew > > > > becomes possible but is limited by the single retry. TTL buys > > > you nothing > > > > with this E_D_TOV, Error_Detect Timeout. > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf > Of > > > > > Raj Bhagwat > > > > > Sent: Sunday, September 10, 2000 10:57 PM > > > > > To: Vivek Kashyap; ipfc@standards.gadzoox.com > > > > > Cc: ips@ece.cmu.edu > > > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > > > > > > > Vivek, > > > > > > > > > > In a properly implemented FC-only fabric, duplicates should > > > not happen. > > > A > > > > > frame that enters a fabric should either be delivered > within E_D_TOV > > > > > (typically 2 sec) or be discarded. FC devices don't retransmit a > frame > > > > > before the E_D_TOV timer expires. > > > > > > > > > > You do bring up an interesting point when two SAN islands are > > > > > bridged across > > > > > an IP network. Once an FC frame enters the IP network, it is hard > > > > > to say how > > > > > the E_D_TOV is enforced by that part of the network. As authors of > the > > > > > FCoverIP draft, we have given some thought to the possibility of > > > > > delivering > > > > > duplicate frames when both the TCP stack at the FCIP > gateway and the > > > > > original FC device initiate recovery/retransmission > procedure in an > > > > > overlapping time period. This is one of the major reasons why we > > > initially > > > > > chose to make the FCIP gateway a stateless device with no > > > buffering and > > > no > > > > > retransmission. > > > > > > > > > > The good news is that the {IP encapsulated} FC frame will be > protected > > > by > > > > > the IP TTL once it enters the IP network. The bad news is that > > > > > detecting the > > > > > E_D_TOV timeout while the frame is inside the IP network may be > > > difficult. > > > > > > > > > > Regards, > > > > > > > > > > Raj Bhagwat > > > > > LightSand Communications > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > From: Vivek Kashyap > > > > > To: > > > > > Sent: Sunday, September 10, 2000 3:17 PM > > > > > Subject: FCIP and wandering duplicates > > > > > > > > > > > > > > > > I read the FCIP document recently. It does not address the issue > of > > > > > > duplicates. > > > > > > > > > > > > Does fibre channel protect itself from duplicates ? I looked at > the > > > > > > header and it does not have a time to live field. Thus > a duplicate > > > > > > caught in a loop and then released at the 'right' time could be > > > > > > accepted by the receiver. Even if the FC fabric path > algorithms do > > > > > > ensure loop free paths transient loops due to errors, switch > > > > > > misconfigurations or failures are possible. Does FC handle > > > such a case > > > ? > > > > > > > > > > > > If it does not then FCIP will exacerbate the problem because the > IP > > > > > > network could have 'transient' loops. Such a 'wandering > duplicate' > > > > > > could be re-injected into the FC island and be accepted > by a newer > > > > > > incarnation of a FC inter-island exchange. Furthermore with > > > no TTL one > > > > > > can't really provide a timout between starting two > conversations. > > > > > > > > > > > > > > > > > > > > > > > > Vivek > > > > > > -- > > > > > > Vivek Kashyap > > > > > > IBM-NumaQ > > > > > > viv@sequent.com > > > > > > > > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Sep 11 22:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31090; 11 Sep 2000 18:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA25164 for ips-outgoing; Mon, 11 Sep 2000 18:05:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (truffle.Stanford.EDU [171.64.73.147]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA25157 for ; Mon, 11 Sep 2000 18:05:31 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 536258E35; Mon, 11 Sep 2000 15:02:42 -0700 (PDT) To: ips@ece.cmu.edu Subject: Avoiding deadlock in iSCSI Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 11 Sep 2000 15:02:42 -0700 Message-ID: Lines: 62 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk The problem: iSCSI, as currently spec'ed, allows SCSI commands and data to be interleaved fairly freely on a TCP connection. A target that stops reading from a TCP connection to avoid reading more command packets also prevents itself from reading data packets. Those data packets may be criticial to making progress on the currently executing command. Note the issue appears with one TCP connection for control and data and even appears in many of the multiple connection schemes. Data in iSCSI comes in two forms: 1) solicited - data requested by target via RTT - data requested by initiator via a SCSI command 2) unsolicited - data sent by initiator without having received an RTT The analysis below assumes that unsolicited data travels over the same TCP connection as SCSI commands. Otherwise, you run the risk of receiving unsolicited data before the relevant SCSI command (thus making implementations more complex). Four solutions: 1) Don't overflow the command queue (i.e. use credits) - and what do you do if a misbehaving initiator overflows your command queue anyway? Drop the connection? - requires you to reserve resources per initiator. some people may want to overcommit 2) Allow dropping of SCSI commands when queue fills - how do you clean up after a dropped SCSI command? - there may be other commands in the pipeline One approach: On command drop, the target enters an error state. While in the error state, all newly received commands terminate with an error until the initiator explicitly clears the error state using a "clear error state" message. You might think that TASK SET FULL and ACA mechanisms from SCSI could be used to attack this problem. However, TASK SET FULL errors don't trigger ACA (in my reading of the SAM). Also, ACA is only triggered by the current enabled command, not by random commands entered into the task set. 3) Put solicited data on a dedicated TCP connection. Require that unsolicited data MUST follow the command, ideally in the same iSCSI PDU 4) (Do it like NFS) Make all transfers from initiator to target unsolicited. Make sure unsolicited data follows the command immediately. Of all the options, #1 and #4 sound the easiest to implement. #2 is more sophisticated than #1. #3 is just plain clever but that's rarely a good thing. :) #4 has large ramifications on current SCSI target designs. -Costa From owner-ips@ECE.cmu.edu Mon Sep 11 23:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31233; 11 Sep 2000 19:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA25798 for ips-outgoing; Mon, 11 Sep 2000 18:22:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from galileo5.galileo.co.il (pop3.galileo.co.il [199.203.130.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA25793 for ; Mon, 11 Sep 2000 18:22:37 -0400 (EDT) Received: from galileo.co.il ([10.2.3.2]) by galileo.co.il (8.8.5/8.8.5) with ESMTP id BAA13641; Tue, 12 Sep 2000 01:22:17 +0200 (GMT-2) Message-ID: <39BD7636.F64A1FC8@galileo.co.il> Date: Tue, 12 Sep 2000 01:17:58 +0100 From: Nafea Bishara Organization: Galileo Technology X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley CC: ips Subject: Re: Command Queue Depth (was asymmetric/Symmetric) References: <39B8781F.CAA9B067@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Matt regarding Ethernet Switches, i can tell you from my experience in architecting wide range of those: For a Gigabit switch, the latency of a 2KByte packet is 16us in today arch and going below 10us in next generation. note the following: - the numbers given there is from last-bit in to first-bit out, so, it is different than the "normal" latency which is first-bit in to first-bit out - whether the switch is acting on Layer-2 or Layer-3 does not matter since all switching decision done on the fly - Ethernet switch latency depends on packet length due to store and forward behaviour. - In Our devices, priority is given based on the TCP/UDP connection, so iSCSI connection always get higher priority than other data ( FTP, HTTP ), and thus, experience the lowest latency possible ( 16us for 2KByte ) -Nafea Matt Wakeley wrote: > Jim McGrath wrote: > > > I agree that BB credit has nothing to do with commands per se, but > > illustrates the problem we have had with deciding on policies for the > > distributed allocation of device resources over multiple initiators. In my > > postscript I noted that the same problems have arisen on command queue (how > > many queue slots do you get), with similarly no satisfactory solution. > > So it sounds to me like this distribution of (command) resources across > initiators is a T10 SCSI issue, and should be solved there, not by each > individual transport (yesterday FC - which didn't solve it, today iSCSI, > tomorrow IB or whatever). > > You've lost me on the following two paragraphs... Ethernet doesn't require > credits to send frames - it's a ship and pray model. I don't know what the > latency through ethernet switches is, but I'd hope it wasn't in the ms range. > Finally, a "usefull" FC-AL will have many devices on it (say 20 drives in a > JBOD) and the latency of the elastic store really starts adding up. So I still > contend that FC-AL is not a low latency medium. > > -Matt > > > On FC-AL, the latency to get an initial credit is typically measured in us > > for a couple of reasons. First, much of that logic has been automated in > > the interface hardware (indeed, the major source of delay is typically the > > elasticity buffer, which is not store and forward and so has a very low > > latency compared to many switches or routers). Second, the distances are > > very small (e.g. hundreds of meters), so both transmission time and the > > opportunity for intervening devices to increase latency is lower than in the > > general internet world. > > > > So generally 2 credits (of 2Kbyte frames) is enough to cover the latency and > > get you into streaming. If the latency of the system was measured in ms, > > then a transport on a Gibt wire would require more like 50 initial credits > > (or more) to cover the latency. Unless we are designing for a low latency > > environment for the exchange of credits (like those where FC-AL are used), > > then we probably need to allocate so much buffer space that it becomes > > difficult to promise initial credits to a lot of potential initiators. > > > > Jim > > > > PS general Fibre Channel (e.g. with switches and the like) is a bit > > different. > > > > -----Original Message----- > > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > > Sent: Wednesday, September 06, 2000 8:23 PM > > To: ips > > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > > Jim, > > > > I agree that FC has tried (unsuccessfully) to address this command queue > > allocation problem. > > > > However, the "login bb credit" mechanism in FC does not address the command > > queue depth issue at all. BB credit is used to receive commands and/or > > data, > > and the target has no clue in advance what is coming. BB credit is just > > there > > to ensure that there is a lowest layer buffer available to receive the FC > > frame > > (as opposed to dropping it on the floor if there is no "mac" buffer, like > > ethernet does). It does not mean the command queue has any room for the > > frame. > > > > At one time, there was a big push to have "data" credits and "command" > > credits > > to take care of this problem, but it couldn't be made to work and be > > "backwards > > compatible". > > > > > The issue of buffer space allocation for multiple initiators has a long > > and > > > troubled history in SCSI. We have never been able to come up with a good > > > answer. > > > > > > Fibre Channel tried to fix this with the notion of "login BB credit" - > > when > > > you login you get a minimum number of credits you are always guaranteed > > when > > > you start data transfers. The problem with this is that storage devices > > had > > > no realistic ability to discriminate between initiators or to change the > > > login BB credit. In addition, the expectation is that all possible > > > initiators would get these credits on login. So storage devices vendors > > > have played it safe and kept this number low (at 0 until recently, now > > > around 2). For iSCSI the number of initial credits you need to "prime the > > > pump" until normal data flow is established is probably large (given the > > > latencies are higher than in Fibre Channel, especially FC-AL), and the > > > > How is the latency low on FC-AL?? given that you need to arbitrate and win > > the > > loop, then receive bb credit, before you can send anything? > > > > -Matt > > > > > > > > number of potential initiators larger than in Fibre Channel, making this a > > > whole lot worse for the storage device. > > > > > > As soon as we allow the devices to start adjusting these credits, then you > > > have the protocol problem of making sure people know when their credits > > are > > > adjusted and the policy problem of how, who, and when to adjust the > > credits. > > > Changing everyone's credit when you add a new initiator can get into a > > > notification nightmare, although it is "fair." Any policy brings up all > > > sorts of nasty issues regarding fairness vs efficient use of the > > > transmission media. > > > > > > Jim > > > > > > Note: the same problem has plagued other attempts to allocate device > > > resources between multiple initiators, like command queue space. In > > general > > > policies with respect to multiple initiators are not really standard in > > the > > > SCSI world. > > > > > > -----Original Message----- > > > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > > > Sent: Wednesday, September 06, 2000 2:56 PM > > > To: ips > > > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > > > > Joshua Tseng wrote: > > > > > > > James, > > > > > > > > I agree with others that there may be an issue with the > > > > command windowing mechanism in the existing iSCSI spec. It is like > > > > "TCP in reverse", in that the target determines the size of the window, > > > and > > > > not the initiator as in TCP. Rather, I believe that everything that > > this > > > > windowing mechanism is attempting to achieve can be more easily obtained > > > > by having the target communicate its buffer size to the initiator at > > > > iSCSI login. It should be the role of the initiator to determine how > > > > many commands to put in flight simultaneously, given this input on > > > available > > > > buffer size from the target. > > > > > > As more initiators connect to a target, it may need to scale back the > > amount > > > of > > > this buffering it has allocated to each previously logged in initiator (to > > > prevent rejecting new logins). > > > > > > > > > > > > > > > As far as multiple initiators, could this not be resolved by the target > > > > refusing additional logins beyond the number of initiators it can safely > > > > support? Not being a storage expert, this is my best guess/suggestion > > > > at how to do it. > > > > > > I believe John already answered this... > > > > > > -Matt -- Nafea \\|// (o o) ~~~~~~~~~~~~oOOo~(_)~oOOo~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ _/_/_/_/ _/_/_/_/_/_/ _/_/_/_/_/_/_/_/ _/_/_/_/_/_/_/_/ _/_/_/_/ Nafea Bishara _/_/_/ Director, Product definition _/_/_/ _/_/_/_/ Galileo Technology Ltd. _/_/_/ _/_/_/ Email - nafea@galileo.co.il _/_/_/ _/_/_/ Snail Mail- D.N. Misgav 20184, Moshav Manof, ISRAEL. _/_/_/_/_/_/_/_/ Tel - Manof + 972 4 9999555 ext. 0 _/_/_/_/_/_/ Haifa + 972 4 8225046 ext. 417 _/_/_/_/ Mobile+ 972 54 995417 FAX - + 972 4 8326420 Check our Web site: http://www.galileoT.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From owner-ips@ECE.cmu.edu Mon Sep 11 23:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31218; 11 Sep 2000 18:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA24957 for ips-outgoing; Mon, 11 Sep 2000 17:59:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA24952 for ; Mon, 11 Sep 2000 17:59:52 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA09357; Mon, 11 Sep 2000 14:49:53 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA09710; Mon, 11 Sep 2000 14:47:33 -0700 From: "Douglas Otis" To: , Subject: RE: Data in SCSI Response or SCSI Data Date: Mon, 11 Sep 2000 15:02:14 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, The "bridging patch" is to reconcile iSCSI expected length against SCSI CDB length. For a definition see FCP FCP_RESID. I could not find any information relating to this in the SAM specification as suggested. It is not directly related to SCSI nor is it directly related with SCSI non-error residual. By combining status with the Read PDU 0x45, the ability to "oops" on perhaps a substantial transfer is removed. Once started, there is no means to report a problem with the transfer. This combining of status has limited merit as a series of smaller reads carries this added residual and status as wasted overhead in all but the last transfer. The Read PDU moves the point of processing status optionally to the beginning or middle of the command rather than at the end as in all other cases. This READ PDU implies perfection in transport as it predicts the future with no end of transfer confirmation. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Saturday, September 09, 2000 12:57 AM > To: ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > > > Overrun/Underrun situations are a "bridging patch" from the block count > model > used by the "classical SCSI" (perhaps matched by some buffering model in > some transports) and the byte count model specified by SAM. > > As a command has a byte count associated to it and the CDB has a block > count > the resulting inconsistency (if any) is reported in overrun/underrun byte > count delivered by > the target. > > As for a non-zero residual being an error, as far as I can read it - it is > today a target decision > and except for some legacy devices most do not report it as an error ( a > far cry from > the 360 channel days where it was an error unless explicitely > requesting it > not to be by > a bit in the command). > > Julo > > "Douglas Otis" on 08/09/2000 21:50:16 > > Please respond to "Douglas Otis" > > To: "Robert Snively" , "Stephen Bailey" > , ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Data in SCSI Response or SCSI Data > > > > > Bob, > > Thanks, that helps an understanding of this PDU 0x45. If iSCSI is to copy > FCP then it should send a response structure as FCP and not one or two and > use it according to FCP. Undefined information of what some consider SCSI > values cause variants in use. I have not seen tape drives use Good Sense > residual but vary in how Check Sense residual is presented. If > iSCSI is to > copy FC, then why deviate on response symmetry? At least this provides an > easier bridge. Iterative read operations will carry unused values to be > examined and opens the door for further variants. Status presented prior > to > data must then ensure sequence without end confirmation. > > Doug > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > Sent: Friday, September 08, 2000 9:20 AM > > To: 'Douglas Otis'; Stephen Bailey; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > In FCP, because FC is a multi-protocol environment that may have > > drivers independent of the SCSI drivers, an actual buffer allocation > > was transmitted independently of the SCSI command. Both overrun > > and underrun indicators and a residual are returned. They are > > protocol related, not SCSI related. > > > > There are cases where incorrect length indications are provided > > in the SCSI model for both under-run and over-run cases, but they > > are rare and typically associated with legacy tape drive programs. > > > > Bob > > > > > -----Original Message----- > > > From: Douglas Otis [mailto:dotis@sanlight.net] > > > Sent: Wednesday, September 06, 2000 6:26 PM > > > To: Stephen Bailey; ips@ece.cmu.edu > > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > > > Steph, > > > > > > Although a logical use of a residual, I can not see where it > > > is defined > > > should the allocation length be greater than the returned > > > length. Should > > > the allocation be less than the response, this is a Check > > > Condition and the > > > residual is defined. It would seem a residual in a case > > > with adequate > > > allocation is not interesting or defined. Do you know where > > > this mechanism > > > you describe is defined? In normal use, without a Check > > > Condition such > > > information is not returned to the application. > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > > Stephen Bailey > > > > Sent: Wednesday, September 06, 2000 3:45 PM > > > > To: ips@ece.cmu.edu > > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > > > > When do you get GOOD status and residual counts on a read? > > > > What is causing > > > > > the target to get the length wrong? > > > > > > > > One example is an INQUIRY command. The inquiry data length is > > > > target-specific. Typically the CDB allocation length (and > > > DL) are set > > > > to some arbitrary large value (0xff), and the target sends back > > > > everything it has. The transfer ends with success status. > > > > > > > > There can certainly be transfer residual and no SCSI error status. > > > > > > > > Steph > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Sep 11 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31427; 11 Sep 2000 19:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA27188 for ips-outgoing; Mon, 11 Sep 2000 19:02:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA27183 for ; Mon, 11 Sep 2000 19:01:56 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 405A515 for ; Mon, 11 Sep 2000 16:01:55 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id QAA20747; Mon, 11 Sep 2000 16:01:53 -0700 (PDT) Message-ID: <39BD65ED.DAD80C83@hp.com> Date: Mon, 11 Sep 2000 16:08:29 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Symmetric vs Asymmetric Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk 1) I don't like the asymmetric model because when solving the synchronization of the commands by sending the commands on a same TCP connexion, it just moves the synchronization problem in an other place : how to synchronize the commands and their data (pb described but Matt and Somesh). 2)The advantage of the asymmetric model (when two connections at least): "avoid the problem of commands flow controlled by large transfer of data" is not an advantage compared to the symmetric model. Because to do so the asymmetric model needs at least two connections. In this case, the symmetric model could avoid this problem too : - opening two TCP connections one for the normal command (read/write,...) and a second one for the urgent commands (task management etc...) 3) As stated it seems that we needs a flow control of the commands because if we use the SCSI one (TASK SET FULL) the traffic we will instable and composed of a sequence of bursts with regularly a full RoundTripTime*Throughput/size of a command number of commands to retry. The command window in the iSCSI draft seems not to be a good mechanism to flow control the commands, because if there is a mix of slow and fast commands, the result will be a task set half full but a window closed, hence the traffic blocked. Why not replace in the draft the "MAXCmdRN" by a credit that is a number of commands in flight allowed (not yet completed) independent of the "CmdRN"? Doing that, the flow will not be blocked by a slow command but will blocked only upon the filling of the task set. Pierre From owner-ips@ECE.cmu.edu Tue Sep 12 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31512; 11 Sep 2000 19:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA26260 for ips-outgoing; Mon, 11 Sep 2000 18:35:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA26248 for ; Mon, 11 Sep 2000 18:35:18 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA10875 for ; Mon, 11 Sep 2000 15:35:17 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.154.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA26891 for ; Mon, 11 Sep 2000 15:35:13 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id PAA00836; Mon, 11 Sep 2000 15:35:13 -0700 (PDT) Message-Id: <200009112235.PAA00836@ha10nwk.EBay.Sun.COM> Date: Mon, 11 Sep 2000 15:35:13 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: jRx8oF5QdE8GTxj6irPTLQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk I think in following this discussion the terminology has been confusing me. When I read "unsolicited data" I interpreted that to mean data for which no command has yet been sent. In general I consider that to be a bug and the receiver should just drop the data on the floor. The only possible scenerio where it might not be a bug is if a command was sent on one connection and the data on the data connection arrived first, thus it is unsolicited. My first assumption is that the sender would not send commands C1 and C2 and data D2 and D1 on the same connection. Doing that creates nasty ordering problems we want to avoid. So if the receiver simply allows the data connection TCP window to shrink the unsolicted data will flow control to a stop until the command queue catches up. With multiple data connections, some may flow control but the active command will be able to make progress on one connection. This may not be the most efficient mechanism but it is "safe". Preferably the data will either follow the command on the same data/command connection or the sender will request a RTT (aka R2T). It is also a sender bug to request a connection for data transfer that it has already sent "unsolicited" data. Unless my assumptions and definitions are wrong, I don't see the issue. -David > The problem: > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > interleaved fairly freely on a TCP connection. A target that stops > reading from a TCP connection to avoid reading more command packets > also prevents itself from reading data packets. Those data packets > may be criticial to making progress on the currently executing > command. > > Note the issue appears with one TCP connection for control and data > and even appears in many of the multiple connection schemes. > > Data in iSCSI comes in two forms: > > 1) solicited - data requested by target via RTT > - data requested by initiator via a SCSI command > 2) unsolicited - data sent by initiator without having received an RTT > > The analysis below assumes that unsolicited data travels over the same > TCP connection as SCSI commands. Otherwise, you run the risk of receiving > unsolicited data before the relevant SCSI command (thus making > implementations more complex). > > Four solutions: > > 1) Don't overflow the command queue (i.e. use credits) > - and what do you do if a misbehaving initiator overflows > your command queue anyway? Drop the connection? > > - requires you to reserve resources per initiator. some people > may want to overcommit > > 2) Allow dropping of SCSI commands when queue fills > - how do you clean up after a dropped SCSI command? > - there may be other commands in the pipeline > > One approach: On command drop, the target enters an error > state. While in the error state, all newly received commands > terminate with an error until the initiator explicitly clears > the error state using a "clear error state" message. > > You might think that TASK SET FULL and ACA mechanisms from SCSI > could be used to attack this problem. However, TASK SET FULL errors > don't trigger ACA (in my reading of the SAM). Also, ACA is only > triggered by the current enabled command, not by random commands > entered into the task set. > > 3) Put solicited data on a dedicated TCP connection. Require that > unsolicited data MUST follow the command, ideally in the same iSCSI > PDU > > 4) (Do it like NFS) Make all transfers from initiator to target > unsolicited. Make sure unsolicited data follows the command > immediately. > > > Of all the options, #1 and #4 sound the easiest to implement. #2 is more > sophisticated than #1. #3 is just plain clever but that's rarely a good > thing. :) #4 has large ramifications on current SCSI target designs. > > -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 01:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31937; 11 Sep 2000 21:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28207 for ips-outgoing; Mon, 11 Sep 2000 19:32:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28199 for ; Mon, 11 Sep 2000 19:32:09 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G0Q000NVWZUBJ@mta5.snfc21.pbi.net> for IPS@ECE.CMU.edu; Mon, 11 Sep 2000 16:23:55 -0700 (PDT) Date: Mon, 11 Sep 2000 16:15:14 -0700 From: Peter Johansson Subject: RE: Command Queue Depth (was asymmetric/Symmetric) In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000911160718.00c673a0@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:38 PM 9/10/00, Jim McGrath wrote: >So yes, I think the SBP-2 style of target initiated actions might make >some sense in heavily loaded systems, which a more traditional approach is >appropriate for lightly loaded situations. The trick is having the >distributed targets and initiators figuring out when to use each one >(difficult) or focusing on one (as we have traditionally) and basically >optimizing latency for light or heavy loads, but not both. Jim, I agree with what you've said (except to correct a small nit, responded to out of order below so as not to distract from the main topic) and think it's especially interesting in the light of the SBP-2 study group convening at T10 this week in Huntington Beach. When the working group first developed SBP-2 we were pleased with our efforts in one particular aspect: the protocol became more efficient the more heavily it was loaded with commands. Now that we have two years experience, there is a better understanding that it is worthwhile to have a special optimization to reduce startup latency when a target is idle, with no commands to process. In the case of SBP-2 this seems to be simple to do; I look forward to your comments if your able to be there on Friday. >In the extreme the target would only has to accept out of the blue an >alert message (just a bit of information) from each initiator, telling it >that the initiator has something for it ... This is exactly what SBP-2 does with the DOORBELL register (that signals the target that new work awaits in the initiator). Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Tue Sep 12 01:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31948; 11 Sep 2000 21:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28609 for ips-outgoing; Mon, 11 Sep 2000 19:44:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28603 for ; Mon, 11 Sep 2000 19:44:30 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA38982 for ; Mon, 11 Sep 2000 19:42:44 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8BNiTb110948 for ; Mon, 11 Sep 2000 17:44:29 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 11 Sep 2000 16:42:49 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/11/2000 05:44:29 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, I think I agree with the comment from David Robinson, who said (among other things) "...I don't see the issue.". At least I do not see the issue with a command and a data Asymmetric Session. Perhaps all we have to do is make a rule, that no unsolicited data can be sent, before its command is sent AND that unsolicited data must be sent in the same order as the commands that reference it. If that rule is followed I do not see the problem. The Data is (or should be) always ready to be read off the data connection queue when needed by the command. If the point is, even if the commands and data arrive in order, some commands my execute quicker then others, then I would say that is even true when the data follows immediately behind the command (on the same connection), and it is the responsibility of a good controller to move the command and data into an execution buffer/cache so that other commands can get through and execute. We do this today. If your controller does not have enough storage to get the slower commands out of the way, then, they may block until the memory/buffer/cache is made available (we all have this kind of mechanism), but there should not be any Dead Locks. If the concern is that in a TCP/IP network there may be some additional memory needed in the storage controller to make the Blocking go away (at least for 99+% of the time), I would say, sounds OK to me, that is part of the system engineering that we all do today. So Costa, I think I probably missed your point, and if you would be so kind as to lay the problem out again, perhaps addressing my points above, I might finally "get it". . . . John L. Hufferd csapuntz@cisco.com@ece.cmu.edu on 09/11/2000 03:02:42 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: csapuntz@cisco.com Subject: Avoiding deadlock in iSCSI The problem: iSCSI, as currently spec'ed, allows SCSI commands and data to be interleaved fairly freely on a TCP connection. A target that stops reading from a TCP connection to avoid reading more command packets also prevents itself from reading data packets. Those data packets may be criticial to making progress on the currently executing command. Note the issue appears with one TCP connection for control and data and even appears in many of the multiple connection schemes. Data in iSCSI comes in two forms: 1) solicited - data requested by target via RTT - data requested by initiator via a SCSI command 2) unsolicited - data sent by initiator without having received an RTT The analysis below assumes that unsolicited data travels over the same TCP connection as SCSI commands. Otherwise, you run the risk of receiving unsolicited data before the relevant SCSI command (thus making implementations more complex). Four solutions: 1) Don't overflow the command queue (i.e. use credits) - and what do you do if a misbehaving initiator overflows your command queue anyway? Drop the connection? - requires you to reserve resources per initiator. some people may want to overcommit 2) Allow dropping of SCSI commands when queue fills - how do you clean up after a dropped SCSI command? - there may be other commands in the pipeline One approach: On command drop, the target enters an error state. While in the error state, all newly received commands terminate with an error until the initiator explicitly clears the error state using a "clear error state" message. You might think that TASK SET FULL and ACA mechanisms from SCSI could be used to attack this problem. However, TASK SET FULL errors don't trigger ACA (in my reading of the SAM). Also, ACA is only triggered by the current enabled command, not by random commands entered into the task set. 3) Put solicited data on a dedicated TCP connection. Require that unsolicited data MUST follow the command, ideally in the same iSCSI PDU 4) (Do it like NFS) Make all transfers from initiator to target unsolicited. Make sure unsolicited data follows the command immediately. Of all the options, #1 and #4 sound the easiest to implement. #2 is more sophisticated than #1. #3 is just plain clever but that's rarely a good thing. :) #4 has large ramifications on current SCSI target designs. -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32300; 11 Sep 2000 22:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA28954 for ips-outgoing; Mon, 11 Sep 2000 19:54:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA28948 for ; Mon, 11 Sep 2000 19:54:55 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA83114 for ; Mon, 11 Sep 2000 19:48:55 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8BNsrb107464 for ; Mon, 11 Sep 2000 17:54:53 -0600 Importance: Normal Subject: Re: Symmetric vs Asymmetric To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 11 Sep 2000 16:51:30 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/11/2000 05:54:53 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, I am not sure that you factored in Immediate/Unsolicited Data into your argument. . . . John L. Hufferd Pierre Labat @ece.cmu.edu on 09/11/2000 04:08:29 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Symmetric vs Asymmetric 1) I don't like the asymmetric model because when solving the synchronization of the commands by sending the commands on a same TCP connexion, it just moves the synchronization problem in an other place : how to synchronize the commands and their data (pb described but Matt and Somesh). 2)The advantage of the asymmetric model (when two connections at least): "avoid the problem of commands flow controlled by large transfer of data" is not an advantage compared to the symmetric model. Because to do so the asymmetric model needs at least two connections. In this case, the symmetric model could avoid this problem too : - opening two TCP connections one for the normal command (read/write,...) and a second one for the urgent commands (task management etc...) 3) As stated it seems that we needs a flow control of the commands because if we use the SCSI one (TASK SET FULL) the traffic we will instable and composed of a sequence of bursts with regularly a full RoundTripTime*Throughput/size of a command number of commands to retry. The command window in the iSCSI draft seems not to be a good mechanism to flow control the commands, because if there is a mix of slow and fast commands, the result will be a task set half full but a window closed, hence the traffic blocked. Why not replace in the draft the "MAXCmdRN" by a credit that is a number of commands in flight allowed (not yet completed) independent of the "CmdRN"? Doing that, the flow will not be blocked by a slow command but will blocked only upon the filling of the task set. Pierre From owner-ips@ECE.cmu.edu Tue Sep 12 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32330; 11 Sep 2000 23:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA01905 for ips-outgoing; Mon, 11 Sep 2000 21:36:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA01896 for ; Mon, 11 Sep 2000 21:36:20 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id SAA29230 for ; Mon, 11 Sep 2000 18:36:16 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.188.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id SAA12628 for ; Mon, 11 Sep 2000 18:36:15 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id SAA10716; Mon, 11 Sep 2000 18:36:16 -0700 (PDT) Message-Id: <200009120136.SAA10716@ha10nwk.EBay.Sun.COM> Date: Mon, 11 Sep 2000 18:36:15 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: rmz+zsQiFV5U6j95o6mfdA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks for the information, I think part of my confusion is the difference in mapping SCSI from a datagram protocol onto a reliable stream protocol. In a datagram protocol if the data is sent without the receiver's cooperation, the receiver's buffers may not be adequate and the data must get discarded. Credits and RTT can be used to handle this case. With a reliable stream transport like TCP, you don't get into this situation because the receiver will never open the TCP window beyond it's buffer capacity. For low amounts of buffering it might not be as efficient as using RTT, but there is no correctness or deadlock issues. Because each sender has its own connection and own flow control they are independantly handled. Likewise with seperate data connections, each is also flow controlled so "unsolicted" data is not an issue. "Overflow" conditions simply never occur. The only major design criteria is that the sender MUST maintain ordering of data sent on any connection. Data Dn MUST always be sent before data Dm where n < m. In particular, if unsolicated data and RTT is mixed, the sender cannot send data Dm before it has recieved an RTT for data Dn if both are to use the same connection. -David > I think people have been meaning "unsolicited data" to really mean data sent > to a receiver without that receiver having first indicated that there is > enough buffering to hold the data. For initiators acting as receivers they > have to verify this before they initiate the command (not enough space for > the whole command? Then break up the command.) For Targets this requires > something like a credit mechanism with RTTs being used. > > So there is an "unsolicited command" problem and an "unsolicited data" > problem. In both cases the sender creates the problem by not first > reserving with the receiver enough resources for the commands/data. > > In the command case there is no SCSI mechanism to reserve resources (QUEUE > FULL is used to indicate overflows). Historically it has been assumed that > queues of commands do not overflow often in practice. In reality initiators > have often artificially limited the number of commands they are willing to > try and queue at the target in order to avoid this rejection (a loss > opportunity in my mind). > > In the data case there is no "DATA QUEUE FULL" - instead, an explicit credit > model of some sort is used to indicate the receiver has reserved space for > the data (REQs in parallel SCSI, BB credits in Fibre Channel). In this case > the assumption was that data overflows would occur a lot otherwise. > > You can solve these problems by rejecting the overflow cleanly (as SCSI does > with commands), which is low latency and works well under light loads. Or > you can do credits. Credits add latency, or get you into the problem of > credit allocation, which can be optimized for light load (over allocate > credits) or heavy loads (allocate only what you have), but not both at once. > > Historically, SCSI has used rejection for commands and credits for data, > optimized for heavy loads. But this is only a T10 given rule, not a God > given rule (although some of us who have served on T10 can get that confused > at times :-)). > > Hope this helps. > > Jim > > > > > > cases there are well known mechanisms to reserve the > > -----Original Message----- > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > Sent: Monday, September 11, 2000 3:35 PM > To: ips@ece.cmu.edu > Subject: Re: Avoiding deadlock in iSCSI > > > I think in following this discussion the terminology has been > confusing me. When I read "unsolicited data" I interpreted that > to mean data for which no command has yet been sent. In general > I consider that to be a bug and the receiver should just drop the > data on the floor. The only possible scenerio where it might > not be a bug is if a command was sent on one connection and the > data on the data connection arrived first, thus it is unsolicited. > My first assumption is that the sender would not send commands > C1 and C2 and data D2 and D1 on the same connection. Doing that > creates nasty ordering problems we want to avoid. So if the > receiver simply allows the data connection TCP window to shrink > the unsolicted data will flow control to a stop until the command > queue catches up. With multiple data connections, some may flow > control but the active command will be able to make progress on > one connection. This may not be the most efficient mechanism but > it is "safe". Preferably the data will either follow the command > on the same data/command connection or the sender will request a > RTT (aka R2T). It is also a sender bug to request a connection > for data transfer that it has already sent "unsolicited" data. > > Unless my assumptions and definitions are wrong, I don't see the issue. > > -David > > > The problem: > > > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > > interleaved fairly freely on a TCP connection. A target that stops > > reading from a TCP connection to avoid reading more command packets > > also prevents itself from reading data packets. Those data packets > > may be criticial to making progress on the currently executing > > command. > > > > Note the issue appears with one TCP connection for control and data > > and even appears in many of the multiple connection schemes. > > > > Data in iSCSI comes in two forms: > > > > 1) solicited - data requested by target via RTT > > - data requested by initiator via a SCSI command > > 2) unsolicited - data sent by initiator without having received an > RTT > > > > The analysis below assumes that unsolicited data travels over the same > > TCP connection as SCSI commands. Otherwise, you run the risk of receiving > > unsolicited data before the relevant SCSI command (thus making > > implementations more complex). > > > > Four solutions: > > > > 1) Don't overflow the command queue (i.e. use credits) > > - and what do you do if a misbehaving initiator overflows > > your command queue anyway? Drop the connection? > > > > - requires you to reserve resources per initiator. some people > > may want to overcommit > > > > 2) Allow dropping of SCSI commands when queue fills > > - how do you clean up after a dropped SCSI command? > > - there may be other commands in the pipeline > > > > One approach: On command drop, the target enters an error > > state. While in the error state, all newly received commands > > terminate with an error until the initiator explicitly clears > > the error state using a "clear error state" message. > > > > You might think that TASK SET FULL and ACA mechanisms from SCSI > > could be used to attack this problem. However, TASK SET FULL > errors > > don't trigger ACA (in my reading of the SAM). Also, ACA is only > > triggered by the current enabled command, not by random commands > > entered into the task set. > > > > 3) Put solicited data on a dedicated TCP connection. Require that > > unsolicited data MUST follow the command, ideally in the same iSCSI > > PDU > > > > 4) (Do it like NFS) Make all transfers from initiator to target > > unsolicited. Make sure unsolicited data follows the command > > immediately. > > > > > > Of all the options, #1 and #4 sound the easiest to implement. #2 is more > > sophisticated than #1. #3 is just plain clever but that's rarely a good > > thing. :) #4 has large ramifications on current SCSI target designs. > > > > -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32350; 11 Sep 2000 23:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA29945 for ips-outgoing; Mon, 11 Sep 2000 20:26:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA29941 for ; Mon, 11 Sep 2000 20:26:35 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLT95; Mon, 11 Sep 2000 17:24:21 -0700 From: "Y P Cheng" To: Subject: RE: Avoiding deadlock in iSCSI Date: Mon, 11 Sep 2000 17:25:26 -0700 Message-ID: <005c01c01c4f$f2cba3c0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa wrote: >iSCSI, as currently spec'ed, allows SCSI commands and data to be >interleaved fairly freely on a TCP connection. A target that stops >reading from a TCP connection to avoid reading more command packets >also prevents itself from reading data packets. Those data packets >may be criticial to making progress on the currently executing >command. > >Note the issue appears with one TCP connection for control and data >and even appears in many of the multiple connection schemes. > >Data in iSCSI comes in two forms: > 1) solicited - data requested by target via RTT > - data requested by initiator via a SCSI command > 2) unsolicited - data sent by initiator without having received an RTT I appreciate your examples. On incoming SCSI commands, the SCSI-QUEUE-FULL status is used to tell an initiator that it needs to resend the command again. Most initiator avoids this status by tracking how many commands being sent to each SCSI device. On unsolicited data, it is tough because a target does not have unlimited resources with so many potential initiators on the net. RTT can force the data being solicited. Another method is to use the queue-depth parameters to each initiators and taking a chance on over-subscribing. However, when an incoming PDU is dropped, the target must inform the initiator. This can be done in the microcode of a NIC adapter easily but not necessarily in an IP device driver. In our NIC implementation, all incoming data must be solicited. Even so, some PDUs can still be dropped because of slow host system bus. No error goes unreported. Even so, the initiator still has the responsibility of detecting if PDU's are dropped due to traffic jam on the gateway or due to broken connections. Y.P. Cheng, CTO, ConnectCom Solutions Corp. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of csapuntz@cisco.com Sent: Monday, September 11, 2000 3:03 PM To: ips@ece.cmu.edu Cc: csapuntz@cisco.com Subject: Avoiding deadlock in iSCSI The problem: iSCSI, as currently spec'ed, allows SCSI commands and data to be interleaved fairly freely on a TCP connection. A target that stops reading from a TCP connection to avoid reading more command packets also prevents itself from reading data packets. Those data packets may be criticial to making progress on the currently executing command. Note the issue appears with one TCP connection for control and data and even appears in many of the multiple connection schemes. Data in iSCSI comes in two forms: 1) solicited - data requested by target via RTT - data requested by initiator via a SCSI command 2) unsolicited - data sent by initiator without having received an RTT The analysis below assumes that unsolicited data travels over the same TCP connection as SCSI commands. Otherwise, you run the risk of receiving unsolicited data before the relevant SCSI command (thus making implementations more complex). Four solutions: 1) Don't overflow the command queue (i.e. use credits) - and what do you do if a misbehaving initiator overflows your command queue anyway? Drop the connection? - requires you to reserve resources per initiator. some people may want to overcommit 2) Allow dropping of SCSI commands when queue fills - how do you clean up after a dropped SCSI command? - there may be other commands in the pipeline One approach: On command drop, the target enters an error state. While in the error state, all newly received commands terminate with an error until the initiator explicitly clears the error state using a "clear error state" message. You might think that TASK SET FULL and ACA mechanisms from SCSI could be used to attack this problem. However, TASK SET FULL errors don't trigger ACA (in my reading of the SAM). Also, ACA is only triggered by the current enabled command, not by random commands entered into the task set. 3) Put solicited data on a dedicated TCP connection. Require that unsolicited data MUST follow the command, ideally in the same iSCSI PDU 4) (Do it like NFS) Make all transfers from initiator to target unsolicited. Make sure unsolicited data follows the command immediately. Of all the options, #1 and #4 sound the easiest to implement. #2 is more sophisticated than #1. #3 is just plain clever but that's rarely a good thing. :) #4 has large ramifications on current SCSI target designs. -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32339; 11 Sep 2000 23:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA01704 for ips-outgoing; Mon, 11 Sep 2000 21:28:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA01698 for ; Mon, 11 Sep 2000 21:28:03 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id SAA09428; Mon, 11 Sep 2000 18:18:02 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id SAA10256; Mon, 11 Sep 2000 18:15:41 -0700 From: "Douglas Otis" To: "David Robinson" , Subject: RE: Avoiding deadlock in iSCSI Date: Mon, 11 Sep 2000 18:30:24 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <200009112235.PAA00836@ha10nwk.EBay.Sun.COM> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > David Robinson > Sent: Monday, September 11, 2000 3:35 PM > To: ips@ece.cmu.edu > Subject: Re: Avoiding deadlock in iSCSI > > > I think in following this discussion the terminology has been > confusing me. When I read "unsolicited data" I interpreted that > to mean data for which no command has yet been sent. David, Here is my understanding. Unsolicited may mean data sent with a command. In SCSI, the target controls exchanges and thus makes all solicitations. >From the aspect of the SCSI model, unsolicited would imply data sent ahead of a request from the target by the normal means of R2T. > In general > I consider that to be a bug and the receiver should just drop the > data on the floor. Not as I understand solicit. > The only possible scenerio where it might > not be a bug is if a command was sent on one connection and the > data on the data connection arrived first, thus it is unsolicited. Yes, unsolicited and out of sequence. > My first assumption is that the sender would not send commands > C1 and C2 and data D2 and D1 on the same connection. Doing that > creates nasty ordering problems we want to avoid. Order on the wire can not be controlled. Only the ULP can avoid such. > So if the > receiver simply allows the data connection TCP window to shrink > the unsolicted data will flow control to a stop until the command > queue catches up. Resources are held until associated data is received to complete operations. If the resource limit is not the data buffer nor freed by content already within the data buffer, this will result in discarding commands. > With multiple data connections, some may flow > control but the active command will be able to make progress on > one connection. This may not be the most efficient mechanism but > it is "safe". One connection per LUN or one connection per command, safe but expensive? > Preferably the data will either follow the command > on the same data/command connection or the sender will request a > RTT (aka R2T). It is also a sender bug to request a connection > for data transfer that it has already sent "unsolicited" data. As a means for freeing resources, data is to be discarded within the iSCSI architecture. As such, even unsolicited data may be requested by the target. Doug > Unless my assumptions and definitions are wrong, I don't see the issue. > > -David > > > The problem: > > > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > > interleaved fairly freely on a TCP connection. A target that stops > > reading from a TCP connection to avoid reading more command packets > > also prevents itself from reading data packets. Those data packets > > may be criticial to making progress on the currently executing > > command. > > > > Note the issue appears with one TCP connection for control and data > > and even appears in many of the multiple connection schemes. > > > > Data in iSCSI comes in two forms: > > > > 1) solicited - data requested by target via RTT > > - data requested by initiator via a SCSI command > > 2) unsolicited - data sent by initiator without having > received an RTT > > > > The analysis below assumes that unsolicited data travels over the same > > TCP connection as SCSI commands. Otherwise, you run the risk of > receiving > > unsolicited data before the relevant SCSI command (thus making > > implementations more complex). > > > > Four solutions: > > > > 1) Don't overflow the command queue (i.e. use credits) > > - and what do you do if a misbehaving initiator overflows > > your command queue anyway? Drop the connection? > > > > - requires you to reserve resources per initiator. some people > > may want to overcommit > > > > 2) Allow dropping of SCSI commands when queue fills > > - how do you clean up after a dropped SCSI command? > > - there may be other commands in the pipeline > > > > One approach: On command drop, the target enters an error > > state. While in the error state, all newly received commands > > terminate with an error until the initiator explicitly clears > > the error state using a "clear error state" message. > > > > You might think that TASK SET FULL and ACA mechanisms from SCSI > > could be used to attack this problem. However, TASK SET > FULL errors > > don't trigger ACA (in my reading of the SAM). Also, ACA is only > > triggered by the current enabled command, not by random commands > > entered into the task set. > > > > 3) Put solicited data on a dedicated TCP connection. Require that > > unsolicited data MUST follow the command, ideally in the same iSCSI > > PDU > > > > 4) (Do it like NFS) Make all transfers from initiator to target > > unsolicited. Make sure unsolicited data follows the command > > immediately. > > > > > > Of all the options, #1 and #4 sound the easiest to implement. #2 is more > > sophisticated than #1. #3 is just plain clever but that's rarely a good > > thing. :) #4 has large ramifications on current SCSI target designs. > > > > -Costa > From owner-ips@ECE.cmu.edu Tue Sep 12 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32321; 11 Sep 2000 23:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA01446 for ips-outgoing; Mon, 11 Sep 2000 21:17:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA01442 for ; Mon, 11 Sep 2000 21:17:45 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id SAA07708; Mon, 11 Sep 2000 18:17:41 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 11 Sep 2000 18:17:39 -0700 Message-ID: From: Jim McGrath To: "'David Robinson'" , ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Date: Mon, 11 Sep 2000 18:17:39 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I think people have been meaning "unsolicited data" to really mean data sent to a receiver without that receiver having first indicated that there is enough buffering to hold the data. For initiators acting as receivers they have to verify this before they initiate the command (not enough space for the whole command? Then break up the command.) For Targets this requires something like a credit mechanism with RTTs being used. So there is an "unsolicited command" problem and an "unsolicited data" problem. In both cases the sender creates the problem by not first reserving with the receiver enough resources for the commands/data. In the command case there is no SCSI mechanism to reserve resources (QUEUE FULL is used to indicate overflows). Historically it has been assumed that queues of commands do not overflow often in practice. In reality initiators have often artificially limited the number of commands they are willing to try and queue at the target in order to avoid this rejection (a loss opportunity in my mind). In the data case there is no "DATA QUEUE FULL" - instead, an explicit credit model of some sort is used to indicate the receiver has reserved space for the data (REQs in parallel SCSI, BB credits in Fibre Channel). In this case the assumption was that data overflows would occur a lot otherwise. You can solve these problems by rejecting the overflow cleanly (as SCSI does with commands), which is low latency and works well under light loads. Or you can do credits. Credits add latency, or get you into the problem of credit allocation, which can be optimized for light load (over allocate credits) or heavy loads (allocate only what you have), but not both at once. Historically, SCSI has used rejection for commands and credits for data, optimized for heavy loads. But this is only a T10 given rule, not a God given rule (although some of us who have served on T10 can get that confused at times :-)). Hope this helps. Jim cases there are well known mechanisms to reserve the -----Original Message----- From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] Sent: Monday, September 11, 2000 3:35 PM To: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI I think in following this discussion the terminology has been confusing me. When I read "unsolicited data" I interpreted that to mean data for which no command has yet been sent. In general I consider that to be a bug and the receiver should just drop the data on the floor. The only possible scenerio where it might not be a bug is if a command was sent on one connection and the data on the data connection arrived first, thus it is unsolicited. My first assumption is that the sender would not send commands C1 and C2 and data D2 and D1 on the same connection. Doing that creates nasty ordering problems we want to avoid. So if the receiver simply allows the data connection TCP window to shrink the unsolicted data will flow control to a stop until the command queue catches up. With multiple data connections, some may flow control but the active command will be able to make progress on one connection. This may not be the most efficient mechanism but it is "safe". Preferably the data will either follow the command on the same data/command connection or the sender will request a RTT (aka R2T). It is also a sender bug to request a connection for data transfer that it has already sent "unsolicited" data. Unless my assumptions and definitions are wrong, I don't see the issue. -David > The problem: > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > interleaved fairly freely on a TCP connection. A target that stops > reading from a TCP connection to avoid reading more command packets > also prevents itself from reading data packets. Those data packets > may be criticial to making progress on the currently executing > command. > > Note the issue appears with one TCP connection for control and data > and even appears in many of the multiple connection schemes. > > Data in iSCSI comes in two forms: > > 1) solicited - data requested by target via RTT > - data requested by initiator via a SCSI command > 2) unsolicited - data sent by initiator without having received an RTT > > The analysis below assumes that unsolicited data travels over the same > TCP connection as SCSI commands. Otherwise, you run the risk of receiving > unsolicited data before the relevant SCSI command (thus making > implementations more complex). > > Four solutions: > > 1) Don't overflow the command queue (i.e. use credits) > - and what do you do if a misbehaving initiator overflows > your command queue anyway? Drop the connection? > > - requires you to reserve resources per initiator. some people > may want to overcommit > > 2) Allow dropping of SCSI commands when queue fills > - how do you clean up after a dropped SCSI command? > - there may be other commands in the pipeline > > One approach: On command drop, the target enters an error > state. While in the error state, all newly received commands > terminate with an error until the initiator explicitly clears > the error state using a "clear error state" message. > > You might think that TASK SET FULL and ACA mechanisms from SCSI > could be used to attack this problem. However, TASK SET FULL errors > don't trigger ACA (in my reading of the SAM). Also, ACA is only > triggered by the current enabled command, not by random commands > entered into the task set. > > 3) Put solicited data on a dedicated TCP connection. Require that > unsolicited data MUST follow the command, ideally in the same iSCSI > PDU > > 4) (Do it like NFS) Make all transfers from initiator to target > unsolicited. Make sure unsolicited data follows the command > immediately. > > > Of all the options, #1 and #4 sound the easiest to implement. #2 is more > sophisticated than #1. #3 is just plain clever but that's rarely a good > thing. :) #4 has large ramifications on current SCSI target designs. > > -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 03:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32406; 11 Sep 2000 23:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA01937 for ips-outgoing; Mon, 11 Sep 2000 21:37:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA01932 for ; Mon, 11 Sep 2000 21:37:41 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id SAA10035; Mon, 11 Sep 2000 18:37:39 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 11 Sep 2000 18:37:38 -0700 Message-ID: From: Jim McGrath To: "'csapuntz@cisco.com'" , ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Date: Mon, 11 Sep 2000 18:37:37 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Note that SCSI targets, when faced with getting a command queue full, do not stop reading from the interconnect. If more commands are received then they respond with a QUEUE FULL status. If data is received then they receive that data without regard for the status of the command queue (as long as it is data for an already queued command). This eliminates the potential for deadlock between command and data queues. Potentially you can have a situation where multiple commands already at the target have only part of their data transmitted, with the remainder still at the initiator(s), and then run out of buffer space for data. If the target uses a credit model to pace the reception of data, it can also make sure this never happens. Unsolicited data, even for commands already queued, can end up creating this deadlock - which is why unsolicited data systems either have to have a tight limit on the resources it can use (e.g. low login BB credit in Fibre Channel terms) or some sort of clean (i.e. not IO terminating) rejection mechanism from target to initiator (like in USB). If data is not received by the target, then it thinks the initiator has more credits than the initiator thinks it does. If the data was delayed in transit, then the target still thinks the credit is outstanding to the initiator, and so should not reuse it yet for that or any other initiator. You may get a pause in the data transfer, but you should not get a deadlock. If the data was lost then some sort of timeout catches the problem. In sum, if existing SCSI rules are used there should not be a problem except for receiving data before its command or getting delayed data. The target can distinguish between the two cases. In the first case you probably should drop the data (but notify the initiator so recovery can be done). In the second, you just wait until the data catches up - as long as the delay is small, things are manageable. Otherwise drop the data and notify the initiator (i.e. handle just like the case of data before a command). Obviously the initiator should not try and generate these problems for the target, but as someone pointed out, the target has to have some defined behavior if the initiator (or fabric) introduces the problem. Jim -----Original Message----- From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] Sent: Monday, September 11, 2000 3:03 PM To: ips@ece.cmu.edu Cc: csapuntz@cisco.com Subject: Avoiding deadlock in iSCSI The problem: iSCSI, as currently spec'ed, allows SCSI commands and data to be interleaved fairly freely on a TCP connection. A target that stops reading from a TCP connection to avoid reading more command packets also prevents itself from reading data packets. Those data packets may be criticial to making progress on the currently executing command. Note the issue appears with one TCP connection for control and data and even appears in many of the multiple connection schemes. Data in iSCSI comes in two forms: 1) solicited - data requested by target via RTT - data requested by initiator via a SCSI command 2) unsolicited - data sent by initiator without having received an RTT The analysis below assumes that unsolicited data travels over the same TCP connection as SCSI commands. Otherwise, you run the risk of receiving unsolicited data before the relevant SCSI command (thus making implementations more complex). Four solutions: 1) Don't overflow the command queue (i.e. use credits) - and what do you do if a misbehaving initiator overflows your command queue anyway? Drop the connection? - requires you to reserve resources per initiator. some people may want to overcommit 2) Allow dropping of SCSI commands when queue fills - how do you clean up after a dropped SCSI command? - there may be other commands in the pipeline One approach: On command drop, the target enters an error state. While in the error state, all newly received commands terminate with an error until the initiator explicitly clears the error state using a "clear error state" message. You might think that TASK SET FULL and ACA mechanisms from SCSI could be used to attack this problem. However, TASK SET FULL errors don't trigger ACA (in my reading of the SAM). Also, ACA is only triggered by the current enabled command, not by random commands entered into the task set. 3) Put solicited data on a dedicated TCP connection. Require that unsolicited data MUST follow the command, ideally in the same iSCSI PDU 4) (Do it like NFS) Make all transfers from initiator to target unsolicited. Make sure unsolicited data follows the command immediately. Of all the options, #1 and #4 sound the easiest to implement. #2 is more sophisticated than #1. #3 is just plain clever but that's rarely a good thing. :) #4 has large ramifications on current SCSI target designs. -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 03:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32418; 11 Sep 2000 23:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA02394 for ips-outgoing; Mon, 11 Sep 2000 21:53:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA02384 for ; Mon, 11 Sep 2000 21:52:34 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id SAA02278 for ; Mon, 11 Sep 2000 18:52:23 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.187.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id SAA23546 for ; Mon, 11 Sep 2000 18:52:22 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id SAA18183; Mon, 11 Sep 2000 18:52:22 -0700 (PDT) Message-Id: <200009120152.SAA18183@ha10nwk.EBay.Sun.COM> Date: Mon, 11 Sep 2000 18:52:22 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: HL/oaMaRGKfbSLdp4blLpQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks for your clarifying comments. > > In general > > I consider that to be a bug and the receiver should just drop the > > data on the floor. > > Not as I understand solicit. This I don't understand, it you get data that no command has been sent yet, as the reciever you can either wait and hope the command arrives consuming buffer space but eventually dropping it after some timeout, or drop it on the spot. In either case this seems like a bad design we should try to avoid. > > My first assumption is that the sender would not send commands > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > creates nasty ordering problems we want to avoid. > > Order on the wire can not be controlled. Only the ULP can avoid such. Yes it can, this is exactly the advantage of using a reliable stream protocol like TCP, the session layer never sees out of order packets. With multiple data connections and the appropriate ordering constraints we have no deadlock or buffer management issues. > Resources are held until associated data is received to complete operations. > If the resource limit is not the data buffer nor freed by content already > within the data buffer, this will result in discarding commands. But with a reliable stream no commands need to be discarded, the transport flow controls so the commands are held at the sender. > > With multiple data connections, some may flow > > control but the active command will be able to make progress on > > one connection. This may not be the most efficient mechanism but > > it is "safe". > > One connection per LUN or one connection per command, safe but expensive? Define "expensive", not in terms of performance as one TCP connection can saturate the link layer or not in terms of memory as the mux/demux state has to be held either in the transport or the session layer. I am not advocating a connection per command as that is just a bad datagram protocol, but either a connection per LUN or per target should work just fine. > As a means for freeing resources, data is to be discarded within the iSCSI > architecture. As such, even unsolicited data may be requested by the > target. I don't understand this statement. Short of target errors or connection errors, why does data need to be discarded? The sender should never send data without a command, and on a given connection the data MUST always be sent after the corresponding command and data from two commands must always be sent in order if on the same connection. -David From owner-ips@ECE.cmu.edu Tue Sep 12 06:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00680; 12 Sep 2000 2:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA08175 for ips-outgoing; Tue, 12 Sep 2000 01:35:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (adsl-63-192-219-28.dsl.snfc21.pacbell.net [63.192.219.28]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA08171 for ; Tue, 12 Sep 2000 01:35:33 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 4BDE88E48; Mon, 11 Sep 2000 22:32:44 -0700 (PDT) To: Jim McGrath Cc: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 11 Sep 2000 22:32:44 -0700 In-Reply-To: Jim McGrath's message of "Mon, 11 Sep 2000 18:37:37 -0700" Message-ID: Lines: 42 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk > Note that SCSI targets, when faced with getting a command queue full, do not > stop reading from the interconnect. If more commands are received then they > respond with a QUEUE FULL status. If data is received then they receive > that data without regard for the status of the command queue (as long as it > is data for an already queued command). This eliminates the potential for > deadlock between command and data queues. Jim, I believe there is problem with the current SCSI behavior. Consider the following scenario for a host with a pathological queue of 1 1) Initiator sends command 1 (ORDERED attribute) 2) Initiator sends command 2 (ORDERED attribute) 3) Initiator sends command 3 (ORDERED attributge) 4) Target reads command 1 5) Target reads command 2 6) Target returns queue full for command 2 7) Command 1 completes 8) Target reads command 3 9) Target executes command 3 We have just violated the ordering constraints of the application by doing command 3 before command 2. > Potentially you can have a situation where multiple commands already at the > target have only part of their data transmitted, with the remainder still at > the initiator(s), and then run out of buffer space for data. If the target > uses a credit model to pace the reception of data, it can also make sure > this never happens. Unsolicited data, even for commands already queued, can > end up creating this deadlock - which is why unsolicited data systems either > have to have a tight limit on the resources it can use (e.g. low login BB > credit in Fibre Channel terms) or some sort of clean (i.e. not IO > terminating) rejection mechanism from target to initiator (like in USB). Unsolicited data is NOT a problem with the current iSCSI spec. We allow the target to always drop data and request data transfers with an RTT. -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 06:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00685; 12 Sep 2000 2:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA08052 for ips-outgoing; Tue, 12 Sep 2000 01:30:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (adsl-63-192-219-28.dsl.snfc21.pacbell.net [63.192.219.28]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA08047 for ; Tue, 12 Sep 2000 01:30:53 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 448248E48; Mon, 11 Sep 2000 22:28:03 -0700 (PDT) To: "John Hufferd/San Jose/IBM" Cc: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 11 Sep 2000 22:28:03 -0700 In-Reply-To: "John Hufferd/San Jose/IBM"'s message of "Mon, 11 Sep 2000 16:42:49 -0700" Message-ID: Lines: 52 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk > I think I agree with the comment from David Robinson, who said (among > other things) "...I don't see the issue.". I'm starting to think that maybe I'm missing something here. :-) > At least I do not see the issue with a command and a data Asymmetric > Session. Perhaps all we have to do is make a rule, that no unsolicited > data can be sent, before its command is sent AND that unsolicited data must > be sent in the same order as the commands that reference it. The above is a good rule. I agree with it heartily. Immediate/unsolicited data should probably be stapled onto the SCSI command. But I do not believe it is sufficient to prevent problems. > If that rule > is followed I do not see the problem. The Data is (or should be) always > ready to be read off the data connection queue when needed by the command. Here is the scenario I proposed. This scenario applies to the iSCSI model as presented in Pittsburg, in which there is only one TCP connection (shared between commands and data). There is no unsolicited data. 1) Initiator sends command 1 (WRITE BLOCK) 2) Initiator sends command 2 (WRITE BLOCK) 3) Target reads command 1 4) Target sends RTT to initiator for data in command 1 5) Target reads from connection and notices comand 2 6) Target stops reading from connection because it does not have room for command 2 7) Initiator sends WRITE data for command 1. WRITE data is stuck behind command 2 in the stream. There are basically three points where you can solve the problem: Make sure #2 never happens (initiator never overflows a target queue) or ... Make #6 to be Target drops command 2 or ... Make #7 is Initiator sends data for command 1 on a separate dedicated, solicited-data only TCP connection. Target reads data off the dedicated, solicited-data only TCP conneciton -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 08:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01275; 12 Sep 2000 4:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA09892 for ips-outgoing; Tue, 12 Sep 2000 02:55:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA09887 for ; Tue, 12 Sep 2000 02:55:06 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id BAA16018 for ; Tue, 12 Sep 2000 01:48:48 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8C6t5b82646 for ; Tue, 12 Sep 2000 00:55:05 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 11 Sep 2000 23:50:05 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/12/2000 12:55:05 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, I think I understand your points, but please also consider the Asymmetric approach. Please consider the Asymmetric approach and also as I said in my last note, suppose that we "make a rule, that no unsolicited data can be sent, before its command is sent AND that unsolicited data must be sent in the same order as the commands that reference it. Along with that rule, and the statements below from David -- will, in your opinion, the Asymmetric approach work even with unsolicited data? . . . John L. Hufferd ---------------------- Forwarded by John Hufferd/San Jose/IBM on 09/11/2000 11:19 PM --------------------------- David Robinson @ece.cmu.edu on 09/11/2000 06:52:22 PM Please respond to David Robinson Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: Avoiding deadlock in iSCSI Thanks for your clarifying comments. > > In general > > I consider that to be a bug and the receiver should just drop the > > data on the floor. > > Not as I understand solicit. This I don't understand, it you get data that no command has been sent yet, as the reciever you can either wait and hope the command arrives consuming buffer space but eventually dropping it after some timeout, or drop it on the spot. In either case this seems like a bad design we should try to avoid. > > My first assumption is that the sender would not send commands > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > creates nasty ordering problems we want to avoid. > > Order on the wire can not be controlled. Only the ULP can avoid such. Yes it can, this is exactly the advantage of using a reliable stream protocol like TCP, the session layer never sees out of order packets. With multiple data connections and the appropriate ordering constraints we have no deadlock or buffer management issues. > Resources are held until associated data is received to complete operations. > If the resource limit is not the data buffer nor freed by content already > within the data buffer, this will result in discarding commands. But with a reliable stream no commands need to be discarded, the transport flow controls so the commands are held at the sender. > > With multiple data connections, some may flow > > control but the active command will be able to make progress on > > one connection. This may not be the most efficient mechanism but > > it is "safe". > > One connection per LUN or one connection per command, safe but expensive? Define "expensive", not in terms of performance as one TCP connection can saturate the link layer or not in terms of memory as the mux/demux state has to be held either in the transport or the session layer. I am not advocating a connection per command as that is just a bad datagram protocol, but either a connection per LUN or per target should work just fine. > As a means for freeing resources, data is to be discarded within the iSCSI > architecture. As such, even unsolicited data may be requested by the > target. I don't understand this statement. Short of target errors or connection errors, why does data need to be discarded? The sender should never send data without a command, and on a given connection the data MUST always be sent after the corresponding command and data from two commands must always be sent in order if on the same connection. -David From owner-ips@ECE.cmu.edu Tue Sep 12 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02582; 12 Sep 2000 8:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA25610 for ips-outgoing; Tue, 12 Sep 2000 07:52:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA25605 for ; Tue, 12 Sep 2000 07:52:29 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA00965; Tue, 12 Sep 2000 06:52:58 -0500 Message-ID: <39BE1919.5C9F8588@stewart.chicago.il.us> Date: Tue, 12 Sep 2000 06:52:57 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: csapuntz@cisco.com CC: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk csapuntz@cisco.com wrote: > > The problem: > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > interleaved fairly freely on a TCP connection. A target that stops > reading from a TCP connection to avoid reading more command packets > also prevents itself from reading data packets. Those data packets > may be criticial to making progress on the currently executing > command. > > Note the issue appears with one TCP connection for control and data > and even appears in many of the multiple connection schemes. > > Data in iSCSI comes in two forms: > > 1) solicited - data requested by target via RTT > - data requested by initiator via a SCSI command > 2) unsolicited - data sent by initiator without having received an RTT > > The analysis below assumes that unsolicited data travels over the same > TCP connection as SCSI commands. Otherwise, you run the risk of receiving > unsolicited data before the relevant SCSI command (thus making > implementations more complex). > > Four solutions: > > 1) Don't overflow the command queue (i.e. use credits) > - and what do you do if a misbehaving initiator overflows > your command queue anyway? Drop the connection? > > - requires you to reserve resources per initiator. some people > may want to overcommit > > 2) Allow dropping of SCSI commands when queue fills > - how do you clean up after a dropped SCSI command? > - there may be other commands in the pipeline > > One approach: On command drop, the target enters an error > state. While in the error state, all newly received commands > terminate with an error until the initiator explicitly clears > the error state using a "clear error state" message. > > You might think that TASK SET FULL and ACA mechanisms from SCSI > could be used to attack this problem. However, TASK SET FULL errors > don't trigger ACA (in my reading of the SAM). Also, ACA is only > triggered by the current enabled command, not by random commands > entered into the task set. > > 3) Put solicited data on a dedicated TCP connection. Require that > unsolicited data MUST follow the command, ideally in the same iSCSI > PDU > > 4) (Do it like NFS) Make all transfers from initiator to target > unsolicited. Make sure unsolicited data follows the command > immediately. > > > Of all the options, #1 and #4 sound the easiest to implement. #2 is more > sophisticated than #1. #3 is just plain clever but that's rarely a good > thing. :) #4 has large ramifications on current SCSI target designs. > > -Costa Costa: If you were using SCTP you could easily do #3 by not using a seperate connection but instead use a specified stream in the command for the data to come on... Another options as well would be put each command/and data on a different stream modulo the number of streams you were using. This way one could always seperate out what one was reading :) Some similar things could be worked out with TCP it would just require a bit more work in the iSCSI layer... i.e. one could pass a connection number in the command and this would be which TCP connection you would read from :) R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 12 13:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02597; 12 Sep 2000 9:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA25649 for ips-outgoing; Tue, 12 Sep 2000 07:54:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sphmraaa.compuserve.com (hs-img-rel-1.compuserve.com [149.174.177.156]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA25644 for ; Tue, 12 Sep 2000 07:54:08 -0400 (EDT) Received: (from mailgate@localhost) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id HAA09700 for ips@ece.cmu.edu; Tue, 12 Sep 2000 07:53:33 -0400 (EDT) Received: from compuserve.com (sfr-tgn-sfe-vty11.as.wcom.net [216.192.7.11]) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id HAA09683 for ; Tue, 12 Sep 2000 07:53:30 -0400 (EDT) Message-ID: <39BDFD0F.5F6E3405@compuserve.com> Date: Tue, 12 Sep 2000 04:53:20 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Technical Coordinators References: <10009111715.AA18716@maia.east.isi.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Allison Mankin wrote: > ... And it's worth noting there should be plenary discussions > of all the documents, certainly by the time they mature. Would that be the IETF Plenary or some other plenary? Thanks. Ralph Weber From owner-ips@ECE.cmu.edu Tue Sep 12 13:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02614; 12 Sep 2000 9:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA26399 for ips-outgoing; Tue, 12 Sep 2000 08:19:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from east.isi.edu (east.isi.edu [38.245.76.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA26393 for ; Tue, 12 Sep 2000 08:19:29 -0400 (EDT) Received: from maia.east.isi.edu (maia.east.isi.edu [38.245.76.14]) by east.isi.edu (8.9.2/8.9.2) with SMTP id IAA23505; Tue, 12 Sep 2000 08:19:49 -0400 (EDT) Posted-Date: Tue, 12 Sep 2000 08:22:49 -0400 Message-Id: <10009121222.AA19647@maia.east.isi.edu> Received: from LOCALHOST.east.isi.edu by maia.east.isi.edu (4.1/4.0.3-6) id ; Tue, 12 Sep 00 08:22:53 EDT To: ENDL_TX@computer.org Cc: ips@ece.cmu.edu Reply-To: mankin@east.isi.edu Subject: Re: Technical Coordinators In-Reply-To: Your message of Tue, 12 Sep 2000 04:53:20 -0500. <39BDFD0F.5F6E3405@compuserve.com> Date: Tue, 12 Sep 2000 08:22:49 -0400 From: Allison Mankin Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, I meant plenary within the working group - not the IETF plenary - sorry for the confusion. Allison From owner-ips@ECE.cmu.edu Tue Sep 12 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03097; 12 Sep 2000 10:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA28869 for ips-outgoing; Tue, 12 Sep 2000 09:31:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mickey.dsi.nus.edu.sg (IDENT:root@mickey.dsi.nus.edu.sg [137.132.30.30]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA09392 for ; Tue, 12 Sep 2000 02:29:32 -0400 (EDT) Received: from singnet.com.sg ([137.132.30.87]) by mickey.dsi.nus.edu.sg (8.9.3/8.9.3) with ESMTP id OAA07445 for ; Tue, 12 Sep 2000 14:32:48 +0800 Message-ID: <39BDCB1E.4366F10D@singnet.com.sg> Date: Tue, 12 Sep 2000 14:20:14 +0800 From: cheng-ann tan Reply-To: chengann@singnet.com.sg X-Mailer: Mozilla 4.6 [en] (Win98; I) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: patent question Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk hi, all: i did a patent search on the net and came across emc's patent (5996024) which describes using scsi devices over network protocols. it covers ip, of course. pardon my ignorance, but if a patent has already been granted, how would this affect further work on i-scsi, ip storage, etc? -cheng From owner-ips@ECE.cmu.edu Tue Sep 12 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03027; 12 Sep 2000 10:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA00271 for ips-outgoing; Tue, 12 Sep 2000 09:58:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA00267 for ; Tue, 12 Sep 2000 09:58:23 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id JAA11990; Tue, 12 Sep 2000 09:58:20 -0400 Message-ID: <004701c01cbe$d54f6c90$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: , Cc: Subject: Re: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 09:39:11 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Layering iSCSI on VI avoids the problem below. see: http://search.ietf.org/internet-drafts/draft-csapuntz-ips-iscsivi-00.txt VI layer flow control will be invoked when target is unable to accept more commands, but this will not impede flow of data in response to RDMA read requests by the target. Functionally this is equivalent to option 1 below. -----Original Message----- From: csapuntz@cisco.com To: ips@ece.cmu.edu Cc: csapuntz@cisco.com Date: Monday, September 11, 2000 6:14 PM Subject: Avoiding deadlock in iSCSI > >The problem: > >iSCSI, as currently spec'ed, allows SCSI commands and data to be >interleaved fairly freely on a TCP connection. A target that stops >reading from a TCP connection to avoid reading more command packets >also prevents itself from reading data packets. Those data packets >may be criticial to making progress on the currently executing >command. > >Note the issue appears with one TCP connection for control and data >and even appears in many of the multiple connection schemes. > >Data in iSCSI comes in two forms: > > 1) solicited - data requested by target via RTT > - data requested by initiator via a SCSI command > 2) unsolicited - data sent by initiator without having received an RTT > >The analysis below assumes that unsolicited data travels over the same >TCP connection as SCSI commands. Otherwise, you run the risk of receiving >unsolicited data before the relevant SCSI command (thus making >implementations more complex). > >Four solutions: > >1) Don't overflow the command queue (i.e. use credits) > - and what do you do if a misbehaving initiator overflows > your command queue anyway? Drop the connection? > > - requires you to reserve resources per initiator. some people > may want to overcommit > >2) Allow dropping of SCSI commands when queue fills > - how do you clean up after a dropped SCSI command? > - there may be other commands in the pipeline > > One approach: On command drop, the target enters an error > state. While in the error state, all newly received commands > terminate with an error until the initiator explicitly clears > the error state using a "clear error state" message. > > You might think that TASK SET FULL and ACA mechanisms from SCSI > could be used to attack this problem. However, TASK SET FULL errors > don't trigger ACA (in my reading of the SAM). Also, ACA is only > triggered by the current enabled command, not by random commands > entered into the task set. > >3) Put solicited data on a dedicated TCP connection. Require that >unsolicited data MUST follow the command, ideally in the same iSCSI >PDU > >4) (Do it like NFS) Make all transfers from initiator to target >unsolicited. Make sure unsolicited data follows the command >immediately. > > >Of all the options, #1 and #4 sound the easiest to implement. #2 is more >sophisticated than #1. #3 is just plain clever but that's rarely a good >thing. :) #4 has large ramifications on current SCSI target designs. > >-Costa > From owner-ips@ECE.cmu.edu Tue Sep 12 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03133; 12 Sep 2000 10:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA01998 for ips-outgoing; Tue, 12 Sep 2000 10:35:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA01992 for ; Tue, 12 Sep 2000 10:35:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id QAA15314 for ; Tue, 12 Sep 2000 16:34:27 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA92556 for ; Tue, 12 Sep 2000 16:34:28 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256958.00500C65 ; Tue, 12 Sep 2000 16:34:20 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 12 Sep 2000 10:01:25 +0300 Subject: Re: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, The deadlock - that we keep coming back to - can be avoided in the asymmetric case by a set of simple rules. On any GIVEN connection send the data in the same order you sent the commands (and I mean ALL the data from a command). This combined with the fact that data arrive on known connections (Kalman's proposal) will enable a target to accept data when executing commands and not having to keep around data when it doesn't need it. For the symmetric case follow the same rule but ADD THE FOLLOWING: if on a connection you have read a command that is out of order stop reading from it until the gap is filled (I assume that on any GIVEN connection commands have non-decreasing numbers) If you intend to use execution orders of the more exotic kind then never use unsolicited data. Again we are not far better of with neither asymmetric or symmetric - but the asymmetric is simpler. And if you have a bad initiator - the only worry we should have is that he should not harm others until we can fence him off. On the asymmetric case you could start fencing if the initiator is sending data on any given connection out of order; in the symmetric case if it sends commands or data out of order. Again not that different. Julo csapuntz@cisco.com on 12/09/2000 01:02:42 Please respond to csapuntz@cisco.com To: ips@ece.cmu.edu cc: csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: Avoiding deadlock in iSCSI The problem: iSCSI, as currently spec'ed, allows SCSI commands and data to be interleaved fairly freely on a TCP connection. A target that stops reading from a TCP connection to avoid reading more command packets also prevents itself from reading data packets. Those data packets may be criticial to making progress on the currently executing command. Note the issue appears with one TCP connection for control and data and even appears in many of the multiple connection schemes. Data in iSCSI comes in two forms: 1) solicited - data requested by target via RTT - data requested by initiator via a SCSI command 2) unsolicited - data sent by initiator without having received an RTT The analysis below assumes that unsolicited data travels over the same TCP connection as SCSI commands. Otherwise, you run the risk of receiving unsolicited data before the relevant SCSI command (thus making implementations more complex). Four solutions: 1) Don't overflow the command queue (i.e. use credits) - and what do you do if a misbehaving initiator overflows your command queue anyway? Drop the connection? - requires you to reserve resources per initiator. some people may want to overcommit 2) Allow dropping of SCSI commands when queue fills - how do you clean up after a dropped SCSI command? - there may be other commands in the pipeline One approach: On command drop, the target enters an error state. While in the error state, all newly received commands terminate with an error until the initiator explicitly clears the error state using a "clear error state" message. You might think that TASK SET FULL and ACA mechanisms from SCSI could be used to attack this problem. However, TASK SET FULL errors don't trigger ACA (in my reading of the SAM). Also, ACA is only triggered by the current enabled command, not by random commands entered into the task set. 3) Put solicited data on a dedicated TCP connection. Require that unsolicited data MUST follow the command, ideally in the same iSCSI PDU 4) (Do it like NFS) Make all transfers from initiator to target unsolicited. Make sure unsolicited data follows the command immediately. Of all the options, #1 and #4 sound the easiest to implement. #2 is more sophisticated than #1. #3 is just plain clever but that's rarely a good thing. :) #4 has large ramifications on current SCSI target designs. -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03138; 12 Sep 2000 10:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA02021 for ips-outgoing; Tue, 12 Sep 2000 10:35:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA02009 for ; Tue, 12 Sep 2000 10:35:21 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA198930 for ; Tue, 12 Sep 2000 16:34:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA38014 for ; Tue, 12 Sep 2000 16:34:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256958.005013CA ; Tue, 12 Sep 2000 16:34:39 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 12 Sep 2000 10:50:11 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, You are correct. Ordering is sufficient to make it dead-lock free (see my previous note as response to Costa). I would like to note that on it is also important not to mix or switch from unsolicited to solicited. In the symmetric model we have also mandated serving R2T by the initiator to allow oversubscription. Julo David Robinson on 12/09/2000 04:36:15 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI Thanks for the information, I think part of my confusion is the difference in mapping SCSI from a datagram protocol onto a reliable stream protocol. In a datagram protocol if the data is sent without the receiver's cooperation, the receiver's buffers may not be adequate and the data must get discarded. Credits and RTT can be used to handle this case. With a reliable stream transport like TCP, you don't get into this situation because the receiver will never open the TCP window beyond it's buffer capacity. For low amounts of buffering it might not be as efficient as using RTT, but there is no correctness or deadlock issues. Because each sender has its own connection and own flow control they are independantly handled. Likewise with seperate data connections, each is also flow controlled so "unsolicted" data is not an issue. "Overflow" conditions simply never occur. The only major design criteria is that the sender MUST maintain ordering of data sent on any connection. Data Dn MUST always be sent before data Dm where n < m. In particular, if unsolicated data and RTT is mixed, the sender cannot send data Dm before it has recieved an RTT for data Dn if both are to use the same connection. -David > I think people have been meaning "unsolicited data" to really mean data sent > to a receiver without that receiver having first indicated that there is > enough buffering to hold the data. For initiators acting as receivers they > have to verify this before they initiate the command (not enough space for > the whole command? Then break up the command.) For Targets this requires > something like a credit mechanism with RTTs being used. > > So there is an "unsolicited command" problem and an "unsolicited data" > problem. In both cases the sender creates the problem by not first > reserving with the receiver enough resources for the commands/data. > > In the command case there is no SCSI mechanism to reserve resources (QUEUE > FULL is used to indicate overflows). Historically it has been assumed that > queues of commands do not overflow often in practice. In reality initiators > have often artificially limited the number of commands they are willing to > try and queue at the target in order to avoid this rejection (a loss > opportunity in my mind). > > In the data case there is no "DATA QUEUE FULL" - instead, an explicit credit > model of some sort is used to indicate the receiver has reserved space for > the data (REQs in parallel SCSI, BB credits in Fibre Channel). In this case > the assumption was that data overflows would occur a lot otherwise. > > You can solve these problems by rejecting the overflow cleanly (as SCSI does > with commands), which is low latency and works well under light loads. Or > you can do credits. Credits add latency, or get you into the problem of > credit allocation, which can be optimized for light load (over allocate > credits) or heavy loads (allocate only what you have), but not both at once. > > Historically, SCSI has used rejection for commands and credits for data, > optimized for heavy loads. But this is only a T10 given rule, not a God > given rule (although some of us who have served on T10 can get that confused > at times :-)). > > Hope this helps. > > Jim > > > > > > cases there are well known mechanisms to reserve the > > -----Original Message----- > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > Sent: Monday, September 11, 2000 3:35 PM > To: ips@ece.cmu.edu > Subject: Re: Avoiding deadlock in iSCSI > > > I think in following this discussion the terminology has been > confusing me. When I read "unsolicited data" I interpreted that > to mean data for which no command has yet been sent. In general > I consider that to be a bug and the receiver should just drop the > data on the floor. The only possible scenerio where it might > not be a bug is if a command was sent on one connection and the > data on the data connection arrived first, thus it is unsolicited. > My first assumption is that the sender would not send commands > C1 and C2 and data D2 and D1 on the same connection. Doing that > creates nasty ordering problems we want to avoid. So if the > receiver simply allows the data connection TCP window to shrink > the unsolicted data will flow control to a stop until the command > queue catches up. With multiple data connections, some may flow > control but the active command will be able to make progress on > one connection. This may not be the most efficient mechanism but > it is "safe". Preferably the data will either follow the command > on the same data/command connection or the sender will request a > RTT (aka R2T). It is also a sender bug to request a connection > for data transfer that it has already sent "unsolicited" data. > > Unless my assumptions and definitions are wrong, I don't see the issue. > > -David > > > The problem: > > > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > > interleaved fairly freely on a TCP connection. A target that stops > > reading from a TCP connection to avoid reading more command packets > > also prevents itself from reading data packets. Those data packets > > may be criticial to making progress on the currently executing > > command. > > > > Note the issue appears with one TCP connection for control and data > > and even appears in many of the multiple connection schemes. > > > > Data in iSCSI comes in two forms: > > > > 1) solicited - data requested by target via RTT > > - data requested by initiator via a SCSI command > > 2) unsolicited - data sent by initiator without having received an > RTT > > > > The analysis below assumes that unsolicited data travels over the same > > TCP connection as SCSI commands. Otherwise, you run the risk of receiving > > unsolicited data before the relevant SCSI command (thus making > > implementations more complex). > > > > Four solutions: > > > > 1) Don't overflow the command queue (i.e. use credits) > > - and what do you do if a misbehaving initiator overflows > > your command queue anyway? Drop the connection? > > > > - requires you to reserve resources per initiator. some people > > may want to overcommit > > > > 2) Allow dropping of SCSI commands when queue fills > > - how do you clean up after a dropped SCSI command? > > - there may be other commands in the pipeline > > > > One approach: On command drop, the target enters an error > > state. While in the error state, all newly received commands > > terminate with an error until the initiator explicitly clears > > the error state using a "clear error state" message. > > > > You might think that TASK SET FULL and ACA mechanisms from SCSI > > could be used to attack this problem. However, TASK SET FULL > errors > > don't trigger ACA (in my reading of the SAM). Also, ACA is only > > triggered by the current enabled command, not by random commands > > entered into the task set. > > > > 3) Put solicited data on a dedicated TCP connection. Require that > > unsolicited data MUST follow the command, ideally in the same iSCSI > > PDU > > > > 4) (Do it like NFS) Make all transfers from initiator to target > > unsolicited. Make sure unsolicited data follows the command > > immediately. > > > > > > Of all the options, #1 and #4 sound the easiest to implement. #2 is more > > sophisticated than #1. #3 is just plain clever but that's rarely a good > > thing. :) #4 has large ramifications on current SCSI target designs. > > > > -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03128; 12 Sep 2000 10:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA01609 for ips-outgoing; Tue, 12 Sep 2000 10:25:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA01592 for ; Tue, 12 Sep 2000 10:25:36 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id HAA06169 for ; Tue, 12 Sep 2000 07:25:33 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Tue, 12 Sep 2000 07:25:30 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1A1@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 07:25:31 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa [mailto:csapuntz@cisco.com] wrote: > I believe there is problem with the current SCSI behavior. > Consider the > following scenario for a host with a pathological queue of 1 > > > 1) Initiator sends command 1 (ORDERED attribute) > 2) Initiator sends command 2 (ORDERED attribute) > 3) Initiator sends command 3 (ORDERED attributge) > 4) Target reads command 1 > 5) Target reads command 2 > 6) Target returns queue full for command 2 > 7) Command 1 completes > 8) Target reads command 3 > 9) Target executes command 3 > > We have just violated the ordering constraints of the application by > doing command 3 before command 2. I think this is a T10 bug, and iSCSI should defer to T10 to fix it. From the proposed charter (David, is there an approved charter yet?): "The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable and in that case will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. " In practice, this bug doesn't bite very often because the only devices that commonly implement command queuing are disks, and since most disk commands are stateless, they are not sent as ordered commands. This bug does present a problem for stateful devices such as tapes or communication controllers. To accomodate WAN connections to those devices, one might like to enable command-queuing, to pipeline around the communication latency incurred in turning around from completion status of the previous command to initating the next command. This bug makes it impractical to implement command-queuing on stateful devices. I suggest that it is T10 which needs to fix this bug, not the iSCSI transport. For purposes of developing the iSCSI transport, I suggest we ignore the bug, and assume the target will respond with TASK SET FULL or BUSY, as appropriate. We should additionally document the problem and forward it to T10 with a request to resolve the bug in SAM-2. A possible fix might be to add a mode page bit that would require the target to establish ACA on TASK SET FULL. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Tue Sep 12 15:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03300; 12 Sep 2000 11:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA02020 for ips-outgoing; Tue, 12 Sep 2000 10:35:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA02010 for ; Tue, 12 Sep 2000 10:35:21 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA82664 for ; Tue, 12 Sep 2000 16:34:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA92576 for ; Tue, 12 Sep 2000 16:34:49 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256958.00500EB8 ; Tue, 12 Sep 2000 16:34:26 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 12 Sep 2000 10:08:25 +0300 Subject: Re: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I apologize for all of us. By unsolicited we mean unsolicited by an explicit R2T - i.e. immediate or in separate data blocks but not "armed" with a target tag (a target tag presumably indicates that there is already somebody waiting at the target). Data arriving before the command is but a particular case of it. As for deadlocks see my previous notes for some simple recipes for good behavior. Julo David Robinson on 12/09/2000 01:35:13 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Avoiding deadlock in iSCSI I think in following this discussion the terminology has been confusing me. When I read "unsolicited data" I interpreted that to mean data for which no command has yet been sent. In general I consider that to be a bug and the receiver should just drop the data on the floor. The only possible scenerio where it might not be a bug is if a command was sent on one connection and the data on the data connection arrived first, thus it is unsolicited. My first assumption is that the sender would not send commands C1 and C2 and data D2 and D1 on the same connection. Doing that creates nasty ordering problems we want to avoid. So if the receiver simply allows the data connection TCP window to shrink the unsolicted data will flow control to a stop until the command queue catches up. With multiple data connections, some may flow control but the active command will be able to make progress on one connection. This may not be the most efficient mechanism but it is "safe". Preferably the data will either follow the command on the same data/command connection or the sender will request a RTT (aka R2T). It is also a sender bug to request a connection for data transfer that it has already sent "unsolicited" data. Unless my assumptions and definitions are wrong, I don't see the issue. -David > The problem: > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > interleaved fairly freely on a TCP connection. A target that stops > reading from a TCP connection to avoid reading more command packets > also prevents itself from reading data packets. Those data packets > may be criticial to making progress on the currently executing > command. > > Note the issue appears with one TCP connection for control and data > and even appears in many of the multiple connection schemes. > > Data in iSCSI comes in two forms: > > 1) solicited - data requested by target via RTT > - data requested by initiator via a SCSI command > 2) unsolicited - data sent by initiator without having received an RTT > > The analysis below assumes that unsolicited data travels over the same > TCP connection as SCSI commands. Otherwise, you run the risk of receiving > unsolicited data before the relevant SCSI command (thus making > implementations more complex). > > Four solutions: > > 1) Don't overflow the command queue (i.e. use credits) > - and what do you do if a misbehaving initiator overflows > your command queue anyway? Drop the connection? > > - requires you to reserve resources per initiator. some people > may want to overcommit > > 2) Allow dropping of SCSI commands when queue fills > - how do you clean up after a dropped SCSI command? > - there may be other commands in the pipeline > > One approach: On command drop, the target enters an error > state. While in the error state, all newly received commands > terminate with an error until the initiator explicitly clears > the error state using a "clear error state" message. > > You might think that TASK SET FULL and ACA mechanisms from SCSI > could be used to attack this problem. However, TASK SET FULL errors > don't trigger ACA (in my reading of the SAM). Also, ACA is only > triggered by the current enabled command, not by random commands > entered into the task set. > > 3) Put solicited data on a dedicated TCP connection. Require that > unsolicited data MUST follow the command, ideally in the same iSCSI > PDU > > 4) (Do it like NFS) Make all transfers from initiator to target > unsolicited. Make sure unsolicited data follows the command > immediately. > > > Of all the options, #1 and #4 sound the easiest to implement. #2 is more > sophisticated than #1. #3 is just plain clever but that's rarely a good > thing. :) #4 has large ramifications on current SCSI target designs. > > -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 15:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03305; 12 Sep 2000 11:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA02078 for ips-outgoing; Tue, 12 Sep 2000 10:36:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA02071 for ; Tue, 12 Sep 2000 10:36:31 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id HAA08057 for ; Tue, 12 Sep 2000 07:36:16 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Tue, 12 Sep 2000 07:36:13 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1A2@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 07:36:14 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim Williams [mailto:jimw@giganet.com] wrote: > Layering iSCSI on VI avoids the problem below. > > see: > http://search.ietf.org/internet-drafts/draft-csapuntz-ips-iscs > ivi-00.txt > > VI layer flow control will be invoked when target is unable > to accept more > commands, but this will not impede flow of data in response > to RDMA read > requests by the target. Functionally this is equivalent to > option 1 below. I haven't seen much discussion of iSCSI over VI here. I think it makes sense. Using VI makes the iSCSI remote DMA hardware acceleration available to other applications besides iSCSI. This broader market should greatly increase the utility of iSCSI hardware, and thus speed the market penetration of iSCSI, bringing the usual benefits of customer acceptance, lower cost, and better implementions courtesy of darwinian competition. And, as an added bonus, you get flow control on commands for free :-) Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Tue Sep 12 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03613; 12 Sep 2000 12:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06568 for ips-outgoing; Tue, 12 Sep 2000 12:12:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA06563 for ; Tue, 12 Sep 2000 12:12:10 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id C39067F3 for ; Tue, 12 Sep 2000 09:12:08 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id JAA14563 for ; Tue, 12 Sep 2000 09:12:07 -0700 (PDT) Received: from agilent.com ([15.3.251.54]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA48CB for ; Tue, 12 Sep 2000 09:12:03 -0700 Message-ID: <39BE3DF3.288C4D89@agilent.com> Date: Tue, 12 Sep 2000 07:30:11 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Enhancements for the iSCSI References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk It's hard enough getting what runs on the wire standardized. I don't think you'd get anyone to agree on the HBA API. Luckily for us, we are only defining the wire protocol. -Matt Nelson Nahum wrote: > Ok, there are asynchronous events from the device to the iSCSI HBA, good!. > The question is how these events are passed from the iSCSI HBA to the iSCSI > driver and from the iSCSI driver to the application?. > Applications like SAN File Systems and Virtualization would benefit from > that. > May be, the solution is to define into the standard how the iSCSI driver > will interface with the applications in order to pass the information that > is not strictly SCSI (IOCTLs etc.). > Today, every FC HBAs has its own APIs, this difficult too much the > applications that need to retrieve > FC information from the HBA. > > Nelson Nahum > StoreAge Networking Technologies > > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Saturday, September 09, 2000 6:37 AM > To: IPS Reflector > Subject: Re: Enhancements for the iSCSI > > Nelson Nahum wrote: > > > I think that the iSCSI is a special opportunity to enhance the current > SCSI > > protocol especially it networking capabilities. > > SCSI was designed with the concept of Initiator for the HBA and Target for > > the devices. The Initator sends commands to the Targets that are passive. > > But the networking world supposes every one talking with every one. The > fact > > that iSCSI is implemented over TCP/IP can help to develop these > enhacements. > > > > Modern SAN appliances need to "talk" with the storage devices but also > need > > a way to update information in the HBAs in a asynchronous manner. > > > > My questions are the following: > > > > 1 - Is iSCSI designed in such a way that every Target can also be > Initiator > > and viceversa or it supposes different charasterisitics (memory, CPU power > > etc.) between the Initiator and the Target? > > yep. > > > 2 - Is possible to add asynchronous events that the Target could send to > the > > Initiators at any time? > > That's in the spec too. I guess you haven't read it very closely. > > > 3 - Another important enhancement could be the possibility to "broadcast" > > information to many iSCSI devices or HBAs. > > Why would you want to do that? > > > Nelson Nahum > > CTO > > StoreAge Networking Technologies > > Matt Wakeley > Agilent Technologies From owner-ips@ECE.cmu.edu Tue Sep 12 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03609; 12 Sep 2000 12:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06189 for ips-outgoing; Tue, 12 Sep 2000 12:05:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA06180 for ; Tue, 12 Sep 2000 12:05:14 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id KAA38798 for ; Tue, 12 Sep 2000 10:58:57 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8CG5D299694 for ; Tue, 12 Sep 2000 10:05:13 -0600 X-Priority: 1 (High) Importance: Normal Subject: Avoiding deadlock in iSCSI To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 12 Sep 2000 09:02:06 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/12/2000 10:05:12 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, We seem to be going around and around on this. Julian, David Robinson, and myself (I hope among others) seem to think that with Asymmetric, and a simple ordering rule, everything works with TCP/IP (because of what TCP/IP itself does). There maybe some buffer management in the SCSI "application" (this is the normal stuff that SCSI does), but it is not an iSCSI/TCP issue. Can we agree on that for the Asymmetric case? For the Symmetric case, Julian suggested a few more "minor" rules, and he claims that then it can work as deadlock free as the above Asymmetric case. Can we agree on that for the Symmetric case? . . . John L. Hufferd From owner-ips@ECE.cmu.edu Tue Sep 12 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03655; 12 Sep 2000 12:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06008 for ips-outgoing; Tue, 12 Sep 2000 12:01:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA05986 for ; Tue, 12 Sep 2000 12:00:54 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA108958 for ; Tue, 12 Sep 2000 18:00:22 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA43450 for ; Tue, 12 Sep 2000 18:00:21 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256958.0057EA5A ; Tue, 12 Sep 2000 18:00:16 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 12 Sep 2000 18:24:35 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Correct. And lack of queue space should lead to a state in which all the commands are rejected until the state is explicitly restored. Consider also that in the asymmetric case with TCP in charge of the window and only small amounts of immediate data that is very unlikely to happen. Julo "John Hufferd/San Jose/IBM" on 12/09/2000 09:50:05 Please respond to "John Hufferd/San Jose/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI Costa, I think I understand your points, but please also consider the Asymmetric approach. Please consider the Asymmetric approach and also as I said in my last note, suppose that we "make a rule, that no unsolicited data can be sent, before its command is sent AND that unsolicited data must be sent in the same order as the commands that reference it. Along with that rule, and the statements below from David -- will, in your opinion, the Asymmetric approach work even with unsolicited data? . . . John L. Hufferd ---------------------- Forwarded by John Hufferd/San Jose/IBM on 09/11/2000 11:19 PM --------------------------- David Robinson @ece.cmu.edu on 09/11/2000 06:52:22 PM Please respond to David Robinson Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: Avoiding deadlock in iSCSI Thanks for your clarifying comments. > > In general > > I consider that to be a bug and the receiver should just drop the > > data on the floor. > > Not as I understand solicit. This I don't understand, it you get data that no command has been sent yet, as the reciever you can either wait and hope the command arrives consuming buffer space but eventually dropping it after some timeout, or drop it on the spot. In either case this seems like a bad design we should try to avoid. > > My first assumption is that the sender would not send commands > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > creates nasty ordering problems we want to avoid. > > Order on the wire can not be controlled. Only the ULP can avoid such. Yes it can, this is exactly the advantage of using a reliable stream protocol like TCP, the session layer never sees out of order packets. With multiple data connections and the appropriate ordering constraints we have no deadlock or buffer management issues. > Resources are held until associated data is received to complete operations. > If the resource limit is not the data buffer nor freed by content already > within the data buffer, this will result in discarding commands. But with a reliable stream no commands need to be discarded, the transport flow controls so the commands are held at the sender. > > With multiple data connections, some may flow > > control but the active command will be able to make progress on > > one connection. This may not be the most efficient mechanism but > > it is "safe". > > One connection per LUN or one connection per command, safe but expensive? Define "expensive", not in terms of performance as one TCP connection can saturate the link layer or not in terms of memory as the mux/demux state has to be held either in the transport or the session layer. I am not advocating a connection per command as that is just a bad datagram protocol, but either a connection per LUN or per target should work just fine. > As a means for freeing resources, data is to be discarded within the iSCSI > architecture. As such, even unsolicited data may be requested by the > target. I don't understand this statement. Short of target errors or connection errors, why does data need to be discarded? The sender should never send data without a command, and on a given connection the data MUST always be sent after the corresponding command and data from two commands must always be sent in order if on the same connection. -David From owner-ips@ECE.cmu.edu Tue Sep 12 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03622; 12 Sep 2000 12:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06018 for ips-outgoing; Tue, 12 Sep 2000 12:01:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA05991 for ; Tue, 12 Sep 2000 12:00:56 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA177040 for ; Tue, 12 Sep 2000 18:00:25 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA93928 for ; Tue, 12 Sep 2000 18:00:22 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256958.0057E7C7 ; Tue, 12 Sep 2000 18:00:09 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 12 Sep 2000 18:15:26 +0300 Subject: Re: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, The case you are describing is not a deadlock. If a target has no room for commands it can reject them and enter ACA. A real deadlock (i.e. target has resources but can't use them properly) can appear only if unsolicited data appear out-of-order. In this case a good target will fence-off the specific initiator (disconnect and wait for administrative action). I think that both symmetric and asymmetric models behave well against a good initiator. The only think we should make sure (and I am not sure that we are though it) is that a bad initiator can't stop forever a good initiator - or do him serious QoS harm (something that looks like a denial of service). And taking into account that initiators can be software driven desktops we should be really careful here. Julo csapuntz@cisco.com on 12/09/2000 08:28:03 Please respond to csapuntz@cisco.com To: John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu, csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: Re: Avoiding deadlock in iSCSI > I think I agree with the comment from David Robinson, who said (among > other things) "...I don't see the issue.". I'm starting to think that maybe I'm missing something here. :-) > At least I do not see the issue with a command and a data Asymmetric > Session. Perhaps all we have to do is make a rule, that no unsolicited > data can be sent, before its command is sent AND that unsolicited data must > be sent in the same order as the commands that reference it. The above is a good rule. I agree with it heartily. Immediate/unsolicited data should probably be stapled onto the SCSI command. But I do not believe it is sufficient to prevent problems. > If that rule > is followed I do not see the problem. The Data is (or should be) always > ready to be read off the data connection queue when needed by the command. Here is the scenario I proposed. This scenario applies to the iSCSI model as presented in Pittsburg, in which there is only one TCP connection (shared between commands and data). There is no unsolicited data. 1) Initiator sends command 1 (WRITE BLOCK) 2) Initiator sends command 2 (WRITE BLOCK) 3) Target reads command 1 4) Target sends RTT to initiator for data in command 1 5) Target reads from connection and notices comand 2 6) Target stops reading from connection because it does not have room for command 2 7) Initiator sends WRITE data for command 1. WRITE data is stuck behind command 2 in the stream. There are basically three points where you can solve the problem: Make sure #2 never happens (initiator never overflows a target queue) or ... Make #6 to be Target drops command 2 or ... Make #7 is Initiator sends data for command 1 on a separate dedicated, solicited-data only TCP connection. Target reads data off the dedicated, solicited-data only TCP conneciton -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03611; 12 Sep 2000 12:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA04869 for ips-outgoing; Tue, 12 Sep 2000 11:37:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA04863 for ; Tue, 12 Sep 2000 11:37:46 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDL4N6; Tue, 12 Sep 2000 08:35:36 -0700 From: "Y P Cheng" To: Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 08:36:40 -0700 Message-ID: <000501c01ccf$3ea3bce0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-reply-to: Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa Wrote: > Here is the scenario I proposed. This scenario applies to the iSCSI > model as presented in Pittsburg, in which there is only one TCP connection > (shared between commands and data). There is no unsolicited data. > > 1) Initiator sends command 1 (WRITE BLOCK) > 2) Initiator sends command 2 (WRITE BLOCK) > 3) Target reads command 1 > 4) Target sends RTT to initiator for data in command 1 > 5) Target reads from connection and notices comand 2 > 6) Target stops reading from connection because it does > not have room for command 2 > 7) Initiator sends WRITE data for command 1. WRITE data > is stuck behind command 2 in the stream. > > There are basically three points where you can solve the problem: > Thank you for the example. The step #6 does not happen in the NIC adapter or iSCSI driver I was referring to. When the adapter or driver send out a SCSI command, it already has a table entry created to serve the RTT. This is because the interface between the SCSI-class driver and the iSCSI driver defines an "atomic" SCSI request. This interface does not require the ULP, i.e. SCSI-class driver, to respond to the RTT with a write data call. The data transfer takes place immediately from the NIC adapter or the iSCSI driver after receiving the RTT and ahead of command 2 -- just like the asymmetric model. Your other example showing "command out order due to QUEUE-FULL status" is accurate. To avoid such problem, the initiator must adhere the QUEUE-DEPTH rule by not sending more commands to overflow the target. The fundamental assumption in my position is that the NIC adapter and iSCSI driver handle hundreds or even thousands of commands without blocking. In other words, the commands just flow through the adapter or the driver without ever being blocked. This requires the initiator not waiting for an ACK of the TCP from the target. Ultimately, the ACK will come from the status PDU or the RTT. The initiator is self regulating because its resource limits the number of commands sent. The initiator pre-allocates buffers for all incoming data. The target limits the command by giving queue-depth parameters to each initiator. Waiting for ACK on a network with long latency between two endpoints is a very expensive proposition. This is why I talked about UDP. Then, I realized that this subject was discussed before. What we need is the "reliable reception" -- a term used in VI -- provided in TCP. However, the same can also be achieved by having the NIC adapter or the iSCSI driver managing the RTT and status PDU without involving the ULP. The case of a lost PDU due to traffic jam or broken connection is also detected and handled by the same. A smooth recovery is a necessary condition for the "reliable delivery" protocol. From owner-ips@ECE.cmu.edu Tue Sep 12 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03624; 12 Sep 2000 12:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06010 for ips-outgoing; Tue, 12 Sep 2000 12:01:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA05988 for ; Tue, 12 Sep 2000 12:00:55 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA254108 for ; Tue, 12 Sep 2000 18:00:23 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA93926 for ; Tue, 12 Sep 2000 18:00:22 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256958.0057EA1F ; Tue, 12 Sep 2000 18:00:15 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 12 Sep 2000 18:19:21 +0300 Subject: Re: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Well - if the targets enters ACA - that state can be reset only explicitely and then you resend the commands in order. Julo csapuntz@cisco.com on 12/09/2000 08:32:44 Please respond to csapuntz@cisco.com To: Jim McGrath cc: ips@ece.cmu.edu, csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: Re: Avoiding deadlock in iSCSI > Note that SCSI targets, when faced with getting a command queue full, do not > stop reading from the interconnect. If more commands are received then they > respond with a QUEUE FULL status. If data is received then they receive > that data without regard for the status of the command queue (as long as it > is data for an already queued command). This eliminates the potential for > deadlock between command and data queues. Jim, I believe there is problem with the current SCSI behavior. Consider the following scenario for a host with a pathological queue of 1 1) Initiator sends command 1 (ORDERED attribute) 2) Initiator sends command 2 (ORDERED attribute) 3) Initiator sends command 3 (ORDERED attributge) 4) Target reads command 1 5) Target reads command 2 6) Target returns queue full for command 2 7) Command 1 completes 8) Target reads command 3 9) Target executes command 3 We have just violated the ordering constraints of the application by doing command 3 before command 2. > Potentially you can have a situation where multiple commands already at the > target have only part of their data transmitted, with the remainder still at > the initiator(s), and then run out of buffer space for data. If the target > uses a credit model to pace the reception of data, it can also make sure > this never happens. Unsolicited data, even for commands already queued, can > end up creating this deadlock - which is why unsolicited data systems either > have to have a tight limit on the resources it can use (e.g. low login BB > credit in Fibre Channel terms) or some sort of clean (i.e. not IO > terminating) rejection mechanism from target to initiator (like in USB). Unsolicited data is NOT a problem with the current iSCSI spec. We allow the target to always drop data and request data transfers with an RTT. -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03702; 12 Sep 2000 12:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06550 for ips-outgoing; Tue, 12 Sep 2000 12:12:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA06540 for ; Tue, 12 Sep 2000 12:11:57 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id 23A63776 for ; Tue, 12 Sep 2000 09:11:56 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id JAA14550 for ; Tue, 12 Sep 2000 09:11:55 -0700 (PDT) Received: from agilent.com ([15.3.251.54]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA48C4 for ; Tue, 12 Sep 2000 09:11:50 -0700 Message-ID: <39BE32FC.41586525@agilent.com> Date: Tue, 12 Sep 2000 06:43:25 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua Tseng wrote: > John, > > "A lot of data" is relative. The fastest TCP implementations available > today > don't come close to what will be required for iSCSI, since they peak out > at about 200Mbps. And the vast majority of applications are get much worse > performance than that out of TCP--I would guess typically about 20Mbps max. > > iSCSI will use TCP like never before. I believe FCP has flow control/ > command recovery mechanisms below SCSI. Nope. FCP does not have *any* flow control mechanisms (Fibre Channel itself does, but *not* the FCP protocol). In addition, FCP does *not* have any command recovery mechisms either. It uses the "if the command doesn't complete within x amount of time, blow it away and try again" recovery mechanism. FCP-2 (which is still under developement, and not deployed) is attempting to "recover" lost commands and data frames. And this "recovery" is only meant for devices such as tapes, and not discs. -Matt > Unlike iSCSI, FCP has the benefit of operating in a low latency environment > (<10us). To think iSCSI can do > without it and still be able to function reliably is, well iffy at best from > what I can see. > > Regads, > > Josh From owner-ips@ECE.cmu.edu Tue Sep 12 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03722; 12 Sep 2000 12:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06585 for ips-outgoing; Tue, 12 Sep 2000 12:12:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA06573 for ; Tue, 12 Sep 2000 12:12:21 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id B7DD627C for ; Tue, 12 Sep 2000 12:12:20 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id JAA14574 for ; Tue, 12 Sep 2000 09:12:19 -0700 (PDT) Received: from agilent.com ([15.3.251.54]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA48D8 for ; Tue, 12 Sep 2000 09:12:15 -0700 Message-ID: <39BE506A.FB93D71F@agilent.com> Date: Tue, 12 Sep 2000 08:48:58 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: TCP speed References: <000601c01c05$26275520$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P Cheng wrote: > Julo wrote: > >Our experience is the same. TCP is FAST. > >The only remaining trouble is memory copy from TCP buffers > >to application buffers. Unless > >handled properly this may slow you down considerably. > > The issue is not the TCP memory to memory copy speed, it is the latency time > of receiving TCP acknowledges. Between two endpoints of New York and Los > Angeles, latency is in milliseconds if not in seconds. On a one-gigabit > network, for each millisecond there are 100K of data, or 66 1.5K datagrams > being transferred. In fibre channel, there is this EE-credit, End-to-End. This is used in Fibre Channel class 1 and class 2 service, neither of which is mainstream fibre channel. Most implementations use class 3 - no EE credit. But I fail to see your point. You state how FC has EE credit, and make it sound like a good think, and then say it's a bad reason for using TCP... > > If the sending party has 10 EE credits, it can't send more than 10 > datagrams. EE-credit manages the TCP sliding window currently discussed in > iSCSI. After sending 10 datagrams, one must wait for acknowledges that may > take several hundred milliseconds to come. > > I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an > initiator is inherently acknowledged by its response from a target. > Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. > > The real question is how many datagrams can we pipe to a receiving node > without causing overrun. This issue is being addressed by the number of > receiving buffers inside the NIC adapter that moves data directly to the > buffers of an application (known as remote DMA in VI for InfiniBand, or > physical writes in 1394, or exchange handling in fibre channel.) It is a > challenge for people who design the NIC adapter hardware which must keep up > with the speed of connecting media at two gigabits per second today and 10 > gigabits next year. > > The TCP memory to memory copy speed is irrelevant if we have a NIC adapter > that can transfer data directly to buffers of application software. If we > don't have such an adapter, it is an impossible task in trying to define a > protocol regulating hundreds of SCSI target devices returning data to a > single SCSI initiator at the same time. If we slow it down by allowing only > one SCSI target to return data at a time, then, with the long latency time > between two nodes, iSCSI does not stand a chance in the world of OC-192, > gigabit Ethernet, Fibre Channel, InfiniBand, and even 1394B. -Matt From owner-ips@ECE.cmu.edu Tue Sep 12 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03747; 12 Sep 2000 13:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07187 for ips-outgoing; Tue, 12 Sep 2000 12:25:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07175 for ; Tue, 12 Sep 2000 12:25:00 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA09618; Tue, 12 Sep 2000 09:14:53 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA12432; Tue, 12 Sep 2000 09:12:33 -0700 From: "Douglas Otis" To: "John Hufferd/San Jose/IBM" , Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 09:27:22 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk John, With separate TCP connections, there is no guarantee which connection will be sending with the lowest latency. One would expect a command stream to be less, but it may be without inertia. A command given to the NIC ahead of data may arrive after the data. The arrival time of two streams sending different quantities would be difficult to predict. As such, out of sequence delivery. Would you recommend a skew buffer? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > John Hufferd/San Jose/IBM > Sent: Monday, September 11, 2000 11:50 PM > To: ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > > Costa, > I think I understand your points, but please also consider the Asymmetric > approach. Please consider the Asymmetric approach and also as I > said in my > last note, suppose that we "make a rule, that no unsolicited data can be > sent, before its command is sent AND that unsolicited data must be sent in > the same order as the commands that reference it. Along with that rule, > and the statements below from David -- will, in your opinion, the > Asymmetric approach work even with unsolicited data? > > . > . > . > John L. Hufferd > > ---------------------- Forwarded by John Hufferd/San Jose/IBM on > 09/11/2000 > 11:19 PM --------------------------- > > David Robinson @ece.cmu.edu on 09/11/2000 > 06:52:22 PM > > Please respond to David Robinson > > Sent by: owner-ips@ece.cmu.edu > > > To: ips@ece.cmu.edu > cc: > Subject: RE: Avoiding deadlock in iSCSI > > > > Thanks for your clarifying comments. > > > > In general > > > I consider that to be a bug and the receiver should just drop the > > > data on the floor. > > > > Not as I understand solicit. > > This I don't understand, it you get data that no command has been > sent yet, as the reciever you can either wait and hope the > command arrives consuming buffer space but eventually dropping > it after some timeout, or drop it on the spot. In either case this > seems like a bad design we should try to avoid. > > > > My first assumption is that the sender would not send commands > > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > > creates nasty ordering problems we want to avoid. > > > > Order on the wire can not be controlled. Only the ULP can avoid such. > > Yes it can, this is exactly the advantage of using a reliable stream > protocol like TCP, the session layer never sees out of order packets. > With multiple data connections and the appropriate ordering constraints > we have no deadlock or buffer management issues. > > > Resources are held until associated data is received to complete > operations. > > If the resource limit is not the data buffer nor freed by > content already > > within the data buffer, this will result in discarding commands. > > But with a reliable stream no commands need to be discarded, the > transport flow controls so the commands are held at the sender. > > > > With multiple data connections, some may flow > > > control but the active command will be able to make progress on > > > one connection. This may not be the most efficient mechanism but > > > it is "safe". > > > > One connection per LUN or one connection per command, safe but > expensive? > > Define "expensive", not in terms of performance as one TCP > connection can saturate the link layer or not in terms of memory > as the mux/demux state has to be held either in the transport or > the session layer. I am not advocating a connection per command > as that is just a bad datagram protocol, but either a connection per > LUN or per target should work just fine. > > > As a means for freeing resources, data is to be discarded within the > iSCSI > > architecture. As such, even unsolicited data may be requested by the > > target. > > I don't understand this statement. Short of target errors or connection > errors, why does data need to be discarded? The sender should never > send data without a command, and on a given connection the data MUST > always be sent after the corresponding command and data from two > commands must always be sent in order if on the same connection. > > -David > > > > From owner-ips@ECE.cmu.edu Tue Sep 12 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03895; 12 Sep 2000 13:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07524 for ips-outgoing; Tue, 12 Sep 2000 12:33:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07519 for ; Tue, 12 Sep 2000 12:33:31 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id JAA29291 for ; Tue, 12 Sep 2000 09:33:29 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Tue, 12 Sep 2000 09:33:26 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1A5@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 09:33:24 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] wrote: > The case you are describing is not a deadlock. If a target > has no room for > commands it > can reject them and enter ACA. IIRC, the SAM-2 spec doesn't allow a target to establish an ACA for TASK_SET_FULL or BUSY status. Are you suggesting that a command-queue-full condition should result in a CHECK_CONDITION? Regards, -Steve From owner-ips@ECE.cmu.edu Tue Sep 12 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03914; 12 Sep 2000 13:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07191 for ips-outgoing; Tue, 12 Sep 2000 12:25:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07179 for ; Tue, 12 Sep 2000 12:25:02 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 12 Sep 2000 12:24:31 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F71@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI: Partial Session Consensus Date: Tue, 12 Sep 2000 12:24:25 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk In reading the mailing list traffic on Asymmetric vs. Symmetric models for multiple TCP connections in an iSCSI session, I don't believe that there is consensus on the issue phrased in that fashion. I believe that there is consensus on a couple of underlying issues: (1) An iSCSI session containing a single TCP connection should not be required to use the currently specified iSCSI command reference numbers and sliding window mechanism because TCP will deliver commands in order. (2) Use of more than one TCP connection per iSCSI session is OPTIONAL. The consensus on (1) is rough; if anyone disagrees with this for a reason other than wanting to use the Symmetric model when an iSCSI session contains multiple TCP connections, please say so on the list. My reading of consensus is based on the fact that much of the recent discussion of the Asymmetric model has been motivated by single connection sessions, whereas essentially all of the recent discussion of the Symmetric model seems to have been focussed on multiple connection sessions. The consensus on (2) is sufficiently long standing to be a closed issue. The resolution to the "deadlock" scenarios posted by Costa is that the target must not stop reading from the TCP connection - SCSI provides means for a target to throw away things it can't deal with (e.g., TASK SET FULL), and with regard to the ordered command issues, I believe Steve Byan is correct when he says: I think this is a T10 bug, and iSCSI should defer to T10 to fix it. In addition to Steve noting that disks tend not to use ordered commands, I would note that a single tape device/drive tends not to have multiple simultaneous initiators sending commands to it, both of which limit the practical impact of this potential problem. I don't see consensus on the model for iSCSI sessions that contain multiple TCP connections. The consensus on (1) above for single TCP connection sessions does not take the Symmetric model out of consideration for multiple TCP connection sessions. If multiple sessions are negotiated on connection establishment, command reference numbers could be added to subsequent headers as a result of that successful negotiation. In order to make progress, this additional complexity should not be used as an argument against the consensus in (1) for single connection sessions. I hope the consensus on (1), or something close to it holds, as if it does not, we may have to form an offline design team or teams to work on this set of session issues, and that could take some time ... meanwhile, discussion of Asymmetric vs. Symmetric for multiple connection sessions should continue. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Sep 12 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04102; 12 Sep 2000 14:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09225 for ips-outgoing; Tue, 12 Sep 2000 13:13:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA09215 for ; Tue, 12 Sep 2000 13:13:12 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDL4SV; Tue, 12 Sep 2000 10:11:06 -0700 From: "Y P Cheng" To: "David Robinson" , Subject: RE: a vote for asymmetric connections in a session Date: Tue, 12 Sep 2000 10:12:08 -0700 Message-ID: <000b01c01cdc$954b5960$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-reply-to: <200009111827.LAA04921@ha10nwk.EBay.Sun.COM> Sender: owner-ips@ece.cmu.edu Precedence: bulk >From David Robinson > Sent: Monday, September 11, 2000 11:27 AM > Having worked on NFS over UDP on high latency lossy links (old ARPAnet) > the allure of saving a transport level ACK by leveraging the > fact that the session layer is datagram-like is fatally flawed > in practice. Works great on a LAN, but when faced with packet > loss due to congestion the session layer is required to maintain > adaptive timers, track round trip times, and base retransmissions > primarily on timeouts. The dropped packet creates huge latency > bubbles while waiting for timeouts, overly aggressive retransmissions > further clog congestion, all of these problems are solveable but the > result looks remarkably like TCP. In fact this is what we did do to > NFS over UDP, implementing the various algorithms found in TCP, albeit > poorly. Today, most NFS clients speak TCP by default with a minor > penalty for local servers but a major win for distant servers. I admit that simply stating UDP will replace TCP in iSCSI was a mistake. What I tried to point out was the long latency of the ACK's of the TCP between two endpoints on a WAN can be very costly. The SCSI protocol by having the status PDU will ensure "reliable reception." With or without ACK, a new NIC adapter has taken on a lot of TCP tasks such as reordering, retransmission, and congestion flow control. (See my other posting on how the RTT and data transfer are handled like having an asymmetric queue.) In fact, the adapters with VI implementation can easily perform RDMA for TCP, hence, eliminating the TCP memory-to-memory copy overhead. Without a NIC card, we can implement equivalent functions in the iSCSI driver. I hope in defining the iSCSI protocol we should consider the power of the new NIC adapters without worrying too much on the old programming model of TCP queuing and stack processing. From owner-ips@ECE.cmu.edu Tue Sep 12 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04149; 12 Sep 2000 14:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09822 for ips-outgoing; Tue, 12 Sep 2000 13:24:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA09804 for ; Tue, 12 Sep 2000 13:24:37 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id KAA10387 for ; Tue, 12 Sep 2000 10:24:20 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Tue, 12 Sep 2000 10:24:17 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1A6@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 10:24:13 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter Johansson [mailto:PJohansson@ACM.org] wrote: > The "this" Stephen refers to, summarized below, is not a T10 bug. [snip] > The malfunction is a property of the interaction of the > transport protocol > (T10 terminology) with SAM. In the case of iSCSI, it has to > be designed so > that no new commands can be accepted after the QUEUE FULL > condition until > the target has confirmed knowledge that the initiator is aware of the > error. That is, the ordered pipeline of commands between the > two endpoints > must be resynchronized before work may resume. Peter, thanks for the correction. Would someone knowlegeable in Fibre Channel please explain how FCP meets this requirement? Thanks in advance. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Tue Sep 12 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04164; 12 Sep 2000 14:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA08842 for ips-outgoing; Tue, 12 Sep 2000 13:04:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tnt.isi.edu (tnt.isi.edu [128.9.128.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA08831 for ; Tue, 12 Sep 2000 13:03:53 -0400 (EDT) Received: from moo.isi.edu (moo.isi.edu [128.9.160.187]) by tnt.isi.edu (8.8.7/8.8.6) with ESMTP id KAA00796; Tue, 12 Sep 2000 10:03:48 -0700 (PDT) Received: (from finn@localhost) by moo.isi.edu (8.9.3/8.8.6) id KAA12959; Tue, 12 Sep 2000 10:03:48 -0700 (PDT) Date: Tue, 12 Sep 2000 10:03:48 -0700 (PDT) Message-Id: <200009121703.KAA12959@moo.isi.edu> From: Greg Finn To: chengann@singnet.com.sg CC: ips@ece.cmu.edu In-reply-to: <39BDCB1E.4366F10D@singnet.com.sg> (message from cheng-ann tan on Tue, 12 Sep 2000 14:20:14 +0800) Subject: Re: patent question Sender: owner-ips@ece.cmu.edu Precedence: bulk Patents are often issued that are in error. Does it fall afoul of prior art? What is the date on the patent? IP/SCSI was prototyped and published pre-IETF years ago. Talk to Rodney Van Meter regarding dates. If the patent is more general than just IP, then other link-layer protocol work encoding SCSI would equally apply as prior art. For those dates talk to others on this list. From owner-ips@ECE.cmu.edu Tue Sep 12 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04147; 12 Sep 2000 14:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA08102 for ips-outgoing; Tue, 12 Sep 2000 12:45:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA08065 for ; Tue, 12 Sep 2000 12:45:24 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G0S007YI8GP20@mta5.snfc21.pbi.net> for IPS@ECE.CMU.edu; Tue, 12 Sep 2000 09:29:14 -0700 (PDT) Date: Tue, 12 Sep 2000 09:29:15 -0700 From: Peter Johansson Subject: RE: Avoiding deadlock in iSCSI In-reply-to: <8133266FE373D11190CD00805FA768BF055BD1A1@shrcmsg1.tdh.qntm.com> X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000912092424.00bb5350@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:25 AM 9/12/00, Stephen Byan wrote: >I think this is a T10 bug, and iSCSI should defer to T10 to fix it. The "this" Stephen refers to, summarized below, is not a T10 bug. >1) Initiator sends command 1 (ORDERED attribute) >2) Initiator sends command 2 (ORDERED attribute) >3) Initiator sends command 3 (ORDERED attributge) >4) Target reads command 1 >5) Target reads command 2 >6) Target returns queue full for command 2 >7) Command 1 completes >8) Target reads command 3 >9) Target executes command 3 > >We have just violated the ordering constraints of the application by doing >command 3 before command 2. The malfunction is a property of the interaction of the transport protocol (T10 terminology) with SAM. In the case of iSCSI, it has to be designed so that no new commands can be accepted after the QUEUE FULL condition until the target has confirmed knowledge that the initiator is aware of the error. That is, the ordered pipeline of commands between the two endpoints must be resynchronized before work may resume. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Tue Sep 12 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04142; 12 Sep 2000 14:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09833 for ips-outgoing; Tue, 12 Sep 2000 13:24:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA09814 for ; Tue, 12 Sep 2000 13:24:42 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA09676; Tue, 12 Sep 2000 10:14:34 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA12626; Tue, 12 Sep 2000 10:12:13 -0700 From: "Douglas Otis" To: "David Robinson" , Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 10:27:03 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <200009120152.SAA18183@ha10nwk.EBay.Sun.COM> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > David Robinson > Sent: Monday, September 11, 2000 6:52 PM > To: ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > > Thanks for your clarifying comments. > > > > In general > > > I consider that to be a bug and the receiver should just drop the > > > data on the floor. > > > > Not as I understand solicit. > > This I don't understand, it you get data that no command has been > sent yet, as the reciever you can either wait and hope the > command arrives consuming buffer space but eventually dropping > it after some timeout, or drop it on the spot. In either case this > seems like a bad design we should try to avoid. As you can not predict latency on two TCP connections, there would be no assurance of delivery order. A skew buffer would seem a requirement. > > > My first assumption is that the sender would not send commands > > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > > creates nasty ordering problems we want to avoid. > > > > Order on the wire can not be controlled. Only the ULP can avoid such. > > Yes it can, this is exactly the advantage of using a reliable stream > protocol like TCP, the session layer never sees out of order packets. > With multiple data connections and the appropriate ordering constraints > we have no deadlock or buffer management issues. Each TCP stream can deliver sequential data, but not within multiple streams. As such, sequential delivery goes out the window with respect to aggregation. As far as the wire, TCP is part of the Upper Layer Protocol. > > Resources are held until associated data is received to > complete operations. > > If the resource limit is not the data buffer nor freed by > content already > > within the data buffer, this will result in discarding commands. > > But with a reliable stream no commands need to be discarded, the > transport flow controls so the commands are held at the sender. The balance between commands and data are not within the control of the target via flow-control. As such, either a data or command resource may become exhausted. At some point, either data or commands may be stopped without necessarily stopping TCP. The means for stopping a command is Check Condition, and for data, discarding. > > > > With multiple data connections, some may flow > > > control but the active command will be able to make progress on > > > one connection. This may not be the most efficient mechanism but > > > it is "safe". > > > > One connection per LUN or one connection per command, safe but > expensive? > > Define "expensive", not in terms of performance as one TCP > connection can saturate the link layer or not in terms of memory > as the mux/demux state has to be held either in the transport or > the session layer. I am not advocating a connection per command > as that is just a bad datagram protocol, but either a connection per > LUN or per target should work just fine. Should there be a TCP connection per LUN, the number of TCP connections would be large if a controller is sitting on 48 LUNs. With asymmetrical connections that would imply 96 TCP sessions per client possible plus fail over connections. On the network, TCP shares on a session basis. This would mean a device with a single connection on the same network would then enjoy only 1% of the bandwidth. This says nothing about TCP overhead. > > As a means for freeing resources, data is to be discarded > within the iSCSI > > architecture. As such, even unsolicited data may be requested by the > > target. > > I don't understand this statement. Short of target errors or connection > errors, why does data need to be discarded? The sender should never > send data without a command, and on a given connection the data MUST > always be sent after the corresponding command and data from two > commands must always be sent in order if on the same connection. The iSCSI means of limiting the amount of data presented in an unsolicited fashion is to discard. For a given amount of buffer space, the number of commands associated with this space is unknown. The overhead for staging these commands would add an additional overhead on a command basis not a data basis. As such, it would be like adding water to a box of rice. Within a margin that stays out of trouble, how much of the buffer would you be wasting to handle all situations? What if you oops due to the latency of responding? Doug From owner-ips@ECE.cmu.edu Tue Sep 12 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04162; 12 Sep 2000 14:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09550 for ips-outgoing; Tue, 12 Sep 2000 13:20:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA09535 for ; Tue, 12 Sep 2000 13:20:03 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA06623 for ; Tue, 12 Sep 2000 10:20:02 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.189.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id KAA17569 for ; Tue, 12 Sep 2000 10:20:01 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id KAA01367; Tue, 12 Sep 2000 10:20:01 -0700 (PDT) Message-Id: <200009121720.KAA01367@ha10nwk.EBay.Sun.COM> Date: Tue, 12 Sep 2000 10:20:01 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: fVuhv7cDNt3rlhUl+lkWFg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > With separate TCP connections, there is no guarantee which connection will > be sending with the lowest latency. One would expect a command stream to be > less, but it may be without inertia. A command given to the NIC ahead of > data may arrive after the data. The arrival time of two streams sending > different quantities would be difficult to predict. As such, out of > sequence delivery. Would you recommend a skew buffer? This is not an issue. If you preserve the ordering of data to match the ordering of the commands things work. You will need to buffer data from the data connections until the commands are read and processed. A target that opens N connections without N*max_window_size buffers available is poorly configured so buffering should not be an issue. -David From owner-ips@ECE.cmu.edu Tue Sep 12 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04324; 12 Sep 2000 14:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10928 for ips-outgoing; Tue, 12 Sep 2000 13:48:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from crufty.research.bell-labs.com (crufty.research.bell-labs.com [204.178.16.49]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id NAA10923 for ; Tue, 12 Sep 2000 13:48:04 -0400 (EDT) Received: from grubby.research.bell-labs.com ([135.104.2.9]) by crufty; Tue Sep 12 13:47:33 EDT 2000 Received: from zydeco.research.bell-labs.com ([135.104.120.150]) by grubby; Tue Sep 12 13:47:32 EDT 2000 Received: (from jkf@localhost) by zydeco.research.bell-labs.com (8.9.1/8.9.1) id NAA05786; Tue, 12 Sep 2000 13:47:32 -0400 (EDT) Date: Tue, 12 Sep 2000 13:47:32 -0400 (EDT) From: Jeff Fellin Message-Id: <200009121747.NAA05786@zydeco.research.bell-labs.com> To: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI Cc: hufferd@us.ibm.com X-Sun-Charset: US-ASCII Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with John's conclusion. Julian's proposed rules for connections deals eith the deadlock issues raised by Costa, and should be enough for concensus on this issue. Jeff Fellin Bell Labs > Subject: Avoiding deadlock in iSCSI > To: > From: "John Hufferd/San Jose/IBM" > Date: Tue, 12 Sep 2000 09:02:06 -0700 > Sender: owner-ips@ece.cmu.edu > > Folks, > We seem to be going around and around on this. > > Julian, David Robinson, and myself (I hope among others) seem to think that > with Asymmetric, and a simple ordering rule, everything works with TCP/IP > (because of what TCP/IP itself does). There maybe some buffer management > in the SCSI "application" (this is the normal stuff that SCSI does), but it > is not an iSCSI/TCP issue. > > Can we agree on that for the Asymmetric case? > > For the Symmetric case, Julian suggested a few more "minor" rules, and he > claims that then it can work as deadlock free as the above Asymmetric case. > > Can we agree on that for the Symmetric case? > > . > . > . > John L. Hufferd > > From owner-ips@ECE.cmu.edu Tue Sep 12 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04367; 12 Sep 2000 14:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA11926 for ips-outgoing; Tue, 12 Sep 2000 14:07:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA11922 for ; Tue, 12 Sep 2000 14:07:51 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA26023; Tue, 12 Sep 2000 11:07:50 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.151.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA00633; Tue, 12 Sep 2000 11:07:49 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA29970; Tue, 12 Sep 2000 11:07:49 -0700 (PDT) Message-Id: <200009121807.LAA29970@ha10nwk.EBay.Sun.COM> Date: Tue, 12 Sep 2000 11:07:49 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu, dotis@sanlight.net MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: L/qscbYNM9fSHC5HNBo6Ig== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > As you can not predict latency on two TCP connections, there would be no > assurance of delivery order. A skew buffer would seem a requirement. If I understand your definition of a "skew buffer", some place to hold the data until the command arrives. This buffer is simply the TCP receive buffer if we require data to be sent on order. Any TCP implementation that offers a total window size (sum of windows on all connections) greater than available buffer space is just plain broken, so this is not an issue. > Each TCP stream can deliver sequential data, but not within multiple > streams. As such, sequential delivery goes out the window with respect to > aggregation. As far as the wire, TCP is part of the Upper Layer Protocol. Yes each connection maintains data ordering, and if we also require the all data is sent in order, the early arrival on one connection or another does not change the ordering. The receiver simply maintains the data buffer and closes the TCP window if flow control is needed until the command arrives, on the command connection, to be processed. Again not an issue. > The balance between commands and data are not within the control of the > target via flow-control. As such, either a data or command resource may > become exhausted. At some point, either data or commands may be stopped > without necessarily stopping TCP. The means for stopping a command is Check > Condition, and for data, discarding. With multiple data connections, any well implemented target will not open the total window size greater than the available buffers. Once a given connection uses up its window it is flow controlled by TCP and no more data is sent. Because the commands are on a seperate connection and we require data ordering, the commands will make progress either using immediate data, or reading data from a data connection that is guarenteed to be available or will be available. Commands should never need to be stopped by iSCSI and neither should the data connections, TCP flow control should be sufficient. There may be rare exceptional conditions where an initiator or target fails to follow the protocol (bug), drops a connection, or other rare event. The the recovery mechanism should simply handle this. > Should there be a TCP connection per LUN, the number of TCP connections > would be large if a controller is sitting on 48 LUNs. With asymmetrical > connections that would imply 96 TCP sessions per client possible plus fail > over connections. On the network, TCP shares on a session basis. This > would mean a device with a single connection on the same network would then > enjoy only 1% of the bandwidth. This says nothing about TCP overhead. Although I like a connection per LUN, the WG seems to be in concensus that the protocol allows multiple LUNs per connection which if you choose can implement just one. But there is no requirement to restrict it to just one. With one connection per LUN there is very little reason to have a seperate data connection and instead it is best to simply provide all the data immediately after the command. Having seperate data connections allows better concurrency when commands are for many LUNs, the concurrency for a single LUN is like to be low, if not sequential in the common case, so extra data connections are not necessary. With either a connection per LUN or all LUNs on one connection, if a connection can consume 100% of the bandwidth of the link, with 100 LUNs each will only get 1% of the bandwidth. The only difference is if the multiplexing is done at the TCP layer or the iSCSI layer, the complexity and overhead is the same (modulo a small constant). The only difference is that TCP already has the mux/demux capability and we will have to add it to iSCSI. I prefer the simplicity of leaving it to TCP, but lost the battle, but it is simply not a big enough issue to worry about. > The iSCSI means of limiting the amount of data presented in an unsolicited > fashion is to discard. For a given amount of buffer space, the number of > commands associated with this space is unknown. The overhead for staging > these commands would add an additional overhead on a command basis not a > data basis. As such, it would be like adding water to a box of rice. Within > a margin that stays out of trouble, how much of the buffer would you be > wasting to handle all situations? What if you oops due to the latency of > responding? Unlike a datagram protocol where you accept either all of the data or none of it, a reliable stream allows you to accept as much or as little as the receiver desires. You never overflow because you never advertise a TCP window larger than your available buffer space. With N connections you never allow any one connection to consume more than 1/N of the buffer space. You may waste buffer space but memory is cheap and you can control how many connections you allow. Latency is not an issue, if data arrives before a command it is simply flow controlled, in a really bizarre implementation with minimal buffering you could hold all data TCP windows to zero until a command arrives that indicates its data is on that connection. I perverse method of implementing RTT/R2T but it shows that the scheme works in the extreme. -David From owner-ips@ECE.cmu.edu Tue Sep 12 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04433; 12 Sep 2000 15:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA12913 for ips-outgoing; Tue, 12 Sep 2000 14:27:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from manganese.network-alchemy.com (NAT-10-net.Network-Alchemy.COM [199.46.17.205]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA12905 for ; Tue, 12 Sep 2000 14:27:20 -0400 (EDT) Received: from manganese.network-alchemy.com (localhost [127.0.0.1]) by manganese.network-alchemy.com (8.9.3/8.8.8) with ESMTP id LAA16954; Tue, 12 Sep 2000 11:27:50 -0700 (PDT) (envelope-from rdv@manganese.network-alchemy.com) Message-Id: <200009121827.LAA16954@manganese.network-alchemy.com> X-Mailer: exmh version 2.0.2 2/24/98 To: Greg Finn cc: chengann@singnet.com.sg, ips@ece.cmu.edu Subject: Re: patent question In-reply-to: finn's message of Tue, 12 Sep 2000 10:03:48 -0700. <200009121703.KAA12959@moo.isi.edu> Reply-To: rdv@Network-Alchemy.COM Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 12 Sep 2000 11:27:50 -0700 From: Rod Van Meter Sender: owner-ips@ece.cmu.edu Precedence: bulk It's worth looking up the patent at www.patents.ibm.com. Steve Blumenau might even be lurking on this list; comments, Steve? My reading of this (and certainly I'm no lawyer) is that it's only about having two computers that already have SCSI devices attached to them pass SCSI commands between them to be proxied to the attached devices. It has nothing to do with having the devices themselves speak SCSI over a network protocol. It does appear to intend to apply to any network protocol, not simply IP. Mentions ethernet a lot. So, as I see it, it's possible that this could affect host implementations, but doesn't seem at all relevant to device-side implementations of iSCSI. Seems to me that it's pretty obvious once you have a device that speaks SCSI over a network that the host has to be able to package up the commands to send to it. I suspect it's only EMC's remote mirroring that's truly defended by this patent. The patent doesn't really have a "related work" section, but I find it unlikely that Blumenau was unaware of things like SCSI-GPP and VAXclusters' MSCP, which have been discussed here before, even if he wasn't aware of the research at LLNL or ISI. The submission date on the patent, btw, is Jan. 14, 1998, and it was issued Nov. 30, 1999. --Rod Greg Finn and Cheng-Ann Tan wrote: > > Patents are often issued that are in error. Does it fall afoul of > prior art? What is the date on the patent? > > IP/SCSI was prototyped and published pre-IETF years ago. Talk to > Rodney Van Meter regarding dates. > > If the patent is more general than just IP, then other link-layer > protocol work encoding SCSI would equally apply as prior art. For > those dates talk to others on this list. > hi, all: > i did a patent search on the net and came across emc's > patent (5996024) which describes using scsi devices > over network protocols. it covers ip, of course. > > pardon my ignorance, but if a patent has already been > granted, how would this affect further work on i-scsi, > ip storage, etc? > > -cheng From owner-ips@ECE.cmu.edu Tue Sep 12 19:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04459; 12 Sep 2000 15:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14424 for ips-outgoing; Tue, 12 Sep 2000 14:55:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e23.nc.us.ibm.com (e23.nc.us.ibm.com [32.97.136.229]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14414 for ; Tue, 12 Sep 2000 14:55:40 -0400 (EDT) Received: from southrelay02.raleigh.ibm.com (southrelay02.raleigh.ibm.com [9.37.3.209]) by e23.nc.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA13858 for ; Tue, 12 Sep 2000 14:31:28 -0500 Received: from d04nms52.raleigh.ibm.com (d04nms51nms52.raleigh.ibm.com [9.67.228.55]) by southrelay02.raleigh.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id OAA51658 for ; Tue, 12 Sep 2000 14:55:38 -0400 Importance: Normal Subject: RE: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu From: "John H Dowdy/Raleigh/IBM" Date: Tue, 12 Sep 2000 14:55:37 -0400 Message-ID: X-MIMETrack: Serialize by Router on D04NMS52/04/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/12/2000 02:55:38 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David Robinson wrote: > Any TCP implementation that offers a total window size (sum of > windows on all connections) greater than available buffer space > is just plain broken, so this is not an issue. Actually, I think this is the issue. Specifically, like airlines, we would like to advertise a total window which is greater than the available buffer space. Generally, this helps "keep the most seats full" (i.e., utilize bandwidth the best). I won't argue that this is "not broken" in design. However, I have products in the field that do just that, and no one has thought them to be broken, because never have all the tcp receive buffers been "full" at the same time. Having said that, I do agree with David that when you advertise a TCP window, you are implying that you can buffer that much data. JD. From owner-ips@ECE.cmu.edu Tue Sep 12 19:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04465; 12 Sep 2000 15:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA13271 for ips-outgoing; Tue, 12 Sep 2000 14:33:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA13259 for ; Tue, 12 Sep 2000 14:33:46 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by palrel3.hp.com (Postfix) with ESMTP id 1CD71118; Tue, 12 Sep 2000 11:33:45 -0700 (PDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id LAA23968; Tue, 12 Sep 2000 11:33:43 -0700 (PDT) X-OpenMail-Hops: 1 Date: Tue, 12 Sep 2000 11:33:31 -0700 Message-Id: Subject: RE: Avoiding deadlock in iSCSI MIME-Version: 1.0 To: David.Robinson@EBay.Sun.COM, ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: David.Robinson@EBay.Sun.COM [mailto:David.Robinson@EBay.Sun.COM] > Sent: Monday, September 11, 2000 6:36 PM > To: ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > > Thanks for the information, I think part of my confusion is > the difference > in mapping SCSI from a datagram protocol onto a reliable > stream protocol. > In a datagram protocol if the data is sent without the receiver's > cooperation, the receiver's buffers may not be adequate and the > data must get discarded. Credits and RTT can be used to handle > this case. > > With a reliable stream transport like TCP, you don't get into > this situation because the receiver will never open the TCP > window beyond it's buffer capacity. For low amounts of buffering > it might not be as efficient as using RTT, but there is no > correctness or deadlock issues. Because each sender has its own > connection and own flow control they are independantly handled. > Likewise with seperate data connections, each is also flow controlled > so "unsolicted" data is not an issue. "Overflow" conditions simply > never occur. > > The only major design criteria is that the sender MUST maintain > ordering of data sent on any connection. Data Dn MUST always be > sent before data Dm where n < m. In particular, if unsolicated > data and RTT is mixed, the sender cannot send data Dm before it > has recieved an RTT for data Dn if both are to use the same > connection. If we are using multiple connections, this requirement has ramifications on the implementation on SMP systems. The initiator will be sending command on one connection, and data on another connection, which are two different actions. To ensure ordering would require taking a lock across the two actions (or otherwise ensuring that they always happen on the same CPU). This would be quite expensive actually. > > -David > > > I think people have been meaning "unsolicited data" to > really mean data sent > > to a receiver without that receiver having first indicated > that there is > > enough buffering to hold the data. For initiators acting > as receivers they > > have to verify this before they initiate the command (not > enough space for > > the whole command? Then break up the command.) For > Targets this requires > > something like a credit mechanism with RTTs being used. > > > > So there is an "unsolicited command" problem and an > "unsolicited data" > > problem. In both cases the sender creates the problem by not first > > reserving with the receiver enough resources for the commands/data. > > > > In the command case there is no SCSI mechanism to reserve > resources (QUEUE > > FULL is used to indicate overflows). Historically it has > been assumed that > > queues of commands do not overflow often in practice. In > reality initiators > > have often artificially limited the number of commands they > are willing to > > try and queue at the target in order to avoid this rejection (a loss > > opportunity in my mind). > > > > In the data case there is no "DATA QUEUE FULL" - instead, > an explicit credit > > model of some sort is used to indicate the receiver has > reserved space for > > the data (REQs in parallel SCSI, BB credits in Fibre > Channel). In this case > > the assumption was that data overflows would occur a lot otherwise. > > > > You can solve these problems by rejecting the overflow > cleanly (as SCSI does > > with commands), which is low latency and works well under > light loads. Or > > you can do credits. Credits add latency, or get you into > the problem of > > credit allocation, which can be optimized for light load > (over allocate > > credits) or heavy loads (allocate only what you have), but > not both at once. > > > > Historically, SCSI has used rejection for commands and > credits for data, > > optimized for heavy loads. But this is only a T10 given > rule, not a God > > given rule (although some of us who have served on T10 can > get that confused > > at times :-)). > > > > Hope this helps. > > > > Jim > > > > > > > > > > > > cases there are well known mechanisms to reserve the > > > > -----Original Message----- > > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > > Sent: Monday, September 11, 2000 3:35 PM > > To: ips@ece.cmu.edu > > Subject: Re: Avoiding deadlock in iSCSI > > > > > > I think in following this discussion the terminology has been > > confusing me. When I read "unsolicited data" I interpreted that > > to mean data for which no command has yet been sent. In general > > I consider that to be a bug and the receiver should just drop the > > data on the floor. The only possible scenerio where it might > > not be a bug is if a command was sent on one connection and the > > data on the data connection arrived first, thus it is unsolicited. > > My first assumption is that the sender would not send commands > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > creates nasty ordering problems we want to avoid. So if the > > receiver simply allows the data connection TCP window to shrink > > the unsolicted data will flow control to a stop until the command > > queue catches up. With multiple data connections, some may flow > > control but the active command will be able to make progress on > > one connection. This may not be the most efficient mechanism but > > it is "safe". Preferably the data will either follow the command > > on the same data/command connection or the sender will request a > > RTT (aka R2T). It is also a sender bug to request a connection > > for data transfer that it has already sent "unsolicited" data. > > > > Unless my assumptions and definitions are wrong, I don't > see the issue. > > > > -David > > > > > The problem: > > > > > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > > > interleaved fairly freely on a TCP connection. A target that stops > > > reading from a TCP connection to avoid reading more > command packets > > > also prevents itself from reading data packets. Those > data packets > > > may be criticial to making progress on the currently executing > > > command. > > > > > > Note the issue appears with one TCP connection for > control and data > > > and even appears in many of the multiple connection schemes. > > > > > > Data in iSCSI comes in two forms: > > > > > > 1) solicited - data requested by target via RTT > > > - data requested by initiator via a SCSI command > > > 2) unsolicited - data sent by initiator without having > received an > > RTT > > > > > > The analysis below assumes that unsolicited data travels > over the same > > > TCP connection as SCSI commands. Otherwise, you run the > risk of receiving > > > unsolicited data before the relevant SCSI command (thus making > > > implementations more complex). > > > > > > Four solutions: > > > > > > 1) Don't overflow the command queue (i.e. use credits) > > > - and what do you do if a misbehaving initiator overflows > > > your command queue anyway? Drop the connection? > > > > > > - requires you to reserve resources per initiator. some people > > > may want to overcommit > > > > > > 2) Allow dropping of SCSI commands when queue fills > > > - how do you clean up after a dropped SCSI command? > > > - there may be other commands in the pipeline > > > > > > One approach: On command drop, the target enters an error > > > state. While in the error state, all newly received commands > > > terminate with an error until the initiator explicitly clears > > > the error state using a "clear error state" message. > > > > > > You might think that TASK SET FULL and ACA mechanisms from SCSI > > > could be used to attack this problem. However, > TASK SET FULL > > errors > > > don't trigger ACA (in my reading of the SAM). Also, ACA is only > > > triggered by the current enabled command, not by random commands > > > entered into the task set. > > > > > > 3) Put solicited data on a dedicated TCP connection. Require that > > > unsolicited data MUST follow the command, ideally in the > same iSCSI > > > PDU > > > > > > 4) (Do it like NFS) Make all transfers from initiator to target > > > unsolicited. Make sure unsolicited data follows the command > > > immediately. > > > > > > > > > Of all the options, #1 and #4 sound the easiest to > implement. #2 is more > > > sophisticated than #1. #3 is just plain clever but that's > rarely a good > > > thing. :) #4 has large ramifications on current SCSI > target designs. > > > > > > -Costa > From owner-ips@ECE.cmu.edu Tue Sep 12 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04539; 12 Sep 2000 15:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA13673 for ips-outgoing; Tue, 12 Sep 2000 14:41:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA13017 for ; Tue, 12 Sep 2000 14:29:35 -0400 (EDT) Message-ID: <618471D08ABDD31188B6009027E5009352A0D7@apollo.pirus.com> From: "Merhar, Milan" To: "'chengann@singnet.com.sg'" , ips@ece.cmu.edu Subject: RE: patent question, Trademark too. Date: Tue, 12 Sep 2000 14:23:55 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Ah yes; it also appears that "ISCSI" has been trademarked by One World Technologies of Huntington Beach CA, doing business as Ariston Technologies. Serial Number 75-536138, filed August 13, 1998. "Computer hardware and computer software drivers that instruct hardware how to communicate with operating systems and other computer software for use on individual computers and local area networks." - milan -----Original Message----- From: cheng-ann tan [mailto:chengann@singnet.com.sg] Sent: Tuesday, September 12, 2000 2:20 AM To: ips@ece.cmu.edu Subject: patent question hi, all: i did a patent search on the net and came across emc's patent (5996024) which describes using scsi devices over network protocols. it covers ip, of course. pardon my ignorance, but if a patent has already been granted, how would this affect further work on i-scsi, ip storage, etc? -cheng From owner-ips@ECE.cmu.edu Tue Sep 12 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04563; 12 Sep 2000 15:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14578 for ips-outgoing; Tue, 12 Sep 2000 14:58:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14566 for ; Tue, 12 Sep 2000 14:58:22 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA14148; Tue, 12 Sep 2000 11:58:21 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.187.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA16703; Tue, 12 Sep 2000 11:58:20 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA29513; Tue, 12 Sep 2000 11:58:20 -0700 (PDT) Message-Id: <200009121858.LAA29513@ha10nwk.EBay.Sun.COM> Date: Tue, 12 Sep 2000 11:58:20 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Avoiding deadlock in iSCSI To: ips@ece.cmu.edu, somesh_gupta@hp.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: KZg2xIV8ffftUM4bq7qi2g== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > If we are using multiple connections, this requirement has > ramifications on the implementation on SMP systems. The initiator > will be sending command on one connection, and data on another > connection, which are two different actions. To ensure ordering > would require taking a lock across the two actions (or otherwise > ensuring that they always happen on the same CPU). This would > be quite expensive actually. I disagree. You need to maintain a current command counter and for each connection a last sent counter. As with any multithreaded application you need to use locks to protect shared data structures. For commands you just get your command number, check the command connection last sent counter, if your command is next you send it, otherwise you set a conditional variable and sleep on it, when the current sender finishes it simply wakes up any waiters. Same for data connections, with multiple data connections the sender could do a simple round robin and actually get less contention than on the command connection. -David From owner-ips@ECE.cmu.edu Tue Sep 12 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04558; 12 Sep 2000 15:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14445 for ips-outgoing; Tue, 12 Sep 2000 14:56:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel2.hp.com (cosrel2.hp.com [156.153.255.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14441 for ; Tue, 12 Sep 2000 14:56:07 -0400 (EDT) From: somesh_gupta@hp.com Received: from cupom2.cup.hp.com (cupom2.cup.hp.com [15.4.32.64]) by cosrel2.hp.com (Postfix) with ESMTP id 2183180D; Tue, 12 Sep 2000 12:55:49 -0600 (MDT) Received: from localhost (root@localhost) by cupom2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with SMTP id LAA00277; Tue, 12 Sep 2000 11:56:01 -0700 (PDT) X-OpenMail-Hops: 1 Date: Tue, 12 Sep 2000 11:55:46 -0700 Message-Id: Subject: RE: iSCSI: Partial Session Consensus MIME-Version: 1.0 To: Black_David@emc.com, ips@ece.cmu.edu Content-Type: text/plain; charset=US-ASCII; name="BDY.TXT" Content-Disposition: inline; filename="BDY.TXT" Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I still have to dig through all the messages (my mail system was broken for a day), so I don't know if there is anything that will change my mind. But let me record a preliminary objection to the (1). I think there are benefits to command reference numbers and sliding window - First, I would like to argue assuming that the current implementations of targets and initiators may not necessarily be the best way of achieving max throughput on iSCSI, considering the range of latencies that are possible. Then of course, memory has become cheaper - and protocol hardware implementations more and more complex. So what may be ideal is where we allow the initiator and target to be streaming commands, data and responses, with some means of avoiding trouble all the time. 1. A management of the command queue depth should enable better utilization of the pipeline by enabling streaming of commands without going into the recovery case all the time. iSCSI is better for this that the SCSI layer. The SCSI layer can keep posting commands and data buffers to the iSCSI adapter, and the iSCSI adapter can then manage when to send more commands. Otherwise, this information would have to bump up to the host, and the host would then post more commands to the iSCSI adapter - increasing workload and latency It also lets performance vary by amount of memory in the array etc rather than depend on some historic algorithm implemented in the initiator. 2. Numbering in general will make recover from command drops easier. As mentioned by Costa, myself and others, if the initiator drops a command, it drops all commands from then on, sends a notification to the initiator, who then sends an ack of the condition, and starts sending commands from the dropped command. The target may accompany the drop notification with a reduction in the command window. This minimizes the possibility of command queue overflow (and deadlock), while allowing as much throughput as possible based on implementation (algorithm and memory) choices. Somesh > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, September 12, 2000 9:24 AM > To: ips@ece.cmu.edu > Subject: iSCSI: Partial Session Consensus > > > In reading the mailing list traffic on Asymmetric > vs. Symmetric models for multiple TCP connections > in an iSCSI session, I don't believe that there is > consensus on the issue phrased in that fashion. > > I believe that there is consensus on a couple of > underlying issues: > > (1) An iSCSI session containing a single TCP connection > should not be required to use the currently specified > iSCSI command reference numbers and sliding window > mechanism because TCP will deliver commands in order. > (2) Use of more than one TCP connection per iSCSI session > is OPTIONAL. > > The consensus on (1) is rough; if anyone disagrees with > this for a reason other than wanting to use the Symmetric > model when an iSCSI session contains multiple TCP connections, > please say so on the list. My reading of consensus is > based on the fact that much of the recent discussion of the > Asymmetric model has been motivated by single connection > sessions, whereas essentially all of the recent discussion > of the Symmetric model seems to have been focussed on > multiple connection sessions. > > The consensus on (2) is sufficiently long standing to be a > closed issue. The resolution to the "deadlock" scenarios posted > by Costa is that the target must not stop reading from the TCP > connection - SCSI provides means for a target to throw away > things it can't deal with (e.g., TASK SET FULL), and with > regard to the ordered command issues, I believe Steve Byan > is correct when he says: > > I think this is a T10 bug, and iSCSI should defer to > T10 to fix it. > > In addition to Steve noting that disks tend not to use ordered > commands, I would note that a single tape device/drive tends > not to have multiple simultaneous initiators sending commands > to it, both of which limit the practical impact of this potential > problem. > > I don't see consensus on the model for iSCSI sessions that > contain multiple TCP connections. The consensus on (1) > above for single TCP connection sessions does not take the > Symmetric model out of consideration for multiple TCP connection > sessions. If multiple sessions are negotiated on connection > establishment, command reference numbers could be added to > subsequent headers as a result of that successful negotiation. > In order to make progress, this additional complexity should not > be used as an argument against the consensus in (1) for single > connection sessions. > > I hope the consensus on (1), or something close to it holds, > as if it does not, we may have to form an offline design team or > teams to work on this set of session issues, and that could take > some time ... meanwhile, discussion of Asymmetric vs. > Symmetric for multiple connection sessions should continue. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Tue Sep 12 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04568; 12 Sep 2000 15:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA14841 for ips-outgoing; Tue, 12 Sep 2000 15:03:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA14836 for ; Tue, 12 Sep 2000 15:02:58 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 12 Sep 2000 14:59:34 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F76@corpmx9.isus.emc.com> To: chengann@singnet.com.sg, ips@ece.cmu.edu Subject: RE: patent question Date: Tue, 12 Sep 2000 14:59:33 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk This is the first I've heard of that patent. I'm working on dealing with it so that it doesn't impede iSCSI standardization -- stay tuned. The IETF has established procedures and precedents for dealing with patents that cover material in standards track documents - see RFC 2026 and the material linked to: http://www.ietf.org/ipr.html --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: cheng-ann tan [SMTP:chengann@singnet.com.sg] > Sent: Tuesday, September 12, 2000 2:20 AM > To: ips@ece.cmu.edu > Subject: patent question > > hi, all: > > i did a patent search on the net and came across emc's > patent (5996024) which describes using scsi devices > over network protocols. it covers ip, of course. > > pardon my ignorance, but if a patent has already been > granted, how would this affect further work on i-scsi, > ip storage, etc? > > -cheng From owner-ips@ECE.cmu.edu Tue Sep 12 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04648; 12 Sep 2000 15:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15314 for ips-outgoing; Tue, 12 Sep 2000 15:12:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15300 for ; Tue, 12 Sep 2000 15:12:21 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDL4YJ; Tue, 12 Sep 2000 12:10:16 -0700 From: "Y P Cheng" To: Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 12:11:17 -0700 Message-ID: <001601c01ced$3a4b1580$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-reply-to: <8133266FE373D11190CD00805FA768BF055BD1A6@shrcmsg1.tdh.qntm.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve wrote: > Would someone knowlegeable in Fibre Channel please explain how FCP meets > this requirement? Thanks in advance. FCP does not meet this requirement. If there are multiple commands in the pipe, they are sent sequentially without waiting for the "status sequence" of a previous command. Therefore, the QUEUE-FULL status does not stop next command from coming. This problem is avoided by having the driver or the microcode of the NIC card tracking the commands sent to a target. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Tue Sep 12 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04679; 12 Sep 2000 15:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15546 for ips-outgoing; Tue, 12 Sep 2000 15:17:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15542 for ; Tue, 12 Sep 2000 15:17:20 -0400 (EDT) Received: from p7020-img-nt.cisco.com (fred-hm-dhcp3.cisco.com [171.69.128.118]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id MAA03309; Tue, 12 Sep 2000 12:17:01 -0700 (PDT) Message-Id: <5.0.0.25.2.20000912114413.0255a700@flipper.cisco.com> X-Sender: fred@flipper.cisco.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Tue, 12 Sep 2000 11:45:09 -0700 To: "Y P Cheng" From: Fred Baker Subject: RE: TCP speed Cc: , In-Reply-To: <000601c01c05$26275520$65c809c0@yp_portable.advansys.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:30 AM 9/11/00 -0700, Y P Cheng wrote: >NFS is implemented on UDP. only the kind that hoses networks. It is also able to be run over TCP, which is very nice when you are connected to a bunch of file servers remotely. Says one who knows... From owner-ips@ECE.cmu.edu Tue Sep 12 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04674; 12 Sep 2000 15:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15577 for ips-outgoing; Tue, 12 Sep 2000 15:17:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sj-msg-core-2.cisco.com (sj-msg-core-2.cisco.com [171.69.43.88]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15568 for ; Tue, 12 Sep 2000 15:17:47 -0400 (EDT) Received: from p7020-img-nt.cisco.com (fred-hm-dhcp3.cisco.com [171.69.128.118]) by sj-msg-core-2.cisco.com (8.9.3/8.9.1) with ESMTP id MAA03323; Tue, 12 Sep 2000 12:17:02 -0700 (PDT) Message-Id: <5.0.0.25.2.20000912114603.02547eb0@flipper.cisco.com> X-Sender: fred@flipper.cisco.com X-Mailer: QUALCOMM Windows Eudora Version 5.0 Date: Tue, 12 Sep 2000 11:55:07 -0700 To: "Greg Joyce" From: Fred Baker Subject: RE: TCP speed Cc: "Y P Cheng" , In-Reply-To: References: <000601c01c05$26275520$65c809c0@yp_portable.advansys.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 12:12 PM 9/11/00 -0500, Greg Joyce wrote: >Once you build that on top of UDP, you'll have something very TCP-like >without the benefit of years of tuning and refinement. I think you'd also >find that the performance was comparable. the difference will no doubt be that UDP has an optional checksum, while TCP requires its checksum to be used. If you have a hardware checksum generator, as has been suggested, mox nix. From owner-ips@ECE.cmu.edu Tue Sep 12 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05021; 12 Sep 2000 17:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA19378 for ips-outgoing; Tue, 12 Sep 2000 16:09:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA19373 for ; Tue, 12 Sep 2000 16:09:08 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id NAA09224; Tue, 12 Sep 2000 13:09:03 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Tue, 12 Sep 2000 13:09:02 -0700 Message-ID: From: Jim McGrath To: "'csapuntz@cisco.com'" , Jim McGrath Cc: ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 13:09:00 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with Stephen that this is a T10 issue. BTW, it has been discussed before in T10. The issue is that for commands ending with a CHECK CONDITION (error) status the ACA mechanism provides a very targeted means of solving this problem (use ACA for your ORDERED commands and you are fine). The QErr bit in the Control Mode page provides a similar global capability for the entire LUN. The problem is that there are other status conditions that may want to be treated in a similar manner, but which (to my knowledge) are not. The first one we looked at was RESERVATION CONFLICT. The second is QUEUE FULL. Both may create a desire to pause execution from the command queue. The current answer is simply not to get into this problem to begin with. That is, if you are sending an ORDERED command, make sure you do not send out the next command (ORDERED or UNORDERED) until you (the initiator) know that the first command has been received by the target. You clearly know if a command has been received by an explicit acknowledgement (as in parallel SCSI) or by a subsequent action (a data transfer or status transfer) that only makes sense if the command was received. You can argue that this degrades performance. For most of the interconnects considered up to now, with relatively low latency, this has not bee as much of a concern. In addition, there has been a feeling that the cases were ORDERED commands are used are so few that you either accepted the degradation (such as it is) when you use them, or you do a vendor unique approach around the problem. If this is considered to be a big issue, then I'd raise it to T10. I suggest you propose a solution that mimics the existing ACA and QErr solutions, since that will be easiest and quickest to adopt. One possibility is adding the status QUEUE FULL to the things that generate ACAs. Another is to propose a new QACA bit that can be used like ACA but for Queue Full (so users can continue to use the old ACA bit for backward compatibility). Since you either introduce a compatibility issue (the first approach) or need a bit (the second approach), the answer is not cost free, so T10 will want some justification for creating a compatibility issue or (more likely) to spend the bit. Jim -----Original Message----- From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] Sent: Monday, September 11, 2000 10:33 PM To: Jim McGrath Cc: ips@ece.cmu.edu; csapuntz@cisco.com Subject: Re: Avoiding deadlock in iSCSI > Note that SCSI targets, when faced with getting a command queue full, do not > stop reading from the interconnect. If more commands are received then they > respond with a QUEUE FULL status. If data is received then they receive > that data without regard for the status of the command queue (as long as it > is data for an already queued command). This eliminates the potential for > deadlock between command and data queues. Jim, I believe there is problem with the current SCSI behavior. Consider the following scenario for a host with a pathological queue of 1 1) Initiator sends command 1 (ORDERED attribute) 2) Initiator sends command 2 (ORDERED attribute) 3) Initiator sends command 3 (ORDERED attributge) 4) Target reads command 1 5) Target reads command 2 6) Target returns queue full for command 2 7) Command 1 completes 8) Target reads command 3 9) Target executes command 3 We have just violated the ordering constraints of the application by doing command 3 before command 2. > Potentially you can have a situation where multiple commands already at the > target have only part of their data transmitted, with the remainder still at > the initiator(s), and then run out of buffer space for data. If the target > uses a credit model to pace the reception of data, it can also make sure > this never happens. Unsolicited data, even for commands already queued, can > end up creating this deadlock - which is why unsolicited data systems either > have to have a tight limit on the resources it can use (e.g. low login BB > credit in Fibre Channel terms) or some sort of clean (i.e. not IO > terminating) rejection mechanism from target to initiator (like in USB). Unsolicited data is NOT a problem with the current iSCSI spec. We allow the target to always drop data and request data transfers with an RTT. -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 21:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05050; 12 Sep 2000 17:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA21526 for ips-outgoing; Tue, 12 Sep 2000 16:42:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA21519 for ; Tue, 12 Sep 2000 16:42:32 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 08B338EBA; Tue, 12 Sep 2000 13:39:42 -0700 (PDT) To: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 12 Sep 2000 13:39:42 -0700 In-Reply-To: Jim McGrath's message of "Tue, 12 Sep 2000 13:09:00 -0700" Message-ID: Lines: 14 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Part 1) Add a new task management command: CLEAR TASK SET FULL Part 2) Once a target returns TASK SET FULL, it must return TASK SET FULL for all non-ACA attribute commands until it receives a target reset, LUN reset, ABORT TASK SET, or CLEAR TASK SET FULL. ACA attribute commands may proceed against a target, even when the TASK SET FULL error condition is present. I'm not sure about backward compatibility. I'd appreciate feedback on that. -Costa From owner-ips@ECE.cmu.edu Tue Sep 12 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05133; 12 Sep 2000 17:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA21719 for ips-outgoing; Tue, 12 Sep 2000 16:47:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tnt.isi.edu (tnt.isi.edu [128.9.128.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA21712 for ; Tue, 12 Sep 2000 16:47:19 -0400 (EDT) Received: from moo.isi.edu (moo.isi.edu [128.9.160.187]) by tnt.isi.edu (8.8.7/8.8.6) with ESMTP id NAA18950; Tue, 12 Sep 2000 13:47:12 -0700 (PDT) Received: (from finn@localhost) by moo.isi.edu (8.9.3/8.8.6) id NAA13038; Tue, 12 Sep 2000 13:47:12 -0700 (PDT) Date: Tue, 12 Sep 2000 13:47:12 -0700 (PDT) Message-Id: <200009122047.NAA13038@moo.isi.edu> From: Greg Finn To: rdv@Network-Alchemy.COM CC: chengann@singnet.com.sg, ips@ece.cmu.edu In-reply-to: <200009121827.LAA16954@manganese.network-alchemy.com> (message from Rod Van Meter on Tue, 12 Sep 2000 11:27:50 -0700) Subject: Re: patent question Sender: owner-ips@ece.cmu.edu Precedence: bulk > The submission date on the patent, btw, is Jan. 14, 1998, and it was > issued Nov. 30, 1999. WRT that patent IP/SCSI would appear not to be an issue. The idea was openly discussed in the community in 1997 and probably much earlier. --- ggf From owner-ips@ECE.cmu.edu Tue Sep 12 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05526; 12 Sep 2000 19:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA23368 for ips-outgoing; Tue, 12 Sep 2000 17:25:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA23364 for ; Tue, 12 Sep 2000 17:25:05 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id XAA05190 for ; Tue, 12 Sep 2000 23:24:33 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id XAA26822 for ; Tue, 12 Sep 2000 23:24:34 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256958.007595BD ; Tue, 12 Sep 2000 23:24:20 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 00:07:07 +0300 Subject: Re: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk That is the way ACA works - and that is what I understood from Ralph's messages on the CA/autosense track (I was unwilling to give up on CA exactly for this type of action - until I understood that ACA is replacing my old fashioned CA). Julo csapuntz@cisco.com on 12/09/2000 23:39:42 Please respond to csapuntz@cisco.com To: ips@ece.cmu.edu cc: csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: Re: Avoiding deadlock in iSCSI Part 1) Add a new task management command: CLEAR TASK SET FULL Part 2) Once a target returns TASK SET FULL, it must return TASK SET FULL for all non-ACA attribute commands until it receives a target reset, LUN reset, ABORT TASK SET, or CLEAR TASK SET FULL. ACA attribute commands may proceed against a target, even when the TASK SET FULL error condition is present. I'm not sure about backward compatibility. I'd appreciate feedback on that. -Costa From owner-ips@ECE.cmu.edu Wed Sep 13 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05812; 12 Sep 2000 19:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA27249 for ips-outgoing; Tue, 12 Sep 2000 19:16:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA23950 for ; Tue, 12 Sep 2000 17:39:51 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by palrel1.hp.com (Postfix) with ESMTP id 26477EB8; Tue, 12 Sep 2000 14:39:50 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA22162; Tue, 12 Sep 2000 15:39:46 -0600 (MDT) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Tue, 12 Sep 2000 15:39:42 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex3)" To: "'David.Robinson@EBay.Sun.COM'" , ips@ece.cmu.edu, "GUPTA,SOMESH (HP-Cupertino,om5)" Subject: RE: Avoiding deadlock in iSCSI Date: Tue, 12 Sep 2000 15:39:40 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I agree that any of the various forms of synchronization will work (note the command numbering) Somesh -----Original Message----- From: David.Robinson@EBay.Sun.COM [mailto:David.Robinson@EBay.Sun.COM] Sent: Tuesday, September 12, 2000 11:58 AM To: ips@ece.cmu.edu; somesh_gupta@hp.com Subject: RE: Avoiding deadlock in iSCSI > If we are using multiple connections, this requirement has > ramifications on the implementation on SMP systems. The initiator > will be sending command on one connection, and data on another > connection, which are two different actions. To ensure ordering > would require taking a lock across the two actions (or otherwise > ensuring that they always happen on the same CPU). This would > be quite expensive actually. I disagree. You need to maintain a current command counter and for each connection a last sent counter. As with any multithreaded application you need to use locks to protect shared data structures. For commands you just get your command number, check the command connection last sent counter, if your command is next you send it, otherwise you set a conditional variable and sleep on it, when the current sender finishes it simply wakes up any waiters. Same for data connections, with multiple data connections the sender could do a simple round robin and actually get less contention than on the command connection. -David From owner-ips@ECE.cmu.edu Wed Sep 13 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05830; 12 Sep 2000 19:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA27161 for ips-outgoing; Tue, 12 Sep 2000 19:15:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA26703 for ; Tue, 12 Sep 2000 18:58:49 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id PAA09857; Tue, 12 Sep 2000 15:48:38 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA13575; Tue, 12 Sep 2000 15:46:17 -0700 From: "Douglas Otis" To: "Vivek Kashyap" Cc: "Raj Bhagwat" , , Subject: RE: FCIP and wandering duplicates (FCoverIP) Date: Tue, 12 Sep 2000 16:01:09 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <200009122226.PAA03330@eng4.sequent.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Vivek, If you use SCTP to encapsulate FC, as example http://search.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-00.txt which implements http://search.ietf.org/internet-drafts/draft-ietf-sigtran-sctp-13.txt and use the pending U-SCTP modification by Randall R. Stewart where only one retransmission is allowed, then on a MAN, such duplication should be avoided in all cases. http://search.ietf.org/internet-drafts/draft-xie-stewart-usctp-00.txt The present U-SCTP disables all retransmissions however it is to be changed to allow a single retransmission. This should allow a reasonable connection suitable for FC and suitable for standard IP equipment. Doug > -----Original Message----- > From: Vivek Kashyap [mailto:viv@sequent.com] > Sent: Tuesday, September 12, 2000 3:27 PM > To: Douglas Otis > Cc: Raj Bhagwat; Vivek Kashyap; ipfc@standards.gadzoox.com; > ips@ece.cmu.edu > Subject: Re: FCIP and wandering duplicates > > > Raj, > > I am, after reading your and Doug's replies, still unclear > whether FC handles > duplicates or not. It appears to me that it does not. > > E_D_TOV seems to be a time out that is set for retransmission between > end points. However, it is possible to introduce transient loops in > the fabric when a topology change (addtion of a new switch or a > link/switch going down) occurs, due to the re-calculation of paths > unless such changes are made atomic to the traffic. In such a case > there could be duplicates. > > Of course, if the FC switches are quiesced during the switch updates > then loops won't matter. Or is there a different mechanism or an > algorithm that guarantees looplessnes? > > > TTL retuning will at most reduce the duplicates but can't guarantee > elimination of duplicates. So if FC somehow guarantees duplicate > free fabrics and does not handle duplicates then it appears to me that > FCIP always will have the danger of corrupting data due to duplicates > inserted by IP. > > > Vivek > > > > > Raj, > > > > I do not understand what tuning TTL does to benefit any > situation. The path > > between locations is not fixed so such tuning may induce lost > packets and is > > unrelated to time in transit. Depending on the domain, a packet may be > > repeated by equipment that does not necessarily decrement TTL. > Although not > > a desired feature, duplication may happen if the packet > transmit status is > > made uncertain by local signaling. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Raj Bhagwat > > > Sent: Monday, September 11, 2000 12:26 PM > > > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com > > > Cc: ips@ece.cmu.edu > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > Doug, > > > > > > I understand how exactly TTL is used. That is why I stated > that TTL should > > > be set to a value larger than the number of hops between the > source and > > > destination. An IP end station may not know this and it may > choose to set > > > the TTL to the max value of 255. However, an FCIP gateway, with some > > > intelligence, can perhaps know the # of hops to other FCIP > > > gateways it talks > > > to, especially in enterprise networks. > > > > > > It is not clear to me how there can be duplicate frames > without the source > > > sending twice. Could you please explain that? > > > > > > Thanks, > > > > > > Raj. > > > > > > ----- Original Message ----- > > > From: Douglas Otis > > > To: Raj Bhagwat ; Vivek Kashyap ; > > > > > > Cc: > > > Sent: Monday, September 11, 2000 11:53 AM > > > Subject: RE: FCIP and wandering duplicates > > > > > > > > > > Raj, > > > > > > > > You can see duplications without the source sending twice. > Reducing TTL > > > > value may prevent the packet from ever arriving. The number of > > > times TTL > > > is > > > > decremented depends on the path and not time in transit. > > > > > > > > Doug > > > > > > > > > -----Original Message----- > > > > > From: Raj Bhagwat [mailto:rajb@lightsand.com] > > > > > Sent: Monday, September 11, 2000 11:31 AM > > > > > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com > > > > > Cc: ips@ece.cmu.edu > > > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > > > > > > > Doug, > > > > > > > > > > Your point on TTL is well taken. It is there to prevent a > packet from > > > > > getting stuck in a routing loop. Duplicates of unicast packets can > > > happen > > > > > when the sender retransmits a packet and both the original and the > > > > > retransmitted packets reach the destination. If the > sender does not > > > > > retransmit for an adequate amount of time, duplicates of > > > unicast packets > > > > > should not happen. This is true even when there are > routing loops or > > > > > topology changes in the network. > > > > > > > > > > As you have pointed out, there is no direct correlation between > > > > > E_D_TOV and > > > > > TTL. And that could be a problem. However, setting the TTL to a > > > relatively > > > > > smaller value (larger than the # of hops between the source and > > > > > destination > > > > > FCIP gateways) can reduce the possibility of duplicates when > > > the sender > > > of > > > > > the FC frame waits for at least E_D_TOV before retransmitting > > > the frame. > > > > > > > > > > I don't think SCTP at the FCIP gateway will solve the duplicate > > > > > problem even > > > > > with a single retry. There is no synchronization between > the FC device > > > and > > > > > the FCIP gateway for error recovery. So, the FC device > can retransmit > > > > > independent of the SCTP at the FCIP gateway. One way to solve this > > > problem > > > > > is to fine tune the timeout values on the FC side and at the > > > SCTP level > > > on > > > > > the FCIP gateway. > > > > > > > > > > Regards, > > > > > > > > > > Raj Bhagwat > > > > > LightSand Communications > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > From: Douglas Otis > > > > > To: Raj Bhagwat ; Vivek Kashyap > ; > > > > > > > > > > Cc: > > > > > Sent: Monday, September 11, 2000 10:14 AM > > > > > Subject: RE: FCIP and wandering duplicates > > > > > > > > > > > > > > > > Raj, > > > > > > > > > > > > Unless you are willing to wrap the FC frame within some > protocol, > > > > > duplicates > > > > > > can happen and this has nothing to do with TTL. TTL > > > prevents circular > > > > > paths > > > > > > from forever sending a packet within a loop. A modified > > > SCTP protocol > > > > > that > > > > > > will solve both of these problems, if restricted one > time retry. A > > > > > > Metro-Area-Network should induce about 5ms RTT delay such that > > > > > a 10ms skew > > > > > > becomes possible but is limited by the single retry. TTL buys > > > > > you nothing > > > > > > with this E_D_TOV, Error_Detect Timeout. > > > > > > > > > > > > Doug > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: owner-ips@ece.cmu.edu > > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > > Of > > > > > > > Raj Bhagwat > > > > > > > Sent: Sunday, September 10, 2000 10:57 PM > > > > > > > To: Vivek Kashyap; ipfc@standards.gadzoox.com > > > > > > > Cc: ips@ece.cmu.edu > > > > > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > > > > > > > > > > > > > Vivek, > > > > > > > > > > > > > > In a properly implemented FC-only fabric, duplicates should > > > > > not happen. > > > > > A > > > > > > > frame that enters a fabric should either be delivered > > > within E_D_TOV > > > > > > > (typically 2 sec) or be discarded. FC devices don't > retransmit a > > > frame > > > > > > > before the E_D_TOV timer expires. > > > > > > > > > > > > > > You do bring up an interesting point when two SAN islands are > > > > > > > bridged across > > > > > > > an IP network. Once an FC frame enters the IP > network, it is hard > > > > > > > to say how > > > > > > > the E_D_TOV is enforced by that part of the network. > As authors of > > > the > > > > > > > FCoverIP draft, we have given some thought to the > possibility of > > > > > > > delivering > > > > > > > duplicate frames when both the TCP stack at the FCIP > > > gateway and the > > > > > > > original FC device initiate recovery/retransmission > > > procedure in an > > > > > > > overlapping time period. This is one of the major > reasons why we > > > > > initially > > > > > > > chose to make the FCIP gateway a stateless device with no > > > > > buffering and > > > > > no > > > > > > > retransmission. > > > > > > > > > > > > > > The good news is that the {IP encapsulated} FC frame will be > > > protected > > > > > by > > > > > > > the IP TTL once it enters the IP network. The bad news is that > > > > > > > detecting the > > > > > > > E_D_TOV timeout while the frame is inside the IP > network may be > > > > > difficult. > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Raj Bhagwat > > > > > > > LightSand Communications > > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > From: Vivek Kashyap > > > > > > > To: > > > > > > > Sent: Sunday, September 10, 2000 3:17 PM > > > > > > > Subject: FCIP and wandering duplicates > > > > > > > > > > > > > > > > > > > > > > I read the FCIP document recently. It does not > address the issue > > > of > > > > > > > > duplicates. > > > > > > > > > > > > > > > > Does fibre channel protect itself from duplicates ? > I looked at > > > the > > > > > > > > header and it does not have a time to live field. Thus > > > a duplicate > > > > > > > > caught in a loop and then released at the 'right' > time could be > > > > > > > > accepted by the receiver. Even if the FC fabric path > > > algorithms do > > > > > > > > ensure loop free paths transient loops due to errors, switch > > > > > > > > misconfigurations or failures are possible. Does FC handle > > > > > such a case > > > > > ? > > > > > > > > > > > > > > > > If it does not then FCIP will exacerbate the > problem because the > > > IP > > > > > > > > network could have 'transient' loops. Such a 'wandering > > > duplicate' > > > > > > > > could be re-injected into the FC island and be accepted > > > by a newer > > > > > > > > incarnation of a FC inter-island exchange. Furthermore with > > > > > no TTL one > > > > > > > > can't really provide a timout between starting two > > > conversations. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Vivek > > > > > > > > -- > > > > > > > > Vivek Kashyap > > > > > > > > IBM-NumaQ > > > > > > > > viv@sequent.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Vivek Kashyap > Advisory Engineer > IBM-NumaQ > viv@sequent.com > vivk@us.ibm.com > 503 578 3422 (o) > From owner-ips@ECE.cmu.edu Wed Sep 13 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05823; 12 Sep 2000 19:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA27216 for ips-outgoing; Tue, 12 Sep 2000 19:16:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from gateway.sequent.com (gateway.sequent.com [192.148.1.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA25729 for ; Tue, 12 Sep 2000 18:27:21 -0400 (EDT) Received: from eng4.sequent.com (eng4.sequent.com [138.95.7.64]) by gateway.sequent.com (8.9.3/8.8.5) with ESMTP id PAA20234; Tue, 12 Sep 2000 15:26:45 -0700 (PDT) Received: (from viv@localhost) by eng4.sequent.com (8.8.5/8.8.5/token.aware-1.2) id PAA03330; Tue, 12 Sep 2000 15:26:43 -0700 (PDT) From: Vivek Kashyap Message-Id: <200009122226.PAA03330@eng4.sequent.com> Subject: Re: FCIP and wandering duplicates To: dotis@sanlight.net (Douglas Otis) Date: Tue, 12 Sep 2000 15:26:43 -0700 (PDT) Cc: rajb@lightsand.com (Raj Bhagwat), viv@sequent.com (Vivek Kashyap), ipfc@standards.gadzoox.com, ips@ece.cmu.edu In-Reply-To: from "Douglas Otis" at Sep 11, 2000 03:09:14 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Raj, I am, after reading your and Doug's replies, still unclear whether FC handles duplicates or not. It appears to me that it does not. E_D_TOV seems to be a time out that is set for retransmission between end points. However, it is possible to introduce transient loops in the fabric when a topology change (addtion of a new switch or a link/switch going down) occurs, due to the re-calculation of paths unless such changes are made atomic to the traffic. In such a case there could be duplicates. Of course, if the FC switches are quiesced during the switch updates then loops won't matter. Or is there a different mechanism or an algorithm that guarantees looplessnes? TTL retuning will at most reduce the duplicates but can't guarantee elimination of duplicates. So if FC somehow guarantees duplicate free fabrics and does not handle duplicates then it appears to me that FCIP always will have the danger of corrupting data due to duplicates inserted by IP. Vivek > > Raj, > > I do not understand what tuning TTL does to benefit any situation. The path > between locations is not fixed so such tuning may induce lost packets and is > unrelated to time in transit. Depending on the domain, a packet may be > repeated by equipment that does not necessarily decrement TTL. Although not > a desired feature, duplication may happen if the packet transmit status is > made uncertain by local signaling. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Raj Bhagwat > > Sent: Monday, September 11, 2000 12:26 PM > > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com > > Cc: ips@ece.cmu.edu > > Subject: Re: FCIP and wandering duplicates > > > > > > Doug, > > > > I understand how exactly TTL is used. That is why I stated that TTL should > > be set to a value larger than the number of hops between the source and > > destination. An IP end station may not know this and it may choose to set > > the TTL to the max value of 255. However, an FCIP gateway, with some > > intelligence, can perhaps know the # of hops to other FCIP > > gateways it talks > > to, especially in enterprise networks. > > > > It is not clear to me how there can be duplicate frames without the source > > sending twice. Could you please explain that? > > > > Thanks, > > > > Raj. > > > > ----- Original Message ----- > > From: Douglas Otis > > To: Raj Bhagwat ; Vivek Kashyap ; > > > > Cc: > > Sent: Monday, September 11, 2000 11:53 AM > > Subject: RE: FCIP and wandering duplicates > > > > > > > Raj, > > > > > > You can see duplications without the source sending twice. Reducing TTL > > > value may prevent the packet from ever arriving. The number of > > times TTL > > is > > > decremented depends on the path and not time in transit. > > > > > > Doug > > > > > > > -----Original Message----- > > > > From: Raj Bhagwat [mailto:rajb@lightsand.com] > > > > Sent: Monday, September 11, 2000 11:31 AM > > > > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com > > > > Cc: ips@ece.cmu.edu > > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > > > > Doug, > > > > > > > > Your point on TTL is well taken. It is there to prevent a packet from > > > > getting stuck in a routing loop. Duplicates of unicast packets can > > happen > > > > when the sender retransmits a packet and both the original and the > > > > retransmitted packets reach the destination. If the sender does not > > > > retransmit for an adequate amount of time, duplicates of > > unicast packets > > > > should not happen. This is true even when there are routing loops or > > > > topology changes in the network. > > > > > > > > As you have pointed out, there is no direct correlation between > > > > E_D_TOV and > > > > TTL. And that could be a problem. However, setting the TTL to a > > relatively > > > > smaller value (larger than the # of hops between the source and > > > > destination > > > > FCIP gateways) can reduce the possibility of duplicates when > > the sender > > of > > > > the FC frame waits for at least E_D_TOV before retransmitting > > the frame. > > > > > > > > I don't think SCTP at the FCIP gateway will solve the duplicate > > > > problem even > > > > with a single retry. There is no synchronization between the FC device > > and > > > > the FCIP gateway for error recovery. So, the FC device can retransmit > > > > independent of the SCTP at the FCIP gateway. One way to solve this > > problem > > > > is to fine tune the timeout values on the FC side and at the > > SCTP level > > on > > > > the FCIP gateway. > > > > > > > > Regards, > > > > > > > > Raj Bhagwat > > > > LightSand Communications > > > > > > > > > > > > ----- Original Message ----- > > > > From: Douglas Otis > > > > To: Raj Bhagwat ; Vivek Kashyap ; > > > > > > > > Cc: > > > > Sent: Monday, September 11, 2000 10:14 AM > > > > Subject: RE: FCIP and wandering duplicates > > > > > > > > > > > > > Raj, > > > > > > > > > > Unless you are willing to wrap the FC frame within some protocol, > > > > duplicates > > > > > can happen and this has nothing to do with TTL. TTL > > prevents circular > > > > paths > > > > > from forever sending a packet within a loop. A modified > > SCTP protocol > > > > that > > > > > will solve both of these problems, if restricted one time retry. A > > > > > Metro-Area-Network should induce about 5ms RTT delay such that > > > > a 10ms skew > > > > > becomes possible but is limited by the single retry. TTL buys > > > > you nothing > > > > > with this E_D_TOV, Error_Detect Timeout. > > > > > > > > > > Doug > > > > > > > > > > > -----Original Message----- > > > > > > From: owner-ips@ece.cmu.edu > > [mailto:owner-ips@ece.cmu.edu]On Behalf > > Of > > > > > > Raj Bhagwat > > > > > > Sent: Sunday, September 10, 2000 10:57 PM > > > > > > To: Vivek Kashyap; ipfc@standards.gadzoox.com > > > > > > Cc: ips@ece.cmu.edu > > > > > > Subject: Re: FCIP and wandering duplicates > > > > > > > > > > > > > > > > > > Vivek, > > > > > > > > > > > > In a properly implemented FC-only fabric, duplicates should > > > > not happen. > > > > A > > > > > > frame that enters a fabric should either be delivered > > within E_D_TOV > > > > > > (typically 2 sec) or be discarded. FC devices don't retransmit a > > frame > > > > > > before the E_D_TOV timer expires. > > > > > > > > > > > > You do bring up an interesting point when two SAN islands are > > > > > > bridged across > > > > > > an IP network. Once an FC frame enters the IP network, it is hard > > > > > > to say how > > > > > > the E_D_TOV is enforced by that part of the network. As authors of > > the > > > > > > FCoverIP draft, we have given some thought to the possibility of > > > > > > delivering > > > > > > duplicate frames when both the TCP stack at the FCIP > > gateway and the > > > > > > original FC device initiate recovery/retransmission > > procedure in an > > > > > > overlapping time period. This is one of the major reasons why we > > > > initially > > > > > > chose to make the FCIP gateway a stateless device with no > > > > buffering and > > > > no > > > > > > retransmission. > > > > > > > > > > > > The good news is that the {IP encapsulated} FC frame will be > > protected > > > > by > > > > > > the IP TTL once it enters the IP network. The bad news is that > > > > > > detecting the > > > > > > E_D_TOV timeout while the frame is inside the IP network may be > > > > difficult. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Raj Bhagwat > > > > > > LightSand Communications > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > From: Vivek Kashyap > > > > > > To: > > > > > > Sent: Sunday, September 10, 2000 3:17 PM > > > > > > Subject: FCIP and wandering duplicates > > > > > > > > > > > > > > > > > > > I read the FCIP document recently. It does not address the issue > > of > > > > > > > duplicates. > > > > > > > > > > > > > > Does fibre channel protect itself from duplicates ? I looked at > > the > > > > > > > header and it does not have a time to live field. Thus > > a duplicate > > > > > > > caught in a loop and then released at the 'right' time could be > > > > > > > accepted by the receiver. Even if the FC fabric path > > algorithms do > > > > > > > ensure loop free paths transient loops due to errors, switch > > > > > > > misconfigurations or failures are possible. Does FC handle > > > > such a case > > > > ? > > > > > > > > > > > > > > If it does not then FCIP will exacerbate the problem because the > > IP > > > > > > > network could have 'transient' loops. Such a 'wandering > > duplicate' > > > > > > > could be re-injected into the FC island and be accepted > > by a newer > > > > > > > incarnation of a FC inter-island exchange. Furthermore with > > > > no TTL one > > > > > > > can't really provide a timout between starting two > > conversations. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Vivek > > > > > > > -- > > > > > > > Vivek Kashyap > > > > > > > IBM-NumaQ > > > > > > > viv@sequent.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Vivek Kashyap Advisory Engineer IBM-NumaQ viv@sequent.com vivk@us.ibm.com 503 578 3422 (o) From owner-ips@ECE.cmu.edu Wed Sep 13 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05821; 12 Sep 2000 19:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA26805 for ips-outgoing; Tue, 12 Sep 2000 19:01:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA26797 for ; Tue, 12 Sep 2000 19:01:08 -0400 (EDT) Message-ID: <618471D08ABDD31188B6009027E500930F13D0@apollo.pirus.com> From: "Hall, Howard" To: ips@ece.cmu.edu Subject: Comments on the Draft Date: Tue, 12 Sep 2000 19:12:38 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Wed Sep 13 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05863; 12 Sep 2000 20:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA27583 for ips-outgoing; Tue, 12 Sep 2000 19:29:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA27578 for ; Tue, 12 Sep 2000 19:29:24 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id QAA17707 for ; Tue, 12 Sep 2000 16:29:23 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.189.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id QAA03233 for ; Tue, 12 Sep 2000 16:29:19 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id QAA06862; Tue, 12 Sep 2000 16:29:19 -0700 (PDT) Message-Id: <200009122329.QAA06862@ha10nwk.EBay.Sun.COM> Date: Tue, 12 Sep 2000 16:29:19 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: Comments on the Draft (sync loss) To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: +91taYWwi8xO0O1P2RTAMg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > Section: General > There's no framing of the headers and data on the buffers from TCP. If > anything goes wrong with the parsing, its difficult if not impossible to > recover. It only takes one length field to be 'off'. If this happens the > target will probably generate lots of "Opcode not understood" messages. We > suggest one of two methods: 1) after seeing consecutive "Opcode not > understood" messages it should shut down the connection if this doesn't > solve the problem then reset the target, or 2) When the target finds that > it is out of sync with the initiator ( on receipt of an "Opcode not > understood"), it will send a new iSCSI "Out of Sync" command to the > initiator. The initiator will assume at the reception of the "Out of Sync" > command that all unacknowledged outstanding requests have been dropped. The > initiator then sends the next command with the OOB (out of band) bit set, > and with the OOB offset pointing to the beginning of the iSCSI header. The > target, after sending the "Out of Sync" command, should ignore every thing > on that connection and wait for the OOB data to re-sync again. This > exchange could also work if sent from the initiator to the target. The loss of sync is an extremely rare event, either the sender sent a request shorter than it indicated, the TCP stack caused corruption, or the receiver misinterpreted the request. All are severe bugs somewhere, in practice with NFS over TCP which has no framing, sync errors are so rare that it is ignored. Instead of going through a complex process of setting OOB to resync, then determine which messages need to be resent, why not instead simply drop the connection and do the common lost connection recovery. The symptoms and state that needs to be recovered is virtually identical. Lets try to stay simple and not overly complicate the recovery process. -David From owner-ips@ECE.cmu.edu Wed Sep 13 02:10:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06477; 12 Sep 2000 22:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA00155 for ips-outgoing; Tue, 12 Sep 2000 20:53:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA00149 for ; Tue, 12 Sep 2000 20:53:41 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id UAA06701; Tue, 12 Sep 2000 20:53:33 -0400 (EDT) Date: Tue, 12 Sep 2000 20:53:33 -0400 (EDT) From: Scott Bradner Message-Id: <200009130053.UAA06701@newdev.harvard.edu> To: ips@ece.cmu.edu, julian_satran@il.ibm.com, ycheng@advansys.com Subject: RE: TCP speed Sender: owner-ips@ece.cmu.edu Precedence: bulk > I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an > initiator is inherently acknowledged by its response from a target. > Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. the ips working group can decide to do something over UDP but it should note that the result has to deal with congestion in a TCP-friendly way - I expect that the WG will find that its not an easy requirement to meet - I personally think that it would not be a good use of the WGs time to spend a good chunk of a year trying to get that right Scott (with AD hat on) From owner-ips@ECE.cmu.edu Wed Sep 13 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06980; 12 Sep 2000 23:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA03849 for ips-outgoing; Tue, 12 Sep 2000 23:07:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA03837 for ; Tue, 12 Sep 2000 23:07:29 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id WAA03474; Tue, 12 Sep 2000 22:07:53 -0500 Message-ID: <39BEEF89.3B02BBA@stewart.chicago.il.us> Date: Tue, 12 Sep 2000 22:07:53 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: David Robinson CC: ips@ece.cmu.edu, dotis@sanlight.net Subject: Re: Avoiding deadlock in iSCSI References: <200009121807.LAA29970@ha10nwk.EBay.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David: Question for you... Your reply leads me to believe a tight coupling exists between the iSCSI layer and the TCP stack. I.e. iSCSI is going in and "tweaking on a dynamic basis" the TCP implemenations rwnd. Other email that has crossed on this subject implies that "a TCP implementation is broken if it offers more buffers than it has"... This in some ways contradicts iSCSI controlling TCP rwnd and TCP controlling its own rwnd... could you please clearify this for me? Is it TCP in control of rwnd or is there a tight coupling here? Thanks R David Robinson wrote: > > > As you can not predict latency on two TCP connections, there would be no > > assurance of delivery order. A skew buffer would seem a requirement. > > If I understand your definition of a "skew buffer", some place to > hold the data until the command arrives. This buffer is simply > the TCP receive buffer if we require data to be sent on order. > Any TCP implementation that offers a total window size (sum of > windows on all connections) greater than available buffer space > is just plain broken, so this is not an issue. > > > Each TCP stream can deliver sequential data, but not within multiple > > streams. As such, sequential delivery goes out the window with respect to > > aggregation. As far as the wire, TCP is part of the Upper Layer Protocol. > > Yes each connection maintains data ordering, and if we also require > the all data is sent in order, the early arrival on one connection > or another does not change the ordering. The receiver simply maintains > the data buffer and closes the TCP window if flow control is needed > until the command arrives, on the command connection, to be processed. > Again not an issue. > > > The balance between commands and data are not within the control of the > > target via flow-control. As such, either a data or command resource may > > become exhausted. At some point, either data or commands may be stopped > > without necessarily stopping TCP. The means for stopping a command is Check > > Condition, and for data, discarding. > > With multiple data connections, any well implemented target will not > open the total window size greater than the available buffers. Once > a given connection uses up its window it is flow controlled by TCP > and no more data is sent. Because the commands are on a seperate > connection and we require data ordering, the commands will make > progress either using immediate data, or reading data from a > data connection that is guarenteed to be available or will be available. > Commands should never need to be stopped by iSCSI and neither should > the data connections, TCP flow control should be sufficient. > > There may be rare exceptional conditions where an initiator or target > fails to follow the protocol (bug), drops a connection, or other rare > event. The the recovery mechanism should simply handle this. > > > Should there be a TCP connection per LUN, the number of TCP connections > > would be large if a controller is sitting on 48 LUNs. With asymmetrical > > connections that would imply 96 TCP sessions per client possible plus fail > > over connections. On the network, TCP shares on a session basis. This > > would mean a device with a single connection on the same network would then > > enjoy only 1% of the bandwidth. This says nothing about TCP overhead. > > Although I like a connection per LUN, the WG seems to be in concensus that > the protocol allows multiple LUNs per connection which if you choose > can implement just one. But there is no requirement to restrict it > to just one. With one connection per LUN there is very little > reason to have a seperate data connection and instead it is > best to simply provide all the data immediately after the command. > Having seperate data connections allows better concurrency when > commands are for many LUNs, the concurrency for a single LUN is > like to be low, if not sequential in the common case, so extra data > connections are not necessary. > > With either a connection per LUN or all LUNs on one connection, if a > connection can consume 100% of the bandwidth of the link, with 100 > LUNs each will only get 1% of the bandwidth. The only difference is > if the multiplexing is done at the TCP layer or the iSCSI layer, > the complexity and overhead is the same (modulo a small constant). > The only difference is that TCP already has the mux/demux capability > and we will have to add it to iSCSI. I prefer the simplicity of > leaving it to TCP, but lost the battle, but it is simply not a big > enough issue to worry about. > > > The iSCSI means of limiting the amount of data presented in an unsolicited > > fashion is to discard. For a given amount of buffer space, the number of > > commands associated with this space is unknown. The overhead for staging > > these commands would add an additional overhead on a command basis not a > > data basis. As such, it would be like adding water to a box of rice. Within > > a margin that stays out of trouble, how much of the buffer would you be > > wasting to handle all situations? What if you oops due to the latency of > > responding? > > Unlike a datagram protocol where you accept either all of the data or > none of it, a reliable stream allows you to accept as much or as > little as the receiver desires. You never overflow because you > never advertise a TCP window larger than your available buffer space. > With N connections you never allow any one connection to consume > more than 1/N of the buffer space. You may waste buffer space but > memory is cheap and you can control how many connections you allow. > Latency is not an issue, if data arrives before a command it > is simply flow controlled, in a really bizarre implementation with > minimal buffering you could hold all data TCP windows to zero > until a command arrives that indicates its data is on that connection. > I perverse method of implementing RTT/R2T but it shows that the scheme > works in the extreme. > > -David > -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 13 04:00:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06982; 12 Sep 2000 23:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA03431 for ips-outgoing; Tue, 12 Sep 2000 22:53:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA03427 for ; Tue, 12 Sep 2000 22:53:28 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id VAA03445; Tue, 12 Sep 2000 21:53:51 -0500 Message-ID: <39BEEC3F.E95677C4@stewart.chicago.il.us> Date: Tue, 12 Sep 2000 21:53:51 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Scott Bradner CC: ips@ece.cmu.edu, julian_satran@il.ibm.com, ycheng@advansys.com Subject: Re: TCP speed References: <200009130053.UAA06701@newdev.harvard.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Scott Bradner wrote: > > > I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an > > initiator is inherently acknowledged by its response from a target. > > Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. > > the ips working group can decide to do something over UDP but > it should note that the result has to deal with congestion in a TCP-friendly > way - I expect that the WG will find that its not an easy requirement > to meet - I personally think that it would not be a good use of the WGs time > to spend a good chunk of a year trying to get that right > > Scott (with AD hat on) If one were to go down this path (foolishly I might add), I think one year would not be enough time.. two would be closer to the mark.. It is quite a lot of work, it can be both fun and educating, but I do think that IPS has a lot of alternatives besides treading these paths :) R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 13 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07964; 13 Sep 2000 3:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA08916 for ips-outgoing; Wed, 13 Sep 2000 02:27:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA08912 for ; Wed, 13 Sep 2000 02:27:52 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA317910 for ; Wed, 13 Sep 2000 08:27:21 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA86254 for ; Wed, 13 Sep 2000 08:27:20 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.002375A6 ; Wed, 13 Sep 2000 08:27:18 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 08:42:43 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Where would you suggest it should be done without violating layering constraints? Julo Peter Johansson on 12/09/2000 19:29:15 Please respond to Peter Johansson To: IP Storage cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI At 07:25 AM 9/12/00, Stephen Byan wrote: >I think this is a T10 bug, and iSCSI should defer to T10 to fix it. The "this" Stephen refers to, summarized below, is not a T10 bug. >1) Initiator sends command 1 (ORDERED attribute) >2) Initiator sends command 2 (ORDERED attribute) >3) Initiator sends command 3 (ORDERED attributge) >4) Target reads command 1 >5) Target reads command 2 >6) Target returns queue full for command 2 >7) Command 1 completes >8) Target reads command 3 >9) Target executes command 3 > >We have just violated the ordering constraints of the application by doing >command 3 before command 2. The malfunction is a property of the interaction of the transport protocol (T10 terminology) with SAM. In the case of iSCSI, it has to be designed so that no new commands can be accepted after the QUEUE FULL condition until the target has confirmed knowledge that the initiator is aware of the error. That is, the ordered pipeline of commands between the two endpoints must be resynchronized before work may resume. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Wed Sep 13 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07989; 13 Sep 2000 3:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA08943 for ips-outgoing; Wed, 13 Sep 2000 02:28:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA08923 for ; Wed, 13 Sep 2000 02:28:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA163278 for ; Wed, 13 Sep 2000 08:27:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA63444 for ; Wed, 13 Sep 2000 08:27:28 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.002376F7 ; Wed, 13 Sep 2000 08:27:22 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 09:07:15 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Having to wait for an ack for an ordered command even if the transport guarantees ordered delivery seems pretty ugly. If T10 doesn't handle it we will need some acceptance guarantee (back to counters!). Some old (good) channels had for this an initial status (!) but not SCSI. Julo Jim McGrath on 12/09/2000 23:09:00 Please respond to Jim McGrath To: "'csapuntz@cisco.com'" , Jim McGrath cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI I agree with Stephen that this is a T10 issue. BTW, it has been discussed before in T10. The issue is that for commands ending with a CHECK CONDITION (error) status the ACA mechanism provides a very targeted means of solving this problem (use ACA for your ORDERED commands and you are fine). The QErr bit in the Control Mode page provides a similar global capability for the entire LUN. The problem is that there are other status conditions that may want to be treated in a similar manner, but which (to my knowledge) are not. The first one we looked at was RESERVATION CONFLICT. The second is QUEUE FULL. Both may create a desire to pause execution from the command queue. The current answer is simply not to get into this problem to begin with. That is, if you are sending an ORDERED command, make sure you do not send out the next command (ORDERED or UNORDERED) until you (the initiator) know that the first command has been received by the target. You clearly know if a command has been received by an explicit acknowledgement (as in parallel SCSI) or by a subsequent action (a data transfer or status transfer) that only makes sense if the command was received. You can argue that this degrades performance. For most of the interconnects considered up to now, with relatively low latency, this has not bee as much of a concern. In addition, there has been a feeling that the cases were ORDERED commands are used are so few that you either accepted the degradation (such as it is) when you use them, or you do a vendor unique approach around the problem. If this is considered to be a big issue, then I'd raise it to T10. I suggest you propose a solution that mimics the existing ACA and QErr solutions, since that will be easiest and quickest to adopt. One possibility is adding the status QUEUE FULL to the things that generate ACAs. Another is to propose a new QACA bit that can be used like ACA but for Queue Full (so users can continue to use the old ACA bit for backward compatibility). Since you either introduce a compatibility issue (the first approach) or need a bit (the second approach), the answer is not cost free, so T10 will want some justification for creating a compatibility issue or (more likely) to spend the bit. Jim -----Original Message----- From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] Sent: Monday, September 11, 2000 10:33 PM To: Jim McGrath Cc: ips@ece.cmu.edu; csapuntz@cisco.com Subject: Re: Avoiding deadlock in iSCSI > Note that SCSI targets, when faced with getting a command queue full, do not > stop reading from the interconnect. If more commands are received then they > respond with a QUEUE FULL status. If data is received then they receive > that data without regard for the status of the command queue (as long as it > is data for an already queued command). This eliminates the potential for > deadlock between command and data queues. Jim, I believe there is problem with the current SCSI behavior. Consider the following scenario for a host with a pathological queue of 1 1) Initiator sends command 1 (ORDERED attribute) 2) Initiator sends command 2 (ORDERED attribute) 3) Initiator sends command 3 (ORDERED attributge) 4) Target reads command 1 5) Target reads command 2 6) Target returns queue full for command 2 7) Command 1 completes 8) Target reads command 3 9) Target executes command 3 We have just violated the ordering constraints of the application by doing command 3 before command 2. > Potentially you can have a situation where multiple commands already at the > target have only part of their data transmitted, with the remainder still at > the initiator(s), and then run out of buffer space for data. If the target > uses a credit model to pace the reception of data, it can also make sure > this never happens. Unsolicited data, even for commands already queued, can > end up creating this deadlock - which is why unsolicited data systems either > have to have a tight limit on the resources it can use (e.g. low login BB > credit in Fibre Channel terms) or some sort of clean (i.e. not IO > terminating) rejection mechanism from target to initiator (like in USB). Unsolicited data is NOT a problem with the current iSCSI spec. We allow the target to always drop data and request data transfers with an RTT. -Costa From owner-ips@ECE.cmu.edu Wed Sep 13 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08038; 13 Sep 2000 3:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA08934 for ips-outgoing; Wed, 13 Sep 2000 02:28:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA08917 for ; Wed, 13 Sep 2000 02:27:57 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA130088 for ; Wed, 13 Sep 2000 08:27:25 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA86258 for ; Wed, 13 Sep 2000 08:27:25 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.00237527 ; Wed, 13 Sep 2000 08:27:17 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 08:25:58 +0300 Subject: d-o-s atack Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, I was wrong. A bad initiator not respecting order can do harm to himself only. A good target design will NOT drop packets as it may open itself to d-o-s attacks. Rejected commands and ACA should be handled with a long execution time to avoid attacks. Regards, Julo From owner-ips@ECE.cmu.edu Wed Sep 13 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07973; 13 Sep 2000 3:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA08944 for ips-outgoing; Wed, 13 Sep 2000 02:28:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA08931 for ; Wed, 13 Sep 2000 02:28:02 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA125952 for ; Wed, 13 Sep 2000 08:27:31 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA33356 for ; Wed, 13 Sep 2000 08:27:30 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.002375F8 ; Wed, 13 Sep 2000 08:27:19 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 08:50:21 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Skew doesn't matter. Data Order has to be maintained on ANY GIVEN CONNECTION. The rest is TCP. If an initiator does not follow order it will get stuck. Julo "Douglas Otis" on 12/09/2000 20:27:03 Please respond to "Douglas Otis" To: "David Robinson" , ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI David, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > David Robinson > Sent: Monday, September 11, 2000 6:52 PM > To: ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > > Thanks for your clarifying comments. > > > > In general > > > I consider that to be a bug and the receiver should just drop the > > > data on the floor. > > > > Not as I understand solicit. > > This I don't understand, it you get data that no command has been > sent yet, as the reciever you can either wait and hope the > command arrives consuming buffer space but eventually dropping > it after some timeout, or drop it on the spot. In either case this > seems like a bad design we should try to avoid. As you can not predict latency on two TCP connections, there would be no assurance of delivery order. A skew buffer would seem a requirement. > > > My first assumption is that the sender would not send commands > > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > > creates nasty ordering problems we want to avoid. > > > > Order on the wire can not be controlled. Only the ULP can avoid such. > > Yes it can, this is exactly the advantage of using a reliable stream > protocol like TCP, the session layer never sees out of order packets. > With multiple data connections and the appropriate ordering constraints > we have no deadlock or buffer management issues. Each TCP stream can deliver sequential data, but not within multiple streams. As such, sequential delivery goes out the window with respect to aggregation. As far as the wire, TCP is part of the Upper Layer Protocol. > > Resources are held until associated data is received to > complete operations. > > If the resource limit is not the data buffer nor freed by > content already > > within the data buffer, this will result in discarding commands. > > But with a reliable stream no commands need to be discarded, the > transport flow controls so the commands are held at the sender. The balance between commands and data are not within the control of the target via flow-control. As such, either a data or command resource may become exhausted. At some point, either data or commands may be stopped without necessarily stopping TCP. The means for stopping a command is Check Condition, and for data, discarding. > > > > With multiple data connections, some may flow > > > control but the active command will be able to make progress on > > > one connection. This may not be the most efficient mechanism but > > > it is "safe". > > > > One connection per LUN or one connection per command, safe but > expensive? > > Define "expensive", not in terms of performance as one TCP > connection can saturate the link layer or not in terms of memory > as the mux/demux state has to be held either in the transport or > the session layer. I am not advocating a connection per command > as that is just a bad datagram protocol, but either a connection per > LUN or per target should work just fine. Should there be a TCP connection per LUN, the number of TCP connections would be large if a controller is sitting on 48 LUNs. With asymmetrical connections that would imply 96 TCP sessions per client possible plus fail over connections. On the network, TCP shares on a session basis. This would mean a device with a single connection on the same network would then enjoy only 1% of the bandwidth. This says nothing about TCP overhead. > > As a means for freeing resources, data is to be discarded > within the iSCSI > > architecture. As such, even unsolicited data may be requested by the > > target. > > I don't understand this statement. Short of target errors or connection > errors, why does data need to be discarded? The sender should never > send data without a command, and on a given connection the data MUST > always be sent after the corresponding command and data from two > commands must always be sent in order if on the same connection. The iSCSI means of limiting the amount of data presented in an unsolicited fashion is to discard. For a given amount of buffer space, the number of commands associated with this space is unknown. The overhead for staging these commands would add an additional overhead on a command basis not a data basis. As such, it would be like adding water to a box of rice. Within a margin that stays out of trouble, how much of the buffer would you be wasting to handle all situations? What if you oops due to the latency of responding? Doug From owner-ips@ECE.cmu.edu Wed Sep 13 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07966; 13 Sep 2000 3:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA08949 for ips-outgoing; Wed, 13 Sep 2000 02:28:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA08924 for ; Wed, 13 Sep 2000 02:28:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA07964 for ; Wed, 13 Sep 2000 08:27:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA33350 for ; Wed, 13 Sep 2000 08:27:28 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.002376BC ; Wed, 13 Sep 2000 08:27:21 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 08:59:28 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk That is a big issue - if what I hear is true. It isn't even related to latency but with having a queue. If you have a long running command and a queue of 1 (awaiting execution) you can have this error. I was (mistakenly) confident that it is handled by ACA. Is it safe to asume that T10 will handle it? Julo Jim McGrath on 12/09/2000 23:09:00 Please respond to Jim McGrath To: "'csapuntz@cisco.com'" , Jim McGrath cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI I agree with Stephen that this is a T10 issue. BTW, it has been discussed before in T10. The issue is that for commands ending with a CHECK CONDITION (error) status the ACA mechanism provides a very targeted means of solving this problem (use ACA for your ORDERED commands and you are fine). The QErr bit in the Control Mode page provides a similar global capability for the entire LUN. The problem is that there are other status conditions that may want to be treated in a similar manner, but which (to my knowledge) are not. The first one we looked at was RESERVATION CONFLICT. The second is QUEUE FULL. Both may create a desire to pause execution from the command queue. The current answer is simply not to get into this problem to begin with. That is, if you are sending an ORDERED command, make sure you do not send out the next command (ORDERED or UNORDERED) until you (the initiator) know that the first command has been received by the target. You clearly know if a command has been received by an explicit acknowledgement (as in parallel SCSI) or by a subsequent action (a data transfer or status transfer) that only makes sense if the command was received. You can argue that this degrades performance. For most of the interconnects considered up to now, with relatively low latency, this has not bee as much of a concern. In addition, there has been a feeling that the cases were ORDERED commands are used are so few that you either accepted the degradation (such as it is) when you use them, or you do a vendor unique approach around the problem. If this is considered to be a big issue, then I'd raise it to T10. I suggest you propose a solution that mimics the existing ACA and QErr solutions, since that will be easiest and quickest to adopt. One possibility is adding the status QUEUE FULL to the things that generate ACAs. Another is to propose a new QACA bit that can be used like ACA but for Queue Full (so users can continue to use the old ACA bit for backward compatibility). Since you either introduce a compatibility issue (the first approach) or need a bit (the second approach), the answer is not cost free, so T10 will want some justification for creating a compatibility issue or (more likely) to spend the bit. Jim -----Original Message----- From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] Sent: Monday, September 11, 2000 10:33 PM To: Jim McGrath Cc: ips@ece.cmu.edu; csapuntz@cisco.com Subject: Re: Avoiding deadlock in iSCSI > Note that SCSI targets, when faced with getting a command queue full, do not > stop reading from the interconnect. If more commands are received then they > respond with a QUEUE FULL status. If data is received then they receive > that data without regard for the status of the command queue (as long as it > is data for an already queued command). This eliminates the potential for > deadlock between command and data queues. Jim, I believe there is problem with the current SCSI behavior. Consider the following scenario for a host with a pathological queue of 1 1) Initiator sends command 1 (ORDERED attribute) 2) Initiator sends command 2 (ORDERED attribute) 3) Initiator sends command 3 (ORDERED attributge) 4) Target reads command 1 5) Target reads command 2 6) Target returns queue full for command 2 7) Command 1 completes 8) Target reads command 3 9) Target executes command 3 We have just violated the ordering constraints of the application by doing command 3 before command 2. > Potentially you can have a situation where multiple commands already at the > target have only part of their data transmitted, with the remainder still at > the initiator(s), and then run out of buffer space for data. If the target > uses a credit model to pace the reception of data, it can also make sure > this never happens. Unsolicited data, even for commands already queued, can > end up creating this deadlock - which is why unsolicited data systems either > have to have a tight limit on the resources it can use (e.g. low login BB > credit in Fibre Channel terms) or some sort of clean (i.e. not IO > terminating) rejection mechanism from target to initiator (like in USB). Unsolicited data is NOT a problem with the current iSCSI spec. We allow the target to always drop data and request data transfers with an RTT. -Costa From owner-ips@ECE.cmu.edu Wed Sep 13 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08036; 13 Sep 2000 3:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA08947 for ips-outgoing; Wed, 13 Sep 2000 02:28:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA08927 for ; Wed, 13 Sep 2000 02:28:01 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA126206 for ; Wed, 13 Sep 2000 08:27:30 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA33354 for ; Wed, 13 Sep 2000 08:27:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.00237634 ; Wed, 13 Sep 2000 08:27:20 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 08:36:57 +0300 Subject: Re: iSCSI: Partial Session Consensus Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Counters as an option is quite ugly and violates some of our design rules - that said that a protocol analyzer capturing the traffic on the wire will be able to read headers starting from any arbitrary point. We can add a next header thing or a flag - but I did not hear explicit objections to the counters. And if you come to think about it most implementations will want to support everything so they will have the counters implemented and we will have the added complexity of a next header field or 2 lengths for headers - none too appealing. I suggest we wait a couple of days? Julo Black_David@emc.com on 12/09/2000 19:24:25 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: iSCSI: Partial Session Consensus In reading the mailing list traffic on Asymmetric vs. Symmetric models for multiple TCP connections in an iSCSI session, I don't believe that there is consensus on the issue phrased in that fashion. I believe that there is consensus on a couple of underlying issues: (1) An iSCSI session containing a single TCP connection should not be required to use the currently specified iSCSI command reference numbers and sliding window mechanism because TCP will deliver commands in order. (2) Use of more than one TCP connection per iSCSI session is OPTIONAL. The consensus on (1) is rough; if anyone disagrees with this for a reason other than wanting to use the Symmetric model when an iSCSI session contains multiple TCP connections, please say so on the list. My reading of consensus is based on the fact that much of the recent discussion of the Asymmetric model has been motivated by single connection sessions, whereas essentially all of the recent discussion of the Symmetric model seems to have been focussed on multiple connection sessions. The consensus on (2) is sufficiently long standing to be a closed issue. The resolution to the "deadlock" scenarios posted by Costa is that the target must not stop reading from the TCP connection - SCSI provides means for a target to throw away things it can't deal with (e.g., TASK SET FULL), and with regard to the ordered command issues, I believe Steve Byan is correct when he says: I think this is a T10 bug, and iSCSI should defer to T10 to fix it. In addition to Steve noting that disks tend not to use ordered commands, I would note that a single tape device/drive tends not to have multiple simultaneous initiators sending commands to it, both of which limit the practical impact of this potential problem. I don't see consensus on the model for iSCSI sessions that contain multiple TCP connections. The consensus on (1) above for single TCP connection sessions does not take the Symmetric model out of consideration for multiple TCP connection sessions. If multiple sessions are negotiated on connection establishment, command reference numbers could be added to subsequent headers as a result of that successful negotiation. In order to make progress, this additional complexity should not be used as an argument against the consensus in (1) for single connection sessions. I hope the consensus on (1), or something close to it holds, as if it does not, we may have to form an offline design team or teams to work on this set of session issues, and that could take some time ... meanwhile, discussion of Asymmetric vs. Symmetric for multiple connection sessions should continue. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 13 07:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08062; 13 Sep 2000 3:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA08946 for ips-outgoing; Wed, 13 Sep 2000 02:28:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA08922 for ; Wed, 13 Sep 2000 02:28:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA22102 for ; Wed, 13 Sep 2000 08:27:28 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA43448 for ; Wed, 13 Sep 2000 08:27:27 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.00237732 ; Wed, 13 Sep 2000 08:27:22 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 09:11:15 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, There is a misunderstanding. You are required to keep order only on a given connection. SMP or no SMP they are represented by ONE data structure (I hope). You are not implying order across connection as it has no meaning to the receiver. Julo somesh_gupta@hp.com on 12/09/2000 21:33:31 Please respond to somesh_gupta@hp.com To: David.Robinson@EBay.Sun.COM, ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI > -----Original Message----- > From: David.Robinson@EBay.Sun.COM [mailto:David.Robinson@EBay.Sun.COM] > Sent: Monday, September 11, 2000 6:36 PM > To: ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > > Thanks for the information, I think part of my confusion is > the difference > in mapping SCSI from a datagram protocol onto a reliable > stream protocol. > In a datagram protocol if the data is sent without the receiver's > cooperation, the receiver's buffers may not be adequate and the > data must get discarded. Credits and RTT can be used to handle > this case. > > With a reliable stream transport like TCP, you don't get into > this situation because the receiver will never open the TCP > window beyond it's buffer capacity. For low amounts of buffering > it might not be as efficient as using RTT, but there is no > correctness or deadlock issues. Because each sender has its own > connection and own flow control they are independantly handled. > Likewise with seperate data connections, each is also flow controlled > so "unsolicted" data is not an issue. "Overflow" conditions simply > never occur. > > The only major design criteria is that the sender MUST maintain > ordering of data sent on any connection. Data Dn MUST always be > sent before data Dm where n < m. In particular, if unsolicated > data and RTT is mixed, the sender cannot send data Dm before it > has recieved an RTT for data Dn if both are to use the same > connection. If we are using multiple connections, this requirement has ramifications on the implementation on SMP systems. The initiator will be sending command on one connection, and data on another connection, which are two different actions. To ensure ordering would require taking a lock across the two actions (or otherwise ensuring that they always happen on the same CPU). This would be quite expensive actually. > > -David > > > I think people have been meaning "unsolicited data" to > really mean data sent > > to a receiver without that receiver having first indicated > that there is > > enough buffering to hold the data. For initiators acting > as receivers they > > have to verify this before they initiate the command (not > enough space for > > the whole command? Then break up the command.) For > Targets this requires > > something like a credit mechanism with RTTs being used. > > > > So there is an "unsolicited command" problem and an > "unsolicited data" > > problem. In both cases the sender creates the problem by not first > > reserving with the receiver enough resources for the commands/data. > > > > In the command case there is no SCSI mechanism to reserve > resources (QUEUE > > FULL is used to indicate overflows). Historically it has > been assumed that > > queues of commands do not overflow often in practice. In > reality initiators > > have often artificially limited the number of commands they > are willing to > > try and queue at the target in order to avoid this rejection (a loss > > opportunity in my mind). > > > > In the data case there is no "DATA QUEUE FULL" - instead, > an explicit credit > > model of some sort is used to indicate the receiver has > reserved space for > > the data (REQs in parallel SCSI, BB credits in Fibre > Channel). In this case > > the assumption was that data overflows would occur a lot otherwise. > > > > You can solve these problems by rejecting the overflow > cleanly (as SCSI does > > with commands), which is low latency and works well under > light loads. Or > > you can do credits. Credits add latency, or get you into > the problem of > > credit allocation, which can be optimized for light load > (over allocate > > credits) or heavy loads (allocate only what you have), but > not both at once. > > > > Historically, SCSI has used rejection for commands and > credits for data, > > optimized for heavy loads. But this is only a T10 given > rule, not a God > > given rule (although some of us who have served on T10 can > get that confused > > at times :-)). > > > > Hope this helps. > > > > Jim > > > > > > > > > > > > cases there are well known mechanisms to reserve the > > > > -----Original Message----- > > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > > Sent: Monday, September 11, 2000 3:35 PM > > To: ips@ece.cmu.edu > > Subject: Re: Avoiding deadlock in iSCSI > > > > > > I think in following this discussion the terminology has been > > confusing me. When I read "unsolicited data" I interpreted that > > to mean data for which no command has yet been sent. In general > > I consider that to be a bug and the receiver should just drop the > > data on the floor. The only possible scenerio where it might > > not be a bug is if a command was sent on one connection and the > > data on the data connection arrived first, thus it is unsolicited. > > My first assumption is that the sender would not send commands > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > creates nasty ordering problems we want to avoid. So if the > > receiver simply allows the data connection TCP window to shrink > > the unsolicted data will flow control to a stop until the command > > queue catches up. With multiple data connections, some may flow > > control but the active command will be able to make progress on > > one connection. This may not be the most efficient mechanism but > > it is "safe". Preferably the data will either follow the command > > on the same data/command connection or the sender will request a > > RTT (aka R2T). It is also a sender bug to request a connection > > for data transfer that it has already sent "unsolicited" data. > > > > Unless my assumptions and definitions are wrong, I don't > see the issue. > > > > -David > > > > > The problem: > > > > > > iSCSI, as currently spec'ed, allows SCSI commands and data to be > > > interleaved fairly freely on a TCP connection. A target that stops > > > reading from a TCP connection to avoid reading more > command packets > > > also prevents itself from reading data packets. Those > data packets > > > may be criticial to making progress on the currently executing > > > command. > > > > > > Note the issue appears with one TCP connection for > control and data > > > and even appears in many of the multiple connection schemes. > > > > > > Data in iSCSI comes in two forms: > > > > > > 1) solicited - data requested by target via RTT > > > - data requested by initiator via a SCSI command > > > 2) unsolicited - data sent by initiator without having > received an > > RTT > > > > > > The analysis below assumes that unsolicited data travels > over the same > > > TCP connection as SCSI commands. Otherwise, you run the > risk of receiving > > > unsolicited data before the relevant SCSI command (thus making > > > implementations more complex). > > > > > > Four solutions: > > > > > > 1) Don't overflow the command queue (i.e. use credits) > > > - and what do you do if a misbehaving initiator overflows > > > your command queue anyway? Drop the connection? > > > > > > - requires you to reserve resources per initiator. some people > > > may want to overcommit > > > > > > 2) Allow dropping of SCSI commands when queue fills > > > - how do you clean up after a dropped SCSI command? > > > - there may be other commands in the pipeline > > > > > > One approach: On command drop, the target enters an error > > > state. While in the error state, all newly received commands > > > terminate with an error until the initiator explicitly clears > > > the error state using a "clear error state" message. > > > > > > You might think that TASK SET FULL and ACA mechanisms from SCSI > > > could be used to attack this problem. However, > TASK SET FULL > > errors > > > don't trigger ACA (in my reading of the SAM). Also, ACA is only > > > triggered by the current enabled command, not by random commands > > > entered into the task set. > > > > > > 3) Put solicited data on a dedicated TCP connection. Require that > > > unsolicited data MUST follow the command, ideally in the > same iSCSI > > > PDU > > > > > > 4) (Do it like NFS) Make all transfers from initiator to target > > > unsolicited. Make sure unsolicited data follows the command > > > immediately. > > > > > > > > > Of all the options, #1 and #4 sound the easiest to > implement. #2 is more > > > sophisticated than #1. #3 is just plain clever but that's > rarely a good > > > thing. :) #4 has large ramifications on current SCSI > target designs. > > > > > > -Costa > From owner-ips@ECE.cmu.edu Wed Sep 13 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08578; 13 Sep 2000 4:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA10814 for ips-outgoing; Wed, 13 Sep 2000 03:33:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA10805 for ; Wed, 13 Sep 2000 03:33:50 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id F20B72C2 for ; Wed, 13 Sep 2000 00:33:49 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id AAA26640 for ; Wed, 13 Sep 2000 00:33:49 -0700 (PDT) Received: from agilent.com ([15.3.253.47]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA5D07 for ; Wed, 13 Sep 2000 00:33:45 -0700 Message-ID: <39BF2942.A1FEBD2C@agilent.com> Date: Wed, 13 Sep 2000 00:14:10 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Avoiding deadlock in iSCSI References: <4.3.2.7.2.20000912092424.00bb5350@PacBell.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I just keep hearing more and more reasons for having separate control and data TCP connections (read: a minimum of TWO). That way, if the command queue in the target fills up, no big deal, it simply stops reading from the command TCP stream and lets TCP do its window management. Mean while, the data TCP connection is still free and clear to transfer the data for the commands accepted. -Matt Wakeley Agilent Technologies Peter Johansson wrote: > At 07:25 AM 9/12/00, Stephen Byan wrote: > > >I think this is a T10 bug, and iSCSI should defer to T10 to fix it. > > The "this" Stephen refers to, summarized below, is not a T10 bug. > > >1) Initiator sends command 1 (ORDERED attribute) > >2) Initiator sends command 2 (ORDERED attribute) > >3) Initiator sends command 3 (ORDERED attributge) > >4) Target reads command 1 > >5) Target reads command 2 > >6) Target returns queue full for command 2 > >7) Command 1 completes > >8) Target reads command 3 > >9) Target executes command 3 > > > >We have just violated the ordering constraints of the application by doing > >command 3 before command 2. > > The malfunction is a property of the interaction of the transport protocol > (T10 terminology) with SAM. In the case of iSCSI, it has to be designed so > that no new commands can be accepted after the QUEUE FULL condition until > the target has confirmed knowledge that the initiator is aware of the > error. That is, the ordered pipeline of commands between the two endpoints > must be resynchronized before work may resume. > > Regards, > > Peter Johansson > > Congruent Software, Inc. > 98 Colorado Avenue > Berkeley, CA 94707 > > (510) 527-3926 > (510) 527-3856 FAX > > PJohansson@ACM.org From owner-ips@ECE.cmu.edu Wed Sep 13 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08573; 13 Sep 2000 4:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA11428 for ips-outgoing; Wed, 13 Sep 2000 03:58:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA11424 for ; Wed, 13 Sep 2000 03:58:09 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id 5FA87285 for ; Wed, 13 Sep 2000 03:58:08 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id AAA26892 for ; Wed, 13 Sep 2000 00:58:07 -0700 (PDT) Received: from agilent.com ([15.3.253.47]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA5D58 for ; Wed, 13 Sep 2000 00:58:03 -0700 Message-ID: <39BF33B1.FA5F4CE8@agilent.com> Date: Wed, 13 Sep 2000 00:58:41 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Avoiding deadlock in iSCSI References: <8133266FE373D11190CD00805FA768BF055BD1A2@shrcmsg1.tdh.qntm.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't think VI/TCP helps at all. The VI is implemented on top of a TCP stream. If the TCP stream delivers the iSCSI command to VI, and VI has no place to put it, what is VI going to do? It has to block the TCP stream, and that in turn will block and "RDMA" from occurring. -Matt Stephen Byan wrote: > Jim Williams [mailto:jimw@giganet.com] wrote: > > > Layering iSCSI on VI avoids the problem below. > > > > see: > > http://search.ietf.org/internet-drafts/draft-csapuntz-ips-iscs > > ivi-00.txt > > > > VI layer flow control will be invoked when target is unable > > to accept more > > commands, but this will not impede flow of data in response > > to RDMA read > > requests by the target. Functionally this is equivalent to > > option 1 below. > > I haven't seen much discussion of iSCSI over VI here. I think it makes > sense. Using VI makes the iSCSI remote DMA hardware acceleration available > to other applications besides iSCSI. This broader market should greatly > increase the utility of iSCSI hardware, and thus speed the market > penetration of iSCSI, bringing the usual benefits of customer acceptance, > lower cost, and better implementions courtesy of darwinian competition. > > And, as an added bonus, you get flow control on commands for free :-) > > Regards, > -Steve > > Steve Byan > > Design Engineer > MS 1-3/E23 > 333 South Street > Shrewsbury, MA 01545 > (508)770-3414 > fax: (508)770-2604 From owner-ips@ECE.cmu.edu Wed Sep 13 09:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08916; 13 Sep 2000 5:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA20569 for ips-outgoing; Wed, 13 Sep 2000 04:42:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA20562 for ; Wed, 13 Sep 2000 04:42:23 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 5FF8276D for ; Wed, 13 Sep 2000 01:42:22 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id BAA22320; Wed, 13 Sep 2000 01:42:21 -0700 (PDT) Message-ID: <39BF3F79.23429EBE@hp.com> Date: Wed, 13 Sep 2000 01:48:58 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk >From the discussion going on the symmetric versus asymmetric model and dead lock avoidance, here is a proposal that tries to combine (as far as possible...) the advantages of both models. The proposal is a slightly modified symmetric model where the symmetric model incorporate an idea of the current asymmetric model. This idea is that the initiator indicates to the target on which connection the next command in order arrives. The asymmetric model does that using an extra connection transporting the commands, and for each command in the header there is the connection number where the data will come. Description of the proposal: --------------------- In the command header a field is added. This field "next connection" indicates the TCP connection (connection id) in the session on which the next command will arrive. This field builds a linked list of commands across the several symmetric connections of the session. The first time the initiator posts a command, it determines the connection on which it post the command and the connection on which it will post the next command. Then for each command the initiator determines only the connection it will use to post the next command. For each command, the connexion id of the next command is added in the header field "next connection" of each command. Doing that, leads to the same algorithm as in the asymmetric model to process the commands on the target side: - a thread of execution pulls the cmd/data in order from the TCP windows. After pulling a cmd/data, it looks at the "next connection" field and move to the next connection to pull a new cmd/data. There is no need of extra storage out of the TCP windows and no need to re-order. In the asymmetric model the thread empties the command connection, here it follows the chain of commands. The advantage of this proposal are: - on the target, as opposed to the original symmetric model, it avoids the re-ordering process of the commands once extracted off the TCP window. - it keeps cmd/data/status on a same connexion avoiding the synchronization penalty for the READS on the initiator (described by Somesh Gupta) - it saves one TCP connection compared to the asymmetric model. Anyway any ordering implementation adds overhead, thus, as in most cases (disks) it is not required, It would be possible to deactivate it. Regards, Pierre From owner-ips@ECE.cmu.edu Wed Sep 13 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa09303; 13 Sep 2000 7:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA22636 for ips-outgoing; Wed, 13 Sep 2000 06:24:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA22632 for ; Wed, 13 Sep 2000 06:24:49 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA118100 for ; Wed, 13 Sep 2000 12:24:16 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA20642 for ; Wed, 13 Sep 2000 12:24:16 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.003925F6 ; Wed, 13 Sep 2000 12:24:12 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 13:21:40 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, That is the essence of Kalman's proposal (and that differentiates it from draft-00 and from an unplanned-allegiance that we discussed in the loby of a hote in Adelaide 100 years ago. Julo Pierre Labat on 13/09/2000 11:48:58 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI >From the discussion going on the symmetric versus asymmetric model and dead lock avoidance, here is a proposal that tries to combine (as far as possible...) the advantages of both models. The proposal is a slightly modified symmetric model where the symmetric model incorporate an idea of the current asymmetric model. This idea is that the initiator indicates to the target on which connection the next command in order arrives. The asymmetric model does that using an extra connection transporting the commands, and for each command in the header there is the connection number where the data will come. Description of the proposal: --------------------- In the command header a field is added. This field "next connection" indicates the TCP connection (connection id) in the session on which the next command will arrive. This field builds a linked list of commands across the several symmetric connections of the session. The first time the initiator posts a command, it determines the connection on which it post the command and the connection on which it will post the next command. Then for each command the initiator determines only the connection it will use to post the next command. For each command, the connexion id of the next command is added in the header field "next connection" of each command. Doing that, leads to the same algorithm as in the asymmetric model to process the commands on the target side: - a thread of execution pulls the cmd/data in order from the TCP windows. After pulling a cmd/data, it looks at the "next connection" field and move to the next connection to pull a new cmd/data. There is no need of extra storage out of the TCP windows and no need to re-order. In the asymmetric model the thread empties the command connection, here it follows the chain of commands. The advantage of this proposal are: - on the target, as opposed to the original symmetric model, it avoids the re-ordering process of the commands once extracted off the TCP window. - it keeps cmd/data/status on a same connexion avoiding the synchronization penalty for the READS on the initiator (described by Somesh Gupta) - it saves one TCP connection compared to the asymmetric model. Anyway any ordering implementation adds overhead, thus, as in most cases (disks) it is not required, It would be possible to deactivate it. Regards, Pierre From owner-ips@ECE.cmu.edu Wed Sep 13 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10169; 13 Sep 2000 10:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA26909 for ips-outgoing; Wed, 13 Sep 2000 09:22:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA26897 for ; Wed, 13 Sep 2000 09:22:14 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id GAA14488 for ; Wed, 13 Sep 2000 06:22:09 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Wed, 13 Sep 2000 06:22:07 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1AC@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: IPS Reflector Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Date: Wed, 13 Sep 2000 06:22:08 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote: > I don't think VI/TCP helps at all. The VI is implemented on > top of a TCP > stream. If the TCP stream delivers the iSCSI command to VI, > and VI has no > place to put it, what is VI going to do? It has to block the > TCP stream, and > that in turn will block and "RDMA" from occurring. If I understand correctly, VI/TCP has a credit-based flow control mechanism on its message queues (which would be used to implement the iSCSI command reception queue), and so the initiator would never send an iSCSI command without a target buffer in which to store it. So RDMA cannot block on commands in the TCP stream. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Wed Sep 13 15:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10464; 13 Sep 2000 11:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA01396 for ips-outgoing; Wed, 13 Sep 2000 11:03:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from storage.network.com (storage.network.com [129.191.1.4]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA01379 for ; Wed, 13 Sep 2000 11:03:26 -0400 (EDT) Received: from anubis.network.com (anubis.network.com [129.191.18.1]) by storage.network.com (8.9.3+Sun/8.9.3) with ESMTP id KAA20543 for ; Wed, 13 Sep 2000 10:03:24 -0500 (CDT) Received: from network.com ([129.191.40.66]) by anubis.network.com (8.9.3/8.9.3) with ESMTP id KAA08795 for ; Wed, 13 Sep 2000 10:03:21 -0500 (CDT) Message-ID: <39BF9732.387C072A@network.com> Date: Wed, 13 Sep 2000 10:03:14 -0500 From: "David A. Peterson" Organization: StorageTek X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: a vote for asymmetric connections in a session References: <39BE32FC.41586525@agilent.com> Content-Type: multipart/mixed; boundary="------------EC6A31DCDC5ACF8B90E63A49" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------EC6A31DCDC5ACF8B90E63A49 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit I would consider the FCP_XFER_RDY IU a "flow control" mechanism (granted it is currently only defined for writes). Also, the FCP-2 sequence level recovery has been developed and is deployed in the field. It does work for class 3 in-order delivery with no known problems other than the recently discovered ambiguous command issue for which there is an agreed upon solution. Matt Wakeley wrote: > Joshua Tseng wrote: > > > John, > > > > "A lot of data" is relative. The fastest TCP implementations available > > today > > don't come close to what will be required for iSCSI, since they peak out > > at about 200Mbps. And the vast majority of applications are get much worse > > performance than that out of TCP--I would guess typically about 20Mbps max. > > > > iSCSI will use TCP like never before. I believe FCP has flow control/ > > command recovery mechanisms below SCSI. > > Nope. FCP does not have *any* flow control mechanisms (Fibre Channel itself > does, but *not* the FCP protocol). > In addition, FCP does *not* have any command recovery mechisms either. It uses > the "if the command doesn't complete within x amount of time, blow it away and > try again" recovery mechanism. > > FCP-2 (which is still under developement, and not deployed) is attempting to > "recover" lost commands and data frames. And this "recovery" is only meant for > devices such as tapes, and not discs. > > -Matt > > > Unlike iSCSI, FCP has the benefit of operating in a low latency environment > > (<10us). To think iSCSI can do > > without it and still be able to function reliably is, well iffy at best from > > what I can see. > > > > Regads, > > > > Josh --------------EC6A31DCDC5ACF8B90E63A49 Content-Type: text/x-vcard; charset=us-ascii; name="dap.vcf" Content-Transfer-Encoding: 7bit Content-Description: Card for David A. Peterson Content-Disposition: attachment; filename="dap.vcf" begin:vcard n:Peterson;David A. tel;cell:612-251-6229 tel;work:763-391-1008 x-mozilla-html:FALSE org:StorageTek;Minnesota Research and Development Center adr:;;;;;; version:2.1 email;internet:dap@network.com title:Chief Architect - Network Interface Development fn:David A. Peterson end:vcard --------------EC6A31DCDC5ACF8B90E63A49-- From owner-ips@ECE.cmu.edu Wed Sep 13 15:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10459; 13 Sep 2000 11:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA00180 for ips-outgoing; Wed, 13 Sep 2000 10:41:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA00071 for ; Wed, 13 Sep 2000 10:40:17 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Wed, 13 Sep 2000 07:41:29 -0700 Message-ID: From: Robert Snively To: Douglas Otis , Robert Reynolds , "'Stephen Bailey'" , ips@ece.cmu.edu Subject: RE: Data in SCSI Response or SCSI Data Date: Wed, 13 Sep 2000 07:38:41 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, It is true that CHECK CONDITION is not presented for transfers shorter than the allocation when the SILI bit is set on tape drives. However, for all other cases, it is. In any case, FCP provides an independent under-run/over-run condition that provides the necessary information to "use other methods" that identify shorter or longer than expected transfers. These were provided because the Fibre Channel transport mechanism is potentially independent from the SCSI command set operations. Basically, it is always there when the application program desires to use it. Normally, variable length parameter fields such as INQUIRY data fields, are self-defining in length. You merely set out a large allocation space and then parse the resulting data object to find out where it ends and if it was truncated in an undesirable way. These are interesting items to keep in mind, and are one of the reasons we recommend the FCP_RSP format for the actual response. Bob > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Monday, September 11, 2000 11:45 AM > To: Robert Reynolds; 'Stephen Bailey'; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > Bob, > > SAM-2 > "As specified in clause 5, the application client may > request autosense > service for any SCSI command. If supported > by the protocol and logical unit and requested by the > application client, > the device server shall only return sense > data in this manner coincident with the completion of a > command with a > status of CHECK CONDITION." > > SSC > "If the suppress incorrect length indicator (SILI) bit is > one and the FIXED > bit is zero, the device server shall: > a) Report CHECK CONDITION status for an incorrect length > condition only if > the overlength condition > exists and the BLOCK LENGTH field in the mode parameter > block descriptor is > nonzero (see SPC-2); or > b) not report CHECK CONDITION status if the only error is > the underlength > condition, or if the only error > is the overlength condition and the BLOCK LENGTH field of the mode > parameters block descriptor is zero. > > NOTE 9 Since the residue information normally provided in > the INFORMATION > field of the sense data may not > be available when the SILI bit is set, other methods for > determining the > actual block length should be used > (e.g. including length information in the data block). > > If the SILI bit is one and the FIXED bit is one, the device > server shall > terminate the command with CHECK > CONDITION status and the sense key shall be set to ILLEGAL > REQUEST with an > additional sense code > and an additional sense code qualifier of INVALID FIELD IN CDB. > > If the SILI bit is zero and an incorrect length block is read, CHECK > CONDITION status shall be returned > and the ILI and VALID bits shall be set to one in the sense > data with an > additional sense code and an > additional sense code qualifier of NO ADDITIONAL SENSE > INFORMATION. Upon > termination, the > logical position shall be after the incorrect length block > (end-of-partition > side). If the FIXED bit is one, the > INFORMATION field shall be set to the requested transfer > length minus the > actual number of blocks read > (not including the incorrect length block). If the FIXED bit > is zero, the > INFORMATION field shall be set to the > requested transfer length minus the actual block length. > Logical units that > do not support negative values > shall set the INFORMATION field to zero if the overlength > condition exists." > > From these definitions, do not expect any defined > information regarding > residue without a Check Condition. The residue of iSCSI > within the Read PDU > 0x45 is not SCSI related nor does it represent the > underlength condition you > suggest. Such underlength data is normally lost and > uninteresting if the > SILI bit is set. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Robert Reynolds > Sent: Monday, September 11, 2000 8:07 AM > To: 'Stephen Bailey'; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > > -----Original Message----- > > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > > Sent: Wednesday, September 06, 2000 5:45 PM > > To: ips@ece.cmu.edu > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > When do you get GOOD status and residual counts on a read? > > What is causing > > > the target to get the length wrong? > > > > One example is an INQUIRY command. The inquiry data length is > > target-specific. Typically the CDB allocation length (and DL) are set > > to some arbitrary large value (0xff), and the target sends back > > everything it has. The transfer ends with success status. > > > > There can certainly be transfer residual and no SCSI error status. > > > Another example is on variable block tape reads. The Initiator > uses an arbitrary size to read the block and the target sends all > the data in that block. In many cases the amount of data returned > is less than the amount of data asked for but it is still a GOOD > status. > > Bob > From owner-ips@ECE.cmu.edu Wed Sep 13 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10893; 13 Sep 2000 13:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03639 for ips-outgoing; Wed, 13 Sep 2000 11:51:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03631 for ; Wed, 13 Sep 2000 11:51:43 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 62EDDA30 for ; Wed, 13 Sep 2000 08:51:38 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id IAA29166; Wed, 13 Sep 2000 08:51:20 -0700 (PDT) Message-ID: <39BFA405.70B4F001@hp.com> Date: Wed, 13 Sep 2000 08:57:57 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > Pierre, > > That is the essence of Kalman's proposal (and that differentiates it from > draft-00 and from an unplanned-allegiance that we > discussed in the loby of a hote in Adelaide 100 years ago. I disagree, there is a fundamental difference with the Kalman draft: in the proposal below always cmd/data/status goes on the same TCP connection. It is not the case in the Kalman draft. The fact that in the Kalman draft the end of data notification and command status comes on two different TCP connection (thus 2 NICs most of the time) implies an important CPU overhead on the initiator to manage the READ command completion as described by Somesh Gupta. You need for each READ to work accross two NICs. You will double the work on each read completion. Moreover you will double the number of cache misses because you need to read completion information from two NICs. And cache misses are really stalling your CPUs. It's a reason for what i can't agree on the Kalman proposal. You don't have this penalty in the proposal below. Regards, Pierre > > > Julo > > Pierre Labat on 13/09/2000 11:48:58 > > Please respond to Pierre Labat > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Avoiding deadlock in iSCSI > > >From the discussion going on the symmetric versus asymmetric model and > dead lock avoidance, here is a proposal that tries to combine (as far > as possible...) > the advantages of both models. > > The proposal is a slightly modified symmetric model where > the symmetric model incorporate an idea of the current > asymmetric model. > > This idea is that the initiator indicates to the target > on which connection the next command in order arrives. > The asymmetric model does that using an extra connection > transporting the commands, and for each command in the header > there is the connection number where the data will come. > > Description of the proposal: > --------------------- > In the command header a field is added. This field > "next connection" indicates the TCP connection (connection id) > in the session on which the next command will arrive. > This field builds a linked list of commands across the several > symmetric connections of the session. > > The first time the initiator posts a command, it determines > the connection on which it post the command and the connection > on which it will post the next command. > Then for each command the initiator determines only > the connection it will use to post the next command. > For each command, the connexion id of the next command is added in the > header > field "next connection" of each command. > > Doing that, leads to the same algorithm as in the asymmetric model > to process the commands on the target side: > - a thread of execution pulls the cmd/data in order from the TCP > windows. After pulling a cmd/data, it looks at the "next connection" > field and move to the next connection to pull a new cmd/data. > There is no need of extra storage out of the TCP windows > and no need to re-order. In the asymmetric model the thread > empties the command connection, here it follows the chain of commands. > > The advantage of this proposal are: > - on the target, as opposed to the original symmetric model, it avoids > the re-ordering process of the commands once extracted > off the TCP window. > - it keeps cmd/data/status on a same connexion avoiding > the synchronization penalty for the READS on the initiator (described > by Somesh Gupta) > - it saves one TCP connection compared to the asymmetric model. > > Anyway any ordering implementation adds overhead, thus, as in most > cases (disks) it is not required, It would be possible to deactivate it. > > Regards, > > Pierre From owner-ips@ECE.cmu.edu Wed Sep 13 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10958; 13 Sep 2000 13:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA04848 for ips-outgoing; Wed, 13 Sep 2000 12:16:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA04832 for ; Wed, 13 Sep 2000 12:16:40 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA10142; Wed, 13 Sep 2000 09:06:21 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA16177; Wed, 13 Sep 2000 09:04:01 -0700 From: "Douglas Otis" To: "Robert Snively" , "Robert Reynolds" , "'Stephen Bailey'" , Subject: RE: Data in SCSI Response or SCSI Data Date: Wed, 13 Sep 2000 09:19:02 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, Explicit definitions found in FCP are better than implied. Data transfers should be punctuated with an end flag and all data transfers terminated with a FCP_RSP like structure. Any length data payload may be split across frames, and to allow recovery without sideband signaling, a terminating response is required. If the goal is to save a few bytes, reduce header size but do not change operation on some conditional basis. Each implementation will likely abuse intended use as it creates further conditional implementations based on a few bits here and there and opinions as to what small is. I do not see elegance in an option of where to put a response structure. (In front or behind the actual data transfer.) The FCP residual value is to reconcile bytes indicated within the SAM request. The initiator is also be able to directly calculate bytes transferred and residual. The difficulty is with overlays by muddled targets. As such, let the target sort it out. Doug > -----Original Message----- > From: Robert Snively [mailto:rsnively@Brocade.COM] > Sent: Wednesday, September 13, 2000 7:39 AM > To: Douglas Otis; Robert Reynolds; 'Stephen Bailey'; ips@ece.cmu.edu > Subject: RE: Data in SCSI Response or SCSI Data > > > > Doug, > > It is true that CHECK CONDITION is not presented for transfers > shorter than the allocation when the SILI bit is set on tape > drives. However, for all other cases, it is. > > In any case, FCP provides an independent under-run/over-run > condition that provides the necessary information to > "use other methods" that identify shorter or longer than > expected transfers. These were provided because the Fibre > Channel transport mechanism is potentially independent from > the SCSI command set operations. > > Basically, it is always there when the application program > desires to use it. Normally, variable length parameter fields > such as INQUIRY data fields, are self-defining in length. You > merely set out a large allocation space and then parse the > resulting data object to find out where it ends and if it > was truncated in an undesirable way. > > These are interesting items to keep in mind, and are one of the > reasons we recommend the FCP_RSP format for the actual response. > > Bob > > > > > > -----Original Message----- > > From: Douglas Otis [mailto:dotis@sanlight.net] > > Sent: Monday, September 11, 2000 11:45 AM > > To: Robert Reynolds; 'Stephen Bailey'; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > Bob, > > > > SAM-2 > > "As specified in clause 5, the application client may > > request autosense > > service for any SCSI command. If supported > > by the protocol and logical unit and requested by the > > application client, > > the device server shall only return sense > > data in this manner coincident with the completion of a > > command with a > > status of CHECK CONDITION." > > > > SSC > > "If the suppress incorrect length indicator (SILI) bit is > > one and the FIXED > > bit is zero, the device server shall: > > a) Report CHECK CONDITION status for an incorrect length > > condition only if > > the overlength condition > > exists and the BLOCK LENGTH field in the mode parameter > > block descriptor is > > nonzero (see SPC-2); or > > b) not report CHECK CONDITION status if the only error is > > the underlength > > condition, or if the only error > > is the overlength condition and the BLOCK LENGTH field of the mode > > parameters block descriptor is zero. > > > > NOTE 9 Since the residue information normally provided in > > the INFORMATION > > field of the sense data may not > > be available when the SILI bit is set, other methods for > > determining the > > actual block length should be used > > (e.g. including length information in the data block). > > > > If the SILI bit is one and the FIXED bit is one, the device > > server shall > > terminate the command with CHECK > > CONDITION status and the sense key shall be set to ILLEGAL > > REQUEST with an > > additional sense code > > and an additional sense code qualifier of INVALID FIELD IN CDB. > > > > If the SILI bit is zero and an incorrect length block is read, CHECK > > CONDITION status shall be returned > > and the ILI and VALID bits shall be set to one in the sense > > data with an > > additional sense code and an > > additional sense code qualifier of NO ADDITIONAL SENSE > > INFORMATION. Upon > > termination, the > > logical position shall be after the incorrect length block > > (end-of-partition > > side). If the FIXED bit is one, the > > INFORMATION field shall be set to the requested transfer > > length minus the > > actual number of blocks read > > (not including the incorrect length block). If the FIXED bit > > is zero, the > > INFORMATION field shall be set to the > > requested transfer length minus the actual block length. > > Logical units that > > do not support negative values > > shall set the INFORMATION field to zero if the overlength > > condition exists." > > > > From these definitions, do not expect any defined > > information regarding > > residue without a Check Condition. The residue of iSCSI > > within the Read PDU > > 0x45 is not SCSI related nor does it represent the > > underlength condition you > > suggest. Such underlength data is normally lost and > > uninteresting if the > > SILI bit is set. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Robert Reynolds > > Sent: Monday, September 11, 2000 8:07 AM > > To: 'Stephen Bailey'; ips@ece.cmu.edu > > Subject: RE: Data in SCSI Response or SCSI Data > > > > > > > -----Original Message----- > > > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > > > Sent: Wednesday, September 06, 2000 5:45 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: Data in SCSI Response or SCSI Data > > > > > > > > > > When do you get GOOD status and residual counts on a read? > > > What is causing > > > > the target to get the length wrong? > > > > > > One example is an INQUIRY command. The inquiry data length is > > > target-specific. Typically the CDB allocation length (and DL) are set > > > to some arbitrary large value (0xff), and the target sends back > > > everything it has. The transfer ends with success status. > > > > > > There can certainly be transfer residual and no SCSI error status. > > > > > > Another example is on variable block tape reads. The Initiator > > uses an arbitrary size to read the block and the target sends all > > the data in that block. In many cases the amount of data returned > > is less than the amount of data asked for but it is still a GOOD > > status. > > > > Bob > > > From owner-ips@ECE.cmu.edu Wed Sep 13 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11538; 13 Sep 2000 15:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA11730 for ips-outgoing; Wed, 13 Sep 2000 14:19:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA11724 for ; Wed, 13 Sep 2000 14:19:42 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA23451 for ; Wed, 13 Sep 2000 11:19:41 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.142.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA15355 for ; Wed, 13 Sep 2000 11:19:40 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA17540; Wed, 13 Sep 2000 11:19:40 -0700 (PDT) Message-ID: <39BFC3D3.9EB59E73@ebay.sun.com> Date: Wed, 13 Sep 2000 11:13:39 -0700 From: David Robinson X-Mailer: Mozilla 4.74 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: <200009121807.LAA29970@ha10nwk.EBay.Sun.COM> <39BEEF89.3B02BBA@stewart.chicago.il.us> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "Randall R. Stewart" wrote: > Your reply leads me to believe a tight coupling exists between > the iSCSI layer and the TCP stack. I.e. iSCSI is going in > and "tweaking on a dynamic basis" the TCP implemenations > rwnd. Other email that has crossed on this subject implies > that "a TCP implementation is broken if it offers more > buffers than it has"... This in some ways contradicts iSCSI > controlling TCP rwnd and TCP controlling its own rwnd... > > could you please clearify this for me? Is it TCP in control > of rwnd or is there a tight coupling here? I envision that iSCSI controls the rwnd by simply making a choice on whether to read from that connection or not. It of course tells the TCP layer what its initial buffers are (ala setsockopt) but if it doesn't read from a data connection that will cause the window to shrink in response to data arriving causing flow control. In my extreme example where iSCSI maintains a zero window, this would require iSCSI to have a hand into the TCP stack. I am not advocating this but just making a point on what could be done at the extreme, but iSCSI should not require an implentation to violate layering. -David From owner-ips@ECE.cmu.edu Wed Sep 13 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11559; 13 Sep 2000 15:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA10375 for ips-outgoing; Wed, 13 Sep 2000 14:00:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA10322 for ; Wed, 13 Sep 2000 14:00:37 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id UAA32198 for ; Wed, 13 Sep 2000 20:00:05 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id UAA48684 for ; Wed, 13 Sep 2000 20:00:05 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256959.0062DE76 ; Wed, 13 Sep 2000 19:59:54 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 13 Sep 2000 20:57:09 +0300 Subject: RE: Comments on the Draft Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk On 1 - depends on where thing go with asymmetric vs. symmetric. 2 - Explicit error codes - isn't that the sense function? 3 - For the flag - probably yes- but I am not sure. Status implies it so you need it only if status is bad - which can be before the end. Julo "Hall, Howard" on 13/09/2000 20:50:10 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Comments on the Draft Julo, Thank you very much for your thoughtful response. What do you think of some of the other issues (reference the original email): - whether CmdRN, MaxCmdRN, etc. should be used for in-order delivery of commands across multiple TCP connections in a session (Comments for Section 2.2.2). - more explicit error codes - the proposed flag described at the end of the email -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 4:52 AM To: Hall, Howard Subject: Re: Comments on the Draft 1. General - Loss of synchronization - we hope to be able one day to get to a better solution but I would suggest to recommend the implementer to drop the connections as it won't be able to parse a reset too, reinstate them and then try a reset to distinguish it clearly from other events. If it happens in a target - drop - and wait for a reconnect then send an AE to indicate the event and wait for reset. As for inconsistencies - I've added your proposal to the todo list - thank you. 2. The view is a qualifier for the name - it allows a target to a second level of refinement about what he things it should show the initiator. The first part of the name itself is the first level - but for those that are conservative in the use of names they are left with the view. 3. We structured it to follow closely FCP - for bridging reasons. As far as I can recall Response comes first followed by sense. We will state it explicitly in the document thanks. 4.timeouts - yes you are right and I stated it already in an older note about bridging. We where thinking about three timers: - command delivery - data delivery - status delivery Those should be set at the session initiation and be known to the initiator and target but they don't have to be necessarily sent over the wire (as they imply mostly local actions in the endpoints). 5. immediate data length negotiation - will fix the negotiation parameters - thanks. We think also about limiting it by default to 64k(?) 6. iSCSI check - the reason I choose to make it SCSI check too is to get a "hook" into ACA behavior. I think that is mandatory - without it all hell breaks loose! Both initiator and target could be built the with ACA on check condition. We can remove the iSCSI status field and add a set of basic sense bytes detailing the error. Let us keep talking on this item. 7. The security parts are being rewritten just as we talk and we hope that we will all see them soon 8. Target reset will be changed to have two types - soft and hard reset. 9. closing connection. Yes you are right it needs more text. And we already agreed to add a logout (on a different connection to force connection close) and remove the RID from login Thanks for your careful reading and hope to see you on row 1 or 2 at the next IETF meeting, Julo "Hall, Howard" on 13/09/2000 02:12:38 Please respond to "Hall, Howard" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Comments on the Draft Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Wed Sep 13 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12022; 13 Sep 2000 16:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA14240 for ips-outgoing; Wed, 13 Sep 2000 15:11:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA14227 for ; Wed, 13 Sep 2000 15:11:16 -0400 (EDT) Message-ID: <618471D08ABDD31188B6009027E500930F13DD@apollo.pirus.com> From: "Hall, Howard" To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: Comments on the Draft Date: Wed, 13 Sep 2000 15:22:56 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk On issue 2 its not the sense issue, it is the argument that: Since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (eg. not_logged_on, data_in_timeout, buffer_address etc..) So, for example, in section 2.2.3 it states "Any message sent on a TCP connection before this connection gets into full feature phase at the initiator should be rejected by the initiator. A message reaching a target on a TCP connection before the full feature phase will be rejected with an iSCSI check condition." Why not send an iSCSI status that is more descriptive like not_logged_on? -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 1:57 PM To: ips@ece.cmu.edu Subject: RE: Comments on the Draft On 1 - depends on where thing go with asymmetric vs. symmetric. 2 - Explicit error codes - isn't that the sense function? 3 - For the flag - probably yes- but I am not sure. Status implies it so you need it only if status is bad - which can be before the end. Julo "Hall, Howard" on 13/09/2000 20:50:10 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Comments on the Draft Julo, Thank you very much for your thoughtful response. What do you think of some of the other issues (reference the original email): - whether CmdRN, MaxCmdRN, etc. should be used for in-order delivery of commands across multiple TCP connections in a session (Comments for Section 2.2.2). - more explicit error codes - the proposed flag described at the end of the email -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 4:52 AM To: Hall, Howard Subject: Re: Comments on the Draft 1. General - Loss of synchronization - we hope to be able one day to get to a better solution but I would suggest to recommend the implementer to drop the connections as it won't be able to parse a reset too, reinstate them and then try a reset to distinguish it clearly from other events. If it happens in a target - drop - and wait for a reconnect then send an AE to indicate the event and wait for reset. As for inconsistencies - I've added your proposal to the todo list - thank you. 2. The view is a qualifier for the name - it allows a target to a second level of refinement about what he things it should show the initiator. The first part of the name itself is the first level - but for those that are conservative in the use of names they are left with the view. 3. We structured it to follow closely FCP - for bridging reasons. As far as I can recall Response comes first followed by sense. We will state it explicitly in the document thanks. 4.timeouts - yes you are right and I stated it already in an older note about bridging. We where thinking about three timers: - command delivery - data delivery - status delivery Those should be set at the session initiation and be known to the initiator and target but they don't have to be necessarily sent over the wire (as they imply mostly local actions in the endpoints). 5. immediate data length negotiation - will fix the negotiation parameters - thanks. We think also about limiting it by default to 64k(?) 6. iSCSI check - the reason I choose to make it SCSI check too is to get a "hook" into ACA behavior. I think that is mandatory - without it all hell breaks loose! Both initiator and target could be built the with ACA on check condition. We can remove the iSCSI status field and add a set of basic sense bytes detailing the error. Let us keep talking on this item. 7. The security parts are being rewritten just as we talk and we hope that we will all see them soon 8. Target reset will be changed to have two types - soft and hard reset. 9. closing connection. Yes you are right it needs more text. And we already agreed to add a logout (on a different connection to force connection close) and remove the RID from login Thanks for your careful reading and hope to see you on row 1 or 2 at the next IETF meeting, Julo "Hall, Howard" on 13/09/2000 02:12:38 Please respond to "Hall, Howard" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Comments on the Draft Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Wed Sep 13 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12030; 13 Sep 2000 16:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA14331 for ips-outgoing; Wed, 13 Sep 2000 15:14:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA14326 for ; Wed, 13 Sep 2000 15:13:59 -0400 (EDT) Message-ID: <618471D08ABDD31188B6009027E500930F13DE@apollo.pirus.com> From: "Hall, Howard" To: "'David Robinson'" , ips@ece.cmu.edu Subject: RE: Comments on the Draft (sync loss) Date: Wed, 13 Sep 2000 15:25:39 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, There is precedent for this type of framing recovery: >From rfc1831, RPC, - August 1995 10. RECORD MARKING STANDARD When RPC messages are passed on top of a byte stream transport protocol (like TCP), it is necessary to delimit one message from another in order to detect and possibly recover from protocol errors. This is called record marking (RM). One RPC message fits into one RM record. A record is composed of one or more record fragments. A record fragment is a four-byte header followed by 0 to (2**31) - 1 bytes of fragment data. The bytes encode an unsigned binary number; as with XDR integers, the byte order is from highest to lowest. The number encodes two values -- a boolean which indicates whether the fragment is the last fragment of the record (bit value 1 implies the fragment is the last fragment) and a 31-bit unsigned binary value which is the length in bytes of the fragment's data. The boolean value is the highest-order bit of the header; the length is the 31 low-order bits. (Note that this record specification is NOT in XDR standard form!) -Howard -----Original Message----- From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] Sent: Tuesday, September 12, 2000 7:29 PM To: ips@ece.cmu.edu Subject: Re: Comments on the Draft (sync loss) > Section: General > There's no framing of the headers and data on the buffers from TCP. If > anything goes wrong with the parsing, its difficult if not impossible to > recover. It only takes one length field to be 'off'. If this happens the > target will probably generate lots of "Opcode not understood" messages. We > suggest one of two methods: 1) after seeing consecutive "Opcode not > understood" messages it should shut down the connection if this doesn't > solve the problem then reset the target, or 2) When the target finds that > it is out of sync with the initiator ( on receipt of an "Opcode not > understood"), it will send a new iSCSI "Out of Sync" command to the > initiator. The initiator will assume at the reception of the "Out of Sync" > command that all unacknowledged outstanding requests have been dropped. The > initiator then sends the next command with the OOB (out of band) bit set, > and with the OOB offset pointing to the beginning of the iSCSI header. The > target, after sending the "Out of Sync" command, should ignore every thing > on that connection and wait for the OOB data to re-sync again. This > exchange could also work if sent from the initiator to the target. The loss of sync is an extremely rare event, either the sender sent a request shorter than it indicated, the TCP stack caused corruption, or the receiver misinterpreted the request. All are severe bugs somewhere, in practice with NFS over TCP which has no framing, sync errors are so rare that it is ignored. Instead of going through a complex process of setting OOB to resync, then determine which messages need to be resent, why not instead simply drop the connection and do the common lost connection recovery. The symptoms and state that needs to be recovered is virtually identical. Lets try to stay simple and not overly complicate the recovery process. -David From owner-ips@ECE.cmu.edu Wed Sep 13 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12032; 13 Sep 2000 16:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15578 for ips-outgoing; Wed, 13 Sep 2000 15:35:25 -0400 (EDT) Received: from mail.wrs.com (unknown-1-11.wrs.com [147.11.1.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15570 for ; Wed, 13 Sep 2000 15:35:15 -0400 (EDT) Received: (from root@localhost) by mail.wrs.com (8.9.3/8.9.1) id MAA13715; Wed, 13 Sep 2000 12:34:40 -0700 (PDT) Received: from mail03-oak.pilot.net (mail-oak-3.pilot.net [198.232.147.18]) by mail.wrs.com (8.9.3/8.9.1) with ESMTP id TAA16200 for ; Tue, 12 Sep 2000 19:18:20 -0700 (PDT) Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by mail03-oak.pilot.net with ESMTP id TAA03621 for ; Tue, 12 Sep 2000 19:18:32 -0700 (PDT) Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA00155 for ips-outgoing; Tue, 12 Sep 2000 20:53:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA00149 for ; Tue, 12 Sep 2000 20:53:41 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id UAA06701; Tue, 12 Sep 2000 20:53:33 -0400 (EDT) Date: Tue, 12 Sep 2000 20:53:33 -0400 (EDT) From: Scott Bradner Message-Id: <200009130053.UAA06701@newdev.harvard.edu> To: ips@ece.cmu.edu, julian_satran@il.ibm.com, ycheng@advansys.com Subject: RE: TCP speed Sender: owner-ips@ece.cmu.edu Precedence: bulk > I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an > initiator is inherently acknowledged by its response from a target. > Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. the ips working group can decide to do something over UDP but it should note that the result has to deal with congestion in a TCP-friendly way - I expect that the WG will find that its not an easy requirement to meet - I personally think that it would not be a good use of the WGs time to spend a good chunk of a year trying to get that right Scott (with AD hat on) From owner-ips@ECE.cmu.edu Wed Sep 13 22:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12517; 13 Sep 2000 17:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA19171 for ips-outgoing; Wed, 13 Sep 2000 16:52:27 -0400 (EDT) Received: from mail.wrs.com (unknown-1-11.wrs.com [147.11.1.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA19160 for ; Wed, 13 Sep 2000 16:52:20 -0400 (EDT) Received: (from root@localhost) by mail.wrs.com (8.9.3/8.9.1) id NAA06553; Wed, 13 Sep 2000 13:52:10 -0700 (PDT) Received: from mail02-oak.pilot.net (mail-oak-2.pilot.net [198.232.147.17]) by mail.wrs.com (8.9.3/8.9.1) with ESMTP id VAA08370 for ; Tue, 12 Sep 2000 21:16:44 -0700 (PDT) Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by mail02-oak.pilot.net with ESMTP id VAA03128 for ; Tue, 12 Sep 2000 21:16:56 -0700 (PDT) Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA03431 for ips-outgoing; Tue, 12 Sep 2000 22:53:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA03427 for ; Tue, 12 Sep 2000 22:53:28 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id VAA03445; Tue, 12 Sep 2000 21:53:51 -0500 Message-ID: <39BEEC3F.E95677C4@stewart.chicago.il.us> Date: Tue, 12 Sep 2000 21:53:51 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Scott Bradner CC: ips@ece.cmu.edu, julian_satran@il.ibm.com, ycheng@advansys.com Subject: Re: TCP speed References: <200009130053.UAA06701@newdev.harvard.edu> Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Scott Bradner wrote: > > > I do believe TCP is a wrong protocol for iSCSI. A SCSI request from an > > initiator is inherently acknowledged by its response from a target. > > Therefore, UDP for iSCSI is a better choice. NFS is implemented on UDP. > > the ips working group can decide to do something over UDP but > it should note that the result has to deal with congestion in a TCP-friendly > way - I expect that the WG will find that its not an easy requirement > to meet - I personally think that it would not be a good use of the WGs time > to spend a good chunk of a year trying to get that right > > Scott (with AD hat on) If one were to go down this path (foolishly I might add), I think one year would not be enough time.. two would be closer to the mark.. It is quite a lot of work, it can be both fun and educating, but I do think that IPS has a lot of alternatives besides treading these paths :) R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 13 22:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12528; 13 Sep 2000 17:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA19110 for ips-outgoing; Wed, 13 Sep 2000 16:51:09 -0400 (EDT) Received: from mail.wrs.com (unknown-1-11.wrs.com [147.11.1.11]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA19104 for ; Wed, 13 Sep 2000 16:51:03 -0400 (EDT) Received: (from root@localhost) by mail.wrs.com (8.9.3/8.9.1) id NAA04588; Wed, 13 Sep 2000 13:50:53 -0700 (PDT) Received: from mail01-oak.pilot.net (mail-oak-1.pilot.net [198.232.147.16]) by mail.wrs.com (8.9.3/8.9.1) with ESMTP id VAA06957 for ; Tue, 12 Sep 2000 21:11:48 -0700 (PDT) Received: from ece.cmu.edu (ECE.CMU.EDU [128.2.236.200]) by mail01-oak.pilot.net with ESMTP id VAA15915 for ; Tue, 12 Sep 2000 21:12:00 -0700 (PDT) Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA03849 for ips-outgoing; Tue, 12 Sep 2000 23:07:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA03837 for ; Tue, 12 Sep 2000 23:07:29 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id WAA03474; Tue, 12 Sep 2000 22:07:53 -0500 Message-ID: <39BEEF89.3B02BBA@stewart.chicago.il.us> Date: Tue, 12 Sep 2000 22:07:53 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: David Robinson CC: ips@ece.cmu.edu, dotis@sanlight.net Subject: Re: Avoiding deadlock in iSCSI References: <200009121807.LAA29970@ha10nwk.EBay.Sun.COM> Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David: Question for you... Your reply leads me to believe a tight coupling exists between the iSCSI layer and the TCP stack. I.e. iSCSI is going in and "tweaking on a dynamic basis" the TCP implemenations rwnd. Other email that has crossed on this subject implies that "a TCP implementation is broken if it offers more buffers than it has"... This in some ways contradicts iSCSI controlling TCP rwnd and TCP controlling its own rwnd... could you please clearify this for me? Is it TCP in control of rwnd or is there a tight coupling here? Thanks R David Robinson wrote: > > > As you can not predict latency on two TCP connections, there would be no > > assurance of delivery order. A skew buffer would seem a requirement. > > If I understand your definition of a "skew buffer", some place to > hold the data until the command arrives. This buffer is simply > the TCP receive buffer if we require data to be sent on order. > Any TCP implementation that offers a total window size (sum of > windows on all connections) greater than available buffer space > is just plain broken, so this is not an issue. > > > Each TCP stream can deliver sequential data, but not within multiple > > streams. As such, sequential delivery goes out the window with respect to > > aggregation. As far as the wire, TCP is part of the Upper Layer Protocol. > > Yes each connection maintains data ordering, and if we also require > the all data is sent in order, the early arrival on one connection > or another does not change the ordering. The receiver simply maintains > the data buffer and closes the TCP window if flow control is needed > until the command arrives, on the command connection, to be processed. > Again not an issue. > > > The balance between commands and data are not within the control of the > > target via flow-control. As such, either a data or command resource may > > become exhausted. At some point, either data or commands may be stopped > > without necessarily stopping TCP. The means for stopping a command is Check > > Condition, and for data, discarding. > > With multiple data connections, any well implemented target will not > open the total window size greater than the available buffers. Once > a given connection uses up its window it is flow controlled by TCP > and no more data is sent. Because the commands are on a seperate > connection and we require data ordering, the commands will make > progress either using immediate data, or reading data from a > data connection that is guarenteed to be available or will be available. > Commands should never need to be stopped by iSCSI and neither should > the data connections, TCP flow control should be sufficient. > > There may be rare exceptional conditions where an initiator or target > fails to follow the protocol (bug), drops a connection, or other rare > event. The the recovery mechanism should simply handle this. > > > Should there be a TCP connection per LUN, the number of TCP connections > > would be large if a controller is sitting on 48 LUNs. With asymmetrical > > connections that would imply 96 TCP sessions per client possible plus fail > > over connections. On the network, TCP shares on a session basis. This > > would mean a device with a single connection on the same network would then > > enjoy only 1% of the bandwidth. This says nothing about TCP overhead. > > Although I like a connection per LUN, the WG seems to be in concensus that > the protocol allows multiple LUNs per connection which if you choose > can implement just one. But there is no requirement to restrict it > to just one. With one connection per LUN there is very little > reason to have a seperate data connection and instead it is > best to simply provide all the data immediately after the command. > Having seperate data connections allows better concurrency when > commands are for many LUNs, the concurrency for a single LUN is > like to be low, if not sequential in the common case, so extra data > connections are not necessary. > > With either a connection per LUN or all LUNs on one connection, if a > connection can consume 100% of the bandwidth of the link, with 100 > LUNs each will only get 1% of the bandwidth. The only difference is > if the multiplexing is done at the TCP layer or the iSCSI layer, > the complexity and overhead is the same (modulo a small constant). > The only difference is that TCP already has the mux/demux capability > and we will have to add it to iSCSI. I prefer the simplicity of > leaving it to TCP, but lost the battle, but it is simply not a big > enough issue to worry about. > > > The iSCSI means of limiting the amount of data presented in an unsolicited > > fashion is to discard. For a given amount of buffer space, the number of > > commands associated with this space is unknown. The overhead for staging > > these commands would add an additional overhead on a command basis not a > > data basis. As such, it would be like adding water to a box of rice. Within > > a margin that stays out of trouble, how much of the buffer would you be > > wasting to handle all situations? What if you oops due to the latency of > > responding? > > Unlike a datagram protocol where you accept either all of the data or > none of it, a reliable stream allows you to accept as much or as > little as the receiver desires. You never overflow because you > never advertise a TCP window larger than your available buffer space. > With N connections you never allow any one connection to consume > more than 1/N of the buffer space. You may waste buffer space but > memory is cheap and you can control how many connections you allow. > Latency is not an issue, if data arrives before a command it > is simply flow controlled, in a really bizarre implementation with > minimal buffering you could hold all data TCP windows to zero > until a command arrives that indicates its data is on that connection. > I perverse method of implementing RTT/R2T but it shows that the scheme > works in the extreme. > > -David > -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 13 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13099; 13 Sep 2000 19:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23569 for ips-outgoing; Wed, 13 Sep 2000 18:37:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23561 for ; Wed, 13 Sep 2000 18:37:38 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id 5DFB8DCF for ; Wed, 13 Sep 2000 15:37:31 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id PAA27115 for ; Wed, 13 Sep 2000 15:37:18 -0700 (PDT) Received: from agilent.com ([15.3.253.134]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA6D54 for ; Wed, 13 Sep 2000 15:37:14 -0700 Message-ID: <39C001BC.C5ADD793@agilent.com> Date: Wed, 13 Sep 2000 15:37:48 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: <39BF3F79.23429EBE@hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre All you have done is to suggest an alternative method of "re-ordering" commands that come down different tcp connections. I don't see how it has any advantages over the existing symmetric model. -Matt Pierre Labat wrote: > From the discussion going on the symmetric versus asymmetric model and > dead lock avoidance, here is a proposal that tries to combine (as far > as possible...) > the advantages of both models. > > The proposal is a slightly modified symmetric model where > the symmetric model incorporate an idea of the current > asymmetric model. > > This idea is that the initiator indicates to the target > on which connection the next command in order arrives. > The asymmetric model does that using an extra connection > transporting the commands, and for each command in the header > there is the connection number where the data will come. > > Description of the proposal: > --------------------- > In the command header a field is added. This field > "next connection" indicates the TCP connection (connection id) > in the session on which the next command will arrive. > This field builds a linked list of commands across the several > symmetric connections of the session. > > The first time the initiator posts a command, it determines > the connection on which it post the command and the connection > on which it will post the next command. > Then for each command the initiator determines only > the connection it will use to post the next command. > For each command, the connexion id of the next command is added in the > header > field "next connection" of each command. > > Doing that, leads to the same algorithm as in the asymmetric model > to process the commands on the target side: > - a thread of execution pulls the cmd/data in order from the TCP > windows. After pulling a cmd/data, it looks at the "next connection" > field and move to the next connection to pull a new cmd/data. > There is no need of extra storage out of the TCP windows > and no need to re-order. In the asymmetric model the thread > empties the command connection, here it follows the chain of commands. > > The advantage of this proposal are: > - on the target, as opposed to the original symmetric model, it avoids > the re-ordering process of the commands once extracted > off the TCP window. > - it keeps cmd/data/status on a same connexion avoiding > the synchronization penalty for the READS on the initiator (described > by Somesh Gupta) > - it saves one TCP connection compared to the asymmetric model. > > Anyway any ordering implementation adds overhead, thus, as in most > cases (disks) it is not required, It would be possible to deactivate it. > > Regards, > > Pierre From owner-ips@ECE.cmu.edu Wed Sep 13 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13101; 13 Sep 2000 19:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23632 for ips-outgoing; Wed, 13 Sep 2000 18:40:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23621 for ; Wed, 13 Sep 2000 18:39:54 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id PAA25603; Wed, 13 Sep 2000 15:38:16 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Wed, 13 Sep 2000 15:38:14 -0700 Message-ID: From: Jim McGrath To: "'Greg Finn'" , rdv@Network-Alchemy.COM Cc: chengann@singnet.com.sg, ips@ece.cmu.edu Subject: RE: patent question Date: Wed, 13 Sep 2000 15:38:12 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Usually patents are hard to enforce for general claims, much easier for specific claims. I would just make sure to avoid specific algorithms/formats, etc.. in the patent, since they are usually the more defendable (if the filers did their job right). Note that just talking about iSCSI in general will not constitute prior art for some of these more specific claims - but then again they usually prove easy to work around if you are aware of them ahead of time. Jim -----Original Message----- From: Greg Finn [mailto:finn@ISI.EDU] Sent: Tuesday, September 12, 2000 1:47 PM To: rdv@Network-Alchemy.COM Cc: chengann@singnet.com.sg; ips@ece.cmu.edu Subject: Re: patent question > The submission date on the patent, btw, is Jan. 14, 1998, and it was > issued Nov. 30, 1999. WRT that patent IP/SCSI would appear not to be an issue. The idea was openly discussed in the community in 1997 and probably much earlier. --- ggf From owner-ips@ECE.cmu.edu Wed Sep 13 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13092; 13 Sep 2000 19:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA24111 for ips-outgoing; Wed, 13 Sep 2000 18:57:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from tnt.isi.edu (tnt.isi.edu [128.9.128.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA24105 for ; Wed, 13 Sep 2000 18:56:57 -0400 (EDT) Received: from moo.isi.edu (moo.isi.edu [128.9.160.187]) by tnt.isi.edu (8.8.7/8.8.6) with ESMTP id PAA06002; Wed, 13 Sep 2000 15:56:48 -0700 (PDT) Received: (from finn@localhost) by moo.isi.edu (8.8.7/8.8.6) id PAA13521; Wed, 13 Sep 2000 15:56:48 -0700 (PDT) Date: Wed, 13 Sep 2000 15:56:48 -0700 (PDT) Message-Id: <200009132256.PAA13521@moo.isi.edu> From: Greg Finn To: Jim.McGrath@quantum.com CC: rdv@Network-Alchemy.COM, chengann@singnet.com.sg, ips@ece.cmu.edu In-reply-to: (message from Jim McGrath on Wed, 13 Sep 2000 15:38:12 -0700) Subject: Re: patent question Sender: owner-ips@ece.cmu.edu Precedence: bulk Work on iSCSI was underway in 1997 in my ARPA project and perhaps earlier. Open publication of the idea of a SCSI net interface was published in a 1996 conference proceedings by our project and probably by others earlier. Specific patent claims of course I have no knowledge of. A general claim would fall afoul of obvious-to-the-field by 1998. iSCSI has already been a common and openly debated topic and iSCSI overhead work had already been published. From owner-ips@ECE.cmu.edu Thu Sep 14 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14206; 13 Sep 2000 23:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA29106 for ips-outgoing; Wed, 13 Sep 2000 21:44:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from yogi.ece.cmu.edu (IDENT:root@YOGI.ECE.CMU.EDU [128.2.252.128]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA29102 for ; Wed, 13 Sep 2000 21:44:45 -0400 (EDT) Received: (from bassoon@localhost) by yogi.ece.cmu.edu (8.8.7/8.8.7) id VAA21338; Wed, 13 Sep 2000 21:44:44 -0400 Message-Id: <200009140144.VAA21338@yogi.ece.cmu.edu> To: ips@ece.cmu.edu Subject: another question (was Re: patent question) Date: Wed, 13 Sep 2000 21:44:44 -0400 From: Dave Nagle Sender: owner-ips@ece.cmu.edu Precedence: bulk Really from From: cheng-ann tan Reply-To: chengann@singnet.com.sg thanks a lot for the clarification. i have a better idea now about the criteria of prior art. but isn't it strange that the patent was granted anyhow? i have another quick question. for a local area application, why hasn't anyone done scsi over ethernet? (ie encapsulating scsi commands into the ethernet package). why use a higher level protocol and work with all the latency and overhead problem? any insight? - -cheng ------- End of Forwarded Message From owner-ips@ECE.cmu.edu Thu Sep 14 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14985; 14 Sep 2000 1:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA03733 for ips-outgoing; Thu, 14 Sep 2000 00:33:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA03727 for ; Thu, 14 Sep 2000 00:33:14 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 14 Sep 2000 00:32:42 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F83@corpmx9.isus.emc.com> To: bassoon@yogi.ece.cmu.edu, ips@ece.cmu.edu Subject: RE: another question (was Re: patent question) Date: Thu, 14 Sep 2000 00:32:42 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > i have another quick question. for a local area application, > why hasn't anyone done scsi over ethernet? (ie encapsulating > scsi commands into the ethernet package). why use a higher > level protocol and work with all the latency and overhead problem? Check the list archives for info on an Adaptec demo of this, and I'm sure Paul von Stamwitz can provide additional information (off the list, please). Encapsulating directly on Ethernet runs into scaling problems - the first time one needs to get off the LAN or VLAN that the traffic started on, one discovers that an IP header is an immensely useful thing to have. Assuming that the "latency and overhead" comment is about TCP, I suggest reading: http://www.ietf.org/internet-drafts/draft-floyd-cong-04.txt A lot of lessons about congestion control have been learned the hard way; as has already been indicated a number of times, congestion control is REQUIRED in protocols standardized by this WG. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Sep 14 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14986; 14 Sep 2000 1:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA04027 for ips-outgoing; Thu, 14 Sep 2000 00:45:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA04020 for ; Thu, 14 Sep 2000 00:45:01 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id GAA296146 for ; Thu, 14 Sep 2000 06:44:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id GAA25216 for ; Thu, 14 Sep 2000 06:44:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695A.001A0A32 ; Thu, 14 Sep 2000 06:44:25 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 14 Sep 2000 07:41:44 +0300 Subject: RE: Comments on the Draft Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Howard, The status info has traditionaly being terse (1 byte with flags indicating the exception class - like busy, unit check, controller check etc.) and the sense detailing it even there at two layers. It made also for a lower footprint software with modules handling a specific exception being brought-in only when needed. I think we would want to keep this type of structure. Julo "Hall, Howard" on 13/09/2000 22:22:56 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Comments on the Draft On issue 2 its not the sense issue, it is the argument that: Since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (eg. not_logged_on, data_in_timeout, buffer_address etc..) So, for example, in section 2.2.3 it states "Any message sent on a TCP connection before this connection gets into full feature phase at the initiator should be rejected by the initiator. A message reaching a target on a TCP connection before the full feature phase will be rejected with an iSCSI check condition." Why not send an iSCSI status that is more descriptive like not_logged_on? -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 1:57 PM To: ips@ece.cmu.edu Subject: RE: Comments on the Draft On 1 - depends on where thing go with asymmetric vs. symmetric. 2 - Explicit error codes - isn't that the sense function? 3 - For the flag - probably yes- but I am not sure. Status implies it so you need it only if status is bad - which can be before the end. Julo "Hall, Howard" on 13/09/2000 20:50:10 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Comments on the Draft Julo, Thank you very much for your thoughtful response. What do you think of some of the other issues (reference the original email): - whether CmdRN, MaxCmdRN, etc. should be used for in-order delivery of commands across multiple TCP connections in a session (Comments for Section 2.2.2). - more explicit error codes - the proposed flag described at the end of the email -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 4:52 AM To: Hall, Howard Subject: Re: Comments on the Draft 1. General - Loss of synchronization - we hope to be able one day to get to a better solution but I would suggest to recommend the implementer to drop the connections as it won't be able to parse a reset too, reinstate them and then try a reset to distinguish it clearly from other events. If it happens in a target - drop - and wait for a reconnect then send an AE to indicate the event and wait for reset. As for inconsistencies - I've added your proposal to the todo list - thank you. 2. The view is a qualifier for the name - it allows a target to a second level of refinement about what he things it should show the initiator. The first part of the name itself is the first level - but for those that are conservative in the use of names they are left with the view. 3. We structured it to follow closely FCP - for bridging reasons. As far as I can recall Response comes first followed by sense. We will state it explicitly in the document thanks. 4.timeouts - yes you are right and I stated it already in an older note about bridging. We where thinking about three timers: - command delivery - data delivery - status delivery Those should be set at the session initiation and be known to the initiator and target but they don't have to be necessarily sent over the wire (as they imply mostly local actions in the endpoints). 5. immediate data length negotiation - will fix the negotiation parameters - thanks. We think also about limiting it by default to 64k(?) 6. iSCSI check - the reason I choose to make it SCSI check too is to get a "hook" into ACA behavior. I think that is mandatory - without it all hell breaks loose! Both initiator and target could be built the with ACA on check condition. We can remove the iSCSI status field and add a set of basic sense bytes detailing the error. Let us keep talking on this item. 7. The security parts are being rewritten just as we talk and we hope that we will all see them soon 8. Target reset will be changed to have two types - soft and hard reset. 9. closing connection. Yes you are right it needs more text. And we already agreed to add a logout (on a different connection to force connection close) and remove the RID from login Thanks for your careful reading and hope to see you on row 1 or 2 at the next IETF meeting, Julo "Hall, Howard" on 13/09/2000 02:12:38 Please respond to "Hall, Howard" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Comments on the Draft Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Thu Sep 14 06:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15378; 14 Sep 2000 2:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA05144 for ips-outgoing; Thu, 14 Sep 2000 01:36:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA05140 for ; Thu, 14 Sep 2000 01:36:43 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 14 Sep 2000 01:36:11 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F88@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI: T10 meeting Date: Thu, 14 Sep 2000 01:36:10 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk I spent today at the T10 meeting, despite United Airlines' best attempts to keep me out of the Pacific time zone :-). I did quite a bit of explaining about IETF and ips logistics, procedures, and status, but did manage to squeeze in some technical discussion. Here are a few items of relevance to iSCSI: - As indicated earlier, T10 would like to see iSCSI (and other encapsulations of SCSI) follow the data formats used by FCP to the extent possible and reasonable. Departures for good functional reasons are ok (e.g., iSCSI separates out task management in a way that FCP does not, and there may be good reasons to do that), even though they complicate bridges. OTOH, format differences that lack a functional reason should be avoided. The recent proposal by Charles Monia to harmonize iSCSI and FCP packet formats deserves serious attention. - There has been some discussion of stateless bridges or gateways on this list in the past. T10's view of this area is that such devices tend to be inherently stateful because they usually have to keep state related to proxying multiple initiators accessing them onto a single initiator that accesses the actual target. Mapping command tags as part of this is not a big deal, and hence T10 does not attempt to mandate a single SCSI tag format -- in fact a 64 bit tag for SCSI over VI was recommended as part of today's meeting. The rationale for that tag may be worth considering for iSCSI - an initiator running a 64 bit OS can stick a memory address in the tag, which makes it easier to handle responses. In any case, it's not important to match FCP's tag structure exactly. - I want to reinforce Steve Byan's earlier email on multiple ordered commands and resource exhaustion. As indicated by words in our (still draft) charter, this sort of issue expressible entirely in SCSI w/o reference to TCP or any other transport is in T10's domain. That means that not only is it T10's responsibility to deal with this issue, it is also T10's responsibility to determine whether the issue is serious enough to merit any changes. This IETF WG should be cautious about redesigning things like ACA - it would be undesirable for ACA to behave differently in iSCSI than in other SCSI transports. >From my viewpoint, a good working relationship with T10 is well underway. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Sep 14 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15588; 14 Sep 2000 3:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA06113 for ips-outgoing; Thu, 14 Sep 2000 02:29:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA06109 for ; Thu, 14 Sep 2000 02:29:15 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 38DC35D4 for ; Wed, 13 Sep 2000 23:29:14 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id XAA14550; Wed, 13 Sep 2000 23:29:12 -0700 (PDT) Message-ID: <39C071C4.2BF66C7E@hp.com> Date: Wed, 13 Sep 2000 23:35:49 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: Julo, > Pierre, > > Sorry (and excuses to Kalman) - I misread you. > I think that the idea behind the symmetric model is to stripe the traffic > command-by-command. > You could hardly ask that an initiator make decisions based only on the > past (as your recommendation would imply) and without examining some of the > command parameters. It is a "recent" past, for one command. That means that the N+1 command is posted on a connexion based on a state of the connexions established after the posting of the command number N. Hence the context of the connections regarding the cmd/data enqueued has not changed a lot. You can examine the command parameters to maintain a "state" of each connexion (how much data/command queued). But as you say even if the initiator knows the "state" of each connexion, it must take a decision only based on that and not knowing the nature of the next command. > > > The what if between this command and the next the connection goes away? You get the same kind of problem when the command connexion of the asymmetric model goes away. You need to restart the commands from the last command completed in order. On this point i think there is no difference between this model and the asymmetric one. > > > And on the target - with your scheme you can do only one command read at a > time (no parallel operation of the pipes) admittedly not that disastrous. > You can do the same thing you do with the asymmetric model. You can read ahead some commands following the chain and processing them in parallel. It is the same thing as you can do dequeuing several READS from the command connection and processing them in parallel. In fact the chain of command materializes in an other way the queue of commands you have in the command connection tcp window of the asymmetric model. Hence on this point i think there is no difference. > > How about chaining back (each command indicates where the previous came > from)? > > Will it work better? (probably not - all relative schemes are bad when they > loose sync.) Well, the command connexion materializes too, a relative scheme. The relation is the order of the commands in the TCP window of this connexion. If you loose this connexion you are in a bad shape too. > > Can you try? (I am on the road and in a hurry - that's why I missed your > context). I'll try. Regards, Pierre > > > We will probably polish Kalman's proposal to ease en-of-data and status > sync - but the essence > is that the parties know ahead of time where the data will be coming. > > Regards, > Julo > > > > > > Pierre Labat on 13/09/2000 11:48:58 > > > > Please respond to Pierre Labat > > > > To: ips@ece.cmu.edu > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: RE: Avoiding deadlock in iSCSI > > > > >From the discussion going on the symmetric versus asymmetric model and > > dead lock avoidance, here is a proposal that tries to combine (as far > > as possible...) > > the advantages of both models. > > > > The proposal is a slightly modified symmetric model where > > the symmetric model incorporate an idea of the current > > asymmetric model. > > > > This idea is that the initiator indicates to the target > > on which connection the next command in order arrives. > > The asymmetric model does that using an extra connection > > transporting the commands, and for each command in the header > > there is the connection number where the data will come. > > > > Description of the proposal: > > --------------------- > > In the command header a field is added. This field > > "next connection" indicates the TCP connection (connection id) > > in the session on which the next command will arrive. > > This field builds a linked list of commands across the several > > symmetric connections of the session. > > > > The first time the initiator posts a command, it determines > > the connection on which it post the command and the connection > > on which it will post the next command. > > Then for each command the initiator determines only > > the connection it will use to post the next command. > > For each command, the connexion id of the next command is added in the > > header > > field "next connection" of each command. > > > > Doing that, leads to the same algorithm as in the asymmetric model > > to process the commands on the target side: > > - a thread of execution pulls the cmd/data in order from the TCP > > windows. After pulling a cmd/data, it looks at the "next connection" > > field and move to the next connection to pull a new cmd/data. > > There is no need of extra storage out of the TCP windows > > and no need to re-order. In the asymmetric model the thread > > empties the command connection, here it follows the chain of commands. > > > > The advantage of this proposal are: > > - on the target, as opposed to the original symmetric model, it avoids > > the re-ordering process of the commands once extracted > > off the TCP window. > > - it keeps cmd/data/status on a same connexion avoiding > > the synchronization penalty for the READS on the initiator (described > > by Somesh Gupta) > > - it saves one TCP connection compared to the asymmetric model. > > > > Anyway any ordering implementation adds overhead, thus, as in most > > cases (disks) it is not required, It would be possible to deactivate it. > > > > Regards, > > > > Pierre From owner-ips@ECE.cmu.edu Thu Sep 14 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15630; 14 Sep 2000 3:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA06337 for ips-outgoing; Thu, 14 Sep 2000 02:36:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA06329 for ; Thu, 14 Sep 2000 02:35:57 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id XAA10434; Wed, 13 Sep 2000 23:25:32 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id XAA18391; Wed, 13 Sep 2000 23:23:12 -0700 From: "Douglas Otis" To: "Dave Nagle" , Subject: RE: another question (was Re: patent question) Date: Wed, 13 Sep 2000 23:38:19 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <200009140144.VAA21338@yogi.ece.cmu.edu> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Cheng, TCP/IP was developed outside normal device interfaces but made to approximate standard file I/O. This IP interface uses buffers in a clumsy manner and requires examination for things like checksums. Unfortunately IP does not trust hardware, perhaps due to the Telco environment. Letting a few bits rot on things like DSLAM (Digital Subscriber Line Access Multiplexor) does not cause excitement. The concept is with enough hardware, you can make a file (byte stream) interface look like a SCSI (block) interface to then have the OS transform that back to a byte stream again. Why? SCSI is the storage interface and yet the traditional means of sending anything in a friendly manner over IP is by TCP. On that, there is agreement. Why a block interface at the storage device? To abstract storage into variable length objects requires about 1% ram to hard space to contain a robust incremental extent, variable name space, and expansive permissions all involved in high level real-time decisions. But this cheap ram is still 100 times the expense of hard space so you have doubled cost of storage and effective use of ram requires close proximity to the CPUs due to nothing more than the speed of light. Within the OS there is massive bandwidth between threads making a final decision if to write/read or hold in cache all based on these highly dynamic abstractions. Once past this decision process, the bandwidth to the block device is tolerable with a simple block offset and length request. SCSI over Ethernet is not an innovation, it is the bigger hammer in hopes to use the now ubiquitous IP. Some disagree with keeping storage interface a simple block. See http://www.nsic.org/nasd/1998-jun/oodreq3.pdf along with clustering. When a dominate OS representative was asked how would you recover from a cluster error, "Reset the system." was not a surprising answer with all this intelligence getting tangled in chaos. But how would you be sure of obtaining the correct object after loss of system coherency? "Our customers would rather have bad data than no data." Again, not a surprise. Placing real-time decisions on the net with back and forth open this, close that does not seem like an innovation to justify a change from something perhaps a bit more predictable. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Dave Nagle > Sent: Wednesday, September 13, 2000 6:45 PM > To: ips@ece.cmu.edu > Subject: another question (was Re: patent question) > > > Really from > From: cheng-ann tan > Reply-To: chengann@singnet.com.sg > > thanks a lot for the clarification. > i have a better idea now about the criteria of > prior art. but isn't it strange that the patent was granted > anyhow? > > i have another quick question. for a local area application, > why hasn't anyone done scsi over ethernet? (ie encapsulating > scsi commands into the ethernet package). why use a higher > level protocol and work with all the latency and overhead problem? > > any insight? > > - -cheng > > ------- End of Forwarded Message > From owner-ips@ECE.cmu.edu Thu Sep 14 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15570; 14 Sep 2000 3:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA06694 for ips-outgoing; Thu, 14 Sep 2000 02:56:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA06689 for ; Thu, 14 Sep 2000 02:56:15 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 3BDA34BC for ; Wed, 13 Sep 2000 23:56:15 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id XAA14870; Wed, 13 Sep 2000 23:56:13 -0700 (PDT) Message-ID: <39C0781A.65E80F3E@hp.com> Date: Thu, 14 Sep 2000 00:02:50 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: <39BF3F79.23429EBE@hp.com> <39C001BC.C5ADD793@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt Wakeley wrote: > Pierre > > All you have done is to suggest an alternative method of "re-ordering" > commands that come down different tcp connections. I don't see how it has any > advantages over the existing symmetric model. Matt, The advantage is on the target side. The target doesn't need extra storage to re-order commands. I mean by "extra" storage that is not allocated to the TCP windows. In the symmetric model as described in the draft, the target needs to - read commands/data from the TCP connexions store them in a temporary location, - continue to do that till the "holes" between the out of order commands are filled - then process the commands Hence the target needs to manage this extra space that is difficult to size because it depends on the degree of "out-of-order" the commands arrive. In the proposal below the target can process the command as soon as it extracts it from the TCP window. It seems to me that avoiding this work to the target could help it. But i am not an expert in target, hence may be this re-ordering is not a problem and needs only extra cheap memory? Regards, Pierre > > > -Matt > > Pierre Labat wrote: > > > From the discussion going on the symmetric versus asymmetric model and > > dead lock avoidance, here is a proposal that tries to combine (as far > > as possible...) > > the advantages of both models. > > > > The proposal is a slightly modified symmetric model where > > the symmetric model incorporate an idea of the current > > asymmetric model. > > > > This idea is that the initiator indicates to the target > > on which connection the next command in order arrives. > > The asymmetric model does that using an extra connection > > transporting the commands, and for each command in the header > > there is the connection number where the data will come. > > > > Description of the proposal: > > --------------------- > > In the command header a field is added. This field > > "next connection" indicates the TCP connection (connection id) > > in the session on which the next command will arrive. > > This field builds a linked list of commands across the several > > symmetric connections of the session. > > > > The first time the initiator posts a command, it determines > > the connection on which it post the command and the connection > > on which it will post the next command. > > Then for each command the initiator determines only > > the connection it will use to post the next command. > > For each command, the connexion id of the next command is added in the > > header > > field "next connection" of each command. > > > > Doing that, leads to the same algorithm as in the asymmetric model > > to process the commands on the target side: > > - a thread of execution pulls the cmd/data in order from the TCP > > windows. After pulling a cmd/data, it looks at the "next connection" > > field and move to the next connection to pull a new cmd/data. > > There is no need of extra storage out of the TCP windows > > and no need to re-order. In the asymmetric model the thread > > empties the command connection, here it follows the chain of commands. > > > > The advantage of this proposal are: > > - on the target, as opposed to the original symmetric model, it avoids > > the re-ordering process of the commands once extracted > > off the TCP window. > > - it keeps cmd/data/status on a same connexion avoiding > > the synchronization penalty for the READS on the initiator (described > > by Somesh Gupta) > > - it saves one TCP connection compared to the asymmetric model. > > > > Anyway any ordering implementation adds overhead, thus, as in most > > cases (disks) it is not required, It would be possible to deactivate it. > > > > Regards, > > > > Pierre From owner-ips@ECE.cmu.edu Thu Sep 14 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18182; 14 Sep 2000 12:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA27016 for ips-outgoing; Thu, 14 Sep 2000 11:06:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA27011 for ; Thu, 14 Sep 2000 11:06:50 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id RAA187644 for ; Thu, 14 Sep 2000 17:06:18 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA51162 for ; Thu, 14 Sep 2000 17:06:03 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695A.0052F083 ; Thu, 14 Sep 2000 17:05:55 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 14 Sep 2000 17:05:21 +0300 Subject: Re: iSCSI: 2.2.6. Naming & mapping Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Not again (what is the sign for frustration?)... I mean not before the next version. Julo Raghavendra Rao on 14/09/2000 21:04:02 Please respond to Raghavendra Rao To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI: 2.2.6. Naming & mapping I feel that a LUN should be very much part of the Naming scheme that has been proposed in the draft, regardless of the level of enforcement of the scheme by implementors. LUNs are very much part of SCSI addressing, and where LUN doesn't exist, zero is assumed. Without a LUN identifier, Naming section doesn't look complete. Do you have insights why it is omitted ? Is this by design or overlook ? Thanks. -JP From owner-ips@ECE.cmu.edu Thu Sep 14 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18189; 14 Sep 2000 12:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA27006 for ips-outgoing; Thu, 14 Sep 2000 11:06:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA27002 for ; Thu, 14 Sep 2000 11:06:37 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id RAA187640 for ; Thu, 14 Sep 2000 17:06:02 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA51160 for ; Thu, 14 Sep 2000 17:06:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695A.0052F122 ; Thu, 14 Sep 2000 17:05:56 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 14 Sep 2000 17:12:45 +0300 Subject: Re: iSCSI: Creation of new LUNs Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Is somebody else doing it differently? Julo Raghavendra Rao on 14/09/2000 21:19:58 Please respond to Raghavendra Rao To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI: Creation of new LUNs What is the recommended/approved way of announcing the availability of new LUNs in the target ? Currently I can do the following, but if it isn't explicitly documented, implementors tend to do their own things (causing interoperability problems) Upon creation/deletion of a LUN in the target, a target shall generate an asynchronous notification with the SCSI event indicator set to either 2 or 4, followed by sense data with appropriate ASC and ASCQ indicating that LUNs have changed. The initiator, on receipt of this notification will issue a REPORT_LUN command to get a list of LUNs. -JP From owner-ips@ECE.cmu.edu Thu Sep 14 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18180; 14 Sep 2000 12:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA28420 for ips-outgoing; Thu, 14 Sep 2000 11:36:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA28414 for ; Thu, 14 Sep 2000 11:36:25 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA37992 for ; Thu, 14 Sep 2000 17:35:53 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA67646 for ; Thu, 14 Sep 2000 17:35:52 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695A.0055ADA4 ; Thu, 14 Sep 2000 17:35:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 14 Sep 2000 18:22:57 +0300 Subject: RE: Comments on status responses Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Will add to my list of to consider items. Thanks, Julo "Merhar, Milan" on 14/09/2000 17:12:59 Please respond to "Merhar, Milan" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Comments on status responses Julo, I agree with you that status returned from a SCSI target device needs to be transported exactly, so that existing drivers, etc. (which are unaware of the intervening iSCSI transport) work as expected. But, I think the question here concerns status reported by the iSCSI layer itself. Basically, my opinion is that if we expect the other end's error handler to be something like if ( errno ) then abort; it makes little or no difference what we return; 'OK' = 0 and 'ERR' = 1 works as well as an elaborate list of nonzero error codes. But, if there's a chance the other end could implement a more robust and/or less disruptive error recovery procedure, it makes sense to pass some "hints" along, through a broader set of exception codes. As an example, I suggest one might want to create a client-side recovery procedure that handles a storage device response timeout differently, depending on whether it was caused by a target too busy to process the command in a timely fashion, versus a timeout caused by excessive network congestion or repeated TCP error retrys. - milan -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 1:57 PM To: ips@ece.cmu.edu Subject: RE: Comments on the Draft On 1 - depends on where thing go with asymmetric vs. symmetric. 2 - Explicit error codes - isn't that the sense function? 3 - For the flag - probably yes- but I am not sure. Status implies it so you need it only if status is bad - which can be before the end. Julo "Hall, Howard" on 13/09/2000 20:50:10 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Comments on the Draft Julo, Thank you very much for your thoughtful response. What do you think of some of the other issues (reference the original email): - whether CmdRN, MaxCmdRN, etc. should be used for in-order delivery of commands across multiple TCP connections in a session (Comments for Section 2.2.2). - more explicit error codes - the proposed flag described at the end of the email -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 4:52 AM To: Hall, Howard Subject: Re: Comments on the Draft 1. General - Loss of synchronization - we hope to be able one day to get to a better solution but I would suggest to recommend the implementer to drop the connections as it won't be able to parse a reset too, reinstate them and then try a reset to distinguish it clearly from other events. If it happens in a target - drop - and wait for a reconnect then send an AE to indicate the event and wait for reset. As for inconsistencies - I've added your proposal to the todo list - thank you. 2. The view is a qualifier for the name - it allows a target to a second level of refinement about what he things it should show the initiator. The first part of the name itself is the first level - but for those that are conservative in the use of names they are left with the view. 3. We structured it to follow closely FCP - for bridging reasons. As far as I can recall Response comes first followed by sense. We will state it explicitly in the document thanks. 4.timeouts - yes you are right and I stated it already in an older note about bridging. We where thinking about three timers: - command delivery - data delivery - status delivery Those should be set at the session initiation and be known to the initiator and target but they don't have to be necessarily sent over the wire (as they imply mostly local actions in the endpoints). 5. immediate data length negotiation - will fix the negotiation parameters - thanks. We think also about limiting it by default to 64k(?) 6. iSCSI check - the reason I choose to make it SCSI check too is to get a "hook" into ACA behavior. I think that is mandatory - without it all hell breaks loose! Both initiator and target could be built the with ACA on check condition. We can remove the iSCSI status field and add a set of basic sense bytes detailing the error. Let us keep talking on this item. 7. The security parts are being rewritten just as we talk and we hope that we will all see them soon 8. Target reset will be changed to have two types - soft and hard reset. 9. closing connection. Yes you are right it needs more text. And we already agreed to add a logout (on a different connection to force connection close) and remove the RID from login Thanks for your careful reading and hope to see you on row 1 or 2 at the next IETF meeting, Julo "Hall, Howard" on 13/09/2000 02:12:38 Please respond to "Hall, Howard" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Comments on the Draft Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Thu Sep 14 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18472; 14 Sep 2000 13:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA29807 for ips-outgoing; Thu, 14 Sep 2000 12:10:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA24312 for ; Thu, 14 Sep 2000 10:01:09 -0400 (EDT) Message-ID: <618471D08ABDD31188B6009027E5009352A0E2@apollo.pirus.com> From: "Merhar, Milan" To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: Comments on status responses Date: Thu, 14 Sep 2000 10:12:59 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, I agree with you that status returned from a SCSI target device needs to be transported exactly, so that existing drivers, etc. (which are unaware of the intervening iSCSI transport) work as expected. But, I think the question here concerns status reported by the iSCSI layer itself. Basically, my opinion is that if we expect the other end's error handler to be something like if ( errno ) then abort; it makes little or no difference what we return; 'OK' = 0 and 'ERR' = 1 works as well as an elaborate list of nonzero error codes. But, if there's a chance the other end could implement a more robust and/or less disruptive error recovery procedure, it makes sense to pass some "hints" along, through a broader set of exception codes. As an example, I suggest one might want to create a client-side recovery procedure that handles a storage device response timeout differently, depending on whether it was caused by a target too busy to process the command in a timely fashion, versus a timeout caused by excessive network congestion or repeated TCP error retrys. - milan -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 1:57 PM To: ips@ece.cmu.edu Subject: RE: Comments on the Draft On 1 - depends on where thing go with asymmetric vs. symmetric. 2 - Explicit error codes - isn't that the sense function? 3 - For the flag - probably yes- but I am not sure. Status implies it so you need it only if status is bad - which can be before the end. Julo "Hall, Howard" on 13/09/2000 20:50:10 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Comments on the Draft Julo, Thank you very much for your thoughtful response. What do you think of some of the other issues (reference the original email): - whether CmdRN, MaxCmdRN, etc. should be used for in-order delivery of commands across multiple TCP connections in a session (Comments for Section 2.2.2). - more explicit error codes - the proposed flag described at the end of the email -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 4:52 AM To: Hall, Howard Subject: Re: Comments on the Draft 1. General - Loss of synchronization - we hope to be able one day to get to a better solution but I would suggest to recommend the implementer to drop the connections as it won't be able to parse a reset too, reinstate them and then try a reset to distinguish it clearly from other events. If it happens in a target - drop - and wait for a reconnect then send an AE to indicate the event and wait for reset. As for inconsistencies - I've added your proposal to the todo list - thank you. 2. The view is a qualifier for the name - it allows a target to a second level of refinement about what he things it should show the initiator. The first part of the name itself is the first level - but for those that are conservative in the use of names they are left with the view. 3. We structured it to follow closely FCP - for bridging reasons. As far as I can recall Response comes first followed by sense. We will state it explicitly in the document thanks. 4.timeouts - yes you are right and I stated it already in an older note about bridging. We where thinking about three timers: - command delivery - data delivery - status delivery Those should be set at the session initiation and be known to the initiator and target but they don't have to be necessarily sent over the wire (as they imply mostly local actions in the endpoints). 5. immediate data length negotiation - will fix the negotiation parameters - thanks. We think also about limiting it by default to 64k(?) 6. iSCSI check - the reason I choose to make it SCSI check too is to get a "hook" into ACA behavior. I think that is mandatory - without it all hell breaks loose! Both initiator and target could be built the with ACA on check condition. We can remove the iSCSI status field and add a set of basic sense bytes detailing the error. Let us keep talking on this item. 7. The security parts are being rewritten just as we talk and we hope that we will all see them soon 8. Target reset will be changed to have two types - soft and hard reset. 9. closing connection. Yes you are right it needs more text. And we already agreed to add a logout (on a different connection to force connection close) and remove the RID from login Thanks for your careful reading and hope to see you on row 1 or 2 at the next IETF meeting, Julo "Hall, Howard" on 13/09/2000 02:12:38 Please respond to "Hall, Howard" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Comments on the Draft Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Thu Sep 14 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18677; 14 Sep 2000 14:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA03019 for ips-outgoing; Thu, 14 Sep 2000 13:27:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA03015 for ; Thu, 14 Sep 2000 13:27:46 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA10598; Thu, 14 Sep 2000 10:17:16 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA20039; Thu, 14 Sep 2000 10:14:55 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI: 2.2.6. Naming & mapping Date: Thu, 14 Sep 2000 10:30:08 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, Consider placing abstractions for names, proxies, paths, and settings found in readable form as something derived from an LDAP server and keep information sent to the device strictly a one-way binary, then abstractions at a management agent could change every two days without affecting storage transport. The storage device would expose fewer avenues of attack and have a smaller workload making conversions through some extraordinary means beyond the IP transport, ICMP and ARP. Conversely, should the transport change, management could still use the same structures. Otherwise, transport protocol will slowly assume the role of LDAP and DNS. The sign for frustration is ... Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Thursday, September 14, 2000 7:05 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: 2.2.6. Naming & mapping > > > > > Not again (what is the sign for frustration?)... I mean not > before the next > version. > > Julo > > Raghavendra Rao on 14/09/2000 21:04:02 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: 2.2.6. Naming & mapping > > > > > > I feel that a LUN should be very much part of the Naming scheme > that has been proposed in the draft, regardless of the level of > enforcement of the scheme by implementors. LUNs are very much > part of SCSI addressing, and where LUN doesn't exist, zero is > assumed. > > Without a LUN identifier, Naming section doesn't look complete. > > Do you have insights why it is omitted ? Is this by design or overlook ? > > Thanks. > -JP > > > > From owner-ips@ECE.cmu.edu Thu Sep 14 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18679; 14 Sep 2000 14:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA01851 for ips-outgoing; Thu, 14 Sep 2000 13:02:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA01846 for ; Thu, 14 Sep 2000 13:02:01 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id C1869D8C for ; Thu, 14 Sep 2000 10:00:41 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA23639; Thu, 14 Sep 2000 10:00:16 -0700 (PDT) Message-ID: <39C105AE.7362FD9A@hp.com> Date: Thu, 14 Sep 2000 10:06:54 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: 2.2.6. Naming & mapping References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk julian_satran@il.ibm.com wrote: > Not again (what is the sign for frustration?)... I mean not before the next > version. > > Julo > > Raghavendra Rao on 14/09/2000 21:04:02 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: 2.2.6. Naming & mapping > > I feel that a LUN should be very much part of the Naming scheme > that has been proposed in the draft, regardless of the level of > enforcement of the scheme by implementors. LUNs are very much > part of SCSI addressing, and where LUN doesn't exist, zero is > assumed. > > Without a LUN identifier, Naming section doesn't look complete. > > Do you have insights why it is omitted ? Is this by design or overlook ? > > Thanks. > -JP About the topic of naming a LU, what seems to be a benefit for me, is to adopt the same requirement as fibre channel: each LU MUST provide a unique identifier (from the Device Identification Page). >From what i read it seems that it is not a big deal to add this page in a LU. However, this unique identifier is not needed for iSCSI protocol to work. It doesn't interact with the protocol. But from an administration point of view, to configure a server using storage through iSCSI, the existence of this unique identifier helps a lot. It allows the configuration product to know/check if various LUNs correspond to the same LU, it could help to manage the LU migration (the LUN is changed inside a target for the same LU). It simplifies the configuration software by avoiding it to fake a unique LU identifier. It will help in having a configuration tool for iSCSI that can be closer to the one used with FC. Is somebody knowing if it is planned to incorporate this Device Identification Page in the iSCSI LU? Will it be mandatory? Which format (identifier type) will be used? FC uses the type 3 (FC_PH Name_Identifier). Regards, Pierre From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24910; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA23516 for ips-outgoing; Thu, 14 Sep 2000 23:16:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA23511 for ; Thu, 14 Sep 2000 23:16:40 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 14 Sep 2000 23:16:05 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F99@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: PLEASE TAG EMAILS! Date: Thu, 14 Sep 2000 23:16:01 -0400 Importance: high X-Priority: 1 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, A gentle reminder that this list is the forum for discussion of both iSCSI and FC over IP. Please make sure that your emails that concern only one of the two are tagged with either the string "iSCSI" or "FCIP" in the Subject line, viz: Subject: iSCSI: Sessions Subject: Sequences in FCIP Email concerning both protocols need not be tagged. This will be greatly appreciated by your colleagues who are already dealing with too much email. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24907; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA27492 for ips-outgoing; Fri, 15 Sep 2000 02:23:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA27487 for ; Fri, 15 Sep 2000 02:23:11 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA45664 for ; Fri, 15 Sep 2000 08:22:39 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA25288 for ; Fri, 15 Sep 2000 08:22:24 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695B.0022FF6A ; Fri, 15 Sep 2000 08:22:16 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 15 Sep 2000 09:16:00 +0300 Subject: RE: Comments on status responses Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Very good analysis. I would like however to point out that althouth that sequence of information WAS DESIGNED albeit not by T10 but by the good people that designed 360 channels (status and sense) and 370 (status, sense and extended sense). We migth end up however having more than 1 bit of iSCSI staus to indicate the troubled area. Julo Jim McGrath on 14/09/2000 21:29:14 Please respond to Jim McGrath To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Comments on status responses Traditionally error reporting performs two different functions. The first is to direct recovery actions. The second is to passively log information for future examination. SCSI was never really "designed," so is not pure in this area. But error reporting does come in three pieces of information: STATUS, SENSE KEY, SENSE CODE. The last two you traditionally get in response to a REQUEST SENSE command, but can also be provided directly via an auto sense function. As a general rule SENSE CODE information is logged, but not used for real time recovery. The combination of STATUS (i.e. CHECK CONDITION) and SENSE KEY is used to drive error recovery. Some errors are already recovered (e.g. Recovered Error key), others cannot be easily recovered (Illegal Request or Hardware Error key), others indicate a retry might work OK (Media Error key). The lesson here is that you should start out with your expected recovery actions, and from that map appropriate STATUS, KEY, and CODES. Don't do something because it sounds reasonable to a human - these are computers we are talking about afterall. The other, related issue is layering. Error recovery may take place at different layers in your stack. Some information is appropriate for the lowest layer - for instance, STATUS can be used by adapter hardware to issue immediate retries (for things like BUSY) or REQUEST SENSE. The latter allows that layer to provide an Auto sense like appearance to higher layers, even though the target does not support Auto sense per se. Similarly, if the target contains layers, you may want to get separate information from each layer (e.g. target, LUN, sub LUNs, etc...). The SCSI Controller Commands document I think provides some interesting examples of this. In general, I would advise iSCSI to focus on new and novel recovery operations required and new layering concepts. In those areas where new stuff can be identified, ask T10 to extend the STATUS/SENSE structure accordingly. Personally I would try and keep layering as strong as possible between SCSI and the underlying transport. Jim -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Thursday, September 14, 2000 8:23 AM To: ips@ece.cmu.edu Subject: RE: Comments on status responses Will add to my list of to consider items. Thanks, Julo "Merhar, Milan" on 14/09/2000 17:12:59 Please respond to "Merhar, Milan" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Comments on status responses Julo, I agree with you that status returned from a SCSI target device needs to be transported exactly, so that existing drivers, etc. (which are unaware of the intervening iSCSI transport) work as expected. But, I think the question here concerns status reported by the iSCSI layer itself. Basically, my opinion is that if we expect the other end's error handler to be something like if ( errno ) then abort; it makes little or no difference what we return; 'OK' = 0 and 'ERR' = 1 works as well as an elaborate list of nonzero error codes. But, if there's a chance the other end could implement a more robust and/or less disruptive error recovery procedure, it makes sense to pass some "hints" along, through a broader set of exception codes. As an example, I suggest one might want to create a client-side recovery procedure that handles a storage device response timeout differently, depending on whether it was caused by a target too busy to process the command in a timely fashion, versus a timeout caused by excessive network congestion or repeated TCP error retrys. - milan -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 1:57 PM To: ips@ece.cmu.edu Subject: RE: Comments on the Draft On 1 - depends on where thing go with asymmetric vs. symmetric. 2 - Explicit error codes - isn't that the sense function? 3 - For the flag - probably yes- but I am not sure. Status implies it so you need it only if status is bad - which can be before the end. Julo "Hall, Howard" on 13/09/2000 20:50:10 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Comments on the Draft Julo, Thank you very much for your thoughtful response. What do you think of some of the other issues (reference the original email): - whether CmdRN, MaxCmdRN, etc. should be used for in-order delivery of commands across multiple TCP connections in a session (Comments for Section 2.2.2). - more explicit error codes - the proposed flag described at the end of the email -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 4:52 AM To: Hall, Howard Subject: Re: Comments on the Draft 1. General - Loss of synchronization - we hope to be able one day to get to a better solution but I would suggest to recommend the implementer to drop the connections as it won't be able to parse a reset too, reinstate them and then try a reset to distinguish it clearly from other events. If it happens in a target - drop - and wait for a reconnect then send an AE to indicate the event and wait for reset. As for inconsistencies - I've added your proposal to the todo list - thank you. 2. The view is a qualifier for the name - it allows a target to a second level of refinement about what he things it should show the initiator. The first part of the name itself is the first level - but for those that are conservative in the use of names they are left with the view. 3. We structured it to follow closely FCP - for bridging reasons. As far as I can recall Response comes first followed by sense. We will state it explicitly in the document thanks. 4.timeouts - yes you are right and I stated it already in an older note about bridging. We where thinking about three timers: - command delivery - data delivery - status delivery Those should be set at the session initiation and be known to the initiator and target but they don't have to be necessarily sent over the wire (as they imply mostly local actions in the endpoints). 5. immediate data length negotiation - will fix the negotiation parameters - thanks. We think also about limiting it by default to 64k(?) 6. iSCSI check - the reason I choose to make it SCSI check too is to get a "hook" into ACA behavior. I think that is mandatory - without it all hell breaks loose! Both initiator and target could be built the with ACA on check condition. We can remove the iSCSI status field and add a set of basic sense bytes detailing the error. Let us keep talking on this item. 7. The security parts are being rewritten just as we talk and we hope that we will all see them soon 8. Target reset will be changed to have two types - soft and hard reset. 9. closing connection. Yes you are right it needs more text. And we already agreed to add a logout (on a different connection to force connection close) and remove the RID from login Thanks for your careful reading and hope to see you on row 1 or 2 at the next IETF meeting, Julo "Hall, Howard" on 13/09/2000 02:12:38 Please respond to "Hall, Howard" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Comments on the Draft Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24922; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA09353 for ips-outgoing; Thu, 14 Sep 2000 15:26:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA09344 for ; Thu, 14 Sep 2000 15:26:30 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA10681; Thu, 14 Sep 2000 12:15:59 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA20399; Thu, 14 Sep 2000 12:13:36 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI: T10 meeting Date: Thu, 14 Sep 2000 12:28:50 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100F88@corpmx9.isus.emc.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, The FCoverIP specification has already defined FC encapsulation. Should there be additional information required to transport an FC frame over IP, that should become a header prefix rather than a redefinition of structures. >From that point, changes can be determined within a narrower perspective. The present solution is less than satisfactory in the basic aspect of control. Solutions already exist within FC. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > Sent: Wednesday, September 13, 2000 10:36 PM > To: ips@ece.cmu.edu > Subject: iSCSI: T10 meeting > > > I spent today at the T10 meeting, despite United Airlines' > best attempts to keep me out of the Pacific time zone :-). > I did quite a bit of explaining about IETF and ips > logistics, procedures, and status, but did manage > to squeeze in some technical discussion. > > Here are a few items of relevance to iSCSI: > > - As indicated earlier, T10 would like to see iSCSI > (and other encapsulations of SCSI) follow the data > formats used by FCP to the extent possible and > reasonable. Departures for good functional reasons > are ok (e.g., iSCSI separates out task management > in a way that FCP does not, and there may be good > reasons to do that), even though they complicate > bridges. OTOH, format differences that lack a > functional reason should be avoided. The recent > proposal by Charles Monia to harmonize iSCSI and > FCP packet formats deserves serious attention. > > - There has been some discussion of stateless bridges > or gateways on this list in the past. T10's view of > this area is that such devices tend to be inherently > stateful because they usually have to keep state > related to proxying multiple initiators accessing them > onto a single initiator that accesses the actual > target. Mapping command tags as part of this is not > a big deal, and hence T10 does not attempt to mandate > a single SCSI tag format -- in fact a 64 bit tag for > SCSI over VI was recommended as part of today's meeting. > The rationale for that tag may be worth considering for > iSCSI - an initiator running a 64 bit OS can stick a > memory address in the tag, which makes it easier to > handle responses. In any case, it's not important > to match FCP's tag structure exactly. > > - I want to reinforce Steve Byan's earlier email on > multiple ordered commands and resource exhaustion. > As indicated by words in our (still draft) charter, > this sort of issue expressible entirely in SCSI > w/o reference to TCP or any other transport is in > T10's domain. That means that not only is it T10's > responsibility to deal with this issue, it is also > T10's responsibility to determine whether the issue > is serious enough to merit any changes. This IETF > WG should be cautious about redesigning things like > ACA - it would be undesirable for ACA to behave > differently in iSCSI than in other SCSI transports. > > >From my viewpoint, a good working relationship > with T10 is well underway. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24908; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA26483 for ips-outgoing; Fri, 15 Sep 2000 01:35:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA26478 for ; Fri, 15 Sep 2000 01:34:56 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id WAA26631 for ; Thu, 14 Sep 2000 22:34:55 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.187.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id WAA02843 for ; Thu, 14 Sep 2000 22:34:54 -0700 (PDT) Received: from quark by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id WAA14996; Thu, 14 Sep 2000 22:34:53 -0700 (PDT) Message-Id: <200009150534.WAA14996@ha10nwk.EBay.Sun.COM> Date: Fri, 15 Sep 2000 07:06:42 -0400 (EDT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: iSCSI: 2.2.6. Naming & mapping To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: U9Z/6EPFHlTTyCmmynXfGg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.3.4p_5 SunOS 5.7 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > > Once the iSCSI session is established, the Rules of SCSI define how the LUs > are addressed. > So what is the scope of the SCSI URL name format suggested in the spec ? Is this intended to establish an iSCSI session or something beyond this ? My original question was to find out why a SCSI LUN identifier is missing in the SCSI URL name, if it is by design. -JP From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24901; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA15676 for ips-outgoing; Fri, 15 Sep 2000 09:47:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA15669 for ; Fri, 15 Sep 2000 09:47:34 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 6997B8FD; Fri, 15 Sep 2000 06:47:33 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id GAA13175; Fri, 15 Sep 2000 06:47:25 -0700 (PDT) Message-Id: <4.2.2.20000915063004.00b5fc20@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 15 Sep 2000 06:37:12 -0700 To: Stephen Byan , IPS Reflector From: Michael Krause Subject: RE: VI (Was: Avoiding deadlock in iSCSI) In-Reply-To: <8133266FE373D11190CD00805FA768BF055BD1AC@shrcmsg1.tdh.qntm .com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 06:22 AM 9/13/00 -0700, Stephen Byan wrote: >Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote: > > > I don't think VI/TCP helps at all. The VI is implemented on > > top of a TCP > > stream. If the TCP stream delivers the iSCSI command to VI, > > and VI has no > > place to put it, what is VI going to do? It has to block the > > TCP stream, and > > that in turn will block and "RDMA" from occurring. > >If I understand correctly, VI/TCP has a credit-based flow control mechanism >on its message queues (which would be used to implement the iSCSI command >reception queue), and so the initiator would never send an iSCSI command >without a target buffer in which to store it. So RDMA cannot block on >commands in the TCP stream. InfiniBand has this as well. One exchanges credits for the number of outstanding receive buffers that can be posted. If one exceeds the credit, the responder returns a RNR-NAK (receiver not ready) with a time to wait before trying again. This is completely independent of RDMA. RDMA is implemented either as a set of READ (fetch data from address X and deposit in my local memory - very nice for thin devices) or WRITE (send this data to address Y from my local memory). The memory must be registered, i.e. pinned / mapped, to avoid having to perform the V-P translations and to insure it is present for DMA operations. The advantage to using RDMA is that it pays the price for buffer management during the heavy weight registration process up front and the subsequent operations are therefore much lighter weight. It also does not have a flow control issue for RDMA WRITEs since if a command is allowed to target a given memory area, it can always target that memory. For RDMA READs things are a bit more complex w.r.t. ordering rules, resource management (need the ability to control how many are outstanding since the responder must be able to store the read command and thus requires some form of a command queue). In general, there are many benefits to RDMA support and I would like to see TCP have a RDMA capability but they are not required to make iSCSI operate (might improve some operations though). Mike From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24904; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA28618 for ips-outgoing; Fri, 15 Sep 2000 03:13:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA28613 for ; Fri, 15 Sep 2000 03:13:04 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id CAA85024; Fri, 15 Sep 2000 02:06:48 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8F7D3e109438; Fri, 15 Sep 2000 01:13:03 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: TCP speed To: "Chee-Siong, Lee" Cc: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 15 Sep 2000 00:11:06 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/15/2000 01:13:03 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk I have no idea what you said or asked! . . . John L. Hufferd "Chee-Siong, Lee" @ece.cmu.edu on 09/14/2000 07:18:52 PM Sent by: owner-ips@ece.cmu.edu To: cc: Subject: Re: TCP speed Is the kernel will use both cpu to process the TCP queue if running a SMP machine? When using multithread UDP process, we can balance the load in transmission with 2 CPUs. --------- cslee From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24917; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA20711 for ips-outgoing; Thu, 14 Sep 2000 21:16:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nister.prisa.com (fwuser@gatekeeper.prisa.com [204.94.67.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA20704 for ; Thu, 14 Sep 2000 21:16:33 -0400 (EDT) Received: from don ([172.16.129.136]) by nister.prisa.com (8.9.3/8.9.3) with SMTP id SAA56591; Thu, 14 Sep 2000 18:13:26 -0700 (PDT) (envelope-from don@prisa.com) Message-Id: <3.0.5.32.20000914181802.00a22a90@mailhost.prisa.com> X-Sender: don@mailhost.prisa.com X-Mailer: QUALCOMM Windows Eudora Pro Version 3.0.5 (32) Date: Thu, 14 Sep 2000 18:18:02 -0700 To: "DeKoning, Rod" From: Don Deel Subject: RE: iSCSI: 2.2.6. Naming & mapping Cc: "'Pierre Labat'" , ips@ece.cmu.edu In-Reply-To: Mime-Version: 1.0 Content-Type: text/enriched; charset="us-ascii" Sender: owner-ips@ece.cmu.edu Precedence: bulk Agreed! Unique identification of LUs is a requirement for manageability of the resulting Storage Network. However, as Pierre also observes, this is not a requirement for iSCSI... Don Deel don@prisa.com At 03:18 PM 09/14/2000 -0500, DeKoning, Rod wrote: >>>> Since he hit one of my hot buttons regarding mapping (and therefore, discovery), . . . I heartily endorse Pierre's assertion that the Device Identification Page be used to identify Logical Units (Specifically the Volume WWN flavor). This is extremely important in an environment that may access volumes (logical units) through many paths and potentially through multiple underlying ports or physical devices. The only question is: does this fit in a protocol spec such as iSCSI? Rod DeKoning Manager Storage Solutions Architecture LSI Logic Storage Systems Inc. ph:316-636-8842 fax:x8889 rod.dekoning@lsil.com -----Original Message----- From: Pierre Labat [<mailto:pierre_labat@hp.com] Sent: Thursday, September 14, 2000 12:07 PM To: ips@ece.cmu.edu Subject: Re: iSCSI: 2.2.6. Naming & mapping julian_satran@il.ibm.com wrote: > Not again (what is the sign for frustration?)... I mean not before the next > version. > > Julo > > Raghavendra Rao < on 14/09/2000 21:04:02 > > Please respond to Raghavendra Rao < > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: 2.2.6. Naming & mapping > > I feel that a LUN should be very much part of the Naming scheme > that has been proposed in the draft, regardless of the level of > enforcement of the scheme by implementors. LUNs are very much > part of SCSI addressing, and where LUN doesn't exist, zero is > assumed. > > Without a LUN identifier, Naming section doesn't look complete. > > Do you have insights why it is omitted ? Is this by design or overlook ? > > Thanks. > -JP About the topic of naming a LU, what seems to be a benefit for me, is to adopt the same requirement as fibre channel: each LU MUST provide a unique identifier (from the Device Identification Page). From what i read it seems that it is not a big deal to add this page in a LU. However, this unique identifier is not needed for iSCSI protocol to work. It doesn't interact with the protocol. But from an administration point of view, to configure a server using storage through iSCSI, the existence of this unique identifier helps a lot. It allows the configuration product to know/check if various LUNs correspond to the same LU, it could help to manage the LU migration (the LUN is changed inside a target for the same LU). It simplifies the configuration software by avoiding it to fake a unique LU identifier. It will help in having a configuration tool for iSCSI that can be closer to the one used with FC. Is somebody knowing if it is planned to incorporate this Device Identification Page in the iSCSI LU? Will it be mandatory? Which format (identifier type) will be used? FC uses the type 3 (FC_PH Name_Identifier). Regards, Pierre <<<<<<<< From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24902; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA11564 for ips-outgoing; Fri, 15 Sep 2000 07:27:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ns.falconstor.com.tw (ns1.falconstor.com.tw [211.75.17.117]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA11560 for ; Fri, 15 Sep 2000 07:27:03 -0400 (EDT) Received: from NEWTON (h1-51.falconstor.com.tw [172.30.1.51]) by ns.falconstor.com.tw (8.10.1/8.10.1) with SMTP id e8FBR2024380; Fri, 15 Sep 2000 19:27:02 +0800 Message-ID: <004301c01f07$fd23d530$33011eac@NEWTON> From: "Chee-Siong, Lee" To: "John Hufferd/San Jose/IBM" Cc: References: Subject: Re: TCP speed Date: Fri, 15 Sep 2000 19:27:53 +0800 Organization: FalconStor Inc. MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry for poor description.. A multithread program will use different cpu in different thread. For example, we use one cpu for receving packets to receive queue and the other CPU for sending packets from sending queue. Hence we can have more CPU power in iSCSI path. An UDP program with its sending and receiving can archive this situation. I think this will give a significant performance than TCP if TCP can't use both CPUs proccesing power. TCP process its own queue in kernel mode. And will this TCP queue is processed with 2 CPU? ----- Original Message ----- From: "John Hufferd/San Jose/IBM" To: "Chee-Siong, Lee" Cc: Sent: Friday, September 15, 2000 3:11 PM Subject: Re: TCP speed > I have no idea what you said or asked! > > . > . > . > John L. Hufferd > > > > "Chee-Siong, Lee" @ece.cmu.edu on 09/14/2000 > 07:18:52 PM > > Sent by: owner-ips@ece.cmu.edu > > > To: > cc: > Subject: Re: TCP speed > > > > > Is the kernel will use both cpu to process the TCP queue if running a > SMP > machine? > When using multithread UDP process, we can balance the load in transmission > with 2 CPUs. > > --------- > cslee > > > > > From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24899; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA15667 for ips-outgoing; Fri, 15 Sep 2000 09:47:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA15661 for ; Fri, 15 Sep 2000 09:47:27 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id F094A973; Fri, 15 Sep 2000 06:47:25 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id GAA13165; Fri, 15 Sep 2000 06:47:12 -0700 (PDT) Message-Id: <4.2.2.20000915061854.00b5f370@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 15 Sep 2000 06:19:43 -0700 To: Fred Baker , "Greg Joyce" From: Michael Krause Subject: RE: TCP speed Cc: "Y P Cheng" , In-Reply-To: <5.0.0.25.2.20000912114603.02547eb0@flipper.cisco.com> References: <000601c01c05$26275520$65c809c0@yp_portable.advansys.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:55 AM 9/12/00 -0700, Fred Baker wrote: >At 12:12 PM 9/11/00 -0500, Greg Joyce wrote: >>Once you build that on top of UDP, you'll have something very TCP-like >>without the benefit of years of tuning and refinement. I think you'd also >>find that the performance was comparable. > >the difference will no doubt be that UDP has an optional checksum, while >TCP requires its checksum to be used. If you have a hardware checksum >generator, as has been suggested, mox nix. I believe everyone implements UDP checksum and it may now be required. Most NICs implement checksum off-load for UDP and TCP so this isn't an issue from a performance point of view. Mike From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24915; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA20043 for ips-outgoing; Thu, 14 Sep 2000 20:50:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA20039 for ; Thu, 14 Sep 2000 20:50:31 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id DBBCDDE5; Thu, 14 Sep 2000 17:50:30 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id RAA03581; Thu, 14 Sep 2000 17:50:29 -0700 (PDT) Message-ID: <39C173E2.E7F77984@hp.com> Date: Thu, 14 Sep 2000 17:57:06 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: Re: iSCSI: 2.2.6. Naming & mapping References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk John Hufferd/San Jose/IBM wrote: > Folks, > We are again mixing issues and layers again. > > There is NO SUCH THING as an iSCSI LU. There is an iSCSI device which is a > Controller which will have a SCSI layer which in turn supports an LU. Once > the iSCSI session is established, the Rules of SCSI define how the LUs are > addressed. I agree. > There maybe additional Database (LDAP) processes and > information that attempts to relate LU #3 known to Host xyz to some name > "abcd...." etc. I don't want so much, i would want just to have a unique identifier per LU. It doesn't cost a lot and it will have the advantages described in the first mail below. > But that is NOT an iSCSI Transport Protocol. We may need > to work on this at some point but it is not an iSCSI transport issue. > Yes, it is not a transport issue, but why not request that now before people start building iSCSI controllers if it simplifies the life of everybody? I don't know how FC managed to have their WWN but why could not we do the same thing? Regards, Pierre > > . > . > . > John L. Hufferd > > Pierre Labat @ece.cmu.edu on 09/14/2000 10:06:54 AM > > Sent by: owner-ips@ece.cmu.edu > > To: ips@ece.cmu.edu > cc: > Subject: Re: iSCSI: 2.2.6. Naming & mapping > > julian_satran@il.ibm.com wrote: > > > Not again (what is the sign for frustration?)... I mean not before the > next > > version. > > > > Julo > > > > Raghavendra Rao on 14/09/2000 21:04:02 > > > > Please respond to Raghavendra Rao > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: > > Subject: iSCSI: 2.2.6. Naming & mapping > > > > I feel that a LUN should be very much part of the Naming scheme > > that has been proposed in the draft, regardless of the level of > > enforcement of the scheme by implementors. LUNs are very much > > part of SCSI addressing, and where LUN doesn't exist, zero is > > assumed. > > > > Without a LUN identifier, Naming section doesn't look complete. > > > > Do you have insights why it is omitted ? Is this by design or overlook ? > > > > Thanks. > > -JP > > About the topic of naming a LU, what seems to be a benefit for me, is to > adopt the same requirement as fibre channel: each LU MUST provide a > unique identifier (from the Device Identification Page). > >From what i read it seems that it is not a big deal to add this page in a > LU. > However, this unique identifier is not needed for iSCSI protocol to work. > It doesn't interact with the protocol. > > But from an administration point of view, to configure a server using > storage > through iSCSI, the existence of this unique identifier helps a lot. > It allows the configuration product to know/check if various LUNs > correspond > to the same LU, it could help to manage the LU migration (the LUN is > changed > > inside a target for the same LU). > It simplifies the configuration software by avoiding it to fake a unique LU > identifier. > It will help in having a configuration tool for iSCSI that can be closer to > the one > used with FC. > > Is somebody knowing if it is planned to incorporate this Device > Identification > Page > in the iSCSI LU? Will it be mandatory? Which format (identifier type) will > be > used? > FC uses the type 3 (FC_PH Name_Identifier). > > Regards, > > Pierre From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24911; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA23303 for ips-outgoing; Thu, 14 Sep 2000 23:07:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA23298 for ; Thu, 14 Sep 2000 23:07:44 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id TAA10793; Thu, 14 Sep 2000 19:57:09 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id TAA21561; Thu, 14 Sep 2000 19:54:49 -0700 From: "Douglas Otis" To: , Subject: RE: FC encapsulation Date: Thu, 14 Sep 2000 20:10:06 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100F94@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I understand a desire to start with a clean slate and then look to see how FCP can map into whatever is created. A valid alternative would be to take FCP structures 'as is' and define use within constructs of IP transport. A considerable advantage from control and documentation would immediately fall into place. Unless there is something abhorrent in starting from a known point, it should receive consideration. No one should expect IP transport to be tightly coupled to SCSI as it is currently conceived. Wide-open traffic problems found with FC will become apparent with IP and dealt with in much the same manner. The value of such a standard would be in terms of facilitating both simple bridges with minimal states as well as native IP access. There are solutions for perceived limitations with the FC transport. A simple buffer and a timestamp prefix, as example. If you wish to combine connections, there are already IP standards for doing so. This effort may be years ahead to use what already exists and define the practice as applied. Discussing tags for 256 commands in a drive from an initiator? 64 bit tags would allow mapping to an address on a high end machine. Most often this is mapped to a 32 bit process ID and so perhaps 32 bits would be okay. A justification for tossing the ugly baby? Now we have a SCSI level flow control of check and dump. Hardly the improvement expected over FCP. Gaps in FC documentation only illustrates effort required to strike a new path. With much to be done once communications structures become defined, I would expect a far easier time explaining why something was added rather than how it was remapped. Doug > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Thursday, September 14, 2000 6:42 PM > To: dotis@sanlight.net; Black_David@emc.com; ips@ece.cmu.edu > Subject: FC encapsulation > > > Doug, > > I think you missed the point ... > > > The FCoverIP specification has already defined FC encapsulation. Should > > there be additional information required to transport an FC frame over > IP, > > I suggest you reread Charles Monia's proposal for use of FCP formats in > iSCSI - this is the sort of thing that T10 is asking for, although the > request > is not for slavish adherence to FCP in all aspects. This is fundamentally > different from the FC over IP approach of encapsulating SCSI in FCP in > FC-2 and then in an IP transport of some form. It's not at all clear that > FC-2 is necessary in this context. > > FWIW, the current version of FC over IP doesn't even encapsulate all > of FC-2. For example, the ABTS basic link service used by FCP to > abort a SCSI task is missing. I presume that this is an oversight that > will be corrected in a future version of the FC over IP draft, > but I suspect > that this isn't the only basic or extended link service that needs to be > added. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24921; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA11331 for ips-outgoing; Thu, 14 Sep 2000 16:21:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lsi.lsil.com (lsi.lsil.com [147.145.40.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA11326 for ; Thu, 14 Sep 2000 16:21:31 -0400 (EDT) Received: from mhbs.lsil.com ([147.145.31.100]) by lsi.lsil.com (8.9.3+Sun/8.9.1) with ESMTP id NAA06778 for ; Thu, 14 Sep 2000 13:21:24 -0700 (PDT) Received: from inca.co.lsil.com by mhbs.lsil.com with ESMTP; Thu, 14 Sep 2000 13:21:21 -0700 Received: from exw-kansas.ks.lsil.com (exw-kansas.ks.lsil.com [153.79.8.7]) by inca.co.lsil.com (8.9.3/8.9.3) with ESMTP id OAA24074; Thu, 14 Sep 2000 14:21:19 -0600 (MDT) Received: by exw-kansas.ks.lsil.com with Internet Mail Service (5.5.2650.21) id ; Thu, 14 Sep 2000 15:19:00 -0500 Message-Id: From: "DeKoning, Rod" To: "'Pierre Labat'" , ips@ece.cmu.edu Subject: RE: iSCSI: 2.2.6. Naming & mapping Date: Thu, 14 Sep 2000 15:18:59 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C01E89.0482C72A" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C01E89.0482C72A Content-Type: text/plain; charset="iso-8859-1" Since he hit one of my hot buttons regarding mapping (and therefore, discovery), . . . I heartily endorse Pierre's assertion that the Device Identification Page be used to identify Logical Units (Specifically the Volume WWN flavor). This is extremely important in an environment that may access volumes (logical units) through many paths and potentially through multiple underlying ports or physical devices. The only question is: does this fit in a protocol spec such as iSCSI? Rod DeKoning Manager Storage Solutions Architecture LSI Logic Storage Systems Inc. ph:316-636-8842 fax:x8889 rod.dekoning@lsil.com -----Original Message----- From: Pierre Labat [mailto:pierre_labat@hp.com] Sent: Thursday, September 14, 2000 12:07 PM To: ips@ece.cmu.edu Subject: Re: iSCSI: 2.2.6. Naming & mapping julian_satran@il.ibm.com wrote: > Not again (what is the sign for frustration?)... I mean not before the next > version. > > Julo > > Raghavendra Rao on 14/09/2000 21:04:02 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: 2.2.6. Naming & mapping > > I feel that a LUN should be very much part of the Naming scheme > that has been proposed in the draft, regardless of the level of > enforcement of the scheme by implementors. LUNs are very much > part of SCSI addressing, and where LUN doesn't exist, zero is > assumed. > > Without a LUN identifier, Naming section doesn't look complete. > > Do you have insights why it is omitted ? Is this by design or overlook ? > > Thanks. > -JP About the topic of naming a LU, what seems to be a benefit for me, is to adopt the same requirement as fibre channel: each LU MUST provide a unique identifier (from the Device Identification Page). >From what i read it seems that it is not a big deal to add this page in a LU. However, this unique identifier is not needed for iSCSI protocol to work. It doesn't interact with the protocol. But from an administration point of view, to configure a server using storage through iSCSI, the existence of this unique identifier helps a lot. It allows the configuration product to know/check if various LUNs correspond to the same LU, it could help to manage the LU migration (the LUN is changed inside a target for the same LU). It simplifies the configuration software by avoiding it to fake a unique LU identifier. It will help in having a configuration tool for iSCSI that can be closer to the one used with FC. Is somebody knowing if it is planned to incorporate this Device Identification Page in the iSCSI LU? Will it be mandatory? Which format (identifier type) will be used? FC uses the type 3 (FC_PH Name_Identifier). Regards, Pierre ------_=_NextPart_001_01C01E89.0482C72A Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: iSCSI: 2.2.6. Naming & mapping

Since he hit one of my hot buttons regarding mapping = (and therefore, discovery), . . . I heartily endorse Pierre's assertion = that the Device Identification Page be used to identify Logical Units = (Specifically the Volume WWN flavor).  This is extremely important = in an environment that may access volumes (logical units) through many = paths and potentially through multiple underlying ports or physical = devices.

The only question is: does this fit in a protocol = spec such as iSCSI?

Rod DeKoning
Manager Storage Solutions Architecture
LSI Logic Storage Systems Inc.
ph:316-636-8842
fax:x8889
rod.dekoning@lsil.com


-----Original Message-----
From: Pierre Labat [mailto:pierre_labat@hp.com]
Sent: Thursday, September 14, 2000 12:07 PM
To: ips@ece.cmu.edu
Subject: Re: iSCSI: 2.2.6. Naming & = mapping


julian_satran@il.ibm.com wrote:

> Not again (what is the sign for frustration?)... = I mean not before the next
> version.
>
> Julo
>
> Raghavendra Rao = <Jp.Raghavendra@EBay.Sun.COM> on 14/09/2000 21:04:02
>
> Please respond to Raghavendra Rao = <Jp.Raghavendra@EBay.Sun.COM>
>
> To:   Julian = Satran/Haifa/IBM@IBMIL
> cc:
> Subject:  iSCSI: 2.2.6. Naming & = mapping
>
> I feel that a LUN should be very much part of = the Naming scheme
> that has been proposed in the draft, regardless = of the level of
> enforcement of the scheme by implementors. LUNs = are very much
> part of SCSI addressing, and where LUN doesn't = exist, zero is
> assumed.
>
> Without a LUN identifier, Naming section = doesn't look complete.
>
> Do you have insights why it is omitted ? Is = this by design or overlook ?
>
> Thanks.
> -JP

About the topic of naming a LU, what seems to = be  a benefit for me, is to
adopt the same requirement as fibre channel: each LU = MUST provide a
unique identifier (from the Device Identification = Page).
From what i read it seems that it is not a big deal = to add this page in a LU.
However, this unique identifier is not needed for = iSCSI protocol to work.
It doesn't interact with the protocol.

But from an administration point of view, to = configure a server using storage
through iSCSI, the existence of this unique = identifier helps a lot.
It allows the configuration product to know/check&nbs= p; if various LUNs correspond
to the same LU, it could help to manage  = the  LU migration (the LUN is changed

inside a target for the same LU).
It simplifies the configuration software by avoiding = it to fake a unique LU
identifier.
It will help in having a configuration tool for = iSCSI that can be closer to
the one
used with FC.

Is somebody knowing if it is planned to incorporate = this Device Identification
Page
in the iSCSI LU? Will it be mandatory? Which format = (identifier type) will be
used?
FC  uses the type 3 (FC_PH = Name_Identifier).

Regards,

Pierre



------_=_NextPart_001_01C01E89.0482C72A-- From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24982; 15 Sep 2000 12:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA03723 for ips-outgoing; Thu, 14 Sep 2000 13:41:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from manganese.network-alchemy.com (NAT-10-net.Network-Alchemy.COM [199.46.17.205]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA03716 for ; Thu, 14 Sep 2000 13:41:25 -0400 (EDT) Received: from manganese.network-alchemy.com (localhost [127.0.0.1]) by manganese.network-alchemy.com (8.9.3/8.8.8) with ESMTP id KAA37561; Thu, 14 Sep 2000 10:42:21 -0700 (PDT) (envelope-from rdv@manganese.network-alchemy.com) Message-Id: <200009141742.KAA37561@manganese.network-alchemy.com> X-Mailer: exmh version 2.0.2 2/24/98 To: Dave Nagle cc: ips@ece.cmu.edu Subject: Re: another question (was Re: patent question) In-reply-to: bassoon's message of Wed, 13 Sep 2000 21:44:44 -0400. <200009140144.VAA21338@yogi.ece.cmu.edu> Reply-To: rdv@network-alchemy.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 14 Sep 2000 10:42:18 -0700 From: Rod Van Meter Sender: owner-ips@ece.cmu.edu Precedence: bulk The first case I actually know of doing literally SCSI over ethernet was Odetics, who did it for some of their big autochangers in ~1993. As far as I know, they never attempted to standardize it, and I presume it took special software on a host to control the thing. --Rod From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24905; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA28987 for ips-outgoing; Fri, 15 Sep 2000 03:33:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA28983 for ; Fri, 15 Sep 2000 03:33:36 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id DAA14874; Fri, 15 Sep 2000 03:31:51 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8F7XYe157854; Fri, 15 Sep 2000 01:33:34 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: 2.2.6. Naming & mapping To: Pierre Labat Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 15 Sep 2000 00:32:22 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/15/2000 01:33:33 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, You said "Yes, it is not a transport issue, but why not request that now before people start building iSCSI controllers if it simplifies the life of everybody? I don't know how FC managed to have their WWN but why could not we do the same thing?" Again, you mixed concepts, that is a "port" attribute (WWN) which is a Transport item, and LUs specific names which is a SCSI issue. However, there are still things associated with iSCSI which may address some of the things you are interested in. Specifically, there are reasons for a Connection to be "bound" to a specific set of LUs We are defining a concept of a LU view, that will permit, the storage controller to know what LU Views to give to any specific connection. This is of value to the Storage Controller as an entity, and the names and associations will probably be saved in the Discovery Database (LDAP). However, this is my hallucination, and has not been defined by the Workgroup. The Workgroup has already agreed to work on the items that are needed for Discovery, etc. in one of the next work assignments following the definition of the iSCSI protocol. There will also be the need to have a process that defines Names and View Mapping (see section 2.2.6). There will need to be, of course a way for the setting and naming the LU View maps. All of that is beyond the iSCSI protocol specification. The iSCSI protocol (2.2.6 & 3.17), however, has specified the protocol needed to carry the View information. Now, please lets focus on the iSCSI Transport Protocol Draft and take on these LU naming issues at the approprate time. . . . John L. Hufferd From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24906; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA27485 for ips-outgoing; Fri, 15 Sep 2000 02:23:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA27481 for ; Fri, 15 Sep 2000 02:23:03 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA13330 for ; Fri, 15 Sep 2000 08:22:32 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA54042 for ; Fri, 15 Sep 2000 08:22:32 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695B.0022FAD6 ; Fri, 15 Sep 2000 08:22:04 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 15 Sep 2000 08:25:08 +0300 Subject: RE: Comments on status responses Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I still don't see what difference does it make if you have to check the status or the sense bytes. But perhaps we can differentiate end-point iSCSI issues from transport iSCSI issues ? We will have to give it some more thought. Julo "Merhar, Milan" on 14/09/2000 17:12:59 Please respond to "Merhar, Milan" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Comments on status responses Julo, I agree with you that status returned from a SCSI target device needs to be transported exactly, so that existing drivers, etc. (which are unaware of the intervening iSCSI transport) work as expected. But, I think the question here concerns status reported by the iSCSI layer itself. Basically, my opinion is that if we expect the other end's error handler to be something like if ( errno ) then abort; it makes little or no difference what we return; 'OK' = 0 and 'ERR' = 1 works as well as an elaborate list of nonzero error codes. But, if there's a chance the other end could implement a more robust and/or less disruptive error recovery procedure, it makes sense to pass some "hints" along, through a broader set of exception codes. As an example, I suggest one might want to create a client-side recovery procedure that handles a storage device response timeout differently, depending on whether it was caused by a target too busy to process the command in a timely fashion, versus a timeout caused by excessive network congestion or repeated TCP error retrys. - milan -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 1:57 PM To: ips@ece.cmu.edu Subject: RE: Comments on the Draft On 1 - depends on where thing go with asymmetric vs. symmetric. 2 - Explicit error codes - isn't that the sense function? 3 - For the flag - probably yes- but I am not sure. Status implies it so you need it only if status is bad - which can be before the end. Julo "Hall, Howard" on 13/09/2000 20:50:10 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Comments on the Draft Julo, Thank you very much for your thoughtful response. What do you think of some of the other issues (reference the original email): - whether CmdRN, MaxCmdRN, etc. should be used for in-order delivery of commands across multiple TCP connections in a session (Comments for Section 2.2.2). - more explicit error codes - the proposed flag described at the end of the email -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 4:52 AM To: Hall, Howard Subject: Re: Comments on the Draft 1. General - Loss of synchronization - we hope to be able one day to get to a better solution but I would suggest to recommend the implementer to drop the connections as it won't be able to parse a reset too, reinstate them and then try a reset to distinguish it clearly from other events. If it happens in a target - drop - and wait for a reconnect then send an AE to indicate the event and wait for reset. As for inconsistencies - I've added your proposal to the todo list - thank you. 2. The view is a qualifier for the name - it allows a target to a second level of refinement about what he things it should show the initiator. The first part of the name itself is the first level - but for those that are conservative in the use of names they are left with the view. 3. We structured it to follow closely FCP - for bridging reasons. As far as I can recall Response comes first followed by sense. We will state it explicitly in the document thanks. 4.timeouts - yes you are right and I stated it already in an older note about bridging. We where thinking about three timers: - command delivery - data delivery - status delivery Those should be set at the session initiation and be known to the initiator and target but they don't have to be necessarily sent over the wire (as they imply mostly local actions in the endpoints). 5. immediate data length negotiation - will fix the negotiation parameters - thanks. We think also about limiting it by default to 64k(?) 6. iSCSI check - the reason I choose to make it SCSI check too is to get a "hook" into ACA behavior. I think that is mandatory - without it all hell breaks loose! Both initiator and target could be built the with ACA on check condition. We can remove the iSCSI status field and add a set of basic sense bytes detailing the error. Let us keep talking on this item. 7. The security parts are being rewritten just as we talk and we hope that we will all see them soon 8. Target reset will be changed to have two types - soft and hard reset. 9. closing connection. Yes you are right it needs more text. And we already agreed to add a logout (on a different connection to force connection close) and remove the RID from login Thanks for your careful reading and hope to see you on row 1 or 2 at the next IETF meeting, Julo "Hall, Howard" on 13/09/2000 02:12:38 Please respond to "Hall, Howard" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Comments on the Draft Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24913; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA22470 for ips-outgoing; Thu, 14 Sep 2000 22:32:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA22461 for ; Thu, 14 Sep 2000 22:32:21 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 1AD1760A; Thu, 14 Sep 2000 19:32:16 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA04604; Thu, 14 Sep 2000 19:31:56 -0700 (PDT) Message-Id: <4.2.2.20000912081718.00b35f00@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 12 Sep 2000 08:21:07 -0700 To: Scott Bradner From: Michael Krause Subject: Re: Command Queue Depth (was asymmetric/Symmetric) Cc: ips@ece.cmu.edu In-Reply-To: <200009081114.HAA04644@newdev.harvard.edu> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:14 AM 9/8/00 -0400, Scott Bradner wrote: > > I don't know what the > > latency through ethernet switches is, but I'd hope it wasn't in the ms > range. > >processing time is in the 10 to 50 microsecond range Don't know whose switches you are evaluating. They range from sub 1-usec to perhaps 5-10 usec - it is all a matter of how much additional processing is going on within the switch. For example, many switches examine not just the ethernet headers/tags but also look at the IP headers and provide additional functionality based on their content. This is where many switch vendors try to differentiate and is not a function of whether they can switch fast or not. The actual switching elements are usually crossbars that operate in the 100-500ns range - chip process improvements will lower that by quite a bit over the next couple of years. Mike From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24923; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA07029 for ips-outgoing; Thu, 14 Sep 2000 14:29:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA07020 for ; Thu, 14 Sep 2000 14:29:23 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id LAA27422; Thu, 14 Sep 2000 11:29:17 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Thu, 14 Sep 2000 11:29:14 -0700 Message-ID: From: Jim McGrath To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: Comments on status responses Date: Thu, 14 Sep 2000 11:29:14 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Traditionally error reporting performs two different functions. The first is to direct recovery actions. The second is to passively log information for future examination. SCSI was never really "designed," so is not pure in this area. But error reporting does come in three pieces of information: STATUS, SENSE KEY, SENSE CODE. The last two you traditionally get in response to a REQUEST SENSE command, but can also be provided directly via an auto sense function. As a general rule SENSE CODE information is logged, but not used for real time recovery. The combination of STATUS (i.e. CHECK CONDITION) and SENSE KEY is used to drive error recovery. Some errors are already recovered (e.g. Recovered Error key), others cannot be easily recovered (Illegal Request or Hardware Error key), others indicate a retry might work OK (Media Error key). The lesson here is that you should start out with your expected recovery actions, and from that map appropriate STATUS, KEY, and CODES. Don't do something because it sounds reasonable to a human - these are computers we are talking about afterall. The other, related issue is layering. Error recovery may take place at different layers in your stack. Some information is appropriate for the lowest layer - for instance, STATUS can be used by adapter hardware to issue immediate retries (for things like BUSY) or REQUEST SENSE. The latter allows that layer to provide an Auto sense like appearance to higher layers, even though the target does not support Auto sense per se. Similarly, if the target contains layers, you may want to get separate information from each layer (e.g. target, LUN, sub LUNs, etc...). The SCSI Controller Commands document I think provides some interesting examples of this. In general, I would advise iSCSI to focus on new and novel recovery operations required and new layering concepts. In those areas where new stuff can be identified, ask T10 to extend the STATUS/SENSE structure accordingly. Personally I would try and keep layering as strong as possible between SCSI and the underlying transport. Jim -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Thursday, September 14, 2000 8:23 AM To: ips@ece.cmu.edu Subject: RE: Comments on status responses Will add to my list of to consider items. Thanks, Julo "Merhar, Milan" on 14/09/2000 17:12:59 Please respond to "Merhar, Milan" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Comments on status responses Julo, I agree with you that status returned from a SCSI target device needs to be transported exactly, so that existing drivers, etc. (which are unaware of the intervening iSCSI transport) work as expected. But, I think the question here concerns status reported by the iSCSI layer itself. Basically, my opinion is that if we expect the other end's error handler to be something like if ( errno ) then abort; it makes little or no difference what we return; 'OK' = 0 and 'ERR' = 1 works as well as an elaborate list of nonzero error codes. But, if there's a chance the other end could implement a more robust and/or less disruptive error recovery procedure, it makes sense to pass some "hints" along, through a broader set of exception codes. As an example, I suggest one might want to create a client-side recovery procedure that handles a storage device response timeout differently, depending on whether it was caused by a target too busy to process the command in a timely fashion, versus a timeout caused by excessive network congestion or repeated TCP error retrys. - milan -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 1:57 PM To: ips@ece.cmu.edu Subject: RE: Comments on the Draft On 1 - depends on where thing go with asymmetric vs. symmetric. 2 - Explicit error codes - isn't that the sense function? 3 - For the flag - probably yes- but I am not sure. Status implies it so you need it only if status is bad - which can be before the end. Julo "Hall, Howard" on 13/09/2000 20:50:10 Please respond to "Hall, Howard" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: Comments on the Draft Julo, Thank you very much for your thoughtful response. What do you think of some of the other issues (reference the original email): - whether CmdRN, MaxCmdRN, etc. should be used for in-order delivery of commands across multiple TCP connections in a session (Comments for Section 2.2.2). - more explicit error codes - the proposed flag described at the end of the email -Howard -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, September 13, 2000 4:52 AM To: Hall, Howard Subject: Re: Comments on the Draft 1. General - Loss of synchronization - we hope to be able one day to get to a better solution but I would suggest to recommend the implementer to drop the connections as it won't be able to parse a reset too, reinstate them and then try a reset to distinguish it clearly from other events. If it happens in a target - drop - and wait for a reconnect then send an AE to indicate the event and wait for reset. As for inconsistencies - I've added your proposal to the todo list - thank you. 2. The view is a qualifier for the name - it allows a target to a second level of refinement about what he things it should show the initiator. The first part of the name itself is the first level - but for those that are conservative in the use of names they are left with the view. 3. We structured it to follow closely FCP - for bridging reasons. As far as I can recall Response comes first followed by sense. We will state it explicitly in the document thanks. 4.timeouts - yes you are right and I stated it already in an older note about bridging. We where thinking about three timers: - command delivery - data delivery - status delivery Those should be set at the session initiation and be known to the initiator and target but they don't have to be necessarily sent over the wire (as they imply mostly local actions in the endpoints). 5. immediate data length negotiation - will fix the negotiation parameters - thanks. We think also about limiting it by default to 64k(?) 6. iSCSI check - the reason I choose to make it SCSI check too is to get a "hook" into ACA behavior. I think that is mandatory - without it all hell breaks loose! Both initiator and target could be built the with ACA on check condition. We can remove the iSCSI status field and add a set of basic sense bytes detailing the error. Let us keep talking on this item. 7. The security parts are being rewritten just as we talk and we hope that we will all see them soon 8. Target reset will be changed to have two types - soft and hard reset. 9. closing connection. Yes you are right it needs more text. And we already agreed to add a logout (on a different connection to force connection close) and remove the RID from login Thanks for your careful reading and hope to see you on row 1 or 2 at the next IETF meeting, Julo "Hall, Howard" on 13/09/2000 02:12:38 Please respond to "Hall, Howard" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Comments on the Draft Here are some detailed questions, issues, and proposals we have come up with after reviewing the draft closely. -Howard Howard Hall Pirus Networks www.pirus.com ----------------------- Section: General We need to explicitly describe in the draft how to handle loss of iSCSI parsing synchronization on a TCP stream. Resynchronization can be accomplished by shutting down the TCP connection. If it's the command stream that gets reset, iSCSI needs to "remember" an error code for this session (for how long?), so that it can let its client know what happened when the command session gets reestablished. Similarly, suggestions should be made on how to respond when the other side sends messages that don't make sense, i.e.: a target sends bogus RTT messages, or repeated "opcode not understood" We propose that the draft, in general, should state that initiators and targets should 'play safe' - especially avoiding bogus reads and writes. Initiators can use the 'escalating big hammer' approach, terminating the task, terminating the connection/session, resetting the device. Finally, what should a target do if it detects inconsistencies in the data buffers for a write command e.g.. offsets over end of total transfer size, offset overlaps, data portion greater than total transfer length, transfer tags which don't correspond with any in progress etc. What should an initiator do when it decides that at target is behaving illegally? For example, what if a 512 byte write is sent and then we receive an RTT for offset 2048? We propose that the draft specifies to abort the write command and send a response with the ISCSI status set to 1 ( iscsi check ). Better yet - since we have 8 bits of iSCSI status, why not have more values than just "good" and "iscsi check" (e.g. not_logged_on, data_in_timeout, buffer_address_inconsistency). After seeing a couple consecutively it should shut down the connection or better yet send the proposed "Out of Sync" command described below. Section: General There's no framing of the headers and data on the buffers from TCP. If anything goes wrong with the parsing, its difficult if not impossible to recover. It only takes one length field to be 'off'. If this happens the target will probably generate lots of "Opcode not understood" messages. We suggest one of two methods: 1) after seeing consecutive "Opcode not understood" messages it should shut down the connection if this doesn't solve the problem then reset the target, or 2) When the target finds that it is out of sync with the initiator ( on receipt of an "Opcode not understood"), it will send a new iSCSI "Out of Sync" command to the initiator. The initiator will assume at the reception of the "Out of Sync" command that all unacknowledged outstanding requests have been dropped. The initiator then sends the next command with the OOB (out of band) bit set, and with the OOB offset pointing to the beginning of the iSCSI header. The target, after sending the "Out of Sync" command, should ignore every thing on that connection and wait for the OOB data to re-sync again. This exchange could also work if sent from the initiator to the target. Section 2.2.2: Ordering and iSCSI numbering "The initiator and target are assumed to have three registers that define the allocation mechanism - CmdRN, ExpCmdRN, MaxCmdRN.... The target and initiator registers are supposed to uphold causal ordering." This indicates that these registers guarantee ordered delivery of commands among multiple TCP connections in a session. However, the spec continues in the following paragraphs with the following statement which seems to negate this: "iSCSI targets are not required to use the numbering scheme for ordered delivery even when they support multiple connections." We believe that the draft should clarify the issue by either: 1) stating that in-order delivery MUST be guaranteed, or 2) limiting the use of the above three registers only to indicate the command queue depth of the target, and remove any mention of their role for in-order delivery. Section 2.2.6: What is the view used for? Is it's intent so different customers get different LUNs views (like FC zoning)? Section: 3.1.1 The draft suggests that the payload to the SCSI response (opcode 0x41) could contain response data and sense data together. However, the description is ambiguous in terms of explaining how to interpret any such data that follows the header - for instance if it contains both, which one comes first, could there be a gap between them, are there alignment issues etc. Should a target's response or sense data be included in a SCSI data response buffer? In any case, its seems more natural that an adapter would provide these separately ( response data for inquiry, sense for check condition etc ). So, in the interests of simplicity we suggest limiting the scsi response messages to contain EITHER response data, OR sense data, but not both. We know which because either the res_len or the sense_len will have a non zero value. Section: 3.10 There should a be a timeout interval for getting in all the data buffers associated with a write and a read. Incomplete data may hang around indefinitely if the TCP session stays open. We propose that the target needs to have a timeout for WRITE, and the initiator needs to have a timeout for READ. The spec should specify the length of the timeout and what the recovery action is. One suggestion is a timeout in the order of seconds, with the recovery action of closing the TCP connection, starting a new connection, and re-issuing the command. The time out parameter could be passed in the Login Parameters field. Section: 2.2.4 It states in section 2.2.4 that "an initiator may request, at login, to send immediate data of any size and a target may indicate the size of immediate data blocks it is ready to accept in its response." But the draft is ambiguous as to how this size negotiation gets done. Is it a text command format? If so what is the syntax? We propose: "ImmediateData: MaxSize" where MaxSize is the maximum size of immediate data the target is willing to accept in bytes. Section: 3.3 The SCSI Response message defines a field 'iSCSI Status'. The value 1 means 'iSCSI Check'. In some error cases the command is being rejected without being sent to the scsi layer, because the iscsi target layer found a problem. Section 3.3.4 of the draft states: "if the iscsi field is not 0 the command status will indicate CHECK CONDITION". But the Command Status field is supposed to be the SCSI status (section 3.3.3). If the command never went to the SCSI layer, then a specific value should not put in that field. We propose in these cases to change iscsi_status to the condition (more values than just "good" and "iscsi check" i.e. not_logged_on, data_in_timeout, buffer_address_inconsistency), and command status to 0. This informs the initiator the iscsi target layer had a problem with this command and never passed it on to the SCSI layer, allowing an unambiguous separation of SCSI-related error conditions from iSCSI ones. Section: 3.11.3 This section states that if the key is not recognized, it should be ignored. How should malformed Text commands be handled? It can be handled by: 1) ignore the command, 2) Send a response with no keys, 3) Close down the session, 4) error response (however this may provide information to an attacker) Section: 3.13.4 the login "parameters passed for a clear-text password authentication are: Initiator:[/modifier] Target:[/modifier] Authenticator:open-sesame Access-Id:value" Access-Id is ambiguous? How does it differ from the Initiator? It is not described in Appendix B. Sections: 3.14.2, 3.11.3, and 3.12.3 A Text Command is sent because the Login Response indicated "additional authentication required." The login can now complete, and the target should send a Login Response indicating "accept login." Does the target have to send a Text Response in addition to the Login Response? The text response description makes it sound like all text keys must be sent back to the initiator, if they are accepted. This would only be the case for certain keys (such as UseRTT), correct? Section: 3.17 The map command is very ambiguous and needs further definition. Does the iSCSI initiator or target issue this command? Sections 3.6-3.8 When an iSCSI target receives a Task Management command specifying "Target Reset", the iSCSI target may send an Asynchronous Event to any initiators connected to the target device. What is the intended session shutdown sequence? Does each initiator wait for the target to close the TCP connection? Does each initiator immediately close its TCP connection? Are the initiators expected to wait for a certain amount of time prior to re-opening their TCP connections? This area needs more work. General Proposal: We propose that a flag be added to the SCSI Data PDU to indicate that a PDU is the last one for the "current data transfer". This is most useful when the current data transfer is the data being sent in response to an RTT: The target knows not to expect any more data for that RTT when it sees the bit set. This simplifies the termination of the data transfer, particularly if overlapping data requests are sent or the initiator fails to send all of the requested data. The bit might also be useful for data transfers to the initiator in response to a SCSI command, where the data transfer can be terminated and checked without having to watch for a SCSI Response packet. Finally, it may belong in the SCSI Command PDU (for immediate data) and perhaps the SCSI Response PDU (if non-sense data can show up there) for consistency of implementation. From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24900; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA15497 for ips-outgoing; Fri, 15 Sep 2000 09:44:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA15491 for ; Fri, 15 Sep 2000 09:44:53 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 4B58BED2; Fri, 15 Sep 2000 06:44:52 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id GAA13124; Fri, 15 Sep 2000 06:44:43 -0700 (PDT) Message-Id: <4.2.2.20000915060545.00b45d70@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Fri, 15 Sep 2000 06:10:22 -0700 To: "Y P Cheng" , "David Robinson" , From: Michael Krause Subject: RE: a vote for asymmetric connections in a session In-Reply-To: <000b01c01cdc$954b5960$65c809c0@yp_portable.advansys.com> References: <200009111827.LAA04921@ha10nwk.EBay.Sun.COM> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 10:12 AM 9/12/00 -0700, Y P Cheng wrote: >With or without ACK, a new NIC adapter has taken on a lot of TCP tasks such >as reordering, retransmission, and congestion flow control. (See my other >posting on how the RTT and data transfer are handled like having an >asymmetric queue.) In fact, the adapters with VI implementation can easily >perform RDMA for TCP, hence, eliminating the TCP memory-to-memory copy >overhead. Zero processor copy implementations of TCP have been implemented and can be implemented with an iSCSI engine in hardware. I don't see this as an issue. I do agree that the use of RDMA is a very compelling and would like to see this specified. Back at the first BOF, I requested that the RDMA be aligned with InfiniBand's semantics and potentially headers (would fit in the TCP options space) since this would allow simple bridging to be implemented independent of the underlying I/O interconnect (PCI, PCI-X, InfiniBand) in the long-run while enabling higher-level storage protocols such as SVP to re-use the iSCSI hardware (I believe some one else noted this benefit as well). >Without a NIC card, we can implement equivalent functions in the >iSCSI driver. I hope in defining the iSCSI protocol we should consider the >power of the new NIC adapters without worrying too much on the old >programming model of TCP queuing and stack processing. It is not difficult to implement in hardware nor requires that much in terms of driver modifications to accommodate. The issue is how one implements the h/w / s/w interface and as many FC implementations have shown, this is not that difficult to do and achieve zero processor copy solutions. Mike From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24918; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA17759 for ips-outgoing; Thu, 14 Sep 2000 19:33:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA17746 for ; Thu, 14 Sep 2000 19:33:21 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id TAA73234; Thu, 14 Sep 2000 19:27:20 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8ENXKe19020; Thu, 14 Sep 2000 17:33:20 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: 2.2.6. Naming & mapping To: Pierre Labat Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Thu, 14 Sep 2000 16:31:55 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/14/2000 05:33:19 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, We are again mixing issues and layers again. There is NO SUCH THING as an iSCSI LU. There is an iSCSI device which is a Controller which will have a SCSI layer which in turn supports an LU. Once the iSCSI session is established, the Rules of SCSI define how the LUs are addressed. There maybe additional Database (LDAP) processes and information that attempts to relate LU #3 known to Host xyz to some name "abcd...." etc. But that is NOT an iSCSI Transport Protocol. We may need to work on this at some point but it is not an iSCSI transport issue. . . . John L. Hufferd Pierre Labat @ece.cmu.edu on 09/14/2000 10:06:54 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: iSCSI: 2.2.6. Naming & mapping julian_satran@il.ibm.com wrote: > Not again (what is the sign for frustration?)... I mean not before the next > version. > > Julo > > Raghavendra Rao on 14/09/2000 21:04:02 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: 2.2.6. Naming & mapping > > I feel that a LUN should be very much part of the Naming scheme > that has been proposed in the draft, regardless of the level of > enforcement of the scheme by implementors. LUNs are very much > part of SCSI addressing, and where LUN doesn't exist, zero is > assumed. > > Without a LUN identifier, Naming section doesn't look complete. > > Do you have insights why it is omitted ? Is this by design or overlook ? > > Thanks. > -JP About the topic of naming a LU, what seems to be a benefit for me, is to adopt the same requirement as fibre channel: each LU MUST provide a unique identifier (from the Device Identification Page). >From what i read it seems that it is not a big deal to add this page in a LU. However, this unique identifier is not needed for iSCSI protocol to work. It doesn't interact with the protocol. But from an administration point of view, to configure a server using storage through iSCSI, the existence of this unique identifier helps a lot. It allows the configuration product to know/check if various LUNs correspond to the same LU, it could help to manage the LU migration (the LUN is changed inside a target for the same LU). It simplifies the configuration software by avoiding it to fake a unique LU identifier. It will help in having a configuration tool for iSCSI that can be closer to the one used with FC. Is somebody knowing if it is planned to incorporate this Device Identification Page in the iSCSI LU? Will it be mandatory? Which format (identifier type) will be used? FC uses the type 3 (FC_PH Name_Identifier). Regards, Pierre From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24903; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA29448 for ips-outgoing; Fri, 15 Sep 2000 04:00:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA29440 for ; Fri, 15 Sep 2000 03:59:58 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA10566; Fri, 15 Sep 2000 09:59:26 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA31562; Fri, 15 Sep 2000 09:59:26 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695B.002BE1D7 ; Fri, 15 Sep 2000 09:59:18 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Pierre Labat , ips@ece.cmu.edu Message-ID: Date: Fri, 15 Sep 2000 10:59:11 +0300 Subject: Re: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk The current iSCSI draft allows for (successful) status to be sent with the last data PDU. This should also be done in the asymetric case, precisely for the reason articulated by Pierre. The NIC receiveing the data will know when to perform the interrupt when the data transfer has completed and return a good status, and only one interrupt will be required to complete the data transfer. (The data buffers are registered to the NIC together with the Initiator Task Tag [or Transfer Tag], so successful status received on the data NIC can be easily associated with the original command, and there is no need for an interrupt on the command NIC.) - Kalman Pierre Labat on 13/09/2000 18:57:57 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: (bcc: Kalman Meth/Haifa/IBM) Subject: Re: Avoiding deadlock in iSCSI <... deleted ...> The fact that in the Kalman draft the end of data notification and command status comes on two different TCP connection (thus 2 NICs most of the time) implies an important CPU overhead on the initiator to manage the READ command completion as described by Somesh Gupta. You need for each READ to work accross two NICs. You will double the work on each read completion. Moreover you will double the number of cache misses because you need to read completion information from two NICs. And cache misses are really stalling your CPUs. It's a reason for what i can't agree on the Kalman proposal. You don't have this penalty in the proposal below. Regards, Pierre ,... deleted ...> From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24909; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA23729 for ips-outgoing; Thu, 14 Sep 2000 23:25:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA23725 for ; Thu, 14 Sep 2000 23:25:52 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 14 Sep 2000 23:25:17 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F9A@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI and FC encapsulation Date: Thu, 14 Sep 2000 23:25:17 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I understand a desire to start with a clean slate and then look to see how > FCP can map into whatever is created. A valid alternative would be to take > FCP structures 'as is' and define use within constructs of IP transport. A > considerable advantage from control and documentation would immediately fall > into place. Unless there is something abhorrent in starting from a known > point, it should receive consideration. If you want to see something other than what Charles Monia has proposed, I suggest writing and submitting an Internet-Draft with an equivalent level of detail to the current drafts. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24920; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA15820 for ips-outgoing; Thu, 14 Sep 2000 18:31:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from rly-ip02.mx.aol.com (rly-ip02.mx.aol.com [152.163.225.160]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA15816 for ; Thu, 14 Sep 2000 18:31:47 -0400 (EDT) Received: from tot-tf.proxy.aol.com (tot-tf.proxy.aol.com [152.163.197.131]) by rly-ip02.mx.aol.com (8.8.8/8.8.8/AOL-5.0.0) with ESMTP id SAA16929 for ; Thu, 14 Sep 2000 18:31:12 -0400 (EDT) Received: from VAIO.PacBell.net (AC9A7E23.ipt.aol.com [172.154.126.35]) by tot-tf.proxy.aol.com (8.10.0/8.10.0) with ESMTP id e8EMV9h06201 for ; Thu, 14 Sep 2000 18:31:10 -0400 (EDT) Message-Id: <4.3.2.7.2.20000914060947.00bb28a0@PacBell.net> X-Sender: Celeborn@PacBell.net X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Thu, 14 Sep 2000 06:52:01 -0700 To: IP Storage From: Peter Johansson Subject: RE: Avoiding deadlock in iSCSI In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Apparently-From: PJohansson@aol.com Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:42 AM 9/13/00, julian_satran@il.ibm.com wrote: >Where would you suggest it should be done without violating layering >constraints? Your query is a bit laconic, Julo, but to put it in the context of how to preserve ordering in the event of a queue full condition, I misspoke earlier. I was under the mistaken impression that a QUEUE FULL condition resulted in a CHECK CONDITION and the concomitant creation of ACA. Jim McGrath pointed out that this is not so. Of the remedies he outlined, of course it's possible for the initiator to refrain from issuing more than one ordered task at a time. This is reasonable for many environments, for example those in which the preponderance of tasks are unordered with the occasional need of an ordered task to fence some synchronization point. In other environments, either purely ordered or one in which a high number of tasks are ordered, this could be inefficient. I like Jim's suggestion of a QACA bit because it does not lump the QUEUE FULL condition together with other ACA events. It wasn't clear whether Jim was referring to a QACA bit in a mode page or in a CDB or both. Although this is, strictly speaking, a T10 matter, it involves interactions with the iSCSI protocol's capabilities and limitations and therefore would benefit from participation by this working group. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24912; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA22173 for ips-outgoing; Thu, 14 Sep 2000 22:18:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ns.falconstor.com.tw (ns1.falconstor.com.tw [211.75.17.117]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA22169 for ; Thu, 14 Sep 2000 22:18:06 -0400 (EDT) Received: from NEWTON (h1-51.falconstor.com.tw [172.30.1.51]) by ns.falconstor.com.tw (8.10.1/8.10.1) with SMTP id e8F2I5019726 for ; Fri, 15 Sep 2000 10:18:05 +0800 Message-ID: <003901c01ebb$4d0206f0$33011eac@NEWTON> From: "Chee-Siong, Lee" To: References: <200009130053.UAA06701@newdev.harvard.edu> Subject: Re: TCP speed Date: Fri, 15 Sep 2000 10:18:52 +0800 Organization: FalconStor Inc. MIME-Version: 1.0 Content-Type: text/plain; charset="big5" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk Is the kernel will use both cpu to process the TCP queue if running a SMP machine? When using multithread UDP process, we can balance the load in transmission with 2 CPUs. --------- cslee From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24919; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA16958 for ips-outgoing; Thu, 14 Sep 2000 19:07:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from rly-ip02.mx.aol.com (rly-ip02.mx.aol.com [152.163.225.160]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA16949 for ; Thu, 14 Sep 2000 19:07:07 -0400 (EDT) Received: from tot-tf.proxy.aol.com (tot-tf.proxy.aol.com [152.163.197.131]) by rly-ip02.mx.aol.com (8.8.8/8.8.8/AOL-5.0.0) with ESMTP id TAA26204 for ; Thu, 14 Sep 2000 19:06:36 -0400 (EDT) Received: from VAIO.PacBell.net (ACA0FD12.ipt.aol.com [172.160.253.18]) by tot-tf.proxy.aol.com (8.10.0/8.10.0) with ESMTP id e8EN6Wh26041 for ; Thu, 14 Sep 2000 19:06:33 -0400 (EDT) Message-Id: <4.3.2.7.2.20000914153732.00bbc390@PacBell.net> X-Sender: Celeborn@PacBell.net X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Thu, 14 Sep 2000 15:42:05 -0700 To: IP Storage From: Peter Johansson Subject: RE: iSCSI: 2.2.6. Naming & mapping In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Apparently-From: PJohansson@aol.com Sender: owner-ips@ece.cmu.edu Precedence: bulk At 03:18 PM 9/14/00, DeKoning, Rod wrote: >Since he hit one of my hot buttons regarding mapping (and therefore, >discovery), . . . I heartily endorse Pierre's assertion that the Device >Identification Page be used to identify Logical Units (Specifically the >Volume WWN flavor). This is extremely important in an environment that >may access volumes (logical units) through many paths and potentially >through multiple underlying ports or physical devices. I very much agree with both Pierre's and Rod's observation: LUs should have distinctive unique identifiers, else it is impractical to sort out multiple path access to the same device instance. >The only question is: does this fit in a protocol spec such as iSCSI? YES. It is entirely appropriate for a part of iSCSI to profile what it expects of the targets that utilize it. From the SAM, SPC, etc. smorgasbord, iSCSI may insist on some features (autosense, unique identifiers for each LU and so on), prohibit others and note the optionality of the remainder. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24916; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA21424 for ips-outgoing; Thu, 14 Sep 2000 21:42:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA21420 for ; Thu, 14 Sep 2000 21:42:39 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 14 Sep 2000 21:41:56 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F94@corpmx9.isus.emc.com> To: dotis@sanlight.net, Black_David@emc.com, ips@ece.cmu.edu Subject: FC encapsulation Date: Thu, 14 Sep 2000 21:41:55 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, I think you missed the point ... > The FCoverIP specification has already defined FC encapsulation. Should > there be additional information required to transport an FC frame over IP, I suggest you reread Charles Monia's proposal for use of FCP formats in iSCSI - this is the sort of thing that T10 is asking for, although the request is not for slavish adherence to FCP in all aspects. This is fundamentally different from the FC over IP approach of encapsulating SCSI in FCP in FC-2 and then in an IP transport of some form. It's not at all clear that FC-2 is necessary in this context. FWIW, the current version of FC over IP doesn't even encapsulate all of FC-2. For example, the ABTS basic link service used by FCP to abort a SCSI task is missing. I presume that this is an oversight that will be corrected in a future version of the FC over IP draft, but I suspect that this isn't the only basic or extended link service that needs to be added. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Sep 15 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24914; 15 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA21904 for ips-outgoing; Thu, 14 Sep 2000 22:05:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA21900 for ; Thu, 14 Sep 2000 22:05:23 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 14 Sep 2000 22:04:46 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100F96@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: Creation of new LUNs Date: Thu, 14 Sep 2000 22:04:46 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk This sort of proposal should go to T10. A potential issue is that not all SCSI transports support Asynchronous Event Notification (AEN). iSCSI could make AEN mandatory to implement as part of addressing this situation, but standardizing the use of AEN to report LUN addition needs to be addressed in T10. An I-D summarizing the problem and proposed solution without reference to iSCSI or any other transport might be a good start on this. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Thursday, September 14, 2000 10:13 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Creation of new LUNs > > > > Is somebody else doing it differently? > > Julo > > Raghavendra Rao on 14/09/2000 21:19:58 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: Creation of new LUNs > > > > > > What is the recommended/approved way of announcing the availability > of new LUNs in the target ? Currently I can do the following, but if > it isn't explicitly documented, implementors tend to do their own > things (causing interoperability problems) > > Upon creation/deletion of a LUN in the target, a target shall generate > an asynchronous notification with the SCSI event indicator set to > either 2 or 4, followed by sense data with appropriate ASC and ASCQ > indicating that LUNs have changed. The initiator, on receipt of this > notification will issue a REPORT_LUN command to get a list of LUNs. > > -JP > From owner-ips@ECE.cmu.edu Fri Sep 15 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25001; 15 Sep 2000 13:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA21159 for ips-outgoing; Fri, 15 Sep 2000 11:52:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA21151 for ; Fri, 15 Sep 2000 11:51:54 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA52752 for ; Fri, 15 Sep 2000 11:50:09 -0400 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8FFppa29576 for ; Fri, 15 Sep 2000 09:51:52 -0600 Importance: Normal Subject: RE: iSCSI: Creation of new LUNs (and AENs) To: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Fri, 15 Sep 2000 08:51:50 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Release 5.0.4a |July 24, 2000) at 09/15/2000 08:51:51 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, Whenever anybody suggests AEN, I always want to ask the question: To what initiator or initiators does a target send an AEN? The target shouldn't "broadcast" this sort of thing. It has to be more picky about it, *especially* in this networked world. So, if you suggest AEN for anything, you have to answer (and spec the answer) to this question. Creating new LUs has already been spec'd. If LU inventory changes for any reason, a REPORTED LUNS DATA HAS CHANGED unit attention condition is set up. See SPC-2, rev18, page 141 in the REPORT LUNS command section. (I might agree that this isn't necessarily completely sufficient, but it does address the issue in most cases. But I don't think iSCSI needs to get involved in this issue.) Jim Hafner Black_David@emc.com@ece.cmu.edu on 09-14-2000 07:04:46 PM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Creation of new LUNs This sort of proposal should go to T10. A potential issue is that not all SCSI transports support Asynchronous Event Notification (AEN). iSCSI could make AEN mandatory to implement as part of addressing this situation, but standardizing the use of AEN to report LUN addition needs to be addressed in T10. An I-D summarizing the problem and proposed solution without reference to iSCSI or any other transport might be a good start on this. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Thursday, September 14, 2000 10:13 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Creation of new LUNs > > > > Is somebody else doing it differently? > > Julo > > Raghavendra Rao on 14/09/2000 21:19:58 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: Creation of new LUNs > > > > > > What is the recommended/approved way of announcing the availability > of new LUNs in the target ? Currently I can do the following, but if > it isn't explicitly documented, implementors tend to do their own > things (causing interoperability problems) > > Upon creation/deletion of a LUN in the target, a target shall generate > an asynchronous notification with the SCSI event indicator set to > either 2 or 4, followed by sense data with appropriate ASC and ASCQ > indicating that LUNs have changed. The initiator, on receipt of this > notification will issue a REPORT_LUN command to get a list of LUNs. > > -JP > From owner-ips@ECE.cmu.edu Fri Sep 15 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25018; 15 Sep 2000 13:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA22585 for ips-outgoing; Fri, 15 Sep 2000 12:25:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA22575 for ; Fri, 15 Sep 2000 12:25:36 -0400 (EDT) Received: from rajlaptop (dhcp049.lightsand.com [192.168.1.49]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id JAA01838; Fri, 15 Sep 2000 09:24:25 -0700 (PDT) From: "Raj Bhagwat" To: , , Subject: RE: FC encapsulation Date: Fri, 15 Sep 2000 09:23:34 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100F94@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk David, The 'FC over IP' draft specifies how to encapsulate any FC frame with no regard to what is inside the FC frame. BLS/ELS commands have regular FC frame format (FT-1) with 24-byte FC header and optional payload. The things 'FC over IP' cannot encapsulate are primitive signals and primitive sequences because they are not FC frames. These are meant for physical link level operation only. Regards, Raj. -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Black_David@emc.com Sent: Thursday, September 14, 2000 6:42 PM To: dotis@sanlight.net; Black_David@emc.com; ips@ece.cmu.edu Subject: FC encapsulation Doug, I think you missed the point ... > The FCoverIP specification has already defined FC encapsulation. Should > there be additional information required to transport an FC frame over IP, I suggest you reread Charles Monia's proposal for use of FCP formats in iSCSI - this is the sort of thing that T10 is asking for, although the request is not for slavish adherence to FCP in all aspects. This is fundamentally different from the FC over IP approach of encapsulating SCSI in FCP in FC-2 and then in an IP transport of some form. It's not at all clear that FC-2 is necessary in this context. FWIW, the current version of FC over IP doesn't even encapsulate all of FC-2. For example, the ABTS basic link service used by FCP to abort a SCSI task is missing. I presume that this is an oversight that will be corrected in a future version of the FC over IP draft, but I suspect that this isn't the only basic or extended link service that needs to be added. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Sep 15 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25066; 15 Sep 2000 13:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA22230 for ips-outgoing; Fri, 15 Sep 2000 12:16:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA22222 for ; Fri, 15 Sep 2000 12:16:45 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id LAA24356 for ; Fri, 15 Sep 2000 11:10:26 -0500 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8FGGea111502 for ; Fri, 15 Sep 2000 10:16:41 -0600 Importance: Normal Subject: Re: iSCSI: 2.2.6. Naming & mapping To: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Fri, 15 Sep 2000 09:16:39 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Release 5.0.4a |July 24, 2000) at 09/15/2000 09:16:40 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, In spite of John's request to not go into this topic now (and concentrate on the protocol stuff), I want to get my two cents in as this is one of my hot buttons too. In this space, you have to make a distinction between "name" and "address" (e.g., ipname, vs ipaddress vs MAC address). Names are good for two purposes: (a) management interfaces and (b) passing between third parties. Addresses are good when an initiator wants to talk to a target and it has no (well almost no) value for the other purposes. For names, the amount of information in a name string depends on the context. For management interfaces, you'd want to have ipname plus the LU identifier. For third party operations, that should be sufficient as well, but it further depends on the context. For example, EXTENDED COPY commands already have mechanisms for either identifying the "device" and the LU OR for identifying the LU with LU WWIdentifiers. The device field is currently too short (only 8 bytes) but that is being addressed in T10. The LU is currently spec'd as LUN, but enhancements to that have been approved (proxy tokens in access controls -- t10/99-245r9) that solve the third party addressing problem. The EVPD page 83h LU WWIdentifier (or whatever it's called) spec'd in SPC-2 has just be made mandatory for SPC-2 (at yesterday's T10 meeting). So iSCSI needn't ask this as a requirement. Frankly, I think iSCSI needs only to spec the mechanism for a host to "walk the bus" (i.e., how does the host determine what storage devices it should talk to on the network). After connecting to that device, the standard SCSI logical unit discovery takes over and no more is needed from iSCSI. About LU Views: I STRONGLY suggest that iSCSI stay away from this topic. The access controls provides a SCSI solution to this issue that is in fact well suited to the iSCSI environment. (There will need to be a small addition to the access controls for iSCSI, namely, TransportIDs, but that's a joint T10/IPS detail that can be handled later.) Adding additional layers on top of that in the iSCSI space is (IMHO) a mistake. Thanks for giving me the forum for today! Jim Hafner John Hufferd/San Jose/IBM@IBMUS@ece.cmu.edu on 09-15-2000 12:32:22 AM Sent by: owner-ips@ece.cmu.edu To: Pierre Labat cc: ips@ece.cmu.edu Subject: Re: iSCSI: 2.2.6. Naming & mapping Pierre, You said "Yes, it is not a transport issue, but why not request that now before people start building iSCSI controllers if it simplifies the life of everybody? I don't know how FC managed to have their WWN but why could not we do the same thing?" Again, you mixed concepts, that is a "port" attribute (WWN) which is a Transport item, and LUs specific names which is a SCSI issue. However, there are still things associated with iSCSI which may address some of the things you are interested in. Specifically, there are reasons for a Connection to be "bound" to a specific set of LUs We are defining a concept of a LU view, that will permit, the storage controller to know what LU Views to give to any specific connection. This is of value to the Storage Controller as an entity, and the names and associations will probably be saved in the Discovery Database (LDAP). However, this is my hallucination, and has not been defined by the Workgroup. The Workgroup has already agreed to work on the items that are needed for Discovery, etc. in one of the next work assignments following the definition of the iSCSI protocol. There will also be the need to have a process that defines Names and View Mapping (see section 2.2.6). There will need to be, of course a way for the setting and naming the LU View maps. All of that is beyond the iSCSI protocol specification. The iSCSI protocol (2.2.6 & 3.17), however, has specified the protocol needed to carry the View information. Now, please lets focus on the iSCSI Transport Protocol Draft and take on these LU naming issues at the approprate time. . . . John L. Hufferd From owner-ips@ECE.cmu.edu Fri Sep 15 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25427; 15 Sep 2000 14:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA24199 for ips-outgoing; Fri, 15 Sep 2000 13:04:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA24192 for ; Fri, 15 Sep 2000 13:04:18 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA10980; Fri, 15 Sep 2000 09:53:37 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA23603; Fri, 15 Sep 2000 09:51:17 -0700 From: "Douglas Otis" To: "Raj Bhagwat" , , Subject: RE: FC encapsulation (FCIP) Date: Fri, 15 Sep 2000 10:06:41 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Raj, You should review the document. There is encoding for these primitives as both prefix and suffix. An alternative scheme might be to place both prefix and suffix into the prefix. This would prevent a simple linear encode, but would make accessing these items a bit easier. Of course, you could use the encapsulated frame size parameter as a means to find the suffix directly. As far as being a complete document, implemented within SCTP actually provides a much higher level of documentation than is present within iSCSI specification. Much of the configuration management, I argue, does not belong within the transport specifications. LDAP does a much better job in providing configuration management. Using pre-existing tools allows flexibility in management required whether it is the client or the provider. Within LDAP, you may wish to add flags to prohibit acceptance of various settings from various clients on the control stream as example. What you will find missing in this document is how to go about mapping and filtering frames. Something iSCSI has yet to address as well. As some point, you may wish to map a subset to FC or Ethernet cable as example. Should this standard be used as a bridge, a timestamp could be used within a prefix to decide if the frame is stale. Such an option would not be required if it was not acting as a bridge. A simple buffer will add adequate latency to provide effective bandwidth control within a bridge by taking advantage of burst limits. Doug > -----Original Message----- > From: Raj Bhagwat [mailto:rajb@lightsand.com] > Sent: Friday, September 15, 2000 9:24 AM > To: Black_David@emc.com; dotis@sanlight.net; ips@ece.cmu.edu > Subject: RE: FC encapsulation > > > David, > > The 'FC over IP' draft specifies how to encapsulate any FC frame with no > regard to what is inside the FC frame. BLS/ELS commands have regular FC > frame format (FT-1) with 24-byte FC header and optional payload. > The things > 'FC over IP' cannot encapsulate are primitive signals and primitive > sequences because they are not FC frames. These are meant for > physical link > level operation only. > > Regards, > > Raj. > > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > Sent: Thursday, September 14, 2000 6:42 PM > To: dotis@sanlight.net; Black_David@emc.com; ips@ece.cmu.edu > Subject: FC encapsulation > > > Doug, > > I think you missed the point ... > > > The FCoverIP specification has already defined FC encapsulation. Should > > there be additional information required to transport an FC frame over > IP, > > I suggest you reread Charles Monia's proposal for use of FCP formats in > iSCSI - this is the sort of thing that T10 is asking for, although the > request > is not for slavish adherence to FCP in all aspects. This is fundamentally > different from the FC over IP approach of encapsulating SCSI in FCP in > FC-2 and then in an IP transport of some form. It's not at all clear that > FC-2 is necessary in this context. > > FWIW, the current version of FC over IP doesn't even encapsulate all > of FC-2. For example, the ABTS basic link service used by FCP to > abort a SCSI task is missing. I presume that this is an oversight that > will be corrected in a future version of the FC over IP draft, > but I suspect > that this isn't the only basic or extended link service that needs to be > added. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > From owner-ips@ECE.cmu.edu Fri Sep 15 23:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26841; 15 Sep 2000 19:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA07268 for ips-outgoing; Fri, 15 Sep 2000 18:19:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA07264 for ; Fri, 15 Sep 2000 18:19:04 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 15 Sep 2000 15:19:47 -0700 Message-ID: From: Charles Monia To: Stephen Byan , IPS Reflector Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Date: Fri, 15 Sep 2000 15:19:37 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Wednesday, September 13, 2000 6:22 AM > To: IPS Reflector > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote: > > > I don't think VI/TCP helps at all. The VI is implemented on > > top of a TCP > > stream. If the TCP stream delivers the iSCSI command to VI, > > and VI has no > > place to put it, what is VI going to do? It has to block the > > TCP stream, and > > that in turn will block and "RDMA" from occurring. > > If I understand correctly, VI/TCP has a credit-based flow > control mechanism > on its message queues (which would be used to implement the > iSCSI command > reception queue), and so the initiator would never send an > iSCSI command > without a target buffer in which to store it. So RDMA cannot block on > commands in the TCP stream. > In the context of generic ULP "flow control" issues, the following observations are offered. Assuming one VI/TCP message queue handles all SCSI control traffic for a target, using flow control on messages to prevent the "queue-full" condition has undesirable side effects. Specifically, back pressure due to lack of command context resources within a logical unit would stop the flow of commands and task management requests to all other LUNS (a form of "head of line" blocking). There's another issue that is perhaps less obvious. Even if multiple message queues are used, say one per LUN, stopping the flow of messages due to a queue full condition would also stop the flow of task management messages as well. Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Sat Sep 16 01:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27384; 15 Sep 2000 21:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA10734 for ips-outgoing; Fri, 15 Sep 2000 20:35:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA10728 for ; Fri, 15 Sep 2000 20:34:55 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id RAA11099; Fri, 15 Sep 2000 17:08:18 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id RAA24681; Fri, 15 Sep 2000 17:05:58 -0700 From: "Douglas Otis" To: "Charles Monia" , "Stephen Byan" , "IPS Reflector" Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Date: Fri, 15 Sep 2000 17:21:25 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, Depending on the capabilities of the Net-SCSI adapter, such direct copies would be possible without resorting to VI. Not using VI but allowing direct placement also provides lower overhead and a safer system as pointers going astray due to a muddled target or bit rot is not a concern. SAM requires buffers to be in place prior to issuing the request. These buffers are at the client which is where VI becomes useful. Unless both ends of this communication becomes VI, you have not provided a meaningful solution. Adding VI modifies TCP. I do not think TCP is a good choice for implementing VI, but to make such modifications, you could equally allow direct copies via SCSI as well. In an attempt to improve sending FCP protocol, which is where the problem exists, these improvements have resulted in a lossy channel. I am confident a solution will be found. I am also confident one could make improvements within the transport without affecting SCSI to achieve desired results. Should these become a form of credit (VI like) or command/data ready signaling, these solutions should apply directly to FCP structures equally well. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Monia > Sent: Friday, September 15, 2000 3:20 PM > To: Stephen Byan; IPS Reflector > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > > > > -----Original Message----- > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > Sent: Wednesday, September 13, 2000 6:22 AM > > To: IPS Reflector > > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > > > > Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote: > > > > > I don't think VI/TCP helps at all. The VI is implemented on > > > top of a TCP > > > stream. If the TCP stream delivers the iSCSI command to VI, > > > and VI has no > > > place to put it, what is VI going to do? It has to block the > > > TCP stream, and > > > that in turn will block and "RDMA" from occurring. > > > > If I understand correctly, VI/TCP has a credit-based flow > > control mechanism > > on its message queues (which would be used to implement the > > iSCSI command > > reception queue), and so the initiator would never send an > > iSCSI command > > without a target buffer in which to store it. So RDMA cannot block on > > commands in the TCP stream. > > > > In the context of generic ULP "flow control" issues, the following > observations are offered. > > Assuming one VI/TCP message queue handles all SCSI control traffic for a > target, using flow control on messages to prevent the > "queue-full" condition > has undesirable side effects. Specifically, back pressure due to lack of > command context resources within a logical unit would stop the flow of > commands and task management requests to all other LUNS (a form > of "head of > line" blocking). > > There's another issue that is perhaps less obvious. Even if > multiple message > queues are used, say one per LUN, stopping the flow of messages due to a > queue full condition would also stop the flow of task management > messages as > well. > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 > From owner-ips@ECE.cmu.edu Sat Sep 16 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27907; 15 Sep 2000 23:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA13367 for ips-outgoing; Fri, 15 Sep 2000 22:34:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA13362 for ; Fri, 15 Sep 2000 22:34:37 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 15 Sep 2000 19:35:24 -0700 Message-ID: From: Charles Monia To: Black_David@emc.com, julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: Creation of new LUNs Date: Fri, 15 Sep 2000 19:35:17 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: A note on AEN support: Although all SCSI transports provide a way to deliver an AEN, few implementations support it in practice. Since a device can implement both initiator and target functions (as it would for 3rd party copy requests), the usual approach is for the initiator to also present itself as a target that supports the standard SCSI model for "host" devices. The data associated with an asynchronous event is then transferred to the initiator (temporarily acting as a target) by means of the SCSI SEND command. I guess one could characterize this as a temporary role reversal. Of course, a protocol can define some other mechanism that serves the same end. Charles > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Thursday, September 14, 2000 7:05 PM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: iSCSI: Creation of new LUNs > > > This sort of proposal should go to T10. A potential > issue is that not all SCSI transports support > Asynchronous Event Notification (AEN). iSCSI could > make AEN mandatory to implement as part of addressing > this situation, but standardizing the use of AEN to > report LUN addition needs to be addressed in T10. > An I-D summarizing the problem and proposed solution > without reference to iSCSI or any other transport > might be a good start on this. > > --David > From owner-ips@ECE.cmu.edu Sat Sep 16 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27909; 15 Sep 2000 23:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA12923 for ips-outgoing; Fri, 15 Sep 2000 22:13:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA12919 for ; Fri, 15 Sep 2000 22:13:45 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 15 Sep 2000 19:14:32 -0700 Message-ID: From: Charles Monia To: Jim McGrath , "'csapuntz@cisco.com'" Cc: ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Date: Fri, 15 Sep 2000 19:14:23 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi All: While Jim's note sums up the SCSI-related issues nicely, I'd like to add a minor filip to what he said. As I see it, the problem of ordering goes beyond whether or not a command has the ordered attribute. For certain devices, like tapes, ordered behavior may be implicit in the device semantics. Although this is a fine point, it probably needs to be spelled out more clearly in the appropriate SCSI specs. Charles > -----Original Message----- > From: Jim McGrath [mailto:Jim.McGrath@quantum.com] > Sent: Tuesday, September 12, 2000 1:09 PM > To: 'csapuntz@cisco.com'; Jim McGrath > Cc: ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > > > I agree with Stephen that this is a T10 issue. BTW, it has > been discussed > before in T10. The issue is that for commands ending with a > CHECK CONDITION > (error) status the ACA mechanism provides a very targeted > means of solving > this problem (use ACA for your ORDERED commands and you are > fine). The QErr > bit in the Control Mode page provides a similar global > capability for the > entire LUN. > > The problem is that there are other status conditions that > may want to be > treated in a similar manner, but which (to my knowledge) are > not. The first > one we looked at was RESERVATION CONFLICT. The second is > QUEUE FULL. Both > may create a desire to pause execution from the command queue. > > The current answer is simply not to get into this problem to > begin with. > That is, if you are sending an ORDERED command, make sure you > do not send > out the next command (ORDERED or UNORDERED) until you (the > initiator) know > that the first command has been received by the target. You > clearly know if > a command has been received by an explicit acknowledgement > (as in parallel > SCSI) or by a subsequent action (a data transfer or status > transfer) that > only makes sense if the command was received. > > You can argue that this degrades performance. For most of > the interconnects > considered up to now, with relatively low latency, this has > not bee as much > of a concern. In addition, there has been a feeling that the > cases were > ORDERED commands are used are so few that you either accepted the > degradation (such as it is) when you use them, or you do a > vendor unique > approach around the problem. > > If this is considered to be a big issue, then I'd raise it to T10. I > suggest you propose a solution that mimics the existing ACA and QErr > solutions, since that will be easiest and quickest to adopt. One > possibility is adding the status QUEUE FULL to the things > that generate > ACAs. Another is to propose a new QACA bit that can be used > like ACA but > for Queue Full (so users can continue to use the old ACA bit > for backward > compatibility). Since you either introduce a compatibility > issue (the first > approach) or need a bit (the second approach), the answer is > not cost free, > so T10 will want some justification for creating a > compatibility issue or > (more likely) to spend the bit. > > Jim > > > -----Original Message----- > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > Sent: Monday, September 11, 2000 10:33 PM > To: Jim McGrath > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > Subject: Re: Avoiding deadlock in iSCSI > > > > > Note that SCSI targets, when faced with getting a command > queue full, do > not > > stop reading from the interconnect. If more commands are > received then > they > > respond with a QUEUE FULL status. If data is received then > they receive > > that data without regard for the status of the command > queue (as long as > it > > is data for an already queued command). This eliminates > the potential for > > deadlock between command and data queues. > > Jim, > > I believe there is problem with the current SCSI behavior. > Consider the > following scenario for a host with a pathological queue of 1 > > > 1) Initiator sends command 1 (ORDERED attribute) > 2) Initiator sends command 2 (ORDERED attribute) > 3) Initiator sends command 3 (ORDERED attributge) > 4) Target reads command 1 > 5) Target reads command 2 > 6) Target returns queue full for command 2 > 7) Command 1 completes > 8) Target reads command 3 > 9) Target executes command 3 > > We have just violated the ordering constraints of the application by > doing command 3 before command 2. > > > Potentially you can have a situation where multiple > commands already at > the > > target have only part of their data transmitted, with the > remainder still > at > > the initiator(s), and then run out of buffer space for data. If the > target > > uses a credit model to pace the reception of data, it can > also make sure > > this never happens. Unsolicited data, even for commands > already queued, > can > > end up creating this deadlock - which is why unsolicited > data systems > either > > have to have a tight limit on the resources it can use > (e.g. low login BB > > credit in Fibre Channel terms) or some sort of clean (i.e. not IO > > terminating) rejection mechanism from target to initiator > (like in USB). > > Unsolicited data is NOT a problem with the current iSCSI > spec. We allow > the target to always drop data and request data transfers > with an RTT. > > -Costa > From owner-ips@ECE.cmu.edu Sat Sep 16 04:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28046; 16 Sep 2000 0:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA14153 for ips-outgoing; Fri, 15 Sep 2000 23:10:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA00844 for ; Fri, 15 Sep 2000 15:35:41 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Fri, 15 Sep 2000 12:36:52 -0700 Message-ID: From: Robert Snively To: "'meth@il.ibm.com'" , Pierre Labat , ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Date: Fri, 15 Sep 2000 12:34:01 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk This concerns me a little bit. In all other versions of SCSI (FCP, Parallel, etc.) there is a maximum of one completion interrupt per complete SCSI task, regardless of the number of data information units or linked commands executed. That is provided at the time the status (and autosense data if any) for the last command in the I/O operation is presented. The use of the word maximum allows for the possibility of processing multiple SCSI completions if there happens to be more than one completed during the time the interrupt context is active. I hope we can expect the same behavior of iSCSI. Bob > The current iSCSI draft allows for (successful) status to be > sent with the > last data PDU. This should also be done in the asymmetric > case, precisely > for the reason articulated by Pierre. The NIC receiving the > data will know > when to perform the interrupt when the data transfer has > completed and > return a good status, and only one interrupt will be > required to complete > the data transfer. (The data buffers are registered to the > NIC together > with the Initiator Task Tag [or Transfer Tag], so successful status > received on the data NIC can be easily associated with the original > command, and there is no need for an interrupt on the command NIC.) From owner-ips@ECE.cmu.edu Sat Sep 16 04:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28051; 16 Sep 2000 0:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA14181 for ips-outgoing; Fri, 15 Sep 2000 23:11:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA02035 for ; Fri, 15 Sep 2000 16:02:09 -0400 (EDT) Received: by thor with Internet Mail Service (5.5.2650.21) id ; Fri, 15 Sep 2000 13:03:24 -0700 Message-ID: From: Robert Snively To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Date: Fri, 15 Sep 2000 13:00:32 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk There appear to be some problems with our understanding of the ordering of SCSI commands. The architected ordering is ONLY with respect to the command stream from a single initiator to a single logical unit, and then only when specified by an ordering attribute. Streams among different logical units from the same initiator are never required to provide ordering. Streams to the same logical unit from different initiators are never required to be ordered. Streams of task management activities are never required to be ordered with respect to commands. The SCSI wedge drivers, OS file systems, RAIDs, and raw access drivers such as database file systems all deal very well with that architecture and exploit the throughput gains that are associated with selectively restricted ordering. Certainly, if all aspects of all streams are explicitly ordered, the architectural requirement will be forced to be met. However, such restrictions are a significant limitation on the flexibility of the implementation and are certainly not required by the SCSI architecture. Requiring them in iSCSI is probably equally limiting. Bob > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, September 12, 2000 11:11 PM > To: ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > > > > Somesh, > > There is a misunderstanding. You are required to keep order > only on a given > connection. > SMP or no SMP they are represented by ONE data structure (I > hope). You are > not implying order across connection as it has no meaning to > the receiver. > > Julo > > somesh_gupta@hp.com on 12/09/2000 21:33:31 > > Please respond to somesh_gupta@hp.com > > To: David.Robinson@EBay.Sun.COM, ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Avoiding deadlock in iSCSI > > > > > > > > -----Original Message----- > > From: David.Robinson@EBay.Sun.COM > [mailto:David.Robinson@EBay.Sun.COM] > > Sent: Monday, September 11, 2000 6:36 PM > > To: ips@ece.cmu.edu > > Subject: RE: Avoiding deadlock in iSCSI > > > > > > Thanks for the information, I think part of my confusion is > > the difference > > in mapping SCSI from a datagram protocol onto a reliable > > stream protocol. > > In a datagram protocol if the data is sent without the receiver's > > cooperation, the receiver's buffers may not be adequate and the > > data must get discarded. Credits and RTT can be used to handle > > this case. > > > > With a reliable stream transport like TCP, you don't get into > > this situation because the receiver will never open the TCP > > window beyond it's buffer capacity. For low amounts of buffering > > it might not be as efficient as using RTT, but there is no > > correctness or deadlock issues. Because each sender has its own > > connection and own flow control they are independantly handled. > > Likewise with seperate data connections, each is also flow > controlled > > so "unsolicted" data is not an issue. "Overflow" conditions simply > > never occur. > > > > The only major design criteria is that the sender MUST maintain > > ordering of data sent on any connection. Data Dn MUST always be > > sent before data Dm where n < m. In particular, if unsolicated > > data and RTT is mixed, the sender cannot send data Dm before it > > has recieved an RTT for data Dn if both are to use the same > > connection. > > If we are using multiple connections, this requirement has > ramifications on the implementation on SMP systems. The initiator > will be sending command on one connection, and data on another > connection, which are two different actions. To ensure ordering > would require taking a lock across the two actions (or otherwise > ensuring that they always happen on the same CPU). This would > be quite expensive actually. > > > > > -David > > > > > I think people have been meaning "unsolicited data" to > > really mean data sent > > > to a receiver without that receiver having first indicated > > that there is > > > enough buffering to hold the data. For initiators acting > > as receivers they > > > have to verify this before they initiate the command (not > > enough space for > > > the whole command? Then break up the command.) For > > Targets this requires > > > something like a credit mechanism with RTTs being used. > > > > > > So there is an "unsolicited command" problem and an > > "unsolicited data" > > > problem. In both cases the sender creates the problem > by not first > > > reserving with the receiver enough resources for the > commands/data. > > > > > > In the command case there is no SCSI mechanism to reserve > > resources (QUEUE > > > FULL is used to indicate overflows). Historically it has > > been assumed that > > > queues of commands do not overflow often in practice. In > > reality initiators > > > have often artificially limited the number of commands they > > are willing to > > > try and queue at the target in order to avoid this > rejection (a loss > > > opportunity in my mind). > > > > > > In the data case there is no "DATA QUEUE FULL" - instead, > > an explicit credit > > > model of some sort is used to indicate the receiver has > > reserved space for > > > the data (REQs in parallel SCSI, BB credits in Fibre > > Channel). In this case > > > the assumption was that data overflows would occur a lot > otherwise. > > > > > > You can solve these problems by rejecting the overflow > > cleanly (as SCSI does > > > with commands), which is low latency and works well under > > light loads. Or > > > you can do credits. Credits add latency, or get you into > > the problem of > > > credit allocation, which can be optimized for light load > > (over allocate > > > credits) or heavy loads (allocate only what you have), but > > not both at once. > > > > > > Historically, SCSI has used rejection for commands and > > credits for data, > > > optimized for heavy loads. But this is only a T10 given > > rule, not a God > > > given rule (although some of us who have served on T10 can > > get that confused > > > at times :-)). > > > > > > Hope this helps. > > > > > > Jim > > > > > > > > > > > > > > > > > > cases there are well known mechanisms to reserve the > > > > > > -----Original Message----- > > > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > > > Sent: Monday, September 11, 2000 3:35 PM > > > To: ips@ece.cmu.edu > > > Subject: Re: Avoiding deadlock in iSCSI > > > > > > > > > I think in following this discussion the terminology has been > > > confusing me. When I read "unsolicited data" I interpreted that > > > to mean data for which no command has yet been sent. In general > > > I consider that to be a bug and the receiver should just drop the > > > data on the floor. The only possible scenerio where it might > > > not be a bug is if a command was sent on one connection and the > > > data on the data connection arrived first, thus it is > unsolicited. > > > My first assumption is that the sender would not send commands > > > C1 and C2 and data D2 and D1 on the same connection. Doing that > > > creates nasty ordering problems we want to avoid. So if the > > > receiver simply allows the data connection TCP window to shrink > > > the unsolicted data will flow control to a stop until the command > > > queue catches up. With multiple data connections, some may flow > > > control but the active command will be able to make progress on > > > one connection. This may not be the most efficient mechanism but > > > it is "safe". Preferably the data will either follow the command > > > on the same data/command connection or the sender will request a > > > RTT (aka R2T). It is also a sender bug to request a connection > > > for data transfer that it has already sent "unsolicited" data. > > > > > > Unless my assumptions and definitions are wrong, I don't > > see the issue. > > > > > > -David > > > > > > > The problem: > > > > > > > > iSCSI, as currently spec'ed, allows SCSI commands and > data to be > > > > interleaved fairly freely on a TCP connection. A > target that stops > > > > reading from a TCP connection to avoid reading more > > command packets > > > > also prevents itself from reading data packets. Those > > data packets > > > > may be criticial to making progress on the currently executing > > > > command. > > > > > > > > Note the issue appears with one TCP connection for > > control and data > > > > and even appears in many of the multiple connection schemes. > > > > > > > > Data in iSCSI comes in two forms: > > > > > > > > 1) solicited - data requested by target via RTT > > > > - data requested by initiator via a > SCSI command > > > > 2) unsolicited - data sent by initiator without having > > received an > > > RTT > > > > > > > > The analysis below assumes that unsolicited data travels > > over the same > > > > TCP connection as SCSI commands. Otherwise, you run the > > risk of receiving > > > > unsolicited data before the relevant SCSI command (thus making > > > > implementations more complex). > > > > > > > > Four solutions: > > > > > > > > 1) Don't overflow the command queue (i.e. use credits) > > > > - and what do you do if a misbehaving initiator overflows > > > > your command queue anyway? Drop the connection? > > > > > > > > - requires you to reserve resources per initiator. > some people > > > > may want to overcommit > > > > > > > > 2) Allow dropping of SCSI commands when queue fills > > > > - how do you clean up after a dropped SCSI command? > > > > - there may be other commands in the pipeline > > > > > > > > One approach: On command drop, the target enters an error > > > > state. While in the error state, all newly > received commands > > > > terminate with an error until the initiator > explicitly clears > > > > the error state using a "clear error state" message. > > > > > > > > You might think that TASK SET FULL and ACA > mechanisms from SCSI > > > > could be used to attack this problem. However, > > TASK SET FULL > > > errors > > > > don't trigger ACA (in my reading of the SAM). > Also, ACA is only > > > > triggered by the current enabled command, not by > random commands > > > > entered into the task set. > > > > > > > > 3) Put solicited data on a dedicated TCP connection. > Require that > > > > unsolicited data MUST follow the command, ideally in the > > same iSCSI > > > > PDU > > > > > > > > 4) (Do it like NFS) Make all transfers from initiator to target > > > > unsolicited. Make sure unsolicited data follows the command > > > > immediately. > > > > > > > > > > > > Of all the options, #1 and #4 sound the easiest to > > implement. #2 is more > > > > sophisticated than #1. #3 is just plain clever but that's > > rarely a good > > > > thing. :) #4 has large ramifications on current SCSI > > target designs. > > > > > > > > -Costa > > > > > > > From owner-ips@ECE.cmu.edu Sat Sep 16 04:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28132; 16 Sep 2000 0:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA13994 for ips-outgoing; Fri, 15 Sep 2000 23:05:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA19911 for ; Fri, 15 Sep 2000 11:23:04 -0400 (EDT) Message-ID: <618471D08ABDD31188B6009027E5009352A0ED@apollo.pirus.com> From: "Merhar, Milan" To: "'ips@ece.cmu.edu'" Subject: RE: switch latency (was Command Queue Depth, Asymmetric/Symmetric ) Date: Fri, 15 Sep 2000 11:34:53 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Michael, You're quite right about the actual crossbars introducing minimal delays - typically, only a few tens of bit times at 2-3x the port bit rate. But, there's often a store-and-forward delay at the input port, where the first bits received sit until the last bits of the packet arrive (introducing one packet transmission time's delay, or about 12.5 usec for a 1500 byte Gigabit Ethernet packet.) This is also where packet inspection and classification delays can appear, as you correctly point out. There is a similar store-and-forward delay coming out of the crossbar, where the first bits switched may wait until the last bits clear the crossbar, before the actual packet transmission is scheduled to the outgoing port. As the crossbar may run at 2-3 times the input bit rate, this delay is less, often on the order of 4-6 usec for a maximum-size packet. Added together, this gives you 16-20 usec "first bit in to first bit out" delay for a maximum sized Gigabit Ethernet packet, which would be reported in the usual data-sheet convention as about 5 usec "last bit in to first bit out." Some of the big switches introduce a little more store and forward delay, as they use two levels of internal switching, one at the module level, and another at the chassis level. "Sub 1 usec" numbers may indicate the switch selectively uses cut-through forwarding rather than store-and-forward (tricky to get right, but certainly faster,) or may merely be the result of doing the measurements with a minimum-sized packet. As for delay numbers exceeding 1 millisecond, all I can suggest is that someone was looking at the specs for a 10 Megabit Ethernet product (where an 1518 byte packet takes 1.2 msec to transmit,) or for a software-based bridge or router, which can easily build up 1 msec of queuing time. And, of course, when you start considering large WAN networks, with multiple router hops and substantial speed-of-light delays, 30-50 msec starts to look mighty fast. Add in congestion queuing, and you can easily build up to the often quoted 150-300 msec "Internet latency." - milan -----Original Message----- From: Michael Krause [mailto:krause@cup.hp.com] Sent: Tuesday, September 12, 2000 11:21 AM To: Scott Bradner Cc: ips@ece.cmu.edu Subject: Re: Command Queue Depth (was asymmetric/Symmetric) At 07:14 AM 9/8/00 -0400, Scott Bradner wrote: > > I don't know what the > > latency through ethernet switches is, but I'd hope it wasn't in the ms > range. > >processing time is in the 10 to 50 microsecond range Don't know whose switches you are evaluating. They range from sub 1-usec to perhaps 5-10 usec - it is all a matter of how much additional processing is going on within the switch. For example, many switches examine not just the ethernet headers/tags but also look at the IP headers and provide additional functionality based on their content. This is where many switch vendors try to differentiate and is not a function of whether they can switch fast or not. The actual switching elements are usually crossbars that operate in the 100-500ns range - chip process improvements will lower that by quite a bit over the next couple of years. Mike From owner-ips@ECE.cmu.edu Sat Sep 16 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29360; 16 Sep 2000 4:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA18252 for ips-outgoing; Sat, 16 Sep 2000 02:45:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA18247 for ; Sat, 16 Sep 2000 02:45:09 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA59236 for ; Sat, 16 Sep 2000 08:44:38 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA125782 for ; Sat, 16 Sep 2000 08:44:38 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695C.0025090C ; Sat, 16 Sep 2000 08:44:31 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 16 Sep 2000 09:07:07 +0300 Subject: RE: iSCSI: Creation of new LUNs Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk OK- we could make AE mandatory. I thought that every transport supports it as it is arround for more than 30 years (predates SCSI). I am however not sure that this is the only way (or the best way) to announce a new LU, nor that it will play nicely with discovery. Julo Black_David@emc.com on 15/09/2000 05:04:46 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Creation of new LUNs This sort of proposal should go to T10. A potential issue is that not all SCSI transports support Asynchronous Event Notification (AEN). iSCSI could make AEN mandatory to implement as part of addressing this situation, but standardizing the use of AEN to report LUN addition needs to be addressed in T10. An I-D summarizing the problem and proposed solution without reference to iSCSI or any other transport might be a good start on this. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Thursday, September 14, 2000 10:13 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Creation of new LUNs > > > > Is somebody else doing it differently? > > Julo > > Raghavendra Rao on 14/09/2000 21:19:58 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: Creation of new LUNs > > > > > > What is the recommended/approved way of announcing the availability > of new LUNs in the target ? Currently I can do the following, but if > it isn't explicitly documented, implementors tend to do their own > things (causing interoperability problems) > > Upon creation/deletion of a LUN in the target, a target shall generate > an asynchronous notification with the SCSI event indicator set to > either 2 or 4, followed by sense data with appropriate ASC and ASCQ > indicating that LUNs have changed. The initiator, on receipt of this > notification will issue a REPORT_LUN command to get a list of LUNs. > > -JP > From owner-ips@ECE.cmu.edu Sat Sep 16 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29290; 16 Sep 2000 4:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA18246 for ips-outgoing; Sat, 16 Sep 2000 02:45:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA18237 for ; Sat, 16 Sep 2000 02:44:59 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA206018 for ; Sat, 16 Sep 2000 08:44:28 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA32120 for ; Sat, 16 Sep 2000 08:44:27 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695C.0025062F ; Sat, 16 Sep 2000 08:44:24 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 16 Sep 2000 08:32:51 +0300 Subject: RE: A Simple Question Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear Mr. Cheng, The whole discussion thread is based on the assumption that there will be cases in which you will need several adapters (more than one) while you keep assuming only one. Julo "Y P Cheng" on 14/09/2000 22:55:38 Please respond to "Y P Cheng" To: John Hufferd/San Jose/IBM@IBMUS cc: Julian Satran/Haifa/IBM@IBMIL, black_david@emc.com Subject: RE: A Simple Question > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Wednesday, September 13, 2000 7:04 PM > > I was not sure what your HBA does, and what your Drivers do. It sounded > like you were combining the functions of SCSI, Wedge, iSCSI and TCP/IP > driver, into a single Device Driver. If so, that sounds like it > might be a bit of a problem for various vendors storage controllers > (they need their own Wedge Drivers). And since most platforms > has their own SCSI drivers, I hope you are not building another one. > Therefore, the thing which I hope you are doing is just a good iSCSI > Device Driver that works with one or more of your own NICs. > If this is so, and as long as you handle the interoperability > issues (or 1 above) correctly, the things you said sound fine. Thank you for taking time to write. Our adapters follow the industry standards and interoperate with the SCSI, 1394, and Fibre Channel devices provided by the industries. It is our intent to add iSCSI function to our adapter. As long we send out the correct 48-byte iSCSI header and also interpret received iSCSI header correctly, we will interoperate with everyone. Our adapter operates on all OS platforms. They are programmed to support different protocols like FCP, IP, and VI, with iSCSI to be added. Alacritec has an adapter that supports TCP in its microcode. The main difference between my view and that of ips discussions is the semantics on creating and interpreting the iSCSI headers. The working group assumes the iSCSI in using TCP/IP will need several system calls: write command, read/write data, and read status. Hence, iSCSI will have the problem of interlocking. The working group tries to solve the problem with asymmetric and symmetric models. However, if the group assumes a single system call to send an iSCSI request, we can let the transport layer breaks up iSCSI requests into PDUs of command, data, and status. The asymmetric model is no longer needed. In our adapters, the transport layer builds a table for all data/status transactions. There is no queuing for them. Like an asymmetric model, the data/status transactions are serviced on demand. The fibre channel adapter uses class 3 delivery for FCP, which does not have an ACK. The adapter takes full responsibility of detecting and correcting bad transmission and dropped frames efficiently. Therefore, the real question is why solve the problems of some TCP/IP implementation in iSCSI? There are other transport mechanism in IP that can make reliable delivery. The iSCSI task ID, RTT, status PDU, sense data, and SCSI tag-queue-depth, etc. are available to us to achieve a reliable delivery. Please don't mistake me. I believe the working group is doing a great job. All I am doing is to provide some additional information to the group. From owner-ips@ECE.cmu.edu Sat Sep 16 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29295; 16 Sep 2000 4:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA18245 for ips-outgoing; Sat, 16 Sep 2000 02:45:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA18238 for ; Sat, 16 Sep 2000 02:44:59 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id IAA206016 for ; Sat, 16 Sep 2000 08:44:27 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA32118 for ; Sat, 16 Sep 2000 08:44:27 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695C.00250672 ; Sat, 16 Sep 2000 08:44:24 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 16 Sep 2000 08:46:37 +0300 Subject: RE: Avoiding deadlock in iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, I am on the road - and had barely time for a short answer. I made the same assumption like you that this is a Check Condition. Issuing command one at a time is not a decent option. One-at-a-time is already provided by chaining and ordered set where created to enable you, unlike chaining, to hide the latency of command transport and setup. Otherwise who needs them? Why go at such length to deliver things in order. I see it really as a T10 issue. Regards, Julo Peter Johansson on 14/09/2000 16:52:01 Please respond to Peter Johansson To: IP Storage cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Avoiding deadlock in iSCSI At 08:42 AM 9/13/00, julian_satran@il.ibm.com wrote: >Where would you suggest it should be done without violating layering >constraints? Your query is a bit laconic, Julo, but to put it in the context of how to preserve ordering in the event of a queue full condition, I misspoke earlier. I was under the mistaken impression that a QUEUE FULL condition resulted in a CHECK CONDITION and the concomitant creation of ACA. Jim McGrath pointed out that this is not so. Of the remedies he outlined, of course it's possible for the initiator to refrain from issuing more than one ordered task at a time. This is reasonable for many environments, for example those in which the preponderance of tasks are unordered with the occasional need of an ordered task to fence some synchronization point. In other environments, either purely ordered or one in which a high number of tasks are ordered, this could be inefficient. I like Jim's suggestion of a QACA bit because it does not lump the QUEUE FULL condition together with other ACA events. It wasn't clear whether Jim was referring to a QACA bit in a mode page or in a CDB or both. Although this is, strictly speaking, a T10 matter, it involves interactions with the iSCSI protocol's capabilities and limitations and therefore would benefit from participation by this working group. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Sat Sep 16 16:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31686; 16 Sep 2000 12:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA26363 for ips-outgoing; Sat, 16 Sep 2000 11:21:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA26352 for ; Sat, 16 Sep 2000 11:21:44 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 15 Sep 2000 15:19:47 -0700 Message-ID: From: Charles Monia To: Stephen Byan , IPS Reflector Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Date: Fri, 15 Sep 2000 15:19:37 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Wednesday, September 13, 2000 6:22 AM > To: IPS Reflector > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote: > > > I don't think VI/TCP helps at all. The VI is implemented on > > top of a TCP > > stream. If the TCP stream delivers the iSCSI command to VI, > > and VI has no > > place to put it, what is VI going to do? It has to block the > > TCP stream, and > > that in turn will block and "RDMA" from occurring. > > If I understand correctly, VI/TCP has a credit-based flow > control mechanism > on its message queues (which would be used to implement the > iSCSI command > reception queue), and so the initiator would never send an > iSCSI command > without a target buffer in which to store it. So RDMA cannot block on > commands in the TCP stream. > In the context of generic ULP "flow control" issues, the following observations are offered. Assuming one VI/TCP message queue handles all SCSI control traffic for a target, using flow control on messages to prevent the "queue-full" condition has undesirable side effects. Specifically, back pressure due to lack of command context resources within a logical unit would stop the flow of commands and task management requests to all other LUNS (a form of "head of line" blocking). There's another issue that is perhaps less obvious. Even if multiple message queues are used, say one per LUN, stopping the flow of messages due to a queue full condition would also stop the flow of task management messages as well. Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Sat Sep 16 16:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31684; 16 Sep 2000 12:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA26361 for ips-outgoing; Sat, 16 Sep 2000 11:21:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA26354 for ; Sat, 16 Sep 2000 11:21:45 -0400 (EDT) Received: by server1 with Internet Mail Service (5.5.2448.0) id ; Fri, 15 Sep 2000 19:14:32 -0700 Message-ID: From: Charles Monia To: Jim McGrath , "'csapuntz@cisco.com'" Cc: ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI Date: Fri, 15 Sep 2000 19:14:23 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi All: While Jim's note sums up the SCSI-related issues nicely, I'd like to add a minor filip to what he said. As I see it, the problem of ordering goes beyond whether or not a command has the ordered attribute. For certain devices, like tapes, ordered behavior may be implicit in the device semantics. Although this is a fine point, it probably needs to be spelled out more clearly in the appropriate SCSI specs. Charles > -----Original Message----- > From: Jim McGrath [mailto:Jim.McGrath@quantum.com] > Sent: Tuesday, September 12, 2000 1:09 PM > To: 'csapuntz@cisco.com'; Jim McGrath > Cc: ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > > > I agree with Stephen that this is a T10 issue. BTW, it has > been discussed > before in T10. The issue is that for commands ending with a > CHECK CONDITION > (error) status the ACA mechanism provides a very targeted > means of solving > this problem (use ACA for your ORDERED commands and you are > fine). The QErr > bit in the Control Mode page provides a similar global > capability for the > entire LUN. > > The problem is that there are other status conditions that > may want to be > treated in a similar manner, but which (to my knowledge) are > not. The first > one we looked at was RESERVATION CONFLICT. The second is > QUEUE FULL. Both > may create a desire to pause execution from the command queue. > > The current answer is simply not to get into this problem to > begin with. > That is, if you are sending an ORDERED command, make sure you > do not send > out the next command (ORDERED or UNORDERED) until you (the > initiator) know > that the first command has been received by the target. You > clearly know if > a command has been received by an explicit acknowledgement > (as in parallel > SCSI) or by a subsequent action (a data transfer or status > transfer) that > only makes sense if the command was received. > > You can argue that this degrades performance. For most of > the interconnects > considered up to now, with relatively low latency, this has > not bee as much > of a concern. In addition, there has been a feeling that the > cases were > ORDERED commands are used are so few that you either accepted the > degradation (such as it is) when you use them, or you do a > vendor unique > approach around the problem. > > If this is considered to be a big issue, then I'd raise it to T10. I > suggest you propose a solution that mimics the existing ACA and QErr > solutions, since that will be easiest and quickest to adopt. One > possibility is adding the status QUEUE FULL to the things > that generate > ACAs. Another is to propose a new QACA bit that can be used > like ACA but > for Queue Full (so users can continue to use the old ACA bit > for backward > compatibility). Since you either introduce a compatibility > issue (the first > approach) or need a bit (the second approach), the answer is > not cost free, > so T10 will want some justification for creating a > compatibility issue or > (more likely) to spend the bit. > > Jim > > > -----Original Message----- > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > Sent: Monday, September 11, 2000 10:33 PM > To: Jim McGrath > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > Subject: Re: Avoiding deadlock in iSCSI > > > > > Note that SCSI targets, when faced with getting a command > queue full, do > not > > stop reading from the interconnect. If more commands are > received then > they > > respond with a QUEUE FULL status. If data is received then > they receive > > that data without regard for the status of the command > queue (as long as > it > > is data for an already queued command). This eliminates > the potential for > > deadlock between command and data queues. > > Jim, > > I believe there is problem with the current SCSI behavior. > Consider the > following scenario for a host with a pathological queue of 1 > > > 1) Initiator sends command 1 (ORDERED attribute) > 2) Initiator sends command 2 (ORDERED attribute) > 3) Initiator sends command 3 (ORDERED attributge) > 4) Target reads command 1 > 5) Target reads command 2 > 6) Target returns queue full for command 2 > 7) Command 1 completes > 8) Target reads command 3 > 9) Target executes command 3 > > We have just violated the ordering constraints of the application by > doing command 3 before command 2. > > > Potentially you can have a situation where multiple > commands already at > the > > target have only part of their data transmitted, with the > remainder still > at > > the initiator(s), and then run out of buffer space for data. If the > target > > uses a credit model to pace the reception of data, it can > also make sure > > this never happens. Unsolicited data, even for commands > already queued, > can > > end up creating this deadlock - which is why unsolicited > data systems > either > > have to have a tight limit on the resources it can use > (e.g. low login BB > > credit in Fibre Channel terms) or some sort of clean (i.e. not IO > > terminating) rejection mechanism from target to initiator > (like in USB). > > Unsolicited data is NOT a problem with the current iSCSI > spec. We allow > the target to always drop data and request data transfers > with an RTT. > > -Costa > From owner-ips@ECE.cmu.edu Sat Sep 16 19:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32469; 16 Sep 2000 15:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA00431 for ips-outgoing; Sat, 16 Sep 2000 14:50:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA00425 for ; Sat, 16 Sep 2000 14:49:55 -0400 (EDT) Received: from yp_portable (slip-32-102-64-254.ca.us.prserv.net [32.102.64.254]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLY5T; Sat, 16 Sep 2000 11:47:57 -0700 From: "Y P Cheng" To: Subject: RE: A Simple Question Date: Sat, 16 Sep 2000 11:48:24 -0700 Message-ID: <000a01c0200e$b175eb20$fe406620@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > > The whole discussion thread is based on the assumption that there will > be cases in which you will need several adapters (more than one) while you > keep assuming only one. > I guess this highlights all my arguments. :-) My other two postings on "Asymmetric Model" and "Transport Protocol without ACK" should give addition background where I come from. Yes, our adapter is quite capable and will execute hundreds or thousands iSCSI requests concurrently and atomically, i.e. the adapter will break up the iSCSI requests into PDU's, order them, and retransmit them. The adapter deals with traffic jam, not the device driver. There is no deadlock problem in the adapter. BTW, this is how most fibre channel adapter works today to deliver FCP requests. I am not reinventing the wheel. Yes, it is our intent to support multiple adapters in the device driver for load balance and recovery. Yes, it is difficult to synchronizing PDUs of a single iSCSI request in two or more adapters when they don't share the execution state. From owner-ips@ECE.cmu.edu Sat Sep 16 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32482; 16 Sep 2000 15:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA00082 for ips-outgoing; Sat, 16 Sep 2000 14:31:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA00078 for ; Sat, 16 Sep 2000 14:31:52 -0400 (EDT) Received: from yp_portable (slip-32-102-64-254.ca.us.prserv.net [32.102.64.254]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLY5L; Sat, 16 Sep 2000 11:29:59 -0700 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Sat, 16 Sep 2000 11:30:24 -0700 Message-ID: <000701c0200c$2dfe22a0$fe406620@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100F9B@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk David Black wrote: > < From: Y.P. Cheng> > > The main difference between my view and that of ips discussions is the > > semantics on creating and interpreting the iSCSI headers. The working > > group assumes the iSCSI in using TCP/IP will need several system calls: > > write command, read/write data, and read status. Hence, iSCSI will have the > > problem of interlocking. The working group tries to solve the problem > > with asymmetric and symmetric models. However, if the group assumes a > > single system call to send an iSCSI request, we can let the transport layer > > break up iSCSI requests into PDUs of command, data, and status. The > > asymmetricmodel is no longer needed. > I think "system calls" is the wrong word. The whole point of > sessions is to take advantage of network fabric parallelism, > i.e., use two or more NICs on initiator and target. > This requires the transport layer to figure out which PDU > to send on which NIC (i.e., which TCP connection) independent of what's > implemented in hardware or software, and entails logic on the far side to > put things back in order. This logic could be in iSCSI or a > transport like SCTP. > > Because reliable delivery isn't enough. Congestion control is also > required. TCP does congestion control in a fashion that no existing > SCSI transport or mechanism does. It's not the only solution, but it > or an equivalent congestion control solution is REQUIRED. Please read > > http://www.ietf.org/internet-drafts/draft-floyd-cong-04.txt > Thank you for pointing out that in addition to mapping a SCSI request to iSCSI PDU's, the working group is also addressing the delivery of PDU's to prevent deadlock when TCP/IP is used. It is also assumed that TCP/IP provides the congestion and flow control. Whether I like it or not, the reality is that many iSCSI implementation will be TCP/IP. However, I do have concerns if iSCSI requires the ACK to achieve reliable delivery. Let me use the following example to illustrate my concern. Assume we wish to perform backup to a device 3000 miles away using iSCSI protocol. It would take 10 milliseconds for an IP packet to travel from the source to destination. Similarly, it takes another 10 milliseconds for the ACK to come back. Lets also assume that the backbone is capable of 1 gigabit per second throughput. To keep data streaming on this connection, the source needs to send 2 megabytes of data before seeing the first ACK coming back. Similarly, the target must be prepared to buffer 2 megabyte of data. This example becomes much more interesting when we increase the backbone connection speed to that of OC-192 at 10 gigabits per second or if the backup devices are accepting incoming streams from multiple initiators. It needs a lot of memory. We could take a lesson from the fibre channel FCP using class-3 datagram. The SCSI protocol is inherited acknowledged by the returning of status PDU. If an ACK to datagrams is not necessary, then the 2MB buffer requirement in the above example can be reduced to less than 100KB to accommodate the speed variation of the host system bus and the connecting media. By the way, multiple paths does not solve this problem. The problem of lost PDU's due to traffic jam or congested system bus are also still there. However, the congestion and retransmission problem has already been solved in the implementation of a fibre channel adapter. TCP/IP is not the only protocol providing a solution. I guess my point is that the iSCSI should allow a transport protocol that does not require ACK. (My apology for mentioning UDP in my previous postings.) If iSCSI is not limiting itself to TCP/IP, then, the asymmetric model is also a non-issue for people who design a smart NIC adapter. From owner-ips@ECE.cmu.edu Sat Sep 16 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32493; 16 Sep 2000 15:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA00077 for ips-outgoing; Sat, 16 Sep 2000 14:31:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA00072 for ; Sat, 16 Sep 2000 14:31:45 -0400 (EDT) Received: from yp_portable (slip-32-102-64-254.ca.us.prserv.net [32.102.64.254]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLY5J; Sat, 16 Sep 2000 11:29:54 -0700 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: RE: Asymmetric model Date: Sat, 16 Sep 2000 11:30:21 -0700 Message-ID: <000601c0200c$2be506a0$fe406620@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Thursday, September 14, 2000 5:17 PM > > You can do what ever you want in your own adapter, however, if you are > going to talk to a target that does not use your own adapter, then whether > you use Symmetric, or Asymmetric, is of importance since the Target needs > to support what you do. So the Workgroup needs to decide, then you will > need to follow that model. > I do agree with you that if you have the capability to support > "n" distinct instances of your adapter, such that the SCSI > layer sees no different, the manor you use to get the commands and data > onto the approprate adapter instance, is your and your iSCSI > driver's business. But how you interact (Symmetric, Asymmetric, etc.) > determines how well you will work with IBM, EMC, et.al. Storage Controllers. > > But every thing I said here is so fundamental, it makes me believe that > you must have meant something other then what came across to me. > If I have misunderstood what you intended to say, please correct me. Thank you for taking time to write. I begin to realize that we are talking two different layers: the mapping of a SCSI request to iSCSI PDUs and the delivery of PDUs using a transport mechanism. When we choose TCP/IP as a delivery mechanism, the ordering of PDU's creates deadlock problem that must be solved with asymmetric model unless we do just one SCSI request at a time. Of course, the asymmetric model allows parallelism and takes advantage of multiple paths. Instead of using the current TCP/IP implementation which establishes multiple paths with multiple endpoints of (IP-Address, Port-Number), the new INC card supporting fibre channel FCP has a different but very reliable delivery mechanism. It uses class 3 protocol which does not require ACKs for datagrams. This led me to state that we might not need TCP/IP for delivery. However, I do accept that many iSCSI implementations will use TCP/IP as a delivery mechanism and apologize for bringing up an old topic being discussed before. I will repeat my statement that ACKs on a network with milliseconds of latency can be very expensive unless we have a huge number of EE (end-to-end) credits. For example, a millisecond of latency on a gigabit connection accomodates 67 1.5K-datagrams. Taking advantage of the flow and resource control of FCP and extending it to IP with long latency for a reliable iSCSI delivery could be an interesting topic for the working group. From owner-ips@ECE.cmu.edu Sat Sep 16 20:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00073; 16 Sep 2000 16:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA01666 for ips-outgoing; Sat, 16 Sep 2000 15:53:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA01662 for ; Sat, 16 Sep 2000 15:53:50 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Sat, 16 Sep 2000 15:53:06 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704C3568D@corpmx9.isus.emc.com> To: cslee@FalconStor.com.tw, hufferd@us.ibm.com Cc: ips@ece.cmu.edu Subject: TCP vs. UDP on SMP Date: Sat, 16 Sep 2000 15:53:05 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > A multithread program will use different cpu in different thread. For example, we use one > cpu for receving packets to receive queue and the other CPU for sending packets from sending > queue. Hence we can have more CPU power in iSCSI path. An UDP program with its sending > and receiving can archive this situation. I think this will give a significant performance than TCP if > TCP can't use both CPUs proccesing power. > > TCP process its own queue in kernel mode. And will this TCP queue is processed with 2 CPU? The situation is somewhat more complex than this. iSCSI is REQUIRED to implement congestion control (for more information, see draft-floyd-cong-04.txt, which is about to be issued as a Best Current Practice RFC). UDP by itself does not implement congestion control, and adding this to UDP may affect SMP efficiency because congestion control solutions use information from the network (receive queue) to control transmission (send queue); this couples the receive and send processing in a way that tends to reduce SMP effectiveness. My recollection of published results on using multiple processors for TCP in Unix-like operating systems is that the big gains come from using different processors for different connections rather than handling send and receive on a single connection on different processors due in part to the send/receive coupling required by congestion control ... but it's been a while since I've checked/read this literature. So, I think the summary Q&A is approximately: Q1: Can a single TCP connection be processed by multiple CPUs? A1: Yes. Q2: Will this work as well as using separate CPUs to handle UDP send and receive? A2: Not in all cases, because TCP couples send and receive processing in a way that UDP does not. Q3: Is the UDP solution with independent send and receive processing on two CPUs applicable to iSCSI? A3: Probably not. It does not implement congestion control, and implementing congestion control will result in some coupling of send and receive processing. Let me remind everyone that while congestion control and flow control may be implemented in roughly the same area of protocol code, they are rather different functions. Flow control is concerned with the effective (e.g., reliable and high performance) delivery of individual flows of traffic. Congestion control is concerned with the response of the entire network as a system to overload conditions in which the network as a whole cannot deliver all of the offered traffic. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Sat Sep 16 23:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00783; 16 Sep 2000 19:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA04535 for ips-outgoing; Sat, 16 Sep 2000 18:31:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA04530 for ; Sat, 16 Sep 2000 18:31:05 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Sat, 16 Sep 2000 15:31:57 -0700 Message-ID: From: Charles Monia To: Pierre Labat , John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: RE: iSCSI: 2.2.6. Naming & mapping Date: Sat, 16 Sep 2000 15:31:57 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Apologies for the clutter if someone else has already responded to this. > -----Original Message----- > From: Pierre Labat [mailto:pierre_labat@hp.com] > Sent: Thursday, September 14, 2000 5:57 PM > To: John Hufferd/San Jose/IBM; ips@ece.cmu.edu > Subject: Re: iSCSI: 2.2.6. Naming & mapping > > > John Hufferd/San Jose/IBM wrote: > > > Folks, > > We are again mixing issues and layers again. > > > > There is NO SUCH THING as an iSCSI LU. There is an iSCSI > device which is a > > Controller which will have a SCSI layer which in turn > supports an LU. Once > > the iSCSI session is established, the Rules of SCSI define > how the LUs are > > addressed. > > I agree. > > > There maybe additional Database (LDAP) processes and > > information that attempts to relate LU #3 known to Host xyz > to some name > > "abcd...." etc. > > I don't want so much, i would want just to have a unique > identifier per LU. > It doesn't cost a lot and it will have the advantages > described in the first > mail below. > Hi: As I read Pierre's note, he is proposing that support for the heretofore optional Device Identification page be added to the list of features that must be supported by an iSCSI device. I believe that's the issue the ips wg ought to be debating. (See ftp://ftp.t10.org/t10/drafts/spc/spc-r11a.pdf, section 8.4.3, for a description). Charles > > But that is NOT an iSCSI Transport Protocol. We may need > > to work on this at some point but it is not an iSCSI > transport issue. > > > > Yes, it is not a transport issue, but why not request that > now before people > start > building iSCSI controllers if it simplifies the life of everybody? > I don't know how FC managed to have their WWN but why could not we > do the same thing? > > Regards, > > Pierre > > > > > . > > . > > . > > John L. Hufferd > > > > Pierre Labat @ece.cmu.edu on > 09/14/2000 10:06:54 AM > > > > Sent by: owner-ips@ece.cmu.edu > > > > To: ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: 2.2.6. Naming & mapping > > > > julian_satran@il.ibm.com wrote: > > > > > Not again (what is the sign for frustration?)... I mean > not before the > > next > > > version. > > > > > > Julo > > > > > > Raghavendra Rao on > 14/09/2000 21:04:02 > > > > > > Please respond to Raghavendra Rao > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: > > > Subject: iSCSI: 2.2.6. Naming & mapping > > > > > > I feel that a LUN should be very much part of the Naming scheme > > > that has been proposed in the draft, regardless of the level of > > > enforcement of the scheme by implementors. LUNs are very much > > > part of SCSI addressing, and where LUN doesn't exist, zero is > > > assumed. > > > > > > Without a LUN identifier, Naming section doesn't look complete. > > > > > > Do you have insights why it is omitted ? Is this by > design or overlook ? > > > > > > Thanks. > > > -JP > > > > About the topic of naming a LU, what seems to be a benefit > for me, is to > > adopt the same requirement as fibre channel: each LU MUST provide a > > unique identifier (from the Device Identification Page). > > >From what i read it seems that it is not a big deal to add > this page in a > > LU. > > However, this unique identifier is not needed for iSCSI > protocol to work. > > It doesn't interact with the protocol. > > > > But from an administration point of view, to configure a > server using > > storage > > through iSCSI, the existence of this unique identifier helps a lot. > > It allows the configuration product to know/check if various LUNs > > correspond > > to the same LU, it could help to manage the LU migration > (the LUN is > > changed > > > > inside a target for the same LU). > > It simplifies the configuration software by avoiding it to > fake a unique LU > > identifier. > > It will help in having a configuration tool for iSCSI that > can be closer to > > the one > > used with FC. > > > > Is somebody knowing if it is planned to incorporate this Device > > Identification > > Page > > in the iSCSI LU? Will it be mandatory? Which format > (identifier type) will > > be > > used? > > FC uses the type 3 (FC_PH Name_Identifier). > > > > Regards, > > > > Pierre > From owner-ips@ECE.cmu.edu Sun Sep 17 00:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01115; 16 Sep 2000 20:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA05696 for ips-outgoing; Sat, 16 Sep 2000 19:50:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA05692 for ; Sat, 16 Sep 2000 19:50:06 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G100026A7I0NI@mta6.snfc21.pbi.net> for IPS@ECE.CMU.edu; Sat, 16 Sep 2000 16:49:13 -0700 (PDT) Date: Sat, 16 Sep 2000 16:45:23 -0700 From: Peter Johansson Subject: Re: iSCSI: 2.2.6. Naming & mapping In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000916163423.00bb4f00@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed X-Priority: 1 (Highest) Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:31 PM 9/14/00, John Hufferd/San Jose/IBM wrote: >There is NO SUCH THING as an iSCSI LU. I agree, John, that there is no such thing as an iSCSI LU. But that wasn't the issue that was raised. SCSI targets are addressed by an iSCSI method. If there is more than one path by which an iSCSI target is addressable (perhaps a target with more than one IP address or URL?), then you had better require unique IDs of SCSI LUs. Otherwise it is not possible to discover that what appears to be two LUs, reached by different paths, are in fact the same LU. One may associate a unique identifier with an LU by either a) command set methods (SCSI) or b) transport protocol methods (iSCSI in this case). SBP-2 elected to make an LU's unique identifier visible at the transport protocol layer. This was chosen to permit a host to enumerate unique LUs (and identify redundant paths) prior to a login to use the device. I'm not fully conversant with iSCSI device discovery methods, but it might be desirable to make a unique LU identifier visible during this stage of the process, for reasons analogous to those discovered by SBP-2. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Sun Sep 17 01:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01371; 16 Sep 2000 21:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA06324 for ips-outgoing; Sat, 16 Sep 2000 20:33:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA06319 for ; Sat, 16 Sep 2000 20:33:34 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id TAA08655; Sat, 16 Sep 2000 19:33:43 -0500 Message-ID: <39C41167.8078A463@stewart.chicago.il.us> Date: Sat, 16 Sep 2000 19:33:43 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: "'Ips@Ece. Cmu. Edu'" Subject: Re: A Transport Protocol Without ACK References: <000701c0200c$2dfe22a0$fe406620@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P Cheng wrote: > > David Black wrote: > > < From: Y.P. Cheng> > > > The main difference between my view and that of ips discussions is the > > > semantics on creating and interpreting the iSCSI headers. The working > > > group assumes the iSCSI in using TCP/IP will need several system calls: > > > write command, read/write data, and read status. Hence, iSCSI will have > the > > > problem of interlocking. The working group tries to solve the problem > > > with asymmetric and symmetric models. However, if the group assumes a > > > single system call to send an iSCSI request, we can let the transport > layer > > > break up iSCSI requests into PDUs of command, data, and status. The > > > asymmetricmodel is no longer needed. > > I think "system calls" is the wrong word. The whole point of > > sessions is to take advantage of network fabric parallelism, > > i.e., use two or more NICs on initiator and target. > > This requires the transport layer to figure out which PDU > > to send on which NIC (i.e., which TCP connection) independent of what's > > implemented in hardware or software, and entails logic on the far side to > > put things back in order. This logic could be in iSCSI or a > > transport like SCTP. > > David: .. I missed this earlier... has mentioned by me somewhat a while ago.. SCTP can help you only in that it will retransmit on a alternate network. There is a notion of a "primary" transport address. This is where all sends, by default, go. Now the sender (i.e. iSCSI) can override this, by specifing either an override or changing the primary... but I don't think this accomplishes what you are after. I think that multiple streams and the automatic message bounding are the big wins iSCSI can gain from SCTP. The multi-homing will help only in the sense that you gains some fault tolerance... > > Because reliable delivery isn't enough. Congestion control is also > > required. TCP does congestion control in a fashion that no existing > > SCSI transport or mechanism does. It's not the only solution, but it > > or an equivalent congestion control solution is REQUIRED. Please read > > > > http://www.ietf.org/internet-drafts/draft-floyd-cong-04.txt > > > > Thank you for pointing out that in addition to mapping a SCSI request to > iSCSI PDU's, the working group is also addressing the delivery of PDU's to > prevent deadlock when TCP/IP is used. It is also assumed that TCP/IP > provides the congestion and flow control. Whether I like it or not, the > reality is that many iSCSI implementation will be TCP/IP. However, I do > have concerns if iSCSI requires the ACK to achieve reliable delivery. Let > me use the following example to illustrate my concern. > > Assume we wish to perform backup to a device 3000 miles away using iSCSI > protocol. It would take 10 milliseconds for an IP packet to travel from the > source to destination. Similarly, it takes another 10 milliseconds for the > ACK to come back. Lets also assume that the backbone is capable of 1 > gigabit per second throughput. To keep data streaming on this connection, > the source needs to send 2 megabytes of data before seeing the first ACK > coming back. Similarly, the target must be prepared to buffer 2 megabyte of > data. This example becomes much more interesting when we increase the > backbone connection speed to that of OC-192 at 10 gigabits per second or if > the backup devices are accepting incoming streams from multiple initiators. > It needs a lot of memory. > > We could take a lesson from the fibre channel FCP using class-3 datagram. > The SCSI protocol is inherited acknowledged by the returning of status PDU. > If an ACK to datagrams is not necessary, then the 2MB buffer requirement in > the above example can be reduced to less than 100KB to accommodate the speed > variation of the host system bus and the connecting media. By the way, > multiple paths does not solve this problem. The problem of lost PDU's due > to traffic jam or congested system bus are also still there. However, the > congestion and retransmission problem has already been solved in the > implementation of a fibre channel adapter. TCP/IP is not the only protocol > providing a solution. I am a bit confused by the above Y.P. you state " by the returning of status PDU."... Both SCTP and TCP will carry a piggyback ACK with that PDU, so you end up accomplishing the same thing. What are you trying to say that I am missing??? > I guess my point is that the iSCSI should allow a transport protocol that > does not require ACK. (My apology for mentioning UDP in my previous > postings.) If iSCSI is not limiting itself to TCP/IP, then, the asymmetric > model is also a non-issue for people who design a smart NIC adapter. Y.P. please enumerate the protocols that have this property that also provide TCP friendly congestion control. If you could enumerate the exact protocols and pointers to the specifications I would be more than glad to have a look at these and see if I can support them. Making vague references to "not limiting itself to TCP/IP" does not do anything for me and I think nothing for the WG. We need specific transport protocols listed that are capable of transporting iSCSI AND have TCP friendly congestion control principles built into them... Right now I assess that list to be: 1) TCP 2) SCTP Please feel free to add to this list.. Thanks R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Sep 17 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01752; 16 Sep 2000 23:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA07898 for ips-outgoing; Sat, 16 Sep 2000 22:12:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA07894 for ; Sat, 16 Sep 2000 22:12:00 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Sat, 16 Sep 2000 19:12:47 -0700 Message-ID: From: Charles Monia To: John Hufferd/San Jose/IBM , Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Sat, 16 Sep 2000 19:12:46 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Saturday, September 09, 2000 12:05 AM > To: Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: a vote for asymmetric connections in a session > > > Joshua, > I do not understand whether this is a real issue or not. I > know of a lot > of Client Server applications that ship a lot of data on > TCP/IP, and TCP/IP > seems to be adequate. Now I understand there are some differences in > Direct Storage Access, but it is more like the other > applications then it > is different. If we can address Costa's Issue or have at least two > Connection per Asymmetric Session, I am not sure anything > else is a real > life problem. But I can be convinced, however, I would like > to understand > for each problem we come up with, why it is only a problem > for iSCSI and > not for the other real world applications, and why the SCSI > layer can not > handle the problem. > > Hi John: Here's my stab at responding to your concerns (hopefully, this isn't "swinging after the bell"): In my opinion, iSCSI, as a mapping of the SCSI device model, is fundamentally different than other client-server protocols. The difference is that it allows a client (the initiator) to have many transactions concurrently pending against a single object (the LUN). Other protocols, on the other hand, typically allow only one. As I see it, the issues we've talked about in this thread stem from this basic property. In the case of SCSI, some of these transactions may be in flight while others may be pending in the LUN waiting to be performed. When the lun receives new commands that can't be serviced for some reason, the SCSI layer handles the problem by discarding them. If the condition is corrected spontaneously, as might happen if the problem was due to a temporary resource shortage, the lun simply resumes command processing. In my view, the question we should be addressing, then, is whether or not that policy is valid in the global internet environment for which iSCSI is targeted. Over to you.... Charles From owner-ips@ECE.cmu.edu Sun Sep 17 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02001; 16 Sep 2000 23:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA08591 for ips-outgoing; Sat, 16 Sep 2000 22:51:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA08585 for ; Sat, 16 Sep 2000 22:50:54 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id VAA72214; Sat, 16 Sep 2000 21:44:39 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8H2ort87506; Sat, 16 Sep 2000 20:50:53 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: 2.2.6. Naming & mapping To: Peter Johansson Cc: IP Storage X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 16 Sep 2000 19:45:14 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/16/2000 08:50:53 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, As with Fibre Channel (and even multiple SCSI buss connections), once you have acuired a connection to the Storage Controller, you may address LU 0, to get the LU numbers that are authorize to that connection (with some Storage Controllers these are virtual Devices authorized by connection "port" and/or WWN of the initiator). The values returned by the LU0 query are the unique LUNs related to that LU 0 (real or virtual). The Host SCSI layer can then obtain a unique ID from the EVPD page 83h. When the same unique ID is found on different paths then they have discovered an "Alternate Path". All of this is normal SCSI, not transport. Since many Storage Controller create virtual volumes, and these are only visable to authorized Initiators, and also unknown to the transport. It seems that we should keep to the same approach used by Fibre Channel et. al. and keep it away from the (iSCSI) transport. Here is what Jim Hafner said to the ips working group WAY BACK on 6/19/00: {snip} SCSI HAS defined a "WWN associated with a volume" with EVPD page 83h. This is independent of LUN value and also independent of the transport layer. You (should) get the same answer if your logical unit is connected (simultaneously) to FC port and to a parallel bus. And use of this WWN is already supported as a naming specification in other places in SCSI ..... {snip} . . . John L. Hufferd Peter Johansson @ece.cmu.edu on 09/16/2000 04:45:23 PM Sent by: owner-ips@ece.cmu.edu To: IP Storage cc: Subject: Re: iSCSI: 2.2.6. Naming & mapping At 04:31 PM 9/14/00, John Hufferd/San Jose/IBM wrote: >There is NO SUCH THING as an iSCSI LU. I agree, John, that there is no such thing as an iSCSI LU. But that wasn't the issue that was raised. SCSI targets are addressed by an iSCSI method. If there is more than one path by which an iSCSI target is addressable (perhaps a target with more than one IP address or URL?), then you had better require unique IDs of SCSI LUs. Otherwise it is not possible to discover that what appears to be two LUs, reached by different paths, are in fact the same LU. One may associate a unique identifier with an LU by either a) command set methods (SCSI) or b) transport protocol methods (iSCSI in this case). SBP-2 elected to make an LU's unique identifier visible at the transport protocol layer. This was chosen to permit a host to enumerate unique LUs (and identify redundant paths) prior to a login to use the device. I'm not fully conversant with iSCSI device discovery methods, but it might be desirable to make a unique LU identifier visible during this stage of the process, for reasons analogous to those discovered by SBP-2. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Sun Sep 17 04:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02282; 17 Sep 2000 0:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA09519 for ips-outgoing; Sat, 16 Sep 2000 23:52:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA09513 for ; Sat, 16 Sep 2000 23:52:27 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id WAA52560; Sat, 16 Sep 2000 22:46:11 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8H3qQt131722; Sat, 16 Sep 2000 21:52:26 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: a vote for asymmetric connections in a session To: Charles Monia Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 16 Sep 2000 20:49:54 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/16/2000 09:52:25 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, I have had a number of Network Client/Server apps that used TCP/IP and had multiple conversations/connections from the same Client Host to the same Data Base target (plus other Client Host that also had multiple conversations/connections). This is perhaps even normal. We also had to make sure that there was adequate memory space in the application (Database or Client) to support the thruput (both requests and responses) needed. They of course needed a method to coordinate its memory needs with the various client threads. None of this had anything to do with what was done by TCP/IP. We expected TCP/IP to work out its own problems, we did not expect the application to do anything special to help TCP/IP -- and guess what -- TCP/IP worked just fine. There is no important difference here between normal Client Server applications and SCSI. . . . John L. Hufferd Charles Monia on 09/16/2000 07:12:46 PM To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Saturday, September 09, 2000 12:05 AM > To: Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: a vote for asymmetric connections in a session > > > Joshua, > I do not understand whether this is a real issue or not. I > know of a lot > of Client Server applications that ship a lot of data on > TCP/IP, and TCP/IP > seems to be adequate. Now I understand there are some differences in > Direct Storage Access, but it is more like the other > applications then it > is different. If we can address Costa's Issue or have at least two > Connection per Asymmetric Session, I am not sure anything > else is a real > life problem. But I can be convinced, however, I would like > to understand > for each problem we come up with, why it is only a problem > for iSCSI and > not for the other real world applications, and why the SCSI > layer can not > handle the problem. > > Hi John: Here's my stab at responding to your concerns (hopefully, this isn't "swinging after the bell"): In my opinion, iSCSI, as a mapping of the SCSI device model, is fundamentally different than other client-server protocols. The difference is that it allows a client (the initiator) to have many transactions concurrently pending against a single object (the LUN). Other protocols, on the other hand, typically allow only one. As I see it, the issues we've talked about in this thread stem from this basic property. In the case of SCSI, some of these transactions may be in flight while others may be pending in the LUN waiting to be performed. When the lun receives new commands that can't be serviced for some reason, the SCSI layer handles the problem by discarding them. If the condition is corrected spontaneously, as might happen if the problem was due to a temporary resource shortage, the lun simply resumes command processing. In my view, the question we should be addressing, then, is whether or not that policy is valid in the global internet environment for which iSCSI is targeted. Over to you.... Charles From owner-ips@ECE.cmu.edu Sun Sep 17 05:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02295; 17 Sep 2000 0:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA10043 for ips-outgoing; Sun, 17 Sep 2000 00:23:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA10038 for ; Sun, 17 Sep 2000 00:23:12 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA56620; Sun, 17 Sep 2000 00:21:28 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8H4NAt82862; Sat, 16 Sep 2000 22:23:10 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: 2.2.6. Naming & mapping To: Charles Monia Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 16 Sep 2000 21:20:02 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/16/2000 10:23:09 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, Though I think you are being very kind in your interpretation of Pierre's note, you may have a point. Though I do not know why it is optional when you have multiple connections -- the Spec does, however, say that the VPD is optionally returned by the Inquire Command. I think it is universally applied when a Storage Controller has multiple connections, however, your point may be -- that should we just state it as a non optional feature of storage controllers that support iSCSI. I can not see why this would be a problem. Does anyone else? . . . John L. Hufferd Charles Monia on 09/16/2000 03:31:57 PM To: Pierre Labat , John Hufferd/San Jose/IBM@IBMUS, ips@ece.cmu.edu cc: Subject: RE: iSCSI: 2.2.6. Naming & mapping Apologies for the clutter if someone else has already responded to this. > -----Original Message----- > From: Pierre Labat [mailto:pierre_labat@hp.com] > Sent: Thursday, September 14, 2000 5:57 PM > To: John Hufferd/San Jose/IBM; ips@ece.cmu.edu > Subject: Re: iSCSI: 2.2.6. Naming & mapping > > > John Hufferd/San Jose/IBM wrote: > > > Folks, > > We are again mixing issues and layers again. > > > > There is NO SUCH THING as an iSCSI LU. There is an iSCSI > device which is a > > Controller which will have a SCSI layer which in turn > supports an LU. Once > > the iSCSI session is established, the Rules of SCSI define > how the LUs are > > addressed. > > I agree. > > > There maybe additional Database (LDAP) processes and > > information that attempts to relate LU #3 known to Host xyz > to some name > > "abcd...." etc. > > I don't want so much, i would want just to have a unique > identifier per LU. > It doesn't cost a lot and it will have the advantages > described in the first > mail below. > Hi: As I read Pierre's note, he is proposing that support for the heretofore optional Device Identification page be added to the list of features that must be supported by an iSCSI device. I believe that's the issue the ips wg ought to be debating. (See ftp://ftp.t10.org/t10/drafts/spc/spc-r11a.pdf, section 8.4.3, for a description). Charles > > But that is NOT an iSCSI Transport Protocol. We may need > > to work on this at some point but it is not an iSCSI > transport issue. > > > > Yes, it is not a transport issue, but why not request that > now before people > start > building iSCSI controllers if it simplifies the life of everybody? > I don't know how FC managed to have their WWN but why could not we > do the same thing? > > Regards, > > Pierre > > > > > . > > . > > . > > John L. Hufferd > > > > Pierre Labat @ece.cmu.edu on > 09/14/2000 10:06:54 AM > > > > Sent by: owner-ips@ece.cmu.edu > > > > To: ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: 2.2.6. Naming & mapping > > > > julian_satran@il.ibm.com wrote: > > > > > Not again (what is the sign for frustration?)... I mean > not before the > > next > > > version. > > > > > > Julo > > > > > > Raghavendra Rao on > 14/09/2000 21:04:02 > > > > > > Please respond to Raghavendra Rao > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: > > > Subject: iSCSI: 2.2.6. Naming & mapping > > > > > > I feel that a LUN should be very much part of the Naming scheme > > > that has been proposed in the draft, regardless of the level of > > > enforcement of the scheme by implementors. LUNs are very much > > > part of SCSI addressing, and where LUN doesn't exist, zero is > > > assumed. > > > > > > Without a LUN identifier, Naming section doesn't look complete. > > > > > > Do you have insights why it is omitted ? Is this by > design or overlook ? > > > > > > Thanks. > > > -JP > > > > About the topic of naming a LU, what seems to be a benefit > for me, is to > > adopt the same requirement as fibre channel: each LU MUST provide a > > unique identifier (from the Device Identification Page). > > >From what i read it seems that it is not a big deal to add > this page in a > > LU. > > However, this unique identifier is not needed for iSCSI > protocol to work. > > It doesn't interact with the protocol. > > > > But from an administration point of view, to configure a > server using > > storage > > through iSCSI, the existence of this unique identifier helps a lot. > > It allows the configuration product to know/check if various LUNs > > correspond > > to the same LU, it could help to manage the LU migration > (the LUN is > > changed > > > > inside a target for the same LU). > > It simplifies the configuration software by avoiding it to > fake a unique LU > > identifier. > > It will help in having a configuration tool for iSCSI that > can be closer to > > the one > > used with FC. > > > > Is somebody knowing if it is planned to incorporate this Device > > Identification > > Page > > in the iSCSI LU? Will it be mandatory? Which format > (identifier type) will > > be > > used? > > FC uses the type 3 (FC_PH Name_Identifier). > > > > Regards, > > > > Pierre > From owner-ips@ECE.cmu.edu Sun Sep 17 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02544; 17 Sep 2000 1:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA10263 for ips-outgoing; Sun, 17 Sep 2000 00:39:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA10258 for ; Sun, 17 Sep 2000 00:39:07 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA64898 for ; Sun, 17 Sep 2000 00:33:05 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8H4d6t112970 for ; Sat, 16 Sep 2000 22:39:06 -0600 Importance: Normal Subject: RE: iSCSI: 2.2.6. Naming & mapping To: "John Hufferd/San Jose/IBM" Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sat, 16 Sep 2000 21:36:53 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/16/2000 10:39:06 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry folks, I read Jim Hafner's note, after I sent the note. Jim states: {Snip} The EVPD page 83h LU WWIdentifier (or whatever it's called) spec'd in SPC-2 has just be made mandatory for SPC-2 (at yesterday's T10 meeting). So iSCSI needn't ask this as a requirement. {Snip} So please do not answer the question I asked in the attached note. Now lets get back to the iSCSI spec. . . . John L. Hufferd John Hufferd/San Jose/IBM@IBMUS@ece.cmu.edu on 09/16/2000 09:20:02 PM Sent by: owner-ips@ece.cmu.edu To: Charles Monia cc: ips@ece.cmu.edu Subject: RE: iSCSI: 2.2.6. Naming & mapping Charles, Though I think you are being very kind in your interpretation of Pierre's note, you may have a point. Though I do not know why it is optional when you have multiple connections -- the Spec does, however, say that the VPD is optionally returned by the Inquire Command. I think it is universally applied when a Storage Controller has multiple connections, however, your point may be -- that should we just state it as a non optional feature of storage controllers that support iSCSI. I can not see why this would be a problem. Does anyone else? . . . John L. Hufferd Charles Monia on 09/16/2000 03:31:57 PM To: Pierre Labat , John Hufferd/San Jose/IBM@IBMUS, ips@ece.cmu.edu cc: Subject: RE: iSCSI: 2.2.6. Naming & mapping Apologies for the clutter if someone else has already responded to this. > -----Original Message----- > From: Pierre Labat [mailto:pierre_labat@hp.com] > Sent: Thursday, September 14, 2000 5:57 PM > To: John Hufferd/San Jose/IBM; ips@ece.cmu.edu > Subject: Re: iSCSI: 2.2.6. Naming & mapping > > > John Hufferd/San Jose/IBM wrote: > > > Folks, > > We are again mixing issues and layers again. > > > > There is NO SUCH THING as an iSCSI LU. There is an iSCSI > device which is a > > Controller which will have a SCSI layer which in turn > supports an LU. Once > > the iSCSI session is established, the Rules of SCSI define > how the LUs are > > addressed. > > I agree. > > > There maybe additional Database (LDAP) processes and > > information that attempts to relate LU #3 known to Host xyz > to some name > > "abcd...." etc. > > I don't want so much, i would want just to have a unique > identifier per LU. > It doesn't cost a lot and it will have the advantages > described in the first > mail below. > Hi: As I read Pierre's note, he is proposing that support for the heretofore optional Device Identification page be added to the list of features that must be supported by an iSCSI device. I believe that's the issue the ips wg ought to be debating. (See ftp://ftp.t10.org/t10/drafts/spc/spc-r11a.pdf, section 8.4.3, for a description). Charles > > But that is NOT an iSCSI Transport Protocol. We may need > > to work on this at some point but it is not an iSCSI > transport issue. > > > > Yes, it is not a transport issue, but why not request that > now before people > start > building iSCSI controllers if it simplifies the life of everybody? > I don't know how FC managed to have their WWN but why could not we > do the same thing? > > Regards, > > Pierre > > > > > . > > . > > . > > John L. Hufferd > > > > Pierre Labat @ece.cmu.edu on > 09/14/2000 10:06:54 AM > > > > Sent by: owner-ips@ece.cmu.edu > > > > To: ips@ece.cmu.edu > > cc: > > Subject: Re: iSCSI: 2.2.6. Naming & mapping > > > > julian_satran@il.ibm.com wrote: > > > > > Not again (what is the sign for frustration?)... I mean > not before the > > next > > > version. > > > > > > Julo > > > > > > Raghavendra Rao on > 14/09/2000 21:04:02 > > > > > > Please respond to Raghavendra Rao > > > > > > To: Julian Satran/Haifa/IBM@IBMIL > > > cc: > > > Subject: iSCSI: 2.2.6. Naming & mapping > > > > > > I feel that a LUN should be very much part of the Naming scheme > > > that has been proposed in the draft, regardless of the level of > > > enforcement of the scheme by implementors. LUNs are very much > > > part of SCSI addressing, and where LUN doesn't exist, zero is > > > assumed. > > > > > > Without a LUN identifier, Naming section doesn't look complete. > > > > > > Do you have insights why it is omitted ? Is this by > design or overlook ? > > > > > > Thanks. > > > -JP > > > > About the topic of naming a LU, what seems to be a benefit > for me, is to > > adopt the same requirement as fibre channel: each LU MUST provide a > > unique identifier (from the Device Identification Page). > > >From what i read it seems that it is not a big deal to add > this page in a > > LU. > > However, this unique identifier is not needed for iSCSI > protocol to work. > > It doesn't interact with the protocol. > > > > But from an administration point of view, to configure a > server using > > storage > > through iSCSI, the existence of this unique identifier helps a lot. > > It allows the configuration product to know/check if various LUNs > > correspond > > to the same LU, it could help to manage the LU migration > (the LUN is > > changed > > > > inside a target for the same LU). > > It simplifies the configuration software by avoiding it to > fake a unique LU > > identifier. > > It will help in having a configuration tool for iSCSI that > can be closer to > > the one > > used with FC. > > > > Is somebody knowing if it is planned to incorporate this Device > > Identification > > Page > > in the iSCSI LU? Will it be mandatory? Which format > (identifier type) will > > be > > used? > > FC uses the type 3 (FC_PH Name_Identifier). > > > > Regards, > > > > Pierre > From owner-ips@ECE.cmu.edu Sun Sep 17 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02539; 17 Sep 2000 1:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA10338 for ips-outgoing; Sun, 17 Sep 2000 00:44:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA10333 for ; Sun, 17 Sep 2000 00:44:42 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id AAA93156; Sun, 17 Sep 2000 00:42:58 -0400 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8H4iVt57562; Sat, 16 Sep 2000 22:44:31 -0600 Importance: Normal Subject: RE: Avoiding deadlock in iSCSI To: Robert Snively Cc: "Kalman Meth/Haifa/IBM" , Pierre Labat , ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.2a (Intl) 23 November 1999 Message-ID: From: "Prasenjit Sarkar/Almaden/IBM" Date: Sat, 16 Sep 2000 21:44:28 -0700 X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Release 5.0.4a |July 24, 2000) at 09/16/2000 09:44:39 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Actually, because of interrupt coalescing in Gigabit Ethernet adapters, you can have potentially close many SCSI transactions with one interrupt. However, you have hit the nail on the head, parallelism tends to (I'm not saying will always) increase the average number of interrupts per transaction if the parallelism decreases the possibility of interrupt coalescing. And since interrupt coalescing is a statically determined parameter (in current implementations), getting speedups out of parallelism is harder than it appears. Anyways, I dont want to distract the discussion on symmetric and assymetric connections.. Robert Snively @ece.cmu.edu on 09/15/2000 12:34:01 PM Sent by: owner-ips@ece.cmu.edu To: Kalman Meth/Haifa/IBM@IBMIL, Pierre Labat , ips@ece.cmu.edu cc: Subject: RE: Avoiding deadlock in iSCSI This concerns me a little bit. In all other versions of SCSI (FCP, Parallel, etc.) there is a maximum of one completion interrupt per complete SCSI task, regardless of the number of data information units or linked commands executed. That is provided at the time the status (and autosense data if any) for the last command in the I/O operation is presented. The use of the word maximum allows for the possibility of processing multiple SCSI completions if there happens to be more than one completed during the time the interrupt context is active. I hope we can expect the same behavior of iSCSI. Bob > The current iSCSI draft allows for (successful) status to be > sent with the > last data PDU. This should also be done in the asymmetric > case, precisely > for the reason articulated by Pierre. The NIC receiving the > data will know > when to perform the interrupt when the data transfer has > completed and > return a good status, and only one interrupt will be > required to complete > the data transfer. (The data buffers are registered to the > NIC together > with the Initiator Task Tag [or Transfer Tag], so successful status > received on the data NIC can be easily associated with the original > command, and there is no need for an interrupt on the command NIC.) From owner-ips@ECE.cmu.edu Sun Sep 17 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05623; 17 Sep 2000 11:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA18744 for ips-outgoing; Sun, 17 Sep 2000 10:55:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA18733 for ; Sun, 17 Sep 2000 10:55:34 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 5CAF61291; Sun, 17 Sep 2000 07:55:33 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA22365; Sun, 17 Sep 2000 07:55:30 -0700 (PDT) Message-Id: <4.2.2.20000917073512.00b32410@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Sun, 17 Sep 2000 07:43:16 -0700 To: "Y P Cheng" , "'Ips@Ece. Cmu. Edu'" From: Michael Krause Subject: RE: A Transport Protocol Without ACK In-Reply-To: <000701c0200c$2dfe22a0$fe406620@yp_portable.advansys.com> References: <0F31E5C394DAD311B60C00E029101A0704100F9B@corpmx9.isus.emc.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk > >Thank you for pointing out that in addition to mapping a SCSI request to >iSCSI PDU's, the working group is also addressing the delivery of PDU's to >prevent deadlock when TCP/IP is used. It is also assumed that TCP/IP >provides the congestion and flow control. Whether I like it or not, the >reality is that many iSCSI implementation will be TCP/IP. It is a fact of business - there are many applications for hardware based TCP/IP implementations beyond iSCSI. >Assume we wish to perform backup to a device 3000 miles away using iSCSI >protocol. It would take 10 milliseconds for an IP packet to travel from the >source to destination. Similarly, it takes another 10 milliseconds for the >ACK to come back. Lets also assume that the backbone is capable of 1 >gigabit per second throughput. To keep data streaming on this connection, >the source needs to send 2 megabytes of data before seeing the first ACK >coming back. Similarly, the target must be prepared to buffer 2 megabyte of >data. This example becomes much more interesting when we increase the >backbone connection speed to that of OC-192 at 10 gigabits per second or if >the backup devices are accepting incoming streams from multiple initiators. >It needs a lot of memory. Any type of reliable service requires the data to be persistent until acknowledged. It would be a poor implementation that chose to use adapter resources to perform this retransmission buffering. Similarly, the initiator does not send data in TCP unless one has the window space which indicates the target can receive the data. The reception of this data does not require excessive buffering either since one can use flow through DMA techniques to maintain balance. Many of the problems in this area are really implementation specific and with some thought, one can implement a thin device and high performance. The issue is how to insure the windows / buffer resources are kept in balance at any distance without creating jitter. Mike From owner-ips@ECE.cmu.edu Sun Sep 17 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05628; 17 Sep 2000 11:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA18737 for ips-outgoing; Sun, 17 Sep 2000 10:55:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA18727 for ; Sun, 17 Sep 2000 10:55:31 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 0C4D683A; Sun, 17 Sep 2000 07:55:30 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA22357; Sun, 17 Sep 2000 07:55:26 -0700 (PDT) Message-Id: <4.2.2.20000917072436.00ca0ba0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Sun, 17 Sep 2000 07:33:03 -0700 To: Black_David@emc.com, cslee@FalconStor.com.tw, hufferd@us.ibm.com From: Michael Krause Subject: Re: TCP vs. UDP on SMP Cc: ips@ece.cmu.edu In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704C3568D@corpmx9.isus.emc. com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 03:53 PM 9/16/00 -0400, Black_David@emc.com wrote: >effectiveness. My recollection of published results on using multiple >processors for TCP in Unix-like operating systems is that the big gains >come from using different processors for different connections rather than >handling send and receive on a single connection on different processors >due in part to the send/receive coupling required by congestion control >... but it's been a while since I've checked/read this literature. > >So, I think the summary Q&A is approximately: >Q1: Can a single TCP connection be processed by multiple CPUs? >A1: Yes. This can create performance issues. Scalability is achieved by distributing operations across a set of resources. The key is to determine when to distribute the operation - it is a function of how much subsequent processing and system impact (bus utilization, cache pollution, etc.) one can handle. In general, the lower within the TCP/IP stack the better. >Q2: Will this work as well as using separate CPUs to handle UDP send and >receive? >A2: Not in all cases, because TCP couples send and receive processing in a >way that UDP does not. This is an implementation problem which can (and has been to some extent) be solved with some thought. >Q3: Is the UDP solution with independent send and receive processing on two >CPUs applicable to iSCSI? >A3: Probably not. It does not implement congestion control, and implementing >congestion control will result in some coupling of send and receive >processing. The issue is how fast can the system generate workload relative to the ability of the attached fabric and target to consume. If there is no feedback process anywhere along the path, then the source executes in an unbounded fashion which is not acceptable. >Let me remind everyone that while congestion control and flow control may >be implemented in roughly the same area of protocol code, they are rather >different functions. Flow control is concerned with the effective (e.g., >reliable and high performance) delivery of individual flows of >traffic. Congestion control is concerned with the response of the entire >network as a system to overload conditions in which the network as a whole >cannot deliver all of the offered traffic. Thought this was well understood but good to reiterate every once in awhile. Mike From owner-ips@ECE.cmu.edu Sun Sep 17 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05618; 17 Sep 2000 11:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA18725 for ips-outgoing; Sun, 17 Sep 2000 10:55:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA18719 for ; Sun, 17 Sep 2000 10:55:24 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 8DD2FEFF; Sun, 17 Sep 2000 07:55:23 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA22347; Sun, 17 Sep 2000 07:55:19 -0700 (PDT) Message-Id: <4.2.2.20000917070614.00b33630@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Sun, 17 Sep 2000 07:09:51 -0700 To: "Prasenjit Sarkar/Almaden/IBM" , Robert Snively From: Michael Krause Subject: RE: Avoiding deadlock in iSCSI Cc: "Kalman Meth/Haifa/IBM" , Pierre Labat , ips@ece.cmu.edu In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 09:44 PM 9/16/00 -0700, Prasenjit Sarkar/Almaden/IBM wrote: >Actually, because of interrupt coalescing in Gigabit Ethernet adapters, >you can have potentially close many SCSI transactions with one interrupt. > >However, you have hit the nail on the head, parallelism tends to (I'm not >saying will always) increase the average number of interrupts per >transaction if the parallelism decreases the possibility of interrupt >coalescing. > >And since interrupt coalescing is a statically determined parameter (in >current implementations), getting speedups out of parallelism is harder >than it appears. Interrupt management and parallelism are critical to actual application throughput. There are existing solutions that allow throughput to be achieved with intelligent interrupt management. I do not believe iSCSI changes anything along these lines and consider this issue to be implementation specific. Mike From owner-ips@ECE.cmu.edu Sun Sep 17 19:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06516; 17 Sep 2000 14:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA21676 for ips-outgoing; Sun, 17 Sep 2000 13:47:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA21670 for ; Sun, 17 Sep 2000 13:46:57 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA41580; Sun, 17 Sep 2000 13:40:55 -0400 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8HHkuC71754; Sun, 17 Sep 2000 11:46:56 -0600 Importance: Normal Subject: RE: Avoiding deadlock in iSCSI To: Michael Krause Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.2a (Intl) 23 November 1999 Message-ID: From: "Prasenjit Sarkar/Almaden/IBM" Date: Sun, 17 Sep 2000 10:46:42 -0700 X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Release 5.0.4a |July 24, 2000) at 09/17/2000 10:46:55 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Unfortunately, "implementation issues" determine the complexity of a protocol and consequently, whether the benefits of the protocol are worth the cost. BTW, zero-copy TCP/IP stacks have a lot of caveats (e.g. memory alignment etc) which is why they have never made it to any operating system. There are rough implementations in Solaris, BSD and Linux but none of them are particularly close to being robust (I have tried them all). What we need is a rational cost-benfit analysis, not simply whether its just another implementation issue. Prasenjit Prasenjit Sarkar Research Staff Member IBM Almaden Research San Jose Michael Krause on 09/17/2000 07:09:51 AM To: Prasenjit Sarkar/Almaden/IBM@IBMUS, Robert Snively cc: Kalman Meth/Haifa/IBM@IBMIL, Pierre Labat , ips@ece.cmu.edu Subject: RE: Avoiding deadlock in iSCSI At 09:44 PM 9/16/00 -0700, Prasenjit Sarkar/Almaden/IBM wrote: >Actually, because of interrupt coalescing in Gigabit Ethernet adapters, >you can have potentially close many SCSI transactions with one interrupt. > >However, you have hit the nail on the head, parallelism tends to (I'm not >saying will always) increase the average number of interrupts per >transaction if the parallelism decreases the possibility of interrupt >coalescing. > >And since interrupt coalescing is a statically determined parameter (in >current implementations), getting speedups out of parallelism is harder >than it appears. Interrupt management and parallelism are critical to actual application throughput. There are existing solutions that allow throughput to be achieved with intelligent interrupt management. I do not believe iSCSI changes anything along these lines and consider this issue to be implementation specific. Mike From owner-ips@ECE.cmu.edu Sun Sep 17 19:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06586; 17 Sep 2000 15:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA22175 for ips-outgoing; Sun, 17 Sep 2000 14:17:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA22171 for ; Sun, 17 Sep 2000 14:17:30 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G11004YLMOPEF@mta6.snfc21.pbi.net> for IPS@ECE.CMU.edu; Sun, 17 Sep 2000 11:14:49 -0700 (PDT) Date: Sun, 17 Sep 2000 11:14:16 -0700 From: Peter Johansson Subject: Re: iSCSI: 2.2.6. Naming & mapping In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000917110648.00c20670@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed X-Priority: 1 (Highest) Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:45 PM 9/16/00, John Hufferd/San Jose/IBM wrote: >As with Fibre Channel (and even multiple SCSI buss connections), once you >have acuired a connection to the Storage Controller, you may address LU >0, to get the LU numbers that are authorize to that connection (with some >Storage Controllers these are virtual Devices authorized by connection >"port" and/or WWN of the initiator). The values returned by the LU0 query >are the unique LUNs related to that LU 0 (real or virtual). > >The Host SCSI layer can then obtain a unique ID from the EVPD page 83h. >When the same unique ID is found on different paths then they have >discovered an "Alternate Path". All of this is normal SCSI, not transport. This is one of two methods I mentioned, an approach rooted in the command set. I have no disagreement with the usefulness of vital product data for this purpose. >Since many Storage Controller create virtual volumes, and these are only >visable to authorized Initiators, and also unknown to the transport. It >seems that we should keep to the same approach used by Fibre Channel et. >al. and keep it away from the (iSCSI) transport. I was unaware that iSCSI was targeted (no pun intended) solely at Storage Controllers. If there are iSCSI methods to describe the existence of a particular target's LUs PRIOR to session establishment, then it may also be desirable to be able to discover unique IDs at a target or LU level at the same time. Some clients may find it useful to make a decision at this earlier point rather than after issuing a SCSI command. In any case, this feature would be OPTIONAL and complementary to EVPD. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Sun Sep 17 19:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06732; 17 Sep 2000 15:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA22577 for ips-outgoing; Sun, 17 Sep 2000 14:40:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA22573 for ; Sun, 17 Sep 2000 14:40:07 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA36554; Sun, 17 Sep 2000 14:34:05 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8HIe6C98856; Sun, 17 Sep 2000 12:40:06 -0600 Importance: Normal Subject: Re: iSCSI: 2.2.6. Naming & mapping To: Peter Johansson Cc: IP Storage X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sun, 17 Sep 2000 11:37:22 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/17/2000 12:40:06 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, As specified by Jim Hafner, there is no need and we should not do what you suggest. We have talked this through before, many, many times, and the answer has always come out the same. PLEASE, lets not go there again. . . . John L. Hufferd Peter Johansson @ece.cmu.edu on 09/17/2000 11:14:16 AM Sent by: owner-ips@ece.cmu.edu To: IP Storage cc: Subject: Re: iSCSI: 2.2.6. Naming & mapping At 07:45 PM 9/16/00, John Hufferd/San Jose/IBM wrote: >As with Fibre Channel (and even multiple SCSI buss connections), once you >have acuired a connection to the Storage Controller, you may address LU >0, to get the LU numbers that are authorize to that connection (with some >Storage Controllers these are virtual Devices authorized by connection >"port" and/or WWN of the initiator). The values returned by the LU0 query >are the unique LUNs related to that LU 0 (real or virtual). > >The Host SCSI layer can then obtain a unique ID from the EVPD page 83h. >When the same unique ID is found on different paths then they have >discovered an "Alternate Path". All of this is normal SCSI, not transport. This is one of two methods I mentioned, an approach rooted in the command set. I have no disagreement with the usefulness of vital product data for this purpose. >Since many Storage Controller create virtual volumes, and these are only >visable to authorized Initiators, and also unknown to the transport. It >seems that we should keep to the same approach used by Fibre Channel et. >al. and keep it away from the (iSCSI) transport. I was unaware that iSCSI was targeted (no pun intended) solely at Storage Controllers. If there are iSCSI methods to describe the existence of a particular target's LUs PRIOR to session establishment, then it may also be desirable to be able to discover unique IDs at a target or LU level at the same time. Some clients may find it useful to make a decision at this earlier point rather than after issuing a SCSI command. In any case, this feature would be OPTIONAL and complementary to EVPD. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Sun Sep 17 23:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07733; 17 Sep 2000 19:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA26799 for ips-outgoing; Sun, 17 Sep 2000 18:09:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta6.snfc21.pbi.net (mta6.snfc21.pbi.net [206.13.28.240]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA26795 for ; Sun, 17 Sep 2000 18:09:05 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta6.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G1100FLNXDX4C@mta6.snfc21.pbi.net> for IPS@ECE.CMU.edu; Sun, 17 Sep 2000 15:05:58 -0700 (PDT) Date: Sun, 17 Sep 2000 15:03:10 -0700 From: Peter Johansson Subject: Re: iSCSI: 2.2.6. Naming & mapping In-reply-to: X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000917145527.00bfe370@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:37 AM 9/17/00, John Hufferd/San Jose/IBM wrote: >As specified by Jim Hafner, there is no need and we should not do what >you suggest. We have talked this through before, many, many times, and >the answer has always come out the same. PLEASE, lets not go there again. Jim Hafner's observation (that mandatory presence of a unique ID, on a per LU basis, in EVPD appears to be destined for adoption within SPC-2) only reinforces that part of the discussion with which I already agree. It is desirable to have command set-dependent methods to determine the unique ID for a LU. Matter closed? It might also be desirable to have transport-dependent mechanisms to specify unique IDs for those LUs whose existence is discoverable by transport-dependent methods. I think this is still open to discussion. There's not much I can respond to in your comment above, John, because "... we have talked this through before ..." doesn't constitute much of a technical argument. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Sun Sep 17 23:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07738; 17 Sep 2000 19:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA27629 for ips-outgoing; Sun, 17 Sep 2000 18:47:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA27623 for ; Sun, 17 Sep 2000 18:47:26 -0400 (EDT) Received: from yp_portable (slip-32-101-254-46.ny.us.prserv.net [32.101.254.46]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLZCG; Sun, 17 Sep 2000 15:44:42 -0700 From: "Y P Cheng" To: Cc: "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Sun, 17 Sep 2000 15:45:50 -0700 Message-ID: <000001c020f9$096c1e40$2efe6520@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal In-Reply-To: <39C41167.8078A463@stewart.chicago.il.us> X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk (My apology for this long reply. I hope it worthies your reading.) > From: randall@stewart.chicago.il.us > [mailto:randall@stewart.chicago.il.us] > I am a bit confused by the above Y.P. you state " by the returning of > status PDU."... Both SCTP and TCP will carry a piggyback > ACK with that PDU, so you end up accomplishing the same thing. What are > you trying to say that I am missing??? The piggybacked ACK saves extra PDUs but does not solve the buffer requirement for long latency. In my example, if we have 20-milliseconds of round-trip time on a IP network with gigabit backbone, in order to keep the data streaming on the net we must have 2MB of buffer just in case we need to retransmit the data. By the way, in SCSI read/write data transfer, the receiver sends nothing back until status phase. Therefore, there is nothing to piggyback on. The SCSI protocol for data transfer is basically half-duplex, not full-duplex. Sending ACKs requires extra PUDs. > Y.P. please enumerate the protocols that have this property that also > provide TCP friendly congestion control. If you could enumerate the > exact > protocols and pointers to the specifications I would be more than > glad to have a look at these and see if I can support them. Making vague > references to "not limiting itself to TCP/IP" does not do anything for > me and I think nothing for the WG. We need specific transport protocols > listed that are capable of transporting iSCSI AND have TCP friendly > congestion control principles built into them... I am not an expert in making a transport protocol proposal. However, let me use a bottom-up approach by saying how an iSCSI transport layer should work. Other people can help in making it an IETF proposal. I participated in this discussion with an intention to provide the working group information on the latest NIC adapter technology so the iSCSI proposal can better serve the NIC adapter industry as well as the community who uses TCP/IP. In this response, I will address two topics: 1) The inefficiency of using TCP/IP to implement iSCSI 2) What we can do in the transport layer to overcome the inefficiency of iSCSI on TCP/IP (In here, I am stealing ideas from VI, TCP/RDMA, and FCP.) (Disclaimer: my apology in advance if my view on TCP/IP is incorrect herein. After all, I am a career adapter designer.) For iSCSI to use TCP/IP, it uses SOCKET, CONNECT or BIND to first make a connection point which is a (IP address, TCP port) pair. The asymmetric model provides a second TCP port; a multi-path to another node has a second IP address. After connecting -- with one or more connection points and paths -- the iSCSI creates multiple PDUs: command, data, and status. A SCSI initiator uses a WRITE call to tell an IP NIC to send the PDUs. The iSCSI driver is aware that there could be multiple NIC cards. A SCSI target LISTEN to the incoming PDUs. It may listen to multiple NIC cards. I will not repeat the queuing and blocking problems of the iSCSI driver in dealing with multiple application software with many TCP/IP ports, and the issues of connecting to multiple targets or initiators. We will address only the performance issue of the stream- and connection-oriented delivery of TCP/IP. As in the example of my previous posting, to keep write data streaming on a 1 gigabit connection with 20 milliseconds round-trip latency time, the initiator must have 2000 1K buffers hanging around for retransmitting lost data packets. If it has 200 1K buffers allocated for a target, the initiator can only send 200K of data in 2 milliseconds and wait for 18 milliseconds for the first ACK to come back. Therefore, it runs at 10% of the possible maximum throughput. A target uses RTT to control how much resources each initiator can consume. However, it has no choice but to provide 2000 1K buffers to receive the incoming data for maximum possible performance. To get TCP/IP data, the target uses READs to get data from the IP NIC cards. The memory-to-memory copy to process the TCP stack looking for a TCP port number in the IP packet is the greatest culprit of all of the TCP/IP performance problem. Companies like Alacritec builds special TCP adapter to solve the performance problem to doing the port look up in the adapter. The good news is the above performance problem has already been addressed by VI and FCP implementation in the latest NIC adapters. Here is my proposed iSCSI transport layer protocol: A TRANSACTION ORIENTED WITH BULK ACKNOWLEDGMENT protocol. Instead of using READs and WRITEs for data streams for TCP/IP, A iSCSI driver should send a SCSI request or response to a transport layer using SEND-REQUEST and RECEIVE-RESPONSE MESSAGEs. These message contains the IP end-point connection, SCSI command bytes, and data buffer descriptors that supplied by the application software. Each message describes a transaction EXCHANGE, which can have an exchange-ID. (iSCSI calls the Initiator Task Tag, although a task can have multiple SCSI commands.) The iSCSI driver still use SOCKET, CONNECT, and BIND to create connections. It is true using a total connectionless protocol like UDP to transmit 10 megabytes of data on a busy Internet, we will be forever trying to retransmit due to the lost-frame error. However, instead of sending an ACK for every data frame, we can steal the ideal from fibre channel by breaking down a transaction exchange into data sequences each with a collection of data frames. The receiver needs only to acknowledge a sequence which has a unique sequence ID. A sequence with lost data frame will be retransmitted. Using sliding window, multiple sequences can be transmitted. This is how we keep data frames streaming on a network with long latency time. The size of data sequence is of course network dependent. Having the data descriptors provided by application software for a transaction layer is the greatest benefit of this proposal. There is no data buffering like TCP/IP. The transport layer does not have to allocate a huge buffer to keep data frames streaming on a network with long latency delay. It uses the buffers provided the application software. It can always retransmit a data frame because the application software must stay around until the transaction exchange is complete. In VI, the application software allocates a memory segment, gives it a handle, and passes to a remote node to allow remote DMA. Therefore, the data descriptors of this transport protocol can be simply a memory-handle for a memory segment previously created. TCP/RDMA is copying this idea. Each transaction exchange is executed by a NIC driver atomically. Hundred or even thousands of SEND-REQUEST and RECEIVE-RESPONSE can be outstanding in the driver. After sending the SCSI command PDUs, the data and status PDUs are handled on demand by the NIC driver. There is no queue and deadlock problem. The detection of lost data frame is a function of the transport layer which specifies the QoS (Quality of Service). Flow control is done by EE-credit granting by a receiver so no one can overflow its resources. This is the same as the Max---RN discussed in iSCSI. Congestion control is managed by alternative NIC or IP endpoints. Both should be a part of the transport protocol. I don't claim any credit about this transport layer protocol. Every fibre channel and Infiniband adapter designer knows about this protocol -- although there is no standard. I am sure the TCP accelerator card is doing the same. This protocol is a great alternative to the use of TCP/IP and should be incorporated into iSCSI. From owner-ips@ECE.cmu.edu Mon Sep 18 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08731; 17 Sep 2000 23:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA01446 for ips-outgoing; Sun, 17 Sep 2000 21:45:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA01402 for ; Sun, 17 Sep 2000 21:42:17 -0400 (EDT) Received: from yp_portable (slip-32-101-254-66.ny.us.prserv.net [32.101.254.66]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDLZDZ; Sun, 17 Sep 2000 18:39:27 -0700 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Sun, 17 Sep 2000 18:40:38 -0700 Message-ID: <000001c02111$72b50340$42fe6520@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal In-Reply-To: <4.2.2.20000917073512.00b32410@hpindlm.cup.hp.com> X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Sunday, September 17, 2000 7:43 AM > Any type of reliable service requires the data to be persistent until > acknowledged. It would be a poor implementation that chose to > use adapter resources to perform this retransmission buffering. No, the adapter does not buffer the data for recovery. See my detailed descriptions of SEND-REQUEST and RECEIVE-RESPONSE in another posting. The data for transmission and retransmission are directly from the application software. This is what SCSI adapters have been doing for many years. This is what iSCSI should do. No resource is wasted in the adapter. > Similarly, the initiator does not send data in TCP unless one has the > window space which indicates the target can receive the data. The > reception of this data does not require excessive buffering either since > one can use flow through DMA techniques to maintain balance. You are talking about RTT from a target, right? The flow through DMA in the adapter is what I am talking about for not needing buffers. > Many of the problems in this area are really implementation specific and > with some thought, one can implement a thin device and high > performance. The issue is how to insure the windows / buffer > resources are > kept in balance at any distance without creating jitter. I think it is more than implementation. Whenever there are queues, there will be head-of-queue blocking that creates deadlock. Creating asymmetric queue is trying to remove the head-of-queue blocking. When SCSI requests and responses are treated like atomic transactions instead of multiple PDU-reads and -writes in a transport protocol, it is easy to retry on the same connection or retransmit on a different connection. There is no need for additional buffer space. Many have said that after solving all the transmission and congestion problems, eventually one comes back to something similar to TCP/IP. However, the SCSI protocol give us certain attributes that we can have reliable delivery and reception without TCP/IP. The FCP is an example. VI provides different QoS without using stream-oriented READ and WRITE functions. We can certainly have many VI connections in one system. Let me repeat, a fibre channel adapter today is capable of executing hundreds or thousands SCSI-over-FC (FCP) requests without queuing and resource allocation issues. When the FC is replaced by an IP network the probability of retry becomes higher due to dropped packets. We just have to make sure the transport protocol allows retransmission on the same connection or a different connection. From owner-ips@ECE.cmu.edu Mon Sep 18 03:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08885; 17 Sep 2000 23:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA02372 for ips-outgoing; Sun, 17 Sep 2000 22:36:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ertpg14e1.nortelnetworks.com (ertpg14e1.nortelnetworks.com [47.234.0.35]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA02368 for ; Sun, 17 Sep 2000 22:35:58 -0400 (EDT) Received: from zbl6c016.corpeast.baynetworks.com (actually zbl6c016) by ertpg14e1.nortelnetworks.com; Sun, 17 Sep 2000 22:30:54 -0400 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zbl6c016.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id S88WS127; Sun, 17 Sep 2000 22:30:05 -0400 Received: from ftravost.nortelnetworks.com (bne-cnc-166.corpeast.baynetworks.com [132.245.134.166]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id TDJGDJJL; Sun, 17 Sep 2000 22:30:03 -0400 Message-Id: <4.3.1.2.20000917212851.00b80100@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.1 Date: Sun, 17 Sep 2000 22:20:10 -0400 To: Black_David@emc.com, cslee@FalconStor.com.tw, hufferd@us.ibm.com From: "Franco Travostino" Subject: Re: TCP vs. UDP on SMP Cc: ips@ece.cmu.edu In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704C3568D@corpmx9.isus.emc. com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_7226663==_.ALT" Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_7226663==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed > My recollection of published results on using multiple >processors for >TCP in Unix-like operating systems is that the big gains come from using >different >processors for different connections rather than handling send and receive >on a single >connection on different processors due in part to the send/receive coupling >required >by congestion control ... but it's been a while since I've checked/read this >literature. Much agreed. Some quantitative results in: Performance Issues in Parallelized Network Protocols, Erich M. Nahum, David J. Yates, James F. Kurose, and Don Towsley. USENIX Symposium on Operating Systems Design and Implementation (OSDI), Nov., 1994 (Monterey, CA). Now that we're in TCP/IP offload age ... Recent programmable network processors throw more silicon (e.g., hardware threads) and expose lower level of control (e.g., asynchronous memory operations) than general-purpose CPUs in SMP boxes. Kind of an SMP on a single chip, really. While the TCP vs. UDP fundamental parallelization issues are still in the way, I expect that there will be a bunch of parallelization trade-offs more when optimizing TCP on network processors. -franco --=====================_7226663==_.ALT Content-Type: text/html; charset="us-ascii"
  My recollection of published results on using multiple
processors for
TCP in Unix-like operating systems is that the big gains come from using
different
processors for different connections rather than handling send and receive
on a single
connection on different processors due in part to the send/receive coupling
required
by congestion control ... but it's been a while since I've checked/read this
literature.

Much agreed. Some quantitative results in:

Performance Issues in Parallelized Network Protocols, Erich M. Nahum, David J. Yates, James
 F. Kurose, and Don Towsley. USENIX Symposium on Operating Systems Design and
 Implementation (OSDI), Nov., 1994 (Monterey, CA).

Now that we're in TCP/IP offload age ... Recent programmable network processors throw more silicon (e.g., hardware threads) and expose lower level of control (e.g., asynchronous memory operations) than general-purpose CPUs in SMP boxes. Kind of an SMP on a single chip, really. While the TCP vs. UDP  fundamental parallelization issues are still in the way, I expect that there will be a bunch of parallelization trade-offs more when optimizing TCP on network processors.

-franco --=====================_7226663==_.ALT-- From owner-ips@ECE.cmu.edu Mon Sep 18 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10025; 18 Sep 2000 3:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA06666 for ips-outgoing; Mon, 18 Sep 2000 02:14:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (play-doh.Stanford.EDU [128.12.131.42]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA06662 for ; Mon, 18 Sep 2000 02:14:07 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 459738E81; Sun, 17 Sep 2000 23:11:24 -0700 (PDT) To: IP Storage Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] References: <4.3.2.7.2.20000917145527.00bfe370@PacBell.net> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 17 Sep 2000 23:11:24 -0700 In-Reply-To: Peter Johansson's message of "Sun, 17 Sep 2000 15:03:10 -0700" Message-ID: Lines: 29 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Here is a proposal to extend the iSCSI URL scheme to support LUNs and WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This extension is not required for the iSCSI transport protocol, but may be useful for discovery and management. A URL for the target has the following form: scsi://hostname/path/with/ A URL referring to a specific LU has the following form: scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber If no LUN= term appears in the URL, then LUN 0 is assumed. The WWN= term is optional. If present, the party should verify that the WWN in the LU's Device Identification Inquiry Page correspojnds to the WWN. Note, the following URL: scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After connecting, the initiator should verify that the WWN of the LU is 0a050a4bcdefa. -Costa From owner-ips@ECE.cmu.edu Mon Sep 18 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10012; 18 Sep 2000 3:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA06925 for ips-outgoing; Mon, 18 Sep 2000 02:28:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (play-doh.Stanford.EDU [128.12.131.42]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA06920 for ; Mon, 18 Sep 2000 02:28:11 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 41C928E81; Sun, 17 Sep 2000 23:25:28 -0700 (PDT) To: Subject: Re: TCP speed References: <004301c01f07$fd23d530$33011eac@NEWTON> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 17 Sep 2000 23:25:28 -0700 In-Reply-To: "Chee-Siong, Lee"'s message of "Fri, 15 Sep 2000 19:27:53 +0800" Message-ID: Lines: 31 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk "Chee-Siong, Lee" writes: > Sorry for poor description.. > > A multithread program will use different cpu in different thread. For > example, we use one > cpu for receving packets to receive queue and the other CPU for sending > packets from sending > queue. Hence we can have more CPU power in iSCSI path. An UDP program with > its sending > and receiving can archive this situation. I think this will give a > significant performance than TCP if > TCP can't use both CPUs proccesing power. UDP is missing the following featuers: 1) FIFO ordering of SCSI commands across the wire 2) Congestion control 3) Segmentation of payloads (unless you want to set off an IP fragmentation grenade) 4) Discovery when/recovery from dropped segments 5) Protection against lingering packets in the network Unfortunately, by the time you put in the features, you have added dependencies between the transmit and receive paths (mostly related to acknowledgements) which I believe are on the order of TCP in complexity. -Costa From owner-ips@ECE.cmu.edu Mon Sep 18 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10189; 18 Sep 2000 3:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA07390 for ips-outgoing; Mon, 18 Sep 2000 02:55:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA07385 for ; Mon, 18 Sep 2000 02:55:11 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id CAA56740 for ; Mon, 18 Sep 2000 02:53:28 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8I6tAC140588 for ; Mon, 18 Sep 2000 00:55:10 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: IP Storage X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Sun, 17 Sep 2000 23:51:17 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/18/2000 12:55:10 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, I can see how a Host can acquire the WWN information from the Storage Controller. I can even see a management application on that host, building a name similar to what you said, and sending it to a Management Center, (which can report on its status from time to time, etc.). However, I need your help in the "path" part of your name. What do you think this is? Does it have something to do with the initiator Side or the Target Side? Is it a IP address (or a name reducible to an IP address). I am also assuming that your WWN part is the EVPD (page code 83h) that was talked about by Jim Hafner, is that right? . . . John L. Hufferd Sent by: owner-ips@ece.cmu.edu To: IP Storage cc: csapuntz@cisco.com Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Here is a proposal to extend the iSCSI URL scheme to support LUNs and WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This extension is not required for the iSCSI transport protocol, but may be useful for discovery and management. A URL for the target has the following form: scsi://hostname/path/with/ A URL referring to a specific LU has the following form: scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber If no LUN= term appears in the URL, then LUN 0 is assumed. The WWN= term is optional. If present, the party should verify that the WWN in the LU's Device Identification Inquiry Page correspojnds to the WWN. Note, the following URL: scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After connecting, the initiator should verify that the WWN of the LU is 0a050a4bcdefa. -Costa From owner-ips@ECE.cmu.edu Mon Sep 18 12:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11664; 18 Sep 2000 7:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA11318 for ips-outgoing; Mon, 18 Sep 2000 07:07:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA11308 for ; Mon, 18 Sep 2000 07:07:26 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA12606; Mon, 18 Sep 2000 06:07:25 -0500 Message-ID: <39C5F76C.C73019F3@stewart.chicago.il.us> Date: Mon, 18 Sep 2000 06:07:24 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: "'Ips@Ece. Cmu. Edu'" Subject: Re: A Transport Protocol Without ACK References: <000001c020f9$096c1e40$2efe6520@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P Cheng wrote: > > (My apology for this long reply. I hope it worthies your reading.) > > From: randall@stewart.chicago.il.us > > [mailto:randall@stewart.chicago.il.us] > > I am a bit confused by the above Y.P. you state " by the returning of > > status PDU."... Both SCTP and TCP will carry a piggyback > > ACK with that PDU, so you end up accomplishing the same thing. What are > > you trying to say that I am missing??? > > The piggybacked ACK saves extra PDUs but does not solve the buffer > requirement for long latency. In my example, if we have 20-milliseconds of > round-trip time on a IP network with gigabit backbone, in order to keep the > data streaming on the net we must have 2MB of buffer just in case we need to > retransmit the data. By the way, in SCSI read/write data transfer, the > receiver sends nothing back until status phase. Therefore, there is nothing > to piggyback on. The SCSI protocol for data transfer is basically > half-duplex, not full-duplex. Sending ACKs requires extra PUDs. > > > Y.P. please enumerate the protocols that have this property that also > > provide TCP friendly congestion control. If you could enumerate the > > exact > > protocols and pointers to the specifications I would be more than > > glad to have a look at these and see if I can support them. Making vague > > references to "not limiting itself to TCP/IP" does not do anything for > > me and I think nothing for the WG. We need specific transport protocols > > listed that are capable of transporting iSCSI AND have TCP friendly > > congestion control principles built into them... > > I am not an expert in making a transport protocol proposal. However, let me > use a bottom-up approach by saying how an iSCSI transport layer should work. > Other people can help in making it an IETF proposal. I participated in this > discussion with an intention to provide the working group information on the > latest NIC adapter technology so the iSCSI proposal can better serve the NIC > adapter industry as well as the community who uses TCP/IP. In this response, > I will address two topics: > 1) The inefficiency of using TCP/IP to implement iSCSI > 2) What we can do in the transport layer to overcome the inefficiency > of iSCSI on TCP/IP (In here, I am stealing ideas from VI, TCP/RDMA, and > FCP.) > > (Disclaimer: my apology in advance if my view on TCP/IP is incorrect herein. > After all, I am a career adapter designer.) For iSCSI to use TCP/IP, it > uses SOCKET, CONNECT or BIND to first make a connection point which is a (IP > address, TCP port) pair. The asymmetric model provides a second TCP port; a > multi-path to another node has a second IP address. After connecting -- with > one or more connection points and paths -- the iSCSI creates multiple PDUs: > command, data, and status. A SCSI initiator uses a WRITE call to tell an IP > NIC to send the PDUs. The iSCSI driver is aware that there could be > multiple NIC cards. A SCSI target LISTEN to the incoming PDUs. It may > listen to multiple NIC cards. I will not repeat the queuing and blocking > problems of the iSCSI driver in dealing with multiple application software > with many TCP/IP ports, and the issues of connecting to multiple targets or > initiators. We will address only the performance issue of the stream- and > connection-oriented delivery of TCP/IP. As in the example of my previous > posting, to keep write data streaming on a 1 gigabit connection with 20 > milliseconds round-trip latency time, the initiator must have 2000 1K > buffers hanging around for retransmitting lost data packets. If it has 200 > 1K buffers allocated for a target, the initiator can only send 200K of data > in 2 milliseconds and wait for 18 milliseconds for the first ACK to come > back. Therefore, it runs at 10% of the possible maximum throughput. A > target uses RTT to control how much resources each initiator can consume. > However, it has no choice but to provide 2000 1K buffers to receive the > incoming data for maximum possible performance. To get TCP/IP data, the > target uses READs to get data from the IP NIC cards. The memory-to-memory > copy to process the TCP stack looking for a TCP port number in the IP packet > is the greatest culprit of all of the TCP/IP performance problem. Companies > like Alacritec builds special TCP adapter to solve the performance problem > to doing the port look up in the adapter. > > The good news is the above performance problem has already been addressed by > VI and FCP implementation in the latest NIC adapters. Here is my proposed > iSCSI transport layer protocol: A TRANSACTION ORIENTED WITH BULK > ACKNOWLEDGMENT protocol. > > Instead of using READs and WRITEs for data streams for TCP/IP, A iSCSI > driver should send a SCSI request or response to a transport layer using > SEND-REQUEST and RECEIVE-RESPONSE MESSAGEs. These message contains the IP > end-point connection, SCSI command bytes, and data buffer descriptors that > supplied by the application software. Each message describes a transaction > EXCHANGE, which can have an exchange-ID. (iSCSI calls the Initiator Task > Tag, although a task can have multiple SCSI commands.) The iSCSI driver > still use SOCKET, CONNECT, and BIND to create connections. > > It is true using a total connectionless protocol like UDP to transmit 10 > megabytes of data on a busy Internet, we will be forever trying to > retransmit due to the lost-frame error. However, instead of sending an ACK > for every data frame, we can steal the ideal from fibre channel by breaking > down a transaction exchange into data sequences each with a collection of > data frames. The receiver needs only to acknowledge a sequence which has a > unique sequence ID. A sequence with lost data frame will be retransmitted. > Using sliding window, multiple sequences can be transmitted. This is how we > keep data frames streaming on a network with long latency time. The size of > data sequence is of course network dependent. > So unless you have a very very small sequence (network dependant as you say) you end up with one lost frame causes the WHOLE sequence to be retransmitted. Reducing the network goodput to a crawl. If the "sequence" is very very small (say a MTU for the network) then what you have described is a connectionless UDP protocol OR SCTP if it is connection oriented and reliable.... > Having the data descriptors provided by application software for a > transaction layer is the greatest benefit of this proposal. There is no > data buffering like TCP/IP. The transport layer does not have to allocate a > huge buffer to keep data frames streaming on a network with long latency > delay. It uses the buffers provided the application software. It can > always retransmit a data frame because the application software must stay > around until the transaction exchange is complete. In VI, the application > software allocates a memory segment, gives it a handle, and passes to a > remote node to allow remote DMA. Therefore, the data descriptors of this > transport protocol can be simply a memory-handle for a memory segment > previously created. TCP/RDMA is copying this idea. Not having the transport protocol copy data from the user and just passing the buffers is NOT a new idea. It has been around in TCP for quite some time and you have been able to find implemenations that do this for years. Now one question I have for you is that you nicely describe the sender side above.. what about the receiver? It will still need some sort of buffer to read data into.. > > Each transaction exchange is executed by a NIC driver atomically. Hundred > or even thousands of SEND-REQUEST and RECEIVE-RESPONSE can be outstanding in > the driver. After sending the SCSI command PDUs, the data and status PDUs > are handled on demand by the NIC driver. There is no queue and deadlock > problem. The detection of lost data frame is a function of the transport > layer which specifies the QoS (Quality of Service). Your proposal does not describe how you detect a lost frame. The only thing I can see is you must use some sort of ordering of the sequences. Otherwise if you lost a whole sequence you would never be aware that you lost one... > > Flow control is done by EE-credit granting by a receiver so no one can > overflow its resources. This is the same as the Max---RN discussed in > iSCSI. Congestion control is managed by alternative NIC or IP endpoints. > Both should be a part of the transport protocol. How is congestion control managed by the NIC cards or IP endpoint? These are CRITICAL questions that you MUST have answers for... passing this off to an undefined NIC card is UNACCEPTABLE. You MUST be able to show a transport protocol that is responsive to network congestions the way TCP is. 1) What is the indication of network congestion? 2) How does the protocol respond to network congestion when detected by (1)? 3) When do retransmissions happen? Is there a timer? if so does it back off properly? 4) If you have a timer what algorithm are you using to establish it? 5) What limits your data flow? EE-credits granting would need to be shown responsive like TCP cwnd? You so far have describe nothing but "let the iSCSI designer use a unspecifed protocol that send packets" aka UDP. I see no viable transport protocol here and I don't see this conversation of any use unless you get exact details AND point to a internet draft that defines EXACTLY how it works (or possibly some other standards document). > > I don't claim any credit about this transport layer protocol. Every fibre > channel and Infiniband adapter designer knows about this protocol -- > although there is no standard. I am sure the TCP accelerator card is doing > the same. This protocol is a great alternative to the use of TCP/IP and > should be incorporated into iSCSI. No it is not. You are not offering an alternative yet.. R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 18 14:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12200; 18 Sep 2000 9:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA14694 for ips-outgoing; Mon, 18 Sep 2000 09:18:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA14683 for ; Mon, 18 Sep 2000 09:18:33 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id GAA16664 for ; Mon, 18 Sep 2000 06:18:20 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 18 Sep 2000 06:18:18 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1B8@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: IPS Reflector Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Date: Mon, 18 Sep 2000 06:18:19 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles Monia [mailto:cmonia@NishanSystems.com] wrote: > In the context of generic ULP "flow control" issues, the following > observations are offered. > > Assuming one VI/TCP message queue handles all SCSI control > traffic for a > target, using flow control on messages to prevent the > "queue-full" condition > has undesirable side effects. Specifically, back pressure due > to lack of > command context resources within a logical unit would stop the flow of > commands and task management requests to all other LUNS (a > form of "head of > line" blocking). > > There's another issue that is perhaps less obvious. Even if > multiple message > queues are used, say one per LUN, stopping the flow of > messages due to a > queue full condition would also stop the flow of task > management messages as > well. Good points, Charles. So SCSI over VI would imply at least two message queues per LUN, if one of the message queues is to flow control on the SCSI command queue resource. This seems excessive, so VI flow control no longer looks like an attractive way to handle the "ordered command queuing TASK_SET_FULL" problem. Assuming that the message queue does flow control only on the message buffer resource (i.e. the VI flow control substitutes for rejecting a command with BUSY status), then I think it is in principle possible to implement SCSI over VI using only one message queue per device (not that this is necessarily desirable). Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Mon Sep 18 14:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12189; 18 Sep 2000 9:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA13511 for ips-outgoing; Mon, 18 Sep 2000 08:45:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA13506 for ; Mon, 18 Sep 2000 08:45:18 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id HAA12785; Mon, 18 Sep 2000 07:18:46 -0500 Message-ID: <39C60826.9AF587FD@stewart.chicago.il.us> Date: Mon, 18 Sep 2000 07:18:46 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Prasenjit Sarkar/Almaden/IBM CC: Michael Krause , ips@ece.cmu.edu Subject: Re: Avoiding deadlock in iSCSI References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Prasenjit Sarkar/Almaden/IBM wrote: > > Unfortunately, "implementation issues" determine the complexity of a > protocol and > consequently, whether the benefits of the protocol are worth the cost. > > BTW, zero-copy TCP/IP stacks have a lot of caveats (e.g. memory alignment > etc) which > is why they have never made it to any operating system. There are rough > implementations > in Solaris, BSD and Linux but none of them are particularly close to being > robust > (I have tried them all). > > What we need is a rational cost-benfit analysis, not simply whether its > just another > implementation issue. > Prasenjit: I have used a zero-copy TCP/IP stack (if I recall correctly) in VxWorks or was it VRTX... I will grant you these are NOT *NiX systems and are just recently beginning to become PosiX compliant... but I seem to remember the zero-copy symenatics and I don't think there was a memory alignment requriement.. it has been a while so I may be mis-remembering.. :) R > Prasenjit > > Prasenjit Sarkar > Research Staff Member > IBM Almaden Research > San Jose > > Michael Krause on 09/17/2000 07:09:51 AM > > To: Prasenjit Sarkar/Almaden/IBM@IBMUS, Robert Snively > > cc: Kalman Meth/Haifa/IBM@IBMIL, Pierre Labat , > ips@ece.cmu.edu > Subject: RE: Avoiding deadlock in iSCSI > > At 09:44 PM 9/16/00 -0700, Prasenjit Sarkar/Almaden/IBM wrote: > > >Actually, because of interrupt coalescing in Gigabit Ethernet adapters, > >you can have potentially close many SCSI transactions with one interrupt. > > > >However, you have hit the nail on the head, parallelism tends to (I'm not > >saying will always) increase the average number of interrupts per > >transaction if the parallelism decreases the possibility of interrupt > >coalescing. > > > >And since interrupt coalescing is a statically determined parameter (in > >current implementations), getting speedups out of parallelism is harder > >than it appears. > > Interrupt management and parallelism are critical to actual application > throughput. There are existing solutions that allow throughput to be > achieved with intelligent interrupt management. I do not believe iSCSI > changes anything along these lines and consider this issue to be > implementation specific. > > Mike -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 18 15:42:05 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12700; 18 Sep 2000 11:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA16848 for ips-outgoing; Mon, 18 Sep 2000 10:10:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA16839 for ; Mon, 18 Sep 2000 10:10:32 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id HAA23798 for ; Mon, 18 Sep 2000 07:10:30 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 18 Sep 2000 07:10:27 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1B9@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: IPS Reflector Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Date: Mon, 18 Sep 2000 07:07:04 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis [mailto:dotis@sanlight.net] wrote: > Charles, > > Depending on the capabilities of the Net-SCSI adapter, such > direct copies > would be possible without resorting to VI. Not using VI but > allowing direct > placement also provides lower overhead and a safer system as > pointers going > astray due to a muddled target or bit rot is not a concern. > SAM requires > buffers to be in place prior to issuing the request. These > buffers are at > the client which is where VI becomes useful. Unless both ends of this > communication becomes VI, you have not provided a meaningful solution. It may be better/more efficient/easier/whatever to implement iSCSI directly over TCP rather than over VI over TCP. My point was that in return for the pain, we can leverage the utility of the RDMA hardware assists, since the assists would be encapsulated in the VI layer, rather than intimately part of the iSCSI protocol. Consequently the RDMA hardware could be reused to accelerate other new protocols, as well as cluster message passing. I think an enlarged market for the hardware is important in order to ensure the success of iSCSI. iSCSI depends on getting RDMA hardware acceleration NICs into the servers. Consider two product scenarios: 1) a NIC with RDMA hardware acceleration for iSCSI. 2) a NIC with RDMA hardware acceleration for iSCSI, DAFS, MPI, Oracle Parallel Server, DB2 extended edition, and which is open to future applications. Which product do you think would be the easiest to sell to an end-user customer? > Adding VI modifies TCP. I don't believe draft-dicecco-vitcp-00.txt modifies TCP in any way. However, I do think the proposed TCP Message Boundary Option would be useful in order to optimize the implementation of RDMA in either iSCSI or in VI/TCP. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 15:52:05 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 11:42:31 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 16:02:05 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 11:52:31 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 16:12:06 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 12:02:30 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 16:22:05 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 12:12:31 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 16:32:05 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 12:22:30 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 16:42:05 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 12:32:30 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 16:52:05 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 12:42:31 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From owner-ips@ECE.cmu.edu Mon Sep 18 16:52:05 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13118; 18 Sep 2000 12:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA18641 for ips-outgoing; Mon, 18 Sep 2000 11:31:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA18635 for ; Mon, 18 Sep 2000 11:31:11 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA13010; Mon, 18 Sep 2000 11:29:26 -0400 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8IFV6O121860; Mon, 18 Sep 2000 09:31:07 -0600 Importance: Normal Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: csapuntz@cisco.com Cc: IPS@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Mon, 18 Sep 2000 08:31:04 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Release 5.0.4a |July 24, 2000) at 09/18/2000 08:31:06 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, I've seen Daniel's proposal before and I still haven't figured out the context in which such names are to be used. Given that, I'm personally a bit uncomfortable with any naming convention that uses LUN values in a global context. These have no meaning as LUNs are host-specific values. Even worse, LUNs are addresses, not names! I have no problem with WWNs in a global context as that is what they are for! A subtle difficulty, is that a WWN does not give a host an address (either for the target or the LU). Even in SPC-2, EXTENDED COPY's Identification Descriptor Target Descriptor Format (sic?) where WWNs are used, says "instructs the copy manager to locate a target and logical unit that returns a device identification VPD page ..." but gives no hints on how that should be done (e.g., walk the bus and do INQUIRY to everything...?). The "hostname" and DNS provide a canonical method for getting an address for a name, so that part is OK. The only defined way to get an address (LUN) for a LU WWN is by exhaustive INQUIRY to each LU at a given target. On the other hand, I can envision how some of these things might work, in a well-coordinated and well-managed environment: There is a managing application which has the dual responsibility of managing the targets for their host/LUN mapping configurations and also serves as the "walk the bus" discovery resource for hosts. When the manager (human?) determines the distribution of host/target LU resources (after inventorying both hosts and target LUs), the application will send to the targets whatever host/LUN Mapping (access controls?) commands are necessary to configure them. Later, when a host boots it queries the application to get its "walk-the-bus" services. In this, it minimally gets from this application a list of targets that have LUs to which it should have access. There is no REQUIREMENT at this point for the application to give the host anything more as the normal SCSI LU discovery process per target kicks in. On the other hand, I can see some value in the host getting more information about the LUs it will see at each target. In that case, the host could get from the application a list of targets and LU identifiers (these could be LUNs or WWN or both, as you suggested). The important point here, is that these LUNs are valid only in the context of the requesting HOST and are not global! Note that this process requires clear coordination between the "target configurator" and "host discovery of targets". Given an appropriate context for a URL scheme, I'd make two modifications: 1) drop the assumption that no LUN means LUN=0. If no LU qualifier (e.g., no LUN or WWN (or other) is provided), then LUN=0. In all other cases, the alternative names should be sufficient to identify a LU, though, as above, finding a LUN address may require extra work. 2) add ?ProxyToken= as an additional option here (this coordinates well with the changes to EXTENDED COPY approved in the context of the SCSI access controls). Well, that's another two cents! I was hoping to save some of this discussion until the protocol gets worked out, but .... Jim Hafner csapuntz@cisco.com@ece.cmu.edu on 09-17-2000 11:11:24 PM Sent by: owner-ips@ece.cmu.edu To: IP Storage cc: csapuntz@cisco.com Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Here is a proposal to extend the iSCSI URL scheme to support LUNs and WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This extension is not required for the iSCSI transport protocol, but may be useful for discovery and management. A URL for the target has the following form: scsi://hostname/path/with/ A URL referring to a specific LU has the following form: scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber If no LUN= term appears in the URL, then LUN 0 is assumed. The WWN= term is optional. If present, the party should verify that the WWN in the LU's Device Identification Inquiry Page correspojnds to the WWN. Note, the following URL: scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After connecting, the initiator should verify that the WWN of the LU is 0a050a4bcdefa. -Costa From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 17:02:05 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 12:52:31 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 17:12:06 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 13:02:31 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 17:22:05 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 13:12:31 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 17:30:16 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 13:22:30 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out do_ypcall: clnt_call: RPC: Timed out From owner-ips@ECE.cmu.edu Mon Sep 18 17:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13484; 18 Sep 2000 13:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA22676 for ips-outgoing; Mon, 18 Sep 2000 12:53:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA22667 for ; Mon, 18 Sep 2000 12:52:52 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA11488; Mon, 18 Sep 2000 09:41:20 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA01158; Mon, 18 Sep 2000 09:39:02 -0700 From: "Douglas Otis" To: , "IP Storage" Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 18 Sep 2000 09:55:02 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, Why should a transport protocol carry device abstractions? This a function best left to an LDAP server. You could use LDAP as a means to discover devices, for authentication, and any number of binary representations to text abstractions. By keeping this function independent of the transport layer, you allow flexibility in actual management to be either focused at the client or at the provider or perhaps a combination of the two. You should not attempt to solve everything with one specification. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > csapuntz@cisco.com > Sent: Sunday, September 17, 2000 11:11 PM > To: IP Storage > Cc: csapuntz@cisco.com > Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > Here is a proposal to extend the iSCSI URL scheme to support LUNs and > WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This > extension is not required for the iSCSI transport protocol, but may be > useful for discovery and management. > > A URL for the target has the following form: > > scsi://hostname/path/with/ > > A URL referring to a specific LU has the following form: > > scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber > > If no LUN= term appears in the URL, then LUN 0 is assumed. > > The WWN= term is optional. If present, the party should > verify that the WWN in the LU's Device Identification Inquiry > Page correspojnds to the WWN. > > Note, the following URL: > > scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa > > refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After > connecting, the initiator should verify that the WWN > of the LU is 0a050a4bcdefa. > > -Costa > From owner-ips@ECE.cmu.edu Mon Sep 18 17:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13482; 18 Sep 2000 13:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA22269 for ips-outgoing; Mon, 18 Sep 2000 12:44:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA22260 for ; Mon, 18 Sep 2000 12:44:53 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id JAA21354 for ; Mon, 18 Sep 2000 09:44:50 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 18 Sep 2000 09:44:46 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1BB@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: ips@ece.cmu.edu Subject: RE: Zero-copy TCP stacks (Was: Avoiding deadlock in iSCSI) Date: Mon, 18 Sep 2000 09:44:46 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I may be merely increasing the noise floor; if so, I beg forgiveness in advance... Randall R. Stewart [mailto:randall@stewart.chicago.il.us] wrote: > Prasenjit Sarkar/Almaden/IBM wrote: > > BTW, zero-copy TCP/IP stacks have a lot of caveats (e.g. > memory alignment > > etc) which > > is why they have never made it to any operating system. > There are rough > > implementations > > in Solaris, BSD and Linux but none of them are particularly > close to being > > robust > > (I have tried them all). > > Prasenjit: > > I have used a zero-copy TCP/IP stack (if I recall correctly) > in VxWorks > or was it VRTX... I will grant you these are NOT *NiX systems and > are just recently beginning to become PosiX compliant... but I > seem to remember the zero-copy symenatics and I don't think there > was a memory alignment requriement.. it has been a while so I > may be mis-remembering.. :) I'm no TCP expert, but I think the difference between Randall's experience and Prasenjit's experience relates to the extent of the memory protection and security guarantees offered by the operating system. Embedded OS's such as VxWorks and VRTX do not provide separate memory address spaces for each task (aka process). Consequently there's no security problem in handing a pointer to an arbitrary memory buffer to a task - it can already see all of memory anyway. "Real OS's (TM)" do provide separate memory address spaces for each process (aka task). Consequently there is a security problem in handing a pointer to an arbitrary memory buffer to a task - the process could potentially view packets which happen to share the same memory page but which belong to other processes. To solve this problem, the TCP buffers must be aligned to a page boundary, and at most one buffer can be allocated per page. Regards, -Steve P.S. to the TCP experts - did I get this right? Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Mon Sep 18 17:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa13477; 18 Sep 2000 13:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA21473 for ips-outgoing; Mon, 18 Sep 2000 12:26:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA21451 for ; Mon, 18 Sep 2000 12:26:49 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Mon, 18 Sep 2000 09:27:49 -0700 Message-ID: From: Charles Monia To: "Ips (E-mail)" Subject: RE: a vote for asymmetric connections in a session Date: Mon, 18 Sep 2000 09:27:48 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi John: I agree that there is no issue with TCP/IP, per se. What I've been trying to call attention to is the ULP policy specified in SAM for handling the stream of commands and task management functions after they emerge from the TCP/IP pipe. As I've stated below, the issue that ought to be discussed is whether or not SAM's policy is adequate in an environment where the pipeline delays and hence the amount of in-flight traffic may significantly exceed the norms for other SCSI interconnct environments. This, or course, has nothing to do with TCP/IP but is an inherent property of some network environments in which TCP/IP operates. Charles > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Saturday, September 16, 2000 8:14 PM > To: Charles Monia > Subject: RE: a vote for asymmetric connections in a session > > > > Charles, > I have had a number of Network Client/Server apps that used > TCP/IP and had > multiple conversations/connections from the same Client Host > to the same > Data Base target (plus other Client Host that also had multiple > conversations/connections). This is perhaps even normal. We > also had to > make sure that there was adequate memory space in the > application (Database > or Client) to support the thruput (both requests and > responses) needed. > They of course needed a method to coordinate its memory needs with the > various client threads. None of this had anything to do with > what was done > by TCP/IP. We expected TCP/IP to work out its own problems, > we did not > expect the application to do anything special to help TCP/IP > -- and guess > what -- TCP/IP worked just fine. > > There is no important difference here between normal Client Server > applications and SCSI. > > . > . > . > John L. Hufferd > > > Charles Monia on 09/16/2000 07:12:46 PM > > To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng > > cc: ips@ece.cmu.edu > Subject: RE: a vote for asymmetric connections in a session > > > > > > > -----Original Message----- > > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > > Sent: Saturday, September 09, 2000 12:05 AM > > To: Joshua Tseng > > Cc: ips@ece.cmu.edu > > Subject: RE: a vote for asymmetric connections in a session > > > > > > Joshua, > > I do not understand whether this is a real issue or not. I > > know of a lot > > of Client Server applications that ship a lot of data on > > TCP/IP, and TCP/IP > > seems to be adequate. Now I understand there are some > differences in > > Direct Storage Access, but it is more like the other > > applications then it > > is different. If we can address Costa's Issue or have at least two > > Connection per Asymmetric Session, I am not sure anything > > else is a real > > life problem. But I can be convinced, however, I would like > > to understand > > for each problem we come up with, why it is only a problem > > for iSCSI and > > not for the other real world applications, and why the SCSI > > layer can not > > handle the problem. > > > > > > Hi John: > > Here's my stab at responding to your concerns (hopefully, this isn't > "swinging after the bell"): > > In my opinion, iSCSI, as a mapping of the SCSI device model, is > fundamentally different than other client-server protocols. > The difference > is that it allows a client (the initiator) to have many transactions > concurrently pending against a single object (the LUN). Other > protocols, on > the other hand, typically allow only one. As I see it, the > issues we've > talked about in this thread stem from this basic property. > > In the case of SCSI, some of these transactions may be in flight while > others may be pending in the LUN waiting to be performed. > When the lun > receives new commands that can't be serviced for some reason, the SCSI > layer > handles the problem by discarding them. If the condition is corrected > spontaneously, as might happen if the problem was due to a temporary > resource shortage, the lun simply resumes command processing. > > In my view, the question we should be addressing, then, is > whether or not > that policy is valid in the global internet environment for > which iSCSI is > targeted. > > Over to you.... > > Charles > > > > > > > > From ips-ml+@cnoc.pdl.cs.cmu.edu Mon Sep 18 17:40:01 2000 Return-Path: From: root (Cron Daemon) To: ips-ml Subject: Cron /usr/local/bin/inc -silent >&/dev/null X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Mon, 18 Sep 2000 13:30:16 EDT Sender: ips-ml+@cnoc.pdl.cs.cmu.edu do_ypcall: clnt_call: RPC: Unable to receive; errno = Connection refused do_ypcall: clnt_call: RPC: Unable to receive; errno = Connection refused do_ypcall: clnt_call: RPC: Unable to receive; errno = Connection refused do_ypcall: clnt_call: RPC: Unable to receive; errno = Connection refused do_ypcall: clnt_call: RPC: Unable to receive; errno = Connection refused From owner-ips@ECE.cmu.edu Mon Sep 18 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14001; 18 Sep 2000 14:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA25369 for ips-outgoing; Mon, 18 Sep 2000 13:47:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA25358 for ; Mon, 18 Sep 2000 13:47:19 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA11514; Mon, 18 Sep 2000 10:35:52 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA01312; Mon, 18 Sep 2000 10:33:30 -0700 From: "Douglas Otis" To: "Merhar, Milan" , Subject: RE: switch latency (was Command Queue Depth, Asymmetric/Symmetric) Date: Mon, 18 Sep 2000 10:49:29 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <618471D08ABDD31188B6009027E5009352A0ED@apollo.pirus.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Merhar, FIFO depth comes into play as an additional factor. If an average port includes somewhere from 16-64 packet depth for asynchronous traffic, these points of buffering become significant. Expectation of delay should be in milli-seconds as few paths are limited to just a few such points of buffering. 3 micro-seconds per mile in glass provides 150 micro-seconds to travel 50 miles. If the lowest level switch is 1 Gb, then there is about 1.5 micro-seconds per 1.5k packet buffered. Should a buffer depth be 10 at that point in time, then one switch may add an additional 15 micro-seconds. Once data is delivered into the NIC, even more significant buffering takes place. Although it may be possible to consider micro-second delivery within a Metro-Area-Network, drop just one packet and this will more than triple transit time. If you expect a level of performance, you should consider both distance, buffering and head of queue blocking. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Merhar, Milan > Sent: Friday, September 15, 2000 8:35 AM > To: 'ips@ece.cmu.edu' > Subject: RE: switch latency (was Command Queue Depth, > Asymmetric/Symmetric) > > > Michael, > > You're quite right about the actual crossbars introducing > minimal delays - typically, only a few tens of bit times > at 2-3x the port bit rate. > > But, there's often a store-and-forward delay at the input port, > where the first bits received sit until the last bits of the > packet arrive (introducing one packet transmission time's delay, > or about 12.5 usec for a 1500 byte Gigabit Ethernet packet.) > This is also where packet inspection and classification > delays can appear, as you correctly point out. > > There is a similar store-and-forward delay coming out of > the crossbar, where the first bits switched may wait until > the last bits clear the crossbar, before the actual > packet transmission is scheduled to the outgoing port. > As the crossbar may run at 2-3 times the input bit rate, > this delay is less, often on the order of 4-6 usec for > a maximum-size packet. > > Added together, this gives you 16-20 usec "first bit in to first > bit out" delay for a maximum sized Gigabit Ethernet packet, > which would be reported in the usual data-sheet convention > as about 5 usec "last bit in to first bit out." > Some of the big switches introduce a little more store and > forward delay, as they use two levels of internal switching, > one at the module level, and another at the chassis level. > > "Sub 1 usec" numbers may indicate the switch selectively > uses cut-through forwarding rather than store-and-forward > (tricky to get right, but certainly faster,) or may merely be > the result of doing the measurements with a minimum-sized packet. > > As for delay numbers exceeding 1 millisecond, all I can suggest > is that someone was looking at the specs for a 10 Megabit > Ethernet product (where an 1518 byte packet takes 1.2 msec > to transmit,) or for a software-based bridge or router, > which can easily build up 1 msec of queuing time. > > And, of course, when you start considering large WAN networks, > with multiple router hops and substantial speed-of-light delays, > 30-50 msec starts to look mighty fast. Add in congestion queuing, > and you can easily build up to the often quoted 150-300 msec > "Internet latency." > > - milan > > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Tuesday, September 12, 2000 11:21 AM > To: Scott Bradner > Cc: ips@ece.cmu.edu > Subject: Re: Command Queue Depth (was asymmetric/Symmetric) > > > At 07:14 AM 9/8/00 -0400, Scott Bradner wrote: > > > I don't know what the > > > latency through ethernet switches is, but I'd hope it wasn't > in the ms > > range. > > > >processing time is in the 10 to 50 microsecond range > > Don't know whose switches you are evaluating. They range from sub 1-usec > to perhaps 5-10 usec - it is all a matter of how much additional > processing > is going on within the switch. For example, many switches > examine not just > the ethernet headers/tags but also look at the IP headers and provide > additional functionality based on their content. This is where > many switch > vendors try to differentiate and is not a function of whether they can > switch fast or not. The actual switching elements are usually crossbars > that operate in the 100-500ns range - chip process improvements > will lower > that by quite a bit over the next couple of years. > > Mike > From owner-ips@ECE.cmu.edu Mon Sep 18 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14011; 18 Sep 2000 14:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA26066 for ips-outgoing; Mon, 18 Sep 2000 14:00:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA26050 for ; Mon, 18 Sep 2000 14:00:21 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA11518; Mon, 18 Sep 2000 10:49:04 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA01348; Mon, 18 Sep 2000 10:46:46 -0700 From: "Douglas Otis" To: "Douglas Otis" , "Merhar, Milan" , Subject: RE: switch latency (was Command Queue Depth, Asymmetric/Symmetric) Date: Mon, 18 Sep 2000 11:02:46 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Oops, I can not read my own writing. It is 8 micro-seconds per mile. Such that 50 miles would be 402 micro-seconds. Doug > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Monday, September 18, 2000 10:49 AM > To: Merhar, Milan; 'ips@ece.cmu.edu' > Subject: RE: switch latency (was Command Queue Depth, > Asymmetric/Symmetric) > > > Merhar, > > FIFO depth comes into play as an additional factor. If an > average port includes somewhere from 16-64 packet depth for > asynchronous traffic, these points of buffering become > significant. Expectation of delay should be in milli-seconds as > few paths are limited to just a few such points of buffering. 3 > micro-seconds per mile in glass provides 150 micro-seconds to > travel 50 miles. If the lowest level switch is 1 Gb, then there > is about 1.5 micro-seconds per 1.5k packet buffered. Should a > buffer depth be 10 at that point in time, then one switch may add > an additional 15 micro-seconds. Once data is delivered into the > NIC, even more significant buffering takes place. Although it > may be possible to consider micro-second delivery within a > Metro-Area-Network, drop just one packet and this will more than > triple transit time. If you expect a level of performance, you > should consider both distance, buffering and head of queue blocking. > > Doug From owner-ips@ECE.cmu.edu Mon Sep 18 19:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14026; 18 Sep 2000 14:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA26794 for ips-outgoing; Mon, 18 Sep 2000 14:14:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from sphmraaa.compuserve.com (hs-img-rel-1.compuserve.com [149.174.177.156]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA26790 for ; Mon, 18 Sep 2000 14:14:44 -0400 (EDT) Received: (from mailgate@localhost) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id OAA26992 for ips@ece.cmu.edu; Mon, 18 Sep 2000 14:14:13 -0400 (EDT) Received: from compuserve.com (dal-qbu-zol-vty40.as.wcom.net [216.192.244.40]) by sphmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id OAA26940; Mon, 18 Sep 2000 14:14:07 -0400 (EDT) Message-ID: <39C65C3D.A5CAB3E7@compuserve.com> Date: Mon, 18 Sep 2000 13:17:34 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: "T10, Reflector" , IPS Reflector Subject: SAM-2 revision 14 is available Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk SAM-2 (SCSI Architecture Model -2) revision 14 is available at: ftp://ftp.t10.org/t10/drafts/sam2/sam2r14.pdf SAM-2 r14 includes all T10 approved change proposals as of this date and contains the following significant changes: 1) The SMU (SCSI Multi-port Unit) model for devices with multiple ports has been replaced with TBD because T10 is still hashing out the details of how that capability will be defined. 2) The architecture has been modified to allow bi-directional data transfers. 3) The Nexus concept has been reintroduced to the architecture. 4) The TASK ABORTED status has been added as an optional way to notify one initiator when another initiator has aborted its tasks. Enjoy. Ralph... From owner-ips@ECE.cmu.edu Tue Sep 19 01:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15818; 18 Sep 2000 21:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA10935 for ips-outgoing; Mon, 18 Sep 2000 20:07:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA10927 for ; Mon, 18 Sep 2000 20:07:41 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id RAA13057; Mon, 18 Sep 2000 17:07:28 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 18 Sep 2000 17:07:27 -0700 Message-ID: From: Jim McGrath To: "'Randall R. Stewart'" , Y P Cheng Cc: "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Mon, 18 Sep 2000 17:07:25 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Actually, I have not heard any discussion of Netware networking protocols - IPX, SPX, etc... these work fine, are widely supported, and could be using instead of TCP. I am not familiar enough with a comparison to be able and determine the pros and cons, but I would invite someone to express a reasoned opinion. Jim -----Original Message----- From: Randall R. Stewart [mailto:randall@stewart.chicago.il.us] Sent: Saturday, September 16, 2000 5:34 PM To: Y P Cheng Cc: 'Ips@Ece. Cmu. Edu' Subject: Re: A Transport Protocol Without ACK Y P Cheng wrote: > > David Black wrote: > > < From: Y.P. Cheng> > > > The main difference between my view and that of ips discussions is the > > > semantics on creating and interpreting the iSCSI headers. The working > > > group assumes the iSCSI in using TCP/IP will need several system calls: > > > write command, read/write data, and read status. Hence, iSCSI will have > the > > > problem of interlocking. The working group tries to solve the problem > > > with asymmetric and symmetric models. However, if the group assumes a > > > single system call to send an iSCSI request, we can let the transport > layer > > > break up iSCSI requests into PDUs of command, data, and status. The > > > asymmetricmodel is no longer needed. > > I think "system calls" is the wrong word. The whole point of > > sessions is to take advantage of network fabric parallelism, > > i.e., use two or more NICs on initiator and target. > > This requires the transport layer to figure out which PDU > > to send on which NIC (i.e., which TCP connection) independent of what's > > implemented in hardware or software, and entails logic on the far side to > > put things back in order. This logic could be in iSCSI or a > > transport like SCTP. > > David: .. I missed this earlier... has mentioned by me somewhat a while ago.. SCTP can help you only in that it will retransmit on a alternate network. There is a notion of a "primary" transport address. This is where all sends, by default, go. Now the sender (i.e. iSCSI) can override this, by specifing either an override or changing the primary... but I don't think this accomplishes what you are after. I think that multiple streams and the automatic message bounding are the big wins iSCSI can gain from SCTP. The multi-homing will help only in the sense that you gains some fault tolerance... > > Because reliable delivery isn't enough. Congestion control is also > > required. TCP does congestion control in a fashion that no existing > > SCSI transport or mechanism does. It's not the only solution, but it > > or an equivalent congestion control solution is REQUIRED. Please read > > > > http://www.ietf.org/internet-drafts/draft-floyd-cong-04.txt > > > > Thank you for pointing out that in addition to mapping a SCSI request to > iSCSI PDU's, the working group is also addressing the delivery of PDU's to > prevent deadlock when TCP/IP is used. It is also assumed that TCP/IP > provides the congestion and flow control. Whether I like it or not, the > reality is that many iSCSI implementation will be TCP/IP. However, I do > have concerns if iSCSI requires the ACK to achieve reliable delivery. Let > me use the following example to illustrate my concern. > > Assume we wish to perform backup to a device 3000 miles away using iSCSI > protocol. It would take 10 milliseconds for an IP packet to travel from the > source to destination. Similarly, it takes another 10 milliseconds for the > ACK to come back. Lets also assume that the backbone is capable of 1 > gigabit per second throughput. To keep data streaming on this connection, > the source needs to send 2 megabytes of data before seeing the first ACK > coming back. Similarly, the target must be prepared to buffer 2 megabyte of > data. This example becomes much more interesting when we increase the > backbone connection speed to that of OC-192 at 10 gigabits per second or if > the backup devices are accepting incoming streams from multiple initiators. > It needs a lot of memory. > > We could take a lesson from the fibre channel FCP using class-3 datagram. > The SCSI protocol is inherited acknowledged by the returning of status PDU. > If an ACK to datagrams is not necessary, then the 2MB buffer requirement in > the above example can be reduced to less than 100KB to accommodate the speed > variation of the host system bus and the connecting media. By the way, > multiple paths does not solve this problem. The problem of lost PDU's due > to traffic jam or congested system bus are also still there. However, the > congestion and retransmission problem has already been solved in the > implementation of a fibre channel adapter. TCP/IP is not the only protocol > providing a solution. I am a bit confused by the above Y.P. you state " by the returning of status PDU."... Both SCTP and TCP will carry a piggyback ACK with that PDU, so you end up accomplishing the same thing. What are you trying to say that I am missing??? > I guess my point is that the iSCSI should allow a transport protocol that > does not require ACK. (My apology for mentioning UDP in my previous > postings.) If iSCSI is not limiting itself to TCP/IP, then, the asymmetric > model is also a non-issue for people who design a smart NIC adapter. Y.P. please enumerate the protocols that have this property that also provide TCP friendly congestion control. If you could enumerate the exact protocols and pointers to the specifications I would be more than glad to have a look at these and see if I can support them. Making vague references to "not limiting itself to TCP/IP" does not do anything for me and I think nothing for the WG. We need specific transport protocols listed that are capable of transporting iSCSI AND have TCP friendly congestion control principles built into them... Right now I assess that list to be: 1) TCP 2) SCTP Please feel free to add to this list.. Thanks R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 19 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15831; 18 Sep 2000 21:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA10947 for ips-outgoing; Mon, 18 Sep 2000 20:08:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA10943 for ; Mon, 18 Sep 2000 20:08:18 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Mon, 18 Sep 2000 17:09:21 -0700 Message-ID: From: Charles Monia To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: bridging issues -- Converging FCP-2 and iSCSI control struct ures Date: Mon, 18 Sep 2000 17:09:20 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Julo: > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Sunday, September 10, 2000 7:44 AM > To: ips@ece.cmu.edu > Subject: RE: bridging issues -- Converging FCP-2 and iSCSI control > struct ures > > > > > Charles, > > We (the iSCSI design team) had some guidelines that we > followed (to the > best > of our knowledge) when deciding on formats. > > Here are some of those: > > constant header length > no redundant information (as far as possible) to avoid the need for > validity checks > compact coding while keeping implementations efficient > close resemblance to FCP to simplify bridging > > We also considered that requiring a bridge to do stateless > transliteration > is acceptable. > Your proposal has many elements towards which we are neutral. > We might be > wiling to change some coding elements to better fit bridges > as long as they > don't affect iSCSI efficiency but we would like to keep the > header length > constant. > I agree with that goal. However, as you state below, the issue of how best to distribute I/O traffic across multiple TCP/IP connections effects header design and was still in flux. Like you, I felt it was premature to address concerns in this area. > There is some misunderstanding about the CDB length - the > iSCSI is pretty > liberal only that it use a tricky coding of the length field > (not something > we would necessarily want to keep in). > I may well have misunderstood something here. After reading the iSCSI spec, I was not clear as to what the extra data field in the iSCSI command PDU was to be used for. It was my surmise that you may have intended this for CDBs in excess of 16 bytes -- although, I didn't glean that from the wording in the spec. > The FCP task management encoding is strange - if any task > management flag > is set > the CDB is unused; we might go for it if it where not for > that the task > abort command requires us to specify which task we are taking > about and we > don't want to expand every header. The net result is a header > that exceeds > our constant length. > I suspect that the rational for the encapsulation was based on the desire to simplify implementations by having all control blocks be the same size. After all, task mangement requests are very infrequent, so there's not much point in optimizing them. > > > Responses - are simpler. There is an almost 1-to-1 match and the total > length is within bounds. > > For R2T - we can rearange the fields. > > For Data - if we are moving to an asymmetric model with 2 > connections we > may want > to have a streamlined header on the data stream(as in the > 00-draft - no use > for the current clutter). > > THE QUESTION of TAG MAPPING is still open. > > In summary I think that we can converge but we have some more > work to do > to on the commands and I think that before we settle the > asymmetric/symmetric and > related it will be hard to see all the details. > > But let us keep talking and agree to attempt a format > convergence before > the next version of the draft is due (before San Diego). > > Keep in mind that if we go for asymmetric we would like to > get the header > back to the original > 40 bytes (?) if possible. > Regards, Charles From owner-ips@ECE.cmu.edu Tue Sep 19 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15837; 18 Sep 2000 21:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA11434 for ips-outgoing; Mon, 18 Sep 2000 20:25:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub1.almaden.ibm.com (mailhub1.almaden.ibm.com [198.4.83.44]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA11421 for ; Mon, 18 Sep 2000 20:25:20 -0400 (EDT) Received: from catalina.almaden.ibm.com (catalina.almaden.ibm.com [9.1.24.58]) by mailhub1.almaden.ibm.com (8.8.8/8.8.8) with ESMTP id RAA46422 for ; Mon, 18 Sep 2000 17:22:58 -0700 Received: (from dfsmith@localhost) by catalina.almaden.ibm.com (AIX4.3/UCB 8.8.8/8.7) id RAA24462 for ips@ece.cmu.edu; Mon, 18 Sep 2000 17:24:49 -0700 From: Daniel Smith Message-Id: <200009190024.RAA24462@catalina.almaden.ibm.com> Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: ips@ece.cmu.edu Date: Mon, 18 Sep 2000 17:24:49 -0700 (PDT) In-Reply-To: from "Jim Hafner/Almaden/IBM" at Sep 18, 2000 08:31:04 AM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Just in case people are wondering what this proposal is/was (the mailing list seems to have exploded quite a bit since it was released back in May), here it is again, with a handful of [modifications] and [comments]. -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 ---------------------------------------------- Subject: iSCSI naming Date: Mon, 8 May 2000 15:01:17 -0700 (PDT) [mods 18 Sep 2000] iSCSI naming requirements (first [and a half] iteration) ------------------------- It is important to have a consistent naming scheme as the basis of a discovery system for the SCSI over TCP protocol. A basic set of requirements is below, and a method is suggested. Requirements ------------ A name identifies a [SCSI service delivery port], not a LUN. notes: There may be many thousands of LUNs (and virtual LUNs) on a connection. why: The LUNs given to each connection are likely to be dynamic and assigned by the iSCSI manager. The iSCSI names are more static and long-lived, and must be persistent even if the iSCSI manager fails. notes: A separate naming scheme will document the LUN discovery and naming mechanism. [query to LUN 0 to get list of LUNs] The namespace need not be fundamentally [systematically] enumerable. notes: FC can be enumerated, the Internet cannot (practically). why: iSCSI will be deployed on untrusted networks. Having devices that can be enumerated are subject to systematic attack. Also, many devices will be behind firewall machines, and not even contactable until a login procedure has taken place---holes in such an enumerated namespace are not acceptable. notes: Devices on a LAN can be enumerated, and this is acceptable. notes: Enumeration (listing) by a 3rd party is acceptable. The names should be as static as possible. notes: Both in time and location. why: A name given for one iSCSI device really ought to resolve to the same device every time, from everywhere. [device here refers to a SCSI service delivery port, of course the LUs should stay the same too, barring things like security lock-outs] The name should not be tied to the network technology. notes: For example, ethernet MAC address or IPv4/v6 address. why: To cope with future networking systems, the name of the iSCSI device should not rely only on aspects of the transport that are liable to become obsolete, or that are not static enough. notes: IPv4 addresses may be acceptable on a controlled private network, however, these names should not be the canonical names as viewed from the outside. [major problems with NAT if that happens] The naming scheme must be reliable. notes: If parts of the network go down, the names must remain intact. why: iSCSI may form part of the networking infrastructure, meaning that after power loss, for example, the iSCSI devices will be required to bring up the rest of the network. This implies that the devices should be aware of their own name, if necessary for network stability. It also implies that any name service should have redundancy. The naming scheme should be nice to look at. why: Long strings of unformatted numbers are horrible. [and prone to typos] The iSCSI device being addressed should be maskable from observers. why: The device name may contain valuable (secure) information, so it must be possible to hide this information from prying network interfaces. notes: The URL scheme manages this, by separating the hostname from the device name. Proposal -------- We should follow the URL naming scheme for iSCSI devices, thus: scsi:/// Where :=|: is the TCP [destination] identifier... is the DNS or DDNS host name of the iSCSI device and is the TCP port number to connect to. And [ should be ] :=|/|? is the device identifier is a text identifier for the device :=|? is an argument for the device [argument escape character was '&' in the original] The element is reasonably well defined. However, I would prefer to keep the element more free-form. The scsis:/// is also a straightforward extension. [to secure (SSL) iSCSI] The identifier specifies an iSCSI TCP connection identifier only. This may provide access to one or many LUNs that flit in and out of LUN space. One caveat is that some (most?) iSCSI devices will not have access to DNS. Fortunately, devices (targets) do not need to [query the network to find] their names except in the case of trying to resolve another device (target)---i.e., only initiators need DNS resolution. Here are some fun examples of the things I would want to do as a storage manager. The "backup" program does a simple block-to-block copy from one iSCSI device to another... backup scsi://raid.acme.com/lun12 scsi://tape.acme.com/id25 Of course, a more useful operation may be... flashcopy scsi://disk.acme.com/12/2 scsi://disk.acme.com/13/2 where the pathnames are ASCII numbers. (They may even be the WWN or LUN value.) This doesn't address the issue of security. When we have more of those issues resolved, we might want to issue a command like this... kcopy scsis://secure.acme.com/raid12/set2?key=12345?auth=dfsmith ...so that the part, containing the security tokens, is never passed in the clear over the network. It may be possible for a manager to transparently interpret the arguments or pathname into a LUN list in a storage farm. iscsi attach scsi://farm.acme.com/tray17?LUN=0x17?LUN=0x18 /dev/scsi2 iscsi attach scsi://san.acme.com/5A/27/23/99/B7/A2/01/00 /dev/sda The Text/Response commands in the iSCSI Internet-Draft might be accommodated thusly: iscsitext scsi://tape.scsi.acme.com/cart5 "com.ibm.retension:yes" \ "com.ibm.eject:now" > com.ibm.retension:ok retensioning > com.ibm.eject:ok scheduled ETA 2:30 Secure enumeration might be done in the following way, with peculiar examples of possible Un*x integration. iscsi find_domain_manager > iscsi.acme.com httpsget https://iscsi.acme.com:99/root&auth=dfsmith&key=1234 > > > scsi://unit1.raid.acme.com/ > disk1 disk2 disk3 disk4 disk8 disk9 > > > > scsi://bigtape.acme.com/racf4 > cart57 cart58 > > > > scsi://ibmdeskstar75.dfsmith.acme.com/lun1 > > iscsi mount scsi://ibmdeskstar75.dfsmith.acme.com/lun1 /mnt/mydisk > /mnt/mydisk: 73564224 blocks available tar -cf 'scsi://bigtape.acme.com/racf4/cart57&user=dfsmith' ~/* & > [3] 4291 runsocks dd if=/dev/zero of=scsi://dfsmith.blocks4less.com/disk1 & > [4] 4293 And so on. (Note: I am not an XML expert.) [Additional notes: The decision to resolve devices in the host name or in the device name is really up to the implementer. Some companies may already have systems in place for handling the DNS while they have no existing SAN infrastructure. Thus the decision to name a particular service delivery port scsi://disk26.farm-b.acme.com/ or scsi://farm-b.acme.com/disk26 is up to the system designer. The iSCSI URL names a bundle of LUs (one iSCSI "connection"). People love to point out that this is a problem if you have multiple entrance ways into the same SAN, and the same device (say WWN=0x1234) has different LUNs (say LUN=5 on iscsi://scsi.acme.com/bridge1 and LUN=477 on iscsi://scsi.acme.com/bridge2). So don't do that. Or do that and just live with it. Or make sure the different initiator/bridges into the SAN assign the device the same LUN. (There isn't really a problem using a LUN in a global context, as long as it's always attached to its service delivery port.) (Having to query every LUN on a port to get the list of WWNs ought to be addressed by T10, in the same way they covered the "query LUN 0 to get a list of LUNs" problem. IMO!) The security aspects bring up the spectre of LUs being masked out for different iSCSI initiators on different parts of the net. This is fun and nothing to worry about. These URL names are used in several places, most of which are not related to the iSCSI transport itself. (Likewise, html is not particularly related to HTTP, nor SMTP to metamail.) First by users (or users by dint of clobbering an html hyperlink) who want to attach to iSCSI devices. The names are descriptive and are malleable to all sorts of marvelous manipulations by the sysadmins. Second, by directory servers, who can deliver a list of the URLs (and the LUNs that are accessible therein) to the iSCSI clients. Third, by iSCSI itself for the 3rd party copy command. In this case, the URL specifies the destination SAN and the target LUN is carried in the additional data field. ] ---------------------------------------------- Jim Hafner/Almaden/IBM wrote: > > I've seen Daniel's proposal before and I still haven't figured out the > context in which such names are to be used. > > Given that, I'm personally a bit uncomfortable with any naming convention > that uses LUN values in a global context. These have no meaning as LUNs > are host-specific values. Even worse, LUNs are addresses, not names! > > I have no problem with WWNs in a global context as that is what they are > for! > > A subtle difficulty, is that a WWN does not give a host an address (either > for the target or the LU). Even in SPC-2, EXTENDED COPY's Identification > Descriptor Target Descriptor Format (sic?) where WWNs are used, says > "instructs the copy manager to locate a target and logical unit that > returns a device identification VPD page ..." but gives no hints on how > that should be done (e.g., walk the bus and do INQUIRY to everything...?). > > The "hostname" and DNS provide a canonical method for getting an address > for a name, so that part is OK. The only defined way to get an address > (LUN) for a LU WWN is by exhaustive INQUIRY to each LU at a given target. > > On the other hand, I can envision how some of these things might work, in a > well-coordinated and well-managed environment: > > There is a managing application which has the dual responsibility of > managing the targets for their host/LUN mapping configurations and also > serves as the "walk the bus" discovery resource for hosts. > > When the manager (human?) determines the distribution of host/target LU > resources (after inventorying both hosts and target LUs), the > application will send to the targets whatever host/LUN Mapping (access > controls?) commands are necessary to configure them. Later, when a host > boots it queries the application to get its "walk-the-bus" services. In > this, it minimally gets from this application a list of targets that > have LUs to which it should have access. There is no REQUIREMENT at > this point for the application to give the host anything more as the > normal SCSI LU discovery process per target kicks in. On the other > hand, I can see some value in the host getting more information about > the LUs it will see at each target. In that case, the host could get > from the application a list of targets and LU identifiers (these could > be LUNs or WWN or both, as you suggested). The important point here, is > that these LUNs are valid only in the context of the requesting HOST and > are not global! > > Note that this process requires clear coordination between the "target > configurator" and "host discovery of targets". > > > Given an appropriate context for a URL scheme, I'd make two modifications: > 1) drop the assumption that no LUN means LUN=0. If no LU qualifier (e.g., > no LUN or WWN (or other) is provided), then LUN=0. In all other cases, the > alternative names should be sufficient to identify a LU, though, as above, > finding a LUN address may require extra work. > 2) add ?ProxyToken= as an additional option here (this coordinates > well with the changes to EXTENDED COPY approved in the context of the SCSI > access controls). > > Well, that's another two cents! I was hoping to save some of this > discussion until the protocol gets worked out, but .... > > Jim Hafner > > > csapuntz@cisco.com@ece.cmu.edu on 09-17-2000 11:11:24 PM > > Sent by: owner-ips@ece.cmu.edu > > > To: IP Storage > cc: csapuntz@cisco.com > Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > Here is a proposal to extend the iSCSI URL scheme to support LUNs and > WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This > extension is not required for the iSCSI transport protocol, but may be > useful for discovery and management. > > A URL for the target has the following form: > > scsi://hostname/path/with/ > > A URL referring to a specific LU has the following form: > > scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber > > If no LUN= term appears in the URL, then LUN 0 is assumed. > > The WWN= term is optional. If present, the party should > verify that the WWN in the LU's Device Identification Inquiry > Page correspojnds to the WWN. > > Note, the following URL: > > scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa > > refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After > connecting, the initiator should verify that the WWN > of the LU is 0a050a4bcdefa. > > -Costa > > > > From owner-ips@ECE.cmu.edu Tue Sep 19 02:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16023; 18 Sep 2000 22:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA12169 for ips-outgoing; Mon, 18 Sep 2000 20:53:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA12162 for ; Mon, 18 Sep 2000 20:53:45 -0400 (EDT) Received: from yp_portable (slip-32-100-251-193.ny.us.prserv.net [32.100.251.193]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDL5JC; Mon, 18 Sep 2000 17:51:04 -0700 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Mon, 18 Sep 2000 17:52:10 -0700 Message-ID: <000d01c021d3$d77a33a0$c1fb6420@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 In-Reply-To: <39C5F76C.C73019F3@stewart.chicago.il.us> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk From: randall@stewart.chicago.il.us > I see no viable transport protocol here and I don't see this > conversation of any use unless you get exact details AND point > to a internet draft that defines EXACTLY how it works (or possibly > some other standards document). Both I2O and VI are transport protocols which define the format of a request to a transport service provider, i.e. an adapter card. I2O is used but not limited to deliver SCSI requests and VI is used for any payload including IP packets. VI is mapped into FC with the device headers between the FC header and data payload. VI can certainly be used for delivery of SCSI requests too. Both protocols require the service provider to have reliable delivery and reception. VI defines different QoS. > > I don't claim any credit about this transport layer protocol. Every fibre > > channel and Infiniband adapter designer knows about this protocol -- > > although there is no standard. I am sure the TCP accelerator card is doing > > the same. This protocol is a great alternative to the use of TCP/IP and > > should be incorporated into iSCSI. > > No it is not. You are not offering an alternative yet.. I did not imply iSCSI should use I2O or VI. In fact, the purpose iSCSI is to map SCSI requests into IP packets as well as to define the delivery . It seems to me that the working group has set its mind on TCP/IP and is believing this is the only solution. The consensus seems if there is any other solutions that address flow control and congestion, it would end up like TCP/IP. I am simply pointing out if we keep an iSCSI request as a single atomic transaction without separating it into the TCP/IP-stream-oriented Writes and Reads that each deals with a single DU, then, the deadlock problem goes away. While the work group thinks we should take advantage the flow control and congestion management of TCP/IP, there are alternatives known as BB-credit and EE-credit management. The fibre channel adapters make reliable delivery, lost packet detection, and retransmission without TCP/IP. Randall, you are right, I did not spent time to provide the working group a draft defining such transaction-oriented protocol. All I have provided is an idea that besides TCP/IP. The designers for SCSI and fibre channel adapters have solved the head-of-queue blocking, the congestion, and retransmission problems. The transaction-oriented WRITE-REQUEST and READ-RESPONSE, in my humble opinion, allows us to implement iSCSI simpler than that of WRITE and READ stream requests. The performance cost of requiring ACKs on every DU with size greater than MTU on a network with long latency is very expensive.. By defining a greater ACK granularity is an attempt to solve this performance problem. If we do wish to ACK on every DU, then, on a long latency network, we must have a method to stream the PDUs to ensure the performance. The method should not consume a large amount of memory space. One should never ignore the TCP/IP memory-to-memory copy overhead when the backbone will be running at OC-192 speed in the near future. Finally, please don't ever ask two NIC cards to synchronize with each other. It is really hard to do as those of us in business of designing NIC cards can testify. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Tue Sep 19 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16279; 18 Sep 2000 22:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA14223 for ips-outgoing; Mon, 18 Sep 2000 21:55:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel1.hp.com (cosrel1.hp.com [156.153.255.170]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA26233 for ; Mon, 18 Sep 2000 14:03:10 -0400 (EDT) Received: from omgw1.boi.hp.com (omgw1.boi.hp.com [15.56.8.101]) by cosrel1.hp.com (Postfix) with ESMTP id 50E3DA4; Mon, 18 Sep 2000 12:03:19 -0600 (MDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by omgw1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id MAA25247; Mon, 18 Sep 2000 12:03:09 -0600 (MDT) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Mon, 18 Sep 2000 12:03:08 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: John Hufferd/San Jose/IBM , Charles Monia Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Mon, 18 Sep 2000 11:27:06 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Let me use an abstract example to illustrate why differences in underlying transports may be important. Let us assme protocol layer A uses services provided by Protocol layer B. There is an understanding of the functionality as well as other characteristics (like latency or implicit ordering that may be implicit) of the service that B provides to A. When you end up using another protocol layer C instead of B with different characteristics, then there needs to be a convergence so that users of layer A remain umimpacted. This could be achieved by providing the required convergence in C, or in A. I think it is better to address those convergence issues in C (iSCSI) than in A. Most of these are brought about by the desire to operate at full speed in the face of increased latency (without increasing frequency of events which even though theoretically possible in parallel SCSI or FC rarely happen and users may not account for as a normal operating mode). Somesh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Saturday, September 16, 2000 8:50 PM To: Charles Monia Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Charles, I have had a number of Network Client/Server apps that used TCP/IP and had multiple conversations/connections from the same Client Host to the same Data Base target (plus other Client Host that also had multiple conversations/connections). This is perhaps even normal. We also had to make sure that there was adequate memory space in the application (Database or Client) to support the thruput (both requests and responses) needed. They of course needed a method to coordinate its memory needs with the various client threads. None of this had anything to do with what was done by TCP/IP. We expected TCP/IP to work out its own problems, we did not expect the application to do anything special to help TCP/IP -- and guess what -- TCP/IP worked just fine. There is no important difference here between normal Client Server applications and SCSI. . . . John L. Hufferd Charles Monia on 09/16/2000 07:12:46 PM To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Saturday, September 09, 2000 12:05 AM > To: Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: a vote for asymmetric connections in a session > > > Joshua, > I do not understand whether this is a real issue or not. I > know of a lot > of Client Server applications that ship a lot of data on > TCP/IP, and TCP/IP > seems to be adequate. Now I understand there are some differences in > Direct Storage Access, but it is more like the other > applications then it > is different. If we can address Costa's Issue or have at least two > Connection per Asymmetric Session, I am not sure anything > else is a real > life problem. But I can be convinced, however, I would like > to understand > for each problem we come up with, why it is only a problem > for iSCSI and > not for the other real world applications, and why the SCSI > layer can not > handle the problem. > > Hi John: Here's my stab at responding to your concerns (hopefully, this isn't "swinging after the bell"): In my opinion, iSCSI, as a mapping of the SCSI device model, is fundamentally different than other client-server protocols. The difference is that it allows a client (the initiator) to have many transactions concurrently pending against a single object (the LUN). Other protocols, on the other hand, typically allow only one. As I see it, the issues we've talked about in this thread stem from this basic property. In the case of SCSI, some of these transactions may be in flight while others may be pending in the LUN waiting to be performed. When the lun receives new commands that can't be serviced for some reason, the SCSI layer handles the problem by discarding them. If the condition is corrected spontaneously, as might happen if the problem was due to a temporary resource shortage, the lun simply resumes command processing. In my view, the question we should be addressing, then, is whether or not that policy is valid in the global internet environment for which iSCSI is targeted. Over to you.... Charles From owner-ips@ECE.cmu.edu Tue Sep 19 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16293; 18 Sep 2000 22:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA14001 for ips-outgoing; Mon, 18 Sep 2000 21:48:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel1.hp.com (cosrel1.hp.com [156.153.255.170]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA12056 for ; Mon, 18 Sep 2000 20:49:42 -0400 (EDT) Received: from omgw5.rsvl.itc.hp.com (omgw5.rsvl.itc.hp.com [15.34.240.65]) by cosrel1.hp.com (Postfix) with ESMTP id 9BE1331E; Mon, 18 Sep 2000 18:49:50 -0600 (MDT) Received: from xrosebh3.rsvl.itc.hp.com (xrosebh3.rsvl.itc.hp.com [15.34.240.67]) by omgw5.rsvl.itc.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id SAA14035; Mon, 18 Sep 2000 18:49:39 -0600 (MDT) Received: by xrosebh3.rsvl.itc.hp.com with Internet Mail Service (5.5.2650.21) id ; Mon, 18 Sep 2000 18:49:39 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: John Hufferd/San Jose/IBM , Charles Monia , ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Date: Mon, 18 Sep 2000 18:49:38 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Let me use an abstract example to illustrate why differences in underlying transports may be important. Let us assme protocol layer A uses services provided by Protocol layer B. There is an understanding of the functionality as well as other characteristics (like latency or implicit ordering that may be implicit) of the service that B provides to A. When you end up using another protocol layer C instead of B with different characteristics, then there needs to be a convergence so that users of layer A remain umimpacted. This could be achieved by providing the required convergence in C, or in A. I think it is better to address those convergence issues in C (iSCSI) than in A. Most of these are brought about by the desire to operate at full speed in the face of increased latency (without increasing frequency of events which even though theoretically possible in parallel SCSI or FC rarely happen and users may not account for as a normal operating mode). Somesh -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Saturday, September 16, 2000 8:50 PM To: Charles Monia Cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session Charles, I have had a number of Network Client/Server apps that used TCP/IP and had multiple conversations/connections from the same Client Host to the same Data Base target (plus other Client Host that also had multiple conversations/connections). This is perhaps even normal. We also had to make sure that there was adequate memory space in the application (Database or Client) to support the thruput (both requests and responses) needed. They of course needed a method to coordinate its memory needs with the various client threads. None of this had anything to do with what was done by TCP/IP. We expected TCP/IP to work out its own problems, we did not expect the application to do anything special to help TCP/IP -- and guess what -- TCP/IP worked just fine. There is no important difference here between normal Client Server applications and SCSI. . . . John L. Hufferd Charles Monia on 09/16/2000 07:12:46 PM To: John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng cc: ips@ece.cmu.edu Subject: RE: a vote for asymmetric connections in a session > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Saturday, September 09, 2000 12:05 AM > To: Joshua Tseng > Cc: ips@ece.cmu.edu > Subject: RE: a vote for asymmetric connections in a session > > > Joshua, > I do not understand whether this is a real issue or not. I > know of a lot > of Client Server applications that ship a lot of data on > TCP/IP, and TCP/IP > seems to be adequate. Now I understand there are some differences in > Direct Storage Access, but it is more like the other > applications then it > is different. If we can address Costa's Issue or have at least two > Connection per Asymmetric Session, I am not sure anything > else is a real > life problem. But I can be convinced, however, I would like > to understand > for each problem we come up with, why it is only a problem > for iSCSI and > not for the other real world applications, and why the SCSI > layer can not > handle the problem. > > Hi John: Here's my stab at responding to your concerns (hopefully, this isn't "swinging after the bell"): In my opinion, iSCSI, as a mapping of the SCSI device model, is fundamentally different than other client-server protocols. The difference is that it allows a client (the initiator) to have many transactions concurrently pending against a single object (the LUN). Other protocols, on the other hand, typically allow only one. As I see it, the issues we've talked about in this thread stem from this basic property. In the case of SCSI, some of these transactions may be in flight while others may be pending in the LUN waiting to be performed. When the lun receives new commands that can't be serviced for some reason, the SCSI layer handles the problem by discarding them. If the condition is corrected spontaneously, as might happen if the problem was due to a temporary resource shortage, the lun simply resumes command processing. In my view, the question we should be addressing, then, is whether or not that policy is valid in the global internet environment for which iSCSI is targeted. Over to you.... Charles From owner-ips@ECE.cmu.edu Tue Sep 19 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16559; 18 Sep 2000 23:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA15129 for ips-outgoing; Mon, 18 Sep 2000 22:27:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA15095 for ; Mon, 18 Sep 2000 22:27:32 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id TAA00289; Mon, 18 Sep 2000 19:27:30 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Mon, 18 Sep 2000 19:27:29 -0700 Message-ID: From: Jim McGrath To: "'Y P Cheng'" , "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Mon, 18 Sep 2000 19:27:29 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I would expand your search to include non standard protocols (i.e. proprietary ones) as well if they offered something and were adequately understood by the outside world. We do that in storage quite a lot - indeed, some standard protocols are direct descendants of what were once proprietary protocols (e.g. ATA, the most widely used desktop disk interface, and ESCON, a dominant mainframe class interface (both of which originated from IBM proprietary technologies)). Jim -----Original Message----- From: Y P Cheng [mailto:ycheng@advansys.com] Sent: Monday, September 18, 2000 5:52 PM To: 'Ips@Ece. Cmu. Edu' Subject: RE: A Transport Protocol Without ACK From: randall@stewart.chicago.il.us > I see no viable transport protocol here and I don't see this > conversation of any use unless you get exact details AND point > to a internet draft that defines EXACTLY how it works (or possibly > some other standards document). Both I2O and VI are transport protocols which define the format of a request to a transport service provider, i.e. an adapter card. I2O is used but not limited to deliver SCSI requests and VI is used for any payload including IP packets. VI is mapped into FC with the device headers between the FC header and data payload. VI can certainly be used for delivery of SCSI requests too. Both protocols require the service provider to have reliable delivery and reception. VI defines different QoS. > > I don't claim any credit about this transport layer protocol. Every fibre > > channel and Infiniband adapter designer knows about this protocol -- > > although there is no standard. I am sure the TCP accelerator card is doing > > the same. This protocol is a great alternative to the use of TCP/IP and > > should be incorporated into iSCSI. > > No it is not. You are not offering an alternative yet.. I did not imply iSCSI should use I2O or VI. In fact, the purpose iSCSI is to map SCSI requests into IP packets as well as to define the delivery . It seems to me that the working group has set its mind on TCP/IP and is believing this is the only solution. The consensus seems if there is any other solutions that address flow control and congestion, it would end up like TCP/IP. I am simply pointing out if we keep an iSCSI request as a single atomic transaction without separating it into the TCP/IP-stream-oriented Writes and Reads that each deals with a single DU, then, the deadlock problem goes away. While the work group thinks we should take advantage the flow control and congestion management of TCP/IP, there are alternatives known as BB-credit and EE-credit management. The fibre channel adapters make reliable delivery, lost packet detection, and retransmission without TCP/IP. Randall, you are right, I did not spent time to provide the working group a draft defining such transaction-oriented protocol. All I have provided is an idea that besides TCP/IP. The designers for SCSI and fibre channel adapters have solved the head-of-queue blocking, the congestion, and retransmission problems. The transaction-oriented WRITE-REQUEST and READ-RESPONSE, in my humble opinion, allows us to implement iSCSI simpler than that of WRITE and READ stream requests. The performance cost of requiring ACKs on every DU with size greater than MTU on a network with long latency is very expensive.. By defining a greater ACK granularity is an attempt to solve this performance problem. If we do wish to ACK on every DU, then, on a long latency network, we must have a method to stream the PDUs to ensure the performance. The method should not consume a large amount of memory space. One should never ignore the TCP/IP memory-to-memory copy overhead when the backbone will be running at OC-192 speed in the near future. Finally, please don't ever ask two NIC cards to synchronize with each other. It is really hard to do as those of us in business of designing NIC cards can testify. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Tue Sep 19 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16858; 19 Sep 2000 0:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA17395 for ips-outgoing; Mon, 18 Sep 2000 23:53:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA17387 for ; Mon, 18 Sep 2000 23:53:09 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id UAA11659; Mon, 18 Sep 2000 20:41:45 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id UAA02894; Mon, 18 Sep 2000 20:39:27 -0700 From: "Douglas Otis" To: "Daniel Smith" , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 18 Sep 2000 20:55:29 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <200009190024.RAA24462@catalina.almaden.ibm.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Daniel, SCSI is not HTTP and does not provide a human interface as a basis for communication. Ownership of naming abstractions vary from the perspective of the client to that of the provider. The provider may not wish to share management of these naming abstractions with the client and vise-versa. Rather than attempting to keep a permanent naming abstraction as globally shared between both provider and client, these abstractions should be meaningful to the human dealing with the names. For the client it may be Tom, Dick, and Harry, and for the provider it could be Shelf A, E, and N. By keeping these abstractions independent of the transport layer, it allows each end to use such means as LDAP servers to join these views in a coherent fashion. Your IT manager will insist you not concern yourself with the IP of your workstation, as he has control of these assignments through DHCP and LDAP servers, the same should be true for a networked version of SCSI. Do not forget the large task of authentication that may also be managed by either the provider or the client. By keeping the transport as plain as possible, these naming abstractions can become as complex as required without affecting the transport specification. Implanting names into SCSI is a mistake and should be corrected as it limits this valuable freedom. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Daniel Smith > Sent: Monday, September 18, 2000 5:25 PM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Just in case people are wondering what this proposal is/was (the mailing > list seems to have exploded quite a bit since it was released > back in May), > here it is again, with a handful of [modifications] and [comments]. > -- > IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > > > > ---------------------------------------------- > > Subject: iSCSI naming > Date: Mon, 8 May 2000 15:01:17 -0700 (PDT) [mods 18 Sep 2000] > > iSCSI naming requirements (first [and a half] iteration) > ------------------------- > > It is important to have a consistent naming scheme as the basis of a > discovery system for the SCSI over TCP protocol. A basic set of > requirements is below, and a method is suggested. > > Requirements > ------------ > > A name identifies a [SCSI service delivery port], not a LUN. > notes: There may be many thousands of LUNs (and virtual LUNs) on a > connection. > why: The LUNs given to each connection are likely to be dynamic > and assigned > by the iSCSI manager. The iSCSI names are more static and long-lived, and > must be persistent even if the iSCSI manager fails. > notes: A separate naming scheme will document the LUN discovery and naming > mechanism. [query to LUN 0 to get list of LUNs] > > The namespace need not be fundamentally [systematically] enumerable. > notes: FC can be enumerated, the Internet cannot (practically). > why: iSCSI will be deployed on untrusted networks. Having > devices that can > be enumerated are subject to systematic attack. Also, many > devices will be > behind firewall machines, and not even contactable until a login procedure > has taken place---holes in such an enumerated namespace are not > acceptable. > notes: Devices on a LAN can be enumerated, and this is acceptable. > notes: Enumeration (listing) by a 3rd party is acceptable. > > The names should be as static as possible. > notes: Both in time and location. > why: A name given for one iSCSI device really ought to resolve to the same > device every time, from everywhere. [device here refers to a SCSI service > delivery port, of course the LUs should stay the same too, barring things > like security lock-outs] > > The name should not be tied to the network technology. > notes: For example, ethernet MAC address or IPv4/v6 address. > why: To cope with future networking systems, the name of the iSCSI device > should not rely only on aspects of the transport that are liable to become > obsolete, or that are not static enough. > notes: IPv4 addresses may be acceptable on a controlled private network, > however, these names should not be the canonical names as viewed from the > outside. [major problems with NAT if that happens] > > The naming scheme must be reliable. > notes: If parts of the network go down, the names must remain intact. > why: iSCSI may form part of the networking infrastructure, meaning that > after power loss, for example, the iSCSI devices will be required > to bring up > the rest of the network. This implies that the devices should be aware of > their own name, if necessary for network stability. It also implies that > any name service should have redundancy. > > The naming scheme should be nice to look at. > why: Long strings of unformatted numbers are horrible. [and prone > to typos] > > The iSCSI device being addressed should be maskable from observers. > why: The device name may contain valuable (secure) information, > so it must be > possible to hide this information from prying network interfaces. > notes: The URL scheme manages this, by separating the hostname from the > device name. > > Proposal > -------- > We should follow the URL naming scheme for iSCSI devices, thus: > scsi:/// > Where > :=|: is the TCP [destination] identifier... > is the DNS or DDNS host name of the iSCSI device and > is the TCP port number to connect to. > > And [ should be ] > :=|/|? is the device identifier > is a text identifier for the device > :=|? is an argument for the device > [argument escape character was '&' in the original] > > The element is reasonably well defined. However, I would prefer to > keep the element more free-form. The scsis:/// is > also a straightforward extension. [to secure (SSL) iSCSI] > > The identifier specifies an iSCSI TCP connection > identifier only. > This may provide access to one or many LUNs that flit in and out of LUN > space. > > One caveat is that some (most?) iSCSI devices will not have > access to DNS. > Fortunately, devices (targets) do not need to [query the network to find] > their names except in the case of trying to resolve another device > (target)---i.e., only initiators need DNS resolution. > > Here are some fun examples of the things I would want to do as a storage > manager. > > The "backup" program does a simple block-to-block copy from one > iSCSI device > to another... > backup scsi://raid.acme.com/lun12 scsi://tape.acme.com/id25 > Of course, a more useful operation may be... > flashcopy scsi://disk.acme.com/12/2 scsi://disk.acme.com/13/2 > where the pathnames are ASCII numbers. (They may even be the WWN or LUN > value.) > > This doesn't address the issue of security. When we have more of those > issues resolved, we might want to issue a command like this... > kcopy scsis://secure.acme.com/raid12/set2?key=12345?auth=dfsmith > ...so that the part, containing the security tokens, is > never passed > in the clear over the network. > > It may be possible for a manager to transparently interpret the arguments > or pathname into a LUN list in a storage farm. > iscsi attach scsi://farm.acme.com/tray17?LUN=0x17?LUN=0x18 > /dev/scsi2 > iscsi attach scsi://san.acme.com/5A/27/23/99/B7/A2/01/00 /dev/sda > > The Text/Response commands in the iSCSI Internet-Draft might be > accommodated > thusly: > iscsitext scsi://tape.scsi.acme.com/cart5 "com.ibm.retension:yes" \ > "com.ibm.eject:now" > > com.ibm.retension:ok retensioning > > com.ibm.eject:ok scheduled ETA 2:30 > > Secure enumeration might be done in the following way, with peculiar > examples of possible Un*x integration. > iscsi find_domain_manager > > iscsi.acme.com > httpsget https://iscsi.acme.com:99/root&auth=dfsmith&key=1234 > > > > > > scsi://unit1.raid.acme.com/ > > disk1 disk2 disk3 disk4 disk8 disk9 > > > > > > > > scsi://bigtape.acme.com/racf4 > > cart57 cart58 > > > > > > > > scsi://ibmdeskstar75.dfsmith.acme.com/lun1 > > > > > iscsi mount scsi://ibmdeskstar75.dfsmith.acme.com/lun1 /mnt/mydisk > > /mnt/mydisk: 73564224 blocks available > tar -cf 'scsi://bigtape.acme.com/racf4/cart57&user=dfsmith' ~/* & > > [3] 4291 > runsocks dd if=/dev/zero of=scsi://dfsmith.blocks4less.com/disk1 & > > [4] 4293 > And so on. (Note: I am not an XML expert.) > > [Additional notes: > > The decision to resolve devices in the host name or in the device name is > really up to the implementer. Some companies may already have systems in > place for handling the DNS while they have no existing SAN > infrastructure. > Thus the decision to name a particular service delivery port > scsi://disk26.farm-b.acme.com/ > or > scsi://farm-b.acme.com/disk26 > is up to the system designer. > > The iSCSI URL names a bundle of LUs (one iSCSI "connection"). People love > to point out that this is a problem if you have multiple entrance > ways into > the same SAN, and the same device (say WWN=0x1234) has different LUNs (say > LUN=5 on iscsi://scsi.acme.com/bridge1 and LUN=477 on > iscsi://scsi.acme.com/bridge2). So don't do that. Or do that > and just live > with it. Or make sure the different initiator/bridges into the SAN assign > the device the same LUN. (There isn't really a problem using a LUN in a > global context, as long as it's always attached to its service delivery > port.) (Having to query every LUN on a port to get the list of WWNs ought > to be addressed by T10, in the same way they covered the "query > LUN 0 to get > a list of LUNs" problem. IMO!) > > The security aspects bring up the spectre of LUs being masked out for > different iSCSI initiators on different parts of the net. This is fun and > nothing to worry about. > > These URL names are used in several places, most of which are not > related to > the iSCSI transport itself. (Likewise, html is not particularly > related to > HTTP, nor SMTP to metamail.) > > First by users (or users by dint of clobbering an html hyperlink) who want > to attach to iSCSI devices. The names are descriptive and are > malleable to > all sorts of marvelous manipulations by the sysadmins. > > Second, by directory servers, who can deliver a list of the URLs (and the > LUNs that are accessible therein) to the iSCSI clients. > > Third, by iSCSI itself for the 3rd party copy command. In this case, the > URL specifies the destination SAN and the target LUN is carried in the > additional data field. > > ] > > > > > ---------------------------------------------- > > Jim Hafner/Almaden/IBM wrote: > > > > I've seen Daniel's proposal before and I still haven't figured out the > > context in which such names are to be used. > > > > Given that, I'm personally a bit uncomfortable with any naming > convention > > that uses LUN values in a global context. These have no meaning as LUNs > > are host-specific values. Even worse, LUNs are addresses, not names! > > > > I have no problem with WWNs in a global context as that is what they are > > for! > > > > A subtle difficulty, is that a WWN does not give a host an > address (either > > for the target or the LU). Even in SPC-2, EXTENDED COPY's > Identification > > Descriptor Target Descriptor Format (sic?) where WWNs are used, says > > "instructs the copy manager to locate a target and logical unit that > > returns a device identification VPD page ..." but gives no hints on how > > that should be done (e.g., walk the bus and do INQUIRY to > everything...?). > > > > The "hostname" and DNS provide a canonical method for getting an address > > for a name, so that part is OK. The only defined way to get an address > > (LUN) for a LU WWN is by exhaustive INQUIRY to each LU at a > given target. > > > > On the other hand, I can envision how some of these things > might work, in a > > well-coordinated and well-managed environment: > > > > There is a managing application which has the dual responsibility of > > managing the targets for their host/LUN mapping > configurations and also > > serves as the "walk the bus" discovery resource for hosts. > > > > When the manager (human?) determines the distribution of > host/target LU > > resources (after inventorying both hosts and target LUs), the > > application will send to the targets whatever host/LUN > Mapping (access > > controls?) commands are necessary to configure them. Later, > when a host > > boots it queries the application to get its "walk-the-bus" > services. In > > this, it minimally gets from this application a list of targets that > > have LUs to which it should have access. There is no REQUIREMENT at > > this point for the application to give the host anything more as the > > normal SCSI LU discovery process per target kicks in. On the other > > hand, I can see some value in the host getting more information about > > the LUs it will see at each target. In that case, the host could get > > from the application a list of targets and LU identifiers > (these could > > be LUNs or WWN or both, as you suggested). The important > point here, is > > that these LUNs are valid only in the context of the > requesting HOST and > > are not global! > > > > Note that this process requires clear coordination between > the "target > > configurator" and "host discovery of targets". > > > > > > Given an appropriate context for a URL scheme, I'd make two > modifications: > > 1) drop the assumption that no LUN means LUN=0. If no LU > qualifier (e.g., > > no LUN or WWN (or other) is provided), then LUN=0. In all > other cases, the > > alternative names should be sufficient to identify a LU, > though, as above, > > finding a LUN address may require extra work. > > 2) add ?ProxyToken= as an additional option here (this > coordinates > > well with the changes to EXTENDED COPY approved in the context > of the SCSI > > access controls). > > > > Well, that's another two cents! I was hoping to save some of this > > discussion until the protocol gets worked out, but .... > > > > Jim Hafner > > > > > > csapuntz@cisco.com@ece.cmu.edu on 09-17-2000 11:11:24 PM > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > To: IP Storage > > cc: csapuntz@cisco.com > > Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > > > > > Here is a proposal to extend the iSCSI URL scheme to support LUNs and > > WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This > > extension is not required for the iSCSI transport protocol, but may be > > useful for discovery and management. > > > > A URL for the target has the following form: > > > > scsi://hostname/path/with/ > > > > A URL referring to a specific LU has the following form: > > > > scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber > > > > If no LUN= term appears in the URL, then LUN 0 is assumed. > > > > The WWN= term is optional. If present, the party should > > verify that the WWN in the LU's Device Identification Inquiry > > Page correspojnds to the WWN. > > > > Note, the following URL: > > > > scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa > > > > refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After > > connecting, the initiator should verify that the WWN > > of the LU is 0a050a4bcdefa. > > > > -Costa > > > > > > > > > > From owner-ips@ECE.cmu.edu Tue Sep 19 05:10:07 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16947; 19 Sep 2000 1:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA17397 for ips-outgoing; Mon, 18 Sep 2000 23:53:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA17381 for ; Mon, 18 Sep 2000 23:53:06 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id UAA11655; Mon, 18 Sep 2000 20:41:43 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id UAA02891; Mon, 18 Sep 2000 20:39:25 -0700 From: "Douglas Otis" To: "Y P Cheng" , "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Mon, 18 Sep 2000 20:55:28 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <000d01c021d3$d77a33a0$c1fb6420@yp_portable.advansys.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Cheng, The feature you desire is out of sequence data processing. As far as SCSI is concerned, either the request is completed with read data or a write response and has nothing to do with return of ACKs except in the condition where a sequence must be maintained. The principle feature VI offered was out of order delivery and SCSI typically also has this behavior. TCP is a constriction to this freedom and it represents a sizeable performance loss with head of queue blocking. To design a system that requires head of queue blocking, recovery must process at twice a normal rate due to parked frames. A pointer scheme as a TCP option to provide framing and out of sequence processing opens the window of allowable sequences increasing susceptibility to spoofing and muddles the state of the receiver. A second pass of TCP is not likely to produce the same frame boundaries and is ugly to handle. What happens when this pointer is wrong. SCTP provides these features and removes the multiple independent connections. SCSI protocol expects the target to be in control of the exchange. Allowing advanced sending in the form of a credit from the target does not change SCSI protocol. Creating check-conditions as a means of control violates SCSI and will likely choke the applications. Discarding data is a waste of network bandwidth. As it is intended to be the aggregation of many logical devices, IP transport can not provide flow control to each device. You need an additional scheme. In the case of FC, the solicitations from the target seems like a small cost compared to the present specification. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Y > P Cheng > Sent: Monday, September 18, 2000 5:52 PM > To: 'Ips@Ece. Cmu. Edu' > Subject: RE: A Transport Protocol Without ACK > > > From: randall@stewart.chicago.il.us > > I see no viable transport protocol here and I don't see this > > conversation of any use unless you get exact details AND point > > to a internet draft that defines EXACTLY how it works (or possibly > > some other standards document). > > Both I2O and VI are transport protocols which define the format > of a request > to a transport service provider, i.e. an adapter card. I2O is > used but not > limited to deliver SCSI requests and VI is used for any payload > including IP > packets. VI is mapped into FC with the device headers between > the FC header > and data payload. VI can certainly be used for delivery of SCSI requests > too. Both protocols require the service provider to have > reliable delivery > and reception. VI defines different QoS. > > > > I don't claim any credit about this transport layer protocol. Every > fibre > > > channel and Infiniband adapter designer knows about this protocol -- > > > although there is no standard. I am sure the TCP accelerator card is > doing > > > the same. This protocol is a great alternative to the use of > TCP/IP and > > > should be incorporated into iSCSI. > > > > No it is not. You are not offering an alternative yet.. > > I did not imply iSCSI should use I2O or VI. In fact, the purpose iSCSI is > to map SCSI requests into IP packets as well as to define the > delivery . It > seems to me that the working group has set its mind on TCP/IP and is > believing this is the only solution. The consensus seems if there is any > other solutions that address flow control and congestion, it would end up > like TCP/IP. I am simply pointing out if we keep an iSCSI request as a > single atomic transaction without separating it into the > TCP/IP-stream-oriented Writes and Reads that each deals with a single DU, > then, the deadlock problem goes away. While the work group > thinks we should > take advantage the flow control and congestion management of TCP/IP, there > are alternatives known as BB-credit and EE-credit management. The fibre > channel adapters make reliable delivery, lost packet detection, and > retransmission without TCP/IP. > > Randall, you are right, I did not spent time to provide the > working group a > draft defining such transaction-oriented protocol. All I have provided is > an idea that besides TCP/IP. The designers for SCSI and fibre channel > adapters have solved the head-of-queue blocking, the congestion, and > retransmission problems. The transaction-oriented WRITE-REQUEST and > READ-RESPONSE, in my humble opinion, allows us to implement iSCSI simpler > than that of WRITE and READ stream requests. The performance cost of > requiring ACKs on every DU with size greater than MTU on a > network with long > latency is very expensive.. By defining a greater ACK granularity is an > attempt to solve this performance problem. If we do wish to ACK on every > DU, then, on a long latency network, we must have a method to stream the > PDUs to ensure the performance. The method should not consume a large > amount of memory space. One should never ignore the TCP/IP > memory-to-memory > copy overhead when the backbone will be running at OC-192 speed > in the near > future. Finally, please don't ever ask two NIC cards to synchronize with > each other. It is really hard to do as those of us in business > of designing > NIC cards can testify. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. > From owner-ips@ECE.cmu.edu Tue Sep 19 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18059; 19 Sep 2000 4:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA22008 for ips-outgoing; Tue, 19 Sep 2000 03:29:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA22004 for ; Tue, 19 Sep 2000 03:29:47 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA10346; Tue, 19 Sep 2000 09:29:15 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA96016; Tue, 19 Sep 2000 09:29:14 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695F.00291F2A ; Tue, 19 Sep 2000 09:29:09 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Charles Monia , ips@ece.cmu.edu Message-ID: Date: Tue, 19 Sep 2000 09:48:26 +0300 Subject: RE: bridging issues -- Converging FCP-2 and iSCSI control structures Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, The "Additional Data" field in the SCSI Command PDU may be used to send either "immediate data," or a CDB that is longer than 16 bytes, or both. To determine whether the "Additional Data" is "immediate data" or part of a long CDB, the flags (Byte #1, bits 6-7) must be examined. The most expected case is that the CDB fits into the allocated 16 byte space, so only "immediate data" would be in the "Additional Data" field; in this case the flags (bits 6-7) are 00b. The current specification is not very neat with regard to this point, but using extra bytes to separate the "immediate data" length from the extra CDB length would have necessitated expanding the iSCSI header even more. How do others think we should clean up this part of the iSCSI header? The use of "immediate data" is for the case when the data for a command is small, in which case the entire data may be sent together with the SCSI Command PDU as "immediate data," thus avoiding the need to send a separate SCSI Data PDU. - Kalman Charles Monia on 19/09/2000 03:09:20 Please respond to Charles Monia To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: (bcc: Kalman Meth/Haifa/IBM) Subject: RE: bridging issues -- Converging FCP-2 and iSCSI control struct ures <... deleted ...> Julo wrote: > There is some misunderstanding about the CDB length - the > iSCSI is pretty > liberal only that it use a tricky coding of the length field > (not something > we would necessarily want to keep in). > Charles wrote: I may well have misunderstood something here. After reading the iSCSI spec, I was not clear as to what the extra data field in the iSCSI command PDU was to be used for. It was my surmise that you may have intended this for CDBs in excess of 16 bytes -- although, I didn't glean that from the wording in the spec. <... deleted ..> Regards, Charles From owner-ips@ECE.cmu.edu Tue Sep 19 11:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18784; 19 Sep 2000 7:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA26012 for ips-outgoing; Tue, 19 Sep 2000 06:07:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA26006 for ; Tue, 19 Sep 2000 06:07:26 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA185110; Tue, 19 Sep 2000 12:06:44 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA18202; Tue, 19 Sep 2000 12:06:43 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125695F.0037892B ; Tue, 19 Sep 2000 12:06:35 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Peter Johansson , IP Storage Message-ID: Date: Tue, 19 Sep 2000 12:23:49 +0300 Subject: Re: iSCSI: 2.2.6. Naming & mapping Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Peter, By "transport-dependent" I assume you mean dependent on the SCSI transport (in our case iSCSI, similar to the WWN used in FCP) rather than on TCP. You seem to be suggesting that we should define in iSCSI a way to name and discover LUs without performing SCSI Inquiry commands. (Do I understand you correctly?) Can you please give a scenario of the type of case you have in mind with the benefits of introducing this new feature? In your scenario, would the ability to name the particular LU be crucial, or would it still be sufficient to simply name the target device and then perform Inquiry on the target device? Could you also suggest how you think the naming of a particular LU should be specified? Thanks, - Kalman Peter Johansson on 18/09/2000 01:03:10 Please respond to Peter Johansson To: IP Storage cc: (bcc: Kalman Meth/Haifa/IBM) Subject: Re: iSCSI: 2.2.6. Naming & mapping <... deleted ...> Jim Hafner's observation (that mandatory presence of a unique ID, on a per LU basis, in EVPD appears to be destined for adoption within SPC-2) only reinforces that part of the discussion with which I already agree. It is desirable to have command set-dependent methods to determine the unique ID for a LU. Matter closed? It might also be desirable to have transport-dependent mechanisms to specify unique IDs for those LUs whose existence is discoverable by transport-dependent methods. I think this is still open to discussion. <... deleted ...> Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Tue Sep 19 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20004; 19 Sep 2000 11:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA04297 for ips-outgoing; Tue, 19 Sep 2000 10:15:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA29568 for ; Tue, 19 Sep 2000 07:46:57 -0400 (EDT) Received: from cisco.com (sj-isp-nat-pool-34.cisco.com [204.69.198.34]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA15613; Tue, 19 Sep 2000 06:33:36 -0500 Message-ID: <39C74DEB.BBD04720@cisco.com> Date: Tue, 19 Sep 2000 06:28:43 -0500 From: Randall Stewart X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Stephen Byan CC: ips@ece.cmu.edu Subject: Re: Zero-copy TCP stacks (Was: Avoiding deadlock in iSCSI) References: <8133266FE373D11190CD00805FA768BF055BD1BB@shrcmsg1.tdh.qntm.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen: comments in line... (even though I too am not a expert :) Stephen Byan wrote: > I may be merely increasing the noise floor; if so, I beg forgiveness in > advance... > > Randall R. Stewart [mailto:randall@stewart.chicago.il.us] wrote: > > Prasenjit Sarkar/Almaden/IBM wrote: > > > BTW, zero-copy TCP/IP stacks have a lot of caveats (e.g. > > memory alignment > > > etc) which > > > is why they have never made it to any operating system. > > There are rough > > > implementations > > > in Solaris, BSD and Linux but none of them are particularly > > close to being > > > robust > > > (I have tried them all). > > > > Prasenjit: > > > > I have used a zero-copy TCP/IP stack (if I recall correctly) > > in VxWorks > > or was it VRTX... I will grant you these are NOT *NiX systems and > > are just recently beginning to become PosiX compliant... but I > > seem to remember the zero-copy symenatics and I don't think there > > was a memory alignment requriement.. it has been a while so I > > may be mis-remembering.. :) > > I'm no TCP expert, but I think the difference between Randall's experience > and Prasenjit's experience relates to the extent of the memory protection > and security guarantees offered by the operating system. > > Embedded OS's such as VxWorks and VRTX do not provide separate memory > address spaces for each task (aka process). Consequently there's no security > problem in handing a pointer to an arbitrary memory buffer to a task - it > can already see all of memory anyway. > > "Real OS's (TM)" do provide separate memory address spaces for each process > (aka task). Consequently there is a security problem in handing a pointer to > an arbitrary memory buffer to a task - the process could potentially view > packets which happen to share the same memory page but which belong to other > processes. To solve this problem, the TCP buffers must be aligned to a page > boundary, and at most one buffer can be allocated per page. > I think you have it correct here. I have always considered VxWorks and VRTX more of a monitor than a O/S :) ... Now one point here though is that at least one of the ends of the communication that is running I-SCSI may well be in this vain as well i.e the disk side... Oh, I also remember once building something for lynx-o/s .. (this DOES have the MMU turned on,so it meets my defintion of a O/S :>) that I was able to pass a page of speech to it via a system call and it would send it back to me when it was done... This was real specific to a particular application i.e. speech processing but I do NOT see that TCP APIcould not also be "adjusted" to allow this. We have talked a LOT on this list about "special" interfaces to TCP. Not that I am any way advocating making adjustments to TCP :) I think this is a bad idea and I think the working group would do better going with SCTP (of course I am biased ) R > > Regards, > -Steve > > P.S. to the TCP experts - did I get this right? > > Steve Byan > > Design Engineer > MS 1-3/E23 > 333 South Street > Shrewsbury, MA 01545 > (508)770-3414 > fax: (508)770-2604 From owner-ips@ECE.cmu.edu Tue Sep 19 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20099; 19 Sep 2000 11:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA04309 for ips-outgoing; Tue, 19 Sep 2000 10:15:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA29787 for ; Tue, 19 Sep 2000 07:56:28 -0400 (EDT) Received: from cisco.com (sj-isp-nat-pool-34.cisco.com [204.69.198.34]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA15629; Tue, 19 Sep 2000 06:43:05 -0500 Message-ID: <39C75024.6B84287@cisco.com> Date: Tue, 19 Sep 2000 06:38:12 -0500 From: Randall Stewart X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jim McGrath CC: "'Y P Cheng'" , "'Ips@Ece. Cmu. Edu'" Subject: Re: A Transport Protocol Without ACK References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim: Any transport protocol proposal is ok. As long as it can be seen and reviewed. So far I have seen only two TCP and SCTP. Oh, a little side note, any transport protocol proposed MUST be able to show TCP like behavior in the face of congestion. And I think, IMHO, that this means that if it is NOT using RFC2581 procedures it MUST show that it does backoff and share with TCP. It also has a HEAVY burden of proof to show this facility at least in my mind and I would think in the IESG's mind as well... R Jim McGrath wrote: > I would expand your search to include non standard protocols (i.e. > proprietary ones) as well if they offered something and were adequately > understood by the outside world. We do that in storage quite a lot - > indeed, some standard protocols are direct descendants of what were once > proprietary protocols (e.g. ATA, the most widely used desktop disk > interface, and ESCON, a dominant mainframe class interface (both of which > originated from IBM proprietary technologies)). > > Jim > > -----Original Message----- > From: Y P Cheng [mailto:ycheng@advansys.com] > Sent: Monday, September 18, 2000 5:52 PM > To: 'Ips@Ece. Cmu. Edu' > Subject: RE: A Transport Protocol Without ACK > > From: randall@stewart.chicago.il.us > > I see no viable transport protocol here and I don't see this > > conversation of any use unless you get exact details AND point > > to a internet draft that defines EXACTLY how it works (or possibly > > some other standards document). > > Both I2O and VI are transport protocols which define the format of a request > to a transport service provider, i.e. an adapter card. I2O is used but not > limited to deliver SCSI requests and VI is used for any payload including IP > packets. VI is mapped into FC with the device headers between the FC header > and data payload. VI can certainly be used for delivery of SCSI requests > too. Both protocols require the service provider to have reliable delivery > and reception. VI defines different QoS. > > > > I don't claim any credit about this transport layer protocol. Every > fibre > > > channel and Infiniband adapter designer knows about this protocol -- > > > although there is no standard. I am sure the TCP accelerator card is > doing > > > the same. This protocol is a great alternative to the use of TCP/IP and > > > should be incorporated into iSCSI. > > > > No it is not. You are not offering an alternative yet.. > > I did not imply iSCSI should use I2O or VI. In fact, the purpose iSCSI is > to map SCSI requests into IP packets as well as to define the delivery . It > seems to me that the working group has set its mind on TCP/IP and is > believing this is the only solution. The consensus seems if there is any > other solutions that address flow control and congestion, it would end up > like TCP/IP. I am simply pointing out if we keep an iSCSI request as a > single atomic transaction without separating it into the > TCP/IP-stream-oriented Writes and Reads that each deals with a single DU, > then, the deadlock problem goes away. While the work group thinks we should > take advantage the flow control and congestion management of TCP/IP, there > are alternatives known as BB-credit and EE-credit management. The fibre > channel adapters make reliable delivery, lost packet detection, and > retransmission without TCP/IP. > > Randall, you are right, I did not spent time to provide the working group a > draft defining such transaction-oriented protocol. All I have provided is > an idea that besides TCP/IP. The designers for SCSI and fibre channel > adapters have solved the head-of-queue blocking, the congestion, and > retransmission problems. The transaction-oriented WRITE-REQUEST and > READ-RESPONSE, in my humble opinion, allows us to implement iSCSI simpler > than that of WRITE and READ stream requests. The performance cost of > requiring ACKs on every DU with size greater than MTU on a network with long > latency is very expensive.. By defining a greater ACK granularity is an > attempt to solve this performance problem. If we do wish to ACK on every > DU, then, on a long latency network, we must have a method to stream the > PDUs to ensure the performance. The method should not consume a large > amount of memory space. One should never ignore the TCP/IP memory-to-memory > copy overhead when the backbone will be running at OC-192 speed in the near > future. Finally, please don't ever ask two NIC cards to synchronize with > each other. It is really hard to do as those of us in business of designing > NIC cards can testify. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Tue Sep 19 18:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20722; 19 Sep 2000 14:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA12647 for ips-outgoing; Tue, 19 Sep 2000 13:31:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA12641 for ; Tue, 19 Sep 2000 13:31:45 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id NAA44862; Tue, 19 Sep 2000 13:30:02 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8JHVfi160644; Tue, 19 Sep 2000 11:31:41 -0600 X-Priority: 1 (High) Importance: Normal Subject: We are drifting To: black_david@emc.com Cc: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 19 Sep 2000 10:30:44 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/19/2000 11:31:40 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Perhaps I am wrong but it seems we are drifting and not focusing on the search for concurrence regarding Symmetric vs Asymmetric. What is your view? . . . John L. Hufferd From owner-ips@ECE.cmu.edu Tue Sep 19 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20735; 19 Sep 2000 14:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA11312 for ips-outgoing; Tue, 19 Sep 2000 13:02:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA11302 for ; Tue, 19 Sep 2000 13:02:23 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id KAA17413 for ; Tue, 19 Sep 2000 10:02:21 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Tue, 19 Sep 2000 10:02:16 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1C0@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: A Transport Protocol Without ACK Date: Tue, 19 Sep 2000 10:02:11 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P Cheng [mailto:ycheng@advansys.com] wrote: > While the work group > thinks we should > take advantage the flow control and congestion management of > TCP/IP, there > are alternatives known as BB-credit and EE-credit management. These are flow control mechanisms, not congestion control mechanisms. Fibre channel and InfiniBand do not have congestion control of any sort. > The fibre > channel adapters make reliable delivery, lost packet detection, and > retransmission without TCP/IP. > > Randall, you are right, I did not spent time to provide the > working group a > draft defining such transaction-oriented protocol. All I > have provided is > an idea that besides TCP/IP. The designers for SCSI and fibre channel > adapters have solved the head-of-queue blocking, the congestion, and > retransmission problems. They have not solved the congestion problem. Please read RFC RFC 2914 "Congestion Control Principles" and RFC 896 "Congestion Control in IP/TCP Internetworks". Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Tue Sep 19 18:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20774; 19 Sep 2000 14:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA12966 for ips-outgoing; Tue, 19 Sep 2000 13:37:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA12961 for ; Tue, 19 Sep 2000 13:37:32 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id C025029A for ; Tue, 19 Sep 2000 10:37:27 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA07537; Tue, 19 Sep 2000 10:37:26 -0700 (PDT) Message-ID: <39C7A5E6.B8E703CC@hp.com> Date: Tue, 19 Sep 2000 10:44:06 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: LU access through an iSCSI session Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, Thinking about what could be an implementation of an iSCSI session, several questions came to my mind for which i was unable to find an answer in the draft. 1) Do commands directed to a particular LU through a session are able to use any of the TCP connections within the session? As the TCP connections could connect to various "service delivery ports" it is not guarantee that a particular LU is visible through all the connections. 2) Do commands directed to a particular LU through a session need to be associated to different LUNs depending on the TCP connection used or is it always the same LUN for all the connections? As the TCP connections could connect to various "service delivery ports" of the target, it seems that the LUN could differ depending on the connection used. In the iSCSI draft in 2.2.1 there is: "The group of TCP connections linking an initiator with a target forms a session (loosely equivalent to a SCSI nexus)." Does it means that the same LUN is used across the TCP connections? Depending on the answer to these questions the failover/load balancing can be handled in different ways: -> If the answer for 1) is "All TCP connections can be used" and the answer for 2) is "Same LUN regardless of the connexion" For fail over and load balancing inside a session, there is no need to work at the (thin) LU granularity, working at the granularity of the TCP connexion is sufficient. For example in case of failed TCP connexion all the traffic for this failed TCP connexion can be directed on the other connections regardless of the LU. It is very quick, straight forward and may add value compared to the legacy solutions that are doing the fail over on a LU basis (one LU command returns an error or timeout, the next commands for this LU will use an alternate path) In the case where a large number (N) of LUs access are multiplexed to a target port, N switch over are needed. The same result could be achieved in one operation (TCP connection switch over). -> If the answer for 1) is "All TCP connections can be used" and the answer for 2) is "different LUN depending on the connexion" We need to work at the LU granularity. For each command the right LUN as to be determined depending on the tcp connection selected. -> If the answer for 1) is "not all TCP connections can be used, it is LU dependent" On top of the previous case extra work, for each LU it must be a list of TCP connections usable that needs to be looked into before routing the command to one TCP connection. It means that some LUs access using a session could survive to a TCP connection failure some other LUs no. Regards, Pierre From owner-ips@ECE.cmu.edu Tue Sep 19 20:10:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22063; 19 Sep 2000 16:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA16275 for ips-outgoing; Tue, 19 Sep 2000 14:43:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch11.nsmg.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA16264 for ; Tue, 19 Sep 2000 14:43:07 -0400 (EDT) Received: by lmoxch11.nsmg.veritas.com with Internet Mail Service (5.5.2448.0) id ; Tue, 19 Sep 2000 14:43:53 -0400 Message-ID: From: Roger Cummings To: "'ips@ece.cmu.edu'" Subject: RE: A Transport Protocol Without ACK Date: Tue, 19 Sep 2000 14:43:52 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I'm trying for a pre-emptive strike on this particular rathole before it opens. Fibre Channel does in fact have some tools with which to implement a congestion control mechanism, but they are implemented in a class of service (Class 2) that is not widely used today. The tools are a buffer-to-buffer flow control scheme (separate from end-to-end flow control) to prevent a particular fabric element being overwhelmed, and the ability for FC fabrics to return a special frame (called F_Busy) indicating that onward progress was not possible - basic Explicit Congestion Notification (ECN). There had even been talk about the equivalent of a slow-start algorithm, though in the terms of RFC2914 we had never quite got to AIMD (Additive-Increase Multiplicative Decrease). I am NOT proposing that this WG look at these processes. Quite to the contrary, I'm trying to point out that there a number of significant warts on this scheme - the fact that it is port-based, not connection or flow-based, and that the transmitter backoff is implied rather than closely defined. Whatever we are going to do for congestion control in IP Storage, it cannot be based on this scheme, which does not even pass a top-level inspection for suitability. In summary, Steve wasn't quite correct. Yes FC does have some tools for responding to the congestion problem in its own domain, but No they are definitely not what we want for IP Storage. Regards, Roger Cummings VERITAS roger.cummings@veritas.com > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Tuesday, September 19, 2000 1:02 PM > To: 'ips@ece.cmu.edu' > Subject: RE: A Transport Protocol Without ACK > > > Y P Cheng [mailto:ycheng@advansys.com] wrote: > > > While the work group > > thinks we should > > take advantage the flow control and congestion management of > > TCP/IP, there > > are alternatives known as BB-credit and EE-credit management. > > These are flow control mechanisms, not congestion control > mechanisms. Fibre > channel and InfiniBand do not have congestion control of any sort. > > > The fibre > > channel adapters make reliable delivery, lost packet detection, and > > retransmission without TCP/IP. > > > > Randall, you are right, I did not spent time to provide the > > working group a > > draft defining such transaction-oriented protocol. All I > > have provided is > > an idea that besides TCP/IP. The designers for SCSI and > fibre channel > > adapters have solved the head-of-queue blocking, the congestion, and > > retransmission problems. > > They have not solved the congestion problem. Please read RFC RFC 2914 > "Congestion Control Principles" and RFC 896 "Congestion > Control in IP/TCP > Internetworks". > > Regards, > -Steve > > Steve Byan > > Design Engineer > MS 1-3/E23 > 333 South Street > Shrewsbury, MA 01545 > (508)770-3414 > fax: (508)770-2604 > > From owner-ips@ECE.cmu.edu Tue Sep 19 20:10:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22046; 19 Sep 2000 16:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA18268 for ips-outgoing; Tue, 19 Sep 2000 15:23:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA18262 for ; Tue, 19 Sep 2000 15:23:05 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 19 Sep 2000 15:22:24 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FBE@corpmx9.isus.emc.com> To: hufferd@us.ibm.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: iSCSI: Session Partial Resolution Date: Tue, 19 Sep 2000 15:22:20 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Perhaps I am wrong but it seems we are drifting and not focusing on the > search for concurrence regarding Symmetric vs Asymmetric. What is your > view? John's acquiring the prescient talent of causing the co-chair to appear and take action ... as long as he doesn't start calling himself a "wizard" based on these summoning powers, I won't object :-). In any case ... In the past week, I have seen at most one objection to each of the following two proposed points of consensus: (1) An iSCSI session containing a single TCP connection should not be required to use the currently specified iSCSI command reference numbers and sliding window mechanism because TCP will deliver commands in order. (2) Use of more than one TCP connection per iSCSI session is OPTIONAL. Therefore I declare these to be the WG rough consensus on these issues, and the next version of the iSCSI draft should remove the command reference numbers and sliding window mechanism from the iSCSI header. Somesh Gupta's objection to (1) and Matt Wakeley's continued objection to (2) are noted as part of declaring these to be the WG rough consensus. Anyone else who objects to this declaration of rough consensus should email me directly with the reasons for the objection. OTOH, I do not see consensus on the session model for multiple connection sessions among the Asymmetric model, the Symmetric model, and Pierre Labat's proposal. In order to make progress on iSCSI, I see no alternative to separating multi-connection sessions from the main iSCSI spec. Significant effort and email traffic has been invested in this topic for at least 6 weeks and the issue is not settled -- I don't think holding up the iSCSI spec for another 6+ weeks in hopes of settling this issue on the list is an effective way to make progress, but I'm prepared to listen to dissenting opinions (e.g., if someone thinks there is rough consensus, and I've missed it); please send such opinions directly to me rather than using the list. I've already had one offline comment from an outside observer expressing amazement at the willingness of this community to discuss multi-connection sessions "ad nauseum". Therefore, I would ask that the authors of the next version of the iSCSI draft delete all specification of multiple connection sessions from the next version of the except for a note that they will be handled in a separate document. Producing that separate document is going to require an offline design team. The design team can either be chartered to write a compromise session specification or to evaluate competing specifications and choose one. My current inclination is to do the latter, which would involve having the design team produce a set of requirements and guidelines for session specifications in consultation with the co-chairs, evaluate Internet-Drafts documenting the specifications, and recommend an approach to the WG. Comments on this process are solicited - either on the list or to me directly. Further discussion of multi-connection sessions on the list is probably not a good use of list bandwidth. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Sep 19 20:10:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22058; 19 Sep 2000 16:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA18482 for ips-outgoing; Tue, 19 Sep 2000 15:28:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA18473 for ; Tue, 19 Sep 2000 15:28:11 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 1C31512F5 for ; Tue, 19 Sep 2000 12:28:09 -0700 (PDT) Received: from mk731912 (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA07525 for ; Tue, 19 Sep 2000 12:28:00 -0700 (PDT) Message-Id: <4.2.2.20000919092445.00b3df00@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 19 Sep 2000 09:33:34 -0700 To: From: Michael Krause Subject: RE: VI (Was: Avoiding deadlock in iSCSI) In-Reply-To: <8133266FE373D11190CD00805FA768BF055BD1B9@shrcmsg1.tdh.qntm .com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:07 AM 9/18/00 -0700, Stephen Byan wrote: > My point was that in return for the >pain, we can leverage the utility of the RDMA hardware assists, since the >assists would be encapsulated in the VI layer, rather than intimately part >of the iSCSI protocol. Consequently the RDMA hardware could be reused to >accelerate other new protocols, as well as cluster message passing. RDMA != VI though VI does use RDMA technologies. Prefer to see discussion focused on what RDMA operations are required, what are the error and ordering requirements, etc. Much of this has been improved upon within InfiniBand and would ideally just be "lifted" from that architecture with minimal if any modifications. The rest of the functionality within VI or InfiniBand could be used / avoided as required. >I think an enlarged market for the hardware is important in order to ensure >the success of iSCSI. iSCSI depends on getting RDMA hardware acceleration >NICs into the servers. Consider two product scenarios: > >1) a NIC with RDMA hardware acceleration for iSCSI. >2) a NIC with RDMA hardware acceleration for iSCSI, DAFS, MPI, Oracle >Parallel Server, DB2 extended edition, and which is open to future >applications. Focus is how to consolidate the protocol and technology used to deliver storage, IPC, etc. across IP-based networks. This will provide customers with the lowest total cost of ownership, reduce customer management / physical component requirements, reduce vendor development costs, and reduce hardware requirements (e.g. reduce the number of I/O chips / slots required per server / storage endnode). As such, a general purpose RDMA solution which operates over TCP/IP is the optimal solution to pursue since it will lead to the broadest industry and customer adoption rate. Mike From owner-ips@ECE.cmu.edu Tue Sep 19 20:30:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22363; 19 Sep 2000 16:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA19047 for ips-outgoing; Tue, 19 Sep 2000 15:40:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA19022 for ; Tue, 19 Sep 2000 15:39:41 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 19 Sep 2000 15:39:10 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FC0@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI: SCTP, credits, proposals Date: Tue, 19 Sep 2000 15:39:09 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk About two weeks ago, I proposed: > - Recognize the merits of SCTP as well as TCP, plan for both > with the anticipation that TCP will be used first. Aside from a concern that this not hold up specification of iSCSI over TCP, I have seen no objection to this proposal, and hence instruct the WG that we should proceed in this fashion. What this means is that the layering boundary between iSCSI and its transport is significant, and while iSCSI must work well over TCP, we need to design it in a fashion that assumes it will eventually operate over SCTP and hence does not do things that are fundamentally at odds with SCTP. Fortunately, we have an SCTP expert on this list (Randall Stewart) who should be able to flag things that are at odds with SCTP in a timely fashion. The recent consensus to remove the command numbers and sliding windows from the standard header works well with SCTP, as use of multiple SCTP connections in a single iSCSI session seems rather unlikely. The thread ending in the proposal of BB and EE credits was an unfortunate use of list bandwidth. As was stated toward the end of that thread, these are flow control mechanisms that do not implement congestion control. I strongly suggest that everyone who has not already done so read RFC 2914, which is the RFC version of the floyd draft on congestion that I've referred to in the past. The next person who leads us down the rathole of asserting that flow control mechanisms solve congestion control problems should not expect a polite and accommodating response. Credits could form the basis of a congestion control mechanism, but one should expect to invest at least a year, probably longer in design and prototyping to get it to pass muster with the co-chairs, ADs, and IESG. There are a number of "proposals" that have surfaced recently. Due to list traffic, emails get swamped in the archive quickly (which message had that proposal on X)? I strongly encourage those with substantial proposals to write Internet Drafts - this makes it easy to find the document 387 emails later. I realize that writing one's first I-D is a hurdle, but it's worth doing as it makes the proposal much more accessible to the WG. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 00:20:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23881; 19 Sep 2000 20:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA27721 for ips-outgoing; Tue, 19 Sep 2000 19:05:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA27712 for ; Tue, 19 Sep 2000 19:04:57 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id QAA27816; Tue, 19 Sep 2000 16:04:54 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Tue, 19 Sep 2000 16:04:53 -0700 Message-ID: From: Jim McGrath To: "'Randall Stewart'" , Jim McGrath Cc: "'Y P Cheng'" , "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Tue, 19 Sep 2000 16:04:50 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Actually the burden of proof issues is why I suggest we look at some things that are actually being used today (since you don't have to guess how they behave). That is one of TCP's great strengths, and a bit of a weakness for SCTP (no offense to SCTP supporters, but it certainly does not have a big and long "track record" yet, and so I can understand the concerns others may have as to whether things would work out as well in practice as they do in proposal). Jim PS Personally, I'm a big believer is copying stuff that works, making the minimum amount of required changes, and then doing a rapid but controlled deployment (I've been involved in a lot of those sorts of things in ATA and SCSI). Having been involved in both these sorts of endevors and the opposite (big, clean sheet of paper efforts, like 1394 (no offense to 1394/Firewire supporters, but I was working on it a decade ago)), I know how easy it is to underestimate the work required by the latter, and to be turned off by the "inelegance" of the former. For me, life has become too short - I'm willing to accept inelegance as the price for speed of deployment. -----Original Message----- From: Randall Stewart [mailto:rrs@cisco.com] Sent: Tuesday, September 19, 2000 4:38 AM To: Jim McGrath Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' Subject: Re: A Transport Protocol Without ACK Jim: Any transport protocol proposal is ok. As long as it can be seen and reviewed. So far I have seen only two TCP and SCTP. Oh, a little side note, any transport protocol proposed MUST be able to show TCP like behavior in the face of congestion. And I think, IMHO, that this means that if it is NOT using RFC2581 procedures it MUST show that it does backoff and share with TCP. It also has a HEAVY burden of proof to show this facility at least in my mind and I would think in the IESG's mind as well... R Jim McGrath wrote: > I would expand your search to include non standard protocols (i.e. > proprietary ones) as well if they offered something and were adequately > understood by the outside world. We do that in storage quite a lot - > indeed, some standard protocols are direct descendants of what were once > proprietary protocols (e.g. ATA, the most widely used desktop disk > interface, and ESCON, a dominant mainframe class interface (both of which > originated from IBM proprietary technologies)). > > Jim > > -----Original Message----- > From: Y P Cheng [mailto:ycheng@advansys.com] > Sent: Monday, September 18, 2000 5:52 PM > To: 'Ips@Ece. Cmu. Edu' > Subject: RE: A Transport Protocol Without ACK > > From: randall@stewart.chicago.il.us > > I see no viable transport protocol here and I don't see this > > conversation of any use unless you get exact details AND point > > to a internet draft that defines EXACTLY how it works (or possibly > > some other standards document). > > Both I2O and VI are transport protocols which define the format of a request > to a transport service provider, i.e. an adapter card. I2O is used but not > limited to deliver SCSI requests and VI is used for any payload including IP > packets. VI is mapped into FC with the device headers between the FC header > and data payload. VI can certainly be used for delivery of SCSI requests > too. Both protocols require the service provider to have reliable delivery > and reception. VI defines different QoS. > > > > I don't claim any credit about this transport layer protocol. Every > fibre > > > channel and Infiniband adapter designer knows about this protocol -- > > > although there is no standard. I am sure the TCP accelerator card is > doing > > > the same. This protocol is a great alternative to the use of TCP/IP and > > > should be incorporated into iSCSI. > > > > No it is not. You are not offering an alternative yet.. > > I did not imply iSCSI should use I2O or VI. In fact, the purpose iSCSI is > to map SCSI requests into IP packets as well as to define the delivery . It > seems to me that the working group has set its mind on TCP/IP and is > believing this is the only solution. The consensus seems if there is any > other solutions that address flow control and congestion, it would end up > like TCP/IP. I am simply pointing out if we keep an iSCSI request as a > single atomic transaction without separating it into the > TCP/IP-stream-oriented Writes and Reads that each deals with a single DU, > then, the deadlock problem goes away. While the work group thinks we should > take advantage the flow control and congestion management of TCP/IP, there > are alternatives known as BB-credit and EE-credit management. The fibre > channel adapters make reliable delivery, lost packet detection, and > retransmission without TCP/IP. > > Randall, you are right, I did not spent time to provide the working group a > draft defining such transaction-oriented protocol. All I have provided is > an idea that besides TCP/IP. The designers for SCSI and fibre channel > adapters have solved the head-of-queue blocking, the congestion, and > retransmission problems. The transaction-oriented WRITE-REQUEST and > READ-RESPONSE, in my humble opinion, allows us to implement iSCSI simpler > than that of WRITE and READ stream requests. The performance cost of > requiring ACKs on every DU with size greater than MTU on a network with long > latency is very expensive.. By defining a greater ACK granularity is an > attempt to solve this performance problem. If we do wish to ACK on every > DU, then, on a long latency network, we must have a method to stream the > PDUs to ensure the performance. The method should not consume a large > amount of memory space. One should never ignore the TCP/IP memory-to-memory > copy overhead when the backbone will be running at OC-192 speed in the near > future. Finally, please don't ever ask two NIC cards to synchronize with > each other. It is really hard to do as those of us in business of designing > NIC cards can testify. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Wed Sep 20 00:20:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23883; 19 Sep 2000 20:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA27000 for ips-outgoing; Tue, 19 Sep 2000 18:42:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA26992 for ; Tue, 19 Sep 2000 18:42:24 -0400 (EDT) Received: from westrelay01.boulder.ibm.com (westrelay01.boulder.ibm.com [9.99.132.204]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA39720; Tue, 19 Sep 2000 18:40:42 -0400 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay01.boulder.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id QAA156446; Tue, 19 Sep 2000 16:42:22 -0600 Importance: Normal Subject: Re: LU access through an iSCSI session To: Pierre Labat Cc: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Tue, 19 Sep 2000 15:42:21 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Release 5.0.4a |July 24, 2000) at 09/19/2000 03:42:22 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, The answer to your questions (in my mind) are: > 1) Do commands directed to a particular LU through a session are > able to use any of the TCP connections within the session? I think this should be "Yes" and is a very desirable design point (but I don't know if that's what we're going to get). > 2) Do commands directed to a particular LU through a session > need to be associated to different LUNs depending on the TCP > connection used or is it always the same LUN for all > the connections? This has a more complicated answer, namely: *it depends*. Vendors can implement what they like, so in some of today's FC devices, the answer is that you might get different LUNs on different ports. Though, in that case, it's not clear if different ports are conceptually connected in anything that resembles an initiator/target session. The "same LUN for all connections" was a hoped-for side effect of the SCSI access controls model, since in that model, LUN Maps are defined per initiator, regardless of port connection. If the initiator in an I_T nexus is coalesced across multiple connections, then you should see this. On the other hand, the model doesn't preclude more complex implementations and other behaviour. Such implementations would not necessarily be contrary to the SCSI standards (they are silent on this point) but would definitely be extra-standard. I agree with you that this behaviour is undesirable for many reasons, but there's nothing to inhibit it except market forces (I think). I don't think IETF through iSCSI can mandate this behavior particularly since T10 through SAM/SPC doesn't. Also, I agree with your assessment of the situation. Jim Hafner Pierre Labat @ece.cmu.edu on 09-19-2000 10:44:06 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: LU access through an iSCSI session Julo, Thinking about what could be an implementation of an iSCSI session, several questions came to my mind for which i was unable to find an answer in the draft. 1) Do commands directed to a particular LU through a session are able to use any of the TCP connections within the session? As the TCP connections could connect to various "service delivery ports" it is not guarantee that a particular LU is visible through all the connections. 2) Do commands directed to a particular LU through a session need to be associated to different LUNs depending on the TCP connection used or is it always the same LUN for all the connections? As the TCP connections could connect to various "service delivery ports" of the target, it seems that the LUN could differ depending on the connection used. In the iSCSI draft in 2.2.1 there is: "The group of TCP connections linking an initiator with a target forms a session (loosely equivalent to a SCSI nexus)." Does it means that the same LUN is used across the TCP connections? Depending on the answer to these questions the failover/load balancing can be handled in different ways: -> If the answer for 1) is "All TCP connections can be used" and the answer for 2) is "Same LUN regardless of the connexion" For fail over and load balancing inside a session, there is no need to work at the (thin) LU granularity, working at the granularity of the TCP connexion is sufficient. For example in case of failed TCP connexion all the traffic for this failed TCP connexion can be directed on the other connections regardless of the LU. It is very quick, straight forward and may add value compared to the legacy solutions that are doing the fail over on a LU basis (one LU command returns an error or timeout, the next commands for this LU will use an alternate path) In the case where a large number (N) of LUs access are multiplexed to a target port, N switch over are needed. The same result could be achieved in one operation (TCP connection switch over). -> If the answer for 1) is "All TCP connections can be used" and the answer for 2) is "different LUN depending on the connexion" We need to work at the LU granularity. For each command the right LUN as to be determined depending on the tcp connection selected. -> If the answer for 1) is "not all TCP connections can be used, it is LU dependent" On top of the previous case extra work, for each LU it must be a list of TCP connections usable that needs to be looked into before routing the command to one TCP connection. It means that some LUs access using a session could survive to a TCP connection failure some other LUs no. Regards, Pierre From owner-ips@ECE.cmu.edu Wed Sep 20 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25061; 20 Sep 2000 0:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA04930 for ips-outgoing; Tue, 19 Sep 2000 23:05:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA27954 for ; Tue, 19 Sep 2000 19:13:05 -0400 (EDT) Received: from cisco.com (dhcp-71-148-229.cisco.com [171.71.148.229]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id SAA16858; Tue, 19 Sep 2000 18:12:52 -0500 Message-ID: <39C7AB65.72FC418B@cisco.com> Date: Tue, 19 Sep 2000 13:07:34 -0500 From: Randall Stewart X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: ips@ece.cmu.edu Subject: Re: iSCSI: SCTP, credits, proposals References: <0F31E5C394DAD311B60C00E029101A0704100FC0@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com wrote: > Fortunately, we have an SCTP expert on this list > (Randall Stewart) who should be able to flag things > that are at odds with SCTP in a timely fashion. The > recent consensus to remove the command numbers > and sliding windows from the standard header works > well with SCTP, as use of multiple SCTP connections > in a single iSCSI session seems rather unlikely. > David: I will only acknowledge the title of expert in the sense that: a) Ex means former and b) spert is a drip.. In other words, I will own up to being a former drip :) On the serious side of things... So far I see nothing in iSCSI that does not make SCTP a minor drop in replacement for TCP. The only thing is you would not want to put in multiple connections. And since we will be getting a seperate document for multiple connections, may I make a suggestion that this document possibly also take up the use of multiple streams in place of multiple connections when SCTP is used in place of TCP... I will even volunteer to write some text for the document... my plate is emptying out (don't tell anyone :->) and I will have some cycles I can devote to this. I will be glad to help out in whatever small ways are possible for me to apply my limited talents :) > > The thread ending in the proposal of BB and EE credits > was an unfortunate use of list bandwidth. As was stated > toward the end of that thread, these are flow control > mechanisms that do not implement congestion control. > I strongly suggest that everyone who has not already > done so read RFC 2914, which is the RFC version of the > floyd draft on congestion that I've referred to in the past. > The next person who leads us down the rathole of asserting > that flow control mechanisms solve congestion control problems > should not expect a polite and accommodating response. > Credits could form the basis of a congestion control > mechanism, but one should expect to invest at least a > year, probably longer in design and prototyping to get it > to pass muster with the co-chairs, ADs, and IESG. > Yes, I think we have poured enough concrete in that Rat-Hole and we have a consenses :) R > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25056; 20 Sep 2000 0:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA04985 for ips-outgoing; Tue, 19 Sep 2000 23:07:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel1.hp.com (cosrel1.hp.com [156.153.255.170]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA02762 for ; Tue, 19 Sep 2000 21:46:36 -0400 (EDT) Received: from omgw5.rsvl.itc.hp.com (omgw5.rsvl.itc.hp.com [15.34.240.65]) by cosrel1.hp.com (Postfix) with ESMTP id 82BB11BF; Tue, 19 Sep 2000 19:46:44 -0600 (MDT) Received: from xboibrg1.boi.hp.com (xboibrg1.boi.hp.com [15.56.8.167]) by omgw5.rsvl.itc.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id TAA22957; Tue, 19 Sep 2000 19:46:30 -0600 (MDT) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2650.21) id ; Tue, 19 Sep 2000 19:46:29 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: Black_David@emc.com, hufferd@us.ibm.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Session Partial Resolution Date: Tue, 19 Sep 2000 19:46:28 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Would it be reasonable to ask for a "show of hands" on these from the participants on this WG on the two issues? I thought that at least regarding 1, there was some consensus building (or at least a number of people were in favor of) need to avoid the queue full condition etc and to account for the increased latencies of IP networks (without leaving it upto T10 to fix our problems). Somesh -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Tuesday, September 19, 2000 12:22 PM To: hufferd@us.ibm.com; Black_David@emc.com Cc: ips@ece.cmu.edu Subject: iSCSI: Session Partial Resolution > Perhaps I am wrong but it seems we are drifting and not focusing on the > search for concurrence regarding Symmetric vs Asymmetric. What is your > view? John's acquiring the prescient talent of causing the co-chair to appear and take action ... as long as he doesn't start calling himself a "wizard" based on these summoning powers, I won't object :-). In any case ... In the past week, I have seen at most one objection to each of the following two proposed points of consensus: (1) An iSCSI session containing a single TCP connection should not be required to use the currently specified iSCSI command reference numbers and sliding window mechanism because TCP will deliver commands in order. (2) Use of more than one TCP connection per iSCSI session is OPTIONAL. Therefore I declare these to be the WG rough consensus on these issues, and the next version of the iSCSI draft should remove the command reference numbers and sliding window mechanism from the iSCSI header. Somesh Gupta's objection to (1) and Matt Wakeley's continued objection to (2) are noted as part of declaring these to be the WG rough consensus. Anyone else who objects to this declaration of rough consensus should email me directly with the reasons for the objection. OTOH, I do not see consensus on the session model for multiple connection sessions among the Asymmetric model, the Symmetric model, and Pierre Labat's proposal. In order to make progress on iSCSI, I see no alternative to separating multi-connection sessions from the main iSCSI spec. Significant effort and email traffic has been invested in this topic for at least 6 weeks and the issue is not settled -- I don't think holding up the iSCSI spec for another 6+ weeks in hopes of settling this issue on the list is an effective way to make progress, but I'm prepared to listen to dissenting opinions (e.g., if someone thinks there is rough consensus, and I've missed it); please send such opinions directly to me rather than using the list. I've already had one offline comment from an outside observer expressing amazement at the willingness of this community to discuss multi-connection sessions "ad nauseum". Therefore, I would ask that the authors of the next version of the iSCSI draft delete all specification of multiple connection sessions from the next version of the except for a note that they will be handled in a separate document. Producing that separate document is going to require an offline design team. The design team can either be chartered to write a compromise session specification or to evaluate competing specifications and choose one. My current inclination is to do the latter, which would involve having the design team produce a set of requirements and guidelines for session specifications in consultation with the co-chairs, evaluate Internet-Drafts documenting the specifications, and recommend an approach to the WG. Comments on this process are solicited - either on the list or to me directly. Further discussion of multi-connection sessions on the list is probably not a good use of list bandwidth. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26178; 20 Sep 2000 4:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA10963 for ips-outgoing; Wed, 20 Sep 2000 03:17:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA10959 for ; Wed, 20 Sep 2000 03:17:16 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id C71C84B7 for ; Wed, 20 Sep 2000 03:17:15 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id AAA06450 for ; Wed, 20 Sep 2000 00:17:14 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA2694 for ; Wed, 20 Sep 2000 00:17:10 -0700 Message-ID: <39C8646A.BFAF55FB@agilent.com> Date: Wed, 20 Sep 2000 00:16:58 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: LU access through an iSCSI session References: <39C7A5E6.B8E703CC@hp.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre Labat wrote: > Julo, > > Thinking about what could be an implementation of an iSCSI > session, several questions came to my mind for which > i was unable to find an answer in the draft. > > 1) Do commands directed to a particular LU through a session are > able to use any of the TCP connections within the session? > > As the TCP connections could connect to various "service delivery ports" > it is not guarantee that a particular LU is visible through all the > connections. The multiple TCP connections are simply "lanes" that make up the session. The "session" makes the single delivery port. The "session" combines the lanes to "look" like a single wire. -Matt From owner-ips@ECE.cmu.edu Wed Sep 20 08:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26208; 20 Sep 2000 4:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA10766 for ips-outgoing; Wed, 20 Sep 2000 03:09:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA10760 for ; Wed, 20 Sep 2000 03:09:23 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id E3CA8570 for ; Wed, 20 Sep 2000 03:09:22 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id AAA06370 for ; Wed, 20 Sep 2000 00:09:21 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA2642 for ; Wed, 20 Sep 2000 00:09:18 -0700 Message-ID: <39C86291.D9AC868D@agilent.com> Date: Wed, 20 Sep 2000 00:09:05 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: A Transport Protocol Without ACK References: <000d01c021d3$d77a33a0$c1fb6420@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P Cheng wrote: > The fibre > channel adapters make reliable delivery, lost packet detection, and > retransmission without TCP/IP. No, they don't. That is the fallacy of fibre channel. Especially class 2 is thought to be "reliable". It's simply not true. The FCP-2 group has been working on this for a couple years now, and it's still not finished. Problems really arise when trying to address "out of order" frame delivery. Class 3 (the most common implementation of FC) is "connectionless" - you ship a frame and hope it gets there. Class 2 (with its ACKs) over advertises EE credit to each node that connects to it. It ultimately falls back to BB credit, which causes a pile up of frames from the receiving node all the way through the fabric. And with this model, you only know that the frame got to where it's going. If you don't get the ACK back, you don't know if it got there or not. And there is no mechanism to request retransmission of a particular frame. -Matt From owner-ips@ECE.cmu.edu Wed Sep 20 08:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26217; 20 Sep 2000 4:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA11230 for ips-outgoing; Wed, 20 Sep 2000 03:31:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA11225 for ; Wed, 20 Sep 2000 03:31:07 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id B5681408 for ; Wed, 20 Sep 2000 03:31:06 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id AAA06542 for ; Wed, 20 Sep 2000 00:31:05 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA26CD for ; Wed, 20 Sep 2000 00:31:03 -0700 Message-ID: <39C867AA.F9CA5841@agilent.com> Date: Wed, 20 Sep 2000 00:30:50 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: LU access through an iSCSI session References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim Hafner/Almaden/IBM wrote: > Pierre, > > The answer to your questions (in my mind) are: > > > 1) Do commands directed to a particular LU through a session are > > able to use any of the TCP connections within the session? > > I think this should be "Yes" and is a very desirable design point (but I > don't know if that's what we're going to get). As I said in a previous reply, the *only* answer is "Yes", which leads to... > > > > 2) Do commands directed to a particular LU through a session > > need to be associated to different LUNs depending on the TCP > > connection used or is it always the same LUN for all > > the connections? > > This has a more complicated answer, namely: *it depends*. Again, the *only* answer is "Yes". The multiple TCP connections make up a *single* session. The session just happens to have lots of "TCP connections" to make it look like it has a "big pipe" (for example, 4 1Gbit connections can be used to emulate a 4Gbit link). If you want different "LU views" per TCP connection, then each TCP connection would be a separate, independent iscsi session. -Matt > > Vendors can > implement what they like, so in some of today's FC devices, the answer is > that you might get different LUNs on different ports. Though, in that > case, it's not clear if different ports are conceptually connected in > anything that resembles an initiator/target session. The "same LUN for all > connections" was a hoped-for side effect of the SCSI access controls model, > since in that model, LUN Maps are defined per initiator, regardless of port > connection. If the initiator in an I_T nexus is coalesced across multiple > connections, then you should see this. On the other hand, the model > doesn't preclude more complex implementations and other behaviour. Such > implementations would not necessarily be contrary to the SCSI standards > (they are silent on this point) but would definitely be extra-standard. > > I agree with you that this behaviour is undesirable for many reasons, but > there's nothing to inhibit it except market forces (I think). I don't > think IETF through iSCSI can mandate this behavior particularly since T10 > through SAM/SPC doesn't. > > Also, I agree with your assessment of the situation. > > Jim Hafner From owner-ips@ECE.cmu.edu Wed Sep 20 08:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26210; 20 Sep 2000 4:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA10881 for ips-outgoing; Wed, 20 Sep 2000 03:14:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA10877 for ; Wed, 20 Sep 2000 03:14:40 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id AAA11970; Wed, 20 Sep 2000 00:03:03 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id AAA06896; Wed, 20 Sep 2000 00:00:45 -0700 From: "Douglas Otis" To: "Jim McGrath" , "'Randall Stewart'" Cc: "'Y P Cheng'" , "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Wed, 20 Sep 2000 00:17:00 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, I understand a desire to stick with what works. Regardless of the IP transport, the traffic to each LUN will require some other flow control mechanism. Yes, we could re-invent the wheel as it applies to SCSI, but if you examine FC Class 3 FCP, you will see an appropriate flow control mechanism in place. It uses Buffer-to-Buffer credit tokens generated by comma frame delimiters. These frame delimiters are defined within the FC-encapsulation documentation. Simple, direct and easy. See http://search.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-00.txt. To facilitate processing frame delimiters within software, both could be presented before the frame rather than as shown in the rough draft. Perhaps even a Null CRC option could be added if one trusts SCTP checksum for software implementations. Once flow control is in place, there is little need for extending command tags, CRN, or anything associated with the LUN as these structures are then independent of transport bandwidth. I doubt there is a great benefit in having more than 256 commands pending on a nexus. Expanding any field only makes converting to a drive interface state-full, difficult and far less reliable. With the FCP flow control mechanism, T10 does not need to redefine SCSI for initiators that overwhelm the target. The target would have adequate control of resources. Should IP-SCSI be driven by controller design? Caching, volume management, reservations, and nearly every feature offered by a controller is significantly reduced in value should the controller be placed next to the drive. If you are in a facility 35 miles from a location holding drives, you may find 50 miles of fiber transversed creating some 800+ micro-seconds of round-trip time simply due to the speed of light. You may shudder to think about any NIC buffer. Where would you want the controller and where would you want the drive? The controller must remain on the client side of the network. As such, drive design should steer the IP-SCSI standard. At least with FCP, the drive manufacturers have already spoken. Those making controllers will just have to make more of them and develop controller locking protocols should this controller be part of a remote cluster. If you examine FCP documentation, you will find that you can send data with the command as an option. You can also send the response at the end of data as an option. Every vital feature used to justify tossing FCP structures become moot. Should just an 8M byte FIFO buffer be placed between an IP agent and a FC agent, as much as 65 milli-seconds of latency can be created. Merely this additional latency will greatly facilitate rate differences between these two agents. FCP flow control and burst limits could easily finish the task. You speak of TCP as a proven technology, but TCP is not being suggested for IP-SCSI. TCP with some other mechanism is used to solve ills created by a persistent single byte stream. This is not proven technology, nor likely to function properly without major tweaking. At least if you wish to have a hand at creating a suitable API for multi-object-streams far and away more suitable for SCSI, now is the time. Perhaps either Randall Stewart's U-SCTP or a stale frame timer should be added to prevent overlapping retry mechanisms if this protocol is used as a bridge to FC. As far as the configuration effort, convert these requirements into LDAP structures. This would allow a single database to manage all aspects of configuration. Stuffing this information across the transport only weakens security. A bad idea and makes deciding who manages difficult. Networks will always have a means to identify equipment in some binary fashion, and LDAP and DHCP servers combine this information into meaningful structures with meaningful names. All values required for the various transport layers would be derived from these standard servers. As far as what to do with Stream 0- revision negotiations, FC-domain mapping, SRC-DST filtering done in purely binary form would be the best means at getting equipment to accept commands without a high overhead. The equipment does not care what the binary number represents. As far as a clever means of doing remote DNS, SCTP has that covered. Again, this information comes from an LDAP server accessed by the driver and not the SAM interface or the SCSI transport layer. Yes, there are many options within FC that should be excluded. If FCP structures can be used, perhaps while holding one's nose, they should be. There are far too many benefits for doing so, and too few benefits for not. In the end, a better product would have a common set of structures to speak SAN. If you wish to make round wheels out of square blocks, don't let me stop you. I think I see a set of wheels already. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jim McGrath > Sent: Tuesday, September 19, 2000 4:05 PM > To: 'Randall Stewart'; Jim McGrath > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > Subject: RE: A Transport Protocol Without ACK > > > > Actually the burden of proof issues is why I suggest we look at > some things > that are actually being used today (since you don't have to guess how they > behave). That is one of TCP's great strengths, and a bit of a > weakness for > SCTP (no offense to SCTP supporters, but it certainly does not have a big > and long "track record" yet, and so I can understand the concerns > others may > have as to whether things would work out as well in practice as they do in > proposal). > > Jim > > PS Personally, I'm a big believer is copying stuff that works, making the > minimum amount of required changes, and then doing a rapid but controlled > deployment (I've been involved in a lot of those sorts of things > in ATA and > SCSI). Having been involved in both these sorts of endevors and the > opposite (big, clean sheet of paper efforts, like 1394 (no offense to > 1394/Firewire supporters, but I was working on it a decade ago)), > I know how > easy it is to underestimate the work required by the latter, and to be > turned off by the "inelegance" of the former. For me, life has become too > short - I'm willing to accept inelegance as the price for speed of > deployment. > > > -----Original Message----- > From: Randall Stewart [mailto:rrs@cisco.com] > Sent: Tuesday, September 19, 2000 4:38 AM > To: Jim McGrath > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > Subject: Re: A Transport Protocol Without ACK > > > Jim: > > Any transport protocol proposal is ok. As long as it can be seen and > reviewed. So far I have seen only two TCP and SCTP. > > Oh, a little side note, any transport protocol proposed MUST be able to > show TCP like behavior in the face of congestion. And I think, IMHO, that > this means that if it is NOT using RFC2581 procedures it MUST show that > it does backoff and share with TCP. It also has a HEAVY burden of proof to > show this facility at least in my mind and I would think in the > IESG's mind > as well... > > R > > > Jim McGrath wrote: > > > I would expand your search to include non standard protocols (i.e. > > proprietary ones) as well if they offered something and were adequately > > understood by the outside world. We do that in storage quite a lot - > > indeed, some standard protocols are direct descendants of what were once > > proprietary protocols (e.g. ATA, the most widely used desktop disk > > interface, and ESCON, a dominant mainframe class interface > (both of which > > originated from IBM proprietary technologies)). > > > > Jim > > > > -----Original Message----- > > From: Y P Cheng [mailto:ycheng@advansys.com] > > Sent: Monday, September 18, 2000 5:52 PM > > To: 'Ips@Ece. Cmu. Edu' > > Subject: RE: A Transport Protocol Without ACK > > > > From: randall@stewart.chicago.il.us > > > I see no viable transport protocol here and I don't see this > > > conversation of any use unless you get exact details AND point > > > to a internet draft that defines EXACTLY how it works (or possibly > > > some other standards document). > > > > Both I2O and VI are transport protocols which define the format of a > request > > to a transport service provider, i.e. an adapter card. I2O is used but > not > > limited to deliver SCSI requests and VI is used for any payload > including > IP > > packets. VI is mapped into FC with the device headers between the FC > header > > and data payload. VI can certainly be used for delivery of > SCSI requests > > too. Both protocols require the service provider to have reliable > delivery > > and reception. VI defines different QoS. > > > > > > I don't claim any credit about this transport layer protocol. Every > > fibre > > > > channel and Infiniband adapter designer knows about this protocol -- > > > > although there is no standard. I am sure the TCP > accelerator card is > > doing > > > > the same. This protocol is a great alternative to the use of TCP/IP > and > > > > should be incorporated into iSCSI. > > > > > > No it is not. You are not offering an alternative yet.. > > > > I did not imply iSCSI should use I2O or VI. In fact, the > purpose iSCSI is > > to map SCSI requests into IP packets as well as to define the delivery . > It > > seems to me that the working group has set its mind on TCP/IP and is > > believing this is the only solution. The consensus seems if > there is any > > other solutions that address flow control and congestion, it > would end up > > like TCP/IP. I am simply pointing out if we keep an iSCSI request as a > > single atomic transaction without separating it into the > > TCP/IP-stream-oriented Writes and Reads that each deals with a > single DU, > > then, the deadlock problem goes away. While the work group thinks we > should > > take advantage the flow control and congestion management of > TCP/IP, there > > are alternatives known as BB-credit and EE-credit management. The fibre > > channel adapters make reliable delivery, lost packet detection, and > > retransmission without TCP/IP. > > > > Randall, you are right, I did not spent time to provide the > working group > a > > draft defining such transaction-oriented protocol. All I have > provided is > > an idea that besides TCP/IP. The designers for SCSI and fibre channel > > adapters have solved the head-of-queue blocking, the congestion, and > > retransmission problems. The transaction-oriented WRITE-REQUEST and > > READ-RESPONSE, in my humble opinion, allows us to implement > iSCSI simpler > > than that of WRITE and READ stream requests. The performance cost of > > requiring ACKs on every DU with size greater than MTU on a network with > long > > latency is very expensive.. By defining a greater ACK granularity is an > > attempt to solve this performance problem. If we do wish to > ACK on every > > DU, then, on a long latency network, we must have a method to stream the > > PDUs to ensure the performance. The method should not consume a large > > amount of memory space. One should never ignore the TCP/IP > memory-to-memory > > copy overhead when the backbone will be running at OC-192 speed in the > near > > future. Finally, please don't ever ask two NIC cards to > synchronize with > > each other. It is really hard to do as those of us in business of > designing > > NIC cards can testify. > > > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. > From owner-ips@ECE.cmu.edu Wed Sep 20 09:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26636; 20 Sep 2000 5:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA12235 for ips-outgoing; Wed, 20 Sep 2000 04:16:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA12230 for ; Wed, 20 Sep 2000 04:16:17 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA64680; Wed, 20 Sep 2000 10:15:40 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA21716; Wed, 20 Sep 2000 10:15:40 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256960.002D5F29 ; Wed, 20 Sep 2000 10:15:34 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Pierre Labat , ips@ece.cmu.edu Message-ID: Date: Wed, 20 Sep 2000 11:06:28 +0300 Subject: Re: LU access through an iSCSI session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, All accessible LUs may be accessed through all TCP connections of a single session. The session is conceptually the "service delivery port." LU numbers are identical across all of the TCP connections of a single iSCSI session. Each iSCSI SCSI Command PDU must specify the LUN to which it is referring, and it doesn't matter across which TCP connection (of the same session) it is sent. - Kalman Meth Pierre Labat on 19/09/2000 20:44:06 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: (bcc: Kalman Meth/Haifa/IBM) Subject: LU access through an iSCSI session Julo, Thinking about what could be an implementation of an iSCSI session, several questions came to my mind for which i was unable to find an answer in the draft. 1) Do commands directed to a particular LU through a session are able to use any of the TCP connections within the session? As the TCP connections could connect to various "service delivery ports" it is not guarantee that a particular LU is visible through all the connections. 2) Do commands directed to a particular LU through a session need to be associated to different LUNs depending on the TCP connection used or is it always the same LUN for all the connections? <... deleted ...> Regards, Pierre From owner-ips@ECE.cmu.edu Wed Sep 20 10:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26837; 20 Sep 2000 6:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA13275 for ips-outgoing; Wed, 20 Sep 2000 05:05:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA13270 for ; Wed, 20 Sep 2000 05:05:29 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 6CC106C4 for ; Wed, 20 Sep 2000 05:05:28 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id CAA13980 for ; Wed, 20 Sep 2000 02:05:27 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA27D1; Wed, 20 Sep 2000 02:05:21 -0700 Message-ID: <39C87DC4.D26366F4@agilent.com> Date: Wed, 20 Sep 2000 02:05:08 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com Cc: hufferd@us.ibm.com, ips@ece.cmu.edu Subject: Re: iSCSI: Session Partial Resolution References: <0F31E5C394DAD311B60C00E029101A0704100FBE@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com wrote: > OTOH, I do not see consensus on the session model for > multiple connection sessions among the Asymmetric model, > the Symmetric model, and Pierre Labat's proposal. In > order to make progress on iSCSI, I see no alternative to > separating multi-connection sessions from the main iSCSI > spec. Significant effort and email traffic has been > invested in this topic for at least 6 weeks and the issue > is not settled -- I don't think holding up the iSCSI spec > for another 6+ weeks in hopes of settling this issue on > the list is an effective way to make progress, but I'm > prepared to listen to dissenting opinions (e.g., if > someone thinks there is rough consensus, and I've missed > it); please send such opinions directly to me rather than > using the list. I've already had one offline comment from > an outside observer expressing amazement at the willingness > of this community to discuss multi-connection sessions > "ad nauseum". > > Therefore, I would ask that the authors of the next > version of the iSCSI draft delete all specification > of multiple connection sessions from the next version > of the except for a note that they will be handled in > a separate document. The "iSCSI Requirements" document requires this capability: "[R] High bandwidth, bandwidth aggregation. [D] The bandwidth (transfer rate, MB/sec) supported by storage controllers is rapidly increasing, due to several factors: (1) Increase in disk spindle and controller performance; (2) Use of ever-larger caches, and improved caching algorithms; (3) Increased scale of storage controllers (number of supported spindles, speed of interconnects). Not only must the iSCSI provide for full utilization of available link bandwidth, it also must exploit parallelism (multiple connections) at the device interfaces and within the interconnect fabric." Unless there is "consensus" that this is not required, I think the multiple connections/session must remain. If nothing else, at least the "hooks" should remain in the document (such as for login, etc) so as to minimize incompatibilities with future versions of the specification. The mechanisms to "bundle" tcp connections into a session are the same regardless of whether they're used "synchronously" or "asynchronously". Also, it's much easier to ignore unimplemented fields in a header than to add fields after the fact. -Matt Wakeley Agilent Technologies > > > Producing that separate document is going to require > an offline design team. The design team can either be > chartered to write a compromise session specification > or to evaluate competing specifications and choose one. > My current inclination is to do the latter, which would > involve having the design team produce a set of requirements > and guidelines for session specifications in consultation > with the co-chairs, evaluate Internet-Drafts documenting > the specifications, and recommend an approach to the WG. > Comments on this process are solicited - either on the > list or to me directly. Further discussion of multi-connection > sessions on the list is probably not a good use of list > bandwidth. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 10:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26844; 20 Sep 2000 6:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA13803 for ips-outgoing; Wed, 20 Sep 2000 05:28:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA13799 for ; Wed, 20 Sep 2000 05:28:45 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel3.hp.com (Postfix) with ESMTP id BDEEF94D for ; Wed, 20 Sep 2000 02:28:44 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id CAA14112 for ; Wed, 20 Sep 2000 02:28:43 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA2812 for ; Wed, 20 Sep 2000 02:28:40 -0700 Message-ID: <39C8833C.33572620@agilent.com> Date: Wed, 20 Sep 2000 02:28:28 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: ISCSI: flow control Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Well, David thinks there's consensus on flow control, but I just wanted to make sure some issues are clear: > (1) An iSCSI session containing a single TCP connection > should not be required to use the currently specified > iSCSI command reference numbers and sliding window > mechanism because TCP will deliver commands in order. > The iSCSI spec originally implemented command reference numbers to "order" commands that are delivered across multiple TCP connections. The sliding window was added so that if a tcp connection dies, the remaining tcp connections don't flood the target with (now potentially) out of order commands. It's true that an iSCSI session containing a single TCP connection, or an iSCSI session containing a multiple TCP connections but using the "asynchronous" model does not require command re-ordering or the sliding window. Only the "synchronous" model with multiple TCP connections requires both. However, some have wanted to use the sliding window to solve a different problem. They want to enable the target to advertise to the initiator how many commands the target is able to receive. This apparently is not specified in SAM-2, and SAM-2 deals with overflowing the command queue in an ugly way - throw the commands away and enter an error state. The question here is whether iSCSI should attempt to solve the command queue overflow problem, or allow T10 to deal with it. Other protocols (fibre channel) already deal with it in the "ugly" fashion - perhaps so should iSCSI. What we need consensus on is whether or not iSCSI is going to deal with this. -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Wed Sep 20 11:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27103; 20 Sep 2000 6:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA15201 for ips-outgoing; Wed, 20 Sep 2000 06:32:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA15197 for ; Wed, 20 Sep 2000 06:32:28 -0400 (EDT) Received: from centralmail1.Central.Sun.COM ([129.147.62.10]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id EAA10092; Wed, 20 Sep 2000 04:32:23 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail1.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id EAA14596; Wed, 20 Sep 2000 04:32:22 -0600 (MDT) Received: from sun.com (hobo26.Central.Sun.COM [129.147.8.26]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id EAA27941; Wed, 20 Sep 2000 04:32:05 -0600 (MDT) Message-ID: <39C892D9.61E12D9C@sun.com> Date: Wed, 20 Sep 2000 04:35:05 -0600 From: "Mark A. Carlson" Reply-To: mark.carlson@sun.com Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.61 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: meth@il.ibm.com CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: iSCSI: Session Partial Resolution References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Support for multiple sessions has always been optional. So lets get a specification out there that shows how to do iSCSI when you don't support multiple sessions. The only other thing we need specify then, is some way to say that the implementation does support multiple connections, and then that work can proceed in another document on another time schedule and companies that were not going to implement multiple sessions initially anyway can start work. If you like, this can be the spec for the degenerate single connection symmetric model since I think there is consensus that a single connection cannot be asymmetric. David is just trying to move us along and eliminate possible rat holes that will hold up the end result. -- mark meth@il.ibm.com wrote: > > David, > > If we eliminate all support for multiple connections from the next version > of the draft and have a separate draft for multiple connections, then we'll > eventually end up with 2 different (incompatible) protocols. Depending on > whether we have a single connection or multiple connections, the initiator > and target will have to implement one or the other or both protocols. Some > products that implement only one version will not be compatible with other > products, etc, and we will have shot ourselves in the foot. The wide > acceptance of iSCSI will be strongly influenced by the ability to > inter-operate with all kinds of devices and products, many of which will > greatly benefit from one of the various multiple-connection models. > > I recommend to try to focus on the question of the multiple-connection > model to see if we can at least agree that one or more of them sufficiently > satisfies the requirements, and then choose one of the satisfactory models. > Even if we can't agree on which model is the best, I think we can > more-or-less agree on which of the models is at least good enough. > > - Kalman Meth > > Black_David@emc.com on 19/09/2000 22:22:20 > > Please respond to Black_David@emc.com > > To: John Hufferd/San Jose/IBM@IBMUS, Black_David@emc.com > cc: ips@ece.cmu.edu (bcc: Kalman Meth/Haifa/IBM) > Subject: iSCSI: Session Partial Resolution > > <... deleted ...> > > OTOH, I do not see consensus on the session model for > multiple connection sessions among the Asymmetric model, > the Symmetric model, and Pierre Labat's proposal. In > order to make progress on iSCSI, I see no alternative to > separating multi-connection sessions from the main iSCSI > spec. Significant effort and email traffic has been > invested in this topic for at least 6 weeks and the issue > is not settled -- I don't think holding up the iSCSI spec > for another 6+ weeks in hopes of settling this issue on > the list is an effective way to make progress, but I'm > prepared to listen to dissenting opinions (e.g., if > someone thinks there is rough consensus, and I've missed > it); please send such opinions directly to me rather than > using the list. I've already had one offline comment from > an outside observer expressing amazement at the willingness > of this community to discuss multi-connection sessions > "ad nauseum". > > Therefore, I would ask that the authors of the next > version of the iSCSI draft delete all specification > of multiple connection sessions from the next version > of the except for a note that they will be handled in > a separate document. > > <... deleted ...> > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 11:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27093; 20 Sep 2000 6:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA14535 for ips-outgoing; Wed, 20 Sep 2000 06:02:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA14531 for ; Wed, 20 Sep 2000 06:02:40 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id MAA01934; Wed, 20 Sep 2000 12:02:09 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA61314; Wed, 20 Sep 2000 12:01:59 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256960.00371C0A ; Wed, 20 Sep 2000 12:01:56 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com, ips@ece.cmu.edu Message-ID: Date: Wed, 20 Sep 2000 13:01:48 +0300 Subject: Re: iSCSI: Session Partial Resolution Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, If we eliminate all support for multiple connections from the next version of the draft and have a separate draft for multiple connections, then we'll eventually end up with 2 different (incompatible) protocols. Depending on whether we have a single connection or multiple connections, the initiator and target will have to implement one or the other or both protocols. Some products that implement only one version will not be compatible with other products, etc, and we will have shot ourselves in the foot. The wide acceptance of iSCSI will be strongly influenced by the ability to inter-operate with all kinds of devices and products, many of which will greatly benefit from one of the various multiple-connection models. I recommend to try to focus on the question of the multiple-connection model to see if we can at least agree that one or more of them sufficiently satisfies the requirements, and then choose one of the satisfactory models. Even if we can't agree on which model is the best, I think we can more-or-less agree on which of the models is at least good enough. - Kalman Meth Black_David@emc.com on 19/09/2000 22:22:20 Please respond to Black_David@emc.com To: John Hufferd/San Jose/IBM@IBMUS, Black_David@emc.com cc: ips@ece.cmu.edu (bcc: Kalman Meth/Haifa/IBM) Subject: iSCSI: Session Partial Resolution <... deleted ...> OTOH, I do not see consensus on the session model for multiple connection sessions among the Asymmetric model, the Symmetric model, and Pierre Labat's proposal. In order to make progress on iSCSI, I see no alternative to separating multi-connection sessions from the main iSCSI spec. Significant effort and email traffic has been invested in this topic for at least 6 weeks and the issue is not settled -- I don't think holding up the iSCSI spec for another 6+ weeks in hopes of settling this issue on the list is an effective way to make progress, but I'm prepared to listen to dissenting opinions (e.g., if someone thinks there is rough consensus, and I've missed it); please send such opinions directly to me rather than using the list. I've already had one offline comment from an outside observer expressing amazement at the willingness of this community to discuss multi-connection sessions "ad nauseum". Therefore, I would ask that the authors of the next version of the iSCSI draft delete all specification of multiple connection sessions from the next version of the except for a note that they will be handled in a separate document. <... deleted ...> --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 11:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27098; 20 Sep 2000 6:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA14866 for ips-outgoing; Wed, 20 Sep 2000 06:16:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA14862 for ; Wed, 20 Sep 2000 06:16:29 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id 44080189 for ; Wed, 20 Sep 2000 03:16:16 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id DAA14403 for ; Wed, 20 Sep 2000 03:16:15 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA2895 for ; Wed, 20 Sep 2000 03:16:11 -0700 Message-ID: <39C88E59.398616EE@agilent.com> Date: Wed, 20 Sep 2000 03:15:53 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: ISCSI: asymmetric vs symmetric and multiple connections Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David thinks we have come to "consensus". I'm not so sure. I would like to summarize what the issues are with symmetric vs asymmetric and multiple tcp connections. definitions: Symmetric model means commands and their associated data flow down any of the TCP connections that make up a session. Asymmetric model means that all commands flow on one TCP connection, and all data flows on the remaining TCP connections that make up the session. Note that with the above definitions, an iSCSI session with a single tcp connection is by definition a symmetric model. Asymmetric model pros: - does not require a "command reordering" mechanism. - commands (relatively small messages) are generally not flow controlled by TCP like they would be if multiplexed with large data transfers. - may be easier to implement in hardware - no multiplexing of commands and data on the same TCP connection. Asymmetric model cons: - requires a minimum of two TCP connections: one for commands, one for data. - iSCSI sessions using multiple TCP connections across different iSCSI NICs to provide aggregation of links requires double the control and setup on the processor I/O bus to perform a SCSI I/O. This is because one NIC must be set up to send the command, the other for the data. This would not be and issue for an iSCSI NIC that had multiple MACs or connections to the fabric. - may make the implementation of "immediate data" on targets difficult (the data could arrive before the command). - may require more TCP context storage due to a minimum of two TCP connections per session (but not nearly as much as those advocating a connection per LU!). Symmetric model pros: - the "double control overhead" of the asymmetric model does not apply. - all TCP connections are the same - may be easier to implement in hardware because there are no "special" connections. - implementation of immediate data easier on targets - data always follows command on same TCP connection. Symmetric model cons: - requires command reference numbers to re-order commands on multi TCP connection sessions (could be optional for a single TCP connection iSCSI session). - commands may be stalled behind data due to TCP flow control. - sliding window for commands implemented only to address the case where a TCP connection dies. (not to address the SCSI command queue overflow issue.) -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Wed Sep 20 11:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27176; 20 Sep 2000 7:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA15026 for ips-outgoing; Wed, 20 Sep 2000 06:24:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA15022 for ; Wed, 20 Sep 2000 06:24:17 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8KAOCU04689; Wed, 20 Sep 2000 05:24:12 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA04290; Wed, 20 Sep 00 05:22:29 -0500 Message-Id: <10009201022.AA04290@candide.cs.uchicago.edu> To: bassoon@yogi.ece.cmu.edu, ips@ece.cmu.edu Subject: Re: another question (was Re: patent question) In-Reply-To: Message from Black_David@emc.com of "Thu, 14 Sep 2000 00:32:42 -0400." <0F31E5C394DAD311B60C00E029101A0704100F83@corpmx9.isus.emc.com> Date: Wed, 20 Sep 2000 19:23:19 +0900 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > > i have another quick question. for a local area application, > > why hasn't anyone done scsi over ethernet? (ie encapsulating > > scsi commands into the ethernet package). why use a higher > > level protocol and work with all the latency and overhead problem? > > Check the list archives for info on an Adaptec demo of this, and I'm > sure Paul von Stamwitz can provide additional information (off the > list, please). Encapsulating directly on Ethernet runs into scaling > problems - the first time one needs to get off the LAN or VLAN that > the traffic started on, one discovers that an IP header is an immensely > useful thing to have. I can add that SCSI on ST (SST) is another implemented form of SCSI on Ethernet with existing, hardware accelerated implementations. See www.genroco.com for more information. What David says is correct. You CAN build SANs this way, but you must careful control the configuration, which is another reason why scalability is limited. In some cases, having an Ethernet SAN which requires this careful configuration control is still better than needing two network attachment infrastructures (Ethernet & FCP). Steph From owner-ips@ECE.cmu.edu Wed Sep 20 13:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27655; 20 Sep 2000 9:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA17605 for ips-outgoing; Wed, 20 Sep 2000 08:15:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA17601 for ; Wed, 20 Sep 2000 08:15:04 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id IAA68344 for ; Wed, 20 Sep 2000 08:09:01 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8KCF3003640 for ; Wed, 20 Sep 2000 06:15:03 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: Session Partial Resolution To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 20 Sep 2000 05:10:31 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/20/2000 06:15:03 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I had thought that we had "agreed" (perhaps an overstatement) that a two connection per session (Asymmetric Session) solved a number of problems, one of which was brought up by Costa. This would still fit the other stuff you said. Note: I did not say that we had consensus of using Asymmetric vs Symmetric, just that an Asymmetric Session (of at least two connections) solved some potential problems. Also this did not say anything about having the double Connections within a Session being on different NICs. The value of at least two connections seemed to have value (solved certain problems) even within a single NIC. . . . John L. Hufferd meth@il.ibm.com@ece.cmu.edu on 09/20/2000 03:01:48 AM Sent by: owner-ips@ece.cmu.edu To: Black_David@emc.com, ips@ece.cmu.edu cc: Subject: Re: iSCSI: Session Partial Resolution David, If we eliminate all support for multiple connections from the next version of the draft and have a separate draft for multiple connections, then we'll eventually end up with 2 different (incompatible) protocols. Depending on whether we have a single connection or multiple connections, the initiator and target will have to implement one or the other or both protocols. Some products that implement only one version will not be compatible with other products, etc, and we will have shot ourselves in the foot. The wide acceptance of iSCSI will be strongly influenced by the ability to inter-operate with all kinds of devices and products, many of which will greatly benefit from one of the various multiple-connection models. I recommend to try to focus on the question of the multiple-connection model to see if we can at least agree that one or more of them sufficiently satisfies the requirements, and then choose one of the satisfactory models. Even if we can't agree on which model is the best, I think we can more-or-less agree on which of the models is at least good enough. - Kalman Meth Black_David@emc.com on 19/09/2000 22:22:20 Please respond to Black_David@emc.com To: John Hufferd/San Jose/IBM@IBMUS, Black_David@emc.com cc: ips@ece.cmu.edu (bcc: Kalman Meth/Haifa/IBM) Subject: iSCSI: Session Partial Resolution <... deleted ...> OTOH, I do not see consensus on the session model for multiple connection sessions among the Asymmetric model, the Symmetric model, and Pierre Labat's proposal. In order to make progress on iSCSI, I see no alternative to separating multi-connection sessions from the main iSCSI spec. Significant effort and email traffic has been invested in this topic for at least 6 weeks and the issue is not settled -- I don't think holding up the iSCSI spec for another 6+ weeks in hopes of settling this issue on the list is an effective way to make progress, but I'm prepared to listen to dissenting opinions (e.g., if someone thinks there is rough consensus, and I've missed it); please send such opinions directly to me rather than using the list. I've already had one offline comment from an outside observer expressing amazement at the willingness of this community to discuss multi-connection sessions "ad nauseum". Therefore, I would ask that the authors of the next version of the iSCSI draft delete all specification of multiple connection sessions from the next version of the except for a note that they will be handled in a separate document. <... deleted ...> --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 13:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27662; 20 Sep 2000 9:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA17130 for ips-outgoing; Wed, 20 Sep 2000 07:59:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA17126 for ; Wed, 20 Sep 2000 07:59:27 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id HAA70156; Wed, 20 Sep 2000 07:57:45 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8KBxP055826; Wed, 20 Sep 2000 05:59:25 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: Session Partial Resolution To: mark.carlson@sun.com Cc: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 20 Sep 2000 04:56:51 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/20/2000 05:59:25 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, Just for correct communication, you might want to make your comments consistent between the words Session and Connection. They are not the same thing. What was talked about by David was Multiple Connections. The context was Multiple Connections per Session. David did not make any statements with regard to Multiple Sessions. I would think there could be Multiple Sessions between a Host and a Storage Controller (yet still be consistent with David's statements) as long as No overarching iSCSI relationship existed between the commands and data in the different sessions. This is the case with FC today. If there is multiple Sessions, a Wedge Driver can be used to balance or be used to provide for vendor specific alternate path recovery. . . . John L. Hufferd "Mark A. Carlson" @ece.cmu.edu on 09/20/2000 03:35:05 AM Please respond to mark.carlson@sun.com Sent by: owner-ips@ece.cmu.edu To: Kalman Meth/Haifa/IBM@IBMIL cc: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: iSCSI: Session Partial Resolution Support for multiple sessions has always been optional. So lets get a specification out there that shows how to do iSCSI when you don't support multiple sessions. The only other thing we need specify then, is some way to say that the implementation does support multiple connections, and then that work can proceed in another document on another time schedule and companies that were not going to implement multiple sessions initially anyway can start work. If you like, this can be the spec for the degenerate single connection symmetric model since I think there is consensus that a single connection cannot be asymmetric. David is just trying to move us along and eliminate possible rat holes that will hold up the end result. -- mark meth@il.ibm.com wrote: > > David, > > If we eliminate all support for multiple connections from the next version > of the draft and have a separate draft for multiple connections, then we'll > eventually end up with 2 different (incompatible) protocols. Depending on > whether we have a single connection or multiple connections, the initiator > and target will have to implement one or the other or both protocols. Some > products that implement only one version will not be compatible with other > products, etc, and we will have shot ourselves in the foot. The wide > acceptance of iSCSI will be strongly influenced by the ability to > inter-operate with all kinds of devices and products, many of which will > greatly benefit from one of the various multiple-connection models. > > I recommend to try to focus on the question of the multiple-connection > model to see if we can at least agree that one or more of them sufficiently > satisfies the requirements, and then choose one of the satisfactory models. > Even if we can't agree on which model is the best, I think we can > more-or-less agree on which of the models is at least good enough. > > - Kalman Meth > > Black_David@emc.com on 19/09/2000 22:22:20 > > Please respond to Black_David@emc.com > > To: John Hufferd/San Jose/IBM@IBMUS, Black_David@emc.com > cc: ips@ece.cmu.edu (bcc: Kalman Meth/Haifa/IBM) > Subject: iSCSI: Session Partial Resolution > > <... deleted ...> > > OTOH, I do not see consensus on the session model for > multiple connection sessions among the Asymmetric model, > the Symmetric model, and Pierre Labat's proposal. In > order to make progress on iSCSI, I see no alternative to > separating multi-connection sessions from the main iSCSI > spec. Significant effort and email traffic has been > invested in this topic for at least 6 weeks and the issue > is not settled -- I don't think holding up the iSCSI spec > for another 6+ weeks in hopes of settling this issue on > the list is an effective way to make progress, but I'm > prepared to listen to dissenting opinions (e.g., if > someone thinks there is rough consensus, and I've missed > it); please send such opinions directly to me rather than > using the list. I've already had one offline comment from > an outside observer expressing amazement at the willingness > of this community to discuss multi-connection sessions > "ad nauseum". > > Therefore, I would ask that the authors of the next > version of the iSCSI draft delete all specification > of multiple connection sessions from the next version > of the except for a note that they will be handled in > a separate document. > > <... deleted ...> > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28101; 20 Sep 2000 10:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA20469 for ips-outgoing; Wed, 20 Sep 2000 09:37:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA20457 for ; Wed, 20 Sep 2000 09:36:59 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id GAA04831 for ; Wed, 20 Sep 2000 06:36:58 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Wed, 20 Sep 2000 06:36:55 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1C5@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: ips@ece.cmu.edu Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Date: Wed, 20 Sep 2000 06:36:39 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Michael Krause [mailto:krause@cup.hp.com] wrote: > [snip] > RDMA != VI though VI does use RDMA technologies. Agreed. It is an example > Prefer to > see discussion > focused on what RDMA operations are required, what are the error and > ordering requirements, etc. I think RDMA reads and writes to the initiator are the only operations needed. As someone here recently argued in the context of TCP/IP performance, it is the storage clients which most need to offload CPU. I haven't thought through error and ordering requirements, although the notion is of course to allow out-of-order processing of RDMA, given some framing assist from TCP. > [snip] > As such, a general > purpose RDMA > solution which operates over TCP/IP is the optimal solution > to pursue since > it will lead to the broadest industry and customer adoption rate. I think we are in complete agreement. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Wed Sep 20 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28106; 20 Sep 2000 10:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA21766 for ips-outgoing; Wed, 20 Sep 2000 10:06:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA21761 for ; Wed, 20 Sep 2000 10:06:26 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id C72BA109; Wed, 20 Sep 2000 07:06:24 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA26831; Wed, 20 Sep 2000 07:06:23 -0700 (PDT) Message-Id: <4.2.2.20000920054007.00cca140@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 20 Sep 2000 05:41:58 -0700 To: Black_David@emc.com From: Michael Krause Subject: Re: iSCSI: Session Partial Resolution Cc: hufferd@us.ibm.com, ips@ece.cmu.edu In-Reply-To: <39C87DC4.D26366F4@agilent.com> References: <0F31E5C394DAD311B60C00E029101A0704100FBE@corpmx9.isus.emc.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 02:05 AM 9/20/00 -0700, Matt Wakeley wrote: >Black_David@emc.com wrote: >\ >The "iSCSI Requirements" document requires this capability: > >"[R] High bandwidth, bandwidth aggregation. > >[D] The bandwidth (transfer rate, MB/sec) supported by storage >controllers is rapidly increasing, due to several factors: (1) >Increase in disk spindle and controller performance; (2) Use >of ever-larger caches, and improved caching algorithms; (3) >Increased scale of storage controllers (number of supported >spindles, speed of interconnects). Not only must the iSCSI >provide for full utilization of available link bandwidth, it >also must exploit parallelism (multiple connections) at the >device interfaces and within the interconnect fabric." > > >Unless there is "consensus" that this is not required, I think the multiple >connections/session must remain. If nothing else, at least the "hooks" should >remain in the document (such as for login, etc) so as to minimize >incompatibilities with future versions of the specification. The mechanisms >to "bundle" tcp connections into a session are the same regardless of whether >they're used "synchronously" or "asynchronously". Also, it's much easier to >ignore unimplemented fields in a header than to add fields after the fact. I'd rather pre-enable for the future which does not really impact today's implementation costs and avoid a major rework later, hence I concur with Matt. Mike From owner-ips@ECE.cmu.edu Wed Sep 20 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28172; 20 Sep 2000 10:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA21772 for ips-outgoing; Wed, 20 Sep 2000 10:06:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA21768 for ; Wed, 20 Sep 2000 10:06:33 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id A5080B4C for ; Wed, 20 Sep 2000 07:06:32 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA26848 for ; Wed, 20 Sep 2000 07:06:32 -0700 (PDT) Message-Id: <4.2.2.20000920054837.00cca4d0@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 20 Sep 2000 05:54:51 -0700 To: From: Michael Krause Subject: Re: ISCSI: flow control In-Reply-To: <39C8833C.33572620@agilent.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 02:28 AM 9/20/00 -0700, Matt Wakeley wrote: >Well, David thinks there's consensus on flow control, but I just wanted to >make sure some issues are clear: > > > > (1) An iSCSI session containing a single TCP connection > > should not be required to use the currently specified > > iSCSI command reference numbers and sliding window > > mechanism because TCP will deliver commands in order. > > > >The iSCSI spec originally implemented command reference numbers to "order" >commands that are delivered across multiple TCP connections. The sliding >window was added so that if a tcp connection dies, the remaining tcp >connections don't flood the target with (now potentially) out of order >commands. > >It's true that an iSCSI session containing a single TCP connection, or an >iSCSI session containing a multiple TCP connections but using the >"asynchronous" model does not require command re-ordering or the sliding >window. Only the "synchronous" model with multiple TCP connections requires >both. > >However, some have wanted to use the sliding window to solve a different >problem. They want to enable the target to advertise to the initiator how >many commands the target is able to receive. This apparently is not specified >in SAM-2, and SAM-2 deals with overflowing the command queue in an ugly way - >throw the commands away and enter an error state. > >The question here is whether iSCSI should attempt to solve the command queue >overflow problem, or allow T10 to deal with it. Other protocols (fibre >channel) already deal with it in the "ugly" fashion - perhaps so should >iSCSI. What we need consensus on is whether or not iSCSI is going to deal >with this. The "ugly" mechanism is undesirable for a variety of reasons including fast convergence / recovery upon data loss. A credit system is simple to implement and provides a way to bound this problem and the implementation complexity (whether in software or hardware or if operating across a set of adapters). With a single connection implementation, this is trivial to accommodate and future multi-port or multi-adapter solutions can take advantage of it as you note with minimal effort. Given this does not really impact a single connection solution, what compelling reason is there for removing its definition / requirement within the spec at this point? Note: I believe there are also other advantages to always having this present beyond the stated problem. Mike From owner-ips@ECE.cmu.edu Wed Sep 20 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28200; 20 Sep 2000 10:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA22561 for ips-outgoing; Wed, 20 Sep 2000 10:25:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA22556 for ; Wed, 20 Sep 2000 10:25:00 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 20 Sep 2000 10:21:53 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FC1@corpmx9.isus.emc.com> To: cmonia@NishanSystems.com Cc: ips@ece.cmu.edu Subject: RE: iSCSI: Session Partial Resolution Date: Wed, 20 Sep 2000 10:21:51 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles Monia asked me a couple off off-line questions whose answers are of general interest: > As I interpret the above, you are saying that there is a rough consensus in > favor of the asynchronous model -- ie. one TCP connection to handle all > control traffic. As an option, additional connections may be created to > carry data traffic. Is that correct? No. The rough consensus is only for whether to put the command numbers and sliding window information in the headers in the single connection case. Both the symmetric and asymmetric model could be negotiated up from there. > I thought that that the command reference number and sliding window in the > control PDUs was needed to make the symmetrical session model work. As I > recall, these allowed commands arriving on different TCP connections to be > deskewed by iSCSI and sequenced for in-order delivery to the ULP. If these > are no longer present in the iSCSI header, is any meaningful consideration > of the symmetrical model possible? They could be added by negotiation at the start of the session, but it does raise the bar for consideration of the Symmetric model. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28215; 20 Sep 2000 11:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA23334 for ips-outgoing; Wed, 20 Sep 2000 10:41:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA16984 for ; Wed, 20 Sep 2000 07:53:48 -0400 (EDT) Received: from cisco.com (sj-isp-nat-pool-34.cisco.com [204.69.198.34]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA18247; Wed, 20 Sep 2000 06:40:16 -0500 Message-ID: <39C8A102.A70CAED9@cisco.com> Date: Wed, 20 Sep 2000 06:35:30 -0500 From: Randall Stewart X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jim McGrath CC: "'Y P Cheng'" , "'Ips@Ece. Cmu. Edu'" Subject: Re: A Transport Protocol Without ACK References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim McGrath wrote: > Actually the burden of proof issues is why I suggest we look at some things > that are actually being used today (since you don't have to guess how they > behave). That is one of TCP's great strengths, and a bit of a weakness for > SCTP (no offense to SCTP supporters, but it certainly does not have a big > and long "track record" yet, and so I can understand the concerns others may > have as to whether things would work out as well in practice as they do in > proposal). > Jim: SCTP's congestion control comes right from RFC2581 and it is the same has TCP. Now it is true you may get bad implemenations of congestion control in SCTP... but this same fact holds true for TCP. There are good and bad implemenations of TCP CC in the market... most of the bad have dwindled away (hopefully). My point is that IF the WG decides to go ahead with some UDPish transport protocol and they DO NOT follow RFC2581 then there is a extreme burden of proof on that transport protocol. SCTP's behavior is known.. it is the same as TCP. Is all of the individual implementations behavior known... i.e. how folks implemented it? no, and neither do we know all the behaviors of the TCP implemenation... the postive thing for TCP is that MOST of the implemenations now out there have been corrected ... R > Jim > > PS Personally, I'm a big believer is copying stuff that works, making the > minimum amount of required changes, and then doing a rapid but controlled > deployment (I've been involved in a lot of those sorts of things in ATA and > SCSI). Having been involved in both these sorts of endevors and the > opposite (big, clean sheet of paper efforts, like 1394 (no offense to > 1394/Firewire supporters, but I was working on it a decade ago)), I know how > easy it is to underestimate the work required by the latter, and to be > turned off by the "inelegance" of the former. For me, life has become too > short - I'm willing to accept inelegance as the price for speed of > deployment. > > -----Original Message----- > From: Randall Stewart [mailto:rrs@cisco.com] > Sent: Tuesday, September 19, 2000 4:38 AM > To: Jim McGrath > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > Subject: Re: A Transport Protocol Without ACK > > Jim: > > Any transport protocol proposal is ok. As long as it can be seen and > reviewed. So far I have seen only two TCP and SCTP. > > Oh, a little side note, any transport protocol proposed MUST be able to > show TCP like behavior in the face of congestion. And I think, IMHO, that > this means that if it is NOT using RFC2581 procedures it MUST show that > it does backoff and share with TCP. It also has a HEAVY burden of proof to > show this facility at least in my mind and I would think in the IESG's mind > as well... > > R > > Jim McGrath wrote: > > > I would expand your search to include non standard protocols (i.e. > > proprietary ones) as well if they offered something and were adequately > > understood by the outside world. We do that in storage quite a lot - > > indeed, some standard protocols are direct descendants of what were once > > proprietary protocols (e.g. ATA, the most widely used desktop disk > > interface, and ESCON, a dominant mainframe class interface (both of which > > originated from IBM proprietary technologies)). > > > > Jim > > > > -----Original Message----- > > From: Y P Cheng [mailto:ycheng@advansys.com] > > Sent: Monday, September 18, 2000 5:52 PM > > To: 'Ips@Ece. Cmu. Edu' > > Subject: RE: A Transport Protocol Without ACK > > > > From: randall@stewart.chicago.il.us > > > I see no viable transport protocol here and I don't see this > > > conversation of any use unless you get exact details AND point > > > to a internet draft that defines EXACTLY how it works (or possibly > > > some other standards document). > > > > Both I2O and VI are transport protocols which define the format of a > request > > to a transport service provider, i.e. an adapter card. I2O is used but > not > > limited to deliver SCSI requests and VI is used for any payload including > IP > > packets. VI is mapped into FC with the device headers between the FC > header > > and data payload. VI can certainly be used for delivery of SCSI requests > > too. Both protocols require the service provider to have reliable > delivery > > and reception. VI defines different QoS. > > > > > > I don't claim any credit about this transport layer protocol. Every > > fibre > > > > channel and Infiniband adapter designer knows about this protocol -- > > > > although there is no standard. I am sure the TCP accelerator card is > > doing > > > > the same. This protocol is a great alternative to the use of TCP/IP > and > > > > should be incorporated into iSCSI. > > > > > > No it is not. You are not offering an alternative yet.. > > > > I did not imply iSCSI should use I2O or VI. In fact, the purpose iSCSI is > > to map SCSI requests into IP packets as well as to define the delivery . > It > > seems to me that the working group has set its mind on TCP/IP and is > > believing this is the only solution. The consensus seems if there is any > > other solutions that address flow control and congestion, it would end up > > like TCP/IP. I am simply pointing out if we keep an iSCSI request as a > > single atomic transaction without separating it into the > > TCP/IP-stream-oriented Writes and Reads that each deals with a single DU, > > then, the deadlock problem goes away. While the work group thinks we > should > > take advantage the flow control and congestion management of TCP/IP, there > > are alternatives known as BB-credit and EE-credit management. The fibre > > channel adapters make reliable delivery, lost packet detection, and > > retransmission without TCP/IP. > > > > Randall, you are right, I did not spent time to provide the working group > a > > draft defining such transaction-oriented protocol. All I have provided is > > an idea that besides TCP/IP. The designers for SCSI and fibre channel > > adapters have solved the head-of-queue blocking, the congestion, and > > retransmission problems. The transaction-oriented WRITE-REQUEST and > > READ-RESPONSE, in my humble opinion, allows us to implement iSCSI simpler > > than that of WRITE and READ stream requests. The performance cost of > > requiring ACKs on every DU with size greater than MTU on a network with > long > > latency is very expensive.. By defining a greater ACK granularity is an > > attempt to solve this performance problem. If we do wish to ACK on every > > DU, then, on a long latency network, we must have a method to stream the > > PDUs to ensure the performance. The method should not consume a large > > amount of memory space. One should never ignore the TCP/IP > memory-to-memory > > copy overhead when the backbone will be running at OC-192 speed in the > near > > future. Finally, please don't ever ask two NIC cards to synchronize with > > each other. It is really hard to do as those of us in business of > designing > > NIC cards can testify. > > > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Wed Sep 20 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28374; 20 Sep 2000 11:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA23342 for ips-outgoing; Wed, 20 Sep 2000 10:41:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA17153 for ; Wed, 20 Sep 2000 08:00:26 -0400 (EDT) Received: from cisco.com (sj-isp-nat-pool-34.cisco.com [204.69.198.34]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id HAA18285; Wed, 20 Sep 2000 07:00:08 -0500 Message-ID: <39C8A5AA.393B4B68@cisco.com> Date: Wed, 20 Sep 2000 06:55:23 -0500 From: Randall Stewart X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Matt Wakeley CC: IPS Reflector Subject: Re: LU access through an iSCSI session References: <39C867AA.F9CA5841@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt Wakeley wrote: > .....snip....snip.... > Again, the *only* answer is "Yes". The multiple TCP connections make up a > *single* session. The session just happens to have lots of "TCP connections" > to make it look like it has a "big pipe" (for example, 4 1Gbit connections can > be used to emulate a 4Gbit link). > > If you want different "LU views" per TCP connection, then each TCP connection > would be a separate, independent iscsi session. ....snip....snip..... The above talk of making "lots of TCP connections look like a big pipe" sounds once again like attempting to get around congestion control in the network... I know we had this conversation before.. but what is stated above strongly implies this .... and I don't think this is the behavior one would want on the Big I-internet. I can see a private network doing this ... but I don't see how it could be supported in a standard manner that can be used on the Iternet... R From owner-ips@ECE.cmu.edu Wed Sep 20 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28376; 20 Sep 2000 11:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA23434 for ips-outgoing; Wed, 20 Sep 2000 10:43:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA23430 for ; Wed, 20 Sep 2000 10:43:30 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 20 Sep 2000 10:42:52 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FC2@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI: Session Partial Resolution Clarifications Date: Wed, 20 Sep 2000 10:42:51 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk A few clarifications based on the last day's worth of email. - The "flow control" issue is definitely still open. The sliding window mechanism is one way of addressing this, but I'd venture that it may not be the one that would get chosen if this were the only goal. So the "flow control" issue is definitely open, and if that results in putting back the reference numbers and sliding windows, so be it ... but not without a serious consideration of other design alternatives such as capacity advertisement on login and credit-based mechanisms. Keep in mind that this is iSCSI-level flow control; the WG does not have a license to tinker with TCP's flow and congestion control. - The requirement for OPTIONAL support of multiple connection sessions has not gone away; the intent is to address it in a separate document in order to make progress on the main document. I'm assuming (and should have stated, sorry), that the main iSCSI document will continue to contain a negotiation mechanism sufficient to set up any of the connection models that have been proposed. My understanding is that contains sufficient "hooks" as that term has been used by Matt Wakeley and Michael Krause, but alternative views (e.g., we need to set aside n bytes of Reserved fields for future use in the headers now) are welcome. - Those who want to continue to debate multiple connection sessions on the list need to explain what will be different that will avoid spending another 6 weeks and not getting to consensus. In the absence of such an explanation, expect to see active discouragement of such traffic on the list. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28397; 20 Sep 2000 11:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA24269 for ips-outgoing; Wed, 20 Sep 2000 11:01:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA24265 for ; Wed, 20 Sep 2000 11:00:58 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id HAA12085; Wed, 20 Sep 2000 07:49:20 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id HAA08086; Wed, 20 Sep 2000 07:47:02 -0700 From: "Douglas Otis" To: "Matt Wakeley" , Cc: , Subject: RE: iSCSI: Session Partial Resolution Date: Wed, 20 Sep 2000 08:03:22 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <39C87DC4.D26366F4@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Wednesday, September 20, 2000 2:05 AM > To: Black_David@emc.com > Cc: hufferd@us.ibm.com; ips@ece.cmu.edu > Subject: Re: iSCSI: Session Partial Resolution > > Black_David@emc.com wrote: > > > > > OTOH, I do not see consensus on the session model for > > multiple connection sessions among the Asymmetric model, > > the Symmetric model, and Pierre Labat's proposal. In > > order to make progress on iSCSI, I see no alternative to > > separating multi-connection sessions from the main iSCSI > > spec. Significant effort and email traffic has been > > invested in this topic for at least 6 weeks and the issue > > is not settled -- I don't think holding up the iSCSI spec > > for another 6+ weeks in hopes of settling this issue on > > the list is an effective way to make progress, but I'm > > prepared to listen to dissenting opinions (e.g., if > > someone thinks there is rough consensus, and I've missed > > it); please send such opinions directly to me rather than > > using the list. I've already had one offline comment from > > an outside observer expressing amazement at the willingness > > of this community to discuss multi-connection sessions > > "ad nauseum". > > > > Therefore, I would ask that the authors of the next > > version of the iSCSI draft delete all specification > > of multiple connection sessions from the next version > > of the except for a note that they will be handled in > > a separate document. > > The "iSCSI Requirements" document requires this capability: > > "[R] High bandwidth, bandwidth aggregation. This is a general feature of IP. > [D] The bandwidth (transfer rate, MB/sec) supported by storage > controllers is rapidly increasing, due to several factors: (1) > Increase in disk spindle and controller performance; The linear increase in the performance of the mechanical aspects of the drive have not kept pace with the exponential data density improvements. Unless dealing with a single massive transfer, the overall effect of data density improvements is a small reduction in latency lost by the large latency within a MAN network. > (2) Use > of ever-larger caches, and improved caching algorithms; Network latency negates use of remote caching. Such caching must be near the client and not the drive. If you wish to define a controller specification, this should be done using appropriate technologies. Clearly, if the controller is on site, interface requirements change substantially. For redundancy, there may even be one per client system! Now the traffic that needs some work would be to ensure such controllers stay synchronized if used in a cluster. Either way, you will not see a higher performance moving the controller next to the drive. > (3) > Increased scale of storage controllers (number of supported > spindles, speed of interconnects). Not only must the iSCSI > provide for full utilization of available link bandwidth, it > also must exploit parallelism (multiple connections) at the > device interfaces and within the interconnect fabric." Again, this is a general feature of IP. If access is to the drive, as it should be, there is no problem scaling especially if you are using IP. > Unless there is "consensus" that this is not required, I think > the multiple > connections/session must remain. Not one reason for forcing aggregation into a single connection. If you wish to bind multiple adapters, IP supports this. I doubt you will ever need to do this however. Doug > If nothing else, at least the > "hooks" should > remain in the document (such as for login, etc) so as to minimize > incompatibilities with future versions of the specification. The > mechanisms > to "bundle" tcp connections into a session are the same > regardless of whether > they're used "synchronously" or "asynchronously". Also, it's > much easier to > ignore unimplemented fields in a header than to add fields after the fact. > > -Matt Wakeley > Agilent Technologies > > > Producing that separate document is going to require > > an offline design team. The design team can either be > > chartered to write a compromise session specification > > or to evaluate competing specifications and choose one. > > My current inclination is to do the latter, which would > > involve having the design team produce a set of requirements > > and guidelines for session specifications in consultation > > with the co-chairs, evaluate Internet-Drafts documenting > > the specifications, and recommend an approach to the WG. > > Comments on this process are solicited - either on the > > list or to me directly. Further discussion of multi-connection > > sessions on the list is probably not a good use of list > > bandwidth. > > > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Sep 20 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28422; 20 Sep 2000 11:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA24035 for ips-outgoing; Wed, 20 Sep 2000 10:56:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA24026 for ; Wed, 20 Sep 2000 10:56:31 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id KAA87626 for ; Wed, 20 Sep 2000 10:54:50 -0400 Received: from d03nmx42.almaden.ibm.com (d03nmx42.almaden.ibm.com [9.1.24.146]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8KEuTX123820 for ; Wed, 20 Sep 2000 08:56:29 -0600 Importance: Normal Subject: Bidirectional SCSI commands and iSCSI To: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Wed, 20 Sep 2000 07:56:27 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NMX42/03/M/IBM(Release 5.0.4a |July 24, 2000) at 09/20/2000 07:56:28 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, In case you missed this, T10 just approved changes to SAM-2 (rev14) which remove the restriction that a single SCSI command can have a data flow in only one direction (DataIn xor DataOut). Transport specs (that want to allow this) will need to change to allow for two sets of Data buffer fields in the headers (length, over/underrun, etc.). Is anybody looking to modify the iSCSI spec to get these concepts supported? Jim Hafner From owner-ips@ECE.cmu.edu Wed Sep 20 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28967; 20 Sep 2000 13:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA00279 for ips-outgoing; Wed, 20 Sep 2000 13:10:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA00275 for ; Wed, 20 Sep 2000 13:10:10 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 2EE1B115D for ; Wed, 20 Sep 2000 13:09:31 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id KAA18971 for ; Wed, 20 Sep 2000 10:09:29 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA2FA1 for ; Wed, 20 Sep 2000 10:09:26 -0700 Message-ID: <39C8EF39.3BCB7139@agilent.com> Date: Wed, 20 Sep 2000 10:09:14 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: LU access through an iSCSI session References: <39C867AA.F9CA5841@agilent.com> <39C8A5AA.393B4B68@cisco.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk The talk of making "lots of TCP connections look like a big pipe" is NOT meant for congestion control. It is meant to obtain higher bandwidth on a local network, such as in a data room. For example, one could have multiple 1Gbit TCP connections from a host to say a switch. The switch might be connected via a very high speed backbone to another area where the storage devices are, where there is another such switch. Similarly, a disc array may be connected via multiple 1Gbit TCP connections to that switch. There is lots of early discussions on why multiple TCP connections are required to obtain parallel paths through the fabric. You might check out the email archives, especially messages posted by Randy Haagens. -Matt Randall Stewart wrote: > Matt Wakeley wrote: > > > .....snip....snip.... > > Again, the *only* answer is "Yes". The multiple TCP connections make up a > > *single* session. The session just happens to have lots of "TCP connections" > > to make it look like it has a "big pipe" (for example, 4 1Gbit connections can > > be used to emulate a 4Gbit link). > > > > If you want different "LU views" per TCP connection, then each TCP connection > > would be a separate, independent iscsi session. > > ....snip....snip..... > > The above talk of making "lots of TCP connections look like a big pipe" sounds > once > again like attempting to get around congestion control in the network... I know > we > had this conversation before.. but what is stated above strongly implies this > .... and > I don't think this is the behavior one would want on the Big I-internet. I can > see > a private network doing this ... but I don't see how it could be supported in a > standard manner that can be used on the Iternet... > > R From owner-ips@ECE.cmu.edu Wed Sep 20 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29019; 20 Sep 2000 13:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA28371 for ips-outgoing; Wed, 20 Sep 2000 12:26:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA28367 for ; Wed, 20 Sep 2000 12:26:30 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel3.hp.com (Postfix) with ESMTP id 2956E4BF for ; Wed, 20 Sep 2000 09:26:29 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id JAA29434; Wed, 20 Sep 2000 09:26:28 -0700 (PDT) Message-ID: <39C8E6C4.838E92E2@hp.com> Date: Wed, 20 Sep 2000 09:33:08 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: LU access through an iSCSI session Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk It seems that most people assume/wish that one session correspond conceptually to a "service delivery port". Hence the LU number used are the same across the connections. Could it be explicitly stated in the iSCSI draft (or the in the Appendix about multiple connections sessions)? Regards, Pierre From owner-ips@ECE.cmu.edu Wed Sep 20 19:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29456; 20 Sep 2000 15:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA02342 for ips-outgoing; Wed, 20 Sep 2000 13:57:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA02337 for ; Wed, 20 Sep 2000 13:57:18 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id KAA03346 for ; Wed, 20 Sep 2000 10:57:11 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.153.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id KAA02678 for ; Wed, 20 Sep 2000 10:57:11 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id KAA08209; Wed, 20 Sep 2000 10:57:10 -0700 (PDT) Message-Id: <200009201757.KAA08209@ha10nwk.EBay.Sun.COM> Date: Wed, 20 Sep 2000 10:57:10 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: iSCSI: Flow Control To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: UgqxxmqtMbOT3o5EP+hZZQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > - The "flow control" issue is definitely still > open. The sliding window mechanism is one way of > addressing this, but I'd venture that it may not > be the one that would get chosen if this were the > only goal. So the "flow control" issue is definitely > open, and if that results in putting back the > reference numbers and sliding windows, so be it ... > but not without a serious consideration of other > design alternatives such as capacity advertisement > on login and credit-based mechanisms. Keep in > mind that this is iSCSI-level flow control; the > WG does not have a license to tinker with TCP's > flow and congestion control. Could someone please clearly state the problem that iSCSI "flow control" is attempting to solve. Given the reliable stream nature of TCP with its flow control, what problem exists? Also with an iSCSI layer flow control will we not also be victim to some nasty hysteresis effects due to TCP's flow control as well? -david From owner-ips@ECE.cmu.edu Wed Sep 20 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29562; 20 Sep 2000 15:37 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA03910 for ips-outgoing; Wed, 20 Sep 2000 14:32:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA03902 for ; Wed, 20 Sep 2000 14:32:28 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA12156; Wed, 20 Sep 2000 11:20:31 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA08629; Wed, 20 Sep 2000 11:18:14 -0700 From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: LU access through an iSCSI session Date: Wed, 20 Sep 2000 11:34:35 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <39C8EF39.3BCB7139@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, As you are clearly designing an interface for an on-site controller, why use a large numbers of TCP connections? For low latency, would an ST like protocol be a better choice? See http://www.ietf.org/rfc/rfc1819.txt. Confirming placement of the controller being at the client, you have little concern about the treatment of the network which is obviously dedicated. Why bother with IETF? There are many alternative interconnects and more coming, why re-invent this interface if it is indeed local and dedicated? Surely, even with hardware assist, you will be better off with other solutions. My idea of an ideal controller would be one that bolts into the system and finds access via SCTP. Intel will suggest Infiniband. What does Sun, IBM and HP use to talk to local controllers? Fibre-Channel? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Wednesday, September 20, 2000 10:09 AM > To: IPS Reflector > Subject: Re: LU access through an iSCSI session > > > The talk of making "lots of TCP connections look like a big pipe" > is NOT meant for > congestion control. It is meant to obtain higher bandwidth on a > local network, such > as in a data room. > > For example, one could have multiple 1Gbit TCP connections from a > host to say a > switch. The switch might be connected via a very high speed > backbone to another > area where the storage devices are, where there is another such > switch. Similarly, > a disc array may be connected via multiple 1Gbit TCP connections > to that switch. > > There is lots of early discussions on why multiple TCP > connections are required to > obtain parallel paths through the fabric. You might check out > the email archives, > especially messages posted by Randy Haagens. > > -Matt > > Randall Stewart wrote: > > > Matt Wakeley wrote: > > > > > .....snip....snip.... > > > Again, the *only* answer is "Yes". The multiple TCP > connections make up a > > > *single* session. The session just happens to have lots of > "TCP connections" > > > to make it look like it has a "big pipe" (for example, 4 > 1Gbit connections can > > > be used to emulate a 4Gbit link). > > > > > > If you want different "LU views" per TCP connection, then > each TCP connection > > > would be a separate, independent iscsi session. > > > > ....snip....snip..... > > > > The above talk of making "lots of TCP connections look like a > big pipe" sounds > > once > > again like attempting to get around congestion control in the > network... I know > > we > > had this conversation before.. but what is stated above > strongly implies this > > .... and > > I don't think this is the behavior one would want on the Big > I-internet. I can > > see > > a private network doing this ... but I don't see how it could > be supported in a > > standard manner that can be used on the Iternet... > > > > R > From owner-ips@ECE.cmu.edu Wed Sep 20 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30094; 20 Sep 2000 17:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA08387 for ips-outgoing; Wed, 20 Sep 2000 16:05:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA08383 for ; Wed, 20 Sep 2000 16:05:44 -0400 (EDT) From: pat_thaler@agilent.com Received: from msgrel1.cos.agilent.com (msgrel1.cos.agilent.com [130.29.152.77]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 65F7CE8; Wed, 20 Sep 2000 14:05:43 -0600 (MDT) Received: from axcsbh2.cs.itc.hp.com (axcsbh2.cs.itc.hp.com [15.69.24.34]) by msgrel1.cos.agilent.com (Postfix) with SMTP id 358CCBC; Wed, 20 Sep 2000 14:05:43 -0600 (MDT) Received: from 15.69.24.34 by axcsbh2.cs.itc.hp.com (InterScan E-Mail VirusWall NT); Wed, 20 Sep 2000 14:05:42 -0600 (Mountain Daylight Time) Received: by axcsbh2.cs.itc.hp.com with Internet Mail Service (5.5.2650.21) id ; Wed, 20 Sep 2000 14:05:42 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A002F2E0B1@axcs04.cs.itc.hp.com> To: steph@cs.uchicago.edu, bassoon@yogi.ece.cmu.edu, ips@ece.cmu.edu Subject: RE: another question (was Re: patent question) Date: Wed, 20 Sep 2000 14:05:32 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Even when staying within a LAN or VLAN, as the network scales up in size, congestion control becomes necessary. Neither flow control nor end-to-end flow control provide effective congestion control in large networks. For scalability, one needs an adaptive flow control that will react to congestion within the network by throttling the congestion sources with minimal impact on sources that are not causing congestion. The other alternative is resource reservation over the whole path rather than just at the end devices. However, given reasonably bursty traffic, strict resource reservation wastes a lot of bandwidth and therefore has its own scaling problems. For a small network, careful configuration control may work. Efficient, reliable operation beyond that requires congestion control. Developing effective, stable congestion control algorithms has been very challenging (to put it mildly). TCP provides an effective congestion control that operates across a wide variety of network sizes and conditions. Pat -----Original Message----- From: Stephen Bailey [mailto:steph@cs.uchicago.edu] Sent: Wednesday, September 20, 2000 3:23 AM To: bassoon@yogi.ece.cmu.edu; ips@ece.cmu.edu Subject: Re: another question (was Re: patent question) > > i have another quick question. for a local area application, > > why hasn't anyone done scsi over ethernet? (ie encapsulating > > scsi commands into the ethernet package). why use a higher > > level protocol and work with all the latency and overhead problem? > > Check the list archives for info on an Adaptec demo of this, and I'm > sure Paul von Stamwitz can provide additional information (off the > list, please). Encapsulating directly on Ethernet runs into scaling > problems - the first time one needs to get off the LAN or VLAN that > the traffic started on, one discovers that an IP header is an immensely > useful thing to have. I can add that SCSI on ST (SST) is another implemented form of SCSI on Ethernet with existing, hardware accelerated implementations. See www.genroco.com for more information. What David says is correct. You CAN build SANs this way, but you must careful control the configuration, which is another reason why scalability is limited. In some cases, having an Ethernet SAN which requires this careful configuration control is still better than needing two network attachment infrastructures (Ethernet & FCP). Steph From owner-ips@ECE.cmu.edu Wed Sep 20 23:40:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30675; 20 Sep 2000 19:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA14655 for ips-outgoing; Wed, 20 Sep 2000 18:51:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA14647 for ; Wed, 20 Sep 2000 18:50:56 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA29343 for ; Wed, 20 Sep 2000 15:50:55 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.142.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA16347 for ; Wed, 20 Sep 2000 15:50:54 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id PAA26743; Wed, 20 Sep 2000 15:50:53 -0700 (PDT) Message-Id: <200009202250.PAA26743@ha10nwk.EBay.Sun.COM> Date: Wed, 20 Sep 2000 15:50:53 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: iSCSI: Flow Control To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: yPQTzmWfhDe9mPTfy5PDXA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > As I understand it, TCP should not have any intelligence > of the iSCSI layer above it. Thus, TCP does not know if > the oncoming data is an iSCSI data PDU or a command PDU. > If it is receiving a long stream of data, TCP can't tell > if it's 1 short command PDU and a very long data PDU, or > 100 short command PDU's. I agree completely. > The latter case could overwhelm the > SCSI command queue, but still be within the TCP window of > the receiving node. This I don't understand. The act of TCP receiving data into its window (and thus causing the window to shrink) is complete independant of the application (aka iSCSI) grabbing the data (causing the window to grow). As an extreme example a naught initiator sends 1,000 commands down to a target that has a command queue depth of 2, will it be overwhelmed? I think not. A sane implementation of the target will detect that there is data on the connection, read a chunk large enough to grab the first command and stuff it in the command queue, read the next command and stuff it in the command queue, then simply waits for one of the commands to finish leaving the other 998 commands in the TCP receive buffers. When one of the commands completes it sucks in the next command and puts it in the queue. > That is, unless you want iSCSI to talk > to TCP to tell it to close the window. But that would > violate layering principles, I believe. The above example violates not layering, the TCP window closes because the application does not read all of the data because its command queue is too small. Now a clever implementation can do interesting tricks in hardware where it copies the data stream straight into the command queue based on its interpretation of the data stream. The overly clever (and thus dangerous) implementation could also play with the window and not open it up until *after* the command is executed. But again this is an implementation hack that saves a copy but then violates layering, but the proposed protocol does not violate layering. Still don't see the issue. -David From owner-ips@ECE.cmu.edu Wed Sep 20 23:50:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30680; 20 Sep 2000 19:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA14093 for ips-outgoing; Wed, 20 Sep 2000 18:33:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA14088 for ; Wed, 20 Sep 2000 18:33:09 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 20 Sep 2000 15:34:41 -0700 Message-ID: From: Joshua Tseng To: David Robinson , ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Wed, 20 Sep 2000 15:34:41 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi David, >Could someone please clearly state the problem that >iSCSI "flow control" is attempting to solve. Given >the reliable stream nature of TCP with its flow >control, what problem exists? Also with an iSCSI >layer flow control will we not also be victim to some >nasty hysteresis effects due to TCP's flow control as >well? > > -david As I understand it, TCP should not have any intelligence of the iSCSI layer above it. Thus, TCP does not know if the oncoming data is an iSCSI data PDU or a command PDU. If it is receiving a long stream of data, TCP can't tell if it's 1 short command PDU and a very long data PDU, or 100 short command PDU's. The latter case could overwhelm the SCSI command queue, but still be within the TCP window of the receiving node. That is, unless you want iSCSI to talk to TCP to tell it to close the window. But that would violate layering principles, I believe. Josh From owner-ips@ECE.cmu.edu Wed Sep 20 23:50:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30693; 20 Sep 2000 19:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA13485 for ips-outgoing; Wed, 20 Sep 2000 18:15:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA13481 for ; Wed, 20 Sep 2000 18:15:11 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id C307FDA8 for ; Wed, 20 Sep 2000 18:15:06 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id PAA12691 for ; Wed, 20 Sep 2000 15:15:00 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA37EC for ; Wed, 20 Sep 2000 15:14:56 -0700 Message-ID: <39C936D4.FBA24863@agilent.com> Date: Wed, 20 Sep 2000 15:14:44 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI: Flow Control References: <200009201757.KAA08209@ha10nwk.EBay.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David Robinson wrote: > Could someone please clearly state the problem that > iSCSI "flow control" is attempting to solve. Given > the reliable stream nature of TCP with its flow > control, what problem exists? Also with an iSCSI > layer flow control will we not also be victim to some > nasty hysteresis effects due to TCP's flow control as > well? > > -david I stated the issue in my earlier message of the same title: > However, some have wanted to use the sliding window to solve a different > problem. They want to enable the target to advertise to the initiator how > many commands the target is able to receive. This apparently is not specified > in SAM-2, and SAM-2 deals with overflowing the command queue in an ugly way - > throw the commands away and enter an error state. > > The question here is whether iSCSI should attempt to solve the command queue > overflow problem, or allow T10 to deal with it. Other protocols (fibre > channel) already deal with it in the "ugly" fashion - perhaps so should > iSCSI. What we need consensus on is whether or not iSCSI is going to deal > with this. > There is LOTs of discussion on this topic in earlier postings. -Matt From owner-ips@ECE.cmu.edu Wed Sep 20 23:50:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30718; 20 Sep 2000 19:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA14837 for ips-outgoing; Wed, 20 Sep 2000 18:56:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA14833 for ; Wed, 20 Sep 2000 18:56:30 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA01416 for ; Wed, 20 Sep 2000 15:56:29 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.142.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA17761 for ; Wed, 20 Sep 2000 15:56:28 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id PAA29630; Wed, 20 Sep 2000 15:56:28 -0700 (PDT) Message-Id: <200009202256.PAA29630@ha10nwk.EBay.Sun.COM> Date: Wed, 20 Sep 2000 15:56:28 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: Re: iSCSI: Flow Control To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 06ZxbE77SYb0u6yGLmOJUQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > > However, some have wanted to use the sliding window to solve a different > > problem. They want to enable the target to advertise to the initiator how > > many commands the target is able to receive. This apparently is not specified > > in SAM-2, and SAM-2 deals with overflowing the command queue in an ugly way - > > throw the commands away and enter an error state. > > > > The question here is whether iSCSI should attempt to solve the command queue > > overflow problem, or allow T10 to deal with it. Other protocols (fibre > > channel) already deal with it in the "ugly" fashion - perhaps so should > > iSCSI. What we need consensus on is whether or not iSCSI is going to deal > > with this. FC has a command queue overflow problem because it is a datagram protocol and cannot at the transport layer finely control the sender. But with a reliable stream protocol I don't understand how you could ever get a command queue overflow. If you command queue is full you simply stop reading from the TCP stream which will eventually cause the TCP input buffers to fill and the window to shrink to zero causing flow to stop. Short of a naive implementation that reads the TCP stream (thus keeping the window open) with no place to put the command it just read, I still don't see the problem. -David From owner-ips@ECE.cmu.edu Thu Sep 21 00:10:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30808; 20 Sep 2000 20:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA15004 for ips-outgoing; Wed, 20 Sep 2000 19:01:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA15000 for ; Wed, 20 Sep 2000 19:01:44 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 20 Sep 2000 16:03:21 -0700 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: iSCSI Security Protocol Date: Wed, 20 Sep 2000 16:03:20 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I just did a brief review of the document draft-klein-iscsi-security-00.txt. What is the current consensus (if any) on this document? Is there agreement to use SSH as the security mechanism for iSCSI? Josh From owner-ips@ECE.cmu.edu Thu Sep 21 01:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31063; 20 Sep 2000 20:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA17146 for ips-outgoing; Wed, 20 Sep 2000 20:13:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA17140 for ; Wed, 20 Sep 2000 20:13:11 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id RAA27683 for ; Wed, 20 Sep 2000 17:13:07 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.187.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id RAA25119 for ; Wed, 20 Sep 2000 17:13:06 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id RAA10925; Wed, 20 Sep 2000 17:13:05 -0700 (PDT) Message-Id: <200009210013.RAA10925@ha10nwk.EBay.Sun.COM> Date: Wed, 20 Sep 2000 17:13:05 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: iSCSI: Flow Control To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: LX3PmlhTgXUYMyLr4aWulQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > I'm not a s/w engineer, so I could be getting myself into trouble here, > but I still see an issue.... > > iSCSI will not know whether the next PDU in the TCP buffer is a command > or data PDU until it actually reads the data. iSCSI should NEVER stop > reading > data from TCP or else deadlock if the next message is a data PDU (as was so > eloquently explained to me in a prior message). It must continue reading > data (thus emptying the TCP receive window), and if the SCSI command queue > is full, then SCSI will respond with TASK SET FULL. This works okay > (depending > on who you ask) in a parallel SCSI or FC world where latency is measured in > us, > but in iSCSI it could be 100ms and 1000+ commands later before the initiator > ever finds out. The TCP connection can have either a command or data. In either case reading a short fixed amount of data can determine if it is one or the other. With a single connection I can only see deadlock occuring if the initiator sends Command1, Command2, then Data1 on a single command/data connection. With a single connection this seems like a broken initiator as doing the right thing of C1, D1, C2 shouldn't be any more expensive and it is definately safer. Well one other case can occur is if the initiator wants the target to issue an RTT and it interposes another command before the RTT is received, this too seems like a broken things to do as I don't see much value in doing RTT on a single connection. In the multiple connection asymettric case this isn't an issue as data is never interleved with commands. So is there a fundamental requirement that the target must handle the case of C1, C2, then D1? It seems easy to just write the iSCSI initiator to never issue such a sequence and if the target gets one return an error when C2 arrives before D1. It again seems like we are trying to impose or solve datagram problems in a stream world. Does parallel SCSI have this problem? -David From owner-ips@ECE.cmu.edu Thu Sep 21 01:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31065; 20 Sep 2000 20:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA16600 for ips-outgoing; Wed, 20 Sep 2000 19:54:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA16596 for ; Wed, 20 Sep 2000 19:54:30 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 20 Sep 2000 16:56:07 -0700 Message-ID: From: Joshua Tseng To: David Robinson , ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Wed, 20 Sep 2000 16:56:06 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk I'm not a s/w engineer, so I could be getting myself into trouble here, but I still see an issue.... iSCSI will not know whether the next PDU in the TCP buffer is a command or data PDU until it actually reads the data. iSCSI should NEVER stop reading data from TCP or else deadlock if the next message is a data PDU (as was so eloquently explained to me in a prior message). It must continue reading data (thus emptying the TCP receive window), and if the SCSI command queue is full, then SCSI will respond with TASK SET FULL. This works okay (depending on who you ask) in a parallel SCSI or FC world where latency is measured in us, but in iSCSI it could be 100ms and 1000+ commands later before the initiator ever finds out. Josh -----Original Message----- From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] Sent: Wednesday, September 20, 2000 3:51 PM To: ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control > As I understand it, TCP should not have any intelligence > of the iSCSI layer above it. Thus, TCP does not know if > the oncoming data is an iSCSI data PDU or a command PDU. > If it is receiving a long stream of data, TCP can't tell > if it's 1 short command PDU and a very long data PDU, or > 100 short command PDU's. I agree completely. > The latter case could overwhelm the > SCSI command queue, but still be within the TCP window of > the receiving node. This I don't understand. The act of TCP receiving data into its window (and thus causing the window to shrink) is complete independant of the application (aka iSCSI) grabbing the data (causing the window to grow). As an extreme example a naught initiator sends 1,000 commands down to a target that has a command queue depth of 2, will it be overwhelmed? I think not. A sane implementation of the target will detect that there is data on the connection, read a chunk large enough to grab the first command and stuff it in the command queue, read the next command and stuff it in the command queue, then simply waits for one of the commands to finish leaving the other 998 commands in the TCP receive buffers. When one of the commands completes it sucks in the next command and puts it in the queue. > That is, unless you want iSCSI to talk > to TCP to tell it to close the window. But that would > violate layering principles, I believe. The above example violates not layering, the TCP window closes because the application does not read all of the data because its command queue is too small. Now a clever implementation can do interesting tricks in hardware where it copies the data stream straight into the command queue based on its interpretation of the data stream. The overly clever (and thus dangerous) implementation could also play with the window and not open it up until *after* the command is executed. But again this is an implementation hack that saves a copy but then violates layering, but the proposed protocol does not violate layering. Still don't see the issue. -David From owner-ips@ECE.cmu.edu Thu Sep 21 02:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31337; 20 Sep 2000 21:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA17651 for ips-outgoing; Wed, 20 Sep 2000 20:32:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA17638 for ; Wed, 20 Sep 2000 20:31:57 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id RAA01287; Wed, 20 Sep 2000 17:31:38 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Wed, 20 Sep 2000 17:31:37 -0700 Message-ID: From: Jim McGrath To: "'Black_David@emc.com'" , ips@ece.cmu.edu Subject: RE: iSCSI: Session Partial Resolution Clarifications Date: Wed, 20 Sep 2000 17:31:36 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I think our experience with Fibre Channel illustrates that establishing initial credit at login ("login BB credit") will not get what people want. One possible solution is to keep the concept of an initial credit, but to allow the target to dynamically manage this credit allocation with its initiators. For instance, if one set of initiators starts becoming very active, then the target may allocate more initial credit to them and subtract it from others. In practice this might solve the allocation problems that have hindered this feature in Fibre Channel. Note that a target could accomplish this by logging out and then logging back in with a new credit setting, but this approach has two practical problems. First, there will be a period of time when the initiator/target pair are not logged in with respect to one another, and so cannot execute commands. Second, doing this frequently will probably drive the software folks nuts - using login in this manner would appear to be similar to the overuse of RESET or MODE SELECT to solve all problems (i.e. the sledgehammer approach). So if there is a side protocol for changing initial credit allocations, then we can keep everyone always logged in, but not freeze a credit allocation at login. In particular, targets will actually be able to use this if they had the freedom to readjust credits as the loads change (always remembering that the credits are not reclaimed until the initiator confirms their deallocation). I'd recommend that this side protocol look like something familiar (ie. the login process or the dynamic credit flow control process) rather than invent something entirely new. Jim -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Wednesday, September 20, 2000 7:43 AM To: ips@ece.cmu.edu Subject: iSCSI: Session Partial Resolution Clarifications A few clarifications based on the last day's worth of email. - The "flow control" issue is definitely still open. The sliding window mechanism is one way of addressing this, but I'd venture that it may not be the one that would get chosen if this were the only goal. So the "flow control" issue is definitely open, and if that results in putting back the reference numbers and sliding windows, so be it ... but not without a serious consideration of other design alternatives such as capacity advertisement on login and credit-based mechanisms. Keep in mind that this is iSCSI-level flow control; the WG does not have a license to tinker with TCP's flow and congestion control. - The requirement for OPTIONAL support of multiple connection sessions has not gone away; the intent is to address it in a separate document in order to make progress on the main document. I'm assuming (and should have stated, sorry), that the main iSCSI document will continue to contain a negotiation mechanism sufficient to set up any of the connection models that have been proposed. My understanding is that contains sufficient "hooks" as that term has been used by Matt Wakeley and Michael Krause, but alternative views (e.g., we need to set aside n bytes of Reserved fields for future use in the headers now) are welcome. - Those who want to continue to debate multiple connection sessions on the list need to explain what will be different that will avoid spending another 6 weeks and not getting to consensus. In the absence of such an explanation, expect to see active discouragement of such traffic on the list. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Thu Sep 21 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31512; 20 Sep 2000 22:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA18500 for ips-outgoing; Wed, 20 Sep 2000 21:00:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA18493 for ; Wed, 20 Sep 2000 21:00:03 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id RAA05218; Wed, 20 Sep 2000 17:59:42 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Wed, 20 Sep 2000 17:59:41 -0700 Message-ID: From: Jim McGrath To: "'Douglas Otis'" , Jim McGrath , "'Randall Stewart'" Cc: "'Y P Cheng'" , "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Wed, 20 Sep 2000 17:59:38 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, I agreed that FCP offers some instructive ideas. I would like to decouple the allocation of initial credits from the login process (as per a previous message) and allow the target to really dynamically allocate them on a per initiator basis. On the general BB credit model, the only real issue I have there is that in FC the credits are for frames (in FC often 2K bytes) not bytes. I agree that commands can just be stored by the target like any other data, but there is big difference in size between a command frame and a user data frame. We don't need byte level granularity, but keeping the "credit unit" to something like 512 byes (or smaller) would allow for more efficient target memory management at modest controller complexity. Note you could still send things like 2K byte payloads, you just end up using 4 512 byte credits rather than a single frame credit. It was the coupling of 1 credit per frame, and then the need for large frames for efficient bus utilization that got us into trouble. In FC one objection to making a lot more smaller credits is the number of primitive tokens you would have to send (since each mapped to one credit), but here we need control packets anyway, so we can free up and use credits in bunches rather than individually (similar to what is done in FCP). In theory an initiator could send down multiple commands and then start sending down data sort of randomly between the commands, creating potential starvation issues. But no initiator that I know of does anything like that. I've never seen one that will send down some write data, jump to another command and send data, and then go back to the first command (maybe someone else has?). As long as the amount of data you can send with credits is smaller than the TCP window size, then you should never get starvation as far as I can tell (am I missing something)? Jim -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Wednesday, September 20, 2000 12:17 AM To: Jim McGrath; 'Randall Stewart' Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' Subject: RE: A Transport Protocol Without ACK Jim, I understand a desire to stick with what works. Regardless of the IP transport, the traffic to each LUN will require some other flow control mechanism. Yes, we could re-invent the wheel as it applies to SCSI, but if you examine FC Class 3 FCP, you will see an appropriate flow control mechanism in place. It uses Buffer-to-Buffer credit tokens generated by comma frame delimiters. These frame delimiters are defined within the FC-encapsulation documentation. Simple, direct and easy. See http://search.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-00.txt. To facilitate processing frame delimiters within software, both could be presented before the frame rather than as shown in the rough draft. Perhaps even a Null CRC option could be added if one trusts SCTP checksum for software implementations. Once flow control is in place, there is little need for extending command tags, CRN, or anything associated with the LUN as these structures are then independent of transport bandwidth. I doubt there is a great benefit in having more than 256 commands pending on a nexus. Expanding any field only makes converting to a drive interface state-full, difficult and far less reliable. With the FCP flow control mechanism, T10 does not need to redefine SCSI for initiators that overwhelm the target. The target would have adequate control of resources. Should IP-SCSI be driven by controller design? Caching, volume management, reservations, and nearly every feature offered by a controller is significantly reduced in value should the controller be placed next to the drive. If you are in a facility 35 miles from a location holding drives, you may find 50 miles of fiber transversed creating some 800+ micro-seconds of round-trip time simply due to the speed of light. You may shudder to think about any NIC buffer. Where would you want the controller and where would you want the drive? The controller must remain on the client side of the network. As such, drive design should steer the IP-SCSI standard. At least with FCP, the drive manufacturers have already spoken. Those making controllers will just have to make more of them and develop controller locking protocols should this controller be part of a remote cluster. If you examine FCP documentation, you will find that you can send data with the command as an option. You can also send the response at the end of data as an option. Every vital feature used to justify tossing FCP structures become moot. Should just an 8M byte FIFO buffer be placed between an IP agent and a FC agent, as much as 65 milli-seconds of latency can be created. Merely this additional latency will greatly facilitate rate differences between these two agents. FCP flow control and burst limits could easily finish the task. You speak of TCP as a proven technology, but TCP is not being suggested for IP-SCSI. TCP with some other mechanism is used to solve ills created by a persistent single byte stream. This is not proven technology, nor likely to function properly without major tweaking. At least if you wish to have a hand at creating a suitable API for multi-object-streams far and away more suitable for SCSI, now is the time. Perhaps either Randall Stewart's U-SCTP or a stale frame timer should be added to prevent overlapping retry mechanisms if this protocol is used as a bridge to FC. As far as the configuration effort, convert these requirements into LDAP structures. This would allow a single database to manage all aspects of configuration. Stuffing this information across the transport only weakens security. A bad idea and makes deciding who manages difficult. Networks will always have a means to identify equipment in some binary fashion, and LDAP and DHCP servers combine this information into meaningful structures with meaningful names. All values required for the various transport layers would be derived from these standard servers. As far as what to do with Stream 0- revision negotiations, FC-domain mapping, SRC-DST filtering done in purely binary form would be the best means at getting equipment to accept commands without a high overhead. The equipment does not care what the binary number represents. As far as a clever means of doing remote DNS, SCTP has that covered. Again, this information comes from an LDAP server accessed by the driver and not the SAM interface or the SCSI transport layer. Yes, there are many options within FC that should be excluded. If FCP structures can be used, perhaps while holding one's nose, they should be. There are far too many benefits for doing so, and too few benefits for not. In the end, a better product would have a common set of structures to speak SAN. If you wish to make round wheels out of square blocks, don't let me stop you. I think I see a set of wheels already. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jim McGrath > Sent: Tuesday, September 19, 2000 4:05 PM > To: 'Randall Stewart'; Jim McGrath > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > Subject: RE: A Transport Protocol Without ACK > > > > Actually the burden of proof issues is why I suggest we look at > some things > that are actually being used today (since you don't have to guess how they > behave). That is one of TCP's great strengths, and a bit of a > weakness for > SCTP (no offense to SCTP supporters, but it certainly does not have a big > and long "track record" yet, and so I can understand the concerns > others may > have as to whether things would work out as well in practice as they do in > proposal). > > Jim > > PS Personally, I'm a big believer is copying stuff that works, making the > minimum amount of required changes, and then doing a rapid but controlled > deployment (I've been involved in a lot of those sorts of things > in ATA and > SCSI). Having been involved in both these sorts of endevors and the > opposite (big, clean sheet of paper efforts, like 1394 (no offense to > 1394/Firewire supporters, but I was working on it a decade ago)), > I know how > easy it is to underestimate the work required by the latter, and to be > turned off by the "inelegance" of the former. For me, life has become too > short - I'm willing to accept inelegance as the price for speed of > deployment. > > > -----Original Message----- > From: Randall Stewart [mailto:rrs@cisco.com] > Sent: Tuesday, September 19, 2000 4:38 AM > To: Jim McGrath > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > Subject: Re: A Transport Protocol Without ACK > > > Jim: > > Any transport protocol proposal is ok. As long as it can be seen and > reviewed. So far I have seen only two TCP and SCTP. > > Oh, a little side note, any transport protocol proposed MUST be able to > show TCP like behavior in the face of congestion. And I think, IMHO, that > this means that if it is NOT using RFC2581 procedures it MUST show that > it does backoff and share with TCP. It also has a HEAVY burden of proof to > show this facility at least in my mind and I would think in the > IESG's mind > as well... > > R > > > Jim McGrath wrote: > > > I would expand your search to include non standard protocols (i.e. > > proprietary ones) as well if they offered something and were adequately > > understood by the outside world. We do that in storage quite a lot - > > indeed, some standard protocols are direct descendants of what were once > > proprietary protocols (e.g. ATA, the most widely used desktop disk > > interface, and ESCON, a dominant mainframe class interface > (both of which > > originated from IBM proprietary technologies)). > > > > Jim > > > > -----Original Message----- > > From: Y P Cheng [mailto:ycheng@advansys.com] > > Sent: Monday, September 18, 2000 5:52 PM > > To: 'Ips@Ece. Cmu. Edu' > > Subject: RE: A Transport Protocol Without ACK > > > > From: randall@stewart.chicago.il.us > > > I see no viable transport protocol here and I don't see this > > > conversation of any use unless you get exact details AND point > > > to a internet draft that defines EXACTLY how it works (or possibly > > > some other standards document). > > > > Both I2O and VI are transport protocols which define the format of a > request > > to a transport service provider, i.e. an adapter card. I2O is used but > not > > limited to deliver SCSI requests and VI is used for any payload > including > IP > > packets. VI is mapped into FC with the device headers between the FC > header > > and data payload. VI can certainly be used for delivery of > SCSI requests > > too. Both protocols require the service provider to have reliable > delivery > > and reception. VI defines different QoS. > > > > > > I don't claim any credit about this transport layer protocol. Every > > fibre > > > > channel and Infiniband adapter designer knows about this protocol -- > > > > although there is no standard. I am sure the TCP > accelerator card is > > doing > > > > the same. This protocol is a great alternative to the use of TCP/IP > and > > > > should be incorporated into iSCSI. > > > > > > No it is not. You are not offering an alternative yet.. > > > > I did not imply iSCSI should use I2O or VI. In fact, the > purpose iSCSI is > > to map SCSI requests into IP packets as well as to define the delivery . > It > > seems to me that the working group has set its mind on TCP/IP and is > > believing this is the only solution. The consensus seems if > there is any > > other solutions that address flow control and congestion, it > would end up > > like TCP/IP. I am simply pointing out if we keep an iSCSI request as a > > single atomic transaction without separating it into the > > TCP/IP-stream-oriented Writes and Reads that each deals with a > single DU, > > then, the deadlock problem goes away. While the work group thinks we > should > > take advantage the flow control and congestion management of > TCP/IP, there > > are alternatives known as BB-credit and EE-credit management. The fibre > > channel adapters make reliable delivery, lost packet detection, and > > retransmission without TCP/IP. > > > > Randall, you are right, I did not spent time to provide the > working group > a > > draft defining such transaction-oriented protocol. All I have > provided is > > an idea that besides TCP/IP. The designers for SCSI and fibre channel > > adapters have solved the head-of-queue blocking, the congestion, and > > retransmission problems. The transaction-oriented WRITE-REQUEST and > > READ-RESPONSE, in my humble opinion, allows us to implement > iSCSI simpler > > than that of WRITE and READ stream requests. The performance cost of > > requiring ACKs on every DU with size greater than MTU on a network with > long > > latency is very expensive.. By defining a greater ACK granularity is an > > attempt to solve this performance problem. If we do wish to > ACK on every > > DU, then, on a long latency network, we must have a method to stream the > > PDUs to ensure the performance. The method should not consume a large > > amount of memory space. One should never ignore the TCP/IP > memory-to-memory > > copy overhead when the backbone will be running at OC-192 speed in the > near > > future. Finally, please don't ever ask two NIC cards to > synchronize with > > each other. It is really hard to do as those of us in business of > designing > > NIC cards can testify. > > > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. > From owner-ips@ECE.cmu.edu Thu Sep 21 03:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31698; 20 Sep 2000 23:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA19806 for ips-outgoing; Wed, 20 Sep 2000 21:44:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA19799 for ; Wed, 20 Sep 2000 21:44:07 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 20 Sep 2000 18:45:45 -0700 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Wed, 20 Sep 2000 18:45:44 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, >be any more expensive and it is definately safer. Well one other >case can occur is if the initiator wants the target to issue an RTT >and it interposes another command before the RTT is received, this >too seems like a broken things to do as I don't see much value in doing RTT >on a single connection. Not doing RTT means each write command must be completed atomically before proceeding on to the next command. There will be some very large data PDU's hogging the single connection. How about task management functions which the initiator may want to deliver asynchronously? With a large data PDU stuck in the connection, (and commands stuck in the pipeline) it may require a more catastrophic abort/reset of the entire TCP connection, which might not have been originally necessary. Somehow, it seems to me that the SCSI folks put RTT in there for a purpose, and iSCSI would be losing something by eliminating it. I don't know exactly what this is, but I would think it includes AT LEAST some performance impact. Josh From owner-ips@ECE.cmu.edu Thu Sep 21 03:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31711; 20 Sep 2000 23:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA20198 for ips-outgoing; Wed, 20 Sep 2000 22:00:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA20194 for ; Wed, 20 Sep 2000 22:00:30 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 20 Sep 2000 19:01:56 -0700 Message-ID: From: Charles Monia To: David Robinson , ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Wed, 20 Sep 2000 19:01:55 -0700 Importance: low X-Priority: 5 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > Sent: Wednesday, September 20, 2000 3:56 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Flow Control > > > > > However, some have wanted to use the sliding window to > solve a different > > > problem. They want to enable the target to advertise to > the initiator how > > > many commands the target is able to receive. This > apparently is not > specified > > > in SAM-2, and SAM-2 deals with overflowing the command > queue in an ugly way > - > > > throw the commands away and enter an error state. > > > > > > The question here is whether iSCSI should attempt to > solve the command queue > > > overflow problem, or allow T10 to deal with it. Other > protocols (fibre > > > channel) already deal with it in the "ugly" fashion - > perhaps so should > > > iSCSI. What we need consensus on is whether or not iSCSI > is going to deal > > > with this. > > FC has a command queue overflow problem because it is a > datagram protocol > and cannot at the transport layer finely control the sender. Not true, but not relevant either -- take my word for it. >..............................................................But > with a reliable stream protocol I don't understand how you could > ever get a command queue overflow. If you command queue is full > you simply stop reading from the TCP stream which will eventually > cause the TCP input buffers to fill and the window to shrink to > zero causing flow to stop. Short of a naive implementation that > reads the TCP stream (thus keeping the window open) with no place > to put the command it just read, I still don't see the problem. > You're almost there. Since the TCP connection carries control traffic for all logical units, you don't want the flow of control information to stop due to the command queue on one LUN becoming full (a kind of head-of-line blocking). SAM gets around this by implementing what you refer to as the naive policy. i.e., it maintains continuous flow through the pipeline by dumping commands when the queue fills. By the way, even if the TCP connection only serviced a single LUN, you'd still want to keep the pipeline flowing to insure that SCSI task management commands would eventually get serviced. Charles From owner-ips@ECE.cmu.edu Thu Sep 21 04:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31996; 21 Sep 2000 0:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA21308 for ips-outgoing; Wed, 20 Sep 2000 22:45:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA21294 for ; Wed, 20 Sep 2000 22:44:41 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id TAA16880; Wed, 20 Sep 2000 19:43:52 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Wed, 20 Sep 2000 19:43:50 -0700 Message-ID: From: Jim McGrath To: "'David Robinson'" , ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Wed, 20 Sep 2000 19:43:50 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes, C1, C2, D1 is very common and is required. The reason is that the target can often dramatically improve throughput (different from latency) by executing commands out of order. So you want to give the target ALL of your queued commands (or up to a reasonable number, like 256) and let it drive the execution. Note that even the initial latency is quickly hidden if you really do have a lot of commands to execute. However, this is still not a problem as long as the window is big enough (i.e. more than the credits you have outstanding), and you took into account the buffer space needed for commands as well as data (as per a previous email). Since the target window and credits should reflect the same buffer constraints, I'm thinking that there may be a problem if the maximum window and the maximum credits are changeable at different times (i.e. it does not good to realize on the fly that your underutilized buffer can supports a lot more credits than you initially thought if changing the maximum window size takes too long to take advantage of these resources). Any thoughts as to whether this is an actual problem? (I am thinking of maximum changes occurring on the minute level time scale). Jim -----Original Message----- From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] Sent: Wednesday, September 20, 2000 5:13 PM To: ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control > I'm not a s/w engineer, so I could be getting myself into trouble here, > but I still see an issue.... > > iSCSI will not know whether the next PDU in the TCP buffer is a command > or data PDU until it actually reads the data. iSCSI should NEVER stop > reading > data from TCP or else deadlock if the next message is a data PDU (as was so > eloquently explained to me in a prior message). It must continue reading > data (thus emptying the TCP receive window), and if the SCSI command queue > is full, then SCSI will respond with TASK SET FULL. This works okay > (depending > on who you ask) in a parallel SCSI or FC world where latency is measured in > us, > but in iSCSI it could be 100ms and 1000+ commands later before the initiator > ever finds out. The TCP connection can have either a command or data. In either case reading a short fixed amount of data can determine if it is one or the other. With a single connection I can only see deadlock occuring if the initiator sends Command1, Command2, then Data1 on a single command/data connection. With a single connection this seems like a broken initiator as doing the right thing of C1, D1, C2 shouldn't be any more expensive and it is definately safer. Well one other case can occur is if the initiator wants the target to issue an RTT and it interposes another command before the RTT is received, this too seems like a broken things to do as I don't see much value in doing RTT on a single connection. In the multiple connection asymettric case this isn't an issue as data is never interleved with commands. So is there a fundamental requirement that the target must handle the case of C1, C2, then D1? It seems easy to just write the iSCSI initiator to never issue such a sequence and if the target gets one return an error when C2 arrives before D1. It again seems like we are trying to impose or solve datagram problems in a stream world. Does parallel SCSI have this problem? -David From owner-ips@ECE.cmu.edu Thu Sep 21 05:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32266; 21 Sep 2000 1:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA22553 for ips-outgoing; Wed, 20 Sep 2000 23:39:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA22546 for ; Wed, 20 Sep 2000 23:39:08 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id UAA12382; Wed, 20 Sep 2000 20:27:25 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id UAA10063; Wed, 20 Sep 2000 20:25:08 -0700 From: "Douglas Otis" To: "Jim McGrath" , "'Randall Stewart'" Cc: "'Y P Cheng'" , "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Wed, 20 Sep 2000 20:41:31 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, I agree drives are primarily used in sector units of 512 bytes, but even the lowly CD use 2048 and include the headers. A header that targets the initiator could act to exchange credits. The danger in not keeping units confined to FC structures means there could end up a disagreement. A signal when login credits are available (in case the initiator becomes confused) should be a means to reconcile accounts. The same signal echoed (Now at Login Credit) could act as acknowledgement. The PDU used to acknowledge would then be the PDU size below login. Rather than using a single token, a token count could act to dynamically allocate additional credit in addition to acknowledging used credit being returned. If you signal login credit, should there be an over allocation and not just a reminder, waiting for acknowledgement does not seem onerous. At least, things stay compatible. 512 or 2k, hard to decide. To match FC, it would be 2k. What is 1536 bytes between friends? Doug > -----Original Message----- > From: Jim McGrath [mailto:Jim.McGrath@quantum.com] > Sent: Wednesday, September 20, 2000 6:00 PM > To: 'Douglas Otis'; Jim McGrath; 'Randall Stewart' > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > Subject: RE: A Transport Protocol Without ACK > > > > Doug, > > I agreed that FCP offers some instructive ideas. I would like to decouple > the allocation of initial credits from the login process (as per > a previous > message) and allow the target to really dynamically allocate them on a per > initiator basis. > > On the general BB credit model, the only real issue I have there > is that in > FC the credits are for frames (in FC often 2K bytes) not bytes. I agree > that commands can just be stored by the target like any other data, but > there is big difference in size between a command frame and a user data > frame. We don't need byte level granularity, but keeping the > "credit unit" > to something like 512 byes (or smaller) would allow for more efficient > target memory management at modest controller complexity. Note you could > still send things like 2K byte payloads, you just end up using 4 512 byte > credits rather than a single frame credit. It was the coupling > of 1 credit > per frame, and then the need for large frames for efficient bus > utilization > that got us into trouble. > > In FC one objection to making a lot more smaller credits is the number of > primitive tokens you would have to send (since each mapped to one credit), > but here we need control packets anyway, so we can free up and use credits > in bunches rather than individually (similar to what is done in FCP). > > In theory an initiator could send down multiple commands and then start > sending down data sort of randomly between the commands, creating > potential > starvation issues. But no initiator that I know of does anything > like that. > I've never seen one that will send down some write data, jump to another > command and send data, and then go back to the first command > (maybe someone > else has?). As long as the amount of data you can send with credits is > smaller than the TCP window size, then you should never get starvation as > far as I can tell (am I missing something)? > > Jim > > > > > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Wednesday, September 20, 2000 12:17 AM > To: Jim McGrath; 'Randall Stewart' > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > Subject: RE: A Transport Protocol Without ACK > > > Jim, > > I understand a desire to stick with what works. Regardless of the IP > transport, the traffic to each LUN will require some other flow control > mechanism. Yes, we could re-invent the wheel as it applies to > SCSI, but if > you examine FC Class 3 FCP, you will see an appropriate flow control > mechanism in place. It uses Buffer-to-Buffer credit tokens generated by > comma frame delimiters. These frame delimiters are defined within the > FC-encapsulation documentation. Simple, direct and easy. See > http://search.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-00.txt. To > facilitate processing frame delimiters within software, both could be > presented before the frame rather than as shown in the rough > draft. Perhaps > even a Null CRC option could be added if one trusts SCTP checksum for > software implementations. > > Once flow control is in place, there is little need for extending command > tags, CRN, or anything associated with the LUN as these > structures are then > independent of transport bandwidth. I doubt there is a great benefit in > having more than 256 commands pending on a nexus. Expanding any > field only > makes converting to a drive interface state-full, difficult and far less > reliable. With the FCP flow control mechanism, T10 does not need to > redefine SCSI for initiators that overwhelm the target. The target would > have adequate control of resources. > > Should IP-SCSI be driven by controller design? Caching, volume > management, > reservations, and nearly every feature offered by a controller is > significantly reduced in value should the controller be placed next to the > drive. If you are in a facility 35 miles from a location holding drives, > you may find 50 miles of fiber transversed creating some 800+ > micro-seconds > of round-trip time simply due to the speed of light. You may shudder to > think about any NIC buffer. Where would you want the controller and where > would you want the drive? The controller must remain on the > client side of > the network. As such, drive design should steer the IP-SCSI standard. At > least with FCP, the drive manufacturers have already spoken. Those making > controllers will just have to make more of them and develop controller > locking protocols should this controller be part of a remote cluster. > > If you examine FCP documentation, you will find that you can send > data with > the command as an option. You can also send the response at the > end of data > as an option. Every vital feature used to justify tossing FCP structures > become moot. Should just an 8M byte FIFO buffer be placed between an IP > agent and a FC agent, as much as 65 milli-seconds of latency can > be created. > Merely this additional latency will greatly facilitate rate differences > between these two agents. FCP flow control and burst limits could easily > finish the task. > > You speak of TCP as a proven technology, but TCP is not being > suggested for > IP-SCSI. TCP with some other mechanism is used to solve ills created by a > persistent single byte stream. This is not proven technology, > nor likely to > function properly without major tweaking. At least if you wish to have a > hand at creating a suitable API for multi-object-streams far and away more > suitable for SCSI, now is the time. Perhaps either Randall > Stewart's U-SCTP > or a stale frame timer should be added to prevent overlapping retry > mechanisms if this protocol is used as a bridge to FC. > > As far as the configuration effort, convert these requirements into LDAP > structures. This would allow a single database to manage all aspects of > configuration. Stuffing this information across the transport > only weakens > security. A bad idea and makes deciding who manages difficult. Networks > will always have a means to identify equipment in some binary fashion, and > LDAP and DHCP servers combine this information into meaningful structures > with meaningful names. All values required for the various > transport layers > would be derived from these standard servers. > > As far as what to do with Stream 0- revision negotiations, FC-domain > mapping, SRC-DST filtering done in purely binary form would be the best > means at getting equipment to accept commands without a high > overhead. The > equipment does not care what the binary number represents. As far as a > clever means of doing remote DNS, SCTP has that covered. Again, this > information comes from an LDAP server accessed by the driver and > not the SAM > interface or the SCSI transport layer. > > Yes, there are many options within FC that should be excluded. If FCP > structures can be used, perhaps while holding one's nose, they should be. > There are far too many benefits for doing so, and too few > benefits for not. > In the end, a better product would have a common set of > structures to speak > SAN. If you wish to make round wheels out of square blocks, don't let me > stop you. I think I see a set of wheels already. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Jim McGrath > > Sent: Tuesday, September 19, 2000 4:05 PM > > To: 'Randall Stewart'; Jim McGrath > > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > > Subject: RE: A Transport Protocol Without ACK > > > > > > > > Actually the burden of proof issues is why I suggest we look at > > some things > > that are actually being used today (since you don't have to > guess how they > > behave). That is one of TCP's great strengths, and a bit of a > > weakness for > > SCTP (no offense to SCTP supporters, but it certainly does not > have a big > > and long "track record" yet, and so I can understand the concerns > > others may > > have as to whether things would work out as well in practice as > they do in > > proposal). > > > > Jim > > > > PS Personally, I'm a big believer is copying stuff that works, > making the > > minimum amount of required changes, and then doing a rapid but > controlled > > deployment (I've been involved in a lot of those sorts of things > > in ATA and > > SCSI). Having been involved in both these sorts of endevors and the > > opposite (big, clean sheet of paper efforts, like 1394 (no offense to > > 1394/Firewire supporters, but I was working on it a decade ago)), > > I know how > > easy it is to underestimate the work required by the latter, and to be > > turned off by the "inelegance" of the former. For me, life has > become too > > short - I'm willing to accept inelegance as the price for speed of > > deployment. > > > > > > -----Original Message----- > > From: Randall Stewart [mailto:rrs@cisco.com] > > Sent: Tuesday, September 19, 2000 4:38 AM > > To: Jim McGrath > > Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' > > Subject: Re: A Transport Protocol Without ACK > > > > > > Jim: > > > > Any transport protocol proposal is ok. As long as it can be seen and > > reviewed. So far I have seen only two TCP and SCTP. > > > > Oh, a little side note, any transport protocol proposed MUST be able to > > show TCP like behavior in the face of congestion. And I think, > IMHO, that > > this means that if it is NOT using RFC2581 procedures it MUST show that > > it does backoff and share with TCP. It also has a HEAVY burden > of proof to > > show this facility at least in my mind and I would think in the > > IESG's mind > > as well... > > > > R > > > > > > Jim McGrath wrote: > > > > > I would expand your search to include non standard protocols (i.e. > > > proprietary ones) as well if they offered something and were > adequately > > > understood by the outside world. We do that in storage quite a lot - > > > indeed, some standard protocols are direct descendants of > what were once > > > proprietary protocols (e.g. ATA, the most widely used desktop disk > > > interface, and ESCON, a dominant mainframe class interface > > (both of which > > > originated from IBM proprietary technologies)). > > > > > > Jim > > > > > > -----Original Message----- > > > From: Y P Cheng [mailto:ycheng@advansys.com] > > > Sent: Monday, September 18, 2000 5:52 PM > > > To: 'Ips@Ece. Cmu. Edu' > > > Subject: RE: A Transport Protocol Without ACK > > > > > > From: randall@stewart.chicago.il.us > > > > I see no viable transport protocol here and I don't see this > > > > conversation of any use unless you get exact details AND point > > > > to a internet draft that defines EXACTLY how it works (or possibly > > > > some other standards document). > > > > > > Both I2O and VI are transport protocols which define the format of a > > request > > > to a transport service provider, i.e. an adapter card. I2O > is used but > > not > > > limited to deliver SCSI requests and VI is used for any payload > > including > > IP > > > packets. VI is mapped into FC with the device headers between the FC > > header > > > and data payload. VI can certainly be used for delivery of > > SCSI requests > > > too. Both protocols require the service provider to have reliable > > delivery > > > and reception. VI defines different QoS. > > > > > > > > I don't claim any credit about this transport layer > protocol. Every > > > fibre > > > > > channel and Infiniband adapter designer knows about this > protocol -- > > > > > although there is no standard. I am sure the TCP > > accelerator card is > > > doing > > > > > the same. This protocol is a great alternative to the > use of TCP/IP > > and > > > > > should be incorporated into iSCSI. > > > > > > > > No it is not. You are not offering an alternative yet.. > > > > > > I did not imply iSCSI should use I2O or VI. In fact, the > > purpose iSCSI is > > > to map SCSI requests into IP packets as well as to define the > delivery . > > It > > > seems to me that the working group has set its mind on TCP/IP and is > > > believing this is the only solution. The consensus seems if > > there is any > > > other solutions that address flow control and congestion, it > > would end up > > > like TCP/IP. I am simply pointing out if we keep an iSCSI > request as a > > > single atomic transaction without separating it into the > > > TCP/IP-stream-oriented Writes and Reads that each deals with a > > single DU, > > > then, the deadlock problem goes away. While the work group thinks we > > should > > > take advantage the flow control and congestion management of > > TCP/IP, there > > > are alternatives known as BB-credit and EE-credit management. > The fibre > > > channel adapters make reliable delivery, lost packet detection, and > > > retransmission without TCP/IP. > > > > > > Randall, you are right, I did not spent time to provide the > > working group > > a > > > draft defining such transaction-oriented protocol. All I have > > provided is > > > an idea that besides TCP/IP. The designers for SCSI and fibre channel > > > adapters have solved the head-of-queue blocking, the congestion, and > > > retransmission problems. The transaction-oriented WRITE-REQUEST and > > > READ-RESPONSE, in my humble opinion, allows us to implement > > iSCSI simpler > > > than that of WRITE and READ stream requests. The performance cost of > > > requiring ACKs on every DU with size greater than MTU on a > network with > > long > > > latency is very expensive.. By defining a greater ACK > granularity is an > > > attempt to solve this performance problem. If we do wish to > > ACK on every > > > DU, then, on a long latency network, we must have a method to > stream the > > > PDUs to ensure the performance. The method should not consume a large > > > amount of memory space. One should never ignore the TCP/IP > > memory-to-memory > > > copy overhead when the backbone will be running at OC-192 speed in the > > near > > > future. Finally, please don't ever ask two NIC cards to > > synchronize with > > > each other. It is really hard to do as those of us in business of > > designing > > > NIC cards can testify. > > > > > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. > > > From owner-ips@ECE.cmu.edu Thu Sep 21 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00050; 21 Sep 2000 2:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA25875 for ips-outgoing; Thu, 21 Sep 2000 01:49:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25871 for ; Thu, 21 Sep 2000 01:49:36 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id 2BCC77BB for ; Thu, 21 Sep 2000 01:49:35 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA06026 for ; Wed, 20 Sep 2000 22:49:33 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA3FA1 for ; Wed, 20 Sep 2000 22:49:30 -0700 Message-ID: <39C9A159.3EB1193B@agilent.com> Date: Wed, 20 Sep 2000 22:49:13 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 Cc: "Reflector, IPS" Subject: Re: iSCSI: Flow Control References: <200009202250.PAA26743@ha10nwk.EBay.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David Robinson wrote: > > The latter case could overwhelm the > > SCSI command queue, but still be within the TCP window of > > the receiving node. > > This I don't understand. The act of TCP receiving data into > its window (and thus causing the window to shrink) is > complete independant of the application (aka iSCSI) grabbing > the data (causing the window to grow). As an extreme example > a naught initiator sends 1,000 commands down to a target that > has a command queue depth of 2, will it be overwhelmed? I think > not. And let's also stipulate that there is only one TCP connection, and that the commands are all write commands... > A sane implementation of the target will detect that there > is data on the connection, read a chunk large enough to grab > the first command and stuff it in the command queue, read the > next command and stuff it in the command queue, then simply > waits for one of the commands to finish leaving the other 998 > commands in the TCP receive buffers. When one of the commands > completes it sucks in the next command and puts it in the queue. Ok, so now the SCSI processes the first command, and sends an RTT (XFER_RDY in FC terms) to the initiator. Now, the initiator sends the data down the same TCP connection, and it gets stuck behind all those 998 commands in the TCP receive buffers. The command can't complete because it can't get the data, and the data can't be delivered because there's no room for the commands in front of it. Deadlock. Do you see the issue now? (this is a good example of why the single TCP connection model, be it synchronous or asynchronous, is bad). -Matt > > > That is, unless you want iSCSI to talk > > to TCP to tell it to close the window. But that would > > violate layering principles, I believe. > > The above example violates not layering, the TCP window closes because > the application does not read all of the data because its command > queue is too small. Now a clever implementation can do interesting > tricks in hardware where it copies the data stream straight into > the command queue based on its interpretation of the data stream. > The overly clever (and thus dangerous) implementation could also > play with the window and not open it up until *after* the command > is executed. But again this is an implementation hack that > saves a copy but then violates layering, but the proposed protocol > does not violate layering. > > Still don't see the issue. > > -David > From owner-ips@ECE.cmu.edu Thu Sep 21 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00039; 21 Sep 2000 2:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA26132 for ips-outgoing; Thu, 21 Sep 2000 02:00:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA26125 for ; Thu, 21 Sep 2000 02:00:02 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id XAA04244 for ; Wed, 20 Sep 2000 23:00:01 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.154.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id XAA04368 for ; Wed, 20 Sep 2000 23:00:01 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id XAA23760; Wed, 20 Sep 2000 23:00:00 -0700 (PDT) Message-ID: <39C9A3E1.F264C83F@ebay.sun.com> Date: Wed, 20 Sep 2000 23:00:01 -0700 From: David Robinson X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: Flow Control References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua Tseng wrote: > Not doing RTT means each write command must be completed atomically > before proceeding on to the next command. There will be some very > large data PDU's hogging the single connection. How about task > management functions which the initiator may want to deliver > asynchronously? With a large data PDU stuck in the connection, > (and commands stuck in the pipeline) it may require a more > catastrophic abort/reset of the entire TCP connection, which might > not have been originally necessary. > > Somehow, it seems to me that the SCSI folks put RTT in there for > a purpose, and iSCSI would be losing something by eliminating it. > I don't know exactly what this is, but I would think it includes > AT LEAST some performance impact. I understand the basis of RTT is in the SCSI legacy of targets with very small buffers where limits might be in the order of single digits. With modern buffers and cheap memory I question its need. -David From owner-ips@ECE.cmu.edu Thu Sep 21 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00055; 21 Sep 2000 2:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA25589 for ips-outgoing; Thu, 21 Sep 2000 01:38:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25585 for ; Thu, 21 Sep 2000 01:37:59 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel2.hp.com (Postfix) with ESMTP id 99D07106A for ; Thu, 21 Sep 2000 01:37:58 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id WAA05898 for ; Wed, 20 Sep 2000 22:37:57 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA3F87 for ; Wed, 20 Sep 2000 22:37:51 -0700 Message-ID: <39C99EA3.9993E347@agilent.com> Date: Wed, 20 Sep 2000 22:37:39 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: "Reflector, IPS" Subject: Re: iSCSI: Session Partial Resolution References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > Matt, > > > > The "iSCSI Requirements" document requires this capability: > > > > "[R] High bandwidth, bandwidth aggregation. > > This is a general feature of IP. No, it's not. Please go back and read Randy Haagens' memo from August 5, available here http://ips.pdl.cs.cmu.edu/mail/msg00412.html > > > > [D] The bandwidth (transfer rate, MB/sec) supported by storage > > controllers is rapidly increasing, due to several factors: (1) > > Increase in disk spindle and controller performance; > > The linear increase in the performance of the mechanical aspects of the > drive have not kept pace with the exponential data density improvements. > Unless dealing with a single massive transfer, the overall effect of data > density improvements is a small reduction in latency lost by the large > latency within a MAN network. We are not talking about drive controllers, we're talking about storage array (raid) controllers, where there is lots of drives behind it. I don't think anyone considers attaching individual disk drives to the internet (we'd very quickly run out of IP addresses). > > > > (2) Use > > of ever-larger caches, and improved caching algorithms; > > Network latency negates use of remote caching. Such caching must be near > the client and not the drive. If you wish to define a controller > specification, this should be done using appropriate technologies. Clearly, > if the controller is on site, interface requirements change substantially. > For redundancy, there may even be one per client system! Now the traffic > that needs some work would be to ensure such controllers stay synchronized > if used in a cluster. Either way, you will not see a higher performance > moving the controller next to the drive. > > > (3) > > Increased scale of storage controllers (number of supported > > spindles, speed of interconnects). Not only must the iSCSI > > provide for full utilization of available link bandwidth, it > > also must exploit parallelism (multiple connections) at the > > device interfaces and within the interconnect fabric." > > Again, this is a general feature of IP. If access is to the drive, as it > should be, there is no problem scaling especially if you are using IP. > > > Unless there is "consensus" that this is not required, I think > > the multiple > > connections/session must remain. > > Not one reason for forcing aggregation into a single connection. If you > wish to bind multiple adapters, IP supports this. I doubt you will ever > need to do this however. I'm not going to try and convince you of the need for iSCSI link aggregation, and how IP does not support this. Bottom line is that major system and storage vendors such as HP think we need it. Once again, re-read Randy's message. > > > Doug -Matt From owner-ips@ECE.cmu.edu Thu Sep 21 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00057; 21 Sep 2000 2:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA26314 for ips-outgoing; Thu, 21 Sep 2000 02:07:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA26309 for ; Thu, 21 Sep 2000 02:06:59 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id XAA05497 for ; Wed, 20 Sep 2000 23:06:58 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.188.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id XAA13519 for ; Wed, 20 Sep 2000 23:06:58 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id XAA27174; Wed, 20 Sep 2000 23:06:56 -0700 (PDT) Message-ID: <39C9A582.BE76A0C1@ebay.sun.com> Date: Wed, 20 Sep 2000 23:06:58 -0700 From: David Robinson X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Reflector, IPS" Subject: Re: iSCSI: Flow Control References: <200009202250.PAA26743@ha10nwk.EBay.Sun.COM> <39C9A159.3EB1193B@agilent.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt Wakeley wrote: > Ok, so now the SCSI processes the first command, and sends an RTT (XFER_RDY in > FC terms) to the initiator. Now, the initiator sends the data down the same > TCP connection, and it gets stuck behind all those 998 commands in the TCP > receive buffers. The command can't complete because it can't get the data, > and the data can't be delivered because there's no room for the commands in > front of it. Deadlock. Do you see the issue now? (this is a good example of > why the single TCP connection model, be it synchronous or asynchronous, is > bad). As I said in other e-mail I question the use of RTT in a modern environment with large buffering. Data immediately following the command makes much better sense to me, but if we must support this environment then you are right that deadlock can occur. -David From owner-ips@ECE.cmu.edu Thu Sep 21 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00065; 21 Sep 2000 2:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA25982 for ips-outgoing; Thu, 21 Sep 2000 01:54:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA25977 for ; Thu, 21 Sep 2000 01:53:56 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id WAA03065 for ; Wed, 20 Sep 2000 22:53:55 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.154.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id WAA03913 for ; Wed, 20 Sep 2000 22:53:55 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id WAA21211; Wed, 20 Sep 2000 22:53:54 -0700 (PDT) Message-ID: <39C9A273.F8BA5F1@ebay.sun.com> Date: Wed, 20 Sep 2000 22:53:55 -0700 From: David Robinson X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: Flow Control References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > You're almost there. Since the TCP connection carries control traffic for > all logical units, you don't want the flow of control information to stop > due to the command queue on one LUN becoming full (a kind of head-of-line > blocking). SAM gets around this by implementing what you refer to as the > naive policy. i.e., it maintains continuous flow through the pipeline by > dumping commands when the queue fills. I have always been advocating a connection per LUN so this problem seems to reinforce my belief. But I agree that if the WG wants multiple LUNs per connection this problem needs to be addressed. My lack of understanding comes from my NFS background where NFS over TCP in practice doesn't have a flow control problem even though it is common to mux multiple mount points over a single connection. The simple reason why this is not a problem is because the initiator (client) doesn't issue more requests to the target (export point) than it can buffer. The question is if the protocol expects the initiator to send more commands than a LUN can queue. If so this can be defended against either statically at session establishment (assumes command queues lengths are static) or dynamic flow control. My misunderstanding seems to be rooted in a different philosophy between the SCSI model and the distributed filesystem model. -David From owner-ips@ECE.cmu.edu Thu Sep 21 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00369; 21 Sep 2000 4:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA26971 for ips-outgoing; Thu, 21 Sep 2000 02:40:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA26966 for ; Thu, 21 Sep 2000 02:40:34 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA32206; Thu, 21 Sep 2000 08:40:02 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA20048; Thu, 21 Sep 2000 08:40:01 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256961.00249EA7 ; Thu, 21 Sep 2000 08:39:59 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: David Robinson , ips@ece.cmu.edu Message-ID: Date: Thu, 21 Sep 2000 09:39:50 +0300 Subject: Re: iSCSI: Flow Control Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Suppose we have many TCP connections using the symmetric model and suppose that one of the commands is delayed because some packet was dropped. Suppose further that we want to maintain order of delivery (using the command reference numbers). Commands arrive on the other TCP connections and fill up the command queue while we are waiting for the missing comand. (Without using Pierre's suggestion) we don't know on which TCP connection the missing command will arrive, so we have to keep reading commands from the TCP connections in order to get to the missing command. Note that we only have to read a single command from each TCP connection until we find the missing command. - Kalman Meth David Robinson on 21/09/2000 01:56:28 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Kalman Meth/Haifa/IBM) Subject: Re: iSCSI: Flow Control > > However, some have wanted to use the sliding window to solve a different > > problem. They want to enable the target to advertise to the initiator how > > many commands the target is able to receive. This apparently is not specified > > in SAM-2, and SAM-2 deals with overflowing the command queue in an ugly way - > > throw the commands away and enter an error state. > > > > The question here is whether iSCSI should attempt to solve the command queue > > overflow problem, or allow T10 to deal with it. Other protocols (fibre > > channel) already deal with it in the "ugly" fashion - perhaps so should > > iSCSI. What we need consensus on is whether or not iSCSI is going to deal > > with this. FC has a command queue overflow problem because it is a datagram protocol and cannot at the transport layer finely control the sender. But with a reliable stream protocol I don't understand how you could ever get a command queue overflow. If you command queue is full you simply stop reading from the TCP stream which will eventually cause the TCP input buffers to fill and the window to shrink to zero causing flow to stop. Short of a naive implementation that reads the TCP stream (thus keeping the window open) with no place to put the command it just read, I still don't see the problem. -David From owner-ips@ECE.cmu.edu Thu Sep 21 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00687; 21 Sep 2000 4:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA28415 for ips-outgoing; Thu, 21 Sep 2000 03:36:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA28409 for ; Thu, 21 Sep 2000 03:36:14 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id 6EBFA800 for ; Thu, 21 Sep 2000 03:36:13 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id AAA10314 for ; Thu, 21 Sep 2000 00:36:12 -0700 (PDT) Received: from agilent.com ([15.26.30.186]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA4130 for ; Thu, 21 Sep 2000 00:36:07 -0700 Message-ID: <39C9BA5B.15273500@agilent.com> Date: Thu, 21 Sep 2000 00:35:55 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: "Reflector, IPS" Subject: Re: iSCSI: Flow Control References: <200009202250.PAA26743@ha10nwk.EBay.Sun.COM> <39C9A159.3EB1193B@agilent.com> <39C9A582.BE76A0C1@ebay.sun.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Well, "modern" disk arrays imployed on SCSI and FC use XFER_RDYs (RTT). They don't seem to question it. -Matt David Robinson wrote: > Matt Wakeley wrote: > > > Ok, so now the SCSI processes the first command, and sends an RTT (XFER_RDY in > > FC terms) to the initiator. Now, the initiator sends the data down the same > > TCP connection, and it gets stuck behind all those 998 commands in the TCP > > receive buffers. The command can't complete because it can't get the data, > > and the data can't be delivered because there's no room for the commands in > > front of it. Deadlock. Do you see the issue now? (this is a good example of > > why the single TCP connection model, be it synchronous or asynchronous, is > > bad). > > As I said in other e-mail I question the use of RTT in a modern > environment with large buffering. Data immediately following the > command makes much better sense to me, but if we must support > this environment then you are right that deadlock can occur. > > -David From owner-ips@ECE.cmu.edu Thu Sep 21 09:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01038; 21 Sep 2000 5:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA29633 for ips-outgoing; Thu, 21 Sep 2000 04:21:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from radmail.rad.co.il (radmail.rad.co.il [207.232.32.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA29629 for ; Thu, 21 Sep 2000 04:21:32 -0400 (EDT) Received: from sanrad.com ([192.114.26.219]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-54115U1200L1200S0V35) with ESMTP id il; Thu, 21 Sep 2000 11:24:26 +0300 Message-ID: <39C9E126.EE9170C9@sanrad.com> Date: Thu, 21 Sep 2000 11:21:26 +0100 From: Yaron Klein Organization: SANRAD X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: Joshua Tseng CC: ips@ece.cmu.edu Subject: Re: iSCSI Security Protocol References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Joshua, Mark Bakke, Kalman Meth, Costa and I are now working on the new scheme of security within the iSCSI. The key points so far are: For full security (authentication and encryption) use external protocol, e.g., IPsec. You can define an IPsec policy for encrypting everything (not feasible for most cases) or just the first 48 bytes (headers) and so on. However, IPsec may cause some problems since it is IP oriented (connection oriented and not session oriented). Moreover, you are forcing the client to have IPsec, which is not always true. The security scheme in the iSCSI draft includes authorization and authentication. The authorization is done in the login phase with the negotiation (detailed in the draft), and authentication is achieved by a trailer that checks the integrity of the data and the header (either simple CRC or some mac algorithm). Everything is flexible and negotiable. I hope we release the new draft very soon. Regards, Yaron Joshua Tseng wrote: > I just did a brief review of the document draft-klein-iscsi-security-00.txt. > What is the current consensus (if any) on this document? Is there > agreement to use SSH as the security mechanism for iSCSI? > > Josh From owner-ips@ECE.cmu.edu Thu Sep 21 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01956; 21 Sep 2000 8:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA04779 for ips-outgoing; Thu, 21 Sep 2000 07:35:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA04774 for ; Thu, 21 Sep 2000 07:35:24 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id GAA85952 for ; Thu, 21 Sep 2000 06:29:09 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8LBY6j33848 for ; Thu, 21 Sep 2000 05:34:06 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: Flow Control To: "Reflector, IPS" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Thu, 21 Sep 2000 04:31:30 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/21/2000 05:34:05 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, I am not sure that you answered David's Question. Even if the SCSI layer uses RTT, what if ISCSI did not? What would be the result? . . . John L. Hufferd "Matt Wakeley" @ece.cmu.edu on 09/21/2000 12:35:55 AM Please respond to Matt Wakeley Sent by: owner-ips@ece.cmu.edu To: "Reflector, IPS" cc: Subject: Re: iSCSI: Flow Control Well, "modern" disk arrays imployed on SCSI and FC use XFER_RDYs (RTT). They don't seem to question it. -Matt David Robinson wrote: > Matt Wakeley wrote: > > > Ok, so now the SCSI processes the first command, and sends an RTT (XFER_RDY in > > FC terms) to the initiator. Now, the initiator sends the data down the same > > TCP connection, and it gets stuck behind all those 998 commands in the TCP > > receive buffers. The command can't complete because it can't get the data, > > and the data can't be delivered because there's no room for the commands in > > front of it. Deadlock. Do you see the issue now? (this is a good example of > > why the single TCP connection model, be it synchronous or asynchronous, is > > bad). > > As I said in other e-mail I question the use of RTT in a modern > environment with large buffering. Data immediately following the > command makes much better sense to me, but if we must support > this environment then you are right that deadlock can occur. > > -David From owner-ips@ECE.cmu.edu Thu Sep 21 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03060; 21 Sep 2000 12:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA14291 for ips-outgoing; Thu, 21 Sep 2000 11:54:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA14287 for ; Thu, 21 Sep 2000 11:54:47 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Thu, 21 Sep 2000 08:56:29 -0700 Message-ID: From: Joshua Tseng To: Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: iSCSI Security Protocol Date: Thu, 21 Sep 2000 08:56:28 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi Yaron, >For full security (authentication and encryption) use external protocol, e.g., >IPsec. You can define an IPsec policy for encrypting everything (not feasible >for most cases) or just the first 48 bytes (headers) and so on. I agree we should look at IPSec. Today, all SSH implementations are software-based as far as I know. IPSec hardware is much more readily available. >However, IPsec may cause some problems since it is IP oriented (connection >oriented and not session oriented). Moreover, you are forcing the client to have >IPsec, which is not always true. I'm not sure what you mean here. IPSec is a layer-3 protocol, and it has no knowledge of TCP. Conversely, TCP is unaware of IPSec operating at the layer below it, and IPSec should not interfere at all with TCP. The only drawback I see is that if you selectively apply a security policy to iSCSI conversations which may go over the same TCP connection, some TCP segments may be encrypted and others will not. For example, a policy may dictate no security for one iSCSI conversation, but ESP tunneling w/3DES for a different iSCSI conversation. Both conversations use the same TCP connection. If this happens, you will not see a difference at the TCP end points, but in the network you'll see some segments in a TCP connection "disappear", as they have been encrypted, while others may be left alone and visible to the network, IP and TCP headers and all. This effect may confuse firewalls and other monitoring points in the network. But I would rather have the flexibility to do this than to be forced to apply a single uniform security policy to all iSCSI traffic using a TCP connection. I also don't know what you mean by "forcing the client to have IPSec". If the client doesn't have IPSec, this can be negotiated out at iSCSI login. Or, if you want to do authentication before iSCSI login, IKE (Internet Key Exchange) can easily be implemented in software. >The security scheme in the iSCSI draft includes authorization and >authentication. The authorization is done in the login phase with the >negotiation (detailed in the draft), and authentication is achieved by a trailer >that checks the integrity of the data and the header (either simple CRC or some >mac algorithm). So it seems you do not intend to leverage the Authentication Headers (AH) protocol and will imbed the authentication & data integrity mechanism within the iSCSI protocol? Regards, Josh From owner-ips@ECE.cmu.edu Thu Sep 21 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03049; 21 Sep 2000 12:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA13164 for ips-outgoing; Thu, 21 Sep 2000 11:29:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ipperformance.com ([206.225.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA13157 for ; Thu, 21 Sep 2000 11:29:48 -0400 (EDT) From: rbg@ipperformance.com Received: from rbg.gilhooley.com ([10.1.0.66]) by ipperformance.com (8.9.3/8.9.3) with ESMTP id KAA64492; Thu, 21 Sep 2000 10:29:23 -0500 (CDT) (envelope-from rbg@ipperformance.com) Received: from localhost (localhost [127.0.0.1]) by rbg.gilhooley.com (8.9.3/8.9.3) with ESMTP id KAA07223; Thu, 21 Sep 2000 10:28:55 -0500 (CDT) (envelope-from rbg@ipperformance.com) Date: Thu, 21 Sep 2000 10:28:55 -0500 (CDT) Message-Id: <20000921.102855.102249224.rbg@ipperformance.com> To: matt_wakeley@agilent.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI: Flow Control In-Reply-To: <39C9BA5B.15273500@agilent.com> References: <39C9A159.3EB1193B@agilent.com> <39C9A582.BE76A0C1@ebay.sun.com> <39C9BA5B.15273500@agilent.com> X-Mailer: Mew version 1.95b43 on Emacs 20.4 / Mule 4.0 (HANANOEN) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk In "modern" disk arrays, what problem does XFER_RDYs solve (or is it a feature) ? Is there a concept of "command+immediate data" for these disk arrays ? Just as a general question.. do we need to comprehend and accomadate all legacy issues ? Matt, Are you suggesting that iSCSI should be limited to disk arrays ? > On Thu, 21 Sep 2000 00:35:55 -0700 you said: > Well, "modern" disk arrays imployed on SCSI and FC use XFER_RDYs (RTT). They don't > seem to question it. > > -Matt > > David Robinson wrote: > > > Matt Wakeley wrote: > > > > > Ok, so now the SCSI processes the first command, and sends an RTT (XFER_RDY in > > > FC terms) to the initiator. Now, the initiator sends the data down the same > > > TCP connection, and it gets stuck behind all those 998 commands in the TCP > > > receive buffers. The command can't complete because it can't get the data, > > > and the data can't be delivered because there's no room for the commands in > > > front of it. Deadlock. Do you see the issue now? (this is a good example of > > > why the single TCP connection model, be it synchronous or asynchronous, is > > > bad). > > > > As I said in other e-mail I question the use of RTT in a modern > > environment with large buffering. Data immediately following the > > command makes much better sense to me, but if we must support > > this environment then you are right that deadlock can occur. > > > > -David > _____________________________________________________________ Robert Gordon rbg@ipperf.com IP Performance, Inc Austin, Texas. From owner-ips@ECE.cmu.edu Thu Sep 21 17:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03079; 21 Sep 2000 13:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA13425 for ips-outgoing; Thu, 21 Sep 2000 11:34:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lsi.lsil.com (lsi.lsil.com [147.145.40.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA13402 for ; Thu, 21 Sep 2000 11:34:42 -0400 (EDT) Received: from mhbs.lsil.com ([147.145.31.100]) by lsi.lsil.com (8.9.3+Sun/8.9.1) with ESMTP id IAA12687 for ; Thu, 21 Sep 2000 08:34:35 -0700 (PDT) Received: from inca.co.lsil.com by mhbs.lsil.com with ESMTP for ips@ece.cmu.edu; Thu, 21 Sep 2000 08:34:12 -0700 Received: from exw-kansas.ks.lsil.com (exw-kansas.ks.lsil.com [153.79.8.7]) by inca.co.lsil.com (8.9.3/8.9.3) with ESMTP id JAA29011 for ; Thu, 21 Sep 2000 09:34:11 -0600 (MDT) Received: by exw-kansas.ks.lsil.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 10:31:34 -0500 Message-Id: From: "Binford, Charles" To: "Reflector, IPS" Subject: RE: iSCSI: Flow Control Date: Thu, 21 Sep 2000 10:31:32 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C023E1.05D159D6" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C023E1.05D159D6 Content-Type: text/plain; charset="iso-8859-1" I do not understand the question of "Even if the SCSI layer uses RTT, what if ISCSI did not? What would be the result?", so I may be also misunderstanding the general issue. However, let me add my two cents to what I think I'm hearing. It appears to me that David Robinson is suggesting that there is no need for targets to ask for the data (RTT), the initiator should just send it immediately after the CDB. "Modern" RAID controllers with large amounts of memory can buffer it. I disagree with that line of thinking. SCSI has always had the target in control of when the data flowed, and for good reason. Regardless of the amount of cache the target has, the host can and *will* fill it up (disks are still slower than the transport). It is imperative for the target to be able to manage his own cache, and that includes being in control of timing of the data transfer. By having the target specify when the data is transferred you leave the data in the host until the target has space for it. If not, the data ends up in the transport layer with only flow control to slow it down, causing unneeded congestion. FC allows for the option of having data sent immediately after the CDB. However, there is a strict limit on the amount of data that can be delivered in that initial chunk (set via a mode page). If the Write is larger that what is allowed by the initial burst size, then the target uses XFER_RDY to ask for the remainder of the data. This type of scheme is acceptable because a target can set aside a chunk of data memory for each queue slot it has and use the XFER_RDY to manage the receipt of any data for that I/O that didn't fit in the pre-allocated buffer. (Of course you still have the queue-full management problem, but that is a separate problem than what I'm addressing here.) Charles Binford LSI Logic Storage Systems (316) 636-8566 > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Thursday, September 21, 2000 6:32 AM > To: Reflector, IPS > Subject: Re: iSCSI: Flow Control > > > > Matt, > I am not sure that you answered David's Question. Even if > the SCSI layer uses RTT, what if ISCSI did not? What would be the result? > . > John L. Hufferd > > > > "Matt Wakeley" @ece.cmu.edu on 09/21/2000 > 12:35:55 AM > Please respond to Matt Wakeley > Sent by: owner-ips@ece.cmu.edu > > To: "Reflector, IPS" > Subject: Re: iSCSI: Flow Control > > Well, "modern" disk arrays imployed on SCSI and FC use > XFER_RDYs (RTT). They don't seem to question it. > > -Matt > > David Robinson wrote: > > > Matt Wakeley wrote: > > > > > Ok, so now the SCSI processes the first command, and sends an RTT > (XFER_RDY in FC terms) to the initiator. Now, the initiator sends the > data down the same TCP connection, and it gets stuck behind all those 998 > commands in the TCP receive buffers. The command can't complete because it > can't get the data, and the data can't be delivered because there's no room for the > commands in front of it. Deadlock. Do you see the issue now? (this > is a good example of why the single TCP connection model, be it synchronous or > asynchronous, is bad). > > > > As I said in other e-mail I question the use of RTT in a modern > > environment with large buffering. Data immediately following the > > command makes much better sense to me, but if we must support > > this environment then you are right that deadlock can occur. > > > > -David ------_=_NextPart_001_01C023E1.05D159D6 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: iSCSI: Flow Control

I do not understand the question of "Even if the = SCSI layer uses RTT, what if ISCSI did not?  What would be the = result?", so I may be also misunderstanding the general = issue.  However, let me add my two cents to what I think I'm = hearing.

It appears to me that David Robinson is suggesting = that there is no need for targets to ask for the data (RTT), the = initiator should just send it immediately after the CDB.  = "Modern" RAID controllers with large amounts of memory can = buffer it. 

I disagree with that line of thinking.  SCSI has = always had the target in control of when the data flowed, and for good = reason.  Regardless of the amount of cache the target has, the = host can and *will* fill it up (disks are still slower than the = transport).  It is imperative for the target to be able to manage = his own cache, and that includes being in control of timing of the data = transfer.  By having the target specify when the data is = transferred you leave the data in the host until the target has space = for it.  If not, the data ends up in the transport layer with only = flow control to slow it down, causing unneeded congestion.

FC allows for the option of having data sent = immediately after the CDB.  However, there is a strict limit on = the amount of data that can be delivered in that initial chunk (set via = a mode page).  If the Write is larger that what is allowed by the = initial burst size, then the target uses XFER_RDY to ask for the = remainder of the data.  This type of scheme is acceptable because = a target can set aside a chunk of data memory for each queue slot it = has and use the XFER_RDY to manage the receipt of any data for that I/O = that didn't fit in the pre-allocated buffer.  (Of course you still = have the queue-full management problem, but that is a separate problem = than what I'm addressing here.)




Charles Binford
LSI Logic Storage Systems
(316) 636-8566


> -----Original Message-----
> From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com]=
> Sent: Thursday, September 21, 2000 6:32 = AM
> To: Reflector, IPS
> Subject: Re: iSCSI: Flow Control
>
>
>
> Matt,
> I am not sure that you answered David's = Question.  Even if
> the SCSI layer uses RTT, what if ISCSI did = not?  What would be the result?
> .
> John L. Hufferd
>
>
>
> "Matt Wakeley" = <matt_wakeley@agilent.com>@ece.cmu.edu on 09/21/2000
> 12:35:55 AM
> Please respond to Matt Wakeley = <matt_wakeley@agilent.com>
> Sent by:  owner-ips@ece.cmu.edu
>
> To:   "Reflector, IPS" = <ips@ece.cmu.edu>
> Subject:  Re: iSCSI: Flow Control
>
> Well, "modern" disk arrays imployed = on SCSI and FC use
> XFER_RDYs (RTT). They don't seem to question = it.
>
> -Matt
>
> David Robinson wrote:
>
> > Matt Wakeley wrote:
> >
> > > Ok, so now the SCSI processes the = first command, and sends an RTT
> (XFER_RDY in FC terms) to the initiator.  = Now, the initiator sends the
> data down the same TCP connection, and it gets = stuck behind all those 998
> commands in the TCP receive buffers.  The = command can't complete because it
> can't get the data, and the data can't be = delivered because there's no room for the
> commands in front of it.  Deadlock.  = Do you see the issue now?  (this
> is a good example of why the single TCP = connection model, be it synchronous or
> asynchronous, is bad).
> >
> > As I said in other e-mail I question the = use of RTT in a modern
> > environment with large buffering. Data = immediately following the
> > command makes much better sense to me, but = if we must support
> > this environment then you are right that = deadlock can occur.
> >
> = >         -David

------_=_NextPart_001_01C023E1.05D159D6-- From owner-ips@ECE.cmu.edu Thu Sep 21 18:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03427; 21 Sep 2000 14:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA17236 for ips-outgoing; Thu, 21 Sep 2000 13:05:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA17228 for ; Thu, 21 Sep 2000 13:05:02 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id 208849F7 for ; Thu, 21 Sep 2000 10:05:00 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id KAA21846 for ; Thu, 21 Sep 2000 10:04:58 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA4840; Thu, 21 Sep 2000 10:04:51 -0700 Message-ID: <39CA3FCD.C4DF707C@agilent.com> Date: Thu, 21 Sep 2000 10:05:18 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: rbg@ipperformance.com, ips@ece.cmu.edu Subject: Re: iSCSI: Flow Control References: <39C9A159.3EB1193B@agilent.com> <39C9A582.BE76A0C1@ebay.sun.com> <39C9BA5B.15273500@agilent.com> <20000921.102855.102249224.rbg@ipperformance.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk rbg@ipperformance.com wrote: > In "modern" disk arrays, what problem does XFER_RDYs solve (or is it a > feature) ? Is there a concept of "command+immediate data" for these > disk arrays ? Charles Binford answers this question very well in a separate message. > Just as a general question.. do we need to comprehend and accomadate > all legacy issues ? > > Matt, Are you suggesting that iSCSI should be limited to disk arrays ? No, but disk array storage attach (and tapes) is the major market. -Matt > > > > On Thu, 21 Sep 2000 00:35:55 -0700 you said: > > Well, "modern" disk arrays imployed on SCSI and FC use XFER_RDYs (RTT). They don't > > seem to question it. > > > > -Matt > > > > David Robinson wrote: > > > > > Matt Wakeley wrote: > > > > > > > Ok, so now the SCSI processes the first command, and sends an RTT (XFER_RDY in > > > > FC terms) to the initiator. Now, the initiator sends the data down the same > > > > TCP connection, and it gets stuck behind all those 998 commands in the TCP > > > > receive buffers. The command can't complete because it can't get the data, > > > > and the data can't be delivered because there's no room for the commands in > > > > front of it. Deadlock. Do you see the issue now? (this is a good example of > > > > why the single TCP connection model, be it synchronous or asynchronous, is > > > > bad). > > > > > > As I said in other e-mail I question the use of RTT in a modern > > > environment with large buffering. Data immediately following the > > > command makes much better sense to me, but if we must support > > > this environment then you are right that deadlock can occur. > > > > > > -David > > > _____________________________________________________________ > Robert Gordon rbg@ipperf.com > IP Performance, Inc > Austin, Texas. From owner-ips@ECE.cmu.edu Thu Sep 21 19:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03864; 21 Sep 2000 15:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA21025 for ips-outgoing; Thu, 21 Sep 2000 14:37:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA21018 for ; Thu, 21 Sep 2000 14:37:11 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Thu, 21 Sep 2000 11:38:24 -0700 Message-ID: From: Charles Monia To: David Robinson , ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Thu, 21 Sep 2000 11:38:22 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > Sent: Wednesday, September 20, 2000 10:54 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Flow Control > > > > You're almost there. Since the TCP connection carries > control traffic for > > all logical units, you don't want the flow of control > information to stop > > due to the command queue on one LUN becoming full (a kind > of head-of-line > > blocking). SAM gets around this by implementing what you > refer to as the > > naive policy. i.e., it maintains continuous flow through > the pipeline by > > dumping commands when the queue fills. > > I have always been advocating a connection per LUN so this > problem seems > to reinforce my belief. But I agree that if the WG wants multiple > LUNs per connection this problem needs to be addressed. As I said earlier: "..........even if the TCP connection only serviced a single LUN, you'd still want to keep the pipeline flowing to insure that SCSI task management commands would eventually get serviced." > .......................................................My lack of > understanding comes from my NFS background where NFS over TCP in > practice doesn't have a flow control problem even though it is common > to mux multiple mount points over a single connection. The simple > reason why this is not a problem is because the initiator (client) > doesn't issue more requests to the target (export point) than it can > buffer. The question is if the protocol expects the initiator to > send more commands than a LUN can queue. If so this can be defended > against either statically at session establishment (assumes command > queues lengths are static) or dynamic flow control. Agreed. If there is agreement that this is an issue, a solution along those lines might be possible. As I recall, though, there seemed to be a rough consensus that any policy for regulating control traffic should be specified in SAM-xx (i.e. by the T10 committee), not within iSCSI. There seemed to be no consensus on whether or not this was an ips problem, however. Charles From owner-ips@ECE.cmu.edu Thu Sep 21 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04400; 21 Sep 2000 19:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA29310 for ips-outgoing; Thu, 21 Sep 2000 18:07:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA27569 for ; Thu, 21 Sep 2000 17:18:55 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 14:20:17 -0700 Message-ID: From: Robert Snively To: "'ips@ece.cmu.edu'" Subject: FW: LU access through an iSCSI session (resend) Date: Thu, 21 Sep 2000 14:17:16 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: Robert Snively Sent: Thursday, September 21, 2000 8:43 AM To: 'meth@il.ibm.com'; Pierre Labat; ips@ece.cmu.edu Subject: RE: LU access through an iSCSI session > LU numbers are identical across all of the TCP connections of a > single iSCSI session. This is an intriguing concept. At present, SCSI does not require logical units to have the same LUN if they are accessed through different NICs. However, present SCSI is essentially one NIC per session. If you have an iSCSI session using NIC A and NIC B and a second session using NIC B and NIC C as physical paths, by this definition, NIC B could have two different LUNs for the same logical unit, one from each session. That should not be a problem, right? From owner-ips@ECE.cmu.edu Thu Sep 21 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04402; 21 Sep 2000 19:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA28964 for ips-outgoing; Thu, 21 Sep 2000 17:58:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA13835 for ; Thu, 21 Sep 2000 11:44:26 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 08:45:45 -0700 Message-ID: From: Robert Snively To: "'meth@il.ibm.com'" , Pierre Labat , ips@ece.cmu.edu Subject: RE: LU access through an iSCSI session Date: Thu, 21 Sep 2000 08:42:44 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > LU numbers are identical across all of the TCP connections of a > single iSCSI session. This is an intriguing concept. At present, SCSI does not require logical units to have the same LUN if they are accessed through different NICs. However, present SCSI is essentially one NIC per session. If you have an iSCSI session using NIC A and NIC B and a second session using NIC B and NIC C as physical paths, by this definition, NIC B could have two different LUNs for the same logical unit, one from each session. That should not be a problem, right? From owner-ips@ECE.cmu.edu Thu Sep 21 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04404; 21 Sep 2000 19:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA29295 for ips-outgoing; Thu, 21 Sep 2000 18:07:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA27542 for ; Thu, 21 Sep 2000 17:18:07 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 14:19:29 -0700 Message-ID: From: Robert Snively To: "'ips@ece.cmu.edu'" Subject: FW: A Transport Protocol Without ACK (resend) Date: Thu, 21 Sep 2000 14:16:27 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: Robert Snively Sent: Thursday, September 21, 2000 8:35 AM To: 'Douglas Otis'; Jim McGrath; 'Randall Stewart' Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' Subject: RE: A Transport Protocol Without ACK A minor correction: > > If you examine FCP documentation, you will find that you can > send data with > the command as an option. You can also send the response at > the end of data > as an option. Every vital feature used to justify tossing > FCP structures > become moot. While this is true in FCP, the function has been made "obsolete" in FCP-2 because there is no identifiable benefit to it and because it complicates retry. I believe the same would be true of any reasonable iSCSI implementation. A reasonable iSCSI implementation is defined as one that has essentially zero additional overhead in managing two transfer units as opposed to managing one transfer unit. From owner-ips@ECE.cmu.edu Thu Sep 21 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04440; 21 Sep 2000 19:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA29098 for ips-outgoing; Thu, 21 Sep 2000 18:01:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA19599 for ; Thu, 21 Sep 2000 14:04:50 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 11:06:11 -0700 Message-ID: From: Robert Snively To: "'David Robinson'" , ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Thu, 21 Sep 2000 11:03:10 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > > The TCP connection can have either a command or data. In either > case reading a short fixed amount of data can determine if it is > one or the other. With a single connection I can only see deadlock > occuring if the initiator sends Command1, Command2, then > Data1 on a single > command/data connection. With a single connection this seems like > a broken initiator as doing the right thing of C1, D1, C2 shouldn't > be any more expensive and it is definately safer. Well one other > case can occur is if the initiator wants the target to issue an RTT > and it interposes another command before the RTT is received, this > too seems like a broken things to do as I don't see much > value in doing RTT > on a single connection. > > In the multiple connection asymettric case this isn't an > issue as data > is never interleved with commands. > > So is there a fundamental requirement that the target must handle > the case of C1, C2, then D1? It seems easy to just write the iSCSI > initiator to never issue such a sequence and if the target gets > one return an error when C2 arrives before D1. > David, This is one of the most valuable capabilities of SCSI and is very important to high performance behavior. It is managed quite successfully in those "ugly" ways that were previously described. Basically, a storage device has the option to choose among a large number of commands so that they will be completed in an optimum order. This typically boosts raw performance of disk drives from about 90 IOs/second to well in excess of 200. Even greater performance improvements can be achieved in storage controllers, since they have deeper caching and far more powerful and reliable data buffering. I have heard numbers as high as 100,000 IOPs/sec from a single storage controller. If you can't do this well in TCP/IP iSCSI, then you do not have a SCSI solution. The problem is independent of datagram or not. The same problem will show up with symmetric or asymmetric multiple connections and will typically interact with the sort of out-of-band flow control and congestion management being proposed for iSCSI in pathological ways. FWIW, FC is really not a datagram service, but a messaging service (Sequence = message) with acknowledgments as required at the link level and/or the protocol level. Bob > It again seems like we are trying to impose or solve > datagram problems > in a stream world. Does parallel SCSI have this problem? > > -David > From owner-ips@ECE.cmu.edu Thu Sep 21 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04435; 21 Sep 2000 19:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA29043 for ips-outgoing; Thu, 21 Sep 2000 18:00:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA15527 for ; Thu, 21 Sep 2000 12:23:06 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 09:24:27 -0700 Message-ID: From: Robert Snively To: "'Michael Krause'" , ips@ece.cmu.edu Subject: RE: ISCSI: flow control Date: Thu, 21 Sep 2000 09:21:26 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > >The question here is whether iSCSI should attempt to solve > the command queue > >overflow problem, or allow T10 to deal with it. Other > protocols (fibre > >channel) already deal with it in the "ugly" fashion - > perhaps so should > >iSCSI. What we need consensus on is whether or not iSCSI > is going to deal > >with this. > > The "ugly" mechanism is undesirable for a variety of reasons > including fast > convergence / recovery upon data loss. A credit system is simple to > implement and provides a way to bound this problem and the > implementation > complexity (whether in software or hardware or if operating > across a set of > adapters). With a single connection implementation, this > is trivial to > accommodate and future multi-port or multi-adapter solutions > can take > advantage of it as you note with minimal effort. Given this > does not > really impact a single connection solution, what compelling > reason is there > for removing its definition / requirement within the spec at > this point? Actually, the ugly problem is not made much less ugly by having a credit system. The problem is that the command receipt resources of a "target" are oversubscribed during periods of high activity on a properly configured system simply because of the statistical characteristics of storage access. The command resources are also shared among multiple sessions. The only way to guarantee credit is to share the credit out among the multiple sessions. This necessarily leaves a lot of unused resources lying around for sessions that are not very busy at a given instant and unnecessarily limits the capabilities of those sessions that are busy and would otherwise be able to make constructive use of those resources. The result is higher latencies and lower throughputs than you would hope. What you really need is dynamic credit allocation, which will always leave the possibility that a particular session may find itself oversubscribed and forcing busy indications. Which is effectively the mechanism SCSI uses today. Bob Snively Brocade Communications Phone 408 487 8135 1745 Technology Drive San Jose, CA 95110 Email rsnively@brocade.com From owner-ips@ECE.cmu.edu Thu Sep 21 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04421; 21 Sep 2000 19:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA28951 for ips-outgoing; Thu, 21 Sep 2000 17:57:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA13511 for ; Thu, 21 Sep 2000 11:36:34 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 08:37:43 -0700 Message-ID: From: Robert Snively To: "'Douglas Otis'" , Jim McGrath , "'Randall Stewart'" Cc: "'Y P Cheng'" , "'Ips@Ece. Cmu. Edu'" Subject: RE: A Transport Protocol Without ACK Date: Thu, 21 Sep 2000 08:34:41 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk A minor correction: > > If you examine FCP documentation, you will find that you can > send data with > the command as an option. You can also send the response at > the end of data > as an option. Every vital feature used to justify tossing > FCP structures > become moot. While this is true in FCP, the function has been made "obsolete" in FCP-2 because there is no identifiable benefit to it and because it complicates retry. I believe the same would be true of any reasonable iSCSI implementation. A reasonable iSCSI implementation is defined as one that has essentially zero additional overhead in managing two transfer units as opposed to managing one transfer unit. From owner-ips@ECE.cmu.edu Thu Sep 21 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04428; 21 Sep 2000 19:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA29278 for ips-outgoing; Thu, 21 Sep 2000 18:06:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA27455 for ; Thu, 21 Sep 2000 17:16:42 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 14:18:04 -0700 Message-ID: From: Robert Snively To: "'ips@ece.cmu.edu'" Subject: FW: iSCSI: Flow Control (Resend) Date: Thu, 21 Sep 2000 14:15:03 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: Robert Snively Sent: Thursday, September 21, 2000 11:03 AM To: 'David Robinson'; ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control > > The TCP connection can have either a command or data. In either > case reading a short fixed amount of data can determine if it is > one or the other. With a single connection I can only see deadlock > occuring if the initiator sends Command1, Command2, then > Data1 on a single > command/data connection. With a single connection this seems like > a broken initiator as doing the right thing of C1, D1, C2 shouldn't > be any more expensive and it is definately safer. Well one other > case can occur is if the initiator wants the target to issue an RTT > and it interposes another command before the RTT is received, this > too seems like a broken things to do as I don't see much > value in doing RTT > on a single connection. > > In the multiple connection asymettric case this isn't an > issue as data > is never interleved with commands. > > So is there a fundamental requirement that the target must handle > the case of C1, C2, then D1? It seems easy to just write the iSCSI > initiator to never issue such a sequence and if the target gets > one return an error when C2 arrives before D1. > David, This is one of the most valuable capabilities of SCSI and is very important to high performance behavior. It is managed quite successfully in those "ugly" ways that were previously described. Basically, a storage device has the option to choose among a large number of commands so that they will be completed in an optimum order. This typically boosts raw performance of disk drives from about 90 IOs/second to well in excess of 200. Even greater performance improvements can be achieved in storage controllers, since they have deeper caching and far more powerful and reliable data buffering. I have heard numbers as high as 100,000 IOPs/sec from a single storage controller. If you can't do this well in TCP/IP iSCSI, then you do not have a SCSI solution. The problem is independent of datagram or not. The same problem will show up with symmetric or asymmetric multiple connections and will typically interact with the sort of out-of-band flow control and congestion management being proposed for iSCSI in pathological ways. FWIW, FC is really not a datagram service, but a messaging service (Sequence = message) with acknowledgments as required at the link level and/or the protocol level. Bob > It again seems like we are trying to impose or solve > datagram problems > in a stream world. Does parallel SCSI have this problem? > > -David > From owner-ips@ECE.cmu.edu Thu Sep 21 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04426; 21 Sep 2000 19:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA29285 for ips-outgoing; Thu, 21 Sep 2000 18:06:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA27494 for ; Thu, 21 Sep 2000 17:17:17 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 14:18:39 -0700 Message-ID: From: Robert Snively To: "'ips@ece.cmu.edu'" Subject: FW: ISCSI: flow control (resend) Date: Thu, 21 Sep 2000 14:15:38 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: Robert Snively Sent: Thursday, September 21, 2000 9:21 AM To: 'Michael Krause'; ips@ece.cmu.edu Subject: RE: ISCSI: flow control > >The question here is whether iSCSI should attempt to solve > the command queue > >overflow problem, or allow T10 to deal with it. Other > protocols (fibre > >channel) already deal with it in the "ugly" fashion - > perhaps so should > >iSCSI. What we need consensus on is whether or not iSCSI > is going to deal > >with this. > > The "ugly" mechanism is undesirable for a variety of reasons > including fast > convergence / recovery upon data loss. A credit system is simple to > implement and provides a way to bound this problem and the > implementation > complexity (whether in software or hardware or if operating > across a set of > adapters). With a single connection implementation, this > is trivial to > accommodate and future multi-port or multi-adapter solutions > can take > advantage of it as you note with minimal effort. Given this > does not > really impact a single connection solution, what compelling > reason is there > for removing its definition / requirement within the spec at > this point? Actually, the ugly problem is not made much less ugly by having a credit system. The problem is that the command receipt resources of a "target" are oversubscribed during periods of high activity on a properly configured system simply because of the statistical characteristics of storage access. The command resources are also shared among multiple sessions. The only way to guarantee credit is to share the credit out among the multiple sessions. This necessarily leaves a lot of unused resources lying around for sessions that are not very busy at a given instant and unnecessarily limits the capabilities of those sessions that are busy and would otherwise be able to make constructive use of those resources. The result is higher latencies and lower throughputs than you would hope. What you really need is dynamic credit allocation, which will always leave the possibility that a particular session may find itself oversubscribed and forcing busy indications. Which is effectively the mechanism SCSI uses today. Bob Snively Brocade Communications Phone 408 487 8135 1745 Technology Drive San Jose, CA 95110 Email rsnively@brocade.com From owner-ips@ECE.cmu.edu Thu Sep 21 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04497; 21 Sep 2000 19:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA00683 for ips-outgoing; Thu, 21 Sep 2000 18:44:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from charon.Exabyte.COM ([12.17.207.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA00679 for ; Thu, 21 Sep 2000 18:44:20 -0400 (EDT) From: JoeBre@exabyte.com Received: from prime1.Exabyte.COM (prime1.Exabyte.COM [161.81.18.231]) by charon.Exabyte.COM (8.9.1a/8.9.1) with ESMTP id QAA16936; Thu, 21 Sep 2000 16:44:19 -0600 (MDT) Received: from bouldermta.exabyte.com (bouldermta.Exabyte.COM [161.81.192.16]) by prime1.Exabyte.COM (8.6.12/8.6.10) with ESMTP id QAA21647; Thu, 21 Sep 2000 16:44:19 -0600 Received: by bouldermta.Exabyte.COM with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 16:43:37 -0600 Message-ID: <118A72FD3F02D211AD1000805F9AA58B02ADC1B6@boulderp03.Exabyte.COM> To: David.Robinson@EBay.Sun.COM, ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Thu, 21 Sep 2000 16:43:49 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C0241D.68DA91A0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C0241D.68DA91A0 Content-Type: text/plain; charset="iso-8859-1" > -----Original Message----- > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > Sent: Thursday, September 21, 2000 12:00 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Flow Control > ... > > I understand the basis of RTT is in the SCSI legacy of targets > with very small buffers where limits might be in the order of > single digits. With modern buffers and cheap memory I question > its need. > > -David > We implement a device with a buffer of 32MB. Maybe that is not a 'modern buffer' - I dunno. When a transfer exceeds that size, a RTT-like mechanism will be needed. No matter how modern the buffer is, implemented with arbitrarily cheap memory, there will be some limit that cannot be exceeded without breaking the desired transfer into multiple physical transfers, with the target informing the initiator when it can accept more data (RTT). Joe ------_=_NextPart_001_01C0241D.68DA91A0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: iSCSI: Flow Control

> -----Original Message-----
> From: David Robinson [mailto:David.Robinson@EBay.S= un.COM]
> Sent: Thursday, September 21, 2000 12:00 = AM
> To: ips@ece.cmu.edu
> Subject: Re: iSCSI: Flow Control
>
...
>
> I understand the basis of RTT is in the SCSI = legacy of targets
> with very small buffers where limits might be = in the order of
> single digits.  With modern buffers and = cheap memory I question
> its need.
>
>       -David
>

We implement a device with a buffer of 32MB. Maybe = that is not a 'modern buffer' - I dunno. When a transfer exceeds that = size, a RTT-like mechanism will be needed. No matter how modern the = buffer is, implemented with arbitrarily cheap memory, there will be = some limit that cannot be exceeded without breaking the desired = transfer into multiple physical transfers, with the target informing = the initiator when it can accept more data (RTT).

Joe

------_=_NextPart_001_01C0241D.68DA91A0-- From owner-ips@ECE.cmu.edu Fri Sep 22 01:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04787; 21 Sep 2000 21:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA03777 for ips-outgoing; Thu, 21 Sep 2000 20:27:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA03773 for ; Thu, 21 Sep 2000 20:27:10 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by palrel1.hp.com (Postfix) with ESMTP id 526FB39C for ; Thu, 21 Sep 2000 17:27:09 -0700 (PDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id RAA13804 for ; Thu, 21 Sep 2000 17:27:08 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA5484 for ; Thu, 21 Sep 2000 17:27:04 -0700 Message-ID: <39CAA772.E262C4BD@agilent.com> Date: Thu, 21 Sep 2000 17:27:30 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: Flow Control References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles Monia wrote: > As I recall, though, there seemed to be a rough > consensus that any policy for regulating control traffic should be specified > in SAM-xx (i.e. by the T10 committee), not within iSCSI. There seemed to be > no consensus on whether or not this was an ips problem, however. Ok, I'll bite. What's the difference between iSCSI and ips? > > > Charles From owner-ips@ECE.cmu.edu Fri Sep 22 01:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04788; 21 Sep 2000 21:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA03533 for ips-outgoing; Thu, 21 Sep 2000 20:19:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA03528 for ; Thu, 21 Sep 2000 20:19:30 -0400 (EDT) Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by atlrel1.hp.com (Postfix) with ESMTP id 5EEE8749 for ; Thu, 21 Sep 2000 20:19:29 -0400 (EDT) Received: from cassin.rose.hp.com (nsmail@cassin.rose.hp.com [15.43.210.194]) by mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id RAA13665 for ; Thu, 21 Sep 2000 17:19:28 -0700 (PDT) Received: from agilent.com ([156.140.234.188]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA5447 for ; Thu, 21 Sep 2000 17:19:25 -0700 Message-ID: <39CAA5A7.5746D00E@agilent.com> Date: Thu, 21 Sep 2000 17:19:51 -0700 From: "Matt Wakeley" Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: LU access through an iSCSI session References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Robert Snively wrote: > > LU numbers are identical across all of the TCP connections of a > > single iSCSI session. > > This is an intriguing concept. At present, SCSI does not require > logical units to have the same LUN if they are accessed through > different NICs. However, present SCSI is essentially one NIC per session. > > If you have an iSCSI session using NIC A and NIC B and a second session > using NIC B and NIC C as physical paths, by this definition, > NIC B could have two different LUNs for the same logical unit, one > from each session. > > That should not be a problem, right? Right. And the reason is because NIC B by itself does not access the LUNs. The iSCSI session running through NICs B and C has a view of LUNs that could be different than the iSCSI session running through NICs A and B. Remember, the iSCSI session is simply using TCP connections through the NICs. The TCP connections by themselves know nothing about iSCSI or LUNs. -Matt From owner-ips@ECE.cmu.edu Fri Sep 22 01:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04795; 21 Sep 2000 21:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA02962 for ips-outgoing; Thu, 21 Sep 2000 19:57:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA02951 for ; Thu, 21 Sep 2000 19:56:53 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id QAA13117; Thu, 21 Sep 2000 16:56:40 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 16:56:39 -0700 Message-ID: From: Jim McGrath To: "'David Robinson'" , ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Thu, 21 Sep 2000 16:56:35 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk While memory may be getting cheaper, latency and transfer rates are getting higher. We have gone from 25 m parallel SCSI buses to transcontinental TCP/IP connections; from 1 MB/s to 100 MB/s (and greater) transfer rates. These combine to make the maximum amount of data in flight that keeps the connection full to be growing much faster than memory cost is declining. (Exponential growth rates are applied to both memory cost and transmission speed; distance also appears to be growing very fast, although perhaps not exponentially). So while your argument is works if you keep the fabric size the same and increase the transfer rate (as it has been with the ATA interface - buffer costs have declined over the years), it does not work if the fabric keeps on growing as well. If a fabric introduces 1 ms (two orders of magnitude less than the worse cases I have heard) at Gbit speed, then we need 100 Kbytes of buffer space for a connection. We don't have enough buffer to reserve this for all possible connections we could get (Fibre Channel designs could not reserve 4 KByte for a smaller number of potential connections until recently). Jim PS if we actually are starting to need windows greater than 64 KBytes, is this a problem? My understanding is that deployed TCP/IP products do not easily support extremely large windows. This argues for spreading a single SCSI command across multiple TCP/IP connections for pipelining to overcome latency, not for bandwidth. -----Original Message----- From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] Sent: Wednesday, September 20, 2000 11:00 PM To: ips@ece.cmu.edu Subject: Re: iSCSI: Flow Control Joshua Tseng wrote: > Not doing RTT means each write command must be completed atomically > before proceeding on to the next command. There will be some very > large data PDU's hogging the single connection. How about task > management functions which the initiator may want to deliver > asynchronously? With a large data PDU stuck in the connection, > (and commands stuck in the pipeline) it may require a more > catastrophic abort/reset of the entire TCP connection, which might > not have been originally necessary. > > Somehow, it seems to me that the SCSI folks put RTT in there for > a purpose, and iSCSI would be losing something by eliminating it. > I don't know exactly what this is, but I would think it includes > AT LEAST some performance impact. I understand the basis of RTT is in the SCSI legacy of targets with very small buffers where limits might be in the order of single digits. With modern buffers and cheap memory I question its need. -David From owner-ips@ECE.cmu.edu Fri Sep 22 04:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05303; 22 Sep 2000 0:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA07986 for ips-outgoing; Thu, 21 Sep 2000 23:22:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA07980 for ; Thu, 21 Sep 2000 23:22:29 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 21 Sep 2000 23:21:53 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704C35CB2@corpmx9.isus.emc.com> To: jtseng@NishanSystems.com, ips@ece.cmu.edu Subject: RE: iSCSI Security Protocol Date: Thu, 21 Sep 2000 23:21:51 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk There is no formal consensus on that document at the moment. --David (w/WG co-chair hat firmly in place). --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Joshua Tseng [SMTP:jtseng@nishansystems.com] > Sent: Wednesday, September 20, 2000 7:03 PM > To: ips@ece.cmu.edu > Subject: iSCSI Security Protocol > > I just did a brief review of the document > draft-klein-iscsi-security-00.txt. > What is the current consensus (if any) on this document? Is there > agreement to use SSH as the security mechanism for iSCSI? > > Josh From owner-ips@ECE.cmu.edu Fri Sep 22 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05650; 22 Sep 2000 2:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA10143 for ips-outgoing; Fri, 22 Sep 2000 00:53:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA10134 for ; Fri, 22 Sep 2000 00:53:52 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 22 Sep 2000 00:53:21 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FCC@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: ips WG Charter Approved! Date: Fri, 22 Sep 2000 00:53:19 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I've just been informed that the IESG has approved the ips charter. Congratulations - we are now officially a Working Group, and thanks to all who contributed. The official charter will appear on the IETF web site in the near future, linked to: http://www.ietf.org/html.charters/wg-dir.html#Transport_Area Here's a close approximation to what was approved; I believe the differences between this and the official version are limited to typos, formatting, and the like: IP Storage (ips) Working Group Proposed Charter - DRAFT 7 Mailing list: ips@ece.cmu.edu To subscribe, send email to ips-request@ece.cmu.edu with "subscribe ips" in the first line of the body. The list is archived at: http://ips.pdl.cs.cmu.edu/mail/maillist.html There is significant interest in using IP-based networks to transport block storage traffic. This group will pursue the pragmatic approach of encapsulating existing protocols, such as SCSI and Fibre Channel, in an IP-based transport or transports. The group will focus on the transport or transports and related issues (e.g., security, naming, discovery, and configuration), as opposed to modifying existing protocols. Standards for the protocols to be encapsulated are controlled by other standards organizations (e.g., T10 [SCSI] and T11 [Fibre Channel]). The WG cannot assume that any changes it desires will be made in these standards, and hence will pursue approaches that do not depend on such changes unless they are unavoidable. In that case the WG will create a document to be forwarded to the standards group responsible for the technology explaining the issue and requesting the desired changes be considered. The WG will endeavor to ensure high quality communications with these standards organizations. The WG will consider whether a layered architecture providing common transport, security, and/or other functionality for its encapsulations is the best technical approach. The protocols to be encapsulated expect a reliable transport, in that failure to deliver data is considered to be a rare event for which time-consuming recovery at higher levels is acceptable. This has implications for both the choice of transport protocols and design of the encapsulation(s). The WG's encapsulations may require quality of service assurances (e.g., bounded latency) to operate successfully; the WG will consider what assurances are appropriate and how to provide them in shared traffic environments (e.g., the Internet) based on existing IETF QoS mechanisms such as Differentiated Services. Use of IP-based transports raises issues that do not occur in the existing transports for the protocols to be encapsulated. The WG will address at least the following: - - Congestion control suitable for shared traffic network environments such as the Internet. - - Security measures, including authentication and privacy, sufficient to defend against threats up to and including those that can be expected on a public network. - - Naming and discovery mechanisms for the encapsulated protocols on IP-based networks, including both discovery of resources (e.g., storage) for access by the discovering entity, and discovery for management. - - Management, including appropriate MIB definition(s). The WG will address security and congestion control as an integral part of its protocol encapsulation(s); naming, discovery, and management are important related issues, but may be addressed in companion documents. The WG specifications will provide support for bridges and gateways that connect to existing implementations of the encapsulated protocols. The WG will preserve the approaches to discovery, multi-pathing, booting, and similar issues taken by the protocols it encapsulates to the extent feasible. It may be necessary for traffic utilizing the WG's encapsulations to pass through Network Address Translators (NATs) and/or firewalls in some circumstances; the WG will endeavor to design NAT- and firewall-friendly protocols that do not dynamically select target ports or require Application Level Gateways. Effective implementations of some IP transports for the encapsulated protocols are likely to require hardware acceleration; the WG will consider issues concerning the effective implementation of its protocols in hardware. The standard internet checksum is weaker than the checksums used by other implementations of the protocols to be encapsulated. The WG will consider what levels of data integrity assurance are required and how they should be achieved. The WG will produce a framework document that provides an overview of the environments in which its encapsulated protocols and related protocols are expected to operate. The WG will produce requirements and specification documents for each protocol encapsulation, and may produce applicability statements. The requirements and specification documents will consider both disk and tape devices, taking note of the variation in scale from single drives to large disk arrays and tape libraries, although the requirements and specifications need not encompass all such devices. The WG will not work on: - - Extensions to existing protocols such as SCSI and Fibre Channel beyond those strictly necessary for the use of IP-based transports. - - Modifications to internet transport protocols or approaches requiring transport protocol options that are not widely supported, although the WG may recommend use of such options for block storage traffic. - - Support for environments in which significant data loss or data corruption is acceptable. - - File system protocols. Operational Structure: Due to the scope of the task and the need for parallel progress on multiple work items, the WG effort is organized as follows: A technical coordinator will be identified and selected for each protocol encapsulation adopted as a work item by the group. This person will be responsible for coordinating the technical efforts of the group with respect to that encapsulation, working with and motivating the document editors, and evangelizing the group's work within both the community and relevant external organizations such as T10 and T11. In addition to the normal responsibilities of IETF working group chairs, the IPS chairs hold primary responsibility for selection of coordinators, identifying areas of technical commonality and building cross-technology efforts within the group. Coordinators for initially important encapsulations: SCSI over IP (aka iSCSI): TBD Fibre Channel (FC-2) over IP: TBD Milestones Oct 00 Post initial versions of requirements and specification drafts for the initial protocol encapsulations as working group Internet-Drafts. Nov 00 Submit initial version of framework document as an Internet-Draft. Dec 00 Discuss drafts and issues at the IETF meeting in San Diego. Feb 01 Submit final versions of requirements drafts to the IESG for consideration as Informational RFCs. Mar 01 Discuss framework, specification and related drafts (e.g., MIBs, discovery) for the protocol encapsulations at IETF meeting in Minneapolis. May 01 Submit protocol specification drafts to the IESG for consideration as Proposed Standard RFCs. Jun 01 Begin revision of WG charter in consultation with the Area Directors. Aug 01 Meet at IETF meeting to close any open issues and finish any outstanding work items, including MIB, discovery, and framework drafts. Sep 01 Submit MIB, discovery, framework, and any other WG drafts to the IESG for consideration as appropriate to each draft. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Fri Sep 22 10:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06371; 22 Sep 2000 6:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA15535 for ips-outgoing; Fri, 22 Sep 2000 05:21:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA15525 for ; Fri, 22 Sep 2000 05:20:54 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id CAA12749; Fri, 22 Sep 2000 02:08:49 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id CAA14426; Fri, 22 Sep 2000 02:06:29 -0700 From: "Douglas Otis" To: "Jim McGrath" , "'David Robinson'" , Subject: RE: iSCSI: Flow Control Date: Fri, 22 Sep 2000 02:23:06 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jim McGrath > Sent: Thursday, September 21, 2000 4:57 PM > To: 'David Robinson'; ips@ece.cmu.edu > Subject: RE: iSCSI: Flow Control > > While memory may be getting cheaper, latency and transfer rates > are getting > higher. We have gone from 25 m parallel SCSI buses to transcontinental > TCP/IP connections; from 1 MB/s to 100 MB/s (and greater) transfer rates. > These combine to make the maximum amount of data in flight that keeps the > connection full to be growing much faster than memory cost is declining. > (Exponential growth rates are applied to both memory cost and transmission > speed; distance also appears to be growing very fast, although perhaps not > exponentially). > > So while your argument is works if you keep the fabric size the same and > increase the transfer rate (as it has been with the ATA interface - buffer > costs have declined over the years), it does not work if the > fabric keeps on > growing as well. > > If a fabric introduces 1 ms (two orders of magnitude less than the worse > cases I have heard) at Gbit speed, then we need 100 Kbytes of buffer space > for a connection. We don't have enough buffer to reserve this for all > possible connections we could get (Fibre Channel designs could > not reserve 4 > KByte for a smaller number of potential connections until recently). > > Jim > > PS if we actually are starting to need windows greater than 64 KBytes, is > this a problem? My understanding is that deployed TCP/IP products do not > easily support extremely large windows. This argues for > spreading a single > SCSI command across multiple TCP/IP connections for pipelining to overcome > latency, not for bandwidth. Realize that regardless of the number of connections, data does not move any faster unless you intend to crowd out other traffic. It would not seem wise to sit upon a single transfer until completion before moving to the next if that was your reasoning. Transfers could be broken down to 2k units as with FC using overlapping R2T and reads. As far as a safe estimation, I would tend toward using an expectation of 5 ms for TCP on a MAN and 100+ ms for anything farther (8us per mile) and you may get there by way of Canada. I doubt a process will keep buffers lean and you have retry and head of queue blocking with TCP. An agent translating for legacy drives extends buffer space. Imagine placing a raid controller behind this horrendous latency. No number of connections would save you. Doug > > -----Original Message----- > From: David Robinson [mailto:David.Robinson@EBay.Sun.COM] > Sent: Wednesday, September 20, 2000 11:00 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Flow Control > > > Joshua Tseng wrote: > > Not doing RTT means each write command must be completed atomically > > before proceeding on to the next command. There will be some very > > large data PDU's hogging the single connection. How about task > > management functions which the initiator may want to deliver > > asynchronously? With a large data PDU stuck in the connection, > > (and commands stuck in the pipeline) it may require a more > > catastrophic abort/reset of the entire TCP connection, which might > > not have been originally necessary. > > > > Somehow, it seems to me that the SCSI folks put RTT in there for > > a purpose, and iSCSI would be losing something by eliminating it. > > I don't know exactly what this is, but I would think it includes > > AT LEAST some performance impact. > > I understand the basis of RTT is in the SCSI legacy of targets > with very small buffers where limits might be in the order of > single digits. With modern buffers and cheap memory I question > its need. > > -David > From owner-ips@ECE.cmu.edu Fri Sep 22 10:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa06376; 22 Sep 2000 6:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA15693 for ips-outgoing; Fri, 22 Sep 2000 05:30:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA15683 for ; Fri, 22 Sep 2000 05:30:27 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id CAA12753; Fri, 22 Sep 2000 02:18:28 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id CAA14453; Fri, 22 Sep 2000 02:16:11 -0700 From: "Douglas Otis" To: "Matt Wakeley" , "Reflector, IPS" Subject: RE: iSCSI: Session Partial Resolution Date: Fri, 22 Sep 2000 02:32:49 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <39C99EA3.9993E347@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Wednesday, September 20, 2000 10:38 PM > To: Reflector, IPS > Subject: Re: iSCSI: Session Partial Resolution > > > Douglas Otis wrote: > > > Matt, > > > > > > > The "iSCSI Requirements" document requires this capability: > > > > > > "[R] High bandwidth, bandwidth aggregation. > > > > This is a general feature of IP. > > No, it's not. Please go back and read Randy Haagens' memo from August 5, > available here > http://ips.pdl.cs.cmu.edu/mail/msg00412.html > > > > > > > > [D] The bandwidth (transfer rate, MB/sec) supported by storage > > > controllers is rapidly increasing, due to several factors: (1) > > > Increase in disk spindle and controller performance; > > > > The linear increase in the performance of the mechanical aspects of the > > drive have not kept pace with the exponential data density improvements. > > Unless dealing with a single massive transfer, the overall > effect of data > > density improvements is a small reduction in latency lost by the large > > latency within a MAN network. > > We are not talking about drive controllers, we're talking about > storage array > (raid) controllers, where there is lots of drives behind it. > > I don't think anyone considers attaching individual disk drives > to the internet > (we'd very quickly run out of IP addresses). Matt, It depends on how you make the connection. You should review my draft. You will not get the performance expected with an architecture as you propose. A bad idea. With SCTP you can drop into 10.xx.xx.xx space. What do you mean not enough IPs? iSCSI is not a good way to aggregate as it requires a stateful bridge into drives. Are you really advocating placing a controller behind 5 ms+ latency? The impact on drive performance would be far less noticeable as the controller already hides much of the latency. But I guess you have a better idea? Doug > (2) Use > > > of ever-larger caches, and improved caching algorithms; > > > > Network latency negates use of remote caching. Such caching > must be near > > the client and not the drive. If you wish to define a controller > > specification, this should be done using appropriate > technologies. Clearly, > > if the controller is on site, interface requirements change > substantially. > > For redundancy, there may even be one per client system! Now the traffic > > that needs some work would be to ensure such controllers stay > synchronized > > if used in a cluster. Either way, you will not see a higher performance > > moving the controller next to the drive. > > > > > (3) > > > Increased scale of storage controllers (number of supported > > > spindles, speed of interconnects). Not only must the iSCSI > > > provide for full utilization of available link bandwidth, it > > > also must exploit parallelism (multiple connections) at the > > > device interfaces and within the interconnect fabric." > > > > Again, this is a general feature of IP. If access is to the > drive, as it > > should be, there is no problem scaling especially if you are using IP. > > > > > Unless there is "consensus" that this is not required, I think > > > the multiple > > > connections/session must remain. > > > > Not one reason for forcing aggregation into a single connection. If you > > wish to bind multiple adapters, IP supports this. I doubt you will ever > > need to do this however. > > I'm not going to try and convince you of the need for iSCSI link > aggregation, > and how IP does not support this. Bottom line is that major > system and storage > vendors such as HP think we need it. Once again, re-read Randy's message. > > > > > > > Doug > > -Matt > From owner-ips@ECE.cmu.edu Fri Sep 22 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07492; 22 Sep 2000 13:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA26367 for ips-outgoing; Fri, 22 Sep 2000 11:40:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from www.indstorage.com (windjo.ppp.frii.com [216.17.133.220] (may be forged)) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA26352 for ; Fri, 22 Sep 2000 11:40:26 -0400 (EDT) Received: from aegis ([192.168.1.12]) by www.indstorage.com (8.9.3/8.9.2) with SMTP id JAA13411; Fri, 22 Sep 2000 09:35:55 -0600 From: "Mark Bradley" To: "Matt Wakeley" , Subject: iSCSI vs IPS Date: Fri, 22 Sep 2000 09:35:41 -0600 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600 In-Reply-To: <39CAA772.E262C4BD@agilent.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk When IPS was originally conceived by Carl Madison and myself within the context of the IETF, we had in mind the general concept of 'Storage over IP', not SCSI over IP. IPS is IPStorage, not IPSCSI. -- markb > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Thursday, September 21, 2000 6:27 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Flow Control > > Ok, I'll bite. What's the difference between iSCSI and ips? > > > > > > > Charles > From owner-ips@ECE.cmu.edu Fri Sep 22 18:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07677; 22 Sep 2000 14:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA00949 for ips-outgoing; Fri, 22 Sep 2000 13:37:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA00940 for ; Fri, 22 Sep 2000 13:37:41 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA12845; Fri, 22 Sep 2000 10:25:29 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA15667; Fri, 22 Sep 2000 10:23:09 -0700 From: "Douglas Otis" To: "Michael Krause" Cc: Subject: RE: iSCSI: Session Partial Resolution Date: Fri, 22 Sep 2000 10:39:50 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <4.2.2.20000920133238.00cba410@hpindlm.cup.hp.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Wednesday, September 20, 2000 1:36 PM > To: Douglas Otis > Cc: ips@ece.cmu.edu > Subject: RE: iSCSI: Session Partial Resolution > > > At 08:03 AM 9/20/00 -0700, Douglas Otis wrote: > > > > [D] The bandwidth (transfer rate, MB/sec) supported by storage > > > controllers is rapidly increasing, due to several factors: (1) > > > Increase in disk spindle and controller performance; > > > >The linear increase in the performance of the mechanical aspects of the > >drive have not kept pace with the exponential data density improvements. > >Unless dealing with a single massive transfer, the overall effect of data > >density improvements is a small reduction in latency lost by the large > >latency within a MAN network. > > Product deployment across all distances with a large percentage > within the > data center itself since IP / Ethernet backbone is and will remain the > dominate technology. As such, the latency benefits within the > data center > and LAN environments should not be dismissed. If the point of splicing a WAN into the normal client<->controller<->storage is between the controller and storage, then all residing within the same facility will not see significant impact nor is the existing client interfaces altered by necessity. Placing the WAN between the client and the controller, you have removed normal protection offered by the controller against latency and by necessity altered the client interfaces. Such an architecture is at a serious disadvantage. A WAN interface between controller and storage also prevents a controller from being a point of failure with respect to the remote facility. Doug > > > (2) Use > > > of ever-larger caches, and improved caching algorithms; > > > >Network latency negates use of remote caching. > > Same counter argument w.r.t. latency. > > Mike > From owner-ips@ECE.cmu.edu Fri Sep 22 18:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07682; 22 Sep 2000 14:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA29451 for ips-outgoing; Fri, 22 Sep 2000 13:04:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA29447 for ; Fri, 22 Sep 2000 13:04:10 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id C9E311BB0; Fri, 22 Sep 2000 10:04:08 -0700 (PDT) Received: from mk731912 (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA22251; Fri, 22 Sep 2000 10:04:10 -0700 (PDT) Message-Id: <4.2.2.20000920133238.00cba410@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 20 Sep 2000 13:36:06 -0700 To: "Douglas Otis" From: Michael Krause Subject: RE: iSCSI: Session Partial Resolution Cc: In-Reply-To: References: <39C87DC4.D26366F4@agilent.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:03 AM 9/20/00 -0700, Douglas Otis wrote: > > [D] The bandwidth (transfer rate, MB/sec) supported by storage > > controllers is rapidly increasing, due to several factors: (1) > > Increase in disk spindle and controller performance; > >The linear increase in the performance of the mechanical aspects of the >drive have not kept pace with the exponential data density improvements. >Unless dealing with a single massive transfer, the overall effect of data >density improvements is a small reduction in latency lost by the large >latency within a MAN network. Product deployment across all distances with a large percentage within the data center itself since IP / Ethernet backbone is and will remain the dominate technology. As such, the latency benefits within the data center and LAN environments should not be dismissed. > > (2) Use > > of ever-larger caches, and improved caching algorithms; > >Network latency negates use of remote caching. Same counter argument w.r.t. latency. Mike From owner-ips@ECE.cmu.edu Fri Sep 22 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07842; 22 Sep 2000 15:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA02612 for ips-outgoing; Fri, 22 Sep 2000 14:18:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA02608 for ; Fri, 22 Sep 2000 14:18:25 -0400 (EDT) Received: from amrelay1.boi.hp.com (amrelay1.boi.hp.com [15.56.8.24]) by atlrel1.hp.com (Postfix) with ESMTP id A7ECB2BC; Fri, 22 Sep 2000 14:18:20 -0400 (EDT) Received: from xboibrg1.boi.hp.com (xboibrg1.boi.hp.com [15.56.8.167]) by amrelay1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA01557; Fri, 22 Sep 2000 12:18:16 -0600 (MDT) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2650.21) id ; Fri, 22 Sep 2000 12:18:16 -0600 Message-ID: <499DC368E25AD411B3F100902740AD6501F8705F@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "IPS (E-mail)" Cc: Robert Snively Subject: RE: A Transport Protocol Without ACK (resend) Date: Fri, 22 Sep 2000 12:04:26 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C024BF.8BB9FFB0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C024BF.8BB9FFB0 Content-Type: text/plain; charset="iso-8859-1" Robert, The reason we proposed immediate write data for iSCSI was as an optimization for short writes over "long" networks. As network diameter increases, minimizing round-trip signalling delays becomes important. Sending write data immediately following the command eliminates the need for an RTT from the target, and thus eliminates a round-trip delay. If immediate data threatens to overflow the target's cache, the target may, on an emergency basis, discard the data; and, when ready, request data again using the RTT protocol. This behavior should be minimized, however, as it will tend to congest the network. Note the use of SCSI-layer flow control (RTT mechanism). I agree with those that say a controller must manage its cache, regardless of the size of the cache. This is most apparent when one considers multiple hosts sharing a storage controller. In order to allocate performance between the hosts, the controller must ensure that a write burst from a single host cannot fill its cache. RTT provides a means for doing this that still allows other commands (read commands, for example) from the same host to proceed. However, reserving a few MB of cache for each host, to accommodate immediate writes, seems to me reasonable. The command windowing mechanism (another flow control mechanism) will prevent the overrunning of these immediate write buffers, assuming that the controller has reserved space in proportion to the largest immediate buffer allowed (which is negotiated) x the maximum number of commands. Upon writing this, it occurs to me that it may be useful to add an XOF/XON-like mechanism for temporarily disallowing immediate write data. This would give the controller another tool for cache management. When its cache became congested, the controller could signal one or more of the requesting hosts to stop sending immediate write data with commands; later, another message could allow immediate writes to resume. These messages would need to make it to the iSCSI layer in the hosts, as they will affect how the host does DMA chain construction. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com -----Original Message----- From: Robert Snively [mailto:rsnively@Brocade.COM] Sent: Thursday, September 21, 2000 2:16 PM To: 'ips@ece.cmu.edu' Subject: FW: A Transport Protocol Without ACK (resend) -----Original Message----- From: Robert Snively Sent: Thursday, September 21, 2000 8:35 AM To: 'Douglas Otis'; Jim McGrath; 'Randall Stewart' Cc: 'Y P Cheng'; 'Ips@Ece. Cmu. Edu' Subject: RE: A Transport Protocol Without ACK A minor correction: > > If you examine FCP documentation, you will find that you can > send data with > the command as an option. You can also send the response at > the end of data > as an option. Every vital feature used to justify tossing > FCP structures > become moot. While this is true in FCP, the function has been made "obsolete" in FCP-2 because there is no identifiable benefit to it and because it complicates retry. I believe the same would be true of any reasonable iSCSI implementation. A reasonable iSCSI implementation is defined as one that has essentially zero additional overhead in managing two transfer units as opposed to managing one transfer unit. ------_=_NextPart_000_01C024BF.8BB9FFB0 Content-Type: application/octet-stream; name="Randy Haagens (E-mail).vcf" Content-Disposition: attachment; filename="Randy Haagens (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000609T224154Z END:VCARD ------_=_NextPart_000_01C024BF.8BB9FFB0-- From owner-ips@ECE.cmu.edu Sat Sep 23 00:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa08867; 22 Sep 2000 20:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA12871 for ips-outgoing; Fri, 22 Sep 2000 19:04:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA09636 for ; Fri, 22 Sep 2000 17:19:33 -0400 (EDT) From: csapuntz@csapuntz-u1.cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 47BBA8E5B; Fri, 22 Sep 2000 14:15:06 -0700 (PDT) To: ips@ece.cmu.edu Subject: iSCSI: remove recovery from transport-layer connection failure(?) Content-Type: text/plain; charset=US-ASCII Date: 22 Sep 2000 14:15:06 -0700 Message-ID: Lines: 19 User-Agent: Gnus/5.0807 (Gnus v5.8.7) XEmacs/21.1 (Bryce Canyon) MIME-Version: 1.0 Sender: owner-ips@ece.cmu.edu Precedence: bulk Currently, iSCSI is spec'ed to recover from transport-layer connection failures. The main motivation for this decision was to support tape backup applications that are quite sensitive to any failures that get propogated to their layer. However, there are other ways of doing backup over IP than SCSI/iSCSI. NDMP (www.ndmp.org) and devices that implement it are already shipping. So, perhaps we can remove the requirement of recovering from transport-layer connection failures in iSCSI. This would simplify the protocol somewhat. Thoughts? -Costa From owner-ips@ECE.cmu.edu Sat Sep 23 07:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10114; 23 Sep 2000 3:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA22078 for ips-outgoing; Sat, 23 Sep 2000 02:49:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA22074 for ; Sat, 23 Sep 2000 02:49:12 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id CAA80632 for ; Sat, 23 Sep 2000 02:43:07 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8N6nBT156246 for ; Sat, 23 Sep 2000 00:49:11 -0600 Importance: Normal Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Fri, 22 Sep 2000 23:46:32 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/23/2000 12:49:10 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, This is not a replacement for Tape. Tape is used for other things besides Backup. Also, NDMP is a Client/Server protocol. Someone still needs to talk to the tape with SCSI or FC. The actual tape needs to be locatable anywhere. This is still the need for iSCSI. . . . John L. Hufferd csapuntz@csapuntz-u1.cisco.com@ece.cmu.edu on 09/22/2000 02:15:06 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: iSCSI: remove recovery from transport-layer connection failure(?) Currently, iSCSI is spec'ed to recover from transport-layer connection failures. The main motivation for this decision was to support tape backup applications that are quite sensitive to any failures that get propogated to their layer. However, there are other ways of doing backup over IP than SCSI/iSCSI. NDMP (www.ndmp.org) and devices that implement it are already shipping. So, perhaps we can remove the requirement of recovering from transport-layer connection failures in iSCSI. This would simplify the protocol somewhat. Thoughts? -Costa From owner-ips@ECE.cmu.edu Sat Sep 23 14:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11189; 23 Sep 2000 10:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA28149 for ips-outgoing; Sat, 23 Sep 2000 09:04:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA16906 for ; Fri, 22 Sep 2000 22:05:01 -0400 (EDT) From: csapuntz@csapuntz-u1.cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 6CF658EBA; Fri, 22 Sep 2000 19:00:34 -0700 (PDT) To: "IP Storage" Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] References: Content-Type: text/plain; charset=US-ASCII Date: 22 Sep 2000 19:00:33 -0700 In-Reply-To: "Douglas Otis"'s message of "Mon, 18 Sep 2000 09:55:02 -0700" Message-ID: Lines: 40 User-Agent: Gnus/5.0807 (Gnus v5.8.7) XEmacs/21.1 (Bryce Canyon) MIME-Version: 1.0 Sender: owner-ips@ece.cmu.edu Precedence: bulk Just for clarification... I was not proposing to add the extended URL scheme for the transport spec. It isn't necessary. In this thread, Doug makes an excellent point about LDAP being a superior mechanism for describing how to connect to the storage. Directory services such as LDAP, as Doug has pointed out, will be critical to managing large quantities of storage. A host can ask such a directory service for a list of storage devices it should mount and how to connect to those storage devices. The query against the directory server that returns this information can be based on machine ID, user ID, operating system ID, or even the owner's birthday. One of the things discovery will end up doing, no doubt, is defining LDAP schemas that describe how to connect to storage (i.e. use SCTP or TCP, what port, what target name, what LUN, what WWN, how to authenticate, etc.). However, there is one place where the transport protocol has to define a name: third party commands. There needs to be some kind of global name which the initiator can pass to the target. The name must be distillable into a string. The target must understand the name and be able to use the information to establish a connection to another target. One could say that the string that is passed is not specified by the standard but instead specified by some management software. I think this will lead to poor interoperability. The SCSI URL-type name is the current proposal for target name. Why is SCSI target name made up of a hostname + a path? Why is the hostname + path passed on connection setup? There are two reasons. I think NAT and IPv6 makes passing hostnames rather than addresses in protocols more desirable. Hostnames can be re-resolved as you cross addressing boundaries. The path is there so that the name can support multiple targets behind a single IP address without having to add entries to the DNS server. At Cisco, for example, I have no control over the local DNS servers and cannot add DNS entries for the ATAPI DVD and floppy in my computer. _Costa From owner-ips@ECE.cmu.edu Sat Sep 23 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11396; 23 Sep 2000 11:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA29246 for ips-outgoing; Sat, 23 Sep 2000 10:17:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA29242 for ; Sat, 23 Sep 2000 10:17:10 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 8B4F6CBA; Sat, 23 Sep 2000 07:17:09 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA15707; Sat, 23 Sep 2000 07:17:11 -0700 (PDT) Message-Id: <4.2.2.20000923065323.00cd5a30@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Sat, 23 Sep 2000 07:13:48 -0700 To: Jim McGrath , "'David Robinson'" , ips@ece.cmu.edu From: Michael Krause Subject: RE: iSCSI: Flow Control In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:56 PM 9/21/00 -0700, Jim McGrath wrote: >While memory may be getting cheaper, latency and transfer rates are getting >higher. We have gone from 25 m parallel SCSI buses to transcontinental >TCP/IP connections; from 1 MB/s to 100 MB/s (and greater) transfer rates. >These combine to make the maximum amount of data in flight that keeps the >connection full to be growing much faster than memory cost is declining. >(Exponential growth rates are applied to both memory cost and transmission >speed; distance also appears to be growing very fast, although perhaps not >exponentially). > >So while your argument is works if you keep the fabric size the same and >increase the transfer rate (as it has been with the ATA interface - buffer >costs have declined over the years), it does not work if the fabric keeps on >growing as well. > >If a fabric introduces 1 ms (two orders of magnitude less than the worse >cases I have heard) at Gbit speed, then we need 100 Kbytes of buffer space >for a connection. We don't have enough buffer to reserve this for all >possible connections we could get (Fibre Channel designs could not reserve 4 >KByte for a smaller number of potential connections until recently). Something to think about w.r.t. this problem: RDMA semantics: Pros: - Sender only targets memory that it knows is available to use and thus does not inject more data than what the receiver can use. This mitigate the overflow problem. - End-to-end ULP ACKs provide an implicit credit scheme for the associated target resources. Con: - One must "slice" up the target resources among a set of senders which can create scalability problems depending upon the resources required per session. This is where SEND semantics have their advantages - one can use statistical access to deal with burst with minimal buffer overflow reserves and combine this with the idea described below. - RDMA support requires additional buffer access / tracking logic within the endnode to track the impacted memory. The semantics are not difficult to implement but it is additional cost within the implementation. Note: SEND semantics have DMA chain costs as well so the actual delta in implementation will vary depending upon the amount of resources one can effectively map /register at a given time. - For small messages, RDMA does not always provide any cost/benefit advantage which is why most implementations support SEND and RDMA semantics. >Jim > >PS if we actually are starting to need windows greater than 64 KBytes, is >this a problem? My understanding is that deployed TCP/IP products do not >easily support extremely large windows. This argues for spreading a single >SCSI command across multiple TCP/IP connections for pipelining to overcome >latency, not for bandwidth. Large window support is not difficult to implement and is supported in many endnodes. However, memory even in large endnodes is still limited and subject to oversubscription so if a link cannot replenish its buffers quickly enough, it drops the incoming packet and the transport retransmission / congestion management takes over and adjusts the injection rate. The question is whether one would like to implement a WRED (weight random early detection - used today in routing elements) type of system within an endnode (server, storage, etc.) whereby it would drop inbound packets when resources are tight based on some criteria of the inbound packet (IP addr, QoS, TCP port, etc.). This would allow the endnode to control which services should have priority when the workload approaches / exceeds the available buffer resources. This would also allow one to vary the amount of "emergency" reserve buffers discussed by others without having to communicate any of this end-to-end or specify it within the architecture beyond the interface and drop value interpretation. I believe there is value in creating the policy interfaces to communicate whether a given connection has any special policies associated with it and one of these policies can be where it is in the drop priority list when circumstances warrant it. The actual policy would be outside of iSCSI (see the previous e-mail discussions about QoS and policy from this summer for other areas where a policy interface would have benefit) to keep iSCSI opaque to the upper layer / application requirements. Mike From owner-ips@ECE.cmu.edu Sat Sep 23 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11389; 23 Sep 2000 11:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA29231 for ips-outgoing; Sat, 23 Sep 2000 10:16:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA29226 for ; Sat, 23 Sep 2000 10:16:19 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 85785922 for ; Sat, 23 Sep 2000 07:16:18 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA15687 for ; Sat, 23 Sep 2000 07:16:21 -0700 (PDT) Message-Id: <4.2.2.20000923063737.00cd1c50@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Sat, 23 Sep 2000 06:43:09 -0700 To: From: Michael Krause Subject: RE: A Transport Protocol Without ACK (resend) In-Reply-To: <499DC368E25AD411B3F100902740AD6501F8705F@xrose03.rose.hp.c om> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 12:04 PM 9/22/00 -0600, HAAGENS,RANDY (HP-Roseville,ex1) wrote: >Robert, > >The reason we proposed immediate write data for iSCSI was as an optimization >for short writes over "long" networks. As network diameter increases, >minimizing round-trip signalling delays becomes important. Sending write >data immediately following the command eliminates the need for an RTT from >the target, and thus eliminates a round-trip delay. > >If immediate data threatens to overflow the target's cache, the target may, >on an emergency basis, discard the data; and, when ready, request data again >using the RTT protocol. This behavior should be minimized, however, as it >will tend to congest the network. > >Note the use of SCSI-layer flow control (RTT mechanism). I agree with those >that say a controller must manage its cache, regardless of the size of the >cache. This is most apparent when one considers multiple hosts sharing a >storage controller. In order to allocate performance between the hosts, the >controller must ensure that a write burst from a single host cannot fill its >cache. RTT provides a means for doing this that still allows other commands >(read commands, for example) from the same host to proceed. > >However, reserving a few MB of cache for each host, to accommodate immediate >writes, seems to me reasonable. The command windowing mechanism (another >flow control mechanism) will prevent the overrunning of these immediate >write buffers, assuming that the controller has reserved space in proportion >to the largest immediate buffer allowed (which is negotiated) x the maximum >number of commands. > >Upon writing this, it occurs to me that it may be useful to add an >XOF/XON-like mechanism for temporarily disallowing immediate write data. >This would give the controller another tool for cache management. When its >cache became congested, the controller could signal one or more of the >requesting hosts to stop sending immediate write data with commands; later, >another message could allow immediate writes to resume. These messages >would need to make it to the iSCSI layer in the hosts, as they will affect >how the host does DMA chain construction. Given the distances involved and the subsequent latency, I'd suggested lifting the technique of issuing a RNR-NAK (receiver not ready) with a time period the sender should wait upon receiving the NAK before it retries the operation (0 indicates immediate which may be acceptable if one knows the sender is far away). Allows the receiver to allocate the requested buffer space within what it believes to be a reasonable amount of time, eliminates the need for the receiver to track the sender's state for this problem, and eliminates the additional message associated with the XOFF/XON mechanism. In general, we should strive wherever possible to architect such that only one side needs to track the state of a given attribute - this leads to simplicity in implementation and future development of the architecture. Mike From owner-ips@ECE.cmu.edu Sat Sep 23 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12156; 23 Sep 2000 16:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA05203 for ips-outgoing; Sat, 23 Sep 2000 15:28:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA05198 for ; Sat, 23 Sep 2000 15:28:04 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDL0FK; Sat, 23 Sep 2000 12:25:46 -0700 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: RE: An IPS Transport Protocol (was A Transport Protocol Without ACK) Date: Sat, 23 Sep 2000 12:26:03 -0700 Message-ID: <001401c02594$1cbcd9e0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal In-Reply-To: <39C75024.6B84287@cisco.com> X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: stewrtrs@stewart.chicago.il.us > > Any transport protocol proposal is ok. As long as it can be seen and > reviewed. So far I have seen only two TCP and SCTP. > > Oh, a little side note, any transport protocol proposed MUST be able to > show TCP like behavior in the face of congestion. And I think, IMHO, that > this means that if it is NOT using RFC2581 procedures it MUST show that > it does backoff and share with TCP. It also has a HEAVY burden of proof to > show this facility at least in my mind and I would think in the > IESG's mind > as well... I will try to describe a transport protocol for iSCSI herein. This proposal addresses the RFC2581 for congestion management as well as queuing and resource management for iSCSI initiator and target devices. I will call this IPS (IP Storage) Protocol which is a hybrid between FCP of fibre channel and TCP of IP. The way this email is written, it is not a formal proposal by any stretch of imagination. I am a career adapter designer and I don't do RFC or windows and floors. Therefore, in describing this IPS Protocol if I misuse any words that have specific meanings to RFCs, my sincere apology to this working group. Herein I assume the iSCSI IETF effort can be broken into two parts: one for mapping a SCSI request and response to one or more iSCSI PDUs and another for accommodating a transport protocol such as TCP, SCTP, or this proposed protocol, IPS. This proposal addresses the second effort of the IETF. If this assumption is wrong, hit the delete key now so you won't waste any more time. 1. The Needs The speed of light travels about 5 us per kilometer or 8 us per mile. With 3000 miles between New York and Los Angeles, the Round Trip Time (RTT) is 3000 x 8 x 2, or 48 msec,, not counting the queuings and delays in the switches and routers. Comparing to the latency of just a few microseconds on locally attached devices, to make iSCSI device a meaningful alternative, it must have an appropriate transport protocol that deals with the long latency. Furthermore, the congestion of the Internet Network that drops and duplicates datagrams demands an efficient and reliable detection of error and retransmission. Finally, given TCP/IP is a well accepted and proven transport protocol, iSCSI must support TCP/IP. 2. Executive Summary For those do not have time to read this long posting, this IPS proposal describes the processing -- both creating and parsing -- of an iSCSI PDU encapsulated within an Internet TCP/IP datagram. Hence the proposal complements the current IETF effort that defines the iSCSI PDUs. An iSCSI PDU starts with a media header such as Ethernet or Fibre Channel, followed by an IP header, an TCP header, an iSCSI header, and, finally, the data payload with CRC. An iSCSI service provider, either a iSCSI driver running on top of a simple old fashion NIC adapter or a sophisticated fiber-channel-like-iSCSI adapter with large amount of microcode and local memory, will perform the protocol processing. This proposal describes the processing -- the semantics -- that solves the iSCSI needs above. Since, the iSCSI PDU has a TCP/IP header, this proposal does not preclude the using of TCP/IP protocol for iSCSI. This IPS protocol addresses congestion management like that in RFC2581 that describes a "good citizenship behavior" of a protocol on how to start and to retransmit data segments on a busy network. This protocol modifies the RFC2581 to deal with long Internet latency of delivery of datagrams. The protocol ensures efficient and yet reliable delivery. By stealing some ideas from fibre channel adapters, which is now targeted for 50,000 IOs per second, this protocol also describes the creation of an exchange table which deals with thousands of concurrent iSCSI requests and responses without the problems of deadlock and resource allocations. 3. Terms A segment -- a term used in the RFC2581, same as an iSCSI PDU ACK and ACK-0 -- an acknowledge PDU. Refer ACK-0 to the FC-PH spec. An Exchange -- roughly like a session defined by the working group except it is executed on a single TCP connection An iSCSI Request/Response Message -- an APL to an iSCSI Provider describing sending/receiving an iSCSI request/response. BB-Credit -- refer to the FC-PH spec. cwnd and rwnd -- Congestion and Receive Windows, terms used in the RFC2581. They have the same value in this protocol SOCKET, CONNECT, BIND Systems Calls -- same meaning as the TCP/IP implementation Delay Constant -- the time units between transfers of sequences Data Descriptors -- in the form of a memory handle or a scatter/gather list in an iSCSI request/response for sending/receiving segments DMA -- Direct Memory Access to transfer iSCSI data payloads to/from iSCSI application software using the data descriptors inside a iSCSI request/response message EE-Credit -- refer to the FC-PH spec. Exchange ID -- OX_ID and RX_ID, please refer to the FC-PH spec. iSCSI Provider -- an iSCSI driver together with an old fashion NIC adapter or a modern superfast iSCSI adapter iSCSI PDU -- as defined by this working group Sequences -- an exchange has many sequences each of which has many segments Tag Queuing -- refer to the SCSI SAM spec. TCP Connection -- A pair of IP-Address and TCP port that uniquely identifies an application process that transmits/receives an iSCSI PDU. Retransmission -- A part of error recovery to retransmit a lost sequence 4. Congestion Management The RFC2581 is not specific to TCP. It should be used by every transport protocol sharing the network, although the authors of the RFC based their experiments and conclusions using TCP. The RFC covers four specific topics: slow start, congestion avoidance, fast transmit, and fast recovery. If other protocols on the network are not following the same rules, while a TCP client/server using slow start waits patiently on a congested network, other protocols will continue flood the network with new data segments, hence, defeating the congestion management. The RFC2581 definitely is not the best thing for a network with extreme long latency. Let me use an example to describe the problem before describing the solution. Assume the latency delay or round-trip time of two iSCSI devices between N.Y. and L.A. is 50 msec. In addition, assume data segment is 2K. Using the slow start algorithm of the RFC2581, a sender only sends two segments at beginning and waits for the ACKs before increase its cwnd. After waiting 50 msec, the sender increases its cwnd to 3, sends 3 segments, and waits again. On a not-so-busy network, to send one MB of data or 500 segments, the sender being a good citizen on the network, will repeat the wait 32 times to send all 500 2K segments. The total time for delivering one MB of data is 50 msec times 32, or about 1.6 seconds. One may argue that given enough time, the cwnd can be increased to 500 and the whole one MB of data can be transferred once. However, any lost packet or out-of-order delivery -- which we assume happening often and is the reason for having the slow start -- the sender seeing the duplicated ACKs slows down immediately by reducing cwnd quickly. Furthermore, the RFC also does slow start after some idle time. This is because the network congestion status is no longer known after some idle time. In this super fast Internet era, when we are designing adapters to process each fibre channel request in 20 microseconds and 50,000 IO's per second, the 50 msec wait and 1.6 sec for moving one MB of data using slow start simply sounds awful. This problem becomes much worse when the MTU is not 2K but reduced to 512 bytes. In this case, there are 2000 segments for a one MB transfer. I don't need to challenge your imagination when the iSCSI is used to back up one TB of data. Now the solution. In the IPS protocol breaks down the 1MB data into 25 20K-sequences. Each sequence has ten 2K segments. Each sequence will be acknowledged individually. We define a Delay Constant between the transfer of two consecutive sequences. On a not-so-busy network, the delay should be zero. Hence, the sender sends all 25 sequences or 500 segments without delay. Using a 1 Gb adapter, the whole 1 MB of data goes out in 10 msec. 25 msec later they arrive at the destination. Each sequence is acknowledged individually. 25 msec later, all 25 ACKs come back to the sender. The whole one 1 MB is transferred in 60 msec, not 1.6 sec. Comparing to the 10 msec transfer on a local network, 60 msec is not so great, but it is the best we can do because the 50 msec delay is contributed by the speed-of-light. A thousand TCP connections will not rid the 50 msec delay. If we decide not to keep this IPS Protocol simple and stupid, we can make the ACK a little more specific by specifying which particular segment is missing. Only missing segments are retransmitted. We can even bundle the missing segments from different sequences by defining a transmitted sequence which contains only retransmitted segments. As an adapter design, I prefer keeping it simple and stupid by retransmit the whole sequence. Instead, we fine tuning it by changing the size of a sequence. When retransmit is necessary, the sender will act as a good citizen by increase the delay constant between sequences. On successful transmit, the sender will decrease the delay constant. Exactly how aggressively should we back away from a congested network -- by a large jump of the delay constant -- will be left for simulation. I do believe the result will depend on the segment sizes the latency values. Note, the performance of this protocol does not depend on the MTU size because it is designed to stream the segments. Notice, this IPS protocol takes an optimistic view about the Internet traffic, i.e., assuming the traffic is light. If not true, it backs off quickly. I believe this is necessary for a network with long latency delay because we can't afford the slow start. A second thing about this IPS Protocol is that one ACK is generated on each sequence instead of each segment. Using the bulk ACK on a busy network with long latency reduces the ACK traffic. The third thing about the IPS is it assumes the receiver is very intelligent to generate the bulk ACK. Of course, if an ACK is missing, the missing sequence is detected by timeout and must be retransmitted. We should also use the ACK-0 of the fibre channel to signal the sender that everything is OK even some ACKs are not received by the sender. ACK-0 will greatly reduce the retransmission by a missing ACK. 5. Queuing Management An IPS request/response message is transaction-oriented, i.e. the whole "iSCSI session" is described in a single request/response message to the IPS provider. Within a request, SCSI command, one or more endpoints, i.e. IP address and TCP port pairs, and data descriptors in the form of a memory handle or a scatter/gather list, and other needed variables are provided. The IPS request/response message is sent to a iSCSI provider that is responsible for creating outgoing PDUs and receiving incoming PDUs. To the provider, each message is an exchange between two endpoints. The initiator give it an OX_ID and the target gives it a RX_ID. Each exchange is executed atomically, i.e. the IPS provider is responsible for sequencing the SCSI command, data, and status. There are no command queuing or head-of-queue deadlock problems. This is because the IPS provider creates a giant exchange table. Whenever a data PDU is received, using OX_ID or RX_ID to find the exchange, the IPS refers to the exchange table to determine what to do. Data PDUs are served on demand, hence, no head-of-queue blocking problem. Outgoing data PDUs are broken down into sequences. After the transfer of each sequence the IPS provider can switch to another exchange to avoid long delay behind a large exchange. For those who familiar with a fibre channel adapter, executing an IPS request is like executing an FCP request, except for the congestion management described earlier. If more than one endpoint is in the iSCSI request/response message, the IPS provider can take the liberty of selecting another endpoint to transmit or retransmit. However, when a different endpoint is used, the whole message, or session, is repeated. A Task Management PDU like ABORT may be needed to avoid confusion on the receiver side. I do appreciate that some people will implement the iSCSI provider in the old fashion stream-oriented TCP protocol instead of this IPS protocol. I don't have any problem for the working group in trying to solve their problems. Personally, I will never implement an iSCSI provider using TCP stream oriented protocol. I will implement the aforementioned congestion management in a fibre channel adapter today as an IPS provider. As long as an IPS provider deals with the PDU's correctly, it should always interoperate with another node which uses TCP stream oriented protocol. Of course, how do two endpoints generate the ACKs must be uniform. In dealing with an IPS provider using TCP, the concept of transfer sequence disappear. Each sequence is a single segment which is ACK'ed individually. By the way, I will never consider multiple TCP paths to reduce latency time because the IPS provider like a fibre channel adapter is targeted to deliver 50,000 IOs per second going to 100,000 IOs in the near future. The context switch time between multiple TCP paths will make the 100,000 IOs impossible. Keeping the segments streaming on the same connection path is the only good solution for long latency delay. 6. Resource Management There are three layers of resource management. First, the BB credit takes care of two nodes connecting point-to-point or on the same arbitrated loop. Using BB credit, one node can never overrun the incoming buffer of another node. This does not apply to iSCSI device connecting to Ethernet due to the collision avoidance protocol, i.e, one has no control of the sender of the incoming segments. Second, the EE credit is equivalent to the rwnd variable of the RFC2581. It manages how many segments a receiver is willing to receive. The EE credit concept is unpractical on a network with long latency. Using the example of the one MB transfer earlier, if the EE credit is small, the sender must wait after its EE credit is exhausted. Only ACKs can replenish the EE credit. The wait is 50 msec each time. In fact, it is imperative for an IPS provider to use DMA to empty incoming segments from its buffer in lieu of EE-credit management. Using EE-credit to slow down the sender on a network with long latency makes the performance unpractical. Finally, the third, the number of SCSI commands can be sent to a target device is governed by the SCSI tag queuing concept. The initiator is always aware of the number of SCSI commands can be sent to a target. It simply does not make sense to send ten commands to a target who can only accept five. After command #6 is rejected with queue busy, the is no guarantee that command #7 will also be rejected. This is because command #1 could be completed before #7 arrives. If #7 is not rejected, then, #6 and #7 will be executed out of order and not acceptable. With the exception of SCSI tag queuing, an IPS provider can not use either BB or EE credits. It must use DMA to empty the incoming segments quickly. For those who implement the IPS provider in TCP, EE credit can be used. Then, one must pay the price of a network with long latency delay. Last, but not least, in the IPS protocol, an IPS provider never needs to allocate cache memory to receive PDUs. This is because it uses the memory supplied by application software with the data descriptor in the request/response message. Each message sets up one exchange table entry which saves the data descriptor. When a PDU is received without an exchange table entry, the segment is unsolicited and thrown away. In other words, the IPS provider is not responsible for an incoming segment when there is no application program waiting for it. This is like the TCP receiving an incoming segment which has an invalid port number. Like setting up the TCP port, an application program must always instruct the IPS provider to create an exchange table entry to receive incoming iSCSI segments. It is OK to send data to a target right after a SCSI command without waiting the Read-To-Transfer from the target. This is known as streaming transfer. When a target uses a IPS message to receive a SCSI command, it can also have the option to provide data descriptors to receive the streamed data without the need of returning R2T first. The streaming transfer is OK'ed when a connection is made. 7. Multiple NICs We certainly do not exclude multiple IPS providers. I believe a wedge driver sits on top of the IPS providers may choose different one for load balance as long as they can reach the same destination. Note, since each IPS request/response message is executed atomically by one IPS provider, there is no synchronization between them. One the receiving end, the application software can set up multiple IPS provider to receive incoming requests. I don't know enough about this area to make meaningful comments. 8. Multiple Paths to Same Destination The IPS protocol uses the SOCKET, CONNECT, and BIND system calls to make a TCP connection. It is assumed that when there are multiple IP addresses to reach a same destination, the SOCKET data structure will provide such information which in turn will be given to the IPS provider for retransmission consideration. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Sat Sep 23 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12290; 23 Sep 2000 17:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA06271 for ips-outgoing; Sat, 23 Sep 2000 16:19:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA06264 for ; Sat, 23 Sep 2000 16:19:23 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id WAA36298; Sat, 23 Sep 2000 22:18:47 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id WAA121092; Sat, 23 Sep 2000 22:18:42 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256963.006F91A1 ; Sat, 23 Sep 2000 22:18:37 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Robert Snively , ips@ece.cmu.edu Message-ID: Date: Sat, 23 Sep 2000 23:18:35 +0300 Subject: RE: LU access through an iSCSI session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk There should be no problem. It is similar to the case where there are 2 different SCSI busses connecting a host to a controller. Each iSCSI session is like a separate virtual SCSI bus. The LU names for the same device that is reached via the different paths may be different. The way for the host to tell that they are actually the same device is though the Vital Product Data, which uniquely identifies the LU. - Kalman Robert Snively on 21/09/2000 18:42:44 Please respond to Robert Snively To: Kalman Meth/Haifa/IBM@IBMIL, Pierre Labat , ips@ece.cmu.edu cc: Subject: RE: LU access through an iSCSI session > LU numbers are identical across all of the TCP connections of a > single iSCSI session. This is an intriguing concept. At present, SCSI does not require logical units to have the same LUN if they are accessed through different NICs. However, present SCSI is essentially one NIC per session. If you have an iSCSI session using NIC A and NIC B and a second session using NIC B and NIC C as physical paths, by this definition, NIC B could have two different LUNs for the same logical unit, one from each session. That should not be a problem, right? From owner-ips@ECE.cmu.edu Sun Sep 24 01:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12828; 23 Sep 2000 21:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA10627 for ips-outgoing; Sat, 23 Sep 2000 19:57:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA10623 for ; Sat, 23 Sep 2000 19:56:57 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Sat, 23 Sep 2000 16:58:55 -0700 Message-ID: From: Charles Monia To: meth@il.ibm.com, Robert Snively , ips@ece.cmu.edu Subject: RE: LU access through an iSCSI session Date: Sat, 23 Sep 2000 16:58:47 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: Strangely enough, while we're on the subject, I don't believe SAM-xx requires a target to present the same view of its logical units to every initiator regardless of the number of NICs. I suspect there may even be products out there that exploit this "feature". Charles > -----Original Message----- > From: meth@il.ibm.com [mailto:meth@il.ibm.com] > Sent: Saturday, September 23, 2000 1:19 PM > To: Robert Snively; ips@ece.cmu.edu > Subject: RE: LU access through an iSCSI session > > > > > > There should be no problem. It is similar to the case where > there are 2 > different SCSI busses connecting a host to a controller. Each > iSCSI session > is like a separate virtual SCSI bus. The LU names for the > same device that > is reached via the different paths may be different. The way > for the host > to tell that they are actually the same device is though the > Vital Product > Data, which uniquely identifies the LU. > > - Kalman > > Robert Snively on 21/09/2000 18:42:44 > > Please respond to Robert Snively > > To: Kalman Meth/Haifa/IBM@IBMIL, Pierre Labat , > ips@ece.cmu.edu > cc: > Subject: RE: LU access through an iSCSI session > > > > > > > LU numbers are identical across all of the TCP connections of a > > single iSCSI session. > > This is an intriguing concept. At present, SCSI does not require > logical units to have the same LUN if they are accessed through > different NICs. However, present SCSI is essentially one NIC > per session. > > If you have an iSCSI session using NIC A and NIC B and a > second session > using NIC B and NIC C as physical paths, by this definition, > NIC B could have two different LUNs for the same logical unit, one > from each session. > > That should not be a problem, right? > > > > From owner-ips@ECE.cmu.edu Sun Sep 24 14:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15140; 24 Sep 2000 9:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA23439 for ips-outgoing; Sun, 24 Sep 2000 08:41:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from radmail.rad.co.il (radmail.rad.co.il [207.232.32.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA23434 for ; Sun, 24 Sep 2000 08:41:27 -0400 (EDT) Received: from sanrad.com ([192.114.26.219]) by radmail.rad.co.il (Post.Office MTA v3.5.3 release 223 ID# 0-54115U1200L1200S0V35) with ESMTP id il; Sun, 24 Sep 2000 15:44:19 +0300 Message-ID: <39CE127B.43CF4025@sanrad.com> Date: Sun, 24 Sep 2000 15:41:00 +0100 From: Yaron Klein Organization: SANRAD X-Mailer: Mozilla 4.7 [en] (WinNT; I) X-Accept-Language: en MIME-Version: 1.0 To: Joshua Tseng CC: ips@ece.cmu.edu Subject: Re: iSCSI Security Protocol References: Content-Type: text/plain; charset=iso-8859-1 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id IAB23439 Joshua, What I meant in =93forcing to use IPsec=94 is that you must have an authe= ntication mechanism in the iSCSI spec if your client does not support IPsec. Several other drawbacks for IPsec: =B7 The iSCSI will use different AH for the header and the data, thus ena= ble fast performance in iSCSI gateways applications (where the header is modified = and the message is forwarded =96 only the header AH need to be calculated). =B7 The point you mentioned, using tunnel mode IPsec may cause some probl= ems while traveling through gateways and proxies. So, the system administrator can set and configure the system as he wishe= s. If the system supports IPsec, iSCSI authentication will not be needed. If you have any results from tests with IPsec performance, let us know! Regards, Yaron Joshua Tseng wrote: > Hi Yaron, > > >For full security (authentication and encryption) use external protoco= l, > e.g., > >IPsec. You can define an IPsec policy for encrypting everything (not > feasible > >for most cases) or just the first 48 bytes (headers) and so on. > > I agree we should look at IPSec. Today, all SSH implementations are > software-based as far as I know. IPSec hardware is much more readily > available. > > >However, IPsec may cause some problems since it is IP oriented (connec= tion > >oriented and not session oriented). Moreover, you are forcing the clie= nt to > have > >IPsec, which is not always true. > > I'm not sure what you mean here. IPSec is a layer-3 protocol, and it h= as > no knowledge of TCP. Conversely, TCP is unaware of IPSec operating at = the > layer below it, and IPSec should not interfere at all with TCP. The on= ly > drawback I see is that if you selectively apply a security policy to iS= CSI > conversations which may go over the same TCP connection, some TCP segme= nts > may be encrypted and others will not. For example, a policy may dictat= e > no security for one iSCSI conversation, but ESP tunneling w/3DES for a > different iSCSI conversation. Both conversations use the same TCP > connection. > If this happens, you will not see a difference at the TCP end points, b= ut in > the network you'll see some segments in a TCP connection "disappear", a= s > they > have been encrypted, while others may be left alone and visible to the > network, > IP and TCP headers and all. This effect may confuse firewalls and othe= r > monitoring points in the network. But I would rather have the flexibil= ity > to do this than to be forced to apply a single uniform security policy = to > all > iSCSI traffic using a TCP connection. > > I also don't know what you mean by "forcing the client to have IPSec". = If > the > client doesn't have IPSec, this can be negotiated out at iSCSI login. = Or, > if > you want to do authentication before iSCSI login, IKE (Internet Key > Exchange) > can easily be implemented in software. > > >The security scheme in the iSCSI draft includes authorization and > >authentication. The authorization is done in the login phase with the > >negotiation (detailed in the draft), and authentication is achieved by= a > trailer > >that checks the integrity of the data and the header (either simple CR= C or > some > >mac algorithm). > > So it seems you do not intend to leverage the Authentication Headers (A= H) > protocol > and will imbed the authentication & data integrity mechanism within the > iSCSI protocol? > > Regards, > > Josh From owner-ips@ECE.cmu.edu Sun Sep 24 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15880; 24 Sep 2000 14:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA28250 for ips-outgoing; Sun, 24 Sep 2000 13:44:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA28246 for ; Sun, 24 Sep 2000 13:43:56 -0400 (EDT) Received: from colosus2.cup.hp.com (colosus2.cup.hp.com [15.13.128.145]) by palrel1.hp.com (Postfix) with ESMTP id 834ACFB8; Sun, 24 Sep 2000 10:43:55 -0700 (PDT) Received: from hp.com (IDENT:plabat@pl703521.cup.hp.com [15.13.133.216]) by colosus2.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA09009; Sun, 24 Sep 2000 10:43:53 -0700 (PDT) Message-ID: <39CE3F8D.F56A4F0B@hp.com> Date: Sun, 24 Sep 2000 10:53:17 -0700 From: Pierre Labat Organization: Hewlett Packard ATM-SISL X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: Robert Snively Cc: ips@ece.cmu.edu Subject: Re: ISCSI: flow control References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Robert Snively wrote: > > >The question here is whether iSCSI should attempt to solve > > the command queue > > >overflow problem, or allow T10 to deal with it. Other > > protocols (fibre > > >channel) already deal with it in the "ugly" fashion - > > perhaps so should > > >iSCSI. What we need consensus on is whether or not iSCSI > > is going to deal > > >with this. > > > > The "ugly" mechanism is undesirable for a variety of reasons > > including fast > > convergence / recovery upon data loss. A credit system is simple to > > implement and provides a way to bound this problem and the > > implementation > > complexity (whether in software or hardware or if operating > > across a set of > > adapters). With a single connection implementation, this > > is trivial to > > accommodate and future multi-port or multi-adapter solutions > > can take > > advantage of it as you note with minimal effort. Given this > > does not > > really impact a single connection solution, what compelling > > reason is there > > for removing its definition / requirement within the spec at > > this point? > > Actually, the ugly problem is not made much less ugly by having > a credit system. The problem is that the command receipt resources > of a "target" are oversubscribed during periods of high activity on > a properly configured system simply because of the statistical > characteristics of storage access. The command resources are also > shared among multiple sessions. The only way to guarantee credit > is to share the credit out among the multiple sessions. This > necessarily leaves a lot of unused resources lying around for > sessions that are not very busy at a given instant and unnecessarily > limits the capabilities of those sessions that are busy and would > otherwise be able to make constructive use of those resources. > The result is higher latencies and lower throughputs than you would > hope. What you really need is dynamic credit allocation, which will > always leave the possibility that a particular session may find > itself oversubscribed and forcing busy indications. Which is effectively > the mechanism SCSI uses today. Robert, As you say, the target can inform the initiator whenever it wants, based on its own policy of resources management, that it can't handle no more command. It has the advantage to be dynamic and to take into account all the initiators. However, this mechanism has the following weaknesses: 1) to inform the initiator to stop sending command, the target drops commands (and data). It implies retransmission It's a waste of bandwidth. 2) the initiator after detecting [several] TASK SET FULL/BUSY status, must handle blindly the return to the full depth of the command window. It does that in a slow way. At that time you under use the capabilities of the LU. I explain: in a typical SCSI implementation, when a TASK SET FULL set occurs, the initiator decrements the max queue depth (command on flight for the LU) by one. Then after a number of commands completed (say N=32) the initiator increments the max queue depth to return to the initial value. The initiator has to comeback to the initial queue depth slowly and cautiously because never the target informs it, the queue depth available for it. This behavior looks like a target congestion control mechanism 1) = you drops 2) = it's a kind of slow start Why stick with this mechanism that on one side waste bandwidth and on the other side "under use" the capabilities of a LU because the slow start? It would be better to add a comand flow control mechanism that acts before we hit this target congestion control mechanism. T10 could add an advertising of a command credit. For example when the status is returned the target gives a command credit. Doing that way the target is able to slow down the command flow without dropping, and then drive the initiator in order it catches up quickly with the recovered capability/availibility of the LU for it. Regards, Pierre > > > Bob Snively > Brocade Communications Phone 408 487 8135 > 1745 Technology Drive > San Jose, CA 95110 Email rsnively@brocade.com From owner-ips@ECE.cmu.edu Sun Sep 24 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16129; 24 Sep 2000 16:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA00180 for ips-outgoing; Sun, 24 Sep 2000 15:14:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA00163 for ; Sun, 24 Sep 2000 15:14:05 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA13115; Sun, 24 Sep 2000 12:01:29 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA22582; Sun, 24 Sep 2000 11:59:08 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI: remove recovery from transport-layer connection failure(?) Date: Sun, 24 Sep 2000 12:16:13 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, Is this using RPC function calls? Are you suggesting access via servers? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > csapuntz@csapuntz-u1.cisco.com > Sent: Friday, September 22, 2000 2:15 PM > To: ips@ece.cmu.edu > Subject: iSCSI: remove recovery from transport-layer connection > failure(?) > > > > Currently, iSCSI is spec'ed to recover from transport-layer > connection failures. > > The main motivation for this decision was to support tape backup > applications that are quite sensitive to any failures that get > propogated to their layer. > > However, there are other ways of doing backup over IP than > SCSI/iSCSI. NDMP (www.ndmp.org) and devices that implement it are > already shipping. > > So, perhaps we can remove the requirement of recovering from > transport-layer connection failures in iSCSI. This would simplify > the protocol somewhat. > > Thoughts? > > -Costa > From owner-ips@ECE.cmu.edu Sun Sep 24 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16139; 24 Sep 2000 16:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA00765 for ips-outgoing; Sun, 24 Sep 2000 15:38:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA00760 for ; Sun, 24 Sep 2000 15:38:19 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA13119; Sun, 24 Sep 2000 12:14:57 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA22620; Sun, 24 Sep 2000 12:12:41 -0700 From: "Douglas Otis" To: , "IP Storage" Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Sun, 24 Sep 2000 12:29:47 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, Should the IP-SCSI spec indicate a reference to an additional standard that describes methods of extracting information from an LDAP server, you will see the desired uniformity in implementations. This key information will be related to the intrinsic network identification used also by DHCP and would provide the normal behavior of local drives. Such references may also include system user identification or some other pop-up identification to allow a virtual local drive that follows the user or application. All of these reference keys lay beyond any transport standards. How this information ultimately is derived may be a local proxy by means of menus or shared data between LDAP servers. These servers also include their own authentication. Again, as these methods evolve, it would be beneficial to exclude everything except this standards reference. Try to isolate transport encapsulation from transport/SCSI configuration. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > csapuntz@csapuntz-u1.cisco.com > Sent: Friday, September 22, 2000 7:01 PM > To: IP Storage > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > Just for clarification... I was not proposing to add the extended > URL scheme for the transport spec. It isn't necessary. > > In this thread, Doug makes an excellent point about LDAP being a > superior mechanism for describing how to connect to the > storage. Directory services such as LDAP, as Doug has pointed out, > will be critical to managing large quantities of storage. A host can > ask such a directory service for a list of storage devices it should > mount and how to connect to those storage devices. The query against > the directory server that returns this information can be based on > machine ID, user ID, operating system ID, or even the owner's > birthday. One of the things discovery will end up doing, no doubt, is > defining LDAP schemas that describe how to connect to storage > (i.e. use SCTP or TCP, what port, what target name, what LUN, what > WWN, how to authenticate, etc.). > > However, there is one place where the transport protocol has to define > a name: third party commands. There needs to be some kind of global > name which the initiator can pass to the target. The name must be > distillable into a string. The target must understand the name and > be able to use the information to establish a connection to > another target. > > One could say that the string that is passed is not specified by the > standard but instead specified by some management software. I think > this will lead to poor interoperability. > > The SCSI URL-type name is the current proposal for target name. > > Why is SCSI target name made up of a hostname + a path? Why is the > hostname + path passed on connection setup? There are two reasons. > I think NAT and IPv6 makes passing hostnames rather than addresses in > protocols more desirable. Hostnames can be re-resolved as you cross > addressing boundaries. The path is there so that the name can > support multiple targets behind a single IP address without having > to add entries to the DNS server. At Cisco, for example, > I have no control over the local DNS servers and cannot > add DNS entries for the ATAPI DVD and floppy in my computer. > > _Costa > From owner-ips@ECE.cmu.edu Mon Sep 25 06:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17746; 25 Sep 2000 2:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA14748 for ips-outgoing; Mon, 25 Sep 2000 01:32:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA14742 for ; Mon, 25 Sep 2000 01:32:32 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id HAA41574 for ; Mon, 25 Sep 2000 07:31:58 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA120246 for ; Mon, 25 Sep 2000 07:31:54 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256965.001E616D ; Mon, 25 Sep 2000 07:31:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 25 Sep 2000 08:29:36 +0300 Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I am afaraid it will suffer from the same issue. Recovery was there to simplify upper layers and avoid costly multiple operations restarts. Julo csapuntz@csapuntz-u1.cisco.com on 23/09/2000 00:15:06 Please respond to csapuntz@csapuntz-u1.cisco.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: iSCSI: remove recovery from transport-layer connection failure(?) Currently, iSCSI is spec'ed to recover from transport-layer connection failures. The main motivation for this decision was to support tape backup applications that are quite sensitive to any failures that get propogated to their layer. However, there are other ways of doing backup over IP than SCSI/iSCSI. NDMP (www.ndmp.org) and devices that implement it are already shipping. So, perhaps we can remove the requirement of recovering from transport-layer connection failures in iSCSI. This would simplify the protocol somewhat. Thoughts? -Costa From owner-ips@ECE.cmu.edu Mon Sep 25 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18190; 25 Sep 2000 4:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA17190 for ips-outgoing; Mon, 25 Sep 2000 03:35:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA17183 for ; Mon, 25 Sep 2000 03:35:40 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA65128 for ; Mon, 25 Sep 2000 09:35:08 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA163274 for ; Mon, 25 Sep 2000 09:34:52 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256965.0029A489 ; Mon, 25 Sep 2000 09:34:50 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 25 Sep 2000 10:32:06 +0300 Subject: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, I was out on a week vacation and was a bit surprised by the turn of events. I will try to summarize today why the whole design team, and many more, thought that multiple connections are a central feature to iSCSI and why removing it will harm the effort. I thought that we have a rough consensus on this and what we are not fully confident about is if to go for an asymmetric scheme (with separate control and data connections) or a symmetric scheme. Julo From owner-ips@ECE.cmu.edu Mon Sep 25 09:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18421; 25 Sep 2000 5:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA18442 for ips-outgoing; Mon, 25 Sep 2000 04:40:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA18436 for ; Mon, 25 Sep 2000 04:40:13 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id EAA88458 for ; Mon, 25 Sep 2000 04:34:07 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8P8eCT31412 for ; Mon, 25 Sep 2000 02:40:12 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: "IP Storage" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 25 Sep 2000 01:38:33 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/25/2000 02:40:11 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk OK folks, Lets go over this name thing again. First off -- the LU names comes ONLY from the authorized initiator addressing the LU-0 which the Storage Controller permits the (Authorized) initiator to see (yes each initiator may see different LUs, and LU numbers, including different LU-0s). Some of these LUs may be made up of the same storage components but can have different LU numbers as seen by different (Authorized) Initiators. The only way of determining the LUs that an (Authorized) Initiator may address (other then LU-0) is for the (Authorized) Initiator to first perform a Report LUNS command to the LU-0 that an(Authorized) Initiator is presented by the Storage Controller. Then the (Authorized) Initiator will use the information returned by the Report LUNS Command to identify the LU numbers that can be addressed by this specific (Authorized) initiator. The Host's (Authorized) Initiator will now be able to issue the "Inquire" Command to the LU numbers identified above, and at VPD page number 83h find the unique name that can be used to identify the LU. This unique identifier will be the same regardless of the initiator or LU number used to get VPD 83h. It is the matching up of the different LU identifiers which causes Hosts to understand that they have more then one path to a given LU. Now if we understand the above, we can now do a little more about naming. First understand that only the Hosts which are authorized to address specific LUs know what the unique names are, and know the corresponding LU numbers, which can be used by their Initiators. Hosts that do not have authorized initiators, can not address the LUs, even if they know the LU number that is used by another Host's authorized Initiator. Even two different Host with authorized Initiators will not necessarily see the specific LU with the same LU number. So the value of placing the LU number and ID in a Database is more limited then you might have otherwise thought. No system that doesn't already have the authorization and the capability of acquiring the knowledge about the LU can use the information about the LU. This means that the only systems that can place information about an LU into the Database are the ones that are capable of directly using that information, and which are also the ones that can discover the LU information when ever they startup. Having said that, there may be management reasons for having an LU id in the Database. A few of these reasons are for error reporting, error diagnostics, throughput analysis, etc. Below, I will try to explain how the Database may be used with the above information (By the way, I am also in agreement with those folks that believe that the place to record this is in an LDAP Database.) The most important thing to be recorded, in the LDAP Database, is the information about the Device (Storage Controllers), and the Hosts. That is, after a set of IP discovery processes (to be discussed later) and the extraction of the approprate MIB information. With the IP discovered information, about the various storage controllers, an administrator can assign human understandable names. It is also possible for an administrator to have given a name to the Storage Controller before attaching it onto the general network. In this case, the Storage Controller may have already updated the DNS (Distributed Name Server) with its own IP address and Human name. I would also except, again in this case, the Management Software to extracted that information from the DNS and populate its LDAP Database with the approprate name. Given that the names are included in the LDAP Database, an administrator will need to assign the various Storage Controllers to the approprate hosts, and insure that the hosts have paths to get to the Storage Controllers (both physically and logically). That is, if their is to be a VPN setup, the administrator must not only insure that a physical path exist between the Host and its authorized storage controllers, but that a logical path exists. The best way this can be done, of course, is for each IP entity to have a Human understandable name. Once the various entities have Human understandable names, it will be possible for management software to create a methods that will tell the initiators what Target IP address they can use, along with what security tokens to use, etc. Like wise it should be possible for the management software to tell the targets what initiators IP addresses are valid and what security tokens to accept and send etc. You will notice in the above, there is no talk about the LU names. At the moment, each Storage Controller has their own proprietary interface for administrators to use to create LUs and assign them to various initiators. This will, in my opinion need to be done also in the iSCSI environment, and should be done with Human Readable Names. That is, the iSCSI Storage Controller will need to understand what LUs go with what Host initiator names. It is not clear how the LU names will be associated with the approprate LU numbers such that they can some how be matched with the VPD 83h. It is not even clear that there will be a name associated with a LU number for a specific Host initiator name. It might, however, make since that when we define the management process we might want to provide a way for a Central administrator to assign the various LU views to a specific Host initiators, for all the Storage Controllers within some domain. But without that management approach, there will not be a way for the Central Management function, to know what LUs can be addressed without the various Hosts telling it. So I think that before we start talking about the makeup of a name, we should understand, how it is created, and how it will be used. When folks start focusing on Naming, I think we need to put the various naming approaches into the above scenarios, and then we can begin to understand what is of use and of value and what is not. It would also be useful to take the various proposals, along with the above, and insure that they some how fit with the Naming and Mapping section (2.2.6) of the iSCSI Document including the definition of what is a Target Acquired Name (TAN), and how the Map command (3.17) fits into the same picture. Would someone like to take up that challenge and extend the above scenario, that is, carefully layout how they think the names will be created, discoverd, used, and fit with the above. I think I have taken this scenario as far as I can. . . . John L. Hufferd csapuntz@csapuntz-u1.cisco.com@ece.cmu.edu on 09/22/2000 07:00:33 PM Sent by: owner-ips@ece.cmu.edu To: "IP Storage" cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Just for clarification... I was not proposing to add the extended URL scheme for the transport spec. It isn't necessary. In this thread, Doug makes an excellent point about LDAP being a superior mechanism for describing how to connect to the storage. Directory services such as LDAP, as Doug has pointed out, will be critical to managing large quantities of storage. A host can ask such a directory service for a list of storage devices it should mount and how to connect to those storage devices. The query against the directory server that returns this information can be based on machine ID, user ID, operating system ID, or even the owner's birthday. One of the things discovery will end up doing, no doubt, is defining LDAP schemas that describe how to connect to storage (i.e. use SCTP or TCP, what port, what target name, what LUN, what WWN, how to authenticate, etc.). However, there is one place where the transport protocol has to define a name: third party commands. There needs to be some kind of global name which the initiator can pass to the target. The name must be distillable into a string. The target must understand the name and be able to use the information to establish a connection to another target. One could say that the string that is passed is not specified by the standard but instead specified by some management software. I think this will lead to poor interoperability. The SCSI URL-type name is the current proposal for target name. Why is SCSI target name made up of a hostname + a path? Why is the hostname + path passed on connection setup? There are two reasons. I think NAT and IPv6 makes passing hostnames rather than addresses in protocols more desirable. Hostnames can be re-resolved as you cross addressing boundaries. The path is there so that the name can support multiple targets behind a single IP address without having to add entries to the DNS server. At Cisco, for example, I have no control over the local DNS servers and cannot add DNS entries for the ATAPI DVD and floppy in my computer. _Costa From owner-ips@ECE.cmu.edu Mon Sep 25 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19283; 25 Sep 2000 11:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA26950 for ips-outgoing; Mon, 25 Sep 2000 10:04:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA26946 for ; Mon, 25 Sep 2000 10:04:51 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id ACE0BEDD; Mon, 25 Sep 2000 07:04:49 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA15618; Mon, 25 Sep 2000 07:04:31 -0700 (PDT) Message-Id: <4.2.2.20000924204600.00b4ad40@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Sun, 24 Sep 2000 20:51:04 -0700 To: Pierre Labat , Robert Snively From: Michael Krause Subject: Re: ISCSI: flow control Cc: ips@ece.cmu.edu In-Reply-To: <39CE3F8D.F56A4F0B@hp.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 10:53 AM 9/24/00 -0700, Pierre Labat wrote: >T10 could add an advertising of a command credit. For example >when the status is returned the target gives a command credit. In essence, this is what InfiniBand does and others have been advocating. When the ACK (SCSI response) is returned it encodes a credit to inform the sender of how many receives buffers (available command queue slots) have been posted. This allows the sender to know whether it can continue to issue send operations. The cost is trivial in hardware and greatly simplifies the software / control logic since it can post as many commands as it wants on the requester and these will be processed at the rate the receiver can actually consume them. When combined with the RNR NAK operation (still applicable since there may be other reasons for returning this), one can achieve a fairly robust and simple implementation that works across a wide range of products. Mike From owner-ips@ECE.cmu.edu Mon Sep 25 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19219; 25 Sep 2000 11:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA27465 for ips-outgoing; Mon, 25 Sep 2000 10:16:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from nu.nuspeed.com (IDENT:postfix@nu.nuspeed.com [63.169.23.10]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA27450 for ; Mon, 25 Sep 2000 10:16:50 -0400 (EDT) Received: from mail.nuspeed.com (mail.nuspeed.com [172.18.1.3]) by nu.nuspeed.com (Postfix) with ESMTP id 7CD8C8380B; Mon, 25 Sep 2000 09:16:49 -0500 (CDT) Received: from nuspeed.com (marvin.eng.nuspeed.com [172.18.48.16]) by mail.nuspeed.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id TTGX408F; Mon, 25 Sep 2000 09:25:21 -0500 Message-ID: <39CF5E51.16EEA2@nuspeed.com> Date: Mon, 25 Sep 2000 09:16:49 -0500 From: Mark Bakke X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.16-3 i686) X-Accept-Language: en, de MIME-Version: 1.0 To: John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with John that we should keep it. We had put it in so we could recover from lost connections for any SCSI device. Without this mechanism, the only device that would be recoverable is disk. BTW, NDMP is a file-level protocol; it is used by backup products to instruct a file server to write certain files to tape. We still need to be able to write tape at the block level, either directly or via a third-party copy. Even if we end up with a single connection per session, we will still have to acknowledge receipt of status to the target, and be able to recover sessions if the TCP stream is lost for some reason. -- Mark John Hufferd/San Jose/IBM wrote: > > Costa, > This is not a replacement for Tape. Tape is used for other things besides > Backup. Also, NDMP is a Client/Server protocol. Someone still needs to > talk to the tape with SCSI or FC. The actual tape needs to be locatable > anywhere. This is still the need for iSCSI. > > . > . > . > John L. Hufferd > > csapuntz@csapuntz-u1.cisco.com@ece.cmu.edu on 09/22/2000 02:15:06 PM > > Sent by: owner-ips@ece.cmu.edu > > To: ips@ece.cmu.edu > cc: > Subject: iSCSI: remove recovery from transport-layer connection failure(?) > > Currently, iSCSI is spec'ed to recover from transport-layer > connection failures. > > The main motivation for this decision was to support tape backup > applications that are quite sensitive to any failures that get > propogated to their layer. > > However, there are other ways of doing backup over IP than > SCSI/iSCSI. NDMP (www.ndmp.org) and devices that implement it are > already shipping. > > So, perhaps we can remove the requirement of recovering from > transport-layer connection failures in iSCSI. This would simplify > the protocol somewhat. > > Thoughts? > > -Costa -- Mark A. Bakke NuSpeed, Inc. mark.bakke@nuspeed.com 763.398.1054 From owner-ips@ECE.cmu.edu Mon Sep 25 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19230; 25 Sep 2000 11:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA28250 for ips-outgoing; Mon, 25 Sep 2000 10:36:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA28238 for ; Mon, 25 Sep 2000 10:36:04 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 10:35:14 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FD2@corpmx9.isus.emc.com> To: ycheng@advansys.com, ips@ece.cmu.edu Subject: RE: An IPS Transport Protocol (was A Transport Protocol Without A CK) Date: Mon, 25 Sep 2000 10:34:26 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Now the solution. In the IPS protocol breaks down the 1MB data into 25 > 20K-sequences. Each sequence has ten 2K segments. Each sequence will be > acknowledged individually. We define a Delay Constant between the transfer > of two consecutive sequences. On a not-so-busy network, the delay should be > zero. Hence, the sender sends all 25 sequences or 500 segments without > delay. Using a 1 Gb adapter, the whole 1 MB of data goes out in 10 msec. > 25 msec later they arrive at the destination. Each sequence is acknowledged > individually. 25 msec later, all 25 ACKs come back to the sender. The > whole one 1 MB is transferred in 60 msec, not 1.6 sec. Sorry, that doesn't achieve the goal of complying with RFC 2581. If the "Delay Constant" is set incorrectly for the network, the result is disastrous - the whole point of slow start is to set and adjust parameters like the "Delay Constant" automatically in a fashion independent of network conditions. Also see the discussion in RFC 2581 about initial window size. > Personally, I will never implement an iSCSI provider using TCP > stream oriented protocol. I will implement the aforementioned congestion > management in a fibre channel adapter today as an IPS provider. As long as > an IPS provider deals with the PDU's correctly, it should always > interoperate with another node which uses TCP stream oriented protocol. In other words, change the TCP protocol to make it go faster for your traffic without changing the TCP header format. Not only is this not a good idea, but the approved WG charter specifically PROHIBITS the WG from working on this, unless the argument can be made that the changes are required for storage traffic. I believe the current WG consensus is that TCP congestion control is good enough for storage traffic, and hence would like to put an end to this thread. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Sep 25 15:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19324; 25 Sep 2000 11:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA28826 for ips-outgoing; Mon, 25 Sep 2000 10:48:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA28820 for ; Mon, 25 Sep 2000 10:48:20 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 10:47:47 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FD3@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: multiple connections Date: Mon, 25 Sep 2000 10:47:19 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, I have to ask you not to do that. There is consensus in the WG that multiple connection sessions are an important feature that needs to be specified, but should be optional to implement. There is NOT consensus on what the right design approach is. I have off-line email from proponents of both the Asymmetric and Symmetric multi-connection session models expressing dismay at the separation of them from the main specification and arguing that their preferred approach is the right one. These reinforce my observations that there is no consensus on the issue, and that the mailing list discussion is unlikely to achieve consensus. As I stated in earlier email, the requirement for multiple connection sessions has not been removed, but spending the next 6-8 weeks discussing it on the mailing list does not appear likely to achieve consensus. We need to try something else, namely an off-line design team. In the near term, list bandwidth is better used to make progress on issues where progress is still possible, such as flow control. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Monday, September 25, 2000 3:32 AM > To: ips@ece.cmu.edu > Subject: multiple connections > > > > Dear colleagues, > > I was out on a week vacation and was a bit surprised by the turn of > events. > I will try to summarize today why the whole design team, and many more, > thought that multiple connections are a central feature to iSCSI and why > removing it will harm the effort. > I thought that we have a rough consensus on this and what we are not fully > confident about is if to go for an asymmetric scheme (with separate > control > and data connections) or a symmetric scheme. > > Julo > From owner-ips@ECE.cmu.edu Mon Sep 25 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19371; 25 Sep 2000 11:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA29420 for ips-outgoing; Mon, 25 Sep 2000 11:01:15 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA29412 for ; Mon, 25 Sep 2000 11:01:10 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id LAA18725 for ; Mon, 25 Sep 2000 11:01:02 -0400 Message-ID: <00a601c026fe$bac427e0$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: Subject: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Mon, 25 Sep 2000 10:41:36 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id LAB29420 Stephen Byan wrote: >Michael Krause [mailto:krause@cup.hp.com] wrote: > >> [snip] >> RDMA !=3D VI though VI does use RDMA technologies. > >Agreed. It is an example > >> Prefer to see discussion focused on what RDMA operations are >> required, what are the error and ordering requirements, etc. > [snip] >> [snip] >> As such, a general >> purpose RDMA solution which operates over TCP/IP is the optimal >> solution to pursue since it will lead to the broadest industry >> and customer adoption rate. > >I think we are in complete agreement. > >Regards, >-Steve In response to this I would offer the following proposal with the caution that it is very preliminary and has not been analyzed or reviewed. But I thought it might be worth posting in order to see what the general response to this approach is. ############################################################## RDMA / TCP 1. Abstract This document describes a format for encapsulating RDMA (remote direct memory access) information within a TCP data stream. No changes or modification to TCP of any sort are required. This is not intended to be a protocol, but rather a common format that may be shared by multiple client protocols, for instance VI/TCP and iSCSI. By using a common format it is hoped that design of NICs supporting these multiple protocols can be simplified. Sufficient information is included in the RDMA message format to allow determination of the protocol message units, as will as the ability to process an incoming RDMA request even if previous packets are missing and awaiting retransmission. In addition a CRC-32 is included in each segment to enhance the checksum coverage included in TCP. 2. Overview Data transfers consist of a sequence of messages. Each message is of one of four types: Send, RDMA_write, RDMA_Read_Request, and RDMA_Read_Response. The maximum size of a message is approximately 2^32. Each message is divided into one or more segments. It is RECOMMENDED that each TCP segment contain exactly one RDMA segment. The receive end of the connection cannot assume any alignment between the RDMA segments and TCP segments, however a receiver SHOULD optimize performance for the case where each TCP segment contains exactly one RDMA segment. 3. RDMA Segment Format The format of an RDMA segment depends on the message type. Shown below are the formats for the four different types of messages. All multibyte formats are to be represented in network byte order (i.e., big-endian). 3.1 Send and RDMA_Read_Response Message Type: | Byte 0 | Byte 1 | Byte 2 | Byte 3 | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ | Version | res |B|E|typ| Segment Length | +---------------+---------------+---------------+---------------+ | | + Connection ID + | | +---------------+---------------+---------------+---------------+ | Message Number | +---------------+---------------+---------------+---------------+ | order | res. | CLEN | +---------------+---------------+---------------+---------------+ | Data Offset | +---------------+---------------+---------------+---------------+ | | | Control Data | | | +---------------+---------------+---------------+---------------+ | | | | | Payload Data | | | | | + +---------------+---------------+ | | Padding | +---------------+---------------+---------------+---------------+ | CRC-32 | +---------------+---------------+---------------+---------------+ 3.2 RDMA_Write Message Type: | Byte 0 | Byte 1 | Byte 2 | Byte 3 | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ | Version | res |B|E|typ| Segment Length | +---------------+---------------+---------------+---------------+ | | + Connection ID + | | +---------------+---------------+---------------+---------------+ | Message Number | +---------------+---------------+---------------+---------------+ | order | res. | CLEN | +---------------+---------------+---------------+---------------+ | RDMA Buffer ID | +---------------+---------------+---------------+---------------+ | RDMA Buffer offset | +---------------+---------------+---------------+---------------+ | RDMA Length | +---------------+---------------+---------------+---------------+ | | | Control Data | | | +---------------+---------------+---------------+---------------+ | | | Payload Data | | | | | + +---------------+---------------+ | | Padding | +---------------+---------------+---------------+---------------+ | CRC-32 | +---------------+---------------+---------------+---------------+ 3.3 RDMA_Read_Request Message Type: | Byte 0 | Byte 1 | Byte 2 | Byte 3 | |7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0|7 6 5 4 3 2 1 0| +---------------+---------------+---------------+---------------+ | Version | res |B|E|typ| Segment Length | +---------------+---------------+---------------+---------------+ | | + Connection ID + | | +---------------+---------------+---------------+---------------+ | Message Number | +---------------+---------------+---------------+---------------+ | order | res. | CLEN | +---------------+---------------+---------------+---------------+ | RDMA Buffer ID | +---------------+---------------+---------------+---------------+ | RDMA Buffer offset | +---------------+---------------+---------------+---------------+ | RDMA Length | +---------------+---------------+---------------+---------------+ | | | Control Data | | | +---------------+---------------+---------------+---------------+ | CRC-32 | +---------------+---------------+---------------+---------------+ Note that RDMA_Read_Request messages always consist of exactly one segment and contain no payload data. 3.4 Segment Field Definitions Version: The version number refers to the version of the RDMA format, not to that of the client protocol. This document defines version 1, so this field should contain 0x1. Res: These four bits are reserved and not used by the RDMA mechanism. They may be used by the client protocol. B, E: These are the begin and end bits. The indicate that this segment is the beginning or end, respectively, of the message to which is belongs. Either, both, or neither of these bit may be set for a given segment. Type: Four Types of messages are supported. These are 0 =96 Send, 1 =96 RDMA_Write, 2 =96 RDMA_Read_Request, and 3- RDMA_Read_Response. Segment Length: This field contains the length of the RDMA segment in bytes. This length includes the RDMA segment header and payload up to, but not including, the padding and CRC. Connection ID: The Connection ID is a 64 bit value selected at random. This value is selected by the client side of the connection and included in the first message segment sent over the connection. The same value is then used for all subsequent segments sent in either direction. It is RECOMMENDED that a secure un-guessable random number generator be used to generate these values. The Connection ID serves two purposes. It allows framing to be recovered after a dropped segment, and it provides security against blind attacks. Message Number: For Send, RDMA_Write, and RDMA_Read_Request messages, the message number is assigned sequentially for each message, wrapping from 2^32-1 to 0. All three types of messages are part of a single sequence. For RDMA_Read_Response messages, the message number should be set equal to the originating RDMA_Read_Request. The initial message number for the first (non RDMA_Read_Response) message sent in each direction SHOULD be selected at random. Order: This 16 bit field defines the ordering requirements on a message. A value of "N" in the order field indicates that the payload data may not be read from or written to its ultimate source or destination until all EXCEPT the preceding N messages have been processed. The value of 0xFFFF (all one bits) is reserved to indicate that the operation may be done immediately when received unconditionally. As an example of the above, if a sequence of RDMA_Write and RDMA_Read_request messages are received with an order field containing zero, then the operations must be done in the order received, however the individual segments of a given RDMA_Write message may be written to the target buffer in arbitrary order. If an RDMA_Write or RDMA_Read_Request message is received with an order field containing 5 and a message number of 97, then message number 91 and all previous messages must have been processed. For a Send message, out of order processing implies that the client protocol actually receives these messages out of order. If, for instance, the send messages contain commands, the value in the order field of these messages should not allow reordering unless the client protocol is allowed to process the contained commands out of order. CLEN, control data: This 8 bit CLEN field defines the length of the control data included with a message. The value of CLEN is the number of 32 bit words of control data included. The meaning of the control data is determined by the client protocol, but the significance is that it is not part of the RDMA transfer and should not be written to the RDMA target or read response buffer. For Send messages, the control data designation is only for the convenience of the client protocol, and the only difference between control and payload data is that control data is not counted towards the computation of the data offset. Typically the control data will contain header information for the client protocol in addition to that provided by the RDMA segment format. Data Offset: This field is contained in Send and RDMA_Read_Response messages. The first segment of a message must have a data offset of zero. In each subsequent segment of the message, the offset will be equal to the number of payload bytes sent in all previous segments of the message. RDMA Buffer ID, RDMA Buffer Offset: These values determine the target address of an RDMA read or write. The value of the RDMA Buffer ID must be constant across all segments of an RDMA_Write. The value of the RDMA Buffer Offset can be anything in the first segment of an RDMA_Write, but must be incremented in each subsequent segment by the number of bytes transferred. The exact interpretation of these values is determined by the client protocol, however it is expected that the RDMA Buffer ID, possibly combined with some bits from the RDMA buffer offset, will be used as an index into a table of buffers. The actual data transfer will occur to or from this buffer starting at an offset determined by the RDMA buffer offset, or some bits extracted from the RDMA buffer offset. Unfortunately because of the differing addressing models used by different client protocols, it is not possible to exactly specify how buffer ID and offset are resolved to a physical address in the NIC. It is hoped, however, that even with this protocol dependent feature, the commonality in the RDMA format should allow more efficient implementation of protocol accelerating NICs that support multiple protocols requiring RDMA. RDMA Length: Indicates the number of bytes to be transferred in an RDMA operation. In the case of an RDMA_Write, this is the total number of bytes in the entire message, and the same value must be repeated in each segment of the message. Padding: Between 0 and 3 bytes of padding are used to make the segment a multiple of 4 bytes in length. The padding MUST be set to zero by the sender and ignored by the receiver. CRC-32 The CRC-32 is calculated across the entire segment (but does not cover other segments of the same message, or lower level protocol headers such as TCP). The algorithm used to calculate the CRC is exactly that used for the ethernet CRC except that a different generator polynomial is used. The generator polynomial for the RDMA CRC is x^32 + x^31 + x^30 + x^28 + x^27 + x^25 + x^24 + x^22 + x^21 + x^20 + x^16 + x^10 + x^9 + x^6 + 1. This polynomial is the standard ethernet polynomial with a left-right reversal. (Or mathematically, substitute y =3D x^-1 and multiply by y^32). In hex format with the x^32 term removed, this is 0xDB710641. It is desirable to use a different polynomial than ethernet so that when an RDMA segment is carried in an ethernet packet, the combined protection of two different polynomials is achieved, rather than checking twice with the same polynomial. [ Add reverence for ethernet CRC and detailed computation algorithm. ] 4. Segments and Messages The four types of messages are divided into two groups. The first group consists of Send, RDMA_Write, and RDMA_Read_Request messages, and the second group consists of RDMA_Read_Response messages. Within each of these two groups, all messages must be sent in order. Each message is divided into one or more segments, and all the segments of a particular message are sent in order. All segments of one message must be sent before the first segment of the next message is sent. However between the two groups, segments may be interleaved arbitrarily. [ Show example of a series of segments following these rules. ] 5. Determination of Framing The beginning of the first segment on a TCP connection occurs of course starting with the first data byte sent. Given the start of a segment, the start of the next segment can be determined by noting the length field in the header of the segment, rounding up to the next multiple of four (to account for padding) and adding four (for the CRC) and moving forward that many bytes in the TCP data stream. In this manner, the beginning of each segment can be determined from the last. When a packet is dropped, however, it is desirable to recover framing on subsequent segments so that they might be processed by the NIC and their payload data placed directly in its ultimate destination. The recommended method for doing this is to assume that the RDMA segment is aligned with a TCP segment, and verify the correctness of the header fields of the RDMA segment. If these header fields are not correct, then the NIC should fall back to buffering the packet until it can be processed in order. In particular, the 64 bit connection ID field was selected at random, so the only way that could match payload data is by pure chance. It can be easily shown that even if a miss-aligned packet arrives every 2us, the MTBF of mistakenly identifying this as an aligned packet is greater than one million years. Checking the message number, CRC, and other fields only enhances the confidence in this determination. From owner-ips@ECE.cmu.edu Mon Sep 25 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19369; 25 Sep 2000 11:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA29728 for ips-outgoing; Mon, 25 Sep 2000 11:09:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA29721 for ; Mon, 25 Sep 2000 11:09:07 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 11:08:37 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FD5@corpmx9.isus.emc.com> To: jimw@giganet.com, ips@ece.cmu.edu Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Mon, 25 Sep 2000 11:08:36 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > In response to this I would offer the following proposal with > the caution that it is very preliminary and has not been > analyzed or reviewed. But I thought it might be worth posting > in order to see what the general response to this approach is. I would ask those who make these sorts of proposals to PLEASE write them up as Internet-Draft and submit them to the I-D servers. Outside the few weeks before an IETF meeting, the submission turn-around is rapid, and having something like this written up as an I-D avoids it vanishing into the mailing list archive where it's more difficult to retrieve. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Sep 25 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19364; 25 Sep 2000 11:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA29608 for ips-outgoing; Mon, 25 Sep 2000 11:06:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA29603 for ; Mon, 25 Sep 2000 11:06:12 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 11:05:26 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FD4@corpmx9.isus.emc.com> To: krause@cup.hp.com Cc: ips@ece.cmu.edu Subject: RE: ISCSI: flow control Date: Mon, 25 Sep 2000 11:05:25 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, > In essence, this is what InfiniBand does and others have been > advocating. When the ACK (SCSI response) is returned it encodes a credit > to inform the sender of how many receives buffers (available command queue > slots) have been posted. Could you post, or provide a pointer to a self-contained specification of that mechanism? If this is a pointer to InfiniBand specs, a heads-up on any intellectual property issues is in order. A concern that has been raised in this discussion is how credit information relates to the concurrency and ordering (esp. lack thereof) of transmission and processing of SCSI commands and the transmission of responses. My understanding of the FCP approach to buffer management (and I assume InfiniBand is similar) is that traffic cannot be sent unless the sender knows that there is space in the receiver's buffer to accommodate it (i.e., the sender has a credit or credits indicating space in the receiver's buffer). This implies is that if for some reason the receiver stalled, all the in-flight commands and data could be successfully received. In contrast, I've seen discussion on this list of long distance connections in which there is potentially more traffic in flight than the receiver could accommodate if the receiver stopped. I believe that whether to allow this is an open issue, but the underlying cause is valid - there is a desire to use iSCSI in situations where the initiator to target coupling is looser (in this case, due to distance) than is typical for SCSI and Fibre Channel. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Sep 25 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19572; 25 Sep 2000 13:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA01359 for ips-outgoing; Mon, 25 Sep 2000 11:46:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA01348 for ; Mon, 25 Sep 2000 11:46:45 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDL074; Mon, 25 Sep 2000 08:44:47 -0700 From: "Y P Cheng" To: , Subject: RE: Last Word on An IPS Transport Protocol? Date: Mon, 25 Sep 2000 08:44:51 -0700 Message-ID: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100FD2@corpmx9.isus.emc.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > "Delay Constant" is set incorrectly for the network, the result is > disastrous - the whole point of slow start is to set and adjust parameters > like the "Delay Constant" automatically in a fashion independent of network > conditions. Also see the discussion in RFC 2581 about initial window size. If slow start is needed to avoid disaster on the network, I don't have any problem with a slow start. However, the cost on performance -- taking 1.6 sec to transfer one MB of data using slow start -- must be considered. When the MTU is small, 512 instead of 2K, the cost is even higher. How long does it take to move one gigabytes of data with iSCSI on a network with 50 msec round-trip delay? > In other words, change the TCP protocol to make it go faster for your > traffic without changing the TCP header format. Not only is this > not a good idea, but the approved WG charter specifically PROHIBITS the > WG from working on this, unless the argument can be made that the > changes are required for storage traffic. I believe the current WG consensus > is that TCP congestion control is good enough for storage traffic, > and hence would like to put an end to this thread. Dave, I think you have missed a very important part of my proposal. We must stream the data transfer with DMA assist to overcome the performance problem of storage traffic on a network with long delay. Changing or not the TCP header is not my point at all. The proposed protocol does not change the TCP protocol either if the sequence size is set to one, i.e. the segment size. My contention is the current TCP congestion control is NOT good enough and the ACK traffic on a network with long latency delay is BAD. We must have streamed transfer on a network with long latency. Therefore, defining the ACK of TCP is critical. The TCP header format is not sacred to me. If the WG does not see this point and believes otherwise, I will submit to the wish of this WG. But I do believe the current discussion has totally missed the effect of the long latency network delay. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Mon Sep 25 17:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19580; 25 Sep 2000 13:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA02143 for ips-outgoing; Mon, 25 Sep 2000 12:04:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lmoxch11.nsmg.veritas.com (london-bridge.east.veritas.com [207.30.27.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA02139 for ; Mon, 25 Sep 2000 12:04:25 -0400 (EDT) Received: by lmoxch11.nsmg.veritas.com with Internet Mail Service (5.5.2448.0) id ; Mon, 25 Sep 2000 12:05:13 -0400 Message-ID: From: Roger Cummings To: Charles Monia Cc: ips@ece.cmu.edu Subject: RE: LU access through an iSCSI session Date: Mon, 25 Sep 2000 12:05:06 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, Jim Hafner said the same thing when the question came up with regards to his Access Control proposal in T10, and a bunch of people (including myself) went and looked at SAM as a result, and I believe that there was a good consensus in T10 that this is correct. Regards, Roger > -----Original Message----- > From: Charles Monia [mailto:cmonia@NishanSystems.com] > Sent: Saturday, September 23, 2000 7:59 PM > To: meth@il.ibm.com; Robert Snively; ips@ece.cmu.edu > Subject: RE: LU access through an iSCSI session > > > Hi: > > Strangely enough, while we're on the subject, I don't believe SAM-xx > requires a target to present the same view of its logical > units to every > initiator regardless of the number of NICs. I suspect there > may even be > products out there that exploit this "feature". > > Charles > > > > -----Original Message----- > > From: meth@il.ibm.com [mailto:meth@il.ibm.com] > > Sent: Saturday, September 23, 2000 1:19 PM > > To: Robert Snively; ips@ece.cmu.edu > > Subject: RE: LU access through an iSCSI session > > > > > > > > > > > > There should be no problem. It is similar to the case where > > there are 2 > > different SCSI busses connecting a host to a controller. Each > > iSCSI session > > is like a separate virtual SCSI bus. The LU names for the > > same device that > > is reached via the different paths may be different. The way > > for the host > > to tell that they are actually the same device is though the > > Vital Product > > Data, which uniquely identifies the LU. > > > > - Kalman > > > > Robert Snively on 21/09/2000 18:42:44 > > > > Please respond to Robert Snively > > > > To: Kalman Meth/Haifa/IBM@IBMIL, Pierre Labat > , > > ips@ece.cmu.edu > > cc: > > Subject: RE: LU access through an iSCSI session > > > > > > > > > > > > > LU numbers are identical across all of the TCP connections of a > > > single iSCSI session. > > > > This is an intriguing concept. At present, SCSI does not require > > logical units to have the same LUN if they are accessed through > > different NICs. However, present SCSI is essentially one NIC > > per session. > > > > If you have an iSCSI session using NIC A and NIC B and a > > second session > > using NIC B and NIC C as physical paths, by this definition, > > NIC B could have two different LUNs for the same logical unit, one > > from each session. > > > > That should not be a problem, right? > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Sep 25 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 19:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 19:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 20:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 20:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 20:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 20:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 21:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 21:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 22:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 22:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 22:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 23:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 23:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 23:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Sep 25 23:50:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 00:10:04 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 00:20:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 00:30:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 00:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 01:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 01:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 01:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 01:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 02:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 02:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 02:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 02:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 03:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 03:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 03:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 04:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 04:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 04:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 04:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 04:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 05:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 05:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 06:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 06:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 06:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 06:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 06:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 07:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 07:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 07:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 07:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 08:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 08:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 09:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 09:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 09:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 09:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 09:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 10:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 10:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 10:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 10:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 10:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 11:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 11:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 11:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 11:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 11:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 12:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 12:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 12:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 12:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 12:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 12:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 13:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 13:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 13:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 13:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 13:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 14:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 14:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 14:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 14:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 15:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 15:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 16:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 16:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 16:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 17:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 17:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 17:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 18:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 18:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 19:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 19:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 20:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 20:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 20:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 21:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 22:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 23:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 23:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 23:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Tue Sep 26 23:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 00:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 00:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 00:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 00:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 01:00:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 01:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 01:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 02:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 02:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 02:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 02:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 03:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 03:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 03:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 03:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 04:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 04:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 04:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 04:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 05:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 06:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 06:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 06:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 06:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 06:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 07:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 07:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 08:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 08:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 08:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 09:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 09:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 09:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 09:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 09:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 10:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 10:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 10:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 10:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 10:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 11:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 11:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 11:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 11:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 11:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 12:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 12:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 12:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 12:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 12:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 12:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 13:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 13:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 13:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 13:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 13:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 14:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 14:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 14:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 15:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 16:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 16:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 16:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 16:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 18:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 18:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 19:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 19:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 20:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 20:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 20:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 20:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 21:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 21:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 21:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 22:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 22:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 23:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 23:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 23:30:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 23:40:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Wed Sep 27 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 00:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 00:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 00:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 01:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 01:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 01:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 02:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 02:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 02:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 02:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 03:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 03:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 03:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 03:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 04:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 04:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 04:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 05:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 05:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 05:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 06:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 06:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 06:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 06:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 06:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 07:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 07:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 07:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 08:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 08:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 09:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 09:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 09:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 09:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 09:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 10:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 10:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 10:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 10:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 10:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 10:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 11:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 11:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 11:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 11:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 11:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 11:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 12:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 12:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 12:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 12:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 12:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 12:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 13:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 13:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 13:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 13:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 14:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 14:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 14:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 15:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 15:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 15:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 16:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 16:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 17:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 17:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 18:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 18:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 18:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 19:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 19:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 20:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 20:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 20:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 21:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 21:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 22:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 22:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 23:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 23:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 23:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Thu Sep 28 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 00:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 00:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 00:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 00:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 01:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 01:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 01:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 02:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 02:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 02:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 03:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 03:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 04:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 04:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 04:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 04:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 05:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 05:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 06:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 06:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 06:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 06:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 06:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 07:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 07:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 07:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 08:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 09:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 09:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 09:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 09:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 09:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 10:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 10:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 10:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 10:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 10:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 11:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 11:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 11:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 11:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 11:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 11:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 12:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 12:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 12:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 12:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 12:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 12:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 13:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 13:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 13:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 13:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 13:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 13:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 14:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 14:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 14:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 15:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 15:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 16:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 16:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 16:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 16:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 17:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 17:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 18:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 18:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 19:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 19:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 19:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 20:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 20:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 20:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 21:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 21:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 21:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 22:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 22:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 22:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 23:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 23:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 23:30:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 23:40:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Fri Sep 29 23:50:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 00:00:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 00:10:08 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 00:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 00:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 01:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 01:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 01:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 01:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 02:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 02:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 02:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 02:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 03:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 03:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 03:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 04:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 04:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 04:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 05:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 05:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 06:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 06:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 06:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 06:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 06:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 06:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 07:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 07:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 07:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 07:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 07:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 08:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 08:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 09:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 09:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 09:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 09:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 09:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 10:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 10:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 10:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 10:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 10:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 11:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 11:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 11:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 11:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 11:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 12:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 12:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 12:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 12:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 12:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 12:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 13:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 13:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 13:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 13:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 13:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 13:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 14:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 14:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 14:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 14:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 15:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 15:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 15:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 16:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 16:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 17:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 17:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 18:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 18:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 19:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 19:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 19:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 19:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 20:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 20:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 20:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 20:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 21:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 21:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 21:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 22:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 22:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 22:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 23:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 23:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 23:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 23:30:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 23:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sat Sep 30 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 00:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 00:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 00:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 00:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 01:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 01:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 01:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 01:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 01:50:03 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 02:00:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 02:10:08 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 02:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 02:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 02:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 03:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 03:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 03:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 04:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 04:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 04:50:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 05:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 05:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 05:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 05:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 05:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 06:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 06:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 06:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 06:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 06:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 07:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 07:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 07:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 07:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 08:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 08:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 09:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 09:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 09:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 09:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 09:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 09:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 10:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 10:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 10:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 10:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 10:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 11:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 11:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 11:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 11:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 11:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 11:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 12:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 12:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 12:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 12:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 12:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 12:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 13:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 13:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 13:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 13:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 13:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 14:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 14:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 14:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 14:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 15:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 15:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 15:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 16:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 16:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 17:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 17:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 18:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 18:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 19:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 19:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 19:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 19:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 20:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 20:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 20:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 20:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 21:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 21:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 21:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 21:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 22:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 22:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 22:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 22:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 22:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 23:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 23:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 23:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 23:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 23:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Sun Oct 01 23:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 00:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 00:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 00:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 00:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 01:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 01:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 01:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 01:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 01:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 01:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 02:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 02:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 02:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 02:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 02:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 02:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 03:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 03:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 03:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 03:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 03:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 04:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 04:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 04:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 04:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 04:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 05:10:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 05:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 05:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 05:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 05:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 06:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 06:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 06:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 06:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 06:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 06:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 07:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 07:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 07:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 07:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 07:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 07:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 08:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 08:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 08:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 08:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 08:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 09:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 09:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 09:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 09:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 09:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 09:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 10:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 10:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 10:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 10:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 10:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 10:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 11:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 11:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 11:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 11:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 11:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 11:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 12:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 12:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 12:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 12:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 12:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 12:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 13:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 13:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 13:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 13:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 13:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 14:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 14:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 14:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 14:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 14:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 14:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 15:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 15:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 15:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 15:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 16:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 16:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 16:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 16:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 16:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 17:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 17:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 17:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 17:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 17:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27798; 27 Sep 2000 13:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA13323 for ips-outgoing; Wed, 27 Sep 2000 12:07:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA13313 for ; Wed, 27 Sep 2000 12:07:38 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id SAA309594 for ; Wed, 27 Sep 2000 18:06:57 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA41940 for ; Wed, 27 Sep 2000 18:06:56 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256967.005884BC ; Wed, 27 Sep 2000 18:06:51 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 27 Sep 2000 19:04:34 +0300 Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Although I think that this a very interesting and relevant thread and we are only scraping its surface can't we just postpone it until after we have the protocol fixed and we start handling discovery and management? Julo ---------------------- Forwarded by Julian Satran/Haifa/IBM on 27/09/2000 19:01 --------------------------- csapuntz@cisco.com on 18/09/2000 09:11:24 Please respond to csapuntz@cisco.com To: IP Storage cc: csapuntz@cisco.com (bcc: Julian Satran/Haifa/IBM) Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Here is a proposal to extend the iSCSI URL scheme to support LUNs and WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This extension is not required for the iSCSI transport protocol, but may be useful for discovery and management. A URL for the target has the following form: scsi://hostname/path/with/ A URL referring to a specific LU has the following form: scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber If no LUN= term appears in the URL, then LUN 0 is assumed. The WWN= term is optional. If present, the party should verify that the WWN in the LU's Device Identification Inquiry Page correspojnds to the WWN. Note, the following URL: scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After connecting, the initiator should verify that the WWN of the LU is 0a050a4bcdefa. -Costa Peter Johansson on 18/09/2000 01:03:10 Please respond to Peter Johansson To: IP Storage cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI: 2.2.6. Naming & mapping At 11:37 AM 9/17/00, John Hufferd/San Jose/IBM wrote: >As specified by Jim Hafner, there is no need and we should not do what >you suggest. We have talked this through before, many, many times, and >the answer has always come out the same. PLEASE, lets not go there again. Jim Hafner's observation (that mandatory presence of a unique ID, on a per LU basis, in EVPD appears to be destined for adoption within SPC-2) only reinforces that part of the discussion with which I already agree. It is desirable to have command set-dependent methods to determine the unique ID for a LU. Matter closed? It might also be desirable to have transport-dependent mechanisms to specify unique IDs for those LUs whose existence is discoverable by transport-dependent methods. I think this is still open to discussion. There's not much I can respond to in your comment above, John, because "... we have talked this through before ..." doesn't constitute much of a technical argument. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org "Jim Hafner/Almaden/IBM" on 18/09/2000 18:31:04 Please respond to "Jim Hafner/Almaden/IBM" To: csapuntz@cisco.com cc: IPS@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Costa, I've seen Daniel's proposal before and I still haven't figured out the context in which such names are to be used. Given that, I'm personally a bit uncomfortable with any naming convention that uses LUN values in a global context. These have no meaning as LUNs are host-specific values. Even worse, LUNs are addresses, not names! I have no problem with WWNs in a global context as that is what they are for! A subtle difficulty, is that a WWN does not give a host an address (either for the target or the LU). Even in SPC-2, EXTENDED COPY's Identification Descriptor Target Descriptor Format (sic?) where WWNs are used, says "instructs the copy manager to locate a target and logical unit that returns a device identification VPD page ..." but gives no hints on how that should be done (e.g., walk the bus and do INQUIRY to everything...?). The "hostname" and DNS provide a canonical method for getting an address for a name, so that part is OK. The only defined way to get an address (LUN) for a LU WWN is by exhaustive INQUIRY to each LU at a given target. On the other hand, I can envision how some of these things might work, in a well-coordinated and well-managed environment: There is a managing application which has the dual responsibility of managing the targets for their host/LUN mapping configurations and also serves as the "walk the bus" discovery resource for hosts. When the manager (human?) determines the distribution of host/target LU resources (after inventorying both hosts and target LUs), the application will send to the targets whatever host/LUN Mapping (access controls?) commands are necessary to configure them. Later, when a host boots it queries the application to get its "walk-the-bus" services. In this, it minimally gets from this application a list of targets that have LUs to which it should have access. There is no REQUIREMENT at this point for the application to give the host anything more as the normal SCSI LU discovery process per target kicks in. On the other hand, I can see some value in the host getting more information about the LUs it will see at each target. In that case, the host could get from the application a list of targets and LU identifiers (these could be LUNs or WWN or both, as you suggested). The important point here, is that these LUNs are valid only in the context of the requesting HOST and are not global! Note that this process requires clear coordination between the "target configurator" and "host discovery of targets". Given an appropriate context for a URL scheme, I'd make two modifications: 1) drop the assumption that no LUN means LUN=0. If no LU qualifier (e.g., no LUN or WWN (or other) is provided), then LUN=0. In all other cases, the alternative names should be sufficient to identify a LU, though, as above, finding a LUN address may require extra work. 2) add ?ProxyToken= as an additional option here (this coordinates well with the changes to EXTENDED COPY approved in the context of the SCSI access controls). Well, that's another two cents! I was hoping to save some of this discussion until the protocol gets worked out, but .... Jim Hafner csapuntz@cisco.com@ece.cmu.edu on 09-17-2000 11:11:24 PM Sent by: owner-ips@ece.cmu.edu To: IP Storage cc: csapuntz@cisco.com Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Here is a proposal to extend the iSCSI URL scheme to support LUNs and WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This extension is not required for the iSCSI transport protocol, but may be useful for discovery and management. A URL for the target has the following form: scsi://hostname/path/with/ A URL referring to a specific LU has the following form: scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber If no LUN= term appears in the URL, then LUN 0 is assumed. The WWN= term is optional. If present, the party should verify that the WWN in the LU's Device Identification Inquiry Page correspojnds to the WWN. Note, the following URL: scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After connecting, the initiator should verify that the WWN of the LU is 0a050a4bcdefa. -Costa From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25593; 27 Sep 2000 1:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA23551 for ips-outgoing; Wed, 27 Sep 2000 00:09:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA23543 for ; Wed, 27 Sep 2000 00:08:58 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8R48wU17413 for ; Tue, 26 Sep 2000 23:08:58 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA03037; Tue, 26 Sep 00 23:07:17 -0500 Message-Id: <10009270407.AA03037@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI: Session Partial Resolution In-Reply-To: Message from Charles Monia of "Tue, 26 Sep 2000 17:35:22 MST." Date: Tue, 26 Sep 2000 18:07:32 -1000 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > In general, such requests are "think-time" limited and therefore non > blocking, so this seems not to be a problem in practice. The abort task operation can be pretty heavyweight. It frequently involves a lot of non-nominal, antioptimized (slowed down so something else can be fast) behavior. That said, I'm not sure why it matters whether task management is fast or slow. Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31066; 28 Sep 2000 7:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA17744 for ips-outgoing; Thu, 28 Sep 2000 06:39:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA17737 for ; Thu, 28 Sep 2000 06:39:47 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id MAA183906 for ; Thu, 28 Sep 2000 12:39:15 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA46004 for ; Thu, 28 Sep 2000 12:39:15 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256968.003A83D7 ; Thu, 28 Sep 2000 12:39:08 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 28 Sep 2000 13:36:40 +0300 Subject: Re: LU access through an iSCSI session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Pierre, Sorry for the delay. 1. Yes. The session is the delivery-port (in SAM terminology). 2.As in 1 - it is the same LUN as a session is a delivery-port and yes you can do load balancing based on connection capalities and the need perceived by the initiator. Thanks for your patience, Julo Pierre Labat on 19/09/2000 20:44:06 Please respond to Pierre Labat To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: LU access through an iSCSI session Julo, Thinking about what could be an implementation of an iSCSI session, several questions came to my mind for which i was unable to find an answer in the draft. 1) Do commands directed to a particular LU through a session are able to use any of the TCP connections within the session? As the TCP connections could connect to various "service delivery ports" it is not guarantee that a particular LU is visible through all the connections. 2) Do commands directed to a particular LU through a session need to be associated to different LUNs depending on the TCP connection used or is it always the same LUN for all the connections? As the TCP connections could connect to various "service delivery ports" of the target, it seems that the LUN could differ depending on the connection used. In the iSCSI draft in 2.2.1 there is: "The group of TCP connections linking an initiator with a target forms a session (loosely equivalent to a SCSI nexus)." Does it means that the same LUN is used across the TCP connections? Depending on the answer to these questions the failover/load balancing can be handled in different ways: -> If the answer for 1) is "All TCP connections can be used" and the answer for 2) is "Same LUN regardless of the connexion" For fail over and load balancing inside a session, there is no need to work at the (thin) LU granularity, working at the granularity of the TCP connexion is sufficient. For example in case of failed TCP connexion all the traffic for this failed TCP connexion can be directed on the other connections regardless of the LU. It is very quick, straight forward and may add value compared to the legacy solutions that are doing the fail over on a LU basis (one LU command returns an error or timeout, the next commands for this LU will use an alternate path) In the case where a large number (N) of LUs access are multiplexed to a target port, N switch over are needed. The same result could be achieved in one operation (TCP connection switch over). -> If the answer for 1) is "All TCP connections can be used" and the answer for 2) is "different LUN depending on the connexion" We need to work at the LU granularity. For each command the right LUN as to be determined depending on the tcp connection selected. -> If the answer for 1) is "not all TCP connections can be used, it is LU dependent" On top of the previous case extra work, for each LU it must be a list of TCP connections usable that needs to be looked into before routing the command to one TCP connection. It means that some LUs access using a session could survive to a TCP connection failure some other LUs no. Regards, Pierre From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20302; 25 Sep 2000 17:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA13746 for ips-outgoing; Mon, 25 Sep 2000 16:26:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA13741 for ; Mon, 25 Sep 2000 16:25:56 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 16:25:24 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FE1@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, Black_David@emc.com Cc: ips@ece.cmu.edu Subject: RE: multiple connections Date: Mon, 25 Sep 2000 16:25:20 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Thanks for your prompt answer. However I will try to summarize the views as I heard them > only to keep tab on where we stand at the point the discussion was interrupted and state > the (rather long) line of thought that makes us all feel that the design should include > multiple connections from the outset. We can then stop discussing it for a while. If such a message is posted it will set off more discussion ... this community seems reluctant to grant anyone the proverbial "last word" in that sort of fashion. Please consider whether posting such a message is likely to produce progress towards consensus. > I feel also that the community had no chance to see a draft including an asymmetric > multiple connection model and it might be a worthwhile exercise to present > one as an (optional text) with the next draft. Please put the (optional text) into a separate draft - as long as the Asymmetric vs. Symmetric issue is open, neither should be described in the main draft, lest people not familiar with the day to day happenings on this list draw the wrong conclusions. Preparing such a draft with all the details worked out would indeed be useful, as would corresponding drafts on the other possible models. > I would like also - only to be fair - to point out - > as others have done before - that by removing the command counters and the sliding > window you have, for all practical purposes, closed the door to the > symmetric multiple connection version. Subsequent to saying that the command numbers should be removed, I got a bunch of complaints that the numbers were also useful for flow control, even for single connection sessions and hence reopened the flow control discussion. If consensus is that flow control really needs the command numbers, putting them back in is fine. It would also be reasonable to advocate that the fields that contained the command numbers be Reserved in the single connection case. On the list, I see a set of people who seem to be primarily interested in single connection sessions, and hence for whom simplicity of the protocol in the single connection case is an important consideration. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31807; 28 Sep 2000 12:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA26412 for ips-outgoing; Thu, 28 Sep 2000 11:29:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA26383 for ; Thu, 28 Sep 2000 11:29:13 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 08:30:39 -0700 Message-ID: From: Robert Snively To: "'Matt Wakeley'" , IPS Reflector Subject: RE:Why FCP doesn't need RDMA? It has a better way. Date: Thu, 28 Sep 2000 08:28:38 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > I object to mandating iSCSI use an RDMA option because: > > - (main reason) there isn't any standardized mechanism now, and I > don't want to hold up iSCSI while one crawls through the > standards process. > > - I don't think RDMA is needed. FCP doesn't use it, and it > works great with > the > FC protocol chips that "accelerate" FCP. Actually, RDMA is not needed in FCP because all protocol chips implemented perform a real peer-to-peer DMA straight to the data areas specified by the user's interaction with the operating systems allocation algorithms. The combination of the FCP/SCSI pointer structure, task tagging, and the FC relative offset perform the function you would otherwise have to use RDMA to accomplish. Bob Snively Brocade Communications Phone 408 487 8135 1745 Technology Drive San Jose, CA 95110 Email rsnively@brocade.com From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32469; 28 Sep 2000 16:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA06463 for ips-outgoing; Thu, 28 Sep 2000 15:42:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA06452 for ; Thu, 28 Sep 2000 15:41:59 -0400 (EDT) Received: from yp_portable (slip-32-100-202-170.ma.us.prserv.net [32.100.202.170]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDMDTZ; Thu, 28 Sep 2000 12:39:07 -0700 From: "Y P Cheng" To: , Subject: RE: A Transport Protocol Without ACK Date: Thu, 28 Sep 2000 12:39:32 -0700 Message-ID: <000701c02983$d30a5c20$c6d46520@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Thursday, September 28, 2000 2:53 AM > > Dear Mr. Cheng, > I am baffled by your taxonomy. VI and I2O are basically interface (API) > definitions and each of them has a different purpose. VI > was ment to be a User-Space-to-User-Space communication interface and > I2O well...a panacea for drivers. > We are trying to build a protocol (to which an API is assumed but not > mandated). > Could you be a bit more specific in what you criticize or propose? I misused the term "transport protocol" for VI and I2O. I meant that VI and I2O could be the API to pass an iSCSI request to an iSCSI service provider, i.e. either an iSCSI driver above the TCP/IP layer or an adapter that performs proper TCP/IP transport. VI further defines connection-oriented or connectionless for delivery of different QoS. I2O has nothing to do with transport. My apology. What I tried to point out was that this iSCSI working group is trying to solve problems of using TCP/IP reads and writes to send PDUs that encounters deadlock and head-of-queue problems. If the WG can assume the use of VI- and I2O-like API to send an iSCSI request and allow an iSCSI service provider taking responsibility of delivery of PDUs in proper order, then, the deadlock and head-of-queue problems are non-issues. An iSCSI service provider WILL send proper TCP/IP packets as well as follow the RFC2581 for congestion avoidance. For people who choose to implement an iSCSI provider using TCP reads and writes, they do have an implementation problem. But, should the WG address the problems of every possible implementation? We certainly could suggest a method of using TCP/IP stack. But, the asymmetric model with multiple connections is not my top choice, because synchronizing among connections is non-trivial. I will specify an exchange table like that in a fibre channel adapter. We could use the initiator and target task tags to index into the exchange table. I am changing a fibre channel adapter to an iSCSI service provider that can handle thousands of iSCSI requests without any queuing or resource allocation problem. The adapter CAN correctly interpret the iSCSI PDUs and follow TCP/IP delivery rules including congestion control. It also confirms the SAM-2. All the discussions on asymmetric-multiple-connections and lack-of-resource problems do not apply to the implementation of this adapter. (I described in another posting on how the fibre channel adapters handle thousands of requests concurrently.) IMHO, the important discussions for iSCSI is to have a transport mechanism that allows the service provider to stream data to the other side of the world 10,000 miles away with 80 msec latency and 160 msec roundtrip time. At 1Gb speed, there are 8MB of data on route to a destination in 80 msec before the destination even has a chance to send its very first ACK back. The originator needs to stream 16MB of data before seeing the first ACK. We need a lot of buffers if we implements iSCSI in TCP stack. What if 10 Gb InfiniBand, Ethernet, and Fibre are introduced in the next 12 months? Obviously we can use flow control to ensure no overrun. But, isn't it more important to have a mechanism to move data at 10 Gb speed on an iSCSI connection? The WG assumes RFC2581 will handle indefinite transfer size with indefinite delay. I am not sure. On a busy network, the isochronous traffic like voice-over-IP using UDP has no ACKs for congestion control. Hence, the TCP/IP implementing RFC2581 will have unfair disadvantage on the Big I-nternet. To ensure iSCSI being a viable proposal beyond its application to MAN where delay is small there should be discussions on the fairness on the Big I-nternet. Virtual circuits with paid services could be a solution. I don't know SCTP enough to say it is the right solution. I do understand that the WG's charter is not to question the TCP congestion control. Therefore, I can not raise this issue. We have entered an era where a fibre channel adapter is capable of 30,000 IO's today and going to 50,000 and 100,000 shortly. I don't believe the current discussions appreciate the nature of having only 10 usec to process an IO request. Is it even possible to deliver 50,000 IOs using iSCSI without discussing of mechanism of streaming 160MB of data at 10 Gb speed to a destination 10,000 miles away? I do believe it is possible if we can keep the outgoing/incoming PDUs streaming with a sensible recovery of lost packets. But, to do this we have to have a good mechanism handling the ACK traffic. Introducing ACK-0 of fibre channel to deal with dropped ACK is critical, IMHO. However, TCP/IP is sacred and not be questioned at this time. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31004; 28 Sep 2000 7:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA17332 for ips-outgoing; Thu, 28 Sep 2000 06:16:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA17327 for ; Thu, 28 Sep 2000 06:16:32 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id MAA35406 for ; Thu, 28 Sep 2000 12:15:55 +0200 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA72244 for ; Thu, 28 Sep 2000 12:15:53 +0200 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256968.00384329 ; Thu, 28 Sep 2000 12:14:32 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: "Douglas Otis" cc: dfsmith@almaden.ibm.com, ips@ece.cmu.edu Message-ID: Date: Thu, 28 Sep 2000 13:11:29 +0300 Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, There seem to be practical and philosophical issues that you consistently ignore. Textual names are more stable than underlying network addresses and thus better for applications (I assume that your card starts with your name not phone number and that the former has changed less than the later). I the iSCSI realm - and that is the only one under scrutiny - a pertinent observation was made by David Black a century ago during our (too) long naming discussions - that a scheme that will allow the third party commands to make the name to address conversion at use time is better than one that requires this mapping to be made ahead of time. Unfortunately that is not yet possible with SCSI but we are informed that it soon be. And for authentication the textual address in login is yet another mean to prevent errors and/or unauthorized access if the target chooses to use it. Julo "Douglas Otis" on 19/09/2000 06:55:29 Please respond to "Douglas Otis" To: Daniel Smith/Almaden/IBM@IBMUS, ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Daniel, SCSI is not HTTP and does not provide a human interface as a basis for communication. Ownership of naming abstractions vary from the perspective of the client to that of the provider. The provider may not wish to share management of these naming abstractions with the client and vise-versa. Rather than attempting to keep a permanent naming abstraction as globally shared between both provider and client, these abstractions should be meaningful to the human dealing with the names. For the client it may be Tom, Dick, and Harry, and for the provider it could be Shelf A, E, and N. By keeping these abstractions independent of the transport layer, it allows each end to use such means as LDAP servers to join these views in a coherent fashion. Your IT manager will insist you not concern yourself with the IP of your workstation, as he has control of these assignments through DHCP and LDAP servers, the same should be true for a networked version of SCSI. Do not forget the large task of authentication that may also be managed by either the provider or the client. By keeping the transport as plain as possible, these naming abstractions can become as complex as required without affecting the transport specification. Implanting names into SCSI is a mistake and should be corrected as it limits this valuable freedom. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Daniel Smith > Sent: Monday, September 18, 2000 5:25 PM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Just in case people are wondering what this proposal is/was (the mailing > list seems to have exploded quite a bit since it was released > back in May), > here it is again, with a handful of [modifications] and [comments]. > -- > IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > > > > ---------------------------------------------- > > Subject: iSCSI naming > Date: Mon, 8 May 2000 15:01:17 -0700 (PDT) [mods 18 Sep 2000] > > iSCSI naming requirements (first [and a half] iteration) > ------------------------- > > It is important to have a consistent naming scheme as the basis of a > discovery system for the SCSI over TCP protocol. A basic set of > requirements is below, and a method is suggested. > > Requirements > ------------ > > A name identifies a [SCSI service delivery port], not a LUN. > notes: There may be many thousands of LUNs (and virtual LUNs) on a > connection. > why: The LUNs given to each connection are likely to be dynamic > and assigned > by the iSCSI manager. The iSCSI names are more static and long-lived, and > must be persistent even if the iSCSI manager fails. > notes: A separate naming scheme will document the LUN discovery and naming > mechanism. [query to LUN 0 to get list of LUNs] > > The namespace need not be fundamentally [systematically] enumerable. > notes: FC can be enumerated, the Internet cannot (practically). > why: iSCSI will be deployed on untrusted networks. Having > devices that can > be enumerated are subject to systematic attack. Also, many > devices will be > behind firewall machines, and not even contactable until a login procedure > has taken place---holes in such an enumerated namespace are not > acceptable. > notes: Devices on a LAN can be enumerated, and this is acceptable. > notes: Enumeration (listing) by a 3rd party is acceptable. > > The names should be as static as possible. > notes: Both in time and location. > why: A name given for one iSCSI device really ought to resolve to the same > device every time, from everywhere. [device here refers to a SCSI service > delivery port, of course the LUs should stay the same too, barring things > like security lock-outs] > > The name should not be tied to the network technology. > notes: For example, ethernet MAC address or IPv4/v6 address. > why: To cope with future networking systems, the name of the iSCSI device > should not rely only on aspects of the transport that are liable to become > obsolete, or that are not static enough. > notes: IPv4 addresses may be acceptable on a controlled private network, > however, these names should not be the canonical names as viewed from the > outside. [major problems with NAT if that happens] > > The naming scheme must be reliable. > notes: If parts of the network go down, the names must remain intact. > why: iSCSI may form part of the networking infrastructure, meaning that > after power loss, for example, the iSCSI devices will be required > to bring up > the rest of the network. This implies that the devices should be aware of > their own name, if necessary for network stability. It also implies that > any name service should have redundancy. > > The naming scheme should be nice to look at. > why: Long strings of unformatted numbers are horrible. [and prone > to typos] > > The iSCSI device being addressed should be maskable from observers. > why: The device name may contain valuable (secure) information, > so it must be > possible to hide this information from prying network interfaces. > notes: The URL scheme manages this, by separating the hostname from the > device name. > > Proposal > -------- > We should follow the URL naming scheme for iSCSI devices, thus: > scsi:/// > Where > :=|: is the TCP [destination] identifier... > is the DNS or DDNS host name of the iSCSI device and > is the TCP port number to connect to. > > And [ should be ] > :=|/|? is the device identifier > is a text identifier for the device > :=|? is an argument for the device > [argument escape character was '&' in the original] > > The element is reasonably well defined. However, I would prefer to > keep the element more free-form. The scsis:/// is > also a straightforward extension. [to secure (SSL) iSCSI] > > The identifier specifies an iSCSI TCP connection > identifier only. > This may provide access to one or many LUNs that flit in and out of LUN > space. > > One caveat is that some (most?) iSCSI devices will not have > access to DNS. > Fortunately, devices (targets) do not need to [query the network to find] > their names except in the case of trying to resolve another device > (target)---i.e., only initiators need DNS resolution. > > Here are some fun examples of the things I would want to do as a storage > manager. > > The "backup" program does a simple block-to-block copy from one > iSCSI device > to another... > backup scsi://raid.acme.com/lun12 scsi://tape.acme.com/id25 > Of course, a more useful operation may be... > flashcopy scsi://disk.acme.com/12/2 scsi://disk.acme.com/13/2 > where the pathnames are ASCII numbers. (They may even be the WWN or LUN > value.) > > This doesn't address the issue of security. When we have more of those > issues resolved, we might want to issue a command like this... > kcopy scsis://secure.acme.com/raid12/set2?key=12345?auth=dfsmith > ...so that the part, containing the security tokens, is > never passed > in the clear over the network. > > It may be possible for a manager to transparently interpret the arguments > or pathname into a LUN list in a storage farm. > iscsi attach scsi://farm.acme.com/tray17?LUN=0x17?LUN=0x18 > /dev/scsi2 > iscsi attach scsi://san.acme.com/5A/27/23/99/B7/A2/01/00 /dev/sda > > The Text/Response commands in the iSCSI Internet-Draft might be > accommodated > thusly: > iscsitext scsi://tape.scsi.acme.com/cart5 "com.ibm.retension:yes" \ > "com.ibm.eject:now" > > com.ibm.retension:ok retensioning > > com.ibm.eject:ok scheduled ETA 2:30 > > Secure enumeration might be done in the following way, with peculiar > examples of possible Un*x integration. > iscsi find_domain_manager > > iscsi.acme.com > httpsget https://iscsi.acme.com:99/root&auth=dfsmith&key=1234 > > > > > > scsi://unit1.raid.acme.com/ > > disk1 disk2 disk3 disk4 disk8 disk9 > > > > > > > > scsi://bigtape.acme.com/racf4 > > cart57 cart58 > > > > > > > > scsi://ibmdeskstar75.dfsmith.acme.com/lun1 > > > > > iscsi mount scsi://ibmdeskstar75.dfsmith.acme.com/lun1 /mnt/mydisk > > /mnt/mydisk: 73564224 blocks available > tar -cf 'scsi://bigtape.acme.com/racf4/cart57&user=dfsmith' ~/* & > > [3] 4291 > runsocks dd if=/dev/zero of=scsi://dfsmith.blocks4less.com/disk1 & > > [4] 4293 > And so on. (Note: I am not an XML expert.) > > [Additional notes: > > The decision to resolve devices in the host name or in the device name is > really up to the implementer. Some companies may already have systems in > place for handling the DNS while they have no existing SAN > infrastructure. > Thus the decision to name a particular service delivery port > scsi://disk26.farm-b.acme.com/ > or > scsi://farm-b.acme.com/disk26 > is up to the system designer. > > The iSCSI URL names a bundle of LUs (one iSCSI "connection"). People love > to point out that this is a problem if you have multiple entrance > ways into > the same SAN, and the same device (say WWN=0x1234) has different LUNs (say > LUN=5 on iscsi://scsi.acme.com/bridge1 and LUN=477 on > iscsi://scsi.acme.com/bridge2). So don't do that. Or do that > and just live > with it. Or make sure the different initiator/bridges into the SAN assign > the device the same LUN. (There isn't really a problem using a LUN in a > global context, as long as it's always attached to its service delivery > port.) (Having to query every LUN on a port to get the list of WWNs ought > to be addressed by T10, in the same way they covered the "query > LUN 0 to get > a list of LUNs" problem. IMO!) > > The security aspects bring up the spectre of LUs being masked out for > different iSCSI initiators on different parts of the net. This is fun and > nothing to worry about. > > These URL names are used in several places, most of which are not > related to > the iSCSI transport itself. (Likewise, html is not particularly > related to > HTTP, nor SMTP to metamail.) > > First by users (or users by dint of clobbering an html hyperlink) who want > to attach to iSCSI devices. The names are descriptive and are > malleable to > all sorts of marvelous manipulations by the sysadmins. > > Second, by directory servers, who can deliver a list of the URLs (and the > LUNs that are accessible therein) to the iSCSI clients. > > Third, by iSCSI itself for the 3rd party copy command. In this case, the > URL specifies the destination SAN and the target LUN is carried in the > additional data field. > > ] > > > > > ---------------------------------------------- > > Jim Hafner/Almaden/IBM wrote: > > > > I've seen Daniel's proposal before and I still haven't figured out the > > context in which such names are to be used. > > > > Given that, I'm personally a bit uncomfortable with any naming > convention > > that uses LUN values in a global context. These have no meaning as LUNs > > are host-specific values. Even worse, LUNs are addresses, not names! > > > > I have no problem with WWNs in a global context as that is what they are > > for! > > > > A subtle difficulty, is that a WWN does not give a host an > address (either > > for the target or the LU). Even in SPC-2, EXTENDED COPY's > Identification > > Descriptor Target Descriptor Format (sic?) where WWNs are used, says > > "instructs the copy manager to locate a target and logical unit that > > returns a device identification VPD page ..." but gives no hints on how > > that should be done (e.g., walk the bus and do INQUIRY to > everything...?). > > > > The "hostname" and DNS provide a canonical method for getting an address > > for a name, so that part is OK. The only defined way to get an address > > (LUN) for a LU WWN is by exhaustive INQUIRY to each LU at a > given target. > > > > On the other hand, I can envision how some of these things > might work, in a > > well-coordinated and well-managed environment: > > > > There is a managing application which has the dual responsibility of > > managing the targets for their host/LUN mapping > configurations and also > > serves as the "walk the bus" discovery resource for hosts. > > > > When the manager (human?) determines the distribution of > host/target LU > > resources (after inventorying both hosts and target LUs), the > > application will send to the targets whatever host/LUN > Mapping (access > > controls?) commands are necessary to configure them. Later, > when a host > > boots it queries the application to get its "walk-the-bus" > services. In > > this, it minimally gets from this application a list of targets that > > have LUs to which it should have access. There is no REQUIREMENT at > > this point for the application to give the host anything more as the > > normal SCSI LU discovery process per target kicks in. On the other > > hand, I can see some value in the host getting more information about > > the LUs it will see at each target. In that case, the host could get > > from the application a list of targets and LU identifiers > (these could > > be LUNs or WWN or both, as you suggested). The important > point here, is > > that these LUNs are valid only in the context of the > requesting HOST and > > are not global! > > > > Note that this process requires clear coordination between > the "target > > configurator" and "host discovery of targets". > > > > > > Given an appropriate context for a URL scheme, I'd make two > modifications: > > 1) drop the assumption that no LUN means LUN=0. If no LU > qualifier (e.g., > > no LUN or WWN (or other) is provided), then LUN=0. In all > other cases, the > > alternative names should be sufficient to identify a LU, > though, as above, > > finding a LUN address may require extra work. > > 2) add ?ProxyToken= as an additional option here (this > coordinates > > well with the changes to EXTENDED COPY approved in the context > of the SCSI > > access controls). > > > > Well, that's another two cents! I was hoping to save some of this > > discussion until the protocol gets worked out, but .... > > > > Jim Hafner > > > > > > csapuntz@cisco.com@ece.cmu.edu on 09-17-2000 11:11:24 PM > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > To: IP Storage > > cc: csapuntz@cisco.com > > Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > > > > > Here is a proposal to extend the iSCSI URL scheme to support LUNs and > > WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This > > extension is not required for the iSCSI transport protocol, but may be > > useful for discovery and management. > > > > A URL for the target has the following form: > > > > scsi://hostname/path/with/ > > > > A URL referring to a specific LU has the following form: > > > > scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber > > > > If no LUN= term appears in the URL, then LUN 0 is assumed. > > > > The WWN= term is optional. If present, the party should > > verify that the WWN in the LU's Device Identification Inquiry > > Page correspojnds to the WWN. > > > > Note, the following URL: > > > > scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa > > > > refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After > > connecting, the initiator should verify that the WWN > > of the LU is 0a050a4bcdefa. > > > > -Costa > > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28569; 27 Sep 2000 17:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA26003 for ips-outgoing; Wed, 27 Sep 2000 16:32:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub1.almaden.ibm.com (mailhub1.almaden.ibm.com [198.4.83.44]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA25979 for ; Wed, 27 Sep 2000 16:32:18 -0400 (EDT) Received: from catalina.almaden.ibm.com (catalina.almaden.ibm.com [9.1.24.58]) by mailhub1.almaden.ibm.com (8.8.8/8.8.8) with ESMTP id NAA66696 for ; Wed, 27 Sep 2000 13:32:57 -0700 Received: (from dfsmith@localhost) by catalina.almaden.ibm.com (AIX4.3/UCB 8.8.8/8.7) id NAA48020 for ips@ece.cmu.edu; Wed, 27 Sep 2000 13:31:47 -0700 From: Daniel Smith Message-Id: <200009272031.NAA48020@catalina.almaden.ibm.com> Subject: Re: New List: rdma@cisco.com: to discuss RDMA To: ips@ece.cmu.edu Date: Wed, 27 Sep 2000 13:31:46 -0700 (PDT) In-Reply-To: from "csapuntz@cisco.com" at Sep 27, 2000 10:16:29 AM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk csapuntz@cisco.com wrote: > > Does anybody on the list object to specifying an RDMA mechanism for > use with iSCSI? Does anybody on the list object to mandating an RDMA > mechanism? Please include your reasons. Mandate: strongly against. Recommend: neutral, slightly favorable. Unless... By `mandate' you mean supporting a TEXT flag com.cisco.RDMAEnableRFCxxxx: goforit to which the target can respond com.cisco.RDMAEnableRFCxxxx: nowaybuddy because the target does not have enough system privilage to alter any TCP options. Daniel Smith. -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29895; 28 Sep 2000 1:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA11000 for ips-outgoing; Thu, 28 Sep 2000 00:24:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e33.esmtp.ibm.com (e33.co.us.ibm.com [32.97.110.131]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA10996 for ; Thu, 28 Sep 2000 00:24:28 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e33.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id XAA08478 for ; Wed, 27 Sep 2000 23:18:17 -0500 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8S4NC397380 for ; Wed, 27 Sep 2000 22:23:12 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 27 Sep 2000 21:17:40 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/27/2000 10:23:11 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas, not every song you sing should have the same tune over and over and over. We have heard it very well. In fact I look forward to the point in time that we will have completed the TCP/IP version of iSCSI and turn our focus at whether SCTP is approprate and how it will benefit iSCSI. Now against my better judgement I will try to engage this one more time. A careful reading of my note implies nothing like you have read into it. We looked at the value of the functions that Costa had in his original proposal for OPTIONAL extensions to TCP/IP. When these were used they permitted, in my estimation and others, a positive effect on a number of TCP/IP applications. It just so happened that one of the applications was iSCSI. Now, the arguments that I put in my note stated why we chose to move the TCP/IP RDMA option to the background. I think this very thread captures the essence of the reason why it was put on a back burner. There were also strong feeling that TCP/IP done right could move us forward in a significant way even without the RDMA option. We did not want to sacrifice the Very Very Good for the Perfect. If Costa and group can in another reflector and a separate set of interactions with the IETF (perhaps with our support) get acceptance for the TCP/IP RDMA option then that would be good for many TCP/IP applications including iSCSI. And I, for one wish them good luck. Since this thread was intended to give Costa my thoughts, which I have, there is nothing to gain to continue this discussion thread further. . . . John L. Hufferd "Douglas Otis" on 09/27/2000 06:50:20 PM To: John Hufferd/San Jose/IBM@IBMUS, cc: Subject: RE: John, If the intent of the draft-satran-iscsi-01.txt is to eventually modify TCP in a significant way, these issues should be addressed ahead of concluding such features as RDMA with TCP will become available. As much of the optimization required in the use of TCP for SCSI hinge on such modifications, it would appear you have already concluded such modifications *will* take place. Most should agree frame alignment, out of sequence processing, independent streams sharing a single control, and zero-copy data handling are deciding issues. Separate TCP buffers to control resources, frame alignment, unique TCP APIs are between the lines in this proposal. Before havoc ensues, there is an alternative. Rather than bending TCP to fit, SCTP has already laid a foundation. As such, *no* transport modifications are required and yet customers still get friendly TCP behavior. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > John Hufferd/San Jose/IBM > Sent: Wednesday, September 27, 2000 3:08 PM > To: ips@ece.cmu.edu > Subject: > > > Costa, > If you remember way back when we first got together, you had a good RDMA > proposal for use with TCP/IP using the various optional fields. We all > seemed to love it at the time. I for one was disappointed when we dropped > it out of the iSCSI proposal (even as an optional consideration). The > thoughts were, that if we started to push this onto TCP/IP folks, we would > spend much time on that and not get the iSCSI stuff done. So it was > dropped. > > I for one would find it useful to bring it back, but only as an optional > feature, and only if that did not get in the way of the iSCSI Stuff, since > you can see how that has already begin to fill up the current ips > reflector. But I think working it as a side issue, on another reflector, > until it is fully baked seems OK to me. > > Summary: > RDMA -- Yes -- the one you originally had proposed > Optional --Yes > > . > . > . > John L. Hufferd > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32086; 28 Sep 2000 14:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA29348 for ips-outgoing; Thu, 28 Sep 2000 12:50:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA29329 for ; Thu, 28 Sep 2000 12:50:24 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA14574; Thu, 28 Sep 2000 09:37:11 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA04102; Thu, 28 Sep 2000 09:34:57 -0700 From: "Douglas Otis" To: Cc: "Charles Monia" , , "Jim Williams" , Subject: RE: New List: rdma@cisco.com: to discuss RDMA Date: Thu, 28 Sep 2000 09:52:45 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <39D336A8.D007675E@stewart.chicago.il.us> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Randall, With protocols like SCSI, the data payload is contained within an additional structure. Although one could attempt to constrain data to a stream and make exchanges dynamically allocated to streams, it would be easier to extract payload from the encapsulation. From this extraction, either generically or specifically, transport the payload as described. Yes, such optimization would be implementation specific but very possible with SCTP. Doug > -----Original Message----- > From: randall@stewart.chicago.il.us > [mailto:randall@stewart.chicago.il.us] > Sent: Thursday, September 28, 2000 5:17 AM > To: Douglas Otis > Cc: Charles Monia; csapuntz@cisco.com; Jim Williams; ips@ece.cmu.edu > Subject: Re: New List: rdma@cisco.com: to discuss RDMA > > > Douglas: > > I may be missing something here, but sendmsg() and recvmsg() both > have scatter/gather arrays within the berkley sockets api. > I am not sure that one can uses sendmsg() recvmsg() with TCP.. I > have never tried. As far as SCTP goes, we have defined a sockets > mapping... the draft is released but real rough.. an next version > is in the works.... the current "rough" cut can be found at: > > http://search.ietf.org/internet-drafts/draft-stewart-sctpsocket-si > gtran-00.txt > > Now it may be that I am missing a key point in this discussion.. were > you > thinking other than the iovec structure... or were you thinking a > modification > that allows the pages to be stolen with no copy... the no copy becomes > a implemenation issue.. I have done something like it in the past > where if the application gives a 4k page in the send, the O/S underneath > does a swap of a page from kernel space with the user space one... > > But as I say, this is real implemenation dependant :) > > R > > Douglas Otis wrote: > > > > Charles, > > > > With respect to SCTP, features added by a TCP option for RDMA > are not needed > > to support alignment and out of sequence processing that > ultimately alters > > the TCP API. SCTP adds these features without disruption or > modification to > > TCP. The intent of VI is to allow scatter/gather function to > handled by the > > target. A safer scheme would be to adhere to SCSI conventions > and implement > > zero copy and out of sequence processing using SCTP and SAM structures > > related to locally pre-arranged transfer structures. This > would keep the > > initiator or client in intimate control of memory and not > reliant on targets > > eliminating boundary checking. Such a feature will slightly > impact SCTP to > > add a means to generally encapsulate a data payload associated with > > structure of pointers equipped with scatter-gather lists and relative > > offsets. Specifically, this could be defined as the FCP structure data > > structure in SCSI implementations. I would whole-heartedly endorse such > > features within SCTP. I would dissuade such features from > being added to > > TCP. One could view the scatter-gather list structure as a > token to be used > > by the target in more conventional VI methods. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Charles Monia > > > Sent: Wednesday, September 27, 2000 11:49 AM > > > To: csapuntz@cisco.com; Jim Williams > > > Cc: ips@ece.cmu.edu > > > Subject: RE: New List: rdma@cisco.com: to discuss RDMA > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > > > > Sent: Wednesday, September 27, 2000 10:16 AM > > > > To: Jim Williams > > > > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > > > > Subject: Re: New List: rdma@cisco.com: to discuss RDMA > > > > > > > > > > > > > > > > Does anybody on the list object to specifying an RDMA mechanism for > > > > use with iSCSI? Does anybody on the list object to mandating an RDMA > > > > mechanism? Please include your reasons. > > > > > > > > > > I'd object to mandationg the use of RDMA in iSCSI. However, I > > > would support > > > structuring the spec so that an RDMA transport mechanism could be used > > > underneath (I guess that's motherhood). If the iSCSI folks > decided this > > > wasn't a priority issue, that's ok with me too. > > > > > > Charles > > > > > -- > Randall R. Stewart > randall@stewart.chicago.il.us or rrs@cisco.com > 815-342-5222 (cell) 815-477-2127 (work) > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24216; 26 Sep 2000 16:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA06082 for ips-outgoing; Tue, 26 Sep 2000 15:36:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA06042 for ; Tue, 26 Sep 2000 15:35:57 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id PAA19153; Tue, 26 Sep 2000 15:35:52 -0400 Message-ID: <01ea01c027ee$4b0f0870$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: "Douglas Otis" , Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Tue, 26 Sep 2000 15:16:33 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk The starting assumptions for the proposal were that the transport must be TCP, and TCP could not be modified in any way. If you don't buy the starting assumptions, it may be a waste of time to discuss the specifics of the proposal. NO changes or new options to TCP are proposed. Frame alignment is not a requirement, only a recommendation, so no existing TCP APIs and kernel implementations need to change. (Perhaps I sould soften "recommendation" and make it "possible feature to be considered by implementators when convenient".) The CRC is certainly open for discussion. It looks to me like I can do a software CRC in 3 risc instructions per byte of data using a moderately sized table lookup (less that 32KB). Is a CRC a good thing or a bad thing? Should it be optional or required? -----Original Message----- From: Douglas Otis To: Jim Williams ; ips@ece.cmu.edu Date: Tuesday, September 26, 2000 2:52 PM Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) >Jim, > >In summary, you expect kernel modifications of TCP API to allow frame >alignment, a hardware based 32-bit CRC that uses a different polynomial than >FCP and a direct copy method at the NIC. As SCSI does not benefit from VI >with zero copy already possible with frame alignment, the area where there >could be commonality has been missed with the choice of CRC. Rather than >modifying thousands of TCP implementations, it would seem appropriate to >leave TCP as is and concentrate on providing these improved features using >SCTP designed to handle these requirements together with additional features >yet to be resolved with TCP. You need not convince systems to change TCP to >benefit from rather important features found with SCTP. Do not encumber TCP >with disruptive options. SCTP can emerge using UDP and become supported >without making these changes to TCP that you advocate as evolutionary. Once >you implement frame alignment, the API for TCP becomes unusable as it does >not afford means of discerning frames. SCTP implementations already address >these problems and so I do not see your approach as evolutionary but rather >revolutionary. You do not get my vote on this approach. > >Doug From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20492; 25 Sep 2000 18:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA17453 for ips-outgoing; Mon, 25 Sep 2000 17:50:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA17444 for ; Mon, 25 Sep 2000 17:50:48 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA13486; Mon, 25 Sep 2000 14:38:08 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA26435; Mon, 25 Sep 2000 14:35:53 -0700 From: "Douglas Otis" To: "Jim Williams" , Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Mon, 25 Sep 2000 14:53:11 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <00a601c026fe$bac427e0$cb08efd0@sachmo.giganet.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, > RDMA / TCP > 1. Abstract > > This document describes a format for encapsulating RDMA (remote direct > memory access) information within a TCP data stream. No changes or > modification to TCP of any sort are required. You suggest a frame aligned formatting scheme is possible using an unmodified TCP. > This is not intended to > be a protocol, but rather a common format that may be shared by > multiple client protocols, for instance VI/TCP and iSCSI. By using a > common format it is hoped that design of NICs supporting these multiple > protocols can be simplified. > > Sufficient information is included in the RDMA message format to allow > determination of the protocol message units, as will as the ability to > process an incoming RDMA request even if previous packets are missing > and awaiting retransmission. In addition a CRC-32 is included in each > segment to enhance the checksum coverage included in TCP. By allowing out of sequence delivery of TCP, are you opening the window for spoofing? One need not guess the sequence, just get in front of it a bit. After that, you will never recover state. Do you expect the RDMA option to be universally supported, and if not? Do you see this level of CRC done in software? > 2. Overview > > Data transfers consist of a sequence of messages. Each message is of > one of four types: Send, RDMA_write, RDMA_Read_Request, and > RDMA_Read_Response. The maximum size of a message is approximately > 2^32. Each message is divided into one or more segments. It is > RECOMMENDED that each TCP segment contain exactly one RDMA segment. How do you achieve this recommendation? Sending 4GB exchanges, you still have queue blocking if not head of queue blocking. > The receive end of the connection cannot assume any alignment between > the RDMA segments and TCP segments, however a receiver SHOULD optimize > performance for the case where each TCP segment contains exactly one > RDMA segment. Can one suggest frame alignment within TCP? Have you investigated the use of SCTP? SCTP does solve all the related problems without violating protocol or protocol API. Doug From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20310; 25 Sep 2000 17:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA14290 for ips-outgoing; Mon, 25 Sep 2000 16:38:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA12966 for ; Mon, 25 Sep 2000 16:08:07 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 13:09:34 -0700 Message-ID: From: Robert Snively To: "'Pierre Labat'" , Robert Snively Cc: ips@ece.cmu.edu Subject: RE: ISCSI: flow control Date: Mon, 25 Sep 2000 13:07:34 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > > Actually, the ugly problem is not made much less ugly by having > > a credit system. The problem is that the command receipt resources > > of a "target" are oversubscribed during periods of high activity on > > a properly configured system simply because of the statistical > > characteristics of storage access. The command resources are also > > shared among multiple sessions. The only way to guarantee credit > > is to share the credit out among the multiple sessions. This > > necessarily leaves a lot of unused resources lying around for > > sessions that are not very busy at a given instant and > unnecessarily > > limits the capabilities of those sessions that are busy and would > > otherwise be able to make constructive use of those resources. > > The result is higher latencies and lower throughputs than you would > > hope. What you really need is dynamic credit allocation, > which will > > always leave the possibility that a particular session may find > > itself oversubscribed and forcing busy indications. Which > is effectively > > the mechanism SCSI uses today. > > Robert, > > As you say, the target can inform the initiator whenever it > wants, based on > its > own policy of resources management, that it can't handle no more > command. It has the advantage to be dynamic and to take into account > all the initiators. > However, this mechanism has the following weaknesses: > 1) to inform the initiator to stop sending command, the target > drops commands (and data). It implies retransmission > It's a waste of bandwidth. This is one of many reasons I do not believe that sending immediate data is a constructive exercise. Having to resend a few bytes of command information on those rare occasions that your system is so underconfigured that you run out of command queuing resources doesn't seem to me to be a major problem. However, running out of data buffering (as opposed to command queuing) resources could be a frequent occurrence and should be avoided by using the appropriate Ready To Receive indication from the device. > > 2) the initiator after detecting [several] TASK SET FULL/BUSY status, > must handle blindly the return to the full depth of the > command window. > It does that in a slow way. At that time you under use the > capabilities > of the LU. My point is that a properly implemented system does not do this. Instead, it uses an optimistic start up algorithm. By reserving one slot for each likely initiator, the SCSI device is always able to accept at least one command from that initiator. Dynamically allocating the remainder of the command queuing resources allows any combination of additional commands from any initiator to be queued up until the queue is full (except for the reserved slots for initiators with no activity). If one of those initiators then starts up, it starts by optimistically sending all its commands. Most of the time, there will be plenty of space and the target will cheerfully absorb the commands. Sometimes, the queue will be all the way full and only one command will be absorbed. If that is the case, the remaining commands are returned with a queue full indication. Eventually, the one queue command from that initiator will be completed and sent back to the initiator. At that time, it makes the assumption that lots of other commands have also been cleared out by other initiators, and sends to the target all the commands it wants to send. By this time, there will again usually be plenty of space. Some implementations may have a sense of how much queue space is typically available and will limit their first optimistic delivery of requests to some number, but if demand is high for a particular initiator, that initiator will push the numbers beyond that limit very quickly. If you have a situation where an initiator is frequently encountering no command queuing resources at a target, you have an improperly configured system and must either: a) sacrifice performance (not total throughput, but individual observed latency); b) reduce the number of initiators and/or the load provided by each; or c) increase the capability of the target. Note that this has nothing to do with congestion management anywhere, since all these packets are very small. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26230; 27 Sep 2000 4:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA28220 for ips-outgoing; Wed, 27 Sep 2000 03:14:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA28216 for ; Wed, 27 Sep 2000 03:14:30 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id JAA196234 for ; Wed, 27 Sep 2000 09:13:58 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA66120 for ; Wed, 27 Sep 2000 09:13:58 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256967.0027BA17 ; Wed, 27 Sep 2000 09:13:55 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 27 Sep 2000 10:05:17 +0300 Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, This goes to the hearth of problem. To avoid this effect we propose (like FCP) unordered messages to go always through - allow the initiator to decide which task management messages he deems "urgent". But if you have a single connection and the TCP window is closed you are left with only one think to do - drop the connection (and that is exactly what FCP is doing for similar reasons!). Julo Charles Monia on 16/09/2000 01:19:37 Please respond to Charles Monia To: Stephen Byan , IPS Reflector cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > -----Original Message----- > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > Sent: Wednesday, September 13, 2000 6:22 AM > To: IPS Reflector > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote: > > > I don't think VI/TCP helps at all. The VI is implemented on > > top of a TCP > > stream. If the TCP stream delivers the iSCSI command to VI, > > and VI has no > > place to put it, what is VI going to do? It has to block the > > TCP stream, and > > that in turn will block and "RDMA" from occurring. > > If I understand correctly, VI/TCP has a credit-based flow > control mechanism > on its message queues (which would be used to implement the > iSCSI command > reception queue), and so the initiator would never send an > iSCSI command > without a target buffer in which to store it. So RDMA cannot block on > commands in the TCP stream. > In the context of generic ULP "flow control" issues, the following observations are offered. Assuming one VI/TCP message queue handles all SCSI control traffic for a target, using flow control on messages to prevent the "queue-full" condition has undesirable side effects. Specifically, back pressure due to lack of command context resources within a logical unit would stop the flow of commands and task management requests to all other LUNS (a form of "head of line" blocking). There's another issue that is perhaps less obvious. Even if multiple message queues are used, say one per LUN, stopping the flow of messages due to a queue full condition would also stop the flow of task management messages as well. Charles Monia Senior Technology Consultant Nishan Systems Corporation email: cmonia@nishansystems.com voice: (408) 519-3986 fax: (408) 435-8385 From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27630; 27 Sep 2000 12:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA10612 for ips-outgoing; Wed, 27 Sep 2000 11:04:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA10606 for ; Wed, 27 Sep 2000 11:04:35 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id RAA247768 for ; Wed, 27 Sep 2000 17:03:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA29976 for ; Wed, 27 Sep 2000 17:03:46 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256967.0052BA06 ; Wed, 27 Sep 2000 17:03:35 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 27 Sep 2000 18:01:18 +0300 Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I think several of us have stated that we would love and RDMA that is good for iSCSI, NFS, etc.. I doubt that a full-VI is the right answer but an RDMA mechanism satisfying both would be fine. But this will take time - and although related - should not stop us from moving ahead. Julo Stephen Byan on 18/09/2000 17:07:04 Please respond to Stephen Byan To: IPS Reflector cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Douglas Otis [mailto:dotis@sanlight.net] wrote: > Charles, > > Depending on the capabilities of the Net-SCSI adapter, such > direct copies > would be possible without resorting to VI. Not using VI but > allowing direct > placement also provides lower overhead and a safer system as > pointers going > astray due to a muddled target or bit rot is not a concern. > SAM requires > buffers to be in place prior to issuing the request. These > buffers are at > the client which is where VI becomes useful. Unless both ends of this > communication becomes VI, you have not provided a meaningful solution. It may be better/more efficient/easier/whatever to implement iSCSI directly over TCP rather than over VI over TCP. My point was that in return for the pain, we can leverage the utility of the RDMA hardware assists, since the assists would be encapsulated in the VI layer, rather than intimately part of the iSCSI protocol. Consequently the RDMA hardware could be reused to accelerate other new protocols, as well as cluster message passing. I think an enlarged market for the hardware is important in order to ensure the success of iSCSI. iSCSI depends on getting RDMA hardware acceleration NICs into the servers. Consider two product scenarios: 1) a NIC with RDMA hardware acceleration for iSCSI. 2) a NIC with RDMA hardware acceleration for iSCSI, DAFS, MPI, Oracle Parallel Server, DB2 extended edition, and which is open to future applications. Which product do you think would be the easiest to sell to an end-user customer? > Adding VI modifies TCP. I don't believe draft-dicecco-vitcp-00.txt modifies TCP in any way. However, I do think the proposed TCP Message Boundary Option would be useful in order to optimize the implementation of RDMA in either iSCSI or in VI/TCP. Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32533; 28 Sep 2000 16:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA08601 for ips-outgoing; Thu, 28 Sep 2000 16:28:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA08588 for ; Thu, 28 Sep 2000 16:28:01 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8SKS0U17696 for ; Thu, 28 Sep 2000 15:28:00 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA05035; Thu, 28 Sep 00 15:26:20 -0500 Message-Id: <10009282026.AA05035@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Status summary on multiple connections In-Reply-To: Message from Robert Snively of "Wed, 27 Sep 2000 08:46:26 PDT." Date: Thu, 28 Sep 2000 15:26:31 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > A third alternative does exist and may be preferable. That is > to reflect the present broad usage of SCSI that uses wedge drivers > to achieve parallelization, bandwidth aggregation, and high availability. > By that definition, the proper number of connections per I_T nexus > is one and the proper number of connections per session is also one. I agree with Bob on this. Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24967; 26 Sep 2000 21:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA16229 for ips-outgoing; Tue, 26 Sep 2000 20:05:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA16224 for ; Tue, 26 Sep 2000 20:05:04 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA13961 for ; Tue, 26 Sep 2000 16:51:56 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA30441 for ; Tue, 26 Sep 2000 16:49:41 -0700 From: "Douglas Otis" To: Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Tue, 26 Sep 2000 17:07:12 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <01ea01c027ee$4b0f0870$cb08efd0@sachmo.giganet.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, Protection from spoofing as proposed is obtained with frame alignment, if out of sequence processing is allowed. To allow reception of frames without headers, as possible without alignment, non-IDed sequences are not excluded and must be held and verified by other means adding significant complexity. Out of sequence processing would be practical should sending and receiving agents both support an alignment. As your proposal would not merit benefits without alignment for both zero copy and out of sequence processing, it is specious to suggest such alignment implementation is only suggested. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Jim Williams > Sent: Tuesday, September 26, 2000 12:17 PM > To: Douglas Otis; ips@ece.cmu.edu > Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in > iSCSI)) > > > The starting assumptions for the proposal were that the transport > must be TCP, and TCP could not be modified in any way. If you > don't buy the starting assumptions, it may be a waste > of time to discuss the specifics of the proposal. > > NO changes or new options to TCP are proposed. > Frame alignment is not a requirement, > only a recommendation, so no existing TCP APIs and kernel > implementations need to change. (Perhaps I sould soften > "recommendation" and make it "possible feature to be > considered by implementators when convenient".) > > The CRC is certainly open for discussion. It looks > to me like I can do a software CRC in 3 risc instructions > per byte of data using a moderately sized table lookup > (less that 32KB). Is a CRC a good thing or a bad thing? > Should it be optional or required? > > -----Original Message----- > From: Douglas Otis > To: Jim Williams ; ips@ece.cmu.edu > Date: Tuesday, September 26, 2000 2:52 PM > Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) > > > >Jim, > > > >In summary, you expect kernel modifications of TCP API to allow frame > >alignment, a hardware based 32-bit CRC that uses a different polynomial > than > >FCP and a direct copy method at the NIC. As SCSI does not > benefit from VI > >with zero copy already possible with frame alignment, the area > where there > >could be commonality has been missed with the choice of CRC. Rather than > >modifying thousands of TCP implementations, it would seem appropriate to > >leave TCP as is and concentrate on providing these improved > features using > >SCTP designed to handle these requirements together with additional > features > >yet to be resolved with TCP. You need not convince systems to change TCP > to > >benefit from rather important features found with SCTP. Do not encumber > TCP > >with disruptive options. SCTP can emerge using UDP and become supported > >without making these changes to TCP that you advocate as evolutionary. > Once > >you implement frame alignment, the API for TCP becomes unusable > as it does > >not afford means of discerning frames. SCTP implementations already > address > >these problems and so I do not see your approach as evolutionary > but rather > >revolutionary. You do not get my vote on this approach. > > > >Doug > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31950; 28 Sep 2000 13:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA27693 for ips-outgoing; Thu, 28 Sep 2000 12:07:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA27689 for ; Thu, 28 Sep 2000 12:07:45 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id CB2CE131 for ; Thu, 28 Sep 2000 09:07:43 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id JAA04666 for ; Thu, 28 Sep 2000 09:08:19 -0700 (PDT) Message-ID: <39D36CC3.5796C52A@agilent.com> Date: Thu, 28 Sep 2000 09:07:31 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt Content-Type: multipart/mixed; boundary="------------F6426221160C9DE3382727CA" Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. --------------F6426221160C9DE3382727CA Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit --------------F6426221160C9DE3382727CA Content-Type: message/rfc822 Content-Transfer-Encoding: 7bit Content-Disposition: inline Return-Path: Received: from atlrel1.hp.com ([15.10.176.10]) by cassin.rose.hp.com (Netscape Messaging Server 3.6) with ESMTP id AAA73BC for ; Thu, 28 Sep 2000 07:26:00 -0700 Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by atlrel1.hp.com (Postfix) with ESMTP id 3F1047D1 for ; Thu, 28 Sep 2000 10:25:59 -0400 (EDT) Received: from matt.rose.agilent.com (matt.rose.agilent.com [156.140.233.79]) by andom1.an.hp.com (Postfix) with ESMTP id 9C4E5179 for ; Thu, 28 Sep 2000 10:25:35 -0400 (EDT) Received: (from matt@localhost) by matt.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) id HAA10424 for matt@cassin.rose.hp.com; Thu, 28 Sep 2000 07:25:35 -0700 (PDT) Received: from rtl.rose.agilent.com (root@rtl.rose.agilent.com [156.140.232.231]) by matt.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id HAA10421 for ; Thu, 28 Sep 2000 07:25:33 -0700 (PDT) Received: from andom1.an.hp.com (postfix@andom1.an.hp.com [15.4.128.104]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id HAA28397 for ; Thu, 28 Sep 2000 07:25:30 -0700 (PDT) Received: from msgbas1tx.cos.agilent.com (msgbas1t.cos.agilent.com [130.29.152.59]) by andom1.an.hp.com (Postfix) with ESMTP id EECEE144 for ; Thu, 28 Sep 2000 10:25:28 -0400 (EDT) Received: from stamp.agilent.com (unknown [192.6.9.4]) by msgbas1tx.cos.agilent.com (Postfix) with ESMTP id 02E95100; Thu, 28 Sep 2000 08:25:28 -0600 (MDT) Received: from loki.ietf.org (loki.ietf.org [132.151.1.177]) by stamp.agilent.com (Postfix) with ESMTP id 68C861816; Thu, 28 Sep 2000 08:25:27 -0600 (MDT) Received: (from adm@localhost) by loki.ietf.org (8.9.1b+Sun/8.9.1) id IAA02150 for ietf-123-outbound.01@ietf.org; Thu, 28 Sep 2000 08:55:00 -0400 (EDT) Received: from ietf.org (odin.ietf.org [10.27.2.28]) by loki.ietf.org (8.9.1b+Sun/8.9.1) with ESMTP id GAA01351 for ; Thu, 28 Sep 2000 06:46:23 -0400 (EDT) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA16390 for ; Thu, 28 Sep 2000 06:46:24 -0400 (EDT) Message-Id: <200009281046.GAA16390@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; From: Internet-Drafts@ietf.org Reply-To: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-wakeley-iscsi-msgbndry-00.txt Date: Thu, 28 Sep 2000 06:46:24 -0400 Sender: nsyracus@cnri.reston.va.us --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : iSCSI Message Boundary Detection Proposal Author(s) : M. Wakeley Filename : draft-wakeley-iscsi-msgbndry-00.txt Pages : 4 Date : 27-Sep-00 The iSCSI working group is currently considering the framing/delimiting of iSCSI messages in a TCP stream. This I-D presents a proposal to discover iSCSI message boundaries using the TCP urgent pointer. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-wakeley-iscsi-msgbndry-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-wakeley-iscsi-msgbndry-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-wakeley-iscsi-msgbndry-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20000927140011.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-wakeley-iscsi-msgbndry-00.txt --OtherAccess Content-Type: Message/External-body; name="draft-wakeley-iscsi-msgbndry-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20000927140011.I-D@ietf.org> --OtherAccess-- --NextPart-- --------------F6426221160C9DE3382727CA-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27818; 27 Sep 2000 13:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA15387 for ips-outgoing; Wed, 27 Sep 2000 12:52:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA15382 for ; Wed, 27 Sep 2000 12:52:14 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA14174; Wed, 27 Sep 2000 09:39:16 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA00450; Wed, 27 Sep 2000 09:37:02 -0700 From: "Douglas Otis" To: "Stephen Bailey" , Subject: RE: iSCSI: remove recovery from transport-layer connection failure(?) Date: Wed, 27 Sep 2000 09:54:40 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <10009270415.AA03051@candide.cs.uchicago.edu> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, Any IP-SCSI spec could be seen both as a connection to a controller and as bridge to existing drives. With 5+ milli-second network latency, controllers will remain adjacent to the client as a means of protection against this latency in much the same manner a controller protects from drive latency. As both modes of operation are legitimate, assumptions about transport should be tempered by these possibilities. SCTP does provide for a more immediate recovery. It would also be irresponsible to promote modification to TCP to support features already found within SCTP. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Bailey > Sent: Tuesday, September 26, 2000 9:16 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: remove recovery from transport-layer connection > failure(?) > > > > Currently, iSCSI is spec'ed to recover from transport-layer > > connection failures. > > > > The main motivation for this decision was to support tape backup > > applications that are quite sensitive to any failures that get > > propogated to their layer. > > > > So, perhaps we can remove the requirement of recovering from > > transport-layer connection failures in iSCSI. This would simplify > > the protocol somewhat. > > > > Thoughts? > > I'm all for eliminating command recovery. > > There seem to be several reasons advanced for command recovery. > > The first seems to be based upon an inappropriate analogy to FCP. > Command recovery had to be added to FCP-2 because the FC layer is > unreliable. A single dropped FC frame leads to a failed FCP command. > This clearly upsets tape operation even when the link is performing > nominally. In FCP, without command recovery, with some observable > frequency, you will get an expected error that leads to complete, > irrecoverable failure of a transfer stream. The other thing that > makes FCP-2 command recovery work well is when you are doing a write, > which is 90% (maybe it's 99%?) of tape operation, the target can > return an early indication of most frame drops, rather than waiting > for a timer to expire. > > TCP's reliability solves this problem in another way. By the time you > get a TCP connection failure, you have already exhausted a set of > reliability mechanisms which guarantee, with high certainty, that > further data can not be transferred between the two endpoints. > > `the two endpoints' phrase suggests the other reason advanced for > command recovery. That is, to permit path failover for commands which > are not idempotent, such as tape write sequential. The > problem with this, is that it is not clear HOW iSCSI command recovery > can actually work properly, given a TCP connection failure indication. > It takes a long time for a TCP connection to fail, and by that time, > I'm not sure recovery would reasonably be possible. Perhaps I'm in > error on this assumption. Can a tape guru (Joe from Exabyte?) comment > on whether recovery would be possible after many seconds (tens, > hundreds) have elapsed? > > The SCSI layer has never been solely responsible for ensuring reliable > backup. Macro scale things go wrong with tape (run off the end, get > eaten, etc..) with relatively high frequency. A low level backup > engine like tar or dump will fail on a SCSI error, and that's OK. > There must also be a higher level software component like Amanda, > which manages retries, including operator intervention, to ensure > reliable backup. > > It seems like whether iSCSI has a command recovery mechanism should be > a function of whether somebody can stand up and say for sure that it > solves a real problem. So far it only seems like it MIGHT solve a > problem. Who can say `this solves MY problem!'? > > Steph > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23463; 26 Sep 2000 12:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA19964 for ips-outgoing; Tue, 26 Sep 2000 10:51:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA19960 for ; Tue, 26 Sep 2000 10:51:31 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by atlrel1.hp.com (Postfix) with ESMTP id 0CF84413; Tue, 26 Sep 2000 10:51:30 -0400 (EDT) Received: from xboibrg1.boi.hp.com (xboibrg1.boi.hp.com [15.56.8.167]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id IAA21745; Tue, 26 Sep 2000 08:51:27 -0600 (MDT) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2650.21) id ; Tue, 26 Sep 2000 08:51:26 -0600 Message-ID: <499DC368E25AD411B3F100902740AD652E980B@xrose03.rose.hp.com> From: "HAAGENS,RANDY (HP-Roseville,ex1)" To: "David Black (E-mail)" Cc: "IPS (E-mail)" Subject: RE: iSCSI: Session Partial Resolution Date: Tue, 26 Sep 2000 08:51:24 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C027C9.3D89BA80" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C027C9.3D89BA80 Content-Type: text/plain; charset="iso-8859-1" David, I. Re: proposed points of consensus Add my name to the list of those objecting to (1) below. OTOH, (2) The use of more than one TCP connection per session *is* optional in the current draft, or at least in the last version of the draft that I read, and the one that we're all implementing. (1) Command sequencing was added to the draft for several reasons, but the important point here is that command flow control is necessary when using a single TCP connection in order to prevent a command queue full condition from preventing the reception of write data. We could just leave command flow control to the good behavior of the SCSI layer (there is no SCSI mechanism defined for it, and so it is implementation-dependent), but we decided that it was legitimate to treat it at the iSCSI "transport" layer, since from SCSI's pov, it is a transport problem. Command sequencing also solved two other problems (ordered command striping and recovery from TCP connection failure), so it seemed like an efficient mechanism overall. Some have objected that command windowing may result in the command queue's being blocked, and urgent commands' not getting through. If I recall the draft correctly, we made provision for this with the notion of unnumbered commands. These can be sent at any time by the initiator; but since they do not have a reserved place in the command queue, their reception must be considered unreliable: the target can drop them at its discretion. I believe this situation is familiar to the T10 community, and acceptable from a SAM-2 perspective. But if not, we could create a separate, sequenced, command queue for urgent commands. The question of the storage controller's overcommitting command queue credits has been raised. Our initial thought was that this would not be allowed. But if it must be allowed, then we could extend the command sequencing scheme to allow to target to drop commands and later demand their retransmission. The command sequence numbers would ensure that, ultimately, all commands are delivered in order, with no commands lost or duplicated. In general, if one is going to maintain queues of things of different types, then intermixing these things on a single TCP connection is going to require higher-level flow control, in order to prevent congestion at the TCP connection level. The alternative is to separate these queues into separate TCP connections: one connection for normal commands, one for urgent commands, one or more for write data, etc. (2) Use of more than one TCP connection per iSCSI session currently is optional. I don't think that there's any requirement for a target to support or for an initiator to use more than one TCP connection per session. But note that a single connection is necessarily symmetrical. Starting from the desire to support simple iSCSI sessions consisting of only one connection, one is led to symmetry and then to a solution to the command/data multiplexing problem (1). In summary: (a) A single TCP connection per session is considered desirable by many. This single connection is necessarily "symmetrical". And it requires a solution to the command/data head-of-line blocking problem. Command sequencing provides an elegant solution to this problem, and several others as well. Given a solution to these problems, it's possible additionally to implement a symmetrical connection model where multiple TCP connections are aggregated into one session. But this aggregation capability is not required. (b) Alternatively, if we use TCP connections liberally and asymmetrically, then we can dispense with higher-level flow control mechanisms, relying instead on TCP flow control for each distinct message type. But in this case, we cannot build a session consisting of a single connection. It must include at least one connection of each type. II. Re: "session model for multiple connection sessions" Here the real issue is not symmetrical/asymmetrical. Rather, it's whether to support connection/bandwidth aggregation at all. Possibly we should revisit the requirement. But I doubt your plan of separating it from the main body of the iSCSI work, as I'm fearful of not being able to re-integrate it later. To my mind, aggregation is an obvious requirement: at every stage of technology, its desirable to use N of the fastest things you have at once, to go even faster. SCSI defines a lot of task queue management features that are dependent on commands' being delivered in order by the SCSI transport. So any single-transport aggregation technique must support ordered command delivery. Today, we all build hosts and storage controllers with multiple Fibre Channel interfaces. And until recently, the SCSI SAM-2 document explicitly recognized multiport storage controllers. One limitation of today's devices, however, is that they must treat their multiple ports as independent. They constitute separate SCSI "Service Deliver Ports" (transport end points), and it's not possible to uphold SCSI ordering constraints across SCSI commands delivered on these separate ports. This is fine for load balancing of disk IOs, which frequently are unordered anyway. But it doesn't work for two important applications: writes to sequential devices (tape) and asynchronous (but ordered) disk mirroring. These are two important MAN/WAN applications. We could exclude them from the iSCSI requirements if we chose; but I'd like to see some debate on that point at the requirements level. iSCSI command numbering permits command ordering to be maintained across the several TCP connections in a session. This allows bandwidth aggregation through the technique of "ordered command striping". Command striping has one other desirable property for aggregating even commands that have no ordering requirement: Striping round-robin across N connections means that the aggregate will proceed at N times the rate of the slowest connection. Yes, this is a desirable property, because it means that if one connection experiences congestion, the entire aggregate will back off. It probably is not appropriate simply to shift load to a faster (not-yet-congested) connection, as that practice likely defeats TCP's congestion control. This result is conter-intuitive to the max, and contradicts the practice of current path-balancing software; but I think it deserves further consideration. R Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. tel. +1 916 785 4578 e-mail: Randy_Haagens@hp.com -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Tuesday, September 19, 2000 12:22 PM To: hufferd@us.ibm.com; Black_David@emc.com Cc: ips@ece.cmu.edu Subject: iSCSI: Session Partial Resolution > Perhaps I am wrong but it seems we are drifting and not focusing on the > search for concurrence regarding Symmetric vs Asymmetric. What is your > view? John's acquiring the prescient talent of causing the co-chair to appear and take action ... as long as he doesn't start calling himself a "wizard" based on these summoning powers, I won't object :-). In any case ... In the past week, I have seen at most one objection to each of the following two proposed points of consensus: (1) An iSCSI session containing a single TCP connection should not be required to use the currently specified iSCSI command reference numbers and sliding window mechanism because TCP will deliver commands in order. (2) Use of more than one TCP connection per iSCSI session is OPTIONAL. Therefore I declare these to be the WG rough consensus on these issues, and the next version of the iSCSI draft should remove the command reference numbers and sliding window mechanism from the iSCSI header. Somesh Gupta's objection to (1) and Matt Wakeley's continued objection to (2) are noted as part of declaring these to be the WG rough consensus. Anyone else who objects to this declaration of rough consensus should email me directly with the reasons for the objection. OTOH, I do not see consensus on the session model for multiple connection sessions among the Asymmetric model, the Symmetric model, and Pierre Labat's proposal. In order to make progress on iSCSI, I see no alternative to separating multi-connection sessions from the main iSCSI spec. Significant effort and email traffic has been invested in this topic for at least 6 weeks and the issue is not settled -- I don't think holding up the iSCSI spec for another 6+ weeks in hopes of settling this issue on the list is an effective way to make progress, but I'm prepared to listen to dissenting opinions (e.g., if someone thinks there is rough consensus, and I've missed it); please send such opinions directly to me rather than using the list. I've already had one offline comment from an outside observer expressing amazement at the willingness of this community to discuss multi-connection sessions "ad nauseum". Therefore, I would ask that the authors of the next version of the iSCSI draft delete all specification of multiple connection sessions from the next version of the except for a note that they will be handled in a separate document. Producing that separate document is going to require an offline design team. The design team can either be chartered to write a compromise session specification or to evaluate competing specifications and choose one. My current inclination is to do the latter, which would involve having the design team produce a set of requirements and guidelines for session specifications in consultation with the co-chairs, evaluate Internet-Drafts documenting the specifications, and recommend an approach to the WG. Comments on this process are solicited - either on the list or to me directly. Further discussion of multi-connection sessions on the list is probably not a good use of list bandwidth. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- Randy Haagens Director, Networked Storage Architecture Storage Organization Hewlett-Packard Co. e-mail: Randy_Haagens@hp.com tel: +1 916 785 4578 fax: +1 916 785 1911 <> ------_=_NextPart_000_01C027C9.3D89BA80 Content-Type: application/octet-stream; name="Haagens, Randy (E-mail).vcf" Content-Disposition: attachment; filename="Haagens, Randy (E-mail).vcf" BEGIN:VCARD VERSION:2.1 N:Haagens;Randy;;; FN:Randy Haagens (E-mail) ORG:Hewlett-Packard Company;Architecture and Performance TITLE:Director, Networked Storage Architecture TEL;WORK;VOICE:+1 (916) 785-4578 TEL;CELL;VOICE: TEL;WORK;FAX:+1 (916) 785-1911 ADR;WORK:;Roseville, R5U-P5/R5;8000 Foothills Blvd. MS 5668;Roseville;CA;95747-5668;United States of America LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Roseville, R5U-P5/R5=0D=0A8000 Foothills Blvd. MS 5668=0D=0ARoseville, CA 95= 747-5668=0D=0AUnited States of America EMAIL;PREF;INTERNET:Randy_Haagens@hp.com REV:20000607T194543Z END:VCARD ------_=_NextPart_000_01C027C9.3D89BA80-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20497; 25 Sep 2000 18:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA16849 for ips-outgoing; Mon, 25 Sep 2000 17:37:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lightsand.com ([208.50.99.84]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA16842 for ; Mon, 25 Sep 2000 17:36:59 -0400 (EDT) Received: from cx418298b (cx418298-b.orng1.occa.home.com [24.1.179.117]) by lightsand.com (8.9.3+Sun/8.9.1) with SMTP id OAA18651; Mon, 25 Sep 2000 14:36:01 -0700 (PDT) Message-ID: <013501c02739$b0d14de0$75b30118@orng1.occa.home.com> From: "Murali Rajagopal" To: "Douglas Otis" , "Ips" , "Elizabeth G Rodriguez \(Elizabeth\)" , "fc" , "T10" References: Subject: Re: Revised Fibre-Channel over SCTP/IP Date: Mon, 25 Sep 2000 14:43:49 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk All: A clarification is in order: Note that the IPS WG still has not chosen (by rough consensus) which method to encapsulate FC. Currently, there are two proposals on the table: IP and SCTP. Soon, there will a third one using TCP. I beleive that at the San Diego metting (December), we will know. Regards, Murali Rajagopal LightSand Communications ----- Original Message ----- From: "Douglas Otis" To: "Ips" Cc: "Elizabeth G Rodriguez (Elizabeth)" ; "fc" ; "T10" Sent: Monday, September 25, 2000 11:50 AM Subject: Revised Fibre-Channel over SCTP/IP > All, > > The IPS (Internet Protocol Storage) Work Group is officially under IETF > Transport Area. > > Included are changes to illustrate a strategy involved in allowing both > native IP access and FC encapsulation. In addition, a credit token scheme > similar to FC Class 3 has been implemented. The disscussion reflector is at > ips@ece.cmu.edu > to subscribe send email to majordomo@ece.cmu.edu and within the message > body- subscribe ips > the website is at http://www.ece.cmu.edu/~ips. > > > http://www.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-01.txt > > Doug > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24008; 26 Sep 2000 15:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA02221 for ips-outgoing; Tue, 26 Sep 2000 14:20:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA02216 for ; Tue, 26 Sep 2000 14:19:57 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 26 Sep 2000 14:19:17 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FF1@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: iSCSI: Connection status summary corrections Date: Tue, 26 Sep 2000 14:19:07 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk A couple of important corrections to Julian's summary: - There is long standing WG consensus that multiple TCP connections per session are optional. Therefore the Pure Asymmetric model must also support sending both commands and data over the same connection when only one TCP connection is in the session, and this needs to be added to the list of PA drawbacks. - The discussion of Asymmetric models overlooked Pierre Labat's proposal for spreading commands across multiple connections, but using a single command sequence in which the initiator tells the target which connection to expect the next command on. Since this provides the initiator with some ability to do load balancing, Balanced Asymmetric (BA) might be a good name. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Tuesday, September 26, 2000 12:48 PM > To: ips@ece.cmu.edu > Subject: Status summary on multiple connections > > > > Dear colleagues, > > I am attempting to summarize where we stand with regard to the multiple > connection issue > and to the two possible models - Symmetric (S) and Asymmetric (A). > > Many of us feel strongly that the multiple connection issue is central to > the whole design > and cannot be added as an afterthought. Moreover designing the hooks to > allow it later > will certainly already force us to make a decision. And both the hardware > and the software designers will be ill-served if we hand them a > half-backed solution. > > However this is not an invitation to reiterate positions that where > already stated. > If you feel that I have grossly misstated anything in this note please > write me and use > the mailing list only if my answer is not satisfying. > > And yes - like the chairman - I would like to make progress but I don't > see any way to > do it without satisfactorily closing this issue. > > The reasons for multiple connections where discussed at some length and > where very nicely summarized in a series of notes by Michael Krause > (beginning of August). > > The core reasons for having multiple connections where the need for more > bandwidth and availability than a single link can supply with a level of > complexity affordable for simple > installations and with a traffic engineering and management clearly > separated from the transport users (SCSI). > > The session is embodying this requirement. > > The only major objections I have heard against this where those requiring > to go all > the way in having one TCP connection/LU - and after a short debate this > objection > was practically removed. > > The other objections we heard where that this is basically a transport > issue and it should be solved at transport level. > That might be true - but since many, if not most, of TCP applications do > not have this > requirement it is highly unlikely that TCP is going to do connection > trunking in the foreseeable > future. > > iSCSI can be designed to use multiple TCP connections in one of two ways: > > -Asymmetric - one TCP flow only carries commands the others carry only > data > > -Symmetric - every flow carries both commands and their associated data > > The S version is designed in the I-D version 01 > The S version requires an command ordering scheme and that is provided by > a > command counter and a sliding window scheme. It was argued that ordering > needs might be more prevalent than usually thought and a good conservative > design should preserve ordering. > Ordering-per-LU (as it is designed in FCP-2 version 4) was considered > impractical as it > required initiators to maintain state for each LU - while the rest of the > design required initiators only to maintain state for outstanding > commands. > Several comments on this list suggested that this windowing mechanism > could > also be used as a command-flow-control mechanism and that is a "bonus" of > the scheme. > > The A version - comes in two flawors: > > - pure A (PA) in which ONLY commands flow on one TCP connection while > data flow > on DIFFERENT connections (with only one data connection being selected > for a > command); this scheme requires a minimum of 2 TCP connections > although not > necessarily on different physical links. > > - collapsed A (CA) in which commands flow one a single TCP connection > while data > flow on ANY connection; this scheme requires as a minimum 1 TCP > connection. > > Here is first attempt to list the benefits and drawbacks of all of them: > > > - S - benefits > - well understood > - simple hardware setup and/or TCP API activation > - window mechanism can be also used for flow control > - the minimum required is a single TCP connection > > - drawbacks > - need to maintain a window mechanism > - a multiplexing mechanism has to be carefully crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > - PA - benefits > - TCP will both order the commands and provide for flow control > through the TCP window mechanism > - the multiplexing mechanism is simpler as closing TCP windows > will > never affect command flow > - data flow can use a streamlined header (and processing) > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > > - CA - benefits > - TCP will order the commands > - needs a single TCP connection as a minimum > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > - if command flow control is required counters and the sliding > window mechanism > are required > - a multiplexing mechanism has to be carefully crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > From the above it should be apparent - as was already pointed out by Matt > Wakeley - > that the CA inherits the drawbacks of S and PA and as such the choice is > really > between S and PA. > > Julo > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32278; 28 Sep 2000 15:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA03625 for ips-outgoing; Thu, 28 Sep 2000 14:36:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA03619 for ; Thu, 28 Sep 2000 14:35:59 -0400 (EDT) Received: from ebaymail2.EBay.Sun.COM ([129.150.111.20]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA16640 for ; Thu, 28 Sep 2000 11:35:58 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.152.210]) by ebaymail2.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA20100 for ; Thu, 28 Sep 2000 11:35:58 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA07686; Thu, 28 Sep 2000 11:35:57 -0700 (PDT) Message-Id: <200009281835.LAA07686@ha10nwk.EBay.Sun.COM> Date: Thu, 28 Sep 2000 11:35:57 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Why FCP doesn't need RDMA? It has a better way. To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: J+7pMV4k/Hb+Xq3BEVON6g== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > From an iSCSI viewpoint, I don't see iSCSI by itself as being sufficient > to motivate a protocol-independent RDMA - an iSCSI HBA could understand > the iSCSI headers and interact with DMA in the same fashion as existing > HBAs. The task before those interested in RDMA is to identify a set > of protocols for which a common RDMA mechanism makes sense from > an engineering standpoint. I tend to agree with the previous emails > that iSCSI could make optional use of a common RDMA mechanism > if available, but must not REQUIRE its use. As another data point to show that RDMA is not required, a recently reported group acheiving 980+Mbps on a gigabit ethernet using NFS. NFS is not nearly as simple to parse headers as FCP or the current iSCSI draft, and the authors were able to reprogram an Altheon card's firmware to acheive it. I suspect a good design engineer could do the same for iSCSI without RDMA. http://people.freebsd.org/~ken/zero_copy/ So while RDMA is interesting, I see no reason to REQUIRE it for iSCSI. -David From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20382; 25 Sep 2000 18:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA14942 for ips-outgoing; Mon, 25 Sep 2000 16:52:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA14937 for ; Mon, 25 Sep 2000 16:52:33 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 16:51:58 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FE3@corpmx9.isus.emc.com> To: matt_wakeley@agilent.com, Black_David@emc.com, ips@ece.cmu.edu Subject: iSCSI: multiple connection process Date: Mon, 25 Sep 2000 16:51:48 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Exactly. There has been NO consensus that multi-connection sessions should be > removed from the main specification (only your proposal to do that). Just > because we can't (yet) agree on something, does not mean "let's not do it at > all, or put it off until later." Sorry, lack of rough consensus means one of those two. Documents don't leave the WG without rough consensus, so either the document gets delayed for consensus, or the feature gets removed. Needless to say, we don't have rough consensus at the moment, nor do I see much in the way of common ground emerging among the different points of view. > We already had the offline design team and we came up with something that > everyone is arguing about. So let's have some more discussion on what it is > that people don't like, and what the requirements are, so that another offline > design team has something to work with. Actually, offline design teams came up with both the Asymmetric and Symmetric models ... and now there's a third. It appears to me that continued list discussion is an active version of "put it off until later" rather than a passive one. I could be wrong, but someone will need to explain what will be different about continued list discussion that will lead to a consensus in a way that discussion since Pittsburgh has not. The alternative process I have in mind is more of an evaluation team than a design team - form a team of people who publish evaluation criteria for multiple-connection sessions, evaluate drafts describing the various models against those criteria, and make a recommendation to the WG. One possible way to make continued progress on the list is to discuss what those evaluation criteria ought to be rather than asking an off-line team to write them. Doing this really requires someone to track the criteria as they evolve - a volunteer for this task is hereby solicited, preferably someone who has not taken a position on this session issue. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23057; 26 Sep 2000 9:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA08316 for ips-outgoing; Tue, 26 Sep 2000 08:07:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA08310 for ; Tue, 26 Sep 2000 08:07:08 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id HAA08475; Tue, 26 Sep 2000 07:07:09 -0500 Message-ID: <39D0916D.13D83DC9@stewart.chicago.il.us> Date: Tue, 26 Sep 2000 07:07:09 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: Black_David@emc.com, matt_wakeley@agilent.com, ips@ece.cmu.edu Subject: Re: iSCSI: multiple connection process References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julain: I have not been involved in any of the offline emails that I am aware of and I don't see a consenses on the list. I think David is just trying to move us forward and I support this. I don't see that he is saying remove it completly.. just lets get something finished and then open this up in another draft... I think it is a good way to move forward... R julian_satran@il.ibm.com wrote: > > David, > > Only for correctness - the SAME design team has discussed both the > symmetric and > asymmetric model and choose the symmetric because it did not have a good > enough > asymmetric solution; that does not hold now anymore and that is why I > opened > this thread (should I say I am sorry?) > > I have two objections to the direction you have taken: > > - you considered objections that where voiced to you and not to the mailing > list; > the community should judge if they are core or marginal; I would > appreciate if > you could summarize them > > - the key issues with any of the two approaches are understood - an > evaluation > committee will only broaden the "discontent" > > Julo > > Black_David@emc.com on 25/09/2000 23:51:48 > > Please respond to Black_David@emc.com > > To: matt_wakeley@agilent.com, Black_David@emc.com, ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: iSCSI: multiple connection process > > > Exactly. There has been NO consensus that multi-connection sessions > should be > > removed from the main specification (only your proposal to do that). > Just > > because we can't (yet) agree on something, does not mean "let's not do it > at > > all, or put it off until later." > > Sorry, lack of rough consensus means one of those two. Documents don't > leave the WG without rough consensus, so either the document gets delayed > for consensus, or the feature gets removed. Needless to say, we don't have > rough consensus at the moment, nor do I see much in the way of common > ground emerging among the different points of view. > > > We already had the offline design team and we came up with something that > > everyone is arguing about. So let's have some more discussion on what it > is > > that people don't like, and what the requirements are, so that another > offline > > design team has something to work with. > > Actually, offline design teams came up with both the Asymmetric and > Symmetric > models ... and now there's a third. It appears to me that continued list > discussion is > an active version of "put it off until later" rather than a passive one. I > could be > wrong, but someone will need to explain what will be different about > continued list > discussion that will lead to a consensus in a way that discussion since > Pittsburgh > has not. > > The alternative process I have in mind is more of an evaluation team than a > design > team - form a team of people who publish evaluation criteria for > multiple-connection > sessions, evaluate drafts describing the various models against those > criteria, > and make a recommendation to the WG. One possible way to make continued > progress on the list is to discuss what those evaluation criteria ought to > be > rather than asking an off-line team to write them. Doing this really > requires someone > to track the criteria as they evolve - a volunteer for this task is hereby > solicited, > preferably someone who has not taken a position on this session issue. > > Thanks, > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31955; 28 Sep 2000 13:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA28593 for ips-outgoing; Thu, 28 Sep 2000 12:31:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA28589 for ; Thu, 28 Sep 2000 12:31:21 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 12:30:50 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101007@corpmx9.isus.emc.com> To: rsnively@Brocade.COM, matt_wakeley@agilent.com, ips@ece.cmu.edu Subject: RE: Why FCP doesn't need RDMA? It has a better way. Date: Thu, 28 Sep 2000 12:30:46 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk With my WG co-chair hat off: > Actually, RDMA is not needed in FCP because all protocol chips > implemented perform a real peer-to-peer DMA straight to the > data areas specified by the user's interaction with the operating > systems allocation algorithms. The combination of the FCP/SCSI > pointer structure, task tagging, and the FC relative offset perform the > function you would otherwise have to use RDMA to accomplish. And this illuminates the design tradeoff that may motivate RDMA. If one only wants to accelerate one protocol (SCSI/FCP in the above example) then having hardware understand its headers and doing the DMA on that basis is a fairly obvious way to go - HBAs for both parallel SCSI and Fibre Channel (SCSI/FCP) do this. RDMA may be interesting if there are multiple protocols involved, and there are engineering concerns that lead to not wanting to implement hardware support for all of them. >From an iSCSI viewpoint, I don't see iSCSI by itself as being sufficient to motivate a protocol-independent RDMA - an iSCSI HBA could understand the iSCSI headers and interact with DMA in the same fashion as existing HBAs. The task before those interested in RDMA is to identify a set of protocols for which a common RDMA mechanism makes sense from an engineering standpoint. I tend to agree with the previous emails that iSCSI could make optional use of a common RDMA mechanism if available, but must not REQUIRE its use. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20148; 25 Sep 2000 16:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA08483 for ips-outgoing; Mon, 25 Sep 2000 14:28:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA08477 for ; Mon, 25 Sep 2000 14:28:51 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id OAA25578 for ; Mon, 25 Sep 2000 14:28:50 -0400 Message-ID: <013301c0271b$c1acee30$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: Subject: RDMA over TCP Date: Mon, 25 Sep 2000 14:09:32 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Another Try: ftp://coke.giganet.com/rdma-tcp.txt From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25125; 26 Sep 2000 22:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA18117 for ips-outgoing; Tue, 26 Sep 2000 21:05:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA18110 for ; Tue, 26 Sep 2000 21:05:03 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id RAA13979; Tue, 26 Sep 2000 17:52:15 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id RAA30598; Tue, 26 Sep 2000 17:50:00 -0700 From: "Douglas Otis" To: "Michael Krause" , Cc: Subject: RE: ISCSI: flow control Date: Tue, 26 Sep 2000 18:07:30 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <4.2.2.20000926115300.00ba8500@hpindlm.cup.hp.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Mike, You will find a similar flow control proposal in the current fc-sctp-ip draft. http://www.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-01.txt This proposal assumes the SCTP<->FC interface will receive credits and relay them as part of the general header. In other words, within every response, credit may be updated for the individual connector that terminates each stream. The SCTP agent may also act to accommodate additional credit owing to interconnecting FIFOs to these connectors to assist in allowing greater network distance than normally supported by FC. Should the end point not be a connector but rather a port to a controller, then fabric login and credits would be handled directly by the SCTP agent and additional credits may be retracted and made available to other ports upon acknowledgement of the retraction. The proposal advisory was included to illustrate use of the FC header to allow a comparison to that of alternative proposals. You will find it relatively easy to implement. As FC starts out with limited credit due to expected low latency, a means to extend credit beyond login is essential. An Ordered Set Sequence or credit message can be sent without affecting credit as well. Once flow-control is offered, the FC structures do not require change as these structures become independent of transport speed. Each stream would originate as an initiator. The use of CRC is optional for native IP access and the entire non-FC frame information is placed within a prefix for easy firmware manipulation. The intent of this proposal was to strike a balance between bridge-only encapsulation and native access. The solutions found for FC access to controllers or devices remain unchanged by this proposal. The communication structures are defined by existing standards which also remain unchanged. All streams, ports and connectors are able to share a common flow. There would be no blocking or credit uncertainty as credit would remain defined as a FC frame carried as an SCTP chunk. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Michael Krause > Sent: Tuesday, September 26, 2000 3:54 PM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu > Subject: RE: ISCSI: flow control > > > At 11:05 AM 9/25/00 -0400, Black_David@emc.com wrote: > >Mike, > > > > > In essence, this is what InfiniBand does and others have been > > > advocating. When the ACK (SCSI response) is returned it > encodes a credit > > > to inform the sender of how many receives buffers (available > command queue > > > > > slots) have been posted. > > > >Could you post, or provide a pointer to a self-contained specification of > >that > >mechanism? If this is a pointer to InfiniBand specs, a heads-up on any > >intellectual property issues is in order. > > The V1.0 InfiniBand spec is about to be made public and as such, I would > refer people to it to understand the specifics of that architecture. The > problem being addressed in InfiniBand and here is rather generic > in nature > - how to avoid overflowing a receive queue using a credit > scheme. InfiniBand's scheme is unique in terms of the specifics > (encoding, > ACK message formats, etc.) to it but the essence is the same. > I'll try to > paraphrase the scheme here in more general purpose terms - if a RFC draft > is required, let me know. > > This credit scheme is implemented as follows: > > (1) Responder encodes a N-bit credit within the ACK (iSCSI response) > message. Credits are absolute values, i.e. one "snapshots" and > encodes the > current responder's credit value to return in the ACK message. If the > endnode does not support credits the requester shall assume an infinite > value. > > (2) Credits are on a per connection or per session basis. Simplicity > favors the per connection basis but if the session layer is load > balancing > commands across multiple connections and given the completion processing > and resource management for commands is at the session layer, it > may not be > a performance / implementation inhibitor to implement this within the > session layer itself. In general, this can be implemented across > multiple > ports or multiple NICs, entirely in software or hardware or a mix with > minimal overhead. > > (3) Requester maintains a current credit count and decrements this value > for each outstanding request. When new credit is received, the requester > updates its credit window and determines whether new requests may be > injected into the network. > > (4) If a requester does not receive any credits for a period of time and > there are no outstanding requests, it may probe the responder by > issuing a > single request. The responder may respond with a RNR NAK or an > ACK with a > credit update. This prevents deadlock. Ideally, one would allow an > unsolicited ACK to be sent by the responder when new credit arrives and > there are no outstanding requests being processed. The advantage for > unsolicited ACKs is simplicity - the requester never generates an > operation > without credit and the responder only returns credit thus making the > implementation simpler for both sides. > > (5) Responder increments its credit value each time a receive > descriptor / > command queue element is posted / available. Again this value may be per > connection or per session depending upon the resource / coherency > strategy > pursued. > > (6) To support long-distance implementation, one would like to > stretch the > number of credits under the assumption that a number of responses > are also > in-flight at a given time. If this is implemented, then a RNR > NAK / QUEUE > FULL algorithm is needed as is an unsolicited ACK / grant credit > message. An implementation would need to understand the dynamic rate of > commands completions and perform optimistic calculations for what this > stretched "credit" window is. When it receives a RNR NAK / QUEUE FULL > message, it would reduce the injection rate by a moderate amount (avoid > large oscillations) - some modeling would be needed to understand > what this > reduction would be. > > (7) Requester's can transmit requests that do not consume responder > resources, e.g. RDMA READ, RDMA WRITE without immediate data, etc. > > >A concern that has been raised in this discussion is how credit > >information relates to the concurrency and ordering (esp. lack > thereof) of > >transmission and processing of SCSI commands and the transmission of > >responses. My understanding of the FCP approach to buffer > management (and > >I assume InfiniBand is similar) is that traffic cannot be sent > unless the > >sender knows that there is space in the receiver's buffer to accommodate > >it (i.e., the sender has a credit or credits indicating space in the > >receiver's buffer). > > In general, this is correct for InfiniBand - one cannot initiate a SEND > operation unless credit is available. It should be kept in mind that > InfiniBand was designed for the data center, i.e. 300 meters for a given > link instance. As such, some optimizations were made that may not be > acceptable w.r.t. this workgroup's focus. > > >This implies is that if for some reason the receiver stalled, all the > >in-flight commands and data could be successfully received. In > contrast, > >I've seen discussion on this list of long distance connections in which > >there is potentially more traffic in flight than the receiver could > >accommodate if the receiver stopped. I believe that whether to > allow this > >is an open issue, but the underlying cause is valid - there is a > desire to > >use iSCSI in situations where the initiator to target coupling is looser > >(in this case, due to distance) than is typical for SCSI and > Fibre Channel. > > > Mike > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24776; 26 Sep 2000 20:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA14111 for ips-outgoing; Tue, 26 Sep 2000 18:55:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA14107 for ; Tue, 26 Sep 2000 18:55:15 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 0F5A478; Tue, 26 Sep 2000 15:55:14 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA22761; Tue, 26 Sep 2000 15:55:19 -0700 (PDT) Message-Id: <4.2.2.20000926115300.00ba8500@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 26 Sep 2000 15:53:53 -0700 To: Black_David@emc.com From: Michael Krause Subject: RE: ISCSI: flow control Cc: ips@ece.cmu.edu In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100FD4@corpmx9.isus.emc. com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:05 AM 9/25/00 -0400, Black_David@emc.com wrote: >Mike, > > > In essence, this is what InfiniBand does and others have been > > advocating. When the ACK (SCSI response) is returned it encodes a credit > > to inform the sender of how many receives buffers (available command queue > > > slots) have been posted. > >Could you post, or provide a pointer to a self-contained specification of >that >mechanism? If this is a pointer to InfiniBand specs, a heads-up on any >intellectual property issues is in order. The V1.0 InfiniBand spec is about to be made public and as such, I would refer people to it to understand the specifics of that architecture. The problem being addressed in InfiniBand and here is rather generic in nature - how to avoid overflowing a receive queue using a credit scheme. InfiniBand's scheme is unique in terms of the specifics (encoding, ACK message formats, etc.) to it but the essence is the same. I'll try to paraphrase the scheme here in more general purpose terms - if a RFC draft is required, let me know. This credit scheme is implemented as follows: (1) Responder encodes a N-bit credit within the ACK (iSCSI response) message. Credits are absolute values, i.e. one "snapshots" and encodes the current responder's credit value to return in the ACK message. If the endnode does not support credits the requester shall assume an infinite value. (2) Credits are on a per connection or per session basis. Simplicity favors the per connection basis but if the session layer is load balancing commands across multiple connections and given the completion processing and resource management for commands is at the session layer, it may not be a performance / implementation inhibitor to implement this within the session layer itself. In general, this can be implemented across multiple ports or multiple NICs, entirely in software or hardware or a mix with minimal overhead. (3) Requester maintains a current credit count and decrements this value for each outstanding request. When new credit is received, the requester updates its credit window and determines whether new requests may be injected into the network. (4) If a requester does not receive any credits for a period of time and there are no outstanding requests, it may probe the responder by issuing a single request. The responder may respond with a RNR NAK or an ACK with a credit update. This prevents deadlock. Ideally, one would allow an unsolicited ACK to be sent by the responder when new credit arrives and there are no outstanding requests being processed. The advantage for unsolicited ACKs is simplicity - the requester never generates an operation without credit and the responder only returns credit thus making the implementation simpler for both sides. (5) Responder increments its credit value each time a receive descriptor / command queue element is posted / available. Again this value may be per connection or per session depending upon the resource / coherency strategy pursued. (6) To support long-distance implementation, one would like to stretch the number of credits under the assumption that a number of responses are also in-flight at a given time. If this is implemented, then a RNR NAK / QUEUE FULL algorithm is needed as is an unsolicited ACK / grant credit message. An implementation would need to understand the dynamic rate of commands completions and perform optimistic calculations for what this stretched "credit" window is. When it receives a RNR NAK / QUEUE FULL message, it would reduce the injection rate by a moderate amount (avoid large oscillations) - some modeling would be needed to understand what this reduction would be. (7) Requester's can transmit requests that do not consume responder resources, e.g. RDMA READ, RDMA WRITE without immediate data, etc. >A concern that has been raised in this discussion is how credit >information relates to the concurrency and ordering (esp. lack thereof) of >transmission and processing of SCSI commands and the transmission of >responses. My understanding of the FCP approach to buffer management (and >I assume InfiniBand is similar) is that traffic cannot be sent unless the >sender knows that there is space in the receiver's buffer to accommodate >it (i.e., the sender has a credit or credits indicating space in the >receiver's buffer). In general, this is correct for InfiniBand - one cannot initiate a SEND operation unless credit is available. It should be kept in mind that InfiniBand was designed for the data center, i.e. 300 meters for a given link instance. As such, some optimizations were made that may not be acceptable w.r.t. this workgroup's focus. >This implies is that if for some reason the receiver stalled, all the >in-flight commands and data could be successfully received. In contrast, >I've seen discussion on this list of long distance connections in which >there is potentially more traffic in flight than the receiver could >accommodate if the receiver stopped. I believe that whether to allow this >is an open issue, but the underlying cause is valid - there is a desire to >use iSCSI in situations where the initiator to target coupling is looser >(in this case, due to distance) than is typical for SCSI and Fibre Channel. Mike From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32108; 28 Sep 2000 14:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA29349 for ips-outgoing; Thu, 28 Sep 2000 12:50:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA29331 for ; Thu, 28 Sep 2000 12:50:28 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA14562; Thu, 28 Sep 2000 09:37:08 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA04093; Thu, 28 Sep 2000 09:34:54 -0700 From: "Douglas Otis" To: "Stephen Byan" , Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Thu, 28 Sep 2000 09:52:42 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <8133266FE373D11190CD00805FA768BF055BD1EB@shrcmsg1.tdh.qntm.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, I would agree hardware acceleration would be beneficial. SCTP would provide significant advantages if accelerated. With alignment maintained and out of sequence processing allowed, the effect of accelerating SCTP would be to allow competitive products to FC and cLAN. TCP would only provide a marginal improvement and never to the same levels unless you alter TCP significantly. Such alternation of TCP would not be beneficial however. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Byan > Sent: Thursday, September 28, 2000 8:26 AM > To: 'ips@ece.cmu.edu' > Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in > iSCSI)) > > > Douglas Otis [mailto:dotis@sanlight.net] wrote: > > > As SCSI does not benefit from VI > > with zero copy already possible with frame alignment, the > > area where there > > could be commonality has been missed with the choice of CRC. > > I believe the point in factoring out the RDMA mechanism from iSCSI is to > make the same hardware assists available to other application > protocols. The > benefit this provides to iSCSI is that it enables iSCSI adapters to be > marketed as general-purpose application protocol accelerators, not just as > iSCSI accelerators. This is not a technical benefit, but rather > an economic > and marketing benefit. > > Doug, do you disagree that having a generic RDMA facility would be an > economic and marketing benefit? > > Regards, > -Steve > > Steve Byan > > Design Engineer > MS 1-3/E23 > 333 South Street > Shrewsbury, MA 01545 > (508)770-3414 > fax: (508)770-2604 > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29375; 27 Sep 2000 22:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA06187 for ips-outgoing; Wed, 27 Sep 2000 21:48:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA06182 for ; Wed, 27 Sep 2000 21:47:56 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id SAA14396; Wed, 27 Sep 2000 18:34:54 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id SAA01920; Wed, 27 Sep 2000 18:32:39 -0700 From: "Douglas Otis" To: "John Hufferd/San Jose/IBM" , Subject: RE: Date: Wed, 27 Sep 2000 18:50:20 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk John, If the intent of the draft-satran-iscsi-01.txt is to eventually modify TCP in a significant way, these issues should be addressed ahead of concluding such features as RDMA with TCP will become available. As much of the optimization required in the use of TCP for SCSI hinge on such modifications, it would appear you have already concluded such modifications *will* take place. Most should agree frame alignment, out of sequence processing, independent streams sharing a single control, and zero-copy data handling are deciding issues. Separate TCP buffers to control resources, frame alignment, unique TCP APIs are between the lines in this proposal. Before havoc ensues, there is an alternative. Rather than bending TCP to fit, SCTP has already laid a foundation. As such, *no* transport modifications are required and yet customers still get friendly TCP behavior. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > John Hufferd/San Jose/IBM > Sent: Wednesday, September 27, 2000 3:08 PM > To: ips@ece.cmu.edu > Subject: > > > Costa, > If you remember way back when we first got together, you had a good RDMA > proposal for use with TCP/IP using the various optional fields. We all > seemed to love it at the time. I for one was disappointed when we dropped > it out of the iSCSI proposal (even as an optional consideration). The > thoughts were, that if we started to push this onto TCP/IP folks, we would > spend much time on that and not get the iSCSI stuff done. So it was > dropped. > > I for one would find it useful to bring it back, but only as an optional > feature, and only if that did not get in the way of the iSCSI Stuff, since > you can see how that has already begin to fill up the current ips > reflector. But I think working it as a side issue, on another reflector, > until it is fully baked seems OK to me. > > Summary: > RDMA -- Yes -- the one you originally had proposed > Optional --Yes > > . > . > . > John L. Hufferd > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31794; 28 Sep 2000 12:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA26238 for ips-outgoing; Thu, 28 Sep 2000 11:25:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA26231 for ; Thu, 28 Sep 2000 11:25:54 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id IAA24690 for ; Thu, 28 Sep 2000 08:25:48 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 08:25:45 -0700 Message-ID: <8133266FE373D11190CD00805FA768BF055BD1EB@shrcmsg1.tdh.qntm.com> From: Stephen Byan To: "'ips@ece.cmu.edu'" Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Thu, 28 Sep 2000 08:25:47 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis [mailto:dotis@sanlight.net] wrote: > As SCSI does not benefit from VI > with zero copy already possible with frame alignment, the > area where there > could be commonality has been missed with the choice of CRC. I believe the point in factoring out the RDMA mechanism from iSCSI is to make the same hardware assists available to other application protocols. The benefit this provides to iSCSI is that it enables iSCSI adapters to be marketed as general-purpose application protocol accelerators, not just as iSCSI accelerators. This is not a technical benefit, but rather an economic and marketing benefit. Doug, do you disagree that having a generic RDMA facility would be an economic and marketing benefit? Regards, -Steve Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23864; 26 Sep 2000 14:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA26028 for ips-outgoing; Tue, 26 Sep 2000 12:51:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA25963 for ; Tue, 26 Sep 2000 12:51:10 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id SAA195530 for ; Tue, 26 Sep 2000 18:50:39 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA155422 for ; Tue, 26 Sep 2000 18:50:38 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256966.005C8461 ; Tue, 26 Sep 2000 18:50:32 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 26 Sep 2000 19:48:17 +0300 Subject: Status summary on multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear colleagues, I am attempting to summarize where we stand with regard to the multiple connection issue and to the two possible models - Symmetric (S) and Asymmetric (A). Many of us feel strongly that the multiple connection issue is central to the whole design and cannot be added as an afterthought. Moreover designing the hooks to allow it later will certainly already force us to make a decision. And both the hardware and the software designers will be ill-served if we hand them a half-backed solution. However this is not an invitation to reiterate positions that where already stated. If you feel that I have grossly misstated anything in this note please write me and use the mailing list only if my answer is not satisfying. And yes - like the chairman - I would like to make progress but I don't see any way to do it without satisfactorily closing this issue. The reasons for multiple connections where discussed at some length and where very nicely summarized in a series of notes by Michael Krause (beginning of August). The core reasons for having multiple connections where the need for more bandwidth and availability than a single link can supply with a level of complexity affordable for simple installations and with a traffic engineering and management clearly separated from the transport users (SCSI). The session is embodying this requirement. The only major objections I have heard against this where those requiring to go all the way in having one TCP connection/LU - and after a short debate this objection was practically removed. The other objections we heard where that this is basically a transport issue and it should be solved at transport level. That might be true - but since many, if not most, of TCP applications do not have this requirement it is highly unlikely that TCP is going to do connection trunking in the foreseeable future. iSCSI can be designed to use multiple TCP connections in one of two ways: -Asymmetric - one TCP flow only carries commands the others carry only data -Symmetric - every flow carries both commands and their associated data The S version is designed in the I-D version 01 The S version requires an command ordering scheme and that is provided by a command counter and a sliding window scheme. It was argued that ordering needs might be more prevalent than usually thought and a good conservative design should preserve ordering. Ordering-per-LU (as it is designed in FCP-2 version 4) was considered impractical as it required initiators to maintain state for each LU - while the rest of the design required initiators only to maintain state for outstanding commands. Several comments on this list suggested that this windowing mechanism could also be used as a command-flow-control mechanism and that is a "bonus" of the scheme. The A version - comes in two flawors: - pure A (PA) in which ONLY commands flow on one TCP connection while data flow on DIFFERENT connections (with only one data connection being selected for a command); this scheme requires a minimum of 2 TCP connections although not necessarily on different physical links. - collapsed A (CA) in which commands flow one a single TCP connection while data flow on ANY connection; this scheme requires as a minimum 1 TCP connection. Here is first attempt to list the benefits and drawbacks of all of them: - S - benefits - well understood - simple hardware setup and/or TCP API activation - window mechanism can be also used for flow control - the minimum required is a single TCP connection - drawbacks - need to maintain a window mechanism - a multiplexing mechanism has to be carefully crafted to avoid closing TCP windows to severely affect command flow and performance - PA - benefits - TCP will both order the commands and provide for flow control through the TCP window mechanism - the multiplexing mechanism is simpler as closing TCP windows will never affect command flow - data flow can use a streamlined header (and processing) - drawbacks - more complex hardware and/or software API activation - need for a minimum of two TCP connections (not links) - CA - benefits - TCP will order the commands - needs a single TCP connection as a minimum - drawbacks - more complex hardware and/or software API activation - need for a minimum of two TCP connections (not links) - if command flow control is required counters and the sliding window mechanism are required - a multiplexing mechanism has to be carefully crafted to avoid closing TCP windows to severely affect command flow and performance >From the above it should be apparent - as was already pointed out by Matt Wakeley - that the CA inherits the drawbacks of S and PA and as such the choice is really between S and PA. Julo From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20292; 25 Sep 2000 17:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA13059 for ips-outgoing; Mon, 25 Sep 2000 16:10:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1tx.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA13054 for ; Mon, 25 Sep 2000 16:10:07 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1tx.cos.agilent.com (Postfix) with ESMTP id 4D9DD48F; Mon, 25 Sep 2000 14:10:02 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 4A69228; Mon, 25 Sep 2000 16:10:01 -0400 (EDT) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.188]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id NAA02519; Mon, 25 Sep 2000 13:10:00 -0700 (PDT) Message-ID: <39CFB136.C5626E24@agilent.com> Date: Mon, 25 Sep 2000 13:10:30 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: multiple connections References: <0F31E5C394DAD311B60C00E029101A0704100FD3@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Black_David@emc.com wrote: > Julian, > > I have to ask you not to do that. There is consensus > in the WG that multiple connection sessions are an > important feature that needs to be specified, but > should be optional to implement. agreed. > There is NOT consensus > on what the right design approach is. agreed. > I have off-line > email from proponents of both the Asymmetric and Symmetric > multi-connection session models expressing dismay > at the separation of them from the main specification Exactly. There has been NO consensus that multi-connection sessions should be removed from the main specification (only your proposal to do that). Just because we can't (yet) agree on something, does not mean "let's not do it at all, or put it off until later." Any well performing iSCSI implementation will have to have some level of hardware implementation. Putting off the discussion of multiple connections will either delay hardware, or require hardware rolls to implement the new functionality after the initial hardware is out. > > and arguing that their preferred approach is the right > one. These reinforce my observations that there is > no consensus on the issue, and that the mailing list > discussion is unlikely to achieve consensus. As I > stated in earlier email, the requirement for > multiple connection sessions has not been removed, > but spending the next 6-8 weeks discussing it on the > mailing list does not appear likely to achieve consensus. > We need to try something else, namely an off-line > design team. We already had the offline design team and we came up with something that everyone is arguing about. So let's have some more discussion on what it is that people don't like, and what the requirements are, so that another offline design team has something to work with. > In the near term, list bandwidth is > better used to make progress on issues where > progress is still possible, such as flow control. In my mind, the flow control is directly related to how many connections there are, and how they will work. > > > Thanks, > --David Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27789; 27 Sep 2000 13:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA14659 for ips-outgoing; Wed, 27 Sep 2000 12:36:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (play-doh.Stanford.EDU [128.12.131.42]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA14654 for ; Wed, 27 Sep 2000 12:36:09 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 429F28E35; Wed, 27 Sep 2000 09:31:18 -0700 (PDT) To: "Jim Williams" Cc: Subject: Re: New List: rdma@cisco.com: to discuss RDMA References: <008e01c02892$d45d6280$cb08efd0@sachmo.giganet.com> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 27 Sep 2000 09:31:18 -0700 In-Reply-To: "Jim Williams"'s message of "Wed, 27 Sep 2000 10:54:21 -0400" Message-ID: Lines: 1 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19884; 25 Sep 2000 14:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06719 for ips-outgoing; Mon, 25 Sep 2000 13:48:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06713 for ; Mon, 25 Sep 2000 13:48:09 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Mon, 25 Sep 2000 10:49:25 -0700 Message-ID: From: Joshua Tseng To: Yaron Klein , Joshua Tseng Cc: ips@ece.cmu.edu Subject: RE: iSCSI Security Protocol Date: Mon, 25 Sep 2000 10:49:23 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id NAA06715 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id NAB06719 Yaron, The inconvenience of using IPSec through proxies and firewalls is not a new one, but it does not seem to be prohibitively burdenesome on the end-user, at least in IPSec deployments I have seen so far. The problem with proxies is that you have introduced a new entity into your trust model, and IPSec forces you to account for that entity, which effectively is a middle-man. Should that middle-man be required to authenticate itself? IPSec says yes. But of course, that's more work on the system deployment side. The downside of including the authentication within iSCSI is that the proxy middle-man does not authenticate itself. It may be easier for the end-user to set up and manage their network, but there is no defense if the proxy has its security compromised. Regardless, I would support including an authentication mechanism within iSCSI, although I disagree that it would make things perform substantially faster (MD5 is pretty cheap). Rather, an iSCSI authentication block would give the end-user more flexibility and options to use proxies. Also, I would not support an encryption mechanism within iSCSI, since that can=20 effectively be taken care of with IPSec. As far as h/w performance, I am aware of several chip vendors who=20 claim gigabit performance with 56-bit encryption. Here is one public reference. This is an old product I believe--the next generation should be much faster. http://www.chrysalis-its.com/products/pdfs/luna_340_datasheet.pdf Josh -----Original Message----- From: Yaron Klein [mailto:klein@sanrad.com] Sent: Sunday, September 24, 2000 7:41 AM To: Joshua Tseng Cc: ips@ece.cmu.edu Subject: Re: iSCSI Security Protocol Joshua, What I meant in "forcing to use IPsec" is that you must have an authentication mechanism in the iSCSI spec if your client does not support IPsec. Several other drawbacks for IPsec: =B7 The iSCSI will use different AH for the header and the data, thus ena= ble fast performance in iSCSI gateways applications (where the header is modified = and the message is forwarded - only the header AH need to be calculated). =B7 The point you mentioned, using tunnel mode IPsec may cause some probl= ems while traveling through gateways and proxies. So, the system administrator can set and configure the system as he wishe= s. If the system supports IPsec, iSCSI authentication will not be needed. If you have any results from tests with IPsec performance, let us know! Regards, Yaron Joshua Tseng wrote: > Hi Yaron, > > >For full security (authentication and encryption) use external protoco= l, > e.g., > >IPsec. You can define an IPsec policy for encrypting everything (not > feasible > >for most cases) or just the first 48 bytes (headers) and so on. > > I agree we should look at IPSec. Today, all SSH implementations are > software-based as far as I know. IPSec hardware is much more readily > available. > > >However, IPsec may cause some problems since it is IP oriented (connection > >oriented and not session oriented). Moreover, you are forcing the clie= nt to > have > >IPsec, which is not always true. > > I'm not sure what you mean here. IPSec is a layer-3 protocol, and it h= as > no knowledge of TCP. Conversely, TCP is unaware of IPSec operating at = the > layer below it, and IPSec should not interfere at all with TCP. The on= ly > drawback I see is that if you selectively apply a security policy to iS= CSI > conversations which may go over the same TCP connection, some TCP segme= nts > may be encrypted and others will not. For example, a policy may dictat= e > no security for one iSCSI conversation, but ESP tunneling w/3DES for a > different iSCSI conversation. Both conversations use the same TCP > connection. > If this happens, you will not see a difference at the TCP end points, b= ut in > the network you'll see some segments in a TCP connection "disappear", a= s > they > have been encrypted, while others may be left alone and visible to the > network, > IP and TCP headers and all. This effect may confuse firewalls and othe= r > monitoring points in the network. But I would rather have the flexibil= ity > to do this than to be forced to apply a single uniform security policy = to > all > iSCSI traffic using a TCP connection. > > I also don't know what you mean by "forcing the client to have IPSec". = If > the > client doesn't have IPSec, this can be negotiated out at iSCSI login. = Or, > if > you want to do authentication before iSCSI login, IKE (Internet Key > Exchange) > can easily be implemented in software. > > >The security scheme in the iSCSI draft includes authorization and > >authentication. The authorization is done in the login phase with the > >negotiation (detailed in the draft), and authentication is achieved by= a > trailer > >that checks the integrity of the data and the header (either simple CR= C or > some > >mac algorithm). > > So it seems you do not intend to leverage the Authentication Headers (A= H) > protocol > and will imbed the authentication & data integrity mechanism within the > iSCSI protocol? > > Regards, > > Josh From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32506; 28 Sep 2000 16:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA07543 for ips-outgoing; Thu, 28 Sep 2000 16:04:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA06210 for ; Thu, 28 Sep 2000 15:37:56 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 12:39:23 -0700 Message-ID: From: Robert Snively To: "'Douglas Otis'" , John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: RE: not every song Date: Thu, 28 Sep 2000 12:37:21 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > FCP > offers WRITE-DATA and READ-RESPONSE structures if desired. Reminder: No, it doesn't. FCP-2 (the new and improved FCP) took the function out because nobody implemented it and it did not make sense. It would have required additional recovery mechanisms and it did not improve performance. Bob From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32467; 28 Sep 2000 16:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA05349 for ips-outgoing; Thu, 28 Sep 2000 15:17:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA05345 for ; Thu, 28 Sep 2000 15:16:56 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA17788 for ; Thu, 28 Sep 2000 21:16:14 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id VAA53408 for ; Thu, 28 Sep 2000 21:16:14 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256968.0069D9F3 ; Thu, 28 Sep 2000 21:16:10 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 28 Sep 2000 22:10:53 +0300 Subject: Re: Bidirectional SCSI commands and iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, What is the intended use? Are there already commands that use it? Is there a major/minor distinction - like output data & input results or output parameters & input data - or is it intended to be completely symmetric? Julo "Jim Hafner/Almaden/IBM" on 20/09/2000 17:56:27 Please respond to "Jim Hafner/Almaden/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Bidirectional SCSI commands and iSCSI Folks, In case you missed this, T10 just approved changes to SAM-2 (rev14) which remove the restriction that a single SCSI command can have a data flow in only one direction (DataIn xor DataOut). Transport specs (that want to allow this) will need to change to allow for two sets of Data buffer fields in the headers (length, over/underrun, etc.). Is anybody looking to modify the iSCSI spec to get these concepts supported? Jim Hafner From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25589; 27 Sep 2000 1:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA23926 for ips-outgoing; Wed, 27 Sep 2000 00:17:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA23917 for ; Wed, 27 Sep 2000 00:17:37 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8R4HaU17557 for ; Tue, 26 Sep 2000 23:17:36 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA03051; Tue, 26 Sep 00 23:15:56 -0500 Message-Id: <10009270415.AA03051@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) In-Reply-To: Message from csapuntz@csapuntz-u1.cisco.com of "22 Sep 2000 14:15:06 MST." Date: Tue, 26 Sep 2000 18:16:12 -1000 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Currently, iSCSI is spec'ed to recover from transport-layer > connection failures. > > The main motivation for this decision was to support tape backup > applications that are quite sensitive to any failures that get > propogated to their layer. > > So, perhaps we can remove the requirement of recovering from > transport-layer connection failures in iSCSI. This would simplify > the protocol somewhat. > > Thoughts? I'm all for eliminating command recovery. There seem to be several reasons advanced for command recovery. The first seems to be based upon an inappropriate analogy to FCP. Command recovery had to be added to FCP-2 because the FC layer is unreliable. A single dropped FC frame leads to a failed FCP command. This clearly upsets tape operation even when the link is performing nominally. In FCP, without command recovery, with some observable frequency, you will get an expected error that leads to complete, irrecoverable failure of a transfer stream. The other thing that makes FCP-2 command recovery work well is when you are doing a write, which is 90% (maybe it's 99%?) of tape operation, the target can return an early indication of most frame drops, rather than waiting for a timer to expire. TCP's reliability solves this problem in another way. By the time you get a TCP connection failure, you have already exhausted a set of reliability mechanisms which guarantee, with high certainty, that further data can not be transferred between the two endpoints. `the two endpoints' phrase suggests the other reason advanced for command recovery. That is, to permit path failover for commands which are not idempotent, such as tape write sequential. The problem with this, is that it is not clear HOW iSCSI command recovery can actually work properly, given a TCP connection failure indication. It takes a long time for a TCP connection to fail, and by that time, I'm not sure recovery would reasonably be possible. Perhaps I'm in error on this assumption. Can a tape guru (Joe from Exabyte?) comment on whether recovery would be possible after many seconds (tens, hundreds) have elapsed? The SCSI layer has never been solely responsible for ensuring reliable backup. Macro scale things go wrong with tape (run off the end, get eaten, etc..) with relatively high frequency. A low level backup engine like tar or dump will fail on a SCSI error, and that's OK. There must also be a higher level software component like Amanda, which manages retries, including operator intervention, to ensure reliable backup. It seems like whether iSCSI has a command recovery mechanism should be a function of whether somebody can stand up and say for sure that it solves a real problem. So far it only seems like it MIGHT solve a problem. Who can say `this solves MY problem!'? Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19877; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA07438 for ips-outgoing; Mon, 25 Sep 2000 14:04:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA07432 for ; Mon, 25 Sep 2000 14:04:47 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id OAA25043; Mon, 25 Sep 2000 14:04:45 -0400 Message-ID: <00fb01c02718$645da150$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: , Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Mon, 25 Sep 2000 13:45:27 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is not baked enough to be an Internet-Draft, however the following URL is available in the mean time. ftp:\\coke.giganet.com\rdma-tcp.txt I thought it might be short enough to get away with sending in email. Evidently not, so I apologize. >> In response to this I would offer the following proposal with >> the caution that it is very preliminary and has not been >> analyzed or reviewed. But I thought it might be worth posting >> in order to see what the general response to this approach is. > >I would ask those who make these sorts of proposals to PLEASE >write them up as Internet-Draft and submit them to the I-D servers. >Outside the few weeks before an IETF meeting, the submission >turn-around is rapid, and having something like this written up as an >I-D avoids it vanishing into the mailing list archive where it's more >difficult to retrieve. > >Thanks, >--David From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27978; 27 Sep 2000 14:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA16572 for ips-outgoing; Wed, 27 Sep 2000 13:21:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (play-doh.Stanford.EDU [128.12.131.42]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA16568 for ; Wed, 27 Sep 2000 13:21:19 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 630EF8DC0; Wed, 27 Sep 2000 10:16:29 -0700 (PDT) To: "Jim Williams" Cc: Subject: Re: New List: rdma@cisco.com: to discuss RDMA References: <008e01c02892$d45d6280$cb08efd0@sachmo.giganet.com> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 27 Sep 2000 10:16:29 -0700 In-Reply-To: "Jim Williams"'s message of "Wed, 27 Sep 2000 10:54:21 -0400" Message-ID: Lines: 18 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk Does anybody on the list object to specifying an RDMA mechanism for use with iSCSI? Does anybody on the list object to mandating an RDMA mechanism? Please include your reasons. Current RDMA proposals: draft-csapuntz-tcprdma-00.txt draft-dicecco-vitcp-00.txt -Costa > Before going too far down this road, it is important to > understand if there is support for using an RDMA mechanism > as a basis for iSCSI. Will the next draft of the iSCSI > protocol actually be based on an RDMA mechanism that > would be defined? The answers to these questions and > the specific pros and cons need to be the driving > force behind the RDMA discussion. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32103; 28 Sep 2000 14:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA00742 for ips-outgoing; Thu, 28 Sep 2000 13:23:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA29903 for ; Thu, 28 Sep 2000 13:05:37 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 10:06:57 -0700 Message-ID: From: Robert Snively To: "'julian_satran@il.ibm.com'" , Douglas Otis Cc: dfsmith@almaden.ibm.com, ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Thu, 28 Sep 2000 10:04:55 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, I am sorry, but I disagree with you both practically and philosophically. Textual names are practically useless as identifiers, as witness the challenge of finding "Bob Smith" in an American phonebook. Every time you walk into a store where your name is kept in a data base, they ask for your phone number, not your name. Your passport, credit card, driver's license, and all other useful identifiers use a registered number set that allows for no (legal) duplication. SCSI has taken the same approach with LUN identification, recommending one of several registered numbers as the sole and immutable designation of the logical unit. Fibre Channel and Ethernet have taken the same approach at the Port/Node WWN and MAC address level. Use of a URL is a virtualization of the underlying name structure and should not be implemented except as a temporary convenience for the highest level client who is constrained to a very small search subset by his login authorizations. The virtualization is always at risk of security breaches. In the storage environment, the underlying structure is of vital interest. Any given data is present on specified physical and logical units and consistency of the data is maintained on the basis of an intermediate view of the lower level addressing structure which must also have the proper uniqueness guarantees. Address it with a URL on the browser if you wish, but understand that you are talking to a data block on a single storage unit attached to a network server having a specified IP address mapped to a registered MAC address. Bob > > There seem to be practical and philosophical issues that you > consistently > ignore. > Textual names are more stable than underlying network > addresses and thus > better > for applications (I assume that your card starts with your > name not phone > number and > that the former has changed less than the later). > I the iSCSI realm - and that is the only one under scrutiny > - a pertinent > observation was > made by David Black a century ago during our (too) long > naming discussions > - that a scheme that will allow the third party commands to > make the name > to address conversion > at use time is better than one that requires this mapping to > be made ahead > of time. > Unfortunately that is not yet possible with SCSI but we are > informed that > it soon be. > And for authentication the textual address in login is yet > another mean to > prevent > errors and/or unauthorized access if the target chooses to use it. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32525; 28 Sep 2000 16:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA07526 for ips-outgoing; Thu, 28 Sep 2000 16:03:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA05636 for ; Thu, 28 Sep 2000 15:24:28 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA28760 for ; Thu, 28 Sep 2000 21:23:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id VAA83576 for ; Thu, 28 Sep 2000 21:23:51 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256968.006A8AB8 ; Thu, 28 Sep 2000 21:23:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 28 Sep 2000 22:20:46 +0300 Subject: Re: LU access through an iSCSI session Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Randal, It is NOT an attempt to get around congestion control but to get more bandwidth and availability. And congestion control is good enough even in a world in which you can't possible control what a connect PC will do (open one or a zillion of connections). Julo Randall Stewart on 20/09/2000 14:55:23 Please respond to Randall Stewart To: Matt Wakeley cc: IPS Reflector (bcc: Julian Satran/Haifa/IBM) Subject: Re: LU access through an iSCSI session Matt Wakeley wrote: > .....snip....snip.... > Again, the *only* answer is "Yes". The multiple TCP connections make up a > *single* session. The session just happens to have lots of "TCP connections" > to make it look like it has a "big pipe" (for example, 4 1Gbit connections can > be used to emulate a 4Gbit link). > > If you want different "LU views" per TCP connection, then each TCP connection > would be a separate, independent iscsi session. ....snip....snip..... The above talk of making "lots of TCP connections look like a big pipe" sounds once again like attempting to get around congestion control in the network... I know we had this conversation before.. but what is stated above strongly implies this .... and I don't think this is the behavior one would want on the Big I-internet. I can see a private network doing this ... but I don't see how it could be supported in a standard manner that can be used on the Iternet... R From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26221; 27 Sep 2000 4:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA28213 for ips-outgoing; Wed, 27 Sep 2000 03:14:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA28203 for ; Wed, 27 Sep 2000 03:14:19 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA90326 for ; Wed, 27 Sep 2000 09:13:47 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA102096 for ; Wed, 27 Sep 2000 09:13:45 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256967.0027B521 ; Wed, 27 Sep 2000 09:13:42 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 27 Sep 2000 08:25:48 +0300 Subject: Re: iSCSI: Connection status summary corrections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Regarding Pierre Labat's proposal - although certainly interesting - it is another form of numbering (as Matt Wakeley has already pointed out). It will be hard to accept as any ordering scheme needs a limit to the number of yet unordered items you are going to keep. As such I would put it as another version of S. Regarding the requirement to have multiple connections or not I would not say that it precludes ONE connection for control and ONE for data as many other transport mechanisms (some buses, some protocols like FTP) use. Again I am somewhat biased for the PA but I also think that S is viable if PA is flawed. Julo Black_David@emc.com on 26/09/2000 21:19:07 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: iSCSI: Connection status summary corrections A couple of important corrections to Julian's summary: - There is long standing WG consensus that multiple TCP connections per session are optional. Therefore the Pure Asymmetric model must also support sending both commands and data over the same connection when only one TCP connection is in the session, and this needs to be added to the list of PA drawbacks. - The discussion of Asymmetric models overlooked Pierre Labat's proposal for spreading commands across multiple connections, but using a single command sequence in which the initiator tells the target which connection to expect the next command on. Since this provides the initiator with some ability to do load balancing, Balanced Asymmetric (BA) might be a good name. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Tuesday, September 26, 2000 12:48 PM > To: ips@ece.cmu.edu > Subject: Status summary on multiple connections > > > > Dear colleagues, > > I am attempting to summarize where we stand with regard to the multiple > connection issue > and to the two possible models - Symmetric (S) and Asymmetric (A). > > Many of us feel strongly that the multiple connection issue is central to > the whole design > and cannot be added as an afterthought. Moreover designing the hooks to > allow it later > will certainly already force us to make a decision. And both the hardware > and the software designers will be ill-served if we hand them a > half-backed solution. > > However this is not an invitation to reiterate positions that where > already stated. > If you feel that I have grossly misstated anything in this note please > write me and use > the mailing list only if my answer is not satisfying. > > And yes - like the chairman - I would like to make progress but I don't > see any way to > do it without satisfactorily closing this issue. > > The reasons for multiple connections where discussed at some length and > where very nicely summarized in a series of notes by Michael Krause > (beginning of August). > > The core reasons for having multiple connections where the need for more > bandwidth and availability than a single link can supply with a level of > complexity affordable for simple > installations and with a traffic engineering and management clearly > separated from the transport users (SCSI). > > The session is embodying this requirement. > > The only major objections I have heard against this where those requiring > to go all > the way in having one TCP connection/LU - and after a short debate this > objection > was practically removed. > > The other objections we heard where that this is basically a transport > issue and it should be solved at transport level. > That might be true - but since many, if not most, of TCP applications do > not have this > requirement it is highly unlikely that TCP is going to do connection > trunking in the foreseeable > future. > > iSCSI can be designed to use multiple TCP connections in one of two ways: > > -Asymmetric - one TCP flow only carries commands the others carry only > data > > -Symmetric - every flow carries both commands and their associated data > > The S version is designed in the I-D version 01 > The S version requires an command ordering scheme and that is provided by > a > command counter and a sliding window scheme. It was argued that ordering > needs might be more prevalent than usually thought and a good conservative > design should preserve ordering. > Ordering-per-LU (as it is designed in FCP-2 version 4) was considered > impractical as it > required initiators to maintain state for each LU - while the rest of the > design required initiators only to maintain state for outstanding > commands. > Several comments on this list suggested that this windowing mechanism > could > also be used as a command-flow-control mechanism and that is a "bonus" of > the scheme. > > The A version - comes in two flawors: > > - pure A (PA) in which ONLY commands flow on one TCP connection while > data flow > on DIFFERENT connections (with only one data connection being selected > for a > command); this scheme requires a minimum of 2 TCP connections > although not > necessarily on different physical links. > > - collapsed A (CA) in which commands flow one a single TCP connection > while data > flow on ANY connection; this scheme requires as a minimum 1 TCP > connection. > > Here is first attempt to list the benefits and drawbacks of all of them: > > > - S - benefits > - well understood > - simple hardware setup and/or TCP API activation > - window mechanism can be also used for flow control > - the minimum required is a single TCP connection > > - drawbacks > - need to maintain a window mechanism > - a multiplexing mechanism has to be carefully crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > - PA - benefits > - TCP will both order the commands and provide for flow control > through the TCP window mechanism > - the multiplexing mechanism is simpler as closing TCP windows > will > never affect command flow > - data flow can use a streamlined header (and processing) > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > > - CA - benefits > - TCP will order the commands > - needs a single TCP connection as a minimum > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > - if command flow control is required counters and the sliding > window mechanism > are required > - a multiplexing mechanism has to be carefully crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > From the above it should be apparent - as was already pointed out by Matt > Wakeley - > that the CA inherits the drawbacks of S and PA and as such the choice is > really > between S and PA. > > Julo > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20741; 25 Sep 2000 20:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA19410 for ips-outgoing; Mon, 25 Sep 2000 18:45:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA19406 for ; Mon, 25 Sep 2000 18:45:27 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 18:44:56 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FE6@corpmx9.isus.emc.com> To: dotis@sanlight.net, Black_David@emc.com, ycheng@advansys.com, ips@ece.cmu.edu Subject: RE: Last Word on An IPS Transport Protocol? Date: Mon, 25 Sep 2000 18:44:54 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Yes, SCTP is a suitable protocol *because* it complies with RFC 2581; I thought this was clear from prior discussion on the list. --David > -----Original Message----- > From: Douglas Otis [SMTP:dotis@sanlight.net] > Sent: Monday, September 25, 2000 5:53 PM > To: Black_David@emc.com; ycheng@advansys.com; ips@ece.cmu.edu > Subject: RE: Last Word on An IPS Transport Protocol? > > David, > > Some have interpreted your comment to infer SCTP is not a suitable > protocol > for discussion. As SCTP does include TCP like congestion control (RFC > 2581 > compliance), could you clarify SCTP as a suitable protocol of merit. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Black_David@emc.com > > Sent: Monday, September 25, 2000 11:10 AM > > To: ycheng@advansys.com; Black_David@emc.com; ips@ece.cmu.edu > > Subject: RE: Last Word on An IPS Transport Protocol? > > > > > > > My contention is the current TCP congestion control is NOT good enough > and > > > the ACK traffic on a network with long latency delay is BAD. We must > have > > > streamed transfer on a network with long latency. Therefore, defining > the > > > ACK of TCP is critical. The TCP header format is not sacred to me. > > > > It's time to put my WG co-chair hat on and play "bad cop" ... > > > > There are experimental and production results indicating that TCP is > > capable of saturating arbitrarily high bandwidth networks with > arbitrarily > > long delays. Buffering proportional to the bandwidth-delay product is > > a good idea, so this doesn't come for free. Streaming transfer > > can be achieved without playing these sort of ACK games - of course > > if congestion is encountered, TCP backs off dramatically. > > > > This WG does not have the license to fundamentally change TCP's > > congestion control algorithm or to use a transport that does not > implement > > congestion control in a sufficiently TCP-like manner (RFC 2581 > compliance > > is sufficient); the co-chairs and ADs will reject any document that > > tries to do either of these things. Please don't consume list bandwidth > > in further discussion of this. > > > > Developing a new transport with sufficient congestion control is going > to > > take time. If the WG were to go in this direction, at least a year > should > > be added to all of the completion milestones in the charter. > > > > --David > > > > --------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 42 South St., Hopkinton, MA 01748 > > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > -------------------------------------------------- > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20153; 25 Sep 2000 16:34 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA09914 for ips-outgoing; Mon, 25 Sep 2000 15:02:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA09899 for ; Mon, 25 Sep 2000 15:02:39 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 15:02:08 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FDF@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: Internet Drafts and TCP RDMA Date: Mon, 25 Sep 2000 15:02:05 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > This is not baked enough to be an Internet-Draft Internet-Drafts do not need to be fully-baked or even half-baked. They're working documents and some of their value is that they're easy to refer to and find. I've written or contributed to some Internet-Drafts that were significantly lacking in the baking department (including at least one idea that was fundamentally wrong in 20/20 hindsight); an Internet-Draft is a good mechanism to get ideas out to see if they require more baking or are digging in the proverbial "wrong place". Working Group drafts (names of the form draft-ietf--...) are expected to reflect some level of WG consensus, and hence be somewhat baked, but there's no corresponding expectation for individual submissions, and especially not for -00 versions of individual submissions. As an example of the utility of Internet-Drafts, the TCP RDMA non-draft has some similarities to: draft-csapuntz-tcprdma-00.txt Now, isn't that easier to find than a months-old email message in the archives? In addition, there's a Cisco intellectual property statement covering that draft on the IETF intellectual property rights page: http://www.ietf.org/ipr.html Please take the time to write up substantial technical proposals as Internet-Drafts. I hope Jim isn't offended or annoyed by this, as he's not the only person ... there are a bunch of other things in the archives that should have been Internet-Drafts. This gets more important as the community on this list expands because the list archives are already to the point where I would not expect newcomers to review them in their entirety. Doug Otis and Costa Sapuntzakis get credit for following this exhortation in the past ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22707; 26 Sep 2000 7:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id GAA05512 for ips-outgoing; Tue, 26 Sep 2000 06:05:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA05508 for ; Tue, 26 Sep 2000 06:05:10 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id MAA100670; Tue, 26 Sep 2000 12:04:37 +0200 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id MAA110014; Tue, 26 Sep 2000 12:04:37 +0200 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256966.003752AF ; Tue, 26 Sep 2000 12:04:16 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: matt_wakeley@agilent.com, Black_David@emc.com, ips@ece.cmu.edu Message-ID: Date: Tue, 26 Sep 2000 11:50:17 +0300 Subject: Re: iSCSI: multiple connection process Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Only for correctness - the SAME design team has discussed both the symmetric and asymmetric model and choose the symmetric because it did not have a good enough asymmetric solution; that does not hold now anymore and that is why I opened this thread (should I say I am sorry?) I have two objections to the direction you have taken: - you considered objections that where voiced to you and not to the mailing list; the community should judge if they are core or marginal; I would appreciate if you could summarize them - the key issues with any of the two approaches are understood - an evaluation committee will only broaden the "discontent" Julo Black_David@emc.com on 25/09/2000 23:51:48 Please respond to Black_David@emc.com To: matt_wakeley@agilent.com, Black_David@emc.com, ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: iSCSI: multiple connection process > Exactly. There has been NO consensus that multi-connection sessions should be > removed from the main specification (only your proposal to do that). Just > because we can't (yet) agree on something, does not mean "let's not do it at > all, or put it off until later." Sorry, lack of rough consensus means one of those two. Documents don't leave the WG without rough consensus, so either the document gets delayed for consensus, or the feature gets removed. Needless to say, we don't have rough consensus at the moment, nor do I see much in the way of common ground emerging among the different points of view. > We already had the offline design team and we came up with something that > everyone is arguing about. So let's have some more discussion on what it is > that people don't like, and what the requirements are, so that another offline > design team has something to work with. Actually, offline design teams came up with both the Asymmetric and Symmetric models ... and now there's a third. It appears to me that continued list discussion is an active version of "put it off until later" rather than a passive one. I could be wrong, but someone will need to explain what will be different about continued list discussion that will lead to a consensus in a way that discussion since Pittsburgh has not. The alternative process I have in mind is more of an evaluation team than a design team - form a team of people who publish evaluation criteria for multiple-connection sessions, evaluate drafts describing the various models against those criteria, and make a recommendation to the WG. One possible way to make continued progress on the list is to discuss what those evaluation criteria ought to be rather than asking an off-line team to write them. Doing this really requires someone to track the criteria as they evolve - a volunteer for this task is hereby solicited, preferably someone who has not taken a position on this session issue. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32283; 28 Sep 2000 15:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA03851 for ips-outgoing; Thu, 28 Sep 2000 14:41:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA03840 for ; Thu, 28 Sep 2000 14:41:03 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA18596 for ; Thu, 28 Sep 2000 11:41:02 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.188.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA04563 for ; Thu, 28 Sep 2000 11:40:58 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA10602; Thu, 28 Sep 2000 11:40:58 -0700 (PDT) Message-Id: <200009281840.LAA10602@ha10nwk.EBay.Sun.COM> Date: Thu, 28 Sep 2000 11:40:57 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Status summary on multiple connections To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: Mjj4K43tocck6Y8ZlmY6WQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk ------------- Begin Forwarded Message ------------- From: Robert Snively To: "'David Robinson'" , Robert Snively Subject: RE: Status summary on multiple connections Date: Thu, 28 Sep 2000 09:36:50 -0700 Dave, You ask some interesting questions with non-short answers. If you would like, and if you think it useful, you may post this out to the ips reflector. > > The single connection alternative allows a simplistic ordering > > structure, a simple recovery mechanism, and does not require > > state sharing among multiple NICs. It allows bandwidth aggregation > > across any set of boundaries that is required. Because command > > queuing is the rule among high performance SCSI environments, > > latency appears only as an increment in host buffer requirements > > except during writes that perform a commit function. > Those traditionally > > have been taken out of the performance path by using local > > non-volatile RAM to perform the commit functions, using slower > > high latency writes with less strict ordering requirements relative > > to reads to actually perform the write to media. > > Can you clarify something for me, in my previous questions on > flow control it was strongly indicated that the target must drain > the stream in order to allow commands to flow when the command queue > filled up. You seem to indicate here that command queuing and the > flow control needed to handle overflow is being done at the > SCSI layer and not the transport. Is it correct that it really > is a command level function and not a transport function? Without > considering the TCP window management, SCSI will cause > command flow to > stop when the queue fills up? If this is true then most of > the arguments for at least two connections are no relivent. You > may need to still discard commands if the target over advertises its > total queue space but that seems to be more of an implementation > bug. SCSI manages two independent sets of resources. One is the resources required to receive and process command states. The other is the resources required to buffer and process data to be transferred as a result of processing the commands. At the initiator, all resources for the execution of a command, including both the command state resources and the explicitly specified buffer area, are defined at the time the command is delivered to the SCSI stack. Those resources are locked down until the SCSI command is finished, at which time the command state resources (by this time a response packet) and control of the buffer are passed back to the application client (user, driver, operating system, file system, application program or whatever). The beauty of SCSI is that all the transfer management is done by the target (which knows exactly what is going on and exactly what is needed), not by the initiator. The target also has two sets of resources. The command is received into a command buffer (perhaps implemented as a large single buffer or perhaps implemented as a large number of smaller buffers at each logical unit). The rules on the command queueing I explained before, but basically all commands are posted into the same buffer from whatever initiator they were received from with some kind of time/order stamp. A well-behaved device is always capable of receiving at least one simple or ordered queued command and one head of queue command for each logical unit/initiator nexus that is supported by the device. Present devices support from 16 to 64 initiators per logical unit. Typically at least one additional slot is available for task management functions. The remaining locations for commands in the queue are dynamically portioned out to whatever commands come in, regardless of initiator or logical unit. When there is no more dynamic space left and all the pre-allocated locations for a particular ITL nexus are also full, the next command gets a queue full indication returned. Because of the dynamic assignment area, this will typically be rare in a properly configured system. The initiator then resends the command and all subsequent commands after at least one command comes back completed, indicating that at least one (and probably a whole stack more) slots are again available. Note that there is a possibility that commands that are inflight and have ordering constraints may be accepted out of order, a question that has caused lots of agonizing, but is apparently reasonably well managed by most file systems today by the selective use of ordering only for blocking boundaries of a particular logical stream of commands. The target then begins sorting commands for optimum execution order, to exploit pre-buffered data, and to perform any coalescence of streaming operations to the device and begins to execute the commands in ITS desired order, modified by the ordered queueing restrictions, if any. If data is required from the initiator, buffers are set aside for the data in the target and the data (already locked down in the initiator buffers) is requested from the initiator. If data is to be sent to the initiator, it is assembled in the target buffers and shipped off to the specified buffers in the initiator. In large storage subsystems, this is typically going on for multiple initiators and in both directions at the same time. The initiator buffers are identified by the command context in the initiator. The command context is selected by the Initiator/Target/Logical unit/Queue Tag (ITLQ) nexus carried from the target with the data or the data request. As a result, SCSI, independent of transport (IEEE 1394, Parallel SCSI, FC, and I hope iSCSI) has complete flow control at the initiator and at the target with respect to all command and data transfers. SCSI, being a storage protocol and having bursty traffic characteristics, is traditionally configured such that over-subscription is of short duration and has little effect on average latencies except for very short periods. Of course, underconfigured SCSI transports will increase latency, but they will still be well-behaved in terms of throughput and they will not block. Depending on the particular transport implementation, they may be more or less well-behaved in terms of IT nexus fairness. As an example, older parallel SCSI implementations may exhibit higher throughput on high priority IT nexi than on low priority IT nexi. However, if additional flow controls or congestion management exist in the transport layer, they can interact in some pathological ways with the basic SCSI function. I believe it is possible that such structures could create head of queue blocking or throttling behaviors in the transport switches if those mechanisms are not implemented properly. Note that this is 100% outside the scope of the SCSI behaviors. I believe that the conclusions in your note are well founded. ------------- End Forwarded Message ------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27825; 27 Sep 2000 13:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA14926 for ips-outgoing; Wed, 27 Sep 2000 12:41:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA14917 for ; Wed, 27 Sep 2000 12:41:08 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 27 Sep 2000 09:42:49 -0700 Message-ID: From: Charles Monia To: John Hufferd/San Jose/IBM Cc: "Ips (E-mail)" Subject: RE: iSCSI: Session Partial Resolution Date: Wed, 27 Sep 2000 09:42:40 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] > Sent: Tuesday, September 26, 2000 8:48 PM > To: IPS (E-mail) > Subject: RE: iSCSI: Session Partial Resolution > > > > Charles, > I thought we were all together until you said, ...have the > initiator budget > one "credit" to be used for this purpose. > Up until that statement there had been no mention of Credit, > and I am not > sure why we need to bring that concept in. > Hi John: I was using the notion of budgeting one "credit" as shorthand for " budgeting enough resources to perform one task managment operation". Sorry about the confusion. Charles From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28217; 27 Sep 2000 15:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA18990 for ips-outgoing; Wed, 27 Sep 2000 14:14:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA18983 for ; Wed, 27 Sep 2000 14:14:41 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 27 Sep 2000 11:16:33 -0700 Message-ID: From: Charles Monia To: Stephen Bailey , ips@ece.cmu.edu Subject: RE: iSCSI: Session Partial Resolution Date: Wed, 27 Sep 2000 11:16:30 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > Sent: Tuesday, September 26, 2000 9:08 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Session Partial Resolution > > > > In general, such requests are "think-time" limited and therefore non > > blocking, so this seems not to be a problem in practice. > > The abort task operation can be pretty heavyweight. It frequently > involves a lot of non-nominal, antioptimized (slowed down so something > else can be fast) behavior. > > That said, I'm not sure why it matters whether task management is fast > or slow. > Yes -- you're right, speed is not the issue. The point I should have been making is that the one-at-a-time rule for task management functions is ok because such requests are relatively infrequent and finish within some "reasonably bounded time". Charles From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20384; 25 Sep 2000 18:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA15407 for ips-outgoing; Mon, 25 Sep 2000 17:02:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hqmail3.Crossroads.com ([63.237.99.226]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id RAA15401 for ; Mon, 25 Sep 2000 17:02:43 -0400 (EDT) Received: by hqmail3.Crossroads.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 15:55:25 -0500 Message-ID: <670C86291BFCD311AFE500508B94D4B6630CB6@hqmail1.crossroads.com> From: Robert Reynolds To: "'ips@ece.cmu.edu'" Subject: RE: ISCSI: flow control Date: Mon, 25 Sep 2000 16:04:00 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Would this credit be on an LU<->Initiator pair? If not, then the LU could advertise a credit of 10 to Initiator A. Initiator A would then send 10 commands without any knowledge that some other initiator may have also sent commands to that same LU. The end result is that the targets command queue is still overrun. If the credit is on an LU<->Initiator pair then how does the target divide up it's resources in a multi-initiator environment where a new initiator could come on-line at any time? > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Sunday, September 24, 2000 10:51 PM > To: Pierre Labat; Robert Snively > Cc: ips@ece.cmu.edu > Subject: Re: ISCSI: flow control > > > At 10:53 AM 9/24/00 -0700, Pierre Labat wrote: > > > >T10 could add an advertising of a command credit. For example > >when the status is returned the target gives a command credit. > > In essence, this is what InfiniBand does and others have been > advocating. When the ACK (SCSI response) is returned it > encodes a credit > to inform the sender of how many receives buffers (available > command queue > slots) have been posted. This allows the sender to know > whether it can > continue to issue send operations. The cost is trivial in > hardware and > greatly simplifies the software / control logic since it can > post as many > commands as it wants on the requester and these will be > processed at the > rate the receiver can actually consume them. When combined > with the RNR > NAK operation (still applicable since there may be other reasons for > returning this), one can achieve a fairly robust and simple > implementation > that works across a wide range of products. > > Mike > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31368; 28 Sep 2000 9:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA19701 for ips-outgoing; Thu, 28 Sep 2000 08:17:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA19696 for ; Thu, 28 Sep 2000 08:16:58 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id HAA16214; Thu, 28 Sep 2000 07:16:40 -0500 Message-ID: <39D336A8.D007675E@stewart.chicago.il.us> Date: Thu, 28 Sep 2000 07:16:40 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis CC: Charles Monia , csapuntz@cisco.com, Jim Williams , ips@ece.cmu.edu Subject: Re: New List: rdma@cisco.com: to discuss RDMA References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas: I may be missing something here, but sendmsg() and recvmsg() both have scatter/gather arrays within the berkley sockets api. I am not sure that one can uses sendmsg() recvmsg() with TCP.. I have never tried. As far as SCTP goes, we have defined a sockets mapping... the draft is released but real rough.. an next version is in the works.... the current "rough" cut can be found at: http://search.ietf.org/internet-drafts/draft-stewart-sctpsocket-sigtran-00.txt Now it may be that I am missing a key point in this discussion.. were you thinking other than the iovec structure... or were you thinking a modification that allows the pages to be stolen with no copy... the no copy becomes a implemenation issue.. I have done something like it in the past where if the application gives a 4k page in the send, the O/S underneath does a swap of a page from kernel space with the user space one... But as I say, this is real implemenation dependant :) R Douglas Otis wrote: > > Charles, > > With respect to SCTP, features added by a TCP option for RDMA are not needed > to support alignment and out of sequence processing that ultimately alters > the TCP API. SCTP adds these features without disruption or modification to > TCP. The intent of VI is to allow scatter/gather function to handled by the > target. A safer scheme would be to adhere to SCSI conventions and implement > zero copy and out of sequence processing using SCTP and SAM structures > related to locally pre-arranged transfer structures. This would keep the > initiator or client in intimate control of memory and not reliant on targets > eliminating boundary checking. Such a feature will slightly impact SCTP to > add a means to generally encapsulate a data payload associated with > structure of pointers equipped with scatter-gather lists and relative > offsets. Specifically, this could be defined as the FCP structure data > structure in SCSI implementations. I would whole-heartedly endorse such > features within SCTP. I would dissuade such features from being added to > TCP. One could view the scatter-gather list structure as a token to be used > by the target in more conventional VI methods. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Charles Monia > > Sent: Wednesday, September 27, 2000 11:49 AM > > To: csapuntz@cisco.com; Jim Williams > > Cc: ips@ece.cmu.edu > > Subject: RE: New List: rdma@cisco.com: to discuss RDMA > > > > > > > > > > > -----Original Message----- > > > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > > > Sent: Wednesday, September 27, 2000 10:16 AM > > > To: Jim Williams > > > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > > > Subject: Re: New List: rdma@cisco.com: to discuss RDMA > > > > > > > > > > > > Does anybody on the list object to specifying an RDMA mechanism for > > > use with iSCSI? Does anybody on the list object to mandating an RDMA > > > mechanism? Please include your reasons. > > > > > > > I'd object to mandationg the use of RDMA in iSCSI. However, I > > would support > > structuring the spec so that an RDMA transport mechanism could be used > > underneath (I guess that's motherhood). If the iSCSI folks decided this > > wasn't a priority issue, that's ok with me too. > > > > Charles > > -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24010; 26 Sep 2000 15:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA03815 for ips-outgoing; Tue, 26 Sep 2000 14:52:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA03803 for ; Tue, 26 Sep 2000 14:51:53 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA13843; Tue, 26 Sep 2000 11:39:03 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA29600; Tue, 26 Sep 2000 11:36:48 -0700 From: "Douglas Otis" To: "Jim Williams" , Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Tue, 26 Sep 2000 11:54:16 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: <019d01c027db$304d3060$cb08efd0@sachmo.giganet.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, In summary, you expect kernel modifications of TCP API to allow frame alignment, a hardware based 32-bit CRC that uses a different polynomial than FCP and a direct copy method at the NIC. As SCSI does not benefit from VI with zero copy already possible with frame alignment, the area where there could be commonality has been missed with the choice of CRC. Rather than modifying thousands of TCP implementations, it would seem appropriate to leave TCP as is and concentrate on providing these improved features using SCTP designed to handle these requirements together with additional features yet to be resolved with TCP. You need not convince systems to change TCP to benefit from rather important features found with SCTP. Do not encumber TCP with disruptive options. SCTP can emerge using UDP and become supported without making these changes to TCP that you advocate as evolutionary. Once you implement frame alignment, the API for TCP becomes unusable as it does not afford means of discerning frames. SCTP implementations already address these problems and so I do not see your approach as evolutionary but rather revolutionary. You do not get my vote on this approach. Doug > -----Original Message----- > From: Jim Williams [mailto:jimw@giganet.com] > Sent: Tuesday, September 26, 2000 10:00 AM > To: Douglas Otis; ips@ece.cmu.edu > Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in > iSCSI)) > > > > -----Original Message----- > From: Douglas Otis > To: Jim Williams ; ips@ece.cmu.edu > Date: Monday, September 25, 2000 5:56 PM > Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) > > > >You suggest a frame aligned formatting scheme is possible using an > >unmodified TCP. > > Yes. > > >By allowing out of sequence delivery of TCP, are you opening the > window for > >spoofing? One need not guess the sequence, just get in front of > it a bit. > >After that, you will never recover state. > > A 64 bit randomly selected connection ID should prevent blind spoofing. > For an attacker who snoops valid packets and then spoofs, we would > be vulnerable. However no window is opened, just an already open window > failed to close tight. > > > Do you expect the RDMA option to be universally supported, and if not? > > The proposal is not an option, it is a format. The format could > potentially > be shared by multiple protocols. It is by no means universal; it > applies only to those protocols that adopt it. > > >Do you see this level of CRC done in software? > > I see it being done in hardware in a NIC by anyone who cares about > performance. That's largely the case for TCP checksums now. Non > performance critical applications could do it in software. > > >> It is > >> RECOMMENDED that each TCP segment contain exactly one RDMA segment. > > > >How do you achieve this recommendation? > > With cooperation of the TCP implementation. Without that, it cannot > in general be guaranteed. Typically for performance critical > applications, NICs will handle both TCP and RDMA protocol processing, > and in this case it is easy. > > >Sending 4GB exchanges, you still have queue blocking if not head of queue > >blocking. > > Yes. If that's a problem, don't send 4GB messages. All I am saying > is that the format allows 4GB messages. > > > >Can one suggest frame alignment within TCP? > > Certainly one can suggest it. Will the suggestion be supported? > Does it violate any rules that can't be changed? Perhaps I > will find out. > > >Have you investigated the use > >of SCTP? SCTP does solve all the related problems without violating > >protocol or protocol API. > > SCTP is an interesting protocol with a lot of merrit and a lot of > good ideas. But my guess is that the market wants one transport > protocol and wants it to be TCP. My vote would be to stick with > TCP and evolve it in a way that meets the requirements of the > market and is fully compatable with the existing infrastructure. > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26223; 27 Sep 2000 4:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA28214 for ips-outgoing; Wed, 27 Sep 2000 03:14:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA28209 for ; Wed, 27 Sep 2000 03:14:25 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id JAA62672 for ; Wed, 27 Sep 2000 09:13:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA88516 for ; Wed, 27 Sep 2000 09:13:51 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256967.0027B76A ; Wed, 27 Sep 2000 09:13:48 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 27 Sep 2000 08:43:36 +0300 Subject: Re: iSCSI: Behaviors Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk JP, We will try to fix it in the next draft. Thanks, Julo Raghavendra Rao on 14/09/2000 21:13:18 Please respond to Raghavendra Rao To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI: Behaviors It would be great if the draft can clearly point to a SCSI document (appropriate version) for greater details where little is mentioned in the draft, particularly with regard to target and LUN resets. 1. How are the outstanding commands affected for the initiator that performed the reset ? 2. What is the action on the newly (just) received commands ? 3. How are the outstanding/new commands sent from a different initiator (other than the one that performed the reset) affected ? 4. Will devices such as Tapes rewind after reset ? (They currently do). Also, What is the reason for not generating a LUN reset asynchronous notification from the targets ? (it is only done for target reset) Thanks -JP From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24195; 26 Sep 2000 16:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA07315 for ips-outgoing; Tue, 26 Sep 2000 16:05:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA07301 for ; Tue, 26 Sep 2000 16:04:55 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA13885; Tue, 26 Sep 2000 12:52:08 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA29812; Tue, 26 Sep 2000 12:49:53 -0700 From: "Douglas Otis" To: "Jim Williams" , Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Tue, 26 Sep 2000 13:07:22 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Importance: Normal In-Reply-To: <01ea01c027ee$4b0f0870$cb08efd0@sachmo.giganet.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, Manage frame based data using a byte stream, especially when out of sequence processing is permitted? The API must allow for this type of data handling foreign to TCP. As such, this proposal changes the API to allow these modifications with respect to both send and receive regardless how the proposal is worded. Both FDDI and FC have selected a CRC polynomial ignored by your proposal. Recommending, suggesting, or offering a feature of frame alignment and out of sequence processing rend the TCP API. There are better alternatives to your proposal in SCTP, so I fail to see the justification. Doug > -----Original Message----- > From: Jim Williams [mailto:jimw@giganet.com] > Sent: Tuesday, September 26, 2000 12:17 PM > To: Douglas Otis; ips@ece.cmu.edu > Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in > iSCSI)) > > > The starting assumptions for the proposal were that the transport > must be TCP, and TCP could not be modified in any way. If you > don't buy the starting assumptions, it may be a waste > of time to discuss the specifics of the proposal. > > NO changes or new options to TCP are proposed. > Frame alignment is not a requirement, > only a recommendation, so no existing TCP APIs and kernel > implementations need to change. (Perhaps I sould soften > "recommendation" and make it "possible feature to be > considered by implementators when convenient".) > > The CRC is certainly open for discussion. It looks > to me like I can do a software CRC in 3 risc instructions > per byte of data using a moderately sized table lookup > (less that 32KB). Is a CRC a good thing or a bad thing? > Should it be optional or required? > > -----Original Message----- > From: Douglas Otis > To: Jim Williams ; ips@ece.cmu.edu > Date: Tuesday, September 26, 2000 2:52 PM > Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) > > > >Jim, > > > >In summary, you expect kernel modifications of TCP API to allow frame > >alignment, a hardware based 32-bit CRC that uses a different polynomial > than > >FCP and a direct copy method at the NIC. As SCSI does not > benefit from VI > >with zero copy already possible with frame alignment, the area > where there > >could be commonality has been missed with the choice of CRC. Rather than > >modifying thousands of TCP implementations, it would seem appropriate to > >leave TCP as is and concentrate on providing these improved > features using > >SCTP designed to handle these requirements together with additional > features > >yet to be resolved with TCP. You need not convince systems to change TCP > to > >benefit from rather important features found with SCTP. Do not encumber > TCP > >with disruptive options. SCTP can emerge using UDP and become supported > >without making these changes to TCP that you advocate as evolutionary. > Once > >you implement frame alignment, the API for TCP becomes unusable > as it does > >not afford means of discerning frames. SCTP implementations already > address > >these problems and so I do not see your approach as evolutionary > but rather > >revolutionary. You do not get my vote on this approach. > > > >Doug > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20043; 25 Sep 2000 15:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09392 for ips-outgoing; Mon, 25 Sep 2000 14:49:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA09385 for ; Mon, 25 Sep 2000 14:49:22 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA13340; Mon, 25 Sep 2000 11:35:56 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA25886; Mon, 25 Sep 2000 11:33:39 -0700 From: "Douglas Otis" To: "Ips" Cc: "Elizabeth G Rodriguez \(Elizabeth\)" , "fc" , "T10" Subject: Revised Fibre-Channel over SCTP/IP Date: Mon, 25 Sep 2000 11:50:55 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk All, The IPS (Internet Protocol Storage) Work Group is officially under IETF Transport Area. Included are changes to illustrate a strategy involved in allowing both native IP access and FC encapsulation. In addition, a credit token scheme similar to FC Class 3 has been implemented. The disscussion reflector is at ips@ece.cmu.edu to subscribe send email to majordomo@ece.cmu.edu and within the message body- subscribe ips the website is at http://www.ece.cmu.edu/~ips. http://www.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-01.txt Doug From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31523; 28 Sep 2000 10:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA22143 for ips-outgoing; Thu, 28 Sep 2000 09:39:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA22117 for ; Thu, 28 Sep 2000 09:39:00 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id PAA126258 for ; Thu, 28 Sep 2000 15:38:22 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id PAA109652 for ; Thu, 28 Sep 2000 15:38:22 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256968.004AE9C0 ; Thu, 28 Sep 2000 15:38:15 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 28 Sep 2000 16:30:58 +0300 Subject: twist on iSCSI asymmetric model Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk In the asymmetric model, one TCP connection (the control connection) is used for sending all of the commands, thus ensuring ordered delivery of the SCSI commands. If (a large amount of) data is also sent on the control connection, there is the possibility that the data would fill up the window, and critical task management requests couldn't get through. On the other hand, it has been pointed that some implementations will prefer a single (and only) TCP connection for both commands and data. I would like to suggest the possibility of combining these 2 cases; either (1) have a single connection for everything or (2) have one connection for control and one or more data connections. In this case, no (WRITE or READ) data may be sent over the control connection. The initiator and target negotiate during the first login as to which of the 2 possibilities they will use. In (1), it is the responsibility of the initiator to worry about not overloading the one and only connection so as not to delay delivery of critical task management messages. In (2), the iSCSI SCSI Command PDU specifies which data connection to use for the data of this command. On the data channels, where we will have iSCSI Data and R2T PDUs, we shouldn't need to use the long headers that we use to specify an iSCSI SCSI Command PDU. We will therefore have a shortened header for the data channels. In (1), since data will travel on the control channel, the iSCSI Data PDU and R2T headers will be padded with zeros so that all headers are of the same length on a particular connection. The use of R2T is negotiated during the first login of the iSCSI session. If R2T is to be used, then the initiator may not send data before it is requested by the target. Thus there will always be buffers ready to accept the data. If the target agrees to not use R2T, then the target implicitly agrees to have some number of buffers available to receive data sent by the initiator, even if the data arrives before the command arrives. In this case the initiator MUST send data in the data connections in the same order that the corresponding commands were sent. (This avoids deadlocks.) If the target runs out of buffers, then it will let data back up in the TCP window of the data connections, while it is processing the data from earlier commands. The only main drawback remaining with the asymmetric model is that 2 separate operations may be required to set up sending a command and to register its DMA buffers. If the control and data connections happen to be on the same NIC, then this might not be a problem. Only one interrupt is required upon completing a READ data transfer, if we mark the last iSCSI Data PDU of the data transfer. - Kalman Meth. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28206; 27 Sep 2000 15:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA18951 for ips-outgoing; Wed, 27 Sep 2000 14:13:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA18939 for ; Wed, 27 Sep 2000 14:13:34 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA14241; Wed, 27 Sep 2000 11:00:38 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA00716; Wed, 27 Sep 2000 10:58:20 -0700 From: "Douglas Otis" To: , , Subject: RE: iSCSI: Connection status summary corrections Date: Wed, 27 Sep 2000 11:15:59 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100FFC@corpmx9.isus.emc.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Has EtherChannel technology been investigated? As far as redundant paths, SCTP already provides this feature. What is missing? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > Sent: Wednesday, September 27, 2000 6:32 AM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: iSCSI: Connection status summary corrections > > > > Regarding the requirement to have multiple connections or not I > would not > say that it > > precludes ONE connection for control and ONE for data as many other > > transport mechanisms (some buses, some protocols like FTP) use. > > And as the person who called that consensus, I am informing Julian that he > is incorrect. I know that Matt Wakeley has a standing objection to this > consensus, and now Julian makes two (although I hope that's a > misunderstanding). > I continue to be amazed at the lack of ability to make progress on these > session issues - does anyone else want to reopen the issue of whether > more than one TCP connections will be REQUIRED in an iSCSI session? > > At the moment, continued discussion of sessions appears to be making > negative progress ... > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28703; 27 Sep 2000 18:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA28093 for ips-outgoing; Wed, 27 Sep 2000 17:17:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA28079 for ; Wed, 27 Sep 2000 17:17:21 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id A65D64A4 for ; Wed, 27 Sep 2000 15:17:12 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id C24E114A for ; Wed, 27 Sep 2000 17:17:11 -0400 (EDT) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.188]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA24204 for ; Wed, 27 Sep 2000 14:17:10 -0700 (PDT) Message-ID: <39D26405.2CD8DC85@agilent.com> Date: Wed, 27 Sep 2000 14:17:57 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: New List: rdma@cisco.com: to discuss RDMA References: <008e01c02892$d45d6280$cb08efd0@sachmo.giganet.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I object to mandating iSCSI use an RDMA option because: - (main reason) there isn't any standardized mechanism now, and I don't want to hold up iSCSI while one crawls through the standards process. - I don't think RDMA is needed. FCP doesn't use it, and it works great with the FC protocol chips that "accelerate" FCP. -Matt Wakeley csapuntz@cisco.com wrote: > Does anybody on the list object to specifying an RDMA mechanism for > use with iSCSI? Does anybody on the list object to mandating an RDMA > mechanism? Please include your reasons. > > Current RDMA proposals: > draft-csapuntz-tcprdma-00.txt > draft-dicecco-vitcp-00.txt > > -Costa > > > Before going too far down this road, it is important to > > understand if there is support for using an RDMA mechanism > > as a basis for iSCSI. Will the next draft of the iSCSI > > protocol actually be based on an RDMA mechanism that > > would be defined? The answers to these questions and > > the specific pros and cons need to be the driving > > force behind the RDMA discussion. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27635; 27 Sep 2000 12:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA11092 for ips-outgoing; Wed, 27 Sep 2000 11:14:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA11083 for ; Wed, 27 Sep 2000 11:14:10 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id LAA04232; Wed, 27 Sep 2000 11:13:37 -0400 Message-ID: <008e01c02892$d45d6280$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: , Cc: Subject: Re: New List: rdma@cisco.com: to discuss RDMA Date: Wed, 27 Sep 2000 10:54:21 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk Before going too far down this road, it is important to understand if there is support for using an RDMA mechanism as a basis for iSCSI. Will the next draft of the iSCSI protocol actually be based on an RDMA mechanism that would be defined? The answers to these questions and the specific pros and cons need to be the driving force behind the RDMA discussion. -----Original Message----- From: csapuntz@cisco.com To: ips@ece.cmu.edu Cc: csapuntz@cisco.com Date: Tuesday, September 26, 2000 7:22 PM Subject: New List: rdma@cisco.com: to discuss RDMA > >There have been a couple proposals for RDMA on top of IP over the past >few months. It seems that there is interest in discussing and >developing this topic. The ips working group mailing alias may not >be the best place for a focused discussion on this topic. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28317; 27 Sep 2000 16:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA21952 for ips-outgoing; Wed, 27 Sep 2000 15:14:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA21941 for ; Wed, 27 Sep 2000 15:14:13 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 27 Sep 2000 12:16:04 -0700 Message-ID: From: Charles Monia To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Date: Wed, 27 Sep 2000 12:15:57 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Wednesday, September 27, 2000 12:05 AM > To: ips@ece.cmu.edu > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > > > Charles, > > This goes to the hearth of problem. > To avoid this effect we propose (like FCP) unordered messages > to go always > through - > allow the initiator to decide which task management messages he deems > "urgent". > But if you have a single connection and the TCP window is > closed you are > left with only > one think to do - drop the connection (and that is exactly what FCP is > doing for similar > reasons!). > > Julo > Hi Julo: Does dropping the connection blow away the iSCSI session? If so, FCP does nothing like that. In FCP terms, only the command is discarded. The "session" established at FC login time remains intact. That is, other pending commands and initiator-specific device context (mode page settings, reservations and the like) are unaffected. Charles From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23533; 26 Sep 2000 12:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA21650 for ips-outgoing; Tue, 26 Sep 2000 11:31:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA21641 for ; Tue, 26 Sep 2000 11:30:52 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDMB1M; Tue, 26 Sep 2000 08:28:00 -0700 From: "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" Subject: RE: iSCSI: multiple connection process Date: Tue, 26 Sep 2000 08:28:57 -0700 Message-ID: <000a01c027ce$7cbbc2c0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: <39D0916D.13D83DC9@stewart.chicago.il.us> Sender: owner-ips@ece.cmu.edu Precedence: bulk I watched this thread with curiosity. Isn't the WG in defining a multiple connection process solving a specific implementation problem? The head-of-queue blocking problem results from placing a bunch of PDUs from several iSCSI requests into a single queue. Therefore, the WG suggest another connection to separate commands from data to avoid deadlock. This problem is easily solvable with the creation of an exchange table from which we can serve the data PDUs without having them waiting in a separate queue. The WG should not worry about if someone chooses a wrong method of delivery that results to head-of-queue blocking. By wrong method, I mean using the TCP reads and writes to deliver/receive the PDUs. BTW, multiple connections for different application programs is natural and should be supported. This is similar to having multiple VI's in a single system. If an iSCSI implementation chooses multiple connections to solve the head-of-queue problem, it would be fine too. But, should the WG tell everyone how to do that? Am I wrong? If not, may be we should move forward. Y.P. Cheng, CTO, ConnectCom Solutions Corp. > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Randall R. Stewart > Sent: Tuesday, September 26, 2000 5:07 AM > To: julian_satran@il.ibm.com > Cc: Black_David@emc.com; matt_wakeley@agilent.com; ips@ece.cmu.edu > Subject: Re: iSCSI: multiple connection process > > > Julain: > > I have not been involved in any of the offline emails > that I am aware of and I don't see a consenses on the > list. I think David is just trying to move us > forward and I support this. I don't see that he is > saying remove it completly.. just lets get something > finished and then open this up in another draft... > > I think it is a good way to move forward... > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32285; 28 Sep 2000 15:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA02584 for ips-outgoing; Thu, 28 Sep 2000 14:07:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA02576 for ; Thu, 28 Sep 2000 14:06:54 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA14608; Thu, 28 Sep 2000 10:53:24 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA04312; Thu, 28 Sep 2000 10:51:11 -0700 From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt Date: Thu, 28 Sep 2000 11:09:00 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <39D36CC3.5796C52A@agilent.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, Reliance on the urgent pointer at the beginning if each message block is problematic. What pointer is valid with more than one such message within the pointer range? It is not clear how such a mechanism would work with existing implementations or if this pointer would vector to a message within the TCP segment or stagger by the send buffer. This would not be the intended use but at least some pointer would be better than nothing. To allow out of sequence processing, protection from spoofing should also be afforded. This seems rather messy when a clean solution already exists. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Thursday, September 28, 2000 9:08 AM > To: IPS Reflector > Subject: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt > > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28708; 27 Sep 2000 18:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA28614 for ips-outgoing; Wed, 27 Sep 2000 17:31:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA28610 for ; Wed, 27 Sep 2000 17:31:47 -0400 (EDT) Received: from amrelay1.boi.hp.com (amrelay1.boi.hp.com [15.56.8.24]) by palrel3.hp.com (Postfix) with ESMTP id C69A5A0E; Wed, 27 Sep 2000 14:31:45 -0700 (PDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA28254; Wed, 27 Sep 2000 15:31:45 -0600 (MDT) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Wed, 27 Sep 2000 15:31:44 -0600 Message-ID: <499DC368E25AD411B3F100902740AD654448D3@xrose03.rose.hp.com> From: "TALBOTT,GLENN (HP-Roseville,ex1)" To: "'csapuntz@cisco.com'" Cc: ips@ece.cmu.edu Subject: RE: New List: rdma@cisco.com: to discuss RDMA Date: Wed, 27 Sep 2000 15:31:35 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, et al., I believe that specifying an RDMA mechanism for iSCSI is the right thing to do. RDMA (especially when combined with a frame recovery mechanism such as the one described by Jim Williams) will enable iSCSI NIC implementers to eliminate the massive on-card buffering and double processing required to implement SACK in the face of long, fast, fat pipes. The NIC is able to push each inbound RDMA frame off into it's proper place in host memory while waiting for the missing packet instead of buffering the receive stream until the missing packet is received to recover synchronization. A general purpose RDMA and framing solution will benefit both iSCSI and VI/TCP. I do not believe that an RDMA mechanism should be mandatory for iSCSI. Manufacturers of lower cost thin storage devices may opt to limit applications to inside the data center, throw large amounts of off chip RAM at the problem (possibly not constrained to PCI NIC form factors), or accept lower performance by not implementing SACK, rather than go to the expense of the added complexity that RDMA may add to an implementation. I can see a future where iSCSI server initiators typically implement RDMA and targets typically do not. Then SCSI reads are satisfied by the target sending a sequence of RDMA writes to the initiator, and SCSI writes are satisfied by the target sending a sequence of RDMA reads to the initiator. This also eliminates the need for the target to send R2T. Each RDMA read is an implicit R2T. Glenn Talbott > -----Original Message----- > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > Sent: Wednesday, September 27, 2000 10:16 AM > To: Jim Williams > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > Subject: Re: New List: rdma@cisco.com: to discuss RDMA > > > > Does anybody on the list object to specifying an RDMA mechanism for > use with iSCSI? Does anybody on the list object to mandating an RDMA > mechanism? Please include your reasons. > > Current RDMA proposals: > draft-csapuntz-tcprdma-00.txt > draft-dicecco-vitcp-00.txt > > -Costa > > > Before going too far down this road, it is important to > > understand if there is support for using an RDMA mechanism > > as a basis for iSCSI. Will the next draft of the iSCSI > > protocol actually be based on an RDMA mechanism that > > would be defined? The answers to these questions and > > the specific pros and cons need to be the driving > > force behind the RDMA discussion. > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27113; 27 Sep 2000 9:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA03971 for ips-outgoing; Wed, 27 Sep 2000 07:59:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA03964 for ; Wed, 27 Sep 2000 07:59:37 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA12124; Wed, 27 Sep 2000 06:59:38 -0500 Message-ID: <39D1E12A.CF4D6C75@stewart.chicago.il.us> Date: Wed, 27 Sep 2000 06:59:38 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com CC: ips@ece.cmu.edu Subject: Re: VI (Was: Avoiding deadlock in iSCSI) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian: julian_satran@il.ibm.com wrote: > > Charles, > > This goes to the hearth of problem. > To avoid this effect we propose (like FCP) unordered messages to go always > through - > allow the initiator to decide which task management messages he deems > "urgent". > But if you have a single connection and the TCP window is closed you are > left with only > one think to do - drop the connection (and that is exactly what FCP is > doing for similar > reasons!). > This is exactly what SCTP's un-ordered message services was added for. It solves this very issue. R > Julo > > Charles Monia on 16/09/2000 01:19:37 > > Please respond to Charles Monia > > To: Stephen Byan , IPS Reflector > > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > -----Original Message----- > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > Sent: Wednesday, September 13, 2000 6:22 AM > > To: IPS Reflector > > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > > > > Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote: > > > > > I don't think VI/TCP helps at all. The VI is implemented on > > > top of a TCP > > > stream. If the TCP stream delivers the iSCSI command to VI, > > > and VI has no > > > place to put it, what is VI going to do? It has to block the > > > TCP stream, and > > > that in turn will block and "RDMA" from occurring. > > > > If I understand correctly, VI/TCP has a credit-based flow > > control mechanism > > on its message queues (which would be used to implement the > > iSCSI command > > reception queue), and so the initiator would never send an > > iSCSI command > > without a target buffer in which to store it. So RDMA cannot block on > > commands in the TCP stream. > > > > In the context of generic ULP "flow control" issues, the following > observations are offered. > > Assuming one VI/TCP message queue handles all SCSI control traffic for a > target, using flow control on messages to prevent the "queue-full" > condition > has undesirable side effects. Specifically, back pressure due to lack of > command context resources within a logical unit would stop the flow of > commands and task management requests to all other LUNS (a form of "head of > line" blocking). > > There's another issue that is perhaps less obvious. Even if multiple > message > queues are used, say one per LUN, stopping the flow of messages due to a > queue full condition would also stop the flow of task management messages > as > well. > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28851; 27 Sep 2000 19:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA29602 for ips-outgoing; Wed, 27 Sep 2000 17:57:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA18453 for ; Wed, 27 Sep 2000 14:02:24 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Wed, 27 Sep 2000 11:01:36 -0700 Message-ID: From: Robert Snively To: "'julian_satran@il.ibm.com'" , ips@ece.cmu.edu Subject: RE: Status summary on multiple connections Date: Wed, 27 Sep 2000 08:46:26 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, A third alternative does exist and may be preferable. That is to reflect the present broad usage of SCSI that uses wedge drivers to achieve parallelization, bandwidth aggregation, and high availability. By that definition, the proper number of connections per I_T nexus is one and the proper number of connections per session is also one. This recognizes the reality that there is really little to gain in virtualizing multiple connections into a single session image when you can perform even more flexible virtualization through a wedge driver at a higher level. The single connection alternative allows a simplistic ordering structure, a simple recovery mechanism, and does not require state sharing among multiple NICs. It allows bandwidth aggregation across any set of boundaries that is required. Because command queuing is the rule among high performance SCSI environments, latency appears only as an increment in host buffer requirements except during writes that perform a commit function. Those traditionally have been taken out of the performance path by using local non-volatile RAM to perform the commit functions, using slower high latency writes with less strict ordering requirements relative to reads to actually perform the write to media. The overheads associated with handling a multiple connection session of any type are basically indistinguishable from the overheads associated with device virtualization through wedge drivers. If you consider software instruction path lengths for the total functionality, you can conceive of the multiple connection session as simply a TCP/IP wedge driver. Bob > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, September 26, 2000 9:48 AM > To: ips@ece.cmu.edu > Subject: Status summary on multiple connections > > > > > Dear colleagues, > > I am attempting to summarize where we stand with regard to > the multiple > connection issue > and to the two possible models - Symmetric (S) and Asymmetric (A). > > Many of us feel strongly that the multiple connection issue > is central to > the whole design > and cannot be added as an afterthought. Moreover designing > the hooks to > allow it later > will certainly already force us to make a decision. And both > the hardware > and the software designers will be ill-served if we hand > them a half-backed > solution. > > However this is not an invitation to reiterate positions > that where already > stated. > If you feel that I have grossly misstated anything in this > note please > write me and use > the mailing list only if my answer is not satisfying. > > And yes - like the chairman - I would like to make progress > but I don't see > any way to > do it without satisfactorily closing this issue. > > The reasons for multiple connections where discussed at some > length and > where very nicely summarized in a series of notes by Michael Krause > (beginning of August). > > The core reasons for having multiple connections where the > need for more > bandwidth and availability than a single link can supply > with a level of > complexity affordable for simple > installations and with a traffic engineering and management clearly > separated from the transport users (SCSI). > > The session is embodying this requirement. > > The only major objections I have heard against this where > those requiring > to go all > the way in having one TCP connection/LU - and after a short > debate this > objection > was practically removed. > > The other objections we heard where that this is basically a > transport > issue and it should be > solved at transport level. > That might be true - but since many, if not most, of TCP > applications do > not have this > requirement it is highly unlikely that TCP is going to do connection > trunking in the foreseeable > future. > > iSCSI can be designed to use multiple TCP connections in one > of two ways: > > -Asymmetric - one TCP flow only carries commands the others > carry only data > > -Symmetric - every flow carries both commands and their > associated data > > The S version is designed in the I-D version 01 > The S version requires an command ordering scheme and that > is provided by a > command counter and a sliding window scheme. It was argued > that ordering > needs might be more prevalent than usually thought and a > good conservative > design should preserve ordering. > Ordering-per-LU (as it is designed in FCP-2 version 4) was > considered > impractical as it > required initiators to maintain state for each LU - while > the rest of the > design required initiators only to maintain state for > outstanding commands. > Several comments on this list suggested that this windowing > mechanism could > also be used as a command-flow-control mechanism and that is > a "bonus" of > the scheme. > > The A version - comes in two flawors: > > - pure A (PA) in which ONLY commands flow on one TCP > connection while data > flow > on DIFFERENT connections (with only one data connection > being selected > for a > command); this scheme requires a minimum of 2 TCP > connections although > not > necessarily on different physical links. > > - collapsed A (CA) in which commands flow one a single TCP > connection while > data > flow on ANY connection; this scheme requires as a minimum 1 TCP > connection. > > Here is first attempt to list the benefits and drawbacks of > all of them: > > > - S - benefits > - well understood > - simple hardware setup and/or TCP API activation > - window mechanism can be also used for flow control > - the minimum required is a single TCP connection > > - drawbacks > - need to maintain a window mechanism > - a multiplexing mechanism has to be carefully > crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > - PA - benefits > - TCP will both order the commands and provide for > flow control > through the TCP window mechanism > - the multiplexing mechanism is simpler as closing > TCP windows > will > never affect command flow > - data flow can use a streamlined header (and processing) > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > > - CA - benefits > - TCP will order the commands > - needs a single TCP connection as a minimum > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > - if command flow control is required counters > and the sliding > window mechanism > are required > - a multiplexing mechanism has to be carefully > crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > From the above it should be apparent - as was already > pointed out by Matt > Wakeley - > that the CA inherits the drawbacks of S and PA and as such > the choice is > really > between S and PA. > > Julo > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20472; 25 Sep 2000 18:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA17445 for ips-outgoing; Mon, 25 Sep 2000 17:50:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA17438 for ; Mon, 25 Sep 2000 17:50:44 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA13490; Mon, 25 Sep 2000 14:38:09 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA26438; Mon, 25 Sep 2000 14:35:54 -0700 From: "Douglas Otis" To: , , Subject: RE: Last Word on An IPS Transport Protocol? Date: Mon, 25 Sep 2000 14:53:11 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704100FDA@corpmx9.isus.emc.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Some have interpreted your comment to infer SCTP is not a suitable protocol for discussion. As SCTP does include TCP like congestion control (RFC 2581 compliance), could you clarify SCTP as a suitable protocol of merit. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > Sent: Monday, September 25, 2000 11:10 AM > To: ycheng@advansys.com; Black_David@emc.com; ips@ece.cmu.edu > Subject: RE: Last Word on An IPS Transport Protocol? > > > > My contention is the current TCP congestion control is NOT good > enough and > > the ACK traffic on a network with long latency delay is BAD. > We must have > > streamed transfer on a network with long latency. Therefore, > defining the > > ACK of TCP is critical. The TCP header format is not sacred to me. > > It's time to put my WG co-chair hat on and play "bad cop" ... > > There are experimental and production results indicating that TCP is > capable of saturating arbitrarily high bandwidth networks with arbitrarily > long delays. Buffering proportional to the bandwidth-delay product is > a good idea, so this doesn't come for free. Streaming transfer > can be achieved without playing these sort of ACK games - of course > if congestion is encountered, TCP backs off dramatically. > > This WG does not have the license to fundamentally change TCP's > congestion control algorithm or to use a transport that does not implement > congestion control in a sufficiently TCP-like manner (RFC 2581 compliance > is sufficient); the co-chairs and ADs will reject any document that > tries to do either of these things. Please don't consume list bandwidth > in further discussion of this. > > Developing a new transport with sufficient congestion control is going to > take time. If the WG were to go in this direction, at least a year should > be added to all of the completion milestones in the charter. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > -------------------------------------------------- > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27816; 27 Sep 2000 13:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA14976 for ips-outgoing; Wed, 27 Sep 2000 12:42:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA14961 for ; Wed, 27 Sep 2000 12:42:07 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA05792 for ; Wed, 27 Sep 2000 18:41:27 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA30044 for ; Wed, 27 Sep 2000 18:41:27 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256967.005BACC7 ; Wed, 27 Sep 2000 18:41:20 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 27 Sep 2000 19:39:07 +0300 Subject: RE: Zero-copy TCP stacks (Was: Avoiding deadlock in iSCSI) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't know how relevant this thread is. I assume that most DMA mechanism will not have any trouble starting at any boundary but will have trouble crossing page boundaries in system that use some form of mapping for either protection or virtual memory or both. Those have good scather-gather mechanisms might work in those cases too. Julo Stephen Byan on 18/09/2000 19:44:46 Please respond to Stephen Byan To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Zero-copy TCP stacks (Was: Avoiding deadlock in iSCSI) I may be merely increasing the noise floor; if so, I beg forgiveness in advance... Randall R. Stewart [mailto:randall@stewart.chicago.il.us] wrote: > Prasenjit Sarkar/Almaden/IBM wrote: > > BTW, zero-copy TCP/IP stacks have a lot of caveats (e.g. > memory alignment > > etc) which > > is why they have never made it to any operating system. > There are rough > > implementations > > in Solaris, BSD and Linux but none of them are particularly > close to being > > robust > > (I have tried them all). > > Prasenjit: > > I have used a zero-copy TCP/IP stack (if I recall correctly) > in VxWorks > or was it VRTX... I will grant you these are NOT *NiX systems and > are just recently beginning to become PosiX compliant... but I > seem to remember the zero-copy symenatics and I don't think there > was a memory alignment requriement.. it has been a while so I > may be mis-remembering.. :) I'm no TCP expert, but I think the difference between Randall's experience and Prasenjit's experience relates to the extent of the memory protection and security guarantees offered by the operating system. Embedded OS's such as VxWorks and VRTX do not provide separate memory address spaces for each task (aka process). Consequently there's no security problem in handing a pointer to an arbitrary memory buffer to a task - it can already see all of memory anyway. "Real OS's (TM)" do provide separate memory address spaces for each process (aka task). Consequently there is a security problem in handing a pointer to an arbitrary memory buffer to a task - the process could potentially view packets which happen to share the same memory page but which belong to other processes. To solve this problem, the TCP buffers must be aligned to a page boundary, and at most one buffer can be allocated per page. Regards, -Steve P.S. to the TCP experts - did I get this right? Steve Byan Design Engineer MS 1-3/E23 333 South Street Shrewsbury, MA 01545 (508)770-3414 fax: (508)770-2604 From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20164; 25 Sep 2000 16:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA10336 for ips-outgoing; Mon, 25 Sep 2000 15:11:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA10327 for ; Mon, 25 Sep 2000 15:11:24 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id VAA211138 for ; Mon, 25 Sep 2000 21:10:49 +0200 Received: from d12mta05.de.ibm.com (d12mta05_cs0 [9.165.222.239]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id VAA123776 for ; Mon, 25 Sep 2000 21:10:49 +0200 Received: by d12mta05.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256965.00695A6D ; Mon, 25 Sep 2000 21:10:44 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Black_David@emc.com cc: ips@ece.cmu.edu Message-ID: Date: Mon, 25 Sep 2000 22:08:23 +0300 Subject: RE: multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Thanks for your prompt answer. However I will try to summarize the views as I heard them only to keep tab on where we stand at the point the discussion was interrupted and state the (rather long) line of thought that makes us all feel that the design should include multiple connections from the outset. We can then stop discussing it for a while. I feel also that the community had no chance to see a draft including an asymmetric multiple connection model and it might be a worthwhile exercise to present one as an (optional text) with the next draft. I would like also - only to be fair - to point out - as others have done before - that by removing the command counters and the sliding window you have, for all practical purposes, closed the door to the symmetric multiple connection version. Julo Black_David@emc.com on 25/09/2000 17:47:19 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: multiple connections Julian, I have to ask you not to do that. There is consensus in the WG that multiple connection sessions are an important feature that needs to be specified, but should be optional to implement. There is NOT consensus on what the right design approach is. I have off-line email from proponents of both the Asymmetric and Symmetric multi-connection session models expressing dismay at the separation of them from the main specification and arguing that their preferred approach is the right one. These reinforce my observations that there is no consensus on the issue, and that the mailing list discussion is unlikely to achieve consensus. As I stated in earlier email, the requirement for multiple connection sessions has not been removed, but spending the next 6-8 weeks discussing it on the mailing list does not appear likely to achieve consensus. We need to try something else, namely an off-line design team. In the near term, list bandwidth is better used to make progress on issues where progress is still possible, such as flow control. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: julian_satran@il.ibm.com [SMTP:julian_satran@il.ibm.com] > Sent: Monday, September 25, 2000 3:32 AM > To: ips@ece.cmu.edu > Subject: multiple connections > > > > Dear colleagues, > > I was out on a week vacation and was a bit surprised by the turn of > events. > I will try to summarize today why the whole design team, and many more, > thought that multiple connections are a central feature to iSCSI and why > removing it will harm the effort. > I thought that we have a rough consensus on this and what we are not fully > confident about is if to go for an asymmetric scheme (with separate > control > and data connections) or a symmetric scheme. > > Julo > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20048; 25 Sep 2000 15:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA07712 for ips-outgoing; Mon, 25 Sep 2000 14:10:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA07693 for ; Mon, 25 Sep 2000 14:10:40 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 25 Sep 2000 14:10:10 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FDA@corpmx9.isus.emc.com> To: ycheng@advansys.com, Black_David@emc.com, ips@ece.cmu.edu Subject: RE: Last Word on An IPS Transport Protocol? Date: Mon, 25 Sep 2000 14:10:09 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. It's time to put my WG co-chair hat on and play "bad cop" ... There are experimental and production results indicating that TCP is capable of saturating arbitrarily high bandwidth networks with arbitrarily long delays. Buffering proportional to the bandwidth-delay product is a good idea, so this doesn't come for free. Streaming transfer can be achieved without playing these sort of ACK games - of course if congestion is encountered, TCP backs off dramatically. This WG does not have the license to fundamentally change TCP's congestion control algorithm or to use a transport that does not implement congestion control in a sufficiently TCP-like manner (RFC 2581 compliance is sufficient); the co-chairs and ADs will reject any document that tries to do either of these things. Please don't consume list bandwidth in further discussion of this. Developing a new transport with sufficient congestion control is going to take time. If the WG were to go in this direction, at least a year should be added to all of the completion milestones in the charter. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 -------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19875; 25 Sep 2000 14:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06785 for ips-outgoing; Mon, 25 Sep 2000 13:50:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06781 for ; Mon, 25 Sep 2000 13:50:01 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id MAA05943; Mon, 25 Sep 2000 12:50:07 -0500 Message-ID: <39CF904F.FBB8543A@stewart.chicago.il.us> Date: Mon, 25 Sep 2000 12:50:07 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: Last Word on An IPS Transport Protocol? References: <000101c02707$8b146d80$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P: It is NOT within the scope of this WG to change ANY Congestion Control parameters of TCP. The working group COULD spend a year or so and define its OWN protocol.. HOWEVER it WOULD HAVE TO HAVE congestion control compliant to RFC 2581! This means you do not get anything even close to your proposal. The bottom line is changing the congestion control is NOT allowed..IMHO R Y P Cheng wrote: > > > Sorry, that doesn't achieve the goal of complying with RFC 2581. If the > > "Delay Constant" is set incorrectly for the network, the result is > > disastrous - the whole point of slow start is to set and adjust parameters > > like the "Delay Constant" automatically in a fashion independent of > network > > conditions. Also see the discussion in RFC 2581 about initial window > size. > > If slow start is needed to avoid disaster on the network, I don't have any > problem with a slow start. However, the cost on performance -- taking 1.6 > sec to transfer one MB of data using slow start -- must be considered. When > the MTU is small, 512 instead of 2K, the cost is even higher. How long does > it take to move one gigabytes of data with iSCSI on a network with 50 msec > round-trip delay? > > > In other words, change the TCP protocol to make it go faster for your > > traffic without changing the TCP header format. Not only is this > > not a good idea, but the approved WG charter specifically PROHIBITS the > > WG from working on this, unless the argument can be made that the > > changes are required for storage traffic. I believe the current WG > consensus > > is that TCP congestion control is good enough for storage traffic, > > and hence would like to put an end to this thread. > > Dave, I think you have missed a very important part of my proposal. We must > stream the data transfer with DMA assist to overcome the performance problem > of storage traffic on a network with long delay. Changing or not the TCP > header is not my point at all. The proposed protocol does not change the > TCP protocol either if the sequence size is set to one, i.e. the segment > size. > > My contention is the current TCP congestion control is NOT good enough and > the ACK traffic on a network with long latency delay is BAD. We must have > streamed transfer on a network with long latency. Therefore, defining the > ACK of TCP is critical. The TCP header format is not sacred to me. > > If the WG does not see this point and believes otherwise, I will submit to > the wish of this WG. But I do believe the current discussion has totally > missed the effect of the long latency network delay. > > Y.P. Cheng, CTO, ConnectCom Solutions Corp. -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28563; 27 Sep 2000 17:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA26844 for ips-outgoing; Wed, 27 Sep 2000 16:50:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA26838 for ; Wed, 27 Sep 2000 16:50:50 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA14318; Wed, 27 Sep 2000 13:37:40 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id NAA01162; Wed, 27 Sep 2000 13:35:25 -0700 From: "Douglas Otis" To: "Charles Monia" , , "Jim Williams" Cc: Subject: RE: New List: rdma@cisco.com: to discuss RDMA Date: Wed, 27 Sep 2000 13:53:05 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, With respect to SCTP, features added by a TCP option for RDMA are not needed to support alignment and out of sequence processing that ultimately alters the TCP API. SCTP adds these features without disruption or modification to TCP. The intent of VI is to allow scatter/gather function to handled by the target. A safer scheme would be to adhere to SCSI conventions and implement zero copy and out of sequence processing using SCTP and SAM structures related to locally pre-arranged transfer structures. This would keep the initiator or client in intimate control of memory and not reliant on targets eliminating boundary checking. Such a feature will slightly impact SCTP to add a means to generally encapsulate a data payload associated with structure of pointers equipped with scatter-gather lists and relative offsets. Specifically, this could be defined as the FCP structure data structure in SCSI implementations. I would whole-heartedly endorse such features within SCTP. I would dissuade such features from being added to TCP. One could view the scatter-gather list structure as a token to be used by the target in more conventional VI methods. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Charles Monia > Sent: Wednesday, September 27, 2000 11:49 AM > To: csapuntz@cisco.com; Jim Williams > Cc: ips@ece.cmu.edu > Subject: RE: New List: rdma@cisco.com: to discuss RDMA > > > > > > -----Original Message----- > > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > > Sent: Wednesday, September 27, 2000 10:16 AM > > To: Jim Williams > > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > > Subject: Re: New List: rdma@cisco.com: to discuss RDMA > > > > > > > > Does anybody on the list object to specifying an RDMA mechanism for > > use with iSCSI? Does anybody on the list object to mandating an RDMA > > mechanism? Please include your reasons. > > > > I'd object to mandationg the use of RDMA in iSCSI. However, I > would support > structuring the spec so that an RDMA transport mechanism could be used > underneath (I guess that's motherhood). If the iSCSI folks decided this > wasn't a priority issue, that's ok with me too. > > Charles > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25586; 27 Sep 2000 1:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA22809 for ips-outgoing; Tue, 26 Sep 2000 23:50:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA22805 for ; Tue, 26 Sep 2000 23:50:29 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id XAA40922 for ; Tue, 26 Sep 2000 23:44:22 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8R3oSN120082 for ; Tue, 26 Sep 2000 21:50:28 -0600 Importance: Normal Subject: RE: iSCSI: Session Partial Resolution To: "IPS (E-mail)" X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 26 Sep 2000 20:47:53 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/26/2000 09:50:28 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, I thought we were all together until you said, ...have the initiator budget one "credit" to be used for this purpose. Up until that statement there had been no mention of Credit, and I am not sure why we need to bring that concept in. Randy (HAAGENS), You said: "The question of the storage controller's overcommitting command queue credits has been raised. Our initial thought was that this would not be allowed. But if it must be allowed, then we could extend the command sequencing scheme to allow to target to drop commands and later demand their retransmission. The command sequence numbers would ensure that, ultimately, all commands are delivered in order, with no commands lost or duplicated." I am confused since I do not find the work "Credits" anywhere in the Draft. With all the stuff that has been going around about credits, perhaps you could either tell me what you meant by credits, or tell me where that is specified in the Draft. The way it has been use recently is to clone the approach that FC uses. This seems confusing, since I do not understand why that is needed with TCP. Therefore, I keep thinking that I am misunderstanding you. In any event, based on what Charles said (in the rest of his note) I got the feeling that there was only Two "channels" (or connections) needed per Session to prevent the blockage folks keep talking about. One for commands and one for data, esp. since Charles made the point that control functions should be on the same connection with commands. Do you agree that a double connection per Session solves most of the problems we have been talking about? . . . John L. Hufferd Charles Monia @ece.cmu.edu on 09/26/2000 05:35:22 PM Sent by: owner-ips@ece.cmu.edu To: "HAAGENS,RANDY (HP-Roseville,ex1)" , "David Black (E-mail)" cc: "IPS (E-mail)" Subject: RE: iSCSI: Session Partial Resolution Hi: See below for a few comments on flow control. > -----Original Message----- > From: HAAGENS,RANDY (HP-Roseville,ex1) [mailto:randy_haagens@hp.com] > Sent: Tuesday, September 26, 2000 7:51 AM > To: David Black (E-mail) > Cc: IPS (E-mail) > Subject: RE: iSCSI: Session Partial Resolution > > > David, > > I. Re: proposed points of consensus > > (1) Command sequencing was added to the draft for several > reasons, but the > important point here is that command flow control is > necessary when using a > single TCP connection in order to prevent a command queue > full condition > from preventing the reception of write data. We could just > leave command > flow control to the good behavior of the SCSI layer (there is no SCSI > mechanism defined for it, and so it is > implementation-dependent), but we > decided that it was legitimate to treat it at the iSCSI > "transport" layer, > since from SCSI's pov, it is a transport problem. Command > sequencing also > solved two other problems (ordered command striping and > recovery from TCP > connection failure), so it seemed like an efficient mechanism overall. > > Some have objected that command windowing may result in the > command queue's > being blocked, and urgent commands' not getting through. If > I recall the > draft correctly, we made provision for this with the notion > of unnumbered > commands. These can be sent at any time by the initiator; > but since they do > not have a reserved place in the command queue, their > reception must be > considered unreliable: the target can drop them at its discretion. I > believe this situation is familiar to the T10 community, and > acceptable from > a SAM-2 perspective. But if not, we could create a separate, > sequenced, > command queue for urgent commands. > Hi: I think the above proposal is headed in the right direction with respect to command windowing. I'd like to make a few observations and suggestions however. 1. I'm inclined to reserve the use of unnumbered "slots" for task management functions, such as "abort task", etc. 2. As I recall (possibly not very accurately) SAM-xx states that an initiator should not have more than one pending task management request at a time. In general, such requests are "think-time" limited and therefore non blocking, so this seems not to be a problem in practice. 3. It's important that this set of control functions flow over the same control connection that's used for commands (ie. these functions need to flow through the command delivery pipe). Otherwise their behavior is indeterminate. An example is an "abort task" function which arrives at the target while the command to be aborted is still in transit. 4. Considering the rule of allowing only one pending task management request at a time, it might be sufficient to have the initiator budget one "credit" to be used for this purpose. > The question of the storage controller's overcommitting command queue > credits has been raised. Our initial thought was that this > would not be > allowed. But if it must be allowed, then we could extend the command > sequencing scheme to allow to target to drop commands and > later demand their > retransmission. The command sequence numbers would ensure > that, ultimately, > all commands are delivered in order, with no commands lost or > duplicated. > A good idea in my opinion. Charles From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23862; 26 Sep 2000 14:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA00304 for ips-outgoing; Tue, 26 Sep 2000 13:43:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA00297 for ; Tue, 26 Sep 2000 13:43:00 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id NAA15629; Tue, 26 Sep 2000 13:19:07 -0400 Message-ID: <019d01c027db$304d3060$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: "Douglas Otis" , Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Tue, 26 Sep 2000 12:59:52 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: Douglas Otis To: Jim Williams ; ips@ece.cmu.edu Date: Monday, September 25, 2000 5:56 PM Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) >You suggest a frame aligned formatting scheme is possible using an >unmodified TCP. Yes. >By allowing out of sequence delivery of TCP, are you opening the window for >spoofing? One need not guess the sequence, just get in front of it a bit. >After that, you will never recover state. A 64 bit randomly selected connection ID should prevent blind spoofing. For an attacker who snoops valid packets and then spoofs, we would be vulnerable. However no window is opened, just an already open window failed to close tight. > Do you expect the RDMA option to be universally supported, and if not? The proposal is not an option, it is a format. The format could potentially be shared by multiple protocols. It is by no means universal; it applies only to those protocols that adopt it. >Do you see this level of CRC done in software? I see it being done in hardware in a NIC by anyone who cares about performance. That's largely the case for TCP checksums now. Non performance critical applications could do it in software. >> It is >> RECOMMENDED that each TCP segment contain exactly one RDMA segment. > >How do you achieve this recommendation? With cooperation of the TCP implementation. Without that, it cannot in general be guaranteed. Typically for performance critical applications, NICs will handle both TCP and RDMA protocol processing, and in this case it is easy. >Sending 4GB exchanges, you still have queue blocking if not head of queue >blocking. Yes. If that's a problem, don't send 4GB messages. All I am saying is that the format allows 4GB messages. >Can one suggest frame alignment within TCP? Certainly one can suggest it. Will the suggestion be supported? Does it violate any rules that can't be changed? Perhaps I will find out. >Have you investigated the use >of SCTP? SCTP does solve all the related problems without violating >protocol or protocol API. SCTP is an interesting protocol with a lot of merrit and a lot of good ideas. But my guess is that the market wants one transport protocol and wants it to be TCP. My vote would be to stick with TCP and evolve it in a way that meets the requirements of the market and is fully compatable with the existing infrastructure. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28862; 27 Sep 2000 19:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA00018 for ips-outgoing; Wed, 27 Sep 2000 18:10:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e2.ny.us.ibm.com (e2.ny.us.ibm.com [32.97.182.102]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA00011 for ; Wed, 27 Sep 2000 18:10:08 -0400 (EDT) Received: from northrelay02.pok.ibm.com (northrelay02.pok.ibm.com [9.117.200.22]) by e2.ny.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA54320 for ; Wed, 27 Sep 2000 18:09:12 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by northrelay02.pok.ibm.com (8.8.8m3/NCO v4.93) with ESMTP id SAA37378 for ; Wed, 27 Sep 2000 18:09:30 -0400 Importance: Normal To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 27 Sep 2000 15:07:58 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/27/2000 04:09:31 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, If you remember way back when we first got together, you had a good RDMA proposal for use with TCP/IP using the various optional fields. We all seemed to love it at the time. I for one was disappointed when we dropped it out of the iSCSI proposal (even as an optional consideration). The thoughts were, that if we started to push this onto TCP/IP folks, we would spend much time on that and not get the iSCSI stuff done. So it was dropped. I for one would find it useful to bring it back, but only as an optional feature, and only if that did not get in the way of the iSCSI Stuff, since you can see how that has already begin to fill up the current ips reflector. But I think working it as a side issue, on another reflector, until it is fully baked seems OK to me. Summary: RDMA -- Yes -- the one you originally had proposed Optional --Yes . . . John L. Hufferd From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa31009; 28 Sep 2000 7:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA16931 for ips-outgoing; Thu, 28 Sep 2000 05:56:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA16925 for ; Thu, 28 Sep 2000 05:56:25 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA108018 for ; Thu, 28 Sep 2000 11:55:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id LAA109904 for ; Thu, 28 Sep 2000 11:55:51 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256968.00368CC2 ; Thu, 28 Sep 2000 11:55:49 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Thu, 28 Sep 2000 12:53:29 +0300 Subject: RE: A Transport Protocol Without ACK Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Dear Mr. Cheng, I am baffled by your taxonomy. VI and I2O are basically interface (API) definitions and each of them has a different purpose. VI was ment to be a User-Space-to-User-Space communication interface and I2O well... a panacea for drivers. We are trying to build a protocol (to which an API is assumed but not mandated). Could you be a bit more specific in what you criticize or propose? Julo "Y P Cheng" on 19/09/2000 03:52:10 Please respond to "Y P Cheng" To: "'Ips@Ece. Cmu. Edu'" cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: A Transport Protocol Without ACK From: randall@stewart.chicago.il.us > I see no viable transport protocol here and I don't see this > conversation of any use unless you get exact details AND point > to a internet draft that defines EXACTLY how it works (or possibly > some other standards document). Both I2O and VI are transport protocols which define the format of a request to a transport service provider, i.e. an adapter card. I2O is used but not limited to deliver SCSI requests and VI is used for any payload including IP packets. VI is mapped into FC with the device headers between the FC header and data payload. VI can certainly be used for delivery of SCSI requests too. Both protocols require the service provider to have reliable delivery and reception. VI defines different QoS. > > I don't claim any credit about this transport layer protocol. Every fibre > > channel and Infiniband adapter designer knows about this protocol -- > > although there is no standard. I am sure the TCP accelerator card is doing > > the same. This protocol is a great alternative to the use of TCP/IP and > > should be incorporated into iSCSI. > > No it is not. You are not offering an alternative yet.. I did not imply iSCSI should use I2O or VI. In fact, the purpose iSCSI is to map SCSI requests into IP packets as well as to define the delivery . It seems to me that the working group has set its mind on TCP/IP and is believing this is the only solution. The consensus seems if there is any other solutions that address flow control and congestion, it would end up like TCP/IP. I am simply pointing out if we keep an iSCSI request as a single atomic transaction without separating it into the TCP/IP-stream-oriented Writes and Reads that each deals with a single DU, then, the deadlock problem goes away. While the work group thinks we should take advantage the flow control and congestion management of TCP/IP, there are alternatives known as BB-credit and EE-credit management. The fibre channel adapters make reliable delivery, lost packet detection, and retransmission without TCP/IP. Randall, you are right, I did not spent time to provide the working group a draft defining such transaction-oriented protocol. All I have provided is an idea that besides TCP/IP. The designers for SCSI and fibre channel adapters have solved the head-of-queue blocking, the congestion, and retransmission problems. The transaction-oriented WRITE-REQUEST and READ-RESPONSE, in my humble opinion, allows us to implement iSCSI simpler than that of WRITE and READ stream requests. The performance cost of requiring ACKs on every DU with size greater than MTU on a network with long latency is very expensive.. By defining a greater ACK granularity is an attempt to solve this performance problem. If we do wish to ACK on every DU, then, on a long latency network, we must have a method to stream the PDUs to ensure the performance. The method should not consume a large amount of memory space. One should never ignore the TCP/IP memory-to-memory copy overhead when the backbone will be running at OC-192 speed in the near future. Finally, please don't ever ask two NIC cards to synchronize with each other. It is really hard to do as those of us in business of designing NIC cards can testify. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24965; 26 Sep 2000 21:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA17170 for ips-outgoing; Tue, 26 Sep 2000 20:33:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA17161 for ; Tue, 26 Sep 2000 20:33:43 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Tue, 26 Sep 2000 17:35:27 -0700 Message-ID: From: Charles Monia To: "HAAGENS,RANDY (HP-Roseville,ex1)" , "David Black (E-mail)" Cc: "IPS (E-mail)" Subject: RE: iSCSI: Session Partial Resolution Date: Tue, 26 Sep 2000 17:35:22 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: See below for a few comments on flow control. > -----Original Message----- > From: HAAGENS,RANDY (HP-Roseville,ex1) [mailto:randy_haagens@hp.com] > Sent: Tuesday, September 26, 2000 7:51 AM > To: David Black (E-mail) > Cc: IPS (E-mail) > Subject: RE: iSCSI: Session Partial Resolution > > > David, > > I. Re: proposed points of consensus > > (1) Command sequencing was added to the draft for several > reasons, but the > important point here is that command flow control is > necessary when using a > single TCP connection in order to prevent a command queue > full condition > from preventing the reception of write data. We could just > leave command > flow control to the good behavior of the SCSI layer (there is no SCSI > mechanism defined for it, and so it is > implementation-dependent), but we > decided that it was legitimate to treat it at the iSCSI > "transport" layer, > since from SCSI's pov, it is a transport problem. Command > sequencing also > solved two other problems (ordered command striping and > recovery from TCP > connection failure), so it seemed like an efficient mechanism overall. > > Some have objected that command windowing may result in the > command queue's > being blocked, and urgent commands' not getting through. If > I recall the > draft correctly, we made provision for this with the notion > of unnumbered > commands. These can be sent at any time by the initiator; > but since they do > not have a reserved place in the command queue, their > reception must be > considered unreliable: the target can drop them at its discretion. I > believe this situation is familiar to the T10 community, and > acceptable from > a SAM-2 perspective. But if not, we could create a separate, > sequenced, > command queue for urgent commands. > Hi: I think the above proposal is headed in the right direction with respect to command windowing. I'd like to make a few observations and suggestions however. 1. I'm inclined to reserve the use of unnumbered "slots" for task management functions, such as "abort task", etc. 2. As I recall (possibly not very accurately) SAM-xx states that an initiator should not have more than one pending task management request at a time. In general, such requests are "think-time" limited and therefore non blocking, so this seems not to be a problem in practice. 3. It's important that this set of control functions flow over the same control connection that's used for commands (ie. these functions need to flow through the command delivery pipe). Otherwise their behavior is indeterminate. An example is an "abort task" function which arrives at the target while the command to be aborted is still in transit. 4. Considering the rule of allowing only one pending task management request at a time, it might be sufficient to have the initiator budget one "credit" to be used for this purpose. > The question of the storage controller's overcommitting command queue > credits has been raised. Our initial thought was that this > would not be > allowed. But if it must be allowed, then we could extend the command > sequencing scheme to allow to target to drop commands and > later demand their > retransmission. The command sequence numbers would ensure > that, ultimately, > all commands are delivered in order, with no commands lost or > duplicated. > A good idea in my opinion. Charles From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa30756; 28 Sep 2000 5:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA15085 for ips-outgoing; Thu, 28 Sep 2000 04:08:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA15079 for ; Thu, 28 Sep 2000 04:08:01 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA89648; Thu, 28 Sep 2000 10:07:29 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA134310; Thu, 28 Sep 2000 10:07:29 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256968.002CA0FF ; Thu, 28 Sep 2000 10:07:27 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: csapuntz@cisco.com, ips@ece.cmu.edu Message-ID: Date: Thu, 28 Sep 2000 11:07:17 +0300 Subject: Re: New List: rdma@cisco.com: to discuss RDMA Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I think we should specify an RDMA mechanism for use with iSCSI, as this will help all interested parties to have interoperable RDMA mechanisms. I don't think RDMA should be mandated for iSCSI; I assume there will be implemenations on hosts and devices that won't have the resources to perform RDMA. - Kalman Meth csapuntz@cisco.com on 27/09/2000 20:16:29 Please respond to csapuntz@cisco.com To: "Jim Williams" cc: ips@ece.cmu.edu, csapuntz@cisco.com (bcc: Kalman Meth/Haifa/IBM) Subject: Re: New List: rdma@cisco.com: to discuss RDMA Does anybody on the list object to specifying an RDMA mechanism for use with iSCSI? Does anybody on the list object to mandating an RDMA mechanism? Please include your reasons. Current RDMA proposals: draft-csapuntz-tcprdma-00.txt draft-dicecco-vitcp-00.txt -Costa > Before going too far down this road, it is important to > understand if there is support for using an RDMA mechanism > as a basis for iSCSI. Will the next draft of the iSCSI > protocol actually be based on an RDMA mechanism that > would be defined? The answers to these questions and > the specific pros and cons need to be the driving > force behind the RDMA discussion. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32081; 28 Sep 2000 14:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA29328 for ips-outgoing; Thu, 28 Sep 2000 12:50:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA29321 for ; Thu, 28 Sep 2000 12:50:17 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA14566; Thu, 28 Sep 2000 09:37:08 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA04096; Thu, 28 Sep 2000 09:34:54 -0700 From: "Douglas Otis" To: "John Hufferd/San Jose/IBM" , Subject: RE: not every song Date: Thu, 28 Sep 2000 09:52:43 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk John, It is clear your aim is providing a connection to a controller. I would hope a standard would support more than this one song you sing and provide access to devices for reasons already mentioned. How do you envision controlling clients? I wonder about your definition of TCP done right. It is also clear you wish to see TCP change, as you have often repeated. FCP offers WRITE-DATA and READ-RESPONSE structures if desired. Once a reasonable flow control akin to FC Class 3 is added, no structural changes are required as shown in http://www.ietf.org/internet-drafts/draft-otis-fc-sctp-ip-01.txt. To advocate extensive change, show justification. Bluster not. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > John Hufferd/San Jose/IBM > Sent: Wednesday, September 27, 2000 9:18 PM > To: ips@ece.cmu.edu > Subject: RE: > > > Douglas, > not every song you sing should have the same tune over and over and over. > We have heard it very well. In fact I look forward to the point in time > that we will have completed the TCP/IP version of iSCSI and turn our focus > at whether SCTP is approprate and how it will benefit iSCSI. > > Now against my better judgement I will try to engage this one > more time. A > careful reading of my note implies nothing like you have read into it. We > looked at the value of the functions that Costa had in his original > proposal for OPTIONAL extensions to TCP/IP. When these were used they > permitted, in my estimation and others, a positive effect on a number of > TCP/IP applications. It just so happened that one of the applications was > iSCSI. > > Now, the arguments that I put in my note stated why we chose to move the > TCP/IP RDMA option to the background. I think this very thread captures > the essence of the reason why it was put on a back burner. There > were also > strong feeling that TCP/IP done right could move us forward in a > significant way even without the RDMA option. We did not want to > sacrifice > the Very Very Good for the Perfect. > > If Costa and group can in another reflector and a separate set of > interactions with the IETF (perhaps with our support) get acceptance for > the TCP/IP RDMA option then that would be good for many TCP/IP > applications > including iSCSI. And I, for one wish them good luck. > > Since this thread was intended to give Costa my thoughts, which I have, > there is nothing to gain to continue this discussion thread further. > > . > . > . > John L. Hufferd > > > > "Douglas Otis" on 09/27/2000 06:50:20 PM > > To: John Hufferd/San Jose/IBM@IBMUS, > cc: > Subject: RE: > > > > John, > > If the intent of the draft-satran-iscsi-01.txt is to eventually modify TCP > in a significant way, these issues should be addressed ahead of concluding > such features as RDMA with TCP will become available. As much of the > optimization required in the use of TCP for SCSI hinge on such > modifications, it would appear you have already concluded such > modifications > *will* take place. > > Most should agree frame alignment, out of sequence processing, independent > streams sharing a single control, and zero-copy data handling are deciding > issues. Separate TCP buffers to control resources, frame > alignment, unique > TCP APIs are between the lines in this proposal. Before havoc ensues, > there > is an alternative. Rather than bending TCP to fit, SCTP has > already laid a > foundation. As such, *no* transport modifications are required and yet > customers still get friendly TCP behavior. > > Doug > > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > John Hufferd/San Jose/IBM > > Sent: Wednesday, September 27, 2000 3:08 PM > > To: ips@ece.cmu.edu > > Subject: > > > > > > Costa, > > If you remember way back when we first got together, you had a good RDMA > > proposal for use with TCP/IP using the various optional fields. We all > > seemed to love it at the time. I for one was disappointed when we > dropped > > it out of the iSCSI proposal (even as an optional consideration). The > > thoughts were, that if we started to push this onto TCP/IP folks, we > would > > spend much time on that and not get the iSCSI stuff done. So it was > > dropped. > > > > I for one would find it useful to bring it back, but only as an optional > > feature, and only if that did not get in the way of the iSCSI Stuff, > since > > you can see how that has already begin to fill up the current ips > > reflector. But I think working it as a side issue, on another > reflector, > > until it is fully baked seems OK to me. > > > > Summary: > > RDMA -- Yes -- the one you originally had proposed > > Optional --Yes > > > > . > > . > > . > > John L. Hufferd > > > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27803; 27 Sep 2000 13:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA13882 for ips-outgoing; Wed, 27 Sep 2000 12:19:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA13877 for ; Wed, 27 Sep 2000 12:19:05 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA61118 for ; Wed, 27 Sep 2000 18:18:33 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA130964 for ; Wed, 27 Sep 2000 18:18:33 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256967.005994DD ; Wed, 27 Sep 2000 18:18:28 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 27 Sep 2000 19:16:11 +0300 Subject: Re: TCP vs. UDP on SMP Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk My experience is similar. And this was also the reason that I stated that using several TCP connections would be beneficial to SMP machines (easing both coupling and some serialization inherent to some stacks). Julo "Franco Travostino" on 18/09/2000 05:20:10 Please respond to "Franco Travostino" To: Black_David@emc.com, cslee@FalconStor.com.tw, John Hufferd/San Jose/IBM@IBMUS cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: Re: TCP vs. UDP on SMP > My recollection of published results on using multiple >processors for >TCP in Unix-like operating systems is that the big gains come from using >different >processors for different connections rather than handling send and receive >on a single >connection on different processors due in part to the send/receive coupling >required >by congestion control ... but it's been a while since I've checked/read this >literature. Much agreed. Some quantitative results in: Performance Issues in Parallelized Network Protocols, Erich M. Nahum, David J. Yates, James F. Kurose, and Don Towsley. USENIX Symposium on Operating Systems Design and Implementation (OSDI), Nov., 1994 (Monterey, CA). Now that we're in TCP/IP offload age ... Recent programmable network processors throw more silicon (e.g., hardware threads) and expose lower level of control (e.g., asynchronous memory operations) than general-purpose CPUs in SMP boxes. Kind of an SMP on a single chip, really. While the TCP vs. UDP fundamental parallelization issues are still in the way, I expect that there will be a bunch of parallelization trade-offs more when optimizing TCP on network processors. -franco From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23882; 26 Sep 2000 14:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA01581 for ips-outgoing; Tue, 26 Sep 2000 14:06:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA01570 for ; Tue, 26 Sep 2000 14:06:18 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Tue, 26 Sep 2000 14:05:41 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FF0@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: iSCSI: multiple connection process Date: Tue, 26 Sep 2000 14:05:40 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I have two objections to the direction you have taken: > > - you considered objections that where voiced to you and not to the mailing list; > the community should judge if they are core or marginal; I would appreciate if > you could summarize them Let me be clear that lack of consensus was clear from the list traffic prior to my saying that consensus was lacking. The subsequent emails served to reinforce the decision, but the decision was not based on them. In general, emails directly to me about process issues are perfectly reasonable, and I prefer to leave it to those who write such emails about whether to post them publicly or contact me directly. I did not see any new significant technical arguments/information on the session issue in the emails; as the mailing list is the working forum, new technical information/arguments do need to go on the mailing list, but I didn't see any in this instance. > - the key issues with any of the two approaches are understood - an evaluation > committee will only broaden the "discontent" Given that Julian's summary has now appeared, attempting to shut down discussion at this point is clearly futile ... I hope everyone involved now has an appreciation for the consequences of lack of consensus ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32665; 28 Sep 2000 17:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA09668 for ips-outgoing; Thu, 28 Sep 2000 16:54:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA09654 for ; Thu, 28 Sep 2000 16:54:27 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8SKsRU18718 for ; Thu, 28 Sep 2000 15:54:27 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA05060; Thu, 28 Sep 00 15:52:46 -0500 Message-Id: <10009282052.AA05060@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: VI (Was: Avoiding deadlock in iSCSI) In-Reply-To: Message from Charles Monia of "Wed, 27 Sep 2000 12:15:57 PDT." Date: Thu, 28 Sep 2000 15:52:58 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Does dropping the connection blow away the iSCSI session? If so, FCP does > nothing like that. If by connection in FCP, you mean login, you actually have a choice. If an initiator choses to recover the session and does it within the defined time, it may do so. Early in FCP, none of the recovery mechanisms worked correctly, so doing this triggered all sorts of horrible bugs. The alternative is just to log in from scratch, which has the effect of killing the session and allowing the target to recover its state without waiting for the timer to expire. The problem with iSCSI seems to be the long maximum recovery timer value. Obviously, the amount of unrecovered state (including orphan operations) increases proportionally. Whether that's a big deal or not seems to depend upon many implementation factors. In FCP if an initator dies and is reborn, when it reconnects, the orphan session state is automatically recovered by merit of the fact that there can be only one session between two endpoints. In iSCSI, there can be multiple sessions between endpoints (sure, it's strongly discouraged, but it would be boneheaded to prohibit it), so starting a new session can not recover state from an old one. One could argue that an iSCSI connection is much more durable than an FCP login, and when it fails, immediate recovery is unlikely anyway. Would reservations be a problem in this approach? If you think so, I'd be grateful for you (or somebody else) to go ahead and make that argument. The reason I say this is that reservations seem to have problems at a SAM level which are independent of transport, and I'm not sure this behavior would make them any worse. Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32454; 28 Sep 2000 16:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA06249 for ips-outgoing; Thu, 28 Sep 2000 15:38:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA06242 for ; Thu, 28 Sep 2000 15:38:22 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA14252 for ; Thu, 28 Sep 2000 15:36:42 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8SJb0276676 for ; Thu, 28 Sep 2000 13:37:01 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: twist on iSCSI asymmetric model To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Thu, 28 Sep 2000 12:31:56 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/28/2000 01:37:00 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Kalman, I am not sure what your new suggestion solves. Especially if what Charles wrote is correct: {Snip} "2. As I recall (possibly not very accurately) SAM-xx states that an initiator should not have more than one pending task management request at a time. In general, such requests are "think-time" limited and therefore non blocking, so this seems not to be a problem in practice. 3. It's important that this set of control functions flow over the same control connection that's used for commands (ie. these functions need to flow through the command delivery pipe). Otherwise their behavior is indeterminate. An example is an "abort task" function which arrives at the target while the command to be aborted is still in transit. 4. Considering the rule of allowing only one pending task management request at a time, it might be sufficient to have the initiator budget one "credit" to be used for this purpose." {Snip} (His use of Credit here meant a reserved Buffer) I believe that you also have determined that Two connections per Session Solves the key problems (especially if what Charles said is correct,) and they are on the same NIC, since you said ".... If the control and data connections happen to be on the same NIC, then this might not be a problem". This, I believe, is the point that Matt has been trying to make. He felt that two connections per session were key and should be required. So lets put it all together; Control should flow on the same connection as Commands, and Data on a separate connection (within the same session). This solves the blockage issue, and both connections within the same NIC solve the interrupt problem. Plus it does not have iSCSI sliding windows etc. etc., the implementation is relatively easy, compatible with the concept of iSCSI HW on the NIC, and the Wedge Drivers still work as they do today. Based on that, I think Matt has a strong point. What do you think? You also added (on your last Paragraph) a statement that I think takes us in a different direction "...Only one interrupt is required upon completing a READ data transfer, if we mark the last iSCSI Data PDU of the data transfer". I am not sure what you are attempting to say here, are you making a new recommendation? In any event, please address the previous questions, before you amplify your last paragraph. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 meth@il.ibm.com@ece.cmu.edu on 09/28/2000 06:30:58 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: twist on iSCSI asymmetric model In the asymmetric model, one TCP connection (the control connection) is used for sending all of the commands, thus ensuring ordered delivery of the SCSI commands. If (a large amount of) data is also sent on the control connection, there is the possibility that the data would fill up the window, and critical task management requests couldn't get through. On the other hand, it has been pointed that some implementations will prefer a single (and only) TCP connection for both commands and data. I would like to suggest the possibility of combining these 2 cases; either (1) have a single connection for everything or (2) have one connection for control and one or more data connections. In this case, no (WRITE or READ) data may be sent over the control connection. The initiator and target negotiate during the first login as to which of the 2 possibilities they will use. In (1), it is the responsibility of the initiator to worry about not overloading the one and only connection so as not to delay delivery of critical task management messages. In (2), the iSCSI SCSI Command PDU specifies which data connection to use for the data of this command. On the data channels, where we will have iSCSI Data and R2T PDUs, we shouldn't need to use the long headers that we use to specify an iSCSI SCSI Command PDU. We will therefore have a shortened header for the data channels. In (1), since data will travel on the control channel, the iSCSI Data PDU and R2T headers will be padded with zeros so that all headers are of the same length on a particular connection. The use of R2T is negotiated during the first login of the iSCSI session. If R2T is to be used, then the initiator may not send data before it is requested by the target. Thus there will always be buffers ready to accept the data. If the target agrees to not use R2T, then the target implicitly agrees to have some number of buffers available to receive data sent by the initiator, even if the data arrives before the command arrives. In this case the initiator MUST send data in the data connections in the same order that the corresponding commands were sent. (This avoids deadlocks.) If the target runs out of buffers, then it will let data back up in the TCP window of the data connections, while it is processing the data from earlier commands. The only main drawback remaining with the asymmetric model is that 2 separate operations may be required to set up sending a command and to register its DMA buffers. If the control and data connections happen to be on the same NIC, then this might not be a problem. Only one interrupt is required upon completing a READ data transfer, if we mark the last iSCSI Data PDU of the data transfer. - Kalman Meth. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa27375; 27 Sep 2000 10:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA06842 for ips-outgoing; Wed, 27 Sep 2000 09:32:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA06835 for ; Wed, 27 Sep 2000 09:32:28 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Wed, 27 Sep 2000 09:31:39 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704100FFC@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: Connection status summary corrections Date: Wed, 27 Sep 2000 09:31:38 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Regarding the requirement to have multiple connections or not I would not say that it > precludes ONE connection for control and ONE for data as many other > transport mechanisms (some buses, some protocols like FTP) use. And as the person who called that consensus, I am informing Julian that he is incorrect. I know that Matt Wakeley has a standing objection to this consensus, and now Julian makes two (although I hope that's a misunderstanding). I continue to be amazed at the lack of ability to make progress on these session issues - does anyone else want to reopen the issue of whether more than one TCP connections will be REQUIRED in an iSCSI session? At the moment, continued discussion of sessions appears to be making negative progress ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20413; 25 Sep 2000 18:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA15679 for ips-outgoing; Mon, 25 Sep 2000 17:09:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from proxy4.ba.best.com (root@proxy4.ba.best.com [206.184.139.15]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA15674 for ; Mon, 25 Sep 2000 17:09:24 -0400 (EDT) Received: from nvsys.com (adsl-63-205-242-26.dsl.snfc21.pacbell.net [63.205.242.26]) by proxy4.ba.best.com (8.9.3/8.9.2/best.out) with ESMTP id OAA09822 for ; Mon, 25 Sep 2000 14:08:42 -0700 (PDT) Message-ID: <39CFBEF6.C997EB74@nvsys.com> Date: Mon, 25 Sep 2000 14:09:10 -0700 From: Bob Quinn-NVS Organization: NVS X-Mailer: Mozilla 4.72 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: "ips@ece.cmu.edu" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk remove From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24717; 26 Sep 2000 20:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA14811 for ips-outgoing; Tue, 26 Sep 2000 19:18:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA14673 for ; Tue, 26 Sep 2000 19:13:24 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 6F3A28EC4; Tue, 26 Sep 2000 16:08:34 -0700 (PDT) To: ips@ece.cmu.edu Subject: New List: rdma@cisco.com: to discuss RDMA Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 26 Sep 2000 16:08:34 -0700 Message-ID: Lines: 21 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk There have been a couple proposals for RDMA on top of IP over the past few months. It seems that there is interest in discussing and developing this topic. The ips working group mailing alias may not be the best place for a focused discussion on this topic. In that vein, I have created a list called rdma@cisco.com. To subscribe to it, send a message to mailer@cisco.com with a blank subject line and a one-line body that says "subscribe rdma address@your.domain". The goals of the list include: - Establishing the utility (if any) of RDMA - Enumerating requirements for RDMA on top of IP - Discussing the current proposals - Co-ordinating proof-of-concept implementations - Gauging interest in proceeding with standardization of RDMA Both TCP and SCTP will be initially considered. -Costa From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32096; 28 Sep 2000 14:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA29367 for ips-outgoing; Thu, 28 Sep 2000 12:50:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA29342 for ; Thu, 28 Sep 2000 12:50:36 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA14570; Thu, 28 Sep 2000 09:37:10 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA04099; Thu, 28 Sep 2000 09:34:56 -0700 From: "Douglas Otis" To: Cc: , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Thu, 28 Sep 2000 09:52:43 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, The point missed is in how names are managed. You wish to include name management within the transport where it does not belong. SCSI third party commands presently map in a binary fashion. In some cases, this binary to name translation should not be sent by the client. Relegating this function to LDAP server as example, both the client and the device server depend on an external management already in use. By attempting to encompass name management rather than leveraging off existing standards, security is weakened and such standard undergo revisions unrelated to transport as naming conventions, associated attributes, and points of definition change. Doug > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, September 28, 2000 3:11 AM > To: Douglas Otis > Cc: dfsmith@almaden.ibm.com; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > Doug, > > There seem to be practical and philosophical issues that you consistently > ignore. > Textual names are more stable than underlying network addresses and thus > better > for applications (I assume that your card starts with your name not phone > number and > that the former has changed less than the later). > I the iSCSI realm - and that is the only one under scrutiny - a pertinent > observation was > made by David Black a century ago during our (too) long naming discussions > - that a scheme that will allow the third party commands to make the name > to address conversion > at use time is better than one that requires this mapping to be made ahead > of time. > Unfortunately that is not yet possible with SCSI but we are informed that > it soon be. > And for authentication the textual address in login is yet another mean to > prevent > errors and/or unauthorized access if the target chooses to use it. > > Julo > > "Douglas Otis" on 19/09/2000 06:55:29 > > Please respond to "Douglas Otis" > > To: Daniel Smith/Almaden/IBM@IBMUS, ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > Daniel, > > SCSI is not HTTP and does not provide a human interface as a basis for > communication. Ownership of naming abstractions vary from the perspective > of the client to that of the provider. The provider may not wish to share > management of these naming abstractions with the client and vise-versa. > Rather than attempting to keep a permanent naming abstraction as globally > shared between both provider and client, these abstractions should be > meaningful to the human dealing with the names. For the client it may be > Tom, Dick, and Harry, and for the provider it could be Shelf A, E, and N. > By keeping these abstractions independent of the transport layer, > it allows > each end to use such means as LDAP servers to join these views in a > coherent > fashion. Your IT manager will insist you not concern yourself with the IP > of your workstation, as he has control of these assignments through DHCP > and > LDAP servers, the same should be true for a networked version of SCSI. > > Do not forget the large task of authentication that may also be managed by > either the provider or the client. By keeping the transport as plain as > possible, these naming abstractions can become as complex as required > without affecting the transport specification. Implanting names into SCSI > is a mistake and should be corrected as it limits this valuable freedom. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Daniel Smith > > Sent: Monday, September 18, 2000 5:25 PM > > To: ips@ece.cmu.edu > > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > Just in case people are wondering what this proposal is/was (the mailing > > list seems to have exploded quite a bit since it was released > > back in May), > > here it is again, with a handful of [modifications] and [comments]. > > -- > > IBM Almaden Research Center, 650 Harry Road, San Jose, CA > 95120-6099, USA > > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > > > > > > > > ---------------------------------------------- > > > > Subject: iSCSI naming > > Date: Mon, 8 May 2000 15:01:17 -0700 (PDT) [mods 18 Sep 2000] > > > > iSCSI naming requirements (first [and a half] iteration) > > ------------------------- > > > > It is important to have a consistent naming scheme as the basis of a > > discovery system for the SCSI over TCP protocol. A basic set of > > requirements is below, and a method is suggested. > > > > Requirements > > ------------ > > > > A name identifies a [SCSI service delivery port], not a LUN. > > notes: There may be many thousands of LUNs (and virtual LUNs) on a > > connection. > > why: The LUNs given to each connection are likely to be dynamic > > and assigned > > by the iSCSI manager. The iSCSI names are more static and long-lived, > and > > must be persistent even if the iSCSI manager fails. > > notes: A separate naming scheme will document the LUN discovery and > naming > > mechanism. [query to LUN 0 to get list of LUNs] > > > > The namespace need not be fundamentally [systematically] enumerable. > > notes: FC can be enumerated, the Internet cannot (practically). > > why: iSCSI will be deployed on untrusted networks. Having > > devices that can > > be enumerated are subject to systematic attack. Also, many > > devices will be > > behind firewall machines, and not even contactable until a login > procedure > > has taken place---holes in such an enumerated namespace are not > > acceptable. > > notes: Devices on a LAN can be enumerated, and this is acceptable. > > notes: Enumeration (listing) by a 3rd party is acceptable. > > > > The names should be as static as possible. > > notes: Both in time and location. > > why: A name given for one iSCSI device really ought to resolve to the > same > > device every time, from everywhere. [device here refers to a > SCSI service > > delivery port, of course the LUs should stay the same too, > barring things > > like security lock-outs] > > > > The name should not be tied to the network technology. > > notes: For example, ethernet MAC address or IPv4/v6 address. > > why: To cope with future networking systems, the name of the > iSCSI device > > should not rely only on aspects of the transport that are liable to > become > > obsolete, or that are not static enough. > > notes: IPv4 addresses may be acceptable on a controlled private network, > > however, these names should not be the canonical names as > viewed from the > > outside. [major problems with NAT if that happens] > > > > The naming scheme must be reliable. > > notes: If parts of the network go down, the names must remain intact. > > why: iSCSI may form part of the networking infrastructure, meaning that > > after power loss, for example, the iSCSI devices will be required > > to bring up > > the rest of the network. This implies that the devices should be aware > of > > their own name, if necessary for network stability. It also > implies that > > any name service should have redundancy. > > > > The naming scheme should be nice to look at. > > why: Long strings of unformatted numbers are horrible. [and prone > > to typos] > > > > The iSCSI device being addressed should be maskable from observers. > > why: The device name may contain valuable (secure) information, > > so it must be > > possible to hide this information from prying network interfaces. > > notes: The URL scheme manages this, by separating the hostname from the > > device name. > > > > Proposal > > -------- > > We should follow the URL naming scheme for iSCSI devices, thus: > > scsi:/// > > Where > > :=|: is the TCP [destination] identifier... > > is the DNS or DDNS host name of the iSCSI device and > > is the TCP port number to connect to. > > > > And [ should be ] > > :=|/|? is the device > identifier > > is a text identifier for the device > > :=|? is an argument for the > device > > [argument escape character was '&' in the original] > > > > The element is reasonably well defined. However, I would prefer > to > > keep the element more free-form. The > scsis:/// is > > also a straightforward extension. [to secure (SSL) iSCSI] > > > > The identifier specifies an iSCSI TCP connection > > identifier only. > > This may provide access to one or many LUNs that flit in and out of LUN > > space. > > > > One caveat is that some (most?) iSCSI devices will not have > > access to DNS. > > Fortunately, devices (targets) do not need to [query the > network to find] > > their names except in the case of trying to resolve another device > > (target)---i.e., only initiators need DNS resolution. > > > > Here are some fun examples of the things I would want to do as a storage > > manager. > > > > The "backup" program does a simple block-to-block copy from one > > iSCSI device > > to another... > > backup scsi://raid.acme.com/lun12 scsi://tape.acme.com/id25 > > Of course, a more useful operation may be... > > flashcopy scsi://disk.acme.com/12/2 scsi://disk.acme.com/13/2 > > where the pathnames are ASCII numbers. (They may even be the WWN or LUN > > value.) > > > > This doesn't address the issue of security. When we have more of those > > issues resolved, we might want to issue a command like this... > > kcopy scsis://secure.acme.com/raid12/set2?key=12345?auth=dfsmith > > ...so that the part, containing the security tokens, is > > never passed > > in the clear over the network. > > > > It may be possible for a manager to transparently interpret the > arguments > > or pathname into a LUN list in a storage farm. > > iscsi attach scsi://farm.acme.com/tray17?LUN=0x17?LUN=0x18 > > /dev/scsi2 > > iscsi attach scsi://san.acme.com/5A/27/23/99/B7/A2/01/00 /dev/sda > > > > The Text/Response commands in the iSCSI Internet-Draft might be > > accommodated > > thusly: > > iscsitext scsi://tape.scsi.acme.com/cart5 "com.ibm.retension:yes" \ > > "com.ibm.eject:now" > > > com.ibm.retension:ok retensioning > > > com.ibm.eject:ok scheduled ETA 2:30 > > > > Secure enumeration might be done in the following way, with peculiar > > examples of possible Un*x integration. > > iscsi find_domain_manager > > > iscsi.acme.com > > httpsget https://iscsi.acme.com:99/root&auth=dfsmith&key=1234 > > > > > > > > > scsi://unit1.raid.acme.com/ > > > disk1 disk2 disk3 disk4 disk8 disk9 > > > > > > > > > > > > scsi://bigtape.acme.com/racf4 > > > cart57 cart58 > > > > > > > > > > > > scsi://ibmdeskstar75.dfsmith.acme.com/lun1 > > > > > > > > iscsi mount scsi://ibmdeskstar75.dfsmith.acme.com/lun1 /mnt/mydisk > > > /mnt/mydisk: 73564224 blocks available > > tar -cf 'scsi://bigtape.acme.com/racf4/cart57&user=dfsmith' ~/* & > > > [3] 4291 > > runsocks dd if=/dev/zero of=scsi://dfsmith.blocks4less.com/disk1 & > > > [4] 4293 > > And so on. (Note: I am not an XML expert.) > > > > [Additional notes: > > > > The decision to resolve devices in the host name or in the > device name is > > really up to the implementer. Some companies may already have > systems in > > place for handling the DNS while they have no existing SAN > > infrastructure. > > Thus the decision to name a particular service delivery port > > scsi://disk26.farm-b.acme.com/ > > or > > scsi://farm-b.acme.com/disk26 > > is up to the system designer. > > > > The iSCSI URL names a bundle of LUs (one iSCSI "connection"). People > love > > to point out that this is a problem if you have multiple entrance > > ways into > > the same SAN, and the same device (say WWN=0x1234) has different LUNs > (say > > LUN=5 on iscsi://scsi.acme.com/bridge1 and LUN=477 on > > iscsi://scsi.acme.com/bridge2). So don't do that. Or do that > > and just live > > with it. Or make sure the different initiator/bridges into the SAN > assign > > the device the same LUN. (There isn't really a problem using a LUN in a > > global context, as long as it's always attached to its service delivery > > port.) (Having to query every LUN on a port to get the list of WWNs > ought > > to be addressed by T10, in the same way they covered the "query > > LUN 0 to get > > a list of LUNs" problem. IMO!) > > > > The security aspects bring up the spectre of LUs being masked out for > > different iSCSI initiators on different parts of the net. This is fun > and > > nothing to worry about. > > > > These URL names are used in several places, most of which are not > > related to > > the iSCSI transport itself. (Likewise, html is not particularly > > related to > > HTTP, nor SMTP to metamail.) > > > > First by users (or users by dint of clobbering an html hyperlink) who > want > > to attach to iSCSI devices. The names are descriptive and are > > malleable to > > all sorts of marvelous manipulations by the sysadmins. > > > > Second, by directory servers, who can deliver a list of the > URLs (and the > > LUNs that are accessible therein) to the iSCSI clients. > > > > Third, by iSCSI itself for the 3rd party copy command. In this > case, the > > URL specifies the destination SAN and the target LUN is carried in the > > additional data field. > > > > ] > > > > > > > > > > ---------------------------------------------- > > > > Jim Hafner/Almaden/IBM wrote: > > > > > > I've seen Daniel's proposal before and I still haven't figured out the > > > context in which such names are to be used. > > > > > > Given that, I'm personally a bit uncomfortable with any naming > > convention > > > that uses LUN values in a global context. These have no meaning as > LUNs > > > are host-specific values. Even worse, LUNs are addresses, not names! > > > > > > I have no problem with WWNs in a global context as that is what they > are > > > for! > > > > > > A subtle difficulty, is that a WWN does not give a host an > > address (either > > > for the target or the LU). Even in SPC-2, EXTENDED COPY's > > Identification > > > Descriptor Target Descriptor Format (sic?) where WWNs are used, says > > > "instructs the copy manager to locate a target and logical unit that > > > returns a device identification VPD page ..." but gives no > hints on how > > > that should be done (e.g., walk the bus and do INQUIRY to > > everything...?). > > > > > > The "hostname" and DNS provide a canonical method for getting an > address > > > for a name, so that part is OK. The only defined way to get > an address > > > (LUN) for a LU WWN is by exhaustive INQUIRY to each LU at a > > given target. > > > > > > On the other hand, I can envision how some of these things > > might work, in a > > > well-coordinated and well-managed environment: > > > > > > There is a managing application which has the dual > responsibility of > > > managing the targets for their host/LUN mapping > > configurations and also > > > serves as the "walk the bus" discovery resource for hosts. > > > > > > When the manager (human?) determines the distribution of > > host/target LU > > > resources (after inventorying both hosts and target LUs), the > > > application will send to the targets whatever host/LUN > > Mapping (access > > > controls?) commands are necessary to configure them. Later, > > when a host > > > boots it queries the application to get its "walk-the-bus" > > services. In > > > this, it minimally gets from this application a list of > targets that > > > have LUs to which it should have access. There is no > REQUIREMENT at > > > this point for the application to give the host anything > more as the > > > normal SCSI LU discovery process per target kicks in. On the other > > > hand, I can see some value in the host getting more information > about > > > the LUs it will see at each target. In that case, the host could > get > > > from the application a list of targets and LU identifiers > > (these could > > > be LUNs or WWN or both, as you suggested). The important > > point here, is > > > that these LUNs are valid only in the context of the > > requesting HOST and > > > are not global! > > > > > > Note that this process requires clear coordination between > > the "target > > > configurator" and "host discovery of targets". > > > > > > > > > Given an appropriate context for a URL scheme, I'd make two > > modifications: > > > 1) drop the assumption that no LUN means LUN=0. If no LU > > qualifier (e.g., > > > no LUN or WWN (or other) is provided), then LUN=0. In all > > other cases, the > > > alternative names should be sufficient to identify a LU, > > though, as above, > > > finding a LUN address may require extra work. > > > 2) add ?ProxyToken= as an additional option here (this > > coordinates > > > well with the changes to EXTENDED COPY approved in the context > > of the SCSI > > > access controls). > > > > > > Well, that's another two cents! I was hoping to save some of this > > > discussion until the protocol gets worked out, but .... > > > > > > Jim Hafner > > > > > > > > > csapuntz@cisco.com@ece.cmu.edu on 09-17-2000 11:11:24 PM > > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > > > > To: IP Storage > > > cc: csapuntz@cisco.com > > > Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > > > > > > > > > > Here is a proposal to extend the iSCSI URL scheme to support LUNs and > > > WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This > > > extension is not required for the iSCSI transport protocol, but may be > > > useful for discovery and management. > > > > > > A URL for the target has the following form: > > > > > > scsi://hostname/path/with/ > > > > > > A URL referring to a specific LU has the following form: > > > > > > scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber > > > > > > If no LUN= term appears in the URL, then LUN 0 is assumed. > > > > > > The WWN= term is optional. If present, the party should > > > verify that the WWN in the LU's Device Identification Inquiry > > > Page correspojnds to the WWN. > > > > > > Note, the following URL: > > > > > > scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa > > > > > > refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After > > > connecting, the initiator should verify that the WWN > > > of the LU is 0a050a4bcdefa. > > > > > > -Costa > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa29024; 27 Sep 2000 20:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA01586 for ips-outgoing; Wed, 27 Sep 2000 18:57:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA01577 for ; Wed, 27 Sep 2000 18:57:03 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA48114 for ; Wed, 27 Sep 2000 18:55:17 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8RMtU3155686 for ; Wed, 27 Sep 2000 16:55:31 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: Status summary on multiple connections To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 27 Sep 2000 15:52:36 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/27/2000 04:55:31 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David Black, Let me vote that there is merit in having two Asymmetric Conversation per Session. Even though I do appreciate the request for a minimum of Two, I am troubled with the fact that I think many folks have already built iSCSI Initiators and Targets that use just one connection per session. Now in Robert Snively note below, he talks about the value of the Wedge Drivers, and some of us also buy into that. However, his arguments were based on spreading the stuff across NICs etc. I do not know if that is required, but having one TCP/IP connection that only carries Commands, and one that only carries Data, seemed to solve almost all the complaining and pacing/credit/sliding window conversations we have been having. This is true even when the second TCP/IP connection (of the session) is on the SAME NIC. (Robert, I think that all your wedge statements still hold true with 2 Asymmetric Connections per session, if both connections are on the same NIC.) Except, for that, much of the rest of Roberts points are "right on". So if we can just for awhile talk about whether the Dual Conversation Asymmetric approach is acceptable, I think that you may very well find that a lot of issue go away. And I think that was Matt's point. If this is correct, then, it is important to have a minimum two conversation per Session (maybe two is all that is really needed). But what about the implementations that are currently in flight. I think we may have to permit the single connection per Session, but strongly discourage it, since when it gets into longer distances, or works with implementation with less memory, or in some kind of network stress, the recovery approaches are much more draconian (dropping the connection etc.). That means that all the implementations that are working today, will continue to work, but will not be nearly as good as the Asymmetric Double Connection per Session (ADCS?) versions. Now Matt's point is that for folks just starting, being able to have code both the Double and the Single (with all the extra recovery/pacing stuff required in the Single Conversation per Session) seems wrong. And I think I have heard him state that updating the Driver code from a Single to a Double (especially if within the same NIC, was a non problem). If my statements about ADCS is correct, the testing effort should be a lot more straight forward since all the other various "boundary conditions" will not have to be tested for, since they will not exist. And if Matt's statements are correct, that makes me wonder why wouldn't we go that way as a requirement? Does anyone that currently has iSCSI drivers have a problem of moving to 2 Asymmetric Connections per Session as a requirement, if so, please state why. . . . John L. Hufferd Robert Snively @ece.cmu.edu on 09/27/2000 08:46:26 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Status summary on multiple connections Julo, A third alternative does exist and may be preferable. That is to reflect the present broad usage of SCSI that uses wedge drivers to achieve parallelization, bandwidth aggregation, and high availability. By that definition, the proper number of connections per I_T nexus is one and the proper number of connections per session is also one. This recognizes the reality that there is really little to gain in virtualizing multiple connections into a single session image when you can perform even more flexible virtualization through a wedge driver at a higher level. The single connection alternative allows a simplistic ordering structure, a simple recovery mechanism, and does not require state sharing among multiple NICs. It allows bandwidth aggregation across any set of boundaries that is required. Because command queuing is the rule among high performance SCSI environments, latency appears only as an increment in host buffer requirements except during writes that perform a commit function. Those traditionally have been taken out of the performance path by using local non-volatile RAM to perform the commit functions, using slower high latency writes with less strict ordering requirements relative to reads to actually perform the write to media. The overheads associated with handling a multiple connection session of any type are basically indistinguishable from the overheads associated with device virtualization through wedge drivers. If you consider software instruction path lengths for the total functionality, you can conceive of the multiple connection session as simply a TCP/IP wedge driver. Bob > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, September 26, 2000 9:48 AM > To: ips@ece.cmu.edu > Subject: Status summary on multiple connections > > > > > Dear colleagues, > > I am attempting to summarize where we stand with regard to > the multiple > connection issue > and to the two possible models - Symmetric (S) and Asymmetric (A). > > Many of us feel strongly that the multiple connection issue > is central to > the whole design > and cannot be added as an afterthought. Moreover designing > the hooks to > allow it later > will certainly already force us to make a decision. And both > the hardware > and the software designers will be ill-served if we hand > them a half-backed > solution. > > However this is not an invitation to reiterate positions > that where already > stated. > If you feel that I have grossly misstated anything in this > note please > write me and use > the mailing list only if my answer is not satisfying. > > And yes - like the chairman - I would like to make progress > but I don't see > any way to > do it without satisfactorily closing this issue. > > The reasons for multiple connections where discussed at some > length and > where very nicely summarized in a series of notes by Michael Krause > (beginning of August). > > The core reasons for having multiple connections where the > need for more > bandwidth and availability than a single link can supply > with a level of > complexity affordable for simple > installations and with a traffic engineering and management clearly > separated from the transport users (SCSI). > > The session is embodying this requirement. > > The only major objections I have heard against this where > those requiring > to go all > the way in having one TCP connection/LU - and after a short > debate this > objection > was practically removed. > > The other objections we heard where that this is basically a > transport > issue and it should be > solved at transport level. > That might be true - but since many, if not most, of TCP > applications do > not have this > requirement it is highly unlikely that TCP is going to do connection > trunking in the foreseeable > future. > > iSCSI can be designed to use multiple TCP connections in one > of two ways: > > -Asymmetric - one TCP flow only carries commands the others > carry only data > > -Symmetric - every flow carries both commands and their > associated data > > The S version is designed in the I-D version 01 > The S version requires an command ordering scheme and that > is provided by a > command counter and a sliding window scheme. It was argued > that ordering > needs might be more prevalent than usually thought and a > good conservative > design should preserve ordering. > Ordering-per-LU (as it is designed in FCP-2 version 4) was > considered > impractical as it > required initiators to maintain state for each LU - while > the rest of the > design required initiators only to maintain state for > outstanding commands. > Several comments on this list suggested that this windowing > mechanism could > also be used as a command-flow-control mechanism and that is > a "bonus" of > the scheme. > > The A version - comes in two flawors: > > - pure A (PA) in which ONLY commands flow on one TCP > connection while data > flow > on DIFFERENT connections (with only one data connection > being selected > for a > command); this scheme requires a minimum of 2 TCP > connections although > not > necessarily on different physical links. > > - collapsed A (CA) in which commands flow one a single TCP > connection while > data > flow on ANY connection; this scheme requires as a minimum 1 TCP > connection. > > Here is first attempt to list the benefits and drawbacks of > all of them: > > > - S - benefits > - well understood > - simple hardware setup and/or TCP API activation > - window mechanism can be also used for flow control > - the minimum required is a single TCP connection > > - drawbacks > - need to maintain a window mechanism > - a multiplexing mechanism has to be carefully > crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > - PA - benefits > - TCP will both order the commands and provide for > flow control > through the TCP window mechanism > - the multiplexing mechanism is simpler as closing > TCP windows > will > never affect command flow > - data flow can use a streamlined header (and processing) > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > > - CA - benefits > - TCP will order the commands > - needs a single TCP connection as a minimum > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > - if command flow control is required counters > and the sliding > window mechanism > are required > - a multiplexing mechanism has to be carefully > crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > From the above it should be apparent - as was already > pointed out by Matt > Wakeley - > that the CA inherits the drawbacks of S and PA and as such > the choice is > really > between S and PA. > > Julo > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:48 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa28294; 27 Sep 2000 16:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA20402 for ips-outgoing; Wed, 27 Sep 2000 14:47:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA20393 for ; Wed, 27 Sep 2000 14:47:32 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 27 Sep 2000 11:49:24 -0700 Message-ID: From: Charles Monia To: csapuntz@cisco.com, Jim Williams Cc: ips@ece.cmu.edu Subject: RE: New List: rdma@cisco.com: to discuss RDMA Date: Wed, 27 Sep 2000 11:49:17 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > Sent: Wednesday, September 27, 2000 10:16 AM > To: Jim Williams > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > Subject: Re: New List: rdma@cisco.com: to discuss RDMA > > > > Does anybody on the list object to specifying an RDMA mechanism for > use with iSCSI? Does anybody on the list object to mandating an RDMA > mechanism? Please include your reasons. > I'd object to mandationg the use of RDMA in iSCSI. However, I would support structuring the spec so that an RDMA transport mechanism could be used underneath (I guess that's motherhood). If the iSCSI folks decided this wasn't a priority issue, that's ok with me too. Charles From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04557; 29 Sep 2000 19:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA24375 for ips-outgoing; Fri, 29 Sep 2000 18:58:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA24371 for ; Fri, 29 Sep 2000 18:58:47 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 18:58:17 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070410102C@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: ips Administrative Matters and framework document Date: Fri, 29 Sep 2000 18:58:15 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk The ips WG charter is now posted on the IETF web site at: http://www.ietf.org/html.charters/ips-charter.html Everyone should read it, as it contains a number of important statements of goals, requirements, and restrictions, in addition to a current set of expected milestones. There are no drafts currently linked to the charter. The current plan is to have new versions of the iSCSI and FC over IP drafts submitted as WG documents (draft-ietf-ips-...) and link them at that point. Things have changed sufficiently since the last versions, that linking them to the web copy of the charter could seriously mislead people who aren't following the mailing list. This should happen by the end of October, and hopefully a lot earlier (yes, this is a public hint to the document authors :-) ). That takes care of the first milestone on the charter. The second milestone is to get an initial version of the framework document submitted as an Internet-Draft. The charter says: "The WG will produce a framework document that provides an overview of the environments in which its encapsulated protocols and related protocols are expected to operate." The framework document is the place for discussions of how, why and where iSCSI and FC-over-IP might be used and what the various other components are that have to be present to make them usable aside from implementations of these protocols. This also includes complementarity of these protocols and discussion of overlap/ situations in which both might be applicable. The initial version of the framework document will be produced by an off-line author team for review and comment by the WG. I'm looking for a few good authors for the document - expertise on networking, SCSI and Fibre Channel will be needed in the author team. Volunteers should have the time available to produce significant text in the next couple of months, and there's a need for one person to put in the time to coordinate this effort and assemble the text as document editor. The rewards are fleeting fame as a co-author of the document; I would guess that it'll go through several serious revisions over the next 6 months or so before it can be submitted for publication as an informational RFC. If you're interested (and I hope a number of folks out there are), please send me email directly. Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00137; 28 Sep 2000 18:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA10947 for ips-outgoing; Thu, 28 Sep 2000 17:23:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e32.bld.us.ibm.com (e32.co.us.ibm.com [32.97.110.130]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA10925 for ; Thu, 28 Sep 2000 17:22:55 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e32.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id RAA09482 for ; Thu, 28 Sep 2000 17:21:18 -0400 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8SLLY238654; Thu, 28 Sep 2000 15:21:35 -0600 Importance: Normal Subject: Re: Bidirectional SCSI commands and iSCSI To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Thu, 28 Sep 2000 14:21:32 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/28/2000 02:21:34 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, The primary motivation is for object-based storage devices (OSDs). The protocol that seems to be needed in that environment is the ability to send a CDB, plus some parameter data (DataOut) AND get back read-type data (DataIn) in the same command sequence. One (non-OSD) command has been proposed in T10 for an XOR type XDREADWRITE command. SAM-2 has now allowed for such commands in the architecture. From this perspective, it is completely symmetric. Transports that support this feature will need to have data structures in there PDUs or IUs (or whatever) to indicate things like DataOut datalength, DataIn datalength, possible virtual addresses for each buffer, etc. In other words, there will potentially be two active buffers (on each end) for use with the same command. So, e.g., iSCSI PDU may need to have both "DataIn expected data transfer length" and "DataOut expected transfer length" fields. I'm assuming that the data direction in the existing iSCSI PDU is derived from the SCSI opcode, unlike FCP where there are fields to indicate the data direction. (I am surprised not to find such things in the iSCSI Command PDU.) There are two proposals in T10 to enable this feature in SPI-4 and in FCP-x (probably FCP-3). Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-28-2000 12:10:53 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, What is the intended use? Are there already commands that use it? Is there a major/minor distinction - like output data & input results or output parameters & input data - or is it intended to be completely symmetric? Julo "Jim Hafner/Almaden/IBM" on 20/09/2000 17:56:27 Please respond to "Jim Hafner/Almaden/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Bidirectional SCSI commands and iSCSI Folks, In case you missed this, T10 just approved changes to SAM-2 (rev14) which remove the restriction that a single SCSI command can have a data flow in only one direction (DataIn xor DataOut). Transport specs (that want to allow this) will need to change to allow for two sets of Data buffer fields in the headers (length, over/underrun, etc.). Is anybody looking to modify the iSCSI spec to get these concepts supported? Jim Hafner From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03255; 29 Sep 2000 11:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA05487 for ips-outgoing; Fri, 29 Sep 2000 10:40:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA05480 for ; Fri, 29 Sep 2000 10:40:20 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id JAA00951; Fri, 29 Sep 2000 09:43:23 -0400 Message-ID: <007701c02a20$8050c170$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: "Douglas Otis" , , Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Fri, 29 Sep 2000 10:21:02 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk -----Original Message----- From: Douglas Otis To: Jim Williams ; ips@ece.cmu.edu Date: Tuesday, September 26, 2000 2:52 PM Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Regarding: ftp://coke.giganet.com/rdma-tcp.txt >Jim, > >In summary, you expect kernel modifications of TCP API to allow frame >alignment, a hardware based 32-bit CRC that uses a different polynomial than >FCP and a direct copy method at the NIC. I do not expect kernel modifications. The motivation for RDMA is to allow a NIC to directly place received data in the target buffer without kernel intervention or host data copy. One can certainly argue how and whether this should be done and whether there should be a protocol independent mechanism for RMDA. However if the received data must be processed by the kernel, it might as well be processed in order using the conventional API without any changes. One could argue that some small performance gains could be had by doing kernel modifications, but my guess that this is way insufficient to justify modifying the API. >Both FDDI and FC have selected a CRC polynomial ignored >by your proposal. Ethernet, FDDI, and FC all use the same CRC polynomial. When encapsulating a CRC within an ethernet frame, it really doesn't make all that much difference what polynomial is used, the only important requirement is that it be relatively prime to the ethernet CRC polynomial so that coverage is additive, not redundant. >Recommending, suggesting, or offering a feature of frame >alignment and out of sequence processing rend the TCP API. There are better >alternatives to your proposal in SCTP, so I fail to see the justification. I agree that there are somewhat easier ways to accomplish the objectives with SCTP, but my goal was to show that it can be done with TCP. The justification is that there are customers that want to buy TCP. In the short term there are some fundamental infrastructure issues that favor TCP. Things like the availability of well tested reference implementations, availibility of test, design, and QA tools that understand TCP, availability of more engineers and other professionals that understand TCP. Ultimately in the long term, it may make sense to support both TCP and SCTP and let the market place decide which will prevail. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15644; 2 Oct 2000 8:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA27654 for ips-outgoing; Mon, 2 Oct 2000 07:54:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA27636 for ; Mon, 2 Oct 2000 07:54:38 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id NAA54364 for ; Mon, 2 Oct 2000 13:54:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id NAA80032 for ; Mon, 2 Oct 2000 13:54:05 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.00415C9B ; Mon, 2 Oct 2000 13:53:55 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 13:05:10 +0300 Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, That is a nice and very long response. But what was the question? Julo "Douglas Otis" on 29/09/2000 22:21:32 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Julo, You would not authenticate exclusively from a single token out of the blue. There must be an additional layer of authentication that indicates IP:Port[] x is customer 'y' and today they wish to connect to target z[]. Only in the most simplest configurations would a flat file provide this information. You could easily have two servers that manage the customer and target databases separately and join them using a 'Key.' This join would be a database construct and not a SCSI. This additional management layer will share with both ends and not pass over SCSI transport. With this management layer, binary representations are adaquate and provide far less information to those wishing to crack data. DNS, if used, would be run by a management layer and not a transport layer. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Friday, September 29, 2000 6:15 AM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > I would add that we wanted the path to be an additional > identifier that the > target could use > to determine what collection of LUs to present to the initiator. > > Julo > > csapuntz@csapuntz-u1.cisco.com on 23/09/2000 05:00:33 > > Please respond to csapuntz@csapuntz-u1.cisco.com > > To: "IP Storage" > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > Just for clarification... I was not proposing to add the extended > URL scheme for the transport spec. It isn't necessary. > > In this thread, Doug makes an excellent point about LDAP being a > superior mechanism for describing how to connect to the > storage. Directory services such as LDAP, as Doug has pointed out, > will be critical to managing large quantities of storage. A host can > ask such a directory service for a list of storage devices it should > mount and how to connect to those storage devices. The query against > the directory server that returns this information can be based on > machine ID, user ID, operating system ID, or even the owner's > birthday. One of the things discovery will end up doing, no doubt, is > defining LDAP schemas that describe how to connect to storage > (i.e. use SCTP or TCP, what port, what target name, what LUN, what > WWN, how to authenticate, etc.). > > However, there is one place where the transport protocol has to define > a name: third party commands. There needs to be some kind of global > name which the initiator can pass to the target. The name must be > distillable into a string. The target must understand the name and > be able to use the information to establish a connection to > another target. > > One could say that the string that is passed is not specified by the > standard but instead specified by some management software. I think > this will lead to poor interoperability. > > The SCSI URL-type name is the current proposal for target name. > > Why is SCSI target name made up of a hostname + a path? Why is the > hostname + path passed on connection setup? There are two reasons. > I think NAT and IPv6 makes passing hostnames rather than addresses in > protocols more desirable. Hostnames can be re-resolved as you cross > addressing boundaries. The path is there so that the name can > support multiple targets behind a single IP address without having > to add entries to the DNS server. At Cisco, for example, > I have no control over the local DNS servers and cannot > add DNS entries for the ATAPI DVD and floppy in my computer. > > _Costa > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03097; 29 Sep 2000 10:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA04759 for ips-outgoing; Fri, 29 Sep 2000 10:19:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA04751 for ; Fri, 29 Sep 2000 10:19:24 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id QAA103698; Fri, 29 Sep 2000 16:18:50 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id QAA131834; Fri, 29 Sep 2000 16:18:45 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256969.004E9C3B ; Fri, 29 Sep 2000 16:18:38 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: Mark Burton cc: ips@ece.cmu.edu Message-ID: Date: Fri, 29 Sep 2000 17:16:16 +0300 Subject: Re: iSCSI query Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Mark, Sorry for the delayed answer. 1. A response with iSCSI status nonzero 2a. same as 1 - sense will contain details 2b. a command with iSCSI errors will not reach execution 3. There are no more document but (I hope!) next version of the draft will have more information. Also the archives of this mailing list if you have the patience to go through them until you reach paydirt Julo Mark Burton on 24/09/2000 17:13:54 Please respond to Mark Burton To: satran@haifa.vnet.ibm.com cc: (bcc: Julian Satran/Haifa/IBM) Subject: iSCSI query Hello Julian, I am reading the July 10, 2000 edition of draft-satran-iscsi-01.txt with view to implementing an iSCSI target. I would be very grateful if you could clarify these points. 1 - Near the end of the second paragraph of section 2.2.2 (Ordering and iSCSI numbering) it says "The target will reject any command outside this range ..." How is the fact that the command has been rejected conveyed to the initiator? 2 - More generally, how are errors at the iSCSI level communicated to the initiator? The only error-like response seems to be OpcodeNotUnderstood. 2 - Section 3.3.4 (iSCSI Status) says "The iSCSI Status field is used to report the status of the command before it was sent by the target to the LUN ..." Does an iSCSI Status = 1 always imply that the command in question has not been sent to the target because the iSCSI layer has detected a problem with the iSCSI command block? 3 - What is the format of the iSCSI sense data and what are the currently defined error codes. 4 - I have your document and the iSCSI requirements document. Are there any other iSCSI documents (or sources of information) that are available for implementors? Thanks, Mark From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03814; 29 Sep 2000 15:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA12986 for ips-outgoing; Fri, 29 Sep 2000 14:08:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel2.hp.com (cosrel2.hp.com [156.153.255.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA12798 for ; Fri, 29 Sep 2000 14:03:01 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by cosrel2.hp.com (Postfix) with ESMTP id C01979A4; Fri, 29 Sep 2000 12:02:25 -0600 (MDT) Received: from xrosebh3.rsvl.itc.hp.com (xrosebh3.rsvl.itc.hp.com [15.34.240.67]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA01429; Fri, 29 Sep 2000 12:02:40 -0600 (MDT) Received: by xrosebh3.rsvl.itc.hp.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 12:02:38 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: Matt Wakeley , IPS Reflector Subject: RE: Why FCP doesn't need RDMA? It has a better way. Date: Fri, 29 Sep 2000 12:02:35 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk This was just to point to what the real underlying requirement a number of people were expressing i.e. to preserve message boundaries in TCP segments. I will respond to the urgent pointer proposal on that thread. Somesh -----Original Message----- From: Matt Wakeley [mailto:matt_wakeley@agilent.com] Sent: Friday, September 29, 2000 8:08 AM To: IPS Reflector Subject: Re: Why FCP doesn't need RDMA? It has a better way. "Randall R. Stewart" wrote: > "GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > > > > Even iSCSI does not really need RDMA. The underlying note in > > the whole debate is really the preservation of message boundaries. > > > > If we could use one of the reserved bits in the TCP header to say, > > do the repacketize or adjust segment boundaries, and each TCP > > segment starts with an iSCSI header, there should be no need for > > another protocol (or TCP option header). > > > > I don't think you can go playing with TCP option bits. This implies > changing TCP and I don't see this in the WG charter. If bits need > to be changed then a draft needs to be written and submited to > the proper WG.. in this case I would say transport area WG... I agree. There is no need to use new option bits. Please read my draft http://www.ietf.org/internet-drafts/draft-wakeley-iscsi-msgbndry-00.txt It describes a proposal to discover message boundaries using the existing TCP mechanisms (the urgent pointer). > > > I think in some respects this is exactly what Costa has done... Now > as far as message boundary preservation, you get this for free > with SCTP.. and has Douglas as already pointed out, you could very > easily implement in SCTP the ability to copy a specific stream > into receive buffers .. i.e. tell the sender, but the data on > stream X and then as it arrives dump it directly to user buffers. Randall, please stop telling us all about the fabulous features of SCTP. We are all sick to death of hearing about SCTP. -Matt > > > This would take a bit of work, but so does the RDMA solution :/ > > R > > > The iSCSI header contains enough information in it to enable > > the recepient to determine where to put the data. > > > > Somesh > > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > Sent: Thursday, September 28, 2000 8:29 AM > > To: 'Matt Wakeley'; IPS Reflector > > Subject: RE:Why FCP doesn't need RDMA? It has a better way. > > > > > I object to mandating iSCSI use an RDMA option because: > > > > > > - (main reason) there isn't any standardized mechanism now, and I > > > don't want to hold up iSCSI while one crawls through the > > > standards process. > > > > > > - I don't think RDMA is needed. FCP doesn't use it, and it > > > works great with > > > the > > > FC protocol chips that "accelerate" FCP. > > > > Actually, RDMA is not needed in FCP because all protocol chips > > implemented perform a real peer-to-peer DMA straight to the > > data areas specified by the user's interaction with the operating > > systems allocation algorithms. The combination of the FCP/SCSI > > pointer structure, task tagging, and the FC relative offset perform the > > function you would otherwise have to use RDMA to accomplish. > > > > Bob Snively > > Brocade Communications Phone 408 487 8135 > > 1745 Technology Drive > > San Jose, CA 95110 Email rsnively@brocade.com > > -- > Randall R. Stewart > randall@stewart.chicago.il.us or rrs@cisco.com > 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16110; 2 Oct 2000 11:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA02229 for ips-outgoing; Mon, 2 Oct 2000 10:28:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA27220 for ; Mon, 2 Oct 2000 07:31:44 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Mon, 2 Oct 2000 04:33:19 -0700 Message-ID: From: Robert Snively To: "'Douglas Otis'" , Robert Snively , John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: RE: not every song Date: Mon, 2 Oct 2000 04:31:11 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, Differentiate for me if you would please: One PDU with two IUs contained: Segmented and transmitted across a link. Two consecutive PDUs with one IU contained in each: (Maybe with the first not requiring segmentation) and transmitted consecutively across a link. Since FCP could not find any performance or latency differences, I would expect that iSCSI would find similar characteristics. Bob > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Thursday, September 28, 2000 1:50 PM > To: Robert Snively; John Hufferd/San Jose/IBM; ips@ece.cmu.edu > Subject: RE: not every song > > > Bob, > > I am not sure I understand the point. Not that I would advocate such > modifications, but should these points become a deciding > factor, the service > parameter for such negotiations has already been defined > within FCP-2 as of > recently. Such a decision for removal may have been premature should > network latency argue for Word 3, bit 3 being set. > > Again, I am agnostic about this issue, but I would rather > stick with FCP > structures. Should there be attempts to provide zero copy, > adopting FCP > structures permit a greater reliance on these structure's stability. > > Doug > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > Sent: Thursday, September 28, 2000 12:37 PM > > To: 'Douglas Otis'; John Hufferd/San Jose/IBM; ips@ece.cmu.edu > > Subject: RE: not every song > > > > > > > FCP > > > offers WRITE-DATA and READ-RESPONSE structures if desired. > > > > Reminder: > > > > No, it doesn't. FCP-2 (the new and improved FCP) took the > > function out because nobody implemented it and it did not make > > sense. It would have required additional recovery mechanisms and > > it did not improve performance. > > > > Bob > > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00762; 28 Sep 2000 22:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA18518 for ips-outgoing; Thu, 28 Sep 2000 21:42:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA17155 for ; Thu, 28 Sep 2000 20:46:38 -0400 (EDT) Received: from omgw2.boi.hp.com (omgw2.boi.hp.com [15.56.8.102]) by atlrel2.hp.com (Postfix) with ESMTP id 4DB501AA; Thu, 28 Sep 2000 20:46:32 -0400 (EDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by omgw2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id SAA03862; Thu, 28 Sep 2000 18:46:28 -0600 (MDT) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 18:46:28 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: Robert Snively , "WAKELEY,MATT (A-Americas,unix1)" , IPS Reflector Subject: RE: Why FCP doesn't need RDMA? It has a better way. Date: Thu, 28 Sep 2000 17:50:27 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Even iSCSI does not really need RDMA. The underlying note in the whole debate is really the preservation of message boundaries. If we could use one of the reserved bits in the TCP header to say, do the repacketize or adjust segment boundaries, and each TCP segment starts with an iSCSI header, there should be no need for another protocol (or TCP option header). The iSCSI header contains enough information in it to enable the recepient to determine where to put the data. Somesh -----Original Message----- From: Robert Snively [mailto:rsnively@Brocade.COM] Sent: Thursday, September 28, 2000 8:29 AM To: 'Matt Wakeley'; IPS Reflector Subject: RE:Why FCP doesn't need RDMA? It has a better way. > I object to mandating iSCSI use an RDMA option because: > > - (main reason) there isn't any standardized mechanism now, and I > don't want to hold up iSCSI while one crawls through the > standards process. > > - I don't think RDMA is needed. FCP doesn't use it, and it > works great with > the > FC protocol chips that "accelerate" FCP. Actually, RDMA is not needed in FCP because all protocol chips implemented perform a real peer-to-peer DMA straight to the data areas specified by the user's interaction with the operating systems allocation algorithms. The combination of the FCP/SCSI pointer structure, task tagging, and the FC relative offset perform the function you would otherwise have to use RDMA to accomplish. Bob Snively Brocade Communications Phone 408 487 8135 1745 Technology Drive San Jose, CA 95110 Email rsnively@brocade.com From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02639; 29 Sep 2000 8:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA00232 for ips-outgoing; Fri, 29 Sep 2000 07:49:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA00224 for ; Fri, 29 Sep 2000 07:49:07 -0400 (EDT) Received: from yp_portable (slip-32-101-27-230.ma.us.prserv.net [32.101.27.230]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDM1JD; Fri, 29 Sep 2000 04:46:28 -0700 From: "Y P Cheng" To: Subject: RE: Status summary on multiple connections -- iSCSI flow control Date: Fri, 29 Sep 2000 04:46:46 -0700 Message-ID: <000101c02a0a$f1cda680$e61b6520@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 Importance: Normal In-Reply-To: <200009281840.LAA10602@ha10nwk.EBay.Sun.COM> X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: Robert Snively > To: "'David Robinson'" , Robert Snively > When there is no more dynamic space left and all the > pre-allocated locations for a particular ITL nexus are also full, > the next command gets a queue full indication returned. Because of > the dynamic assignment area, this will typically be rare in a properly > configured system. The initiator then resends the command and all > subsequent commands after at least one command comes back completed, > indicating that at least one (and probably a whole stack more) slots > are again available. Note that there is a possibility that commands > that are inflight and have ordering constraints may be accepted out of > order, a question that has caused lots of agonizing, but is apparently > reasonably well managed by most file systems today by the selective > use of ordering only for blocking boundaries of a particular logical > stream of commands. Bob, I liked everything you said. The SCSI protocol itself provides lots of mechanisms for a target device to manage its resources. An initiator must commit its resources for all the requests whether they are in queue, inflight, or being processed by a target. However, your description of out-of-order execution begs more clarification of retransmission and the flow control to minimize it. While it is true that the file systems ensure all outstanding commands can be executed out of order -- because if not careful a target device may inadvertently sort the commands to improve performance -- there is no out-of-order execution problem in a 1394 or SCSI adapter. This issue of commands inflight and executed out-of-order is new to fibre channel and iSCSI only. Having said that, the retransmission is more important to this discussion. For a SCSI adapter, the queue-full status prevents any new command being sent until at least an outstanding one is completed. There is no command inflight on SCSI bus. On 1394, the target will never fetch another ORB when there is no room. If the fetch fails, a target device has the option to retry or terminate. A 1394 target can prefetch multiple ORBs and decide to sort them or execute them in order. For fibre channel and iSCSI, no flow control can prevent a command being executed out of order due to a non-zero probability of having a command not arriving at the target. This is due to busy network, packet CRC error, the NIC receive-buffer full, or the TCP window closed. Although not necessary, if one insists, a CmdRN can be placed in the PDU to ensure sequentially. Fibre channel uses BB-credit to eliminate the possibility of the adapter receive-buffer full. But, it can not prevent a frame from having a CRC error and being dropped by a target. Only timeout by an initiator will detect the loss of a command. As you have said, out-of-order execution is NOT a problem. The SCSI command can be transmitted by a single frame or PDU. The retransmission is trivial once the loss of frame or PDU is detected. This is not the case for data PDUs. The key question is what is the granularity of the retransmitted data. For iSCSI, long latency delay and loss of PDU is a matter of life. There is a statistical possibility for hundreds of initiators sending PDUs to a target at the same time. No flow control can prevent the PDU loss due to busy host bus that causes receive-buffer full. In addition to busy host bus, a data PDU can be lost due to, CRC error, TCP window closed or the switch and router being too busy. The lost PDU must be retransmitted. In TCP/IP, an ACK is needed for every PDU, the granularity of retransmit is the smallest. However, the ACK traffic is the highest which causes even more congestion with the highest possibility of the missing of an ACK. The wait of an ACK on a network with long delay is very costly. To stream a large amount of data without a huge buffer on a long-delayed network requires DMA/RDMA. The tradeoff between a greater granularity of retransmit and les s ACK traffic is truly the challenge of this WG. The ACK-0 concept from the fibre channel and the retransmit of a single sequence could be helpful to iSCSI. This WG assumes TCP/IP is such a reliable and proven technology that its flow and congestion control will solve all the problems magically. It spends all its effort trying to fit iSCSI into a stream-oriented TCP/IP implementation. I believe it would be helpful if this WG discusses the need of executing 100,000 IOs per second on an iSCSI adapter and the issue of the granularity of data retransmission on long-delayed network with a high probability of loss of a PDU. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32676; 28 Sep 2000 17:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA10137 for ips-outgoing; Thu, 28 Sep 2000 17:05:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub1.almaden.ibm.com (mailhub1.almaden.ibm.com [198.4.83.44]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA10123 for ; Thu, 28 Sep 2000 17:05:41 -0400 (EDT) Received: from catalina.almaden.ibm.com (catalina.almaden.ibm.com [9.1.24.58]) by mailhub1.almaden.ibm.com (8.8.8/8.8.8) with ESMTP id OAA38876; Thu, 28 Sep 2000 14:06:00 -0700 Received: (from dfsmith@localhost) by catalina.almaden.ibm.com (AIX4.3/UCB 8.8.8/8.7) id OAA36398; Thu, 28 Sep 2000 14:05:09 -0700 From: Daniel Smith Message-Id: <200009282105.OAA36398@catalina.almaden.ibm.com> Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: rsnively@Brocade.COM (Robert Snively) Date: Thu, 28 Sep 2000 14:05:09 -0700 (PDT) Cc: ips@ece.cmu.edu In-Reply-To: from "Robert Snively" at Sep 28, 2000 10:04:55 AM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Robert Snively wrote: > > Textual names are practically useless as identifiers, as witness the > challenge of finding "Bob Smith" in an American phonebook. Every This is not quite the same problem. 1) You don't get a choice as to what you call your URL ("Robert Smith"), because the network naming guys will point out another another Robert.Smith.com already owns that name and you can't have it. 2) The URL is really an arbitrary length base-40ish numeral. The infrastructure is already in place and well tested with HTTP URLs. 3) Once the URL is decoded ("Robert Smith, in Seattle, on Broad St, next to the donut store") it becomes a numeric address---probably a 32-bit integer on most systems. Then you can address the TCP reference number plus the LUN to access your device. 4) In your favor (although it isn't so noticable for men) names can change when circumstances change. However, if you don't receive notification of the change, you probably shouldn't have been trying to resolve the name anyway. B-) (On a side note, the prospect of "redirectors" during login has been brought up, and seems very sensible, if not indispensible.) > Fibre Channel and Ethernet have taken the same approach at the > Port/Node WWN and MAC address level. I'm betting that the committee/person that chose 32 bits for IP wishes they'd/he'd chosen more bits. Thankfully, the naming system allows almost unlimited expansion of the IP-bitspace behind firewalls. People in the Bay Area are used to the threat of their phone numbers being changed en-masse. Luckily the DNS is flexible enough to cope with (pretty much) any amount of oversubscription to limited numbers. > Use of a URL is a virtualization of the underlying name structure > and should not be implemented except as a temporary convenience for the > highest level client who is constrained to a very small search > subset by his login authorizations. The virtualization is always > at risk of security breaches. True. But probably not significantly more so than normal. If your secure e-commerce site is name-hijacked, that's a major problem too. > In the storage environment, the underlying structure is of vital > interest. Any given data is present on specified physical and > logical units and consistency of the data is maintained on the > basis of an intermediate view of the lower level addressing structure > which must also have the proper uniqueness guarantees. This is true for the SCSI environment. The underlying structure is irrelevant in the networking world. All that matters is time-to-the-correct-data (or cost-to...). In fact, it may be important to completely /remove/ any "intermediate views of the lower level addressing" because that view was, say, knocked out by a giant lizard and your current path to the data is being routed _around_ Tokyo now. For example, take two bridges into a legacy SAN. There are multiple paths into the SAN, at different IP addresses. One of the bridge connections is chewed up by a mutant space goat, so now all traffic entering the SAN has to pass though a bridge with a different IP address. With a named scheme, this is a trivial (and automatic) repair (in fact, the DNS would mention that the name is associated with both IP addresses, so if one's down, use the other). With an IP address, all initiators accessing that SAN would need to update their storage pointers. (It's not difficult to solve, but I'm too lazy to want to program thousands of lists of iffy failover names.) > Address it with a URL on the browser if you wish, but understand that > you are talking to a data block on a single storage unit attached to > a network server having a specified IP address mapped to a > registered MAC address. Exactly. Once the URL is resolved and opened, the only reference you needs is a TCP stream and a LUN. (Or an SCTP connection and a LUN.) If you/we feel really strongly about this, then we could specify a maximum length of URL at 255 bytes (plus NUL terminator). That would give you a numeric value. (65280 bits, with admittedly lots of redundancy.) Daniel Smith. P.S., who do I call to remove mutant space goats from our data closet? P.P.S., I haven't made any arguments against using a pure numerical system. But any such system will need to be of similar complexity as a URL scheme. When including methods for redirection around firewalls, NATs, redundant IP addresses, dynamic load balancing and management side-nodes (similar to MX records in DNS), it gets horrendous. My brain hurts just thinking about it. B-P -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04158; 29 Sep 2000 17:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA18100 for ips-outgoing; Fri, 29 Sep 2000 16:09:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA18083 for ; Fri, 29 Sep 2000 16:09:31 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id WAA83912 for ; Fri, 29 Sep 2000 22:08:46 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id WAA157634 for ; Fri, 29 Sep 2000 22:08:46 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256969.006EA697 ; Fri, 29 Sep 2000 22:08:36 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 29 Sep 2000 23:06:10 +0300 Subject: RE: Status summary on multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I think that RDMA out-of-order processing is important only in order not to have data piling up in adapters. It does not mean that data gets really "committed". The commands can be executed in or out-of-order - this is a pure SCSI story. But if they have to be executed in order the ordering provided by SCSI will enable it. And as you and others have pointed out the same mechanism is good for both ordering and flow control. As far as I understand it windowing - although more expensive - is a better technique over a wide variety of latencies than credits. And BTW we even considered credits for a "prefetching mechanism" for chained commands but where told that those are very much "out of fashion" (i.e. not worth speeding up on Elefants). Julo David Robinson on 29/09/2000 21:25:52 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Status summary on multiple connections > I am left with the following impression as to what was indicated here: > - In general, command ordering is not relevant > - If the initiator filesystem detects an ordering dependency, it will wait > until outstanding commands are complete before issuing the dependant > command. > > This may be a reasonable means of operation for the disk world. It is > woefully inadequate for the tape world, as follows: > - In general, command ordering is crucial - out of order command processing > will lead to data corruption. > - This would require the initiator backup application to block on completion > of every single write command of a backup operation before issuing the next > command. > > If this blocking were performed, both the throughput and capacity of a tape > device/media would be negatively impacted by an order of magnitude or more. > This would occur even assuming an instantaneous transport. I am hearing different stories on the issue of ordering. One side is pushing hard for techniques that will allow out of order execution using various RDMA techniques. This clearly states for a certain class of devices (e.g. tapes) ordering is crucial. I thought this problem was already solved at the SCSI layer through the use of ordered commands which in general are not used for disks but always used for tapes? Since FC will reorder this has to be a solved problem. Would not an initiator talking to a tape target simply set the ordering flag? Lastly, for a TCP based connection ordering can easily be made a non-issue, simply don't try to process segments out of order. I will defer to a transport expert, but I believe processing TCP segments by an application out of order might cause problems. In particular since the out of order segment is not ACKed until after the missing segments arrive, they can be retransmitted multiple times. SACK helps this but does not guarentee that segments will not be retransmitted. So to process out of order segments the application must maintain a list of which segments have been processed as well, yuck! -David From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00438; 28 Sep 2000 20:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA13677 for ips-outgoing; Thu, 28 Sep 2000 18:41:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA13668 for ; Thu, 28 Sep 2000 18:41:53 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id C1783576; Thu, 28 Sep 2000 15:41:39 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA11126; Thu, 28 Sep 2000 15:41:43 -0700 (PDT) Message-Id: <4.2.2.20000928152047.00b7a940@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Thu, 28 Sep 2000 15:41:03 -0700 To: Stephen Bailey , ips@ece.cmu.edu From: Michael Krause Subject: Re: New List: rdma@cisco.com: to discuss RDMA In-Reply-To: <10009282122.AA05090@candide.cs.uchicago.edu> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:22 PM 9/28/00 -0500, Stephen Bailey wrote: > > However, I would support structuring the spec so that an RDMA > > transport mechanism could be used underneath (I guess that's > > motherhood). > >Not necessarily. > >You have to ask the implementors (particularly the hardware >implementors), what sort of optional RDMA proposal they'd be happy >with. My answer is none. It's mandatory or not at all. I disagree. We had a similar debate within InfiniBand on just this subject and it really came down to implementation specific requirements. The IB host channel adapters were required to support all of the RDMA semantics in order to insure the base functionality is ubiquitous but the target channel adapters (think of this as a NIC) were allowed to support all, some, or none of the RDMA operations. For example a storage adapter might accept SEND operations which contain the SCSI command. It could then issue RDMA READ operations to obtain the data from the source. The same would apply to issuing RDMA WRITE operations in response to a data read operation. This can be implemented in a fairly thin resource adapter with minimal complexity and one might say is quite similar to any number of existing SCSI or FC implementations. The benefit of RDMA will vary depending upon how much functionality and resources are available on each side of the communication and whether the usage is uni- or bi-directional. It should be noted that RDMA and many of the storage implementations still "lock" up resources for potentially long periods of time which can lead to application-level resource contention and throughput degradation. This is a problem that no one has really addressed - they've mitigated the impact but as bandwidths increase the impacts of such resources caches and application cache contention will worsen. I don't view RDMA or any of these implementations as panacea solutions for this overall resource usage problem. >The reason for using RDMA is to make the implementation of iSCSI >easier in hardware. If there are implementations which do not support >the RDMA protocol, then your hardware implementation will have to >support both the `easy path' (using RDMA) and the `hard path' (no >RDMA). If you have to implement the hard path anyway, there's no >point in implementing the easy path. Debatable. Obviously, iSCSI does require some form of option negotiation at session / connection establishment and RDMA is one of the options that should be included in this negotiation. The benefits of RDMA are associated with the ability to reduce the amount of buffering required within a solution at the various points along the way and deliver zero-processor copy placement and control. These can be implemented in other ways as others have pointed out and thus the cost / benefit needs will vary. It should be noted that other protocols (e.g. SVP) and perhaps the ability to implement higher-level functionality may be more easily implemented in the future if a general-purpose RDMA solution is defined. This is why creating such a solution and enabling iSCSI to take advantage of such implementations is critical at this point in time. >The argument that you could make the hard path infrequent and >implement it in software doesn't wash in this case. It just takes one >implementation that doesn't do RDMA to slow your system to a crawl, >and the competitor who only implemented the non-RDMA path makes you >look like a fool. Let the market decide whether that one implementation should survive or not. Most likely it will die as others will have been smarter in their designs and functional selection. >Fundamentally, RDMA has to be either adopted or punted. Of course, >I'm happy to have somebody prove this statement wrong. The issue is one of the level of gray not black and white. Mike From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32674; 28 Sep 2000 17:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA09404 for ips-outgoing; Thu, 28 Sep 2000 16:48:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA09368 for ; Thu, 28 Sep 2000 16:47:49 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA14709; Thu, 28 Sep 2000 13:34:40 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id NAA04797; Thu, 28 Sep 2000 13:32:26 -0700 From: "Douglas Otis" To: "Robert Snively" , "John Hufferd/San Jose/IBM" , Subject: RE: not every song Date: Thu, 28 Sep 2000 13:50:16 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, I am not sure I understand the point. Not that I would advocate such modifications, but should these points become a deciding factor, the service parameter for such negotiations has already been defined within FCP-2 as of recently. Such a decision for removal may have been premature should network latency argue for Word 3, bit 3 being set. Again, I am agnostic about this issue, but I would rather stick with FCP structures. Should there be attempts to provide zero copy, adopting FCP structures permit a greater reliance on these structure's stability. Doug > -----Original Message----- > From: Robert Snively [mailto:rsnively@Brocade.COM] > Sent: Thursday, September 28, 2000 12:37 PM > To: 'Douglas Otis'; John Hufferd/San Jose/IBM; ips@ece.cmu.edu > Subject: RE: not every song > > > > FCP > > offers WRITE-DATA and READ-RESPONSE structures if desired. > > Reminder: > > No, it doesn't. FCP-2 (the new and improved FCP) took the > function out because nobody implemented it and it did not make > sense. It would have required additional recovery mechanisms and > it did not improve performance. > > Bob > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id ah11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11027; 1 Oct 2000 6:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA01022 for ips-outgoing; Sun, 1 Oct 2000 05:32:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA01017 for ; Sun, 1 Oct 2000 05:32:39 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id LAA59738 for ; Sun, 1 Oct 2000 11:32:08 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id LAA35246 for ; Sun, 1 Oct 2000 11:32:08 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.00345F3C ; Sun, 1 Oct 2000 11:32:02 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 12:29:43 +0300 Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk There is no statement about managing names in the draft, nor in Daniel note. Julo "Douglas Otis" on 28/09/2000 19:52:43 Please respond to "Douglas Otis" To: Julian Satran/Haifa/IBM@IBMIL cc: Daniel Smith/Almaden/IBM@IBMUS, ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Julo, The point missed is in how names are managed. You wish to include name management within the transport where it does not belong. SCSI third party commands presently map in a binary fashion. In some cases, this binary to name translation should not be sent by the client. Relegating this function to LDAP server as example, both the client and the device server depend on an external management already in use. By attempting to encompass name management rather than leveraging off existing standards, security is weakened and such standard undergo revisions unrelated to transport as naming conventions, associated attributes, and points of definition change. Doug > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Thursday, September 28, 2000 3:11 AM > To: Douglas Otis > Cc: dfsmith@almaden.ibm.com; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > Doug, > > There seem to be practical and philosophical issues that you consistently > ignore. > Textual names are more stable than underlying network addresses and thus > better > for applications (I assume that your card starts with your name not phone > number and > that the former has changed less than the later). > I the iSCSI realm - and that is the only one under scrutiny - a pertinent > observation was > made by David Black a century ago during our (too) long naming discussions > - that a scheme that will allow the third party commands to make the name > to address conversion > at use time is better than one that requires this mapping to be made ahead > of time. > Unfortunately that is not yet possible with SCSI but we are informed that > it soon be. > And for authentication the textual address in login is yet another mean to > prevent > errors and/or unauthorized access if the target chooses to use it. > > Julo > > "Douglas Otis" on 19/09/2000 06:55:29 > > Please respond to "Douglas Otis" > > To: Daniel Smith/Almaden/IBM@IBMUS, ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > Daniel, > > SCSI is not HTTP and does not provide a human interface as a basis for > communication. Ownership of naming abstractions vary from the perspective > of the client to that of the provider. The provider may not wish to share > management of these naming abstractions with the client and vise-versa. > Rather than attempting to keep a permanent naming abstraction as globally > shared between both provider and client, these abstractions should be > meaningful to the human dealing with the names. For the client it may be > Tom, Dick, and Harry, and for the provider it could be Shelf A, E, and N. > By keeping these abstractions independent of the transport layer, > it allows > each end to use such means as LDAP servers to join these views in a > coherent > fashion. Your IT manager will insist you not concern yourself with the IP > of your workstation, as he has control of these assignments through DHCP > and > LDAP servers, the same should be true for a networked version of SCSI. > > Do not forget the large task of authentication that may also be managed by > either the provider or the client. By keeping the transport as plain as > possible, these naming abstractions can become as complex as required > without affecting the transport specification. Implanting names into SCSI > is a mistake and should be corrected as it limits this valuable freedom. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Daniel Smith > > Sent: Monday, September 18, 2000 5:25 PM > > To: ips@ece.cmu.edu > > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > Just in case people are wondering what this proposal is/was (the mailing > > list seems to have exploded quite a bit since it was released > > back in May), > > here it is again, with a handful of [modifications] and [comments]. > > -- > > IBM Almaden Research Center, 650 Harry Road, San Jose, CA > 95120-6099, USA > > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > > > > > > > > ---------------------------------------------- > > > > Subject: iSCSI naming > > Date: Mon, 8 May 2000 15:01:17 -0700 (PDT) [mods 18 Sep 2000] > > > > iSCSI naming requirements (first [and a half] iteration) > > ------------------------- > > > > It is important to have a consistent naming scheme as the basis of a > > discovery system for the SCSI over TCP protocol. A basic set of > > requirements is below, and a method is suggested. > > > > Requirements > > ------------ > > > > A name identifies a [SCSI service delivery port], not a LUN. > > notes: There may be many thousands of LUNs (and virtual LUNs) on a > > connection. > > why: The LUNs given to each connection are likely to be dynamic > > and assigned > > by the iSCSI manager. The iSCSI names are more static and long-lived, > and > > must be persistent even if the iSCSI manager fails. > > notes: A separate naming scheme will document the LUN discovery and > naming > > mechanism. [query to LUN 0 to get list of LUNs] > > > > The namespace need not be fundamentally [systematically] enumerable. > > notes: FC can be enumerated, the Internet cannot (practically). > > why: iSCSI will be deployed on untrusted networks. Having > > devices that can > > be enumerated are subject to systematic attack. Also, many > > devices will be > > behind firewall machines, and not even contactable until a login > procedure > > has taken place---holes in such an enumerated namespace are not > > acceptable. > > notes: Devices on a LAN can be enumerated, and this is acceptable. > > notes: Enumeration (listing) by a 3rd party is acceptable. > > > > The names should be as static as possible. > > notes: Both in time and location. > > why: A name given for one iSCSI device really ought to resolve to the > same > > device every time, from everywhere. [device here refers to a > SCSI service > > delivery port, of course the LUs should stay the same too, > barring things > > like security lock-outs] > > > > The name should not be tied to the network technology. > > notes: For example, ethernet MAC address or IPv4/v6 address. > > why: To cope with future networking systems, the name of the > iSCSI device > > should not rely only on aspects of the transport that are liable to > become > > obsolete, or that are not static enough. > > notes: IPv4 addresses may be acceptable on a controlled private network, > > however, these names should not be the canonical names as > viewed from the > > outside. [major problems with NAT if that happens] > > > > The naming scheme must be reliable. > > notes: If parts of the network go down, the names must remain intact. > > why: iSCSI may form part of the networking infrastructure, meaning that > > after power loss, for example, the iSCSI devices will be required > > to bring up > > the rest of the network. This implies that the devices should be aware > of > > their own name, if necessary for network stability. It also > implies that > > any name service should have redundancy. > > > > The naming scheme should be nice to look at. > > why: Long strings of unformatted numbers are horrible. [and prone > > to typos] > > > > The iSCSI device being addressed should be maskable from observers. > > why: The device name may contain valuable (secure) information, > > so it must be > > possible to hide this information from prying network interfaces. > > notes: The URL scheme manages this, by separating the hostname from the > > device name. > > > > Proposal > > -------- > > We should follow the URL naming scheme for iSCSI devices, thus: > > scsi:/// > > Where > > :=|: is the TCP [destination] identifier... > > is the DNS or DDNS host name of the iSCSI device and > > is the TCP port number to connect to. > > > > And [ should be ] > > :=|/|? is the device > identifier > > is a text identifier for the device > > :=|? is an argument for the > device > > [argument escape character was '&' in the original] > > > > The element is reasonably well defined. However, I would prefer > to > > keep the element more free-form. The > scsis:/// is > > also a straightforward extension. [to secure (SSL) iSCSI] > > > > The identifier specifies an iSCSI TCP connection > > identifier only. > > This may provide access to one or many LUNs that flit in and out of LUN > > space. > > > > One caveat is that some (most?) iSCSI devices will not have > > access to DNS. > > Fortunately, devices (targets) do not need to [query the > network to find] > > their names except in the case of trying to resolve another device > > (target)---i.e., only initiators need DNS resolution. > > > > Here are some fun examples of the things I would want to do as a storage > > manager. > > > > The "backup" program does a simple block-to-block copy from one > > iSCSI device > > to another... > > backup scsi://raid.acme.com/lun12 scsi://tape.acme.com/id25 > > Of course, a more useful operation may be... > > flashcopy scsi://disk.acme.com/12/2 scsi://disk.acme.com/13/2 > > where the pathnames are ASCII numbers. (They may even be the WWN or LUN > > value.) > > > > This doesn't address the issue of security. When we have more of those > > issues resolved, we might want to issue a command like this... > > kcopy scsis://secure.acme.com/raid12/set2?key=12345?auth=dfsmith > > ...so that the part, containing the security tokens, is > > never passed > > in the clear over the network. > > > > It may be possible for a manager to transparently interpret the > arguments > > or pathname into a LUN list in a storage farm. > > iscsi attach scsi://farm.acme.com/tray17?LUN=0x17?LUN=0x18 > > /dev/scsi2 > > iscsi attach scsi://san.acme.com/5A/27/23/99/B7/A2/01/00 /dev/sda > > > > The Text/Response commands in the iSCSI Internet-Draft might be > > accommodated > > thusly: > > iscsitext scsi://tape.scsi.acme.com/cart5 "com.ibm.retension:yes" \ > > "com.ibm.eject:now" > > > com.ibm.retension:ok retensioning > > > com.ibm.eject:ok scheduled ETA 2:30 > > > > Secure enumeration might be done in the following way, with peculiar > > examples of possible Un*x integration. > > iscsi find_domain_manager > > > iscsi.acme.com > > httpsget https://iscsi.acme.com:99/root&auth=dfsmith&key=1234 > > > > > > > > > scsi://unit1.raid.acme.com/ > > > disk1 disk2 disk3 disk4 disk8 disk9 > > > > > > > > > > > > scsi://bigtape.acme.com/racf4 > > > cart57 cart58 > > > > > > > > > > > > scsi://ibmdeskstar75.dfsmith.acme.com/lun1 > > > > > > > > iscsi mount scsi://ibmdeskstar75.dfsmith.acme.com/lun1 /mnt/mydisk > > > /mnt/mydisk: 73564224 blocks available > > tar -cf 'scsi://bigtape.acme.com/racf4/cart57&user=dfsmith' ~/* & > > > [3] 4291 > > runsocks dd if=/dev/zero of=scsi://dfsmith.blocks4less.com/disk1 & > > > [4] 4293 > > And so on. (Note: I am not an XML expert.) > > > > [Additional notes: > > > > The decision to resolve devices in the host name or in the > device name is > > really up to the implementer. Some companies may already have > systems in > > place for handling the DNS while they have no existing SAN > > infrastructure. > > Thus the decision to name a particular service delivery port > > scsi://disk26.farm-b.acme.com/ > > or > > scsi://farm-b.acme.com/disk26 > > is up to the system designer. > > > > The iSCSI URL names a bundle of LUs (one iSCSI "connection"). People > love > > to point out that this is a problem if you have multiple entrance > > ways into > > the same SAN, and the same device (say WWN=0x1234) has different LUNs > (say > > LUN=5 on iscsi://scsi.acme.com/bridge1 and LUN=477 on > > iscsi://scsi.acme.com/bridge2). So don't do that. Or do that > > and just live > > with it. Or make sure the different initiator/bridges into the SAN > assign > > the device the same LUN. (There isn't really a problem using a LUN in a > > global context, as long as it's always attached to its service delivery > > port.) (Having to query every LUN on a port to get the list of WWNs > ought > > to be addressed by T10, in the same way they covered the "query > > LUN 0 to get > > a list of LUNs" problem. IMO!) > > > > The security aspects bring up the spectre of LUs being masked out for > > different iSCSI initiators on different parts of the net. This is fun > and > > nothing to worry about. > > > > These URL names are used in several places, most of which are not > > related to > > the iSCSI transport itself. (Likewise, html is not particularly > > related to > > HTTP, nor SMTP to metamail.) > > > > First by users (or users by dint of clobbering an html hyperlink) who > want > > to attach to iSCSI devices. The names are descriptive and are > > malleable to > > all sorts of marvelous manipulations by the sysadmins. > > > > Second, by directory servers, who can deliver a list of the > URLs (and the > > LUNs that are accessible therein) to the iSCSI clients. > > > > Third, by iSCSI itself for the 3rd party copy command. In this > case, the > > URL specifies the destination SAN and the target LUN is carried in the > > additional data field. > > > > ] > > > > > > > > > > ---------------------------------------------- > > > > Jim Hafner/Almaden/IBM wrote: > > > > > > I've seen Daniel's proposal before and I still haven't figured out the > > > context in which such names are to be used. > > > > > > Given that, I'm personally a bit uncomfortable with any naming > > convention > > > that uses LUN values in a global context. These have no meaning as > LUNs > > > are host-specific values. Even worse, LUNs are addresses, not names! > > > > > > I have no problem with WWNs in a global context as that is what they > are > > > for! > > > > > > A subtle difficulty, is that a WWN does not give a host an > > address (either > > > for the target or the LU). Even in SPC-2, EXTENDED COPY's > > Identification > > > Descriptor Target Descriptor Format (sic?) where WWNs are used, says > > > "instructs the copy manager to locate a target and logical unit that > > > returns a device identification VPD page ..." but gives no > hints on how > > > that should be done (e.g., walk the bus and do INQUIRY to > > everything...?). > > > > > > The "hostname" and DNS provide a canonical method for getting an > address > > > for a name, so that part is OK. The only defined way to get > an address > > > (LUN) for a LU WWN is by exhaustive INQUIRY to each LU at a > > given target. > > > > > > On the other hand, I can envision how some of these things > > might work, in a > > > well-coordinated and well-managed environment: > > > > > > There is a managing application which has the dual > responsibility of > > > managing the targets for their host/LUN mapping > > configurations and also > > > serves as the "walk the bus" discovery resource for hosts. > > > > > > When the manager (human?) determines the distribution of > > host/target LU > > > resources (after inventorying both hosts and target LUs), the > > > application will send to the targets whatever host/LUN > > Mapping (access > > > controls?) commands are necessary to configure them. Later, > > when a host > > > boots it queries the application to get its "walk-the-bus" > > services. In > > > this, it minimally gets from this application a list of > targets that > > > have LUs to which it should have access. There is no > REQUIREMENT at > > > this point for the application to give the host anything > more as the > > > normal SCSI LU discovery process per target kicks in. On the other > > > hand, I can see some value in the host getting more information > about > > > the LUs it will see at each target. In that case, the host could > get > > > from the application a list of targets and LU identifiers > > (these could > > > be LUNs or WWN or both, as you suggested). The important > > point here, is > > > that these LUNs are valid only in the context of the > > requesting HOST and > > > are not global! > > > > > > Note that this process requires clear coordination between > > the "target > > > configurator" and "host discovery of targets". > > > > > > > > > Given an appropriate context for a URL scheme, I'd make two > > modifications: > > > 1) drop the assumption that no LUN means LUN=0. If no LU > > qualifier (e.g., > > > no LUN or WWN (or other) is provided), then LUN=0. In all > > other cases, the > > > alternative names should be sufficient to identify a LU, > > though, as above, > > > finding a LUN address may require extra work. > > > 2) add ?ProxyToken= as an additional option here (this > > coordinates > > > well with the changes to EXTENDED COPY approved in the context > > of the SCSI > > > access controls). > > > > > > Well, that's another two cents! I was hoping to save some of this > > > discussion until the protocol gets worked out, but .... > > > > > > Jim Hafner > > > > > > > > > csapuntz@cisco.com@ece.cmu.edu on 09-17-2000 11:11:24 PM > > > > > > Sent by: owner-ips@ece.cmu.edu > > > > > > > > > To: IP Storage > > > cc: csapuntz@cisco.com > > > Subject: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > > > > > > > > > > Here is a proposal to extend the iSCSI URL scheme to support LUNs and > > > WWNs. It was originally proposed by Daniel Smith of IBM Almaden. This > > > extension is not required for the iSCSI transport protocol, but may be > > > useful for discovery and management. > > > > > > A URL for the target has the following form: > > > > > > scsi://hostname/path/with/ > > > > > > A URL referring to a specific LU has the following form: > > > > > > scsi://hostname/path/with/?LUN=lunnumber?WWN=wwnnumber > > > > > > If no LUN= term appears in the URL, then LUN 0 is assumed. > > > > > > The WWN= term is optional. If present, the party should > > > verify that the WWN in the LU's Device Identification Inquiry > > > Page correspojnds to the WWN. > > > > > > Note, the following URL: > > > > > > scsi://gsg.cisco.com/tape/?WWN=0a050a4bcdefa > > > > > > refers to LUN 0 of target scsi://gsg.cisco.com/tape/. After > > > connecting, the initiator should verify that the WWN > > > of the LU is 0a050a4bcdefa. > > > > > > -Costa > > > > > > > > > > > > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05039; 29 Sep 2000 22:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA29089 for ips-outgoing; Fri, 29 Sep 2000 22:04:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA29084 for ; Fri, 29 Sep 2000 22:03:59 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 22:03:29 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101033@corpmx9.isus.emc.com> To: dotis@sanlight.net, Black_David@emc.com, matt_wakeley@agilent.com, ips@ece.cmu.edu Subject: RE: Enough on TCP vs. SCTP, please Date: Fri, 29 Sep 2000 22:03:28 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > I understand how you could hold that view. To implement such a definition, > you would need to make changes to the TCP API as well as to the TCP stack to > allow messages to be marked. What changes? Be specific. Answers that do not contain specific references to RFC 793 (which contains a nominal TCP interface) are not productive. If this is an argument about changes to a sockets API of some form, it's not relevant, as IETF does not standardize sockets. For the record, use of SCTP with iSCSI is still germane to this list, but SCTP vs. TCP battles are not. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00798; 28 Sep 2000 22:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA18746 for ips-outgoing; Thu, 28 Sep 2000 21:51:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA18693 for ; Thu, 28 Sep 2000 21:50:00 -0400 (EDT) Received: from amrelay1.boi.hp.com (amrelay1.boi.hp.com [15.56.8.24]) by atlrel1.hp.com (Postfix) with ESMTP id 86525302; Thu, 28 Sep 2000 21:49:46 -0400 (EDT) Received: from xboibrg1.boi.hp.com (xboibrg1.boi.hp.com [15.56.8.167]) by amrelay1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id TAA09184; Thu, 28 Sep 2000 19:49:45 -0600 (MDT) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 19:49:45 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: John Hufferd/San Jose/IBM , ips@ece.cmu.edu Subject: RE: Status summary on multiple connections Date: Thu, 28 Sep 2000 17:33:54 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I thought we had hashed this thing to death in a previous string. Now we have to go through all the issues all over again. -----Original Message----- From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com] Sent: Wednesday, September 27, 2000 3:53 PM To: ips@ece.cmu.edu Subject: RE: Status summary on multiple connections David Black, Let me vote that there is merit in having two Asymmetric Conversation per Session. Even though I do appreciate the request for a minimum of Two, I am troubled with the fact that I think many folks have already built iSCSI Initiators and Targets that use just one connection per session. Now in Robert Snively note below, he talks about the value of the Wedge Drivers, and some of us also buy into that. However, his arguments were based on spreading the stuff across NICs etc. I do not know if that is required, but having one TCP/IP connection that only carries Commands, and one that only carries Data, seemed to solve almost all the complaining and pacing/credit/sliding window conversations we have been having. This is true even when the second TCP/IP connection (of the session) is on the SAME NIC. (Robert, I think that all your wedge statements still hold true with 2 Asymmetric Connections per session, if both connections are on the same NIC.) Except, for that, much of the rest of Roberts points are "right on". So if we can just for awhile talk about whether the Dual Conversation Asymmetric approach is acceptable, I think that you may very well find that a lot of issue go away. And I think that was Matt's point. If this is correct, then, it is important to have a minimum two conversation per Session (maybe two is all that is really needed). But what about the implementations that are currently in flight. I think we may have to permit the single connection per Session, but strongly discourage it, since when it gets into longer distances, or works with implementation with less memory, or in some kind of network stress, the recovery approaches are much more draconian (dropping the connection etc.). That means that all the implementations that are working today, will continue to work, but will not be nearly as good as the Asymmetric Double Connection per Session (ADCS?) versions. Now Matt's point is that for folks just starting, being able to have code both the Double and the Single (with all the extra recovery/pacing stuff required in the Single Conversation per Session) seems wrong. And I think I have heard him state that updating the Driver code from a Single to a Double (especially if within the same NIC, was a non problem). If my statements about ADCS is correct, the testing effort should be a lot more straight forward since all the other various "boundary conditions" will not have to be tested for, since they will not exist. And if Matt's statements are correct, that makes me wonder why wouldn't we go that way as a requirement? Does anyone that currently has iSCSI drivers have a problem of moving to 2 Asymmetric Connections per Session as a requirement, if so, please state why. . . . John L. Hufferd Robert Snively @ece.cmu.edu on 09/27/2000 08:46:26 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Status summary on multiple connections Julo, A third alternative does exist and may be preferable. That is to reflect the present broad usage of SCSI that uses wedge drivers to achieve parallelization, bandwidth aggregation, and high availability. By that definition, the proper number of connections per I_T nexus is one and the proper number of connections per session is also one. This recognizes the reality that there is really little to gain in virtualizing multiple connections into a single session image when you can perform even more flexible virtualization through a wedge driver at a higher level. The single connection alternative allows a simplistic ordering structure, a simple recovery mechanism, and does not require state sharing among multiple NICs. It allows bandwidth aggregation across any set of boundaries that is required. Because command queuing is the rule among high performance SCSI environments, latency appears only as an increment in host buffer requirements except during writes that perform a commit function. Those traditionally have been taken out of the performance path by using local non-volatile RAM to perform the commit functions, using slower high latency writes with less strict ordering requirements relative to reads to actually perform the write to media. The overheads associated with handling a multiple connection session of any type are basically indistinguishable from the overheads associated with device virtualization through wedge drivers. If you consider software instruction path lengths for the total functionality, you can conceive of the multiple connection session as simply a TCP/IP wedge driver. Bob > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, September 26, 2000 9:48 AM > To: ips@ece.cmu.edu > Subject: Status summary on multiple connections > > > > > Dear colleagues, > > I am attempting to summarize where we stand with regard to > the multiple > connection issue > and to the two possible models - Symmetric (S) and Asymmetric (A). > > Many of us feel strongly that the multiple connection issue > is central to > the whole design > and cannot be added as an afterthought. Moreover designing > the hooks to > allow it later > will certainly already force us to make a decision. And both > the hardware > and the software designers will be ill-served if we hand > them a half-backed > solution. > > However this is not an invitation to reiterate positions > that where already > stated. > If you feel that I have grossly misstated anything in this > note please > write me and use > the mailing list only if my answer is not satisfying. > > And yes - like the chairman - I would like to make progress > but I don't see > any way to > do it without satisfactorily closing this issue. > > The reasons for multiple connections where discussed at some > length and > where very nicely summarized in a series of notes by Michael Krause > (beginning of August). > > The core reasons for having multiple connections where the > need for more > bandwidth and availability than a single link can supply > with a level of > complexity affordable for simple > installations and with a traffic engineering and management clearly > separated from the transport users (SCSI). > > The session is embodying this requirement. > > The only major objections I have heard against this where > those requiring > to go all > the way in having one TCP connection/LU - and after a short > debate this > objection > was practically removed. > > The other objections we heard where that this is basically a > transport > issue and it should be > solved at transport level. > That might be true - but since many, if not most, of TCP > applications do > not have this > requirement it is highly unlikely that TCP is going to do connection > trunking in the foreseeable > future. > > iSCSI can be designed to use multiple TCP connections in one > of two ways: > > -Asymmetric - one TCP flow only carries commands the others > carry only data > > -Symmetric - every flow carries both commands and their > associated data > > The S version is designed in the I-D version 01 > The S version requires an command ordering scheme and that > is provided by a > command counter and a sliding window scheme. It was argued > that ordering > needs might be more prevalent than usually thought and a > good conservative > design should preserve ordering. > Ordering-per-LU (as it is designed in FCP-2 version 4) was > considered > impractical as it > required initiators to maintain state for each LU - while > the rest of the > design required initiators only to maintain state for > outstanding commands. > Several comments on this list suggested that this windowing > mechanism could > also be used as a command-flow-control mechanism and that is > a "bonus" of > the scheme. > > The A version - comes in two flawors: > > - pure A (PA) in which ONLY commands flow on one TCP > connection while data > flow > on DIFFERENT connections (with only one data connection > being selected > for a > command); this scheme requires a minimum of 2 TCP > connections although > not > necessarily on different physical links. > > - collapsed A (CA) in which commands flow one a single TCP > connection while > data > flow on ANY connection; this scheme requires as a minimum 1 TCP > connection. > > Here is first attempt to list the benefits and drawbacks of > all of them: > > > - S - benefits > - well understood > - simple hardware setup and/or TCP API activation > - window mechanism can be also used for flow control > - the minimum required is a single TCP connection > > - drawbacks > - need to maintain a window mechanism > - a multiplexing mechanism has to be carefully > crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > - PA - benefits > - TCP will both order the commands and provide for > flow control > through the TCP window mechanism > - the multiplexing mechanism is simpler as closing > TCP windows > will > never affect command flow > - data flow can use a streamlined header (and processing) > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > > - CA - benefits > - TCP will order the commands > - needs a single TCP connection as a minimum > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > - if command flow control is required counters > and the sliding > window mechanism > are required > - a multiplexing mechanism has to be carefully > crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > From the above it should be apparent - as was already > pointed out by Matt > Wakeley - > that the CA inherits the drawbacks of S and PA and as such > the choice is > really > between S and PA. > > Julo > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16313; 2 Oct 2000 12:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA05013 for ips-outgoing; Mon, 2 Oct 2000 11:35:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA05000 for ; Mon, 2 Oct 2000 11:35:38 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 2 Oct 2000 11:35:03 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070410103B@corpmx9.isus.emc.com> To: ycheng@advansys.com, mark.carlson@sun.com, randall@stewart.chicago.il.us Cc: David.Robinson@EBay.Sun.COM, ips@ece.cmu.edu Subject: RE: Status summary on multiple connections Date: Mon, 2 Oct 2000 11:35:00 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Both of you forget about the case when multiple PDUs are inflight, say N, > N+1, and N+2, and one of them has CRC error, say N+1. The receiver throws > away N+1 because the bad CRC. N+2 is received long before N+1 is > retransmitted after a timeout by the sender. Of course, a sequence number > inside the PDU will ensure sequentially. However, as I stated in another > posting, all software realize such problem and will not count on sequential > delivery by a transport. Let me try to head off some confusion here. PDU is a dangerous acronym because it could refer to layer 2 (e.g. Ethernet) packets, layer 4 TCP segments or iSCSI information units at layer 5. For this discussion, layers 2 and/or 4 are the intended context. If packet/segment N+1 has an (e.g., Ethernet) CRC error, TCP has to buffer N+2 and can't deliver it to the application. The concern that's behind discussions of things like RDMA and the Urgent pointer is in the details of how to buffer N+2. If the N+1/N+2 boundary is not at the start of an iSCSI PDU, N+2 gets tossed into some memory somewhere and copied out when N+1 arrives and it's possible to resume processing of the stream at the iSCSI level; this can be slow. One of the goals of the RDMA and Urgent discussion is to identify the iSCSI boundary in the TCP data, either by forcing the boundary to the TCP N+1/N+2 boundary or providing a pointer to where it is inside N+2 so that all or most of N+2 can go through iSCSI processing on receipt and hit the right place in memory the first time. Handing a SCSI command found in N+2 to the SCSI device before getting the N+1 data is prone to cause a variety of peculiarities. For example, if the command in N+2 is a task abort of a command in N+1, the Initiator is going to be rather surprised when the abort fails and the task that was supposed to be aborted executes after the failed abort (which may be observable via a different iSCSI session). There are doubtless cases in which this can be done safely, but they'll need some careful investigation. David Robinson's contention that the WG shouldn't spend time on optimizing cases in which packets are lost because they're not going to happen often enough for optimizations to make a significant difference is an open issue that the WG will need to come to a conclusion on. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15666; 2 Oct 2000 8:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA27637 for ips-outgoing; Mon, 2 Oct 2000 07:54:39 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA27630 for ; Mon, 2 Oct 2000 07:54:34 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id NAA80144 for ; Mon, 2 Oct 2000 13:54:02 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id NAA130414 for ; Mon, 2 Oct 2000 13:54:02 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.00415AFE ; Mon, 2 Oct 2000 13:53:51 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 12:09:56 +0300 Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, The "URL like" target address and view are sent o the target as part of the text-message at login. If authenthication is being done then this will be part of the message that follows the exchange that establishes the authenticated channel. Julo "Jim Hafner/Almaden/IBM" on 29/09/2000 18:07:54 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Julo, Yet another method for determining the mapping of LUs to LUNs for given initiator? You already have vendor-specific ways (in FC that will surely get ported to iSCSI), the T10 standardized way with AccessID enrollment or with a TransportID (for iSCSI, which is still TBD), and the login authentication process (however that ends up getting defined). Also, your comment implies that the URL is sent to the target in a mode similar to an http-type protocol. Where is this protocol defined in the draft? Is it part of the Text message in the login authentication? Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-29-2000 06:14:44 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] I would add that we wanted the path to be an additional identifier that the target could use to determine what collection of LUs to present to the initiator. Julo csapuntz@csapuntz-u1.cisco.com on 23/09/2000 05:00:33 Please respond to csapuntz@csapuntz-u1.cisco.com To: "IP Storage" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Just for clarification... I was not proposing to add the extended URL scheme for the transport spec. It isn't necessary. In this thread, Doug makes an excellent point about LDAP being a superior mechanism for describing how to connect to the storage. Directory services such as LDAP, as Doug has pointed out, will be critical to managing large quantities of storage. A host can ask such a directory service for a list of storage devices it should mount and how to connect to those storage devices. The query against the directory server that returns this information can be based on machine ID, user ID, operating system ID, or even the owner's birthday. One of the things discovery will end up doing, no doubt, is defining LDAP schemas that describe how to connect to storage (i.e. use SCTP or TCP, what port, what target name, what LUN, what WWN, how to authenticate, etc.). However, there is one place where the transport protocol has to define a name: third party commands. There needs to be some kind of global name which the initiator can pass to the target. The name must be distillable into a string. The target must understand the name and be able to use the information to establish a connection to another target. One could say that the string that is passed is not specified by the standard but instead specified by some management software. I think this will lead to poor interoperability. The SCSI URL-type name is the current proposal for target name. Why is SCSI target name made up of a hostname + a path? Why is the hostname + path passed on connection setup? There are two reasons. I think NAT and IPv6 makes passing hostnames rather than addresses in protocols more desirable. Hostnames can be re-resolved as you cross addressing boundaries. The path is there so that the name can support multiple targets behind a single IP address without having to add entries to the DNS server. At Cisco, for example, I have no control over the local DNS servers and cannot add DNS entries for the ATAPI DVD and floppy in my computer. _Costa From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04969; 29 Sep 2000 22:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA26610 for ips-outgoing; Fri, 29 Sep 2000 20:19:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA26606 for ; Fri, 29 Sep 2000 20:19:32 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 20:19:00 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070410102D@corpmx9.isus.emc.com> To: dotis@sanlight.net, Black_David@emc.com, matt_wakeley@agilent.com, ips@ece.cmu.edu Subject: RE: Enough on TCP vs. SCTP, please Date: Fri, 29 Sep 2000 20:19:00 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk draft-wakeley-iscsi-msgbndry-00.txt says: The proposal is pretty simple. The first byte of every iSCSI message is marked "urgent" - the TCP urgent pointer will point to the first byte of the iSCSI message in the TCP segment. Douglas Otis says: > It is not reasonable to redefine the urgent pointer. Your WG co-chair says: I can't find anything in the draft that redefines the urgent pointer. TCP deliberately does not define exactly what the urgent pointer points to. For example, I wouldn't expect iSCSI and telnet to use the urgent pointer in the same way. In order to continue this argument that an unacceptable change is being made to TCP, it is necessary to cite the text in RFC 793 (or some other RFC that defines TCP) that would have to be changed if the approach in the draft referenced above were adopted. NOTE: this is *not* a statement of consensus that the approach in the draft should be adopted. This is only a procedural statement that consideration of that approach is within scope of the WG. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16101; 2 Oct 2000 11:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA01312 for ips-outgoing; Mon, 2 Oct 2000 10:03:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA01307 for ; Mon, 2 Oct 2000 10:03:44 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 2 Oct 2000 10:03:07 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101039@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: The third alternative Date: Mon, 2 Oct 2000 10:02:31 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > The only argument for SCSI wedge drivers where that they EXIST ALREADY. > Pretty weak argument for those building new equipment and for > interoperability. This severely understates the case. Not only do wedge drivers exist already, but they do a number of things that an iSCSI standard will never do. Wedge drivers tend to contain logic specific to the device that they're providing the wedge for - that's inappropriate to standardize, either in IETF or T10. In addition, there are implementation advantages to building wedge drivers above the SCSI level - one doesn't have to spread SCSI connection state across a failure boundary. > I think that if we keep ourselves honest we have to either: > > - provide for multiple connections at the iSCSI level as it is transport problem > that other TCP applications are not compelled to handle (I hear already BUT SCTP > handles it!) and hope that one day the session concept will drift into pure transport Provide for is fine, as long as they're not required. I would expect to see significant deployment of systems that use single TCP connection per session and handle multiple sessions in wedge drivers. Arguments like the one Bob has made is one of the reasons for multiple connections/session to remain OPTIONAL. > - go to T10 and ask the to standardize wedge drivers! Just say no! Exactly what would you want T10 to standardize? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02911; 29 Sep 2000 9:49 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA02010 for ips-outgoing; Fri, 29 Sep 2000 08:56:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA02005 for ; Fri, 29 Sep 2000 08:56:14 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA19430; Fri, 29 Sep 2000 06:50:11 -0500 Message-ID: <39D481F3.A2974E8B@stewart.chicago.il.us> Date: Fri, 29 Sep 2000 06:50:11 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: "GUPTA,SOMESH (HP-Cupertino,ex1)" CC: Robert Snively , "WAKELEY,MATT (A-Americas,unix1)" , IPS Reflector Subject: Re: Why FCP doesn't need RDMA? It has a better way. References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > > Even iSCSI does not really need RDMA. The underlying note in > the whole debate is really the preservation of message boundaries. > > If we could use one of the reserved bits in the TCP header to say, > do the repacketize or adjust segment boundaries, and each TCP > segment starts with an iSCSI header, there should be no need for > another protocol (or TCP option header). > I don't think you can go playing with TCP option bits. This implies changing TCP and I don't see this in the WG charter. If bits need to be changed then a draft needs to be written and submited to the proper WG.. in this case I would say transport area WG... I think in some respects this is exactly what Costa has done... Now as far as message boundary preservation, you get this for free with SCTP.. and has Douglas as already pointed out, you could very easily implement in SCTP the ability to copy a specific stream into receive buffers .. i.e. tell the sender, but the data on stream X and then as it arrives dump it directly to user buffers. This would take a bit of work, but so does the RDMA solution :/ R > The iSCSI header contains enough information in it to enable > the recepient to determine where to put the data. > > Somesh > > -----Original Message----- > From: Robert Snively [mailto:rsnively@Brocade.COM] > Sent: Thursday, September 28, 2000 8:29 AM > To: 'Matt Wakeley'; IPS Reflector > Subject: RE:Why FCP doesn't need RDMA? It has a better way. > > > I object to mandating iSCSI use an RDMA option because: > > > > - (main reason) there isn't any standardized mechanism now, and I > > don't want to hold up iSCSI while one crawls through the > > standards process. > > > > - I don't think RDMA is needed. FCP doesn't use it, and it > > works great with > > the > > FC protocol chips that "accelerate" FCP. > > Actually, RDMA is not needed in FCP because all protocol chips > implemented perform a real peer-to-peer DMA straight to the > data areas specified by the user's interaction with the operating > systems allocation algorithms. The combination of the FCP/SCSI > pointer structure, task tagging, and the FC relative offset perform the > function you would otherwise have to use RDMA to accomplish. > > Bob Snively > Brocade Communications Phone 408 487 8135 > 1745 Technology Drive > San Jose, CA 95110 Email rsnively@brocade.com -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04147; 29 Sep 2000 17:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA18390 for ips-outgoing; Fri, 29 Sep 2000 16:15:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA18381 for ; Fri, 29 Sep 2000 16:15:04 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G1N00LVJYJQ7L@mta5.snfc21.pbi.net> for IPS@ECE.CMU.edu; Fri, 29 Sep 2000 12:38:15 -0700 (PDT) Date: Fri, 29 Sep 2000 12:37:44 -0700 From: Peter Johansson Subject: RE: Status summary on multiple connections In-reply-to: <200009291825.LAA13471@ha10nwk.EBay.Sun.COM> X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000929122849.00cc18a0@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 11:25 AM 9/29/00, David Robinson wrote: >I am hearing different stories on the issue of ordering. [...] I thought >this problem was already solved at the SCSI layer through the use of >ordered commands which in general are not used for disks but always used >for tapes? Since FC will reorder this has to be a solved problem. Would >not an initiator talking to a tape target simply set the ordering flag? Part of the ordering conundrum is solved within SCSI by virtue of the fact that SAM recognizes ordered and unordered execution models. The choice of execution model is usually a matter for the target's designer. The catch is that in order to enable targets that use an ordered execution model, the transport protocol (iSCSI, in this case) has to be able to communicate the command ordering to the target. So it will do an application little good to declare a task's attribute to be ORDERED if the order in which the tasks were created by the initiator is lost en route to the target. This raises the question as to whether or not a particular session (login to a LU) should be declared ordered or unordered at the time it is instantiated. If there are performance reasons to not care if the commands are delivered in order, then perhaps it would be desirable to configure the manner of command delivery on a session by session basis. Whatever you do, make sure that the ability to signal commands to a target in the same order they were created at the initiator is a mandatory feature of iSCSI. PS Similar considerations apply to the return of command completion information to the initiator IF one wishes to take some short-cuts that eliminate the necessity to transmit GOOD status in an ordered environment. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32725; 28 Sep 2000 17:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA10951 for ips-outgoing; Thu, 28 Sep 2000 17:23:10 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA10926 for ; Thu, 28 Sep 2000 17:22:55 -0400 (EDT) From: meth@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id XAA44888 for ; Thu, 28 Sep 2000 23:22:23 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id XAA24032; Thu, 28 Sep 2000 23:22:22 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256968.0075646E ; Thu, 28 Sep 2000 23:22:14 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: "John Hufferd/San Jose/IBM" , ips@ece.cmu.edu Message-ID: Date: Fri, 29 Sep 2000 00:22:06 +0300 Subject: Re: twist on iSCSI asymmetric model Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk John, I'm not really saying anything new. I'm essentially summing up and choosing the best of the various ideas that were thrown around for the so-called asymmetirc model. The point is to not allow immediate data in the control channel when using the asymmetric model; all bulk data must go in the data channels. This avoids the problem of data clogging up the control channel and thereby preventing a critical task management operation from getting through. Since there are also strong feelings that we should allow a single TCP connection, I prescribed how we should deal with the 2 cases: either a single (and only) TCP connection under the sole responsibility of the initiator to not become clogged, or a purely asymmetric model with no bulk data going over the control channel. I then described a few technical details that I think will allow for more efficient implementations. The last sentence is a repeat of my response to Pierre's objection about requiring 2 interrupts upon the completion of a READ operation: one interrupt on the NIC receiving the data and one interrupt on the control channel NIC for the response to the command. I am simply commenting that we only need the interrupt on the data channel when the last data PDU arrives, while no interrupt is needed on the control channel NIC. - Kalman Meth "John Hufferd/San Jose/IBM" on 28/09/2000 22:31:56 Please respond to "John Hufferd/San Jose/IBM" To: ips@ece.cmu.edu cc: (bcc: Kalman Meth/Haifa/IBM) Subject: Re: twist on iSCSI asymmetric model Kalman, I am not sure what your new suggestion solves. Especially if what Charles wrote is correct: {Snip} "2. As I recall (possibly not very accurately) SAM-xx states that an initiator should not have more than one pending task management request at a time. In general, such requests are "think-time" limited and therefore non blocking, so this seems not to be a problem in practice. 3. It's important that this set of control functions flow over the same control connection that's used for commands (ie. these functions need to flow through the command delivery pipe). Otherwise their behavior is indeterminate. An example is an "abort task" function which arrives at the target while the command to be aborted is still in transit. 4. Considering the rule of allowing only one pending task management request at a time, it might be sufficient to have the initiator budget one "credit" to be used for this purpose." {Snip} (His use of Credit here meant a reserved Buffer) I believe that you also have determined that Two connections per Session Solves the key problems (especially if what Charles said is correct,) and they are on the same NIC, since you said ".... If the control and data connections happen to be on the same NIC, then this might not be a problem". This, I believe, is the point that Matt has been trying to make. He felt that two connections per session were key and should be required. So lets put it all together; Control should flow on the same connection as Commands, and Data on a separate connection (within the same session). This solves the blockage issue, and both connections within the same NIC solve the interrupt problem. Plus it does not have iSCSI sliding windows etc. etc., the implementation is relatively easy, compatible with the concept of iSCSI HW on the NIC, and the Wedge Drivers still work as they do today. Based on that, I think Matt has a strong point. What do you think? You also added (on your last Paragraph) a statement that I think takes us in a different direction "...Only one interrupt is required upon completing a READ data transfer, if we mark the last iSCSI Data PDU of the data transfer". I am not sure what you are attempting to say here, are you making a new recommendation? In any event, please address the previous questions, before you amplify your last paragraph. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSD San Jose Ca (408) 256-0403, Tie: 276-0403 Internet address: hufferd@us.ibm.com Notes address: John Hufferd/San Jose/IBM @ IBMUS VM address: hufferd at IBMUSM54 meth@il.ibm.com@ece.cmu.edu on 09/28/2000 06:30:58 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: twist on iSCSI asymmetric model In the asymmetric model, one TCP connection (the control connection) is used for sending all of the commands, thus ensuring ordered delivery of the SCSI commands. If (a large amount of) data is also sent on the control connection, there is the possibility that the data would fill up the window, and critical task management requests couldn't get through. On the other hand, it has been pointed that some implementations will prefer a single (and only) TCP connection for both commands and data. I would like to suggest the possibility of combining these 2 cases; either (1) have a single connection for everything or (2) have one connection for control and one or more data connections. In this case, no (WRITE or READ) data may be sent over the control connection. The initiator and target negotiate during the first login as to which of the 2 possibilities they will use. In (1), it is the responsibility of the initiator to worry about not overloading the one and only connection so as not to delay delivery of critical task management messages. In (2), the iSCSI SCSI Command PDU specifies which data connection to use for the data of this command. On the data channels, where we will have iSCSI Data and R2T PDUs, we shouldn't need to use the long headers that we use to specify an iSCSI SCSI Command PDU. We will therefore have a shortened header for the data channels. In (1), since data will travel on the control channel, the iSCSI Data PDU and R2T headers will be padded with zeros so that all headers are of the same length on a particular connection. The use of R2T is negotiated during the first login of the iSCSI session. If R2T is to be used, then the initiator may not send data before it is requested by the target. Thus there will always be buffers ready to accept the data. If the target agrees to not use R2T, then the target implicitly agrees to have some number of buffers available to receive data sent by the initiator, even if the data arrives before the command arrives. In this case the initiator MUST send data in the data connections in the same order that the corresponding commands were sent. (This avoids deadlocks.) If the target runs out of buffers, then it will let data back up in the TCP window of the data connections, while it is processing the data from earlier commands. The only main drawback remaining with the asymmetric model is that 2 separate operations may be required to set up sending a command and to register its DMA buffers. If the control and data connections happen to be on the same NIC, then this might not be a problem. Only one interrupt is required upon completing a READ data transfer, if we mark the last iSCSI Data PDU of the data transfer. - Kalman Meth. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id ab11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10951; 1 Oct 2000 5:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA00511 for ips-outgoing; Sun, 1 Oct 2000 04:54:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00504 for ; Sun, 1 Oct 2000 04:54:35 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA239432 for ; Sun, 1 Oct 2000 10:54:03 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA68118 for ; Sun, 1 Oct 2000 10:54:03 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.0030E246 ; Sun, 1 Oct 2000 10:53:56 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 11:51:36 +0300 Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, Assume than in the new wonderfull SAN world you have started a disk-to-tape (or disk-to-disk) long third party copy. The SAN is fine and the copy proceeds for an hour but the lousy initiator-to-copy-manager link (on which accidentally no data transfer took place) fails for a fraction of a second. Should we restart the command under-the-cover or drop it or ask the parties to provide state information to a specific SCSI restart driver? And we can build many similar scenarios. I think that whatever we can do simplify exception handling we should do (the same arguments that hold for multiple connections hold here too). I would add that in Ideal world - I would like to have transport "splice" a new TCP connection with an old TCP connection but failing this to happen (again SCTP is doing it already or not?) we should take care that simple events like a cable taken-out in some obscure part of the network will only seldom affect higher layers. Julo Stephen Bailey on 27/09/2000 07:16:12 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) > Currently, iSCSI is spec'ed to recover from transport-layer > connection failures. > > The main motivation for this decision was to support tape backup > applications that are quite sensitive to any failures that get > propogated to their layer. > > So, perhaps we can remove the requirement of recovering from > transport-layer connection failures in iSCSI. This would simplify > the protocol somewhat. > > Thoughts? I'm all for eliminating command recovery. There seem to be several reasons advanced for command recovery. The first seems to be based upon an inappropriate analogy to FCP. Command recovery had to be added to FCP-2 because the FC layer is unreliable. A single dropped FC frame leads to a failed FCP command. This clearly upsets tape operation even when the link is performing nominally. In FCP, without command recovery, with some observable frequency, you will get an expected error that leads to complete, irrecoverable failure of a transfer stream. The other thing that makes FCP-2 command recovery work well is when you are doing a write, which is 90% (maybe it's 99%?) of tape operation, the target can return an early indication of most frame drops, rather than waiting for a timer to expire. TCP's reliability solves this problem in another way. By the time you get a TCP connection failure, you have already exhausted a set of reliability mechanisms which guarantee, with high certainty, that further data can not be transferred between the two endpoints. `the two endpoints' phrase suggests the other reason advanced for command recovery. That is, to permit path failover for commands which are not idempotent, such as tape write sequential. The problem with this, is that it is not clear HOW iSCSI command recovery can actually work properly, given a TCP connection failure indication. It takes a long time for a TCP connection to fail, and by that time, I'm not sure recovery would reasonably be possible. Perhaps I'm in error on this assumption. Can a tape guru (Joe from Exabyte?) comment on whether recovery would be possible after many seconds (tens, hundreds) have elapsed? The SCSI layer has never been solely responsible for ensuring reliable backup. Macro scale things go wrong with tape (run off the end, get eaten, etc..) with relatively high frequency. A low level backup engine like tar or dump will fail on a SCSI error, and that's OK. There must also be a higher level software component like Amanda, which manages retries, including operator intervention, to ensure reliable backup. It seems like whether iSCSI has a command recovery mechanism should be a function of whether somebody can stand up and say for sure that it solves a real problem. So far it only seems like it MIGHT solve a problem. Who can say `this solves MY problem!'? Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id ab12978; 1 Oct 2000 17:14 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12835; 1 Oct 2000 16:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA11215 for ips-outgoing; Sun, 1 Oct 2000 15:30:34 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA11211 for ; Sun, 1 Oct 2000 15:30:31 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id VAA12588 for ; Sun, 1 Oct 2000 21:29:57 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id VAA42052 for ; Sun, 1 Oct 2000 21:29:56 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.006B1934 ; Sun, 1 Oct 2000 21:29:48 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 22:27:27 +0300 Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph, I made the same points as you a long time ago on this list (before Adelaide) when Costa came out with an RDMA option for TCP. I think that at the time I was vocal in support of such an approach. Watching what happened to it and considering that we have to have widespread support to get to use it for storage I think it will be counterproductive to have iSCSI wait for it. That is what "Julo was not buying" Stephen Bailey on 29/09/2000 00:36:40 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) > I believe the point in factoring out the RDMA mechanism from iSCSI is to > make the same hardware assists available to other application > protocols. I agree. The motivation is primarily economic. If you implement a general solution to 1) data steering, 2) congestion avoidance, 3) security, you own the edge of all networking. If you implement these for storage, you only own storage. Given that it seems such an easy step to make the hardware slightly more general and own the world, I'm stumped on why not. OK, I admit I know the argument why not, because people think the more specific solution more tractable. I disagree. The other reason why not is some people really aren't sold on the necessity for hardware implementation at all. BTW, I made this case at CERN recently: http://hsi.web.cern.ch/HSI/HNF-Europe/Workshop%202000/Presentations/Com_Stor_Net.PDF Julo wasn't buying though. Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32720; 28 Sep 2000 17:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA11147 for ips-outgoing; Thu, 28 Sep 2000 17:27:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from zmamail01.zma.compaq.com (zmamail01.zma.compaq.com [161.114.64.101]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA11106 for ; Thu, 28 Sep 2000 17:26:58 -0400 (EDT) Received: by zmamail01.zma.compaq.com (Postfix, from userid 12345) id 6893B641A; Thu, 28 Sep 2000 17:26:28 -0400 (EDT) Received: from hobbit.cca.cpqcorp.net (hobbit.cca.cpqcorp.net [16.101.176.101]) by zmamail01.zma.compaq.com (Postfix) with SMTP id 255FB6440 for ; Thu, 28 Sep 2000 17:26:28 -0400 (EDT) Received: by hobbit.cca.cpqcorp.net (Smail3.2.0.101/COMPAQ) id m13elCF-000WWLC; Thu, 28 Sep 2000 16:26:27 -0500 (CDT) Message-Id: From: relliott@hobbit.cca.cpqcorp.net (Robert Elliott) Subject: Re: Bidirectional SCSI commands and iSCSI To: ips@ece.cmu.edu Date: Thu, 28 Sep 100 16:26:27 -0500 (CDT) In-Reply-To: from "julian_satran@il.ibm.com" at Sep 28, 2000 10:10:53 PM X-Mailer: ELM [version 2.5 PL0b2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk > Jim, > > What is the intended use? Are there already commands that use > it? Is there a major/minor distinction - like output data > & input results or output parameters & input data - or is it > intended to be completely symmetric? There can be different amounts of write and read data. The data-out size and data-in size are set by the initiator. The target chooses: * whether to accept write data or provide read data first * when to switch direction * how often to switch direction The initiator is not allowed to change the write data after sending the CDB - read-modify-write (where the initiator modifies the data) is not supported. Write-modify-read (where the target modifies the data) is supported. T10/00-315 describes a bidirectional command called XDWRITEREAD useful for RAID operations. The object storage device (OSD) command set plans to use bidirectional commands. Other commands may appear in the future that use this feature. T10/00-314 describes how this is supported on Parallel SCSI. T10/00-377 describes how this is supported on Fibre Channel (FCP-2). The 1394 community plans to revise SBP-2 to add bidirectional command support. (all T10 documents are on http://www.t10.org) > Julo > > "Jim Hafner/Almaden/IBM" on 20/09/2000 17:56:27 > > Please respond to "Jim Hafner/Almaden/IBM" > > To: ips@ece.cmu.edu > Subject: Bidirectional SCSI commands and iSCSI > > Folks, > > In case you missed this, T10 just approved changes to SAM-2 (rev14) which > remove the restriction that a single SCSI command can have a data flow in > only one direction (DataIn xor DataOut). Transport specs (that want to > allow this) will need to change to allow for two sets of Data buffer fields > in the headers (length, over/underrun, etc.). > > Is anybody looking to modify the iSCSI spec to get these concepts > supported? > > Jim Hafner > -- Rob Elliott UNIX mailto:relliott@unixmail.compaq.com Houston, TX PC mailto:Robert.Elliott@compaq.com From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15656; 2 Oct 2000 8:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA27653 for ips-outgoing; Mon, 2 Oct 2000 07:54:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA27635 for ; Mon, 2 Oct 2000 07:54:38 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id NAA66298 for ; Mon, 2 Oct 2000 13:54:05 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id NAA80028 for ; Mon, 2 Oct 2000 13:54:05 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.00415D1E ; Mon, 2 Oct 2000 13:53:56 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 13:46:42 +0300 Subject: Re: Status summary on multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, The picture I had in mind is somewhat simpler. I assume that on any given transport connection, with or without RDMA, iSCSI will get from TCP and hand to SCSI things that are in order (just to keep layering iSCSI over TCP simple and clean). No retransmissions involved in iSCSI here. In case of several connections iSCSI might get things out of order (no ack for them) and may decide to hand them or not to hand them to execution. iSCSI will not require retransmission as we assume that the out of order thing is a temporary artifact of the underlying network. Recovery if needed here is done only when a connection is blown-away and then all outstanding commands that where shipped on that connection are restarted (resent). And again we conceived the window only to limit the number of commands iSCSI has to keep to get things in order but it can as well serve as a flow-control mechanism for the target as the initiator is unconcerned of what is being done with the commands within the window (e.g., a R2T from a command within the window - i.e., not acked yet - is a legal event). Julo David Robinson on 30/09/2000 01:35:11 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Status summary on multiple connections There are two types of out of order processing we need to consider: 1) Commands that have been received and ACK'd by the transport 2) Commands that have been received and not ACK'd In the first case this is purely a SCSI issue on if the commands are ordered and how the target processes them. As far as the transport is concerned the packets are "delivered". In the second case the transport is still required to track the possible retransmissions and process the transport ACK for all of the data when the missing segment arrives. It will overly complicate the iSCSI layer to successfully track which parts of the sequence space it has processed, defend against retransmissions, and handle packets that have more than one command, and retransmissons that may be only part of a command. This is all doable in an implementation (a bit messy at times) but to support this as an explicit feature we need to start adding in an iSCSI layer sequence space. I propose that we leave the processing of transport level un-ACK'd commands and data as an implementation detail and not make it an explicit iSCSI feature. Dropped or reordered TCP segments are rare enough in high performance environments that this is really a non-issue. -David julian_satran@il.ibm.com wrote: > > David, > > I think that RDMA out-of-order processing is important only in order not to > have data > piling up in adapters. It does not mean that data gets really "committed". > The commands can be executed in or out-of-order - this is a pure SCSI > story. > But if they have to be executed in order the ordering provided by SCSI > will enable it. > And as you and others have pointed out the same mechanism is good for both > ordering and flow control. > As far as I understand it windowing - although more expensive - is a better > technique over a wide variety of latencies than credits. > > And BTW we even considered credits for a "prefetching mechanism" for > chained > commands but where told that those are very much "out of fashion" (i.e. not > worth speeding up on Elefants). > > Julo > > David Robinson on 29/09/2000 21:25:52 > > Please respond to David Robinson > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Status summary on multiple connections > > > I am left with the following impression as to what was indicated here: > > - In general, command ordering is not relevant > > - If the initiator filesystem detects an ordering dependency, it will > wait > > until outstanding commands are complete before issuing the dependant > > command. > > > > This may be a reasonable means of operation for the disk world. It is > > woefully inadequate for the tape world, as follows: > > - In general, command ordering is crucial - out of order command > processing > > will lead to data corruption. > > - This would require the initiator backup application to block on > completion > > of every single write command of a backup operation before issuing the > next > > command. > > > > If this blocking were performed, both the throughput and capacity of a > tape > > device/media would be negatively impacted by an order of magnitude or > more. > > This would occur even assuming an instantaneous transport. > > I am hearing different stories on the issue of ordering. One side > is pushing hard for techniques that will allow out of order > execution using various RDMA techniques. This clearly states for > a certain class of devices (e.g. tapes) ordering is crucial. > I thought this problem was already solved at the SCSI layer > through the use of ordered commands which in general are not used > for disks but always used for tapes? Since FC will reorder this > has to be a solved problem. Would not an initiator talking to > a tape target simply set the ordering flag? > > Lastly, for a TCP based connection ordering can easily be made a > non-issue, simply don't try to process segments out of order. I > will defer to a transport expert, but I believe processing > TCP segments by an application out of order might cause problems. > In particular since the out of order segment is not ACKed until > after the missing segments arrive, they can be retransmitted > multiple times. SACK helps this but does not guarentee that > segments will not be retransmitted. So to process out of order > segments the application must maintain a list of which segments > have been processed as well, yuck! > > -David From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05918; 30 Sep 2000 3:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA04110 for ips-outgoing; Sat, 30 Sep 2000 02:22:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA04105 for ; Sat, 30 Sep 2000 02:22:24 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA57634 for ; Sat, 30 Sep 2000 08:21:49 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA147196 for ; Sat, 30 Sep 2000 08:21:49 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696A.0022F133 ; Sat, 30 Sep 2000 08:21:39 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sat, 30 Sep 2000 09:19:19 +0300 Subject: Re: iSCSI: problem with LUN discovery Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk JP, Unless I misunderstood you this is entirely a T10 issue. Presenting a different LU map to each initiator as identified by the port or the ACCESS-ID is intentional (each initiator can view a different set of LUs)and part of the protection mechanisms of SCSI. With iSCSI at later stages (when we start handling discovery and management) the pain of getting to a specific volume might be eased. Julo Raghavendra Rao on 27/09/2000 23:05:22 Please respond to Raghavendra Rao To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: iSCSI: problem with LUN discovery Julian, Do you agree that following is a problem ? If so, can we fix it ? SAM doesn't prohibit a target device from presenting initiator specific values for LUs. So it is believed that similar values for a LU may not necessarily address the same LU inside a target when accessed from different initiators. So the vendor ID page 83 of INQUIRY command is considered to be unique for a LU - There is one caveat to this though, that the page 83 values could optionally be target port specific. Anyway, if the hosts were to use a more persistent values such as page-83 identifier, say after reboot or after reconfiguration, it first needs to translate this page-83 identifier into a LUN. This can be done by first issuing a REPORT_LUN command to LUN 0 and issuing an INQUIRY page-83 command to each LU in the list obtained from REPORT_LUN response until a successful match is found. An initiator could cache the mapping for future speedier lookups, but it still has to linearly probe out all LUs to get the page-83 unique identifiers. I hate to do this static probing; This problem can be fixed by either a) mandating that LUN values are persistent and same for all initiators (more unlikely to be accepted) OR b) introducing a new page code for INQUIRY in which the initiator will pass down the page-83 identifier in the parameter list to LUN 0 and the target will respond with the LUN corresponding to this identifier; Of course, this has to go to T10 for approval. Thanks. -JP From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04942; 29 Sep 2000 22:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA27750 for ips-outgoing; Fri, 29 Sep 2000 21:08:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA27744 for ; Fri, 29 Sep 2000 21:08:22 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 21:07:50 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070410102E@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI sessions: Let's try again Date: Fri, 29 Sep 2000 21:07:50 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Taking another swipe at the hornet's nest labeled "multi-connection iSCSI sessions" ... With the consensus on the minimum number of TCP connections per iSCSI session having come unraveled, we need to start over on this whole area of multiple connection sessions, beginning with requirements and applicability. This only applies to TCP, as SCTP would only use a single connection per session. This email is a mixture of an attempt to summarize discussions and propose WG consensus with some additional technical contributions from yours truly with my co-chair hat off. I've tried to flag the latter with , but I may not have gotten all of them. Multiple connection sessions respond to two requirements: [1] Use of physical parallelism in the network fabric to improve throughput over a single connection that cannot exploit physical parallelism. This is an optimization. [2] Improvements in error recovery over SCSI's usual approach to error recovery. This is an optimization. [3] A TCP connection may get into a state where no data can be sent, requiring another connection. This has been referred to as window closure, and avoiding persistent forms of it is a requirement There appears to be an additional goal that is implicit in some of this discussion: [4] A multiple connection session may simplify implementations that support multiple physical connections by using one SCSI for multiple TCP instances rather than one-for-one. This is a design/implementation-specific optimization. I don't see a requirement that multiple TCP connections per iSCSI session be used to increase utilization of a single physical link, although in practice if we specify multi-connection sessions, it's hard to prevent their usage in this fashion. I don't think this merits further discussion, because even if we prohibit multi-connection iSCSI sessions, we probably can't stop someone from opening up multiple iSCSI sessions over the same physical link. There are three application scenarios of interest: [A] Direct host or server access to disk storage. [B] Access to tape, usually by a host or server, but third party copy engines that support tape targets would fall into this category. [C] Disk storage to disk storage mirroring, replication, etc. So picking up on the requirements and goals: --- [1] Utilize physical network parallelism. - [A] Server/host to disk storage. The conclusion of the long discussion of wedge drivers is that multiple connection sessions are not required to use network parallelism for disk access. - [B] Tape access Wedge drivers don't work for tape, hence multiple TCP sessions per iSCSI session is the only way for iSCSI tape access to take advantage of network parallelism. - [C] Disk mirroring/replication, etc. While these may start out as just remote disk interfaces, in practice, they seem to rapidly acquire additional functionality above and beyond basic SCSI, resulting in a wedge driver-like protocol layer that can handle the ordering issues that arise in using network parallelism. Examples of this include EMC's SRDF, and IBM's PPRC, both of which include significant enhancements above the basic SCSI or ESCON disk interface. Hence this case does not appear to generate a requirement for multi-connection sessions. --- [2] Error recovery [A] and [C] are similar in that there's higher level logic (e.g., a SCSI driver, a wedge driver, a mirroring/replication protocol) above it that can attempt to recover by retrying command. A concern here is that these retries tend to be expensive. [B] tape is different because a command retry may result in the command being executed twice; while disks generally behave correctly in this situation, tapes often do not. The technical case for error recovery seems weak on a couple of grounds. The first is that TCP's response to something not getting through is to try again and again. If TCP has to give up, it has likely done so after a significant delay comparable to the cost of the higher level retry, making the gain from optimizing the retry less clear. There are ways to make TCP give up quickly - something else on the LAN noticing that connectivity is gone and issuing an RST comes to mind, but I wonder if these sort of situations are sufficiently common to be worth optimizing for. Second, TCP handles most/many of the failure situations that are causing the recovery logic to be put into FCP-2. If Fibre Channel drops a frame from a backup session with a tape device, with current FCP the backup has to be aborted. The corresponding situation with TCP is that the frame will be retried and succeed, and the backup session will continue. My conclusion is that I don't see a strong technical case for use of multi-connection sessions for error recovery. --- [3] TCP Window Closure There's been a lot of traffic here, but most of the problems appear to be scenarios that are consequences of poor design choices (e.g., if a large blob of data causes a target not to be able to accept commands for a long time, the target should never have permitted the data to be sent). I haven't seen anything that makes multiple TCP connections per session to avoid the consequences of window closure a requirement. --- [4] Implementation simplification If I were building a multi-port iSCSI over Ethernet controller, I could see how only having one SCSI instance rather than multiple instances might make the design simpler and hence more feasible. The "If" at the start of this sentence seems to make this implementation specific, and hence, while this may be a nice thing to have, it doesn't seem to be a requirement. Summarizing the above discussion: [Consensus Attempt 1] I don't see anything that REQUIRES two TCP connections in all iSCSI sessions, because I haven't seen any TCP window closure scenarios [3] that aren't avoidable by reasonable design/engineering of the target. I therefore propose to re-establish the WG consensus that multiple TCP connections per iSCSI session is OPTIONAL. [Consensus Attempt 2] One of the points of this long email was to explore whether sessions are important enough that they need to be specified as part of iSCSI. Between tape parallelism [B] and implementation optimization [4] , I think sessions are important enough to specify even though they are OPTIONAL to implement. [Procedural Suggestion] It appears to me that the implementation optimization goal [4] that spans all uses of iSCSI is more important than the network parallelism [1] which is only required by tape. This would suggest according more weight to the concerns of those working on implementations, especially hardware. [Issue] The technical arguments for error recovery among the TCP connections in a multi-connection iSCSI session strike me as weak. It is not clear whether error recovery should remain a requirement for multi-connection sessions. Where we go from here: - There are two consensus attempts above, please register any objections to them promptly on the list. - There are a number of places where I've expressed my technical opinion, which are mostly set off by . Feel free to express an alternate opinion. - Comments on the procedural issue would be appreciated, as it may affect how I attempt to call consensus in the future. - In addition to general comments on the error recovery issue, could Joe Breher from Exabyte and anyone else familiar with the specifics of tape behavior comment on whether TCP's error recovery (retry with duplicate detection that achieves reliable delivery in the face of transient communication drops and errors) is good enough for tape? Resolving that specific point will help resolve the general issue. - Can we refrain from discussion of the actual session model proposals until the items raised in this email are relatively settled? Ok, fire away ... Thanks, --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16308; 2 Oct 2000 12:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA04963 for ips-outgoing; Mon, 2 Oct 2000 11:34:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA04949 for ; Mon, 2 Oct 2000 11:34:32 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id LAA48990 for ; Mon, 2 Oct 2000 11:21:39 -0400 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e92FYId140438 for ; Mon, 2 Oct 2000 09:34:18 -0600 Importance: Normal Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Mon, 2 Oct 2000 08:34:15 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/02/2000 08:34:17 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, Are you suggesting that the host send to the target what it thinks the LUN->LU map should be (during the login process)? I think you call this LUN Map, the "view". I don't quite see what the point of this is. The initiator is fully authenticated to the target as is the target to the initiator (via whatever cryptographic or non-cryptographic schemes the security proposal defines). After that, the target knows whose talking and knows the map (based on configuration information established in any number of ways (Access Control commands in-band; vendor specific in-band or out-of-band). I don't see the point of the initiator sending its "view" to the target at any point. If the initiator sends an incorrect view, you've just created an error scenario which needs additional definition. In any case, the target will send its "view" for that initiator in REPORT LUNS SCSI command after login (which is actually the layer that cares about the LUN Map at all anyway). Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 10-02-2000 02:09:56 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Jim, The "URL like" target address and view are sent o the target as part of the text-message at login. If authenthication is being done then this will be part of the message that follows the exchange that establishes the authenticated channel. Julo "Jim Hafner/Almaden/IBM" on 29/09/2000 18:07:54 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Julo, Yet another method for determining the mapping of LUs to LUNs for given initiator? You already have vendor-specific ways (in FC that will surely get ported to iSCSI), the T10 standardized way with AccessID enrollment or with a TransportID (for iSCSI, which is still TBD), and the login authentication process (however that ends up getting defined). Also, your comment implies that the URL is sent to the target in a mode similar to an http-type protocol. Where is this protocol defined in the draft? Is it part of the Text message in the login authentication? Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-29-2000 06:14:44 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] I would add that we wanted the path to be an additional identifier that the target could use to determine what collection of LUs to present to the initiator. Julo csapuntz@csapuntz-u1.cisco.com on 23/09/2000 05:00:33 Please respond to csapuntz@csapuntz-u1.cisco.com To: "IP Storage" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Just for clarification... I was not proposing to add the extended URL scheme for the transport spec. It isn't necessary. In this thread, Doug makes an excellent point about LDAP being a superior mechanism for describing how to connect to the storage. Directory services such as LDAP, as Doug has pointed out, will be critical to managing large quantities of storage. A host can ask such a directory service for a list of storage devices it should mount and how to connect to those storage devices. The query against the directory server that returns this information can be based on machine ID, user ID, operating system ID, or even the owner's birthday. One of the things discovery will end up doing, no doubt, is defining LDAP schemas that describe how to connect to storage (i.e. use SCTP or TCP, what port, what target name, what LUN, what WWN, how to authenticate, etc.). However, there is one place where the transport protocol has to define a name: third party commands. There needs to be some kind of global name which the initiator can pass to the target. The name must be distillable into a string. The target must understand the name and be able to use the information to establish a connection to another target. One could say that the string that is passed is not specified by the standard but instead specified by some management software. I think this will lead to poor interoperability. The SCSI URL-type name is the current proposal for target name. Why is SCSI target name made up of a hostname + a path? Why is the hostname + path passed on connection setup? There are two reasons. I think NAT and IPv6 makes passing hostnames rather than addresses in protocols more desirable. Hostnames can be re-resolved as you cross addressing boundaries. The path is there so that the name can support multiple targets behind a single IP address without having to add entries to the DNS server. At Cisco, for example, I have no control over the local DNS servers and cannot add DNS entries for the ATAPI DVD and floppy in my computer. _Costa From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16318; 2 Oct 2000 12:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA06235 for ips-outgoing; Mon, 2 Oct 2000 12:02:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA06227 for ; Mon, 2 Oct 2000 12:02:51 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id IAA15639; Mon, 2 Oct 2000 08:48:56 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id IAA18129; Mon, 2 Oct 2000 08:46:35 -0700 From: "Douglas Otis" To: , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 2 Oct 2000 09:05:09 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, If you wished to crack data being sent via IP, could you ask for a better tag of such information- scsi://[/modifier] being sent in clear text down the same connection? Of course, the same is true for login as well. Here again "who, what, and where" are nicely presented in easy to read form. Neither SCSI clients nor servers are likely able to authenticate independently or retain textual information. Both the client and server should depend on some outside database. This compromise in security was done to eliminate an IP selection option but then a textual means of including IP was included to ensure all modes of operation remain. Rather than using a secure and opaque means to establish a connection using an independent means of authenticating on a different connection, the entire function was included in this all encompassing draft. This was not a good choice in my view. By creating a separate draft to define an authentication database access (LDAP structures), options remain possible without impact on the transport specification. The only aspect of authentication within the transport should be a cookie. Authentication provides access at some IP and port or perhaps a set of such locations, it could also provide third-party mapping in binary as well. (This mapping could be encoded by various schemes.) As other transports are employed, flexibility becomes important. Why insist targets do DNS lookup? Keep the workload light as such lookups should never be needed beyond the process of authentication rightfully done by other equipment not involved in the time critical data handling. I hope this is comprehendible this time. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Monday, October 02, 2000 3:05 AM > To: ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > Doug, > > That is a nice and very long response. But what was the question? > > Julo > > "Douglas Otis" on 29/09/2000 22:21:32 > > Please respond to "Douglas Otis" > > To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu > cc: > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > Julo, > > You would not authenticate exclusively from a single token out of > the blue. > There must be an additional layer of authentication that indicates > IP:Port[] > x is customer 'y' and today they wish to connect to target z[]. > Only in the > most simplest configurations would a flat file provide this information. > You could easily have two servers that manage the customer and target > databases separately and join them using a 'Key.' This join would be a > database construct and not a SCSI. This additional management layer will > share with both ends and not pass over SCSI transport. With this > management > layer, binary representations are adaquate and provide far less > information > to those wishing to crack data. DNS, if used, would be run by a > management > layer and not a transport layer. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > julian_satran@il.ibm.com > > Sent: Friday, September 29, 2000 6:15 AM > > To: ips@ece.cmu.edu > > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > > > > > I would add that we wanted the path to be an additional > > identifier that the > > target could use > > to determine what collection of LUs to present to the initiator. > > > > Julo > > > > csapuntz@csapuntz-u1.cisco.com on 23/09/2000 05:00:33 > > > > Please respond to csapuntz@csapuntz-u1.cisco.com > > > > To: "IP Storage" > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > > > > > > > Just for clarification... I was not proposing to add the extended > > URL scheme for the transport spec. It isn't necessary. > > > > In this thread, Doug makes an excellent point about LDAP being a > > superior mechanism for describing how to connect to the > > storage. Directory services such as LDAP, as Doug has pointed out, > > will be critical to managing large quantities of storage. A host can > > ask such a directory service for a list of storage devices it should > > mount and how to connect to those storage devices. The query against > > the directory server that returns this information can be based on > > machine ID, user ID, operating system ID, or even the owner's > > birthday. One of the things discovery will end up doing, no doubt, is > > defining LDAP schemas that describe how to connect to storage > > (i.e. use SCTP or TCP, what port, what target name, what LUN, what > > WWN, how to authenticate, etc.). > > > > However, there is one place where the transport protocol has to define > > a name: third party commands. There needs to be some kind of global > > name which the initiator can pass to the target. The name must be > > distillable into a string. The target must understand the name and > > be able to use the information to establish a connection to > > another target. > > > > One could say that the string that is passed is not specified by the > > standard but instead specified by some management software. I think > > this will lead to poor interoperability. > > > > The SCSI URL-type name is the current proposal for target name. > > > > Why is SCSI target name made up of a hostname + a path? Why is the > > hostname + path passed on connection setup? There are two reasons. > > I think NAT and IPv6 makes passing hostnames rather than addresses in > > protocols more desirable. Hostnames can be re-resolved as you cross > > addressing boundaries. The path is there so that the name can > > support multiple targets behind a single IP address without having > > to add entries to the DNS server. At Cisco, for example, > > I have no control over the local DNS servers and cannot > > add DNS entries for the ATAPI DVD and floppy in my computer. > > > > _Costa > > > > > > > > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16090; 2 Oct 2000 11:14 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA02241 for ips-outgoing; Mon, 2 Oct 2000 10:28:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA28151 for ; Mon, 2 Oct 2000 08:21:59 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Mon, 2 Oct 2000 05:23:33 -0700 Message-ID: From: Robert Snively To: "'JoeBre@exabyte.com'" , ips@ece.cmu.edu Subject: RE: Status summary on multiple connections Date: Mon, 2 Oct 2000 05:21:24 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk > This may be a reasonable means of operation for the disk world. > It is woefully inadequate for the tape world, as follows: > > - In general, command ordering is crucial - out of order command > processing will lead to data corruption. > - This would require the initiator backup application to block on > completion of every single write command of a backup operation before > issuing the next command. > > If this blocking were performed, both the throughput and capacity of a tape > device/media would be negatively impacted by an order of magnitude or more. > This would occur even assuming an instantaneous transport. Sorry Joe, You folks already have given up on ordering. Your buffering structure is designed to perform lying writes. You gather in a command, gather in its data, post the completion and then pick up the next command. No present day tape drive except a few of the highly specialized Fibre Channel implementations allows ordered command queuing to be invoked from the host adapter and no driver exploits it because it is so rarely implemented. It is only with Rob Basham's new tape command set proposal that you are really going to get going full speed ahead into command queuing, and that is because the ordering information is contained in the command and the data can be pulled down as desired. The other major tape feature is copy management. That works perfectly, because the recipient or source of data is always a disk drive that is being managed by a tape drive. The tape can manage any re-ordering necessary. Bob From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00152; 28 Sep 2000 19:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA10991 for ips-outgoing; Thu, 28 Sep 2000 17:24:06 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA10986 for ; Thu, 28 Sep 2000 17:24:01 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8SLO1U19889 for ; Thu, 28 Sep 2000 16:24:01 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA05090; Thu, 28 Sep 00 16:22:20 -0500 Message-Id: <10009282122.AA05090@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: New List: rdma@cisco.com: to discuss RDMA In-Reply-To: Message from Charles Monia of "Wed, 27 Sep 2000 11:49:17 PDT." Date: Thu, 28 Sep 2000 16:22:31 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > However, I would support structuring the spec so that an RDMA > transport mechanism could be used underneath (I guess that's > motherhood). Not necessarily. You have to ask the implementors (particularly the hardware implementors), what sort of optional RDMA proposal they'd be happy with. My answer is none. It's mandatory or not at all. The reason for using RDMA is to make the implementation of iSCSI easier in hardware. If there are implementations which do not support the RDMA protocol, then your hardware implementation will have to support both the `easy path' (using RDMA) and the `hard path' (no RDMA). If you have to implement the hard path anyway, there's no point in implementing the easy path. The argument that you could make the hard path infrequent and implement it in software doesn't wash in this case. It just takes one implementation that doesn't do RDMA to slow your system to a crawl, and the competitor who only implemented the non-RDMA path makes you look like a fool. Fundamentally, RDMA has to be either adopted or punted. Of course, I'm happy to have somebody prove this statement wrong. Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01503; 29 Sep 2000 2:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id BAA23598 for ips-outgoing; Fri, 29 Sep 2000 01:36:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id BAA23588 for ; Fri, 29 Sep 2000 01:36:19 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id HAA81712 for ; Fri, 29 Sep 2000 07:35:48 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id HAA116982 for ; Fri, 29 Sep 2000 07:35:48 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256969.001EBC7E ; Fri, 29 Sep 2000 07:35:43 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 29 Sep 2000 08:33:23 +0300 Subject: RE: iSCSI: Flow Control Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, In your example C1, C2, D1, D2 will always work (as will any ordered combination). The troublesome situation is C1, C2+D2 (immediate), C3 (window closed) followed by an R2T for D1. This is why I think we should either explicitly forbid immediate data if R2T is enabled or specify that this behavior may get an initiator into trouble. Julo David Robinson on 21/09/2000 03:13:05 Please respond to David Robinson To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: iSCSI: Flow Control > I'm not a s/w engineer, so I could be getting myself into trouble here, > but I still see an issue.... > > iSCSI will not know whether the next PDU in the TCP buffer is a command > or data PDU until it actually reads the data. iSCSI should NEVER stop > reading > data from TCP or else deadlock if the next message is a data PDU (as was so > eloquently explained to me in a prior message). It must continue reading > data (thus emptying the TCP receive window), and if the SCSI command queue > is full, then SCSI will respond with TASK SET FULL. This works okay > (depending > on who you ask) in a parallel SCSI or FC world where latency is measured in > us, > but in iSCSI it could be 100ms and 1000+ commands later before the initiator > ever finds out. The TCP connection can have either a command or data. In either case reading a short fixed amount of data can determine if it is one or the other. With a single connection I can only see deadlock occuring if the initiator sends Command1, Command2, then Data1 on a single command/data connection. With a single connection this seems like a broken initiator as doing the right thing of C1, D1, C2 shouldn't be any more expensive and it is definately safer. Well one other case can occur is if the initiator wants the target to issue an RTT and it interposes another command before the RTT is received, this too seems like a broken things to do as I don't see much value in doing RTT on a single connection. In the multiple connection asymettric case this isn't an issue as data is never interleved with commands. So is there a fundamental requirement that the target must handle the case of C1, C2, then D1? It seems easy to just write the iSCSI initiator to never issue such a sequence and if the target gets one return an error when C2 arrives before D1. It again seems like we are trying to impose or solve datagram problems in a stream world. Does parallel SCSI have this problem? -David From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00254; 28 Sep 2000 19:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA12710 for ips-outgoing; Thu, 28 Sep 2000 18:11:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA12703 for ; Thu, 28 Sep 2000 18:11:25 -0400 (EDT) Received: from yp_portable (slip-32-101-212-213.ma.us.prserv.net [32.101.212.213]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id S3VDMD5L; Thu, 28 Sep 2000 15:08:44 -0700 From: "Y P Cheng" To: , Subject: RE: Why FCP doesn't need RDMA? It has a better way. Date: Thu, 28 Sep 2000 15:09:04 -0700 Message-ID: <002501c02998$b6f12a40$c6d46520@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101007@corpmx9.isus.emc.com> X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Black_David@emc.com > > With my WG co-chair hat off: > > From an iSCSI viewpoint, I don't see iSCSI by itself as being sufficient > to motivate a protocol-independent RDMA - an iSCSI HBA could understand > the iSCSI headers and interact with DMA in the same fashion as existing > HBAs. The task before those interested in RDMA is to identify a set > of protocols for which a common RDMA mechanism makes sense from > an engineering standpoint. I tend to agree with the previous emails > that iSCSI could make optional use of a common RDMA mechanism > if available, but must not REQUIRE its use. Yes, we must have a common DMA/RDMA mechanism for iSCSI whether the iSCSI PDU travels on Ethernet, 1394, fibre channel, or InifiniBand. (FYI, there are TCP/IP and SCSI implementations on 1394). For most adapters, not all, they get the DMA destination address from the device driver. The incoming data frames only specifies an offset from the destination. The exchange ID or SCSI tag identify a request, hence, the DMA destination. For VI RDMA the initiator sends a memory handle in the FC-VI header to the target, which, in turn, passes the handle back together with an offset to read or write the initiator's memory. For 1394 the target gets an ORB, Operation Request Block, from the initiator. In the ORB, a scatter and gather list of the initiator's memory is provided. The target works through the scatter/gather list using a "physical read or write" protocol unique to 1394. This lack of a commonality puts a lot burden on the adapters to deal with different protocols. IP, FCP, and VI all have different locations for their offsets and not everyone use the fibre channel exchange ID. I have not read the TCP/RDMA proposal, but, something like the ORB of 1394 would be nice. This gives total control to a target with minimum interference to an initiator. We should have a mechanism in iSCSI to allow some adapters to execute 100,000 IOs per second. Of course, not all adapters will be required to support DMA/RDMA. Keep in mind, while God forbids, some iSCSI implementation might not use TCP/IP or SCTP. I hope, this does not violate the charter the WG. Yes, this DMA/RDMA discussion can take months. :-) Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id aa12978; 1 Oct 2000 17:14 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa12825; 1 Oct 2000 16:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA10970 for ips-outgoing; Sun, 1 Oct 2000 15:19:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA10965 for ; Sun, 1 Oct 2000 15:19:07 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id VAA76530 for ; Sun, 1 Oct 2000 21:18:30 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id VAA183120 for ; Sun, 1 Oct 2000 21:18:30 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.006A0E0A ; Sun, 1 Oct 2000 21:18:24 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 22:16:06 +0300 Subject: Re: Bidirectional SCSI commands and iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, Both the examples you gave have something peculiar - the operations are to executed really in sequence (we are no talking about full duplex I/O operations?). I any case, in order no to extend excessively the headers we can either introduce a "next header" field (as in IP V6) or split the operation at the iSCSI level into two phases - each carrying its parameters and each ended with a status - like: - out XOP1, XOP2, OD1, -in S1, ID, S2 In any case commands having only a parameter set (fixed or variable) - like those for the object disk should be handled with the parameters sent as immediate data (as with the three party commands) - parameter size will probably be limited and the performance requirement will be similar to those of block read/write. Do we need for those to care also about residual counts for parameters? Julo "Jim Hafner/Almaden/IBM" on 29/09/2000 00:21:32 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Bidirectional SCSI commands and iSCSI Julo, The primary motivation is for object-based storage devices (OSDs). The protocol that seems to be needed in that environment is the ability to send a CDB, plus some parameter data (DataOut) AND get back read-type data (DataIn) in the same command sequence. One (non-OSD) command has been proposed in T10 for an XOR type XDREADWRITE command. SAM-2 has now allowed for such commands in the architecture. From this perspective, it is completely symmetric. Transports that support this feature will need to have data structures in there PDUs or IUs (or whatever) to indicate things like DataOut datalength, DataIn datalength, possible virtual addresses for each buffer, etc. In other words, there will potentially be two active buffers (on each end) for use with the same command. So, e.g., iSCSI PDU may need to have both "DataIn expected data transfer length" and "DataOut expected transfer length" fields. I'm assuming that the data direction in the existing iSCSI PDU is derived from the SCSI opcode, unlike FCP where there are fields to indicate the data direction. (I am surprised not to find such things in the iSCSI Command PDU.) There are two proposals in T10 to enable this feature in SPI-4 and in FCP-x (probably FCP-3). Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-28-2000 12:10:53 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, What is the intended use? Are there already commands that use it? Is there a major/minor distinction - like output data & input results or output parameters & input data - or is it intended to be completely symmetric? Julo "Jim Hafner/Almaden/IBM" on 20/09/2000 17:56:27 Please respond to "Jim Hafner/Almaden/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Bidirectional SCSI commands and iSCSI Folks, In case you missed this, T10 just approved changes to SAM-2 (rev14) which remove the restriction that a single SCSI command can have a data flow in only one direction (DataIn xor DataOut). Transport specs (that want to allow this) will need to change to allow for two sets of Data buffer fields in the headers (length, over/underrun, etc.). Is anybody looking to modify the iSCSI spec to get these concepts supported? Jim Hafner From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id ai11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa11290; 1 Oct 2000 7:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA01309 for ips-outgoing; Sun, 1 Oct 2000 05:55:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA01305 for ; Sun, 1 Oct 2000 05:55:09 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id LAA272828 for ; Sun, 1 Oct 2000 11:54:35 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id LAA35306 for ; Sun, 1 Oct 2000 11:54:35 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.00366E6E ; Sun, 1 Oct 2000 11:54:32 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 12:52:11 +0300 Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, The trouble we had with numbers is that they tie you up to a specific addressing scheme and not having any hierarch they don't scale well. In a contained environment that is fine but beyond it (and we are there) a flat numbering scheme is bad. I did not want to enter this territory that early but we are forced to choose between IPV4 and IPV6 - and we choose not to choose. Its that simple! Julo Robert Snively on 28/09/2000 20:04:55 Please respond to Robert Snively To: Julian Satran/Haifa/IBM@IBMIL, Douglas Otis cc: Daniel Smith/Almaden/IBM@IBMUS, ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Julo, I am sorry, but I disagree with you both practically and philosophically. Textual names are practically useless as identifiers, as witness the challenge of finding "Bob Smith" in an American phonebook. Every time you walk into a store where your name is kept in a data base, they ask for your phone number, not your name. Your passport, credit card, driver's license, and all other useful identifiers use a registered number set that allows for no (legal) duplication. SCSI has taken the same approach with LUN identification, recommending one of several registered numbers as the sole and immutable designation of the logical unit. Fibre Channel and Ethernet have taken the same approach at the Port/Node WWN and MAC address level. Use of a URL is a virtualization of the underlying name structure and should not be implemented except as a temporary convenience for the highest level client who is constrained to a very small search subset by his login authorizations. The virtualization is always at risk of security breaches. In the storage environment, the underlying structure is of vital interest. Any given data is present on specified physical and logical units and consistency of the data is maintained on the basis of an intermediate view of the lower level addressing structure which must also have the proper uniqueness guarantees. Address it with a URL on the browser if you wish, but understand that you are talking to a data block on a single storage unit attached to a network server having a specified IP address mapped to a registered MAC address. Bob > > There seem to be practical and philosophical issues that you > consistently > ignore. > Textual names are more stable than underlying network > addresses and thus > better > for applications (I assume that your card starts with your > name not phone > number and > that the former has changed less than the later). > I the iSCSI realm - and that is the only one under scrutiny > - a pertinent > observation was > made by David Black a century ago during our (too) long > naming discussions > - that a scheme that will allow the third party commands to > make the name > to address conversion > at use time is better than one that requires this mapping to > be made ahead > of time. > Unfortunately that is not yet possible with SCSI but we are > informed that > it soon be. > And for authentication the textual address in login is yet > another mean to > prevent > errors and/or unauthorized access if the target chooses to use it. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03911; 29 Sep 2000 16:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA16433 for ips-outgoing; Fri, 29 Sep 2000 15:30:53 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA16424 for ; Fri, 29 Sep 2000 15:30:47 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA15114; Fri, 29 Sep 2000 12:17:27 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA08324; Fri, 29 Sep 2000 12:15:14 -0700 From: "Douglas Otis" To: , Subject: RE: A Transport Protocol Without ACK (resend) Date: Fri, 29 Sep 2000 12:33:15 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, I agree, XON/XOFF would not work. A credit token scheme similar to FC Class 3 would. You may wish to review my draft as I have a simple scheme to illustrate the concept. This is also a genetic scheme proposed for SCTP as well. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Friday, September 29, 2000 6:49 AM > To: ips@ece.cmu.edu > Subject: RE: A Transport Protocol Without ACK (resend) > > > > > I don't doubt that such a mechanism could work and be somewhat useful. > Ordering between different XON/XOFFs is a bit more difficult > (the function can be easily piggybacked on R2T but R2Ts have no relative > ordering > or if we choose to do XON/XOFF at command boundary the command response > could > carry this information and responses are ordered). > > However I am under the impression that the TCP window (and in case of the > symmetrical > scheme or command sequence window) will perform the same function whenever > unsolicited data are enabled. > > Am I missing something? > > Julo > > Michael Krause on 23/09/2000 16:43:09 > > Please respond to Michael Krause > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: A Transport Protocol Without ACK (resend) > > > > > At 12:04 PM 9/22/00 -0600, HAAGENS,RANDY (HP-Roseville,ex1) wrote: > >Robert, > > > >The reason we proposed immediate write data for iSCSI was as an > optimization > >for short writes over "long" networks. As network diameter increases, > >minimizing round-trip signalling delays becomes important. Sending write > >data immediately following the command eliminates the need for > an RTT from > >the target, and thus eliminates a round-trip delay. > > > >If immediate data threatens to overflow the target's cache, the target > may, > >on an emergency basis, discard the data; and, when ready, request data > again > >using the RTT protocol. This behavior should be minimized, > however, as it > >will tend to congest the network. > > > >Note the use of SCSI-layer flow control (RTT mechanism). I agree with > those > >that say a controller must manage its cache, regardless of the > size of the > >cache. This is most apparent when one considers multiple hosts sharing a > >storage controller. In order to allocate performance between the hosts, > the > >controller must ensure that a write burst from a single host cannot fill > its > >cache. RTT provides a means for doing this that still allows other > commands > >(read commands, for example) from the same host to proceed. > > > >However, reserving a few MB of cache for each host, to accommodate > immediate > >writes, seems to me reasonable. The command windowing mechanism (another > >flow control mechanism) will prevent the overrunning of these immediate > >write buffers, assuming that the controller has reserved space in > proportion > >to the largest immediate buffer allowed (which is negotiated) x the > maximum > >number of commands. > > > >Upon writing this, it occurs to me that it may be useful to add an > >XOF/XON-like mechanism for temporarily disallowing immediate write data. > >This would give the controller another tool for cache management. When > its > >cache became congested, the controller could signal one or more of the > >requesting hosts to stop sending immediate write data with commands; > later, > >another message could allow immediate writes to resume. These messages > >would need to make it to the iSCSI layer in the hosts, as they > will affect > >how the host does DMA chain construction. > > Given the distances involved and the subsequent latency, I'd suggested > lifting the technique of issuing a RNR-NAK (receiver not ready) > with a time > period the sender should wait upon receiving the NAK before it retries the > operation (0 indicates immediate which may be acceptable if one knows the > sender is far away). Allows the receiver to allocate the requested buffer > space within what it believes to be a reasonable amount of time, > eliminates > the need for the receiver to track the sender's state for this > problem, and > eliminates the additional message associated with the XOFF/XON mechanism. > > In general, we should strive wherever possible to architect such that only > one side needs to track the state of a given attribute - this leads to > simplicity in implementation and future development of the architecture. > > Mike > > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03885; 29 Sep 2000 15:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA14959 for ips-outgoing; Fri, 29 Sep 2000 14:55:19 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel1.hp.com (atlrel1.hp.com [156.153.255.210]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA14936 for ; Fri, 29 Sep 2000 14:54:49 -0400 (EDT) Received: from omgw1.boi.hp.com (omgw1.boi.hp.com [15.56.8.101]) by atlrel1.hp.com (Postfix) with ESMTP id 0DF58698 for ; Fri, 29 Sep 2000 14:54:34 -0400 (EDT) Received: from xrosebh3.rsvl.itc.hp.com (xrosebh3.rsvl.itc.hp.com [15.34.240.67]) by omgw1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id MAA19716; Fri, 29 Sep 2000 12:54:29 -0600 (MDT) Received: by xrosebh3.rsvl.itc.hp.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 12:54:28 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: Matt Wakeley , IPS Reflector Subject: RE: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt Date: Fri, 29 Sep 2000 12:54:23 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, Use of the urgent pointer requires very careful investigation. If you look at the TCP RFC it mentions that that the urgent pointer location is a connection-wide pointer, both on the receiver side and the sender side. >>From the sender side, even if TCP emits a TCP segment for every user send, you are not guaranteed the preservation of the urgent pointer - e.g. 1. Let us say that a retransmission timeout occurs and a number of segments have to be retransmitted. In this case, based on (my interpretation) of RFC 793, only the last value of the urgent pointer is meaningful. 2. Let us say that user data gets queued up due to flow control issues. Again this will cause the last value of the urgent pointer to take effect. the second part will have an impact in the slow start case. A further analysis is required, but it seems that in cases where packets are being dropped, the urgent pointer setting will not be the most optimal (just about when it is needed the most). The urgent pointer proposal may have a benefit when you are loosing exactly one packet per window size (maybe even two with SACK). On the receive side, again the same thing happens when multiple segments with urg pointer are received. Since it is a connection- wide pointer, only the last one should be provided to the user. If the data passes through "intermediate TCP devices" then again the urgent pointer setting will be again be suboptimal. I would urge some TCP expert to jump in here. Somesh From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15661; 2 Oct 2000 8:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA28192 for ips-outgoing; Mon, 2 Oct 2000 08:23:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA28186 for ; Mon, 2 Oct 2000 08:23:54 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id HAA28582; Mon, 2 Oct 2000 07:23:17 -0500 Message-ID: <39D87E35.51050250@stewart.chicago.il.us> Date: Mon, 02 Oct 2000 07:23:17 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: David Robinson CC: ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: <200009291825.LAA13471@ha10nwk.EBay.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David Robinson wrote: > > > I am left with the following impression as to what was indicated here: > > - In general, command ordering is not relevant > > - If the initiator filesystem detects an ordering dependency, it will wait > > until outstanding commands are complete before issuing the dependant > > command. > > > > This may be a reasonable means of operation for the disk world. It is > > woefully inadequate for the tape world, as follows: > > - In general, command ordering is crucial - out of order command processing > > will lead to data corruption. > > - This would require the initiator backup application to block on completion > > of every single write command of a backup operation before issuing the next > > command. > > > > If this blocking were performed, both the throughput and capacity of a tape > > device/media would be negatively impacted by an order of magnitude or more. > > This would occur even assuming an instantaneous transport. > > I am hearing different stories on the issue of ordering. One side > is pushing hard for techniques that will allow out of order > execution using various RDMA techniques. This clearly states for > a certain class of devices (e.g. tapes) ordering is crucial. > I thought this problem was already solved at the SCSI layer > through the use of ordered commands which in general are not used > for disks but always used for tapes? Since FC will reorder this > has to be a solved problem. Would not an initiator talking to > a tape target simply set the ordering flag? > > Lastly, for a TCP based connection ordering can easily be made a > non-issue, simply don't try to process segments out of order. I > will defer to a transport expert, but I believe processing > TCP segments by an application out of order might cause problems. > In particular since the out of order segment is not ACKed until > after the missing segments arrive, they can be retransmitted > multiple times. SACK helps this but does not guarentee that > segments will not be retransmitted. So to process out of order > segments the application must maintain a list of which segments > have been processed as well, yuck! > David: If a TCP stack were to allow delivery out of order... and I don't see how it could since by definition it is a stream of bytes sent in order... it would have to keep track of the segments it has received and prevent duplicates from being propagated up to the application above. As you state, SACK helps, but it can NOT eliminate the duplicates that MAY show up. I think the TCP stack itself would HAVE to track all of this not the application... I really do NOT see how one can allow a TCP to deliver out of order though, oh I know technically how to do it, but it is a fundamental violation of what TCP is supposed to be delivering.... i.e. the bytes being transfered from 0 to N... R > -David > -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04956; 29 Sep 2000 22:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA28498 for ips-outgoing; Fri, 29 Sep 2000 21:39:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA28489 for ; Fri, 29 Sep 2000 21:38:57 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 21:38:26 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101031@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt Date: Fri, 29 Sep 2000 21:38:25 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk This appears to have crossed onto somewhat shaky ground. RFC 793 does specify the coalescing, and hence fiddling with that should not be the first choice option. Somesh is right when he recommends "very careful investigation"; figure out what the common scenarios are that are important to optimize and see how well the urgent pointer works without modification, even if its value is not always optimal for the situation. In general, TCP implementations don't react well to multiple drops, so single drop scenarios are the right place to start looking. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- > -----Original Message----- > From: Matt Wakeley [SMTP:matt_wakeley@agilent.com] > Sent: Friday, September 29, 2000 9:17 PM > To: GUPTA,SOMESH (HP-Cupertino,ex1); IPS Reflector > Subject: Re: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt > > Somesh, > > What you describe is how current TCP stacks typically operate when it does > not > know anything about the protocol running on top of it. A customized TCP > stack > that was closely coupled with the iSCSI protocol running on top of it > could > provide the appropriate urgent pointer in each segment that is > (re)transmitted > without "coalescing" the urgent data. Likewise a customized TCP inbound > path > that was closely coupled with the iSCSI protocol running on top of it > could > pass the urgent information to the iSCSI layer without "coalescing" the > urgent > data. This could be done without any modifications or violations of > current > TCP specifications. > > -Matt > > "GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > > > Matt, > > > > Use of the urgent pointer requires very careful investigation. If you > > look at the TCP RFC it mentions that that the urgent pointer > > location is a connection-wide pointer, both on the receiver side > > and the sender side. > > > > >From the sender side, even if TCP emits a TCP segment for every > > user send, you are not guaranteed the preservation of the urgent > > pointer - e.g. > > > > 1. Let us say that a retransmission timeout occurs and a number > > of segments have to be retransmitted. In this case, based on > > (my interpretation) of RFC 793, only the last value of the > > urgent pointer is meaningful. > > > > 2. Let us say that user data gets queued up due to flow control > > issues. Again this will cause the last value of the urgent > > pointer to take effect. > > > > the second part will have an impact in the slow start case. A > > further analysis is required, but it seems that in cases where > > packets are being dropped, the urgent pointer setting will not > > be the most optimal (just about when it is needed the most). > > The urgent pointer proposal may have a benefit when you are > > loosing exactly one packet per window size (maybe even two > > with SACK). > > > > On the receive side, again the same thing happens when multiple > > segments with urg pointer are received. Since it is a connection- > > wide pointer, only the last one should be provided to the user. > > > > If the data passes through "intermediate TCP devices" then again > > the urgent pointer setting will be again be suboptimal. > > > > I would urge some TCP expert to jump in here. > > > > Somesh From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04477; 29 Sep 2000 19:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA24096 for ips-outgoing; Fri, 29 Sep 2000 18:49:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA24091 for ; Fri, 29 Sep 2000 18:49:38 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 18:49:07 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A070410102B@corpmx9.isus.emc.com> To: dotis@sanlight.net, matt_wakeley@agilent.com, ips@ece.cmu.edu Subject: Enough on TCP vs. FCTP, please Date: Fri, 29 Sep 2000 18:49:07 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > End schemes to fix TCP; address the issues of TCP and SCTP. Do not stand on > desires for TCP over SCTP or that a pact was made to promote TCP. SCTP > performance will be superior to TCP using either hardware or software. SCTP > is easier to accelerate in hardware than TCP and any resulting SCSI standard > will change substantially as a result of the features of SCTP. Even if > there was a generic RDMA option, SCSI would look more like Firewire as a > result. A debased version of RDMA has already been offered that avoids > stepping on accepted options. I think this sort of debate really needs to stop. Matt's draft is a reasonable attempt to apply an existing TCP feature to address a shortcoming in use of TCP for iSCSI. It is not perfect, as is the case for most engineering tradeoffs and compromises. SCTP does have a number of improvements over TCP (e.g., the whole session discussion is moot for SCTP), but TCP has significantly more deployment experience and a great deal of work in progress on hardware acceleration. I am restating the WG consensus that iSCSI needs to anticipate both SCTP and TCP, with the assumption that TCP deployments will come first. Given this, Matt's draft is within scope for the WG. If anyone other than Doug Otis disagrees, please say so and say why on the mailing list. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04487; 29 Sep 2000 19:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23231 for ips-outgoing; Fri, 29 Sep 2000 18:19:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lsi.lsil.com (lsi.lsil.com [147.145.40.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23226 for ; Fri, 29 Sep 2000 18:19:17 -0400 (EDT) Received: from mhbs.lsil.com ([147.145.31.100]) by lsi.lsil.com (8.9.3+Sun/8.9.1) with ESMTP id PAA25805 for ; Fri, 29 Sep 2000 15:19:15 -0700 (PDT) Received: from inca.co.lsil.com by mhbs.lsil.com with ESMTP for ips@ece.cmu.edu; Fri, 29 Sep 2000 15:19:01 -0700 Received: from exw-kansas.ks.lsil.com (exw-kansas.ks.lsil.com [153.79.8.7]) by inca.co.lsil.com (8.9.3/8.9.3) with ESMTP id QAA04273 for ; Fri, 29 Sep 2000 16:18:59 -0600 (MDT) Received: by exw-kansas.ks.lsil.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 17:16:01 -0500 Message-Id: From: "Binford, Charles" To: ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Fri, 29 Sep 2000 17:16:00 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C02A62.D95A8A48" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C02A62.D95A8A48 Content-Type: text/plain; charset="iso-8859-1" Just to clarify things. FCP-2 did not forbid immediate data. Bob Snively's recent posting as to what was removed in FCP-2 was the combining of command and data IN THE SAME SEQUENCE. This is an FC specific issue and does not apply to iSCSI. FCP-2 still has the concept of initial Write XFER_RDY disable. This allows up to burst size (set in a Mode page) amount of data to be sent immediately after the command (but in a different FC sequence). That said, I am not aware of anyone who actually supports this option. If iSCSI decides this is a good thing, then I believe the command queue depth management schemes built into SCSI are not sufficient. Bob Snively has recently spelled out how those work in another posting. I agree they work in an environment where immediate data is not allowed. Retransmitting a short CDB pdu/frame is not a big deal. However, if data was sent and it has to be retransmitted in addition to the CDB, then it suddenly becomes an issue. Also, I believe the current command queue depth handling works acceptably well because most current SCSI implementations (i.e. parallel, fc, 1394, etc.) all have very short transport latency times. If we're talking about retransmitting a "queue-fulled" CDB cross country, then I think we need to consider a method that governs the queue depth and not assume the current SCSI mechanism will work in this environment. Charles Binford LSI Logic Storage Systems (316) 636-8566 > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, September 29, 2000 3:39 PM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: iSCSI: Flow Control > > > > Sorry for not being more explicit. The window can close as > C2+D2 (but > > mainly D2) are not consumed as they are processed after C1. > > > > The scenario holds and the blocking results from a "order > reversal" of the > data in the stream > > caused by enabling both immediate data and R2T. > > In which case, this target should forbid immediate data if its buffer > resources > are that slim; I think we're in violent agreement on that. > > Putting my WG co-chair hat back on: > > The open question to the list is whether there's value in allowing > some amount of immediate data (e.g., for targets that need > fast startup on > long latency connections, and are prepared to deploy the > buffering required > to make it work reliably), or whether we ought to follow > FCP-2 and forbid > immediate data, which will impose a round trip delay (command out, R2T > back) before data starts to flow. I think I've seen a couple > of comments > in favor of this, but more discussion is in order. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > > ------_=_NextPart_001_01C02A62.D95A8A48 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: iSCSI: Flow Control

Just to clarify things.  FCP-2 did not forbid = immediate data.  Bob Snively's recent posting as to what was = removed in FCP-2 was the combining of command and data IN THE SAME = SEQUENCE.  This is an FC specific issue and does not apply to = iSCSI.

FCP-2 still has the concept of initial Write XFER_RDY = disable.  This allows up to burst size (set in a Mode page) amount = of data to be sent immediately after the command (but in a different FC = sequence).

That said, I am not aware of anyone who actually = supports this option.  If iSCSI decides this is a good thing, then = I believe the command queue depth management schemes built into SCSI = are not sufficient.  Bob Snively has recently spelled out how = those work in another posting.  I agree they work in an = environment where immediate data is not allowed.  Retransmitting a = short CDB pdu/frame is not a big deal.  However, if data was sent = and it has to be retransmitted in addition to the CDB, then it suddenly = becomes an issue.

Also, I believe the current command queue depth = handling works acceptably well because most current SCSI = implementations (i.e. parallel, fc, 1394, etc.) all have very short = transport latency times.  If we're talking about retransmitting a = "queue-fulled" CDB cross country, then I think we need to = consider a method that governs the queue depth and not assume the = current SCSI mechanism will work in this environment.


Charles Binford
LSI Logic Storage Systems
(316) 636-8566


> -----Original Message-----
> From: Black_David@emc.com [mailto:Black_David@emc.com]
> Sent: Friday, September 29, 2000 3:39 PM
> To: julian_satran@il.ibm.com; = ips@ece.cmu.edu
> Subject: RE: iSCSI: Flow Control
>
>
> > Sorry for not being more explicit. The = window can close as
> C2+D2 (but
> > mainly D2) are not consumed as they are = processed after C1.
> >
> > The scenario holds and the blocking = results from a "order
> reversal" of the
> data in the stream
> > caused by enabling both immediate data and = R2T.
>
> In which case, this target should forbid = immediate data if its buffer
> resources
> are that slim; I think we're in violent = agreement on that.
>
> Putting my WG co-chair hat back on:
>
> The open question to the list is whether = there's value in allowing
> some amount of immediate data (e.g., for = targets that need
> fast startup on
> long latency connections, and are prepared to = deploy the
> buffering required
> to make it work reliably), or whether we ought = to follow
> FCP-2 and forbid
> immediate data, which will impose a round trip = delay (command out, R2T
> back) before data starts to flow.  I think = I've seen a couple
> of comments
> in favor of this, but more discussion is in = order.
>
> --David
>
> = ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, = MA  01748
> +1 (508) 435-1000 = x75140     FAX: +1 (508) 497-8500
> = black_david@emc.com       Mobile: +1 = (978) 394-7754
> = ---------------------------------------------------
>
>

------_=_NextPart_001_01C02A62.D95A8A48-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id ae11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10965; 1 Oct 2000 5:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA00148 for ips-outgoing; Sun, 1 Oct 2000 04:30:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00133 for ; Sun, 1 Oct 2000 04:29:59 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA35582; Sun, 1 Oct 2000 10:29:26 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA42176; Sun, 1 Oct 2000 10:29:26 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.002E9C56 ; Sun, 1 Oct 2000 10:29:06 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: "Randall R. Stewart" cc: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 09:14:18 +0300 Subject: Re: VI (Was: Avoiding deadlock in iSCSI) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Randall, Many in this community are aware of SCTP. And we will certainly consider it when it will get will get even a small-percentage of the world wide support that TCP has or even earlier. However it is EXTREMELY COUNTERPRODUCTIVE to preach SCTP when we are looking for a solution WITHIN TCP. If your experience with SCTP can help us find a better solution WITHIN TCP please let us know. If not make a note, mental or otherwise, about another good thing in SCTP and keep (or publish) a list of those thinks - we may want it on day - but don't add to the noise here. Thanks, Julo "Randall R. Stewart" on 27/09/2000 14:59:38 Please respond to "Randall R. Stewart" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: VI (Was: Avoiding deadlock in iSCSI) Julian: julian_satran@il.ibm.com wrote: > > Charles, > > This goes to the hearth of problem. > To avoid this effect we propose (like FCP) unordered messages to go always > through - > allow the initiator to decide which task management messages he deems > "urgent". > But if you have a single connection and the TCP window is closed you are > left with only > one think to do - drop the connection (and that is exactly what FCP is > doing for similar > reasons!). > This is exactly what SCTP's un-ordered message services was added for. It solves this very issue. R > Julo > > Charles Monia on 16/09/2000 01:19:37 > > Please respond to Charles Monia > > To: Stephen Byan , IPS Reflector > > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > -----Original Message----- > > From: Stephen Byan [mailto:Stephen.Byan@quantum.com] > > Sent: Wednesday, September 13, 2000 6:22 AM > > To: IPS Reflector > > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > > > > Matt Wakeley [mailto:matt_wakeley@agilent.com] wrote: > > > > > I don't think VI/TCP helps at all. The VI is implemented on > > > top of a TCP > > > stream. If the TCP stream delivers the iSCSI command to VI, > > > and VI has no > > > place to put it, what is VI going to do? It has to block the > > > TCP stream, and > > > that in turn will block and "RDMA" from occurring. > > > > If I understand correctly, VI/TCP has a credit-based flow > > control mechanism > > on its message queues (which would be used to implement the > > iSCSI command > > reception queue), and so the initiator would never send an > > iSCSI command > > without a target buffer in which to store it. So RDMA cannot block on > > commands in the TCP stream. > > > > In the context of generic ULP "flow control" issues, the following > observations are offered. > > Assuming one VI/TCP message queue handles all SCSI control traffic for a > target, using flow control on messages to prevent the "queue-full" > condition > has undesirable side effects. Specifically, back pressure due to lack of > command context resources within a logical unit would stop the flow of > commands and task management requests to all other LUNS (a form of "head of > line" blocking). > > There's another issue that is perhaps less obvious. Even if multiple > message > queues are used, say one per LUN, stopping the flow of messages due to a > queue full condition would also stop the flow of task management messages > as > well. > > Charles Monia > Senior Technology Consultant > Nishan Systems Corporation > email: cmonia@nishansystems.com > voice: (408) 519-3986 > fax: (408) 435-8385 -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16208; 2 Oct 2000 12:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA04280 for ips-outgoing; Mon, 2 Oct 2000 11:18:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA04274 for ; Mon, 2 Oct 2000 11:17:59 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e92FHxS26480 for ; Mon, 2 Oct 2000 10:17:59 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA09402; Mon, 2 Oct 00 10:16:17 -0500 Message-Id: <10010021516.AA09402@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: New List: rdma@cisco.com: to discuss RDMA In-Reply-To: Message from Michael Krause of "Thu, 28 Sep 2000 15:41:03 PDT." <4.2.2.20000928152047.00b7a940@hpindlm.cup.hp.com> Date: Mon, 02 Oct 2000 10:16:11 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > We had a similar debate within InfiniBand on just this subject > and it really came down to implementation specific requirements. I'm not completely up on IB, but what you are describing doesn't appear to be precisely analogous to the negotiable use of RDMA in iSCSI. It seems like you are talking about the subsetting of the capabilities of a general RDMA mechanism for application specific purposes. For example, it makes no sense to require an implementation of RDMA READ on an endpoint is only ever a data sink. Similarly, whether any particular data is transferred as SEND or WRITE also depends upon a choice in the design of the ULP. A general RDMA protocol provides a pallete of options that ULPs can use to accomplish their required data transfer. Asymmetry of requirements is an implication asymmetry of ULP operation. `Host' (general) endpoints are required to implement the whole pallete because it allows ULPs to be designed using any element of the pallete freely as appropriate. Target (application specific) endpoints need only implement the operations required by the ULP, or even just a subset of the ULP they chose implement. They are specialized precisely for their function. Whether iSCSI uses RDMA or an iSCSI specific tagged transfer model doesn't matter from the standpoint of making iSCSI work. The problem with using an iSCSI specific transfer model is that you can't use it for anything else. A general RDMA mechanism can be used for other purposes. > I don't view RDMA or any of these implementations as panacea > solutions for this overall resource usage problem. Was this on the table? Somebody mentioned in an earlier message that SAM-2 requires that the buffers be pinned. However, any transfer mechanism that has bidirectional buffer flow control probably has the potential to support a `rolling pin'. > The benefits of RDMA are associated with the ability to reduce the > amount of buffering required within a solution at the various points > along the way and deliver zero-processor copy placement and control. > These can be implemented in other ways as others have pointed out > and thus the cost / benefit needs will vary. I don't believe that a zero copy placement implemented with a the tagged transfer model is any more costly than one implemented with RDMA. If you are suggesting that the adapter must be thick in one case and thin in the other, I don't see it. I said in my previous message that using RDMA would make iSCSI implementation easier. I take it back. It's not clear that this is actually the case. The real advantage of a general RDMA is that it can bring the benefits traditionally associated with storage adapters (zero copy, and low CPU overhead) to protocols other than SCSI. However, it also seems like an advantage that a high performance iSCSI on RDMA implementation can be built with no iSCSI specific hardware at all. > Let the market decide whether that one implementation should survive or > not. Most likely it will die as others will have been smarter in their > designs and functional selection. Exactly. If the tagged transfer mode is well designed, and RDMA is not required, nobody will do RDMA for iSCSI. Whether to use RDMA or not is a choice that the iSCSI standard is going to have to make. Either way works. However, I do not believe there is a sensible way to allow for both possibilities. Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03978; 29 Sep 2000 16:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA16493 for ips-outgoing; Fri, 29 Sep 2000 15:32:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA16484 for ; Fri, 29 Sep 2000 15:31:59 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 15:31:27 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101023@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Fri, 29 Sep 2000 15:31:26 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk With my WG co-chair hat off: > In your example C1, C2, D1, D2 will always work (as will any ordered combination). > The troublesome situation is C1, C2+D2 (immediate), C3 (window closed) > followed by an R2T for D1. This is why I think we should either explicitly > forbid immediate data if R2T is enabled or specify that this behavior may > get an initiator into trouble. I think we need to be careful about reinventing things that work in the SCSI world. Focus in a discussion like this really needs to be on how the presence of a IP-based network changes things. So, let's work backwards. The window closes because the initiator doesn't receive window updates from the target. Either the updates were lost, or the target didn't send them. If the updates are lost, the R2T will reopen the window, so the problem must be that the target didn't send the updates. If the target is temporarily overloaded, the window will reopen shortly. If it doesn't, then something is broken in the target and possibly the initiator because the same thing would go wrong on any transport. Between the technique already described that allows a SCSI target to set limits on the max size of immediate data and the target's responsibility for controlling resource allocation, a properly engineered target should not leave windows closed for extended periods of time. A target that permits transfer of more data than it can safely buffer is prone to breakage, and is likely to lose market share to targets that don't screw up in this fashion. Recalling that SCSI puts a target in complete control of its resource allocation, if immediate data is allowed, then we need to reuse the existing SCSI mechanism that allows a target to limit the amount of immediate data. Targets that overcommit their buffering resources are responsible for the consequences, and efforts should not be spent here to accommodate poor implementation decisions. We should definitely permit targets to follow Julian's approach of prohibiting immediate data, but I don't think that we should require it. The one thing that does change here is that the presence of the immediate data makes it easier to fill a TCP window, costing a round trip delay to get the next update -- this is an opportunity for implementers to apply some intelligent tuning of the sizes of advertised TCP windows to make this unlikely in practice. Also note that for the case in which C1 and C2 are ordered, the initiator has done something colossally stupid by using R2T for C1's data and sending C2's data in line. This is a case in which the best that can be expected is for things to work, and not a case for which performance ought to be optimized. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04374; 29 Sep 2000 18:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23277 for ips-outgoing; Fri, 29 Sep 2000 18:20:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mta5.snfc21.pbi.net (mta5.snfc21.pbi.net [206.13.28.241]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23271 for ; Fri, 29 Sep 2000 18:20:09 -0400 (EDT) Received: from SERVER.PacBell.net ([63.199.7.253]) by mta5.snfc21.pbi.net (Sun Internet Mail Server sims.3.5.2000.01.05.12.18.p9) with ESMTP id <0G1O0059H4ZHZM@mta5.snfc21.pbi.net> for IPS@ECE.CMU.edu; Fri, 29 Sep 2000 14:57:17 -0700 (PDT) Date: Fri, 29 Sep 2000 14:56:49 -0700 From: Peter Johansson Subject: RE: iSCSI: Flow Control In-reply-to: <0F31E5C394DAD311B60C00E029101A0704101027@corpmx9.isus.emc.com> X-Sender: Celeborn@PacBell.net To: IP Storage Message-id: <4.3.2.7.2.20000929144814.00bd6bd0@PacBell.net> MIME-version: 1.0 X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Content-type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 04:39 PM 9/29/00, Black_David@emc.com wrote: >The open question to the list is whether there's value in allowing some >amount of immediate data (e.g., for targets that need fast startup on >long latency connections, and are prepared to deploy the buffering >required to make it work reliably), or whether we ought to follow FCP-2 >and forbid immediate data, which will impose a round trip delay (command >out, R2T back) before data starts to flow. I think I've seen a couple of >comments in favor of this, but more discussion is in order. I suggest that you split the question: 1) Is immediate data desirable when the initiator has certain knowledge that the LU is idle (with respect to the initiator)? 2) Is immediate data desirable when the LU may be busy. I think the answer to 1) is "Yes", make provision for the initiator to send immediate data---but also make provision for the initiator to obtain some a priori knowledge as to the quantity of data the LU will accept in the idle state. There need not be a negotiation between initiator and LU; it could be sufficient for the LU to advertise its capabilities. In the case of 2), the answer might vary dependent upon whether or not those LU resources used to obtain data from the initiator are fully occupied when the LU is other than idle. If most of the optimization is realized in case 1), it might obviate the need for complicated schemes needed to make immediate data in case 2) useful. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00252; 28 Sep 2000 19:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA13164 for ips-outgoing; Thu, 28 Sep 2000 18:24:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lsi.lsil.com (lsi.lsil.com [147.145.40.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA13158 for ; Thu, 28 Sep 2000 18:24:43 -0400 (EDT) Received: from mhbs.lsil.com ([147.145.31.100]) by lsi.lsil.com (8.9.3+Sun/8.9.1) with ESMTP id PAA16582 for ; Thu, 28 Sep 2000 15:24:42 -0700 (PDT) Received: from inca.co.lsil.com by mhbs.lsil.com with ESMTP for ips@ece.cmu.edu; Thu, 28 Sep 2000 15:24:23 -0700 Received: from exw-kansas.ks.lsil.com (exw-kansas.ks.lsil.com [153.79.8.7]) by inca.co.lsil.com (8.9.3/8.9.3) with ESMTP id QAA22468 for ; Thu, 28 Sep 2000 16:24:22 -0600 (MDT) Received: by exw-kansas.ks.lsil.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 17:21:26 -0500 Message-Id: From: "Binford, Charles" To: ips@ece.cmu.edu Subject: RE: twist on iSCSI asymmetric model Date: Thu, 28 Sep 2000 17:21:25 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C0299A.7092659E" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C0299A.7092659E Content-Type: text/plain; charset="iso-8859-1" See below... Charles Binford LSI Logic Storage Systems (316) 636-8566 > -----Original Message----- > From: meth@il.ibm.com [mailto:meth@il.ibm.com] > Sent: Thursday, September 28, 2000 8:31 AM > To: ips@ece.cmu.edu > Subject: twist on iSCSI asymmetric model > [snip] > > > The use of R2T is negotiated during the first login of the > iSCSI session. > If R2T is to be used, then the initiator may not send data > before it is > requested by the target. Thus there will always be buffers > ready to accept > the data. If the target agrees to not use R2T, then the > target implicitly > agrees to have some number of buffers available to receive > data sent by the > initiator, even if the data arrives before the command > arrives. I could agree with this IF we use the same rule as FCP's Write XFER_RDY Disable bit. When this bit is set, it tells the initiator that it shall send the first burst of data after the command, but before the XFER_RDY. The maximum size of the this first burst of data is set by a mode page. If the I/O is larger that the first burst maximum, the initiator shall wait for an XFER_RDY for the balance of the data. The key here is the target KNOWS ahead of time the maximum number of buffers that will be used per command on an unsolicited basis. This covers the latency issue with small writes. When the initiator sends the occasional large I/O, the first burst goes as normal, then the balance comes when the target asks for it, after it has had time to allocate the additional buffer space. > In this > case the initiator MUST send data in the data connections in > the same order > that the corresponding commands were sent. (This avoids > deadlocks.) If the > target runs out of buffers, then it will let data back up in > the TCP window > of the data connections, while it is processing the data from earlier > commands. > [snip] > > - Kalman Meth. > > > > ------_=_NextPart_001_01C0299A.7092659E Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: twist on iSCSI asymmetric model

See below...

Charles Binford
LSI Logic Storage Systems
(316) 636-8566


> -----Original Message-----
> From: meth@il.ibm.com [mailto:meth@il.ibm.com]
> Sent: Thursday, September 28, 2000 8:31 = AM
> To: ips@ece.cmu.edu
> Subject: twist on iSCSI asymmetric model
>
[snip]
>
>
> The use of R2T is negotiated during the first = login of the
> iSCSI session.
> If R2T is to be used, then the initiator may = not send data
> before it is
> requested by the target. Thus there will always = be buffers
> ready to accept
> the data. If the target agrees to not use R2T, = then the
> target implicitly
> agrees to have some number of buffers available = to receive
> data sent by the
> initiator, even if the data arrives before the = command
> arrives.

I could agree with this IF we use the same rule as = FCP's Write XFER_RDY Disable bit.  When this bit is set, it tells = the initiator that it shall send the first burst of data after the = command, but before the XFER_RDY.  The maximum size of the this = first burst of data is set by a mode page.  If the I/O is larger = that the first burst maximum, the initiator shall wait for an XFER_RDY = for the balance of the data. 

The key here is the target KNOWS ahead of time the = maximum number of buffers that will be used per command on an = unsolicited basis.  This covers the latency issue with small = writes.  When the initiator sends the occasional large I/O, the = first burst goes as normal, then the balance comes when the target asks = for it, after it has had time to allocate the additional buffer = space. 


> In this
> case the initiator MUST send data in the data = connections in
> the same order
> that the corresponding commands were sent. = (This avoids
> deadlocks.) If the
> target runs out of buffers, then it will let = data back up in
> the TCP window
> of the data connections, while it is processing = the data from earlier
> commands.
>
[snip]
>
> - Kalman Meth.
>
>
>
>

------_=_NextPart_001_01C0299A.7092659E-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01093; 29 Sep 2000 0:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA21392 for ips-outgoing; Thu, 28 Sep 2000 23:53:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA21386 for ; Thu, 28 Sep 2000 23:53:22 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id 77B1A4F9 for ; Thu, 28 Sep 2000 20:53:21 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id UAA02133 for ; Thu, 28 Sep 2000 20:53:57 -0700 (PDT) Message-ID: <39D41225.58F47E7@agilent.com> Date: Thu, 28 Sep 2000 20:53:10 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 Cc: IPS Reflector Subject: Re: Why FCP doesn't need RDMA? It has a better way. References: <0F31E5C394DAD311B60C00E029101A0704101007@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Yahoo! for once I agree with David! Nice summary! -Matt Black_David@emc.com wrote: > With my WG co-chair hat off: > > > Actually, RDMA is not needed in FCP because all protocol chips > > implemented perform a real peer-to-peer DMA straight to the > > data areas specified by the user's interaction with the operating > > systems allocation algorithms. The combination of the FCP/SCSI > > pointer structure, task tagging, and the FC relative offset perform the > > function you would otherwise have to use RDMA to accomplish. > > And this illuminates the design tradeoff that may motivate RDMA. If > one only wants to accelerate one protocol (SCSI/FCP in the above > example) then having hardware understand its headers and doing > the DMA on that basis is a fairly obvious way to go - HBAs for both > parallel SCSI and Fibre Channel (SCSI/FCP) do this. RDMA may be > interesting if there are multiple protocols involved, and there are > engineering > concerns that lead to not wanting to implement hardware support for > all of them. > > From an iSCSI viewpoint, I don't see iSCSI by itself as being sufficient > to motivate a protocol-independent RDMA - an iSCSI HBA could understand > the iSCSI headers and interact with DMA in the same fashion as existing > HBAs. The task before those interested in RDMA is to identify a set > of protocols for which a common RDMA mechanism makes sense from > an engineering standpoint. I tend to agree with the previous emails > that iSCSI could make optional use of a common RDMA mechanism > if available, but must not REQUIRE its use. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00445; 28 Sep 2000 20:33 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA15475 for ips-outgoing; Thu, 28 Sep 2000 19:41:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA15466 for ; Thu, 28 Sep 2000 19:40:58 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA14843; Thu, 28 Sep 2000 16:27:46 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA05332; Thu, 28 Sep 2000 16:25:32 -0700 From: "Douglas Otis" To: "Stephen Bailey" , Subject: RE: New List: rdma@cisco.com: to discuss RDMA Date: Thu, 28 Sep 2000 16:43:23 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <10009282122.AA05090@candide.cs.uchicago.edu> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Steve, There should be two generic automatic delivery schemes. One that allows a remote agent to determine what data is to be delivered and where. This would be VI for such things like DAFS. With such a scheme, specifics of where to put data is sent with the data after negotiations of intimate details with highly trusted systems. The other is SCSI where location is understood by reference and only the offset is included, as such, scatter/gather is done locally in a far safer manner. FCP structures are stable for such use. As far as either hardware or software, SCTP would allow both without interference. There would be no hard or easy way, but simply one way. As hardware became available, these features could be added without any noticed change to the data structures or placement. The concepts of RDMA is essentially a restructuring of TCP to aid hardware assistance. Such is not required for SCTP nor would TCP safely allow out of sequence processing without restructuring. SCTP can be implemented without kernel changes whereas RDMA require extensive changes to the TCP kernel and API. A clean and clear path is SCTP for both hardware, software, kernel and non-kernel implementations. In addition, to achieve stability with SCSI structures, retain FCP definitions. These FCP structures have been implemented to facilitate such hardware acceleration. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Stephen Bailey > Sent: Thursday, September 28, 2000 2:23 PM > To: ips@ece.cmu.edu > Subject: Re: New List: rdma@cisco.com: to discuss RDMA > > > > However, I would support structuring the spec so that an RDMA > > transport mechanism could be used underneath (I guess that's > > motherhood). > > Not necessarily. > > You have to ask the implementors (particularly the hardware > implementors), what sort of optional RDMA proposal they'd be happy > with. My answer is none. It's mandatory or not at all. > > The reason for using RDMA is to make the implementation of iSCSI > easier in hardware. If there are implementations which do not support > the RDMA protocol, then your hardware implementation will have to > support both the `easy path' (using RDMA) and the `hard path' (no > RDMA). If you have to implement the hard path anyway, there's no > point in implementing the easy path. > > The argument that you could make the hard path infrequent and > implement it in software doesn't wash in this case. It just takes one > implementation that doesn't do RDMA to slow your system to a crawl, > and the competitor who only implemented the non-RDMA path makes you > look like a fool. > > Fundamentally, RDMA has to be either adopted or punted. Of course, > I'm happy to have somebody prove this statement wrong. > > Steph > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16223; 2 Oct 2000 12:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA04571 for ips-outgoing; Mon, 2 Oct 2000 11:24:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA04558 for ; Mon, 2 Oct 2000 11:24:38 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA08946 for ; Mon, 2 Oct 2000 11:18:06 -0400 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e92FOCd81714 for ; Mon, 2 Oct 2000 09:24:12 -0600 Importance: Normal Subject: Re: Bidirectional SCSI commands and iSCSI To: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Mon, 2 Oct 2000 08:24:09 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/02/2000 08:24:12 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, True, those examples are not true duplex operations. These are only the current thinking or motivation for bi-directional commands. They are not the end of the story, I'm sure. However, the only SAM-2 requirement on bi-directional commands (as now defined) is that DataOut buffer be ready at the time the command is issued (so the target can get/request that data at any time after receipt of the command). In other words, the content of the DataOut buffer should not be time or data-dependent on the contents of the DataIn buffer. This means that as far as SAM-2 is concerned, there is no problem with a true duplex operation. (We just don't have any examples at the moment.) I think your second proposal (using "immediate" data) is only attempting to address the specific examples (OSDs), and not the generic notion of bi-directional commands. How would that solution work if the DataOut buffer was a 1MG chunk of user data for a XDREADWRITE command? The first method (splitting, if I read it correctly) undercuts the motivation that is generating this bi-d command in the first place. The SCSI layer could do that as well, but the issue is one of atomicity of command. (Aside: one principle governing definitions of bi-directional commands in T10 is that there must always be a two (or more) uni-directional variant of the command, to allow a complex operation to be performed on a traditional uni-directional with simpler steps. I read your suggestion as having iSCSI do this for the SCSI layer -- there'd be no point. Besides, doesn't this break layering?). Finally, I would suggest a look at the FCP-2/3 proposal just submitted. I think they are proposing (for legacy reasons) a new IU for bi-directional commands. If you're concerned about cluttering your header, then this group might think about adding a new PDU for these types of commands. I don't quite understand the reluctance to add new fields in the header for this purpose. The current design is not yet cast in stone and now is the perfect opportunity to get it defined in a way that accomodates the future that T10 seems to be moving (rapidly) towards. Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 10-01-2000 12:16:06 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, Both the examples you gave have something peculiar - the operations are to executed really in sequence (we are no talking about full duplex I/O operations?). I any case, in order no to extend excessively the headers we can either introduce a "next header" field (as in IP V6) or split the operation at the iSCSI level into two phases - each carrying its parameters and each ended with a status - like: - out XOP1, XOP2, OD1, -in S1, ID, S2 In any case commands having only a parameter set (fixed or variable) - like those for the object disk should be handled with the parameters sent as immediate data (as with the three party commands) - parameter size will probably be limited and the performance requirement will be similar to those of block read/write. Do we need for those to care also about residual counts for parameters? Julo "Jim Hafner/Almaden/IBM" on 29/09/2000 00:21:32 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Bidirectional SCSI commands and iSCSI Julo, The primary motivation is for object-based storage devices (OSDs). The protocol that seems to be needed in that environment is the ability to send a CDB, plus some parameter data (DataOut) AND get back read-type data (DataIn) in the same command sequence. One (non-OSD) command has been proposed in T10 for an XOR type XDREADWRITE command. SAM-2 has now allowed for such commands in the architecture. From this perspective, it is completely symmetric. Transports that support this feature will need to have data structures in there PDUs or IUs (or whatever) to indicate things like DataOut datalength, DataIn datalength, possible virtual addresses for each buffer, etc. In other words, there will potentially be two active buffers (on each end) for use with the same command. So, e.g., iSCSI PDU may need to have both "DataIn expected data transfer length" and "DataOut expected transfer length" fields. I'm assuming that the data direction in the existing iSCSI PDU is derived from the SCSI opcode, unlike FCP where there are fields to indicate the data direction. (I am surprised not to find such things in the iSCSI Command PDU.) There are two proposals in T10 to enable this feature in SPI-4 and in FCP-x (probably FCP-3). Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-28-2000 12:10:53 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, What is the intended use? Are there already commands that use it? Is there a major/minor distinction - like output data & input results or output parameters & input data - or is it intended to be completely symmetric? Julo "Jim Hafner/Almaden/IBM" on 20/09/2000 17:56:27 Please respond to "Jim Hafner/Almaden/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Bidirectional SCSI commands and iSCSI Folks, In case you missed this, T10 just approved changes to SAM-2 (rev14) which remove the restriction that a single SCSI command can have a data flow in only one direction (DataIn xor DataOut). Transport specs (that want to allow this) will need to change to allow for two sets of Data buffer fields in the headers (length, over/underrun, etc.). Is anybody looking to modify the iSCSI spec to get these concepts supported? Jim Hafner From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03517; 29 Sep 2000 13:21 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA09073 for ips-outgoing; Fri, 29 Sep 2000 12:17:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from charon.Exabyte.COM ([12.17.207.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA09068 for ; Fri, 29 Sep 2000 12:17:22 -0400 (EDT) From: JoeBre@exabyte.com Received: from prime1.Exabyte.COM (prime1.Exabyte.COM [161.81.18.231]) by charon.Exabyte.COM (8.9.1a/8.9.1) with ESMTP id KAA07170 for ; Fri, 29 Sep 2000 10:17:21 -0600 (MDT) Received: from bouldermta.exabyte.com (bouldermta.Exabyte.COM [161.81.192.16]) by prime1.Exabyte.COM (8.6.12/8.6.10) with ESMTP id KAA03789 for ; Fri, 29 Sep 2000 10:17:20 -0600 Received: by bouldermta.Exabyte.COM with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 10:16:10 -0600 Message-ID: <118A72FD3F02D211AD1000805F9AA58B02ADC1C3@boulderp03.Exabyte.COM> To: ips@ece.cmu.edu Subject: RE: Status summary on multiple connections Date: Fri, 29 Sep 2000 10:16:08 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C02A30.939D24C0" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C02A30.939D24C0 Content-Type: text/plain; charset="iso-8859-1" Much good discussion on SCSI device level flow control snipped.... > From: Robert Snively > To: "'David Robinson'" , Robert Snively > > Subject: RE: Status summary on multiple connections > Date: Thu, 28 Sep 2000 09:36:50 -0700 ... > are again available. Note that there is a possibility that commands > that are inflight and have ordering constraints may be accepted out of > order, a question that has caused lots of agonizing, but is apparently > reasonably well managed by most file systems today by the selective > use of ordering only for blocking boundaries of a particular logical > stream of commands. I am left with the following impression as to what was indicated here: - In general, command ordering is not relevant - If the initiator filesystem detects an ordering dependency, it will wait until outstanding commands are complete before issuing the dependant command. This may be a reasonable means of operation for the disk world. It is woefully inadequate for the tape world, as follows: - In general, command ordering is crucial - out of order command processing will lead to data corruption. - This would require the initiator backup application to block on completion of every single write command of a backup operation before issuing the next command. If this blocking were performed, both the throughput and capacity of a tape device/media would be negatively impacted by an order of magnitude or more. This would occur even assuming an instantaneous transport. Joe Breher Exabyte Corp ------_=_NextPart_001_01C02A30.939D24C0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: Status summary on multiple connections

Much good discussion on SCSI device level flow = control snipped....

> From: Robert Snively = <rsnively@Brocade.COM>
> To: "'David Robinson'" = <David.Robinson@EBay.Sun.COM>, Robert Snively
> <rsnively@Brocade.COM>
> Subject: RE: Status summary on multiple = connections
> Date: Thu, 28 Sep 2000 09:36:50 -0700

...

> are again available.  Note that there is a = possibility that commands
> that are inflight and have ordering constraints = may be accepted out of
> order, a question that has caused lots of = agonizing, but is apparently
> reasonably well managed by most file systems = today by the selective
> use of ordering only for blocking boundaries of = a particular logical
> stream of commands.

I am left with the following impression as to what = was indicated here:
- In general, command ordering is not = relevant
- If the initiator filesystem detects an ordering = dependency, it will wait until outstanding commands are complete before = issuing the dependant command.

This may be a reasonable means of operation for the = disk world. It is woefully inadequate for the tape world, as = follows:

- In general, command ordering is crucial - out of = order command processing will lead to data corruption.
- This would require the initiator backup = application to block on completion of every single write command of a = backup operation before issuing the next command.

If this blocking were performed, both the throughput = and capacity of a tape device/media would be negatively impacted by an = order of magnitude or more. This would occur even assuming an = instantaneous transport.

Joe Breher
Exabyte Corp

------_=_NextPart_001_01C02A30.939D24C0-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id ag11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10969; 1 Oct 2000 5:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA00166 for ips-outgoing; Sun, 1 Oct 2000 04:30:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00140 for ; Sun, 1 Oct 2000 04:30:03 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA10752 for ; Sun, 1 Oct 2000 10:29:28 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA68334 for ; Sun, 1 Oct 2000 10:29:27 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.002EA1D8 ; Sun, 1 Oct 2000 10:29:20 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 11:03:39 +0300 Subject: Re: New List: rdma@cisco.com: to discuss RDMA Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree. Mandating such a mechanism for iSCSI would be counterproductive now. Julo Daniel Smith on 27/09/2000 23:31:46 Please respond to Daniel Smith To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: New List: rdma@cisco.com: to discuss RDMA csapuntz@cisco.com wrote: > > Does anybody on the list object to specifying an RDMA mechanism for > use with iSCSI? Does anybody on the list object to mandating an RDMA > mechanism? Please include your reasons. Mandate: strongly against. Recommend: neutral, slightly favorable. Unless... By `mandate' you mean supporting a TEXT flag com.cisco.RDMAEnableRFCxxxx: goforit to which the target can respond com.cisco.RDMAEnableRFCxxxx: nowaybuddy because the target does not have enough system privilage to alter any TCP options. Daniel Smith. -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15950; 2 Oct 2000 10:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA29563 for ips-outgoing; Mon, 2 Oct 2000 09:20:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA29553 for ; Mon, 2 Oct 2000 09:20:36 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id IAA28741; Mon, 2 Oct 2000 08:20:02 -0500 Message-ID: <39D88B82.3420E6D@stewart.chicago.il.us> Date: Mon, 02 Oct 2000 08:20:02 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: "Mark A. Carlson" CC: David Robinson , ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: <200009291825.LAA13471@ha10nwk.EBay.Sun.COM> <39D87E35.51050250@stewart.chicago.il.us> <39D882F1.C46F5243@sun.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "Mark A. Carlson" wrote: > > "Randall R. Stewart" wrote: > > > > I really do NOT see how one can allow a TCP to deliver out of order > > though, oh > > I know technically how to do it, but it is a fundamental violation > > of what TCP is supposed to be delivering.... i.e. the bytes being > > transfered from 0 to N... > > The scenario which everyone seems to be worried about is that > packets are delivered in order, but the ACK for one of them is > lost going back. Everything gets delivered to iSCSI, then the > non-ACKed packet is delivered again. > > Does this really happen in today's stacks? > Well, a transport stack must ALWAYS be prepared for duplicates... However.. has you state below.. if the above happens TCP will recognize it IS a duplicate and drop the duplicate.. The upper layer (iSCSI in this case) will NEVER see the duplicate data... If it DOES see duplicate data then you have a BROKEN TCP stack... R > It seems like the TCP stack would see it as a duplicate that > has already been delivered to the upper layer and drop it. > TCP does have sequence numbers and knows this. The only way > I could imagine this happening is if there was sufficient > data to wrap the TCP sequence space. > > -- mark -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14913; 2 Oct 2000 4:28 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA23607 for ips-outgoing; Mon, 2 Oct 2000 02:57:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA23603 for ; Mon, 2 Oct 2000 02:57:28 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id IAA81802 for ; Mon, 2 Oct 2000 08:56:52 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA38266 for ; Mon, 2 Oct 2000 08:56:52 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.0026287B ; Mon, 2 Oct 2000 08:56:47 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 08:38:41 +0300 Subject: RE: Status summary on multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, Sorry for the cryptic statement - I've implied it under "more complex setup". And only to make it clear - I see it as THE MAJOR DRAWBACK of PA. Julo "GUPTA,SOMESH (HP-Cupertino,ex1)" on 29/09/2000 01:17:17 Please respond to "GUPTA,SOMESH (HP-Cupertino,ex1)" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Status summary on multiple connections You have not included the performance issues that were brought up for the PA model. Somesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Tuesday, September 26, 2000 9:48 AM To: ips@ece.cmu.edu Subject: Status summary on multiple connections Dear colleagues, I am attempting to summarize where we stand with regard to the multiple connection issue and to the two possible models - Symmetric (S) and Asymmetric (A). Many of us feel strongly that the multiple connection issue is central to the whole design and cannot be added as an afterthought. Moreover designing the hooks to allow it later will certainly already force us to make a decision. And both the hardware and the software designers will be ill-served if we hand them a half-backed solution. However this is not an invitation to reiterate positions that where already stated. If you feel that I have grossly misstated anything in this note please write me and use the mailing list only if my answer is not satisfying. And yes - like the chairman - I would like to make progress but I don't see any way to do it without satisfactorily closing this issue. The reasons for multiple connections where discussed at some length and where very nicely summarized in a series of notes by Michael Krause (beginning of August). The core reasons for having multiple connections where the need for more bandwidth and availability than a single link can supply with a level of complexity affordable for simple installations and with a traffic engineering and management clearly separated from the transport users (SCSI). The session is embodying this requirement. The only major objections I have heard against this where those requiring to go all the way in having one TCP connection/LU - and after a short debate this objection was practically removed. The other objections we heard where that this is basically a transport issue and it should be solved at transport level. That might be true - but since many, if not most, of TCP applications do not have this requirement it is highly unlikely that TCP is going to do connection trunking in the foreseeable future. iSCSI can be designed to use multiple TCP connections in one of two ways: -Asymmetric - one TCP flow only carries commands the others carry only data -Symmetric - every flow carries both commands and their associated data The S version is designed in the I-D version 01 The S version requires an command ordering scheme and that is provided by a command counter and a sliding window scheme. It was argued that ordering needs might be more prevalent than usually thought and a good conservative design should preserve ordering. Ordering-per-LU (as it is designed in FCP-2 version 4) was considered impractical as it required initiators to maintain state for each LU - while the rest of the design required initiators only to maintain state for outstanding commands. Several comments on this list suggested that this windowing mechanism could also be used as a command-flow-control mechanism and that is a "bonus" of the scheme. The A version - comes in two flawors: - pure A (PA) in which ONLY commands flow on one TCP connection while data flow on DIFFERENT connections (with only one data connection being selected for a command); this scheme requires a minimum of 2 TCP connections although not necessarily on different physical links. - collapsed A (CA) in which commands flow one a single TCP connection while data flow on ANY connection; this scheme requires as a minimum 1 TCP connection. Here is first attempt to list the benefits and drawbacks of all of them: - S - benefits - well understood - simple hardware setup and/or TCP API activation - window mechanism can be also used for flow control - the minimum required is a single TCP connection - drawbacks - need to maintain a window mechanism - a multiplexing mechanism has to be carefully crafted to avoid closing TCP windows to severely affect command flow and performance - PA - benefits - TCP will both order the commands and provide for flow control through the TCP window mechanism - the multiplexing mechanism is simpler as closing TCP windows will never affect command flow - data flow can use a streamlined header (and processing) - drawbacks - more complex hardware and/or software API activation - need for a minimum of two TCP connections (not links) - CA - benefits - TCP will order the commands - needs a single TCP connection as a minimum - drawbacks - more complex hardware and/or software API activation - need for a minimum of two TCP connections (not links) - if command flow control is required counters and the sliding window mechanism are required - a multiplexing mechanism has to be carefully crafted to avoid closing TCP windows to severely affect command flow and performance >From the above it should be apparent - as was already pointed out by Matt Wakeley - that the CA inherits the drawbacks of S and PA and as such the choice is really between S and PA. Julo From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32683; 28 Sep 2000 17:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA09390 for ips-outgoing; Thu, 28 Sep 2000 16:47:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA09370 for ; Thu, 28 Sep 2000 16:47:50 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA14711; Thu, 28 Sep 2000 13:34:40 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id NAA04800; Thu, 28 Sep 2000 13:32:26 -0700 From: "Douglas Otis" To: "David Robinson" , Subject: RE: Why FCP doesn't need RDMA? It has a better way. Date: Thu, 28 Sep 2000 13:50:16 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <200009281835.LAA07686@ha10nwk.EBay.Sun.COM> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, This example is of .5MB transfers using jumbo frames. It would be difficult to extrapolate performance as related to normal SCSI use. It does suggest that such hardware able to implement scatter/gather is useful. It would be equally useful for any protocol. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > David Robinson > Sent: Thursday, September 28, 2000 11:36 AM > To: ips@ece.cmu.edu > Subject: RE: Why FCP doesn't need RDMA? It has a better way. > > > > From an iSCSI viewpoint, I don't see iSCSI by itself as being sufficient > > to motivate a protocol-independent RDMA - an iSCSI HBA could understand > > the iSCSI headers and interact with DMA in the same fashion as existing > > HBAs. The task before those interested in RDMA is to identify a set > > of protocols for which a common RDMA mechanism makes sense from > > an engineering standpoint. I tend to agree with the previous emails > > that iSCSI could make optional use of a common RDMA mechanism > > if available, but must not REQUIRE its use. > > As another data point to show that RDMA is not required, a recently > reported group acheiving 980+Mbps on a gigabit ethernet using > NFS. NFS is not nearly as simple to parse headers as FCP or > the current iSCSI draft, and the authors were able to reprogram > an Altheon card's firmware to acheive it. I suspect a good design > engineer could do the same for iSCSI without RDMA. > > http://people.freebsd.org/~ken/zero_copy/ > > So while RDMA is interesting, I see no reason to REQUIRE it for iSCSI. > > -David > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16531; 2 Oct 2000 13:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA08045 for ips-outgoing; Mon, 2 Oct 2000 12:47:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ertpg14e1.nortelnetworks.com (ertpg14e1.nortelnetworks.com [47.234.0.35]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA08038 for ; Mon, 2 Oct 2000 12:47:32 -0400 (EDT) Received: from zsc4c002.corpwest.baynetworks.com by ertpg14e1.nortelnetworks.com; Mon, 2 Oct 2000 12:46:21 -0400 Received: from zsc4c006.corpwest.baynetworks.com ([134.177.2.153]) by zsc4c002.corpwest.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id T3GD42Q9; Mon, 2 Oct 2000 09:46:17 -0700 Received: from long-pc.us.nortel.com (LONG-PC [134.177.44.60]) by zsc4c006.corpwest.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id T7ZMVCPF; Mon, 2 Oct 2000 09:46:17 -0700 Message-Id: <4.2.2.20001002092921.00b7b4b0@zsc4c006.corpwest.baynetworks.com> X-Sender: long@zsc4c006.corpwest.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Mon, 02 Oct 2000 09:47:38 -0700 To: Y P Cheng , "Mark A. Carlson" , "Randall R. Stewart" From: "Lyndon Ong" Subject: RE: Status summary on multiple connections Cc: David Robinson , ips In-Reply-To: <001201c02c82$24399ce0$65c809c0@yp_portable.advansys.com> References: <39D882F1.C46F5243@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:05 AM 10/2/2000 -0700, Y P Cheng wrote: >Randall and Mark: > >Both of you forget about the case when multiple PDUs are inflight, say N, >N+1, and N+2, and one of them has CRC error, say N+1. The receiver throws >away N+1 because the bad CRC. N+2 is received long before N+1 is >retransmitted after a timeout by the sender. Of course, a sequence number >inside the PDU will ensure sequentially. However, as I stated in another >posting, all software realize such problem and will not count on sequential >delivery by a transport. > >Y.P. This confuses me. Why would the application ever see the N+2 PDU, wouldn't it be queued by TCP until N+1 is delivered correctly? It shouldn't matter if N+2 was delivered first or if it contains a sequence number at the higher layer, the higher layer would not get N+2 before N+1. TCP would presumably not be aware of the boundaries between messages at the higher layer, but deliver what it sees as a stream of bits. Cheers, L. Ong From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id ad11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10955; 1 Oct 2000 5:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA00160 for ips-outgoing; Sun, 1 Oct 2000 04:30:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00141 for ; Sun, 1 Oct 2000 04:30:03 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA81514 for ; Sun, 1 Oct 2000 10:29:28 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA68336 for ; Sun, 1 Oct 2000 10:29:28 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.002EA18E ; Sun, 1 Oct 2000 10:29:20 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 10:19:38 +0300 Subject: The third alternative Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Bob, This thread went on for a long time and the arguments against leaving it all too the wedge drivers where: - that it makes the same feature available to all environments (even those not using wedge drivers today) - that it is more in line with traffic engineering in networks in that you don't have to work on the application to change the underlying network traffic characteristics and this can be even done dynamically with existing networking tools - that it better reflects layering - that it will simplify all applications and mainly exception handling - that it will be an interoperable solution (are wedge drivers interoperable?) The only argument for SCSI wedge drivers where that they EXIST ALREADY. Pretty weak argument for those building new equipment and for interoperability. I think that if we keep ourselves honest we have to either: - provide for multiple connections at the iSCSI level as it is transport problem that other TCP applications are not compelled to handle (I hear already BUT SCTP handles it!) and hope that one day the session concept will drift into pure transport - go to T10 and ask the to standardize wedge drivers! What would you consider as a "better alternative"? Julo Robert Snively on 27/09/2000 18:46:26 Please respond to Robert Snively To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: Status summary on multiple connections Julo, A third alternative does exist and may be preferable. That is to reflect the present broad usage of SCSI that uses wedge drivers to achieve parallelization, bandwidth aggregation, and high availability. By that definition, the proper number of connections per I_T nexus is one and the proper number of connections per session is also one. This recognizes the reality that there is really little to gain in virtualizing multiple connections into a single session image when you can perform even more flexible virtualization through a wedge driver at a higher level. The single connection alternative allows a simplistic ordering structure, a simple recovery mechanism, and does not require state sharing among multiple NICs. It allows bandwidth aggregation across any set of boundaries that is required. Because command queuing is the rule among high performance SCSI environments, latency appears only as an increment in host buffer requirements except during writes that perform a commit function. Those traditionally have been taken out of the performance path by using local non-volatile RAM to perform the commit functions, using slower high latency writes with less strict ordering requirements relative to reads to actually perform the write to media. The overheads associated with handling a multiple connection session of any type are basically indistinguishable from the overheads associated with device virtualization through wedge drivers. If you consider software instruction path lengths for the total functionality, you can conceive of the multiple connection session as simply a TCP/IP wedge driver. Bob > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Tuesday, September 26, 2000 9:48 AM > To: ips@ece.cmu.edu > Subject: Status summary on multiple connections > > > > > Dear colleagues, > > I am attempting to summarize where we stand with regard to > the multiple > connection issue > and to the two possible models - Symmetric (S) and Asymmetric (A). > > Many of us feel strongly that the multiple connection issue > is central to > the whole design > and cannot be added as an afterthought. Moreover designing > the hooks to > allow it later > will certainly already force us to make a decision. And both > the hardware > and the software designers will be ill-served if we hand > them a half-backed > solution. > > However this is not an invitation to reiterate positions > that where already > stated. > If you feel that I have grossly misstated anything in this > note please > write me and use > the mailing list only if my answer is not satisfying. > > And yes - like the chairman - I would like to make progress > but I don't see > any way to > do it without satisfactorily closing this issue. > > The reasons for multiple connections where discussed at some > length and > where very nicely summarized in a series of notes by Michael Krause > (beginning of August). > > The core reasons for having multiple connections where the > need for more > bandwidth and availability than a single link can supply > with a level of > complexity affordable for simple > installations and with a traffic engineering and management clearly > separated from the transport users (SCSI). > > The session is embodying this requirement. > > The only major objections I have heard against this where > those requiring > to go all > the way in having one TCP connection/LU - and after a short > debate this > objection > was practically removed. > > The other objections we heard where that this is basically a > transport > issue and it should be > solved at transport level. > That might be true - but since many, if not most, of TCP > applications do > not have this > requirement it is highly unlikely that TCP is going to do connection > trunking in the foreseeable > future. > > iSCSI can be designed to use multiple TCP connections in one > of two ways: > > -Asymmetric - one TCP flow only carries commands the others > carry only data > > -Symmetric - every flow carries both commands and their > associated data > > The S version is designed in the I-D version 01 > The S version requires an command ordering scheme and that > is provided by a > command counter and a sliding window scheme. It was argued > that ordering > needs might be more prevalent than usually thought and a > good conservative > design should preserve ordering. > Ordering-per-LU (as it is designed in FCP-2 version 4) was > considered > impractical as it > required initiators to maintain state for each LU - while > the rest of the > design required initiators only to maintain state for > outstanding commands. > Several comments on this list suggested that this windowing > mechanism could > also be used as a command-flow-control mechanism and that is > a "bonus" of > the scheme. > > The A version - comes in two flawors: > > - pure A (PA) in which ONLY commands flow on one TCP > connection while data > flow > on DIFFERENT connections (with only one data connection > being selected > for a > command); this scheme requires a minimum of 2 TCP > connections although > not > necessarily on different physical links. > > - collapsed A (CA) in which commands flow one a single TCP > connection while > data > flow on ANY connection; this scheme requires as a minimum 1 TCP > connection. > > Here is first attempt to list the benefits and drawbacks of > all of them: > > > - S - benefits > - well understood > - simple hardware setup and/or TCP API activation > - window mechanism can be also used for flow control > - the minimum required is a single TCP connection > > - drawbacks > - need to maintain a window mechanism > - a multiplexing mechanism has to be carefully > crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > - PA - benefits > - TCP will both order the commands and provide for > flow control > through the TCP window mechanism > - the multiplexing mechanism is simpler as closing > TCP windows > will > never affect command flow > - data flow can use a streamlined header (and processing) > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > > - CA - benefits > - TCP will order the commands > - needs a single TCP connection as a minimum > > - drawbacks > - more complex hardware and/or software API activation > - need for a minimum of two TCP connections (not links) > - if command flow control is required counters > and the sliding > window mechanism > are required > - a multiplexing mechanism has to be carefully > crafted to avoid > closing TCP windows to severely affect command flow and > performance > > > From the above it should be apparent - as was already > pointed out by Matt > Wakeley - > that the CA inherits the drawbacks of S and PA and as such > the choice is > really > between S and PA. > > Julo > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16103; 2 Oct 2000 11:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA02214 for ips-outgoing; Mon, 2 Oct 2000 10:27:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA27159 for ; Mon, 2 Oct 2000 07:27:34 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id ; Mon, 2 Oct 2000 04:29:08 -0700 Message-ID: From: Robert Snively To: "'Daniel Smith'" , Robert Snively Cc: ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 2 Oct 2000 04:26:58 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry Dan, I am not sure the analogy is valid: > > 2) The URL is really an arbitrary length base-40ish numeral. The > infrastructure is already in place and well tested with HTTP URLs. The referencing is reversed. In the case of the HTTP, the registered value is the domain and the naming below the domain is at the pleasure of the domain owner. There is a 3 to 4 order of magnitude difference between the number of logical units being considered and the number of domains referenced in HTTP URLs. > > 3) Once the URL is decoded ("Robert Smith, in Seattle, on > Broad St, next to > the donut store") it becomes a numeric address---probably a > 32-bit integer > on most systems. Then you can address the TCP reference > number plus the > LUN to access your device. > HTTP is really: Seattle: as the registered domain, and Broad Street/donut store/Robert Smith as a device specific location. What SCSI wants is: Robert Smith: as the registered entity and Broad Street/donut store/Seattle as one way to get to him while 2nd street/around the corner from the restaurant/Seattle as a second way to get to him. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03092; 29 Sep 2000 10:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA03920 for ips-outgoing; Fri, 29 Sep 2000 09:52:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA03915 for ; Fri, 29 Sep 2000 09:52:47 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id PAA189162 for ; Fri, 29 Sep 2000 15:52:11 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id PAA143558 for ; Fri, 29 Sep 2000 15:52:12 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256969.004C2F69 ; Fri, 29 Sep 2000 15:52:08 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 29 Sep 2000 16:49:12 +0300 Subject: RE: A Transport Protocol Without ACK (resend) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I don't doubt that such a mechanism could work and be somewhat useful. Ordering between different XON/XOFFs is a bit more difficult (the function can be easily piggybacked on R2T but R2Ts have no relative ordering or if we choose to do XON/XOFF at command boundary the command response could carry this information and responses are ordered). However I am under the impression that the TCP window (and in case of the symmetrical scheme or command sequence window) will perform the same function whenever unsolicited data are enabled. Am I missing something? Julo Michael Krause on 23/09/2000 16:43:09 Please respond to Michael Krause To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: A Transport Protocol Without ACK (resend) At 12:04 PM 9/22/00 -0600, HAAGENS,RANDY (HP-Roseville,ex1) wrote: >Robert, > >The reason we proposed immediate write data for iSCSI was as an optimization >for short writes over "long" networks. As network diameter increases, >minimizing round-trip signalling delays becomes important. Sending write >data immediately following the command eliminates the need for an RTT from >the target, and thus eliminates a round-trip delay. > >If immediate data threatens to overflow the target's cache, the target may, >on an emergency basis, discard the data; and, when ready, request data again >using the RTT protocol. This behavior should be minimized, however, as it >will tend to congest the network. > >Note the use of SCSI-layer flow control (RTT mechanism). I agree with those >that say a controller must manage its cache, regardless of the size of the >cache. This is most apparent when one considers multiple hosts sharing a >storage controller. In order to allocate performance between the hosts, the >controller must ensure that a write burst from a single host cannot fill its >cache. RTT provides a means for doing this that still allows other commands >(read commands, for example) from the same host to proceed. > >However, reserving a few MB of cache for each host, to accommodate immediate >writes, seems to me reasonable. The command windowing mechanism (another >flow control mechanism) will prevent the overrunning of these immediate >write buffers, assuming that the controller has reserved space in proportion >to the largest immediate buffer allowed (which is negotiated) x the maximum >number of commands. > >Upon writing this, it occurs to me that it may be useful to add an >XOF/XON-like mechanism for temporarily disallowing immediate write data. >This would give the controller another tool for cache management. When its >cache became congested, the controller could signal one or more of the >requesting hosts to stop sending immediate write data with commands; later, >another message could allow immediate writes to resume. These messages >would need to make it to the iSCSI layer in the hosts, as they will affect >how the host does DMA chain construction. Given the distances involved and the subsequent latency, I'd suggested lifting the technique of issuing a RNR-NAK (receiver not ready) with a time period the sender should wait upon receiving the NAK before it retries the operation (0 indicates immediate which may be acceptable if one knows the sender is far away). Allows the receiver to allocate the requested buffer space within what it believes to be a reasonable amount of time, eliminates the need for the receiver to track the sender's state for this problem, and eliminates the additional message associated with the XOFF/XON mechanism. In general, we should strive wherever possible to architect such that only one side needs to track the state of a given attribute - this leads to simplicity in implementation and future development of the architecture. Mike From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16300; 2 Oct 2000 12:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA04932 for ips-outgoing; Mon, 2 Oct 2000 11:33:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA04926 for ; Mon, 2 Oct 2000 11:33:53 -0400 (EDT) Received: from centralmail2.Central.Sun.COM ([129.147.62.11]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id JAA04937; Mon, 2 Oct 2000 09:33:50 -0600 (MDT) Received: from matador.Central.Sun.COM (matador.Central.Sun.COM [129.147.248.1]) by centralmail2.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id JAA06298; Mon, 2 Oct 2000 09:33:48 -0600 (MDT) Received: from sun.com (d-nwk04-151-185.EBay.Sun.COM [129.150.151.185]) by matador.Central.Sun.COM (8.8.8+Sun/8.8.8) with ESMTP id JAA00852; Mon, 2 Oct 2000 09:33:13 -0600 (MDT) Message-ID: <39D8ABCE.765651A1@sun.com> Date: Mon, 02 Oct 2000 09:37:50 -0600 From: "Mark A. Carlson" Reply-To: mark.carlson@sun.com Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.75 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Y P Cheng CC: "Randall R. Stewart" , David Robinson , ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: <001201c02c82$24399ce0$65c809c0@yp_portable.advansys.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Y P Cheng wrote: > > > Randall and Mark: > > Both of you forget about the case when multiple PDUs are inflight, say N, > N+1, and N+2, and one of them has CRC error, say N+1. The receiver throws > away N+1 because the bad CRC. N+2 is received long before N+1 is > retransmitted after a timeout by the sender. Of course, a sequence number > inside the PDU will ensure sequentially. However, as I stated in another > posting, all software realize such problem and will not count on sequential > delivery by a transport. N+2 is received by the TCP stack, but not delivered to iSCSI until N+1 arrives (from the retransmit) and is delivered in sequence. This is how TCP works according to my understanding. -- mark From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03887; 29 Sep 2000 15:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15935 for ips-outgoing; Fri, 29 Sep 2000 15:19:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15925 for ; Fri, 29 Sep 2000 15:19:04 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA15098; Fri, 29 Sep 2000 12:05:44 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA08282; Fri, 29 Sep 2000 12:03:31 -0700 From: "Douglas Otis" To: , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Fri, 29 Sep 2000 12:21:32 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, You would not authenticate exclusively from a single token out of the blue. There must be an additional layer of authentication that indicates IP:Port[] x is customer 'y' and today they wish to connect to target z[]. Only in the most simplest configurations would a flat file provide this information. You could easily have two servers that manage the customer and target databases separately and join them using a 'Key.' This join would be a database construct and not a SCSI. This additional management layer will share with both ends and not pass over SCSI transport. With this management layer, binary representations are adaquate and provide far less information to those wishing to crack data. DNS, if used, would be run by a management layer and not a transport layer. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > julian_satran@il.ibm.com > Sent: Friday, September 29, 2000 6:15 AM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > I would add that we wanted the path to be an additional > identifier that the > target could use > to determine what collection of LUs to present to the initiator. > > Julo > > csapuntz@csapuntz-u1.cisco.com on 23/09/2000 05:00:33 > > Please respond to csapuntz@csapuntz-u1.cisco.com > > To: "IP Storage" > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > > Just for clarification... I was not proposing to add the extended > URL scheme for the transport spec. It isn't necessary. > > In this thread, Doug makes an excellent point about LDAP being a > superior mechanism for describing how to connect to the > storage. Directory services such as LDAP, as Doug has pointed out, > will be critical to managing large quantities of storage. A host can > ask such a directory service for a list of storage devices it should > mount and how to connect to those storage devices. The query against > the directory server that returns this information can be based on > machine ID, user ID, operating system ID, or even the owner's > birthday. One of the things discovery will end up doing, no doubt, is > defining LDAP schemas that describe how to connect to storage > (i.e. use SCTP or TCP, what port, what target name, what LUN, what > WWN, how to authenticate, etc.). > > However, there is one place where the transport protocol has to define > a name: third party commands. There needs to be some kind of global > name which the initiator can pass to the target. The name must be > distillable into a string. The target must understand the name and > be able to use the information to establish a connection to > another target. > > One could say that the string that is passed is not specified by the > standard but instead specified by some management software. I think > this will lead to poor interoperability. > > The SCSI URL-type name is the current proposal for target name. > > Why is SCSI target name made up of a hostname + a path? Why is the > hostname + path passed on connection setup? There are two reasons. > I think NAT and IPv6 makes passing hostnames rather than addresses in > protocols more desirable. Hostnames can be re-resolved as you cross > addressing boundaries. The path is there so that the name can > support multiple targets behind a single IP address without having > to add entries to the DNS server. At Cisco, for example, > I have no control over the local DNS servers and cannot > add DNS entries for the ATAPI DVD and floppy in my computer. > > _Costa > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16210; 2 Oct 2000 12:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03907 for ips-outgoing; Mon, 2 Oct 2000 11:07:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03899 for ; Mon, 2 Oct 2000 11:07:42 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id T9NCD78B; Mon, 2 Oct 2000 08:05:05 -0700 From: "Y P Cheng" To: "Mark A. Carlson" , "Randall R. Stewart" Cc: "David Robinson" , Subject: RE: Status summary on multiple connections Date: Mon, 2 Oct 2000 08:05:03 -0700 Message-ID: <001201c02c82$24399ce0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: <39D882F1.C46F5243@sun.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > > I really do NOT see how one can allow a TCP to deliver out of order > > though, oh > > I know technically how to do it, but it is a fundamental violation > > of what TCP is supposed to be delivering.... i.e. the bytes being > > transfered from 0 to N... > > The scenario which everyone seems to be worried about is that > packets are delivered in order, but the ACK for one of them is > lost going back. Everything gets delivered to iSCSI, then the > non-ACKed packet is delivered again. > > Does this really happen in today's stacks? > > It seems like the TCP stack would see it as a duplicate that > has already been delivered to the upper layer and drop it. > TCP does have sequence numbers and knows this. The only way > I could imagine this happening is if there was sufficient > data to wrap the TCP sequence space. Randall and Mark: Both of you forget about the case when multiple PDUs are inflight, say N, N+1, and N+2, and one of them has CRC error, say N+1. The receiver throws away N+1 because the bad CRC. N+2 is received long before N+1 is retransmitted after a timeout by the sender. Of course, a sequence number inside the PDU will ensure sequentially. However, as I stated in another posting, all software realize such problem and will not count on sequential delivery by a transport. Y.P. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00289; 28 Sep 2000 19:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA14494 for ips-outgoing; Thu, 28 Sep 2000 19:10:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from baucis.sc.intel.com (baucis.sc.intel.com [143.183.152.22]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA14485 for ; Thu, 28 Sep 2000 19:10:35 -0400 (EDT) Received: from SMTP (fmsmsxvs03-1.fm.intel.com [132.233.42.203]) by baucis.sc.intel.com (8.9.1a+p1/8.9.1/d: relay.m4,v 1.31 2000/08/22 00:15:13 dmccart Exp $) with SMTP id QAA24507; Thu, 28 Sep 2000 16:10:31 -0700 (PDT) Received: from fmsmsx19.fm.intel.com ([132.233.48.19]) by 132.233.48.203 (Norton AntiVirus for Internet Email Gateways 1.0) ; Thu, 28 Sep 2000 23:10:22 0000 (GMT) Received: by fmsmsx19.fm.intel.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 16:10:18 -0700 Message-ID: <31E3E2A9D8DDD311AC43009027C68052C7FF8B@orsmsx59.jf.intel.com> From: "Grun, Paul" To: "'Stephen Bailey'" , ips@ece.cmu.edu Subject: RE: New List: rdma@cisco.com: to discuss RDMA Date: Thu, 28 Sep 2000 16:10:07 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" X-MIME-Autoconverted: from quoted-printable to 8bit by ece.cmu.edu id TAA14486 Sender: owner-ips@ece.cmu.edu Precedence: bulk Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ece.cmu.edu id TAB14494 Interesting point of view...if the target end cannot count on the service being available at the initiator end, it does little good. InfiniBand, which has an RDMA mechanism, ran into a similar problem. The solution was asymmetric requirements; requirements placed on the host end permitted less latitude than those placed on the target end. The result = is a set of services upon which the target end knows it can rely. Paul Grun=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=20 Intel Corporation - Enterprise Platform Group (503) 677-6768 - paul.grun@intel.com =20 > -----Original Message----- > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > Sent: Thursday, September 28, 2000 2:23 PM > To: ips@ece.cmu.edu > Subject: Re: New List: rdma@cisco.com: to discuss RDMA=20 >=20 >=20 > > However, I would support structuring the spec so that an RDMA > > transport mechanism could be used underneath (I guess that's > > motherhood). >=20 > Not necessarily. >=20 > You have to ask the implementors (particularly the hardware > implementors), what sort of optional RDMA proposal they'd be happy > with. My answer is none. It's mandatory or not at all. >=20 > The reason for using RDMA is to make the implementation of iSCSI > easier in hardware. If there are implementations which do not support > the RDMA protocol, then your hardware implementation will have to > support both the `easy path' (using RDMA) and the `hard path' (no > RDMA). If you have to implement the hard path anyway, there's no > point in implementing the easy path. >=20 > The argument that you could make the hard path infrequent and > implement it in software doesn't wash in this case. It just takes one > implementation that doesn't do RDMA to slow your system to a crawl, > and the competitor who only implemented the non-RDMA path makes you > look like a fool. >=20 > Fundamentally, RDMA has to be either adopted or punted. Of course, > I'm happy to have somebody prove this statement wrong. >=20 > Steph >=20 From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04479; 29 Sep 2000 19:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23663 for ips-outgoing; Fri, 29 Sep 2000 18:35:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23655 for ; Fri, 29 Sep 2000 18:35:11 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id PAA18764 for ; Fri, 29 Sep 2000 15:35:11 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.142.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id PAA11672 for ; Fri, 29 Sep 2000 15:35:10 -0700 (PDT) Received: from ebay.sun.com by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id PAA09146; Fri, 29 Sep 2000 15:35:09 -0700 (PDT) Message-ID: <39D5191F.52DD9F5@ebay.sun.com> Date: Fri, 29 Sep 2000 15:35:11 -0700 From: David Robinson X-Mailer: Mozilla 4.75 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk There are two types of out of order processing we need to consider: 1) Commands that have been received and ACK'd by the transport 2) Commands that have been received and not ACK'd In the first case this is purely a SCSI issue on if the commands are ordered and how the target processes them. As far as the transport is concerned the packets are "delivered". In the second case the transport is still required to track the possible retransmissions and process the transport ACK for all of the data when the missing segment arrives. It will overly complicate the iSCSI layer to successfully track which parts of the sequence space it has processed, defend against retransmissions, and handle packets that have more than one command, and retransmissons that may be only part of a command. This is all doable in an implementation (a bit messy at times) but to support this as an explicit feature we need to start adding in an iSCSI layer sequence space. I propose that we leave the processing of transport level un-ACK'd commands and data as an implementation detail and not make it an explicit iSCSI feature. Dropped or reordered TCP segments are rare enough in high performance environments that this is really a non-issue. -David julian_satran@il.ibm.com wrote: > > David, > > I think that RDMA out-of-order processing is important only in order not to > have data > piling up in adapters. It does not mean that data gets really "committed". > The commands can be executed in or out-of-order - this is a pure SCSI > story. > But if they have to be executed in order the ordering provided by SCSI > will enable it. > And as you and others have pointed out the same mechanism is good for both > ordering and flow control. > As far as I understand it windowing - although more expensive - is a better > technique over a wide variety of latencies than credits. > > And BTW we even considered credits for a "prefetching mechanism" for > chained > commands but where told that those are very much "out of fashion" (i.e. not > worth speeding up on Elefants). > > Julo > > David Robinson on 29/09/2000 21:25:52 > > Please respond to David Robinson > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: RE: Status summary on multiple connections > > > I am left with the following impression as to what was indicated here: > > - In general, command ordering is not relevant > > - If the initiator filesystem detects an ordering dependency, it will > wait > > until outstanding commands are complete before issuing the dependant > > command. > > > > This may be a reasonable means of operation for the disk world. It is > > woefully inadequate for the tape world, as follows: > > - In general, command ordering is crucial - out of order command > processing > > will lead to data corruption. > > - This would require the initiator backup application to block on > completion > > of every single write command of a backup operation before issuing the > next > > command. > > > > If this blocking were performed, both the throughput and capacity of a > tape > > device/media would be negatively impacted by an order of magnitude or > more. > > This would occur even assuming an instantaneous transport. > > I am hearing different stories on the issue of ordering. One side > is pushing hard for techniques that will allow out of order > execution using various RDMA techniques. This clearly states for > a certain class of devices (e.g. tapes) ordering is crucial. > I thought this problem was already solved at the SCSI layer > through the use of ordered commands which in general are not used > for disks but always used for tapes? Since FC will reorder this > has to be a solved problem. Would not an initiator talking to > a tape target simply set the ordering flag? > > Lastly, for a TCP based connection ordering can easily be made a > non-issue, simply don't try to process segments out of order. I > will defer to a transport expert, but I believe processing > TCP segments by an application out of order might cause problems. > In particular since the out of order segment is not ACKed until > after the missing segments arrive, they can be retransmitted > multiple times. SACK helps this but does not guarentee that > segments will not be retransmitted. So to process out of order > segments the application must maintain a list of which segments > have been processed as well, yuck! > > -David From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id ac11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10953; 1 Oct 2000 5:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA00172 for ips-outgoing; Sun, 1 Oct 2000 04:30:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00156 for ; Sun, 1 Oct 2000 04:30:12 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id KAA313390 for ; Sun, 1 Oct 2000 10:29:36 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA105466 for ; Sun, 1 Oct 2000 10:29:37 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.002EA482 ; Sun, 1 Oct 2000 10:29:27 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 11:14:05 +0300 Subject: RE: New List: rdma@cisco.com: to discuss RDMA Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Just out of curiosity - what is the difference between an RDMA read-request and the R2T you are trying to avoid? Julo "TALBOTT,GLENN (HP-Roseville,ex1)" on 28/09/2000 00:31:35 Please respond to "TALBOTT,GLENN (HP-Roseville,ex1)" To: "'csapuntz@cisco.com'" cc: ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: New List: rdma@cisco.com: to discuss RDMA Costa, et al., I believe that specifying an RDMA mechanism for iSCSI is the right thing to do. RDMA (especially when combined with a frame recovery mechanism such as the one described by Jim Williams) will enable iSCSI NIC implementers to eliminate the massive on-card buffering and double processing required to implement SACK in the face of long, fast, fat pipes. The NIC is able to push each inbound RDMA frame off into it's proper place in host memory while waiting for the missing packet instead of buffering the receive stream until the missing packet is received to recover synchronization. A general purpose RDMA and framing solution will benefit both iSCSI and VI/TCP. I do not believe that an RDMA mechanism should be mandatory for iSCSI. Manufacturers of lower cost thin storage devices may opt to limit applications to inside the data center, throw large amounts of off chip RAM at the problem (possibly not constrained to PCI NIC form factors), or accept lower performance by not implementing SACK, rather than go to the expense of the added complexity that RDMA may add to an implementation. I can see a future where iSCSI server initiators typically implement RDMA and targets typically do not. Then SCSI reads are satisfied by the target sending a sequence of RDMA writes to the initiator, and SCSI writes are satisfied by the target sending a sequence of RDMA reads to the initiator. This also eliminates the need for the target to send R2T. Each RDMA read is an implicit R2T. Glenn Talbott > -----Original Message----- > From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] > Sent: Wednesday, September 27, 2000 10:16 AM > To: Jim Williams > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > Subject: Re: New List: rdma@cisco.com: to discuss RDMA > > > > Does anybody on the list object to specifying an RDMA mechanism for > use with iSCSI? Does anybody on the list object to mandating an RDMA > mechanism? Please include your reasons. > > Current RDMA proposals: > draft-csapuntz-tcprdma-00.txt > draft-dicecco-vitcp-00.txt > > -Costa > > > Before going too far down this road, it is important to > > understand if there is support for using an RDMA mechanism > > as a basis for iSCSI. Will the next draft of the iSCSI > > protocol actually be based on an RDMA mechanism that > > would be defined? The answers to these questions and > > the specific pros and cons need to be the driving > > force behind the RDMA discussion. > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01301; 29 Sep 2000 1:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA22351 for ips-outgoing; Fri, 29 Sep 2000 00:32:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA22345 for ; Fri, 29 Sep 2000 00:32:15 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id 964CB15D for ; Thu, 28 Sep 2000 21:32:14 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id VAA05754 for ; Thu, 28 Sep 2000 21:32:50 -0700 (PDT) Message-ID: <39D41B43.B23804A2@agilent.com> Date: Thu, 28 Sep 2000 21:32:03 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) References: <10009282136.AA05106@candide.cs.uchicago.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen Bailey wrote: > > I believe the point in factoring out the RDMA mechanism from iSCSI is to > > make the same hardware assists available to other application > > protocols. > > I agree. > > The motivation is primarily economic. If you implement a general > solution to 1) data steering, 2) congestion avoidance, 3) security, > you own the edge of all networking. great - go off and work on this in the TCP working group, and when it's done, iSCSI should seamlessly work on top of it, just like FTP, CIFS, HTTP and all other protocols that run over TCP should. -Matt > If you implement these for > storage, you only own storage. > > Given that it seems such an easy step to make the hardware slightly > more general and own the world, I'm stumped on why not. OK, I admit I > know the argument why not, because people think the more specific > solution more tractable. I disagree. The other reason why not is > some people really aren't sold on the necessity for hardware > implementation at all. > > BTW, I made this case at CERN recently: > http://hsi.web.cern.ch/HSI/HNF-Europe/Workshop%202000/Presentations/Com_Stor_Net.PDF > > Julo wasn't buying though. > > Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16194; 2 Oct 2000 12:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA04279 for ips-outgoing; Mon, 2 Oct 2000 11:18:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA04266 for ; Mon, 2 Oct 2000 11:17:54 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e92FHsS26469 for ; Mon, 2 Oct 2000 10:17:54 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA09390; Mon, 2 Oct 00 10:16:12 -0500 Message-Id: <10010021516.AA09390@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: iSCSI: remove recovery from transport-layer connection failur e(?) In-Reply-To: Message from JoeBre@Exabyte.COM of "Fri, 29 Sep 2000 09:35:48 MDT." <118A72FD3F02D211AD1000805F9AA58B02ADC1C2@boulderp03.Exabyte.COM> Date: Mon, 02 Oct 2000 10:16:06 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > If, on the other hand, iSCSI is not to be SAM/SAM-2 compliant... OK, I'm an idiot. Where in SAM-2 does it say that this behavior is required? > A mature tape drive, coupled with a mature backup application, is unlikely > to run off the end or get eaten. I guess I misspoke. Running off the end in this case means `filling up'. Amanda is a mature backup application, it writes on the tape in the drive until it's full. It reminds the operator to change the tape, but it doesn't require it. > I guess 'relatively high frequency' must be qualified. I was not making the claim that `tape is unreliable'; only that it's subject to more and more frequently occuring nonnominal phenomena than winchester disks. The same is true of other changable/ejectable media. For example, your own (Exabyte 8mm) tape drives (dunno if they still do this, but they certainly did once upon a time), refuse to operate if they haven't been cleaned within some particular interval. > The problem area that I foresee is where commands get lost in > transmission. This will not happen in iSCSI. Unlike ||SCSI and FCP, the underlying iSCSI transport is `reliable'. What that means is that if the command (or response, or data) does not get through in iSCSI, there is no immediate action that the ULP (SCSI) can take to rectify the situation. The path between the two endpoint IP addresses is dead. If the ULP choses to change one or the other endpoint IP addresses, there may be a new path, but that's a different kettle of fish. In fact, it IS the kettle of fish that started this discussion, but as David pointed out in 'iSCSI sessions: Let's try again', this is an enhancement over what other SCSI transports offer. A message a few months ago pointed out that FCP-2 does not provide this capability, although it has been cursorily discussed. Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03063; 29 Sep 2000 10:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA03932 for ips-outgoing; Fri, 29 Sep 2000 09:53:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA03917 for ; Fri, 29 Sep 2000 09:52:48 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id PAA59848 for ; Fri, 29 Sep 2000 15:52:11 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id PAA143556 for ; Fri, 29 Sep 2000 15:52:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256969.004C2F0A ; Fri, 29 Sep 2000 15:52:07 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 29 Sep 2000 16:14:44 +0300 Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I would add that we wanted the path to be an additional identifier that the target could use to determine what collection of LUs to present to the initiator. Julo csapuntz@csapuntz-u1.cisco.com on 23/09/2000 05:00:33 Please respond to csapuntz@csapuntz-u1.cisco.com To: "IP Storage" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Just for clarification... I was not proposing to add the extended URL scheme for the transport spec. It isn't necessary. In this thread, Doug makes an excellent point about LDAP being a superior mechanism for describing how to connect to the storage. Directory services such as LDAP, as Doug has pointed out, will be critical to managing large quantities of storage. A host can ask such a directory service for a list of storage devices it should mount and how to connect to those storage devices. The query against the directory server that returns this information can be based on machine ID, user ID, operating system ID, or even the owner's birthday. One of the things discovery will end up doing, no doubt, is defining LDAP schemas that describe how to connect to storage (i.e. use SCTP or TCP, what port, what target name, what LUN, what WWN, how to authenticate, etc.). However, there is one place where the transport protocol has to define a name: third party commands. There needs to be some kind of global name which the initiator can pass to the target. The name must be distillable into a string. The target must understand the name and be able to use the information to establish a connection to another target. One could say that the string that is passed is not specified by the standard but instead specified by some management software. I think this will lead to poor interoperability. The SCSI URL-type name is the current proposal for target name. Why is SCSI target name made up of a hostname + a path? Why is the hostname + path passed on connection setup? There are two reasons. I think NAT and IPv6 makes passing hostnames rather than addresses in protocols more desirable. Hostnames can be re-resolved as you cross addressing boundaries. The path is there so that the name can support multiple targets behind a single IP address without having to add entries to the DNS server. At Cisco, for example, I have no control over the local DNS servers and cannot add DNS entries for the ATAPI DVD and floppy in my computer. _Costa From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00769; 28 Sep 2000 22:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA18480 for ips-outgoing; Thu, 28 Sep 2000 21:41:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA12975 for ; Thu, 28 Sep 2000 18:19:09 -0400 (EDT) Received: from omgw2.boi.hp.com (omgw2.boi.hp.com [15.56.8.102]) by atlrel2.hp.com (Postfix) with ESMTP id EA2761065; Thu, 28 Sep 2000 18:17:38 -0400 (EDT) Received: from xrosebh3.rsvl.itc.hp.com (xrosebh3.rsvl.itc.hp.com [15.34.240.67]) by omgw2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id QAA02964; Thu, 28 Sep 2000 16:17:38 -0600 (MDT) Received: by xrosebh3.rsvl.itc.hp.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 16:17:27 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: Status summary on multiple connections Date: Thu, 28 Sep 2000 16:17:17 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk You have not included the performance issues that were brought up for the PA model. Somesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Tuesday, September 26, 2000 9:48 AM To: ips@ece.cmu.edu Subject: Status summary on multiple connections Dear colleagues, I am attempting to summarize where we stand with regard to the multiple connection issue and to the two possible models - Symmetric (S) and Asymmetric (A). Many of us feel strongly that the multiple connection issue is central to the whole design and cannot be added as an afterthought. Moreover designing the hooks to allow it later will certainly already force us to make a decision. And both the hardware and the software designers will be ill-served if we hand them a half-backed solution. However this is not an invitation to reiterate positions that where already stated. If you feel that I have grossly misstated anything in this note please write me and use the mailing list only if my answer is not satisfying. And yes - like the chairman - I would like to make progress but I don't see any way to do it without satisfactorily closing this issue. The reasons for multiple connections where discussed at some length and where very nicely summarized in a series of notes by Michael Krause (beginning of August). The core reasons for having multiple connections where the need for more bandwidth and availability than a single link can supply with a level of complexity affordable for simple installations and with a traffic engineering and management clearly separated from the transport users (SCSI). The session is embodying this requirement. The only major objections I have heard against this where those requiring to go all the way in having one TCP connection/LU - and after a short debate this objection was practically removed. The other objections we heard where that this is basically a transport issue and it should be solved at transport level. That might be true - but since many, if not most, of TCP applications do not have this requirement it is highly unlikely that TCP is going to do connection trunking in the foreseeable future. iSCSI can be designed to use multiple TCP connections in one of two ways: -Asymmetric - one TCP flow only carries commands the others carry only data -Symmetric - every flow carries both commands and their associated data The S version is designed in the I-D version 01 The S version requires an command ordering scheme and that is provided by a command counter and a sliding window scheme. It was argued that ordering needs might be more prevalent than usually thought and a good conservative design should preserve ordering. Ordering-per-LU (as it is designed in FCP-2 version 4) was considered impractical as it required initiators to maintain state for each LU - while the rest of the design required initiators only to maintain state for outstanding commands. Several comments on this list suggested that this windowing mechanism could also be used as a command-flow-control mechanism and that is a "bonus" of the scheme. The A version - comes in two flawors: - pure A (PA) in which ONLY commands flow on one TCP connection while data flow on DIFFERENT connections (with only one data connection being selected for a command); this scheme requires a minimum of 2 TCP connections although not necessarily on different physical links. - collapsed A (CA) in which commands flow one a single TCP connection while data flow on ANY connection; this scheme requires as a minimum 1 TCP connection. Here is first attempt to list the benefits and drawbacks of all of them: - S - benefits - well understood - simple hardware setup and/or TCP API activation - window mechanism can be also used for flow control - the minimum required is a single TCP connection - drawbacks - need to maintain a window mechanism - a multiplexing mechanism has to be carefully crafted to avoid closing TCP windows to severely affect command flow and performance - PA - benefits - TCP will both order the commands and provide for flow control through the TCP window mechanism - the multiplexing mechanism is simpler as closing TCP windows will never affect command flow - data flow can use a streamlined header (and processing) - drawbacks - more complex hardware and/or software API activation - need for a minimum of two TCP connections (not links) - CA - benefits - TCP will order the commands - needs a single TCP connection as a minimum - drawbacks - more complex hardware and/or software API activation - need for a minimum of two TCP connections (not links) - if command flow control is required counters and the sliding window mechanism are required - a multiplexing mechanism has to be carefully crafted to avoid closing TCP windows to severely affect command flow and performance >>From the above it should be apparent - as was already pointed out by Matt Wakeley - that the CA inherits the drawbacks of S and PA and as such the choice is really between S and PA. Julo From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03270; 29 Sep 2000 12:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA07611 for ips-outgoing; Fri, 29 Sep 2000 11:37:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from charon.Exabyte.COM ([12.17.207.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA07605 for ; Fri, 29 Sep 2000 11:37:03 -0400 (EDT) From: JoeBre@exabyte.com Received: from prime1.Exabyte.COM (prime1.Exabyte.COM [161.81.18.231]) by charon.Exabyte.COM (8.9.1a/8.9.1) with ESMTP id JAA05968; Fri, 29 Sep 2000 09:37:01 -0600 (MDT) Received: from bouldermta.exabyte.com (bouldermta.Exabyte.COM [161.81.192.16]) by prime1.Exabyte.COM (8.6.12/8.6.10) with ESMTP id JAA02006; Fri, 29 Sep 2000 09:37:01 -0600 Received: by bouldermta.Exabyte.COM with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 09:35:50 -0600 Message-ID: <118A72FD3F02D211AD1000805F9AA58B02ADC1C2@boulderp03.Exabyte.COM> To: steph@cs.uchicago.edu, ips@ece.cmu.edu Subject: RE: iSCSI: remove recovery from transport-layer connection failur e(?) Date: Fri, 29 Sep 2000 09:35:48 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C02A2A.F0EB2830" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C02A2A.F0EB2830 Content-Type: text/plain; charset="iso-8859-1" Comments inline... > -----Original Message----- > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > Sent: Tuesday, September 26, 2000 10:16 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: remove recovery from transport-layer connection > failure(?) > > > > Currently, iSCSI is spec'ed to recover from transport-layer > > connection failures. > > > > The main motivation for this decision was to support tape backup > > applications that are quite sensitive to any failures that get > > propogated to their layer. > > > > So, perhaps we can remove the requirement of recovering from > > transport-layer connection failures in iSCSI. This would simplify > > the protocol somewhat. Yes, it would simplify the protocol somewhat, and would also yield a protocol that would not accomodate tapes (or any other sequential device). I guess the question to be answered is wheter or not iSCSI will be a suitable transport/LLP for all SCSI traffic, as defined in SAM/SAM-2. If so, some means of recovering from transport layer problems must be built into the spec. It may be worth pointing out here that one of the most significant barriers to early widespread market acceptance of SANs has been the inability to back up data over the SAN. The most significant barrier to backup was the lack of support in FC for LLP/transport level recovery of commands. As I have mentioned before, issuing the same command twice to a sequential device corrupts data. If, on the other hand, iSCSI is not to be SAM/SAM-2 compliant... > > > > Thoughts? > > I'm all for eliminating command recovery. > > There seem to be several reasons advanced for command recovery. > > The first seems to be based upon an inappropriate analogy to FCP. > Command recovery had to be added to FCP-2 because the FC layer is > unreliable. A single dropped FC frame leads to a failed FCP command. > This clearly upsets tape operation even when the link is performing > nominally. In FCP, without command recovery, with some observable > frequency, you will get an expected error that leads to complete, > irrecoverable failure of a transfer stream. The other thing that > makes FCP-2 command recovery work well is when you are doing a write, > which is 90% (maybe it's 99%?) of tape operation, the target can > return an early indication of most frame drops, rather than waiting > for a timer to expire. > > TCP's reliability solves this problem in another way. By the time you > get a TCP connection failure, you have already exhausted a set of > reliability mechanisms which guarantee, with high certainty, that > further data can not be transferred between the two endpoints. > > `the two endpoints' phrase suggests the other reason advanced for > command recovery. That is, to permit path failover for commands which > are not idempotent, such as tape write sequential. The > problem with this, is that it is not clear HOW iSCSI command recovery > can actually work properly, given a TCP connection failure indication. > It takes a long time for a TCP connection to fail, and by that time, > I'm not sure recovery would reasonably be possible. Perhaps I'm in > error on this assumption. Can a tape guru (Joe from Exabyte?) comment > on whether recovery would be possible after many seconds (tens, > hundreds) have elapsed? Putting my tape guru hat on ;-) There are numerous backup applications that set command timeouts in excess of ten minutes. When one considers the sequential nature of tape, one will realize that this is an eminently reasonable thing to do. > The SCSI layer has never been solely responsible for ensuring reliable > backup. Macro scale things go wrong with tape (run off the end, get > eaten, etc..) with relatively high frequency. A low level backup > engine like tar or dump will fail on a SCSI error, and that's OK. > There must also be a higher level software component like Amanda, > which manages retries, including operator intervention, to ensure > reliable backup. A mature tape drive, coupled with a mature backup application, is unlikely to run off the end or get eaten. I guess 'relatively high frequency' must be qualified. Most installations will encounter neither of these problems ever in the service life of the system. The problem area that I foresee is where commands get lost in transmission. With disks, the command can merely be resent. This is fine, as the associated data goes to/is taken from the same LBA on the media. Whether it was the command or the response that got lost in transmission, data is fine. With sequential devices, if it was truly the command that got lost in transmission, the command can safely be resent. However, if the response gets lost rather than the command, reissuing a WRITE command results in two copies of the data on the tape - one at the LBA the host application expects it to be, and another starting at the next sequential LBA. The host's mapping of what data it thinks is at each LBA is corrupted, leading to its subsequent inability to recover data from that point to the end of the tape. > It seems like whether iSCSI has a command recovery mechanism should be > a function of whether somebody can stand up and say for sure that it > solves a real problem. So far it only seems like it MIGHT solve a > problem. Who can say `this solves MY problem!'? Command recovery solves the problem outlined above. It also solves the problem of being a SAM/SAM-2 compliant LLP. Joe Exbayte > Steph > ------_=_NextPart_001_01C02A2A.F0EB2830 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: iSCSI: remove recovery from transport-layer connection = failure(?)

Comments inline...

> -----Original Message-----
> From: Stephen Bailey [mailto:steph@cs.uchicago.edu]<= /FONT>
> Sent: Tuesday, September 26, 2000 10:16 = PM
> To: ips@ece.cmu.edu
> Subject: Re: iSCSI: remove recovery from = transport-layer connection
> failure(?)
>
>
> > Currently, iSCSI is spec'ed to recover = from transport-layer
> > connection failures.
> >
> > The main motivation for this decision was = to support tape backup
> > applications that are quite sensitive to = any failures that get
> > propogated to their layer.
> >
> > So, perhaps we can remove the requirement = of recovering from
> > transport-layer connection failures in = iSCSI. This would simplify
> > the protocol somewhat.

Yes, it would simplify the protocol somewhat, and = would also yield a protocol that would not accomodate tapes (or any = other sequential device).

I guess the question to be answered is wheter or not = iSCSI will be a suitable transport/LLP for all SCSI traffic, as defined = in SAM/SAM-2. If so, some means of recovering from transport layer = problems must be built into the spec.

It may be worth pointing out here that one of the = most significant barriers to early widespread market acceptance of SANs = has been the inability to back up data over the SAN. The most = significant barrier to backup was the lack of support in FC for = LLP/transport level recovery of commands. As I have mentioned before, = issuing the same command twice to a sequential device corrupts = data.

If, on the other hand, iSCSI is not to be SAM/SAM-2 = compliant...

> >
> > Thoughts?
>
> I'm all for eliminating command = recovery.
>
> There seem to be several reasons advanced for = command recovery.
>
> The first seems to be based upon an = inappropriate analogy to FCP.
> Command recovery had to be added to FCP-2 = because the FC layer is
> unreliable.  A single dropped FC frame = leads to a failed FCP command.
> This clearly upsets tape operation even when = the link is performing
> nominally.  In FCP, without command = recovery, with some observable
> frequency, you will get an expected error that = leads to complete,
> irrecoverable failure of a transfer = stream.  The other thing that
> makes FCP-2 command recovery work well is when = you are doing a write,
> which is 90% (maybe it's 99%?) of tape = operation, the target can
> return an early indication of most frame drops, = rather than waiting
> for a timer to expire.
>
> TCP's reliability solves this problem in = another way.  By the time you
> get a TCP connection failure, you have already = exhausted a set of
> reliability mechanisms which guarantee, with = high certainty, that
> further data can not be transferred between the = two endpoints.
>
> `the two endpoints' phrase suggests the other = reason advanced for
> command recovery.  That is, to permit path = failover for commands which
> are not idempotent, such as tape write = sequential.  The
> problem with this, is that it is not clear HOW = iSCSI command recovery
> can actually work properly, given a TCP = connection failure indication.
> It takes a long time for a TCP connection to = fail, and by that time,
> I'm not sure recovery would reasonably be = possible.  Perhaps I'm in
> error on this assumption.  Can a tape guru = (Joe from Exabyte?) comment
> on whether recovery would be possible after = many seconds (tens,
> hundreds) have elapsed?

Putting my tape guru hat on ;-)

There are numerous backup applications that set = command timeouts in excess of ten minutes. When one considers the = sequential nature of tape, one will realize that this is an eminently = reasonable thing to do.

 
> The SCSI layer has never been solely = responsible for ensuring reliable
> backup.  Macro scale things go wrong with = tape (run off the end, get
> eaten, etc..) with relatively high = frequency.  A low level backup
> engine like tar or dump will fail on a SCSI = error, and that's OK.
> There must also be a higher level software = component like Amanda,
> which manages retries, including operator = intervention, to ensure
> reliable backup.

A mature tape drive, coupled with a mature backup = application, is unlikely to run off the end or get eaten. I guess = 'relatively high frequency' must be qualified. Most installations will = encounter neither of these problems ever in the service life of the = system.

The problem area that I foresee is where commands get = lost in transmission. With disks, the command can merely be resent. = This is fine, as the associated data goes to/is taken from the same LBA = on the media. Whether it was the command or the response that got lost = in transmission, data is fine. With sequential devices, if it was truly = the command that got lost in transmission, the command can safely be = resent. However, if the response gets lost rather than the command, = reissuing a WRITE command results in two copies of the data on the tape = - one at the LBA the host application expects it to be, and another = starting at the next sequential LBA. The host's mapping of what data it = thinks is at each LBA is corrupted, leading to its subsequent inability = to recover data from that point to the end of the tape.

 
> It seems like whether iSCSI has a command = recovery mechanism should be
> a function of whether somebody can stand up and = say for sure that it
> solves a real problem.  So far it only = seems like it MIGHT solve a
> problem.  Who can say `this solves MY = problem!'?

Command recovery solves the problem outlined above. = It also solves the problem of being a SAM/SAM-2 compliant LLP.

Joe
Exbayte

> Steph
> 

------_=_NextPart_001_01C02A2A.F0EB2830-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04361; 29 Sep 2000 18:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA21514 for ips-outgoing; Fri, 29 Sep 2000 17:30:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from hqmail3.Crossroads.com ([63.237.99.226]) by ece.cmu.edu (8.9.2/8.8.8) with SMTP id RAA21497 for ; Fri, 29 Sep 2000 17:29:57 -0400 (EDT) Received: by hqmail3.Crossroads.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 16:22:38 -0500 Message-ID: <670C86291BFCD311AFE500508B94D4B6630CC0@hqmail1.crossroads.com> From: Robert Reynolds To: "'ips@ece.cmu.edu'" Subject: RE: not every song Date: Fri, 29 Sep 2000 16:31:16 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with Douglas. The WRITE-DATA and DATA-RESPONSE message become more important when you are talking longer distances, and longer delays. You can reduce your roundtrip latency by 1/2 on WRITE commands by using a WRITE-DATA message instead of independent WRITE-XFER_RDY-DATA messages. This can be a huge performance increase when talking about doing data replication across longer latency connections. This is the difference between a 2 hour database sync. delay and a 4 hour sync delay. I think it is a mistake that FCP-2 removed those messages because they weren't currently implemented and I think it would be a mistake to remove them from iSCSI. Bob > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Thursday, September 28, 2000 3:50 PM > To: Robert Snively; John Hufferd/San Jose/IBM; ips@ece.cmu.edu > Subject: RE: not every song > > > Bob, > > I am not sure I understand the point. Not that I would advocate such > modifications, but should these points become a deciding > factor, the service > parameter for such negotiations has already been defined > within FCP-2 as of > recently. Such a decision for removal may have been premature should > network latency argue for Word 3, bit 3 being set. > > Again, I am agnostic about this issue, but I would rather > stick with FCP > structures. Should there be attempts to provide zero copy, > adopting FCP > structures permit a greater reliance on these structure's stability. > > Doug > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > Sent: Thursday, September 28, 2000 12:37 PM > > To: 'Douglas Otis'; John Hufferd/San Jose/IBM; ips@ece.cmu.edu > > Subject: RE: not every song > > > > > > > FCP > > > offers WRITE-DATA and READ-RESPONSE structures if desired. > > > > Reminder: > > > > No, it doesn't. FCP-2 (the new and improved FCP) took the > > function out because nobody implemented it and it did not make > > sense. It would have required additional recovery mechanisms and > > it did not improve performance. > > > > Bob > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15651; 2 Oct 2000 8:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA27655 for ips-outgoing; Mon, 2 Oct 2000 07:54:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA27645 for ; Mon, 2 Oct 2000 07:54:41 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id NAA94026 for ; Mon, 2 Oct 2000 13:54:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id NAA80036 for ; Mon, 2 Oct 2000 13:54:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.00415C03 ; Mon, 2 Oct 2000 13:53:53 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 12:19:19 +0300 Subject: RE: iSCSI: remove recovery from transport-layer connection failur e(?) Mime-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=b2dqTioK9GF4Yif0MgWHOlkCfYr6cWCxtTih4PRlyyaniiLC4SWC3vCB" Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk --0__=b2dqTioK9GF4Yif0MgWHOlkCfYr6cWCxtTih4PRlyyaniiLC4SWC3vCB Content-type: text/plain; charset=us-ascii Content-Disposition: inline This pretty much summarizes our arguments for recovery and command ordering. I would like to add that Randy Haagens also made the point that new remote tape and disk applications could hide latency by striping over several (or one) connection achieving a pipelining effect provided that order is preserved. Julo ---------------------- Forwarded by Julian Satran/Haifa/IBM on 02/10/2000 12:15 --------------------------- JoeBre@exabyte.com on 29/09/2000 18:35:48 Please respond to JoeBre@exabyte.com To: steph@cs.uchicago.edu, ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: iSCSI: remove recovery from transport-layer connection failur e(?) Comments inline... > -----Original Message----- > From: Stephen Bailey [mailto:steph@cs.uchicago.edu] > Sent: Tuesday, September 26, 2000 10:16 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: remove recovery from transport-layer connection > failure(?) > > > > Currently, iSCSI is spec'ed to recover from transport-layer > > connection failures. > > > > The main motivation for this decision was to support tape backup > > applications that are quite sensitive to any failures that get > > propogated to their layer. > > > > So, perhaps we can remove the requirement of recovering from > > transport-layer connection failures in iSCSI. This would simplify > > the protocol somewhat. Yes, it would simplify the protocol somewhat, and would also yield a protocol that would not accomodate tapes (or any other sequential device). I guess the question to be answered is wheter or not iSCSI will be a suitable transport/LLP for all SCSI traffic, as defined in SAM/SAM-2. If so, some means of recovering from transport layer problems must be built into the spec. It may be worth pointing out here that one of the most significant barriers to early widespread market acceptance of SANs has been the inability to back up data over the SAN. The most significant barrier to backup was the lack of support in FC for LLP/transport level recovery of commands. As I have mentioned before, issuing the same command twice to a sequential device corrupts data. If, on the other hand, iSCSI is not to be SAM/SAM-2 compliant... > > > > Thoughts? > > I'm all for eliminating command recovery. > > There seem to be several reasons advanced for command recovery. > > The first seems to be based upon an inappropriate analogy to FCP. > Command recovery had to be added to FCP-2 because the FC layer is > unreliable. A single dropped FC frame leads to a failed FCP command. > This clearly upsets tape operation even when the link is performing > nominally. In FCP, without command recovery, with some observable > frequency, you will get an expected error that leads to complete, > irrecoverable failure of a transfer stream. The other thing that > makes FCP-2 command recovery work well is when you are doing a write, > which is 90% (maybe it's 99%?) of tape operation, the target can > return an early indication of most frame drops, rather than waiting > for a timer to expire. > > TCP's reliability solves this problem in another way. By the time you > get a TCP connection failure, you have already exhausted a set of > reliability mechanisms which guarantee, with high certainty, that > further data can not be transferred between the two endpoints. > > `the two endpoints' phrase suggests the other reason advanced for > command recovery. That is, to permit path failover for commands which > are not idempotent, such as tape write sequential. The > problem with this, is that it is not clear HOW iSCSI command recovery > can actually work properly, given a TCP connection failure indication. > It takes a long time for a TCP connection to fail, and by that time, > I'm not sure recovery would reasonably be possible. Perhaps I'm in > error on this assumption. Can a tape guru (Joe from Exabyte?) comment > on whether recovery would be possible after many seconds (tens, > hundreds) have elapsed? Putting my tape guru hat on ;-) There are numerous backup applications that set command timeouts in excess of ten minutes. When one considers the sequential nature of tape, one will realize that this is an eminently reasonable thing to do. > The SCSI layer has never been solely responsible for ensuring reliable > backup. Macro scale things go wrong with tape (run off the end, get > eaten, etc..) with relatively high frequency. A low level backup > engine like tar or dump will fail on a SCSI error, and that's OK. > There must also be a higher level software component like Amanda, > which manages retries, including operator intervention, to ensure > reliable backup. A mature tape drive, coupled with a mature backup application, is unlikely to run off the end or get eaten. I guess 'relatively high frequency' must be qualified. Most installations will encounter neither of these problems ever in the service life of the system. The problem area that I foresee is where commands get lost in transmission. With disks, the command can merely be resent. This is fine, as the associated data goes to/is taken from the same LBA on the media. Whether it was the command or the response that got lost in transmission, data is fine. With sequential devices, if it was truly the command that got lost in transmission, the command can safely be resent. However, if the response gets lost rather than the command, reissuing a WRITE command results in two copies of the data on the tape - one at the LBA the host application expects it to be, and another starting at the next sequential LBA. The host's mapping of what data it thinks is at each LBA is corrupted, leading to its subsequent inability to recover data from that point to the end of the tape. > It seems like whether iSCSI has a command recovery mechanism should be > a function of whether somebody can stand up and say for sure that it > solves a real problem. So far it only seems like it MIGHT solve a > problem. Who can say `this solves MY problem!'? Command recovery solves the problem outlined above. It also solves the problem of being a SAM/SAM-2 compliant LLP. Joe Exbayte > Steph > (Embedded image moved to file: pic31154.pcx) - att1.htm JoeBre@exabyte.com on 29/09/2000 19:16:08 Please respond to JoeBre@exabyte.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: Status summary on multiple connections Much good discussion on SCSI device level flow control snipped.... > From: Robert Snively > To: "'David Robinson'" , Robert Snively > > Subject: RE: Status summary on multiple connections > Date: Thu, 28 Sep 2000 09:36:50 -0700 ... > are again available. Note that there is a possibility that commands > that are inflight and have ordering constraints may be accepted out of > order, a question that has caused lots of agonizing, but is apparently > reasonably well managed by most file systems today by the selective > use of ordering only for blocking boundaries of a particular logical > stream of commands. I am left with the following impression as to what was indicated here: - In general, command ordering is not relevant - If the initiator filesystem detects an ordering dependency, it will wait until outstanding commands are complete before issuing the dependant command. This may be a reasonable means of operation for the disk world. It is woefully inadequate for the tape world, as follows: - In general, command ordering is crucial - out of order command processing will lead to data corruption. - This would require the initiator backup application to block on completion of every single write command of a backup operation before issuing the next command. If this blocking were performed, both the throughput and capacity of a tape device/media would be negatively impacted by an order of magnitude or more. This would occur even assuming an instantaneous transport. Joe Breher Exabyte Corp (Embedded image moved to file: pic29236.pcx) - att1.htm --0__=b2dqTioK9GF4Yif0MgWHOlkCfYr6cWCxtTih4PRlyyaniiLC4SWC3vCB Content-type: application/octet-stream; name="pic31154.pcx" Content-Disposition: attachment; filename="pic31154.pcx" Content-transfer-encoding: base64 CgUBCAAAAAAfAB8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAABIAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAADRD8gPxA/CDw/ED88AxADFD8MPD8QPAA8IDwgPCA8IDwgPCA8IDwgPwgDF D8IPD8QPAAgPCA8IDwgPCA8IDwgPCA8IAAgAxA/CDw/EDwAPCA8IDwgPCA8IDwgPCA8IDwDCCADE D8IPxA8ACA8IDwgPCA8IDwgPCA8IDwgAwwgAww/CD8QPAA8IDwgPCA8IDwgPCA8IDwgPAMQIAMMP D8QPAAgPCA8IDwgPCA8IDwgPCA8IxgAAwg8PxA8ADwgPCA8IDwgPCA8IDwgPCA8IDwgPCA8Awg8P xA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8IDwgPCA8IDwgPCA8A wg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8IDwgPCA8IDwgP CA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8IDwgPCA8I DwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8IDwgP CA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8I DwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgP CA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8I DwgPCA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgP CA8IDwgPCA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8A DwgPCA8IDwgPCA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8P xA8ADwgPCA8IDwgPCA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgA wg8PxA/PAMcAwwDCDw/RD8gPxA/CDw8MAAAAgAAAAIAAgIAAAACAgACAAICAgICAwMDA/wAAAP8A //8AAAD//wD/AP//////AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA --0__=b2dqTioK9GF4Yif0MgWHOlkCfYr6cWCxtTih4PRlyyaniiLC4SWC3vCB Content-type: application/octet-stream; name="pic29236.pcx" Content-Disposition: attachment; filename="pic29236.pcx" Content-transfer-encoding: base64 CgUBCAAAAAAfAB8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAABIAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAADRD8gPxA/CDw/ED88AxADFD8MPD8QPAA8IDwgPCA8IDwgPCA8IDwgPwgDF D8IPD8QPAAgPCA8IDwgPCA8IDwgPCA8IAAgAxA/CDw/EDwAPCA8IDwgPCA8IDwgPCA8IDwDCCADE D8IPxA8ACA8IDwgPCA8IDwgPCA8IDwgAwwgAww/CD8QPAA8IDwgPCA8IDwgPCA8IDwgPAMQIAMMP D8QPAAgPCA8IDwgPCA8IDwgPCA8IxgAAwg8PxA8ADwgPCA8IDwgPCA8IDwgPCA8IDwgPCA8Awg8P xA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8IDwgPCA8IDwgPCA8A wg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8IDwgPCA8IDwgP CA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8IDwgPCA8I DwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8IDwgP CA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgPCA8I DwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8IDwgP CA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgPCA8I DwgPCA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8ADwgP CA8IDwgPCA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8PxA8A DwgPCA8IDwgPCA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgAwg8P xA8ADwgPCA8IDwgPCA8IDwgPCA8IDwgPCA8Awg8PxA8ACA8IDwgPCA8IDwgPCA8IDwgPCA8IDwgA wg8PxA/PAMcAwwDCDw/RD8gPxA/CDw8MAAAAgAAAAIAAgIAAAACAgACAAICAgICAwMDA/wAAAP8A //8AAAD//wD/AP//////AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA --0__=b2dqTioK9GF4Yif0MgWHOlkCfYr6cWCxtTih4PRlyyaniiLC4SWC3vCB-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id af11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10967; 1 Oct 2000 5:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA00154 for ips-outgoing; Sun, 1 Oct 2000 04:30:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00137 for ; Sun, 1 Oct 2000 04:30:00 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id KAA72594; Sun, 1 Oct 2000 10:29:26 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA42174; Sun, 1 Oct 2000 10:29:16 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.002E9E9D ; Sun, 1 Oct 2000 10:29:12 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: "Binford, Charles" cc: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 09:25:41 +0300 Subject: RE: iSCSI: Behaviors Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Thanks, Julo "Binford, Charles" on 27/09/2000 16:44:10 Please respond to "Binford, Charles" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: RE: iSCSI: Behaviors FYI: Just last month at the T10 committee my proposal was approved for inclusion into SAM-2, and SPC-2. It defines a new status value of TASK ABORTED. When enabled via a mode page, this new status is returned to an initiator if his I/O was terminated because of a task management function performed by another initiator. So, if initiator A sends down a LU Reset, all of initiator B's I/Os would be returned to him with a Task Abort status so he knows what happened. See T10/00-229r3 for details. Charles Binford LSI Logic Storage Systems (316) 636-8566 > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Wednesday, September 27, 2000 12:44 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: Behaviors > > > > > JP, > > We will try to fix it in the next draft. > > Thanks, > Julo > > Raghavendra Rao on 14/09/2000 21:13:18 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: Behaviors > > > > > > It would be great if the draft can clearly point to a SCSI document > (appropriate version) for greater details where little is mentioned > in the draft, particularly with regard to target and LUN resets. > > 1. How are the outstanding commands affected for the initiator > that performed the reset ? > > 2. What is the action on the newly (just) received commands ? > > 3. How are the outstanding/new commands sent from a different > initiator (other than the one that performed the reset) > affected ? > > 4. Will devices such as Tapes rewind after reset ? (They > currently > do). > > Also, What is the reason for not generating a LUN reset asynchronous > notification from the targets ? (it is only done for target reset) > > Thanks > -JP > > > > > - att1.htm From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15642; 2 Oct 2000 8:55 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id HAA27629 for ips-outgoing; Mon, 2 Oct 2000 07:54:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id HAA27625 for ; Mon, 2 Oct 2000 07:54:29 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id NAA103740 for ; Mon, 2 Oct 2000 13:53:56 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id NAA130410 for ; Mon, 2 Oct 2000 13:53:56 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.00415BAA ; Mon, 2 Oct 2000 13:53:53 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 13:19:44 +0300 Subject: RE: not every song Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I any case, while we are here to discuss iSCSI, we have the both. Julo Robert Reynolds on 30/09/2000 00:31:16 Please respond to Robert Reynolds To: "'ips@ece.cmu.edu'" cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: not every song I agree with Douglas. The WRITE-DATA and DATA-RESPONSE message become more important when you are talking longer distances, and longer delays. You can reduce your roundtrip latency by 1/2 on WRITE commands by using a WRITE-DATA message instead of independent WRITE-XFER_RDY-DATA messages. This can be a huge performance increase when talking about doing data replication across longer latency connections. This is the difference between a 2 hour database sync. delay and a 4 hour sync delay. I think it is a mistake that FCP-2 removed those messages because they weren't currently implemented and I think it would be a mistake to remove them from iSCSI. Bob > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Thursday, September 28, 2000 3:50 PM > To: Robert Snively; John Hufferd/San Jose/IBM; ips@ece.cmu.edu > Subject: RE: not every song > > > Bob, > > I am not sure I understand the point. Not that I would advocate such > modifications, but should these points become a deciding > factor, the service > parameter for such negotiations has already been defined > within FCP-2 as of > recently. Such a decision for removal may have been premature should > network latency argue for Word 3, bit 3 being set. > > Again, I am agnostic about this issue, but I would rather > stick with FCP > structures. Should there be attempts to provide zero copy, > adopting FCP > structures permit a greater reliance on these structure's stability. > > Doug > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > Sent: Thursday, September 28, 2000 12:37 PM > > To: 'Douglas Otis'; John Hufferd/San Jose/IBM; ips@ece.cmu.edu > > Subject: RE: not every song > > > > > > > FCP > > > offers WRITE-DATA and READ-RESPONSE structures if desired. > > > > Reminder: > > > > No, it doesn't. FCP-2 (the new and improved FCP) took the > > function out because nobody implemented it and it did not make > > sense. It would have required additional recovery mechanisms and > > it did not improve performance. > > > > Bob > > > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03272; 29 Sep 2000 12:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06578 for ips-outgoing; Fri, 29 Sep 2000 11:08:54 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06568 for ; Fri, 29 Sep 2000 11:08:47 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id 217711C4 for ; Fri, 29 Sep 2000 08:08:32 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id IAA20932 for ; Fri, 29 Sep 2000 08:08:54 -0700 (PDT) Message-ID: <39D4B057.6B7AF295@agilent.com> Date: Fri, 29 Sep 2000 08:08:07 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: Why FCP doesn't need RDMA? It has a better way. References: <39D481F3.A2974E8B@stewart.chicago.il.us> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "Randall R. Stewart" wrote: > "GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > > > > Even iSCSI does not really need RDMA. The underlying note in > > the whole debate is really the preservation of message boundaries. > > > > If we could use one of the reserved bits in the TCP header to say, > > do the repacketize or adjust segment boundaries, and each TCP > > segment starts with an iSCSI header, there should be no need for > > another protocol (or TCP option header). > > > > I don't think you can go playing with TCP option bits. This implies > changing TCP and I don't see this in the WG charter. If bits need > to be changed then a draft needs to be written and submited to > the proper WG.. in this case I would say transport area WG... I agree. There is no need to use new option bits. Please read my draft http://www.ietf.org/internet-drafts/draft-wakeley-iscsi-msgbndry-00.txt It describes a proposal to discover message boundaries using the existing TCP mechanisms (the urgent pointer). > > > I think in some respects this is exactly what Costa has done... Now > as far as message boundary preservation, you get this for free > with SCTP.. and has Douglas as already pointed out, you could very > easily implement in SCTP the ability to copy a specific stream > into receive buffers .. i.e. tell the sender, but the data on > stream X and then as it arrives dump it directly to user buffers. Randall, please stop telling us all about the fabulous features of SCTP. We are all sick to death of hearing about SCTP. -Matt > > > This would take a bit of work, but so does the RDMA solution :/ > > R > > > The iSCSI header contains enough information in it to enable > > the recepient to determine where to put the data. > > > > Somesh > > > > -----Original Message----- > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > Sent: Thursday, September 28, 2000 8:29 AM > > To: 'Matt Wakeley'; IPS Reflector > > Subject: RE:Why FCP doesn't need RDMA? It has a better way. > > > > > I object to mandating iSCSI use an RDMA option because: > > > > > > - (main reason) there isn't any standardized mechanism now, and I > > > don't want to hold up iSCSI while one crawls through the > > > standards process. > > > > > > - I don't think RDMA is needed. FCP doesn't use it, and it > > > works great with > > > the > > > FC protocol chips that "accelerate" FCP. > > > > Actually, RDMA is not needed in FCP because all protocol chips > > implemented perform a real peer-to-peer DMA straight to the > > data areas specified by the user's interaction with the operating > > systems allocation algorithms. The combination of the FCP/SCSI > > pointer structure, task tagging, and the FC relative offset perform the > > function you would otherwise have to use RDMA to accomplish. > > > > Bob Snively > > Brocade Communications Phone 408 487 8135 > > 1745 Technology Drive > > San Jose, CA 95110 Email rsnively@brocade.com > > -- > Randall R. Stewart > randall@stewart.chicago.il.us or rrs@cisco.com > 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04559; 29 Sep 2000 19:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA24571 for ips-outgoing; Fri, 29 Sep 2000 19:06:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA24565 for ; Fri, 29 Sep 2000 19:06:07 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Fri, 29 Sep 2000 16:08:23 -0700 Message-ID: From: Charles Monia To: Peter Johansson , IP Storage Subject: RE: iSCSI: Flow Control Date: Fri, 29 Sep 2000 16:08:22 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Peter Johansson [mailto:PJohansson@ACM.org] > Sent: Friday, September 29, 2000 2:57 PM > To: IP Storage > Subject: RE: iSCSI: Flow Control > > > At 04:39 PM 9/29/00, Black_David@emc.com wrote: > > >The open question to the list is whether there's value in > allowing some > >amount of immediate data (e.g., for targets that need fast startup on > >long latency connections, and are prepared to deploy the buffering > >required to make it work reliably), or whether we ought to > follow FCP-2 > >and forbid immediate data, which will impose a round trip > delay (command > >out, R2T back) before data starts to flow. I think I've > seen a couple of > >comments in favor of this, but more discussion is in order. > Since this is testing the water for consensus, I'd say yes -- some way to send "immediate" data on a write command should be in the spec. I also agree that the FCP recipe for doing this, as described elsewhere by Charles Binford, is the right approach. ie. Let's not invent something different. Charles From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15777; 2 Oct 2000 9:32 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA28504 for ips-outgoing; Mon, 2 Oct 2000 08:39:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lukla.Sun.COM (lukla.Sun.COM [192.18.98.31]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA28497 for ; Mon, 2 Oct 2000 08:39:40 -0400 (EDT) Received: from eastmail1.East.Sun.COM ([129.148.1.240]) by lukla.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id GAA06983; Mon, 2 Oct 2000 06:39:38 -0600 (MDT) Received: from swanaba.east (swanaba.East.Sun.COM [129.148.162.54]) by eastmail1.East.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id IAA29983; Mon, 2 Oct 2000 08:39:36 -0400 (EDT) Received: from sun.com (swantty.East.Sun.COM [129.148.162.52]) by swanaba.east (8.8.8+Sun/8.8.8) with ESMTP id IAA09014; Mon, 2 Oct 2000 08:38:43 -0400 (EDT) Message-ID: <39D882F1.C46F5243@sun.com> Date: Mon, 02 Oct 2000 06:43:29 -0600 From: "Mark A. Carlson" Organization: Sun Microsystems, Inc. - Jiro Group http://www.jiro.com X-Mailer: Mozilla 4.75 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Randall R. Stewart" CC: David Robinson , ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: <200009291825.LAA13471@ha10nwk.EBay.Sun.COM> <39D87E35.51050250@stewart.chicago.il.us> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "Randall R. Stewart" wrote: > > I really do NOT see how one can allow a TCP to deliver out of order > though, oh > I know technically how to do it, but it is a fundamental violation > of what TCP is supposed to be delivering.... i.e. the bytes being > transfered from 0 to N... The scenario which everyone seems to be worried about is that packets are delivered in order, but the ACK for one of them is lost going back. Everything gets delivered to iSCSI, then the non-ACKed packet is delivered again. Does this really happen in today's stacks? It seems like the TCP stack would see it as a duplicate that has already been delivered to the upper layer and drop it. TCP does have sequence numbers and knows this. The only way I could imagine this happening is if there was sufficient data to wrap the TCP sequence space. -- mark From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04379; 29 Sep 2000 18:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23366 for ips-outgoing; Fri, 29 Sep 2000 18:24:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23361 for ; Fri, 29 Sep 2000 18:24:18 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id PAA15180; Fri, 29 Sep 2000 15:10:57 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA08951; Fri, 29 Sep 2000 15:08:44 -0700 From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: Why FCP doesn't need RDMA? It has a better way. Date: Fri, 29 Sep 2000 15:26:45 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <39D4B057.6B7AF295@agilent.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, The TCP specification does not end at protocol structures. You should not expect it is okay to go about redefining use of pointers, bits, data placement, buffer use and expect the API to accommodate changes. TCP is not suitable for SCSI and you appear to know this. End schemes to fix TCP; address the issues of TCP and SCTP. Do not stand on desires for TCP over SCTP or that a pact was made to promote TCP. SCTP performance will be superior to TCP using either hardware or software. SCTP is easier to accelerate in hardware than TCP and any resulting SCSI standard will change substantially as a result of the features of SCTP. Even if there was a generic RDMA option, SCSI would look more like Firewire as a result. A debased version of RDMA has already been offered that avoids stepping on accepted options. To talk seriously about your concerns, SCTP is a legitimate aspect of this topic. Your false notion makes TCP even less sacrosanct. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Friday, September 29, 2000 8:08 AM > To: IPS Reflector > Subject: Re: Why FCP doesn't need RDMA? It has a better way. > > > "Randall R. Stewart" wrote: > > > "GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > > > > > > Even iSCSI does not really need RDMA. The underlying note in > > > the whole debate is really the preservation of message boundaries. > > > > > > If we could use one of the reserved bits in the TCP header to say, > > > do the repacketize or adjust segment boundaries, and each TCP > > > segment starts with an iSCSI header, there should be no need for > > > another protocol (or TCP option header). > > > > > > > I don't think you can go playing with TCP option bits. This implies > > changing TCP and I don't see this in the WG charter. If bits need > > to be changed then a draft needs to be written and submited to > > the proper WG.. in this case I would say transport area WG... > > I agree. There is no need to use new option bits. Please read my > draft > http://www.ietf.org/internet-drafts/draft-wakeley-iscsi-msgbndry-00.txt > > It describes a proposal to discover message boundaries using the existing > TCP mechanisms (the urgent pointer). > > > > > > > I think in some respects this is exactly what Costa has done... Now > > as far as message boundary preservation, you get this for free > > with SCTP.. and has Douglas as already pointed out, you could very > > easily implement in SCTP the ability to copy a specific stream > > into receive buffers .. i.e. tell the sender, but the data on > > stream X and then as it arrives dump it directly to user buffers. > > Randall, please stop telling us all about the fabulous features > of SCTP. We > are all sick to death of hearing about SCTP. > > -Matt > > > > > > > This would take a bit of work, but so does the RDMA solution :/ > > > > R > > > > > The iSCSI header contains enough information in it to enable > > > the recepient to determine where to put the data. > > > > > > Somesh > > > > > > -----Original Message----- > > > From: Robert Snively [mailto:rsnively@Brocade.COM] > > > Sent: Thursday, September 28, 2000 8:29 AM > > > To: 'Matt Wakeley'; IPS Reflector > > > Subject: RE:Why FCP doesn't need RDMA? It has a better way. > > > > > > > I object to mandating iSCSI use an RDMA option because: > > > > > > > > - (main reason) there isn't any standardized mechanism now, and I > > > > don't want to hold up iSCSI while one crawls through the > > > > standards process. > > > > > > > > - I don't think RDMA is needed. FCP doesn't use it, and it > > > > works great with > > > > the > > > > FC protocol chips that "accelerate" FCP. > > > > > > Actually, RDMA is not needed in FCP because all protocol chips > > > implemented perform a real peer-to-peer DMA straight to the > > > data areas specified by the user's interaction with the operating > > > systems allocation algorithms. The combination of the FCP/SCSI > > > pointer structure, task tagging, and the FC relative offset > perform the > > > function you would otherwise have to use RDMA to accomplish. > > > > > > Bob Snively > > > Brocade Communications Phone 408 487 8135 > > > 1745 Technology Drive > > > San Jose, CA 95110 Email rsnively@brocade.com > > > > -- > > Randall R. Stewart > > randall@stewart.chicago.il.us or rrs@cisco.com > > 815-342-5222 (cell) 815-477-2127 (work) > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01299; 29 Sep 2000 1:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA22090 for ips-outgoing; Fri, 29 Sep 2000 00:21:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA22085 for ; Fri, 29 Sep 2000 00:21:44 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id 15959465 for ; Thu, 28 Sep 2000 21:21:43 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id VAA04823 for ; Thu, 28 Sep 2000 21:22:18 -0700 (PDT) Message-ID: <39D418CB.3C7D80A2@agilent.com> Date: Thu, 28 Sep 2000 21:21:31 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: New List: rdma@cisco.com: to discuss RDMA References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk meth@il.ibm.com wrote: > I think we should specify an RDMA mechanism for use with iSCSI, as this > will help all interested parties to have interoperable RDMA mechanisms. I > don't think RDMA should be mandated for iSCSI; I assume there will be > implemenations on hosts and devices that won't have the resources to > perform RDMA. > > - Kalman Meth iSCSI does not need an RDMA mechanism because (from David Black's message): > > Actually, RDMA is not needed in FCP because all protocol chips > > implemented perform a real peer-to-peer DMA straight to the > > data areas specified by the user's interaction with the operating > > systems allocation algorithms. The combination of the FCP/SCSI > > pointer structure, task tagging, and the FC relative offset perform the > > function you would otherwise have to use RDMA to accomplish. > > And this illuminates the design tradeoff that may motivate RDMA. If > one only wants to accelerate one protocol (SCSI/FCP in the above > example) then having hardware understand its headers and doing > the DMA on that basis is a fairly obvious way to go - HBAs for both > parallel SCSI and Fibre Channel (SCSI/FCP) do this. The issue is that TCP does not provide message framing such that the out of order messages could be found and processed. Besides, how can you "specify an RDMA mechanism for use with iSCSI" if the mechanism in TCP has not been defined, much less standardized? -Matt From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04574; 29 Sep 2000 19:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA24814 for ips-outgoing; Fri, 29 Sep 2000 19:15:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA24807 for ; Fri, 29 Sep 2000 19:14:54 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA15222; Fri, 29 Sep 2000 16:01:33 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA09117; Fri, 29 Sep 2000 15:59:20 -0700 From: "Douglas Otis" To: , , Subject: RE: Enough on TCP vs. FCTP, please Date: Fri, 29 Sep 2000 16:17:22 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: <0F31E5C394DAD311B60C00E029101A070410102B@corpmx9.isus.emc.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, You and I have a different opinion about reasonable changes to TCP. It is not reasonable to redefine the urgent pointer. When used for SCSI, the urgent pointer is defined to be pointing to the first message boundary within the frame? There is an equal lack of consideration for legitimate alternatives to mucking with TCP. But this is based on my concepts of reasonable. Doug > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, September 29, 2000 3:49 PM > To: dotis@sanlight.net; matt_wakeley@agilent.com; ips@ece.cmu.edu > Subject: Enough on TCP vs. FCTP, please > > > > End schemes to fix TCP; address the issues of TCP and SCTP. Do > not stand > on > > desires for TCP over SCTP or that a pact was made to promote TCP. SCTP > > performance will be superior to TCP using either hardware or software. > SCTP > > is easier to accelerate in hardware than TCP and any resulting SCSI > standard > > will change substantially as a result of the features of SCTP. Even if > > there was a generic RDMA option, SCSI would look more like Firewire as a > > result. A debased version of RDMA has already been offered that avoids > > stepping on accepted options. > > I think this sort of debate really needs to stop. Matt's draft is a > reasonable > attempt to apply an existing TCP feature to address a shortcoming in use > of TCP for iSCSI. It is not perfect, as is the case for most engineering > tradeoffs and compromises. SCTP does have a number of improvements > over TCP (e.g., the whole session discussion is moot for SCTP), but TCP > has significantly more deployment experience and a great deal of work in > progress on hardware acceleration. > > I am restating the WG consensus that iSCSI needs to anticipate both > SCTP and TCP, with the assumption that TCP deployments will come > first. Given this, Matt's draft is within scope for the WG. If anyone > other > than Doug Otis disagrees, please say so and say why on the mailing list. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00760; 28 Sep 2000 22:44 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA18498 for ips-outgoing; Thu, 28 Sep 2000 21:42:20 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from cosrel1.hp.com (cosrel1.hp.com [156.153.255.170]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA16297 for ; Thu, 28 Sep 2000 20:13:51 -0400 (EDT) Received: from omgw5.rsvl.itc.hp.com (omgw5.rsvl.itc.hp.com [15.34.240.65]) by cosrel1.hp.com (Postfix) with ESMTP id 06AC6222; Thu, 28 Sep 2000 18:14:00 -0600 (MDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by omgw5.rsvl.itc.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id SAA01002; Thu, 28 Sep 2000 18:13:48 -0600 (MDT) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id ; Thu, 28 Sep 2000 18:13:47 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: csapuntz@cisco.com, Jim Williams Cc: ips@ece.cmu.edu Subject: RE: New List: rdma@cisco.com: to discuss RDMA Date: Thu, 28 Sep 2000 17:15:15 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Mandating an RDMA mechanism would delay product delivery for sure. I am not familiar enough with the IETF process to understand how it might impact the specification process since it requires a change to TCP. Also RDMA does nothing to help the case where we want to avoid round trip latencies on small (2Kor 4K) writes. -----Original Message----- From: csapuntz@cisco.com [mailto:csapuntz@cisco.com] Sent: Wednesday, September 27, 2000 10:16 AM To: Jim Williams Cc: ips@ece.cmu.edu; csapuntz@cisco.com Subject: Re: New List: rdma@cisco.com: to discuss RDMA Does anybody on the list object to specifying an RDMA mechanism for use with iSCSI? Does anybody on the list object to mandating an RDMA mechanism? Please include your reasons. Current RDMA proposals: draft-csapuntz-tcprdma-00.txt draft-dicecco-vitcp-00.txt -Costa > Before going too far down this road, it is important to > understand if there is support for using an RDMA mechanism > as a basis for iSCSI. Will the next draft of the iSCSI > protocol actually be based on an RDMA mechanism that > would be defined? The answers to these questions and > the specific pros and cons need to be the driving > force behind the RDMA discussion. From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01114; 29 Sep 2000 0:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA20895 for ips-outgoing; Thu, 28 Sep 2000 23:31:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA20891 for ; Thu, 28 Sep 2000 23:31:34 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by atlrel2.hp.com (Postfix) with ESMTP id 6CAACA70 for ; Thu, 28 Sep 2000 23:31:33 -0400 (EDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id UAA29901 for ; Thu, 28 Sep 2000 20:32:08 -0700 (PDT) Message-ID: <39D40D09.D9EF487E@agilent.com> Date: Thu, 28 Sep 2000 20:31:21 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk John Hufferd/San Jose/IBM wrote: > I am troubled with the fact that I think many folks have already built > iSCSI Initiators and Targets that use just one connection per session. > > But what about the implementations that are currently in flight. A couple thoughts on worrying about "implementations in flight". First off, the posted version of the iSCSI spec is the first published version, and was certainly not complete. No one should be hard coding to that spec. Especially in light that David Black has decided that parts of it are to be ripped out (what if I had been implementing a solution that required multiple TCP connections or command reference numbers and then you take it away?). Second, all the current "implementations" are either (a) proprietary, mostly UDP based FCP tunneling implementations, or (b) demo projects to demonstrate feasibility. I don't think we have to worry about existing "in flight" implementations based on the first draft. After a few revs of the document, that will be a different story... -Matt From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05050; 29 Sep 2000 22:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA27883 for ips-outgoing; Fri, 29 Sep 2000 21:13:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA27877 for ; Fri, 29 Sep 2000 21:13:44 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 03F6958C for ; Fri, 29 Sep 2000 19:13:44 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 219D0261; Fri, 29 Sep 2000 21:13:43 -0400 (EDT) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.188]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id SAA19735; Fri, 29 Sep 2000 18:13:41 -0700 (PDT) Message-ID: <39D53E71.AB0F9DC3@agilent.com> Date: Fri, 29 Sep 2000 18:14:26 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: "GUPTA,SOMESH (HP-Cupertino,ex1)" , IPS Reflector Subject: Re: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, What you describe is how current TCP stacks typically operate when it does not know anything about the protocol running on top of it. A customized TCP stack that was closely coupled with the iSCSI protocol running on top of it could provide the appropriate urgent pointer in each segment that is (re)transmitted without "coalescing" the urgent data. Likewise a customized TCP inbound path that was closely coupled with the iSCSI protocol running on top of it could pass the urgent information to the iSCSI layer without "coalescing" the urgent data. This could be done without any modifications or violations of current TCP specifications. -Matt "GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > Matt, > > Use of the urgent pointer requires very careful investigation. If you > look at the TCP RFC it mentions that that the urgent pointer > location is a connection-wide pointer, both on the receiver side > and the sender side. > > From the sender side, even if TCP emits a TCP segment for every > user send, you are not guaranteed the preservation of the urgent > pointer - e.g. > > 1. Let us say that a retransmission timeout occurs and a number > of segments have to be retransmitted. In this case, based on > (my interpretation) of RFC 793, only the last value of the > urgent pointer is meaningful. > > 2. Let us say that user data gets queued up due to flow control > issues. Again this will cause the last value of the urgent > pointer to take effect. > > the second part will have an impact in the slow start case. A > further analysis is required, but it seems that in cases where > packets are being dropped, the urgent pointer setting will not > be the most optimal (just about when it is needed the most). > The urgent pointer proposal may have a benefit when you are > loosing exactly one packet per window size (maybe even two > with SACK). > > On the receive side, again the same thing happens when multiple > segments with urg pointer are received. Since it is a connection- > wide pointer, only the last one should be provided to the user. > > If the data passes through "intermediate TCP devices" then again > the urgent pointer setting will be again be suboptimal. > > I would urge some TCP expert to jump in here. > > Somesh From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa14831; 2 Oct 2000 4:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id CAA23615 for ips-outgoing; Mon, 2 Oct 2000 02:57:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA23610 for ; Mon, 2 Oct 2000 02:57:51 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id IAA81742 for ; Mon, 2 Oct 2000 08:57:11 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id IAA134498 for ; Mon, 2 Oct 2000 08:57:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.00262B14 ; Mon, 2 Oct 2000 08:56:53 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 09:29:18 +0300 Subject: Re: Status summary on multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, Only a small correction. The draft out is the second draft. The first version (00.txt) was a multi-connection asymmetric solution without any connection allegiance for data. The current draft is the first having the symmetric flavor. Julo Matt Wakeley on 29/09/2000 06:31:21 Please respond to Matt Wakeley To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: Status summary on multiple connections John Hufferd/San Jose/IBM wrote: > I am troubled with the fact that I think many folks have already built > iSCSI Initiators and Targets that use just one connection per session. > > But what about the implementations that are currently in flight. A couple thoughts on worrying about "implementations in flight". First off, the posted version of the iSCSI spec is the first published version, and was certainly not complete. No one should be hard coding to that spec. Especially in light that David Black has decided that parts of it are to be ripped out (what if I had been implementing a solution that required multiple TCP connections or command reference numbers and then you take it away?). Second, all the current "implementations" are either (a) proprietary, mostly UDP based FCP tunneling implementations, or (b) demo projects to demonstrate feasibility. I don't think we have to worry about existing "in flight" implementations based on the first draft. After a few revs of the document, that will be a different story... -Matt From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id aa11710; 1 Oct 2000 9:44 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa10949; 1 Oct 2000 5:39 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id EAA00170 for ips-outgoing; Sun, 1 Oct 2000 04:30:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id EAA00159 for ; Sun, 1 Oct 2000 04:30:13 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id KAA223032 for ; Sun, 1 Oct 2000 10:29:27 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id KAA68332 for ; Sun, 1 Oct 2000 10:29:27 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696B.002EA241 ; Sun, 1 Oct 2000 10:29:21 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Sun, 1 Oct 2000 10:30:07 +0300 Subject: RE: VI (Was: Avoiding deadlock in iSCSI) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, No. As long as you can reestablish at least one TCP connection you should be fine. Security context must also be maintained. However it is still an open issue what to do SCSI wise. What I assumed is that target will do nothing and initiator will do a target management function (some form of abort I assume). Julo Charles Monia on 27/09/2000 22:15:57 Please respond to Charles Monia To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Wednesday, September 27, 2000 12:05 AM > To: ips@ece.cmu.edu > Subject: RE: VI (Was: Avoiding deadlock in iSCSI) > > > > > Charles, > > This goes to the hearth of problem. > To avoid this effect we propose (like FCP) unordered messages > to go always > through - > allow the initiator to decide which task management messages he deems > "urgent". > But if you have a single connection and the TCP window is > closed you are > left with only > one think to do - drop the connection (and that is exactly what FCP is > doing for similar > reasons!). > > Julo > Hi Julo: Does dropping the connection blow away the iSCSI session? If so, FCP does nothing like that. In FCP terms, only the command is discarded. The "session" established at FC login time remains intact. That is, other pending commands and initiator-specific device context (mode page settings, reservations and the like) are unaffected. Charles From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa15948; 2 Oct 2000 10:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA29238 for ips-outgoing; Mon, 2 Oct 2000 09:09:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA29229 for ; Mon, 2 Oct 2000 09:09:07 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id IAA28689; Mon, 2 Oct 2000 08:08:26 -0500 Message-ID: <39D888CA.592AC2C@stewart.chicago.il.us> Date: Mon, 02 Oct 2000 08:08:26 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com CC: dotis@sanlight.net, matt_wakeley@agilent.com, ips@ece.cmu.edu Subject: Re: Enough on TCP vs. FCTP, please References: <0F31E5C394DAD311B60C00E029101A070410102B@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David: Let me take a rain-check on this issue... I want to go poke around in a couple of different TCP implemenations and contrast that with Matt's draft.... After doing that I will get back to this thead on my opinion :) I think a study of: A) How current implemenations apply the urgent pointer, B) How this contrasts with Matt's draft, AND C) How all of (A) and (B) contrast with RFC793 must be considered... Now only after a careful assessment of all three will I chime in... My schedule is a bit tight until the end of the week when I will have more time to devote to this subject.. Thats all for now... and by the way Julian and Matt: 1) I have only commented to this WG on SCTP when I deemed it appropriate. Mainly because no one wants to go out and read the RFC/draft on it (it will be published later this week) to figure out what additional features it can apply. 2) I will continue to do so as I see fit.. Not that I have put as many comments as you seem to think I have ... Regards R Black_David@emc.com wrote: > > > End schemes to fix TCP; address the issues of TCP and SCTP. Do not stand > on > > desires for TCP over SCTP or that a pact was made to promote TCP. SCTP > > performance will be superior to TCP using either hardware or software. > SCTP > > is easier to accelerate in hardware than TCP and any resulting SCSI > standard > > will change substantially as a result of the features of SCTP. Even if > > there was a generic RDMA option, SCSI would look more like Firewire as a > > result. A debased version of RDMA has already been offered that avoids > > stepping on accepted options. > > I think this sort of debate really needs to stop. Matt's draft is a > reasonable > attempt to apply an existing TCP feature to address a shortcoming in use > of TCP for iSCSI. It is not perfect, as is the case for most engineering > tradeoffs and compromises. SCTP does have a number of improvements > over TCP (e.g., the whole session discussion is moot for SCTP), but TCP > has significantly more deployment experience and a great deal of work in > progress on hardware acceleration. > > I am restating the WG consensus that iSCSI needs to anticipate both > SCTP and TCP, with the assumption that TCP deployments will come > first. Given this, Matt's draft is within scope for the WG. If anyone > other > than Doug Otis disagrees, please say so and say why on the mailing list. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04129; 29 Sep 2000 17:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA19461 for ips-outgoing; Fri, 29 Sep 2000 16:40:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA19453 for ; Fri, 29 Sep 2000 16:39:56 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Fri, 29 Sep 2000 16:39:25 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101027@corpmx9.isus.emc.com> To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Fri, 29 Sep 2000 16:39:19 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk > Sorry for not being more explicit. The window can close as C2+D2 (but > mainly D2) are not consumed as they are processed after C1. > > The scenario holds and the blocking results from a "order reversal" of the data in the stream > caused by enabling both immediate data and R2T. In which case, this target should forbid immediate data if its buffer resources are that slim; I think we're in violent agreement on that. Putting my WG co-chair hat back on: The open question to the list is whether there's value in allowing some amount of immediate data (e.g., for targets that need fast startup on long latency connections, and are prepared to deploy the buffering required to make it work reliably), or whether we ought to follow FCP-2 and forbid immediate data, which will impose a round trip delay (command out, R2T back) before data starts to flow. I think I've seen a couple of comments in favor of this, but more discussion is in order. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04142; 29 Sep 2000 17:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA18459 for ips-outgoing; Fri, 29 Sep 2000 16:16:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA18450 for ; Fri, 29 Sep 2000 16:16:46 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id WAA19570 for ; Fri, 29 Sep 2000 22:16:14 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id WAA122526 for ; Fri, 29 Sep 2000 22:16:14 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C1256969.006F5579 ; Fri, 29 Sep 2000 22:16:03 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Fri, 29 Sep 2000 23:13:42 +0300 Subject: RE: iSCSI: Flow Control Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, Sorry for not being more explicit. The window can close as C2+D2 (but mainly D2) are not consumed as they are processed after C1. The scenario holds and the blocking results from a "order reversal" of the data in the stream caused by enabling both immediate data and R2T. Julo Black_David@emc.com on 29/09/2000 22:31:26 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: Flow Control With my WG co-chair hat off: > In your example C1, C2, D1, D2 will always work (as will any ordered combination). > The troublesome situation is C1, C2+D2 (immediate), C3 (window closed) > followed by an R2T for D1. This is why I think we should either explicitly > forbid immediate data if R2T is enabled or specify that this behavior may > get an initiator into trouble. I think we need to be careful about reinventing things that work in the SCSI world. Focus in a discussion like this really needs to be on how the presence of a IP-based network changes things. So, let's work backwards. The window closes because the initiator doesn't receive window updates from the target. Either the updates were lost, or the target didn't send them. If the updates are lost, the R2T will reopen the window, so the problem must be that the target didn't send the updates. If the target is temporarily overloaded, the window will reopen shortly. If it doesn't, then something is broken in the target and possibly the initiator because the same thing would go wrong on any transport. Between the technique already described that allows a SCSI target to set limits on the max size of immediate data and the target's responsibility for controlling resource allocation, a properly engineered target should not leave windows closed for extended periods of time. A target that permits transfer of more data than it can safely buffer is prone to breakage, and is likely to lose market share to targets that don't screw up in this fashion. Recalling that SCSI puts a target in complete control of its resource allocation, if immediate data is allowed, then we need to reuse the existing SCSI mechanism that allows a target to limit the amount of immediate data. Targets that overcommit their buffering resources are responsible for the consequences, and efforts should not be spent here to accommodate poor implementation decisions. We should definitely permit targets to follow Julian's approach of prohibiting immediate data, but I don't think that we should require it. The one thing that does change here is that the presence of the immediate data makes it easier to fill a TCP window, costing a round trip delay to get the next update -- this is an opportunity for implementers to apply some intelligent tuning of the sizes of advertised TCP windows to make this unlikely in practice. Also note that for the case in which C1 and C2 are ordered, the initiator has done something colossally stupid by using R2T for C1's data and sending C2's data in line. This is a case in which the best that can be expected is for things to work, and not a case for which performance ought to be optimized. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03058; 29 Sep 2000 10:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA03778 for ips-outgoing; Fri, 29 Sep 2000 09:48:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA03774 for ; Fri, 29 Sep 2000 09:48:05 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id JAA16310; Fri, 29 Sep 2000 09:47:34 -0400 Message-ID: <005401c02a19$26541ed0$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: , "Stephen Bailey" Subject: Re: New List: rdma@cisco.com: to discuss RDMA Date: Fri, 29 Sep 2000 09:28:23 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk David Black wrote: >And this illuminates the design tradeoff that may motivate RDMA. If >one only wants to accelerate one protocol (SCSI/FCP in the above >example) then having hardware understand its headers and doing >the DMA on that basis is a fairly obvious way to go - HBAs for both >parallel SCSI and Fibre Channel (SCSI/FCP) do this. RDMA may be >interesting if there are multiple protocols involved, and there are >engineering concerns that lead to not wanting to implement hardware >support for all of them. > >>From an iSCSI viewpoint, I don't see iSCSI by itself as being sufficient >to motivate a protocol-independent RDMA - an iSCSI HBA could understand >the iSCSI headers and interact with DMA in the same fashion as existing >HBAs. The task before those interested in RDMA is to identify a set >of protocols for which a common RDMA mechanism makes sense from >an engineering standpoint. I think this is a good summary of the situation. I agree with the above. Stephen Bailey wrote: >You have to ask the implementors (particularly the hardware >implementors), what sort of optional RDMA proposal they'd be happy >with. My answer is none. It's mandatory or not at all. > >The reason for using RDMA is to make the implementation of iSCSI >easier in hardware. If there are implementations which do not support >the RDMA protocol, then your hardware implementation will have to >support both the `easy path' (using RDMA) and the `hard path' (no >RDMA). If you have to implement the hard path anyway, there's no >point in implementing the easy path. As a hardware implementor, I tend to agree with the above: manditory or not at all. (Although I am still open to specific proposals that might argue to the contrary.) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa32707; 28 Sep 2000 17:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA09885 for ips-outgoing; Thu, 28 Sep 2000 17:00:32 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA09874 for ; Thu, 28 Sep 2000 17:00:26 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8SL0QU18961 for ; Thu, 28 Sep 2000 16:00:26 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA05065; Thu, 28 Sep 00 15:58:46 -0500 Message-Id: <10009282058.AA05065@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: New List: rdma@cisco.com: to discuss RDMA In-Reply-To: Message from csapuntz@cisco.com of "27 Sep 2000 10:16:29 PDT." Date: Thu, 28 Sep 2000 15:58:57 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > Does anybody on the list object to mandating an RDMA mechanism? I support this. I know you said `speak up if you object', but it's probably useful to see who really cares one way or the other. I apologize if you were trying to stuff the ballot box with all those abstentions and I was to clumsy to pick up on it :^) Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03880; 29 Sep 2000 15:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15215 for ips-outgoing; Fri, 29 Sep 2000 15:01:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15204 for ; Fri, 29 Sep 2000 15:00:59 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA15081; Fri, 29 Sep 2000 11:47:26 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA08221; Fri, 29 Sep 2000 11:45:13 -0700 From: "Douglas Otis" To: "Jim Williams" , , Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) Date: Fri, 29 Sep 2000 12:03:13 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal In-Reply-To: <007701c02a20$8050c170$cb08efd0@sachmo.giganet.com> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, There are practical reasons. Lower CPU overhead is substantial motivation possible with SCTP together with significant reductions in latency. You will not see much improvement using TCP. You may view the CRC polynomial as insignificant, but the present selection is robust. Certainly, you are not arguing an improved CRC is needed. An additional CRC is added when there is *NO* CRC protection within and between pieces of equipment and not because the CRC within Ethernet is not sufficient. If you were to use an adapter to encapsulate FDDI, or FC, then keeping the CRC compatible opens the door to native access. Changing the CRC would have little advantage and significant disadvantage. As you can cleanly add SCTP at this point in time, compared to attempting the same with TCP, customers are more likely to be receptive to these improvements should the NIC interface not change for TCP and yet new features are offered with SCTP. In America, innovation sells. SCTP can claim the TCP heritage together being compatible to VI and SCSI innovations while blowing the doors off. Doug > -----Original Message----- > From: Jim Williams [mailto:jimw@giganet.com] > Sent: Friday, September 29, 2000 7:21 AM > To: Douglas Otis; rdma@cisco.com; ips@ece.cmu.edu > Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in > iSCSI)) > > > > -----Original Message----- > From: Douglas Otis > To: Jim Williams ; ips@ece.cmu.edu > Date: Tuesday, September 26, 2000 2:52 PM > Subject: RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) > > > > Regarding: > ftp://coke.giganet.com/rdma-tcp.txt > > >Jim, > > > >In summary, you expect kernel modifications of TCP API to allow frame > >alignment, a hardware based 32-bit CRC that uses a different polynomial > than > >FCP and a direct copy method at the NIC. > > I do not expect kernel modifications. The motivation for RDMA is to allow > a NIC to directly place received data in the target buffer without > kernel intervention or host data copy. One can certainly argue how > and whether this should be done and whether there should be a protocol > independent mechanism for RMDA. However if the received data must > be processed by the kernel, it might as well be processed in order > using the conventional API without any changes. One could argue > that some small performance gains could be had by doing kernel > modifications, but my guess that this is way insufficient to justify > modifying the API. > > >Both FDDI and FC have selected a CRC polynomial ignored > >by your proposal. > > Ethernet, FDDI, and FC all use the same CRC polynomial. When > encapsulating > a CRC within an ethernet frame, it really doesn't make all that much > difference what polynomial is used, the only important requirement is > that it be relatively prime to the ethernet CRC polynomial so that > coverage is additive, not redundant. > > >Recommending, suggesting, or offering a feature of frame > >alignment and out of sequence processing rend the TCP API. There are > better > >alternatives to your proposal in SCTP, so I fail to see the > justification. > > I agree that there are somewhat easier ways to accomplish the objectives > with SCTP, but my goal was to show that it can be done with TCP. The > justification is that there are customers that want to buy TCP. > In the short term there are some fundamental infrastructure issues > that favor TCP. Things like the availability of well tested reference > implementations, availibility of test, design, and QA tools that > understand TCP, availability of more engineers and other professionals > that understand TCP. Ultimately in the long term, it may make sense to > support both TCP and SCTP and let the market place decide which > will prevail. > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa01294; 29 Sep 2000 1:20 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA22247 for ips-outgoing; Fri, 29 Sep 2000 00:28:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA22240 for ; Fri, 29 Sep 2000 00:28:26 -0400 (EDT) Received: from core.rose.hp.com (core.rose.hp.com [15.43.208.100]) by palrel3.hp.com (Postfix) with ESMTP id 0444F465 for ; Thu, 28 Sep 2000 21:28:26 -0700 (PDT) Received: from agilent.com (wak7854259a.rose.hp.com [15.26.30.186]) by core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id VAA05408 for ; Thu, 28 Sep 2000 21:28:56 -0700 (PDT) Message-ID: <39D41A59.524C1BAB@agilent.com> Date: Thu, 28 Sep 2000 21:28:09 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: New List: rdma@cisco.com: to discuss RDMA References: <10009282122.AA05090@candide.cs.uchicago.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Stephen Bailey wrote: > > However, I would support structuring the spec so that an RDMA > > transport mechanism could be used underneath (I guess that's > > motherhood). > > Not necessarily. > > You have to ask the implementors (particularly the hardware > implementors), what sort of optional RDMA proposal they'd be happy > with. My answer is none. It's mandatory or not at all. great! get rid of it! > > The reason for using RDMA is to make the implementation of iSCSI > easier in hardware. If there are implementations which do not support > the RDMA protocol, then your hardware implementation will have to > support both the `easy path' (using RDMA) and the `hard path' (no > RDMA). If you have to implement the hard path anyway, there's no > point in implementing the easy path. > > The argument that you could make the hard path infrequent and > implement it in software doesn't wash in this case. It just takes one > implementation that doesn't do RDMA to slow your system to a crawl, > and the competitor who only implemented the non-RDMA path makes you > look like a fool. > > Fundamentally, RDMA has to be either adopted or punted. Of course, > I'm happy to have somebody prove this statement wrong. Ok, iSCSI does not need an RDMA mechanism because (from David Black's message): > > Actually, RDMA is not needed in FCP because all protocol chips > > implemented perform a real peer-to-peer DMA straight to the > > data areas specified by the user's interaction with the operating > > systems allocation algorithms. The combination of the FCP/SCSI > > pointer structure, task tagging, and the FC relative offset perform the > > function you would otherwise have to use RDMA to accomplish. > > And this illuminates the design tradeoff that may motivate RDMA. If > one only wants to accelerate one protocol (SCSI/FCP in the above > example) then having hardware understand its headers and doing > the DMA on that basis is a fairly obvious way to go - HBAs for both > parallel SCSI and Fibre Channel (SCSI/FCP) do this. The issue is that TCP does not provide message framing such that the out of order messages could be found and processed. So, instead of focusing on "RDMA", let's focus on "iSCSI message framing". -Matt > > > Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03819; 29 Sep 2000 15:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA13766 for ips-outgoing; Fri, 29 Sep 2000 14:26:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from saturn.sun.com (saturn.Sun.COM [192.9.25.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA13749 for ; Fri, 29 Sep 2000 14:25:53 -0400 (EDT) Received: from ebaymail1.EBay.Sun.COM ([129.150.111.99]) by saturn.sun.com (8.9.3+Sun/8.9.3) with ESMTP id LAA27579 for ; Fri, 29 Sep 2000 11:25:53 -0700 (PDT) Received: from ha10nwk.EBay.Sun.COM (phys-ha10nwka.EBay.Sun.COM [129.150.144.210]) by ebaymail1.EBay.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with ESMTP id LAA29850 for ; Fri, 29 Sep 2000 11:25:52 -0700 (PDT) Received: from jetsun by ha10nwk.EBay.Sun.COM (8.8.8+Sun/SMI-SVR4) id LAA13471; Fri, 29 Sep 2000 11:25:52 -0700 (PDT) Message-Id: <200009291825.LAA13471@ha10nwk.EBay.Sun.COM> Date: Fri, 29 Sep 2000 11:25:52 -0700 (PDT) From: David Robinson Reply-To: David Robinson Subject: RE: Status summary on multiple connections To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 8YZjIiMf+EKtNLBLC1hCAQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > I am left with the following impression as to what was indicated here: > - In general, command ordering is not relevant > - If the initiator filesystem detects an ordering dependency, it will wait > until outstanding commands are complete before issuing the dependant > command. > > This may be a reasonable means of operation for the disk world. It is > woefully inadequate for the tape world, as follows: > - In general, command ordering is crucial - out of order command processing > will lead to data corruption. > - This would require the initiator backup application to block on completion > of every single write command of a backup operation before issuing the next > command. > > If this blocking were performed, both the throughput and capacity of a tape > device/media would be negatively impacted by an order of magnitude or more. > This would occur even assuming an instantaneous transport. I am hearing different stories on the issue of ordering. One side is pushing hard for techniques that will allow out of order execution using various RDMA techniques. This clearly states for a certain class of devices (e.g. tapes) ordering is crucial. I thought this problem was already solved at the SCSI layer through the use of ordered commands which in general are not used for disks but always used for tapes? Since FC will reorder this has to be a solved problem. Would not an initiator talking to a tape target simply set the ordering flag? Lastly, for a TCP based connection ordering can easily be made a non-issue, simply don't try to process segments out of order. I will defer to a transport expert, but I believe processing TCP segments by an application out of order might cause problems. In particular since the out of order segment is not ACKed until after the missing segments arrive, they can be retransmitted multiple times. SACK helps this but does not guarentee that segments will not be retransmitted. So to process out of order segments the application must maintain a list of which segments have been processed as well, yuck! -David From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa04899; 29 Sep 2000 22:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA27988 for ips-outgoing; Fri, 29 Sep 2000 21:16:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA27984 for ; Fri, 29 Sep 2000 21:16:34 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id D698226A for ; Fri, 29 Sep 2000 19:16:33 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id EAFD5266; Fri, 29 Sep 2000 21:16:32 -0400 (EDT) Received: from agilent.com (ros54259wak.rose.agilent.com [156.140.234.188]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id SAA20045; Fri, 29 Sep 2000 18:16:31 -0700 (PDT) Message-ID: <39D53F1C.74B5F879@agilent.com> Date: Fri, 29 Sep 2000 18:17:16 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: "GUPTA,SOMESH (HP-Cupertino,ex1)" , IPS Reflector Subject: Re: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, What you describe is how current TCP stacks typically operate when it does not know anything about the protocol running on top of it. A customized TCP stack that was closely coupled with the iSCSI protocol running on top of it could provide the appropriate urgent pointer in each segment that is (re)transmitted without "coalescing" the urgent data. Likewise a customized TCP inbound path that was closely coupled with the iSCSI protocol running on top of it could pass the urgent information to the iSCSI layer without "coalescing" the urgent data. This could be done without any modifications or violations of current TCP specifications. -Matt "GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > Matt, > > Use of the urgent pointer requires very careful investigation. If you > look at the TCP RFC it mentions that that the urgent pointer > location is a connection-wide pointer, both on the receiver side > and the sender side. > > >From the sender side, even if TCP emits a TCP segment for every > user send, you are not guaranteed the preservation of the urgent > pointer - e.g. > > 1. Let us say that a retransmission timeout occurs and a number > of segments have to be retransmitted. In this case, based on > (my interpretation) of RFC 793, only the last value of the > urgent pointer is meaningful. > > 2. Let us say that user data gets queued up due to flow control > issues. Again this will cause the last value of the urgent > pointer to take effect. > > the second part will have an impact in the slow start case. A > further analysis is required, but it seems that in cases where > packets are being dropped, the urgent pointer setting will not > be the most optimal (just about when it is needed the most). > The urgent pointer proposal may have a benefit when you are > loosing exactly one packet per window size (maybe even two > with SACK). > > On the receive side, again the same thing happens when multiple > segments with urg pointer are received. Since it is a connection- > wide pointer, only the last one should be provided to the user. > > If the data passes through "intermediate TCP devices" then again > the urgent pointer setting will be again be suboptimal. > > I would urge some TCP expert to jump in here. > > Somesh From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa00132; 28 Sep 2000 18:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA11573 for ips-outgoing; Thu, 28 Sep 2000 17:38:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA11566 for ; Thu, 28 Sep 2000 17:38:10 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e8SLcAU20295 for ; Thu, 28 Sep 2000 16:38:10 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA05106; Thu, 28 Sep 00 16:36:29 -0500 Message-Id: <10009282136.AA05106@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI)) In-Reply-To: Message from Stephen Byan of "Thu, 28 Sep 2000 08:25:47 PDT." <8133266FE373D11190CD00805FA768BF055BD1EB@shrcmsg1.tdh.qntm.com> Date: Thu, 28 Sep 2000 16:36:40 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > I believe the point in factoring out the RDMA mechanism from iSCSI is to > make the same hardware assists available to other application > protocols. I agree. The motivation is primarily economic. If you implement a general solution to 1) data steering, 2) congestion avoidance, 3) security, you own the edge of all networking. If you implement these for storage, you only own storage. Given that it seems such an easy step to make the hardware slightly more general and own the world, I'm stumped on why not. OK, I admit I know the argument why not, because people think the more specific solution more tractable. I disagree. The other reason why not is some people really aren't sold on the necessity for hardware implementation at all. BTW, I made this case at CERN recently: http://hsi.web.cern.ch/HSI/HNF-Europe/Workshop%202000/Presentations/Com_Stor_Net.PDF Julo wasn't buying though. Steph From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa02893; 29 Sep 2000 9:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA01811 for ips-outgoing; Fri, 29 Sep 2000 08:50:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from stewart.chicago.il.us (IDENT:root@3ff83343.dsl.flashcom.net [63.248.51.67]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA01798 for ; Fri, 29 Sep 2000 08:50:39 -0400 (EDT) Received: from stewart.chicago.il.us (IDENT:randall@stewart.chicago.il.us [10.1.1.1]) by stewart.chicago.il.us (8.9.3/8.8.7) with ESMTP id GAA19413; Fri, 29 Sep 2000 06:44:02 -0500 Message-ID: <39D48080.6D4ABC4D@stewart.chicago.il.us> Date: Fri, 29 Sep 2000 06:44:00 -0500 From: "Randall R. Stewart" X-Mailer: Mozilla 4.61 [en] (X11; U; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: "GUPTA,SOMESH (HP-Cupertino,ex1)" CC: csapuntz@cisco.com, Jim Williams , ips@ece.cmu.edu Subject: Re: New List: rdma@cisco.com: to discuss RDMA References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk "GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > > Mandating an RDMA mechanism would delay product delivery > for sure. I am not familiar enough with the IETF process to > understand how it might impact the specification process since > it requires a change to TCP. > It seems to me that the RDMA option needs to go to the transport area working group. This is where TCP extensions seem to normally float around... Costa, why don't you direct your draft towards that working group and get comments from there? R -- Randall R. Stewart randall@stewart.chicago.il.us or rrs@cisco.com 815-342-5222 (cell) 815-477-2127 (work) From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa07261; 30 Sep 2000 11:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA11373 for ips-outgoing; Sat, 30 Sep 2000 10:41:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from ertpg14e1.nortelnetworks.com (ertpg14e1.nortelnetworks.com [47.234.0.35]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA11368 for ; Sat, 30 Sep 2000 10:41:22 -0400 (EDT) Received: from zcard00m.ca.nortel.com by ertpg14e1.nortelnetworks.com; Sat, 30 Sep 2000 10:40:58 -0400 Received: from zbl6c002.corpeast.baynetworks.com ([132.245.205.52]) by zcard00m.ca.nortel.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id TZZ7YM55; Sat, 30 Sep 2000 10:40:55 -0400 Received: from ftravost.nortelnetworks.com (bne-cnc-204.corpeast.baynetworks.com [132.245.134.204]) by zbl6c002.corpeast.baynetworks.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2652.39) id TW4RJ5SB; Sat, 30 Sep 2000 10:40:53 -0400 Message-Id: <4.3.1.2.20000930100350.01f05910@zbl6c002.corpeast.baynetworks.com> X-Sender: travos@zbl6c002.corpeast.baynetworks.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.1 Date: Sat, 30 Sep 2000 10:30:17 -0400 To: Matt Wakeley , "GUPTA,SOMESH (HP-Cupertino,ex1)" , IPS Reflector From: "Franco Travostino" Subject: Re: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt In-Reply-To: <39D53E71.AB0F9DC3@agilent.com> References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_120893553==_.ALT" Sender: owner-ips@ece.cmu.edu Precedence: bulk --=====================_120893553==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed IMO, 'Customized' and 'closely coupled' reads also as a protocol layering violation. This raises concerns of architectural and practical nature. For one, low-cost commodity TCP silicons will come in one shape, and they better be agnostic to whatever the client(s) are. The implementation space has historically been fragmented over urgent pointer semantics, I'd hope that we do not raise this entropy with new iSCSI self-serving flavors. Alternately, it there is a contribution on urgent pointer matters that is of general applicability, it could be submitted as a general purpose amendment to TCP for framing (with the same general applicability statement that was applied to the RDMA thread). Other communities (say http 1.x) may add their support. -franco At 09:14 PM 9/29/00, Matt Wakeley wrote: >Somesh, > >What you describe is how current TCP stacks typically operate when it does not >know anything about the protocol running on top of it. A customized TCP stack >that was closely coupled with the iSCSI protocol running on top of it could >provide the appropriate urgent pointer in each segment that is (re)transmitted >without "coalescing" the urgent data. Likewise a customized TCP inbound path >that was closely coupled with the iSCSI protocol running on top of it could >pass the urgent information to the iSCSI layer without "coalescing" the urgent >data. This could be done without any modifications or violations of current >TCP specifications. > >-Matt > >"GUPTA,SOMESH (HP-Cupertino,ex1)" wrote: > > > Matt, > > > > Use of the urgent pointer requires very careful investigation. If you > > look at the TCP RFC it mentions that that the urgent pointer > > location is a connection-wide pointer, both on the receiver side > > and the sender side. > > > > From the sender side, even if TCP emits a TCP segment for every > > user send, you are not guaranteed the preservation of the urgent > > pointer - e.g. > > > > 1. Let us say that a retransmission timeout occurs and a number > > of segments have to be retransmitted. In this case, based on > > (my interpretation) of RFC 793, only the last value of the > > urgent pointer is meaningful. > > > > 2. Let us say that user data gets queued up due to flow control > > issues. Again this will cause the last value of the urgent > > pointer to take effect. > > > > the second part will have an impact in the slow start case. A > > further analysis is required, but it seems that in cases where > > packets are being dropped, the urgent pointer setting will not > > be the most optimal (just about when it is needed the most). > > The urgent pointer proposal may have a benefit when you are > > loosing exactly one packet per window size (maybe even two > > with SACK). > > > > On the receive side, again the same thing happens when multiple > > segments with urg pointer are received. Since it is a connection- > > wide pointer, only the last one should be provided to the user. > > > > If the data passes through "intermediate TCP devices" then again > > the urgent pointer setting will be again be suboptimal. > > > > I would urge some TCP expert to jump in here. > > > > Somesh --=====================_120893553==_.ALT Content-Type: text/html; charset="us-ascii"
IMO, 'Customized' and 'closely coupled' reads also as a protocol layering violation. This raises concerns of architectural and practical nature. For one, low-cost commodity TCP silicons will come in one shape, and they better be agnostic to whatever the client(s) are. The implementation space has historically been fragmented over urgent pointer semantics, I'd hope that we do not raise this entropy with new iSCSI self-serving flavors.

Alternately, it there is a contribution on urgent pointer matters that is of general applicability, it could be submitted as a general purpose amendment to TCP for framing (with the same general applicability statement that was applied to the RDMA thread). Other communities (say http 1.x) may add their support.

-franco


At 09:14 PM 9/29/00, Matt Wakeley wrote:
Somesh,

What you describe is how current TCP stacks typically operate when it does not
know anything about the protocol running on top of it.  A customized TCP stack
that was closely coupled with the iSCSI protocol running on top of it could
provide the appropriate urgent pointer in each segment that is (re)transmitted
without "coalescing" the urgent data. Likewise a customized TCP inbound path
that was closely coupled with the iSCSI protocol running on top of it could
pass the urgent information to the iSCSI layer without "coalescing" the urgent
data. This could be done without any modifications or violations of current
TCP specifications.

-Matt

"GUPTA,SOMESH (HP-Cupertino,ex1)" wrote:

> Matt,
>
> Use of the urgent pointer requires very careful investigation. If you
> look at the TCP RFC it mentions that that the urgent pointer
> location is a connection-wide pointer, both on the receiver side
> and the sender side.
>
> From the sender side, even if TCP emits a TCP segment for every
> user send, you are not guaranteed the preservation of the urgent
> pointer - e.g.
>
> 1. Let us say that a retransmission timeout occurs and a number
>    of segments have to be retransmitted. In this case, based on
>    (my interpretation) of RFC 793, only the last value of the
>    urgent pointer is meaningful.
>
> 2. Let us say that user data gets queued up due to flow control
>    issues. Again this will cause the last value of the urgent
>    pointer to take effect.
>
> the second part will have an impact in the slow start case. A
> further analysis is required, but it seems that in cases where
> packets are being dropped, the urgent pointer setting will not
> be the most optimal (just about when it is needed the most).
> The urgent pointer proposal may have a benefit when you are
> loosing exactly one packet per window size (maybe even two
> with SACK).
>
> On the receive side, again the same thing happens when multiple
> segments with urg pointer are received. Since it is a connection-
> wide pointer, only the last one should be provided to the user.
>
> If the data passes through "intermediate TCP devices" then again
> the urgent pointer setting will be again be suboptimal.
>
> I would urge some TCP expert to jump in here.
>
> Somesh
--=====================_120893553==_.ALT-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa03274; 29 Sep 2000 12:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA06582 for ips-outgoing; Fri, 29 Sep 2000 11:08:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA06574 for ; Fri, 29 Sep 2000 11:08:50 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id KAA57070 for ; Fri, 29 Sep 2000 10:56:12 -0400 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e8TF7vf37974; Fri, 29 Sep 2000 09:07:57 -0600 Importance: Normal Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Fri, 29 Sep 2000 08:07:54 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 09/29/2000 08:07:57 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, Yet another method for determining the mapping of LUs to LUNs for given initiator? You already have vendor-specific ways (in FC that will surely get ported to iSCSI), the T10 standardized way with AccessID enrollment or with a TransportID (for iSCSI, which is still TBD), and the login authentication process (however that ends up getting defined). Also, your comment implies that the URL is sent to the target in a mode similar to an http-type protocol. Where is this protocol defined in the draft? Is it part of the Text message in the login authentication? Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-29-2000 06:14:44 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] I would add that we wanted the path to be an additional identifier that the target could use to determine what collection of LUs to present to the initiator. Julo csapuntz@csapuntz-u1.cisco.com on 23/09/2000 05:00:33 Please respond to csapuntz@csapuntz-u1.cisco.com To: "IP Storage" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Just for clarification... I was not proposing to add the extended URL scheme for the transport spec. It isn't necessary. In this thread, Doug makes an excellent point about LDAP being a superior mechanism for describing how to connect to the storage. Directory services such as LDAP, as Doug has pointed out, will be critical to managing large quantities of storage. A host can ask such a directory service for a list of storage devices it should mount and how to connect to those storage devices. The query against the directory server that returns this information can be based on machine ID, user ID, operating system ID, or even the owner's birthday. One of the things discovery will end up doing, no doubt, is defining LDAP schemas that describe how to connect to storage (i.e. use SCTP or TCP, what port, what target name, what LUN, what WWN, how to authenticate, etc.). However, there is one place where the transport protocol has to define a name: third party commands. There needs to be some kind of global name which the initiator can pass to the target. The name must be distillable into a string. The target must understand the name and be able to use the information to establish a connection to another target. One could say that the string that is passed is not specified by the standard but instead specified by some management software. I think this will lead to poor interoperability. The SCSI URL-type name is the current proposal for target name. Why is SCSI target name made up of a hostname + a path? Why is the hostname + path passed on connection setup? There are two reasons. I think NAT and IPv6 makes passing hostnames rather than addresses in protocols more desirable. Hostnames can be re-resolved as you cross addressing boundaries. The path is there so that the name can support multiple targets behind a single IP address without having to add entries to the DNS server. At Cisco, for example, I have no control over the local DNS servers and cannot add DNS entries for the ATAPI DVD and floppy in my computer. _Costa From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16533; 2 Oct 2000 13:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA07798 for ips-outgoing; Mon, 2 Oct 2000 12:42:23 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA07782 for ; Mon, 2 Oct 2000 12:42:12 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id MAA82112 for ; Mon, 2 Oct 2000 12:29:21 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e92Gg1d130820 for ; Mon, 2 Oct 2000 10:42:01 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: The third alternative To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 2 Oct 2000 09:39:01 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/02/2000 10:42:01 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk OK folks, I think we have agreement between Julian and David Black (hope,hope, hope). David, near the end of his note said "...Provide for is fine, as long as they're not required." Julian, I think this is all your were after anyway. So are we all back together now? . . . John L. Hufferd Black_David@emc.com@ece.cmu.edu on 10/02/2000 07:02:31 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: The third alternative > The only argument for SCSI wedge drivers where that they EXIST ALREADY. > Pretty weak argument for those building new equipment and for > interoperability. This severely understates the case. Not only do wedge drivers exist already, but they do a number of things that an iSCSI standard will never do. Wedge drivers tend to contain logic specific to the device that they're providing the wedge for - that's inappropriate to standardize, either in IETF or T10. In addition, there are implementation advantages to building wedge drivers above the SCSI level - one doesn't have to spread SCSI connection state across a failure boundary. > I think that if we keep ourselves honest we have to either: > > - provide for multiple connections at the iSCSI level as it is transport problem > that other TCP applications are not compelled to handle (I hear already BUT SCTP > handles it!) and hope that one day the session concept will drift into pure transport Provide for is fine, as long as they're not required. I would expect to see significant deployment of systems that use single TCP connection per session and handle multiple sessions in wedge drivers. Arguments like the one Bob has made is one of the reasons for multiple connections/session to remain OPTIONAL. > - go to T10 and ask the to standardize wedge drivers! Just say no! Exactly what would you want T10 to standardize? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa05055; 29 Sep 2000 22:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA28824 for ips-outgoing; Fri, 29 Sep 2000 21:52:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA28820 for ; Fri, 29 Sep 2000 21:52:43 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id SAA15263; Fri, 29 Sep 2000 18:39:19 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id SAA09513; Fri, 29 Sep 2000 18:37:06 -0700 From: "Douglas Otis" To: , , Subject: RE: Enough on TCP vs. SCTP, please Date: Fri, 29 Sep 2000 18:55:09 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: <0F31E5C394DAD311B60C00E029101A070410102D@corpmx9.isus.emc.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, I understand how you could hold that view. To implement such a definition, you would need to make changes to the TCP API as well as to the TCP stack to allow messages to be marked. There would need to be additional sorting algorithms added to search for the first byte of a message. It would also be likely a one byte error would creep in with machines that do not point to byte locations not aligned. To solve that problem, perhaps all messages should be frame aligned. That could be allowed if redefining the urgent pointer is possible. But hey, you would not need the urgent pointer then. Sorry, but when you must re-engineer TCP to work with a SCSI specification, you are no longer using TCP. That is specifically why I think other means be considered. If you wish to see reasonable performance revise TCP? Most should expect without such tweaks on TCP, it will not perform well enough to justify use with SCSI whether being clever to avoid mention or not. Such liberties in changing TCP abound and yet Matt said he was sick of hearing about SCTP. Managing buffers independently is yet another example. I personally see danger in creating unique versions of TCP to suit SCSI whereas drive and controller manufactures may view it as their domain. Perhaps now would be a good time to seek outside consensus on such modifications before re-inventing TCP. If it is true that TCP is up for grabs, then I will be quiet as I would be wrong about this view. Doug > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Friday, September 29, 2000 5:19 PM > To: dotis@sanlight.net; Black_David@emc.com; matt_wakeley@agilent.com; > ips@ece.cmu.edu > Subject: RE: Enough on TCP vs. SCTP, please > > > draft-wakeley-iscsi-msgbndry-00.txt says: > > The proposal is pretty simple. The first byte of every iSCSI > message is marked "urgent" - the TCP urgent pointer will point to > the first byte of the iSCSI message in the TCP segment. > > Douglas Otis says: > > > It is not reasonable to redefine the urgent pointer. > > Your WG co-chair says: > > I can't find anything in the draft that redefines the urgent > pointer. TCP deliberately does not define exactly what the > urgent pointer points to. For example, I wouldn't expect > iSCSI and telnet to use the urgent pointer in the same way. > > In order to continue this argument that an unacceptable change > is being made to TCP, it is necessary to cite the text in RFC 793 > (or some other RFC that defines TCP) that would have to be changed > if the approach in the draft referenced above were adopted. > > NOTE: this is *not* a statement of consensus that the approach > in the draft should be adopted. This is only a procedural statement > that consideration of that approach is within scope of the WG. > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Mon Oct 02 18:29:49 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16199; 2 Oct 2000 12:04 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA03584 for ips-outgoing; Mon, 2 Oct 2000 11:00:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mail.giganet.com (IDENT:root@mail.giganet.com [208.239.8.250]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA03575 for ; Mon, 2 Oct 2000 11:00:41 -0400 (EDT) Received: from sachmo (gn203.giganet.com [208.239.8.203]) by mail.giganet.com (8.8.7/8.8.7) with SMTP id KAA15712; Mon, 2 Oct 2000 10:47:51 -0400 Message-ID: <004801c02c7d$0198f730$cb08efd0@sachmo.giganet.com> From: "Jim Williams" To: "Matt Wakeley" , "GUPTA,SOMESH (HP-Cupertino,ex1)" , "IPS Reflector" , "Franco Travostino" Subject: Re: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt Date: Mon, 2 Oct 2000 10:28:03 -0400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0043_01C02C5B.72648410" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.5 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk This is a multi-part message in MIME format. ------=_NextPart_000_0043_01C02C5B.72648410 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable -----Original Message----- From: Franco Travostino To: Matt Wakeley ; GUPTA,SOMESH = (HP-Cupertino,ex1) ; IPS Reflector = Date: Saturday, September 30, 2000 12:25 PM Subject: Re: ISCSI: draft-wakeley-iscsi-msgbndry-00.txt =20 =20 =20 IMO, 'Customized' and 'closely coupled' reads also as a protocol = layering violation. This raises concerns of architectural and practical = nature. For one, low-cost commodity TCP silicons will come in one shape, = and they better be agnostic to whatever the client(s) are. I would disagree with the above. The actual TCP protocol algorithms are = a small part of the challenge of building an accelerated NIC. Much more important are placing data in = the correct destination buffer so that the host need not do any copies, careful design of = interrupt strategy so that host OS is interrupted only when absolutely necessary, and efficient control = communications between NIC and host so as not to cause the NIC or the host to stall unnecessarily = waiting for IO bus=20 transaction completion. In order to achieve this, the NIC must = understand it detail the client protocol using TCP, and must to some extent merge the = protocol layers. I would argue that merging protocol layers in the implementation is not = a "layering violation" provided that the layered protocol definition as viewed from the wire is = not violated and interoperability is not broken with layered implementations. The implementation space has historically been fragmented over = urgent pointer semantics, I'd hope that we do not raise this entropy = with new iSCSI self-serving flavors. =20 Alternately, it there is a contribution on urgent pointer matters = that is of general applicability, it could be submitted as a general = purpose amendment to TCP for framing (with the same general = applicability statement that was applied to the RDMA thread). Other = communities (say http 1.x) may add their support. The counter argument is that the meaning of urgent data is iSCSI = specific and not general to TCP, and that each TCP client protocol is allowed to define = urgent data to its own liking. To me the operative question (which I won't = presume to=20 answer) is whether Matt's proposal is broken by any current = implementations or could be broken by any reasonable implementations which are in = compliance with the relevant TCP/IP RFCs. ------=_NextPart_000_0043_01C02C5B.72648410 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
 
-----Original = Message-----
From:=20 Franco Travostino <travos@nortelnetworks.com&g= t;
To:=20 Matt Wakeley <matt_wakeley@agilent.com>= ;=20 GUPTA,SOMESH (HP-Cupertino,ex1) <somesh_gupta@am.exch.hp.com>;=20 IPS Reflector <ips@ece.cmu.edu>
Date:=20 Saturday, September 30, 2000 12:25 PM
Subject: Re: = ISCSI:=20 draft-wakeley-iscsi-msgbndry-00.txt


IMO, 'Customized' and 'closely coupled' reads also as a = protocol=20 layering violation. This raises concerns of architectural and = practical=20 nature. For one, low-cost commodity TCP silicons will come in one = shape, and=20 they better be agnostic to whatever the client(s) = are.
I would = disagree with the=20 above.  The actual TCP protocol algorithms are a small part of the=20 challenge
of building=20 an accelerated = NIC.  Much=20 more important are placing data in the correct = destination
buffer so that the = host need not do any copies, careful = design of=20 interrupt strategy so that host
OS is interrupted only when=20 absolutely necessary, = and=20 efficient control communications between NIC
and host so as not to cause the NIC or the host to stall=20 unnecessarily waiting for IO bus
transaction completion.  In order to = achieve this,=20 the NIC must understand it detail
the client protocol using TCP, and must to = some extent=20 merge the protocol layers.
 
I would argue that merging protocol layers in = the=20 implementation is not a "layering violation"
provided that the layered=20 protocol definition as viewed from the wire is not violated
and interoperability is not broken with = layered=20 implementations.
 
The implementation space has historically been fragmented = over urgent=20 pointer semantics, I'd hope that we do not raise this entropy with = new iSCSI=20 self-serving flavors.

Alternately, it there is a contribution = on=20 urgent pointer matters that is of general applicability, it could be = submitted as a general purpose amendment to TCP for framing (with = the same=20 general applicability statement that was applied to the RDMA = thread). Other=20 communities (say http 1.x) may add their = support.
The counter argument is that the meaning of urgent = data is=20 iSCSI specific and not
general to TCP, and that each TCP client protocol is = allowed=20 to define urgent
data to its own liking.  To me the operative = question=20 (which I won't presume to
answer) is whether Matt's proposal is broken by any = current=20 implementations or
could be broken by any reasonable implementations = which are in=20 compliance with
the relevant TCP/IP = RFCs.

 
 
------=_NextPart_000_0043_01C02C5B.72648410-- From owner-ips@ECE.cmu.edu Mon Oct 02 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16841; 2 Oct 2000 14:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10421 for ips-outgoing; Mon, 2 Oct 2000 13:45:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10409 for ; Mon, 2 Oct 2000 13:45:04 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id AA8162EC; Mon, 2 Oct 2000 11:45:01 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 992131EA; Mon, 2 Oct 2000 13:45:00 -0400 (EDT) Received: from agilent.com (cos1nai254106.cs.itc.hp.com [15.3.254.106]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA25182; Mon, 2 Oct 2000 10:44:55 -0700 (PDT) Message-ID: <39D89BA1.F9627E77@agilent.com> Date: Mon, 02 Oct 2000 07:28:49 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: julian_satran@il.ibm.com Cc: ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I think the layering is being blurred. Remember, whether there is RDMA or not, TCP is the layer doing the ACKing, *NOT* iSCSI. And whether or not there is RDMA or not, TCP cannot ACK TCP segments that arrive out of order until the missing ones arrive (it can SACK them, but it can't ACK them - and SACK is only informative). iSCSI cannot deliver commands out of order to the SCSI layer. So even if the command is RDMA'd into a buffer, it cannot be delivered to SCSI until the previous commands are received. -Matt julian_satran@il.ibm.com wrote: > David, > > The picture I had in mind is somewhat simpler. > > I assume that on any given transport connection, with or without RDMA, > iSCSI > will get from TCP and hand to SCSI things that are in order (just to keep > layering iSCSI over TCP simple and clean). No retransmissions involved in > iSCSI here. > > In case of several connections iSCSI might get things out of order (no ack > for them) > and may decide to hand them or not to hand them to execution. > iSCSI will not require retransmission as we assume that the out of order > thing is > a temporary artifact of the underlying network. Recovery if needed here is > done > only when a connection is blown-away and then all outstanding commands > that where shipped on that connection are restarted (resent). > > And again we conceived the window only to limit the number of commands > iSCSI has > to keep to get things in order but it can as well serve as a flow-control > mechanism for > the target as the initiator is unconcerned of what is being done with the > commands within the window (e.g., a R2T from a command within the window - > i.e., not acked yet - is a > legal event). > > Julo > > David Robinson on 30/09/2000 01:35:11 > > Please respond to David Robinson > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: Status summary on multiple connections > > There are two types of out of order processing we need to > consider: > 1) Commands that have been received and ACK'd by the transport > 2) Commands that have been received and not ACK'd > > In the first case this is purely a SCSI issue on if the commands > are ordered and how the target processes them. As far as the > transport is concerned the packets are "delivered". In the second > case the transport is still required to track the possible > retransmissions and process the transport ACK for all of > the data when the missing segment arrives. It will overly > complicate the iSCSI layer to successfully track which parts > of the sequence space it has processed, defend against > retransmissions, and handle packets that have more than one > command, and retransmissons that may be only part of a command. > > This is all doable in an implementation (a bit messy at times) > but to support this as an explicit feature we need to start > adding in an iSCSI layer sequence space. > > I propose that we leave the processing of transport > level un-ACK'd commands and data as an implementation detail > and not make it an explicit iSCSI feature. Dropped or reordered > TCP segments are rare enough in high performance environments > that this is really a non-issue. > > -David > > julian_satran@il.ibm.com wrote: > > > > David, > > > > I think that RDMA out-of-order processing is important only in order not > to > > have data > > piling up in adapters. It does not mean that data gets really > "committed". > > The commands can be executed in or out-of-order - this is a pure SCSI > > story. > > But if they have to be executed in order the ordering provided by SCSI > > will enable it. > > And as you and others have pointed out the same mechanism is good for > both > > ordering and flow control. > > As far as I understand it windowing - although more expensive - is a > better > > technique over a wide variety of latencies than credits. > > > > And BTW we even considered credits for a "prefetching mechanism" for > > chained > > commands but where told that those are very much "out of fashion" (i.e. > not > > worth speeding up on Elefants). > > > > Julo > > > > David Robinson on 29/09/2000 21:25:52 > > > > Please respond to David Robinson > > > > To: ips@ece.cmu.edu > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: RE: Status summary on multiple connections > > > > > I am left with the following impression as to what was indicated here: > > > - In general, command ordering is not relevant > > > - If the initiator filesystem detects an ordering dependency, it will > > wait > > > until outstanding commands are complete before issuing the dependant > > > command. > > > > > > This may be a reasonable means of operation for the disk world. It is > > > woefully inadequate for the tape world, as follows: > > > - In general, command ordering is crucial - out of order command > > processing > > > will lead to data corruption. > > > - This would require the initiator backup application to block on > > completion > > > of every single write command of a backup operation before issuing the > > next > > > command. > > > > > > If this blocking were performed, both the throughput and capacity of a > > tape > > > device/media would be negatively impacted by an order of magnitude or > > more. > > > This would occur even assuming an instantaneous transport. > > > > I am hearing different stories on the issue of ordering. One side > > is pushing hard for techniques that will allow out of order > > execution using various RDMA techniques. This clearly states for > > a certain class of devices (e.g. tapes) ordering is crucial. > > I thought this problem was already solved at the SCSI layer > > through the use of ordered commands which in general are not used > > for disks but always used for tapes? Since FC will reorder this > > has to be a solved problem. Would not an initiator talking to > > a tape target simply set the ordering flag? > > > > Lastly, for a TCP based connection ordering can easily be made a > > non-issue, simply don't try to process segments out of order. I > > will defer to a transport expert, but I believe processing > > TCP segments by an application out of order might cause problems. > > In particular since the out of order segment is not ACKed until > > after the missing segments arrive, they can be retransmitted > > multiple times. SACK helps this but does not guarentee that > > segments will not be retransmitted. So to process out of order > > segments the application must maintain a list of which segments > > have been processed as well, yuck! > > > > -David From owner-ips@ECE.cmu.edu Mon Oct 02 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16839; 2 Oct 2000 14:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10404 for ips-outgoing; Mon, 2 Oct 2000 13:45:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10385 for ; Mon, 2 Oct 2000 13:44:54 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 23C2B24D; Mon, 2 Oct 2000 11:44:52 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 63BC91EB; Mon, 2 Oct 2000 13:44:50 -0400 (EDT) Received: from agilent.com (cos1nai254106.cs.itc.hp.com [15.3.254.106]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA25147; Mon, 2 Oct 2000 10:44:46 -0700 (PDT) Message-ID: <39D8987F.6C1D2813@agilent.com> Date: Mon, 02 Oct 2000 07:15:27 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: David Robinson , ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: <200009291825.LAA13471@ha10nwk.EBay.Sun.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk David Robinson wrote: > > I am left with the following impression as to what was indicated here: > > - In general, command ordering is not relevant > > - If the initiator filesystem detects an ordering dependency, it will wait > > until outstanding commands are complete before issuing the dependant > > command. > > > > This may be a reasonable means of operation for the disk world. It is > > woefully inadequate for the tape world, as follows: > > - In general, command ordering is crucial - out of order command processing > > will lead to data corruption. > > - This would require the initiator backup application to block on completion > > of every single write command of a backup operation before issuing the next > > command. > > > > If this blocking were performed, both the throughput and capacity of a tape > > device/media would be negatively impacted by an order of magnitude or more. > > This would occur even assuming an instantaneous transport. > > I am hearing different stories on the issue of ordering. One side > is pushing hard for techniques that will allow out of order > execution using various RDMA techniques. This clearly states for > a certain class of devices (e.g. tapes) ordering is crucial. > I thought this problem was already solved at the SCSI layer > through the use of ordered commands which in general are not used > for disks but always used for tapes? The ordered commands option in SCSI indicates that the commands shall be executed in order. The only way SCSI knows of the "orderness" of the commands is if they're received in order. > Since FC will reorder this has to be a solved problem. It's solved in the FC layer, not SCSI. FC provides for "command numbering" such that the receiver of the commands verifies the orderness of the commands by sorting them based on the command numbers before delivering them to SCSI. > Would not an initiator talking to a tape target simply set the ordering flag? Nope. The transport layer has to guarantee that the commands are delivered to SCSI in the same order that the transport layer received the commands from SCSI. > Lastly, for a TCP based connection ordering can easily be made a > non-issue, simply don't try to process segments out of order. The issue is not commands on a single TCP connection. The issue is keeping commands in order that were issued on multiple connections (making up a single session). Ordering across TCP connections is not guaranteed, only within a TCP connection. > I will defer to a transport expert, but I believe processing > TCP segments by an application out of order might cause problems. > In particular since the out of order segment is not ACKed until > after the missing segments arrive, they can be retransmitted > multiple times. SACK helps this but does not guarentee that > segments will not be retransmitted. So to process out of order > segments the application must maintain a list of which segments > have been processed as well, yuck! > > -David > -Matt From owner-ips@ECE.cmu.edu Mon Oct 02 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16848; 2 Oct 2000 14:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10442 for ips-outgoing; Mon, 2 Oct 2000 13:45:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10413 for ; Mon, 2 Oct 2000 13:45:07 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id T9NCD81Z; Mon, 2 Oct 2000 10:42:36 -0700 From: "Y P Cheng" To: "Lyndon Ong" , "Mark A. Carlson" , "Randall R. Stewart" Cc: "David Robinson" , "ips" Subject: RE: Status summary on multiple connections Date: Mon, 2 Oct 2000 10:42:32 -0700 Message-ID: <002b01c02c98$24756b60$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: <4.2.2.20001002092921.00b7b4b0@zsc4c006.corpwest.baynetworks.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: Lyndon Ong [mailto:long@nortelnetworks.com] > Sent: Monday, October 02, 2000 9:48 AM > > Both of you forget about the case when multiple PDUs are inflight, say N, > > N+1, and N+2, and one of them has CRC error, say N+1. The receiver throws > > away N+1 because the bad CRC. N+2 is received long before N+1 is > > retransmitted after a timeout by the sender. Of course, a > > sequence number inside the PDU will ensure sequentially. > > This confuses me. Why would the application ever see the N+2 > PDU, wouldn't it be queued by TCP until N+1 is delivered correctly? It > shouldn't matter if N+2 was delivered first or if it contains a > sequence number at the higher layer, the higher layer would not get N+2 > before N+1.TCP would presumably not be aware of the boundaries between > messages at the higher layer, but deliver what it sees as a stream of bits. Sorry, I erred by assuming that each command PDU is delivered by a separate TCP segment. However, on a network with long delay, I do assume that the TCP segment N+2 will be delivered before N+1 is acknowledged. If this assumption is not true, we will have BIG performance problem on a network with long latency delay. From owner-ips@ECE.cmu.edu Mon Oct 02 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16857; 2 Oct 2000 14:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10392 for ips-outgoing; Mon, 2 Oct 2000 13:44:57 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10372 for ; Mon, 2 Oct 2000 13:44:48 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id D74A32BF for ; Mon, 2 Oct 2000 11:44:45 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 41AF41E9 for ; Mon, 2 Oct 2000 13:44:44 -0400 (EDT) Received: from agilent.com (cos1nai254106.cs.itc.hp.com [15.3.254.106]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA25134 for ; Mon, 2 Oct 2000 10:44:40 -0700 (PDT) Message-ID: <39D89639.D2EEA9F0@agilent.com> Date: Mon, 02 Oct 2000 07:05:45 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Status summary on multiple connections References: <118A72FD3F02D211AD1000805F9AA58B02ADC1C3@boulderp03.Exabyte.COM> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk JoeBre@exabyte.com wrote: > Much good discussion on SCSI device level flow control snipped.... > > > From: Robert Snively > > To: "'David Robinson'" , Robert Snively > > > > Subject: RE: Status summary on multiple connections > > Date: Thu, 28 Sep 2000 09:36:50 -0700 > > ... > > > are again available. Note that there is a possibility that commands > > that are inflight and have ordering constraints may be accepted out of > > order, a question that has caused lots of agonizing, but is apparently > > reasonably well managed by most file systems today by the selective > > use of ordering only for blocking boundaries of a particular logical > > stream of commands. > > I am left with the following impression as to what was indicated here: > - In general, command ordering is not relevant > - If the initiator filesystem detects an ordering dependency, it will wait > until outstanding commands are complete before issuing the dependant > command. > > This may be a reasonable means of operation for the disk world. It is > woefully inadequate for the tape world, as follows: >>From what I understand, it's woefully inadequate for disk mirroring applications as well. > > > - In general, command ordering is crucial - out of order command processing > will lead to data corruption. > - This would require the initiator backup application to block on completion > of every single write command of a backup operation before issuing the next > command. > > If this blocking were performed, both the throughput and capacity of a tape > device/media would be negatively impacted by an order of magnitude or more. > This would occur even assuming an instantaneous transport. > > Joe Breher > Exabyte Corp From owner-ips@ECE.cmu.edu Mon Oct 02 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16834; 2 Oct 2000 14:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10482 for ips-outgoing; Mon, 2 Oct 2000 13:46:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1tx.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10475 for ; Mon, 2 Oct 2000 13:46:01 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1tx.cos.agilent.com (Postfix) with ESMTP id 16AE0498 for ; Mon, 2 Oct 2000 11:45:58 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 2FF2C12A for ; Mon, 2 Oct 2000 13:45:57 -0400 (EDT) Received: from agilent.com (cos1nai254106.cs.itc.hp.com [15.3.254.106]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA25440 for ; Mon, 2 Oct 2000 10:45:53 -0700 (PDT) Message-ID: <39D8C943.8B3C6AB3@agilent.com> Date: Mon, 02 Oct 2000 10:43:31 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: A Transport Protocol Without ACK (resend) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > I agree, XON/XOFF would not work. A credit token scheme similar to FC Class > 3 would. FC class 3 does not have an end to end "credit token scheme". It (class 3 bb credit) is simply a credit mechanism between a node and it's neighbor node (the switch port) on whether the node can receive a frame. This means that the fabric has buffering inside it, and will hold onto a frame if the destination node does not have room for it. Thus, if the receiving node temporarily runs out of buffers, it will withold credit from the switch port. The sending node doesn't know that the receiving node has run out of buffers, so the fabric buffers fill up with undeliverable frames. -Matt From owner-ips@ECE.cmu.edu Mon Oct 02 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16850; 2 Oct 2000 14:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA10478 for ips-outgoing; Mon, 2 Oct 2000 13:46:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA10468 for ; Mon, 2 Oct 2000 13:45:59 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 98CA2332; Mon, 2 Oct 2000 11:45:28 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 55953F2; Mon, 2 Oct 2000 13:45:27 -0400 (EDT) Received: from agilent.com (cos1nai254106.cs.itc.hp.com [15.3.254.106]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id KAA25357; Mon, 2 Oct 2000 10:45:20 -0700 (PDT) Message-ID: <39D8A2A6.55D3E913@agilent.com> Date: Mon, 02 Oct 2000 07:58:47 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Robert Reynolds , IPS Reflector Subject: Re: not every song References: <670C86291BFCD311AFE500508B94D4B6630CC0@hqmail1.crossroads.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Robert Reynolds wrote: > I agree with Douglas. The WRITE-DATA and DATA-RESPONSE > message become more important when you are talking longer > distances, and longer delays. You can reduce your roundtrip > latency by 1/2 on WRITE commands by using a WRITE-DATA message > instead of independent WRITE-XFER_RDY-DATA messages. This > can be a huge performance increase when talking about doing > data replication across longer latency connections. This is > the difference between a 2 hour database sync. delay and a 4 > hour sync delay. > > I think it is a mistake that FCP-2 removed those messages because > they weren't currently implemented and I think it would be > a mistake to remove them from iSCSI. Please read Charles Binford's message on flow control that clarifies this issue. To summarize, FC only prohibitted command and data IN THE SAME MESSAGE (fc sequence). It did not prohibit command (message/sequence) followed by data (in a separate message/sequence). -Matt From owner-ips@ECE.cmu.edu Mon Oct 02 18:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa16829; 2 Oct 2000 14:45 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA09988 for ips-outgoing; Mon, 2 Oct 2000 13:35:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA09976 for ; Mon, 2 Oct 2000 13:35:34 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id T9NCD811; Mon, 2 Oct 2000 10:33:15 -0700 From: "Y P Cheng" To: Cc: "'Ips@Ece. Cmu. Edu'" Subject: RE: Status summary on multiple connections Date: Mon, 2 Oct 2000 10:33:11 -0700 Message-ID: <002501c02c96$d64d98a0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal In-Reply-To: <39D8ABCE.765651A1@sun.com> Sender: owner-ips@ece.cmu.edu Precedence: bulk > N+2 is received by the TCP stack, but not delivered to iSCSI > until N+1 arrives > (from the retransmit) and is delivered in sequence. This is how TCP works > according to my understanding. Sorry, I erred by assuming that each command is delivered in a separate TCP segment which is how I would have done by implementing the TCP in an adapter. BTW, may be I should bring up my old broken tune. Have anyone thought about the amount of TCP buffer needed for a one Gb network with 160 msec roundtrip time? It is 16MB. If we have a missing ACK, the buffer need be very big or the wait would be very long. The ten Gb Ethernet is coming very soon! Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Mon Oct 02 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17048; 2 Oct 2000 15:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA12773 for ips-outgoing; Mon, 2 Oct 2000 14:36:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA12756 for ; Mon, 2 Oct 2000 14:36:40 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id UAA159020 for ; Mon, 2 Oct 2000 20:05:18 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id UAA29358 for ; Mon, 2 Oct 2000 20:05:17 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.00635B40 ; Mon, 2 Oct 2000 20:05:14 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 21:02:29 +0300 Subject: Re: Bidirectional SCSI commands and iSCSI Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, I'll take a look at the proposals (I assume they are on the T10 site). My concern about cluttering is related to the attempt to make a fixed-length header as I think that software initiators will be widely used and they are far more efficient with fixed length headers for widely used operations. Thanks, Julo "Jim Hafner/Almaden/IBM" on 02/10/2000 18:22:40 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: Bidirectional SCSI commands and iSCSI Julo, True, those examples are not true duplex operations. These are only the current thinking or motivation for bi-directional commands. They are not the end of the story, I'm sure. However, the only SAM-2 requirement on bi-directional commands (as now defined) is that DataOut buffer be ready at the time the command is issued (so the target can get/request that data at any time after receipt of the command). In other words, the content of the DataOut buffer should not be time or data-dependent on the contents of the DataIn buffer. This means that as far as SAM-2 is concerned, there is no problem with a true duplex operation. (We just don't have any examples at the moment.) I think your second proposal (using "immediate" data) is only attempting to address the specific examples (OSDs), and not the generic notion of bi-directional commands. How would that solution work if the DataOut buffer was a 1MG chunk of user data for a XDREADWRITE command? The first method (splitting, if I read it correctly) undercuts the motivation that is generating this bi-d command in the first place. The SCSI layer could do that as well, but the issue is one of atomicity of command. (Aside: one principle governing definitions of bi-directional commands in T10 is that there must always be a two (or more) uni-directional variant of the command, to allow a complex operation to be performed on a traditional uni-directional with simpler steps. I read your suggestion as having iSCSI do this for the SCSI layer -- there'd be no point. Besides, doesn't this break layering?). Finally, I would suggest a look at the FCP-2/3 proposal just submitted. I think they are proposing (for legacy reasons) a new IU for bi-directional commands. If you're concerned about cluttering your header, then this group might think about adding a new PDU for these types of commands. I don't quite understand the reluctance to add new fields in the header for this purpose. The current design is not yet cast in stone and now is the perfect opportunity to get it defined in a way that accomodates the future that T10 seems to be moving (rapidly) towards. Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 10-01-2000 12:16:06 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, Both the examples you gave have something peculiar - the operations are to executed really in sequence (we are no talking about full duplex I/O operations?). I any case, in order no to extend excessively the headers we can either introduce a "next header" field (as in IP V6) or split the operation at the iSCSI level into two phases - each carrying its parameters and each ended with a status - like: - out XOP1, XOP2, OD1, -in S1, ID, S2 In any case commands having only a parameter set (fixed or variable) - like those for the object disk should be handled with the parameters sent as immediate data (as with the three party commands) - parameter size will probably be limited and the performance requirement will be similar to those of block read/write. Do we need for those to care also about residual counts for parameters? Julo "Jim Hafner/Almaden/IBM" on 29/09/2000 00:21:32 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Bidirectional SCSI commands and iSCSI Julo, The primary motivation is for object-based storage devices (OSDs). The protocol that seems to be needed in that environment is the ability to send a CDB, plus some parameter data (DataOut) AND get back read-type data (DataIn) in the same command sequence. One (non-OSD) command has been proposed in T10 for an XOR type XDREADWRITE command. SAM-2 has now allowed for such commands in the architecture. From this perspective, it is completely symmetric. Transports that support this feature will need to have data structures in there PDUs or IUs (or whatever) to indicate things like DataOut datalength, DataIn datalength, possible virtual addresses for each buffer, etc. In other words, there will potentially be two active buffers (on each end) for use with the same command. So, e.g., iSCSI PDU may need to have both "DataIn expected data transfer length" and "DataOut expected transfer length" fields. I'm assuming that the data direction in the existing iSCSI PDU is derived from the SCSI opcode, unlike FCP where there are fields to indicate the data direction. (I am surprised not to find such things in the iSCSI Command PDU.) There are two proposals in T10 to enable this feature in SPI-4 and in FCP-x (probably FCP-3). Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-28-2000 12:10:53 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, What is the intended use? Are there already commands that use it? Is there a major/minor distinction - like output data & input results or output parameters & input data - or is it intended to be completely symmetric? Julo "Jim Hafner/Almaden/IBM" on 20/09/2000 17:56:27 Please respond to "Jim Hafner/Almaden/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Bidirectional SCSI commands and iSCSI Folks, In case you missed this, T10 just approved changes to SAM-2 (rev14) which remove the restriction that a single SCSI command can have a data flow in only one direction (DataIn xor DataOut). Transport specs (that want to allow this) will need to change to allow for two sets of Data buffer fields in the headers (length, over/underrun, etc.). Is anybody looking to modify the iSCSI spec to get these concepts supported? Jim Hafner From owner-ips@ECE.cmu.edu Mon Oct 02 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17053; 2 Oct 2000 15:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA14145 for ips-outgoing; Mon, 2 Oct 2000 15:04:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA14134 for ; Mon, 2 Oct 2000 15:04:37 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id VAA75638 for ; Mon, 2 Oct 2000 21:04:06 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id VAA119810 for ; Mon, 2 Oct 2000 21:04:06 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.0068BCB1 ; Mon, 2 Oct 2000 21:04:00 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 22:01:42 +0300 Subject: RE: The third alternative Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I think we are. Julo "John Hufferd/San Jose/IBM" on 02/10/2000 19:39:01 Please respond to "John Hufferd/San Jose/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: RE: The third alternative OK folks, I think we have agreement between Julian and David Black (hope,hope, hope). David, near the end of his note said "...Provide for is fine, as long as they're not required." Julian, I think this is all your were after anyway. So are we all back together now? . . . John L. Hufferd Black_David@emc.com@ece.cmu.edu on 10/02/2000 07:02:31 AM Sent by: owner-ips@ece.cmu.edu To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: The third alternative > The only argument for SCSI wedge drivers where that they EXIST ALREADY. > Pretty weak argument for those building new equipment and for > interoperability. This severely understates the case. Not only do wedge drivers exist already, but they do a number of things that an iSCSI standard will never do. Wedge drivers tend to contain logic specific to the device that they're providing the wedge for - that's inappropriate to standardize, either in IETF or T10. In addition, there are implementation advantages to building wedge drivers above the SCSI level - one doesn't have to spread SCSI connection state across a failure boundary. > I think that if we keep ourselves honest we have to either: > > - provide for multiple connections at the iSCSI level as it is transport problem > that other TCP applications are not compelled to handle (I hear already BUT SCTP > handles it!) and hope that one day the session concept will drift into pure transport Provide for is fine, as long as they're not required. I would expect to see significant deployment of systems that use single TCP connection per session and handle multiple sessions in wedge drivers. Arguments like the one Bob has made is one of the reasons for multiple connections/session to remain OPTIONAL. > - go to T10 and ask the to standardize wedge drivers! Just say no! Exactly what would you want T10 to standardize? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 19:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17036; 2 Oct 2000 15:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA13517 for ips-outgoing; Mon, 2 Oct 2000 14:51:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA13508 for ; Mon, 2 Oct 2000 14:51:25 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id UAA20206 for ; Mon, 2 Oct 2000 20:28:12 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id UAA41828 for ; Mon, 2 Oct 2000 20:28:11 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.00657107 ; Mon, 2 Oct 2000 20:28:00 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 21:25:42 +0300 Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim, No. The "view" is just a character string (another accesID if you want) that will help the target understand who the initiator thinks it is talking too. The target will make-up the map accordingly. The view is in fact the "NAME of the VIEW". More than that - in fact the whole string can be used as a selector but we did not want to force the host name to be unique for the view as it is many times done in the "WWW" world (nevertheless we can't prevent it). Julo "Jim Hafner/Almaden/IBM" on 02/10/2000 18:30:53 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Julo, Are you suggesting that the host send to the target what it thinks the LUN->LU map should be (during the login process)? I think you call this LUN Map, the "view". I don't quite see what the point of this is. The initiator is fully authenticated to the target as is the target to the initiator (via whatever cryptographic or non-cryptographic schemes the security proposal defines). After that, the target knows whose talking and knows the map (based on configuration information established in any number of ways (Access Control commands in-band; vendor specific in-band or out-of-band). I don't see the point of the initiator sending its "view" to the target at any point. If the initiator sends an incorrect view, you've just created an error scenario which needs additional definition. In any case, the target will send its "view" for that initiator in REPORT LUNS SCSI command after login (which is actually the layer that cares about the LUN Map at all anyway). Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 10-02-2000 02:09:56 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Jim, The "URL like" target address and view are sent o the target as part of the text-message at login. If authenthication is being done then this will be part of the message that follows the exchange that establishes the authenticated channel. Julo "Jim Hafner/Almaden/IBM" on 29/09/2000 18:07:54 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Julo, Yet another method for determining the mapping of LUs to LUNs for given initiator? You already have vendor-specific ways (in FC that will surely get ported to iSCSI), the T10 standardized way with AccessID enrollment or with a TransportID (for iSCSI, which is still TBD), and the login authentication process (however that ends up getting defined). Also, your comment implies that the URL is sent to the target in a mode similar to an http-type protocol. Where is this protocol defined in the draft? Is it part of the Text message in the login authentication? Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-29-2000 06:14:44 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] I would add that we wanted the path to be an additional identifier that the target could use to determine what collection of LUs to present to the initiator. Julo csapuntz@csapuntz-u1.cisco.com on 23/09/2000 05:00:33 Please respond to csapuntz@csapuntz-u1.cisco.com To: "IP Storage" cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Just for clarification... I was not proposing to add the extended URL scheme for the transport spec. It isn't necessary. In this thread, Doug makes an excellent point about LDAP being a superior mechanism for describing how to connect to the storage. Directory services such as LDAP, as Doug has pointed out, will be critical to managing large quantities of storage. A host can ask such a directory service for a list of storage devices it should mount and how to connect to those storage devices. The query against the directory server that returns this information can be based on machine ID, user ID, operating system ID, or even the owner's birthday. One of the things discovery will end up doing, no doubt, is defining LDAP schemas that describe how to connect to storage (i.e. use SCTP or TCP, what port, what target name, what LUN, what WWN, how to authenticate, etc.). However, there is one place where the transport protocol has to define a name: third party commands. There needs to be some kind of global name which the initiator can pass to the target. The name must be distillable into a string. The target must understand the name and be able to use the information to establish a connection to another target. One could say that the string that is passed is not specified by the standard but instead specified by some management software. I think this will lead to poor interoperability. The SCSI URL-type name is the current proposal for target name. Why is SCSI target name made up of a hostname + a path? Why is the hostname + path passed on connection setup? There are two reasons. I think NAT and IPv6 makes passing hostnames rather than addresses in protocols more desirable. Hostnames can be re-resolved as you cross addressing boundaries. The path is there so that the name can support multiple targets behind a single IP address without having to add entries to the DNS server. At Cisco, for example, I have no control over the local DNS servers and cannot add DNS entries for the ATAPI DVD and floppy in my computer. _Costa From owner-ips@ECE.cmu.edu Mon Oct 02 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17070; 2 Oct 2000 15:42 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA12981 for ips-outgoing; Mon, 2 Oct 2000 14:41:38 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA12977 for ; Mon, 2 Oct 2000 14:41:30 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id TAA130166 for ; Mon, 2 Oct 2000 19:58:21 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id TAA190288 for ; Mon, 2 Oct 2000 19:58:21 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.0062B80D ; Mon, 2 Oct 2000 19:58:16 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 20:55:04 +0300 Subject: Re: New List: rdma@cisco.com: to discuss RDMA Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Steph Bailey said: I said in my previous message that using RDMA would make iSCSI implementation easier. I take it back. It's not clear that this is actually the case. The real advantage of a general RDMA is that it can bring the benefits traditionally associated with storage adapters (zero copy, and low CPU overhead) to protocols other than SCSI. However, it also seems like an advantage that a high performance iSCSI on RDMA implementation can be built with no iSCSI specific hardware at all. > Let the market decide whether that one implementation should survive or > not. Most likely it will die as others will have been smarter in their > designs and functional selection. Exactly. If the tagged transfer mode is well designed, and RDMA is not required, nobody will do RDMA for iSCSI. Whether to use RDMA or not is a choice that the iSCSI standard is going to have to make. Either way works. However, I do not believe there is a sensible way to allow for both possibilities. Steph ________________ Many moths ago I was making the same arguments FOR RDMA on this list and I faced the same strong oposition. It probably happens with every protocol that could benefit from a generic RDMA but its simpler for the job at hand to "roll your own". I suggest all interested parties (I include myself) to take it off to the separate list Costa was so kind to setup and take it on there as a "generic RDMA". Julo From owner-ips@ECE.cmu.edu Mon Oct 02 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17044; 2 Oct 2000 15:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA12739 for ips-outgoing; Mon, 2 Oct 2000 14:36:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA12732 for ; Mon, 2 Oct 2000 14:36:11 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id UAA40276 for ; Mon, 2 Oct 2000 20:35:39 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id UAA55550 for ; Mon, 2 Oct 2000 20:35:39 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.0066217F ; Mon, 2 Oct 2000 20:35:32 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 21:33:09 +0300 Subject: RE: Status summary on multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk A very crisp clarification. I would like only to point out that speed, latency and the out of order delivery by switches are aggravating factors and most DMA schemes attempt only to find a "good home" for the out-of-order arrivals in order not to increase excessively the memory needs of the adapters or of the stack buffers. Nothing is presumed to be delivered from TCP to iSCSI for handling (i.e. processing) out-of-order in the TCP sense. Julo Black_David@emc.com on 02/10/2000 18:35:00 Please respond to Black_David@emc.com To: ycheng@advansys.com, mark.carlson@sun.com, randall@stewart.chicago.il.us cc: David.Robinson@EBay.Sun.COM, ips@ece.cmu.edu (bcc: Julian Satran/Haifa/IBM) Subject: RE: Status summary on multiple connections > Both of you forget about the case when multiple PDUs are inflight, say N, > N+1, and N+2, and one of them has CRC error, say N+1. The receiver throws > away N+1 because the bad CRC. N+2 is received long before N+1 is > retransmitted after a timeout by the sender. Of course, a sequence number > inside the PDU will ensure sequentially. However, as I stated in another > posting, all software realize such problem and will not count on sequential > delivery by a transport. Let me try to head off some confusion here. PDU is a dangerous acronym because it could refer to layer 2 (e.g. Ethernet) packets, layer 4 TCP segments or iSCSI information units at layer 5. For this discussion, layers 2 and/or 4 are the intended context. If packet/segment N+1 has an (e.g., Ethernet) CRC error, TCP has to buffer N+2 and can't deliver it to the application. The concern that's behind discussions of things like RDMA and the Urgent pointer is in the details of how to buffer N+2. If the N+1/N+2 boundary is not at the start of an iSCSI PDU, N+2 gets tossed into some memory somewhere and copied out when N+1 arrives and it's possible to resume processing of the stream at the iSCSI level; this can be slow. One of the goals of the RDMA and Urgent discussion is to identify the iSCSI boundary in the TCP data, either by forcing the boundary to the TCP N+1/N+2 boundary or providing a pointer to where it is inside N+2 so that all or most of N+2 can go through iSCSI processing on receipt and hit the right place in memory the first time. Handing a SCSI command found in N+2 to the SCSI device before getting the N+1 data is prone to cause a variety of peculiarities. For example, if the command in N+2 is a task abort of a command in N+1, the Initiator is going to be rather surprised when the abort fails and the task that was supposed to be aborted executes after the failed abort (which may be observable via a different iSCSI session). There are doubtless cases in which this can be done safely, but they'll need some careful investigation. David Robinson's contention that the WG shouldn't spend time on optimizing cases in which packets are lost because they're not going to happen often enough for optimizations to make a significant difference is an open issue that the WG will need to come to a conclusion on. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 20:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17201; 2 Oct 2000 16:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA14894 for ips-outgoing; Mon, 2 Oct 2000 15:18:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA14881 for ; Mon, 2 Oct 2000 15:18:51 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id VAA36612 for ; Mon, 2 Oct 2000 21:18:15 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id VAA101154 for ; Mon, 2 Oct 2000 21:18:15 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696C.006A0785 ; Mon, 2 Oct 2000 21:18:07 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Mon, 2 Oct 2000 22:15:49 +0300 Subject: Re: Status summary on multiple connections Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I though that this is exactly what I said more than once. iSCSI assumes that it gets data from TCP in order and acked. RDMA is an underlying convenience. For commands received on several connections the things are a bit more complicated. No one of the mechanisms we have built mandates SCSI - in specific contextes to take and execute the commands in the iSCSI order or even to wait until all the previous commands have been received. This is a restriction that we don't have to enforce. Everything - even recovery will work without it. Did I miss something? Julo Matt Wakeley on 02/10/2000 17:28:49 Please respond to Matt Wakeley To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Status summary on multiple connections I think the layering is being blurred. Remember, whether there is RDMA or not, TCP is the layer doing the ACKing, *NOT* iSCSI. And whether or not there is RDMA or not, TCP cannot ACK TCP segments that arrive out of order until the missing ones arrive (it can SACK them, but it can't ACK them - and SACK is only informative). iSCSI cannot deliver commands out of order to the SCSI layer. So even if the command is RDMA'd into a buffer, it cannot be delivered to SCSI until the previous commands are received. -Matt julian_satran@il.ibm.com wrote: > David, > > The picture I had in mind is somewhat simpler. > > I assume that on any given transport connection, with or without RDMA, > iSCSI > will get from TCP and hand to SCSI things that are in order (just to keep > layering iSCSI over TCP simple and clean). No retransmissions involved in > iSCSI here. > > In case of several connections iSCSI might get things out of order (no ack > for them) > and may decide to hand them or not to hand them to execution. > iSCSI will not require retransmission as we assume that the out of order > thing is > a temporary artifact of the underlying network. Recovery if needed here is > done > only when a connection is blown-away and then all outstanding commands > that where shipped on that connection are restarted (resent). > > And again we conceived the window only to limit the number of commands > iSCSI has > to keep to get things in order but it can as well serve as a flow-control > mechanism for > the target as the initiator is unconcerned of what is being done with the > commands within the window (e.g., a R2T from a command within the window - > i.e., not acked yet - is a > legal event). > > Julo > > David Robinson on 30/09/2000 01:35:11 > > Please respond to David Robinson > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: Status summary on multiple connections > > There are two types of out of order processing we need to > consider: > 1) Commands that have been received and ACK'd by the transport > 2) Commands that have been received and not ACK'd > > In the first case this is purely a SCSI issue on if the commands > are ordered and how the target processes them. As far as the > transport is concerned the packets are "delivered". In the second > case the transport is still required to track the possible > retransmissions and process the transport ACK for all of > the data when the missing segment arrives. It will overly > complicate the iSCSI layer to successfully track which parts > of the sequence space it has processed, defend against > retransmissions, and handle packets that have more than one > command, and retransmissons that may be only part of a command. > > This is all doable in an implementation (a bit messy at times) > but to support this as an explicit feature we need to start > adding in an iSCSI layer sequence space. > > I propose that we leave the processing of transport > level un-ACK'd commands and data as an implementation detail > and not make it an explicit iSCSI feature. Dropped or reordered > TCP segments are rare enough in high performance environments > that this is really a non-issue. > > -David > > julian_satran@il.ibm.com wrote: > > > > David, > > > > I think that RDMA out-of-order processing is important only in order not > to > > have data > > piling up in adapters. It does not mean that data gets really > "committed". > > The commands can be executed in or out-of-order - this is a pure SCSI > > story. > > But if they have to be executed in order the ordering provided by SCSI > > will enable it. > > And as you and others have pointed out the same mechanism is good for > both > > ordering and flow control. > > As far as I understand it windowing - although more expensive - is a > better > > technique over a wide variety of latencies than credits. > > > > And BTW we even considered credits for a "prefetching mechanism" for > > chained > > commands but where told that those are very much "out of fashion" (i.e. > not > > worth speeding up on Elefants). > > > > Julo > > > > David Robinson on 29/09/2000 21:25:52 > > > > Please respond to David Robinson > > > > To: ips@ece.cmu.edu > > cc: (bcc: Julian Satran/Haifa/IBM) > > Subject: RE: Status summary on multiple connections > > > > > I am left with the following impression as to what was indicated here: > > > - In general, command ordering is not relevant > > > - If the initiator filesystem detects an ordering dependency, it will > > wait > > > until outstanding commands are complete before issuing the dependant > > > command. > > > > > > This may be a reasonable means of operation for the disk world. It is > > > woefully inadequate for the tape world, as follows: > > > - In general, command ordering is crucial - out of order command > > processing > > > will lead to data corruption. > > > - This would require the initiator backup application to block on > > completion > > > of every single write command of a backup operation before issuing the > > next > > > command. > > > > > > If this blocking were performed, both the throughput and capacity of a > > tape > > > device/media would be negatively impacted by an order of magnitude or > > more. > > > This would occur even assuming an instantaneous transport. > > > > I am hearing different stories on the issue of ordering. One side > > is pushing hard for techniques that will allow out of order > > execution using various RDMA techniques. This clearly states for > > a certain class of devices (e.g. tapes) ordering is crucial. > > I thought this problem was already solved at the SCSI layer > > through the use of ordered commands which in general are not used > > for disks but always used for tapes? Since FC will reorder this > > has to be a solved problem. Would not an initiator talking to > > a tape target simply set the ordering flag? > > > > Lastly, for a TCP based connection ordering can easily be made a > > non-issue, simply don't try to process segments out of order. I > > will defer to a transport expert, but I believe processing > > TCP segments by an application out of order might cause problems. > > In particular since the out of order segment is not ACKed until > > after the missing segments arrive, they can be retransmitted > > multiple times. SACK helps this but does not guarentee that > > segments will not be retransmitted. So to process out of order > > segments the application must maintain a list of which segments > > have been processed as well, yuck! > > > > -David From owner-ips@ECE.cmu.edu Mon Oct 02 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17221; 2 Oct 2000 16:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15826 for ips-outgoing; Mon, 2 Oct 2000 15:36:41 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15071 for ; Mon, 2 Oct 2000 15:22:51 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA15786; Mon, 2 Oct 2000 12:08:43 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA18740; Mon, 2 Oct 2000 12:06:30 -0700 From: "Douglas Otis" To: "Matt Wakeley" , Subject: RE: A Transport Protocol Without ACK (resend) Date: Mon, 2 Oct 2000 12:25:06 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: <39D8C943.8B3C6AB3@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, You would not want an End-to-End scheme. You would want a scheme that prevents any node or sub-node from over-flowing. That is accomplished by Buffer-to-Buffer control. The present scheme now in place is with a rough approximation of command space with any excess data discarded as required. This makes for a lossy channel and one that does not allow control at a specific sub-node that may be at its limits. Restrain the depth of the sub-node with specific control. Just 8M-byte of FIFO will represent 100 millisecond of delay at FC speeds. A token credit scheme ensures no node ever gets to a point of dropping frames in addition to allowing maintenance of the FIFO depth. Should a sub-node fail, only that node stops functioning rather than everything being hung until resolution. A sizable short-coming among many others in aggregating everything into this common scheme. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Monday, October 02, 2000 10:44 AM > To: ips@ece.cmu.edu > Subject: Re: A Transport Protocol Without ACK (resend) > > > Douglas Otis wrote: > > > I agree, XON/XOFF would not work. A credit token scheme > similar to FC Class > > 3 would. > > FC class 3 does not have an end to end "credit token scheme". It > (class 3 bb > credit) is simply a credit mechanism between a node and it's neighbor node > (the > switch port) on whether the node can receive a frame. This means that the > fabric has buffering inside it, and will hold onto a frame if the > destination > node does not have room for it. Thus, if the receiving node > temporarily runs > out of buffers, it will withold credit from the switch port. The > sending node > doesn't know that the receiving node has run out of buffers, so the fabric > buffers fill up with undeliverable frames. > > -Matt > From owner-ips@ECE.cmu.edu Mon Oct 02 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17228; 2 Oct 2000 16:27 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA15266 for ips-outgoing; Mon, 2 Oct 2000 15:26:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub1.almaden.ibm.com (mailhub1.almaden.ibm.com [198.4.83.44]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA15255 for ; Mon, 2 Oct 2000 15:26:24 -0400 (EDT) Received: from catalina.almaden.ibm.com (catalina.almaden.ibm.com [9.1.24.58]) by mailhub1.almaden.ibm.com (8.8.8/8.8.8) with ESMTP id MAA61748; Mon, 2 Oct 2000 12:24:31 -0700 Received: (from dfsmith@localhost) by catalina.almaden.ibm.com (AIX4.3/UCB 8.8.8/8.7) id MAA25676; Mon, 2 Oct 2000 12:24:43 -0700 From: Daniel Smith Message-Id: <200010021924.MAA25676@catalina.almaden.ibm.com> Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: dotis@sanlight.net (Douglas Otis) Date: Mon, 2 Oct 2000 12:24:43 -0700 (PDT) Cc: julian_satran@il.ibm.com, ips@ece.cmu.edu In-Reply-To: from "Douglas Otis" at Oct 02, 2000 09:05:09 AM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I'll reply to this piece-by-piece as there seem to be a lot of misconceptions. (On a side note, I haven't seen any alternative proposals put forward to the URL scheme as noted. If things stay this way, then I'll put it forward as an Internet-Draft.) Okay, off we go. Douglas Otis wrote: > > If you wished to crack data being sent via IP, could you ask for a better > tag of such information- scsi://[/modifier] being sent in clear > text down the same connection? Of course, the same is true for login as will always be sent "in the clear" at some point on the network. There's not a lot we can do about that. The IP address will always be visible in all packets. (Insert comments here about why VPNs are a good thing here....) In an SSL link, or IPsec the [/modifier] part will be encrypted. Indeed, whenever security is tight, the [/modifier] will only be sent after clearance is granted. Why? Let's imagine a large array of fast, reliable and featuresome storage---an IBM Shark box say. B-) This box has many user accounts, and is being used in the context of a storage service provider. The login sequence might go something like this.... I: Hello Shark---I'm an initiator. My authentication ID is "Robert Smith, Seattle, next to the donut store, hjhhaskjanbt112kg9d988". S: Hello initiator. I'll be your server---we need a secure connection: switch to "hjcnneamchjsdkfsdsd394njsdf89423". (Trivial example.) I: Alright! I want to connect to "scsi://niceshark.acme.com/bobsstorage/san3?wwn=0x7b4d72658a7410db" Then we have a choice... S: Great---off you go and have a nice day. or S: Sorry---you haven't got permission to use that any more/it doesn't exist. or (just after they buy more storage) S: Ok, that URL has been changed to "scsi://bobstorage.san3.acme.com:9003/". Please come again. > well. Here again "who, what, and where" are nicely presented in easy to > read form. Neither SCSI clients nor servers are likely able to authenticate > independently or retain textual information. Both the client and server > should depend on some outside database. This compromise in security was If security is needed, then security is needed and there are no short cuts! Whether stored locally or remotely, that information needs to be somewhere, and securely accessible too. > done to eliminate an IP selection option but then a textual means of > including IP was included to ensure all modes of operation remain. Rather > than using a secure and opaque means to establish a connection using an > independent means of authenticating on a different connection, the entire > function was included in this all encompassing draft. This was not a good > choice in my view. There is no requirement that only the Initiator and Target MUST authenticate between /themselves/ only. This may be desirable (in case the authentication database machine goes down.) There are many papers written on this subject; for a simple introduction, see "Secure Distributed Computing", Sci. Am., Nov 1994 pp72--76. This will give you an idea of what the minimum requirements are in an open network authentication realm. > By creating a separate draft to define an authentication database access > (LDAP structures), options remain possible without impact on the transport > specification. URLs have very little impact on the transport. The iSCSI draft just handles arbitrary length text strings to identify resources---and that's all it cares about. The only time interpretation of this is requred is for 3rd-party commands, when we don't have a choice about it---and technically the transport still doesn't care about it. We DO want it standardized. We DON'T want it tied to the transport. In 2003 when iSCSIv5 over SCTP with Microsoft Active Super Directory v3 running over IPv8 on terrabit-RS232, the naming scheme should still be handled by the transport. > The only aspect of authentication within the transport > should be a cookie. Authentication provides access at some IP and port or Please forgive my ignorance, but isn't a cookie an arbitrary length text string (or can be mapped to and from an arbitrary length text string). Cookies as /constant/ entities is an extremely bad idea. That's why we don't recommend it. Okay, cookies are really small delicious disks, sometimes with chocolate chips---what do you mean by cookie? > perhaps a set of such locations, it could also provide third-party mapping > in binary as well. (This mapping could be encoded by various schemes.) As > other transports are employed, flexibility becomes important. Why insist > targets do DNS lookup? Keep the workload light as such lookups should never We don't. Where did you get this idea? If it found its way into the drafts then please point it out so that we can remove it! Target DNS resolution is always optional; though sometimes (perhaps often in the Shark example) it is useful, and perhaps required by the system designer. Initiator DNS resolution is mandatory. (Except in the case where an initiator is allowed interpret, say, "scsi://123.45.67.89".) > be needed beyond the process of authentication rightfully done by other > equipment not involved in the time critical data handling. I hope this is > comprehendible this time. It would be more comprehensible if you gave an example of where the scheme as currently proposed fails. Then we can work to fix it and produce a more baked Internet-Draft for interpreting iSCSI service delivery port names. Daniel Smith. [Mmm. Cookies.] -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 From owner-ips@ECE.cmu.edu Mon Oct 02 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17226; 2 Oct 2000 16:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA16362 for ips-outgoing; Mon, 2 Oct 2000 15:46:55 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA16353 for ; Mon, 2 Oct 2000 15:46:49 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 15DD0CB5 for ; Mon, 2 Oct 2000 12:46:48 -0700 (PDT) Received: from mk731912 (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id MAA25621 for ; Mon, 2 Oct 2000 12:47:00 -0700 (PDT) Message-Id: <4.2.2.20001002122830.00b8f200@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Mon, 02 Oct 2000 12:33:02 -0700 To: ips@ece.cmu.edu From: Michael Krause Subject: Re: Status summary on multiple connections In-Reply-To: <39D89BA1.F9627E77@agilent.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:28 AM 10/2/00 -0700, Matt Wakeley wrote: >I think the layering is being blurred. Remember, whether there is RDMA or >not, TCP is the layer doing the ACKing, *NOT* iSCSI. And whether or not there >is RDMA or not, TCP cannot ACK TCP segments that arrive out of order until the >missing ones arrive (it can SACK them, but it can't ACK them - and SACK is >only informative). > >iSCSI cannot deliver commands out of order to the SCSI layer. So even if the >command is RDMA'd into a buffer, it cannot be delivered to SCSI until the >previous commands are received. This last point is still very important. One can DMA to memory (SEND or RDMA semantics) without posting the completion event to the upper layers (iSCSI) thus relieving the responder from having to buffer all of this within the chip / adapter. There is a transport level ACK from TCP indicating all bytes for a given message have been received and then there is a iSCSI response which is a higher-level completion message which is generated once TCP has indicated all bytes up to a given value have been properly received. Mike From owner-ips@ECE.cmu.edu Mon Oct 02 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17508; 2 Oct 2000 18:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA19806 for ips-outgoing; Mon, 2 Oct 2000 17:04:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA19780 for ; Mon, 2 Oct 2000 17:03:09 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by atlrel2.hp.com (Postfix) with ESMTP id 3EB71974; Mon, 2 Oct 2000 17:03:05 -0400 (EDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA09252; Mon, 2 Oct 2000 15:03:03 -0600 (MDT) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id <41AJD1H0>; Mon, 2 Oct 2000 15:03:01 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: remove recovery from transport-layer connection failur e(?) Date: Mon, 2 Oct 2000 14:47:36 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, If the scenario you point out is correct (a single command lasting for such a long time), then of course we need a mechanism where we can restart the command from the approximate point of failure. However that would be failures lasting for "more than a fraction of a sec". First of all, a TCP connection does not indicate a failure that quickly. Secondly, there are ways to recover from a path failure and still preserve a TCP connection in High-Availability environments. I am sure most system vendors would be implementing such techniques. Somesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Sunday, October 01, 2000 1:52 AM To: ips@ece.cmu.edu Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) Steph, Assume than in the new wonderfull SAN world you have started a disk-to-tape (or disk-to-disk) long third party copy. The SAN is fine and the copy proceeds for an hour but the lousy initiator-to-copy-manager link (on which accidentally no data transfer took place) fails for a fraction of a second. Should we restart the command under-the-cover or drop it or ask the parties to provide state information to a specific SCSI restart driver? And we can build many similar scenarios. I think that whatever we can do simplify exception handling we should do (the same arguments that hold for multiple connections hold here too). I would add that in Ideal world - I would like to have transport "splice" a new TCP connection with an old TCP connection but failing this to happen (again SCTP is doing it already or not?) we should take care that simple events like a cable taken-out in some obscure part of the network will only seldom affect higher layers. Julo Stephen Bailey on 27/09/2000 07:16:12 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) > Currently, iSCSI is spec'ed to recover from transport-layer > connection failures. > > The main motivation for this decision was to support tape backup > applications that are quite sensitive to any failures that get > propogated to their layer. > > So, perhaps we can remove the requirement of recovering from > transport-layer connection failures in iSCSI. This would simplify > the protocol somewhat. > > Thoughts? I'm all for eliminating command recovery. There seem to be several reasons advanced for command recovery. The first seems to be based upon an inappropriate analogy to FCP. Command recovery had to be added to FCP-2 because the FC layer is unreliable. A single dropped FC frame leads to a failed FCP command. This clearly upsets tape operation even when the link is performing nominally. In FCP, without command recovery, with some observable frequency, you will get an expected error that leads to complete, irrecoverable failure of a transfer stream. The other thing that makes FCP-2 command recovery work well is when you are doing a write, which is 90% (maybe it's 99%?) of tape operation, the target can return an early indication of most frame drops, rather than waiting for a timer to expire. TCP's reliability solves this problem in another way. By the time you get a TCP connection failure, you have already exhausted a set of reliability mechanisms which guarantee, with high certainty, that further data can not be transferred between the two endpoints. `the two endpoints' phrase suggests the other reason advanced for command recovery. That is, to permit path failover for commands which are not idempotent, such as tape write sequential. The problem with this, is that it is not clear HOW iSCSI command recovery can actually work properly, given a TCP connection failure indication. It takes a long time for a TCP connection to fail, and by that time, I'm not sure recovery would reasonably be possible. Perhaps I'm in error on this assumption. Can a tape guru (Joe from Exabyte?) comment on whether recovery would be possible after many seconds (tens, hundreds) have elapsed? The SCSI layer has never been solely responsible for ensuring reliable backup. Macro scale things go wrong with tape (run off the end, get eaten, etc..) with relatively high frequency. A low level backup engine like tar or dump will fail on a SCSI error, and that's OK. There must also be a higher level software component like Amanda, which manages retries, including operator intervention, to ensure reliable backup. It seems like whether iSCSI has a command recovery mechanism should be a function of whether somebody can stand up and say for sure that it solves a real problem. So far it only seems like it MIGHT solve a problem. Who can say `this solves MY problem!'? Steph From owner-ips@ECE.cmu.edu Mon Oct 02 22:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17503; 2 Oct 2000 18:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA19529 for ips-outgoing; Mon, 2 Oct 2000 16:57:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA19517 for ; Mon, 2 Oct 2000 16:57:26 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id ; Mon, 2 Oct 2000 16:56:54 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101040@corpmx9.isus.emc.com> To: dfsmith@almaden.ibm.com, ips@ece.cmu.edu Subject: iSCSI URL scheme Date: Mon, 2 Oct 2000 16:56:52 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk With my co-chair hat off ... I've had a long-running concern about the URL naming in iSCSI that Daniel's example allows me to express concisely ... > Let's imagine a large array of fast, reliable and featuresome storage---an > IBM Shark box say. B-) This box has many user accounts, and is being used > in the context of a storage service provider. The login sequence might go > something like this.... > > I: Hello Shark---I'm an initiator. My authentication ID is "Robert Smith, > Seattle, next to the donut store, hjhhaskjanbt112kg9d988". > > S: Hello initiator. I'll be your server---we need a secure connection: > switch to "hjcnneamchjsdkfsdsd394njsdf89423". (Trivial example.) > > I: Alright! I want to connect to > "scsi://niceshark.acme.com/bobsstorage/san3?wwn=0x7b4d72658a7410db" This query is a *major* departure from SCSI practice at this level. The SCSI approach would be more along the lines of: I: Alright! Show me all your LUNs that I can access. (i.e., REPORT LUNS) at which point the Initiator then does additional SCSI commands to figure out what each LUN actually is. One of these commands could be to read (or attempt to read) the WWN for each LUN. For disk storage that is under a volume manager, there's another round of discovery later on in which the volume manager reads the volume label and figures out what the storage is independent of any information that was used to discover it. Returning to the three examples: > Then we have a choice... > > S: Great---off you go and have a nice day. > > or > > S: Sorry---you haven't got permission to use that any more/it > doesn't exist. This is an authorization check that can be implemented by a combination of accepting/denying the connection based on the client authentication information and/or only reporting the LUNs to which the Initiator has access (and after an exhaustive search, the Initiator may discover that a LUN it used to have access to [based on WWN} doesn't seem to be there any more). Fibre Channel makes widespread use of a mechanism similar to the latter at the target discovery level (nameserver based soft zoning, where the fabric nameserver only tells an Initiator about Targets it can access, rather than all the Targets it knows about). > or (just after they buy more storage) > > S: Ok, that URL has been changed to > "scsi://bobstorage.san3.acme.com:9003/". Please come again. This is a common computer science solution to a large set of problems; introduce another level of indirection ;-). While I understand the solution, I'm not at all clear on what the problem is and why it needs to be solved in this fashion. There's been a lot of traffic on this, have I missed something? The counterbalance on the other side of this issue is that iSCSI should fit into existing SCSI frameworks for storage discovery, access, and management (e.g., in operating systems). Every time a new sort of naming is introduced, that integration difficulty goes up considerably. The worked example is that Fibre Channel had to introduce WWNs or their equivalent for some fairly obvious functional reasons, but introducing WWNs to operating systems that started out only understanding parallel SCSI caused no end of pain and suffering. Something that's going to rear its ugly head here is discovery. SCSI discovery is fundamentally based on a "bus walker" paradigm in which low level logic does discovery based on a comprehensive search (i.e., try every possible target on the parallel bus). When Fibre Channel introduced WWNs, this broke the "try every possible target" approach (can't iterate through every 64 bit WWN and still boot quickly), and it was fixed by a multi-round distributed address assignment scheme for FC-AL (don't ask ;-) ) and the ability to query the nameserver for all known ports in FC-SW. FWIW, FC-SW tried to use LDAP for nameservice and gave up; something considerably simpler was implemented. Among the practical problems that resulted from WWN introduction is that booting over Fibre Channel took a while to get working because FC broke the typical "first LUN of first target contains a boot image" mechanism used in boot code. iSCSI has the opportunity to repeat the Fibre Channel experience of changing the way things work in a fashion that will take a while to work its way through systems. If this is done, it needs to be done for a good reason. One question to think about is "If all SCSI targets are named by URLs on the server, where do the URLs come from?" An answer of the form "yet another config file or set of registry entries" is an obstacle to adoption, because tools have to be written to manage these, and administrators have to learn how to use them, and ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Mon Oct 02 22:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17515; 2 Oct 2000 18:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA20061 for ips-outgoing; Mon, 2 Oct 2000 17:10:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA19780 for ; Mon, 2 Oct 2000 17:03:09 -0400 (EDT) Received: from amrelay2.boi.hp.com (amrelay2.boi.hp.com [15.56.8.41]) by atlrel2.hp.com (Postfix) with ESMTP id 3EB71974; Mon, 2 Oct 2000 17:03:05 -0400 (EDT) Received: from xboibrg2.boi.hp.com (xboibrg2.boi.hp.com [15.56.8.172]) by amrelay2.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id PAA09252; Mon, 2 Oct 2000 15:03:03 -0600 (MDT) Received: by xboibrg2.cv.hp.com with Internet Mail Service (5.5.2650.21) id <41AJD1H0>; Mon, 2 Oct 2000 15:03:01 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: remove recovery from transport-layer connection failur e(?) Date: Mon, 2 Oct 2000 14:47:36 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julian, If the scenario you point out is correct (a single command lasting for such a long time), then of course we need a mechanism where we can restart the command from the approximate point of failure. However that would be failures lasting for "more than a fraction of a sec". First of all, a TCP connection does not indicate a failure that quickly. Secondly, there are ways to recover from a path failure and still preserve a TCP connection in High-Availability environments. I am sure most system vendors would be implementing such techniques. Somesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Sunday, October 01, 2000 1:52 AM To: ips@ece.cmu.edu Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) Steph, Assume than in the new wonderfull SAN world you have started a disk-to-tape (or disk-to-disk) long third party copy. The SAN is fine and the copy proceeds for an hour but the lousy initiator-to-copy-manager link (on which accidentally no data transfer took place) fails for a fraction of a second. Should we restart the command under-the-cover or drop it or ask the parties to provide state information to a specific SCSI restart driver? And we can build many similar scenarios. I think that whatever we can do simplify exception handling we should do (the same arguments that hold for multiple connections hold here too). I would add that in Ideal world - I would like to have transport "splice" a new TCP connection with an old TCP connection but failing this to happen (again SCTP is doing it already or not?) we should take care that simple events like a cable taken-out in some obscure part of the network will only seldom affect higher layers. Julo Stephen Bailey on 27/09/2000 07:16:12 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) > Currently, iSCSI is spec'ed to recover from transport-layer > connection failures. > > The main motivation for this decision was to support tape backup > applications that are quite sensitive to any failures that get > propogated to their layer. > > So, perhaps we can remove the requirement of recovering from > transport-layer connection failures in iSCSI. This would simplify > the protocol somewhat. > > Thoughts? I'm all for eliminating command recovery. There seem to be several reasons advanced for command recovery. The first seems to be based upon an inappropriate analogy to FCP. Command recovery had to be added to FCP-2 because the FC layer is unreliable. A single dropped FC frame leads to a failed FCP command. This clearly upsets tape operation even when the link is performing nominally. In FCP, without command recovery, with some observable frequency, you will get an expected error that leads to complete, irrecoverable failure of a transfer stream. The other thing that makes FCP-2 command recovery work well is when you are doing a write, which is 90% (maybe it's 99%?) of tape operation, the target can return an early indication of most frame drops, rather than waiting for a timer to expire. TCP's reliability solves this problem in another way. By the time you get a TCP connection failure, you have already exhausted a set of reliability mechanisms which guarantee, with high certainty, that further data can not be transferred between the two endpoints. `the two endpoints' phrase suggests the other reason advanced for command recovery. That is, to permit path failover for commands which are not idempotent, such as tape write sequential. The problem with this, is that it is not clear HOW iSCSI command recovery can actually work properly, given a TCP connection failure indication. It takes a long time for a TCP connection to fail, and by that time, I'm not sure recovery would reasonably be possible. Perhaps I'm in error on this assumption. Can a tape guru (Joe from Exabyte?) comment on whether recovery would be possible after many seconds (tens, hundreds) have elapsed? The SCSI layer has never been solely responsible for ensuring reliable backup. Macro scale things go wrong with tape (run off the end, get eaten, etc..) with relatively high frequency. A low level backup engine like tar or dump will fail on a SCSI error, and that's OK. There must also be a higher level software component like Amanda, which manages retries, including operator intervention, to ensure reliable backup. It seems like whether iSCSI has a command recovery mechanism should be a function of whether somebody can stand up and say for sure that it solves a real problem. So far it only seems like it MIGHT solve a problem. Who can say `this solves MY problem!'? Steph From owner-ips@ECE.cmu.edu Mon Oct 02 22:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17690; 2 Oct 2000 18:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA22499 for ips-outgoing; Mon, 2 Oct 2000 18:22:51 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA22493 for ; Mon, 2 Oct 2000 18:22:45 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id PAA15860; Mon, 2 Oct 2000 15:08:50 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA19213; Mon, 2 Oct 2000 15:06:38 -0700 From: "Douglas Otis" To: "GUPTA,SOMESH \(HP-Cupertino,ex1\)" , , Subject: RE: iSCSI: remove recovery from transport-layer connection failure(?) Date: Mon, 2 Oct 2000 15:25:15 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, Are you referring to 802.1Q, 802.1D, or HSRP or just EtherChannel in general? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > GUPTA,SOMESH (HP-Cupertino,ex1) > Sent: Monday, October 02, 2000 1:48 PM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: iSCSI: remove recovery from transport-layer connection > failure(?) > > > Julian, > > If the scenario you point out is correct (a single command lasting > for such a long time), then of course we need a mechanism where > we can restart the command from the approximate point of failure. > However that would be failures lasting for "more than a fraction > of a sec". > > First of all, a TCP connection does not indicate a failure that > quickly. Secondly, there are ways to recover from a path failure > and still preserve a TCP connection in High-Availability environments. > I am sure most system vendors would be implementing such techniques. > > Somesh > > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Sunday, October 01, 2000 1:52 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: remove recovery from transport-layer connection > failure(?) > > > > > Steph, > > Assume than in the new wonderfull SAN world you have started a > disk-to-tape > (or disk-to-disk) long third party copy. The SAN is fine and the copy > proceeds for an hour > but the lousy initiator-to-copy-manager link (on which > accidentally no data > transfer took place) fails for a fraction of a second. > Should we restart the command under-the-cover or drop it or ask > the parties to provide state information to a specific SCSI > restart driver? > > And we can build many similar scenarios. > > I think that whatever we can do simplify exception handling we should do > (the same arguments that hold for multiple connections hold here too). > > I would add that in Ideal world - I would like to have transport > "splice" a > new TCP > connection with an old TCP connection but failing this to happen (again > SCTP is doing it already or not?) we should take care that simple events > like a cable taken-out > in some obscure part of the network will only seldom affect higher layers. > > Julo > > Stephen Bailey on 27/09/2000 07:16:12 > > Please respond to Stephen Bailey > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: iSCSI: remove recovery from transport-layer connection > failure(?) > > > > > > Currently, iSCSI is spec'ed to recover from transport-layer > > connection failures. > > > > The main motivation for this decision was to support tape backup > > applications that are quite sensitive to any failures that get > > propogated to their layer. > > > > So, perhaps we can remove the requirement of recovering from > > transport-layer connection failures in iSCSI. This would simplify > > the protocol somewhat. > > > > Thoughts? > > I'm all for eliminating command recovery. > > There seem to be several reasons advanced for command recovery. > > The first seems to be based upon an inappropriate analogy to FCP. > Command recovery had to be added to FCP-2 because the FC layer is > unreliable. A single dropped FC frame leads to a failed FCP command. > This clearly upsets tape operation even when the link is performing > nominally. In FCP, without command recovery, with some observable > frequency, you will get an expected error that leads to complete, > irrecoverable failure of a transfer stream. The other thing that > makes FCP-2 command recovery work well is when you are doing a write, > which is 90% (maybe it's 99%?) of tape operation, the target can > return an early indication of most frame drops, rather than waiting > for a timer to expire. > > TCP's reliability solves this problem in another way. By the time you > get a TCP connection failure, you have already exhausted a set of > reliability mechanisms which guarantee, with high certainty, that > further data can not be transferred between the two endpoints. > > `the two endpoints' phrase suggests the other reason advanced for > command recovery. That is, to permit path failover for commands which > are not idempotent, such as tape write sequential. The > problem with this, is that it is not clear HOW iSCSI command recovery > can actually work properly, given a TCP connection failure indication. > It takes a long time for a TCP connection to fail, and by that time, > I'm not sure recovery would reasonably be possible. Perhaps I'm in > error on this assumption. Can a tape guru (Joe from Exabyte?) comment > on whether recovery would be possible after many seconds (tens, > hundreds) have elapsed? > > The SCSI layer has never been solely responsible for ensuring reliable > backup. Macro scale things go wrong with tape (run off the end, get > eaten, etc..) with relatively high frequency. A low level backup > engine like tar or dump will fail on a SCSI error, and that's OK. > There must also be a higher level software component like Amanda, > which manages retries, including operator intervention, to ensure > reliable backup. > > It seems like whether iSCSI has a command recovery mechanism should be > a function of whether somebody can stand up and say for sure that it > solves a real problem. So far it only seems like it MIGHT solve a > problem. Who can say `this solves MY problem!'? > > Steph > From owner-ips@ECE.cmu.edu Mon Oct 02 22:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17685; 2 Oct 2000 18:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA21235 for ips-outgoing; Mon, 2 Oct 2000 17:44:33 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA21230 for ; Mon, 2 Oct 2000 17:44:27 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA15843; Mon, 2 Oct 2000 14:30:27 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA19110; Mon, 2 Oct 2000 14:28:15 -0700 From: "Douglas Otis" To: "Daniel Smith" Cc: , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 2 Oct 2000 14:46:51 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: <200010021924.MAA25676@catalina.almaden.ibm.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Daniel, > I'll reply to this piece-by-piece as there seem to be a lot of > misconceptions. > > (On a side note, I haven't seen any alternative proposals put > forward to the > URL scheme as noted. If things stay this way, then I'll put it forward as > an Internet-Draft.) > > Okay, off we go. > > Douglas Otis wrote: > > > > If you wished to crack data being sent via IP, could you ask > for a better > > tag of such information- scsi://[/modifier] being > sent in clear > > text down the same connection? Of course, the same is true for login as You base this comment assuming there is no other means. There is no reason for sending the text equivalent of the address within the transport. No equipment will require a text name for routing or delivery. Adding it to the SCSI transport weakens security. > will always be sent "in the clear" at some point on the > network. There's not a lot we can do about that. The IP address will > always be visible in all packets. (Insert comments here about > why VPNs are > a good thing here....) Again, this is not true. Again you assume that somehow sending a name is required. It is not. Communication to an authentication server could be completely secure with no clear text. Would you expect otherwise? > In an SSL link, or IPsec the [/modifier] part will be encrypted. Indeed, > whenever security is tight, the [/modifier] will only be sent after > clearance is granted. > > Why? > > well. Here again "who, what, and where" are nicely presented in easy to > > read form. Neither SCSI clients nor servers are likely able to > authenticate > > independently or retain textual information. Both the client and server > > should depend on some outside database. This compromise in security was > > If security is needed, then security is needed and there are no > short cuts! > Whether stored locally or remotely, that information needs to be > somewhere, > and securely accessible too. There is just being sloppy. Defer these parameters to a better designed server and thus keep the transport layer clean. > > > done to eliminate an IP selection option but then a textual means of > > including IP was included to ensure all modes of operation > remain. Rather > > than using a secure and opaque means to establish a connection using an > > independent means of authenticating on a different connection, > the entire > > function was included in this all encompassing draft. This was > not a good > > choice in my view. > > There is no requirement that only the Initiator and Target MUST > authenticate > between /themselves/ only. This may be desirable (in case the > authentication database machine goes down.) There are many papers > written on > this subject; for a simple introduction, see "Secure Distributed > Computing", > Sci. Am., Nov 1994 pp72--76. This will give you an idea of what > the minimum > requirements are in an open network authentication realm. Once you allow IPv4 or 6 within URL target naming conventions, it is odd to suggest that this scheme affords protection from changes to these standards. There are many means to guard against a server failure besides leaving the front door open. This would suggest targets will contain an entire authentication database. What protocol would exchange this information in a consistent fashion. Perhaps LDAP? > > By creating a separate draft to define an authentication database access > > (LDAP structures), options remain possible without impact on > the transport > > specification. > > URLs have very little impact on the transport. The iSCSI draft > just handles > arbitrary length text strings to identify resources---and that's all it > cares about. The only time interpretation of this is requred is for > 3rd-party commands, when we don't have a choice about it---and technically > the transport still doesn't care about it. We DO want it > standardized. We > DON'T want it tied to the transport. In 2003 when iSCSIv5 over SCTP with > Microsoft Active Super Directory v3 running over IPv8 on > terrabit-RS232, the > naming scheme should still be handled by the transport. You have indeed tied it to the SCSI transport with the use of IP text. You could remove all such information from the spec and thus ensure no future conflicts. Have the authentication process do the mapping and then nothing in the SCSI transport gets touched or exposed. Would you wish to go back and make changes to the transport spec to allow a means for target mapping with additional attributes? By planting this information within the transport spec ensures extensive changes as time marches. I seriously doubt that IPvX will have anything to do with any of these changes. > > The only aspect of authentication within the transport > > should be a cookie. Authentication provides access at some IP > and port or > > Please forgive my ignorance, but isn't a cookie an arbitrary length text > string (or can be mapped to and from an arbitrary length text string). > Cookies as /constant/ entities is an extremely bad idea. That's why we > don't recommend it. Okay, cookies are really small delicious disks, > sometimes with chocolate chips---what do you mean by cookie? It would be a binary digest of shared secrets good for leased use derived from the authentication server. I think you could use a few more chips in your cookie however. The means of fixing the spec is simply the removal of text based mapping. Requiring DNS lookup should be fun. How many seconds would you wait for a response? What if it is required to then authenticate with this third party? Now how long would you wait then? What a bad idea to place this as a part of the real time operation of a SCSI. I know, it is only optional... Then get rid of the option and specify a better means that does not include the SCSI transport. Sending these names in text is broken. Fix it by removing it from the transport specification. Doug > Daniel Smith. > > [Mmm. Cookies.] > -- > IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > From owner-ips@ECE.cmu.edu Mon Oct 02 23:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17703; 2 Oct 2000 18:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA21985 for ips-outgoing; Mon, 2 Oct 2000 18:07:37 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub1.almaden.ibm.com (mailhub1.almaden.ibm.com [198.4.83.44]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA21981 for ; Mon, 2 Oct 2000 18:07:33 -0400 (EDT) Received: from catalina.almaden.ibm.com (catalina.almaden.ibm.com [9.1.24.58]) by mailhub1.almaden.ibm.com (8.8.8/8.8.8) with ESMTP id PAA48214; Mon, 2 Oct 2000 15:06:37 -0700 Received: (from dfsmith@localhost) by catalina.almaden.ibm.com (AIX4.3/UCB 8.8.8/8.7) id PAA25894; Mon, 2 Oct 2000 15:07:01 -0700 From: Daniel Smith Message-Id: <200010022207.PAA25894@catalina.almaden.ibm.com> Subject: Re: iSCSI URL scheme To: Black_David@emc.com Date: Mon, 2 Oct 2000 15:07:01 -0700 (PDT) Cc: ips@ece.cmu.edu In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101040@corpmx9.isus.emc.com> from "Black_David@emc.com" at Oct 02, 2000 04:56:52 PM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Your concerns are quite valid---here's the reasoning though; and yes, it's a significant departure. But not too much. Black_David@emc.com wrote, quoting dfsmith@almaden.ibm.com: > > > > > I: Alright! I want to connect to > > "scsi://niceshark.acme.com/bobsstorage/san3?wwn=0x7b4d72658a7410db" > > This query is a *major* departure from SCSI practice at this level. > The SCSI approach would be more along the lines of: At this point, I will append to the conversation... S: You got it! > I: Alright! Show me all your LUNs that I can access. S: You've got LUN 3. I: What's the WWN of LUN 3? s: 0x7b4d72658a7410d On the other hand, had the the I requested "scsi://.../bobsstorage/san3" it might have received a bigger Request LUNs response. S: You've got LUNs 3, 5, 11, 28 and a bonus number 45. The example given was an SSP-like environment. Due to firewalling, all storage accounts were presented to the outside world through one address: niceshark.acme.com. This presents a problem if there are lots of storage accounts on niceshark---show me all your LUNs may return 10 million of the things! The alternative is to specify a subset (as done here in /bobsstorage/san3, in this particular case a WWN was also specified) /or/ to only present the LUNs that were authenticated as part of the login authentication. We (I?) prefer the former, because I would like to keep the service delivery port name and the authentication procedure separate. I can imagine where the same key can be used to access /bobsstorage/san2 and /bobstorage/san1 as well. > This is an authorization check that can be implemented by a combination > of accepting/denying the connection based on the client authentication > information and/or only reporting the LUNs to which the Initiator has access > (and after an exhaustive search, the Initiator may discover that a LUN it > used to have access to [based on WWN} doesn't seem to be there any more). > Fibre Channel makes widespread use of a mechanism similar to the latter at > the > target discovery level (nameserver based soft zoning, where the fabric > nameserver > only tells an Initiator about Targets it can access, rather than all the > Targets it > knows about). Just as a personal note, this seems rather complex and wasteful. Just as a mailing list contributor, this method is still fully supported. > > S: Ok, that URL has been changed to > > "scsi://bobstorage.san3.acme.com:9003/". Please come again. > > This is a common computer science solution to a large set of problems; > introduce another level of indirection ;-). While I understand the > solution, > I'm not at all clear on what the problem is and why it needs to be solved > in this fashion. There's been a lot of traffic on this, have I missed > something? In this case, management decided to change things without telling anybody. I, too, cannot fathom why this would be useful. B-P (And there hasn't been a lot of list traffic. It's done in the HTTP land all the time, and benefits the servers immensely; though not the users!) Seriously though, this is a management convenience. Consider this situation. You are the haggard and overworked storage administrator in charge of your department's storage needs. As part of your never-ending upgrade plan, you want to swap out those people who use too little storage (or access it too infrequently) to the cheaper server. There are 57 of them. Rather than emailing them to update their SCSI buses, you tell the main server to redirect them instead. (Naturally, you are still peeved that their silly custom installed operating systems don't bother to check the master database when connecting to their SCSI buses.) Whenever you have a `well known name' redirection is important. While FC gets by fine with a WWN, this is slightly different. Imagine, if you will, that all the SANs in the world have been joined together on the InterSAN. (It was Al Gore's initiative too.) To access a device you need a path (LUN) and an identifier (WWN). If the paths are dynamic, you cannot search for an identifier without querying a database of some sort (san.yahoo.com?). But if the database is large, unwieldy and not completely up to date (e.g., Yahoo, Altavista, the DNS servers...), an attempt to access a well known name may yield incorrect results. ("Well, WWN x was always at LUN y before---where has it gone, the database still says it's there.") I think I'll invent a new maxim here. "Any distributed root database of sufficient size is going to update more slowly than the leaf nodes." Has a nice ring to it. ... > for a good reason. One question to think about is "If all SCSI targets > are named by URLs on the server, where do the URLs come from?" > An answer of the form "yet another config file or set of registry entries" > is an obstacle to adoption, because tools have to be written to manage > these, and administrators have to learn how to use them, and ... All good points and I wish I could think of a really good scheme that has no adoption obstacles. If any of you have one, please post it. (Answers of the form "make the 8-byte WWN into a 1-byte path, 4-byte IP address, 2-byte port number and 1-byte id" please note that IPv6 is a 16-byte routing address scheme. And we want iSCSI to be ideally transport (TCP) agnostic, and definitely delivery (IP) agnostic.) Daniel Smith. -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 From owner-ips@ECE.cmu.edu Tue Oct 03 00:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17861; 2 Oct 2000 19:57 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA23460 for ips-outgoing; Mon, 2 Oct 2000 18:58:47 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mailhub1.almaden.ibm.com (mailhub1.almaden.ibm.com [198.4.83.44]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA23451 for ; Mon, 2 Oct 2000 18:58:42 -0400 (EDT) Received: from catalina.almaden.ibm.com (catalina.almaden.ibm.com [9.1.24.58]) by mailhub1.almaden.ibm.com (8.8.8/8.8.8) with ESMTP id PAA76720; Mon, 2 Oct 2000 15:57:46 -0700 Received: (from dfsmith@localhost) by catalina.almaden.ibm.com (AIX4.3/UCB 8.8.8/8.7) id PAA11996; Mon, 2 Oct 2000 15:58:11 -0700 From: Daniel Smith Message-Id: <200010022258.PAA11996@catalina.almaden.ibm.com> Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: dotis@sanlight.net (Douglas Otis) Date: Mon, 2 Oct 2000 15:58:11 -0700 (PDT) Cc: julian_satran@il.ibm.com, ips@ece.cmu.edu In-Reply-To: from "Douglas Otis" at Oct 02, 2000 02:46:51 PM X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > > You base this comment assuming there is no other means. There is no reason ... I aplogize. I'm afraid that I do not know how to respond to your questions and suggestions. I politely request that you propose a naming scheme that meets the minimum requirements, and also specify your assumptions, which seem to differ radically from mine. Then submit the proposal to peer review. The iSCSI Internet-Draft requires a text string to identify a service delivery port. My proposal (which is just a proposal, and was mulled over while the iSCSI draft was being written, with the knowledge of the design team) treats these strings as URLs. It is designed to cope with and blend the storage world with the network world. The Internet especially presents some challenges to the traditional storage notion of addressing. I do not pretend that the solution is a simple one, but neither is the problem simple. But iSCSI (the transport) is distinct from iSCSI (the URL naming scheme), at least until everyone agrees that it is the greatest, most frabulous, brabulous, zip-zoop dabulous way of naming stuff. My starting assumptions (broadly and somewhat imcompletely) are, a) The inter-network is controlled by someone who isn't me. b) I can make a network of my own that I do control, but that can't talk to other networks. c) The Internet is named using DNS, which maps to ephemeral IP addresses. The DNS is moderately, though not totally, secure. d) Firewalls are everywhere. On your outgoing connection and on the second party's incoming connection. Possibly in the middle as well. e) The network world has sufficiently many nuances/protocols/transports that no protocol can totally manage a discovery mechanism within the Internet. This means the naming must be flexible, and the (iSCSI) transport separable from the naming. f) No matter how much I scream and shout, the network administrators still aren't going to change infrastructure that already works for everyone else. g) The network infrastructure will gradually change after iSCSI becomes a noticeable consumer of bandwidth. h) SSPs are going to appear. (Have appeared?) So people are going to be running SCSI over the Internet before the network folks even notice. i) 3rd party commands will become more important as the SAN network increases in size. (Actually, this one is dubious. Let's just assume that some people regard 3-P-Cs of critical importance.) j) Any packet going over the Internet can have is source IP, destination IP and contents sniffed out by an observer. Hence all DNS names can be discovered by inspecting and deresolving the traffic. k) People using SSPs will be paranoid about security. l) SSPs will have clients numbering in the tens of thousands, and storage in the petabyte range. All going through one entry DNS name. m) Who was it who said that something good should be as simple as possible, and no simpler. (Einstein, Twain?) There are, of course, many more assumptions. These seemed the most pertinent. I am going to go into silent mode for a while. Just to see what happens, and if another contender for naming appears. Daniel Smith. -- IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 From owner-ips@ECE.cmu.edu Tue Oct 03 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa17874; 2 Oct 2000 20:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA24256 for ips-outgoing; Mon, 2 Oct 2000 19:29:16 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA24251 for ; Mon, 2 Oct 2000 19:29:10 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id QAA15882; Mon, 2 Oct 2000 16:15:15 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id QAA19383; Mon, 2 Oct 2000 16:13:03 -0700 From: "Douglas Otis" To: "Daniel Smith" Cc: , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 2 Oct 2000 16:31:40 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: <200010021924.MAA25676@catalina.almaden.ibm.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Daniel, While you are right about the Internet using DNS to find public IPs and systems will automatically resolve a name to an IP using DNS unless already entered into the routing table, it is not right to assume this information must be include within a SCSI transport. Developing what could be described as LDAP or a Web CGI and at the same time contend it to be the only means to get the job done is redeveloping these servers within the SCSI transport. A much smaller effort could extend inter-operability with already prevalent servers understood by IT. The means of getting a browser, mail program, and may other network devices configured within an enterprise environment is with an LDAP server. Rather than reinventing the Web CGI or LDAP server and cramming it into a SCSI transport, split it out of the transport layer. Identifying a connection is a small price to pay. In addition, it would provide a higher level of control rather than depending on DNS to provide a point of entry. It is not surprizing to see DNS take a day to update. If you had a problem with your router flapping, even longer. You would want a scheme that the IT manager can control and not a new database you just invented and wedged into a target device. Doug > -----Original Message----- > From: Daniel Smith [mailto:dfsmith@almaden.ibm.com] > Sent: Monday, October 02, 2000 12:25 PM > To: Douglas Otis > Cc: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > I'll reply to this piece-by-piece as there seem to be a lot of > misconceptions. > > (On a side note, I haven't seen any alternative proposals put > forward to the > URL scheme as noted. If things stay this way, then I'll put it forward as > an Internet-Draft.) > > Okay, off we go. > > Douglas Otis wrote: > > > > If you wished to crack data being sent via IP, could you ask > for a better > > tag of such information- scsi://[/modifier] being > sent in clear > > text down the same connection? Of course, the same is true for login as > > will always be sent "in the clear" at some point on the > network. There's not a lot we can do about that. The IP address will > always be visible in all packets. (Insert comments here about > why VPNs are > a good thing here....) > > In an SSL link, or IPsec the [/modifier] part will be encrypted. Indeed, > whenever security is tight, the [/modifier] will only be sent after > clearance is granted. > > Why? > > Let's imagine a large array of fast, reliable and featuresome storage---an > IBM Shark box say. B-) This box has many user accounts, and is being used > in the context of a storage service provider. The login sequence might go > something like this.... > > I: Hello Shark---I'm an initiator. My authentication ID is "Robert Smith, > Seattle, next to the donut store, hjhhaskjanbt112kg9d988". > > S: Hello initiator. I'll be your server---we need a secure connection: > switch to "hjcnneamchjsdkfsdsd394njsdf89423". (Trivial example.) > > I: Alright! I want to connect to > "scsi://niceshark.acme.com/bobsstorage/san3?wwn=0x7b4d72658a7410db" > > Then we have a choice... > > S: Great---off you go and have a nice day. > > or > > S: Sorry---you haven't got permission to use that any more/it > doesn't exist. > > or (just after they buy more storage) > > S: Ok, that URL has been changed to > "scsi://bobstorage.san3.acme.com:9003/". Please come again. > > > well. Here again "who, what, and where" are nicely presented in easy to > > read form. Neither SCSI clients nor servers are likely able to > authenticate > > independently or retain textual information. Both the client and server > > should depend on some outside database. This compromise in security was > > If security is needed, then security is needed and there are no > short cuts! > Whether stored locally or remotely, that information needs to be > somewhere, > and securely accessible too. > > > done to eliminate an IP selection option but then a textual means of > > including IP was included to ensure all modes of operation > remain. Rather > > than using a secure and opaque means to establish a connection using an > > independent means of authenticating on a different connection, > the entire > > function was included in this all encompassing draft. This was > not a good > > choice in my view. > > There is no requirement that only the Initiator and Target MUST > authenticate > between /themselves/ only. This may be desirable (in case the > authentication database machine goes down.) There are many papers > written on > this subject; for a simple introduction, see "Secure Distributed > Computing", > Sci. Am., Nov 1994 pp72--76. This will give you an idea of what > the minimum > requirements are in an open network authentication realm. > > > By creating a separate draft to define an authentication database access > > (LDAP structures), options remain possible without impact on > the transport > > specification. > > URLs have very little impact on the transport. The iSCSI draft > just handles > arbitrary length text strings to identify resources---and that's all it > cares about. The only time interpretation of this is requred is for > 3rd-party commands, when we don't have a choice about it---and technically > the transport still doesn't care about it. We DO want it > standardized. We > DON'T want it tied to the transport. In 2003 when iSCSIv5 over SCTP with > Microsoft Active Super Directory v3 running over IPv8 on > terrabit-RS232, the > naming scheme should still be handled by the transport. > > > The only aspect of authentication within the transport > > should be a cookie. Authentication provides access at some IP > and port or > > Please forgive my ignorance, but isn't a cookie an arbitrary length text > string (or can be mapped to and from an arbitrary length text string). > Cookies as /constant/ entities is an extremely bad idea. That's why we > don't recommend it. Okay, cookies are really small delicious disks, > sometimes with chocolate chips---what do you mean by cookie? > > > perhaps a set of such locations, it could also provide > third-party mapping > > in binary as well. (This mapping could be encoded by various > schemes.) As > > other transports are employed, flexibility becomes important. > Why insist > > targets do DNS lookup? Keep the workload light as such lookups > should never > > We don't. Where did you get this idea? If it found its way into > the drafts > then please point it out so that we can remove it! Target DNS > resolution is > always optional; though sometimes (perhaps often in the Shark > example) it is > useful, and perhaps required by the system designer. Initiator DNS > resolution is mandatory. (Except in the case where an initiator > is allowed > interpret, say, "scsi://123.45.67.89".) > > > be needed beyond the process of authentication rightfully done by other > > equipment not involved in the time critical data handling. I > hope this is > > comprehendible this time. > > It would be more comprehensible if you gave an example of where the scheme > as currently proposed fails. Then we can work to fix it and > produce a more > baked Internet-Draft for interpreting iSCSI service delivery port names. > > Daniel Smith. > > [Mmm. Cookies.] > -- > IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > From owner-ips@ECE.cmu.edu Tue Oct 03 01:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18130; 2 Oct 2000 21:25 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA25733 for ips-outgoing; Mon, 2 Oct 2000 20:23:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1tx.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA25728 for ; Mon, 2 Oct 2000 20:22:56 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1tx.cos.agilent.com (Postfix) with ESMTP id B1F183A1; Mon, 2 Oct 2000 18:22:55 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id A95912D; Mon, 2 Oct 2000 20:22:54 -0400 (EDT) Received: from agilent.com (cos1nai251099.cs.itc.hp.com [15.3.251.99]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id RAA20581; Mon, 2 Oct 2000 17:22:51 -0700 (PDT) Message-ID: <39D926DA.E63268CC@agilent.com> Date: Mon, 02 Oct 2000 17:22:50 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Black_David@emc.com, ips@ece.cmu.edu Subject: Re: iSCSI sessions: Let's try again References: <0F31E5C394DAD311B60C00E029101A070410102E@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Black_David@emc.com wrote: > Summarizing the above discussion: > > [Consensus Attempt 1] I don't see anything that REQUIRES > two TCP connections in all iSCSI sessions, because I haven't > seen any TCP window closure scenarios [3] that aren't avoidable > by reasonable design/engineering of the target. I therefore > propose to re-establish the WG consensus that multiple TCP > connections per iSCSI session is OPTIONAL. Reasons requiring 2 tcp connections per iSCSI session (one for commands the other for data): 1- It allows a target to temporarily flow control commands from the initiator while still allowing data to continue on the "data" connection (allowing commands to complete so that the flow control can be opened again). 2- It allows commands to be received without being stuck behind a large data transfer, allowing the target to begin command execution sooner. 3- if "Consensus Attempt 2" is allowed, it provides a single "model" of iSCSI while allowing more than one data connection. It also simplifies the "command ordering" issues of the symetric model (sending commands on any connection). -Matt Wakeley Agilent Technologies From owner-ips@ECE.cmu.edu Tue Oct 03 01:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18135; 2 Oct 2000 21:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA25790 for ips-outgoing; Mon, 2 Oct 2000 20:25:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA25782 for ; Mon, 2 Oct 2000 20:24:58 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Mon, 2 Oct 2000 17:27:12 -0700 Message-ID: From: Joshua Tseng To: ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 2 Oct 2000 17:27:12 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, >You base this comment assuming there is no other means. There is no reason >for sending the text equivalent of the address within the transport. No >equipment will require a text name for routing or delivery. Adding it to the >SCSI transport weakens security. What about for proxies? A proxy that routes storage data between IP addressing domains will need the original text name to resolve it to a different IP address in a different address domain. NAT is quite common nowadays, and I would think iSCSI needs to preserve mechanisms that allow it to route between IP address domains. As far as the security, is your problem with the ASCII text format itself being readable by a human? If so, I see no difference between using text format compared to "asdj2342zssd" (or whatever), in addressing a remote LUN, as far as security is concerned. On the other hand, if you believe ALL interactions between initiator & target must reference an SPI found in a secure remote authentication server, then IMO you are placing severe and unreasonable deployment requirements on the user. It reminds me of the LANE architecture in ATM--quite unscalable and burdensome. >There are many means to guard against a server failure besides leaving the >front door open. This would suggest targets will contain an entire >authentication database. What protocol would exchange this information in a >consistent fashion. Perhaps LDAP? I see no way around a local authentication database in the target. If the target doesn't track which initiators can do what, then it will have to ask the remote authentication server, which affects performance. >It would be a binary digest of shared secrets good for leased use derived >from the authentication server. I think you could use a few more chips in >your cookie however. The means of fixing the spec is simply the removal of >text based mapping. Requiring DNS lookup should be fun. How many seconds >would you wait for a response? What if it is required to then authenticate >with this third party? Now how long would you wait then? What a bad idea >to place this as a part of the real time operation of a SCSI. I know, it is >only optional... Then get rid of the option and specify a better means that >does not include the SCSI transport. Sending these names in text is broken. >Fix it by removing it from the transport specification. I see no reason why a DNS lookup would be more costly or than an exchange of shared secrets with the authentication server. If anything, the shared secret exchange is more costly, since it must be secured. Josh -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Monday, October 02, 2000 2:47 PM To: Daniel Smith Cc: julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Daniel, > I'll reply to this piece-by-piece as there seem to be a lot of > misconceptions. > > (On a side note, I haven't seen any alternative proposals put > forward to the > URL scheme as noted. If things stay this way, then I'll put it forward as > an Internet-Draft.) > > Okay, off we go. > > Douglas Otis wrote: > > > > If you wished to crack data being sent via IP, could you ask > for a better > > tag of such information- scsi://[/modifier] being > sent in clear > > text down the same connection? Of course, the same is true for login as You base this comment assuming there is no other means. There is no reason for sending the text equivalent of the address within the transport. No equipment will require a text name for routing or delivery. Adding it to the SCSI transport weakens security. > will always be sent "in the clear" at some point on the > network. There's not a lot we can do about that. The IP address will > always be visible in all packets. (Insert comments here about > why VPNs are > a good thing here....) Again, this is not true. Again you assume that somehow sending a name is required. It is not. Communication to an authentication server could be completely secure with no clear text. Would you expect otherwise? > In an SSL link, or IPsec the [/modifier] part will be encrypted. Indeed, > whenever security is tight, the [/modifier] will only be sent after > clearance is granted. > > Why? > > well. Here again "who, what, and where" are nicely presented in easy to > > read form. Neither SCSI clients nor servers are likely able to > authenticate > > independently or retain textual information. Both the client and server > > should depend on some outside database. This compromise in security was > > If security is needed, then security is needed and there are no > short cuts! > Whether stored locally or remotely, that information needs to be > somewhere, > and securely accessible too. There is just being sloppy. Defer these parameters to a better designed server and thus keep the transport layer clean. > > > done to eliminate an IP selection option but then a textual means of > > including IP was included to ensure all modes of operation > remain. Rather > > than using a secure and opaque means to establish a connection using an > > independent means of authenticating on a different connection, > the entire > > function was included in this all encompassing draft. This was > not a good > > choice in my view. > > There is no requirement that only the Initiator and Target MUST > authenticate > between /themselves/ only. This may be desirable (in case the > authentication database machine goes down.) There are many papers > written on > this subject; for a simple introduction, see "Secure Distributed > Computing", > Sci. Am., Nov 1994 pp72--76. This will give you an idea of what > the minimum > requirements are in an open network authentication realm. Once you allow IPv4 or 6 within URL target naming conventions, it is odd to suggest that this scheme affords protection from changes to these standards. There are many means to guard against a server failure besides leaving the front door open. This would suggest targets will contain an entire authentication database. What protocol would exchange this information in a consistent fashion. Perhaps LDAP? > > By creating a separate draft to define an authentication database access > > (LDAP structures), options remain possible without impact on > the transport > > specification. > > URLs have very little impact on the transport. The iSCSI draft > just handles > arbitrary length text strings to identify resources---and that's all it > cares about. The only time interpretation of this is requred is for > 3rd-party commands, when we don't have a choice about it---and technically > the transport still doesn't care about it. We DO want it > standardized. We > DON'T want it tied to the transport. In 2003 when iSCSIv5 over SCTP with > Microsoft Active Super Directory v3 running over IPv8 on > terrabit-RS232, the > naming scheme should still be handled by the transport. You have indeed tied it to the SCSI transport with the use of IP text. You could remove all such information from the spec and thus ensure no future conflicts. Have the authentication process do the mapping and then nothing in the SCSI transport gets touched or exposed. Would you wish to go back and make changes to the transport spec to allow a means for target mapping with additional attributes? By planting this information within the transport spec ensures extensive changes as time marches. I seriously doubt that IPvX will have anything to do with any of these changes. > > The only aspect of authentication within the transport > > should be a cookie. Authentication provides access at some IP > and port or > > Please forgive my ignorance, but isn't a cookie an arbitrary length text > string (or can be mapped to and from an arbitrary length text string). > Cookies as /constant/ entities is an extremely bad idea. That's why we > don't recommend it. Okay, cookies are really small delicious disks, > sometimes with chocolate chips---what do you mean by cookie? It would be a binary digest of shared secrets good for leased use derived from the authentication server. I think you could use a few more chips in your cookie however. The means of fixing the spec is simply the removal of text based mapping. Requiring DNS lookup should be fun. How many seconds would you wait for a response? What if it is required to then authenticate with this third party? Now how long would you wait then? What a bad idea to place this as a part of the real time operation of a SCSI. I know, it is only optional... Then get rid of the option and specify a better means that does not include the SCSI transport. Sending these names in text is broken. Fix it by removing it from the transport specification. Doug > Daniel Smith. > > [Mmm. Cookies.] > -- > IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > From owner-ips@ECE.cmu.edu Tue Oct 03 01:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18179; 2 Oct 2000 21:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA26912 for ips-outgoing; Mon, 2 Oct 2000 21:05:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA26906 for ; Mon, 2 Oct 2000 21:05:30 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id UAA56310 for ; Mon, 2 Oct 2000 20:52:48 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e9315Td54644 for ; Mon, 2 Oct 2000 19:05:29 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: Bidirectional SCSI commands and iSCSI To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Mon, 2 Oct 2000 18:01:27 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/02/2000 07:05:28 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Jim Hafner, Since one of the things we are trying to do is not stray too far from the CDB structure which FC uses, do you or anyone else on the list know how FCP is structuring their CDB for this feature? . . . John L. Hufferd julian_satran@il.ibm.com@ece.cmu.edu on 10/02/2000 11:02:29 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, I'll take a look at the proposals (I assume they are on the T10 site). My concern about cluttering is related to the attempt to make a fixed-length header as I think that software initiators will be widely used and they are far more efficient with fixed length headers for widely used operations. Thanks, Julo "Jim Hafner/Almaden/IBM" on 02/10/2000 18:22:40 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: Subject: Re: Bidirectional SCSI commands and iSCSI Julo, True, those examples are not true duplex operations. These are only the current thinking or motivation for bi-directional commands. They are not the end of the story, I'm sure. However, the only SAM-2 requirement on bi-directional commands (as now defined) is that DataOut buffer be ready at the time the command is issued (so the target can get/request that data at any time after receipt of the command). In other words, the content of the DataOut buffer should not be time or data-dependent on the contents of the DataIn buffer. This means that as far as SAM-2 is concerned, there is no problem with a true duplex operation. (We just don't have any examples at the moment.) I think your second proposal (using "immediate" data) is only attempting to address the specific examples (OSDs), and not the generic notion of bi-directional commands. How would that solution work if the DataOut buffer was a 1MG chunk of user data for a XDREADWRITE command? The first method (splitting, if I read it correctly) undercuts the motivation that is generating this bi-d command in the first place. The SCSI layer could do that as well, but the issue is one of atomicity of command. (Aside: one principle governing definitions of bi-directional commands in T10 is that there must always be a two (or more) uni-directional variant of the command, to allow a complex operation to be performed on a traditional uni-directional with simpler steps. I read your suggestion as having iSCSI do this for the SCSI layer -- there'd be no point. Besides, doesn't this break layering?). Finally, I would suggest a look at the FCP-2/3 proposal just submitted. I think they are proposing (for legacy reasons) a new IU for bi-directional commands. If you're concerned about cluttering your header, then this group might think about adding a new PDU for these types of commands. I don't quite understand the reluctance to add new fields in the header for this purpose. The current design is not yet cast in stone and now is the perfect opportunity to get it defined in a way that accomodates the future that T10 seems to be moving (rapidly) towards. Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 10-01-2000 12:16:06 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, Both the examples you gave have something peculiar - the operations are to executed really in sequence (we are no talking about full duplex I/O operations?). I any case, in order no to extend excessively the headers we can either introduce a "next header" field (as in IP V6) or split the operation at the iSCSI level into two phases - each carrying its parameters and each ended with a status - like: - out XOP1, XOP2, OD1, -in S1, ID, S2 In any case commands having only a parameter set (fixed or variable) - like those for the object disk should be handled with the parameters sent as immediate data (as with the three party commands) - parameter size will probably be limited and the performance requirement will be similar to those of block read/write. Do we need for those to care also about residual counts for parameters? Julo "Jim Hafner/Almaden/IBM" on 29/09/2000 00:21:32 Please respond to "Jim Hafner/Almaden/IBM" To: Julian Satran/Haifa/IBM@IBMIL cc: ips@ece.cmu.edu Subject: Re: Bidirectional SCSI commands and iSCSI Julo, The primary motivation is for object-based storage devices (OSDs). The protocol that seems to be needed in that environment is the ability to send a CDB, plus some parameter data (DataOut) AND get back read-type data (DataIn) in the same command sequence. One (non-OSD) command has been proposed in T10 for an XOR type XDREADWRITE command. SAM-2 has now allowed for such commands in the architecture. From this perspective, it is completely symmetric. Transports that support this feature will need to have data structures in there PDUs or IUs (or whatever) to indicate things like DataOut datalength, DataIn datalength, possible virtual addresses for each buffer, etc. In other words, there will potentially be two active buffers (on each end) for use with the same command. So, e.g., iSCSI PDU may need to have both "DataIn expected data transfer length" and "DataOut expected transfer length" fields. I'm assuming that the data direction in the existing iSCSI PDU is derived from the SCSI opcode, unlike FCP where there are fields to indicate the data direction. (I am surprised not to find such things in the iSCSI Command PDU.) There are two proposals in T10 to enable this feature in SPI-4 and in FCP-x (probably FCP-3). Jim Hafner julian_satran@il.ibm.com@ece.cmu.edu on 09-28-2000 12:10:53 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI Jim, What is the intended use? Are there already commands that use it? Is there a major/minor distinction - like output data & input results or output parameters & input data - or is it intended to be completely symmetric? Julo "Jim Hafner/Almaden/IBM" on 20/09/2000 17:56:27 Please respond to "Jim Hafner/Almaden/IBM" To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Bidirectional SCSI commands and iSCSI Folks, In case you missed this, T10 just approved changes to SAM-2 (rev14) which remove the restriction that a single SCSI command can have a data flow in only one direction (DataIn xor DataOut). Transport specs (that want to allow this) will need to change to allow for two sets of Data buffer fields in the headers (length, over/underrun, etc.). Is anybody looking to modify the iSCSI spec to get these concepts supported? Jim Hafner From owner-ips@ECE.cmu.edu Tue Oct 03 01:50:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18184; 2 Oct 2000 21:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA26965 for ips-outgoing; Mon, 2 Oct 2000 21:07:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1tx.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA26961 for ; Mon, 2 Oct 2000 21:07:20 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1tx.cos.agilent.com (Postfix) with ESMTP id F3F5B314; Mon, 2 Oct 2000 19:07:19 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 159CCEA; Mon, 2 Oct 2000 21:07:19 -0400 (EDT) Received: from agilent.com (cos1nai251099.cs.itc.hp.com [15.3.251.99]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id SAA25829; Mon, 2 Oct 2000 18:07:15 -0700 (PDT) Message-ID: <39D93143.C6E3B62B@agilent.com> Date: Mon, 02 Oct 2000 18:07:15 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis , IPS Reflector Subject: Re: A Transport Protocol Without ACK (resend) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, What good does it to do bring up "buffer to buffer" flow control when the ethernet does not have such a thing? If all commands on an iscsi session were sent on their own TCP connection separate from data, then if the SCSI fifo becomes full, it will stop reading out of TCP, which will close the window on the sender, which will stop commands. That's end-to-end. -Matt Douglas Otis wrote: > Matt, > > You would not want an End-to-End scheme. You would want a scheme that > prevents any node or sub-node from over-flowing. That is accomplished by > Buffer-to-Buffer control. The present scheme now in place is with a rough > approximation of command space with any excess data discarded as required. > This makes for a lossy channel and one that does not allow control at a > specific sub-node that may be at its limits. Restrain the depth of the > sub-node with specific control. Just 8M-byte of FIFO will represent 100 > millisecond of delay at FC speeds. A token credit scheme ensures no node > ever gets to a point of dropping frames in addition to allowing maintenance > of the FIFO depth. Should a sub-node fail, only that node stops functioning > rather than everything being hung until resolution. A sizable short-coming > among many others in aggregating everything into this common scheme. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Matt Wakeley > > Sent: Monday, October 02, 2000 10:44 AM > > To: ips@ece.cmu.edu > > Subject: Re: A Transport Protocol Without ACK (resend) > > > > > > Douglas Otis wrote: > > > > > I agree, XON/XOFF would not work. A credit token scheme > > similar to FC Class > > > 3 would. > > > > FC class 3 does not have an end to end "credit token scheme". It > > (class 3 bb > > credit) is simply a credit mechanism between a node and it's neighbor node > > (the > > switch port) on whether the node can receive a frame. This means that the > > fabric has buffering inside it, and will hold onto a frame if the > > destination > > node does not have room for it. Thus, if the receiving node > > temporarily runs > > out of buffers, it will withold credit from the switch port. The > > sending node > > doesn't know that the receiving node has run out of buffers, so the fabric > > buffers fill up with undeliverable frames. > > > > -Matt > > From owner-ips@ECE.cmu.edu Tue Oct 03 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18358; 2 Oct 2000 22:59 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA28441 for ips-outgoing; Mon, 2 Oct 2000 21:56:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA28435 for ; Mon, 2 Oct 2000 21:56:20 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id SAA15931; Mon, 2 Oct 2000 18:42:19 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id SAA19754; Mon, 2 Oct 2000 18:40:06 -0700 From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: A Transport Protocol Without ACK (resend) Date: Mon, 2 Oct 2000 18:58:45 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: <39D93143.C6E3B62B@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, You are imagining the end of the Ethernet cable is connected to a mythical beast that actually contains the sum of all storage. In reality, there will be many cables heading out of this hydra each acting as a buffer to yet another buffer. The end of the Ethernet cable is not the end for the packet. Should one such cable be allowed to consume all buffer space? It would not be a good design to assume IP flow control will maintain buffer depth to these sub-nodes. Stopping flow is truly the end should it be that a cable was unplugged. This design will also ensure everything runs at the slowest cable rate. Doug > -----Original Message----- > From: Matt Wakeley [mailto:matt_wakeley@agilent.com] > Sent: Monday, October 02, 2000 6:07 PM > To: Douglas Otis; IPS Reflector > Subject: Re: A Transport Protocol Without ACK (resend) > > > Doug, > > What good does it to do bring up "buffer to buffer" flow control when the > ethernet does not have such a thing? > > If all commands on an iscsi session were sent on their own TCP connection > separate from data, then if the SCSI fifo becomes full, it will > stop reading out > of TCP, which will close the window on the sender, which will > stop commands. > That's end-to-end. > > -Matt > > Douglas Otis wrote: > > > Matt, > > > > You would not want an End-to-End scheme. You would want a scheme that > > prevents any node or sub-node from over-flowing. That is > accomplished by > > Buffer-to-Buffer control. The present scheme now in place is > with a rough > > approximation of command space with any excess data discarded > as required. > > This makes for a lossy channel and one that does not allow control at a > > specific sub-node that may be at its limits. Restrain the depth of the > > sub-node with specific control. Just 8M-byte of FIFO will represent 100 > > millisecond of delay at FC speeds. A token credit scheme > ensures no node > > ever gets to a point of dropping frames in addition to allowing > maintenance > > of the FIFO depth. Should a sub-node fail, only that node > stops functioning > > rather than everything being hung until resolution. A sizable > short-coming > > among many others in aggregating everything into this common scheme. > > > > Doug > > > > > -----Original Message----- > > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > > Matt Wakeley > > > Sent: Monday, October 02, 2000 10:44 AM > > > To: ips@ece.cmu.edu > > > Subject: Re: A Transport Protocol Without ACK (resend) > > > > > > > > > Douglas Otis wrote: > > > > > > > I agree, XON/XOFF would not work. A credit token scheme > > > similar to FC Class > > > > 3 would. > > > > > > FC class 3 does not have an end to end "credit token scheme". It > > > (class 3 bb > > > credit) is simply a credit mechanism between a node and it's > neighbor node > > > (the > > > switch port) on whether the node can receive a frame. This > means that the > > > fabric has buffering inside it, and will hold onto a frame if the > > > destination > > > node does not have room for it. Thus, if the receiving node > > > temporarily runs > > > out of buffers, it will withold credit from the switch port. The > > > sending node > > > doesn't know that the receiving node has run out of buffers, > so the fabric > > > buffers fill up with undeliverable frames. > > > > > > -Matt > > > > From owner-ips@ECE.cmu.edu Tue Oct 03 03:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18352; 2 Oct 2000 22:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA29281 for ips-outgoing; Mon, 2 Oct 2000 22:26:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA29271 for ; Mon, 2 Oct 2000 22:26:08 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id TAA15940; Mon, 2 Oct 2000 19:12:06 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id TAA19828; Mon, 2 Oct 2000 19:09:54 -0700 From: "Douglas Otis" To: "Joshua Tseng" , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 2 Oct 2000 19:28:31 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Joshua Tseng > Sent: Monday, October 02, 2000 5:27 PM > To: ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Doug, > > >You base this comment assuming there is no other means. There > is no reason > >for sending the text equivalent of the address within the transport. No > >equipment will require a text name for routing or delivery. Adding it to > the > >SCSI transport weakens security. > > What about for proxies? A proxy that routes storage data between IP > addressing domains will need the original text name to resolve it to > a different IP address in a different address domain. NAT is quite > common nowadays, and I would think iSCSI needs to preserve mechanisms > that allow it to route between IP address domains. Sorry, but that is not how a NAT works. Yes NAT represents a problem only from the perspective of the target. The target should know what goes where and who is authorized. Once the client pokes through the NAT and identifies in an opaque manner, end of problem. A NAT does not use names, however your LDAP server may provide a name to your browser for the purpose of finding the gateway. This name may only resolve locally within your enterprise and not publicly as well. Should you be attempting to connect over the Big I to your drives, then you will need to access the providers authentication database that would include the mapping you desire. This mapping would not be text but rather binary information. The retrieval of this information may be symbolic but those symbols would never touch the SCSI layer. > As far as the security, is your problem with the ASCII text format > itself being readable by a human? If so, I see no difference between > using text format compared to "asdj2342zssd" (or whatever), in > addressing a remote LUN, as far as security is concerned. On the > other hand, if you believe ALL interactions between initiator & target > must reference an SPI found in a secure remote authentication server, > then IMO you are placing severe and unreasonable deployment > requirements on the user. It reminds me of the LANE architecture > in ATM--quite unscalable and burdensome. Placing the authentication server within the SCSI transport does not improve the ability to scale. If anything, the authentication will occupy resources needed for the retrieval of data. Let a server specifically designed to provide secure connections and access to the client and server perform that function. It makes no sense to conclude it becomes difficult to delegate this function to a server already up and running. > > > >There are many means to guard against a server failure besides > leaving the > >front door open. This would suggest targets will contain an entire > >authentication database. What protocol would exchange this > information in > a > >consistent fashion. Perhaps LDAP? > > I see no way around a local authentication database in the target. > If the target doesn't track which initiators can do what, then it > will have to ask the remote authentication server, which affects > performance. Local? You would use the local LDAP server to communicate with the SSP LDAP server. What problem do you see? Each server would control the domain of each entity. Sounds like a good thing. Only the Authentication server would need to expose their IP via DNS and then emit private IPs to ensure the load is balanced and the correct alternative routes are assigned. You should not trust that to DNS. > > > >It would be a binary digest of shared secrets good for leased use derived > >from the authentication server. I think you could use a few > more chips in > >your cookie however. The means of fixing the spec is simply the > removal of > >text based mapping. Requiring DNS lookup should be fun. How > many seconds > >would you wait for a response? What if it is required to then > authenticate > >with this third party? Now how long would you wait then? What > a bad idea > >to place this as a part of the real time operation of a SCSI. I know, it > is > >only optional... Then get rid of the option and specify a better means > that > >does not include the SCSI transport. Sending these names in text is > broken. > >Fix it by removing it from the transport specification. > > I see no reason why a DNS lookup would be more costly or than > an exchange of shared secrets with the authentication server. If > anything, the shared secret exchange is more costly, since > it must be secured. Yes, but not by the SCSI equipment. Expensive for a seldom used server. Not at all expensive for the SCSI transport. The SCSI transport remains dead simple. Adding an authentication server to the SCSI transport together with a naming database is not simple. Just the opposite. Doug > Josh > > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Monday, October 02, 2000 2:47 PM > To: Daniel Smith > Cc: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Daniel, > > > > I'll reply to this piece-by-piece as there seem to be a lot of > > misconceptions. > > > > (On a side note, I haven't seen any alternative proposals put > > forward to the > > URL scheme as noted. If things stay this way, then I'll put it > forward as > > an Internet-Draft.) > > > > Okay, off we go. > > > > Douglas Otis wrote: > > > > > > If you wished to crack data being sent via IP, could you ask > > for a better > > > tag of such information- scsi://[/modifier] being > > sent in clear > > > text down the same connection? Of course, the same is true > for login as > > You base this comment assuming there is no other means. There is > no reason > for sending the text equivalent of the address within the transport. No > equipment will require a text name for routing or delivery. > Adding it to the > SCSI transport weakens security. > > > will always be sent "in the clear" at some point on the > > network. There's not a lot we can do about that. The IP address will > > always be visible in all packets. (Insert comments here about > > why VPNs are > > a good thing here....) > > Again, this is not true. Again you assume that somehow sending a name is > required. It is not. Communication to an authentication server could be > completely secure with no clear text. Would you expect otherwise? > > > In an SSL link, or IPsec the [/modifier] part will be > encrypted. Indeed, > > whenever security is tight, the [/modifier] will only be sent after > > clearance is granted. > > > > Why? > > > > > > well. Here again "who, what, and where" are nicely presented > in easy to > > > read form. Neither SCSI clients nor servers are likely able to > > authenticate > > > independently or retain textual information. Both the client > and server > > > should depend on some outside database. This compromise in > security was > > > > If security is needed, then security is needed and there are no > > short cuts! > > Whether stored locally or remotely, that information needs to be > > somewhere, > > and securely accessible too. > > There is just being sloppy. Defer these parameters to a better designed > server and thus keep the transport layer clean. > > > > > > done to eliminate an IP selection option but then a textual means of > > > including IP was included to ensure all modes of operation > > remain. Rather > > > than using a secure and opaque means to establish a > connection using an > > > independent means of authenticating on a different connection, > > the entire > > > function was included in this all encompassing draft. This was > > not a good > > > choice in my view. > > > > There is no requirement that only the Initiator and Target MUST > > authenticate > > between /themselves/ only. This may be desirable (in case the > > authentication database machine goes down.) There are many papers > > written on > > this subject; for a simple introduction, see "Secure Distributed > > Computing", > > Sci. Am., Nov 1994 pp72--76. This will give you an idea of what > > the minimum > > requirements are in an open network authentication realm. > > Once you allow IPv4 or 6 within URL target naming conventions, it > is odd to > suggest that this scheme affords protection from changes to these > standards. > There are many means to guard against a server failure besides leaving the > front door open. This would suggest targets will contain an entire > authentication database. What protocol would exchange this > information in a > consistent fashion. Perhaps LDAP? > > > > By creating a separate draft to define an authentication > database access > > > (LDAP structures), options remain possible without impact on > > the transport > > > specification. > > > > URLs have very little impact on the transport. The iSCSI draft > > just handles > > arbitrary length text strings to identify resources---and that's all it > > cares about. The only time interpretation of this is requred is for > > 3rd-party commands, when we don't have a choice about it---and > technically > > the transport still doesn't care about it. We DO want it > > standardized. We > > DON'T want it tied to the transport. In 2003 when iSCSIv5 over > SCTP with > > Microsoft Active Super Directory v3 running over IPv8 on > > terrabit-RS232, the > > naming scheme should still be handled by the transport. > > You have indeed tied it to the SCSI transport with the use of IP > text. You > could remove all such information from the spec and thus ensure no future > conflicts. Have the authentication process do the mapping and > then nothing > in the SCSI transport gets touched or exposed. Would you wish to go back > and make changes to the transport spec to allow a means for target mapping > with additional attributes? By planting this information within the > transport spec ensures extensive changes as time marches. I > seriously doubt > that IPvX will have anything to do with any of these changes. > > > > The only aspect of authentication within the transport > > > should be a cookie. Authentication provides access at some IP > > and port or > > > > Please forgive my ignorance, but isn't a cookie an arbitrary length text > > string (or can be mapped to and from an arbitrary length text string). > > Cookies as /constant/ entities is an extremely bad idea. That's why we > > don't recommend it. Okay, cookies are really small delicious disks, > > sometimes with chocolate chips---what do you mean by cookie? > > It would be a binary digest of shared secrets good for leased use derived > from the authentication server. I think you could use a few more chips in > your cookie however. The means of fixing the spec is simply the > removal of > text based mapping. Requiring DNS lookup should be fun. How many seconds > would you wait for a response? What if it is required to then > authenticate > with this third party? Now how long would you wait then? What a bad idea > to place this as a part of the real time operation of a SCSI. I > know, it is > only optional... Then get rid of the option and specify a better > means that > does not include the SCSI transport. Sending these names in text > is broken. > Fix it by removing it from the transport specification. > > > > Doug > > > Daniel Smith. > > > > [Mmm. Cookies.] > > -- > > IBM Almaden Research Center, 650 Harry Road, San Jose, CA > 95120-6099, USA > > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > > > From owner-ips@ECE.cmu.edu Tue Oct 03 03:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18373; 2 Oct 2000 23:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA29277 for ips-outgoing; Mon, 2 Oct 2000 22:26:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA29266 for ; Mon, 2 Oct 2000 22:26:02 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id TAA15944; Mon, 2 Oct 2000 19:12:06 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id TAA19831; Mon, 2 Oct 2000 19:09:54 -0700 From: "Douglas Otis" To: "Matt Wakeley" , , Subject: RE: iSCSI sessions: Let's try again Date: Mon, 2 Oct 2000 19:28:32 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-reply-to: <39D926DA.E63268CC@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Monday, October 02, 2000 5:23 PM > To: Black_David@emc.com; ips@ece.cmu.edu > Subject: Re: iSCSI sessions: Let's try again > > > Black_David@emc.com wrote: > > > > > Summarizing the above discussion: > > > > [Consensus Attempt 1] I don't see anything that REQUIRES > > two TCP connections in all iSCSI sessions, because I haven't > > seen any TCP window closure scenarios [3] that aren't avoidable > > by reasonable design/engineering of the target. I therefore > > propose to re-establish the WG consensus that multiple TCP > > connections per iSCSI session is OPTIONAL. > > Reasons requiring 2 tcp connections per iSCSI session (one for > commands the > other for data): > > 1- It allows a target to temporarily flow control commands from > the initiator > while still allowing data to continue on the "data" connection (allowing > commands to complete so that the flow control can be opened again). How do you go about managing the buffers of these connections using TCP? > 2- It allows commands to be received without being stuck behind a > large data > transfer, allowing the target to begin command execution sooner. You could also break all data transfers into smaller lengths to prevent queue blockage. Doug > 3- if "Consensus Attempt 2" is allowed, it provides a single > "model" of iSCSI > while allowing more than one data connection. It also simplifies > the "command > ordering" issues of the symetric model (sending commands on any > connection). > -Matt Wakeley > Agilent Technologies > From owner-ips@ECE.cmu.edu Tue Oct 03 03:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18458; 2 Oct 2000 23:16 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA29463 for ips-outgoing; Mon, 2 Oct 2000 22:33:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from atlrel2.hp.com (atlrel2.hp.com [156.153.255.202]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA27018 for ; Mon, 2 Oct 2000 21:09:30 -0400 (EDT) Received: from omgw1.boi.hp.com (omgw1.boi.hp.com [15.56.8.101]) by atlrel2.hp.com (Postfix) with ESMTP id 37179DE; Mon, 2 Oct 2000 21:09:29 -0400 (EDT) Received: from xboibrg1.boi.hp.com (xboibrg1.boi.hp.com [15.56.8.167]) by omgw1.boi.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit6.0.6 OpenMail) with ESMTP id TAA19301; Mon, 2 Oct 2000 19:09:27 -0600 (MDT) Received: by xboibrg1.boi.hp.com with Internet Mail Service (5.5.2650.21) id <4DX9CSDS>; Mon, 2 Oct 2000 19:09:25 -0600 Message-ID: From: "GUPTA,SOMESH (HP-Cupertino,ex1)" To: Douglas Otis , "GUPTA,SOMESH (HP-Cupertino,ex1)" , julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI: remove recovery from transport-layer connection failur e(?) Date: Mon, 2 Oct 2000 18:15:45 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The techniques take advantage of the protocol layering. A tcp connection is tied to an IP address and the route table tells you the next hop and the arp table specifies the IP address to MAC address translation. Without giving you the exact details (I don't know the whether proper behavior on the mailing list requires me to spill the beans), you can see that there is significant flexibility to recover from any kind of failure. -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Monday, October 02, 2000 3:25 PM To: GUPTA,SOMESH (HP-Cupertino,ex1); julian_satran@il.ibm.com; ips@ece.cmu.edu Subject: RE: iSCSI: remove recovery from transport-layer connection failure(?) Somesh, Are you referring to 802.1Q, 802.1D, or HSRP or just EtherChannel in general? Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > GUPTA,SOMESH (HP-Cupertino,ex1) > Sent: Monday, October 02, 2000 1:48 PM > To: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: iSCSI: remove recovery from transport-layer connection > failure(?) > > > Julian, > > If the scenario you point out is correct (a single command lasting > for such a long time), then of course we need a mechanism where > we can restart the command from the approximate point of failure. > However that would be failures lasting for "more than a fraction > of a sec". > > First of all, a TCP connection does not indicate a failure that > quickly. Secondly, there are ways to recover from a path failure > and still preserve a TCP connection in High-Availability environments. > I am sure most system vendors would be implementing such techniques. > > Somesh > > -----Original Message----- > From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] > Sent: Sunday, October 01, 2000 1:52 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: remove recovery from transport-layer connection > failure(?) > > > > > Steph, > > Assume than in the new wonderfull SAN world you have started a > disk-to-tape > (or disk-to-disk) long third party copy. The SAN is fine and the copy > proceeds for an hour > but the lousy initiator-to-copy-manager link (on which > accidentally no data > transfer took place) fails for a fraction of a second. > Should we restart the command under-the-cover or drop it or ask > the parties to provide state information to a specific SCSI > restart driver? > > And we can build many similar scenarios. > > I think that whatever we can do simplify exception handling we should do > (the same arguments that hold for multiple connections hold here too). > > I would add that in Ideal world - I would like to have transport > "splice" a > new TCP > connection with an old TCP connection but failing this to happen (again > SCTP is doing it already or not?) we should take care that simple events > like a cable taken-out > in some obscure part of the network will only seldom affect higher layers. > > Julo > > Stephen Bailey on 27/09/2000 07:16:12 > > Please respond to Stephen Bailey > > To: ips@ece.cmu.edu > cc: (bcc: Julian Satran/Haifa/IBM) > Subject: Re: iSCSI: remove recovery from transport-layer connection > failure(?) > > > > > > Currently, iSCSI is spec'ed to recover from transport-layer > > connection failures. > > > > The main motivation for this decision was to support tape backup > > applications that are quite sensitive to any failures that get > > propogated to their layer. > > > > So, perhaps we can remove the requirement of recovering from > > transport-layer connection failures in iSCSI. This would simplify > > the protocol somewhat. > > > > Thoughts? > > I'm all for eliminating command recovery. > > There seem to be several reasons advanced for command recovery. > > The first seems to be based upon an inappropriate analogy to FCP. > Command recovery had to be added to FCP-2 because the FC layer is > unreliable. A single dropped FC frame leads to a failed FCP command. > This clearly upsets tape operation even when the link is performing > nominally. In FCP, without command recovery, with some observable > frequency, you will get an expected error that leads to complete, > irrecoverable failure of a transfer stream. The other thing that > makes FCP-2 command recovery work well is when you are doing a write, > which is 90% (maybe it's 99%?) of tape operation, the target can > return an early indication of most frame drops, rather than waiting > for a timer to expire. > > TCP's reliability solves this problem in another way. By the time you > get a TCP connection failure, you have already exhausted a set of > reliability mechanisms which guarantee, with high certainty, that > further data can not be transferred between the two endpoints. > > `the two endpoints' phrase suggests the other reason advanced for > command recovery. That is, to permit path failover for commands which > are not idempotent, such as tape write sequential. The > problem with this, is that it is not clear HOW iSCSI command recovery > can actually work properly, given a TCP connection failure indication. > It takes a long time for a TCP connection to fail, and by that time, > I'm not sure recovery would reasonably be possible. Perhaps I'm in > error on this assumption. Can a tape guru (Joe from Exabyte?) comment > on whether recovery would be possible after many seconds (tens, > hundreds) have elapsed? > > The SCSI layer has never been solely responsible for ensuring reliable > backup. Macro scale things go wrong with tape (run off the end, get > eaten, etc..) with relatively high frequency. A low level backup > engine like tar or dump will fail on a SCSI error, and that's OK. > There must also be a higher level software component like Amanda, > which manages retries, including operator intervention, to ensure > reliable backup. > > It seems like whether iSCSI has a command recovery mechanism should be > a function of whether somebody can stand up and say for sure that it > solves a real problem. So far it only seems like it MIGHT solve a > problem. Who can say `this solves MY problem!'? > > Steph > From owner-ips@ECE.cmu.edu Tue Oct 03 04:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18637; 3 Oct 2000 0:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA00690 for ips-outgoing; Mon, 2 Oct 2000 23:22:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA00681 for ; Mon, 2 Oct 2000 23:22:31 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id UAA16035; Mon, 2 Oct 2000 20:22:28 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id <41CMD1NS>; Mon, 2 Oct 2000 20:22:26 -0700 Message-ID: From: Jim McGrath To: "'Peter Johansson'" , IP Storage Subject: RE: iSCSI: Flow Control Date: Mon, 2 Oct 2000 20:22:25 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I agree with Peter. We should allow immediate data, but only up to a limit previously set by the target (so we have a chance that it is received). But the target has to be given the ability to dynamically change (in Peter's terms, advertise) the resources available, and be able to do it selectively (i.e. to one initiator and not another). This inability to change the resources deployed for this purpose over time is why current schemes are not implemented (no target wants to get "caught" and so drastically underadvertizes the actual resources it has available). Jim -----Original Message----- From: Peter Johansson [mailto:PJohansson@ACM.org] Sent: Friday, September 29, 2000 2:57 PM To: IP Storage Subject: RE: iSCSI: Flow Control At 04:39 PM 9/29/00, Black_David@emc.com wrote: >The open question to the list is whether there's value in allowing some >amount of immediate data (e.g., for targets that need fast startup on >long latency connections, and are prepared to deploy the buffering >required to make it work reliably), or whether we ought to follow FCP-2 >and forbid immediate data, which will impose a round trip delay (command >out, R2T back) before data starts to flow. I think I've seen a couple of >comments in favor of this, but more discussion is in order. I suggest that you split the question: 1) Is immediate data desirable when the initiator has certain knowledge that the LU is idle (with respect to the initiator)? 2) Is immediate data desirable when the LU may be busy. I think the answer to 1) is "Yes", make provision for the initiator to send immediate data---but also make provision for the initiator to obtain some a priori knowledge as to the quantity of data the LU will accept in the idle state. There need not be a negotiation between initiator and LU; it could be sufficient for the LU to advertise its capabilities. In the case of 2), the answer might vary dependent upon whether or not those LU resources used to obtain data from the initiator are fully occupied when the LU is other than idle. If most of the optimization is realized in case 1), it might obviate the need for complicated schemes needed to make immediate data in case 2) useful. Regards, Peter Johansson Congruent Software, Inc. 98 Colorado Avenue Berkeley, CA 94707 (510) 527-3926 (510) 527-3856 FAX PJohansson@ACM.org From owner-ips@ECE.cmu.edu Tue Oct 03 05:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa18827; 3 Oct 2000 1:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id AAA01912 for ips-outgoing; Tue, 3 Oct 2000 00:11:03 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id AAA01903 for ; Tue, 3 Oct 2000 00:10:59 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Mon, 2 Oct 2000 21:13:51 -0700 Message-ID: From: Joshua Tseng To: Douglas Otis , ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Mon, 2 Oct 2000 21:13:50 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, >Sorry, but that is not how a NAT works. Yes NAT represents a problem only >from the perspective of the target. The target should know what goes where >and who is authorized. Once the client pokes through the NAT and identifies >in an opaque manner, end of problem. A NAT does not use names, however your >LDAP server may provide a name to your browser for the purpose of finding >the gateway. This name may only resolve locally within your enterprise and >not publicly as well. Should you be attempting to connect over the Big I to >your drives, then you will need to access the providers authentication >database that would include the mapping you desire. This mapping would not >be text but rather binary information. The retrieval of this information >may be symbolic but those symbols would never touch the SCSI layer. Thanks, but as a former network engineer, I am fully aware of how NAT works. I was not talking about the NAT mechanism per se, but proxies. The client cannot "poke through the NAT" because the destination IP address is in a different address domain. It's the proxy, not the original client that must do the DNS lookup for the final IP address destination. Conversely, the target will not see the original initiator, but the proxy. And guess how the proxy obtains the information to do a DNS lookup? Hint: It's a critical piece of info in http which is not "binary information", but rather is a name in human-readable form... >Placing the authentication server within the SCSI transport does not improve >the ability to scale. If anything, the authentication will occupy resources >needed for the retrieval of data. Let a server specifically designed to >provide secure connections and access to the client and server perform that >function. It makes no sense to conclude it becomes difficult to delegate >this function to a server already up and running. I don't understand why you equate placing the domain name & path in iSCSI PDU's to "putting the authentication server within the SCSI transport." We will need an authentication LDAP server regardless of whether domain & path go into the iSCSI PDU. Once again, putting the ASCII-based name in iSCSI allows receiving nodes, INCLUDING intermediary proxies, to be able to identify the final destination of the iSCSI traffic. That's all. And as long as DNS is out there and in everyday use, why not use it? I agree it's not perfect, but it works fine and scales. I do believe we need something in addition to DNS for topology/resource discovery, but that can be handled separately using a different mechanism. Authentication is a totally different problem. It involves distributing a shared secret between nodes and using it to create a message digest appended to the end of the iSCSI PDU which can be verified by each endpoint. It should be used only when necessary and required by the operational environment. Maybe I'm misunderstanding your proposal, but it seems you're advocating total reliance on the authentication server, not only for authentication, but for IP endpoint discovery? Josh -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Monday, October 02, 2000 7:29 PM To: Joshua Tseng; ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Joshua, > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Joshua Tseng > Sent: Monday, October 02, 2000 5:27 PM > To: ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Doug, > > >You base this comment assuming there is no other means. There > is no reason > >for sending the text equivalent of the address within the transport. No > >equipment will require a text name for routing or delivery. Adding it to > the > >SCSI transport weakens security. > > What about for proxies? A proxy that routes storage data between IP > addressing domains will need the original text name to resolve it to > a different IP address in a different address domain. NAT is quite > common nowadays, and I would think iSCSI needs to preserve mechanisms > that allow it to route between IP address domains. Sorry, but that is not how a NAT works. Yes NAT represents a problem only from the perspective of the target. The target should know what goes where and who is authorized. Once the client pokes through the NAT and identifies in an opaque manner, end of problem. A NAT does not use names, however your LDAP server may provide a name to your browser for the purpose of finding the gateway. This name may only resolve locally within your enterprise and not publicly as well. Should you be attempting to connect over the Big I to your drives, then you will need to access the providers authentication database that would include the mapping you desire. This mapping would not be text but rather binary information. The retrieval of this information may be symbolic but those symbols would never touch the SCSI layer. > As far as the security, is your problem with the ASCII text format > itself being readable by a human? If so, I see no difference between > using text format compared to "asdj2342zssd" (or whatever), in > addressing a remote LUN, as far as security is concerned. On the > other hand, if you believe ALL interactions between initiator & target > must reference an SPI found in a secure remote authentication server, > then IMO you are placing severe and unreasonable deployment > requirements on the user. It reminds me of the LANE architecture > in ATM--quite unscalable and burdensome. Placing the authentication server within the SCSI transport does not improve the ability to scale. If anything, the authentication will occupy resources needed for the retrieval of data. Let a server specifically designed to provide secure connections and access to the client and server perform that function. It makes no sense to conclude it becomes difficult to delegate this function to a server already up and running. > > > >There are many means to guard against a server failure besides > leaving the > >front door open. This would suggest targets will contain an entire > >authentication database. What protocol would exchange this > information in > a > >consistent fashion. Perhaps LDAP? > > I see no way around a local authentication database in the target. > If the target doesn't track which initiators can do what, then it > will have to ask the remote authentication server, which affects > performance. Local? You would use the local LDAP server to communicate with the SSP LDAP server. What problem do you see? Each server would control the domain of each entity. Sounds like a good thing. Only the Authentication server would need to expose their IP via DNS and then emit private IPs to ensure the load is balanced and the correct alternative routes are assigned. You should not trust that to DNS. > > > >It would be a binary digest of shared secrets good for leased use derived > >from the authentication server. I think you could use a few > more chips in > >your cookie however. The means of fixing the spec is simply the > removal of > >text based mapping. Requiring DNS lookup should be fun. How > many seconds > >would you wait for a response? What if it is required to then > authenticate > >with this third party? Now how long would you wait then? What > a bad idea > >to place this as a part of the real time operation of a SCSI. I know, it > is > >only optional... Then get rid of the option and specify a better means > that > >does not include the SCSI transport. Sending these names in text is > broken. > >Fix it by removing it from the transport specification. > > I see no reason why a DNS lookup would be more costly or than > an exchange of shared secrets with the authentication server. If > anything, the shared secret exchange is more costly, since > it must be secured. Yes, but not by the SCSI equipment. Expensive for a seldom used server. Not at all expensive for the SCSI transport. The SCSI transport remains dead simple. Adding an authentication server to the SCSI transport together with a naming database is not simple. Just the opposite. Doug > Josh > > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Monday, October 02, 2000 2:47 PM > To: Daniel Smith > Cc: julian_satran@il.ibm.com; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Daniel, > > > > I'll reply to this piece-by-piece as there seem to be a lot of > > misconceptions. > > > > (On a side note, I haven't seen any alternative proposals put > > forward to the > > URL scheme as noted. If things stay this way, then I'll put it > forward as > > an Internet-Draft.) > > > > Okay, off we go. > > > > Douglas Otis wrote: > > > > > > If you wished to crack data being sent via IP, could you ask > > for a better > > > tag of such information- scsi://[/modifier] being > > sent in clear > > > text down the same connection? Of course, the same is true > for login as > > You base this comment assuming there is no other means. There is > no reason > for sending the text equivalent of the address within the transport. No > equipment will require a text name for routing or delivery. > Adding it to the > SCSI transport weakens security. > > > will always be sent "in the clear" at some point on the > > network. There's not a lot we can do about that. The IP address will > > always be visible in all packets. (Insert comments here about > > why VPNs are > > a good thing here....) > > Again, this is not true. Again you assume that somehow sending a name is > required. It is not. Communication to an authentication server could be > completely secure with no clear text. Would you expect otherwise? > > > In an SSL link, or IPsec the [/modifier] part will be > encrypted. Indeed, > > whenever security is tight, the [/modifier] will only be sent after > > clearance is granted. > > > > Why? > > > > > > well. Here again "who, what, and where" are nicely presented > in easy to > > > read form. Neither SCSI clients nor servers are likely able to > > authenticate > > > independently or retain textual information. Both the client > and server > > > should depend on some outside database. This compromise in > security was > > > > If security is needed, then security is needed and there are no > > short cuts! > > Whether stored locally or remotely, that information needs to be > > somewhere, > > and securely accessible too. > > There is just being sloppy. Defer these parameters to a better designed > server and thus keep the transport layer clean. > > > > > > done to eliminate an IP selection option but then a textual means of > > > including IP was included to ensure all modes of operation > > remain. Rather > > > than using a secure and opaque means to establish a > connection using an > > > independent means of authenticating on a different connection, > > the entire > > > function was included in this all encompassing draft. This was > > not a good > > > choice in my view. > > > > There is no requirement that only the Initiator and Target MUST > > authenticate > > between /themselves/ only. This may be desirable (in case the > > authentication database machine goes down.) There are many papers > > written on > > this subject; for a simple introduction, see "Secure Distributed > > Computing", > > Sci. Am., Nov 1994 pp72--76. This will give you an idea of what > > the minimum > > requirements are in an open network authentication realm. > > Once you allow IPv4 or 6 within URL target naming conventions, it > is odd to > suggest that this scheme affords protection from changes to these > standards. > There are many means to guard against a server failure besides leaving the > front door open. This would suggest targets will contain an entire > authentication database. What protocol would exchange this > information in a > consistent fashion. Perhaps LDAP? > > > > By creating a separate draft to define an authentication > database access > > > (LDAP structures), options remain possible without impact on > > the transport > > > specification. > > > > URLs have very little impact on the transport. The iSCSI draft > > just handles > > arbitrary length text strings to identify resources---and that's all it > > cares about. The only time interpretation of this is requred is for > > 3rd-party commands, when we don't have a choice about it---and > technically > > the transport still doesn't care about it. We DO want it > > standardized. We > > DON'T want it tied to the transport. In 2003 when iSCSIv5 over > SCTP with > > Microsoft Active Super Directory v3 running over IPv8 on > > terrabit-RS232, the > > naming scheme should still be handled by the transport. > > You have indeed tied it to the SCSI transport with the use of IP > text. You > could remove all such information from the spec and thus ensure no future > conflicts. Have the authentication process do the mapping and > then nothing > in the SCSI transport gets touched or exposed. Would you wish to go back > and make changes to the transport spec to allow a means for target mapping > with additional attributes? By planting this information within the > transport spec ensures extensive changes as time marches. I > seriously doubt > that IPvX will have anything to do with any of these changes. > > > > The only aspect of authentication within the transport > > > should be a cookie. Authentication provides access at some IP > > and port or > > > > Please forgive my ignorance, but isn't a cookie an arbitrary length text > > string (or can be mapped to and from an arbitrary length text string). > > Cookies as /constant/ entities is an extremely bad idea. That's why we > > don't recommend it. Okay, cookies are really small delicious disks, > > sometimes with chocolate chips---what do you mean by cookie? > > It would be a binary digest of shared secrets good for leased use derived > from the authentication server. I think you could use a few more chips in > your cookie however. The means of fixing the spec is simply the > removal of > text based mapping. Requiring DNS lookup should be fun. How many seconds > would you wait for a response? What if it is required to then > authenticate > with this third party? Now how long would you wait then? What a bad idea > to place this as a part of the real time operation of a SCSI. I > know, it is > only optional... Then get rid of the option and specify a better > means that > does not include the SCSI transport. Sending these names in text > is broken. > Fix it by removing it from the transport specification. > > > > Doug > > > Daniel Smith. > > > > [Mmm. Cookies.] > > -- > > IBM Almaden Research Center, 650 Harry Road, San Jose, CA > 95120-6099, USA > > K65B/C2 Phone: +1(408)927-2072 Fax: +1(408)927-3010 > > > From owner-ips@ECE.cmu.edu Tue Oct 03 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19502; 3 Oct 2000 4:31 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA05362 for ips-outgoing; Tue, 3 Oct 2000 03:21:27 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA05357 for ; Tue, 3 Oct 2000 03:21:20 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id JAA27432 for ; Tue, 3 Oct 2000 09:20:43 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA29204 for ; Tue, 3 Oct 2000 09:20:43 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696D.0028584A ; Tue, 3 Oct 2000 09:20:40 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 3 Oct 2000 10:18:19 +0300 Subject: Re: iSCSI URL scheme Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk David, You might be right about us facing a change. Two things drive it. One is minor - addressing - and using the URL we are trying to make it more like other things on the net and allow it to fit into the hierarchical structure of the net instead of the flat SCSI structure. Walking the bus(es) is anyhow not an option anymore and the URL will come from somewhere - a Service Location service or in the extreme (god forbid) a local configuration file. The second (the view name) is again a form of access-id (as in Jim Haffner's scheme adopted by T10) that can be used as a complement or instead of the address to differentiate users (user is here an OS signature). I assume that those naming and authorization schemes will evolve into structures that have many elements in common with other internet services and save us the expense of rediscovering the wheel again and again. As for the simple SCSI walk all the buses scheme it is gone anyhow. Julo Black_David@emc.com on 02/10/2000 23:56:52 Please respond to Black_David@emc.com To: Daniel Smith/Almaden/IBM@IBMUS, ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: iSCSI URL scheme With my co-chair hat off ... I've had a long-running concern about the URL naming in iSCSI that Daniel's example allows me to express concisely ... > Let's imagine a large array of fast, reliable and featuresome storage---an > IBM Shark box say. B-) This box has many user accounts, and is being used > in the context of a storage service provider. The login sequence might go > something like this.... > > I: Hello Shark---I'm an initiator. My authentication ID is "Robert Smith, > Seattle, next to the donut store, hjhhaskjanbt112kg9d988". > > S: Hello initiator. I'll be your server---we need a secure connection: > switch to "hjcnneamchjsdkfsdsd394njsdf89423". (Trivial example.) > > I: Alright! I want to connect to > "scsi://niceshark.acme.com/bobsstorage/san3?wwn=0x7b4d72658a7410db" This query is a *major* departure from SCSI practice at this level. The SCSI approach would be more along the lines of: I: Alright! Show me all your LUNs that I can access. (i.e., REPORT LUNS) at which point the Initiator then does additional SCSI commands to figure out what each LUN actually is. One of these commands could be to read (or attempt to read) the WWN for each LUN. For disk storage that is under a volume manager, there's another round of discovery later on in which the volume manager reads the volume label and figures out what the storage is independent of any information that was used to discover it. Returning to the three examples: > Then we have a choice... > > S: Great---off you go and have a nice day. > > or > > S: Sorry---you haven't got permission to use that any more/it > doesn't exist. This is an authorization check that can be implemented by a combination of accepting/denying the connection based on the client authentication information and/or only reporting the LUNs to which the Initiator has access (and after an exhaustive search, the Initiator may discover that a LUN it used to have access to [based on WWN} doesn't seem to be there any more). Fibre Channel makes widespread use of a mechanism similar to the latter at the target discovery level (nameserver based soft zoning, where the fabric nameserver only tells an Initiator about Targets it can access, rather than all the Targets it knows about). > or (just after they buy more storage) > > S: Ok, that URL has been changed to > "scsi://bobstorage.san3.acme.com:9003/". Please come again. This is a common computer science solution to a large set of problems; introduce another level of indirection ;-). While I understand the solution, I'm not at all clear on what the problem is and why it needs to be solved in this fashion. There's been a lot of traffic on this, have I missed something? The counterbalance on the other side of this issue is that iSCSI should fit into existing SCSI frameworks for storage discovery, access, and management (e.g., in operating systems). Every time a new sort of naming is introduced, that integration difficulty goes up considerably. The worked example is that Fibre Channel had to introduce WWNs or their equivalent for some fairly obvious functional reasons, but introducing WWNs to operating systems that started out only understanding parallel SCSI caused no end of pain and suffering. Something that's going to rear its ugly head here is discovery. SCSI discovery is fundamentally based on a "bus walker" paradigm in which low level logic does discovery based on a comprehensive search (i.e., try every possible target on the parallel bus). When Fibre Channel introduced WWNs, this broke the "try every possible target" approach (can't iterate through every 64 bit WWN and still boot quickly), and it was fixed by a multi-round distributed address assignment scheme for FC-AL (don't ask ;-) ) and the ability to query the nameserver for all known ports in FC-SW. FWIW, FC-SW tried to use LDAP for nameservice and gave up; something considerably simpler was implemented. Among the practical problems that resulted from WWN introduction is that booting over Fibre Channel took a while to get working because FC broke the typical "first LUN of first target contains a boot image" mechanism used in boot code. iSCSI has the opportunity to repeat the Fibre Channel experience of changing the way things work in a fashion that will take a while to work its way through systems. If this is done, it needs to be done for a good reason. One question to think about is "If all SCSI targets are named by URLs on the server, where do the URLs come from?" An answer of the form "yet another config file or set of registry entries" is an obstacle to adoption, because tools have to be written to manage these, and administrators have to learn how to use them, and ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Oct 03 08:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19495; 3 Oct 2000 4:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA05651 for ips-outgoing; Tue, 3 Oct 2000 03:37:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA05646 for ; Tue, 3 Oct 2000 03:37:08 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 0185FA7; Tue, 3 Oct 2000 01:37:06 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 0FAF2106; Tue, 3 Oct 2000 03:37:06 -0400 (EDT) Received: from agilent.com (cos1nai251060.cs.itc.hp.com [15.3.251.60]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id AAA08718; Tue, 3 Oct 2000 00:37:02 -0700 (PDT) Message-ID: <39D98C93.E50301E0@agilent.com> Date: Tue, 03 Oct 2000 00:36:51 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Douglas Otis , IPS Reflector Subject: Re: iSCSI sessions: Let's try again References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > > Reasons requiring 2 tcp connections per iSCSI session (one for > > commands the > > other for data): > > > > 1- It allows a target to temporarily flow control commands from > > the initiator > > while still allowing data to continue on the "data" connection (allowing > > commands to complete so that the flow control can be opened again). > > How do you go about managing the buffers of these connections using TCP? Doug, even you know that if you stop reading from the TCP stream, TCP will eventually stop the initiator from sending commands. Thus, if the SCSI command queue fills up, the target stops reading from TCP, which eventually shuts down the initiator from sending more commands. -Matt From owner-ips@ECE.cmu.edu Tue Oct 03 11:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa19946; 3 Oct 2000 7:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id FAA07508 for ips-outgoing; Tue, 3 Oct 2000 05:41:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraac.compuserve.com (ds-img-rel-3.compuserve.com [149.174.206.154]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id FAA07503 for ; Tue, 3 Oct 2000 05:41:38 -0400 (EDT) Received: (from mailgate@localhost) by spdmraac.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id FAA04286 for ips@ece.cmu.edu; Tue, 3 Oct 2000 05:40:57 -0400 (EDT) Received: from compuserve.com (mid-tgn-ngk-vty10.as.wcom.net [216.192.83.10]) by spdmraac.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id FAA04234 for ; Tue, 3 Oct 2000 05:40:44 -0400 (EDT) Message-ID: <39D9B690.D4F2C91F@compuserve.com> Date: Tue, 03 Oct 2000 05:36:00 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: Bidirectional SCSI commands and iSCSI References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk A proposal for FCP-2 bidirectional changes can be found at: ftp://ftp.t10.org/t10/document.00/00-377r0.pdf Note well: This is a PROPOSAL. It is up for discussion at meetings this week and is subject to change with practically no notice. Thanks. Ralph... John Hufferd/San Jose/IBM wrote: > > Jim Hafner, > Since one of the things we are trying to do is not stray too > far from the CDB structure which FC uses, do you or anyone > else on the list know how FCP is structuring their CDB for > this feature? > . > . > . > John L. Hufferd From owner-ips@ECE.cmu.edu Tue Oct 03 15:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20612; 3 Oct 2000 11:01 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA13864 for ips-outgoing; Tue, 3 Oct 2000 10:05:09 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA05393 for ; Tue, 3 Oct 2000 03:22:46 -0400 (EDT) Received: from divyaroot.India.Sun.COM ([129.158.226.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id AAA09791 for ; Tue, 3 Oct 2000 00:22:44 -0700 (PDT) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with SMTP id MAA23812 for ; Tue, 3 Oct 2000 12:52:43 +0530 (IST) Message-Id: <200010030722.MAA23812@divyaroot.India.Sun.COM> Date: Tue, 3 Oct 2000 12:55:41 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: QIfyQddrlXd2fGGuMrSndg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > out-of-band). I don't see the point of the initiator sending its "view" to > the target at any point. If the initiator sends an incorrect view, you've > just created an error scenario which needs additional definition. Well, to extend what Julian said further, the initiator could ask the target for a LUN value corresponding to a WWN as part of the authentication process, instead of presenting its view - whether right or wrong. But something like this needs to be done as there is no easier/faster way to translate a LU WWN to a LUN value. This may also help in building an authentication method for a LU access where no such thing exists today ? > case, the target will send its "view" for that initiator in REPORT LUNS > SCSI command after login (which is actually the layer that cares about the > LUN Map at all anyway). > This will be a problem if the storage controller has a lot of LUNs. Since REPORT LUNS only returns a list of LUNs, the initiator still has to send an INQUIRY (page 83) request to each LU to to find a matching LU WWN (or until a matching WWN is found). -JP From owner-ips@ECE.cmu.edu Tue Oct 03 15:10:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20627; 3 Oct 2000 11:07 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA14278 for ips-outgoing; Tue, 3 Oct 2000 10:14:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from laime.cs.uchicago.edu (laime.cs.uchicago.edu [128.135.11.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA14271 for ; Tue, 3 Oct 2000 10:14:42 -0400 (EDT) Received: from candide.cs.uchicago.edu (candide.cs.uchicago.edu [128.135.11.62]) by laime.cs.uchicago.edu (8.10.2/8.9.3) with SMTP id e93EEfS03856 for ; Tue, 3 Oct 2000 09:14:41 -0500 (CDT) Received: by candide.cs.uchicago.edu (5.57/4.7) id AA10624; Tue, 3 Oct 00 09:12:59 -0500 Message-Id: <10010031412.AA10624@candide.cs.uchicago.edu> To: ips@ece.cmu.edu Subject: Re: Bidirectional SCSI commands and iSCSI In-Reply-To: Message from julian_satran@il.ibm.com of "Mon, 02 Oct 2000 21:02:29 +0300." Date: Tue, 03 Oct 2000 09:12:46 -0500 From: Stephen Bailey Sender: owner-ips@ece.cmu.edu Precedence: bulk > I think that software initiators will be widely used and they are far > more efficient with fixed length headers for widely used operations. In ST, fixed length headers were a design choice to permit hardware acceleration. Other than on very slow links, it's hard to argue with the performance advantages of fixed length headers under any circumstances. Steph From owner-ips@ECE.cmu.edu Tue Oct 03 16:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20794; 3 Oct 2000 11:50 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA16216 for ips-outgoing; Tue, 3 Oct 2000 10:57:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA16205 for ; Tue, 3 Oct 2000 10:57:20 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id KAA39048; Tue, 3 Oct 2000 10:44:38 -0400 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e93EvIa122970; Tue, 3 Oct 2000 08:57:19 -0600 Importance: Normal Subject: Re: Bidirectional SCSI commands and iSCSI To: Stephen Bailey Cc: ips@ece.cmu.edu From: "Jim Hafner/Almaden/IBM" Date: Tue, 3 Oct 2000 07:57:17 -0700 Message-ID: X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/03/2000 07:57:18 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Folks, I'm not advocating variable length headers here to support bi-d commands. More to the point, I'm suggesting that the fixed length header change (NOW) to accomodate the bi-directional SCSI commands. Better now while the draft is still in flux than later after its finalized and more hardware gets built. BTW, I see two approaches: 1) one universal header with enough fields for both directions 2) one header for classical uni-d commands and an additional header for bi-d commands (as FCP is proposing) My vote would be for option (1). Jim Hafner Stephen Bailey @ece.cmu.edu on 10-03-2000 07:12:46 AM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: Bidirectional SCSI commands and iSCSI > I think that software initiators will be widely used and they are far > more efficient with fixed length headers for widely used operations. In ST, fixed length headers were a design choice to permit hardware acceleration. Other than on very slow links, it's hard to argue with the performance advantages of fixed length headers under any circumstances. Steph From owner-ips@ECE.cmu.edu Tue Oct 03 16:00:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20807; 3 Oct 2000 11:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA17190 for ips-outgoing; Tue, 3 Oct 2000 11:23:00 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from newdev.harvard.edu (newdev.eecs.harvard.edu [140.247.60.212]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA17185 for ; Tue, 3 Oct 2000 11:22:53 -0400 (EDT) Received: (from sob@localhost) by newdev.harvard.edu (8.9.3/8.9.3) id LAA20846 for ips@ece.cmu.edu; Tue, 3 Oct 2000 11:22:53 -0400 (EDT) Date: Tue, 3 Oct 2000 11:22:53 -0400 (EDT) From: Scott Bradner Message-Id: <200010031522.LAA20846@newdev.harvard.edu> To: ips@ece.cmu.edu Subject: RE: Enough on TCP vs. FCTP, please Sender: owner-ips@ece.cmu.edu Precedence: bulk > You and I have a different opinion about reasonable changes to TCP. AD hat on the IESG will not approve of any IPS-specific changes to TCP any proposals for changes to TCP will have to have very good reasons that go well beyond a specific application Scott From owner-ips@ECE.cmu.edu Tue Oct 03 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20968; 3 Oct 2000 12:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA17753 for ips-outgoing; Tue, 3 Oct 2000 11:35:56 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA17744 for ; Tue, 3 Oct 2000 11:35:50 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id <4GAR9LWV>; Tue, 3 Oct 2000 08:37:23 -0700 Message-ID: From: Howard Green To: "'Raghavendra Rao'" , ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 08:35:15 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk I know that the following is kind of a radical question to ask, but has the time arrived to toss out LUNs as a means of identifying logical units, in favor of simply using the logical units' WWNs directly? Currently, an IU containing a SCSI command identifies the target logical unit by a Logical Unit Number. Back in the good ol' days, LUNs were small integers contained directly in each CDB, and were (in practice, if not in theory) a simple index assigned to a specific logical unit throughout its life. SCSI has obviously evolved to a far more sophisticated state of affairs. The LUNs have grown to 64 bits, opaque used to identify a logical unit can vary over time, relative to the interface used, and as a function of the initiator's identity. The WWNs I've seen used to identify logical units currently range from 64 to 128 bits, > -----Original Message----- > From: Raghavendra Rao [mailto:jpr@divyaroot.India.Sun.COM] > Sent: Tuesday, October 03, 2000 10:56 AM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > out-of-band). I don't see the point of the initiator > sending its "view" to > > the target at any point. If the initiator sends an > incorrect view, you've > > just created an error scenario which needs additional definition. > > Well, to extend what Julian said further, the initiator could > ask the target > for a LUN value corresponding to a WWN as part of the > authentication process, > instead of presenting its view - whether right or wrong. But > something like > this needs to be done as there is no easier/faster way to > translate a LU WWN > to a LUN value. > > This may also help in building an authentication method for a > LU access where > no such thing exists today ? > > > > case, the target will send its "view" for that initiator in > REPORT LUNS > > SCSI command after login (which is actually the layer that > cares about the > > LUN Map at all anyway). > > > > This will be a problem if the storage controller has a lot of > LUNs. Since > REPORT LUNS only returns a list of LUNs, the initiator still > has to send an > INQUIRY (page 83) request to each LU to to find a matching LU > WWN (or until > a matching WWN is found). > > -JP > From owner-ips@ECE.cmu.edu Tue Oct 03 16:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa20966; 3 Oct 2000 12:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA18714 for ips-outgoing; Tue, 3 Oct 2000 11:56:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from thor.brocade.COM (asbestos.brocade.com [12.7.224.244]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA18709 for ; Tue, 3 Oct 2000 11:56:21 -0400 (EDT) Received: by thor.brocade.com with Internet Mail Service (5.5.2650.21) id <4GAR9L7Q>; Tue, 3 Oct 2000 08:57:55 -0700 Message-ID: From: Howard Green To: "'Raghavendra Rao'" , ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 08:55:47 -0700 X-Mailer: Internet Mail Service (5.5.2650.21) Sender: owner-ips@ece.cmu.edu Precedence: bulk Sorry for the previous incomplete post. My mailer misfired... I know that the following is kind of a radical question to ask, but has the time arrived to toss out LUNs as a means of identifying logical units, in favor of simply using the logical units' WWNs directly? Back in the good ol' days, LUNs were small integers contained directly in each CDB, and were (in practice, if not in theory) a simple index assigned to a specific logical unit throughout its life. SCSI has obviously evolved to a far more sophisticated state of affairs. The LUNs used to identify a logical unit have grown to 64 bits, and can vary over time, vary relative to the target interface used, and vary as a function of the initiator's identity. Clearly, they're not durable identifiers in any useful sense. Further, they've got to be be set up and administered. My sense here is that they've become just so much bureaucracy, to no particularly useful end. Currently, an IU containing a SCSI command identifies the target logical unit by a Logical Unit Number. So, in effect, the notion here would be to use the logical unit's WWN in place of the LUN in the command IU. The WWNs I've seen used to identify logical units currently range from 64 to 128 bits, so there's no particular additional overhead involved. I think I'm reasonably cognizant of the difficulties imposed by making such a change (changes to SAM, compatibility with existing implementations, etc.)---and that it's really a T10 issue, and not exclusively an iSCSI issue. However, my sense is that the current LUN-based identification mechanism is causing increasing grief, as evidenced by any number of recent postings; and one has to start somewhere... Howard Green > -----Original Message----- > From: Raghavendra Rao [mailto:jpr@divyaroot.India.Sun.COM] > Sent: Tuesday, October 03, 2000 10:56 AM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > out-of-band). I don't see the point of the initiator > sending its "view" to > > the target at any point. If the initiator sends an > incorrect view, you've > > just created an error scenario which needs additional definition. > > Well, to extend what Julian said further, the initiator could > ask the target > for a LUN value corresponding to a WWN as part of the > authentication process, > instead of presenting its view - whether right or wrong. But > something like > this needs to be done as there is no easier/faster way to > translate a LU WWN > to a LUN value. > > This may also help in building an authentication method for a > LU access where > no such thing exists today ? > > > > case, the target will send its "view" for that initiator in > REPORT LUNS > > SCSI command after login (which is actually the layer that > cares about the > > LUN Map at all anyway). > > > > This will be a problem if the storage controller has a lot of > LUNs. Since > REPORT LUNS only returns a list of LUNs, the initiator still > has to send an > INQUIRY (page 83) request to each LU to to find a matching LU > WWN (or until > a matching WWN is found). > > -JP > From owner-ips@ECE.cmu.edu Tue Oct 03 17:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21170; 3 Oct 2000 13:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA19371 for ips-outgoing; Tue, 3 Oct 2000 12:11:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA19354 for ; Tue, 3 Oct 2000 12:10:53 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA67148 for ; Tue, 3 Oct 2000 18:10:16 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA75310 for ; Tue, 3 Oct 2000 18:10:16 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696D.0058D28A ; Tue, 3 Oct 2000 18:10:10 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 3 Oct 2000 19:07:25 +0300 Subject: RE: The third alternative Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I've already agreed that multiple connections are optional. As for the arguments you know that we can go ad nauseam with where is the right place to aggregate. But I cannot accept an argument that distorts what we did in the design of the multiple connections session - that you don't have to spread state over connection boundaries. Julo Black_David@emc.com on 02/10/2000 17:02:31 Please respond to Black_David@emc.com To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: The third alternative > The only argument for SCSI wedge drivers where that they EXIST ALREADY. > Pretty weak argument for those building new equipment and for > interoperability. This severely understates the case. Not only do wedge drivers exist already, but they do a number of things that an iSCSI standard will never do. Wedge drivers tend to contain logic specific to the device that they're providing the wedge for - that's inappropriate to standardize, either in IETF or T10. In addition, there are implementation advantages to building wedge drivers above the SCSI level - one doesn't have to spread SCSI connection state across a failure boundary. > I think that if we keep ourselves honest we have to either: > > - provide for multiple connections at the iSCSI level as it is transport problem > that other TCP applications are not compelled to handle (I hear already BUT SCTP > handles it!) and hope that one day the session concept will drift into pure transport Provide for is fine, as long as they're not required. I would expect to see significant deployment of systems that use single TCP connection per session and handle multiple sessions in wedge drivers. Arguments like the one Bob has made is one of the reasons for multiple connections/session to remain OPTIONAL. > - go to T10 and ask the to standardize wedge drivers! Just say no! Exactly what would you want T10 to standardize? --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Tue Oct 03 17:30:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21179; 3 Oct 2000 13:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA20348 for ips-outgoing; Tue, 3 Oct 2000 12:32:02 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA20334 for ; Tue, 3 Oct 2000 12:31:47 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id SAA74328 for ; Tue, 3 Oct 2000 18:31:16 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id SAA107252 for ; Tue, 3 Oct 2000 18:31:16 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696D.005ABDD4 ; Tue, 3 Oct 2000 18:31:08 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Tue, 3 Oct 2000 19:28:50 +0300 Subject: RE: iSCSI: remove recovery from transport-layer connection failur e(?) Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Somesh, I am sure that a broken TCP connection could be repaired at TCP level as there you have a minimum of context information to care. I've even seen some (academia) papers outlining how to do it and perhaps one day the TCP community will do it. But since we need it now and it is simple to achieve we will have to go for it. The mainframes had it (command restart on channels) for a long time and the it comes almost free. Regards, Julo "GUPTA,SOMESH (HP-Cupertino,ex1)" on 02/10/2000 23:47:36 Please respond to "GUPTA,SOMESH (HP-Cupertino,ex1)" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI: remove recovery from transport-layer connection failur e(?) Julian, If the scenario you point out is correct (a single command lasting for such a long time), then of course we need a mechanism where we can restart the command from the approximate point of failure. However that would be failures lasting for "more than a fraction of a sec". First of all, a TCP connection does not indicate a failure that quickly. Secondly, there are ways to recover from a path failure and still preserve a TCP connection in High-Availability environments. I am sure most system vendors would be implementing such techniques. Somesh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Sunday, October 01, 2000 1:52 AM To: ips@ece.cmu.edu Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) Steph, Assume than in the new wonderfull SAN world you have started a disk-to-tape (or disk-to-disk) long third party copy. The SAN is fine and the copy proceeds for an hour but the lousy initiator-to-copy-manager link (on which accidentally no data transfer took place) fails for a fraction of a second. Should we restart the command under-the-cover or drop it or ask the parties to provide state information to a specific SCSI restart driver? And we can build many similar scenarios. I think that whatever we can do simplify exception handling we should do (the same arguments that hold for multiple connections hold here too). I would add that in Ideal world - I would like to have transport "splice" a new TCP connection with an old TCP connection but failing this to happen (again SCTP is doing it already or not?) we should take care that simple events like a cable taken-out in some obscure part of the network will only seldom affect higher layers. Julo Stephen Bailey on 27/09/2000 07:16:12 Please respond to Stephen Bailey To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI: remove recovery from transport-layer connection failure(?) > Currently, iSCSI is spec'ed to recover from transport-layer > connection failures. > > The main motivation for this decision was to support tape backup > applications that are quite sensitive to any failures that get > propogated to their layer. > > So, perhaps we can remove the requirement of recovering from > transport-layer connection failures in iSCSI. This would simplify > the protocol somewhat. > > Thoughts? I'm all for eliminating command recovery. There seem to be several reasons advanced for command recovery. The first seems to be based upon an inappropriate analogy to FCP. Command recovery had to be added to FCP-2 because the FC layer is unreliable. A single dropped FC frame leads to a failed FCP command. This clearly upsets tape operation even when the link is performing nominally. In FCP, without command recovery, with some observable frequency, you will get an expected error that leads to complete, irrecoverable failure of a transfer stream. The other thing that makes FCP-2 command recovery work well is when you are doing a write, which is 90% (maybe it's 99%?) of tape operation, the target can return an early indication of most frame drops, rather than waiting for a timer to expire. TCP's reliability solves this problem in another way. By the time you get a TCP connection failure, you have already exhausted a set of reliability mechanisms which guarantee, with high certainty, that further data can not be transferred between the two endpoints. `the two endpoints' phrase suggests the other reason advanced for command recovery. That is, to permit path failover for commands which are not idempotent, such as tape write sequential. The problem with this, is that it is not clear HOW iSCSI command recovery can actually work properly, given a TCP connection failure indication. It takes a long time for a TCP connection to fail, and by that time, I'm not sure recovery would reasonably be possible. Perhaps I'm in error on this assumption. Can a tape guru (Joe from Exabyte?) comment on whether recovery would be possible after many seconds (tens, hundreds) have elapsed? The SCSI layer has never been solely responsible for ensuring reliable backup. Macro scale things go wrong with tape (run off the end, get eaten, etc..) with relatively high frequency. A low level backup engine like tar or dump will fail on a SCSI error, and that's OK. There must also be a higher level software component like Amanda, which manages retries, including operator intervention, to ensure reliable backup. It seems like whether iSCSI has a command recovery mechanism should be a function of whether somebody can stand up and say for sure that it solves a real problem. So far it only seems like it MIGHT solve a problem. Who can say `this solves MY problem!'? Steph From owner-ips@ECE.cmu.edu Tue Oct 03 18:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21343; 3 Oct 2000 14:08 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA22297 for ips-outgoing; Tue, 3 Oct 2000 13:18:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA22291 for ; Tue, 3 Oct 2000 13:18:40 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Tue, 3 Oct 2000 10:21:35 -0700 Message-ID: From: Joshua Tseng To: Douglas Otis , ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 10:21:33 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, >If you envision use of a proxy sitting in a private name space rather than >using direct access and wish to embed names as a routing feature, then after >authenticating between the proxy and the client, you will be filtering the >PDUs to extract the embedded "name" to then do a lookup of that name to then >do an authentication of the client against that named target. If an >authentication process has already "embedded" the correct location in a >binary form from negotiations at the client together with an opaque id, >there would not be any lookup required nor any mid-stream authentication. >Why place this in the path of getting the job done? Why not get it out of >the way directly and let the two ends do the work? Doing it your way, you >would need to authenticate between the proxy and the client and then again >between the proxy and the target after doing a name lookup. I would assume >this proxy is passing all the traffic to all the devices, so why add the >burden? Why is a web browser proxy concept better? Why not authenticate >using a secure and public means and then allow each end to confirm a >connection based on information delivered during authentication. Once the >client has discovered the authentication server, all of the heavy lifting is >out of the way. There is no good reason to embed names within the SCSI >transport if your goal is security and performance. Rather than keeping the >process simple, you have made the matter far more complex and far less >likely to scale. What you describe might be possible (although I still think it's a bad idea) if the entire Internet, including all public and private networks, were in a single consolidated address space. But the fact is we are running out of address space, and there is something called NAT defined in RFC1918. Who knows, with IPv6, this may change, or it might not. But it is a reality today. To operate in an environment with NAT, you need proxies. There's no way around it. A client in a public network using registered IP address space should NEVER see a 10.0/8 address. It should NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 address entry in its routing table. It must first talk to a dual-homed proxy with at least one leg using registered IP address space, in order to communicate with a host with a 10.0/8 address. In this environment and with these restrictions, I don't understand how you can remove the involvement of the proxy in the process of what you call "authentication". BTW, it's not just http--e-mail and many other applications today make extensive use of proxy relays as well. Josh From owner-ips@ECE.cmu.edu Tue Oct 03 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21399; 3 Oct 2000 14:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA20632 for ips-outgoing; Tue, 3 Oct 2000 12:38:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA20621 for ; Tue, 3 Oct 2000 12:38:22 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id JAA16105; Tue, 3 Oct 2000 09:24:19 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id JAA21888; Tue, 3 Oct 2000 09:22:07 -0700 From: "Douglas Otis" To: "Joshua Tseng" , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 09:40:52 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, > Thanks, but as a former network engineer, I am fully aware of how NAT > works. I was not talking about the NAT mechanism per se, but proxies. > The client cannot "poke through the NAT" because the destination IP > address is in a different address domain. It's the proxy, not > the original > client that must do the DNS lookup for the final IP address destination. > Conversely, the target will not see the original initiator, but the proxy. > And guess how the proxy obtains the information to do a DNS > lookup? Hint: > It's a critical piece of info in http which is not "binary > information", but > rather is a name in human-readable form... > > >Placing the authentication server within the SCSI transport does not > improve > >the ability to scale. If anything, the authentication will occupy > resources > >needed for the retrieval of data. Let a server specifically designed to > >provide secure connections and access to the client and server > perform that > >function. It makes no sense to conclude it becomes difficult to delegate > >this function to a server already up and running. > > I don't understand why you equate placing the domain name & path in iSCSI > PDU's to "putting the authentication server within the SCSI transport." > We will need an authentication LDAP server regardless of whether domain & > path go into the iSCSI PDU. Once again, putting the ASCII-based name in > iSCSI allows receiving nodes, INCLUDING intermediary proxies, to be able > to identify the final destination of the iSCSI traffic. That's all. And > as long as DNS is out there and in everyday use, why not use it? I agree > it's not perfect, but it works fine and scales. I do believe we need > something in addition to DNS for topology/resource discovery, but that can > be handled separately using a different mechanism. If you envision use of a proxy sitting in a private name space rather than using direct access and wish to embed names as a routing feature, then after authenticating between the proxy and the client, you will be filtering the PDUs to extract the embedded "name" to then do a lookup of that name to then do an authentication of the client against that named target. If an authentication process has already "embedded" the correct location in a binary form from negotiations at the client together with an opaque id, there would not be any lookup required nor any mid-stream authentication. Why place this in the path of getting the job done? Why not get it out of the way directly and let the two ends do the work? Doing it your way, you would need to authenticate between the proxy and the client and then again between the proxy and the target after doing a name lookup. I would assume this proxy is passing all the traffic to all the devices, so why add the burden? Why is a web browser proxy concept better? Why not authenticate using a secure and public means and then allow each end to confirm a connection based on information delivered during authentication. Once the client has discovered the authentication server, all of the heavy lifting is out of the way. There is no good reason to embed names within the SCSI transport if your goal is security and performance. Rather than keeping the process simple, you have made the matter far more complex and far less likely to scale. Doug > Authentication is a totally different problem. It involves distributing > a shared secret between nodes and using it to create a message digest > appended to the end of the iSCSI PDU which can be verified by > each endpoint. > It should be used only when necessary and required by the operational > environment. Maybe I'm misunderstanding your proposal, but it > seems you're > advocating total reliance on the authentication server, not only for > authentication, but for IP endpoint discovery? > > Josh From owner-ips@ECE.cmu.edu Tue Oct 03 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21357; 3 Oct 2000 14:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA21772 for ips-outgoing; Tue, 3 Oct 2000 13:05:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel1.hp.com (palrel1.hp.com [156.153.255.242]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA21767 for ; Tue, 3 Oct 2000 13:05:13 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel1.hp.com (Postfix) with ESMTP id 88944868 for ; Tue, 3 Oct 2000 10:05:12 -0700 (PDT) Received: from mk731912 (mk731912.cup.hp.com [15.8.80.111]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id KAA15832 for ; Tue, 3 Oct 2000 10:05:26 -0700 (PDT) Message-Id: <4.2.2.20001003092838.00b4cb10@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Tue, 03 Oct 2000 09:30:47 -0700 To: From: Michael Krause Subject: RE: iSCSI: Flow Control In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 08:22 PM 10/2/00 -0700, Jim McGrath wrote: >I agree with Peter. > >We should allow immediate data, but only up to a limit previously set by the >target (so we have a chance that it is received). But the target has to be >given the ability to dynamically change (in Peter's terms, advertise) the >resources available, and be able to do it selectively (i.e. to one initiator >and not another). This inability to change the resources deployed for this >purpose over time is why current schemes are not implemented (no target >wants to get "caught" and so drastically underadvertizes the actual >resources it has available). Agree as well. The question is does one only do this at connection set-up time or does one define a iSCSI operation that performs session option negotiation at any time within a session's lifetime. Also if multiple connections are supported, is this negotiation per TCP connection? Recommend doing this per connection within a session and allowing resources per connection and resource to vary even between the same endnode pair. Mike From owner-ips@ECE.cmu.edu Tue Oct 03 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21370; 3 Oct 2000 14:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA23108 for ips-outgoing; Tue, 3 Oct 2000 13:38:36 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA23099 for ; Tue, 3 Oct 2000 13:38:30 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA16147; Tue, 3 Oct 2000 10:24:26 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA22063; Tue, 3 Oct 2000 10:22:11 -0700 From: "Douglas Otis" To: "Matt Wakeley" , "IPS Reflector" Subject: RE: iSCSI sessions: Let's try again Date: Tue, 3 Oct 2000 10:40:57 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <39D98C93.E50301E0@agilent.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Matt, >From the little I understand, there is a command window built into this scheme which should act as a means to regulate flow of commands. You should not need an additional connection to perform that function. I do not think stuffing receive buffers is a good technique but it is a good system test. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Matt Wakeley > Sent: Tuesday, October 03, 2000 12:37 AM > To: Douglas Otis; IPS Reflector > Subject: Re: iSCSI sessions: Let's try again > > > > > Douglas Otis wrote: > > > > Reasons requiring 2 tcp connections per iSCSI session (one for > > > commands the > > > other for data): > > > > > > 1- It allows a target to temporarily flow control commands from > > > the initiator > > > while still allowing data to continue on the "data" > connection (allowing > > > commands to complete so that the flow control can be opened again). > > > > How do you go about managing the buffers of these connections using TCP? > > Doug, even you know that if you stop reading commands> from > the TCP stream, TCP will eventually stop the initiator from > sending commands. > Thus, if the SCSI command queue fills up, the target stops > reading from TCP, > which eventually shuts down the initiator from sending more commands. > > -Matt > > From owner-ips@ECE.cmu.edu Tue Oct 03 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21503; 3 Oct 2000 15:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA24432 for ips-outgoing; Tue, 3 Oct 2000 14:10:25 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA24426 for ; Tue, 3 Oct 2000 14:10:20 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id KAA16168; Tue, 3 Oct 2000 10:56:13 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id KAA22153; Tue, 3 Oct 2000 10:54:01 -0700 From: "Douglas Otis" To: "Joshua Tseng" , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 11:12:47 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, > What you describe might be possible (although I still think it's a bad > idea) if the entire Internet, including all public and private networks, > were in a single consolidated address space. But the fact is we are > running out of address space, and there is something called NAT defined > in RFC1918. Who knows, with IPv6, this may change, or it might not. But > it is a reality today. To operate in an environment with NAT, you need > proxies. There's no way around it. A client in a public network using > registered IP address space should NEVER see a 10.0/8 address. It should > NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 > address entry in its routing table. It must first talk to a dual-homed > proxy with at least one leg using registered IP address space, in order > to communicate with a host with a 10.0/8 address. In this environment > and with these restrictions, I don't understand how you can remove the > involvement of the proxy in the process of what you call "authentication". > > BTW, it's not just http--e-mail and many other applications today make > extensive use of proxy relays as well. > > Josh Yes, and most enterprise environments include a NAT. Even homes with DSL include NAT. A few may even use a proxy. That does not mean private addresses of the target can not be shared at the time of authentication. I would have expected such an exchange. As most of these things work, such permission is in the form of a lease. I would also expect as the map is declared, mapping screens are established based on the permission discovered at the time of authentication. Before and not during use. Using a binary address does not mean PUBLIC addresses. It may not even be IP. It could be SCSI address or perhaps an encoded address. You do not want SCSI to look like an HTTP server. Especially if you wish this application to scale, you do not want to be doing in-band name lookup and authentication. Pleases, this is not a web server, it is a portal to SCSI devices. A client does not need to use a name to get a proxy to listen, try just typing the IP of a web site. The proxy will forgo the lookup. Name lookup is simply a convenience for humans. You would not want to depend on a round-robin selection of IPs from DNS should there be more than one such IP. How would you select the alternative IP, the next in the list? All these parameters can be concisely defined in the authentication exchange. I can not see why someone would wish to place a name on their SCSI portal but they could. The only name that needs to exist is the authentication server. I would not expect an address beyond the SCSI portal to be PUBLIC IPs. I would not expect them to be IP. LDAP is good at doing symbolic lookup. Let it do the work at the time of authentication. Don't invent a SCSI browser. Doug From owner-ips@ECE.cmu.edu Tue Oct 03 19:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21611; 3 Oct 2000 15:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA25855 for ips-outgoing; Tue, 3 Oct 2000 14:39:49 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA25851 for ; Tue, 3 Oct 2000 14:39:45 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Tue, 3 Oct 2000 11:42:43 -0700 Message-ID: From: Joshua Tseng To: Douglas Otis , ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 11:42:43 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk Doug, I'm not sure we understand each other anymore. I will just carefully restate my points, and leave it at that. 1) URL's (domain name & path) are needed in the iSCSI transport to support proxy services. Because of the prevalence of NAT, proxies are necessary. 2) Authentication is a separate issue and has nothing to do with identifying the final destination device/LUN/WWN of the iSCSI traffic. A separate key distribution server may improve scalability of the authentication mechanism, but this has nothing to do with addressing and routing of iSCSI traffic. 3) A LANE-type architecture for addressing and routing of iSCSI traffic is a bad idea due to scalability and management issues. The iSCSI transport must have imbedded routing information in the form of a URL, to allow proxies and destination nodes to route iSCSI traffic to its final device/LUN/WWN destination. Best regards, Josh Tseng -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Tuesday, October 03, 2000 11:13 AM To: Joshua Tseng; ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Joshua, > What you describe might be possible (although I still think it's a bad > idea) if the entire Internet, including all public and private networks, > were in a single consolidated address space. But the fact is we are > running out of address space, and there is something called NAT defined > in RFC1918. Who knows, with IPv6, this may change, or it might not. But > it is a reality today. To operate in an environment with NAT, you need > proxies. There's no way around it. A client in a public network using > registered IP address space should NEVER see a 10.0/8 address. It should > NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 > address entry in its routing table. It must first talk to a dual-homed > proxy with at least one leg using registered IP address space, in order > to communicate with a host with a 10.0/8 address. In this environment > and with these restrictions, I don't understand how you can remove the > involvement of the proxy in the process of what you call "authentication". > > BTW, it's not just http--e-mail and many other applications today make > extensive use of proxy relays as well. > > Josh Yes, and most enterprise environments include a NAT. Even homes with DSL include NAT. A few may even use a proxy. That does not mean private addresses of the target can not be shared at the time of authentication. I would have expected such an exchange. As most of these things work, such permission is in the form of a lease. I would also expect as the map is declared, mapping screens are established based on the permission discovered at the time of authentication. Before and not during use. Using a binary address does not mean PUBLIC addresses. It may not even be IP. It could be SCSI address or perhaps an encoded address. You do not want SCSI to look like an HTTP server. Especially if you wish this application to scale, you do not want to be doing in-band name lookup and authentication. Pleases, this is not a web server, it is a portal to SCSI devices. A client does not need to use a name to get a proxy to listen, try just typing the IP of a web site. The proxy will forgo the lookup. Name lookup is simply a convenience for humans. You would not want to depend on a round-robin selection of IPs from DNS should there be more than one such IP. How would you select the alternative IP, the next in the list? All these parameters can be concisely defined in the authentication exchange. I can not see why someone would wish to place a name on their SCSI portal but they could. The only name that needs to exist is the authentication server. I would not expect an address beyond the SCSI portal to be PUBLIC IPs. I would not expect them to be IP. LDAP is good at doing symbolic lookup. Let it do the work at the time of authentication. Don't invent a SCSI browser. Doug From owner-ips@ECE.cmu.edu Tue Oct 03 20:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21834; 3 Oct 2000 16:19 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA27718 for ips-outgoing; Tue, 3 Oct 2000 15:21:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA27710 for ; Tue, 3 Oct 2000 15:21:53 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA15536 for ; Tue, 3 Oct 2000 15:15:43 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e93JLoa80408 for ; Tue, 3 Oct 2000 13:21:50 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: SCSI URL scheme To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 3 Oct 2000 12:19:19 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/03/2000 01:21:50 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk OK, now I am missing something here. The note below seems to be talking about a Host trying to find a LU and Raghavendra suggest that it is a big effort to do the normal Report Luns and Inquiry. First off, this is suppose to be done (normally) once, but perhaps at various (logical) Plug & Play points. If we are talking about the Plug & Play points in time, then there could be some additional conversation. But independent of that, which so far I think we have been, the Host determines ALL his addressable LUs when he comes up, and today that means "REPORT LUNS", followed by approprate INQUIRYs. This is not a big deal at bring up. I have not followed the turn in the road that would have a Host that should know what is connected to it (and its WWN), needing to find the WWN at some other time. Now it is possible that we are somehow talking about something that is akin to the NAS approach to "Mount" (Or Drive Map). In this mode I expect that the Client Host has been given some Name that needs to get resolved to an LU, or Set of LUs. But even in that case, if the name get resolved so that it targets a given IP Storage Device (Storage Controller), it also should perform a Login, and then it can still perform the normal REPORT LUNs and an approprate Inquiry. In most cases for Desktop systems this should be 1 LU, probably the LU0. But in any event I do not see the problem. The Biggest problem the Host might find is if the information found in the VPD reported by the Storage Controller differs from what is expected. (They call this an administrative error and needs to get fixed by people.) In any event I do not see the value add for the Name to go all the way down to the LU. In fact if does not we can use the normal network processes to connect a Client Host to the approprate Target Storage Controller. Third party commands, however, seem to be a different set of discussion items, so perhaps it would make since to focus on that area where it is not clear what the approach is, rather then work areas that have a technique that (IMHO) work OK. And it might be valuable to discuss the issue of Plug & Play, and how that is affected. (By Plug and Play I mean the process of adding a new LU or taking an LU away while every thing else continues.) . . . John L. Hufferd Raghavendra Rao @ece.cmu.edu on 10/03/2000 10:55:41 AM Please respond to Raghavendra Rao Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > out-of-band). I don't see the point of the initiator sending its "view" to > the target at any point. If the initiator sends an incorrect view, you've > just created an error scenario which needs additional definition. Well, to extend what Julian said further, the initiator could ask the target for a LUN value corresponding to a WWN as part of the authentication process, instead of presenting its view - whether right or wrong. But something like this needs to be done as there is no easier/faster way to translate a LU WWN to a LUN value. This may also help in building an authentication method for a LU access where no such thing exists today ? > case, the target will send its "view" for that initiator in REPORT LUNS > SCSI command after login (which is actually the layer that cares about the > LUN Map at all anyway). > This will be a problem if the storage controller has a lot of LUNs. Since REPORT LUNS only returns a list of LUNs, the initiator still has to send an INQUIRY (page 83) request to each LU to to find a matching LU WWN (or until a matching WWN is found). -JP From owner-ips@ECE.cmu.edu Tue Oct 03 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21847; 3 Oct 2000 16:24 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA29102 for ips-outgoing; Tue, 3 Oct 2000 15:53:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA29094 for ; Tue, 3 Oct 2000 15:53:22 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA55996 for ; Tue, 3 Oct 2000 15:47:10 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e93JrHa79388 for ; Tue, 3 Oct 2000 13:53:17 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: SCSI URL scheme To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 3 Oct 2000 12:50:17 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/03/2000 01:53:16 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Right after I sent the attached, I thought of another reason that folks might me having a problem here, and that is, they might think that the Report Luns reports all the LUNS that are within the Storage Controller. As I understand it, this is not the case, but it only reports the LUs that are authorized to be seen by the questioning Host/Initiator. . . . John L. Hufferd John Hufferd/San Jose/IBM@IBMUS@ece.cmu.edu on 10/03/2000 12:19:19 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme OK, now I am missing something here. The note below seems to be talking about a Host trying to find a LU and Raghavendra suggest that it is a big effort to do the normal Report Luns and Inquiry. First off, this is suppose to be done (normally) once, but perhaps at various (logical) Plug & Play points. If we are talking about the Plug & Play points in time, then there could be some additional conversation. But independent of that, which so far I think we have been, the Host determines ALL his addressable LUs when he comes up, and today that means "REPORT LUNS", followed by approprate INQUIRYs. This is not a big deal at bring up. I have not followed the turn in the road that would have a Host that should know what is connected to it (and its WWN), needing to find the WWN at some other time. Now it is possible that we are somehow talking about something that is akin to the NAS approach to "Mount" (Or Drive Map). In this mode I expect that the Client Host has been given some Name that needs to get resolved to an LU, or Set of LUs. But even in that case, if the name get resolved so that it targets a given IP Storage Device (Storage Controller), it also should perform a Login, and then it can still perform the normal REPORT LUNs and an approprate Inquiry. In most cases for Desktop systems this should be 1 LU, probably the LU0. But in any event I do not see the problem. The Biggest problem the Host might find is if the information found in the VPD reported by the Storage Controller differs from what is expected. (They call this an administrative error and needs to get fixed by people.) In any event I do not see the value add for the Name to go all the way down to the LU. In fact if does not we can use the normal network processes to connect a Client Host to the approprate Target Storage Controller. Third party commands, however, seem to be a different set of discussion items, so perhaps it would make since to focus on that area where it is not clear what the approach is, rather then work areas that have a technique that (IMHO) work OK. And it might be valuable to discuss the issue of Plug & Play, and how that is affected. (By Plug and Play I mean the process of adding a new LU or taking an LU away while every thing else continues.) . . . John L. Hufferd Raghavendra Rao @ece.cmu.edu on 10/03/2000 10:55:41 AM Please respond to Raghavendra Rao Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > out-of-band). I don't see the point of the initiator sending its "view" to > the target at any point. If the initiator sends an incorrect view, you've > just created an error scenario which needs additional definition. Well, to extend what Julian said further, the initiator could ask the target for a LUN value corresponding to a WWN as part of the authentication process, instead of presenting its view - whether right or wrong. But something like this needs to be done as there is no easier/faster way to translate a LU WWN to a LUN value. This may also help in building an authentication method for a LU access where no such thing exists today ? > case, the target will send its "view" for that initiator in REPORT LUNS > SCSI command after login (which is actually the layer that cares about the > LUN Map at all anyway). > This will be a problem if the storage controller has a lot of LUNs. Since REPORT LUNS only returns a list of LUNs, the initiator still has to send an INQUIRY (page 83) request to each LU to to find a matching LU WWN (or until a matching WWN is found). -JP From owner-ips@ECE.cmu.edu Tue Oct 03 21:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21994; 3 Oct 2000 17:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA29692 for ips-outgoing; Tue, 3 Oct 2000 16:09:07 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA29687 for ; Tue, 3 Oct 2000 16:09:03 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id PAA91162 for ; Tue, 3 Oct 2000 15:56:20 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e93K90a126830 for ; Tue, 3 Oct 2000 14:09:00 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 3 Oct 2000 13:04:13 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/03/2000 02:09:00 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Howard, OK, I guess my reading has now caught, up with you, sorry to imply that you did not understand that it was a T10 issue. . . . John L. Hufferd Howard Green @ece.cmu.edu on 10/03/2000 08:55:47 AM Sent by: owner-ips@ece.cmu.edu To: "'Raghavendra Rao'" , ips@ece.cmu.edu cc: Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Sorry for the previous incomplete post. My mailer misfired... I know that the following is kind of a radical question to ask, but has the time arrived to toss out LUNs as a means of identifying logical units, in favor of simply using the logical units' WWNs directly? Back in the good ol' days, LUNs were small integers contained directly in each CDB, and were (in practice, if not in theory) a simple index assigned to a specific logical unit throughout its life. SCSI has obviously evolved to a far more sophisticated state of affairs. The LUNs used to identify a logical unit have grown to 64 bits, and can vary over time, vary relative to the target interface used, and vary as a function of the initiator's identity. Clearly, they're not durable identifiers in any useful sense. Further, they've got to be be set up and administered. My sense here is that they've become just so much bureaucracy, to no particularly useful end. Currently, an IU containing a SCSI command identifies the target logical unit by a Logical Unit Number. So, in effect, the notion here would be to use the logical unit's WWN in place of the LUN in the command IU. The WWNs I've seen used to identify logical units currently range from 64 to 128 bits, so there's no particular additional overhead involved. I think I'm reasonably cognizant of the difficulties imposed by making such a change (changes to SAM, compatibility with existing implementations, etc.)---and that it's really a T10 issue, and not exclusively an iSCSI issue. However, my sense is that the current LUN-based identification mechanism is causing increasing grief, as evidenced by any number of recent postings; and one has to start somewhere... Howard Green > -----Original Message----- > From: Raghavendra Rao [mailto:jpr@divyaroot.India.Sun.COM] > Sent: Tuesday, October 03, 2000 10:56 AM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > out-of-band). I don't see the point of the initiator > sending its "view" to > > the target at any point. If the initiator sends an > incorrect view, you've > > just created an error scenario which needs additional definition. > > Well, to extend what Julian said further, the initiator could > ask the target > for a LUN value corresponding to a WWN as part of the > authentication process, > instead of presenting its view - whether right or wrong. But > something like > this needs to be done as there is no easier/faster way to > translate a LU WWN > to a LUN value. > > This may also help in building an authentication method for a > LU access where > no such thing exists today ? > > > > case, the target will send its "view" for that initiator in > REPORT LUNS > > SCSI command after login (which is actually the layer that > cares about the > > LUN Map at all anyway). > > > > This will be a problem if the storage controller has a lot of > LUNs. Since > REPORT LUNS only returns a list of LUNs, the initiator still > has to send an > INQUIRY (page 83) request to each LU to to find a matching LU > WWN (or until > a matching WWN is found). > > -JP > From owner-ips@ECE.cmu.edu Tue Oct 03 21:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21985; 3 Oct 2000 17:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA29666 for ips-outgoing; Tue, 3 Oct 2000 16:08:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from lsi.lsil.com (lsi.lsil.com [147.145.40.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA29655 for ; Tue, 3 Oct 2000 16:08:35 -0400 (EDT) Received: from mhbs.lsil.com ([147.145.31.100]) by lsi.lsil.com (8.9.3+Sun/8.9.1) with ESMTP id NAA12460 for ; Tue, 3 Oct 2000 13:08:34 -0700 (PDT) Received: from inca.co.lsil.com by mhbs.lsil.com with ESMTP for ips@ece.cmu.edu; Tue, 3 Oct 2000 13:08:32 -0700 Received: from exw-kansas.ks.lsil.com (exw-kansas.ks.lsil.com [153.79.8.7]) by inca.co.lsil.com (8.9.3/8.9.3) with ESMTP id OAA15631 for ; Tue, 3 Oct 2000 14:08:29 -0600 (MDT) Received: by exw-kansas.ks.lsil.com with Internet Mail Service (5.5.2650.21) id ; Tue, 3 Oct 2000 15:05:19 -0500 Message-Id: From: "Binford, Charles" To: IPS Reflector Subject: RE: iSCSI sessions: Let's try again Date: Tue, 3 Oct 2000 15:05:18 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C02D75.40EE4BDA" Sender: owner-ips@ece.cmu.edu Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C02D75.40EE4BDA Content-Type: text/plain; charset="iso-8859-1" Doug, What "command window" are you referring to? I know of no such mechanism in SCSI. Does "this scheme" refer to something in iSCSI that will regulate the flow of new commands to a target, something is generic SCSI (i.e. SAM/SPC-2, etc), or something in TCP? Charles Binford LSI Logic Storage Systems (316) 636-8566 > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Tuesday, October 03, 2000 12:41 PM > To: Matt Wakeley; IPS Reflector > Subject: RE: iSCSI sessions: Let's try again > > > Matt, > > From the little I understand, there is a command window built > into this > scheme which should act as a means to regulate flow of > commands. You should > not need an additional connection to perform that function. > I do not think > stuffing receive buffers is a good technique but it is a good > system test. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Matt Wakeley > > > Sent: Tuesday, October 03, 2000 12:37 AM > > To: Douglas Otis; IPS Reflector > > Subject: Re: iSCSI sessions: Let's try again > > > > > > > > > > Douglas Otis wrote: > > > > > > Reasons requiring 2 tcp connections per iSCSI session (one for > > > > commands the > > > > other for data): > > > > > > > > 1- It allows a target to temporarily flow control commands from > > > > the initiator > > > > while still allowing data to continue on the "data" > > connection (allowing > > > > commands to complete so that the flow control can be > opened again). > > > > > > How do you go about managing the buffers of these > connections using TCP? > > > > Doug, even you know that if you stop reading > commands> from > > the TCP stream, TCP will eventually stop the initiator from > > sending commands. > > Thus, if the SCSI command queue fills up, the target stops > > reading from TCP, > > which eventually shuts down the initiator from sending more > commands. > > > > -Matt > > > > > > ------_=_NextPart_001_01C02D75.40EE4BDA Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: iSCSI sessions: Let's try again

Doug,
  What "command window" are you = referring to?  I know of no such mechanism in SCSI.  Does = "this scheme" refer to something in iSCSI that will regulate = the flow of new commands to a target, something is generic SCSI (i.e. = SAM/SPC-2, etc), or something in TCP?

Charles Binford
LSI Logic Storage Systems
(316) 636-8566


> -----Original Message-----
> From: Douglas Otis [mailto:dotis@sanlight.net]=
> Sent: Tuesday, October 03, 2000 12:41 PM
> To: Matt Wakeley; IPS Reflector
> Subject: RE: iSCSI sessions: Let's try = again
>
>
> Matt,
>
> From the little I understand, there is a = command window built
> into this
> scheme which should act as a means to regulate = flow of
> commands.  You should
> not need an additional connection to perform = that function. 
> I do not think
> stuffing receive buffers is a good technique = but it is a good
> system test.
>
> Doug
>
> > -----Original Message-----
> > From: owner-ips@ece.cmu.edu
> [mailto:owner-ips@ece.cmu.edu]O= n Behalf Of
> > Matt Wakeley
> >
> Sent: Tuesday, October 03, 2000 12:37 AM
> > To: Douglas Otis; IPS Reflector
> > Subject: Re: iSCSI sessions: Let's try = again
> >
> >
> >
> >
> > Douglas Otis wrote:
> >
> > > > Reasons requiring 2 tcp = connections per iSCSI session (one for
> > > > commands the
> > > > other for data):
> > > >
> > > > 1- It allows a target to = temporarily flow control commands from
> > > > the initiator
> > > > while still allowing data to = continue on the "data"
> > connection (allowing
> > > > commands to complete so that the = flow control can be
> opened again).
> > >
> > > How do you go about managing the = buffers of these
> connections using TCP?
> >
> > Doug, even you know that if you stop = reading <in this case the
> > commands> from
> > the TCP stream, TCP will eventually stop = the initiator from
> > sending commands.
> > Thus, if the SCSI command queue fills up, = the target stops
> > reading from TCP,
> > which eventually shuts down the initiator = from sending more
> commands.
> >
> > -Matt
> >
> >
>
>

------_=_NextPart_001_01C02D75.40EE4BDA-- From owner-ips@ECE.cmu.edu Tue Oct 03 21:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa21987; 3 Oct 2000 17:10 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA29511 for ips-outgoing; Tue, 3 Oct 2000 16:03:58 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA29504 for ; Tue, 3 Oct 2000 16:03:51 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id PAA77290 for ; Tue, 3 Oct 2000 15:57:39 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e93K3ka153464 for ; Tue, 3 Oct 2000 14:03:46 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 3 Oct 2000 13:00:35 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/03/2000 02:03:45 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Howard Green, I just put in a couple of note asking some questions about this. (The thread was changed to :SCSI URL scheme.) Those thoughts and question might have some barring here. In any event, assuming your idea is valuable, I think you are talking about major changes to the CDB structure, and that should be deferred to the T10 committee. . . . John L. Hufferd Howard Green @ece.cmu.edu on 10/03/2000 08:35:15 AM Sent by: owner-ips@ece.cmu.edu To: "'Raghavendra Rao'" , ips@ece.cmu.edu cc: Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] I know that the following is kind of a radical question to ask, but has the time arrived to toss out LUNs as a means of identifying logical units, in favor of simply using the logical units' WWNs directly? Currently, an IU containing a SCSI command identifies the target logical unit by a Logical Unit Number. Back in the good ol' days, LUNs were small integers contained directly in each CDB, and were (in practice, if not in theory) a simple index assigned to a specific logical unit throughout its life. SCSI has obviously evolved to a far more sophisticated state of affairs. The LUNs have grown to 64 bits, opaque used to identify a logical unit can vary over time, relative to the interface used, and as a function of the initiator's identity. The WWNs I've seen used to identify logical units currently range from 64 to 128 bits, > -----Original Message----- > From: Raghavendra Rao [mailto:jpr@divyaroot.India.Sun.COM] > Sent: Tuesday, October 03, 2000 10:56 AM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > > out-of-band). I don't see the point of the initiator > sending its "view" to > > the target at any point. If the initiator sends an > incorrect view, you've > > just created an error scenario which needs additional definition. > > Well, to extend what Julian said further, the initiator could > ask the target > for a LUN value corresponding to a WWN as part of the > authentication process, > instead of presenting its view - whether right or wrong. But > something like > this needs to be done as there is no easier/faster way to > translate a LU WWN > to a LUN value. > > This may also help in building an authentication method for a > LU access where > no such thing exists today ? > > > > case, the target will send its "view" for that initiator in > REPORT LUNS > > SCSI command after login (which is actually the layer that > cares about the > > LUN Map at all anyway). > > > > This will be a problem if the storage controller has a lot of > LUNs. Since > REPORT LUNS only returns a list of LUNs, the initiator still > has to send an > INQUIRY (page 83) request to each LU to to find a matching LU > WWN (or until > a matching WWN is found). > > -JP > From owner-ips@ECE.cmu.edu Tue Oct 03 22:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22151; 3 Oct 2000 18:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA02477 for ips-outgoing; Tue, 3 Oct 2000 17:18:22 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1tx.cos.agilent.com (msgbas1tx.cos.agilent.com [192.6.9.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA02466 for ; Tue, 3 Oct 2000 17:18:12 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1tx.cos.agilent.com (Postfix) with ESMTP id 8F08411F for ; Tue, 3 Oct 2000 15:18:11 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 10FFA19F for ; Tue, 3 Oct 2000 17:18:10 -0400 (EDT) Received: from agilent.com (cos1nai248122.cs.itc.hp.com [15.3.248.122]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA15543 for ; Tue, 3 Oct 2000 14:18:06 -0700 (PDT) Message-ID: <39DA4C5C.B967182B@agilent.com> Date: Tue, 03 Oct 2000 14:15:08 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: IPS Reflector Subject: Re: iSCSI sessions: Let's try again References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis wrote: > Matt, > > From the little I understand, there is a command window built into this > scheme which should act as a means to regulate flow of commands. There is not a window in generic SCSI. iSCSI proposed one (the command reference numbers) and everyone hated it. Thus the idea of using a single tcp connection only for commands solves the problem in the absense of command reference numbers. -Matt > You should > not need an additional connection to perform that function. I do not think > stuffing receive buffers is a good technique but it is a good system test. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Matt Wakeley > > Sent: Tuesday, October 03, 2000 12:37 AM > > To: Douglas Otis; IPS Reflector > > Subject: Re: iSCSI sessions: Let's try again > > > > > > > > > > Douglas Otis wrote: > > > > > > Reasons requiring 2 tcp connections per iSCSI session (one for > > > > commands the > > > > other for data): > > > > > > > > 1- It allows a target to temporarily flow control commands from > > > > the initiator > > > > while still allowing data to continue on the "data" > > connection (allowing > > > > commands to complete so that the flow control can be opened again). > > > > > > How do you go about managing the buffers of these connections using TCP? > > > > Doug, even you know that if you stop reading > commands> from > > the TCP stream, TCP will eventually stop the initiator from > > sending commands. > > Thus, if the SCSI command queue fills up, the target stops > > reading from TCP, > > which eventually shuts down the initiator from sending more commands. > > > > -Matt > > > > From owner-ips@ECE.cmu.edu Tue Oct 03 22:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22153; 3 Oct 2000 18:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA02799 for ips-outgoing; Tue, 3 Oct 2000 17:25:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA02794 for ; Tue, 3 Oct 2000 17:25:09 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id OAA16230; Tue, 3 Oct 2000 14:11:03 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id OAA22645; Tue, 3 Oct 2000 14:08:51 -0700 From: "Douglas Otis" To: "Binford, Charles" , "IPS Reflector" Subject: RE: iSCSI sessions: Let's try again Date: Tue, 3 Oct 2000 14:27:38 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Charles, MaxCmdRN - the maximum number to be shipped - defines the queuing capacity of the receiving iSCSI. CmdRN can take any value from ExpCmdRN to MaxCmdRN except 0. iSCSI targets are required to provide ExpCmdRN and MaxCmdRN values that will enable the initiator to make progress. I do not like mixing EE with BB controls as it requires far greater effort. :( Doug -----Original Message----- From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of Binford, Charles Sent: Tuesday, October 03, 2000 1:05 PM To: IPS Reflector Subject: RE: iSCSI sessions: Let's try again Doug, What "command window" are you referring to? I know of no such mechanism in SCSI. Does "this scheme" refer to something in iSCSI that will regulate the flow of new commands to a target, something is generic SCSI (i.e. SAM/SPC-2, etc), or something in TCP? Charles Binford LSI Logic Storage Systems (316) 636-8566 > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Tuesday, October 03, 2000 12:41 PM > To: Matt Wakeley; IPS Reflector > Subject: RE: iSCSI sessions: Let's try again > > > Matt, > > From the little I understand, there is a command window built > into this > scheme which should act as a means to regulate flow of > commands. You should > not need an additional connection to perform that function. > I do not think > stuffing receive buffers is a good technique but it is a good > system test. > > Doug > > > -----Original Message----- > > From: owner-ips@ece.cmu.edu > [mailto:owner-ips@ece.cmu.edu]On Behalf Of > > Matt Wakeley > > > Sent: Tuesday, October 03, 2000 12:37 AM > > To: Douglas Otis; IPS Reflector > > Subject: Re: iSCSI sessions: Let's try again > > > > > > > > > > Douglas Otis wrote: > > > > > > Reasons requiring 2 tcp connections per iSCSI session (one for > > > > commands the > > > > other for data): > > > > > > > > 1- It allows a target to temporarily flow control commands from > > > > the initiator > > > > while still allowing data to continue on the "data" > > connection (allowing > > > > commands to complete so that the flow control can be > opened again). > > > > > > How do you go about managing the buffers of these > connections using TCP? > > > > Doug, even you know that if you stop reading > commands> from > > the TCP stream, TCP will eventually stop the initiator from > > sending commands. > > Thus, if the SCSI command queue fills up, the target stops > > reading from TCP, > > which eventually shuts down the initiator from sending more > commands. > > > > -Matt > > > > > > From owner-ips@ECE.cmu.edu Tue Oct 03 22:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22155; 3 Oct 2000 18:13 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA02467 for ips-outgoing; Tue, 3 Oct 2000 17:18:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.6.9.33]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA02461 for ; Tue, 3 Oct 2000 17:18:10 -0400 (EDT) Received: from andom1.an.hp.com (andom1.an.hp.com [15.4.128.104]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 14D1C154; Tue, 3 Oct 2000 15:18:07 -0600 (MDT) Received: from rtl.rose.agilent.com (rtl.rose.agilent.com [156.140.232.231]) by andom1.an.hp.com (Postfix) with ESMTP id 87FF3155; Tue, 3 Oct 2000 17:18:05 -0400 (EDT) Received: from agilent.com (cos1nai248122.cs.itc.hp.com [15.3.248.122]) by rtl.rose.agilent.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.1.0) with ESMTP id OAA15520; Tue, 3 Oct 2000 14:18:00 -0700 (PDT) Message-ID: <39DA4B18.80502A46@agilent.com> Date: Tue, 03 Oct 2000 14:09:44 -0700 From: Matt Wakeley Reply-To: Matt Wakeley Organization: Agilent Technologies X-Mailer: Mozilla 4.75 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Jim Hafner/Almaden/IBM , ips@ece.cmu.edu Subject: Re: Bidirectional SCSI commands and iSCSI References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk or 3) make the header *NOT* contain the CDB. Instead, let the CDB be part of the iSCSI command message payload (like it is in FC). -Matt Jim Hafner/Almaden/IBM wrote: > Folks, > > I'm not advocating variable length headers here to support bi-d commands. > More to the point, I'm suggesting that the fixed length header change (NOW) > to accomodate the bi-directional SCSI commands. Better now while the draft > is still in flux than later after its finalized and more hardware gets > built. > > BTW, I see two approaches: > 1) one universal header with enough fields for both directions > 2) one header for classical uni-d commands and an additional header for > bi-d commands (as FCP is proposing) > > My vote would be for option (1). > > Jim Hafner > > Stephen Bailey @ece.cmu.edu on 10-03-2000 07:12:46 > AM > > Sent by: owner-ips@ece.cmu.edu > > To: ips@ece.cmu.edu > cc: > Subject: Re: Bidirectional SCSI commands and iSCSI > > > I think that software initiators will be widely used and they are far > > more efficient with fixed length headers for widely used operations. > > In ST, fixed length headers were a design choice to permit hardware > acceleration. > > Other than on very slow links, it's hard to argue with the performance > advantages of fixed length headers under any circumstances. > > Steph From owner-ips@ECE.cmu.edu Tue Oct 03 22:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22170; 3 Oct 2000 18:17 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA02239 for ips-outgoing; Tue, 3 Oct 2000 17:11:26 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id RAA02229 for ; Tue, 3 Oct 2000 17:11:17 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id NAA16222; Tue, 3 Oct 2000 13:57:12 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id NAA22606; Tue, 3 Oct 2000 13:55:00 -0700 From: "Douglas Otis" To: "Joshua Tseng" , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 14:13:47 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, > Doug, > > I'm not sure we understand each other anymore. I will just carefully > restate my points, and leave it at that. > > 1) URL's (domain name & path) are needed in the iSCSI transport to > support proxy services. Because of the prevalence of NAT, proxies > are necessary. Most likely, a DNS response will be inadequate in providing needed information for making a connection if it is more than a simple IP. At this point, there are no SCSI proxy servers that rely on names so there is no prevalent use making a SCSI address translation which depends on a SCSI name server. This is an invention you are suggesting. You are suggesting the use of names is required. That is not true! You are recommending mapping SCSI drives into a directory scheme as denoted by your domain/path. You insist this is the only means of addressing SCSI drives. See: http://www.ietf.org/rfc/rfc2251.txt Lightweight Directory Access Protocol (LDAP) is a distributed, hierarchical directory service access protocol used to access repositories of users and other network related entities. Information such as users found in NIS (flat file) should derive from LDAP as authoritive to take advantage of LDAP scalability. The "DUA" (directory user agent) refers to the LDAP client querying these entities, such as an LDAP to NIS gateway or SCSI services. It is irrelevant whether the DUA and the client reside within the same address space. The DUA giving this information to the client is termed "republishing". You *WILL* map drives to directories because this is how LDAP works! You still have the joy of inventing names to act as place holders. The important aspect of making use of this server, you discover all the mappings before making connection to this proxy or portal. (Access to the portal MUST be routable from the gateway or there would not be any means to connect nor would access to the portal depend on embedded names.) By doing so, there is no need for the portal to do any lookups real-time or need to embed names and in doing so weakening security. If you want a SCSI transport to scale, do not expect name translations done in real-time. > 2) Authentication is a separate issue and has nothing to do with > identifying the final destination device/LUN/WWN of the iSCSI traffic. > A separate key distribution server may improve scalability of the > authentication mechanism, but this has nothing to do with addressing > and routing of iSCSI traffic. Each name must be examined and resolved before an authentication check can be made. By not embedding names, the address would already be resolved by LDAP in advance and far easier to check. > 3) A LANE-type architecture for addressing and routing of iSCSI > traffic is a bad idea due to scalability and management issues. > The iSCSI transport must have imbedded routing information in the > form of a URL, to allow proxies and destination nodes to route > iSCSI traffic to its final device/LUN/WWN destination. It will take less effort to re-route a pre-resolved address than to first resolve the address and then remap it. If the portal only needs the device/lun/wwn to do routing then that should be all that is given. LDAP does not constrain the information embedded, only that it should not be resolved by the portal via your invented SCSI name server. We are not here to re-invent the wheel and it makes no sense to do this live should you wish this equipment to scale. Doug > Best regards, > Josh Tseng > > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Tuesday, October 03, 2000 11:13 AM > To: Joshua Tseng; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Joshua, > > > > What you describe might be possible (although I still think it's a bad > > idea) if the entire Internet, including all public and private networks, > > were in a single consolidated address space. But the fact is we are > > running out of address space, and there is something called NAT defined > > in RFC1918. Who knows, with IPv6, this may change, or it might > not. But > > it is a reality today. To operate in an environment with NAT, you need > > proxies. There's no way around it. A client in a public network using > > registered IP address space should NEVER see a 10.0/8 address. > It should > > NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 > > address entry in its routing table. It must first talk to a dual-homed > > proxy with at least one leg using registered IP address space, in order > > to communicate with a host with a 10.0/8 address. In this environment > > and with these restrictions, I don't understand how you can remove the > > involvement of the proxy in the process of what you call > "authentication". > > > > BTW, it's not just http--e-mail and many other applications today make > > extensive use of proxy relays as well. > > > > Josh > > Yes, and most enterprise environments include a NAT. Even homes with DSL > include NAT. A few may even use a proxy. That does not mean private > addresses of the target can not be shared at the time of > authentication. I > would have expected such an exchange. As most of these things work, such > permission is in the form of a lease. I would also expect as the map is > declared, mapping screens are established based on the permission > discovered > at the time of authentication. Before and not during use. Using a binary > address does not mean PUBLIC addresses. It may not even be IP. > It could be > SCSI address or perhaps an encoded address. You do not want SCSI to look > like an HTTP server. Especially if you wish this application to > scale, you > do not want to be doing in-band name lookup and authentication. > > Pleases, this is not a web server, it is a portal to SCSI > devices. A client > does not need to use a name to get a proxy to listen, try just > typing the IP > of a web site. The proxy will forgo the lookup. Name lookup is simply a > convenience for humans. You would not want to depend on a round-robin > selection of IPs from DNS should there be more than one such IP. > How would > you select the alternative IP, the next in the list? All these parameters > can be concisely defined in the authentication exchange. I can > not see why > someone would wish to place a name on their SCSI portal but they > could. The > only name that needs to exist is the authentication server. I would not > expect an address beyond the SCSI portal to be PUBLIC IPs. I would not > expect them to be IP. LDAP is good at doing symbolic lookup. > Let it do the > work at the time of authentication. Don't invent a SCSI browser. > > Doug > From owner-ips@ECE.cmu.edu Tue Oct 03 22:30:02 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22183; 3 Oct 2000 18:23 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id RAA03830 for ips-outgoing; Tue, 3 Oct 2000 17:51:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from apollo.pirus.com ([63.91.118.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA00500 for ; Tue, 3 Oct 2000 16:27:30 -0400 (EDT) Message-ID: <618471D08ABDD31188B6009027E5009352A13B@apollo.pirus.com> From: "Merhar, Milan" To: ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 16:40:03 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk I'd like to chime in with my agreement, worded a bit differently: 1) I believe large systems (spanning the Internet) will at some point need some form of proxy/gateway/NAT because: - addresses in one realm won't be legal in another realm - the owner of a realm does not want any of its internals revealed - those internals are changeable, and the owner wants to isolate outsiders from the impact of those changes. 2) Authentication, privacy (a.k.a. encryption), etc are indeed a separate issue from end-point naming. 2a) Information obtained by observing and parsing an embedded identifier should have no impact on overall security - We aren't relying on "security by obscurity," after all. 2b) ditto, whether those identifiers are human-readable or not. 3) Let's assume that the available tools and techniques of DNS, URLs, etc are there to be leveraged, and let's see where it takes us. I'd rather spend our time delivering a robust application, rather than inventing optimized replacements for common Internet tools. - Milan Merhar -----Original Message----- From: Joshua Tseng [mailto:jtseng@NishanSystems.com] Sent: Tuesday, October 03, 2000 2:43 PM To: Douglas Otis; ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Doug, I'm not sure we understand each other anymore. I will just carefully restate my points, and leave it at that. 1) URL's (domain name & path) are needed in the iSCSI transport to support proxy services. Because of the prevalence of NAT, proxies are necessary. 2) Authentication is a separate issue and has nothing to do with identifying the final destination device/LUN/WWN of the iSCSI traffic. A separate key distribution server may improve scalability of the authentication mechanism, but this has nothing to do with addressing and routing of iSCSI traffic. 3) A LANE-type architecture for addressing and routing of iSCSI traffic is a bad idea due to scalability and management issues. The iSCSI transport must have imbedded routing information in the form of a URL, to allow proxies and destination nodes to route iSCSI traffic to its final device/LUN/WWN destination. Best regards, Josh Tseng -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Tuesday, October 03, 2000 11:13 AM To: Joshua Tseng; ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Joshua, > What you describe might be possible (although I still think it's a bad > idea) if the entire Internet, including all public and private networks, > were in a single consolidated address space. But the fact is we are > running out of address space, and there is something called NAT defined > in RFC1918. Who knows, with IPv6, this may change, or it might not. But > it is a reality today. To operate in an environment with NAT, you need > proxies. There's no way around it. A client in a public network using > registered IP address space should NEVER see a 10.0/8 address. It should > NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 > address entry in its routing table. It must first talk to a dual-homed > proxy with at least one leg using registered IP address space, in order > to communicate with a host with a 10.0/8 address. In this environment > and with these restrictions, I don't understand how you can remove the > involvement of the proxy in the process of what you call "authentication". > > BTW, it's not just http--e-mail and many other applications today make > extensive use of proxy relays as well. > > Josh Yes, and most enterprise environments include a NAT. Even homes with DSL include NAT. A few may even use a proxy. That does not mean private addresses of the target can not be shared at the time of authentication. I would have expected such an exchange. As most of these things work, such permission is in the form of a lease. I would also expect as the map is declared, mapping screens are established based on the permission discovered at the time of authentication. Before and not during use. Using a binary address does not mean PUBLIC addresses. It may not even be IP. It could be SCSI address or perhaps an encoded address. You do not want SCSI to look like an HTTP server. Especially if you wish this application to scale, you do not want to be doing in-band name lookup and authentication. Pleases, this is not a web server, it is a portal to SCSI devices. A client does not need to use a name to get a proxy to listen, try just typing the IP of a web site. The proxy will forgo the lookup. Name lookup is simply a convenience for humans. You would not want to depend on a round-robin selection of IPs from DNS should there be more than one such IP. How would you select the alternative IP, the next in the list? All these parameters can be concisely defined in the authentication exchange. I can not see why someone would wish to place a name on their SCSI portal but they could. The only name that needs to exist is the authentication server. I would not expect an address beyond the SCSI portal to be PUBLIC IPs. I would not expect them to be IP. LDAP is good at doing symbolic lookup. Let it do the work at the time of authentication. Don't invent a SCSI browser. Doug From owner-ips@ECE.cmu.edu Tue Oct 03 22:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22224; 3 Oct 2000 18:35 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA04367 for ips-outgoing; Tue, 3 Oct 2000 18:04:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from maho3msx2.isus.emc.com (maho3msx2.isus.emc.com [168.159.208.81]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA04359 for ; Tue, 3 Oct 2000 18:04:04 -0400 (EDT) From: Black_David@emc.com Received: by maho3msx2.isus.emc.com with Internet Mail Service (5.5.2650.21) id <4GSADPDG>; Tue, 3 Oct 2000 18:03:21 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101052@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: iSCSI Naming and Discovery Date: Tue, 3 Oct 2000 18:03:19 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The emails from Daniel Smith and Josh Tseng provide a good start on a naming discussion. With my co-chair hat off, let me try to add to it. The concepts surrounding NAT (Network Address Translation) figure strongly in this discussion. RFC 2663 is a good source of background for anyone who needs it. An attempt to group and summarize what I see in Daniel and Josh's emails: [1] Internet infrastructure is beyond the control of storage, including iSCSI [Daniel - a),b),f),g),h)] [2] DNS is the naming structure of the Internet, and hence hosts exporting storage services have to be nameable via DNS [Daniel c)]. [3] Network Address Translation and Network Address Port Translation exist and have to be dealt with [Daniel d), Josh (1)]. [4] Discovery at Internet scale is a hard problem. A flexible naming mechanism keeps options open. [Daniel e)]. [5] SCSI 3rd party commands need to name LUNs, including iSCSI LUNs [Daniel i)] *[6] Source, destination, and contents of any packet on the Internet are public information [Daniel j)]. *[7] Security matters for SSPs [Daniel k)]. [8] SSPs can be expected to connect a lot of clients to a single DNS address [Daniel l)]. *[9] Identification of storage and authentication are separate problems and use separate names. [Josh (2)] [10] Naming/routing information is necessary for proxies to identify which of the entities they are proxying for is involved in traffic [1] and [2] are general descriptions of the Internet. [6], [7], and [9] are (mostly) about security, and aside from noting that [6] is incorrect (all of that info can be hidden by a security gateway using IPsec tunnels and both IPsec and SSL/TLS hide payloads), security discussion might be better deferred as I understand that there will be a serious security proposal in the next version of the iSCSI draft. The biggest underlying problem seems to be how to identify an Initiator or Target - it's part of [3], [5], [8] and [10]. [4] is about Discovery, which compounds the naming issues. At a high level there are three basic ways to identify initiators and targets: - Transport address (e.g., IP address, FC port WWN). - Identification information provided as part of session establishment (e.g., username/password, certificate). - Some combination of the above two. The third alternative may be problematic if it leads to needing both the transport and identification information to determine identity. The discussions I've seen seem to be using transport as a hint that may make the identification easier to verify, which seems like a reasonable optimization to relying on the identification information alone. Moving beyond this (e.g., the *.xyz.com servers may only connect from addresses in the a.b.c/24 netblock) increases the amount of information that has to be configured (e.g., that example is better left to firewalls to enforce). NAT mechanisms contribute to the problem by producing networks in which transport addresses aren't useful for identifying anything on the other side of the NAT. Some NATs can be configured to make this problem somewhat simpler via static assignment of IP addresses in one domain to IP addresses in another. NATs are a thorny subject in IETF - while they are widely deployed, not all the important protocols work through them; IPsec AH is the most notable example, and FTP requires a kludge (er, ah, ALG). IMHO, restrictions on the use of NATs with iSCSI may be ok, provided that the consequences are clearly understood. NATs seem to be a large piece of the forcing function that is leading us away from the transport-based identification information used by other SCSI transports. There are security consequences here (e.g., cryptography for session establishment may become mandatory to implement and use), and it will likely complicate discovery. For example, I noted that booting is an issue - if iSCSI always uses URLs to name storage, the result could be a situation in which a DNS server has to be operational and reachable in order to boot. This seems wrong, and the obvious answer of using an IP address in a URL does not work through NATs, which was the original motivation for using URLs. Third-party naming is a tarpit. Putting my WG co-chair hat back on for this paragraph only, I observe that global context for third party names is an unsolved problem in T10; in general, the Initiator of a 3rd party command must use names that resolve to the desired LUNs from the 3rd party command Target's naming perspective. How to do this when an Initiator and Target don't share a naming context is unspecified :-). While it would be a plus for iSCSI to solve this one via have global names for 3rd party commands, I don't think this is a requirement (and whatever we do will have to be worked through T10, as they have the final say on name formats). WG co-chair hat now comes off ... IMHO, discovery is not getting enough attention. The proposed naming scheme complicates discovery without a compelling solution; I'm concerned that the benefits may not justify the costs. My quasi-random walk through this goes something like: - Not being able to find the boot volume because DNS is down or not responding seems to be a wrong answer. The infamous World Wide Wait is bad enough for a browser; it's unacceptable for a reboot. - Hence the boot volume has to be locatable via IP, a TCP port (which could be implicit, e.g. the default iSCSI port) and a LUN (which could be implicit, e.g., LUN 0). Unlike DNS, I could see putting this into an iSCSI HBA card BIOS. - Consistency suggests the approach of using IP addresses as the primary means of identifying possible storage locations, with the possible addition of non-default TCP port numbers). A nice consequence of this is that one can use ranges (e.g., all the storage is in netblock a.b.c/24 at the default iSCSI port, scan those 256 addresses as part of discovery on boot). The corresponding range wildcarding mechanisms for URLs will be more complex. Even if a centralized configuration repository is used (like Fibre Channel), this sort of address wildcarding still looks useful in managing the repository. The netblock example may be too coarse a wildcard. Returning to the issues at the top of this message: [3] NAT become an issue for network designers/admins. Storage becomes something else that they have to get the IP addressing correct for :-(. There are precedents for this, as the default gateway and DNS resolver are already configured via IP addresses, and if those things move to different IP addresses, stuff breaks (a browser can be very unhappy if its host thinks 0.0.0.0 is the only DNS resolver). The downside is that a centralized config repository containing IP addresses becomes a NAT issue - the ALG required to access that across a NAT is ugly enough that it may be necessary to configure the network so that this never happens (which is not the best answer, but may be workable). [4] Discovery based on IP addresses looks like it works for boot volumes in a way that URLs don't and scales via wildcarding in a fashion superior to URLs. An underlying assumption I'm making is that storage discovery doesn't need to match the scale of DNS, and hence centralizing config info isn't hobbled by NAT issues. [5] Use of IP addresses would better match the other 3rd party addressing modes, and removes a dependency of the third party Target on DNS. The problems created by NATs are similar to problems that already exist in 3rd party addressing, and hence this doesn't make things worse. [8] An IP connection is identified by 2 IP addresses and two ports, the fact that several thousand of them go through a common DNS, or even a common IP address is not a problem. With the exception of the comment on T10 and 3rd party naming, this is all IMHO. Fire away ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Oct 04 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22435; 3 Oct 2000 20:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA06844 for ips-outgoing; Tue, 3 Oct 2000 19:24:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from antipater.hosting.pacbell.net (antipater.hosting.pacbell.net [216.100.99.13]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA05772 for ; Tue, 3 Oct 2000 18:46:46 -0400 (EDT) Received: from VENKAT (64-160-62-156.rhapsodynetworks.com [64.160.62.156] (may be forged)) by antipater.hosting.pacbell.net id SAA06609; Tue, 3 Oct 2000 18:46:43 -0400 (EDT) [ConcentricHost SMTP Relay 1.7] From: "Venkat Rangan" To: Cc: Subject: Re: Status summary on multiple connections Date: Tue, 3 Oct 2000 15:47:08 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 Sender: owner-ips@ece.cmu.edu Precedence: bulk > David: > > If a TCP stack were to allow delivery out of order... and I don't > see how it could since by definition it is a stream of bytes > sent in order... it would have to keep track of the segments > it has received and prevent duplicates from being propagated up > to the application above. As you state, SACK helps, but it can NOT > eliminate the duplicates that MAY show up. I think the TCP stack > itself would HAVE to track all of this not the application... > > I really do NOT see how one can allow a TCP to deliver out of order > though, oh > I know technically how to do it, but it is a fundamental violation > of what TCP is supposed to be delivering.... i.e. the bytes being > transfered from 0 to N... David, I think the discussion is about command reordering issues in the Symmetric model case, with multiple connections. Since commands within a single iSCSI session are distributed on multiple TCP connections, it is quite likely that one TCP connection delivers ahead of another even if each TCP connection delivers an ordered byte stream. Regards, Venkat Rangan venkat@rhapsodynetworks.com www.rhapsodynetworks.com 510-249-0197 From owner-ips@ECE.cmu.edu Wed Oct 04 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22428; 3 Oct 2000 20:00 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA05710 for ips-outgoing; Tue, 3 Oct 2000 18:44:44 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA05705 for ; Tue, 3 Oct 2000 18:44:39 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id PAA16265; Tue, 3 Oct 2000 15:30:30 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id PAA22857; Tue, 3 Oct 2000 15:28:18 -0700 From: "Douglas Otis" To: "Merhar, Milan" , Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Date: Tue, 3 Oct 2000 15:42:55 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <618471D08ABDD31188B6009027E5009352A13B@apollo.pirus.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Milan, > I'd like to chime in with my agreement, worded a bit differently: > > 1) I believe large systems (spanning the Internet) will at some point > need some form of proxy/gateway/NAT because: > - addresses in one realm won't be legal in another realm > - the owner of a realm does not want any of its internals revealed > - those internals are changeable, and the owner wants to isolate > outsiders from the impact of those changes. In suggesting a desire to allow dynamic routing, you are insisting on designing a new type of name server. At some point, even the names would need to be shared with the client. To maintain these names and internal addresses, LDAP provides this services. LDAP would allow isolation and control in both the realm of the client and the provider. > 2) Authentication, privacy (a.k.a. encryption), etc are indeed a separate > issue from end-point naming. Yes, but you would need to resolve the name before checking the authentication. You have made the process of checking and routing more difficult. It is also unlikely that the address space will look anything like an IP address. It is also likely some vendors may require additional parameters shared between the client and provider. LDAP allows for such sharing. > 2a) Information obtained by observing and parsing an embedded identifier > should have no impact on overall security - We aren't relying on > "security by obscurity," after all. You are making security weaker by any measure. Why insist on adding a parser to SCSI transport? Let LDAP do the parsing and providing the directory structures. Why re-invent this? > 2b) ditto, whether those identifiers are human-readable or not. > > > 3) Let's assume that the available tools and techniques of DNS, URLs, > etc are there to be leveraged, and let's see where it takes us. > I'd rather spend our time delivering a robust application, > rather than inventing optimized replacements for common > Internet tools. In my knowledge, there is no SCSI name server. Although there are pre-existing concepts of DNS and URLs, these do not apply to SCSI, they apply to IP. This is not IP, this is SCSI. If you wish leverage, use LDAP. Otherwise, you will spend your time parsing script, managing databases, and making access tools for sharing this information between clients and providers. Doug > > - Milan Merhar > > > > -----Original Message----- > From: Joshua Tseng [mailto:jtseng@NishanSystems.com] > Sent: Tuesday, October 03, 2000 2:43 PM > To: Douglas Otis; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Doug, > > I'm not sure we understand each other anymore. I will just carefully > restate my points, and leave it at that. > > 1) URL's (domain name & path) are needed in the iSCSI transport to > support proxy services. Because of the prevalence of NAT, proxies > are necessary. > > 2) Authentication is a separate issue and has nothing to do with > identifying the final destination device/LUN/WWN of the iSCSI traffic. > A separate key distribution server may improve scalability of the > authentication mechanism, but this has nothing to do with addressing > and routing of iSCSI traffic. > > 3) A LANE-type architecture for addressing and routing of iSCSI > traffic is a bad idea due to scalability and management issues. > The iSCSI transport must have imbedded routing information in the > form of a URL, to allow proxies and destination nodes to route > iSCSI traffic to its final device/LUN/WWN destination. > > Best regards, > Josh Tseng > > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Tuesday, October 03, 2000 11:13 AM > To: Joshua Tseng; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Joshua, > > > > What you describe might be possible (although I still think it's a bad > > idea) if the entire Internet, including all public and private networks, > > were in a single consolidated address space. But the fact is we are > > running out of address space, and there is something called NAT defined > > in RFC1918. Who knows, with IPv6, this may change, or it might > not. But > > it is a reality today. To operate in an environment with NAT, you need > > proxies. There's no way around it. A client in a public network using > > registered IP address space should NEVER see a 10.0/8 address. > It should > > NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 > > address entry in its routing table. It must first talk to a dual-homed > > proxy with at least one leg using registered IP address space, in order > > to communicate with a host with a 10.0/8 address. In this environment > > and with these restrictions, I don't understand how you can remove the > > involvement of the proxy in the process of what you call > "authentication". > > > > BTW, it's not just http--e-mail and many other applications today make > > extensive use of proxy relays as well. > > > > Josh > > Yes, and most enterprise environments include a NAT. Even homes with DSL > include NAT. A few may even use a proxy. That does not mean private > addresses of the target can not be shared at the time of > authentication. I > would have expected such an exchange. As most of these things work, such > permission is in the form of a lease. I would also expect as the map is > declared, mapping screens are established based on the permission > discovered > at the time of authentication. Before and not during use. Using a binary > address does not mean PUBLIC addresses. It may not even be IP. > It could be > SCSI address or perhaps an encoded address. You do not want SCSI to look > like an HTTP server. Especially if you wish this application to > scale, you > do not want to be doing in-band name lookup and authentication. > > Pleases, this is not a web server, it is a portal to SCSI > devices. A client > does not need to use a name to get a proxy to listen, try just > typing the IP > of a web site. The proxy will forgo the lookup. Name lookup is simply a > convenience for humans. You would not want to depend on a round-robin > selection of IPs from DNS should there be more than one such IP. > How would > you select the alternative IP, the next in the list? All these parameters > can be concisely defined in the authentication exchange. I can > not see why > someone would wish to place a name on their SCSI portal but they > could. The > only name that needs to exist is the authentication server. I would not > expect an address beyond the SCSI portal to be PUBLIC IPs. I would not > expect them to be IP. LDAP is good at doing symbolic lookup. > Let it do the > work at the time of authentication. Don't invent a SCSI browser. > > Doug > From owner-ips@ECE.cmu.edu Wed Oct 04 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22440; 3 Oct 2000 20:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id SAA05748 for ips-outgoing; Tue, 3 Oct 2000 18:45:46 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id SAA05740 for ; Tue, 3 Oct 2000 18:45:41 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id SAA11450 for ; Tue, 3 Oct 2000 18:39:32 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e93Mjda57894 for ; Tue, 3 Oct 2000 16:45:39 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: SCSI URL scheme To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 3 Oct 2000 15:42:54 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/03/2000 04:45:38 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Milan, (and others), Since at lest some of you are agreeing on part of this, perhaps we could use someone to step through their thoughts about the Proxy/Gateway/NAT with only the Storage Controller as a Target. That is, for a little while, could we talk about what it take to just get to the IP capable Storage Controller that contains the LUNs we have been authorized to use (without giving the LUNs a name at this time, just assume we will talk to the iSCSI Target Driver on the Storage Controller). If the Storage Controller is known (by some method to its DNS server) what more is needed for a host outside the Proxy/Gateway/NAT to do to get through to the Target Storage Controller? Assume NO changes to current Proxy/Gateway/NAT code. Please assume that the Host has been given a Name that is resolvable, by its DNS, so when that Initiator queries the DNS to get the IP address of the Storage Controller, it gets appropriately resolved. Now I understand that it may only bring the Login request up to the Proxy/Gateway/NAT, but based on the discussion we have been having here, it might be useful to step the process through from the Host to the target Storage Controller including what is really needed to pass through the Proxy/Gateway/NAT and arrive at the Target Storage Controller. Would someone like to try describing that? . . . John L. Hufferd "Merhar, Milan" @ece.cmu.edu on 10/03/2000 01:40:03 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] I'd like to chime in with my agreement, worded a bit differently: 1) I believe large systems (spanning the Internet) will at some point need some form of proxy/gateway/NAT because: - addresses in one realm won't be legal in another realm - the owner of a realm does not want any of its internals revealed - those internals are changeable, and the owner wants to isolate outsiders from the impact of those changes. 2) Authentication, privacy (a.k.a. encryption), etc are indeed a separate issue from end-point naming. 2a) Information obtained by observing and parsing an embedded identifier should have no impact on overall security - We aren't relying on "security by obscurity," after all. 2b) ditto, whether those identifiers are human-readable or not. 3) Let's assume that the available tools and techniques of DNS, URLs, etc are there to be leveraged, and let's see where it takes us. I'd rather spend our time delivering a robust application, rather than inventing optimized replacements for common Internet tools. - Milan Merhar -----Original Message----- From: Joshua Tseng [mailto:jtseng@NishanSystems.com] Sent: Tuesday, October 03, 2000 2:43 PM To: Douglas Otis; ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Doug, I'm not sure we understand each other anymore. I will just carefully restate my points, and leave it at that. 1) URL's (domain name & path) are needed in the iSCSI transport to support proxy services. Because of the prevalence of NAT, proxies are necessary. 2) Authentication is a separate issue and has nothing to do with identifying the final destination device/LUN/WWN of the iSCSI traffic. A separate key distribution server may improve scalability of the authentication mechanism, but this has nothing to do with addressing and routing of iSCSI traffic. 3) A LANE-type architecture for addressing and routing of iSCSI traffic is a bad idea due to scalability and management issues. The iSCSI transport must have imbedded routing information in the form of a URL, to allow proxies and destination nodes to route iSCSI traffic to its final device/LUN/WWN destination. Best regards, Josh Tseng -----Original Message----- From: Douglas Otis [mailto:dotis@sanlight.net] Sent: Tuesday, October 03, 2000 11:13 AM To: Joshua Tseng; ips@ece.cmu.edu Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] Joshua, > What you describe might be possible (although I still think it's a bad > idea) if the entire Internet, including all public and private networks, > were in a single consolidated address space. But the fact is we are > running out of address space, and there is something called NAT defined > in RFC1918. Who knows, with IPv6, this may change, or it might not. But > it is a reality today. To operate in an environment with NAT, you need > proxies. There's no way around it. A client in a public network using > registered IP address space should NEVER see a 10.0/8 address. It should > NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 > address entry in its routing table. It must first talk to a dual-homed > proxy with at least one leg using registered IP address space, in order > to communicate with a host with a 10.0/8 address. In this environment > and with these restrictions, I don't understand how you can remove the > involvement of the proxy in the process of what you call "authentication". > > BTW, it's not just http--e-mail and many other applications today make > extensive use of proxy relays as well. > > Josh Yes, and most enterprise environments include a NAT. Even homes with DSL include NAT. A few may even use a proxy. That does not mean private addresses of the target can not be shared at the time of authentication. I would have expected such an exchange. As most of these things work, such permission is in the form of a lease. I would also expect as the map is declared, mapping screens are established based on the permission discovered at the time of authentication. Before and not during use. Using a binary address does not mean PUBLIC addresses. It may not even be IP. It could be SCSI address or perhaps an encoded address. You do not want SCSI to look like an HTTP server. Especially if you wish this application to scale, you do not want to be doing in-band name lookup and authentication. Pleases, this is not a web server, it is a portal to SCSI devices. A client does not need to use a name to get a proxy to listen, try just typing the IP of a web site. The proxy will forgo the lookup. Name lookup is simply a convenience for humans. You would not want to depend on a round-robin selection of IPs from DNS should there be more than one such IP. How would you select the alternative IP, the next in the list? All these parameters can be concisely defined in the authentication exchange. I can not see why someone would wish to place a name on their SCSI portal but they could. The only name that needs to exist is the authentication server. I would not expect an address beyond the SCSI portal to be PUBLIC IPs. I would not expect them to be IP. LDAP is good at doing symbolic lookup. Let it do the work at the time of authentication. Don't invent a SCSI browser. Doug From owner-ips@ECE.cmu.edu Wed Oct 04 00:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22449; 3 Oct 2000 20:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id TAA06494 for ips-outgoing; Tue, 3 Oct 2000 19:11:45 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id TAA06479 for ; Tue, 3 Oct 2000 19:11:37 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id SAA53658 for ; Tue, 3 Oct 2000 18:58:55 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e93NBaa23426 for ; Tue, 3 Oct 2000 17:11:36 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI Naming and Discovery To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 3 Oct 2000 16:09:16 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/03/2000 05:11:36 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk David Black, Could you please expand (a lot) on the following statement (I am not sure I followed you): "[4] Discovery based on IP addresses looks like it works for boot volumes in a way that URLs don't and scales via wildcarding in a fashion superior to URLs. An underlying assumption I'm making is that storage discovery doesn't need to match the scale of DNS, and hence centralizing config info isn't hobbled by NAT issues." . . . John L. Hufferd Black_David@emc.com@ece.cmu.edu on 10/03/2000 03:03:19 PM Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: iSCSI Naming and Discovery The emails from Daniel Smith and Josh Tseng provide a good start on a naming discussion. With my co-chair hat off, let me try to add to it. The concepts surrounding NAT (Network Address Translation) figure strongly in this discussion. RFC 2663 is a good source of background for anyone who needs it. An attempt to group and summarize what I see in Daniel and Josh's emails: [1] Internet infrastructure is beyond the control of storage, including iSCSI [Daniel - a),b),f),g),h)] [2] DNS is the naming structure of the Internet, and hence hosts exporting storage services have to be nameable via DNS [Daniel c)]. [3] Network Address Translation and Network Address Port Translation exist and have to be dealt with [Daniel d), Josh (1)]. [4] Discovery at Internet scale is a hard problem. A flexible naming mechanism keeps options open. [Daniel e)]. [5] SCSI 3rd party commands need to name LUNs, including iSCSI LUNs [Daniel i)] *[6] Source, destination, and contents of any packet on the Internet are public information [Daniel j)]. *[7] Security matters for SSPs [Daniel k)]. [8] SSPs can be expected to connect a lot of clients to a single DNS address [Daniel l)]. *[9] Identification of storage and authentication are separate problems and use separate names. [Josh (2)] [10] Naming/routing information is necessary for proxies to identify which of the entities they are proxying for is involved in traffic [1] and [2] are general descriptions of the Internet. [6], [7], and [9] are (mostly) about security, and aside from noting that [6] is incorrect (all of that info can be hidden by a security gateway using IPsec tunnels and both IPsec and SSL/TLS hide payloads), security discussion might be better deferred as I understand that there will be a serious security proposal in the next version of the iSCSI draft. The biggest underlying problem seems to be how to identify an Initiator or Target - it's part of [3], [5], [8] and [10]. [4] is about Discovery, which compounds the naming issues. At a high level there are three basic ways to identify initiators and targets: - Transport address (e.g., IP address, FC port WWN). - Identification information provided as part of session establishment (e.g., username/password, certificate). - Some combination of the above two. The third alternative may be problematic if it leads to needing both the transport and identification information to determine identity. The discussions I've seen seem to be using transport as a hint that may make the identification easier to verify, which seems like a reasonable optimization to relying on the identification information alone. Moving beyond this (e.g., the *.xyz.com servers may only connect from addresses in the a.b.c/24 netblock) increases the amount of information that has to be configured (e.g., that example is better left to firewalls to enforce). NAT mechanisms contribute to the problem by producing networks in which transport addresses aren't useful for identifying anything on the other side of the NAT. Some NATs can be configured to make this problem somewhat simpler via static assignment of IP addresses in one domain to IP addresses in another. NATs are a thorny subject in IETF - while they are widely deployed, not all the important protocols work through them; IPsec AH is the most notable example, and FTP requires a kludge (er, ah, ALG). IMHO, restrictions on the use of NATs with iSCSI may be ok, provided that the consequences are clearly understood. NATs seem to be a large piece of the forcing function that is leading us away from the transport-based identification information used by other SCSI transports. There are security consequences here (e.g., cryptography for session establishment may become mandatory to implement and use), and it will likely complicate discovery. For example, I noted that booting is an issue - if iSCSI always uses URLs to name storage, the result could be a situation in which a DNS server has to be operational and reachable in order to boot. This seems wrong, and the obvious answer of using an IP address in a URL does not work through NATs, which was the original motivation for using URLs. Third-party naming is a tarpit. Putting my WG co-chair hat back on for this paragraph only, I observe that global context for third party names is an unsolved problem in T10; in general, the Initiator of a 3rd party command must use names that resolve to the desired LUNs from the 3rd party command Target's naming perspective. How to do this when an Initiator and Target don't share a naming context is unspecified :-). While it would be a plus for iSCSI to solve this one via have global names for 3rd party commands, I don't think this is a requirement (and whatever we do will have to be worked through T10, as they have the final say on name formats). WG co-chair hat now comes off ... IMHO, discovery is not getting enough attention. The proposed naming scheme complicates discovery without a compelling solution; I'm concerned that the benefits may not justify the costs. My quasi-random walk through this goes something like: - Not being able to find the boot volume because DNS is down or not responding seems to be a wrong answer. The infamous World Wide Wait is bad enough for a browser; it's unacceptable for a reboot. - Hence the boot volume has to be locatable via IP, a TCP port (which could be implicit, e.g. the default iSCSI port) and a LUN (which could be implicit, e.g., LUN 0). Unlike DNS, I could see putting this into an iSCSI HBA card BIOS. - Consistency suggests the approach of using IP addresses as the primary means of identifying possible storage locations, with the possible addition of non-default TCP port numbers). A nice consequence of this is that one can use ranges (e.g., all the storage is in netblock a.b.c/24 at the default iSCSI port, scan those 256 addresses as part of discovery on boot). The corresponding range wildcarding mechanisms for URLs will be more complex. Even if a centralized configuration repository is used (like Fibre Channel), this sort of address wildcarding still looks useful in managing the repository. The netblock example may be too coarse a wildcard. Returning to the issues at the top of this message: [3] NAT become an issue for network designers/admins. Storage becomes something else that they have to get the IP addressing correct for :-(. There are precedents for this, as the default gateway and DNS resolver are already configured via IP addresses, and if those things move to different IP addresses, stuff breaks (a browser can be very unhappy if its host thinks 0.0.0.0 is the only DNS resolver). The downside is that a centralized config repository containing IP addresses becomes a NAT issue - the ALG required to access that across a NAT is ugly enough that it may be necessary to configure the network so that this never happens (which is not the best answer, but may be workable). [4] Discovery based on IP addresses looks like it works for boot volumes in a way that URLs don't and scales via wildcarding in a fashion superior to URLs. An underlying assumption I'm making is that storage discovery doesn't need to match the scale of DNS, and hence centralizing config info isn't hobbled by NAT issues. [5] Use of IP addresses would better match the other 3rd party addressing modes, and removes a dependency of the third party Target on DNS. The problems created by NATs are similar to problems that already exist in 3rd party addressing, and hence this doesn't make things worse. [8] An IP connection is identified by 2 IP addresses and two ports, the fact that several thousand of them go through a common DNS, or even a common IP address is not a problem. With the exception of the comment on T10 and 3rd party naming, this is all IMHO. Fire away ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Oct 04 01:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22743; 3 Oct 2000 21:47 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA08455 for ips-outgoing; Tue, 3 Oct 2000 20:25:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA08450 for ; Tue, 3 Oct 2000 20:25:22 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id RAA16304; Tue, 3 Oct 2000 17:11:03 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id RAA23119; Tue, 3 Oct 2000 17:08:50 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI Naming and Discovery Date: Tue, 3 Oct 2000 17:23:27 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101052@corpmx9.isus.emc.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk David, You seem to think there is value in not including a binary address as if this will interfere with NAT. As these would be defined by the realm providing the service, there would be nothing gained in attempting to replace these addresses with names. > The emails from Daniel Smith and Josh Tseng provide > a good start on a naming discussion. With > my co-chair hat off, let me try to add to it. > > The concepts surrounding NAT (Network Address Translation) > figure strongly in this discussion. RFC 2663 is a > good source of background for anyone who needs it. This warns about not depending on IP within the protocol. This is to allow responses which do not depend on the client knowing their IP. The client would be unable to know its own IP and there lies the rub. > An attempt to group and summarize what I see in > Daniel and Josh's emails: > > [1] Internet infrastructure is beyond the control > of storage, including iSCSI [Daniel - a),b),f),g),h)] Only two points of access are required. 1) Authentication (LDAP) 2) Portal > [2] DNS is the naming structure of the Internet, and > hence hosts exporting storage services have to be > nameable via DNS [Daniel c)]. Only the Authentication Server should be required to be published on DNS. > [3] Network Address Translation and Network Address > Port Translation exist and have to be dealt with [Daniel > d), Josh (1)]. No work is required with respect to a NAT. Be sure not to make assumptions about the IP address based on a local sense. Any gateway may translate this IP:Port into a different IP:Port in both directions in a transparent fashion. This is not a problem, it is a feature that allows connections without regard to this function provided the source IP and port are not used to authenticate. > [4] Discovery at Internet scale is a hard problem. A > flexible naming mechanism keeps options open. [Daniel e)]. That discovery is solved in a very simple fashion. It is called LDAP. > [5] SCSI 3rd party commands need to name LUNs, including > iSCSI LUNs [Daniel i)] The mapping to those LUNs would be pre-defined via the LDAP server. The obvious advantage would be both security and speed. > *[6] Source, destination, and contents of any packet > on the Internet are public information [Daniel j)]. Because these packets are public, make an effort not to label the content. > *[7] Security matters for SSPs [Daniel k)]. Through the use of LDAP, which is hierarchical, both the client and the provider are allowed to control their realm via the hierarchy. > [8] SSPs can be expected to connect a lot of clients > to a single DNS address [Daniel l)]. Because DNS can not provide the control desired, specifying the use of an LDAP server in providing the required information allows the requisite controls. > *[9] Identification of storage and authentication are > separate problems and use separate names. [Josh (2)] The association of the user:target would be done via LDAP. The client could control users and the SSP could control general access to their services. > [10] Naming/routing information is necessary for proxies > to identify which of the entities they are proxying for > is involved in traffic This statement is assumes that is better to have a dynamic routing system for SCSI. It is not better nor does it improve the scalability, performance or security. At this point in time, there is no SCSI Name Server (SNS) to provide a means of making routes dynamic. As the SCSI mapping provided to the client at the time of authentication can be leased, routes could only change in a deterministic fashion, but they could change. > [1] and [2] are general descriptions of the Internet. > > [6], [7], and [9] are (mostly) about security, and aside > from noting that [6] is incorrect (all of that info can > be hidden by a security gateway using IPsec tunnels and > both IPsec and SSL/TLS hide payloads), security discussion > might be better deferred as I understand that there will > be a serious security proposal in the next version of the > iSCSI draft. The only security that should be addressed beyond obtaining access to LDAP, would be a means of authenticating the connection in an opaque manner between both the client and the server. This would be based on the secure connection made to the LDAP server and the shared secrets obtained from this database. > The biggest underlying problem seems to be how to identify > an Initiator or Target - it's part of [3], [5], [8] and > [10]. [4] is about Discovery, which compounds the naming > issues. The system would startup using DHCP and obtain information about the location of the LDAP server. From that point, the LDAP server would send the information about the SCSI services. No naming would be required but a convention of symbols for access or class objects should be devised for LDAP retrieval. > At a high level there are three basic ways to identify > initiators and targets: > - Transport address (e.g., IP address, FC port WWN). You would not use the IP address from the client. The IP of the service provider MUST be routable and no name is required. To obtain access to the providers authentication server, this machine should publish on DNS. As far as initiators, this would be identified via LDAP to ascertain user information. This is not an IP or URL matter. The transport MUST provide a point of access that is routable at the gateway. The client needs nothing more with respect to accessing the portal. Once the portal is accessed, the LUN addresses should have been assigned during the authentication. The portal will be required to verify permission based on this LUNs as determined during the authentication process. As such, there would be a notification process sending the results privately to the portal of who has been allowed in and what secret and lun is valid. > - Identification information provided as part of session Do not burden the transport layer with exchanging this information. You will be spend far too much time redefining what already exists. > establishment (e.g., username/password, certificate). Again, a function for LDAP. You should require a secure connection. > - Some combination of the above two. > The third alternative may be problematic if it leads to > needing both the transport and identification information > to determine identity. The discussions I've seen seem to > be using transport as a hint that may make the identification > easier to verify, which seems like a reasonable optimization > to relying on the identification information alone. Moving > beyond this (e.g., the *.xyz.com servers may only connect > from addresses in the a.b.c/24 netblock) increases the > amount of information that has to be configured (e.g., > that example is better left to firewalls to enforce). Again, we are re-inventing. There is not a good reason for re-inventing a SCSI name server used in conjunction with a SCSI protocol proxy. You could just as easily obtain this information from an LDAP server in either a secure or insecure fashion that provides the needed functions and flexibility. Unless your desire is to get into SCSI dynamic router business, I urge, beg, and plead, no don't even think it. > NAT mechanisms contribute to the problem by producing > networks in which transport addresses aren't useful for > identifying anything on the other side of the NAT. Some > NATs can be configured to make this problem somewhat simpler > via static assignment of IP addresses in one domain to IP > addresses in another. Yes, these are nailed down address to allow access such as PRIVATE IP:Port 80 so that you can access an internal web server as example. > NATs are a thorny subject in IETF - while they are widely > deployed, not all the important protocols work through them; > IPsec AH is the most notable example, and FTP requires a kludge > (er, ah, ALG). IMHO, restrictions on the use of NATs with iSCSI > may be ok, provided that the consequences are clearly understood. FTP must use the passive mode to get past the NAT. It should not be a problem for the SCSI transport. I only hope that the NIH does not take over to solve some perceived problem. > NATs seem to be a large piece of the forcing function that is > leading us away from the transport-based identification > information used by other SCSI transports. There are security > consequences here (e.g., cryptography for session establishment > may become mandatory to implement and use), and it will likely > complicate discovery. Not at all. You can not be sure who is using what, unless you know the user. You can not determine the user from the IP. Should you wish to boot the machine, then this drive would only want to know how DHCP identified the machine. You have plenty of flexibility within LDAP to solve all of these issues. > For example, I noted that booting is an issue - if iSCSI always > uses URLs to name storage, the result could be a situation in > which a DNS server has to be operational and reachable in order > to boot. This seems wrong, and the obvious answer of using an > IP address in a URL does not work through NATs, which > was the original motivation for using URLs. BOOTP/DHCP is handled by routers. You simply broadcast 'what am I' and then you receive you answer as well as where to go. I know what the response would be in my case. Yes, you could have secondary LDAP servers to keep things reliable. Chances are you already do. > Third-party naming is a tarpit. Putting my WG co-chair hat > back on for this paragraph only, I observe that global context > for third party names is an unsolved problem in T10; in > general, the Initiator of a 3rd party command must use > names that resolve to the desired LUNs from the 3rd party > command Target's naming perspective. How to do this when > an Initiator and Target don't share a naming context is > unspecified :-). While it would be a plus for iSCSI to solve > this one via have global names for 3rd party commands, > I don't think this is a requirement (and whatever we do > will have to be worked through T10, as they have the final > say on name formats). WG co-chair hat now comes off ... This is not a problem. Simply define the LDAP structures and your done. > IMHO, discovery is not getting enough attention. The proposed > naming scheme complicates discovery without a compelling > solution; I'm concerned that the benefits may not justify the > costs. My quasi-random walk through this goes something like: > > - Not being able to find the boot volume because DNS is down > or not responding seems to be a wrong answer. The infamous > World Wide Wait is bad enough for a browser; it's unacceptable > for a reboot. You would not depend on DNS especially a DNS outside of the facility to obtain boot information in most cases. You machine already can do a BOOTP/DHCP. From there, a TFTP may get you to the next step or you may elect to specify a scheme that extracts the remaining information from an LDAP server so that once the BIOS knows, you have something to talk about. > - Hence the boot volume has to be locatable via IP, a TCP port > (which could be implicit, e.g. the default iSCSI port) > and a LUN (which could be implicit, e.g., LUN 0). Unlike > DNS, I could see putting this into an iSCSI HBA card BIOS. You should investigate BOOTP/DHCP. > - Consistency suggests the approach of using IP addresses as > the primary means of identifying possible storage locations, > with the possible addition of non-default TCP port numbers). Big problem. You have not booted yet so you don't have any clue what IP or name should be used. You don't even know about TCP yet. > A nice consequence of this is that one can use ranges > (e.g., all the storage is in netblock a.b.c/24 at the > default iSCSI port, scan those 256 addresses as part of > discovery on boot). The corresponding range wildcarding > mechanisms for URLs will be more complex. That would not help you. You still need to find the boot drive and you have yet to. Again, because most would wish to key the drive that attaches after the initial boot drives to a user, LDAP is your solution. Make a standard structure than can be acessed by a driver that can talk SCSI transport to determine all the required settings. > Even if a centralized configuration repository is used (like Fibre > Channel), this sort of address wildcarding still looks useful in > managing the repository. The netblock example may be too coarse > a wildcard. Think of LDAP as your hierarchical repository. > Returning to the issues at the top of this message: > > [3] NAT become an issue for network designers/admins. Storage > becomes something else that they have to get the IP addressing > correct for :-(. There are precedents for this, as the > default gateway and DNS resolver are already configured via > IP addresses, and if those things move to different IP > addresses, stuff breaks (a browser can be very unhappy > if its host thinks 0.0.0.0 is the only DNS resolver). The > downside is that a centralized config repository containing > IP addresses becomes a NAT issue - the ALG required to > access that across a NAT is ugly enough that it may be > necessary to configure the network so that this never > happens (which is not the best answer, but may be workable). I fail to understand the concern. NAT is really not that difficult. > [4] Discovery based on IP addresses looks like it works for > boot volumes in a way that URLs don't and scales via wildcarding > in a fashion superior to URLs. An underlying assumption > I'm making is that storage discovery doesn't need to match > the scale of DNS, and hence centralizing config info isn't > hobbled by NAT issues. You could allow the driver an 'If all else fails, use a default IP and LUN via BOOTP to obtain a boot. I don't like it much as it is not very secure. > [5] Use of IP addresses would better match the other 3rd > party addressing modes, and removes a dependency of the > third party Target on DNS. The problems created by NATs > are similar to problems that already exist in 3rd party > addressing, and hence this doesn't make things worse. The problem that already exists is that these addresses are not IPs. Why pretend? > [8] An IP connection is identified by 2 IP addresses and > two ports, the fact that several thousand of them go through > a common DNS, or even a common IP address is not a problem. Whether the port is well-known or assigned does not matter, the problem you must still deal with is that there are 4G IP address of which to pick. Unless you expect a default name and target address will get you in. That would be very dangerous. You would not even be sure what system you would be running at that point. Please Enter Password.... Ha Ha fooled ya. > With the exception of the comment on T10 and 3rd party > naming, this is all IMHO. Fire away ... Doug > > --David > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Wed Oct 04 02:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22792; 3 Oct 2000 21:53 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id UAA09345 for ips-outgoing; Tue, 3 Oct 2000 20:56:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id UAA09336 for ; Tue, 3 Oct 2000 20:55:55 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id RAA16309; Tue, 3 Oct 2000 17:41:48 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id RAA23190; Tue, 3 Oct 2000 17:39:37 -0700 From: "Douglas Otis" To: "John Hufferd/San Jose/IBM" , Subject: RE: SCSI URL scheme Date: Tue, 3 Oct 2000 17:54:14 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk John, > Milan, (and others), > Since at lest some of you are agreeing on part of this, perhaps we could > use someone to step through their thoughts about the > Proxy/Gateway/NAT with > only the Storage Controller as a Target. That is, for a little while, > could we talk about what it take to just get to the IP capable Storage > Controller that contains the LUNs we have been authorized to use (without > giving the LUNs a name at this time, just assume we will talk to the iSCSI > Target Driver on the Storage Controller). There is absolutely no mystery. You ISP or IT manager will ensure the IP address is routable via routing tables in the routers. You nor I have control over these tables. Think of it as magic. You could even allow a DNS server to relate a name to this IP address. You may even wish to establish a well known port to this controller to provide an LDAP server as example which then assigns a port connection. Or this could be a different machine that also tells you the IP and port of another machine. > If the Storage Controller is known (by some method to its DNS server) what > more is needed for a host outside the Proxy/Gateway/NAT to do to get > through to the Target Storage Controller? Assume NO changes to current > Proxy/Gateway/NAT code. Nothing is needed! > Please assume that the Host has been given a Name that is resolvable, by > its DNS, so when that Initiator queries the DNS to get the IP address of > the Storage Controller, it gets appropriately resolved. Now I understand > that it may only bring the Login request up to the Proxy/Gateway/NAT, but > based on the discussion we have been having here, it might be useful to > step the process through from the Host to the target Storage Controller > including what is really needed to pass through the Proxy/Gateway/NAT and > arrive at the Target Storage Controller. > > Would someone like to try describing that? There will be an assignment within the gateway that translates a non-routable IP and port to an IP and port that is routable and exposed to the internet in most cases. This translation runs both directions in a completely transparent manner and only effects the source address of the client. The client should never assume knowledge of its IP address as it appears beyond the gateway. After 4 minutes of not being used, the translation is taken down. The destination IP provided gateway MUST be routable. The client and not the gateway makes the DNS lookup. Only within the SCSI machine, would there be additional addresses needed. I insist the best means of indicating these addresses is via LDAP assigned during authentication. Doug > . > . > John L. Hufferd > > > "Merhar, Milan" @ece.cmu.edu on 10/03/2000 01:40:03 PM > > Sent by: owner-ips@ece.cmu.edu > > > To: ips@ece.cmu.edu > cc: > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > I'd like to chime in with my agreement, worded a bit differently: > > 1) I believe large systems (spanning the Internet) will at some point > need some form of proxy/gateway/NAT because: > - addresses in one realm won't be legal in another realm > - the owner of a realm does not want any of its internals revealed > - those internals are changeable, and the owner wants to isolate > outsiders from the impact of those changes. > > > 2) Authentication, privacy (a.k.a. encryption), etc are indeed a separate > issue from end-point naming. > > 2a) Information obtained by observing and parsing an embedded identifier > should have no impact on overall security - We aren't relying on > "security by obscurity," after all. > > 2b) ditto, whether those identifiers are human-readable or not. > > > 3) Let's assume that the available tools and techniques of DNS, URLs, > etc are there to be leveraged, and let's see where it takes us. > I'd rather spend our time delivering a robust application, > rather than inventing optimized replacements for common > Internet tools. > > - Milan Merhar > > > > -----Original Message----- > From: Joshua Tseng [mailto:jtseng@NishanSystems.com] > Sent: Tuesday, October 03, 2000 2:43 PM > To: Douglas Otis; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Doug, > > I'm not sure we understand each other anymore. I will just carefully > restate my points, and leave it at that. > > 1) URL's (domain name & path) are needed in the iSCSI transport to > support proxy services. Because of the prevalence of NAT, proxies > are necessary. > > 2) Authentication is a separate issue and has nothing to do with > identifying the final destination device/LUN/WWN of the iSCSI traffic. > A separate key distribution server may improve scalability of the > authentication mechanism, but this has nothing to do with addressing > and routing of iSCSI traffic. > > 3) A LANE-type architecture for addressing and routing of iSCSI > traffic is a bad idea due to scalability and management issues. > The iSCSI transport must have imbedded routing information in the > form of a URL, to allow proxies and destination nodes to route > iSCSI traffic to its final device/LUN/WWN destination. > > Best regards, > Josh Tseng > > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Tuesday, October 03, 2000 11:13 AM > To: Joshua Tseng; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Joshua, > > > > What you describe might be possible (although I still think it's a bad > > idea) if the entire Internet, including all public and private networks, > > were in a single consolidated address space. But the fact is we are > > running out of address space, and there is something called NAT defined > > in RFC1918. Who knows, with IPv6, this may change, or it might > not. But > > it is a reality today. To operate in an environment with NAT, you need > > proxies. There's no way around it. A client in a public network using > > registered IP address space should NEVER see a 10.0/8 address. > It should > > NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 > > address entry in its routing table. It must first talk to a dual-homed > > proxy with at least one leg using registered IP address space, in order > > to communicate with a host with a 10.0/8 address. In this environment > > and with these restrictions, I don't understand how you can remove the > > involvement of the proxy in the process of what you call > "authentication". > > > > BTW, it's not just http--e-mail and many other applications today make > > extensive use of proxy relays as well. > > > > Josh > > Yes, and most enterprise environments include a NAT. Even homes with DSL > include NAT. A few may even use a proxy. That does not mean private > addresses of the target can not be shared at the time of > authentication. I > would have expected such an exchange. As most of these things work, such > permission is in the form of a lease. I would also expect as the map is > declared, mapping screens are established based on the permission > discovered > at the time of authentication. Before and not during use. Using a binary > address does not mean PUBLIC addresses. It may not even be IP. It could > be > SCSI address or perhaps an encoded address. You do not want SCSI to look > like an HTTP server. Especially if you wish this application to > scale, you > do not want to be doing in-band name lookup and authentication. > > Pleases, this is not a web server, it is a portal to SCSI devices. A > client > does not need to use a name to get a proxy to listen, try just typing the > IP > of a web site. The proxy will forgo the lookup. Name lookup is simply a > convenience for humans. You would not want to depend on a round-robin > selection of IPs from DNS should there be more than one such IP. > How would > you select the alternative IP, the next in the list? All these parameters > can be concisely defined in the authentication exchange. I can > not see why > someone would wish to place a name on their SCSI portal but they could. > The > only name that needs to exist is the authentication server. I would not > expect an address beyond the SCSI portal to be PUBLIC IPs. I would not > expect them to be IP. LDAP is good at doing symbolic lookup. Let it do > the > work at the time of authentication. Don't invent a SCSI browser. > > Doug > > > From owner-ips@ECE.cmu.edu Wed Oct 04 02:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22921; 3 Oct 2000 22:38 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id WAA11073 for ips-outgoing; Tue, 3 Oct 2000 22:03:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx1.quantum.com (mx1.quantum.com [204.212.103.34]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id WAA11065 for ; Tue, 3 Oct 2000 22:03:23 -0400 (EDT) Received: from milcmima.qntm.com (milcmima.qntm.com [146.174.18.61]) by mx1.quantum.com (8.9.3 (PHNE_18979)/8.9.3) with ESMTP id TAA19635; Tue, 3 Oct 2000 19:00:45 -0700 (PDT) Received: by milcmima.qntm.com with Internet Mail Service (5.5.2650.21) id <4G1JSRB5>; Tue, 3 Oct 2000 19:00:44 -0700 Message-ID: From: Jim McGrath To: "'Michael Krause'" , IPS@ece.cmu.edu Subject: RE: iSCSI: Flow Control Date: Tue, 3 Oct 2000 19:00:43 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk The mapping may be a bit trickier. If the host side manages resources (like buffers) on a per connection basis, then that should be the level. The other dimension is time. Storage traffic is bur sty, so the time span for resetting the resource levels should be within the scope of a "burst time" or "burst cycle". This is probably seconds, or perhaps minutes, but not hours. How long would we expect a session to last? (The problem with previous efforts was it was tied into device/host login, which could last for days). Jim -----Original Message----- From: Michael Krause [mailto:krause@cup.hp.com] Sent: Tuesday, October 03, 2000 9:31 AM To: IPS@ece.cmu.edu Subject: RE: iSCSI: Flow Control At 08:22 PM 10/2/00 -0700, Jim McGrath wrote: >I agree with Peter. > >We should allow immediate data, but only up to a limit previously set by the >target (so we have a chance that it is received). But the target has to be >given the ability to dynamically change (in Peter's terms, advertise) the >resources available, and be able to do it selectively (i.e. to one initiator >and not another). This inability to change the resources deployed for this >purpose over time is why current schemes are not implemented (no target >wants to get "caught" and so drastically underadvertizes the actual >resources it has available). Agree as well. The question is does one only do this at connection set-up time or does one define a iSCSI operation that performs session option negotiation at any time within a session's lifetime. Also if multiple connections are supported, is this negotiation per TCP connection? Recommend doing this per connection within a session and allowing resources per connection and resource to vary even between the same endnode pair. Mike From owner-ips@ECE.cmu.edu Wed Oct 04 02:40:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa22916; 3 Oct 2000 22:36 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id VAA10330 for ips-outgoing; Tue, 3 Oct 2000 21:33:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id VAA10325 for ; Tue, 3 Oct 2000 21:33:24 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id T9NCD0HV; Tue, 3 Oct 2000 18:31:00 -0700 From: "Y P Cheng" To: "Mark A. Carlson" , "Randall R. Stewart" Cc: "David Robinson" , Subject: RE: Status summary on multiple connections Date: Tue, 3 Oct 2000 18:30:42 -0700 Message-ID: <006e01c02da2$b5c7dca0$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk > Randall and Mark: > > Both of you forget about the case when multiple PDUs are > inflight, say N, N+1, and N+2, and one of them has CRC error, say > N+1. The receiver throws away N+1 because the bad CRC. N+2 is > received long before N+1 is retransmitted after a timeout by the > sender. Of course, a sequence number inside the PDU will ensure > sequentialality. OK, last time I didn't do too well with the above example. Let me try again. A target device receives one TCP segment within which there are three PDUs for commands N, N+1, and N+2. The target ACK'ed the TCP segment and passes it to iSCSI who parses the TCP payload. After accepting N, the target has to reject N+1 because of a queue-full condition. Right at this moment, the target finishes one previous command and now has room for one more. Should it accept N+2 or reject all subsequent commands until N+1 is received again? What if N+1 never comes again? On a network with long delay there are many commands inflight. My preference is letting the target accept the N+2 as long as the application software does not mind out-of-order execution. For folks who design tape drives, I don't recall if we ever send multiple writes without interleaving data. In fact, if we need write a large amount of data to a tape drive, instead of multiple commands, we should have one command with a very large block count. With one command at a time, there should not be any out-of-order execution problem. Modern tape drives are doing "lying writes", i.e. accepting write data immediately after the command, then report command complete before data is written to the media. By accepting a write, receiving data, and report completion sequentially, there is no need to accept more than one write command at a time. On reading from a tape, we could send multiple reads to keep the pipeline filled. Then, if we drop one read command, whoops, we are in big trouble! Because the next read command, N+2, -- if not rejected -- will get the data belonging to N+1. Therefore, the only sensible solution is that the initiator should never send more commands than the maximum queue depth allowed by the target. This is what we do in SCSI and 1394 adapters and should be done by a fibre channel and iSCSI adapter. On a network with long delay, we need a large queue to keep enough commands inflight. Am I OK this time? Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Wed Oct 04 05:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa23299; 4 Oct 2000 0:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id XAA13439 for ips-outgoing; Tue, 3 Oct 2000 23:34:28 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id XAA13433 for ; Tue, 3 Oct 2000 23:34:22 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id XAA43464 for ; Tue, 3 Oct 2000 23:21:39 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e943YKa82514 for ; Tue, 3 Oct 2000 21:34:21 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: URL scheme To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Tue, 3 Oct 2000 20:33:34 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/03/2000 09:34:20 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis, Thanks for responding. First let me say that LDAP is wonderful and useful in many ways. Now perhaps we can get that behind us. What I was hoping for was a step by step process so that everyone could conclude what was a problem and what was not. I was trying to keep things directed and pointed to the details of using what currently exists today in IP networks so that we can remove any misunderstandings. So I think much of what you stated will be useful. We have been spending a lot of time trying to name LUs and mixing it with a lot of other things. If we can orderly step through this stuff, then we can appropriately ask what is missing and what we can or should do about it. Again, thank you very much for your response. Any one else? (By the way I updated the thread Name to reflect the iSCSI part of the Workgroup.) . . . John L. Hufferd "Douglas Otis" on 10/03/2000 05:54:14 PM To: John Hufferd/San Jose/IBM@IBMUS, cc: Subject: RE: SCSI URL scheme John, > Milan, (and others), > Since at lest some of you are agreeing on part of this, perhaps we could > use someone to step through their thoughts about the > Proxy/Gateway/NAT with > only the Storage Controller as a Target. That is, for a little while, > could we talk about what it take to just get to the IP capable Storage > Controller that contains the LUNs we have been authorized to use (without > giving the LUNs a name at this time, just assume we will talk to the iSCSI > Target Driver on the Storage Controller). There is absolutely no mystery. You ISP or IT manager will ensure the IP address is routable via routing tables in the routers. You nor I have control over these tables. Think of it as magic. You could even allow a DNS server to relate a name to this IP address. You may even wish to establish a well known port to this controller to provide an LDAP server as example which then assigns a port connection. Or this could be a different machine that also tells you the IP and port of another machine. > If the Storage Controller is known (by some method to its DNS server) what > more is needed for a host outside the Proxy/Gateway/NAT to do to get > through to the Target Storage Controller? Assume NO changes to current > Proxy/Gateway/NAT code. Nothing is needed! > Please assume that the Host has been given a Name that is resolvable, by > its DNS, so when that Initiator queries the DNS to get the IP address of > the Storage Controller, it gets appropriately resolved. Now I understand > that it may only bring the Login request up to the Proxy/Gateway/NAT, but > based on the discussion we have been having here, it might be useful to > step the process through from the Host to the target Storage Controller > including what is really needed to pass through the Proxy/Gateway/NAT and > arrive at the Target Storage Controller. > > Would someone like to try describing that? There will be an assignment within the gateway that translates a non-routable IP and port to an IP and port that is routable and exposed to the internet in most cases. This translation runs both directions in a completely transparent manner and only effects the source address of the client. The client should never assume knowledge of its IP address as it appears beyond the gateway. After 4 minutes of not being used, the translation is taken down. The destination IP provided gateway MUST be routable. The client and not the gateway makes the DNS lookup. Only within the SCSI machine, would there be additional addresses needed. I insist the best means of indicating these addresses is via LDAP assigned during authentication. Doug > . > . > John L. Hufferd > > > "Merhar, Milan" @ece.cmu.edu on 10/03/2000 01:40:03 PM > > Sent by: owner-ips@ece.cmu.edu > > > To: ips@ece.cmu.edu > cc: > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > > I'd like to chime in with my agreement, worded a bit differently: > > 1) I believe large systems (spanning the Internet) will at some point > need some form of proxy/gateway/NAT because: > - addresses in one realm won't be legal in another realm > - the owner of a realm does not want any of its internals revealed > - those internals are changeable, and the owner wants to isolate > outsiders from the impact of those changes. > > > 2) Authentication, privacy (a.k.a. encryption), etc are indeed a separate > issue from end-point naming. > > 2a) Information obtained by observing and parsing an embedded identifier > should have no impact on overall security - We aren't relying on > "security by obscurity," after all. > > 2b) ditto, whether those identifiers are human-readable or not. > > > 3) Let's assume that the available tools and techniques of DNS, URLs, > etc are there to be leveraged, and let's see where it takes us. > I'd rather spend our time delivering a robust application, > rather than inventing optimized replacements for common > Internet tools. > > - Milan Merhar > > > > -----Original Message----- > From: Joshua Tseng [mailto:jtseng@NishanSystems.com] > Sent: Tuesday, October 03, 2000 2:43 PM > To: Douglas Otis; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Doug, > > I'm not sure we understand each other anymore. I will just carefully > restate my points, and leave it at that. > > 1) URL's (domain name & path) are needed in the iSCSI transport to > support proxy services. Because of the prevalence of NAT, proxies > are necessary. > > 2) Authentication is a separate issue and has nothing to do with > identifying the final destination device/LUN/WWN of the iSCSI traffic. > A separate key distribution server may improve scalability of the > authentication mechanism, but this has nothing to do with addressing > and routing of iSCSI traffic. > > 3) A LANE-type architecture for addressing and routing of iSCSI > traffic is a bad idea due to scalability and management issues. > The iSCSI transport must have imbedded routing information in the > form of a URL, to allow proxies and destination nodes to route > iSCSI traffic to its final device/LUN/WWN destination. > > Best regards, > Josh Tseng > > -----Original Message----- > From: Douglas Otis [mailto:dotis@sanlight.net] > Sent: Tuesday, October 03, 2000 11:13 AM > To: Joshua Tseng; ips@ece.cmu.edu > Subject: RE: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] > > > Joshua, > > > > What you describe might be possible (although I still think it's a bad > > idea) if the entire Internet, including all public and private networks, > > were in a single consolidated address space. But the fact is we are > > running out of address space, and there is something called NAT defined > > in RFC1918. Who knows, with IPv6, this may change, or it might > not. But > > it is a reality today. To operate in an environment with NAT, you need > > proxies. There's no way around it. A client in a public network using > > registered IP address space should NEVER see a 10.0/8 address. > It should > > NEVER talk to a 10.0/8 address, and it shouldn't even have a 10.0/8 > > address entry in its routing table. It must first talk to a dual-homed > > proxy with at least one leg using registered IP address space, in order > > to communicate with a host with a 10.0/8 address. In this environment > > and with these restrictions, I don't understand how you can remove the > > involvement of the proxy in the process of what you call > "authentication". > > > > BTW, it's not just http--e-mail and many other applications today make > > extensive use of proxy relays as well. > > > > Josh > > Yes, and most enterprise environments include a NAT. Even homes with DSL > include NAT. A few may even use a proxy. That does not mean private > addresses of the target can not be shared at the time of > authentication. I > would have expected such an exchange. As most of these things work, such > permission is in the form of a lease. I would also expect as the map is > declared, mapping screens are established based on the permission > discovered > at the time of authentication. Before and not during use. Using a binary > address does not mean PUBLIC addresses. It may not even be IP. It could > be > SCSI address or perhaps an encoded address. You do not want SCSI to look > like an HTTP server. Especially if you wish this application to > scale, you > do not want to be doing in-band name lookup and authentication. > > Pleases, this is not a web server, it is a portal to SCSI devices. A > client > does not need to use a name to get a proxy to listen, try just typing the > IP > of a web site. The proxy will forgo the lookup. Name lookup is simply a > convenience for humans. You would not want to depend on a round-robin > selection of IPs from DNS should there be more than one such IP. > How would > you select the alternative IP, the next in the list? All these parameters > can be concisely defined in the authentication exchange. I can > not see why > someone would wish to place a name on their SCSI portal but they could. > The > only name that needs to exist is the authentication server. I would not > expect an address beyond the SCSI portal to be PUBLIC IPs. I would not > expect them to be IP. LDAP is good at doing symbolic lookup. Let it do > the > work at the time of authentication. Don't invent a SCSI browser. > > Doug > > > From owner-ips@ECE.cmu.edu Wed Oct 04 08:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24019; 4 Oct 2000 4:15 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA18382 for ips-outgoing; Wed, 4 Oct 2000 03:10:13 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (play-doh.Stanford.EDU [128.12.131.42]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA18376 for ; Wed, 4 Oct 2000 03:10:09 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 427238E3C; Wed, 4 Oct 2000 00:05:01 -0700 (PDT) To: Subject: Re: iSCSI: URL scheme References: Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 04 Oct 2000 00:05:01 -0700 In-Reply-To: "John Hufferd/San Jose/IBM"'s message of "Tue, 3 Oct 2000 20:33:34 -0700" Message-ID: Lines: 163 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk I would like to propose the following requirement for any naming scheme. * Any naming scheme proposed MUST support multiple targets behind a single IP address * Any naming scheme proposed SHOULD support multiple targets behind a single port. This helps make traffic analysis easier. Consider the topology pictured below. An initiator is connected to a private network that has a gateway to the Internet. Similarly, the target is connected to a private network which is connected by another gateway to the Internet. Initiator - Private Network - Gateway 1 \ \ Internet / / Target - Private Network - Gateway 2 (bob) | Target (fred) How does the Initiator get to talk to the Target bob? ------------- Current model ------------- SCSI device identifier is a string in two parts: a hostname and a host-specific name (HSN). The hostname is either a domain name or an IPv4 or IPv6 address. Steps: 0) Configuration mechanism tells initiator to talk to SCSI device identifier "bob" 1) Do a DNS (or other name service) lookup on bob. Reply arrives with IP address of Gateway #2 2) TCP connect to gateway #2 on well known iSCSI port 3) As part of first login packet, send "Target: bob" 4) Gateway #2 resolves the name "bob" on the internal network and gets the private IP address of the Target Bob 5) Gateway #2 opens a TCP connection to Target Bob and passes all traffic between connections Third party command from bob to fred 0) Configuration tells initiator that bob should talk to SCSI device identifier "fred" 1) Initiator sends command to bob which has SCSI device identifier "fred" 2) Bob resolves fred's name and gets Fred's private Ip address 3) Bob connects to Fred and they do their thing Refinements ----------- Configuration mechanism (e.g. LDAP server) in step 0, along with returning the SCSI device identifier "bob", also returns the IP address of gateway #2. The DNS lookup in #1 is avoided. DNS lookup of bob actually returns gateway #1 (would work for iSCSI and HTTP/1.1 but unlikely to work for other protocols). Gateway #1 then looks up the name on the Internet and gets gateway #2... and so on... ---------------- IP address model ---------------- A target name is made of two parts: an IP address and a target ID. The target ID can be either a string or a fixed-length binary quantity. The target ID is not necessarily globally unique; it can be IP-address specific. Step 0) Initiator queries the configuration mechanisms and gets the IP address of gateway #2 and the target ID for bob. Step 1) Open TCP connection to IP address of gateway #2 on well-known iSCSI port Step 2) As part of first login packet, initiator send the target ID for bob Step 3) Gateway #2 opens connection to bob based on the target ID Step 4) Gateway #2 passes all info between connections Third party commands - passable way -------------------- Step 0) Configuartion mechanism tells initiator that bob should talk to fred. The configuration mechanism tells the initiator about the private IP address/target ID of Fred as seen by Bob (how does it find this out??) Step 1) Initiator sends third party command with Fred's IP address and target ID Step 2) Bob opens connection to Fred's IP address+port Third party commands - icky way -------------------- Step 0) Configuartion mechanism tells initiator that bob should talk to fred. The configuration mechanism tells the initiator about the IP address of gateway #2 and target ID of fred Step 1) Initiator sends third party command with IP address of gateway#2 and target ID of fred Step 2) Gateway #2 intercepts the third party command and rewrites it to have Fred's private IP address (based on the target ID) Step 3) Bob opens connection to Fred's private IP address on well known iSCSI port, sends target ID in the first iSCSI PDU, etc. or Step 2) Bob opens a connection to gateway #2's IP address and passes target ID Step 3) Bob gets redirected to Fred (or... gateway opens connection to Fred and passes all commands between Fred & Bob) In this approach, the gateway must be able to map a target ID to an IP address and a (possibly new) target ID. Note on using port numbers instead of target IDs ------------------------------------------------ The only thing that changes in the analysis above is that the target ID need not be sent on connection open, since the destination port number on the TCP connection identifies the target. In this specific scenario, it seems to me that using strings entirely is cleaner than passing IP addresses which are potentially non-sensical. -Costa From owner-ips@ECE.cmu.edu Wed Oct 04 08:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24032; 4 Oct 2000 4:29 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id DAA19234 for ips-outgoing; Wed, 4 Oct 2000 03:55:14 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-3.de.ibm.com (d12lmsgate-3.de.ibm.com [195.212.91.201]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id DAA19230 for ; Wed, 4 Oct 2000 03:55:09 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-3.de.ibm.com (1.0.0) with ESMTP id JAA47776 for ; Wed, 4 Oct 2000 09:54:37 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id JAA41820 for ; Wed, 4 Oct 2000 09:54:37 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696E.002B7145 ; Wed, 4 Oct 2000 09:54:30 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 4 Oct 2000 10:52:11 +0300 Subject: Re: iSCSI Naming and Discovery Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk I though we went through this several times. We went after the a simple scheme that doesn't force us to choose between IPV4 and 6 and will be widely supported by infrastructure. We also took care of not imposing DNS resolution on everybody - there is a specified string form that represent a LITERAL address (IP4 or 6). This shood take care of simple devices and boot. The only think we where reluctant to do is go for a full-fledged LU WWN naming. Julo Black_David@emc.com on 04/10/2000 01:03:19 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: iSCSI Naming and Discovery The emails from Daniel Smith and Josh Tseng provide a good start on a naming discussion. With my co-chair hat off, let me try to add to it. The concepts surrounding NAT (Network Address Translation) figure strongly in this discussion. RFC 2663 is a good source of background for anyone who needs it. An attempt to group and summarize what I see in Daniel and Josh's emails: [1] Internet infrastructure is beyond the control of storage, including iSCSI [Daniel - a),b),f),g),h)] [2] DNS is the naming structure of the Internet, and hence hosts exporting storage services have to be nameable via DNS [Daniel c)]. [3] Network Address Translation and Network Address Port Translation exist and have to be dealt with [Daniel d), Josh (1)]. [4] Discovery at Internet scale is a hard problem. A flexible naming mechanism keeps options open. [Daniel e)]. [5] SCSI 3rd party commands need to name LUNs, including iSCSI LUNs [Daniel i)] *[6] Source, destination, and contents of any packet on the Internet are public information [Daniel j)]. *[7] Security matters for SSPs [Daniel k)]. [8] SSPs can be expected to connect a lot of clients to a single DNS address [Daniel l)]. *[9] Identification of storage and authentication are separate problems and use separate names. [Josh (2)] [10] Naming/routing information is necessary for proxies to identify which of the entities they are proxying for is involved in traffic [1] and [2] are general descriptions of the Internet. [6], [7], and [9] are (mostly) about security, and aside from noting that [6] is incorrect (all of that info can be hidden by a security gateway using IPsec tunnels and both IPsec and SSL/TLS hide payloads), security discussion might be better deferred as I understand that there will be a serious security proposal in the next version of the iSCSI draft. The biggest underlying problem seems to be how to identify an Initiator or Target - it's part of [3], [5], [8] and [10]. [4] is about Discovery, which compounds the naming issues. At a high level there are three basic ways to identify initiators and targets: - Transport address (e.g., IP address, FC port WWN). - Identification information provided as part of session establishment (e.g., username/password, certificate). - Some combination of the above two. The third alternative may be problematic if it leads to needing both the transport and identification information to determine identity. The discussions I've seen seem to be using transport as a hint that may make the identification easier to verify, which seems like a reasonable optimization to relying on the identification information alone. Moving beyond this (e.g., the *.xyz.com servers may only connect from addresses in the a.b.c/24 netblock) increases the amount of information that has to be configured (e.g., that example is better left to firewalls to enforce). NAT mechanisms contribute to the problem by producing networks in which transport addresses aren't useful for identifying anything on the other side of the NAT. Some NATs can be configured to make this problem somewhat simpler via static assignment of IP addresses in one domain to IP addresses in another. NATs are a thorny subject in IETF - while they are widely deployed, not all the important protocols work through them; IPsec AH is the most notable example, and FTP requires a kludge (er, ah, ALG). IMHO, restrictions on the use of NATs with iSCSI may be ok, provided that the consequences are clearly understood. NATs seem to be a large piece of the forcing function that is leading us away from the transport-based identification information used by other SCSI transports. There are security consequences here (e.g., cryptography for session establishment may become mandatory to implement and use), and it will likely complicate discovery. For example, I noted that booting is an issue - if iSCSI always uses URLs to name storage, the result could be a situation in which a DNS server has to be operational and reachable in order to boot. This seems wrong, and the obvious answer of using an IP address in a URL does not work through NATs, which was the original motivation for using URLs. Third-party naming is a tarpit. Putting my WG co-chair hat back on for this paragraph only, I observe that global context for third party names is an unsolved problem in T10; in general, the Initiator of a 3rd party command must use names that resolve to the desired LUNs from the 3rd party command Target's naming perspective. How to do this when an Initiator and Target don't share a naming context is unspecified :-). While it would be a plus for iSCSI to solve this one via have global names for 3rd party commands, I don't think this is a requirement (and whatever we do will have to be worked through T10, as they have the final say on name formats). WG co-chair hat now comes off ... IMHO, discovery is not getting enough attention. The proposed naming scheme complicates discovery without a compelling solution; I'm concerned that the benefits may not justify the costs. My quasi-random walk through this goes something like: - Not being able to find the boot volume because DNS is down or not responding seems to be a wrong answer. The infamous World Wide Wait is bad enough for a browser; it's unacceptable for a reboot. - Hence the boot volume has to be locatable via IP, a TCP port (which could be implicit, e.g. the default iSCSI port) and a LUN (which could be implicit, e.g., LUN 0). Unlike DNS, I could see putting this into an iSCSI HBA card BIOS. - Consistency suggests the approach of using IP addresses as the primary means of identifying possible storage locations, with the possible addition of non-default TCP port numbers). A nice consequence of this is that one can use ranges (e.g., all the storage is in netblock a.b.c/24 at the default iSCSI port, scan those 256 addresses as part of discovery on boot). The corresponding range wildcarding mechanisms for URLs will be more complex. Even if a centralized configuration repository is used (like Fibre Channel), this sort of address wildcarding still looks useful in managing the repository. The netblock example may be too coarse a wildcard. Returning to the issues at the top of this message: [3] NAT become an issue for network designers/admins. Storage becomes something else that they have to get the IP addressing correct for :-(. There are precedents for this, as the default gateway and DNS resolver are already configured via IP addresses, and if those things move to different IP addresses, stuff breaks (a browser can be very unhappy if its host thinks 0.0.0.0 is the only DNS resolver). The downside is that a centralized config repository containing IP addresses becomes a NAT issue - the ALG required to access that across a NAT is ugly enough that it may be necessary to configure the network so that this never happens (which is not the best answer, but may be workable). [4] Discovery based on IP addresses looks like it works for boot volumes in a way that URLs don't and scales via wildcarding in a fashion superior to URLs. An underlying assumption I'm making is that storage discovery doesn't need to match the scale of DNS, and hence centralizing config info isn't hobbled by NAT issues. [5] Use of IP addresses would better match the other 3rd party addressing modes, and removes a dependency of the third party Target on DNS. The problems created by NATs are similar to problems that already exist in 3rd party addressing, and hence this doesn't make things worse. [8] An IP connection is identified by 2 IP addresses and two ports, the fact that several thousand of them go through a common DNS, or even a common IP address is not a problem. With the exception of the comment on T10 and 3rd party naming, this is all IMHO. Fire away ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Oct 04 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24888; 4 Oct 2000 9:43 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA25814 for ips-outgoing; Wed, 4 Oct 2000 09:07:42 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id CAA17240 for ; Wed, 4 Oct 2000 02:12:33 -0400 (EDT) Received: from divyaroot.India.Sun.COM ([129.158.226.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id XAA25053 for ; Tue, 3 Oct 2000 23:12:30 -0700 (PDT) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with SMTP id LAA25777 for ; Wed, 4 Oct 2000 11:42:29 +0530 (IST) Message-Id: <200010040612.LAA25777@divyaroot.India.Sun.COM> Date: Wed, 4 Oct 2000 11:45:27 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: Re: SCSI URL scheme To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: MnqwMy0v5gjN+QK0W69f4A== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4 SunOS 5.8 sun4u sparc Sender: owner-ips@ece.cmu.edu Precedence: bulk > > First off, this is suppose to be done (normally) once, but perhaps at > various (logical) Plug & Play points. If we are talking about the Plug & > Play points in time, then there could be some additional conversation. But > independent of that, which so far I think we have been, the Host determines > ALL his addressable LUs when he comes up, and today that means "REPORT > LUNS", followed by approprate INQUIRYs. This is not a big deal at bring > up. > It depends on the storage controller and the configuration. If iSCSI indeed becomes very popular, it is not hard to visualize a lot of compute nodes (of the order of 100s just to start with) accessing/sharing a big storage device that could have a lot of LUs (of the order of thousands at the least). It is possible to configure the storage device to selectively report LUs in the REPORT LUN response so as to limit the number of INQUIRYs fired from the initiators - But I see this only as a work around - not a solution to the problem. I think a better way to translate a LU-WWN to LUN value is required to avoid overwhelming number of INQUIRYs fired from all the compute nodes trying to access the storage device; This is not only wasting the network bandwidth, but is also slowing down the host's reconfiguration/booting. -JP From owner-ips@ECE.cmu.edu Wed Oct 04 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24879; 4 Oct 2000 9:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id IAA24563 for ips-outgoing; Wed, 4 Oct 2000 08:34:43 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraab.compuserve.com (ds-img-rel-2.compuserve.com [149.174.206.155]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id IAA24554 for ; Wed, 4 Oct 2000 08:34:37 -0400 (EDT) Received: (from mailgate@localhost) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id IAA27425 for ips@ece.cmu.edu; Wed, 4 Oct 2000 08:34:07 -0400 (EDT) Received: from compuserve.com (mid-tgn-ngu-vty173.as.wcom.net [216.192.87.173]) by spdmraab.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id IAA27349 for ; Wed, 4 Oct 2000 08:33:56 -0400 (EDT) Message-ID: <39DB30E7.A496700B@compuserve.com> Date: Wed, 04 Oct 2000 08:30:16 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & mapping] References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Please use t10@t10.org to propose changes to SCSI. Thanks. Ralph Weber Howard Green wrote: > I know that the following is kind of a radical question > to ask, but has the time arrived to toss out LUNs as a > means of identifying logical units, in favor of simply > using the logical units' WWNs directly? > From owner-ips@ECE.cmu.edu Wed Oct 04 13:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa24881; 4 Oct 2000 9:41 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA25843 for ips-outgoing; Wed, 4 Oct 2000 09:08:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id GAA21589 for ; Wed, 4 Oct 2000 06:04:19 -0400 (EDT) Received: from divyaroot.India.Sun.COM ([129.158.226.35]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id DAA07175 for ; Wed, 4 Oct 2000 03:04:17 -0700 (PDT) Received: from helix (helix [129.158.226.51]) by divyaroot.India.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with SMTP id PAA10191 for ; Wed, 4 Oct 2000 15:34:15 +0530 (IST) Message-Id: <200010041004.PAA10191@divyaroot.India.Sun.COM> Date: Wed, 4 Oct 2000 15:37:13 -0500 (GMT) From: Raghavendra Rao Reply-To: Raghavendra Rao Subject: RE: Status summary on multiple connections To: ips@ece.cmu.edu MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: Wx3QY1lV78vs4rCBGTbWmw== Sender: owner-ips@ece.cmu.edu Precedence: bulk > From: "Y P Cheng" > To: "Mark A. Carlson" , "Randall R. Stewart" > Cc: "David Robinson" , ------- > commands inflight. My preference is letting the target accept the N+2 as > long as the application software does not mind out-of-order execution. For > folks who design tape drives, I don't recall if we ever send multiple writes > without interleaving data. In fact, if we need write a large amount of data > to a tape drive, instead of multiple commands, we should have one command > with a very large block count. With one command at a time, there should not > be any out-of-order execution problem. Modern tape drives are doing "lying > writes", i.e. accepting write data immediately after the command, then > report command complete before data is written to the media. By accepting a > write, receiving data, and report completion sequentially, there is no need > to accept more than one write command at a time. On reading from a tape, we > could send multiple reads to keep the pipeline filled. > Well, on reads, a similar problem could occur, and for both read and write it is just wrong to execute anything out-of-order for a sequential access device, which otherwise will cause reading/writing from wrong/un-intended location; This is simply because you aren't writing/reading from an absolute location rather from the current location (the position of head) - The read or write CDB for a sequential access device doesn't contain the block or file position to write/read. -JP From owner-ips@ECE.cmu.edu Wed Oct 04 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25078; 4 Oct 2000 10:54 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id KAA28216 for ips-outgoing; Wed, 4 Oct 2000 10:16:08 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from palrel3.hp.com (palrel3.hp.com [156.153.255.226]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id KAA28211 for ; Wed, 4 Oct 2000 10:16:04 -0400 (EDT) Received: from hpindlm.cup.hp.com (hpindlm.cup.hp.com [15.13.95.89]) by palrel3.hp.com (Postfix) with ESMTP id 79FD6303 for ; Wed, 4 Oct 2000 07:16:03 -0700 (PDT) Received: from mk731912 (h0100cmk.atl.hp.com [15.113.172.114]) by hpindlm.cup.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.02) with ESMTP id HAA08745 for ; Wed, 4 Oct 2000 07:16:16 -0700 (PDT) Message-Id: <4.2.2.20001004064739.00ba2e40@hpindlm.cup.hp.com> X-Sender: krause@hpindlm.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 04 Oct 2000 06:53:52 -0700 To: IPS@ece.cmu.edu From: Michael Krause Subject: RE: iSCSI: Flow Control In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ips@ece.cmu.edu Precedence: bulk At 07:00 PM 10/3/00 -0700, Jim McGrath wrote: >The mapping may be a bit trickier. If the host side manages resources (like >buffers) on a per connection basis, then that should be the level. The >other dimension is time. Storage traffic is bur sty, so the time span for >resetting the resource levels should be within the scope of a "burst time" >or "burst cycle". This is probably seconds, or perhaps minutes, but not >hours. How long would we expect a session to last? (The problem with >previous efforts was it was tied into device/host login, which could last >for days). A session's duration is a function of the operation being performed so I don't think there is any way to quantify to a "best" value either for a minimum or a maximum (don't really want to track this within an implementation other than what is the current resource). Burst times are also a function of the type of operation being supported - some data base queries or large data set (e.g. technical applications) processing can be very storage intensive for many hours at a time. As such, the login mechanism is a nice initialization point but the dynamic mechanism needs to be present to deal with "real-life" operations which are not steady state activities. I don't see people objecting to this requirement so is this something that can be viewed as consensus? Mike From owner-ips@ECE.cmu.edu Wed Oct 04 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25068; 4 Oct 2000 10:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA27489 for ips-outgoing; Wed, 4 Oct 2000 09:57:01 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA27480 for ; Wed, 4 Oct 2000 09:56:51 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id GAA16449; Wed, 4 Oct 2000 06:42:37 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id GAA25072; Wed, 4 Oct 2000 06:40:25 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI Naming and Discovery Date: Wed, 4 Oct 2000 06:55:10 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, You anticipate building equipment that parses a flavor of SCSI packets to route within a mixed SCSI and IP private address space? I was expecting that once you found the SCSI portal, you were in a SCSI address space. This SCSI/IP router is not going to be easy to build with all the parsing and name lookups and authentication you are assigning to it. Even CISCO manages to separate the task of name lookup, routing and authentication. The client does the name lookup, the router deals with IPs exclusively and if you want inside a private address space, you set up a tunnel. Now, you have a box that must maintain TCP connections, authenticate, parse, do name lookups, route internally over either SCSI or IP, and then of course remember the path back to the original query. Now that is impressive. There are tunneling protocols that handle this in a generic fashion that frees you of these details unless you really want to develop this equipment. Should this be done at the high end, your equipment will be handling 10G-bit connections. I doubt you have anything that can do what you suggest at 100M-bit. I know you do not want to hear this, but I understand that SCTP has a generic solution for this problem. Doug > I though we went through this several times. > > We went after the a simple scheme that doesn't force us to choose > between IPV4 and 6 and will be widely supported by infrastructure. > > We also took care of not imposing DNS resolution on everybody - there > is a specified string form that represent a LITERAL address (IP4 or 6). > This shood take care of simple devices and boot. > > The only think we where reluctant to do is go for a full-fledged LU WWN > naming. > > Julo From owner-ips@ECE.cmu.edu Wed Oct 04 15:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25073; 4 Oct 2000 10:52 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id JAA27470 for ips-outgoing; Wed, 4 Oct 2000 09:56:31 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id JAA27466 for ; Wed, 4 Oct 2000 09:56:26 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 4 Oct 2000 06:58:33 -0700 Message-ID: From: Charles Monia To: "'ENDL_TX@computer.org'" , ips@ece.cmu.edu Subject: T10 proposals [was SCSI URL scheme....] Date: Wed, 4 Oct 2000 06:58:29 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi: To add to Ralph's note, it's my opinion that the interest of the wg would be best served if any proposals to be considered by T10 would be the result of an ips consensus. Individual companies may, of course, submit such proposals, but I suspect that one based on a consensus would be more likely to prevail. Charles > -----Original Message----- > From: Ralph Weber [SMTP:ralphoweber@compuserve.com] > Sent: Wednesday, October 04, 2000 6:30 AM > To: ips@ece.cmu.edu > Subject: Re: SCSI URL scheme [WAS: Re: iSCSI: 2.2.6. Naming & > mapping] > > Please use t10@t10.org to propose changes to SCSI. > > Thanks. > > Ralph Weber > > Howard Green wrote: > > > I know that the following is kind of a radical question > > to ask, but has the time arrived to toss out LUNs as a > > means of identifying logical units, in favor of simply > > using the logical units' WWNs directly? > > > From owner-ips@ECE.cmu.edu Wed Oct 04 16:09:59 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25253; 4 Oct 2000 12:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA00791 for ips-outgoing; Wed, 4 Oct 2000 11:17:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA00666 for ; Wed, 4 Oct 2000 11:14:57 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 4 Oct 2000 08:17:05 -0700 Message-ID: From: Joshua Tseng/Nishan Systems To: julian_satran@il.ibm.com, ips@ece.cmu.edu Subject: RE: iSCSI Naming and Discovery Date: Wed, 4 Oct 2000 08:17:00 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="windows-1252" Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, My opinion is that full-fledged LU and WWN need to be supported at least in order to represent existing Fibre Channel devices which use these naming constructs. Without these, how will you use iSCSI to address a LU on a "legacy" device? That is of course assuming you want iSCSI to be able to interoperate with existing devices. Without LU and WWN, a iSCSI-FC gateway supporting Fibre Channel devices will need another layer of naming abstraction to represent a LU on the legacy side. Josh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, October 04, 2000 12:52 AM To: ips@ece.cmu.edu Subject: Re: iSCSI Naming and Discovery I though we went through this several times. We went after the a simple scheme that doesn't force us to choose between IPV4 and 6 and will be widely supported by infrastructure. We also took care of not imposing DNS resolution on everybody - there is a specified string form that represent a LITERAL address (IP4 or 6). This shood take care of simple devices and boot. The only think we where reluctant to do is go for a full-fledged LU WWN naming. Julo Black_David@emc.com on 04/10/2000 01:03:19 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: iSCSI Naming and Discovery The emails from Daniel Smith and Josh Tseng provide a good start on a naming discussion. With my co-chair hat off, let me try to add to it. The concepts surrounding NAT (Network Address Translation) figure strongly in this discussion. RFC 2663 is a good source of background for anyone who needs it. An attempt to group and summarize what I see in Daniel and Josh's emails: [1] Internet infrastructure is beyond the control of storage, including iSCSI [Daniel - a),b),f),g),h)] [2] DNS is the naming structure of the Internet, and hence hosts exporting storage services have to be nameable via DNS [Daniel c)]. [3] Network Address Translation and Network Address Port Translation exist and have to be dealt with [Daniel d), Josh (1)]. [4] Discovery at Internet scale is a hard problem. A flexible naming mechanism keeps options open. [Daniel e)]. [5] SCSI 3rd party commands need to name LUNs, including iSCSI LUNs [Daniel i)] *[6] Source, destination, and contents of any packet on the Internet are public information [Daniel j)]. *[7] Security matters for SSPs [Daniel k)]. [8] SSPs can be expected to connect a lot of clients to a single DNS address [Daniel l)]. *[9] Identification of storage and authentication are separate problems and use separate names. [Josh (2)] [10] Naming/routing information is necessary for proxies to identify which of the entities they are proxying for is involved in traffic [1] and [2] are general descriptions of the Internet. [6], [7], and [9] are (mostly) about security, and aside from noting that [6] is incorrect (all of that info can be hidden by a security gateway using IPsec tunnels and both IPsec and SSL/TLS hide payloads), security discussion might be better deferred as I understand that there will be a serious security proposal in the next version of the iSCSI draft. The biggest underlying problem seems to be how to identify an Initiator or Target - it's part of [3], [5], [8] and [10]. [4] is about Discovery, which compounds the naming issues. At a high level there are three basic ways to identify initiators and targets: - Transport address (e.g., IP address, FC port WWN). - Identification information provided as part of session establishment (e.g., username/password, certificate). - Some combination of the above two. The third alternative may be problematic if it leads to needing both the transport and identification information to determine identity. The discussions I've seen seem to be using transport as a hint that may make the identification easier to verify, which seems like a reasonable optimization to relying on the identification information alone. Moving beyond this (e.g., the *.xyz.com servers may only connect from addresses in the a.b.c/24 netblock) increases the amount of information that has to be configured (e.g., that example is better left to firewalls to enforce). NAT mechanisms contribute to the problem by producing networks in which transport addresses aren't useful for identifying anything on the other side of the NAT. Some NATs can be configured to make this problem somewhat simpler via static assignment of IP addresses in one domain to IP addresses in another. NATs are a thorny subject in IETF - while they are widely deployed, not all the important protocols work through them; IPsec AH is the most notable example, and FTP requires a kludge (er, ah, ALG). IMHO, restrictions on the use of NATs with iSCSI may be ok, provided that the consequences are clearly understood. NATs seem to be a large piece of the forcing function that is leading us away from the transport-based identification information used by other SCSI transports. There are security consequences here (e.g., cryptography for session establishment may become mandatory to implement and use), and it will likely complicate discovery. For example, I noted that booting is an issue - if iSCSI always uses URLs to name storage, the result could be a situation in which a DNS server has to be operational and reachable in order to boot. This seems wrong, and the obvious answer of using an IP address in a URL does not work through NATs, which was the original motivation for using URLs. Third-party naming is a tarpit. Putting my WG co-chair hat back on for this paragraph only, I observe that global context for third party names is an unsolved problem in T10; in general, the Initiator of a 3rd party command must use names that resolve to the desired LUNs from the 3rd party command Target's naming perspective. How to do this when an Initiator and Target don't share a naming context is unspecified :-). While it would be a plus for iSCSI to solve this one via have global names for 3rd party commands, I don't think this is a requirement (and whatever we do will have to be worked through T10, as they have the final say on name formats). WG co-chair hat now comes off ... IMHO, discovery is not getting enough attention. The proposed naming scheme complicates discovery without a compelling solution; I'm concerned that the benefits may not justify the costs. My quasi-random walk through this goes something like: - Not being able to find the boot volume because DNS is down or not responding seems to be a wrong answer. The infamous World Wide Wait is bad enough for a browser; it's unacceptable for a reboot. - Hence the boot volume has to be locatable via IP, a TCP port (which could be implicit, e.g. the default iSCSI port) and a LUN (which could be implicit, e.g., LUN 0). Unlike DNS, I could see putting this into an iSCSI HBA card BIOS. - Consistency suggests the approach of using IP addresses as the primary means of identifying possible storage locations, with the possible addition of non-default TCP port numbers). A nice consequence of this is that one can use ranges (e.g., all the storage is in netblock a.b.c/24 at the default iSCSI port, scan those 256 addresses as part of discovery on boot). The corresponding range wildcarding mechanisms for URLs will be more complex. Even if a centralized configuration repository is used (like Fibre Channel), this sort of address wildcarding still looks useful in managing the repository. The netblock example may be too coarse a wildcard. Returning to the issues at the top of this message: [3] NAT become an issue for network designers/admins. Storage becomes something else that they have to get the IP addressing correct for :-(. There are precedents for this, as the default gateway and DNS resolver are already configured via IP addresses, and if those things move to different IP addresses, stuff breaks (a browser can be very unhappy if its host thinks 0.0.0.0 is the only DNS resolver). The downside is that a centralized config repository containing IP addresses becomes a NAT issue - the ALG required to access that across a NAT is ugly enough that it may be necessary to configure the network so that this never happens (which is not the best answer, but may be workable). [4] Discovery based on IP addresses looks like it works for boot volumes in a way that URLs don't and scales via wildcarding in a fashion superior to URLs. An underlying assumption I'm making is that storage discovery doesn't need to match the scale of DNS, and hence centralizing config info isn't hobbled by NAT issues. [5] Use of IP addresses would better match the other 3rd party addressing modes, and removes a dependency of the third party Target on DNS. The problems created by NATs are similar to problems that already exist in 3rd party addressing, and hence this doesn't make things worse. [8] An IP connection is identified by 2 IP addresses and two ports, the fact that several thousand of them go through a common DNS, or even a common IP address is not a problem. With the exception of the comment on T10 and 3rd party naming, this is all IMHO. Fire away ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Oct 04 16:09:59 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25255; 4 Oct 2000 12:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA01812 for ips-outgoing; Wed, 4 Oct 2000 11:38:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate.de.ibm.com (d12lmsgate.de.ibm.com [195.212.91.199]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA01773 for ; Wed, 4 Oct 2000 11:37:49 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate.de.ibm.com (1.0.0) with ESMTP id RAA91720 for ; Wed, 4 Oct 2000 17:37:16 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay02.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA100582 for ; Wed, 4 Oct 2000 17:37:18 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696E.0055CC47 ; Wed, 4 Oct 2000 17:37:08 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 4 Oct 2000 18:33:57 +0300 Subject: RE: iSCSI Naming and Discovery Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Joshua, LU numbers are supported. WWNs are an underlying SCSI feature - we don't do any manipulation on them. The only manipulation we are (temporary I hear) forced into is mapping names/addresses into SAM compliant fixed-length addresses for third party commands. We don't propose any character-string substitute for them and they stay fully SAM compliant (8 bytes). Julo "Joshua Tseng/Nishan Systems" on 04/10/2000 18:17:00 Please respond to "Joshua Tseng/Nishan Systems" To: Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu cc: Subject: RE: iSCSI Naming and Discovery Julo, My opinion is that full-fledged LU and WWN need to be supported at least in order to represent existing Fibre Channel devices which use these naming constructs. Without these, how will you use iSCSI to address a LU on a "legacy" device? That is of course assuming you want iSCSI to be able to interoperate with existing devices. Without LU and WWN, a iSCSI-FC gateway supporting Fibre Channel devices will need another layer of naming abstraction to represent a LU on the legacy side. Josh -----Original Message----- From: julian_satran@il.ibm.com [mailto:julian_satran@il.ibm.com] Sent: Wednesday, October 04, 2000 12:52 AM To: ips@ece.cmu.edu Subject: Re: iSCSI Naming and Discovery I though we went through this several times. We went after the a simple scheme that doesn't force us to choose between IPV4 and 6 and will be widely supported by infrastructure. We also took care of not imposing DNS resolution on everybody - there is a specified string form that represent a LITERAL address (IP4 or 6). This shood take care of simple devices and boot. The only think we where reluctant to do is go for a full-fledged LU WWN naming. Julo Black_David@emc.com on 04/10/2000 01:03:19 Please respond to Black_David@emc.com To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: iSCSI Naming and Discovery The emails from Daniel Smith and Josh Tseng provide a good start on a naming discussion. With my co-chair hat off, let me try to add to it. The concepts surrounding NAT (Network Address Translation) figure strongly in this discussion. RFC 2663 is a good source of background for anyone who needs it. An attempt to group and summarize what I see in Daniel and Josh's emails: [1] Internet infrastructure is beyond the control of storage, including iSCSI [Daniel - a),b),f),g),h)] [2] DNS is the naming structure of the Internet, and hence hosts exporting storage services have to be nameable via DNS [Daniel c)]. [3] Network Address Translation and Network Address Port Translation exist and have to be dealt with [Daniel d), Josh (1)]. [4] Discovery at Internet scale is a hard problem. A flexible naming mechanism keeps options open. [Daniel e)]. [5] SCSI 3rd party commands need to name LUNs, including iSCSI LUNs [Daniel i)] *[6] Source, destination, and contents of any packet on the Internet are public information [Daniel j)]. *[7] Security matters for SSPs [Daniel k)]. [8] SSPs can be expected to connect a lot of clients to a single DNS address [Daniel l)]. *[9] Identification of storage and authentication are separate problems and use separate names. [Josh (2)] [10] Naming/routing information is necessary for proxies to identify which of the entities they are proxying for is involved in traffic [1] and [2] are general descriptions of the Internet. [6], [7], and [9] are (mostly) about security, and aside from noting that [6] is incorrect (all of that info can be hidden by a security gateway using IPsec tunnels and both IPsec and SSL/TLS hide payloads), security discussion might be better deferred as I understand that there will be a serious security proposal in the next version of the iSCSI draft. The biggest underlying problem seems to be how to identify an Initiator or Target - it's part of [3], [5], [8] and [10]. [4] is about Discovery, which compounds the naming issues. At a high level there are three basic ways to identify initiators and targets: - Transport address (e.g., IP address, FC port WWN). - Identification information provided as part of session establishment (e.g., username/password, certificate). - Some combination of the above two. The third alternative may be problematic if it leads to needing both the transport and identification information to determine identity. The discussions I've seen seem to be using transport as a hint that may make the identification easier to verify, which seems like a reasonable optimization to relying on the identification information alone. Moving beyond this (e.g., the *.xyz.com servers may only connect from addresses in the a.b.c/24 netblock) increases the amount of information that has to be configured (e.g., that example is better left to firewalls to enforce). NAT mechanisms contribute to the problem by producing networks in which transport addresses aren't useful for identifying anything on the other side of the NAT. Some NATs can be configured to make this problem somewhat simpler via static assignment of IP addresses in one domain to IP addresses in another. NATs are a thorny subject in IETF - while they are widely deployed, not all the important protocols work through them; IPsec AH is the most notable example, and FTP requires a kludge (er, ah, ALG). IMHO, restrictions on the use of NATs with iSCSI may be ok, provided that the consequences are clearly understood. NATs seem to be a large piece of the forcing function that is leading us away from the transport-based identification information used by other SCSI transports. There are security consequences here (e.g., cryptography for session establishment may become mandatory to implement and use), and it will likely complicate discovery. For example, I noted that booting is an issue - if iSCSI always uses URLs to name storage, the result could be a situation in which a DNS server has to be operational and reachable in order to boot. This seems wrong, and the obvious answer of using an IP address in a URL does not work through NATs, which was the original motivation for using URLs. Third-party naming is a tarpit. Putting my WG co-chair hat back on for this paragraph only, I observe that global context for third party names is an unsolved problem in T10; in general, the Initiator of a 3rd party command must use names that resolve to the desired LUNs from the 3rd party command Target's naming perspective. How to do this when an Initiator and Target don't share a naming context is unspecified :-). While it would be a plus for iSCSI to solve this one via have global names for 3rd party commands, I don't think this is a requirement (and whatever we do will have to be worked through T10, as they have the final say on name formats). WG co-chair hat now comes off ... IMHO, discovery is not getting enough attention. The proposed naming scheme complicates discovery without a compelling solution; I'm concerned that the benefits may not justify the costs. My quasi-random walk through this goes something like: - Not being able to find the boot volume because DNS is down or not responding seems to be a wrong answer. The infamous World Wide Wait is bad enough for a browser; it's unacceptable for a reboot. - Hence the boot volume has to be locatable via IP, a TCP port (which could be implicit, e.g. the default iSCSI port) and a LUN (which could be implicit, e.g., LUN 0). Unlike DNS, I could see putting this into an iSCSI HBA card BIOS. - Consistency suggests the approach of using IP addresses as the primary means of identifying possible storage locations, with the possible addition of non-default TCP port numbers). A nice consequence of this is that one can use ranges (e.g., all the storage is in netblock a.b.c/24 at the default iSCSI port, scan those 256 addresses as part of discovery on boot). The corresponding range wildcarding mechanisms for URLs will be more complex. Even if a centralized configuration repository is used (like Fibre Channel), this sort of address wildcarding still looks useful in managing the repository. The netblock example may be too coarse a wildcard. Returning to the issues at the top of this message: [3] NAT become an issue for network designers/admins. Storage becomes something else that they have to get the IP addressing correct for :-(. There are precedents for this, as the default gateway and DNS resolver are already configured via IP addresses, and if those things move to different IP addresses, stuff breaks (a browser can be very unhappy if its host thinks 0.0.0.0 is the only DNS resolver). The downside is that a centralized config repository containing IP addresses becomes a NAT issue - the ALG required to access that across a NAT is ugly enough that it may be necessary to configure the network so that this never happens (which is not the best answer, but may be workable). [4] Discovery based on IP addresses looks like it works for boot volumes in a way that URLs don't and scales via wildcarding in a fashion superior to URLs. An underlying assumption I'm making is that storage discovery doesn't need to match the scale of DNS, and hence centralizing config info isn't hobbled by NAT issues. [5] Use of IP addresses would better match the other 3rd party addressing modes, and removes a dependency of the third party Target on DNS. The problems created by NATs are similar to problems that already exist in 3rd party addressing, and hence this doesn't make things worse. [8] An IP connection is identified by 2 IP addresses and two ports, the fact that several thousand of them go through a common DNS, or even a common IP address is not a problem. With the exception of the comment on T10 and 3rd party naming, this is all IMHO. Fire away ... --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Oct 04 16:09:59 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25267; 4 Oct 2000 12:05 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA01987 for ips-outgoing; Wed, 4 Oct 2000 11:41:11 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from d12lmsgate-2.de.ibm.com (d12lmsgate-2.de.ibm.com [195.212.91.200]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA01978 for ; Wed, 4 Oct 2000 11:41:03 -0400 (EDT) From: julian_satran@il.ibm.com Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com [9.165.215.22]) by d12lmsgate-2.de.ibm.com (1.0.0) with ESMTP id RAA155506 for ; Wed, 4 Oct 2000 17:40:26 +0200 Received: from d12mta02.de.ibm.com (d12mta01_cs0 [9.165.222.237]) by d12relay01.de.ibm.com (8.8.8m3/NCO v4.93) with SMTP id RAA88362 for ; Wed, 4 Oct 2000 17:40:22 +0200 Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5 (863.2 5-20-1999)) id C125696E.00561346 ; Wed, 4 Oct 2000 17:40:10 +0200 X-Lotus-FromDomain: IBMIL@IBMDE To: ips@ece.cmu.edu Message-ID: Date: Wed, 4 Oct 2000 18:37:48 +0300 Subject: Re: iSCSI Naming and Discovery Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, You are correct. The draft proposes only mapping target-names (the part assumed to be transport dependent but limited by SPC to 8 bytes!) into SPC required form. LUN stays as they are. Julo Ralph Weber on 04/10/2000 18:48:36 Please respond to ENDL_TX@computer.org To: ips@ece.cmu.edu cc: (bcc: Julian Satran/Haifa/IBM) Subject: Re: iSCSI Naming and Discovery I've looked at the source messages and believe that these comments are not repeats. Black_David@emc.com wrote: > > [5] SCSI 3rd party commands need to name LUNs, including > iSCSI LUNs [Daniel i)] The EXTENDED COPY command provides fields for the LUN values so target naming (addressing) is the only issue for 3rd party commands and T10 has a work in progress to deal with that ftp://ftp.t10.org/t10/document.00/00-279r0.pdf . Since the T10 work in this area is not yet complete, expeditious input from IPS would be helpful. From owner-ips@ECE.cmu.edu Wed Oct 04 16:09:59 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25260; 4 Oct 2000 12:03 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA00254 for ips-outgoing; Wed, 4 Oct 2000 11:04:48 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA00250 for ; Wed, 4 Oct 2000 11:04:43 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id HAA16475; Wed, 4 Oct 2000 07:50:26 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id HAA25244; Wed, 4 Oct 2000 07:48:15 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI: URL scheme Date: Wed, 4 Oct 2000 08:02:59 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, We should make an attempt to keep the addressing schemes within the SCSI transport SCSI. That means we do not re-invent routers, tunnels, DNS, and everything that goes with it. > I would like to propose the following requirement for any naming scheme. > > * Any naming scheme proposed MUST support multiple targets > behind a single > IP address I have no problem with that except to add that this IP is routable from the client. I call this address the SCSI portal. > * Any naming scheme proposed SHOULD support multiple targets behind > a single port. This helps make traffic analysis easier. Again, a good statement with the following provision, the address space behind the SCSI portal is SCSI and not IP. If you wish to bridge between two portals, then you should setup a connection transparent to the SCSI address space. In otherwords, you would not indicate within the SCSI address space any IPs. This would be done as part of the SCSI address configuration using LDAP. This keeps each address space separate and clean and yet you get the job done. > Consider the topology pictured below. An initiator is connected > to a private network that has a gateway to the Internet. Similarly, > the target is connected to a private network which is connected > by another gateway to the Internet. > Server (Drive D) \ (always Private) SCSI Space ---- SCSI Space \ \ \ \ Routable SCSI Portal 1 Transparent Bridge Connection \ \ \ \ IP Space IP Space / / / / NAT Routable SCSI Portal 2 / / Non-Routable Client (Bob) Server (Drive C) How does Bob get to talk to the Drive C? Bob's system only knows about the Portal 1 however his drive is actually on Portal 2. Bob's system knows about this through a sequence of boot up steps. DHCP informs his system of an LDAP server. Bob's system then queries this server for SCSI services via SCSI class objects. Bob's system leans how to authenticate with Portal 1 and what target/lun/wwn his system will find his Drive C at within the realm of Portal 1. Portal 1 knows that this target/lun/wwn is actually within the realm of Portal 2. At Portal 2, his drive is at yet another target/lun/wwn. Portal 1 will make the translation for Bob without Bob's knowledge into Portal 2. Bob does not know how to Authenticate with Portal 2 nor where Portal 2 keeps his drive. That would be handled transparently for Bob. As Portal 1 always responds back to the client Bob at the IP and port that he came in at, the NAT is not a problem. As far as Portals are concerned, they must be exposed as Routable. I have changed the name Bob into Drive C and used Bob as the Client. Otherwise, I get confused. I have changed Fred into Drive D. I know this is very MSDOS and I could have chosen SCSI directory names such as /home, /bin, or /temp Doug > > ------------- > Current model > ------------- > > SCSI device identifier is a string in two parts: a hostname and a > host-specific > name (HSN). The hostname is either a domain name or an IPv4 or > IPv6 address. > > Steps: > 0) Configuration mechanism tells initiator to talk to SCSI device > identifier "bob" > > 1) Do a DNS (or other name service) lookup on bob. > Reply arrives with IP address of Gateway #2 > > 2) TCP connect to gateway #2 on well known iSCSI port > > 3) As part of first login packet, send "Target: bob" > > 4) Gateway #2 resolves the name "bob" on the internal > network and gets the private IP address of the Target Bob > > 5) Gateway #2 opens a TCP connection to Target Bob and > passes all traffic between connections > > Third party command from bob to fred > 0) Configuration tells initiator that bob should talk to > SCSI device identifier "fred" > > 1) Initiator sends command to bob which has SCSI device identifier > "fred" > > 2) Bob resolves fred's name and gets Fred's private Ip address > > 3) Bob connects to Fred and they do their thing > > > Refinements > ----------- > Configuration mechanism (e.g. LDAP server) in step 0, along with > returning the SCSI device identifier "bob", also returns the IP > address of gateway #2. The DNS lookup in #1 is avoided. > > DNS lookup of bob actually returns gateway #1 (would work for > iSCSI and HTTP/1.1 but unlikely to work for other protocols). > Gateway #1 then looks up the name on the Internet and gets > gateway #2... and so on... > > > ---------------- > IP address model > ---------------- > > A target name is made of two parts: an IP address and a target ID. > The target ID can be either a string or a fixed-length binary > quantity. The target ID is not necessarily globally unique; it > can be IP-address specific. > > > Step 0) Initiator queries the configuration mechanisms and gets > the IP address of gateway #2 and the target ID for bob. > > Step 1) Open TCP connection to IP address of gateway #2 on well-known > iSCSI port > > Step 2) As part of first login packet, initiator send the target > ID for bob > > Step 3) Gateway #2 opens connection to bob based on the target ID > > Step 4) Gateway #2 passes all info between connections > > Third party commands - passable way > -------------------- > > Step 0) Configuartion mechanism tells initiator that bob should talk > to fred. The configuration mechanism tells the initiator about the > private IP address/target ID of Fred as seen by Bob (how does it find > this out??) > > Step 1) Initiator sends third party command with Fred's IP address > and target ID > > Step 2) Bob opens connection to Fred's IP address+port > > > Third party commands - icky way > -------------------- > Step 0) Configuartion mechanism tells initiator that bob should talk > to fred. The configuration mechanism tells the initiator about the > IP address of gateway #2 and target ID of fred > > Step 1) Initiator sends third party command with IP address of gateway#2 > and target ID of fred > > Step 2) Gateway #2 intercepts the third party command and rewrites it > to have Fred's private IP address (based on the target ID) > > Step 3) Bob opens connection to Fred's private IP address on well > known iSCSI > port, sends target ID in the first iSCSI PDU, etc. > > or > > Step 2) Bob opens a connection to gateway #2's IP address and > passes target > ID > > Step 3) Bob gets redirected to Fred (or... gateway opens connection > to Fred and passes all commands between Fred & Bob) > > > In this approach, the gateway must be able to map a target ID to > an IP address > and a (possibly new) target ID. > > > Note on using port numbers instead of target IDs > ------------------------------------------------ > > The only thing that changes in the analysis above is that the target > ID need not be sent on connection open, since the destination port > number on the TCP connection identifies the target. > > > > In this specific scenario, it seems to me that using strings entirely > is cleaner than passing IP addresses which are potentially non-sensical. > > -Costa > > > > > > From owner-ips@ECE.cmu.edu Wed Oct 04 16:09:59 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25281; 4 Oct 2000 12:06 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA00629 for ips-outgoing; Wed, 4 Oct 2000 11:14:04 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA00623 for ; Wed, 4 Oct 2000 11:14:00 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id LAA26032; Wed, 4 Oct 2000 11:07:48 -0400 Received: from f3n42e (d03nm042h.boulder.ibm.com [9.99.140.42]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e94FDtG162650; Wed, 4 Oct 2000 09:13:55 -0600 Importance: Normal Subject: Re: iSCSI: URL scheme To: csapuntz@cisco.com Cc: X-Mailer: Lotus Notes Release 5.0.2a (Intl) 23 November 1999 Message-ID: From: "Prasenjit Sarkar/Almaden/IBM" Date: Wed, 4 Oct 2000 08:13:37 -0700 X-MIMETrack: Serialize by Router on D03NM042/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/04/2000 08:13:55 AM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Hi, Can you clarify what you meant by "target" out here? I've heard there's a lot of work going on in T10 to define what exactly it is, specially in terms of multiple ports. I would also like clarification if SAM allows targets to share ports as implied by requirement (2). Regds, Prasenjit Sent by: owner-ips@ece.cmu.edu To: cc: csapuntz@cisco.com Subject: Re: iSCSI: URL scheme I would like to propose the following requirement for any naming scheme. * Any naming scheme proposed MUST support multiple targets behind a single IP address * Any naming scheme proposed SHOULD support multiple targets behind a single port. This helps make traffic analysis easier. Consider the topology pictured below. An initiator is connected to a private network that has a gateway to the Internet. Similarly, the target is connected to a private network which is connected by another gateway to the Internet. Initiator - Private Network - Gateway 1 \ \ Internet / / Target - Private Network - Gateway 2 (bob) | Target (fred) How does the Initiator get to talk to the Target bob? ------------- Current model ------------- SCSI device identifier is a string in two parts: a hostname and a host-specific name (HSN). The hostname is either a domain name or an IPv4 or IPv6 address. Steps: 0) Configuration mechanism tells initiator to talk to SCSI device identifier "bob" 1) Do a DNS (or other name service) lookup on bob. Reply arrives with IP address of Gateway #2 2) TCP connect to gateway #2 on well known iSCSI port 3) As part of first login packet, send "Target: bob" 4) Gateway #2 resolves the name "bob" on the internal network and gets the private IP address of the Target Bob 5) Gateway #2 opens a TCP connection to Target Bob and passes all traffic between connections Third party command from bob to fred 0) Configuration tells initiator that bob should talk to SCSI device identifier "fred" 1) Initiator sends command to bob which has SCSI device identifier "fred" 2) Bob resolves fred's name and gets Fred's private Ip address 3) Bob connects to Fred and they do their thing Refinements ----------- Configuration mechanism (e.g. LDAP server) in step 0, along with returning the SCSI device identifier "bob", also returns the IP address of gateway #2. The DNS lookup in #1 is avoided. DNS lookup of bob actually returns gateway #1 (would work for iSCSI and HTTP/1.1 but unlikely to work for other protocols). Gateway #1 then looks up the name on the Internet and gets gateway #2... and so on... ---------------- IP address model ---------------- A target name is made of two parts: an IP address and a target ID. The target ID can be either a string or a fixed-length binary quantity. The target ID is not necessarily globally unique; it can be IP-address specific. Step 0) Initiator queries the configuration mechanisms and gets the IP address of gateway #2 and the target ID for bob. Step 1) Open TCP connection to IP address of gateway #2 on well-known iSCSI port Step 2) As part of first login packet, initiator send the target ID for bob Step 3) Gateway #2 opens connection to bob based on the target ID Step 4) Gateway #2 passes all info between connections Third party commands - passable way -------------------- Step 0) Configuartion mechanism tells initiator that bob should talk to fred. The configuration mechanism tells the initiator about the private IP address/target ID of Fred as seen by Bob (how does it find this out??) Step 1) Initiator sends third party command with Fred's IP address and target ID Step 2) Bob opens connection to Fred's IP address+port Third party commands - icky way -------------------- Step 0) Configuartion mechanism tells initiator that bob should talk to fred. The configuration mechanism tells the initiator about the IP address of gateway #2 and target ID of fred Step 1) Initiator sends third party command with IP address of gateway#2 and target ID of fred Step 2) Gateway #2 intercepts the third party command and rewrites it to have Fred's private IP address (based on the target ID) Step 3) Bob opens connection to Fred's private IP address on well known iSCSI port, sends target ID in the first iSCSI PDU, etc. or Step 2) Bob opens a connection to gateway #2's IP address and passes target ID Step 3) Bob gets redirected to Fred (or... gateway opens connection to Fred and passes all commands between Fred & Bob) In this approach, the gateway must be able to map a target ID to an IP address and a (possibly new) target ID. Note on using port numbers instead of target IDs ------------------------------------------------ The only thing that changes in the analysis above is that the target ID need not be sent on connection open, since the destination port number on the TCP connection identifies the target. In this specific scenario, it seems to me that using strings entirely is cleaner than passing IP addresses which are potentially non-sensical. -Costa From owner-ips@ECE.cmu.edu Wed Oct 04 16:20:00 2000 Return-Path: Received: by cnoc.pdl.cs.cmu.edu id aa25349; 4 Oct 2000 12:14 EDT Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25249; 4 Oct 2000 12:02 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA00924 for ips-outgoing; Wed, 4 Oct 2000 11:21:05 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from spdmraaa.compuserve.com (ds-img-rel-1.compuserve.com [149.174.206.140]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA00919 for ; Wed, 4 Oct 2000 11:21:00 -0400 (EDT) Received: (from mailgate@localhost) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) id LAA24511 for ips@ece.cmu.edu; Wed, 4 Oct 2000 11:20:29 -0400 (EDT) Received: from compuserve.com (mid-tgn-ngj-vty34.as.wcom.net [216.192.82.34]) by spdmraaa.compuserve.com (8.9.3/8.9.3/SUN-REL-1.3) with ESMTP id LAA24465 for ; Wed, 4 Oct 2000 11:20:26 -0400 (EDT) Message-ID: <39DB5152.5919894@compuserve.com> Date: Wed, 04 Oct 2000 10:48:36 -0500 From: Ralph Weber Reply-To: ENDL_TX@computer.org X-Mailer: Mozilla 4.7 [en]C-CCK-MCD NSCPCD47 (Win95; I) X-Accept-Language: en,pdf MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI Naming and Discovery References: <0F31E5C394DAD311B60C00E029101A0704101052@corpmx9.isus.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk I've looked at the source messages and believe that these comments are not repeats. Black_David@emc.com wrote: > > [5] SCSI 3rd party commands need to name LUNs, including > iSCSI LUNs [Daniel i)] The EXTENDED COPY command provides fields for the LUN values so target naming (addressing) is the only issue for 3rd party commands and T10 has a work in progress to deal with that ftp://ftp.t10.org/t10/document.00/00-279r0.pdf . Since the T10 work in this area is not yet complete, expeditious input from IPS would be helpful. From owner-ips@ECE.cmu.edu Wed Oct 04 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25387; 4 Oct 2000 12:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA02756 for ips-outgoing; Wed, 4 Oct 2000 12:02:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA02752 for ; Wed, 4 Oct 2000 12:02:12 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id IAA16493; Wed, 4 Oct 2000 08:47:57 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id IAA25389; Wed, 4 Oct 2000 08:45:46 -0700 From: "Douglas Otis" To: "John Hufferd/San Jose/IBM" , Subject: RE: iSCSI: URL scheme Date: Wed, 4 Oct 2000 09:00:31 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk John, The problem that I see with your present scheme that is causing the confusion and difficulty is from trying to create an amalgam of IP and SCSI addresses. Although you may view URLs as extensible and perhaps capable of handling both IP and SCSI, the load created will be horrendous. In judo, they teach you how use inertia in your favor. In this case, you are attempting to change the course of the entire world using a new symbolic router. I am not as clever as you. I only wish to translate firmly establish binary fields that address devices. Yes, I can translate these and perhaps even bridge some of them. I can not parse every packet looking for a symbolic address and then route this packet dynamically in both SCSI and IP domains with your new breed of symbolic router, name and authentication server. On top of that, you wish to see everything extracted from this single connection for authentication and configuration. We end up doing a great deal of looking at these packets and then a great deal of deciding what to do with them. None of this is productive work. You develop a class 2 control for a class 3 world for additional trouble. I would hope that you want a simple solution and not one sure to thwart all efforts. Your truly, Doug From owner-ips@ECE.cmu.edu Wed Oct 04 16:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25385; 4 Oct 2000 12:22 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id LAA01969 for ips-outgoing; Wed, 4 Oct 2000 11:40:59 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pallas.veritas.com (pallas.veritas.com [204.177.156.25]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id LAA01958 for ; Wed, 4 Oct 2000 11:40:52 -0400 (EDT) Received: from megami.veritas.com (megami.veritas.com [192.203.46.101]) by pallas.veritas.com (8.9.1a/8.9.1) with SMTP id IAA07787 for ; Wed, 4 Oct 2000 08:41:10 -0700 (PDT) Received: from veritas.com([172.22.1.51]) (3201 bytes) by megami.veritas.com via sendmail with P:esmtp/R:smart_host/T:smtp (sender: ) id for ; Wed, 4 Oct 2000 08:40:50 -0700 (PDT) (Smail-3.2.0.101 1997-Dec-17 #4 built 1999-Aug-24) Message-ID: <39DB51D5.747E3A60@veritas.com> Date: Wed, 04 Oct 2000 08:50:45 -0700 From: John Muth X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: ips@ece.cmu.edu Subject: Re: iSCSI: problem with LUN discovery References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Julo, FYI. Assuming the initiators are running higher level disk management S/W like VxVM (VERITAS Volume Manager), this won't be a problem. VxVM already deals with device paths moving around by doing it's disk identification based on the label, not the device path. John Muth VERITAS Software julian_satran@il.ibm.com wrote: > > JP, > > Unless I misunderstood you this is entirely a T10 issue. > Presenting a different LU map to each initiator as identified by the port > or the ACCESS-ID is intentional (each initiator can view a different set of > LUs)and part > of the protection mechanisms of SCSI. > > With iSCSI at later stages (when we start handling discovery and > management) > the pain of getting to a specific volume might be eased. > > Julo > > Raghavendra Rao on 27/09/2000 23:05:22 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: problem with LUN discovery > > Julian, > > Do you agree that following is a problem ? If so, can we fix it ? > > SAM doesn't prohibit a target device from presenting initiator specific > values for LUs. So it is believed that similar values for a LU may not > necessarily address the same LU inside a target when accessed from > different > initiators. > > So the vendor ID page 83 of INQUIRY command is considered to be unique > for a LU - There is one caveat to this though, that the page 83 values > could optionally be target port specific. Anyway, if the hosts were to > use a more persistent values such as page-83 identifier, say after reboot > or after reconfiguration, it first needs to translate this page-83 > identifier > into a LUN. This can be done by first issuing a REPORT_LUN command to > LUN 0 and issuing an INQUIRY page-83 command to each LU in the list > obtained from REPORT_LUN response until a successful match is found. > An initiator could cache the mapping for future speedier lookups, but > it still has to linearly probe out all LUs to get the page-83 unique > identifiers. > > I hate to do this static probing; This problem can be fixed by either > > a) mandating that LUN values are persistent and same for all > initiators (more unlikely to be accepted) > > OR > > b) introducing a new page code for INQUIRY in which the initiator > will pass down the page-83 identifier in the parameter list > to LUN 0 and the target will respond with the LUN corresponding > to this identifier; Of course, this has to go to T10 for approval. > > Thanks. > -JP From owner-ips@ECE.cmu.edu Wed Oct 04 17:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25607; 4 Oct 2000 12:56 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA03044 for ips-outgoing; Wed, 4 Oct 2000 12:09:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA03034 for ; Wed, 4 Oct 2000 12:09:14 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id IAA16518; Wed, 4 Oct 2000 08:54:56 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id IAA25430; Wed, 4 Oct 2000 08:52:45 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI Naming and Discovery Date: Wed, 4 Oct 2000 09:07:30 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) In-Reply-To: <39DB5152.5919894@compuserve.com> Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-ips@ece.cmu.edu Precedence: bulk Ralph, In my opinion, the document looks good as is. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Ralph Weber > Sent: Wednesday, October 04, 2000 8:49 AM > To: ips@ece.cmu.edu > Subject: Re: iSCSI Naming and Discovery > > > I've looked at the source messages and believe that these > comments are not repeats. > > Black_David@emc.com wrote: > > > > > > [5] SCSI 3rd party commands need to name LUNs, including > > iSCSI LUNs [Daniel i)] > > The EXTENDED COPY command provides fields for the LUN values > so target naming (addressing) is the only issue for 3rd > party commands and T10 has a work in progress to deal with > that ftp://ftp.t10.org/t10/document.00/00-279r0.pdf . Since > the T10 work in this area is not yet complete, expeditious > input from IPS would be helpful. > > > > > From owner-ips@ECE.cmu.edu Wed Oct 04 17:20:01 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25702; 4 Oct 2000 13:18 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id MAA03955 for ips-outgoing; Wed, 4 Oct 2000 12:29:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mxbh4.isus.emc.com (mxbh4.isus.emc.com [168.159.208.52]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id MAA03951 for ; Wed, 4 Oct 2000 12:29:47 -0400 (EDT) From: Black_David@emc.com Received: by mxbh4.isus.emc.com with Internet Mail Service (5.5.2650.21) id <4GS433N4>; Wed, 4 Oct 2000 12:29:13 -0400 Message-ID: <0F31E5C394DAD311B60C00E029101A0704101059@corpmx9.isus.emc.com> To: ips@ece.cmu.edu Subject: RE: iSCSI Naming and Discovery Date: Wed, 4 Oct 2000 12:29:10 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk All of this is with my WG co-chair hat off: At a high level, my concern about DNS and LDAP is based on Leslie Lamport's infamous (in jest) definition of a distributed system -- one on which he can't get work done because a computer he's never heard of is down :-). While the sort of infrastructure services that run on "a computer he's never heard of" have been getting more and more reliable, the moral I take away from this is to be cautious about introducing dependencies on services. Every introduced dependency provides new ways for things to fail, and this needs to be balanced against the new ways in which things work because of the dependency. IMHO, NATs and IPv4/v6 transparency don't tip the balance far enough to favor DNS or LDAP (but I always reserve the right to change my opinion :-) ). For current block storage, a host's access to its storage either does not depend on any external service (parallel SCSI, FC-AL) or a depends on the Fibre Channel fabric nameserver (FC-SW). The fabric nameserver is usually embedded in the Fibre Channel switches, so that if the nameserver's not working, the switch probably isn't working either (e.g., in practice, if the nameserver isn't responding to the HBAs, the switch gets replaced). In contrast, both DNS and LDAP are external services, probably located on general purpose servers, on which block storage access currently does not depend. LDAP in particular can be rather complex, as some of its leading implementations sit on top of full-fledged relational databases. FWIW, the original design of the FC fabric nameserver was LDAP-based, but this was discovered to be infeasible -- an LDAP server doesn't embed into a switch very well, or at least didn't at the time. The characterization of LDAP as "very simple" is significantly off the mark, IMHO. The discussions of DNS and LDAP on this list have had the flavor of these being ubiquitous completely reliable and available services on which dependencies are no big deal. When everything is working right, that may be the case, but everything doesn't always work right; disasters happen taking down entire facilities and requiring them to be brought up from scratch. An example of what could go wrong is that two DNS servers could be configured in a way that each depends on the other for its storage. If each server is rebooted individually to test the configuration, things work, but if both go down, neither will come back up. This is one more problem for overworked system and network administrators - having to determine which servers must use IP names for storage vs. DNS names, with nasty consequences of getting it wrong. This isn't a fatal problem, but every time management becomes more complicated, it's one more barrier to adoption. This is why I don't put much weight on the argument that IP URLs can be used where DNS or LDAP won't work - there are too many opportunities to screw this up if DNS is allowed in general. Boot is a specific concern, because if DNS or LDAP is involved in locating the boot volume, then the HBA card BIOS has to be able to access DNS or LDAP. There's nothing inherently difficult or infeasible about this, it's just more code in yet another place ... making it harder to deploy/adopt this. Beyond this, I need to toss in a few important scattered comments. Most systems don't boot via BOOTP today; demanding that everything boot via BOOTP in order to avoid problems created by an entirely too clever naming structure sounds like a bad tradeoff. This will lead to iSCSI repeating the Fibre Channel experience of boot not working for a long period of time after deployment of the technology. Customer reactions to that Fibre Channel experience have been negative to the point of just barely printable in some instances. The LDAP vs. nameserver discussion misses the point. The Fibre Channel experience makes it clear that iSCSI is going to need some sort of central repository of configuration information to get the sort of discovery functionality that is expected. That repository is at least a nameserver, and possibly more - the approach of defining LDAP schemas is the result of deciding to use LDAP as the communication protocol for accessing iSCSI nameservers. The [3] discussion of NATs at the end of my previous message was rather verbose. The bottom line is that if IP addresses are stored in and distributed by a central iSCSI config repository, then NATs cause a problem, and a brute force solution that is likely to work in many cases is not to put NATs between the repository and its users. Alternate solutions based on application level gateways and storing addresses with respect to users of the repository rather than the repository server are workable, but more complex. On [4], I see the following advantages of IP addresses over DNS URLs. - IP addresses are much better for booting, and in some cases have to be used (see above). DNS hostnames are then an additional mechanism that adds implementation work and complicates management (see above). - Scanning a range of possible locations of Targets has some attractive properties, because it removes the obligation of a new target to proactively make itself known to some iSCSI configuration repository (such mechanisms do work, Fibre Channel is an example, but the FC solution doesn't directly apply to iSCSI). It seems to me that wildcarding an IP address via a mask or range works better than wildcarding some portion of a URL (e.g., if the wildcard is xyz*.abc.com, there's a fair amount of work involved in finding xyzf, xyz01, and xyz17a in the abc.com domain, by comparison to finding things that match 192.48.27.128-191 or equivalently 192.48.27.128/27). One more note on NATs: > FTP must use the passive mode to get past the NAT. That's not correct. There are NATs deployed that detect FTP traffic and rewrite the IP addresses in the FTP payloads to make everything work right. The official name for this feature/kludge is an Application Level Gateway (ALG), and the FTP features that require it are generally regarded as an example not to be emulated. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Oct 04 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25773; 4 Oct 2000 13:58 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA05897 for ips-outgoing; Wed, 4 Oct 2000 13:16:21 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from storeage_nt.store-age.com ([199.203.178.211]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA05891 for ; Wed, 4 Oct 2000 13:16:15 -0400 (EDT) Received: by storeage_nt.store-age.com with Internet Mail Service (5.5.2650.21) id ; Wed, 4 Oct 2000 20:13:49 +0200 Message-ID: From: Nelson Nahum To: "'John Muth'" , ips@ece.cmu.edu Subject: RE: iSCSI: problem with LUN discovery Date: Wed, 4 Oct 2000 20:13:49 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk John, The problem is that the signature written by the VxVM is not standard and only is recognized by the Veritas VxVM. Due to the fact that the important thing is to identify the media not the controller (as Veritas does), the best way is to have the serial number written in the media in a non accessible sector, and loaded by the controller to one of the Inquiry pages of the LU. In this manner no matter which channel, WWN, LUN the specific LUN is presented can be recognized by every initiator. Nelson Nahum CTO StoreAge Networking Technologies. -----Original Message----- From: John Muth [mailto:muth@veritas.com] Sent: Wednesday, October 04, 2000 4:51 PM To: ips@ece.cmu.edu Subject: Re: iSCSI: problem with LUN discovery Julo, FYI. Assuming the initiators are running higher level disk management S/W like VxVM (VERITAS Volume Manager), this won't be a problem. VxVM already deals with device paths moving around by doing it's disk identification based on the label, not the device path. John Muth VERITAS Software julian_satran@il.ibm.com wrote: > > JP, > > Unless I misunderstood you this is entirely a T10 issue. > Presenting a different LU map to each initiator as identified by the port > or the ACCESS-ID is intentional (each initiator can view a different set of > LUs)and part > of the protection mechanisms of SCSI. > > With iSCSI at later stages (when we start handling discovery and > management) > the pain of getting to a specific volume might be eased. > > Julo > > Raghavendra Rao on 27/09/2000 23:05:22 > > Please respond to Raghavendra Rao > > To: Julian Satran/Haifa/IBM@IBMIL > cc: > Subject: iSCSI: problem with LUN discovery > > Julian, > > Do you agree that following is a problem ? If so, can we fix it ? > > SAM doesn't prohibit a target device from presenting initiator specific > values for LUs. So it is believed that similar values for a LU may not > necessarily address the same LU inside a target when accessed from > different > initiators. > > So the vendor ID page 83 of INQUIRY command is considered to be unique > for a LU - There is one caveat to this though, that the page 83 values > could optionally be target port specific. Anyway, if the hosts were to > use a more persistent values such as page-83 identifier, say after reboot > or after reconfiguration, it first needs to translate this page-83 > identifier > into a LUN. This can be done by first issuing a REPORT_LUN command to > LUN 0 and issuing an INQUIRY page-83 command to each LU in the list > obtained from REPORT_LUN response until a successful match is found. > An initiator could cache the mapping for future speedier lookups, but > it still has to linearly probe out all LUs to get the page-83 unique > identifiers. > > I hate to do this static probing; This problem can be fixed by either > > a) mandating that LUN values are persistent and same for all > initiators (more unlikely to be accepted) > > OR > > b) introducing a new page code for INQUIRY in which the initiator > will pass down the page-83 identifier in the parameter list > to LUN 0 and the target will respond with the LUN corresponding > to this identifier; Of course, this has to go to T10 for approval. > > Thanks. > -JP From owner-ips@ECE.cmu.edu Wed Oct 04 18:00:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25762; 4 Oct 2000 13:51 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA05532 for ips-outgoing; Wed, 4 Oct 2000 13:08:35 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from csapuntz-u1.cisco.com (sj-isp-nat-pool-66.cisco.com [204.69.198.66]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA05525 for ; Wed, 4 Oct 2000 13:08:31 -0400 (EDT) From: csapuntz@cisco.com Received: by csapuntz-u1.cisco.com (Postfix, from userid 500) id 775318E3C; Wed, 4 Oct 2000 10:03:27 -0700 (PDT) To: Black_David@emc.com Cc: ips@ece.cmu.edu Subject: Re: iSCSI Naming and Discovery References: <0F31E5C394DAD311B60C00E029101A0704101059@corpmx9.isus.emc.com> Cc: csapuntz@cisco.com X-csapuntz: outbox Mime-Version: 1.0 (generated by tm-edit 1.5) Content-Type: text/plain; charset=US-ASCII Date: 04 Oct 2000 10:03:27 -0700 In-Reply-To: Black_David@emc.com's message of "Wed, 4 Oct 2000 12:29:10 -0400" Message-ID: Lines: 27 X-Mailer: Gnus v5.6.45/XEmacs 21.1 - "Bryce Canyon" Sender: owner-ips@ece.cmu.edu Precedence: bulk > - Scanning a range of possible locations of Targets has some attractive > properties, because it removes the obligation of a new target > to proactively make itself known to some iSCSI configuration > repository (such mechanisms do work, Fibre Channel is an example, > but the FC solution doesn't directly apply to iSCSI). It seems > to me that wildcarding an IP address via a mask or range works > better than wildcarding some portion of a URL (e.g., if the > wildcard is xyz*.abc.com, there's a fair amount of work involved > in finding xyzf, xyz01, and xyz17a in the abc.com domain, > by comparison to finding things that match 192.48.27.128-191 or > equivalently 192.48.27.128/27). David, I do not believe that wildcarding is compelling. Scanning will be too slow. How many packets do you send to an IP address until you decide that it's not up? After how many seconds to you give up? If you want to discover who on your subnet speaks iSCSI, it'd be much better to use a subnet-local multicast (a multicast message that says: who speaks iSCSI?). If you're going across subnets, then it's probably time to use a directory service. -Costa From owner-ips@ECE.cmu.edu Wed Oct 04 18:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa25866; 4 Oct 2000 14:12 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id NAA06682 for ips-outgoing; Wed, 4 Oct 2000 13:35:52 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from pallas.veritas.com (pallas.veritas.com [204.177.156.25]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id NAA06677 for ; Wed, 4 Oct 2000 13:35:46 -0400 (EDT) Received: from megami.veritas.com (megami.veritas.com [192.203.46.101]) by pallas.veritas.com (8.9.1a/8.9.1) with SMTP id KAA19088 for ; Wed, 4 Oct 2000 10:36:06 -0700 (PDT) Received: from veritas.com(jimi.veritas.com[166.98.10.51]) (4717 bytes) by megami.veritas.com via sendmail with P:esmtp/R:smart_host/T:smtp (sender: ) id for ; Wed, 4 Oct 2000 10:35:45 -0700 (PDT) (Smail-3.2.0.101 1997-Dec-17 #4 built 1999-Aug-24) Message-ID: <39DB6B55.198576D4@veritas.com> Date: Wed, 04 Oct 2000 10:39:33 -0700 From: John Muth X-Mailer: Mozilla 4.51 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 CC: ips@ece.cmu.edu Subject: Re: iSCSI: problem with LUN discovery References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Sender: owner-ips@ece.cmu.edu Precedence: bulk Nelson, I don't disagree, disk serial numbers are a good thing. I was simply trying to point out (perhaps badly) that initiator specific LUN mapping presents no problem to the initiator side since nobody in their right mind uses the path to determine the identity of a disk. Now, if we could only convince all of the disk manufacturers in the world (both real and virtual) to give us unique serial numbers (we've seen drive models were every unit returned the same constant serial number. and don't get me started about arrays....) John Muth VERITAS Software Nelson Nahum wrote: > > John, > > The problem is that the signature written by the VxVM is not standard and > only is recognized by the Veritas VxVM. > Due to the fact that the important thing is to identify the media not the > controller (as Veritas does), the best way is to have the serial number > written in the media in a non accessible sector, and loaded by the > controller to one of the Inquiry pages of the LU. In this manner no matter > which channel, WWN, LUN the specific LUN is presented can be recognized by > every initiator. > > Nelson Nahum > CTO > StoreAge Networking Technologies. > > -----Original Message----- > From: John Muth [mailto:muth@veritas.com] > Sent: Wednesday, October 04, 2000 4:51 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: problem with LUN discovery > > Julo, > > FYI. Assuming the initiators are running higher level disk > management S/W like VxVM (VERITAS Volume Manager), this won't > be a problem. VxVM already deals with device paths moving > around by doing it's disk identification based on the label, > not the device path. > > John Muth > VERITAS Software > > julian_satran@il.ibm.com wrote: > > > > JP, > > > > Unless I misunderstood you this is entirely a T10 issue. > > Presenting a different LU map to each initiator as identified by the port > > or the ACCESS-ID is intentional (each initiator can view a different set > of > > LUs)and part > > of the protection mechanisms of SCSI. > > > > With iSCSI at later stages (when we start handling discovery and > > management) > > the pain of getting to a specific volume might be eased. > > > > Julo > > > > Raghavendra Rao on 27/09/2000 23:05:22 > > > > Please respond to Raghavendra Rao > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: > > Subject: iSCSI: problem with LUN discovery > > > > Julian, > > > > Do you agree that following is a problem ? If so, can we fix it ? > > > > SAM doesn't prohibit a target device from presenting initiator specific > > values for LUs. So it is believed that similar values for a LU may not > > necessarily address the same LU inside a target when accessed from > > different > > initiators. > > > > So the vendor ID page 83 of INQUIRY command is considered to be unique > > for a LU - There is one caveat to this though, that the page 83 values > > could optionally be target port specific. Anyway, if the hosts were to > > use a more persistent values such as page-83 identifier, say after > reboot > > or after reconfiguration, it first needs to translate this page-83 > > identifier > > into a LUN. This can be done by first issuing a REPORT_LUN command to > > LUN 0 and issuing an INQUIRY page-83 command to each LU in the list > > obtained from REPORT_LUN response until a successful match is found. > > An initiator could cache the mapping for future speedier lookups, but > > it still has to linearly probe out all LUs to get the page-83 unique > > identifiers. > > > > I hate to do this static probing; This problem can be fixed by either > > > > a) mandating that LUN values are persistent and same for all > > initiators (more unlikely to be accepted) > > > > OR > > > > b) introducing a new page code for INQUIRY in which the initiator > > will pass down the page-83 identifier in the parameter list > > to LUN 0 and the target will respond with the LUN corresponding > > to this identifier; Of course, this has to go to T10 for approval. > > > > Thanks. > > -JP From owner-ips@ECE.cmu.edu Wed Oct 04 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26007; 4 Oct 2000 15:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA09343 for ips-outgoing; Wed, 4 Oct 2000 14:35:24 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e31.bld.us.ibm.com (e31.co.us.ibm.com [32.97.110.129]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA09339 for ; Wed, 4 Oct 2000 14:35:17 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e31.bld.us.ibm.com (8.9.3/8.9.3) with ESMTP id OAA71454 for ; Wed, 4 Oct 2000 14:29:08 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e94IZFG146120 for ; Wed, 4 Oct 2000 12:35:15 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: SCSI URL scheme To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 4 Oct 2000 11:33:36 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/04/2000 12:35:15 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Raghavendra Rao, Today, and perhaps into the future, it is up to the Storage Controller to permit access, from the approprate Host (initiators), to the approprate LUs (disks). They only return to the Report LUNs requestor those LUNs that the initiators are authorized to see. I think that is a good thing. In the example you focus on, says that it is a problem for 100s of systems issuing a Report LUNs, and Inquiry commands. Since most of the remote systems will probably only request one disk, it will probably be only the logical LU0 which they are permitted to see. I do not see a problem there. In fact even if they have a 100 LUs that they are permitted to get at, I do not see a problem with the Report LUNs sending back the LU#s that each of those systems are authorized to use, and those systems issuing the Inquiry commands to those authorized LUs. (The interaction is trivial.) This is the way things work today, and at the moment I do not think it is broken. It is the ultimate target that is responsible for enforcing the authorization process that the installation establishes. Today, with the lack of good security in the FC network, we have a problem which we can address with iSCSI. But after we get good Authentication, it seems to me that the Storage Controller enforcement of what initiators can address which LUs is good and approprate. . . . John L. Hufferd Raghavendra Rao @ece.cmu.edu on 10/04/2000 09:45:27 AM Please respond to Raghavendra Rao Sent by: owner-ips@ece.cmu.edu To: ips@ece.cmu.edu cc: Subject: Re: SCSI URL scheme > > First off, this is suppose to be done (normally) once, but perhaps at > various (logical) Plug & Play points. If we are talking about the Plug & > Play points in time, then there could be some additional conversation. But > independent of that, which so far I think we have been, the Host determines > ALL his addressable LUs when he comes up, and today that means "REPORT > LUNS", followed by approprate INQUIRYs. This is not a big deal at bring > up. > It depends on the storage controller and the configuration. If iSCSI indeed becomes very popular, it is not hard to visualize a lot of compute nodes (of the order of 100s just to start with) accessing/sharing a big storage device that could have a lot of LUs (of the order of thousands at the least). It is possible to configure the storage device to selectively report LUs in the REPORT LUN response so as to limit the number of INQUIRYs fired from the initiators - But I see this only as a work around - not a solution to the problem. I think a better way to translate a LU-WWN to LUN value is required to avoid overwhelming number of INQUIRYs fired from all the compute nodes trying to access the storage device; This is not only wasting the network bandwidth, but is also slowing down the host's reconfiguration/booting. -JP From owner-ips@ECE.cmu.edu Wed Oct 04 19:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26005; 4 Oct 2000 15:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA08611 for ips-outgoing; Wed, 4 Oct 2000 14:19:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA08605 for ; Wed, 4 Oct 2000 14:19:23 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA16582; Wed, 4 Oct 2000 11:05:07 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA25787; Wed, 4 Oct 2000 11:02:56 -0700 From: "Douglas Otis" To: , Subject: RE: iSCSI Naming and Discovery Date: Wed, 4 Oct 2000 11:17:41 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: <0F31E5C394DAD311B60C00E029101A0704101059@corpmx9.isus.emc.com> Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk David, > All of this is with my WG co-chair hat off: > > At a high level, my concern about DNS and LDAP is based > on Leslie Lamport's infamous (in jest) definition of > a distributed system -- one on which he can't get > work done because a computer he's never heard of is > down :-). While the sort of infrastructure services > that run on "a computer he's never heard of" have been > getting more and more reliable, the moral I take away > from this is to be cautious about introducing > dependencies on services. Every introduced dependency > provides new ways for things to fail, and this needs > to be balanced against the new ways in which things > work because of the dependency. IMHO, NATs and IPv4/v6 > transparency don't tip the balance far enough to favor > DNS or LDAP (but I always reserve the right to change > my opinion :-) ). If your system is fully dependent upon the network to even boot, then your system is using either a BOOTP/DHCP server. It is often the same system that provides IP leases also handles the NIS affairs via LDAP. You would want a database if you wished to scale a to a large enterprise. It is also common to have more than two such servers visible to this booting machine together with assists from the routers for this purpose. If you can not trust the network to assign your network settings and to provide you with the requisite services, then you should install a local drive so that you can continue to work in the dark so to speak. > For current block storage, a host's access to its storage > either does not depend on any external service (parallel > SCSI, FC-AL) or a depends on the Fibre Channel fabric > nameserver (FC-SW). The fabric nameserver is usually > embedded in the Fibre Channel switches, so that if the > nameserver's not working, the switch probably isn't working > either (e.g., in practice, if the nameserver isn't responding > to the HBAs, the switch gets replaced). > > In contrast, both DNS and LDAP are external services, probably > located on general purpose servers, on which block storage > access currently does not depend. LDAP in particular can be > rather complex, as some of its leading implementations sit > on top of full-fledged relational databases. FWIW, the > original design of the FC fabric nameserver was LDAP-based, > but this was discovered to be infeasible -- an LDAP server > doesn't embed into a switch very well, or at least didn't > at the time. The characterization of LDAP as "very simple" > is significantly off the mark, IMHO. LDAP is continuing to improve and I doubt that starting from scratch with an alternate scheme will improve upon the present features offered. At least you will not need to develop additional tools. > The discussions of DNS and LDAP on this list have had the > flavor of these being ubiquitous completely reliable and > available services on which dependencies are no big deal. I have never touted DNS. If fact, perhaps we should forbid the use of DNS for any activity associated with SCSI. At least DNS does not become a point of weakness in the system then. > When everything is working right, that may be the case, but > everything doesn't always work right; disasters happen > taking down entire facilities and requiring them to be > brought up from scratch. An example of what could go > wrong is that two DNS servers could be configured in a > way that each depends on the other for its storage. If > each server is rebooted individually to test the > configuration, things work, but if both go down, neither > will come back up. This is one more problem for overworked > system and network administrators - having to determine > which servers must use IP names for storage vs. DNS names, > with nasty consequences of getting it wrong. This isn't a > fatal problem, but every time management becomes more > complicated, it's one more barrier to adoption. This is > why I don't put much weight on the argument that IP URLs > can be used where DNS or LDAP won't work - there are too > many opportunities to screw this up if DNS is allowed in > general. These few IT managers are handling hundreds and perhaps thousands of systems. Do not give them damaged goods when it comes to tools. You will short change them by providing only the tools you understand. Maintaining a server or router can bring down an entire company and on occasion it happens. Most often this happens with system patches being applied incorrectly following a logon. The OS is damage and is no longer accessible to the IT manager via the network. They then run around like mice from cubical to cubical restoring the system. At least with a network accessible drive, they can make repairs from their workstation. Most of these fellows can bring up any system or server in minutes if you provide them the tools. The tools that let them leverage their abilities. And yes, that might mean a leveraged mistake at times. > Boot is a specific concern, because if DNS or LDAP is > involved in locating the boot volume, then the HBA card > BIOS has to be able to access DNS or LDAP. There's nothing > inherently difficult or infeasible about this, it's just > more code in yet another place ... making it harder to > deploy/adopt this. Not at all. You make it sound difficult. It is not. Again, the servers at play would be DHCP and LDAP. DNS would be an option. Most systems within an enterprise environment will not have access to the network without the DHCP, LDAP or a proprietary MS scheme running. You would not even see your desktop unless the IT manager permitted a local account. You will then find most of your files missing, even the ones on the local drive, until these servers are back up. Don't kid yourself about not needing other servers. As I said, most places use two of them for the reason indicated. > Beyond this, I need to toss in a few important scattered comments. > > Most systems don't boot via BOOTP today; demanding that everything > boot via BOOTP in order to avoid problems created by an entirely too > clever naming structure sounds like a bad tradeoff. This will lead > to iSCSI repeating the Fibre Channel experience of boot not working > for a long period of time after deployment of the technology. > Customer reactions to that Fibre Channel experience have been > negative to the point of just barely printable in some instances. BOOTP and DHCP should be thought of as the same service with different options. Most in fact use DHCP. The newer and better BOOTP. If we standardized the class objects for SCSI under LDAP, there would not be any confusion. By pretending we can do better than LDAP, we will never finish this job. > The LDAP vs. nameserver discussion misses the point. The Fibre > Channel experience makes it clear that iSCSI is going to need > some sort of central repository of configuration information to > get the sort of discovery functionality that is expected. That > repository is at least a nameserver, and possibly more - the > approach of defining LDAP schemas is the result of deciding > to use LDAP as the communication protocol for accessing > iSCSI nameservers. I do not like the idea of inventing a new name server. Let Le Gato or Veritas send their configurations to LDAP or let the IT managers handle it using their own set of tools. Why make yet another server that can fail? You worry about reliability and yet you add another server that must be invented and therefore likely to fail for years to come. > The [3] discussion of NATs at the end of my previous message was > rather verbose. The bottom line is that if IP addresses are stored > in and distributed by a central iSCSI config repository, then NATs > cause a problem, and a brute force solution that is likely to work > in many cases is not to put NATs between the repository and its users. > Alternate solutions based on application level gateways and storing > addresses with respect to users of the repository rather than the > repository server are workable, but more complex. We do not need to re-invent this. LDAP provides a solution for communication. > On [4], I see the following advantages of IP addresses over DNS URLs. > - IP addresses are much better for booting, and in some cases have > to be used (see above). DNS hostnames are then an additional > mechanism that adds implementation work and complicates > management (see above). Agreed. DNS is not needed. > - Scanning a range of possible locations of Targets has some attractive > properties, because it removes the obligation of a new target > to proactively make itself known to some iSCSI configuration > repository (such mechanisms do work, Fibre Channel is an example, > but the FC solution doesn't directly apply to iSCSI). It seems > to me that wildcarding an IP address via a mask or range works > better than wildcarding some portion of a URL (e.g., if the > wildcard is xyz*.abc.com, there's a fair amount of work involved > in finding xyzf, xyz01, and xyz17a in the abc.com domain, > by comparison to finding things that match 192.48.27.128-191 or > equivalently 192.48.27.128/27). Nonsense. If you use just DHCP, which I will bet you dollars to donuts your machine understands and likely uses, these setting can be made at that time whether there is a local drive or not. You do not need anything more complex. What will do wildcarding you suggest? Why? You would need to map to a boot drive that supported your architecture for one, you would need to know what OS you wanted to run, second. Nothing that scans through a default list of drives will give you the results you desire. These server are in place and they work. You are making this much to difficult. Even a simple flat file could replace an LDAP server if desired for the simple case. The simple case is easy to handle. I was concerned about the enterprise solution. > One more note on NATs: > > > FTP must use the passive mode to get past the NAT. > > That's not correct. There are NATs deployed that detect FTP > traffic and rewrite the IP addresses in the FTP payloads > to make everything work right. The official name for > this feature/kludge is an Application Level Gateway (ALG), > and the FTP features that require it are generally regarded > as an example not to be emulated. If you don't have the kludged NAT, you can set the passive flag for FTP. No big deal. Doug > > --David > > --------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 42 South St., Hopkinton, MA 01748 > +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 > black_david@emc.com Mobile: +1 (978) 394-7754 > --------------------------------------------------- > From owner-ips@ECE.cmu.edu Wed Oct 04 19:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26029; 4 Oct 2000 15:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA07904 for ips-outgoing; Wed, 4 Oct 2000 14:04:12 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA07897 for ; Wed, 4 Oct 2000 14:04:06 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id NAA88762 for ; Wed, 4 Oct 2000 13:51:19 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e94I41G55732 for ; Wed, 4 Oct 2000 12:04:01 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: URL scheme To: X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 4 Oct 2000 11:02:20 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/04/2000 12:04:01 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Douglas Otis, Perhaps you miss addressed your note, I do not believe that I either stated what you claim or even believe what you say I believe. Please recheck your source. What I wanted was for folks to start from the basics and tell us step by step what they think they need in the way of a name, why, how they would get it, and how they would use it (including how it would react to current network equipment and current Storage Controllers --which may have an iSCSI interface. I have strong doubts that some of the ideas I have been hearing are needed or reasonable. . . . John L. Hufferd "Douglas Otis" on 10/04/2000 09:00:31 AM To: John Hufferd/San Jose/IBM@IBMUS, cc: Subject: RE: iSCSI: URL scheme John, The problem that I see with your present scheme that is causing the confusion and difficulty is from trying to create an amalgam of IP and SCSI addresses. Although you may view URLs as extensible and perhaps capable of handling both IP and SCSI, the load created will be horrendous. In judo, they teach you how use inertia in your favor. In this case, you are attempting to change the course of the entire world using a new symbolic router. I am not as clever as you. I only wish to translate firmly establish binary fields that address devices. Yes, I can translate these and perhaps even bridge some of them. I can not parse every packet looking for a symbolic address and then route this packet dynamically in both SCSI and IP domains with your new breed of symbolic router, name and authentication server. On top of that, you wish to see everything extracted from this single connection for authentication and configuration. We end up doing a great deal of looking at these packets and then a great deal of deciding what to do with them. None of this is productive work. You develop a class 2 control for a class 3 world for additional trouble. I would hope that you want a simple solution and not one sure to thwart all efforts. Your truly, Doug From owner-ips@ECE.cmu.edu Wed Oct 04 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26092; 4 Oct 2000 15:46 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA10638 for ips-outgoing; Wed, 4 Oct 2000 15:04:30 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from server1.NishanSystems.COM (smtp.nishansystems.com [216.217.36.162]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA10277 for ; Wed, 4 Oct 2000 14:56:13 -0400 (EDT) Received: by IS~SERVER1 with Internet Mail Service (5.5.2448.0) id ; Wed, 4 Oct 2000 11:58:23 -0700 Message-ID: From: Joshua Tseng/Nishan Systems To: Black_David@emc.com, ips@ece.cmu.edu Subject: RE: iSCSI Naming and Discovery Date: Wed, 4 Oct 2000 11:58:16 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-ips@ece.cmu.edu Precedence: bulk David, >down :-). While the sort of infrastructure services >that run on "a computer he's never heard of" have been >getting more and more reliable, the moral I take away >from this is to be cautious about introducing >dependencies on services. Every introduced dependency >provides new ways for things to fail, and this needs >to be balanced against the new ways in which things >work because of the dependency. IMHO, NATs and IPv4/v6 >transparency don't tip the balance far enough to favor >DNS or LDAP (but I always reserve the right to change >my opinion :-) ). If any dependence on any external service is what you're trying to avoid, I think that's unrealistic. If iSCSI is to be deployed through the Public Internet, then at the very least you will have firewall services, not to mention possibly a PKI infrastructure to provide authentication & encryption services. DNS, with all its warts, is pretty reliable, and if it fails, then EVERYONE knows it failed. Storage won't necessarily be the only thing in trouble. >On [4], I see the following advantages of IP addresses over DNS URLs. >- IP addresses are much better for booting, and in some cases have > to be used (see above). DNS hostnames are then an additional > mechanism that adds implementation work and complicates > management (see above). >- Scanning a range of possible locations of Targets has some attractive > properties, because it removes the obligation of a new target > to proactively make itself known to some iSCSI configuration > repository (such mechanisms do work, Fibre Channel is an example, > but the FC solution doesn't directly apply to iSCSI). It seems > to me that wildcarding an IP address via a mask or range works > better than wildcarding some portion of a URL (e.g., if the > wildcard is xyz*.abc.com, there's a fair amount of work involved > in finding xyzf, xyz01, and xyz17a in the abc.com domain, > by comparison to finding things that match 192.48.27.128-191 or > equivalently 192.48.27.128/27). I see a few issues with this idea which I can't begin to understand. Will DHCP support iSCSI, and if so will that cause IP addresses to change? What if addresses are not assigned according to a neatly defined range? How many retries to each IP address before the initiator decides no target exists at that address? Will this lead to a school of homeless IP packets spinning around the network until TTL expires? Josh -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Wednesday, October 04, 2000 9:29 AM To: ips@ece.cmu.edu Subject: RE: iSCSI Naming and Discovery All of this is with my WG co-chair hat off: At a high level, my concern about DNS and LDAP is based on Leslie Lamport's infamous (in jest) definition of a distributed system -- one on which he can't get work done because a computer he's never heard of is down :-). While the sort of infrastructure services that run on "a computer he's never heard of" have been getting more and more reliable, the moral I take away from this is to be cautious about introducing dependencies on services. Every introduced dependency provides new ways for things to fail, and this needs to be balanced against the new ways in which things work because of the dependency. IMHO, NATs and IPv4/v6 transparency don't tip the balance far enough to favor DNS or LDAP (but I always reserve the right to change my opinion :-) ). For current block storage, a host's access to its storage either does not depend on any external service (parallel SCSI, FC-AL) or a depends on the Fibre Channel fabric nameserver (FC-SW). The fabric nameserver is usually embedded in the Fibre Channel switches, so that if the nameserver's not working, the switch probably isn't working either (e.g., in practice, if the nameserver isn't responding to the HBAs, the switch gets replaced). In contrast, both DNS and LDAP are external services, probably located on general purpose servers, on which block storage access currently does not depend. LDAP in particular can be rather complex, as some of its leading implementations sit on top of full-fledged relational databases. FWIW, the original design of the FC fabric nameserver was LDAP-based, but this was discovered to be infeasible -- an LDAP server doesn't embed into a switch very well, or at least didn't at the time. The characterization of LDAP as "very simple" is significantly off the mark, IMHO. The discussions of DNS and LDAP on this list have had the flavor of these being ubiquitous completely reliable and available services on which dependencies are no big deal. When everything is working right, that may be the case, but everything doesn't always work right; disasters happen taking down entire facilities and requiring them to be brought up from scratch. An example of what could go wrong is that two DNS servers could be configured in a way that each depends on the other for its storage. If each server is rebooted individually to test the configuration, things work, but if both go down, neither will come back up. This is one more problem for overworked system and network administrators - having to determine which servers must use IP names for storage vs. DNS names, with nasty consequences of getting it wrong. This isn't a fatal problem, but every time management becomes more complicated, it's one more barrier to adoption. This is why I don't put much weight on the argument that IP URLs can be used where DNS or LDAP won't work - there are too many opportunities to screw this up if DNS is allowed in general. Boot is a specific concern, because if DNS or LDAP is involved in locating the boot volume, then the HBA card BIOS has to be able to access DNS or LDAP. There's nothing inherently difficult or infeasible about this, it's just more code in yet another place ... making it harder to deploy/adopt this. Beyond this, I need to toss in a few important scattered comments. Most systems don't boot via BOOTP today; demanding that everything boot via BOOTP in order to avoid problems created by an entirely too clever naming structure sounds like a bad tradeoff. This will lead to iSCSI repeating the Fibre Channel experience of boot not working for a long period of time after deployment of the technology. Customer reactions to that Fibre Channel experience have been negative to the point of just barely printable in some instances. The LDAP vs. nameserver discussion misses the point. The Fibre Channel experience makes it clear that iSCSI is going to need some sort of central repository of configuration information to get the sort of discovery functionality that is expected. That repository is at least a nameserver, and possibly more - the approach of defining LDAP schemas is the result of deciding to use LDAP as the communication protocol for accessing iSCSI nameservers. The [3] discussion of NATs at the end of my previous message was rather verbose. The bottom line is that if IP addresses are stored in and distributed by a central iSCSI config repository, then NATs cause a problem, and a brute force solution that is likely to work in many cases is not to put NATs between the repository and its users. Alternate solutions based on application level gateways and storing addresses with respect to users of the repository rather than the repository server are workable, but more complex. On [4], I see the following advantages of IP addresses over DNS URLs. - IP addresses are much better for booting, and in some cases have to be used (see above). DNS hostnames are then an additional mechanism that adds implementation work and complicates management (see above). - Scanning a range of possible locations of Targets has some attractive properties, because it removes the obligation of a new target to proactively make itself known to some iSCSI configuration repository (such mechanisms do work, Fibre Channel is an example, but the FC solution doesn't directly apply to iSCSI). It seems to me that wildcarding an IP address via a mask or range works better than wildcarding some portion of a URL (e.g., if the wildcard is xyz*.abc.com, there's a fair amount of work involved in finding xyzf, xyz01, and xyz17a in the abc.com domain, by comparison to finding things that match 192.48.27.128-191 or equivalently 192.48.27.128/27). One more note on NATs: > FTP must use the passive mode to get past the NAT. That's not correct. There are NATs deployed that detect FTP traffic and rewrite the IP addresses in the FTP payloads to make everything work right. The official name for this feature/kludge is an Application Level Gateway (ALG), and the FTP features that require it are generally regarded as an example not to be emulated. --David --------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 42 South St., Hopkinton, MA 01748 +1 (508) 435-1000 x75140 FAX: +1 (508) 497-8500 black_david@emc.com Mobile: +1 (978) 394-7754 --------------------------------------------------- From owner-ips@ECE.cmu.edu Wed Oct 04 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26097; 4 Oct 2000 15:48 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id OAA10156 for ips-outgoing; Wed, 4 Oct 2000 14:53:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id OAA10149 for ; Wed, 4 Oct 2000 14:53:45 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id LAA16603; Wed, 4 Oct 2000 11:39:24 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id LAA25883; Wed, 4 Oct 2000 11:37:12 -0700 From: "Douglas Otis" To: , Cc: Subject: RE: iSCSI Naming and Discovery Date: Wed, 4 Oct 2000 11:51:59 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk Costa, You are inventing yet another server. You already have DHCP that handles such things. You broadcast the request, the routers help, and you get the answer. I doubt you will see such assistance for any new server that does only some new protocol. We should work at reducing the number of servers that will need to be present within an enterprise. DHCP is already present and can handle this function. Stop there. Your done. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > csapuntz@cisco.com > Sent: Wednesday, October 04, 2000 10:03 AM > To: Black_David@emc.com > Cc: ips@ece.cmu.edu; csapuntz@cisco.com > Subject: Re: iSCSI Naming and Discovery > > > > > - Scanning a range of possible locations of Targets has some attractive > > properties, because it removes the obligation of a new target > > to proactively make itself known to some iSCSI configuration > > repository (such mechanisms do work, Fibre Channel is an example, > > but the FC solution doesn't directly apply to iSCSI). It seems > > to me that wildcarding an IP address via a mask or range works > > better than wildcarding some portion of a URL (e.g., if the > > wildcard is xyz*.abc.com, there's a fair amount of work involved > > in finding xyzf, xyz01, and xyz17a in the abc.com domain, > > by comparison to finding things that match 192.48.27.128-191 or > > equivalently 192.48.27.128/27). > > David, > > I do not believe that wildcarding is compelling. > > Scanning will be too slow. How many packets do you send to an IP > address until you decide that it's not up? After how many seconds to > you give up? > > If you want to discover who on your subnet speaks iSCSI, it'd be much > better to use a subnet-local multicast (a multicast message that says: > who speaks iSCSI?). If you're going across subnets, then it's probably > time to use a directory service. > > -Costa > From owner-ips@ECE.cmu.edu Wed Oct 04 19:50:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26071; 4 Oct 2000 15:40 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA10628 for ips-outgoing; Wed, 4 Oct 2000 15:04:18 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from main.connectcom.net (anubis.advansys.com [204.247.22.2]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA10613 for ; Wed, 4 Oct 2000 15:04:09 -0400 (EDT) Received: from yp_portable (anubis.advansys.com [204.247.22.2]) by main.connectcom.net with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id T9NCD094; Wed, 4 Oct 2000 12:02:02 -0700 From: "Y P Cheng" To: "Nelson Nahum" , "'John Muth'" , Subject: RE: iSCSI: problem with LUN discovery Date: Wed, 4 Oct 2000 12:01:40 -0700 Message-ID: <001001c02e35$8734c220$65c809c0@yp_portable.advansys.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0 In-Reply-To: Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-ips@ece.cmu.edu Precedence: bulk > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > Nelson Nahum > > The problem is that the signature written by the VxVM is not standard and > only is recognized by the Veritas VxVM. > Due to the fact that the important thing is to identify the media not the > controller (as Veritas does), the best way is to have the serial number > written in the media in a non accessible sector, and loaded by the > controller to one of the Inquiry pages of the LU. In this manner no matter > which channel, WWN, LUN the specific LUN is presented can be > recognized by every initiator. Every 1394 and Fibre Channel node, including disk drives, has a GUID, Global Unique ID. They are saved in EEPROM by manufacturers. I believe T10 also has a mechanism to report a unique name of a device using mode select. (Page 83h? Someone help!) iSCSI should make sure that each device has a GUID available by mode select for the initiators. No need to reinvent another mechanism. Y.P. Cheng, CTO, ConnectCom Solutions Corp. From owner-ips@ECE.cmu.edu Wed Oct 04 20:30:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26245; 4 Oct 2000 16:26 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA11323 for ips-outgoing; Wed, 4 Oct 2000 15:19:29 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from mx.sanlight.net (adsl-63-202-160-75.dsl.snfc21.pacbell.net [63.202.160.75]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA11310 for ; Wed, 4 Oct 2000 15:19:22 -0400 (EDT) Received: from mail.sanlight.net (adsl-63-202-160-76.dsl.snfc21.pacbell.net [63.202.160.76]) by mx.sanlight.net (8.8.8/8.8.8) with ESMTP id MAA16620; Wed, 4 Oct 2000 12:05:07 -0700 (PDT) Received: from littlejoy (ljoy.sanlight.net [10.0.0.18]) by mail.sanlight.net (8.9.3/8.9.3) with SMTP id MAA25959; Wed, 4 Oct 2000 12:02:56 -0700 From: "Douglas Otis" To: "John Hufferd/San Jose/IBM" , Subject: RE: iSCSI: URL scheme Date: Wed, 4 Oct 2000 12:17:42 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 In-Reply-To: Importance: Normal Sender: owner-ips@ece.cmu.edu Precedence: bulk John, Sorry, but it would seem you are blind to the problem. I was trying very hard to make a point. You can not use symbolic names within a transport. You should exclude these symbols and configuration from the transport. If you wish to define names, make it a class object within an LDAP server. As you again fail to address the basic concerns, I still wonder. Doug > -----Original Message----- > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of > John Hufferd/San Jose/IBM > Sent: Wednesday, October 04, 2000 11:02 AM > To: ips@ece.cmu.edu > Subject: RE: iSCSI: URL scheme > > > > Douglas Otis, > Perhaps you miss addressed your note, I do not believe that I > either stated > what you claim or even believe what you say I believe. Please > recheck your > source. > > What I wanted was for folks to start from the basics and tell us step by > step what they think they need in the way of a name, why, how they would > get it, and how they would use it (including how it would react to current > network equipment and current Storage Controllers --which may have an > iSCSI interface. I have strong doubts that some of the ideas I have been > hearing are needed or reasonable. > . > . > . > John L. Hufferd > > > > "Douglas Otis" on 10/04/2000 09:00:31 AM > > To: John Hufferd/San Jose/IBM@IBMUS, > cc: > Subject: RE: iSCSI: URL scheme > > > > John, > > The problem that I see with your present scheme that is causing the > confusion and difficulty is from trying to create an amalgam of > IP and SCSI > addresses. Although you may view URLs as extensible and perhaps > capable of > handling both IP and SCSI, the load created will be horrendous. In judo, > they teach you how use inertia in your favor. In this case, you are > attempting to change the course of the entire world using a new symbolic > router. I am not as clever as you. I only wish to translate firmly > establish binary fields that address devices. Yes, I can translate these > and perhaps even bridge some of them. I can not parse every > packet looking > for a symbolic address and then route this packet dynamically in both SCSI > and IP domains with your new breed of symbolic router, name and > authentication server. On top of that, you wish to see everything > extracted > from this single connection for authentication and configuration. We end > up > doing a great deal of looking at these packets and then a great deal of > deciding what to do with them. None of this is productive work. You > develop a class 2 control for a class 3 world for additional trouble. I > would hope that you want a simple solution and not one sure to thwart all > efforts. > > Your truly, > Doug > > > > > > > From owner-ips@ECE.cmu.edu Wed Oct 04 20:40:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26262; 4 Oct 2000 16:30 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA12061 for ips-outgoing; Wed, 4 Oct 2000 15:38:40 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA12054 for ; Wed, 4 Oct 2000 15:38:31 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id PAA42350 for ; Wed, 4 Oct 2000 15:25:44 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e94JcQG28234 for ; Wed, 4 Oct 2000 13:38:26 -0600 X-Priority: 1 (High) Importance: Normal Subject: RE: iSCSI: Flow Control To: IPS@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 4 Oct 2000 12:34:30 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/04/2000 01:38:26 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk Michael Krause, First, a session usually lasts from Boot Time till it is Booted again or stopped. The variant to this is a Storage Device being used in a manor like a Mount or Map done with NAS. Even then, most folks, have that setup so the Mounting and Mapping is done at bring up, though it is sometimes done at other times, but even then it is left around. So I think the thing you can say about Session Time as the term is used in the iSCSI context is that it is LONG. I do not think we have consensus about the notification of available buffers. With the way many systems work, is (as stated above), all devices are set up at startup of the Host, and the Session is kept around by the Host, even if there hasn't been anything which use that device all day/week, etc. So I am not sure if having a certain amount of buffer space reserved for each Host (which could be 100s-1000s) would be an especially good idea. . . . John L. Hufferd Michael Krause @ece.cmu.edu on 10/04/2000 06:53:52 AM Sent by: owner-ips@ece.cmu.edu To: IPS@ece.cmu.edu cc: Subject: RE: iSCSI: Flow Control At 07:00 PM 10/3/00 -0700, Jim McGrath wrote: >The mapping may be a bit trickier. If the host side manages resources (like >buffers) on a per connection basis, then that should be the level. The >other dimension is time. Storage traffic is bur sty, so the time span for >resetting the resource levels should be within the scope of a "burst time" >or "burst cycle". This is probably seconds, or perhaps minutes, but not >hours. How long would we expect a session to last? (The problem with >previous efforts was it was tied into device/host login, which could last >for days). A session's duration is a function of the operation being performed so I don't think there is any way to quantify to a "best" value either for a minimum or a maximum (don't really want to track this within an implementation other than what is the current resource). Burst times are also a function of the type of operation being supported - some data base queries or large data set (e.g. technical applications) processing can be very storage intensive for many hours at a time. As such, the login mechanism is a nice initialization point but the dynamic mechanism needs to be present to deal with "real-life" operations which are not steady state activities. I don't see people objecting to this requirement so is this something that can be viewed as consensus? Mike From owner-ips@ECE.cmu.edu Wed Oct 04 21:10:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26419; 4 Oct 2000 17:09 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id QAA13698 for ips-outgoing; Wed, 4 Oct 2000 16:19:50 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from e34.esmtp.ibm.com (e34.co.us.ibm.com [32.97.110.132]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id QAA13683 for ; Wed, 4 Oct 2000 16:19:41 -0400 (EDT) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.99.132.205]) by e34.esmtp.ibm.com (8.9.3/8.9.3) with ESMTP id QAA74844 for ; Wed, 4 Oct 2000 16:06:52 -0400 Received: from f5n70e (d03nm094h.boulder.ibm.com [9.99.140.70]) by westrelay02.boulder.ibm.com (8.11.0m3/NCO v4.93) with ESMTP id e94KJYG129794 for ; Wed, 4 Oct 2000 14:19:34 -0600 X-Priority: 1 (High) Importance: Normal Subject: Re: iSCSI: problem with LUN discovery To: ips@ece.cmu.edu X-Mailer: Lotus Notes Release 5.0.3 (Intl) 21 March 2000 Message-ID: From: "John Hufferd/San Jose/IBM" Date: Wed, 4 Oct 2000 13:14:39 -0700 X-MIMETrack: Serialize by Router on D03NM094/03/M/IBM(Release 5.0.3 (Intl)|21 March 2000) at 10/04/2000 02:19:34 PM MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: owner-ips@ece.cmu.edu Precedence: bulk John Muth, I agree with your main point that "... initiator specific LUN mapping presents no problem". But I think a comment should be made here that a "Disk Serial Number" may not be the right thing when the Storage Controller virtualities the Disk Storage. But to the extent that they present a Virtual Disk Serial Number (I think that is what is in the VPD at page 83h) you are correct. Now at some point I think folks will want a Central Storage Manager to instruct the Storage Controllers (IBM Sharks, or EMC Symmetrix, etc.) how to Virtualize the Disks, (that software may come from Veritas, Tivoli, HP, Compaq, etc.) and that might even include something like "call this virtual Disk xyz". But until that, or something else is done, we will still have the Storage Controller performing the Virtualizations and naming the virtual volumes (LUs), and even then placing the "name" in the VPD page 83h. So even then, your point about "... initiator specific LUN mapping presents no problem" will still remain correct. . . . John L. Hufferd John Muth @ece.cmu.edu on 10/04/2000 10:39:33 AM Sent by: owner-ips@ece.cmu.edu To: cc: ips@ece.cmu.edu Subject: Re: iSCSI: problem with LUN discovery Nelson, I don't disagree, disk serial numbers are a good thing. I was simply trying to point out (perhaps badly) that initiator specific LUN mapping presents no problem to the initiator side since nobody in their right mind uses the path to determine the identity of a disk. Now, if we could only convince all of the disk manufacturers in the world (both real and virtual) to give us unique serial numbers (we've seen drive models were every unit returned the same constant serial number. and don't get me started about arrays....) John Muth VERITAS Software Nelson Nahum wrote: > > John, > > The problem is that the signature written by the VxVM is not standard and > only is recognized by the Veritas VxVM. > Due to the fact that the important thing is to identify the media not the > controller (as Veritas does), the best way is to have the serial number > written in the media in a non accessible sector, and loaded by the > controller to one of the Inquiry pages of the LU. In this manner no matter > which channel, WWN, LUN the specific LUN is presented can be recognized by > every initiator. > > Nelson Nahum > CTO > StoreAge Networking Technologies. > > -----Original Message----- > From: John Muth [mailto:muth@veritas.com] > Sent: Wednesday, October 04, 2000 4:51 PM > To: ips@ece.cmu.edu > Subject: Re: iSCSI: problem with LUN discovery > > Julo, > > FYI. Assuming the initiators are running higher level disk > management S/W like VxVM (VERITAS Volume Manager), this won't > be a problem. VxVM already deals with device paths moving > around by doing it's disk identification based on the label, > not the device path. > > John Muth > VERITAS Software > > julian_satran@il.ibm.com wrote: > > > > JP, > > > > Unless I misunderstood you this is entirely a T10 issue. > > Presenting a different LU map to each initiator as identified by the port > > or the ACCESS-ID is intentional (each initiator can view a different set > of > > LUs)and part > > of the protection mechanisms of SCSI. > > > > With iSCSI at later stages (when we start handling discovery and > > management) > > the pain of getting to a specific volume might be eased. > > > > Julo > > > > Raghavendra Rao on 27/09/2000 23:05:22 > > > > Please respond to Raghavendra Rao > > > > To: Julian Satran/Haifa/IBM@IBMIL > > cc: > > Subject: iSCSI: problem with LUN discovery > > > > Julian, > > > > Do you agree that following is a problem ? If so, can we fix it ? > > > > SAM doesn't prohibit a target device from presenting initiator specific > > values for LUs. So it is believed that similar values for a LU may not > > necessarily address the same LU inside a target when accessed from > > different > > initiators. > > > > So the vendor ID page 83 of INQUIRY command is considered to be unique > > for a LU - There is one caveat to this though, that the page 83 values > > could optionally be target port specific. Anyway, if the hosts were to > > use a more persistent values such as page-83 identifier, say after > reboot > > or after reconfiguration, it first needs to translate this page-83 > > identifier > > into a LUN. This can be done by first issuing a REPORT_LUN command to > > LUN 0 and issuing an INQUIRY page-83 command to each LU in the list > > obtained from REPORT_LUN response until a successful match is found. > > An initiator could cache the mapping for future speedier lookups, but > > it still has to linearly probe out all LUs to get the page-83 unique > > identifiers. > > > > I hate to do this static probing; This problem can be fixed by either > > > > a) mandating that LUN values are persistent and same for all > > initiators (more unlikely to be accepted) > > > > OR > > > > b) introducing a new page code for INQUIRY in which the initiator > > will pass down the page-83 identifier in the parameter list > > to LUN 0 and the target will respond with the LUN corresponding > > to this identifier; Of course, this has to go to T10 for approval. > > > > Thanks. > > -JP From owner-ips@ECE.cmu.edu Wed Oct 04 21:20:00 2000 Return-Path: Received: from ECE.CMU.EDU by cnoc.pdl.cs.cmu.edu id aa26439; 4 Oct 2000 17:11 EDT Received: by ece.cmu.edu (8.9.2/8.8.8) id PAA12835 for ips-outgoing; Wed, 4 Oct 2000 15:58:17 -0400 (EDT) X-Authentication-Warning: ece.cmu.edu: majordom set sender to owner-ips@ece.cmu.edu using -f Received: from storeage_nt.store-age.com ([199.203.178.211]) by ece.cmu.edu (8.9.2/8.8.8) with ESMTP id PAA12831 for ; Wed, 4 Oct 2000 15:58:12 -0400 (EDT) Received: from e4u5e0 (ras4-p114.hfa.netvision.net.il [62.0.148.114]) by storeage_nt.store-age.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id TTT33XGD; Wed, 4 Oct 2000 22:55:45 +0200 Message-ID: <007301c02e45$3b7ad260$7294003e@e4u5e0> From: "Nelson Nahum" To: "Y P Cheng" , "'John Muth'" , References: <001001c02e35$8734c220$65c809c0@yp_portable.advansys.com> Subject: Re: iSCSI: problem with LUN discovery Date: Wed, 4 Oct 2000 22:54:02 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2615.200 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200 Sender: owner-ips@ece.cmu.edu Precedence: bulk The question is if a storage controller keeps a different GUID for every LUN created. If not we are in the same problem as WWN, LUN naming. The question is how to identify a LUN in a storage controller that can create and delete thousands of LUNs, and that the same WWN, LUN combination can address different volumes depend on the intiator. Also to keep this identifier in the EEPROM of the controller is not a good idea, as cause problems when the controller is replaced. I think that the best way is to identify a LUN by a string composed by a (or S/N) and